Tackling Shortcut Learning in Deep Neural Networks: An Iterative
  Approach with Interpretable Models
We use concept-based interpretable models to mitigate shortcut learning.
Existing methods lack interpretability. Beginning with a Blackbox, we
iteratively carve out a mixture of interpretable experts (MoIE) and a residual
network. Each expert explains a subset of data using First Order Logic (FOL).
While explaining a sample, the FOL from biased BB-derived MoIE detects the
shortcut effectively. Finetuning the BB with Metadata Normalization (MDN)
eliminates the shortcut. The FOLs from the finetuned-BB-derived MoIE verify the
elimination of the shortcut. Our experiments show that MoIE does not hurt the
accuracy of the original BB and eliminates shortcuts effectively.
Shantanu Ghosh, Ke Yu, Forough Arabshahi, Kayhan Batmanghelich
# Dividing and Conquering a BlackBox to a Mixture of Interpretable Models: Route, Interpret, Repeat

###### Abstract

ML model design either starts with an interpretable model or a Blackbox and explains it post hoc. Blackbox models are flexible but difficult to explain, while interpretable models are inherently explainable. Yet, interpretable models require extensive ML knowledge and tend to be less flexible and underperforming than their Blackbox variants. This paper aims to blur the distinction between a post hoc explanation of a Blackbox and constructing interpretable models. Beginning with a Blackbox, we iteratively _carve out_ a mixture of interpretable experts (MoIE) and a _residual network_. Each interpretable model specializes in a subset of samples and explains them using First Order Logic (FOL), providing basic reasoning on concepts from the Blackbox. We route the remaining samples through a flexible residual. We repeat the method on the residual network until all the interpretable models explain the desired proportion of data. Our extensive experiments show that our _route, interpret, and repeat_ approach (1) identifies a diverse set of instance-specific concepts with high concept completeness via MoIE without compromising in performance, (2) identifies the relatively "harder" samples to explain via residuals, (3) outperforms the interpretable by-design models by significant margins during test-time interventions, and (4) fixes the shortcut learned by the original Blackbox. The code for MoIE is publicly available at: [https://github.com/batmanlab/ICML-2023-Route-interpret-repeat](https://github.com/batmanlab/ICML-2023-Route-interpret-repeat).

## 1 Introduction

Model explainability is essential in high-stakes applications of AI, _e.g.,_ healthcare. While Blackbox models (_e.g.,_ Deep Learning) offer flexibility and modular design, post hoc explanation is prone to confirmation bias (Wan et al., 2022), lack of fidelity to the original model (Adebayo et al., 2018), and insufficient mechanistic explanation of the decision-making process (Rudin, 2019). Interpretable-by-design models overcome those issues but tend to be less flexible than Blackbox models and demand substantial expertise to design. Using a post hoc explanation or adopting an inherently interpretable model is a mutually exclusive decision to be made at the initial phase of AI model design. This paper blurs the line on that dichotomous model design.

The literature on post hoc explanations is extensive. This includes model attributions ((Simonyan et al., 2013; Selvaraju et al., 2017)), counterfactual approaches (Abid et al., 2021; Singla et al., 2019), and distillation methods (Alharbi et al., 2021; Cheng et al., 2020). Those methods either identify key input features that contribute the most to the network's output (Shrikumar et al., 2016), generate input perturbation to flip the network's output (Samek et al., 2016; Montavon et al., 2018), or estimate simpler functions to approximate the network output locally. Post hoc methods preserve the flexibility and performance of the Blackbox but suffer from a lack of fidelity and mechanistic explanation of the network output (Rudin, 2019). Without a mechanistic explanation, recourse to a model's undesirable behavior is unclear. Interpretable models are alternative designs to the Blackbox without many such drawbacks. For example, modern interpretable methods highlight human understandable _concepts_ that contribute to the downstream prediction.

Several families of interpretable models exist for a long time, such as the rule-based approach and generalized additive models (Hastie and Tibshirani, 1987; Letham et al., 2015; Breiman et al., 1984). They primarily focus on tabular data. Such models for high-dimensional data (_e.g.,_ images) primarily rely on projecting to a lower dimensional human understandable _concept_ or _symbolic_ space (Koh et al., 2020) and predicting the output with an interpretable classifier. Despite their utility, the current State-Of-The-Art (SOTA) are limited in design; for example, they do not model the
interaction between the concepts except for a few exceptions (Ciravegna et al., 2021; Barbiero et al., 2022), offering limited reasoning capabilities and robustness. Furthermore, if a portion of the samples does not fit the template design of the interpretable model, they do not offer any flexibility, compromising performance.

**Our contributions** We propose an interpretable method, aiming to achieve the best of both worlds: not sacrificing Blackbox performance similar to post hoc explainability while still providing actionable interpretation. We hypothesize that a Blackbox encodes several interpretable models, each applicable to a different portion of data. Thus, a single interpretable model may be insufficient to explain all samples. We construct a hybrid neuro-symbolic model by progressively _carving out_ a mixture of interpretable models and a _residual network_ from the given Blackbox. We coin the term _expert_ for each interpretable model, as they specialize over a subset of data. All the interpretable models are termed a "Mixture of Interpretable Experts" (MoIE). Our design identifies a subset of samples and _routes_ them through the interpretable models to explain the samples with FOL, providing basic reasoning on concepts from the Blackbox. The remaining samples are routed through a flexible residual network. On the residual network, we repeat the method until MoIE explains the desired proportion of data. We quantify the sufficiency of the identified concepts to explain the Blackbox's prediction using the concept completeness score (Yeh et al., 2019). Using FOL for interpretable models offers recourse when undesirable behavior is detected in the model. We provide an example of fixing a shortcut learning by modifying the FOL. FOL can be used in human-model interaction (not explored in this paper). Our method is the divide-and-conquer approach, where the instances covered by the residual network need progressively more complicated interpretable models. Such insight can be used to inspect the data and the model further. Finally, our model allows _unexplainable_ category of data, which is currently not allowed in the interpretable models.

## 2 Method

**Notation:** Assume we have a dataset \(\{\mathcal{X}\), \(\mathcal{Y}\), \(\mathcal{C}\}\), where \(\mathcal{X}\), \(\mathcal{Y}\), and \(\mathcal{C}\) are the input images, class labels, and human interpretable attributes, respectively. \(f^{0}:\mathcal{X}\rightarrow\mathcal{Y}\), is our pre-trained initial Blackbox model. We assume that \(f^{0}\) is a composition \(h^{0}\circ\Phi\), where \(\Phi:\mathcal{X}\rightarrow\mathbb{R}^{l}\) is the image embeddings and \(h^{0}:\mathbb{R}^{l}\rightarrow\mathcal{Y}\) is a transformation from the embeddings, \(\Phi\), to the class labels. We denote the learnable function \(t:\mathbb{R}^{l}\rightarrow\mathcal{C}\), projecting the image embeddings to the concept space. The concept space is the space spanned by the attributes \(\mathcal{C}\). Thus, function \(t\) outputs a scalar value representing a concept for each input image.

**Method Overview:** Figure 1 summarizes our approach. We iteratively carve out an interpretable model from the given Blackbox. Each iteration yields an interpretable model (the downward grey paths in Figure 1) and a residual (the straightforward black paths in Figure 1). We start with the initial Blackbox \(f^{0}\). At iteration \(k\), we distill the Blackbox from the previous iteration \(f^{k-1}\) into a neuro-symbolic interpretable model, \(g^{k}:\mathcal{C}\rightarrow\mathcal{Y}\). Our \(g\) is flexible enough to be any interpretable models (Yuksekgonul et al., 2022; Koh et al., 2020; Barbiero et al., 2022). The _residual_\(r^{k}=f^{k-1}-g^{k}\) emphasizes the portion of \(f^{k-1}\) that \(g^{k}\)cannot explain. We then approximate \(r^{k}\) with \(f^{k}=h^{k}\circ\Phi\). \(f^{k}\) will be the Blackbox for the subsequent iteration and be explained by the respective interpretable model. A learnable gating mechanism, denoted by \(\pi^{k}:\mathcal{C}\rightarrow\{0,1\}\) (shown as the _selector_ in Figure 1) routes an input sample towards either \(g^{k}\) or \(r^{k}\). The thickness of the lines in Figure 1 represents the samples covered by the interpretable models (grey line) and the residuals (black line). With every iteration, the cumulative coverage of the interpretable models increases, but the residual decreases. We name our method _route, interpret_ and _repeat_.

### Neuro-Symbolic Knowledge Distillation

Knowledge distillation in our method involves 3 parts: (1) a series of trainable selectors, _routing_ each sample through the interpretable models and the residual networks, (2) a sequence of learnable neuro-symbolic interpretable models, each providing FOL explanations to _interpret_ the Blackbox, and (3) _repeating_ with Residuals for the samples that cannot be explained with their interpretable counterparts. We detail each component below.

Figure 1: Schematic view of _route, interpret_ and _repeat_. At iteration \(k\), the selector _routes_ each sample either towards the interpretable model \(g^{k}\) (to _interpret_) with probability \(\pi^{k}(.)\) or the residual \(r^{k}=f^{k-1}-g^{k}\) with probability \(1-\pi^{k}(.)\) (to _repeat_ in the further iterations). \(f^{k-1}\) is the Blackbox of the \((k-1)^{th}\) iteration. \(g^{k}\) generates FOL-based explanations for the samples it covers. Otherwise, the selector routes through the next step until it either goes through a subsequent interpretable model or reaches the last residual. Components in black and grey indicate the fixed and trainable modules in our model, respectively.
#### 2.1.1 The selector function

As the first step of our method, the selector \(\pi^{k}\)_routes_ the \(j^{th}\) sample through the interpretable model \(g^{k}\) or residual \(r^{k}\) with probability \(\pi^{k}(\mathbf{c_{j}})\) and \(1-\pi^{k}(\mathbf{c_{j}})\) respectively, where \(k\in[0,K]\), with \(K\) being the number of iterations. We define the empirical coverage of the \(k^{th}\) iteration as \(\zeta(\pi^{k})=\frac{1}{m}\sum_{j=1}^{m}\pi^{k}(\mathbf{c_{j}})\), the empirical mean of the samples selected by the selector for the associated interpretable model \(g^{k}\), with \(m\) being the total number of samples in the training set. Thus, the entire selective risk is:

\[\mathcal{R}^{k}(\pi^{k},g^{k})=\frac{\frac{1}{m}\sum_{j=1}^{m}\mathcal{L}^{k}_ {(g^{k},\pi^{k})}\big{(}\mathbf{x_{j}},\mathbf{c_{j}}\big{)}}{\zeta(\pi^{k})}, \tag{1}\]

where \(\mathcal{L}^{k}_{(g^{k},\pi^{k})}\) is the optimization loss used to learn \(g^{k}\) and \(\pi^{k}\) together, discussed in Section 2.1.2. For a given coverage of \(\tau^{k}\in(0,1]\), we solve the following optimization problem:

\[\theta^{*}_{s^{k}},\theta^{*}_{g^{k}}= \operatorname*{arg\,min}_{\theta_{s^{k}},\theta_{g^{k}}}\mathcal{ R}^{k}\Big{(}\pi^{k}(.;\theta_{s^{k}}),g^{k}(.;\theta_{g^{k}})\Big{)}\] \[\text{s.t.}\quad\zeta\big{(}\pi^{k}(:;\theta_{s^{k}})\big{)}\geq \tau^{k}, \tag{2}\]

where \(\theta^{*}_{s^{k}},\theta^{*}_{g^{k}}\) are the optimal parameters at iteration \(k\) for the selector \(\pi^{k}\) and the interpretable model \(g^{k}\) respectively. In this work, \(\pi\)s' of different iterations are neural networks with sigmoid activation. At inference time, the selector routes the \(j^{th}\) sample with concept vector \(\mathbf{c_{j}}\) to \(g^{k}\) if and only if \(\pi^{k}(\mathbf{c_{j}})\geq 0.5\) for \(k\in[0,K]\).

#### 2.1.2 Neuro-Symbolic interpretable models

In this stage, we design interpretable model \(g^{k}\) of \(k^{th}\) iteration to _interpret_ the Blackbox \(f^{k-1}\) from the previous \((k-1)^{th}\) iteration by optimizing the following loss function:

\[\mathcal{L}^{k}_{(g^{k},\pi^{k})}(\mathbf{x_{j}},\mathbf{c_{j}})=\underbrace{\ell \big{(}f^{k-1}(\mathbf{x_{j}}),g^{k}(\mathbf{c_{j}})\big{)}\pi^{k}(c_{j})}_{ \begin{subarray}{c}\text{variable component}\\ \text{for current iteration $k$}\end{subarray}}(\underbrace{\prod_{i=1}^{k-1}(1-\pi^{i}(\mathbf{c_{j}}))}_ {\begin{subarray}{c}\text{fixed component trained}\\ \text{in the previous iterations}\end{subarray}}), \tag{3}\]

where the term \(\pi^{k}(\mathbf{c_{j}})\prod_{i=1}^{k-1}\big{(}1-\pi^{i}(\mathbf{c_{j}})\big{)}\) denotes the probability of \(j^{th}\) sample being routed through the interpretable model \(g^{k}\). It is the probability of the sample going through the residuals for all the previous iterations from \(1\) through \(k-1\) (_i.e.,_\(\prod_{i=1}^{k-1}\big{(}1-\pi^{i}(\mathbf{c_{j}})\big{)}\)) times the probability of going through the interpretable model at iteration \(k\) (_i.e.,_\(\pi^{k}(\mathbf{c_{j}})\)). Refer to Figure 1 for an illustration. We learn \(\pi^{1},\dots\pi^{k-1}\) in the prior iterations and are not trainable at iteration \(k\). As each interpretable model \(g^{k}\) specializes in explaining a specific subset of samples (denoted by coverage \(\tau\)), we refer to it as an _expert_. We use SelectiveNet's (Geifman and El-Yaniv, 2019) optimization method to optimize Equation (2) since selectors need a rejection mechanism to route samples through residuals. Appendix A.4 details the optimization procedure in Equation (3). We refer to the interpretable experts of all the iterations as a "Mixture of Interpretable Experts" (MoIE) cumulatively after training. Furthermore, we utilize E-LEN, _i.e.,_ a Logic Explainable Network (Ciravegna et al., 2023) implemented with an Entropy Layer as first layer (Barbiero et al., 2022) as the interpretable symbolic model \(g\) to construct First Order Logic (FOL) explanations of a given prediction.

#### 2.1.3 The Residuals

The last step is to _repeat_ with the residual \(r^{k}\), as \(r^{k}(\mathbf{x_{j}},\mathbf{c_{j}})=f^{k-1}(\mathbf{x_{j}})-g^{k}(\mathbf{c_{j}})\). We train \(f^{k}=h^{k}\big{(}\Phi(.)\big{)}\) to approximate the residual \(r^{k}\), creating a new Blackbox \(f^{k}\) for the next iteration \((k+1)\). This step is necessary to specialize \(f^{k}\) over samples not covered by \(g^{k}\). Optimizing the following loss function yields \(f^{k}\) for the \(k^{th}\) iteration:

\[\mathcal{L}^{k}_{f}(\mathbf{x_{j}},\mathbf{c_{j}})=\underbrace{\ell\big{(}r^{k}(\mathbf{x _{j}},\mathbf{c_{j}}),f^{k}(\mathbf{x_{j}})\big{)}}_{\begin{subarray}{c}\text{ trainable component}\\ \text{for iteration $k$}\end{subarray}}\underbrace{\prod_{i=1}^{k}\big{(}1-\pi^{i}(\mathbf{c_{j}}) \big{)}}_{\begin{subarray}{c}\text{non-trainable component}\\ \text{for iteration $k$}\end{subarray}} \tag{4}\]

Notice that we fix the embedding \(\Phi(.)\) for all the iterations. Due to computational overhead, we only finetune the last few layers of the Blackbox (\(h^{k}\)) to train \(f^{k}\). At the final iteration \(K\), our method produces a MoIE and a Residual, explaining the interpretable and uninterpretable components of the initial Blackbox \(f^{0}\), respectively. Appendix A.5 describes the training procedure of our model, the extraction of FOL, and the architecture of our model at inference.

**Selecting number of iterations \(K\):** We follow two principles to select the number of iterations \(K\) as a stopping criterion: 1) Each expert should have enough data to be trained reliably ( coverage \(\zeta^{k}\)). If an expert covers insufficient samples, we stop the process. 2) If the final residual (\(r^{K}\)) underperforms a threshold, it is not reliable to distill from the Blackbox. We stop the procedure to ensure that overall accuracy is maintained.

\begin{table}
\begin{tabular}{l c c} \hline \hline DATASET & BLACKBOK & \# EVPRTS \\ \hline CUB-200 (Wih et al., 2011) & RESNET101 (Wih et al., 2016) & 6 \\ CUB-200 (Wih et al., 2011) & VIT (Wang et al., 2012) & 6 \\ AWA22(Cun et al., 2018) & RESNET101 (Wih et al., 2016) & 4 \\ AWA22(Cun et al., 2018) & VIT (Wang et al., 2012) & 6 \\ HAM100 (Tchandl et al., 2018) & INCEPTION (Szeply et al., 2015) & 6 \\ SIM-SSC (Zoeberberg et al., 2012) & INCEPTION (Szeply et al., 2015) & 6 \\ EPFUSION IN MIDECO (Zoebergs et al., 2015) & RESNET121 (Zhang et al., 2017) & 3 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Datasets and Blackboxes.
## 3 Related work

**Post hoc explanations:** Post hoc explanations retain the flexibility and performance of the Blackbox. The post hoc explanation has many categories, including feature attribution (Simonyan et al., 2013; Smilkov et al., 2017; Binder et al., 2016) and counterfactual approaches (Singla et al., 2019; Abid et al., 2021). For example, feature attribution methods associate a measure of importance to features (e.g., pixels) that is proportional to the feature's contribution to BlackBox's predicted output. Many methods were proposed to estimate the importance measure, including gradient-based methods (Selvaraju et al., 2017; Sundararajan et al., 2017), game-theoretic approach (Lundberg and Lee, 2017). The post hoc approaches suffer from a lack of fidelity to input (Adebayo et al., 2018) and ambiguity in explanation due to a lack of correspondence to human-understandable concepts. Recently, Posthoc Concept Bottleneck models (PCBMs) (Yuksekgonul et al., 2022) learn the concepts from a trained Blackbox embedding and use an interpretable classifier for classification. Also, they fit a residual in their hybrid variant (PCBM-h) to mimic the performance of the Blackbox. We will compare against the performance of the PCBMs method. Another major shortcoming is that, due to a lack of mechanistic explanation, post hoc explanations do not provide a recourse when an undesirable property of a Blackbox is identified. Interpretable-by-design provides a remedy to those issues (Rudin, 2019).

**Concept-based interpretable models:** Our approach falls into the category of concept-based interpretable models. Such methods provide a mechanistically interpretable prediction that is a function of human-understandable concepts. The concepts are usually extracted from the activation of the middle layers of the Neural Network (bottleneck). Examples include Concept Bottleneck models (CBMs) (Koh et al., 2020), antheo concept decoder (Sarkar et al., 2022), and a high-dimensional Concept Embedding model (CEMs) (Zarlenga et al., 2022) that uses high dimensional concept embeddings to allow extra supervised learning capacity and achieves SOTA performance in the interpretable-by-design class. Most concept-based interpretable models do not model the interaction between concepts and cannot be used for reasoning. An exception is E-LEN (Barbiero et al., 2022) which uses an entropy-based approach to derive explanations in terms of FOL using the concepts. The underlying assumption of those methods is that one interpretable function can explain the entire set of data, which can limit flexibility and consequently hurt the performance of the models. Our approach relaxes that assumption by allowing multiple interpretable functions and a residual. Each function is appropriate for a portion of the data, and a small portion of the data is allowed to be uninterpretable by the model (_i.e.,_ residual). We will compare our method with CBMs, CEMs, and their E-LEN-enhanced variants.

**Application in fixing the shortcut learning:** Shortcuts are spurious features that correlate with both input and the label on the training dataset but fail to generalize in more challenging real-world scenarios. Explainable AI (X-AI) aims to identify and fix such an undesirable property. Related work in X-AI includes LIME (Ribeiro et al., 2016), utilized to detect spurious background as a shortcut to classify an

Figure 2: MoIE identifies diverse concepts for specific subsets of a class, unlike the generic ones by the baselines. **(i)** We construct the FOL explanations of the samples of, “Bay breasted warbler” in the CUB-200 dataset for VIT-based **(a)** CBM + E-LEN as an _interpretable-by-design_ baseline, **(b)** PCBM + E-LEN as a _posthoc_ baseline, **(c)** experts in MoIE at inference. We highlight the unique concepts for experts 1,2, and 3 in _red, blue_, and _magenta_, respectively. **(ii)** Comparison of FOL explanations by MoIE with the PCBM + E-LEN baselines for HAM10000 **(top)** and ISIC **(down)** to classify Malignant lesion. We highlight unique concepts for experts 3, 5, and 6 in _red, blue_, and _violet_, respectively. For brevity, we combine FOLs for each expert for the samples covered by them.
animal. Recently interpretable model (Rosenzweig et al., 2021), involving local image patches, are used as a proxy to the Blackbox to identify shortcuts. However, both methods operate in pixel space, not concept space. Also, both approaches are post hoc and do not provide a way to eliminate the shortcut learning problem. Our MoIE discovers shortcuts using the high-level concepts in the FOL explanation of the Blackbox's prediction and eliminates them via metadata normalization (MDN) (Lu et al., 2021).

## 4 Experiments

We perform experiments on a variety of vision and medical imaging datasets to show that 1) MoIE captures a diverse set of concepts, 2) the performance of the residuals degrades over successive iterations as they cover "harder" instances, 3) MoIE does not compromise the performance of the Blackbox, 4) MoIE achieves superior performances during test time interventions, and 5) MoIE can fix the shortcuts using the Waterbirds dataset (Sagawa et al., 2019). We repeat our method until MoIE covers at least 90% of samples or the final residual's accuracy falls below 70%. Refer to Table 1 for the datasets and Blackboxes experimented with. For ResNets and Inception, we flatten the feature maps from the last convolutional block to extract the concepts. For VITs, we use the image embeddings from the transformer encoder to perform the same. We use SIIM-ISIC as a real-world transfer learning setting, with the Blackbox trained on HAM10000 and evaluated on a subset of the SIIM-ISIC Melanoma Classification dataset (Yuksekgonul et al., 2022). Appendix A.6 and Appendix A.7 expand on the datasets and hyperparameters.

**Baselines:** We compare our methods to two concept-based baselines - 1) interpretable-by-design and 2) posthoc. They consist of two parts: a) a concept predictor \(\Phi:\mathcal{X}\rightarrow\mathcal{C}\), predicting concepts from images; and b) a label predictor \(g:\mathcal{C}\rightarrow\mathcal{Y}\), predicting labels from the concepts. The end-to-end CEMs and sequential CBMs serve as interpretable-by-design baselines. Similarly, PCBM and PCBM-h serve as post hoc baselines. Convolution-based \(\Phi\) includes all layers till the last convolution block. VIT-based \(\Phi\) consists of the transformer encoder block. The standard CBM and PCBM models do not show how the concepts are composed to make the label prediction. So, we create CBM + E-LEN, PCBM + E-LEN and PCBM-h + E-LEN by using the identical \(g\) of MOIE (shown in Appendix A.7), as a replacement for the standard classifiers of CBM and PCBM. We train the \(\Phi\) and \(g\) in these new baselines to sequentially generate FOLs (Barbiero et al., 2022). Due to the unavailability of concept annotations, we extract the concepts from the Derm7pt dataset (Kawahara et al., 2018) using the pretrained embeddings of the Blackbox (Yuksekgonul et al., 2022) for HAM10000. Thus, we do not have interpretable-by-design baselines for HAM10000 and ISIC.

### Results

#### 4.1.1 Expert driven explanations by MoIE

First, we show that MoIE captures a rich set of diverse instance-specific concepts qualitatively. Next, we show quantitatively that MoIE-identified concepts are faithful to Blackbox's final prediction using the metric "completeness score" and zeroing out relevant concepts.

**Heterogenity of Explanations:** At each iteration of MoIE, the blackbox \(\left(h^{k}(\Phi(.)\right)\) splits into an interpretable expert (\(g^{k}\)) and a residual (\(r^{k}\)). Figure 2i shows this mechanism for VIT-based MoIE and compares the FOLs with CBM + E-LEN and PCBM + E-LEN baselines to classify "Bay Breasted Warbler" of CUB-200. The experts of different iterations specialize in specific instances of "Bay Breasted Warbler". Thus, each expert's FOL comprises its instance-specific concepts of the same class ( Figure 2i-c). For example, the concept, _leg_color_grey_ is unique to expert4, but _belly_pattern_solid_ and _back_pattern_multicolored_ are unique to experts 1 and 2, respectively, to classify the instances of "Bay Breasted Warbler". Unlike MoIE, the baselines employ a single interpretable model \(g\), resulting in a generic FOL with identical concepts for all the samples of "Bay Breasted Warbler" (Figure 2(a-b)). Thus the baselines fail to capture the heterogeneity of explanations. For additional results of CUB-200, refer to Appendix A.10.6.

Figure 2ii shows such diverse explanations for HAM10000 (_top_) and ISIC (_bottom_). In Figure 2ii-(top), the baseline-FOL consists of concepts such as _AtypicalPigmentNetwork_ and _BlueWhitishVeil (BWV)_ to classify "Malignancy" for all the instances for HAM10000. However, expert 3 relies on _RegressionStructures_ along with \(BWV\) to classify the same for the samples it covers while expert 5 utilizes several other concepts _e.g., IrregularStreaks_, _Irregular dots and globules (IrregularDG) etc._ Due to space constraints, Appendix A.10.7 reports similar results for the Awa2 dataset. Also, VIT-based experts compose less concepts per sample than the ResNet-based experts, shown in Appendix A.10.8.

**MoIE-identified concepts attain higher completeness scores.** Figure 5(a-b) shows the completeness scores (Yeh et al., 2019) for varying number of concepts. Completeness score is a post hoc measure, signifying the identified concepts as "sufficient statistic" of the predictive capability of the Blackbox. Recall that \(g\) utilizes E-LEN (Barbiero et al., 2022), associating each concept with an attention weight after training. A concept with high attention weight implies its high predictive significance. Iteratively, we select the top relevant concepts based on their attention weights and compute the completeness scores for the top concepts for MoIE and the PCBM + E-LEN baseline in Fig
ure 5(a-b) ( Appendix A.8 for details). For example, MoIE achieves a completeness score of 0.9 compared to 0.75 of the baseline(\(\sim 20\%\uparrow\)) for the 10 most significant concepts for the CUB-200 dataset with VIT as Blackbox.

**MoIE identifies more meaningful instance-specific concepts.** Figure 5(c-d) reports the drop in accuracy by zeroing out the significant concepts. Any interpretable model (\(g\)) supports concept-intervention (Koh et al., 2020). After identifying the top concepts from \(g\) using the attention weights, as in the last section, we set these concepts' values to zero, compute the model's accuracy drop, and plot in Figure 5(b). When zeroing out the top 10 essential concepts for VIT-based CUB-200 models, MoIE records a drop of 53% compared to 28% and 42% for the CBM + E-LEN and PCBM + E-LEN baselines, respectively, showing the faithfulness of the identified concepts to the prediction.

In both of the last experiments, MoIE outperforms the baselines as the baselines mark the same concepts as significant for all samples of each class. However, MoIE leverages various experts specializing in different subsets of samples of different classes. For results of MIMIC-CXR and Awa2, refer to Appendix A.10.2 and Appendix A.10.4 respectively.

#### 4.1.2 Identification of harder samples by successive residuals

Figure 3 (a-c) display the proportional accuracy of the experts and the residuals of our method per iteration. The proportional accuracy of each model (experts and/or residuals) is defined as the accuracy of that model times its cov

Figure 4: Across architectures test time interventions of concepts on all the samples **(a-d)**, on the “hard” samples **(e)**, covered by only the last two experts of MoIE.

Figure 3: The performance of experts and residuals across iterations. **(a-c)** Coverage and proportional accuracy of the experts and residuals. **(d-f)** We route the samples covered by the residuals across iterations to the initial Blackbox \(f^{0}\) and compare the accuracy of \(f^{0}\) (red bar) with the residual (blue bar). Figures **d-f** show the progressive decline in performance of the residuals across iterations as they cover the samples in the increasing order of “hardness”. We observe the similar abysmal performance of the initial blackbox \(f^{0}\) for these samples.
erage. Recall that the model's coverage is the empirical mean of the samples selected by the selector. Figure 2(a) show that the experts and residual cumulatively achieve an accuracy \(\sim\) 0.92 for the CUB-200 dataset in iteration 1, with more contribution from the residual (black bar) than the expert1 (blue bar). Later iterations cumulatively increase and worsen the performance of the experts and corresponding residuals, respectively. The final iteration curves out the entire interpretable portion from the Blackbox \(f^{0}\) via all the experts, resulting in their more significant contribution to the cumulative performance. The residual of the last iteration covers the "hardest" samples, achieving low accuracy. Tracing these samples back to the original Blackbox \(f^{0}\), it also classifies these samples poorly (Figure 2(d-f)). As shown in the coverage plot, this experiment reinforces Figure 1, where the flow through the experts gradually becomes thicker compared to the narrower flow of the residual with every iteration. Refer to Figure 12 in the Appendix A.10.3 for the results of the ResNet-based MoIEs.

#### 4.1.3 Quantitative analysis of MoIE with the blackbox and baseline

**Comparing with the interpretable-by-design baselines:** Table 2 shows that MoIE achieves comparable performance to the Blackbox. Recall that "MoIE" refers to the mixture of all interpretable experts (\(g\)) only excluding any residuals. MoIE outperforms the interpretable-by-design baselines for all the datasets except Awa2. Since Awa2 is designed for zero-shot learning, its rich concept annotation makes it appropriate for interpretable-by-design models. In general, VIT-derived MoIEs perform better than their ResNet-based variants.

**Comparing with the PCBMs:** Table 2 shows that interpretable MoIE outperforms the interpretable posthoc baselines - PCBM and PCBM + E-LEN for all the datasets, especially by a significant margin for CUB-200 and ISIC. We also report "MoIE + Residual" as the mixture of interpretable experts plus the final residual to compare with the residualized PCBM, _i.e.,_ PCBM-h. Table 2 shows that PCBM-h performs slightly better than MoIE + Residual. Note that PCBM-h learns the residual by fitting the complete dataset to fix the interpretable PCBM's mistakes to replicate the performance of the Blackbox, resulting in better performance for PCBM-h than PCBM. However, we assume the Blackbox to be a combination of interpretable and uninterpretable components. So, we train the experts and the final residual to cover the interpretable and uninterpretable portions of the Blackbox respectively. In each iteration, our method learns the residuals to focus on the samples, which are not covered by the respective interpretable experts. Therefore, residuals are not designed to fix the mistakes made by the experts. In doing so, the final residual in MoIE + Residual covers the "hardest" examples, lowering its overall performance compared to MoIE.

#### 4.1.4 Test time interventions

Figure 4(a-d) shows effect of test time interventions. Any concept-based models (Koh et al., 2020; Zarlenga et al., 2022) allow test time interventions for datasets with concept annotation (_e.g.,_ CUB-200, Awa2). We identify the significant concepts via their attention scores in \(g\), as during the computation of completeness scores, and set their values with the ground truths, considering the ground truth concepts as an oracle. As MoIE identifies a more diverse set of concepts by focusing on different subsets of classes, MoIE

\begin{table}
\begin{tabular}{l l l l l l l l} \hline \hline
**MODEL** & \multicolumn{3}{c}{**Datasets**} \\  & CUB-200 & CUB-200 (VIT) & AWA2 & AWA2 (VIT) & HAM10000 & SIME-SIC & EPpression \\  & (0283TTTTT) & & (0283TTTT) & & & & & \\ \hline BLACKACK & 0.88 & 0.92 & 0.89 & 0.99 & 0.96 & 0.85 & 0.91 \\ \hline
**INTERSPEARLE-RA-V-DISION** & & - & 0.88 \(\pm\) 0.50 & - & NA & NA & 0.76 \(\pm\) 0.00 \\ CIM (Zalalalrega et al., 2022) & 0.77 \(\pm\) 0.22 & - & 0.88 \(\pm\) 0.50 & - & NA & NA & 0.79 \(\pm\) 0.00 \\ CIM (Zalalrega et al., 2022) & 0.65 \(\pm\) 0.37 & 0.86 \(\pm\) 0.24 & 0.88 \(\pm\) 0.35 & 0.94 \(\pm\) 0.28 & NA & NA & 0.79 \(\pm\) 0.00 \\ CIM + E-LEN (Koh et al., 2020; Benlière et al., 2022) & 0.71 \(\pm\) 0.35 & 0.88 \(\pm\) 0.24 & 0.86 \(\pm\) 0.35 & 0.93 \(\pm\) 0.25 & NA & NA & 0.79 \(\pm\) 0.00 \\ \hline
**POSTING** & & & & & & & & \\ PGNL (Vakulappani et al., 2022) & 0.36 \(\pm\) 0.01 & 0.88 \(\pm\) 0.20 & 0.82 \(\pm\) 0.23 & 0.94 \(\pm\) 0.17 & 0.93 \(\pm\) 0.00 & 0.71 \(\pm\) 0.01 & 0.81 \(\pm\) 0.01 \\ PGNL (Vakulappani et al., 2022) & 0.35 \(\pm\) 0.01 & 0.91 \(\pm\) 0.18 & 0.87 \(\pm\) 0.20 & 0.98 \(\pm\) 0.17 & 0.95 \(\pm\) 0.00 & 0.79 \(\pm\) 0.05 & 0.82 \(\pm\) 0.07 \\ PGNL + E-LEN (Vakulappani et al., 2022; Benlière et al., 2022) & 0.80 \(\pm\) 0.36 & 0.89 \(\pm\) 0.26 & 0.85 \(\pm\) 0.25 & 0.96 \(\pm\) 0.18 & 0.94 \(\pm\) 0.02 & 0.73 \(\pm\) 0.01 & 0.83 \(\pm\) 0.01 \\
**POSTING** & & & & & & & & \\ PGNL (Vakulappani et al., 2022; Benlière et al., 2022; Benlière et al., 2022) & 0.88 \(\pm\) 0.24 & 0.98 \(\pm\) 0.20 & 0.95 \(\pm\) 0.03 & 0.82 \(\pm\) 0.05 & 0.87 \(\pm\) 0.03 \\ \hline
**ORES** & & & & & & & & \\ Muffle (COVERAGE) & **0.85 \(\pm\) 0.01 (0.9)** & **0.91 \(\pm\) 0.00** & **0.87 \(\pm\) 0.00** & **0.97 \(\pm\) 0.00** & **0.95 \(\pm\) 0.00** (0.97) \(\pm\) 0.00** & **0.84 \(\pm\) 0.00** & **0.87 \(\pm\) 0.00** \\ MoIE + RESIDUA & & & & & & & & \\ \hline
**MoIE** + **R-SIDUA** & & & & & & & \\ \hline \hline \end{tabular}
\end{table}
Table 2: MoIE does not hurt the performance of the original Blackbox using a held-out test set. We provide the mean and standard errors of AUROC and accuracy for medical imaging (_e.g.,_ HAM10000, ISIC, and Effusion) and vision (_e.g.,_ CUB-200 and Awa2) datasets, respectively, over 5 random seeds. For MoIE, we also report the percentage of test set samples covered by all experts as “coverage”. Here, MoIE + Residual represents the experts with the final residual. Following the setting (Zarlenga et al., 2022), we only report the performance of the convolutional CEM, leaving the construction of VIT-based CEM as a future work. Recall that interpretable-by-design models can not be constructed for HAM10000 and ISIC as they have no concept annotation; we learn the concepts from the Derm7pt dataset. For all the datasets, MoIE covers a significant portion of data (at least 90%) cumulatively. We boldface our results.
outperforms the baselines in terms of accuracy for such test time interventions. Instead of manually deciding the samples to intervene, it is generally preferred to intervene on the "harder" samples, making the process efficient. As per Section 4.1.2, experts of different iterations cover samples with increasing order of "hardness". To intervene efficiently, we perform identical test-time interventions with varying numbers of concepts for the "harder" samples covered by the final two experts and plot the accuracy in Figure 4(e). For the VIT-derived MoIE of CUB-200, intervening only on 20 concepts enhances the accuracy of MoIE from 91% to 96% (\(\sim 6.1\%\) \(\uparrow\)). We cannot perform the same for the baselines as they cannot directly estimate "harder" samples. Also, Figure 4 shows a relatively higher gain for ResNet-based models in general. Appendix A.10.5 demonstrates an example of test time intervention of concepts for relatively "harder" samples, identified by the last two experts of MoIE.

#### 4.1.5 Application in the removal of shortcuts

First, we create the Waterbirds dataset as in (Sagawa et al., 2019)by using forest and bamboo as the spurious land concepts of the Places dataset for landbirds of the CUB-200 dataset. We do the same by using oceans and lakes as the spurious water concepts for waterbirds. We utilize ResNet50 as the Blackbox \(f^{0}\) to identify each bird as a Waterbird or a Landbird. The Blackbox quickly latches on the spurious backgrounds to classify the birds. As a result, the black box's accuracy differs for land-based versus aquatic subsets of the bird species, as shown in Figure 6a. The Waterbird on the water is more accurate than on land ((96% vs. 67 % in the orange bar in the Figure 6a). The FOL from the biased Blackbox-derived MoIE captures the spurious concept _forest_ a waterbird, misclassified as a landbird. Assuming the background concepts as metadata, we minimize the background bias from the representation of the Blackbox using Metadata normalization (MDN) layers (Lu et al., 2021) between two successive layers of the convolutional backbone to fine-tune the biased Blackbox. Next, we train \(t\), using the embedding \(\Phi\) of the robust Blackbox, and compare the accuracy of the spurious concepts with the biased blackbox in Figure 6d. The validation accuracy of all the spurious concepts retrieved from the robust Blackbox falls well short of the predefined threshold 70% compared to the biased Blackbox. Finally, we re-train the MoIE distilling from the new robust Blackbox. Figure 6b illustrates similar accuracies of MoIE for Waterbirds on water vs. Waterbirds on land (91% -88 %). The FOL from the robust Blackbox does not include any background concepts ( 6c, bottom row). Refer to 8 in Appendix A.9 for the flow diagram of this experiment.

Figure 5: Quantitative validation of the extracted concepts. **(a-b)** Completeness scores of the models for a varying number of top concepts. **(c-d)** Drop in accuracy compared to the original model after zeroing out the top significant concepts iteratively. The highest drop for MoIE indicates that MoIE selects more instance-specific concepts than generic ones by the baselines.

Figure 6: MoIE fixes shortcuts. **(a)** Performance of the biased Blackbox. **(b)** Performance of final MoIE extracted from the robust Blackbox after removing the shortcuts using MDN. **(c)** Examples of samples **(top-row)** and their explanations by the biased **(middle-row)** and robust Blackboxes (**bottom-row)**. **(d)** Comparison of accuracies of the spurious concepts extracted from the biased vs. the robust Blackbox.
## 5 Discussion & Conclusions

This paper proposes a novel method to iteratively extract a mixture of interpretable models from a flexible Blackbox. The comprehensive experiments on various datasets demonstrate that our method 1) captures more meaningful instance-specific concepts with high completeness score than baselines without losing the performance of the Blackbox, 2) does not require explicit concept annotation, 3) identifies the "harder" samples using the residuals, 4) achieves significant performance gain than the baselines during test time interventions, 5) eliminate shortcuts effectively. In the future, we aim to apply our method to other modalities, such as text or video. Also, as in the prior work, MoIE-captured concepts may not reflect a causal effect. The assessment of causal concept effects necessitates estimating inter-concept interactions, which will be the subject of future research.

## 6 Acknowledgement

We would like to thank Mert Yukekgonul of Stanford University for providing the code to construct the concept bank of Derm7pt to conduct the skin experiments. This work was partially supported by NIH Award Number 1R01HL141813-01 and the Pennsylvania Department of Health. We are grateful for the computational resources provided by Pittsburgh Super Computing grant number TG-ASC170024.
