# Consistent Explanations in the Face of Model Indeterminacy via Ensembling

Dan Ley Leonard Tang Matthew Nazari
Hongjin Lin Suraj Srinivas Himabindu Lakkaraju
Harvard University, Cambridge, MA
{dley<EMAIL_ADDRESS>
{leonardtang<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

This work addresses the challenge of providing consistent explanations for
predictive models in the presence of model indeterminacy, which arises due to
the existence of multiple (nearly) equally well-performing models for a given
dataset and task. Despite their similar performance, such models often exhibit
inconsistent or even contradictory explanations for their predictions, posing
challenges to end users who rely on these models to make critical decisions.
Recognizing this issue, we introduce ensemble methods as an approach to
enhance the consistency of the explanations provided in these scenarios.
Leveraging insights from recent work on neural network loss landscapes and
mode connectivity, we devise ensemble strategies to efficiently explore the
underspecification set – the set of models with performance variations
resulting solely from changes in the random seed during training. Experiments
on five benchmark financial datasets reveal that ensembling can yield
significant improvements when it comes to explanation similarity, and
demonstrate the potential of existing ensemble methods to explore the
underspecification set efficiently. Our findings highlight the importance of
considering model indeterminacy when interpreting explanations and showcase
the effectiveness of ensembles in enhancing the reliability of explanations in
machine learning.

##  1 Introduction

The rapidly increasing adoption of machine learning (ML) models in a wide
range of applications, including healthcare, finance, criminal justice, and
autonomous systems, underscores the need for transparency and trust in
automated decision-making. However, ensuring the consistency of explanations
offered by these models has proven to be a complex challenge with far-reaching
implications, particularly in high-stakes scenarios, where decisions carry a
substantial impact on individuals and society. This has led to the
introduction of various regulatory principles [2, 18].

A key factor complicating the interpretability of predictive models is model
indeterminacy, which arises from the existence of multiple (nearly) equally
well-performing models for a given dataset and task. Despite comparable
performance, these models often provide inconsistent or even contradictory
explanations for their decisions (Figure 1). Such explanatory multiplicity can
severely undermine transparency efforts, erode trust in automated decision
systems, or lead to potentially harmful outcomes in critical applications such
as risk assessment, medical diagnosis, and credit lending.

For instance, in credit lending, a predictive model may be employed to
determine a person’s creditworthiness. In this context, model indeterminacy
can lead to inconsistent explanations for different loan approvals or
rejections. An individual might be denied a loan based on one model’s
decision, while a nearly equally performing model might provide a different
explanation and approve the loan. In addition, if a model is periodically
retrained without accounting for indeterminacy, it may further exacerbate
inconsistencies and erode trust in the decision-making process.

This prompts us to investigate model indeterminacy in more depth.
Specifically, our work examines the underspecification set [7, 8] – the group
of models whose performance variations arise solely from changes to the random
seed used in training. Similarly performing models from within this set can
each offer markedly different explanations for a given input, and prior work
has also indicated a notable lack of correlation between the prediction of a
model and its explanation [6, 7]. This raises some important questions. Do
equally performing ML models share common patterns in their explanations, or
is explanatory multiplicity largely arbitrary? Furthermore, do there exist
systematic approaches that can be utilized to align explanations, without
compromising on model performance?

Motivated by these questions, we propose the use of ensemble methods to
mitigate explanatory multiplicity among equally performing neural networks
(Figure 1), and explore the application of various ensemble methods towards
improving the consistency of explanations generated for this class of models.
Focusing on neural networks is particularly relevant to this problem setup,
given their expressivity and ability to fit complex functions. Neural networks
of sufficient size can cover a large range of predictors, even for relatively
small tasks. Our aim is to devise efficient ensembling strategies that can
align explanations across various diverse modes, while simultaneously
requiring fewer pre-trained models to do so. The methods deployed are informed
by previous research on neural network loss landscapes and existing
explanation techniques.

More specifically, our overall contributions are as follows:

  1. 1.

In §3, we propose ensemble strategies that target two facets of the neural
network loss landscape: local perturbations on model weights (§3.1); and
global connections between models along paths of near-constant loss (§3.2). We
provide a toy illustration to demonstrate how explanatory multiplicity can be
mitigated through both types of exploration (§3.3).

  2. 2.

In §4.1, we perform ablations to understand the impact of weight perturbations
on explanation similarity, finding that it depends critically on the layer
being perturbed.

  3. 3.

In §4.2, we conduct experiments across five benchmark financial datasets and
three commonly used explanation techniques, demonstrating that ensembles
constructed through local or global exploration of the loss landscape can
yield significant improvements in terms of average pairwise top-$k$111For a
given explanation, the $k$ largest features according to absolute value.
similarity between explanations. We further showcase how a novel combination
of both local and global ensembling can yield performance improvements up to
fivefold over naive ensembling, given a fixed number of pre-trained models.

Figure 1: Illustration of explanatory multiplicity between members of the
underspecification set (model variation due only to random seed). Left:
softmax probabilities of neural networks trained on the two moons dataset,
with test points depicted in black and white. All models achieve similar
performance on test data. Center: average pairwise angular difference between
model explanations in the same region of input space. Gradients with respect
to the input, a proxy for many explanation techniques, are shown for a test
point of high disagreement (i.e. high angular difference) between models.
Right: average pairwise angular difference between ensemble explanations.
Ensembles consist of 10 constituent models sampled from the underspecification
set. Our findings indicate that ensembling results in model alignment,
promoting agreement between explanations in input space.

While our experiments highlight the potential of ensemble techniques in
alleviating the challenges posed by model indeterminacy, the problem of
providing consistent explanations for ML models remains an ongoing area of
research. This work seeks to contribute to the development of more trustworthy
and reliable ML systems that can be implemented responsibly in high-stakes,
real-world applications, and provides an empirical analysis of ensemble
behavior in such regards.

##  2 Related work

###  2.1 Underspecification and model indeterminacy

D’Amour et al. [8] identified underspecification in ML pipelines as a key
reason for poor behavior under deployment, defining an ML pipeline to be
underspecified when it can return various distinct predictors with
equivalently strong test performance. They identify the resulting instability
as a distinct failure mode from previously identified issues arising from
structural mismatches between training and deployment domains. In a similar
vein, Brunet et al. [7] investigated the implications of model indeterminacy
on post-hoc explanations of predictive models, demonstrating that
underspecification can lead to significant explanatory multiplicity, and
highlighting that predictive multiplicity and epistemic uncertainty are not
reliable indicators of explanatory multiplicity. Both works motivate the need
to explicitly account for underspecification in ML pipelines intended for
real-world deployment.

###  2.2 Explanation methods

Various explanation techniques have been proposed to provide interpretability
for ML models, with some of the most commonly used methods being Saliency
[32], its smoothed counterpart SmoothGrad [34], as well as local function
approximators such as LIME [31] and SHAP [28]. Smoothgrad has been highlighted
as particularly faithful to the model being explained [1]. In recent years, an
increasing effort has been lent toward understanding the workings and
limitations of these methods. For instance, Krishna et al. [24] analyzed the
disagreement between different explanation methods for a fixed model,
highlighting the challenge of providing consistent explanations, while Han et
al. [20] attempted to unify popular post-hoc explanation methods.

###  2.3 Ensembles

Ensemble methods have traditionally lent themselves as potent tools within ML,
providing advantages such as reduced generalization error [4, 9], robustness
to adversarial examples [40], and uncertainty estimation [25]. However,
analysis of ensembles alongside explanatory multiplicity has only been briefly
touched. Black et al. [6] introduced selective ensembles, which abstain from
decisions in regions of predictive uncertainty, in an effort to avoid
contradictory predictions and make explanations more consistent, though with
limited analysis on explanations explicitly. Li et al. [26] also proposed a
cross-model consensus of explanations to identify common features used by
various models for classification, finding correlations between consensus
score and model performance for vision tasks.

###  2.4 Loss landscapes and mode connectivity

We are interested in leveraging known properties of neural networks to combat
model indeterminacy and expedite the ensembling process of the
underspecification set. A key aspect in this regard is understanding where
high performing models are located within the loss landscape. Traditionally,
diverse model solutions were viewed as lying in disjoint local minima, until
Garipov et al. [17] and Draxler et al. [10] demonstrated that these minima
could be connected via paths of near constant loss in weight space, a concept
dubbed mode connectivity. This surprising result opened the door for a body of
subsequent research, including applications of mode connectivity in
adversarial robustness [42], discovery of mode connecting volumes [13, 5], and
alignment of models in weight space through permutation symmetries [3, 33,
36]. Although ensemble methods feature prominently in this literature, their
application with respect to model explanations has received very little
attention. Advances in loss landscape understanding and implications of model
indeterminacy have emerged as recent, yet distinct developments, with the two
fields existing almost in parallel. The aim of this work is to serve as a
catalyst for their convergence, and bring about a unified exploration of both
areas.

##  3 Ensemble strategies

Inconsistency between explanations of different ML models poses a significant
challenge, where choices as arbitrary as the random seed used during the
training phase of a deployed ML model can drastically affect the explanation
provided to a particular individual [7, 8]. Ensembling provides a promising
avenue for aligning the explanations across similarly performing models.
However, conventional or vanilla methods may necessitate a prohibitively large
number of models to do so.

Given this context, the primary question driving our research is:

> Can we maximize (in expectation) the explanation similarity between two
> ensembles constructed from $n$ samples of the underspecification set?

This question underscores our exploration of ensemble methods towards bridging
the gap between computational efficiency and explanation consistency. Once a
sufficiently large underspecification set has been formed for a given dataset
and task, samples of equally performing models are used to construct
ensembles. We proceed to outline our ensembling strategies (summarized in
Figure 2).

###  3.1 Ensembles via weight perturbation

While we find that standard ensemble methods are capable of aligning the
explanations of models from the underspecification set, they often require a
large collection of pre-trained models to do so. To reduce the number of
models needed for achieving explanation consistency, we propose a
computationally cheap ensembling method that injects Gaussian noise
$\epsilon\sim\mathcal{N}(0,\sigma^{2})$ to the weights or biases of a given
layer. Figure 2, Center, depicts this in relation to the other methods.

Figure 2: Weight space visualizations of the methods we explore. Left: the
underspecification set of (nearly) equally performing models, trained with the
same hyperparameters but different random initializations. Center: local
weight perturbations to each of the models in the underspecification set,
forming regions of alternate predictors of similar performance. The
perturbations are themselves ensembled to form perturbed models. Right: global
mode connectivity paths of near-constant loss, connecting different members of
the underspecification set. Vanilla ensembling of models increases explanation
consistency but requires an impractically large number of models. Our methods
leverage ensembles formed by local weight perturbations and global mode-
connected models to achieve better explanation consistency with fewer models
and computational overhead.

Constructing an ensemble using this method requires the weights of $n$ pre-
trained models sampled from the underspecification set, each denoted
$\omega_{i}$. The method perturbs each $\omega_{i}$ a total of $m$ times,
leading to $m$ variants of each original model $f_{\omega_{i}}$. We denote the
collection of the $m$ variants as the perturbed model
$f_{\tilde{\omega}_{i}}$. This process of generating $n$ perturbed models does
not involve the computational expense of training $n\times m$ distinct models
from scratch. Instead, we capitalize on the learned knowledge of the $n$ pre-
trained models, exploring the local regions surrounding their weights.

Our approach draws inspiration from Smoothgrad [34], which perturbs the input
to stabilize the gradient estimation at a particular point. Unlike Smoothgrad,
which provides an explanation for a smoothed approximation of a potentially
noisy decision boundary, by perturbing the parameters directly and aggregating
the results, this approach smooths both the explanation and the model’s
decision region. Techniques that perturb the parameters of ML models have been
explored in prior work on adversarial training [39] and robust explanation
generation [37], as well as more generally in the context of Bayesian learning
[16, 29]. Other conceptually similar examples to our approach are dropout
[35], and stochastic weight averaging [22], a technique which promotes
convergence of the SGD learning algorithm by averaging model weights during
the final epochs of training. Informed by these works, we introduce weight
perturbation as a computationally cheap ensembling strategy, with the specific
goal of enhancing explanation similarity.

###  3.2 Ensembles via mode connectivity

Instead of sampling locally around a single model in weight space, we also
consider global explorations between pairs of models along the constant paths
of low loss found via mode connectivity [17]. Quadratic Bezier curves provide
a convenient parametrization of smooth paths, where the curve denoted
$\phi_{\theta}(t)$ with endpoints $\omega_{1}$ and $\omega_{2}$ (sampled from
the underspecification set) is given by

$\phi_{\theta}(t)=(1-t)^{2}\omega_{1}+2t(1-t)\theta+t^{2}\omega_{2},0\leq
t\leq 1$

Once such paths between pairs of models are obtained, ensembles can be
constructed at little extra cost, most simply by uniformally sampling the
scalar $t$, returning model weights along the path (Figure 2, Right). To
construct an ensemble from $n$ pre-trained models, we form connections between
$n/2$ non-overlapping pairs, before sampling along these connections and
ensembling the sampled weights. We discuss the curve finding procedure in §4.

###  3.3 Illustration on a toy example

In Figure 3, we illustrate the problem of explanatory multiplicity on the two
moons dataset, and visualize the effectiveness of the described ensembling
techniques in aligning the input gradients (saliency) of models. Observe that
while vanilla ensembles (Left Center) constructed from four pre-trained models
$f_{\omega_{i}}$ demonstrate reductions in average pairwise explanation
difference compared to single models (Left), ensembles constructed from four
perturbed models $f_{\tilde{\omega}_{i}}$ effectively smooth the explanation
distribution, achieving significantly better alignment (Right Center). We note
similar smoothing effects for ensembles constructed via mode connectivity
(Right), again using four pre-trained models i.e. two mode connected pairs.

Figure 3: Heatmaps of explanation disagreement (average pairwise angular
difference between model gradients) for the described ensembling techniques,
each constructed from four constituent models sampled from the
underspecification set. Left: Baseline disagreements between equally
performing models. Observe the high degree of dissimilarity between gradients
(dark arrows) for a given test point. Left Center to Right: the efficacy of
vanilla ensembles, ensembles via weight perturbation, and ensembles via mode
connectivity on aligning explanations. For the same number of pre-trained
models, local perturbations or global connections demonstrate potential for
superior alignment of gradients between ensembles.

The process of averaging the outputs of local weight perturbations can be
viewed as a mapping from the underspecification set of original models, to a
reduced set of perturbed models, characterized by lower gradient variance.
This shrinking process necessitates fewer perturbed models to reach
convergence in explanation alignment. In contrast, mode connectivity provides
a more extensive traversal of the underspecification set, charting a broad
range of models along the path between two endpoints. The following
experiments demonstrate the efficacy of both exploratory methods in real-world
scenarios. Notably, we find that their relative performance often hinges upon
the unique characteristics of the given dataset, the explanation technique,
and the particular model class used.

##  4 Experiments

To evaluate the effectiveness of these ensemble strategies in maximizing
explanation similarity between ensembles, we conduct a series of experiments
spanning five benchmark financial datasets, three similarity metrics (each
with a focus on distinct aspects of explanation similarity), and three common
explanation methods. In this section, we outline the technical details
involved. In §4.1, we perform ablations to understand the impact of local
weight perturbations on explanation similarity. In §4.2 we, showcase the
merits of both methods in maximizing the expected explanation similarity
between any two ensembles constructed from $n$ samples of the
underspecification set, improvements attained without compromising test
accuracy. Our experiments were conducted in Python [38] on standard consumer-
grade CPUs,222Specifically, an 8-core Apple M1 Pro chip and a
12-core/24-thread AMD Ryzen 9 5900X processor. parallelized with `joblib`
[23].

#### Datasets

We use five publicly available financial datasets: the Home Equity Line of
Credit (HELOC) [12] and German Credit [11] datasets both measure forms of
credit risk, given input information regarding a loanee; Adult Income [11] is
used to predict whether a person’s salary exceeds a threshold of $50,000;
Default Credit [41] and Give Me Some Credit (GMSC) [14] involve predicting the
likelihood of a borrower defaulting on a loan, based on a variety of financial
and personal features. For each dataset, we encode negative outcomes (high
risk, low salary, defaults, etc.) as $y=0$, and compute explanations with
respect to the positive class, to model typical scenarios whereby users seek
advice from model explanations on how to obtain a desired outcome $y=1$.
Categorical features are one-hot encoded, and data is normalized to zero mean
and unit variance, in part to ensure that top-$k$ feature explanations are
meaningful, and not skewed towards features with smaller ranges. We fix a
split of 80% of each dataset for training, and 20% to evaluate explanations
and test accuracy. Full implementation details are provided in Appendix A.1.

#### Model selection

In our experiments, we use `PyTorch` [30] to implement feedforward neural
networks in with fixed architectures: hidden layers of size 128, 64, and 16
are used, to provide sufficient capacity for the networks to capture largely
diverse, but equally performing functional solutions. We use ReLU activations
for the intermediate layers and Softmax for the output. Model selection sweeps
are performed with `RayTune` [27], to identify the best performing model
hyperparameters for a given dataset. Optimal hyperparameters and full
implementation details are included in Appendix A.2.

To explore the underspecification set, we proceed to train 1000 models using
the optimal set of hyperparameters found, changing only the random seed that
controls the initialization of each neural network in weight space. We decide
against randomly shuffling the training data between epochs, focusing on the
effect of underspecification solely in terms of the random initialization.
Notably, we also do not employ practices such as weight decay or dropout, as
these techniques could limit the expressivity of the models and consequently
constrain the variability of explanations, thereby obscuring our ability to
test the effectiveness of ensembling strategies in aligning diverse
explanations.

#### Explanations

We experiment with three common explanation methods. Input gradients, or
Saliency [32], Smoothgrad [34], and DeepSHAP [28]. Specifically, we seek to
verify if our method offers benefits beyond those provided by Smoothgrad,
given its conceptual similarity to weight perturbation. Given the widespread
use of SHAP, particularly in financial domains, we experiment on DeepSHAP,
which uses a selection of background samples to approximate the conditional
expectations of SHAP, described in Lundberg and Lee [28]. In the interests of
compute, we limit the number of test inputs being explained with DeepSHAP to
100. We note that the same train/test samples are used for a given dataset,
and that DeepSHAP values reached convergence, eliminating any potential source
of variation in explanations that might distort the evaluation of our ensemble
methods.

#### Explanation similarity metrics

We are interested in comparing the explanations provided by two models for a
given input, and adopt existing top-$k$ metrics to cover distinct forms of
agreement. Each metric takes a pair of top-$k$ feature sets corresponding to
the two explanations.

Sign-Agreement (SA) computes the fraction of top-$k$ features that appear in
both explanations and share the same sign [24], and can be viewed as the
overlap between two top-$k$ explanations, after accounting for the direction
of features (whether they contributed positively or negatively to the model’s
prediction). We modify two other metrics taken from [7]:

Signed-Set Agreement (SSA) is binary (per-sample), and is 1 when the two
model’s top-$k$ feature sets contain the same features and the features have
the same signed value (the order of the top-$k$ features does not matter). SSA
is thus a stricter form of SA, and is satisfied if and only if SA evaluates to
1.

Consistent Direction of Contribution (CDC) is also binary (per-sample) and
requires that any feature that appears in the top-$k$ of one model has the
same signed value if it appears in the other model. CDC estimates the
likelihood of a top-$k$ explanatory feature changing its contribution
direction when switching between pairs of roughly equivalent models. Such
changes can be confusing for end users.

#### Mode connectivity implementation

We implement mode connectivity following the method outlined in Garipov et al.
[17]. This process involves training a curve between two endpoints, which,
while beneficial, introduces additional computational costs. An alternative is
to train the two endpoints from scratch, using identical hyperparameters to
those used for individual models. This approach requires the same
computational effort as independently training two models, though is often
faster due to internal parallelization. The loss for each batch is calculated
from a uniformly random point sampled along the curve, and the weights of the
endpoints are then updated via backpropagation. We confirm that this results
in endpoint weights closely aligned with those trained independently, serving
as an effective approximation of the mode connectivity process, and detail our
approach in Appendix C. Future work should strive to explore further
properties of the loss landscape in order to expedite mode connectivity,
building upon recent advancements in the field [3, 10, 17, 19, 33, 36].

Figure 4: Effect of $\sigma$ on top-k SA scores (for input gradients and
$k=5$), and test set accuracy, for an increasing number of perturbations. Left
to Right: HELOC, German Credit, and Adult Income datasets, with noise added to
first layer biases, weights, and weights, respectively. Observe how
explanation similarity may continually increase, even as test accuracy drops
(one is not a strong indicator of the other). Error bars represent the central
decile of SA scores over 1000 individuals, and the interquartile ranges of
accuracies over perturbed models.

Figure 5: Effect of $\sigma$ on top-k SA scores (for input gradients and
$k=5$) and test accuracy, over different layers perturbed 100 times. Left to
Right: HELOC, Default Credit, and GMSC. Error bars represent the central
decile of SA scores over 1000 individuals, and the interquartile range of
accuracies over perturbed models.

###  4.1 Impact of weight perturbation

We begin by performing ablations over the value of $\sigma$ used in our weight
perturbation method, conducted on 24 models sampled without replacement from
the underspecification set of 1000 pre-trained models. Each model is perturbed
a number of times, yielding 24 perturbed models (a form of ensemble generated
from just one pre-trained model, described in §3.1). We measure the SA
similarity of the top-5 gradient features between all $\binom{24}{2}$ = 276
unique pairs of perturbed models, averaging over all pairs, i.e. error bars
correspond to the first 1000 individuals in the test set. Test accuracy is
computed over all test inputs, with errors bars corresponding to the 24
perturbed models.

These experiments aim to ascertain the impact of local weight perturbations on
explanation similarity and test accuracy, thereby informing the optimal setup
for our weight perturbation method. Figure 4 indicates that perturbing the
weights or biases of the first layer with Gaussian noise
$\epsilon\sim\mathcal{N}(0,\sigma^{2})$ can lead to significant increases in
explanation similarity (measure between perturbed models). For instance, over
values of $\sigma$ where test accuracy remains approximately constant, SA
scores on HELOC increase from around 0.62 to 0.74, by perturbing just 20 times
the biases of the first layer (in general, convergence required fewer
perturbations when perturbing biases compared to weights). Furthermore, these
gains correspond to perturbing just a single model. The cumulative effect of
constructing ensembles from multiple perturbed models, discussed in §4.2,
leads to additional gains.

We also investigate the effects of perturbing individually each layer of the
network (Figure 5), finding that improvements in explanation similarity depend
critically on the layer being perturbed333For clarity, our naming convention
is such that Layer 1 refers to the weights or biases between the input and the
first hidden layer, Layer 2 refers to the weights/biases between the first and
second hidden layers, etc.. Across all datasets, we observe that the weights
or biases of the first layer tend to be most sensitive to perturbations,
providing potential for improved explanation similarity. For instance, at no
point in Figure 5 do SA scores significantly increase when perturbing
weights/biases beyond the first layer. Further investigations, including full
ablations across all datasets, are located in Appendix B.

Figure 6: Effectiveness of ensemble strategies in stabilizing explanations
across all datasets (top-5 SSA scores for input gradients). Observe the
increases in similarity between ensembles when combining global (mode
connectivity) and local (perturbation) explorations. Average and Majority
denote vanilla ensembles, while Perturb, Connect, and Combine denote weight
perturbation, mode connectivity, and their combination.

###  4.2 Comparison of ensemble techniques

In our experiments, we construct ensembles by randomly sampling 50 non-
overlapping sets of models from the underspecification set of 1000 pre-trained
models. The size of each set differs per experiment, with a maximum size of 20
pre-trained models (covering all 1000 models). For a given $k$ value and
explanation technique, we then compute similarity scores between all
$\binom{50}{2}$ = 1225 unique pairs of ensembles, averaging over all pairs.
Thus, scores represent the range of probability values that individuals have
of satisfying certain similarity criteria. For example, SSA scores estimate
the likelihood that individuals maintain identical top-$k$ features (i.e. with
the same directions, but ignoring ordering), when switching between
approximately equivalent models.

We also trial the combination of both weight perturbation (perturb) and mode
connectivity (connect), as a novel ensembling technique (denoted combine),
which involves uniformally sampling models along each mode connecting path,
and perturbing each of them. Full results across all datasets, explanation
methods, and similarity metrics are provided in Appendix D.

#### Vanilla ensembles can eliminate explanatory multiplicity, when
sufficiently large

Whether the distribution of model explanations across the underspecification
set has low or high variance, the explanations between vanilla ensembles of
sufficient size sampled from the set will eventually converge. We observe this
convergence in Figure 6. While vanilla ensembles of just 20 pre-trained models
cannot achieve perfect similarity in all cases, as the ensembles grow in size
we see a steady increase in average pairwise SSA score (the strictest
criterion). For the GMSC dataset, and vanilla ensembles (blue and orange), the
median probability of an individual retaining the exact same top-$k$ features
when switching between models increases from 50% to 95%.

#### Strategic ensembling reduces pre-training requirements

We demonstrate in Figure 6 that our ensemble strategies consistently expedite
the alignment of model explanations i.e., require fewer pre-trained models to
achieve a given similarity score. The relative performance gains that we
achieve through either local (green) or global (red) exploration often hinge
upon the unique characteristics of the given dataset, explanation technique,
or model class used. We observe that our approach of combining both forms of
exploration achieves superior performance in most cases. Notably, to increase
SSA scores to around 40% in HELOC, from 0% for single models, vanilla
ensembles required 12 pre-trained models (blue and orange). Conversely, the
combined exploration of mode connectivity and weight perturbation required
just 2 pre-trained models (purple). Similarly, the number of pre-trained
models required to surpass SSA scores of 60% on Adult Income, can be reduced
fivefold from 20 to just 4. Table 1 showcases further examples in which our
methods comprehensively improve explanation similarity across SA, SSA and CDC
metrics, for ensembles constructed with four pre-trained models. Note that the
CDC metric typically returned 1 for top-5 explanations, and is the least
strict of the three metrics. Thus, we instead consider top-$d$ CDC, computing
CDC over all $d$ features in the dataset, to stress-test the performance of
our ensembling approaches. In this case, we retain significant improvements
over vanilla ensembles.

#### Ensembling increases test accuracy

A well known property of ensembles is their ability to reduce generalization
error [9, 4], and we observe the same results in our experiments. Our value of
$\sigma$ was chosen to maximize similarity improvements, but with the
constraint that the test accuracy does not drop by more than 1%. In most
cases, our ensembling methods achieved similar test accuracies to vanilla
ensembles, with test accuracy exceeding standard ensembles by up to 1% in some
cases.

Table 1: Results of explanation alignment for Smoothgrad and DeepSHAP, across all datasets and similarity metrics. Ensembles are constructed from just four pre-trained models. Explanation agreement between single models sampled from the underspecification set are shown for reference. We report mean values alongside standard deviation. Best values are shown in bold. Dataset |  Ensemble Method | ——————– Smoothgrad ——————– | ——————— DeepSHAP ———————–
---|---|---|---
Top-5 SA | Top-5 SSA | Top-d CDC | Top-5 SA | Top-5 SSA | Top-d CDC
HELOC
(23 features) | Single | 0.65 $\pm$ 0.09 | 0.06 $\pm$ 0.06 | 0.01 $\pm$ 0.01 | 0.75 $\pm$ 0.09 | 0.18 $\pm$ 0.17 | 0.02 $\pm$ 0.02
Vanilla | 0.81 $\pm$ 0.08 | 0.25 $\pm$ 0.19 | 0.09 $\pm$ 0.09 | 0.84 $\pm$ 0.09 | 0.39 $\pm$ 0.27 | 0.09 $\pm$ 0.09
Perturb | 0.87 $\pm$ 0.06 | 0.43 $\pm$ 0.23 | 0.19 $\pm$ 0.14 | 0.86 $\pm$ 0.07 | 0.44 $\pm$ 0.23 | 0.12 $\pm$ 0.08
Connect | 0.86 $\pm$ 0.07 | 0.39 $\pm$ 0.22 | 0.15 $\pm$ 0.13 | 0.88 $\pm$ 0.08 | 0.51 $\pm$ 0.28 | 0.11 $\pm$ 0.10
Combine | 0.90 $\pm$ 0.06 | 0.55 $\pm$ 0.25 | 0.26 $\pm$ 0.17 | 0.90 $\pm$ 0.07 | 0.55 $\pm$ 0.28 | 0.14 $\pm$ 0.10
German
Credit
(70 features) | Single | 0.50 $\pm$ 0.08 | 0.01 $\pm$ 0.01 | 0.00 $\pm$ 0.00 | 0.68 $\pm$ 0.11 | 0.11 $\pm$ 0.15 | 0.00 $\pm$ 0.00
Vanilla | 0.69 $\pm$ 0.08 | 0.09 $\pm$ 0.07 | 0.00 $\pm$ 0.00 | 0.80 $\pm$ 0.09 | 0.27 $\pm$ 0.24 | 0.00 $\pm$ 0.00
Perturb | 0.73 $\pm$ 0.07 | 0.12 $\pm$ 0.08 | 0.00 $\pm$ 0.00 | 0.83 $\pm$ 0.08 | 0.32 $\pm$ 0.22 | 0.00 $\pm$ 0.00
Connect | 0.71 $\pm$ 0.12 | 0.12 $\pm$ 0.10 | 0.00 $\pm$ 0.00 | 0.77 $\pm$ 0.10 | 0.22 $\pm$ 0.21 | 0.00 $\pm$ 0.00
Combine | 0.76 $\pm$ 0.08 | 0.17 $\pm$ 0.12 | 0.00 $\pm$ 0.00 | 0.81 $\pm$ 0.08 | 0.27 $\pm$ 0.21 | 0.00 $\pm$ 0.00
Adult
Income
(6 features) | Single | 0.83 $\pm$ 0.07 | 0.29 $\pm$ 0.18 | 0.32 $\pm$ 0.15 | 0.92 $\pm$ 0.07 | 0.63 $\pm$ 0.31 | 0.51 $\pm$ 0.17
Vanilla | 0.89 $\pm$ 0.06 | 0.48 $\pm$ 0.24 | 0.50 $\pm$ 0.19 | 0.95 $\pm$ 0.06 | 0.75 $\pm$ 0.26 | 0.66 $\pm$ 0.20
Perturb | 0.90 $\pm$ 0.05 | 0.54 $\pm$ 0.24 | 0.54 $\pm$ 0.19 | 0.95 $\pm$ 0.05 | 0.77 $\pm$ 0.23 | 0.77 $\pm$ 0.19
Connect | 0.90 $\pm$ 0.07 | 0.53 $\pm$ 0.27 | 0.53 $\pm$ 0.22 | 0.96 $\pm$ 0.05 | 0.79 $\pm$ 0.24 | 0.74 $\pm$ 0.20
Combine | 0.92 $\pm$ 0.06 | 0.61 $\pm$ 0.26 | 0.58 $\pm$ 0.21 | 0.96 $\pm$ 0.04 | 0.81 $\pm$ 0.21 | 0.78 $\pm$ 0.20
Default
Credit
(90 features) | Single | 0.49 $\pm$ 0.06 | 0.00 $\pm$ 0.00 | 0.00 $\pm$ 0.00 | 0.82 $\pm$ 0.07 | 0.29 $\pm$ 0.18 | 0.00 $\pm$ 0.00
Vanilla | 0.67 $\pm$ 0.07 | 0.04 $\pm$ 0.04 | 0.01 $\pm$ 0.02 | 0.89 $\pm$ 0.07 | 0.54 $\pm$ 0.27 | 0.02 $\pm$ 0.04
Perturb | 0.72 $\pm$ 0.06 | 0.07 $\pm$ 0.07 | 0.02 $\pm$ 0.03 | 0.90 $\pm$ 0.07 | 0.54 $\pm$ 0.27 | 0.03 $\pm$ 0.04
Connect | 0.70 $\pm$ 0.07 | 0.07 $\pm$ 0.05 | 0.02 $\pm$ 0.03 | 0.91 $\pm$ 0.06 | 0.58 $\pm$ 0.25 | 0.03 $\pm$ 0.03
Combine | 0.75 $\pm$ 0.05 | 0.11 $\pm$ 0.05 | 0.03 $\pm$ 0.06 | 0.91 $\pm$ 0.06 | 0.58 $\pm$ 0.25 | 0.03 $\pm$ 0.02
GMSC
(10 features) | Single | 0.91 $\pm$ 0.03 | 0.57 $\pm$ 0.15 | 0.46 $\pm$ 0.23 | 0.85 $\pm$ 0.07 | 0.39 $\pm$ 0.21 | 0.39 $\pm$ 0.19
Vanilla | 0.95 $\pm$ 0.04 | 0.75 $\pm$ 0.19 | 0.67 $\pm$ 0.25 | 0.90 $\pm$ 0.07 | 0.60 $\pm$ 0.25 | 0.58 $\pm$ 0.22
Perturb | 0.95 $\pm$ 0.04 | 0.76 $\pm$ 0.19 | 0.69 $\pm$ 0.24 | 0.90 $\pm$ 0.07 | 0.60 $\pm$ 0.25 | 0.59 $\pm$ 0.21
Connect | 0.96 $\pm$ 0.04 | 0.82 $\pm$ 0.18 | 0.71 $\pm$ 0.25 | 0.90 $\pm$ 0.06 | 0.58 $\pm$ 0.25 | 0.62 $\pm$ 0.20
Combine | 0.97 $\pm$ 0.04 | 0.83 $\pm$ 0.18 | 0.72 $\pm$ 0.24 | 0.91 $\pm$ 0.06 | 0.62 $\pm$ 0.26 | 0.66 $\pm$ 0.21

##  5 Conclusion and limitations

Here, we tackle the challenge of providing consistent explanations for
predictive models in the presence of model indeterminacy from the
underspecification set. Motivated by local and mode-connected loss landscape
exploration, we develop two novel ensembling methods. On five benchmark
financial datasets, our methods markedly increase the consistency of model
explanations when using a fixed number of pre-trained models. Moreover, our
methods are computationally more affordable than standard ensembling
techniques with respect to the number of trained model constituents, while not
compromising on test accuracy.

However, our work is limited along some dimensions. For each dataset, we
select one set of optimal hyperparameters, though there exist many such sets
of these that can often cover a broad range. Exploring multiple
underspecification sets together is useful future work. Additionally, the
methods we provide traverse the loss landscape along two fundamental axes:
locally (with perturbations) and globally (between models). The specific
methods used for exploring both of these can be further optimized. Finally,
although our methods require no extra training compared to standard
ensembling, approaches to reduce the total number of models included in the
set should be explored to cut inference costs. A more comprehensive discussion
on the limitations and prospective avenues of study, including the exploration
of alternate explanation methods, the alignment of constituent models in
weight space through permutation symmetries [3, 33, 36], and the effects of
distillation [21] and self-distillation [15] are outlined in Appendix E.

It is important to acknowledge that our work does not explicitly consider the
broader impacts and desirable properties of these ensembles in application,
such as for fairness or bias. While our work serves as a foundation for
exploring the impact of ensembling on explanation consistency, we encourage
further research into how our techniques may interact with other aspects of AI
safety.

## References

  * Agarwal et al. [2022] C. Agarwal, S. Krishna, E. Saxena, M. Pawelczyk, N. Johnson, I. Puri, M. Zitnik, and H. Lakkaraju.  OpenXAI: Towards a transparent evaluation of model explanations.  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2022.  URL https://openreview.net/forum?id=MU2495w47rz.
  * AI-Rights [2022] AI-Rights.  Blueprint for an AI Bill of Rights, 2022.  URL https://www.whitehouse.gov/ostp/ai-bill-of-rights/.
  * Ainsworth et al. [2023] S. K. Ainsworth, J. Hayase, and S. Srinivasa.  Git re-basin: Merging models modulo permutation symmetries.  In _International Conference on Learning Representations_ , 2023.
  * Allen-Zhu and Li [2023] Z. Allen-Zhu and Y. Li.  Towards understanding ensemble, knowledge distillation and self-distillation in deep learning.  In _International Conference on Learning Representations_ , 2023.
  * Benton et al. [2021] G. W. Benton, W. J. Maddox, S. Lotfi, and A. G. Wilson.  Loss surface simplexes for mode connecting volumes and fast ensembling.  In _International Conference on Machine Learning_ , 2021.
  * Black et al. [2021] E. Black, K. Leino, and M. Fredrikson.  Selective ensembles for consistent predictions.  In _International Conference on Learning Representations_ , 2021.
  * Brunet et al. [2022] M.-E. Brunet, A. Anderson, and R. Zemel.  Implications of model indeterminacy for explanations of automated decisions.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2022.  URL https://openreview.net/pdf?id=LzbrVf-l0Xq.
  * D’Amour et al. [2022] A. D’Amour, K. Heller, D. Moldovan, B. Adlam, B. Alipanahi, A. Beutel, C. Chen, J. Deaton, J. Eisenstein, M. D. Hoffman, F. Hormozdiari, N. Houlsby, S. Hou, G. Jerfel, A. Karthikesalingam, M. Lucic, Y. Ma, C. McLean, D. Mincu, A. Mitani, A. Montanari, Z. Nado, V. Natarajan, C. Nielson, T. F. Osborne, R. Raman, K. Ramasamy, R. Sayres, J. Schrouff, M. Seneviratne, S. Sequeira, H. Suresh, V. Veitch, M. Vladymyrov, X. Wang, K. Webster, S. Yadlowsky, T. Yun, X. Zhai, and D. Sculley.  Underspecification presents challenges for credibility in modern machine learning.  _Journal of Machine Learning Research_ , 23(226):1–61, 2022.  URL http://jmlr.org/papers/v23/20-1335.html.
  * Dietterich [2000] T. G. Dietterich.  Ensemble methods in machine learning.  In _Multiple Classifier Systems_ , pages 1–15, Berlin, Heidelberg, 2000. Springer Berlin Heidelberg.  ISBN 978-3-540-45014-6.
  * Draxler et al. [2019] F. Draxler, K. Veschgini, M. Salmhofer, and F. A. Hamprecht.  Essentially no barriers in neural network energy landscape.  In _International Conference on Machine Learning_ , 2019.
  * Dua and Graff [2017] D. Dua and C. Graff.  UCI machine learning repository, 2017.  URL http://archive.ics.uci.edu/ml.  Accessed on 2023-05-16.
  * FICO [2018] FICO.  Explainable machine learning challenge, 2018.  URL https://community.fico.com/s/explainable-machine-learning-challenge?tabset-158d9=d157e.  Accessed on 2023-05-16.
  * Fort and Jastrzebski [2019] S. Fort and S. Jastrzebski.  Large scale structure of neural network loss landscapes.  In _Advances in Neural Information Processing Systems_ , volume abs/1906.04724, 2019.  URL http://arxiv.org/abs/1906.04724.
  * Freshcorn [2022] B. Freshcorn.  Give me some credit :: 2011 competition data | kaggle, 2022.  URL https://www.kaggle.com/datasets/brycecf/give-me-some-credit-dataset.  Accessed on 2023-05-16.
  * Furlanello et al. [2018] T. Furlanello, Z. Lipton, M. Tschannen, L. Itti, and A. Anandkumar.  Born again neural networks.  In J. Dy and A. Krause, editors, _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 1607–1616. PMLR, 10–15 Jul 2018.  URL https://proceedings.mlr.press/v80/furlanello18a.html.
  * Gal [2016] Y. Gal.  Uncertainty in deep learning.  _University of Cambridge_ , 1(3), 2016.
  * Garipov et al. [2018] T. Garipov, P. Izmailov, D. Podoprikhin, D. P. Vetrov, and A. G. Wilson.  Loss surfaces, mode connectivity, and fast ensembling of dnns.  In _Advances in Neural Information Processing Systems_ , 2018.
  * GDPR [2018] GDPR.  General Data Protection Regulation (GDPR), 2018.  URL https://gdpr.eu/tag/gdpr/.
  * Gotmare et al. [2018] A. Gotmare, N. S. Keskar, C. Xiong, and R. Socher.  Using mode connectivity for loss landscape analysis.  In _ICML Workshop on Modern Trends in Nonconvex Optimization for Machine Learning,_ , volume abs/1806.06977, 2018.  URL http://arxiv.org/abs/1806.06977.
  * Han et al. [2022] T. Han, S. Srinivas, and H. Lakkaraju.  Which explanation should I choose? A function approximation perspective to characterizing post hoc explanations.  In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=rTvH1_SRyXs.
  * Hinton et al. [2015] G. Hinton, O. Vinyals, and J. Dean.  Distilling the knowledge in a neural network.  In _NIPS Deep Learning and Representation Learning Workshop_ , 2015\.  URL http://arxiv.org/abs/1503.02531.
  * Izmailov et al. [2018] P. Izmailov, D. Podoprikhin, T. Garipov, D. P. Vetrov, and A. G. Wilson.  Averaging weights leads to wider optima and better generalization.  In _Conference on Uncertainty in Artificial Intelligence_ , volume abs/1803.05407, 2018.  URL http://arxiv.org/abs/1803.05407.
  * Joblib Development Team [2020] Joblib Development Team.  Joblib: running python functions as pipeline jobs, 2020.  URL https://joblib.readthedocs.io/.
  * Krishna et al. [2022] S. Krishna, T. Han, A. Gu, J. Pombra, S. Jabbari, S. Wu, and H. Lakkaraju.  The disagreement problem in explainable machine learning: A practitioner’s perspective, 2022.  URL https://arxiv.org/abs/2202.01602.
  * Lakshminarayanan et al. [2017] B. Lakshminarayanan, A. Pritzel, and C. Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  In _Advances in Neural Information Processing Systems_ , 2017.
  * Li et al. [2021] X. Li, H. Xiong, S. Huang, S. Ji, and D. Dou.  Cross-model consensus of explanations and beyond for image classification models: An empirical study, 2021.  URL https://arxiv.org/abs/2109.00707.
  * Liaw et al. [2018] R. Liaw, E. Liang, R. Nishihara, P. Moritz, J. E. Gonzalez, and I. Stoica.  Tune: A research platform for distributed model selection and training.  In _ICML Workshop on AutoML_ , 2018.
  * Lundberg and Lee [2017] S. M. Lundberg and S.-I. Lee.  A unified approach to interpreting model predictions.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf.
  * MacKay [1992] D. J. MacKay.  A practical Bayesian framework for backpropagation networks.  _Neural computation_ , 4(3):448–472, 1992.
  * Paszke et al. [2019] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Ribeiro et al. [2016] M. T. Ribeiro, S. Singh, and C. Guestrin.  "why should i trust you?": Explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’16, page 1135–1144, New York, NY, USA, 2016. Association for Computing Machinery.  ISBN 9781450342322.  doi: 10.1145/2939672.2939778.  URL https://doi.org/10.1145/2939672.2939778.
  * Simonyan et al. [2013] K. Simonyan, A. Vedaldi, and A. Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  _CoRR_ , abs/1312.6034, 2013.
  * Singh and Jaggi [2020] S. P. Singh and M. Jaggi.  Model fusion via optimal transport.  In _Advances in Neural Information Processing Systems_ , volume abs/1910.05653, 2020.  URL http://arxiv.org/abs/1910.05653.
  * Smilkov et al. [2017] D. Smilkov, N. Thorat, B. Kim, F. Viegas, and M. M. Wattenberg.  Smoothgrad: removing noise by adding noise.  In _ICML Workshop on Visualization for Deep Learning_ , Sydney, Australia, 2017.  doi: 0.48550/arXiv.1706.03825.  URL https://arxiv.org/abs/1706.03825.
  * Srivastava et al. [2014] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ , 15(56):1929–1958, 2014.  URL http://jmlr.org/papers/v15/srivastava14a.html.
  * Tatro et al. [2020] N. J. Tatro, P.-Y. Chen, P. Das, I. Melnyk, P. Sattigeri, and R. Lai.  Optimizing mode connectivity via neuron alignment.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Upadhyay et al. [2021] S. Upadhyay, S. Joshi, and H. Lakkaraju.  Towards robust and reliable algorithmic recourse.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, _Advances in Neural Information Processing Systems_ , volume 34, pages 16926–16937. Curran Associates, Inc., 2021.  URL https://proceedings.neurips.cc/paper/2021/file/8ccfb1140664a5fa63177fb6e07352f0-Paper.pdf.
  * Van Rossum and Drake Jr [1995] G. Van Rossum and F. L. Drake Jr.  _Python reference manual_.  Centrum voor Wiskunde en Informatica Amsterdam, 1995.
  * Wu et al. [2020] D. Wu, Y. Wang, and S. Xia.  Adversarial weight perturbation helps robust generalization.  In _Advances in Neural Information Processing Systems_ , volume abs/2004.05884, 2020.  URL https://arxiv.org/abs/2004.05884.
  * Yang et al. [2021] Z. Yang, L. Li, X. Xu, B. Kailkhura, T. Xie, and B. Li.  On the certified robustness for ensemble models and beyond, 2021.  URL https://arxiv.org/abs/2107.10873.
  * Yeh and Lien [2009] I.-C. Yeh and C.-h. Lien.  The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients.  _Expert Systems with Applications_ , 36(2, Part 1):2473–2480, 2009.  ISSN 0957-4174.  doi: https://doi.org/10.1016/j.eswa.2007.12.020.  URL https://www.sciencedirect.com/science/article/pii/S0957417407006719.
  * Zhao et al. [2020] P. Zhao, P. Chen, P. Das, K. N. Ramamurthy, and X. Lin.  Bridging mode connectivity in loss landscapes and adversarial robustness.  In _International Conference on Learning Representations_ , 2020.  URL https://arxiv.org/abs/2005.00060.

## Appendix

This appendix is formatted as follows.

  1. 1.

We discuss the Datasets and models used in our work in Appendix A.

  2. 2.

We discuss Weight perturbation and display full results from ablations in
Appendix B.

  3. 3.

We detail our Mode connectivity implementation in Appendix C.

  4. 4.

We display full Comparison of ensemble techniques in Appendix D.

  5. 5.

We discuss Limitations and future work in Appendix E.

Where necessary, we discuss potential limitations of our work and future
avenues for exploration.

##  Appendix A Datasets and models

Five benchmark datasets are employed in our experiments, all of which describe
binary classification and are publicly available. Details are provided in
Appendix A.1 and in Table 2. Our experiments are also conducted for a fixed
architecture, trained with hyperparameters determined through model selection
sweeps. Details are provided in Appendix A.2 and Table 3.

###  A.1 Datasets

Table 2: Summary of the datasets used in our experiments. Although German
Credit includes continuous features, we find that they have limited effect on
the model both during training and in the resulting explanations.

Name | No. Inputs | Input Dim. | Categorical | Continuous | No. Train | No. Test
---|---|---|---|---|---|---
HELOC | 9871 | 23 | 0 | 23 | 7896* | 1975*
German Credit | 999 | 70* | 17 | 3 | 799 | 200
Adult Income | 32561 | 6 | 0 | 6 | 26048 | 6513
Default Credit | 30000 | 90* | 9 | 14 | 24000 | 6000
GMSC | 11426 | 10 | 0 | 10 | 9140 | 2286

*Denotes values post-processing (one-hot encoding inputs, dropping inputs).

The HELOC (Home Equity Line of Credit) dataset [12] classifies credit risk and
can be obtained from (upon request) and is described in detail at:
https://community.fico.com/s/explainable-machine-learning-challenge. We drop
duplicate inputs, and inputs where all feature values are missing (negative),
and replace remaining missing values in the dataset with median values.

The German Credit dataset [11] classifies credit risk and can be obtained from
and is described at:
https://archive.ics.uci.edu/ml/datasets/statlog+(german+credit+data). The
documentation for this dataset also details a cost matrix, where false
positive predictions induce a higher cost than false negative predictions, but
we ignore this in model training. Note that this dataset includes mostly
categorical features, and is distinct from the common Default Credit dataset,
described below.

The Adult Income dataset [11] classifies whether an individual’s salary
exceeds $50,000, and is obtained from and described at:
https://archive.ics.uci.edu/ml/datasets/adult. We drop categorical features
for this dataset, resulting in a total of 6 features, to include an assessment
of a case where top-5 would encompass most features (n.b., we still see
significant disagreement in this case- either the signs of the 5 features
disagreed, or the least important 6${}^{\text{th}}$ feature was disagreed on).

The Default Credit dataset [41] classifies default risk on customer payments
and is obtained from and described at:
https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients. This
dataset, and German Credit, stress-test increased dimensions (number of
features).

The GMSC (Give Me Some Credit) dataset [14] classifies the probability of
default within the next two years, and is obtained from and described at:
https://www.kaggle.com/datasets/brycecf/give-me-some-credit-dataset. We pre-
process a large random sample of this dataset, ensuring a 50-50 split between
class 0 (no default) and class 1 (default). Summary of these datasets is in
Table 2.

###  A.2 Models

We use `PyTorch` [30] to implement feedforward neural networks with fixed
architectures: hidden layers of size 128, 64, and 16. We use ReLU activations
for the intermediate layers and Softmax for the output. Model selection sweeps
are performed with `RayTune` [27], to identify the best performing model
hyperparameters for a given dataset (shown in Table 3). We performed iterative
searches across learning rates from $10^{-6}$ to $10^{-1}$, batch sizes 16 to
128, and epochs typically between 5 and 100 (though only larger values where
necessary). Notably, there were a wide range of hyperparameters that yielded
near equivalent test accuracy. The investigation of ensembling techniques
should be extended to the Rashomon set for a given architecture and, further,
across various model sizes. Our research focused on the natural starting point
of analyzing a given underspecification set, where the ML pipeline is
optimized up until the choice of the random seed used in training. We employ a
filtering process to discard models that perform more than 1% below the mean
accuracy, though also recognize the importance of seed-induced variability in
the context of automated systems. If the focus is on investigating the effects
of random seeds, it could be argued that all models should be included. We
initially tested both approaches, and found the overall trends in the results
to be largely similar.

Table 3: Hyperparameters used to train our models, and model performance. Name | Epochs | Learning Rate | Batch Size | Train Acc. (%) | Test Acc. (%)
---|---|---|---|---|---
HELOC | 20 | 0.0004 | 32 | 77.97 $\pm$ 0.31 | 72.90 $\pm$ 0.45
German Credit | 100 | 0.004 | 32 | 92.81 $\pm$ 1.98 | 73.47 $\pm$ 1.47
Adult Income | 40 | 0.004 | 128 | 83.50 $\pm$ 0.15 | 82.12 $\pm$ 0.16
Default Credit | 10 | 0.0001 | 128 | 82.32 $\pm$ 0.05 | 82.67 $\pm$ 0.09
GMSC | 20 | 0.0004 | 32 | 75.17 $\pm$ 0.34 | 73.96 $\pm$ 0.34

##  Appendix B Weight perturbation

Figure 7: Effect of $\sigma$ on top-k SA scores (for input gradients and
$k=5$) and test accuracy on HELOC and German Credit datasets. Left and Center:
perturbing the first layer weights, and the first layer biases, respectively.
Right: perturbing layers individually 100 times each. Error bars represent the
central decile of SA scores over 1000 individuals, and the interquartile range
of accuracies over perturbed models.

We present here full ablations across all datasets, to identify the optimal
layer to perturb and the corresponding standard deviation $\sigma$ and number
of perturbations, depicted in Figures 7 and 8. As previously noted, the
effects of random noise perturbations depend critically on the layer being
perturbed. In most cases (bar German Credit or Adult Income), perturbing
layers deeper than the first resulted in no increases in explanation
similarity, and optimal gains were found by perturbing either the weights or
biases of the first layer (i.e. connections between the input and the first
hidden layer of size 128). We also trialled perturbing all layers
cumulatively, or perturbing layers with noise proportional to the loss
gradient of each weight, though could not identify superior results.

Figure 8: Effect of $\sigma$ on top-k SA scores (for input gradients and
$k=5$) and test accuracy on Adult Income, Default Credit and GMSC datasets.
Left and Center: perturbing the first layer weights, and the first layer
biases, respectively. Right: perturbing layers individually 100 times each.
Error bars represent the central decile of SA scores over 1000 individuals,
and the interquartile range of accuracies over perturbed models.

In our ensemble experiments, we perturb the bias of the first layer with
standard deviations 0.05 and 0.5, for GMSC and Default Credit, respectively.
We perturb the weights of the first layer with standard deviations 0.2, 0.2,
and 0.3 for HELOC, German Credit and Adult Income, respectively. We use 50
perturbations in all cases, besides HELOC where we use 100 perturbations.

##  Appendix C Mode connectivity

Figure 9: Mode connectivity approximation for Default Credit. Curve
startpoints are similarly positioned in weight space to the original models,
after training curves from scratch. Left: $\ell_{2}$ distance in weight space
between model pairs in the underspecification set. Center: $\ell_{2}$ distance
in weight space between curve startpoint pairs. There is similar distance
between pairs of curve startpoints after training curves, compared to between
standard model pairs. Right: $\ell_{2}$ distance between trained curve
startpoints and trained models is low.

This section details our mode connectivity implementation. Given two models
from the underspecification set, one can connect the two models in weight
space via paths or near constant loss. We use the publicly available code from
Garipov et al. [17] to do so. This takes two models as a fixed startpoint and
endpoint for the curve. A linear path between the startpoint and endpoint is
initialized, and a fixed number of bends lying along this path are
backpropagated in order to minimize training loss of points sampled
uniformally randomly along the curve.

While this method yielded improvements similar to those detailed in the paper,
it introduces additional training costs, since one must effectively train new
models (the bends along the curve). An alternate approach, described in the
main text, is to initialize the curve with untrained start and end points, and
train the curve from scratch. We trialled this as an effective approximation
to mode connecting process. Specifically, the startpoint of one curve is
initialized with the same random seed as used to train a given model from the
underspecification set (endpoint is initialized randomly). The curve is then
trained with the exact same hyperparameters as the original model was trained
with. The only difference is that loss is sampled along the curve, rather than
just at the startpoint. We demonstrate in Figure 9 how this resulted in the
startpoint of the curve being trained to a similar position in weight space as
the original model (i.e. $\ell_{2}$ distances are low on the right hand graph,
while distances between pairs of curve startpoints after training is similar
to distances between pairs of models after training).

Future work can consider optimizing the mode connectivity process, such that
paths can be found with reduced computational cost. Multiple prior work have
indicated that linear mode connectivity, i.e. straight-line paths between
models of near constant loss, can be achieved via permutation of the weights
used [3, 33, 36]. It is worth noting that the focus of our investigation was
to analyze how local, global, or their combined explorations of the loss
landscape could promote explanation alignment after ensembling. The exact
mechanisms used to explore the loss landscape, and their computational
efficiencies, should build on related advancements, and is a ripe area for
future work.

##  Appendix D Comparison of ensemble techniques

This Appendix displays full results, similar to those in Figure 6, but across
all similarity metrics and explanation techniques. Recall that for the CDC
metric, we aim to stress test our ensembling techniques by assessing top-$d$
similarity, where $d$ is the dimensionality of the data. This means that a
score of 1 is returned only if the signs of all features match between two
ensembles. For any particular number of pre-trained models, 50 ensembles are
constructed, and for each individual in the test set,

Figure 10: Effectiveness of ensemble strategies in stabilizing explanations
across all datasets (top-5 SA, top-5 SSA, and top-$d$ CDC scores for input
gradients). Average and Majority denote vanilla ensembles, while Perturb,
Connect, and Combine denote weight perturbation, mode connectivity, and their
combination respectively.

average similarities are computed across all $\binom{50}{2}$ = 1225 unique
pairs. The median individual similarity is plotted, and error bars indicated
the central decile of individuals.

#### Effects of ensembling on Saliency

Figure 10 details full results for input gradients i.e. Saliency. The central
row is depicted in the main text (Figure 6). Observe similar trends between SA
scores (top row) and SSA scores (central row), as the latter is a stricter
(binary) version of the former. Top-$d$ CDC comparisons demonstrate similar
trends. For instance, to exceed 80% similarity on the GMSC dataset, standard
ensembling (blue and orange) requires around 20 pre-trained models, while the
combined exploration of mode connectivity alongside local perturbations
(purple) requires 4 pre-trained models.

#### Effects of ensembling on Smoothgrad

Figure 11 details full results for Smoothgrad, where we use 50 perturbations
on the input with standard deviation 0.1. Note that this explanation technique
generally improves explanation similarity slightly for standard ensembles
(blue and orange), e.g., for 20 pre-trained models, median SSA scores on Adult
Income increase from 60% in Figure 10 to 70% in Figure 11. However, our
findings suggest that Smoothgrad does not provide significant smoothing
benefits for explanation alignment beyond our ensemble techniques.
Specifically, for the same comparison of median SSA scores on Adult Income,
when considering 20 pre-trained models, both Smoothgrad and saliency
techniques achieve approximately 90% explanation alignment using the combined
approach, while local perturbations or mode connectivity yield around 80%
alignment. Furthermore, while Smoothgrad provides an explanation for a
smoothed approximation of a potentially noisy decision boundary, ensemble
approaches smooth both the explanation and the model’s decision region.

Figure 11: Effectiveness of ensemble strategies in stabilizing explanations
across all datasets (top-5 SA, top-5 SSA, and top-$d$ CDC scores for
Smoothgrad). Average and Majority denote vanilla ensembles, while Perturb,
Connect, and Combine denote weight perturbation, mode connectivity, and their
combination respectively.

#### Effects of ensembling on DeepSHAP

Figure 12 details full results for DeepSHAP, an approximation to SHAP that
leverages knowledge from the neural network directly. Note first that results
may be slightly noisier, given the computational restriction of using this
method (the first 100 test points in each datasets are evaluated, rather than
the first 1000 as in other methods). In spite of this, the various ensembling
techniques proposed, for a given number of pre-trained models, tend to yield
benefits similar to previously e.g. HELOC trends are largely similar after
accounting for noise fluctuations. SA and SSA scores for DeepSHAP tended to
have higher baseline similarity scores (i.e. between single models) compared
to Saliency and Smoothgrad. We see notable improvements for the mode
connected/combined explorations for Adult Income, Default Credit and GMSC.
However, we also observe the one instance where mode connectivity was inferior
to standard ensembling, in German Credit. While this was overcome through the
combined technique of sampling along connected modes and further perturbing
each of the samples (purple), and secondly that local weight perturbation
alone improved performance, this emphasizes the unpredictable nature of
particular explanation techniques, and warrants future research. A
sufficiently large number of pre-trained models may eliminate explanation
multiplicity, and so the focus of future work should be on exploiting further
the properties of the loss landscape to optimize the local and global searches
for ensemble members.

Figure 12: Effectiveness of ensemble strategies in stabilizing explanations
across all datasets (top-5 SA, top-5 SSA, and top-$d$ CDC scores for
DeepSHAP). Average and Majority denote vanilla ensembles, while Perturb,
Connect, and Combine denote weight perturbation, mode connectivity, and their
combination respectively.

##  Appendix E Limitations and future work

This work presents a novel approach to enhancing the consistency of model
explanations by leveraging ensembling methods based on loss landscape
exploration. While we observe promising results, we acknowledge that several
interesting avenues for future work emerge from our study.

#### Underspecification set

In this work, we define the underspecification set as the collection of
optimal models trained with a fixed set of hyperparameters, with the only
source of variation due to the random seed using in training. By focusing on
the underspecification set, we aim to shed light on the intrinsic
inconsistency of model explanations arising purely from indeterminacy within a
specific model configuration, rather than across different configurations.

However, we acknowledge that in real-world scenarios, an exact
underspecification set may not always be found. Often, there are multiple sets
of near-optimal hyperparameters, each potentially giving rise to a different
underspecification set. The focus on one such set in our study is intentional,
as it provides a clearer landscape for assessing the effectiveness of ensemble
techniques in handling model indeterminacy. It is important to contextualize
that addressing the complexities within the underspecification set is a
necessary first step. Without a sound understanding of the explanatory
behavior within a specific model configuration, attempting to reconcile
explanations across broader ranges – such as the entire Rashomon set
encompassing different model classes and hyperparameters – would be overly
ambitious. Thus, our focus on the underspecification set offers an essential
foundation for further research in the quest for more consistent model
explanations.

Looking forward, an interesting direction for future work is the exploration
of multiple underspecification sets concurrently. Comparing and consolidating
explanations across multiple underspecification sets may present additional
challenges but also provide further insights into the behavior of model
explanations. This exploration would segue naturally into investigating the
Rashomon set. Differing from the underspecification set, which captures the
variation within a specific model configuration, a comprehensive exploration
of the Rashomon set would entail an expanded understanding of the variance of
model explanations.

#### Alternate explanation methods

We have chosen to use gradient-based methods for generating explanations in
this study due to their simplicity, computational efficiency, and intuitive
appeal. The gradient of a model with respect to its inputs can provide
valuable insights about how the model responds to changes in those inputs. In
essence, it can inform us about the local sensitivity of the model’s
predictions and thus can be interpreted as a form of local explanation method.
Moreover, gradients can be seen as a natural proxy for counterfactual
explanations (CEs), another popular category of explanation methods. CEs
identify the minimal changes required in the input features to achieve a
different prediction outcome. Because gradients indicate the direction of the
greatest change in the model’s output, they can be seen as giving a first
approximation to CE directions.

However, it is worth noting that CEs, while providing a powerful intuitive
appeal, come with their own set of challenges. They often require more
computation than gradient-based methods, and they can be sensitive to the
specific definition of “minimal change”, which can depend on domain-specific
factors. While we have focused on gradient-based explanations for their
straightforwardness and direct relation to counterfactual reasoning, the field
is open for further investigation and exploration of alternative methods for
enhancing the consistency of model explanations.

We acknowledge that there is a wide array of explanation methods available,
each with its unique advantages and limitations, and many of which could be
considered in the context of our framework. Future work could explore other
types of explanations, including counterfactual methods, or prototype-based
methods, among others. Each of these could provide different perspectives on
the consistency of model explanations and their susceptibility to model
indeterminacy.

#### Weight perturbations and mode connectivity

We employ two strategies for navigating the loss landscape: local exploration
via weight perturbations and global exploration via mode connectivity.

Weight perturbations could be perceived as the ‘safer’ strategy, presenting a
lower risk profile (dependent on the standard deviation of perturbations) but
potentially hitting a performance ceiling based on dataset and model class
(Section 4.1 and Appendix B). The performance of this method can be further
optimized by considering a selection process for the perturbations, possibly
based on training accuracy, or by adjusting the sampling strategy for the
perturbation magnitude, taking into account the hyperspherical distribution of
samples in high-dimensional weight space. On the other hand, mode connectivity
may be perceived as ‘riskier’ due to its broader scope but could yield
benefits that go beyond the reach of simple local exploration, allowing us to
explore more distant regions of the loss landscape, connecting different
locally optimal solutions.

Looking ahead, there is much to learn and optimize about these methods. Future
work would include both refining the methods of weight perturbation, and
exploring alternate paths for mode connectivity [3, 19, 33, 36, 42], in order
to develop better heuristics for navigating the loss landscape. The ultimate
aim is to devise more effective strategies for ensemble creation that balance
the goals of model performance, explanation consistency, and computational
efficiency.

#### Improving inference efficiency

Although our methods require no extra training compared to standard
ensembling, approaches to reduce the total number of models included in the
set should be explored to cut inference costs.444Note additionally that our
implementations may not be fully optimized with respect to parallelization,
etc. For instance, using ensembles with 10 pre-trained models each perturbed
50 times totals 500 models. While our methods offer computational efficiency
compared to standard ensembling techniques with respect to the number of pre-
trained models required, the practicality of this approach might still be
challenging for larger scale applications. Future work could focus on finding
efficient ways to reduce the total number of models in the ensemble. This
could involve techniques such as weight permutations to align models and
subsequently averaging the weights, or investigating methods to find a single
weight configuration that matches the output of ensembles, which we detail in
the following two paragraphs.

#### Exploring permutation symmetries

The initial investigations provided in this work could greatly benefit from
the exploration of permutation symmetries. This emerging direction in research
promises a more efficient traversal of the underspecification set [3, 33, 36].
The strategy involves aligning constituent models in weight space through
permutation symmetries, thereby reducing the complexity of exploring numerous
paths in the loss landscape. Through this approach, we hope to optimize the
exploration process, enabling faster and more effective generation of
ensembles. Not only would this strategy potentially reduce computational
demands, it may also lead to uncovering more consistent explanations across
ensembles, and aid in enhancing our understanding of model indeterminacy.

#### Consolidating ensemble models

The concept of consolidating ensembles into a single point in weight space
stands as another promising direction for future work. The intention behind
this proposed direction is to reap the benefits of ensemble modeling
(explanation alignment, improved predictive performance, robustness against
overfitting or dataset shift, etc.), while reducing the computational cost
associated with operating large numbers of constituent models. Methods such as
weight averaging [22] or model fusion [33] could potentially achieve this
goal.

Furthermore, examining the effects of distillation and self-distillation could
provide additional insights into model indeterminacy and the consistency of
model explanations. Distillation techniques aim to compress the knowledge of
an ensemble into a single, often simpler, model. Analyzing the impact of these
techniques on the consistency and quality of explanations is an exciting
prospect for future investigations. A connection to Bayesian methods might
also be made, which inherently capture model uncertainty and can provide a
measure of variability in single modes. However, these methods often face
challenges related to computational efficiency and robustness to dataset
shift. By contrast, ensemble methods can potentially offer a more widely
accepted and practical alternative. The balance between ensemble techniques
and Bayesian methods and their respective impacts on explanation consistency
is worth considering.

#### Broader impact

The presented ensembling strategies for improving explanation consistency,
while not explicitly designed to address issues of fairness or bias in AI
models, have the potential for significant societal impact. It is expected
that the insights gained from our methods will be beneficial for practitioners
seeking to construct ML models that provide reliable explanations, which is
especially crucial in fields where decision-making based on model outputs has
high-stakes consequences, such as healthcare, finance, and criminal justice.

While the methods we showcase make strides in explanation consistency, they do
not explicitly handle the problem of fairness and bias in AI, a concern that
has been highlighted in numerous studies. Model explanations that are
consistent across different models might still be biased or unfair if the
underlying models or data exhibit these issues. Thus, it’s important to
supplement our techniques with methods explicitly designed to test for and
mitigate bias and unfairness in AI models.

Moreover, our approach does not guarantee perfect explanation consistency, and
there may be cases where explanations between models or different runs might
still vary to a certain degree. This could have implications in scenarios
where consistent explanations are particularly important, such as in the
application of machine learning models in legal or healthcare contexts.

Given these broader impacts, it’s crucial for practitioners applying our
methods to be aware of these considerations. We recommend combining our
ensembling techniques with existing and future fairness-aware methods and
robust validation procedures to ensure comprehensive analysis of model
explanations. As this field evolves, we encourage future research on how to
effectively integrate consideration of explanation consistency, fairness, and
bias in AI model development and evaluation.

#### Closing remarks

In conclusion, our work provides a foundation to pave the way for future
research directions towards leveraging neural network research to improve our
knowledge of explanations amidst model indeterminacy. Although ensemble
methods feature prominently in the neural network analysis literature, their
application with respect to model explanations has received very little
attention. As stated, advances in loss landscape understanding and
implications of model indeterminacy have emerged as recent, yet distinct
developments, with the two fields existing almost in parallel. We emphasize
once more that the aim of this work is to serve as a catalyst for their
convergence, and bring about a unified exploration of both areas. Through
initial alignment of these directions, we have shed light on the interplay
between model indeterminacy and the consistency of model explanations.

Moving forward, a rich and promising set of research directions awaits.
Further exploration of the loss landscape, particularly in the pursuit of
expedited mode connectivity, or distillation techniques, stands as an
intriguing prospect. These directions would build on recent advancements in
the field and potentially lead to more effective and reliable model
explanations. Additionally, the application of permutation symmetries to align
model weights prior to averaging is another promising direction. Each of these
techniques may not only improve computational efficiency but also enhance our
understanding of the loss landscape and its relationship with model
explanations. Such strategies are, in essence, leveraging the mathematical
structure inherent in neural networks to expedite the exploration of the
underspecification set and promote explanation consistency.

We believe that the pursuit of these directions will continue to push the
frontiers of our understanding of explanation consistency in AI models,
contributing positively to the development of reliable, trustworthy, and
interpretable artificial intelligence systems.