# When and How to Fool Explainable Models
(and Humans) with Adversarial Examples

Jon Vadillo
University of the Basque Country UPV/EHU
Spain, 20018
<EMAIL_ADDRESS>
&Roberto Santana
University of the Basque Country UPV/EHU
Spain, 20018
<EMAIL_ADDRESS>
Jose A. Lozano
University of the Basque Country UPV/EHU
Spain, 20018
<EMAIL_ADDRESS>
Basque Center for Applied Mathematics (BCAM)
Spain, 48009
<EMAIL_ADDRESS>

###### Abstract

Reliable deployment of machine learning models such as neural networks
continues to be challenging due to several limitations. Some of the main
shortcomings are the lack of interpretability and the lack of robustness
against adversarial examples or out-of-distribution inputs. In this paper, we
explore the possibilities and limits of adversarial attacks for explainable
machine learning models. First, we extend the notion of adversarial examples
to fit in explainable machine learning scenarios, in which the inputs, the
output classifications and the explanations of the model’s decisions are
assessed by humans. Next, we propose a comprehensive framework to study
whether (and how) adversarial examples can be generated for explainable models
under human assessment, introducing novel attack paradigms. In particular, our
framework considers a wide range of relevant (yet often ignored) factors such
as the type of problem, the user expertise or the objective of the
explanations in order to identify the attack strategies that should be adopted
in each scenario to successfully deceive the model (and the human). These
contributions intend to serve as a basis for a more rigorous and realistic
study of adversarial examples in the field of explainable machine learning.

##  1 Introduction

Deep neural networks still face several weaknesses that hamper the development
and deployment of these technologies, despite their outstanding and ever-
increasing capacity to solve complex artificial intelligence problems. One of
the main shortcoming is their black-box nature, which prevents analyzing and
understanding their reasoning process, while such a requirement is
increasingly demanded in order to guarantee a reliable and transparent use of
artificial intelligence. To overcome this limitation, different strategies
have been proposed in the literature [1], ranging from post-hoc explanation
methods (which try to identify the parts, elements or concepts in the inputs
that most affect the decisions of trained models) [2, 3, 4, 5], to more
proactive approaches which pursue a transparent reasoning by training
inherently interpretable models [6, 7, 8, 9, 10, 11].

Another issue that threatens the reliability of deep neural networks is their
low robustness to adversarial examples [12], which, indeed, can be seen as a
direct implication of their lack of human-like reasoning. Therefore, improving
the explainability of the models is as well a promising direction to achieve
adversarial robustness, hypothesis which is supported by recent works which
show that interpretability and robustness are connected [13, 14, 15, 16]. At
the same time, the assumption of explainable classification models introduces
new questions regarding the definition of adversarial examples: _can
adversarial examples be deployed if humans observe not only the input but also
the output classification or the corresponding explanation?_

In this paper, we shed light on this question by incrementally extending the
notion of adversarial examples for explainable machine learning scenarios, in
which humans can not only assess the input sample, but also compare it to the
output of the model. These extended notions of adversarial examples allow us
to exhaustively analyze the possible attacks that can be produced by means of
adversarially changing the model’s classification and explanations, either
jointly or independently (that is, changing the explanation without altering
the output class, or vice versa). Our analysis leads to a comprehensive
framework that establishes whether (and how) adversarial attacks can be
generated for explainable models under human supervision. Moreover, we
thoroughly describe the requirements that adversarial examples should satisfy
in order to be able to mislead an explainable model (and even a human)
depending on multiple scenarios or factors which, despite their relevance, are
often overlooked in the literature of adversarial examples for explainable
models, such as the expertise of the user or the objective of the explanation.
All these contributions are intended to establish a basis for a more rigorous
study of the vulnerabilities of explainable machine learning in adversarial
scenarios.

##  2 Related work

###  2.1 Overview of explanation methods

In this section, we summarize the explanation methods proposed in the
literature, in order to present the terminology and taxonomy that will be used
in the subsequent sections to develop our analytical framework on adversarial
examples in explainable models.

####  2.1.1 Scope, objective and impact of the explanations

The objective of an explanation is to justify the behavior of a model in a way
that is easily understandable to humans. However, different users might be
interested in different aspects of the model, and, therefore, the explanations
can be generated for different scopes or objectives.

Overall, the scope of an explanation can be categorized as local or global
[1]. On the one hand, local methods aim to characterize or explain the model’s
prediction for each particular input individually, for example, by identifying
the most relevant parts or features of each input. On the other hand, global
methods attempt to expose the general reasoning process of the model, for
instance, summarizing (e.g., using a more simple but interpretable model) when
a certain class will be predicted, or describing to what extent a particular
input-feature is related to one class. Since in this paper we will address the
vulnerability of explainable models to adversarial examples, we we will focus
on local methods.

In addition, explanations can be used, even for the same model, for different
functional purposes. For instance, users querying the model for a credit loan
might be interested in explaining the output obtained for their particular
cases only, whereas a developer might be interested in discovering why that
model misclassifies certain input samples. At the same time, an analyst can be
interested in whether that model is biased against a social group for
unethical reasons. At a higher level, all these purposes are based on
necessities involving ethics, safety or knowledge acquisition, among others
[17]. Based on the functional purpose of the explanations and the particular
problem, domain or scenario in which they are required, another relevant
factor should be taken into consideration: the impact of the explanations,
which can be defined as the consequence of the decisions made based on the
analysis of the explanation. Healthcare domains are clear examples in which
the consequences of the decisions can be severe.

Despite the relevance of these factors, they are often overlooked when local
explanation methods are designed or evaluated [17, 18]. The same happens for
adversarial attacks in explainable models. We argue that the context, the
functional purpose and the impact of explanations should be key factors when
designing adversarial attacks against explainable models, since a different
attack strategy needs to be adopted in each context to successfully deceive
the model (and the human). This will be discussed in detail in Section 3.

####  2.1.2 Types of explanations

Different types of explanations exist depending on how the explanation is
conveyed:

##### Feature-based explanations:

assign an importance score to each feature in the input, based on their
relevance for the output classification. Common feature-based explanations
(specially in the image domain) are activation or saliency maps, which
highlight the most relevant parts of the input. Other approaches are based on
perturbation-based methods, which alter an input (for instance, by removing or
masking some regions) to assess which parts are the most relevant ones for the
prediction. Although feature-based explanations such as saliency maps are the
most employed ones in the literature (particularly in the image domain),
previous works have identified that such explanations can be unreliable and
misleading [4, 8, 19, 20, 21, 22].

##### Example-based explanations:

the explanation is based on comparing the similarity between the input at hand
and a set of prototypical inputs that are representative of the predicted
class. Thus, the classification of a given input sample is justified by the
similarity between it and the prototypes of the predicted class. We will refer
to these types of explanations as prototype-based explanations in the paper,
although different forms of example-based explanation exist, such as the
strategy proposed in [23], where influence functions are employed to estimate
the training images most responsible for a prediction. Recent works have
integrated prototype-based explanations directly in the learning process of
neural networks, so that the classification is based on the similarities
between the input and a set of prototypes [6, 7, 8, 9], achieving a more
interpretable reasoning. The prototypes can represent an entire input
describing one class (e.g., a prototypical handwritten digit "1" in digit
classification) [6], or represent image-parts or semantic concepts [7, 9, 8].

##### Rule-based explanations:

these explanation methods aim to expose the reasoning of a model in a
simplified or human-understandable set of rules, such as logic-rules or if-
then-else rules, which represent a natural form of explanations for humans.
Rule-based explanations are particularly well-suited when the input contains
features which are easily interpretable. In this paper, special attention will
be given to counterfactual explanations [24]. Although counterfactual
explanations can be considered, in their form, as rule-based explanation, we
will consider them separately in some cases due to their conditional or
hypothetical reasoning nature, as the aim is suggesting the possible changes
that should happen in the input to receive a different (and frequently more
positive) output classification (e.g., “a rejected loan request would be
accepted if the subject would have a higher income”).

Overall, the most suitable type of explanation depends on the domain, the
scope and the purpose of the explanation, as well as on the expertise level of
the users querying the model. We refer the reader to [1, 17, 25] for a more
fine-grained overview of explanation methods.

###  2.2 Reliability of explanations under adversarial attacks

Some explanation methods in the literature have been proven to be unreliable
in adversarial settings. In [26, 27, 28, 29, 30], it is shown that small
changes in input samples can produce drastic changes in feature-importance
explanations, while maintaining the output classification. In [26], the
proposed attacks are also evaluated in the example-based explanations proposed
in [23], based on estimating the relevance of each training image for a given
prediction by using influence-functions. In [31], adversarial attacks capable
of changing the explanations while maintaining the outputs are created for
self-explainable (prototype-based) classifiers. In [29, 30], it is shown that
adversarial examples can also produce wrong outputs and (feature-importance)
explanations at the same time, or change the output while maintaining the
explanations [29].

The authors of [32] show that trustworthy explanations can be produced for a
biased or an untrustworthy model, thus manipulating user trust. This approach
is, however, not based on adversarial attacks, as they focus on producing a
global explanation model that closely approximates the original (black-box)
model but which employs trustworthy features instead of sensitive or
discriminatory features (which are actually being used by the original model
to predict). Similarly, in [33, 34] adversarial models are generated, capable
of producing incorrect or misleading explanations whithout harming their
predictive performance. In [33], a fine-tuning procedure is proposed to
adversarially manipulate models, so that saliency map based explanations
drastically change (becoming ineffective in highlighting the relevant regions)
whereas the accuracy of the model is maintained.

Some works have also tried to justify the vulnerability of explanation methods
to adversarial attacks, or the links between them. In [26, 27], the non-smooth
geometry of decision boundaries (of complex models) is blamed, arguing that,
due to these properties, small changes in the inputs implies that the
direction of the gradients (i.e., normal to the decision boundary) can
drastically change. As most explanation methods rely on gradient information,
the change in the gradient direction implies a different explanation. In [29,
30], the vulnerability is attributed to a gap between predictions and
explanations. It is an open question whether this hypothesis holds for self-
explainable models, which have been trained jointly to classify accurately and
to provide explanations. Finally, theoretical connections between explanations
and adversarial examples are established in [13, 35].

###  2.3 Further connections between adversarial examples and interpretability

Paradoxically, using explanations to support or justify the prediction of a
model can imply security breaches, as they might reveal sensitive information
[36]. For instance, an adversary can use explanations of how a black-box model
works (e.g., what features are most relevant in a prediction) in order to
design more effective attacks. Similarly, in this paper we will show that
justifying the classification of the model with an explanation makes it
possible to generate types of deception using adversarial examples that,
without explanations, it would not be possible to generate (e.g., to convince
an expert that a misclassification of the model is correct).

On another note, recent works have shown that robust (e.g., adversarially
trained) models are more interpretable [13, 14, 15, 16]. In [13], this is
justified by showing that the farther the inputs are with respect to the
decision boundaries, the more aligned are the inputs with their saliency maps,
thus, being more interpretable.

Finally, the commonalities between interpretation methods and adversarial
attacks and defenses are analyzed in [37], showing how adversarial methods can
be reinterpreted from an interpretation perspective, and discussing how
techniques from one field can bring advances into the other. Our paper,
however, addresses a different objective. In contrast to [37], which focuses
on highlighting the similarities between particular methods from both
(explainable and adversarial machine learning) fields, in this paper we
propose a comprehensive framework to study if (and how) adversarial examples
can be generated for explainable models under human assessment.

##  3 Extending adversarial examples for explainable machine learning
scenarios

In this section, we extend the notion of adversarial examples to fit in
explainable machine learning contexts. For this purpose, we start from a basic
definition of adversarial examples, and incrementally discuss more
comprehensive scenarios in which the human subjects judge not only the input
sample, but also the model’s decisions, including the output classification
and the corresponding explanations. To the best of our knowledge, no prior
work has addressed comprehensively this type of generalization of adversarial
examples.

This extended definition allows us to provide a general framework that
identifies the way in which an adversary should design an adversarial example
to deploy effective attacks even when a human is assessing the prediction
process. The introduced framework also identifies the most effective ways of
deploying attacks depending on factors such as the type of problem or the way
in which the explanation is conveyed. From an adversary perspective, this
framework provides a comprehensive road map for the design of malicious
attacks in realistic scenarios involving explainable models and a human
assessment of the predictions. From a developer or a defender perspective,
this road map helps to identify the most critical requirements that their
explainable model should satisfy to be reliable.

###  3.1 Scenarios in which human subjects are aware of the model predictions

Regular adversarial attacks are based on the assumption that an adversary can
introduce a perturbation into an input sample, so that:

  1. 1.

The perturbation is not noticeable to humans, and, therefore, the ground-truth
class of the perturbed input does not change.

  2. 2.

The class predicted by a machine learning model changes.

Note that, according to this general definition of adversarial examples, the
human criterion is only considered regarding the input sample, without any
human assessment of the model’s output. Maintaining this assumption, then the
same definition of adversarial examples can be applied in the context of
explainable machine learning. However, this assumption is not realistic, as
the point of explainable models is to assess not only their classification but
also their reasoning. For these reasons, the following question arises: _are
regular adversarial examples useful in practice when the user is aware of the
output?_

To address this question, we start by discussing four different scenarios,
based on the agreement of the following factors: $f(x)$, the model’s
prediction of the input; $h(x)$, the classification performed by a human
subject; and $y_{x}$, the ground-truth class of an input $x$ (which will be
unknown for both the model and the human subject in the prediction phase of
the model). For clarification, we assume that a human misclassification
($h(x)\neq y_{x}$) can occur in scenarios in which the addressed task is of
high complexity (e.g., medical diagnosis) or the label of an input is
ambiguous (e.g., sentiment analysis). Although a human misclassification might
be uncommon in simple problems such as object recognition, even in such cases
ambiguous or challenging inputs can be found. Finally, unless specified, we
will assume expert subjects, that is, subjects with knowledge in the task and
capable of providing well-founded classifications.111Different degrees of
expertise can be considered for a more comprehensive scenario, such as
unskilled subjects, or partially skilled subjects capable of providing basic
judgments about the input (for instance, a subject might not be able to
visually discriminate different species of reptiles, yet be able to visually
classify an animal as a reptile and not as another animal class).

According this framework, the four possible scenarios are as follows:

  1. 1.

Both the model and the human subject agree with the ground-truth class of the
input, that is: $f(x)=h(x)=y_{x}$.

  2. 2.

The model has misclassified the input whereas it is well classified by the
human subject: $f(x)\neq h(x)=y_{x}$. This is, indeed, the case produced when
an adversarial example is classified, as will be described below.

  3. 3.

The human observer (incorrectly) disagrees with the model’s prediction, which
is correct: $h(x)\neq f(x)=y_{x}$.

  4. 4.

Both the model and the human subject wrongly classify the input: $f(x)\neq
y_{x}\ \wedge\ h(x)\neq y_{x}$ If $h(x)=f(x)$, the model’s prediction will be
(wrongly) considered as correct according to the human criterion. Otherwise,
the model’s output will be considered incorrect, but for wrong reasons.

According to the described casuistry, adversarial attacks aim to produce the
second scenario (i.e., ${f(x)\neq h(x)=y_{x}}$), by imperceptibly perturbing
an input $x_{0}$ that satisfies $f(x_{0})=h(x_{0})=y_{x_{0}}$ (i.e., the first
scenario) so that the model’s output is changed, but without altering the
human perception of the input (what, therefore, implies
$h(x)=y_{x}=y_{x_{0}}$). However, assuming that the user is aware of the
output, the fulfillment of the attack is subject to whether human subjects can
correct the detected misclassification, or have control over the implications
of that prediction. For example, an adversarial traffic signal will only
produce a dramatic consequence in autonomous cars if the drivers do not take
the control with sufficient promptness.

Regarding the remaining cases, they do not fit in the definition of an
adversarial attack since either the input is misclassified by the human
subject ($h(x)\neq y_{x}$) or the model is not fooled (${f(x)=h(x)=y_{x}}$).
Nevertheless, assuming a more general definition, scenarios involving human
misclassifications could be potentially interesting for an adversary.
Similarly to regular adversarial attacks, which force the second scenario
departing from the first one, an adversary might be interested in forcing the
fourth scenario departing from the third one. Let us take as an example a
complex computer-aided diagnosis task through medical images, in which an
expert subject fails in its diagnosis while the model is correct. In such
cases, we can induce a human error confirmation attack by forcing the model to
confirm the (wrong) medical diagnosis produced by the expert, that is, forcing
$f(x)=h(x)\neq y_{x}$.

Based on the above discussion, we can determine that some types of adversarial
attacks can still be effective even when the user is aware of the output.
Nonetheless, paradoxically, it is possible to introduce new types of
adversarial attacks when the output classification is supported by
explanations, as we show in the following section.

###  3.2 Scenarios in which human subjects are aware of the explanations

The scenarios described in the previous section can be further extended for
the case of explainable machine learning models, as the explanations for the
predictions come into play. As a consequence, each of the cases defined above
can be subdivided into new subcases depending on whether the explanations
match the output class or whether humans agree with the model’s explanations.
To avoid an exhaustive enumeration of all the possible scenarios, we focus
only on the ones that we identify as interesting from an adversary
perspective. From this standpoint, given an explainable model, adversarial
examples can be generated by perturbing a well classified input (for which the
corresponding explanation is also correct and coherent) with the aim of
changing the output class, the provided explanation or both at the same time.

To formalize these scenarios, let us denote $A_{f}(x)$ the explanation
provided to characterize the decision $f(x)$ of a machine learning model, and
$A_{h}(x)$ the explanation provided by a human-subject according to its
knowledge or criteria. The agreement of a human subject with the explanation
provided by the model will be denoted as $A_{f}(x)\approx A_{h}(x)$ (that is,
the model’s explanation is coherent for a human), whereas the disagreement
will be denoted as $A_{f}(x)\neq A_{h}(x)$. Similarly, we will denote
$A(x)\sim y_{x}$ if an explanation $A(x)$ is consistent with the reasons that
characterize the ground-truth class of the input (that is, if the explanation
correctly characterizes or supports the classification). For simplification,
unless specified, we assume that $h(x)=y_{x}$ and that $A_{h}(x)\sim y_{x}$,
this is, the human classification of an input into one class is correct and is
based on reasons consistent for that class.

The identified scenarios are as follows:

  * A.1

$f(x)=y_{x}\ \wedge\ A_{f}(x)\neq A_{h}(x)$. In this case, the model is right
but the explanations are incorrect or differ from the ones that would be
provided by a human. Adversarial attacks capable of producing such scenarios
have been studied in recent works for post-hoc feature-importance explanations
[26, 27, 28, 29, 30] and for self-explainable prototype-base classifiers [31],
showing that small perturbations can produce a drastic change in the
explanations without changing the output.

    * A.1.1

More particularly, we can imagine a scenario in which $A_{f}(x)\sim y_{x}$
despite ${A_{f}(x)\neq A_{h}(x)}$, for instance, if the explanations point out
to relevant and coherent properties to classify the input as $y_{x}$ (at least
partially), but which do not compose the most relevant or correct explanation
(with respect to the given input) according to a human criterion. From an
adversary’s perspective, changing the explanations without forcing a wrong
classification allows to introduce confusing recommendations. For
illustration, a model can (correctly) reject a loan request but accompanied
with a wrong explanation, preventing the applicant from correcting the
actually relevant deficiencies of the request. Similarly, a wrong explanation
of a medical diagnosis system might lead to a wrong treatment or prescription.
In addition, biased or discriminative explanations could be produced with this
attack scheme, for instance, attributing a loan rejection to sensitive
features (e.g., racial or religious). Such an explanation could make the
models look unreliable or untrustworthy for users. Oppositely, biases could be
hided by producing trustworthy explanations to manipulate user trust [32].

  * A.2

$f(x)\neq y_{x}\ \wedge\ A_{f}(x)\neq A_{h}(x)$. In this case both the
classification and the explanation provided by the model are incorrect.
Adversarial attacks capable of producing such scenarios have been investigated
in recent works [29, 30]. More particularly, we identify two specific sub-
cases as relevant when a human assesses the entire classification process:

    * A.2.1

$A_{f}(x)\sim f(x)$. In this case, the fact that the provided explanation is
coherent with the (incorrectly) predicted class can increase the confidence of
the human in the prediction, being therefore interesting from an adversary
perspective. We identify this case as the most direct extension of adversarial
examples for explainable models, as the model is not only fooled but also
supports its own misclassification with the explanation.

    * A.2.2

$A_{f}(x)\not\sim f(x)$. This case is similar to the previous one (A.2.1),
with the important difference that the model’s explanation is now coherent
with a class $y^{\prime}$ different to $f(x)$ and $y_{x}$ (assuming that
$A_{h}(x)\sim y_{x}$, then $A_{f}(x)\neq A_{h}(x)\Rightarrow A_{f}(x)\not\sim
y_{x}$). Thus, we are in a scenario in which a total mismatch is produced
between all the considered factors. Whereas these attacks are an interesting
case of study, they are also the most challenging to be deployed in practice
without the inconsistencies being noticed.

  * A.3

$f(x)\neq y_{x}\ \wedge\ A_{f}(x)\approx A_{h}(x)\ \wedge\ A_{f}(x)\sim
y_{x}$. In this case, the model’s classification is wrong but the provided
explanations are coherent from a human perspective with respect to the ground-
truth class $y_{x}$. The agreement in the explanations can increase the
confidence in the model, but, at the same time, the output is not consistent
with the explanation. However, the consistency issue might be solved by
finding an input for which the explanation not only satisfies $A_{f}(x)\sim
y_{x}$ but also $A_{f}(x)\sim f(x)$, for instance, by finding an ambiguous
explanation that is applicable to both classes.

###  3.3 Possible attack strategies based on the explanation method employed

Whereas our framework considers models’ explanations in their most general
form, the way in which an explanation is conveyed determines how humans
process and interpret the information. This implies that some attack
strategies might be more suitable for some type of explanations than for
others. Moreover, the way in which an adversarial example is generated for an
explainable machine learning model will also depend on the type of
explanation. For these reasons, in this section we briefly discuss in which
way an adversarial example should be designed depending on the type of
explanation or the particular type of attack to be produced.

  * •

Feature-based explanations: the highlighted parts or features need to be
coherent with the classification, and correspond to (I) human-perceivable,
(II) semantically meaningful and (III) relevant parts. A common criticism to
feature-based explanations such as saliency maps is that they identify the
relevant parts of the inputs, but no how the models are processing such parts
[20]. Thus, an adversarial attack could take advantage of this limitation.
First, a particular region of the input can be highlighted to support a
misclassification of the model and to convince the user (assuming that such
region contributes to predict an incorrect class), which is interesting
particularly for targeted adversarial attacks (that is, when the objective of
the attack is to produce a specific incorrect output). An attack could also
highlight irrelevant parts to mislead the observer, or generate ambiguous
explanations by highlighting multiple regions or providing a uniform map,
which are strategies well-suited for untargeted attacks (that is, when the aim
is to change the output of the model, without any preference in the incorrect
class).

  * •

Prototype-based explanations: in this case, for the human to accept the given
explanation, the key features of the closest prototypes should (I) be
perceptually identifiable in the given input, and, ideally, (II) contain
features correlated with the output class. The contrary should happen for the
farther prototypes, that is, their key features should not be present in the
input nor be correlated with the output class (or, ideally, be opposite). In
order to achieve these objectives, the more general the prototypes (e.g., if
they represent semantic concepts or parts of inputs rather than completely
describing an output class), the higher the chances of producing explanations
that could lead to a wrong classification while being coherent with a human
perception, such as ambiguous explanations.

  * •

Rule-based explanations can be fooled by targeting explanations which are
aligned with the model output (e.g., the explanation justifies the prediction
or at least mimics the model behavior), but which employ reliable, trustworthy
or neutral features [34]. For instance, a model for criminal-recidivism
prediction could provide a negative assessment based on unethical reasons
whereas the explanation is taken as ethical.

  * •

Counterfactual explanations:  in this case, the objective of an adversarial
attack could be forcing a particular counterfactual explanation, suggesting
changes on irrelevant features (thus preventing correcting the actually
relevant deficiencies), or forcing a biased or discriminatory explanations in
detriment of the model’s fairness.

###  3.4 Desiderata for adversarial attacks in different scenarios involving
explainable machine learning models

In Table 1, we describe the main characteristics or desiderata that an
adversarial attack should satisfy in different scenarios in order to be
successful. We build on the idea that common tasks, problems or applications
have common categories, and that explanations or interpretation needs are
different in each of them [17]. Thus, adversarial attacks (or, oppositely, the
defensive countermeasures) should also be designed differently for each type
of explanation, focusing on the more relevant or crucial factor in each case.
The scenarios described in Table 1 comprise different degrees of expertise of
the human in supervision of the classification process and different
functional purposes of the explanation. It is important to note that a
particular problem or task could belong to more than one scenario (i.e.,
scenarios are not mutually-exclusive). Moreover, we emphasize that some of the
scenarios involve factors which are difficult to quantify in a formal way
(e.g., the expertise of a user). Nevertheless, we believe that it is necessary
to consider such detailed scenarios in order to rigorously discuss which type
of adversarial examples can be realizable in practice.

The first scenario described in Table 1 (S1) comprises tasks in which the
implications of the decisions made by the model cannot be controlled by the
user, or cases in which there is no time for a human supervision of the
predictions. Despite the relevance of some tasks that fall into this category
(e.g., autonomous cars or massive content filtering), humans cannot thoroughly
evaluate each possible prediction. For this reason, explanations are not of
practical use in such cases, so the main (or only) goal of an adversary is to
produce an incorrect output.

Nevertheless, interpretability or explainability can be desirable properties
for any model (including those developed for the scenario S1) in order to
debug or validate them (S2 scenario). For instance, a model developer might
want to explain the decisions of a self-driving car (even if the end-user will
not receive explanations when the model is put into practice) to assess why it
has provided an incorrect output, to validate its reasoning process or to gain
knowledge about what the model has learned. In such cases, an adversary could
justify a misclassification of the model (A.2.1, A.3), hide an inappropriate
behavior when the model predicts correct but for the wrong reasons (A.1.1), or
produce wrong outputs and explanations at the same time (A.2). The same attack
strategies are applicable in scenarios in which the models’ decisions are
taken as more relevant or imperative than the experts’ judgment (S3). The main
difference with respect to the scenario S1 is that, in this case, explanations
can be useful or relevant even when the model is deployed or employed by the
end-user, and, therefore, the attack should also take the explanations into
consideration instead of considering only the output class.

In addition, we consider four different scenarios depending on the expertise
level of the user querying the model, which range from no expertise (S4),
medium expertise (S5) and high expertise (S6). The case of no expertise (S4)
is the simplest one from the perspective of the adversary, as any attack
scheme can be produced without arousing suspicions, taking advantage of the
user inexperience. For the same reason, models deployed in such scenarios
should also be the ones with more security measures against adversarial
attacks. If the user’s expertise is medium (S5), the explanation is expected
to clarify the classification to the user. Thus, the explanation should be
sufficiently consistent with the main semantic features in the input (e.g.,
the user might not be able to diagnose a medical image, but can identify the
relevant spots depending on what is being diagnosed, such as darker spots in
skin-cancer diagnosis), and/or sufficiently consistent with the output class
(A.1.1, A.2.1, A.3). Similarly, if the user has a partial expertise (S7),
which could happen in hierarchical classification tasks, then the adversary
needs to ensure that the output and the explanations are coherent only with
the factors or features that are familiar for the user.

A user with high expertise (S6), by definition, will realize that a model is
producing a wrong output or explanation. Therefore, the only way to mislead
the model and convince the human of a wrong prediction is by means of
ambiguity (A.1.1, A.3). For instance, in an image classification task, two
objects can appear at the same time, being possible to produce a wrong class
with a reasonable explanation (e.g., highlighting the secondary object as the
most relevant one). In addition, in problems in which the inputs are
inherently ambiguous, such as natural language processing tasks, different but
reasonable explanations can be produced for the same input.

Finally, in some cases the explanations might be more interesting, necessary
or challenging than the output itself (S8). Some representative tasks are
predictive maintenance (e.g., it might be more interesting knowing why certain
system will fail than just knowing that it will fail) or medical diagnosis
(e.g., discovering why a model has diagnosed a patient as being at high risk
for a particular disease might be the main priority to prevent the disease or
provide a better treatment). For these reasons, a change in the explanation is
critical for such models, being particularly sensitive to the attacks
described in A.1 (or those described in A.2.1, if the misclassification of the
model is difficult to notice by the user).

Scenario | Representative Examples | Applicable Attacks
---|---|---
|  S1: Impossibility of
---
correcting the output
or controlling the
implications of the
decision in time.
| Fast-decision making
---
scenarios (e.g., au-
tonomous cars) or
automatized processes
(e.g., massive online
content filtering).
|  $\bullet$ Any adversarial attack capable of
---
producing a change in the output
class.
|  S2: Model debugging,
---
development,
validation, etc.
| Applicable to any task.
---
$\bullet$ A.2.1, A.3 (justify model’s misclassi- fications). $\bullet$ A.1.1
(mask inappropriate behaviors, e.g., hiding biases by producing trust- worthy
outputs or explanations). $\bullet$ A.2 (produce wrong outputs and
explanations jointly).
|  S3: Models’ decisions
---
are more imperative
than experts’ judgments.
| Risk of criminal
---
recidivism or credit
risk management.
|  S4: User with no
---
expertise.
| Scenarios in which the
---
decision criteria are
secret, hidden, or
unknown (e.g., banking
or financial scenarios,
malware classification
problems, etc.).
|  $\bullet$ Any adversarial attack scheme
---
(taking advantage of the user’s
inexperience).
|  S5: User with medium
---
expertise (the model
is expected to clarify
or support its predic-
tions).
| Challenging scenarios
---
(e.g., complex medical
diagnosis) or unfore-
seeable scenarios (e.g.,
macroeconomic predic-
tions, risk of criminal
recidivism, etc.).
|  $\bullet$ A.1.1, A.2.1, A.3.
---
$\bullet$ The explanation needs to be
consistent with the input patterns
and/or consistent with the output
class.
|  S6: User with high
---
expertise.
| Tasks in which the inputs
---
can be ambiguous (e.g.,
NLP tasks such as senti-
ment analysis or multiple
object detection in the
image domain).
|  $\bullet$ A.1.1, A.3 (Attacks involving
---
generating ambiguous explanations).
|  S7: User with partial
---
or mixed expertise
(i.e., expert in some
factors but clueless
in others).
| Hierarchical
---
classification.
|  $\bullet$ The output and the explanation
---
should be consistent with the known
factors (either regarding input fea-
tures or the output class).
|  S8: Explanations even
---
more relevant than
the classification
itself.
| Predictive maintenance,
---
medical diagnosis or
credit/loan approval
(e.g., with a wrong
explanation users can
not modify or correct
the deficiencies).
|  $\bullet$ A.1, A.2.1 (e.g., maintain the output
---
but produce totally or partially wrong
explanations, or produce unethical
explanations).
Table 1: Possible scenarios in which explainable machine learning models can
be deployed, and a guideline on how adversarial attacks should be designed in
each case in order to pose a realistic threat.

##  4 Conclusions

In this paper, we have introduced a comprehensive framework to rigorously
study the possibilities and limitations of adversarial examples in explainable
machine learning scenarios, in which the input, the model’s predictions and
the explanations are assessed by humans. First, we have extended the notion of
adversarial examples in order to fit in such scenarios, which have allowed us
to examine different adversarial attack paradigms. Furthermore, we thoroughly
analyze how adversarial attacks should be designed in order to mislead
explainable machine learning models (and humans) depending on a wide range of
factors such as the type of task addressed, the expertise of the users
querying the model, as well as the type, scope or impact of the explanation
methods used to justify the decisions of the models. Overall, the proposed
framework provides a comprehensive road map for the design of malicious
attacks in realistic scenarios involving explainable models and a human
supervision, contributing to a more rigorous study of adversarial examples in
the field of explainable machine learning.

##  5 Acknowledgment

This work is supported by the Basque Government (BERC 2018-2021 and ELKARTEK
programs, IT1244-19), by the Spanish Ministry of Economy and Competitiveness
MINECO (project PID2019-104966GB-I00) and by the Spanish Ministry of Science,
Innovation and Universities (FPU19/03231 predoctoral grant). Jose A. Lozano
acknowledges support by the Spanish Ministry of Science, Innovation and
Universities through BCAM Severo Ochoa accreditation (SEV-2017-0718).

## References

  * [1] Yu Zhang, Peter Tiňo, Aleš Leonardis, and Ke Tang.  A Survey on Neural Network Interpretability.  arXiv preprint arXiv:2012.14261, 2021.
  * [2] Matthew D. Zeiler and Rob Fergus.  Visualizing and Understanding Convolutional Networks.  In Computer Vision – ECCV 2014, Lecture Notes in Computer Science (LNCS, volume 8689), pages 818–833. Springer International Publishing, 2014.
  * [3] Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson.  Understanding Neural Networks Through Deep Visualization.  In International Conference on Machine Learning - Deep Learning Workshop 2015, pages 1–12, 2015.
  * [4] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres.  Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV).  In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2668–2677. PMLR, 2018.
  * [5] Amirata Ghorbani, James Wexler, James Y. Zou, and Been Kim.  Towards Automatic Concept-Based Explanations.  In Advances in Neural Information Processing Systems, volume 32, pages 1–10, 2019.
  * [6] Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin.  Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictions.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3530–3537, 2018.
  * [7] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K. Su.  This Looks Like That: Deep Learning for Interpretable Image Recognition.  In Advances in Neural Information Processing Systems, volume 32, pages 1–12, 2019.
  * [8] Peter Hase, Chaofan Chen, Oscar Li, and Cynthia Rudin.  Interpretable Image Recognition with Hierarchical Prototypes.  Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP-19), 7(1):32–40, 2019.
  * [9] David Alvarez-Melis and Tommi Jaakkola.  Towards Robust Interpretability with Self-Explaining Neural Networks.  In Advances in Neural Information Processing Systems, volume 31, pages 1–10, 2018.
  * [10] Sascha Saralajew, Lars Holdijk, Maike Rees, and Ebubekir Asan.  Classification-by-Components: Probabilistic Modeling of Reasoning over a Set of Components.  In Advances in Neural Information Processing Systems, volume 32, pages 1–12, 2019.
  * [11] Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu.  Interpretable Convolutional Neural Networks.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8827–8836. IEEE, 2018.
  * [12] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing Properties of Neural Networks.  In International Conference on Learning Representations (ICLR), pages 1–10, 2014.
  * [13] Christian Etmann, Sebastian Lunz, Peter Maass, and Carola Schoenlieb.  On the Connection Between Adversarial Robustness and Saliency Map Interpretability.  In Proceedings of the 36th International Conference on Machine Learning, pages 1823–1832. PMLR, 2019.
  * [14] Tianyuan Zhang and Zhanxing Zhu.  Interpreting Adversarially Trained Convolutional Neural Networks.  In Proceedings of the 36th International Conference on Machine Learning, pages 7502–7511. PMLR, 2019.
  * [15] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry.  Robustness May Be at Odds with Accuracy.  In International Conference on Learning Representations (ICLR), 2019.
  * [16] Andrew Slavin Ros and Finale Doshi-Velez.  Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing Their Input Gradients.  Proceedings of the AAAI Conference on Artificial Intelligence, 32(1):1660–1669, 2018.
  * [17] Finale Doshi-Velez and Been Kim.  Towards A Rigorous Science of Interpretable Machine Learning.  arXiv preprint arXiv:1702.08608, 2017.
  * [18] Ning Xie, Gabrielle Ras, Marcel van Gerven, and Derek Doran.  Explainable Deep Learning: A Field Guide for the Uninitiated.  arXiv preprint arXiv:2004.14545, 2020.
  * [19] Zhi Chen, Yijie Bei, and Cynthia Rudin.  Concept Whitening for Interpretable Image Recognition.  Nature Machine Intelligence, 2(12):772–782, 2020.
  * [20] Cynthia Rudin.  Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead.  Nature Machine Intelligence, 1(5):206–215, 2019.
  * [21] Zachary C. Lipton.  The Mythos of Model Interpretability: In Machine Learning, the Concept of Interpretability Is Both Important and Slippery.  Queue, 16(3):31–57, 2018.
  * [22] Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Schütt, Sven Dähne, Dumitru Erhan, and Been Kim.  The (Un)reliability of Saliency Methods.  In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, Lecture Notes in Computer Science, pages 267–280. Springer International Publishing, Cham, 2019.
  * [23] Pang Wei Koh and Percy Liang.  Understanding Black-box Predictions via Influence Functions.  In Proceedings of the 34th International Conference on Machine Learning, pages 1885–1894. PMLR, 2017.
  * [24] Sandra Wachter, Brent Mittelstadt, and Chris Russell.  Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR.  Harvard Journal of Law & Technology, 31(2):842–887, 2017.
  * [25] Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal.  Explaining Explanations: An Overview of Interpretability of Machine Learning.  In 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pages 80–89, 2018.
  * [26] Amirata Ghorbani, Abubakar Abid, and James Zou.  Interpretation of Neural Networks Is Fragile.  Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):3681–3688, 2019.
  * [27] Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus-Robert Müller, and Pan Kessel.  Explanations Can Be Manipulated and Geometry Is to Blame.  In Advances in Neural Information Processing Systems, volume 32, 2019.
  * [28] David Alvarez-Melis and Tommi S. Jaakkola.  On the Robustness of Interpretability Methods.  arXiv preprint arXiv:1806.08049, 2018.  Presented at 2018 ICML Workshop on Human Interpretability in Machine Learning (WHI 2018).
  * [29] Xinyang Zhang, Ningfei Wang, Hua Shen, Shouling Ji, Xiapu Luo, and Ting Wang.  Interpretable Deep Learning under Fire.  In 29th USENIX Security Symposium (USENIX Security 20), pages 1659–1676, 2020.
  * [30] Aditya Kuppa and Nhien-An Le-Khac.  Black Box Attacks on Explainable Artificial Intelligence(XAI) methods in Cyber Security.  In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–8, 2020.
  * [31] Haizhong Zheng, Earlence Fernandes, and Atul Prakash.  Analyzing the Interpretability Robustness of Self-Explaining Models.  arXiv preprint arXiv:1905.12429, 2020.
  * [32] Himabindu Lakkaraju and Osbert Bastani.  "How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations.  In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, AIES ’20, pages 79–85, 2020.
  * [33] Juyeon Heo, Sunghwan Joo, and Taesup Moon.  Fooling Neural Network Interpretations via Adversarial Model Manipulation.  In Advances in Neural Information Processing Systems, volume 32, 2019.
  * [34] Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju.  Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods.  In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 180–186, 2020.
  * [35] Alexey Ignatiev, Nina Narodytska, and Joao Marques-Silva.  On Relating Explanations and Adversarial Examples.  In Advances in Neural Information Processing Systems, volume 32, 2019.
  * [36] Luca Viganò and Daniele Magazzeni.  Explainable Security.  In 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS PW), pages 293–300, 2020.
  * [37] Ninghao Liu, Mengnan Du, Ruocheng Guo, Huan Liu, and Xia Hu.  Adversarial Attacks and Defenses: An Interpretation Perspective.  arXiv preprint arXiv:2004.11488, 2020.