# Correcting Classification: A Bayesian Framework Using Explanation Feedback
to Improve Classification Abilities

Yanzhe Bekkemoen
Department of Computer Science
Norwegian University of Science and Technology
Trondheim, Norway
&Helge Langseth
Department of Computer Science
Norwegian University of Science and Technology
Trondheim, Norway

###### Abstract

Neural networks (NNs) have shown high predictive performance, however, with
shortcomings. Firstly, the reasons behind the classifications are not fully
understood. Several explanation methods have been developed, but they do not
provide mechanisms for users to interact with the explanations. Explanations
are social, meaning they are a transfer of knowledge through interactions.
Nonetheless, current explanation methods contribute only to one-way
communication. Secondly, NNs tend to be overconfident, providing unreasonable
uncertainty estimates on out-of-distribution observations. We overcome these
difficulties by training a Bayesian convolutional neural network (CNN) that
uses explanation feedback. After training, the model presents explanations of
training sample classifications to an annotator. Based on the provided
information, the annotator can accept or reject the explanations by providing
feedback. Our proposed method utilizes this feedback for fine-tuning to
correct the model such that the explanations and classifications improve. We
use existing CNN architectures to demonstrate the method’s effectiveness on
one toy dataset (decoy MNIST) and two real-world datasets (Dogs vs. Cats and
ISIC skin cancer). The experiments indicate that few annotated explanations
and fine-tuning epochs are needed to improve the model and predictive
performance, making the model more trustworthy and understandable.

_Keywords_ Explainable/Interpretable Machine Learning $\cdot$ Bayesian
Inference $\cdot$ Deep Learning

##  1 Introduction

During the 20th century, a horse named Clever Hans was claimed to have
performed arithmetic and other intellectual tasks. However, it was later
revealed that the horse responded to the trainer’s involuntary body language
cues. The trainer was unaware that he provided the horse with hints causing it
to display intelligent behavior. This behavior is today referred to as the
Clever Hans effect (Pfungst, 1911).

NNs have displayed high predictive performance in many application areas in
recent years, but they can focus on irrelevant features, thereby displaying
the Clever Hans effect. We define irrelevant features as features not encoding
the data’s true underlying relationship. To make NNs and other machine
learning (ML) methods interpretable and detect such behavior, several methods
for explaining classifications have been developed, specifically for NNs
(Zeiler and Fergus, 2013; Simonyan et al., 2014; Selvaraju et al., 2020;
Shrikumar et al., 2019) and other model-agnostic methods (Ribeiro et al.,
2016). However, these explanation methods do not offer a way to act on the
explanations and remove the Clever Hans effect. As a result, new research has
started to explore the possibility of leveraging explanations to remove or
prevent the effect (Ross et al., 2017; Erion et al., 2020; Rieger et al.,
2020; Teso and Kersting, 2019; Selvaraju et al., 2019). We will refer to these
methods as model correction methods.

Explanations are social, meaning they are a transfer of knowledge through
interactions or conversations (Miller, 2019). Many model correction methods
work as one-way communication where annotations of irrelevant features need to
be provided before training. However, defining such knowledge about the data
before training does not satisfy the explanations’ social aspect.
Additionally, defining such knowledge without considering what the model will
learn through training can be challenging, making model correction methods
hard to use in practice.

Humans’ trust in models is affected by a model’s confidence in its predictive
performance (Zhang et al., 2020). NNs are bad at quantifying uncertainty and
tend to produce overconfident classifications (Lakshminarayanan et al., 2017).
An overconfident model can be dangerous or offensive (Amodei et al., 2016).
The model can be seen as trying to gain false trust that will backlash against
the model if it makes a wrong decision. Consequently, being overconfident can
reduce trust in the model rather than increase it. Providing reasonable
explanations induce trust (Ribeiro et al., 2016), but explanations generated
by these aforementioned explanation methods only capture local behaviors,
meaning an explanation applies only in the vicinity of a sample. This does not
give insight into how a model will perform on out-of-distribution samples.
Hence, to resolve model overconfidence, the model must capture aleatoric and
epistemic uncertainty (Lakshminarayanan et al., 2017). There are several ways
to capture epistemic uncertainty; one way is to frame the NNs within a
Bayesian framework that applies inference on the model’s weights (MacKay,
1992; Blundell et al., 2015).

Model correction and Bayesian inference contribute to robustness and trust in
the model. However, they are difficult to use together because most model
correction methods modify the objective function. The modifications do not
necessarily follow probability calculus and can introduce elements with
unknown distributions.

For a model to have the correct explanation, classification and model
confidence, this work bridges the gap between model correction and Bayesian
inference. Additionally, it takes a step towards making explanations in ML
social. We present a Bayesian framework that uses explanation feedback. After
training, a Bayesian CNN presents explanations of training sample
classifications to a human annotator. The annotator can accept or reject the
explanations by giving feedback as additional evidence. Feedback is
represented as a matrix with the same width and height as a sample specifying
which attribution regions to reject. This feedback is used during fine-tuning
to correct the model such that the explanations and predictive performance
improve.

Figure 1: A “standard” ML pipeline with steps for annotating explanations and
correcting a model. During Step 3, a model explains training sample
classifications to a human annotator who gives feedback on those explanations.
A feedback ${\bm{F}}^{(i)}$ for a sample $i$ is a matrix of the same width and
height as the image. If a feature $k,j$ is irrelevant
${\bm{F}}^{(i)}_{k,j}=1$, otherwise ${\bm{F}}^{(i)}_{k,j}=0$. In Step 4, a
model is fine-tuned with training data and feedback. The goal is to improve
the reasons behind the classifications (explanation in Step 3 vs. Step 5) and
predictive performance.

Our main contributions are:  (1) A pipeline that provides feedback to a model
after training (see Figure 1). This avoids imposing restrictions on the model
during the training phase and results in knowledge transfer being interactive
and model specific.  (2) A Bayesian framework utilizing explanation feedback
to correct a model, resulting in a mathematically grounded objective function
from a probabilistic view.  (3) Experimental results, on one toy dataset and
two real-world datasets, that demonstrate the method’s effectiveness. This
indicates that few annotated explanations and fine-tuning epochs are required
to improve explanations justifying the classifications and the predictive
performance.

##  2 Related Works

##### Explanation Methods.

Several explanation methods have been developed during the past years. These
explanation methods can be divided into model-specific vs. model-agnostic and
global vs. local scope (Lipton, 2017). This work builds upon methods that
create model-specific local explanations for NNs (Zeiler and Fergus, 2013;
Simonyan et al., 2014; Selvaraju et al., 2020; Shrikumar et al., 2019;
Sundararajan et al., 2017). These model-specific local explanation methods
produce importance scores assigned to individual features, e.g., pixels for
image data or words for textual data. We will refer to the importance scores
as feature attributions or attributions for short.

##### Model Correction Methods.

The first work in this area is “Right for the Right Reasons” (RRR) (Ross et
al., 2017) that regularize input gradients to suppress irrelevant features in
the input space. Similarly, Erion et al. (2020) use Expected Gradients (a
modified version of Integrated Gradients (Sundararajan et al., 2017)) to
regularize explanations. Used in a similar fashion as RRR, contextual
decomposition explanation penalization (Rieger et al., 2020) can regularize
feature importance and feature interaction. To encourage NNs to focus on the
same input regions as humans, Human Importance-aware Network Tuning (HINT)
(Selvaraju et al., 2019) uses gradients in the last convolutional layer
(forward direction). HINT encourages NNs to focus on certain regions instead
of suppressing attributions. Similar to active learning, but for explanations,
Teso and Kersting (2019) proposed using data augmentation to actively correct
a model.

##### Bayesian Inference.

Consider a probabilistic model $P({\bm{e}}|{\bm{w}})$, the prior distribution
$P({\bm{w}})$ that encodes the prior knowledge about the parameter ${\bm{w}}$
and the evidence

${\bm{e}}=\\{({\bm{\mathsfit{X}}}^{(i)},{\bm{y}}^{(i)})\\}_{i=1}^{n}$ (1)

with $n$ independent and identically distributed (i.i.d.) samples. We want to
compute the posterior distribution

$P({\bm{w}}|{\bm{e}})=\frac{P({\bm{e}}|{\bm{w}})P({\bm{w}})}{P({\bm{e}})}$ (2)

using Bayes rule. However, exact Bayesian inference is intractable, so Markov
Chain Monte Carlo (MCMC) algorithms or variational inference (VI) are used to
approximate the weights’ posterior distribution. Although VI is more scalable
than MCMC methods, MCMC methods have an asymptotic guarantee that VI methods
do not have. Bayes by Backprop (BBB) (Blundell et al., 2015) is a VI and
backpropagation-compatible method for learning weight distributions. BBB uses
the reparametrization trick (Kingma and Welling, 2014) to calculate a Monte
Carlo (MC) estimator of the evidence lower bound (ELBO) and uses Gaussian
variational distributions. To compute the MC estimator of ELBO, samples are
drawn from the weights, which is computationally expensive. To reduce the
number of samples needed, the local reparameterization trick (LRT) (Kingma et
al., 2015) was proposed. LRT considers uncertainty at the activation level
(before any nonlinear activation function) by computing the activations before
drawing samples. This significantly reduces the number of samples drawn and
makes it computationally cheaper.

######  Example 1 (LRT for a fully connected layer)

Let layer $l$ be a fully connected layer with weights ${\bm{W}}^{m\times r}$
where
$q_{\bm{\lambda}}({W}_{i,j})=\mathcal{N}(\mu_{{W}_{i,j}},\sigma^{2}_{{W}_{i,j}})\leavevmode\nobreak\
\forall{W}_{i,j}\in{\bm{W}}$ and biases ${\bm{B}}^{t\times r}$ where
$q_{\bm{\lambda}}({B}_{i,j})=\mathcal{N}(\mu_{{B}_{i,j}},\sigma^{2}_{{B}_{i,j}})\leavevmode\nobreak\
\forall{B}_{i,j}\in{\bm{B}}$. Given an input ${\bm{Z}}^{t\times m}$, the
activation of layer $l$ is ${\bm{U}}={\bm{Z}}{\bm{W}}+{\bm{B}}$ of size
$t\times r$ where

$q_{\bm{\lambda}}({U}_{i,j}|{\bm{Z}})=\mathcal{N}(\gamma_{i,j},\delta_{i,j})\leavevmode\nobreak\
\forall{U}_{i,j}\in{\bm{U}}$ (3)
$\gamma_{i,j}=\sum^{m}_{k=1}{Z}_{i,k}\mu_{{W}_{k,j}}+\mu_{{B}_{i,j}},\leavevmode\nobreak\
\delta_{i,j}=\sum^{m}_{k=1}{Z}^{2}_{i,k}\sigma^{2}_{{W}_{k,j}}+\sigma^{2}_{{B}_{i,j}}.$
(4)

A new weight matrix is sampled for every training sample to reduce estimator
variance, which has a computational complexity of ${\mathcal{O}}(tmr)$. By
sampling activations rather than weights, the computational complexity is
reduced to ${\mathcal{O}}(tr)$.

######  Example 2 (LRT for a convolutional layer)

Let layer $l$ be a convolutional layer with a convolutional filter
${\bm{\mathsfit{W}}}^{w^{\prime}\times h^{\prime}\times\bar{c}}$ where
$q_{\bm{\lambda}}(\operatorname{vec}({\bm{\mathsfit{W}}}))=\mathcal{N}(\operatorname{vec}({\bm{\mathsfit{M}}}),\operatorname{diag}(\operatorname{vec}({\bm{\mathsfit{V}}}^{2})))$
is a fully factorized Gaussian. Given an input
${\bm{\mathsfit{Z}}}^{\bar{w}\times\bar{h}\times\bar{c}}$, the activation of
layer $l$ is ${\bm{U}}={\bm{\mathsfit{Z}}}*{\bm{\mathsfit{W}}}$ of size
$\hat{w}\times\hat{h}$ where

$q_{\bm{\lambda}}(\operatorname{vec}({\bm{U}})|{\bm{\mathsfit{Z}}})=\mathcal{N}({\bm{\mu}}_{\bm{U}},{\bm{\sigma}}_{\bm{U}}^{2})$
(5)
${\bm{\mu}}_{\bm{U}}=\operatorname{vec}({\bm{\mathsfit{Z}}}*{\bm{\mathsfit{M}}}),\leavevmode\nobreak\
{\bm{\sigma}}_{\bm{U}}^{2}=\operatorname{diag}(\operatorname{vec}({\bm{\mathsfit{Z}}}^{2}*{\bm{\mathsfit{V}}}^{2}))$
(6)

is treated as a fully factorized Gaussian. We define
$\operatorname{vec}(\cdot)$ as a vectorization function, $(\cdot)^{2}$ as an
element-wise operator, $*$ as a convolution operator and
$\operatorname{diag}(\cdot)$ as a diagonal matrix.

##  3 Variational Inference and Explanation Feedback

Section 3.1 outlines the pipeline to construct feedback and correct a model
(Step 3 and 4 in Figure 1). Sections 3.2 and 3.3 outline the objective
function used in the fine-tuning phase and how feedback is induced into a
model by adding additional evidence to ${\bm{e}}$.

###  3.1 Feedback Pipeline

There might be prior knowledge available about a dataset, e.g., digits are
centered in the Modified National Institute of Standards and Technology
(MNIST) (LeCun et al., 1989) dataset. In this case, we can make a model focus
on the center and ignore the edges during training. However, this kind of
knowledge might not be needed and can unnecessarily regularize the model.
Moreover, constructing such knowledge without knowing what the model will
learn through training can be difficult. If irrelevant features were known
before training, training data could be preprocessed instead of introducing a
new technique. We propose a pipeline where, unlike most previous work, the
model asks for feedback after training. By asking for feedback after training,
we can define knowledge specifically targeted at the model.

The pipeline adds two additional steps to a “standard” ML pipeline (Step 3 and
4 in Figure 1). During Step 3, a human annotator provides explanation feedback
on training sample classifications. For explanation-generation, any saliency-
based methods can be used, e.g., Gradient-weighted Class Activation Mapping
(Grad-CAM) (Selvaraju et al., 2020). To determine samples for annotation,
active learning sampling strategies can be used. However, it is not obvious
how to measure explanations’ informativeness, so we leave this issue for
future work. During the experiments, samples are annotated uniformly at random
because the goal of our analysis is a proof of concept and not a fully
specified pipeline.

As for Step 4, a model is fine-tuned using feedback and training data. The
evidence and the objective function used for fine-tuning will be described in
the coming sections.

###  3.2 Add Explanation Feedback to Evidence

We want to compute the posterior distribution $P({\bm{w}}|{\bm{e}})$ of the
weights of a NN. The evaluation involves computation of intractable integrals.
Therefore, approximation techniques are used. VI is one of the most widely
used and defines a variational distribution $q_{\bm{\lambda}}({\bm{w}})$ that
is used to approximate the posterior distribution by minimizing the
Kullback–Leibler (KL) divergence
$D_{\mathrm{KL}}(q_{\bm{\lambda}}({\bm{w}})\|P({\bm{w}}|{\bm{e}}))$.
Minimizing the KL divergence is equivalent to maximizing the ELBO

${\mathcal{L}}_{\text{\lx@glossaries@gls@link{acronym}{elbo}{\leavevmode
ELBO}}}=\underbrace{\mathbb{E}_{q_{\bm{\lambda}}({\bm{w}})}[\log
P({\bm{e}}|{\bm{w}})]}_{\text{Likelihood}}-\underbrace{D_{\mathrm{KL}}(q_{\bm{\lambda}}({\bm{w}})\|P({\bm{w}}))}_{\text{Complexity}}.$
(7)

Let ${{\bm{\mathsfit{X}}}^{(i)}}^{w\times h\times c}$ be an observation with
the label ${\bm{y}}^{(i)}$, a feedback of the sample $i$ is a matrix
${\bm{F}}^{(i)}\in\\{0,1\\}^{w\times h}$ given by a human annotator. If a
feature $k,j$ is irrelevant ${\bm{F}}^{(i)}_{k,j}=1$, otherwise
${\bm{F}}^{(i)}_{k,j}=0$. Furthermore, let $f$ be a Bayesian CNN that takes
${\bm{\mathsfit{X}}}^{(i)}$ as input and outputs the logit vector
$f({\bm{\mathsfit{X}}}^{(i)})=\hat{{\bm{y}}}^{(i)}$.

Consider input gradients
$\nabla_{{\bm{\mathsfit{X}}}^{(i)}}\sum_{j}\hat{{y}}_{j}^{(i)}$ of the
observation ${\bm{\mathsfit{X}}}^{(i)}$ that can be used as an explanation
that specifies feature attributions.

${\bm{\mathsfit{H}}}^{(i)}={\bm{F}}^{(i)}\otimes(\nabla_{{\bm{\mathsfit{X}}}^{(i)}}\sum_{j}\hat{{y}}_{j}^{(i)})$
(8)

is the attributions on irrelevant features that we want to minimize. $\otimes$
signifies broadcasting and an element-wise product. To include explanation
feedback, we add additional evidence to ${\bm{e}}$

${\bm{e}}_{\text{grad}}={\bm{e}}\cup\\{{\bm{\mathsfit{H}}}^{(i)}=\bm{0}\\}_{i=1}^{n}$
(9)

where we set ${\bm{\mathsfit{H}}}^{(i)}=\bm{0}$ to constrain attributions on
irrelevant features, which is similar to previous work (Ross et al., 2017).

However, the distribution of the input gradients is unknown and it is hard to
find analytically. Not knowing the distribution makes us unable to calculate
the likelihood in Equation 7 with ${\bm{e}}_{\text{grad}}$ as evidence. One
way to solve this issue is to look at some quantity with known distribution
instead of the input gradients. Although weight distributions are known, it is
not trivial to map feedback given in the input space to the parameter space.
Therefore, we use the LRT to consider the activation distributions and map the
feedback to activation space. Examples 1 and 2 describe how to obtain
activation distributions.

In the next section, we show how considering the uncertainty at the activation
level enables us to approximate the evidence ${\bm{e}}_{\text{grad}}$ with an
alternative formulation for which the likelihood can be expressed
analytically.

We use CNNs since the activations in convolutional layers retain spatial
information. The spatial information is needed to map feedback defined in the
input space to corresponding activations. An additional advantage of using
activations rather than input gradients is reduced computational complexity
since second-order partial derivatives are not needed.

###  3.3 Methodology

This section details how to add feedback to ${\bm{e}}$ using activations
rather than input gradients. We do this by describing how to map the feedback
from input to activation space. Moreover, we describe how to compute Equation
7 with the new evidence ${\bm{e}}_{\text{act}}$ (to be defined in Equation
13); something we could not do with ${\bm{e}}_{\text{grad}}$.

Consider the same setup as in Section 3.2. Let
${\bm{\mathsfit{A}}}^{(i)}\in\mathbb{R}^{u\times v\times d}$ ($u\leq
w,\leavevmode\nobreak\ v\leq h$) be feature maps (before any nonlinear
activation function) of the last convolutional layer (forward direction)
obtained by forward passing ${\bm{\mathsfit{X}}}^{(i)}$. The last
convolutional layer is used since it contains the highest abstraction level of
features (Zeiler and Fergus, 2013; Zhou et al., 2015; Selvaraju et al., 2019).

####  3.3.1 Map Feedback from Input to Activation Space

The feedback will be mapped to the activation space to include it in the
evidence ${\bm{e}}$. We do this in two steps:  (1) downsample the feedback to
the width and height of the feature maps ${\bm{\mathsfit{A}}}^{(i)}$ and  (2)
find activations in ${\bm{\mathsfit{A}}}^{(i)}$ spatially overlapping with the
feedback and affecting the classification.

We downsample the feedback ${\bm{F}}^{(i)}$ to the width and height of
${\bm{\mathsfit{A}}}^{(i)}$ by computing

${\bm{J}}^{(i)}=\text{AdaptiveMaxPool}({\bm{F}}^{(i)},(u,v)).$ (10)

The function $\text{AdaptiveMaxPool}({\bm{Z}},{\bm{o}})$ takes a matrix
${\bm{Z}}$ and uses max-pooling to transform it to size
${\bm{o}}=(o_{1},o_{2})$. This operation is similar to how Grad-CAM
interpolates attribution maps to the input space to visualize explanations.
Next, we find entries in ${\bm{J}}^{(i)}$ that overlap with
${\bm{\mathsfit{A}}}^{(i)}$ by calculating

${\bm{\mathsfit{G}}}^{(i)}={\bm{\mathsfit{A}}}^{(i)}\otimes{\bm{J}}^{(i)}.$
(11)

CNNs often use rectified linear unit and max-pooling, implying that
classifications are only affected by a subset of the activations. Therefore,
activations not used in the classification will be excluded by computing

${\bm{\mathsfit{L}}}^{(i)}={\bm{\mathsfit{G}}}^{(i)}\odot\bm{1}_{\mathrm{(\nabla_{{\bm{\mathsfit{A}}}^{(i)}}\sum_{j}\hat{{y}}_{j}^{(i)})\neq\bm{0}}}$
(12)

to zero out entries in ${\bm{\mathsfit{G}}}^{(i)}$ that do not affect the
classification. ${\bm{\mathsfit{L}}}^{(i)}$ is a tensor representing the
activations that affect the classification and overlap with all indices $k,j$
where ${\bm{J}}^{(i)}_{k,j}=1$. The goal is to find the random variables where
values in ${\bm{\mathsfit{L}}}^{(i)}\neq 0$ are drawn from. We will refer to
those activations as ${\bm{a}}^{(i)}_{\mathcal{F}}$. The remaining activations
in the network will be denoted ${\bm{a}}^{(i)}_{{\mathcal{F}}^{c}}$. We
substitute input gradients with activations and add feedback to ${\bm{e}}$ by
defining

${\bm{e}}_{\text{act}}={\bm{e}}\cup\\{{\mathbf{a}}^{(i)}_{\mathcal{F}}=\bm{0}\\}_{i=1}^{n}$
(13)

The process of finding ${\bm{a}}^{(i)}_{\mathcal{F}}$incurs an extra forward
pass.

####  3.3.2 Compute the ELBO with Additional Evidence

In this section, we show how to compute the ELBO with ${\bm{e}}_{\text{act}}$.
We follow Kingma et al. (2015) and consider uncertainty at the level of
activations. This implies that the expectation in Equation 7 will be computed
with respect to
$q_{\bm{\lambda}}({\bm{a}}^{(i)}_{{\mathcal{F}}^{c}},{\bm{a}}^{(i)}_{{\mathcal{F}}}|{\bm{\mathsfit{X}}}^{(i)})$
rather than $q_{\bm{\lambda}}({\bm{w}})$.
$q_{\bm{\lambda}}({\bm{a}}^{(i)}_{{\mathcal{F}}^{c}}|{\bm{\mathsfit{X}}}^{(i)},{\mathbf{a}}^{(i)}_{\mathcal{F}}=\bm{0})$
will be written as $q_{\bm{\lambda}}({\bm{a}}^{(i)}_{{\mathcal{F}}^{c}})$ for
short.
${\mathcal{L}}_{\text{\lx@glossaries@gls@link{acronym}{elbo}{\leavevmode
ELBO}}}({\bm{\mathsfit{X}}}^{(1)},...,{\bm{\mathsfit{X}}}^{(n)})=\sum^{n}_{i=1}{\mathcal{L}}_{\text{\lx@glossaries@gls@link{acronym}{elbo}{\leavevmode
ELBO}}}^{(i)}({\bm{\mathsfit{X}}}^{(i)})$ since the samples are i.i.d. We can
reformulate Equation 7 as a loss function using activations and the evidence
${\bm{e}}_{\text{act}}$ for a single sample $i$ as

${\mathcal{L}}_{\text{\lx@glossaries@gls@link{acronym}{lrt}{\leavevmode
LRT}}}^{(i)}=\frac{1}{n}D_{\mathrm{KL}}(q_{\bm{\lambda}}({\bm{w}})\|P({\bm{w}}))-\mathbb{E}_{q_{\bm{\lambda}}({\bm{a}}^{(i)}_{{\mathcal{F}}^{c}})}[\log
P({\bm{e}}_{\text{act}}^{(i)}|{\bm{a}}^{(i)}_{{\mathcal{F}}^{c}})].$ (14)

Expanding ${\bm{e}}_{\text{act}}^{(i)}$ in the log probability in Equation 14
gives

$\begin{split}\log({\bm{e}}_{\text{act}}^{(i)}|{\bm{a}}^{(i)}_{{\mathcal{F}}^{c}})&=\log
P({\bm{y}}^{(i)},{\mathbf{a}}^{(i)}_{\mathcal{F}}=\bm{0}|{\bm{\mathsfit{X}}}^{(i)},{\bm{a}}^{(i)}_{{\mathcal{F}}^{c}})\\\
&=\log
P({\bm{y}}^{(i)}|{\mathbf{a}}^{(i)}_{\mathcal{F}}=\bm{0},{\bm{a}}^{(i)}_{{\mathcal{F}}^{c}})+\log
P({\mathbf{a}}^{(i)}_{\mathcal{F}}=\bm{0}|{\bm{\mathsfit{X}}}^{(i)},{\bm{a}}^{(i)}_{{\mathcal{F}}^{c}})\end{split}$
(15)

where ${\bm{a}}^{(i)}_{{\mathcal{F}}^{c}}$ and
${\bm{a}}^{(i)}_{{\mathcal{F}}}$ are treated as fully factorized Gaussians. To
compute Equation 14, we follow the result of BBB and approximate the negative
log-likelihood by applying MC sampling. When both $q_{\bm{\lambda}}({\bm{w}})$
and $P({\bm{w}})$ are fully factorized Gaussians, the complexity term can be
calculated in closed form (Kingma and Welling, 2014). Equation 14 is an
objective function that uses the evidence ${\bm{e}}_{\text{act}}$ to
incorporate feedback from an annotator and is theoretically justified from a
probabilistic perspective, making it mathematically grounded. Moreover, we do
not need a hyperparameter that deals with feedback penalty; instead, the
objective function trades-off between complexity and data.

######  Example 3 (How to approximate Equation 14)

Consider ${\bm{w}}\in\mathbb{R}^{j}$, where the prior
$P({\bm{w}})=\mathcal{N}(\bm{0},{\bm{I}})$ and the approximate posterior
$q_{\bm{\lambda}}({\bm{w}})=\mathcal{N}({\bm{\mu}},\text{diag}({\bm{\sigma}}^{2}))$
are both fully factorized Gaussians. Hence, Equation 14 can be approximated as

$\begin{split}{\mathcal{L}}_{\text{\lx@glossaries@gls@link{acronym}{lrt}{\leavevmode
LRT}}}^{(i)}&\approx\frac{1}{2n}\sum^{j}_{i=1}(\mu_{i}^{2}+\sigma^{2}_{i}-1-\log\sigma^{2}_{i})\\\
&-\frac{1}{m}\sum^{m}_{t=1}[\log
P({\bm{y}}^{(i)}|{\mathbf{a}}^{(i,t)}_{\mathcal{F}}=\bm{0},{\bm{a}}^{(i,t)}_{{\mathcal{F}}^{c}})+\log
P({\mathbf{a}}^{(i,t)}_{\mathcal{F}}=\bm{0}|{\bm{\mathsfit{X}}}^{(i)},{\bm{a}}^{(i,t)}_{{\mathcal{F}}^{c}})].\end{split}$
(16)

$m$ is the number of MC samples.

To summarize, feedback defined in the input space by an annotator is mapped to
the activations. The feedback is included by adding it as additional evidence
to ${\bm{e}}$. During Step 2, the model is trained with ${\bm{e}}$ as the
evidence and Equation 7 as the objective function. As for Step 4, to fine-tune
the model, Equation 14 is used as the objective function with
${\bm{e}}_{\text{act}}$ as the evidence.

##  4 Experiments and Results

The goal of our experiments is to display the effect of our proposed pipeline
and model correction method. We demonstrate our work on one toy dataset, decoy
MNIST (Ross et al., 2017) and two real-world datasets, Dogs vs. Cats (Kaggle,
2014) and The International Skin Imaging Collaboration (ISIC) skin cancer
(Codella et al., 2019).

The LeNet (Lecun et al., 1998) architecture was used for decoy MNIST. LeNet
models were trained for 100 epochs with early stopping. The AlexNet
(Krizhevsky et al., 2012) architecture with batch normalization (Ioffe and
Szegedy, 2015), but without dropout was used for Dogs vs. Cats and ISIC skin
cancer. AlexNet models were trained for 1000 epochs with early stopping. Both
LeNet and AlexNet models were saved at the best validation F1 score. All
models were trained with a batch size of 256. The Adam optimizer (Kingma and
Ba, 2017) with a learning rate of 0.001 and a prior
$P({\bm{w}})=\mathcal{N}(\bm{0},0.1^{2}{\bm{I}})$ were used. Samples to
annotate with feedback were chosen uniformly at random, as stated in Section
3.1.

To quantify attribution changes before and after feedback, attributions
overlapping with irrelevant features were computed using four methods (see
Table 2): Saliency (Simonyan et al., 2014), Deep Learning Important FeaTures
(DeepLIFT) (Shrikumar et al., 2019), Grad-CAM and Occlusion (Zeiler and
Fergus, 2013). Consider an attribution mask
${{\bm{\mathsfit{T}}}^{(i)}}^{w\times h\times v}$ ($v$ is the number of
channels that depends on the method and the model) of sample $i$ for one of
the attribution methods. $k$ is the number of samples in the test dataset with
annotated explanation (not used in the fine-tuning process). The attribution
overlap in Table 2 is computed as

$\frac{1}{k}\sum^{k}_{i=1}\frac{||\operatorname{vec}({\bm{F}}^{(i)}\otimes{\bm{\mathsfit{T}}}^{(i)})||_{1}}{||\operatorname{vec}({\bm{\mathsfit{T}}}^{(i)})||_{1}}$
(17)

where $||\cdot||_{1}$ is the $\ell_{1}$ norm operator. The effect of fine-
tuning with feedback for a few samples from the test datasets (not used in the
fine-tuning process) can be seen in Figure 2.

Dataset | Precision | Recall | F1 | Accuracy
---|---|---|---|---
NF | F | NF | F | NF | F | NF | F
Decoy MNIST | 0.725 | 0.970 | 0.725 | 0.970 | 0.725 | 0.970 | 0.725 | 0.970
Dogs vs. Cats | 0.918 | 0.923 | 0.857 | 0.870 | 0.887 | 0.896 | 0.886 | 0.894
Skin cancer | 0.280 | 0.320 | 0.904 | 0.798 | 0.427 | 0.457 | 0.815 | 0.799
Skin cancer NP | 0.289 | 0.335 | 0.904 | 0.798 | 0.437 | 0.472 | 0.702 | 0.721
Table 1: Performance metrics of the model trained with no feedback (NF)and
feedback (F). For decoy MNIST, all of the metrics are calculated using micro
average. The skin cancer dataset is tested with and without patch data (NP)
and accuracy is computed with macro average recall, also known as balanced
accuracy.

(a) Decoy MNIST. The model focuses on both the decoys and digits before
feedback. After feedback, only the digits are used for classifications.

(b) Dogs vs. Cats. After feedback, the model becomes sharper and focuses more
on the animals than the background.

(c) ISIC skin cancer. Before feedback, the model uses irrelevant patches and
moles to classify samples. After feedback, it uses only moles on these
samples.

Figure 2: Explanations before and after fine-tuning with feedback visualized
on samples from test datasets. DeepLIFT was used for decoy MNIST and Grad-CAM
for both Dogs vs. Cats and ISIC skin cancer to visualize explanations.

###  4.1 Decoy MNIST

The MNIST dataset consists of $28\times 28$ grayscale images of digits. The
training data has 60000 samples and the test data has 10000 samples. There are
ten classes: $0,1,...,9$. Decoy MNIST is a modified version of MNIST where
every sample in the dataset has a $4\times 4$ square in each corner (see
Figure 2(a)). In the training data, the decoys’ colors correspond with the
label of the digit $y$, ($255-25y$) and in the test data, the colors are
randomly drawn. This decoy rule is a modified version of the one found in Ross
et al. (2017), where only one corner has a decoy and is drawn randomly. The
training data was divided into 90% (54000 samples) for training and 10% (6000
samples) for validation. The model was trained for 1 epoch without feedback
and fine-tuned for 17 epochs. 0.4% (189 samples) of the training samples were
annotated with feedback.

To observe how much of the original predictive performance is recovered
through model correction, the model was trained with MNIST (7 epochs) and
decoy MNIST. The model trained without decoys has 98.7% accuracy. After adding
decoys, the accuracy drops down to 72.5%. By fine-tuning the model with
feedback, the difference between the original accuracy is reduced to 1.7%. The
decoy rule has no randomness, which makes it more difficult to recover the
original accuracy. Most previous works have feedback on all training samples.
In contrast, this approach has feedback on only 0.4% of the training data,
making it more appealing.

Figure 2(a) and Table 2 show that without feedback, the model focuses on the
decoys. After fine-tuning with feedback, the attributions on the decoys are
significantly reduced.

Dataset | Saliency | DeepLIFT | Grad-CAM | Occlusion
---|---|---|---|---
NF | F | NF | F | NF | F | NF | F
Decoy MNIST | 0.080 | 0.031 | 0.078 | 0.019 | 0.063 | 0.025 | 0.181 | 0.050
Dogs vs. Cats | 0.396 | 0.384 | 0.405 | 0.407 | 0.441 | 0.379 | 0.393 | 0.380
Skin cancer | 0.154 | 0.111 | 0.145 | 0.124 | 0.221 | 0.071 | 0.250 | 0.148
Table 2: Attributions overlapping with irrelevant features averaged over all
samples in the test dataset with annotated explanation, as described in
Section 4. The attribution overlap score is bounded $[0,1]$ and a lower score
is better because it implies less attention is focused on irrelevant features.
For Occlusion, a sliding window of size $3\times 3$ was used for decoy MNIST
and $23\times 23$ for the other two datasets.

###  4.2 Dogs vs. Cats

The Dogs vs. Cats dataset consists of RGB images of dogs (label 0) and cats
(label 1) (see Figure 2(b)). All images in the dataset are scaled to
$227\times 227$. It has training data of 25000 samples and test data of 12500
samples, but the latter does not have labels. Thus, only the training data is
used and divided as follows: 90% (22500 samples) training, 5% (2500 samples)
validation and 5% (2500 samples) test. For the construction of feedback, a
pretrained DeepLabV3 ResNet101 (Chen et al., 2017) semantic segmentation
network was used. ${\bm{F}}^{(i)}_{k,j}=1$ when index $k,j$ does not overlap
with an animal. This tells the model to not focus on background information.
The model was trained for 765 epochs without feedback and fine-tuned for 16
epochs. 4.5% (1012 samples) of the training samples were annotated with
feedback.

Dogs vs. Cats is the only dataset without any apparent irrelevant features
repeating in several samples. Therefore, only a slight improvement in the
model’s predictive performance can be seen in Table 1. Furthermore, more
feedback data is needed to affect the model. After fine-tuning with feedback,
attributions overlap more with the animals (see Figure 2(b) and Table 2).
Interestingly, explaining not to focus on background information results in
the model focusing on the animals’ faces instead of the whole body.

###  4.3 ISIC Skin Cancer

The version of the ISIC skin cancer dataset by Rieger et al. (2020) that we
used has 21654 samples where 2284 are malignant (label 1) and 19370 are benign
(label 0). Of those benign samples, 48% (9209 samples) have colorful patches
(see Figure 2(c)). Feedback for samples in this dataset are regions that
contain those colorful patches because they are irrelevant for the
classifications. The dataset was divided into 90% (19488 samples) training, 5%
(1083 samples) validation, and 5% (1083 samples) test. The model was trained
for 997 epochs without feedback and fine-tuned for 1 epoch. 1.5% (292 samples)
of the training samples were annotated with feedback.

The models were tested both with and without patches to observe the patches’
effect. Table 1 shows that with patch data, the model without feedback has
better accuracy than the model fine-tuned with feedback. However, without
patch data, the model fine-tuned with feedback has better accuracy. Moreover,
it is more trustworthy based on the explanations (Figure 2(c)).

Before feedback, the model uses patches to “cheat” on benign samples. When
patches are removed, the model without feedback has the same number of false
positives and true negatives, classifying benign samples randomly. In
contrast, the model fine-tuned with feedback does not do that. Figure 2(c)
demonstrates that the model with feedback focuses much less on the patches.
Table 2 displays the same result quantitatively. Also, the positions of the
patches appear not to matter.

##  5 Conclusion and Future Works

We propose a pipeline and a Bayesian framework using explanation feedback. The
pipeline makes it possible for users to interact with a model after training
to create explanation feedback (summarized in Figure 1). The users create
feedback based on the model’s explanations of training sample classifications.
Our Bayesian framework uses this feedback to correct the model so that
explanations and predictive performance improve, making the model more
trustworthy and understandable. The effectiveness of the proposed approach is
shown on one toy dataset and two real-world datasets. The results indicate
that focus on dataset biases are minimized, and the features representing the
data’s true underlying relationship are targeted more distinctly. Thus, paying
less attention to irrelevant features. Furthermore, few annotated explanations
and fine-tuning epochs are needed to observe this effect.

We see several opportunities for future works, including:  (1) sampling
strategies for finding samples to ask for annotation,  (2) extending beyond
CNN architectures and  (3) richer feedback semantics; currently, only
rejection of attribution regions are supported.

## References

  * Pfungst [1911] O. Pfungst.  _Clever Hans:(the horse of Mr. Von Osten.) a contribution to experimental animal and human psychology_.  Holt, Rinehart and Winston, 1911.
  * Zeiler and Fergus [2013] M. D. Zeiler and R. Fergus.  Visualizing and Understanding Convolutional Networks.  _arXiv:1311.2901_ , 2013.  URL http://arxiv.org/abs/1311.2901.
  * Simonyan et al. [2014] K. Simonyan, A. Vedaldi, and A. Zisserman.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps.  _arXiv:1312.6034_ , 2014.  URL http://arxiv.org/abs/1312.6034.
  * Selvaraju et al. [2020] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization.  _International Journal of Computer Vision_ , 128(2):336–359, 2020.  ISSN 0920-5691, 1573-1405.  doi:10.1007/s11263-019-01228-7.  URL http://arxiv.org/abs/1610.02391.  arXiv: 1610.02391.
  * Shrikumar et al. [2019] A. Shrikumar, P. Greenside, and A. Kundaje.  Learning Important Features Through Propagating Activation Differences.  _arXiv:1704.02685_ , 2019.  URL http://arxiv.org/abs/1704.02685.
  * Ribeiro et al. [2016] M. T. Ribeiro, S. Singh, and C. Guestrin.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier.  _arXiv:1602.04938_ , 2016.  URL http://arxiv.org/abs/1602.04938.
  * Ross et al. [2017] A. S. Ross, M. C. Hughes, and F. Doshi-Velez.  Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations.  _arXiv:1703.03717_ , 2017.  URL http://arxiv.org/abs/1703.03717.
  * Erion et al. [2020] G. Erion, J. D. Janizek, P. Sturmfels, S. Lundberg, and S. Lee.  Improving performance of deep learning models with axiomatic attribution priors and expected gradients.  _arXiv:1906.10670_ , 2020.  URL http://arxiv.org/abs/1906.10670.
  * Rieger et al. [2020] L. Rieger, C. Singh, W. J. Murdoch, and Bin Yu.  Interpretations are useful: penalizing explanations to align neural networks with prior knowledge.  _arXiv:1909.13584_ , 2020.  URL http://arxiv.org/abs/1909.13584.
  * Teso and Kersting [2019] S. Teso and K. Kersting.  Explanatory Interactive Machine Learning.  In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_ , AIES ’19, pages 239–245, New York, NY, USA, January 2019. Association for Computing Machinery.  ISBN 978-1-4503-6324-2.  doi:10.1145/3306618.3314293.  URL https://doi.org/10.1145/3306618.3314293.
  * Selvaraju et al. [2019] R. R. Selvaraju, S. Lee, Y. Shen, H. Jin, S. Ghosh, L. Heck, D. Batra, and D. Parikh.  Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded.  _arXiv:1902.03751_ , 2019.  URL http://arxiv.org/abs/1902.03751.
  * Miller [2019] T. Miller.  Explanation in artificial intelligence: Insights from the social sciences.  _Artificial Intelligence_ , 267:1–38, February 2019.  ISSN 0004-3702.  doi:10.1016/j.artint.2018.07.007.  URL http://www.sciencedirect.com/science/article/pii/S0004370218305988.
  * Zhang et al. [2020] Y. Zhang, Q. V. Liao, and R. K. E. Bellamy.  Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making.  _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , pages 295–305, 2020.  doi:10.1145/3351095.3372852.  URL http://arxiv.org/abs/2001.02114.  arXiv: 2001.02114.
  * Lakshminarayanan et al. [2017] B. Lakshminarayanan, A. Pritzel, and C. Blundell.  Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles.  _arXiv:1612.01474_ , 2017.  URL http://arxiv.org/abs/1612.01474.
  * Amodei et al. [2016] D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané.  Concrete Problems in AI Safety.  _arXiv:1606.06565_ , 2016.  URL http://arxiv.org/abs/1606.06565.
  * MacKay [1992] D. J. C. MacKay.  A Practical Bayesian Framework for Backpropagation Networks.  _Neural Computation_ , 4(3):448–472, May 1992\.  ISSN 0899-7667.  doi:10.1162/neco.1992.4.3.448.
  * Blundell et al. [2015] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra.  Weight Uncertainty in Neural Network.  In _International Conference on Machine Learning_ , pages 1613–1622. PMLR, June 2015.  URL http://proceedings.mlr.press/v37/blundell15.html.  ISSN: 1938-7228.
  * Lipton [2017] Z. C. Lipton.  The Mythos of Model Interpretability.  _arXiv:1606.03490_ , 2017.  URL http://arxiv.org/abs/1606.03490.
  * Sundararajan et al. [2017] M. Sundararajan, A. Taly, and Q. Yan.  Axiomatic Attribution for Deep Networks.  _arXiv:1703.01365_ , 2017.  URL http://arxiv.org/abs/1703.01365.
  * Kingma and Welling [2014] D. P. Kingma and M. Welling.  Auto-Encoding Variational Bayes.  _arXiv:1312.6114_ , 2014.  URL http://arxiv.org/abs/1312.6114.
  * Kingma et al. [2015] D. P. Kingma, T. Salimans, and M. Welling.  Variational Dropout and the Local Reparameterization Trick.  _arXiv:1506.02557_ , 2015.  URL http://arxiv.org/abs/1506.02557.
  * LeCun et al. [1989] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel.  Backpropagation applied to handwritten zip code recognition.  _Neural Computation_ , 1(4):541–551, December 1989.  ISSN 0899-7667.  doi:10.1162/neco.1989.1.4.541.  URL https://doi.org/10.1162/neco.1989.1.4.541.
  * Zhou et al. [2015] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba.  Object Detectors Emerge in Deep Scene CNNs.  _arXiv:1412.6856_ , 2015.  URL http://arxiv.org/abs/1412.6856.
  * Kaggle [2014] Kaggle.  Dogs vs. Cats, January 2014.  URL https://kaggle.com/c/dogs-vs-cats.  Microsoft Research. https://kaggle.com/c/dogs-vs-cats (accessed 28/11/2020).
  * Codella et al. [2019] N. Codella, V. Rotemberg, P. Tschandl, M. E. Celebi, S. Dusza, D. Gutman, B. Helba, A. Kalloo, K. Liopyris, M. Marchetti, H. Kittler, and A. Halpern.  Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC).  _arXiv:1902.03368_ , 2019.  URL http://arxiv.org/abs/1902.03368.
  * Lecun et al. [1998] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, November 1998.  ISSN 1558-2256.  doi:10.1109/5.726791.
  * Krizhevsky et al. [2012] A. Krizhevsky, I. Sutskever, and G. E. Hinton.  ImageNet Classification with Deep Convolutional Neural Networks.  _Advances in Neural Information Processing Systems_ , 25:1097–1105, 2012.  URL https://proceedings.neurips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html.
  * Ioffe and Szegedy [2015] S. Ioffe and C. Szegedy.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.  _arXiv:1502.03167_ , 2015.  URL http://arxiv.org/abs/1502.03167.
  * Kingma and Ba [2017] D. P. Kingma and J. Ba.  Adam: A Method for Stochastic Optimization.  _arXiv:1412.6980_ , 2017.  URL http://arxiv.org/abs/1412.6980.
  * Chen et al. [2017] L. Chen, G. Papandreou, F. Schroff, and H. Adam.  Rethinking Atrous Convolution for Semantic Image Segmentation.  _arXiv:1706.05587_ , 2017.  URL http://arxiv.org/abs/1706.05587.