11institutetext: University of Konstanz, Universitätsstraße 10, 78464
Konstanz, Germany 11email<EMAIL_ADDRESS>
# Introducing the Attribution Stability Indicator:
a Measure for Time Series XAI Attributions

Udo Schlegel 11 0000-0002-8266-0162 Daniel A. Keim 11 0000-0001-7966-9740

###### Abstract

Given the increasing amount and general complexity of time series data in
domains such as finance, weather forecasting, and healthcare, there is a
growing need for state-of-the-art performance models that can provide
interpretable insights into underlying patterns and relationships. Attribution
techniques enable the extraction of explanations from time series models to
gain insights but are hard to evaluate for their robustness and
trustworthiness. We propose the Attribution Stability Indicator ($ASI$), a
measure to incorporate robustness and trustworthiness as properties of
attribution techniques for time series into account. We extend a perturbation
analysis with correlations of the original time series to the perturbed
instance and the attributions to include wanted properties in the measure. We
demonstrate the wanted properties based on an analysis of the attributions in
a dimension-reduced space and the $ASI$ scores distribution over three whole
time series classification datasets.

###### Keywords:

Explainable AI XAI Evaluation XAI for Time Series.

##  1 Introduction

Artificial intelligence (AI) has become an indispensable part of our everyday
lives. We encounter it in various forms, from the tailored advertisements we
receive on social media to the conversational AI (chatbots) that utilize deep
neural networks to answer user and customer queries. However, as deep neural
network models grow in complexity, understanding the rationale behind their
decisions becomes increasingly challenging [9]. This lack of interpretability
can have grave repercussions in critical domains like finance, healthcare, and
transportation, potentially resulting in financial losses, medical errors, or
even loss of life due to incorrect decisions made by intricate models [21]. To
address these concerns, a promising solution lies in the adoption of
explainable artificial intelligence (XAI). XAI aims to shed light on the inner
workings of complex models and the factors influencing their decision-making
[9]. Within this field, one area of particular interest is time series data,
characterized by its sequential nature and interdependencies between
observations. As the volume of data generated by sensors increases and complex
models are applied to tackle more tasks, the need for explainability in time
series analysis becomes evident [32].

In recent years, there has been a growing focus on developing explainable
artificial intelligence (XAI) techniques specifically designed for time series
data [32]. These techniques often rely on the concept of attributions, which
aim to uncover the contributions of individual features and time points to the
overall predictions made by a model [32]. By revealing which parts of the
input data are most relevant to the output, attributions assist users in
understanding the decision-making process of the model [23]. However,
evaluating these attributions poses a non-trivial challenge [25]. To tackle
this challenge, perturbation analysis has emerged as a promising evaluation
technique for assessing the quality of explanations in time series data [22,
29]. Perturbation analysis involves systematically modifying the input data
and assessing the resulting impact on the attributions generated by XAI
methods [25]. By perturbing the input data, it becomes possible to evaluate
the robustness of the explanations provided by XAI methods [32]. For instance,
Schlegel et al. [25] begin to explore the evaluation of attributions in detail
and reveal a gap for an attribution measure incorporating the distance of the
original and perturbed samples.

We propose the Attribution Stability Indicator ($ASI$), which incorporates
various similarity measures between time series, attributions, and
predictions. We incorporate a requirements analysis to reveal the properties
of an attribution technique measure to collect important aspects for working
approaches in the direction of measuring. Based on this requirement analysis,
we propose five essential factors: a class flip, prediction probability
changes, attribution distances, time series perturbation distances, and user-
based weighting. $ASI$ then incorporates our identified requirements by using
the class flip as a binary indicator, the Jensen-Shannon distance [6] for the
prediction probabilities, the Pearson correlation [7] for the change in the
attributions and for the change in the time series as measures. As the ranges
from these measures are between zero and one, we further introduce weighting
to steer the measure in a user-wanted direction. We also provide default
values for these hyperparameters. While the overall $ASI$ approach is quite
general, we focus on time series to demonstrate applicability and incorporate
existing working methods from the literature [2].

Thus, we contribute (1) a requirements analysis of the properties a measure
for attribution techniques needs to evaluate these, (2) a perturbation
analysis-based measure to include time series and attribution properties into
the evaluation process, and (3) a small-scale evaluation to compare various
attribution techniques. At first, we relate our measure to other approaches to
XAI evaluation. Next, we break down the evaluation of attributions for time
series using a perturbation analysis and identify essential requirements for
attributions we want to measure. Afterward, we introduce the attribution
stability indicator parts and definitions based on these requirements. Then,
we use the FordA, FordB, and ElectricDevices datasets [5] on time series
classification to demonstrate the $ASI$ measure against a flip like, e.g.,
Schlegel et al. [22] proposed. Lastly, we conclude our proposed approach and
motivate further extensions.

Results and source code of the experiments is online available at:
https://github.com/visual-xai-for-time-series/attribution-stability-indicator

##  2 Related Work

Explainable AI (XAI) has made significant progress in recent years, driven by
various surveys [9, 1] and techniques such as LIME [20] and SHAP [15]. XAI is
growing in many directions, starting from computer vision [35] and moving
towards time series [32]. However, evaluating explanations remains a nascent
area, with limited efforts to benchmark different techniques against one
another [11]. Several studies have begun to compile a range of evaluation
techniques [19], which have been categorized into five dimensions: mental
model, explanation usefulness and satisfaction, user trust and reliance,
human-AI task performance, and computational measures. The initial dimensions
primarily revolve around evaluating explanations in collaboration with humans,
making them significantly influenced by human factors. On the other hand, the
computational measures focus solely on the automatic evaluation of
explanations, excluding human factors [19]. Here, we examine the computational
measures, explicitly exploring the fidelity of the attribution technique
applied to the model toward an analysis of the fitness of the explanation.

Mainly for time series, we have various levels on which time series model
decisions can be explained [32]. We focus on time-point explanations to work
directly on the raw input time series and attributions based on the taxonomy
from Theissler et al. [32]. Previous research conducted by Schlegel et
al.[22], as well as others[29, 17, 32], has demonstrated the effectiveness of
attribution techniques such as LIME [20], SHAP [15], LRP [3], GradCAM [27],
Integrated Gradients [31], and others [26] in generating meaningful
attributions from time series models. However, in most cases, these
attributions are evaluated using computational measures without further
scrutiny, as observed by Mercier et al. [17]. This limitation calls for a more
in-depth examination computationally and by humans to gain deeper insights
into the attributions.

Schlegel et al.[22] initiated their research by applying perturbation analysis
to attribution techniques in TSC, demonstrating that techniques designed for
images and text are also effective for time series data. Building on these
initial experiments, they further refined their approach by incorporating
additional perturbation functions, resulting in a more comprehensive
evaluation of fidelity[25]. Mercier et al.[17] expanded on these perturbations
by incorporating additional measures from the image domain, such as
(in)fidelity and sensitivity[34]. Simic et al.[29] extended the methods
proposed by Schlegel et al.[25] by incorporating out-of-distribution values
for the perturbation and provided guidelines for selecting attribution
techniques and determining the appropriate window size for perturbation
windows. Similarly, Turbe et al. [33] further enhanced previous approaches by
introducing an additional metric that improves the comparison of attribution
techniques and their ability to demonstrate fidelity towards the underlying
model. However, none of these methods incorporate the distance in the
attribution space and assume that these should also differ.

##  3 Evaluation of Attributions for Time Series

As a starting point for evaluating attributions for time series, we begin with
the perturbation analysis described by Schlegel et al. [25, 24] and use the
definitions and notions of Theissler et al. [32] in their survey.

###  3.1 Definitions for a Perturbation Analysis

We define a time series classification dataset $D=(X,Y)$, where $X$ represents
the time series samples and $Y$ denotes the corresponding time series labels.
The set $X=\\{ts_{1},ts_{2},\dots,ts_{n}\\}$ consists of $n$ time series
samples, each $ts_{i}$ containing $m$ time points represented as an ordered
set of $m$ real-valued time points $ts=\\{tp_{1},tp_{2},\dots,tp_{m}\\}$.
Here, $tp_{1}$ signifies the value at the $i$th time point of $ts$. The set
$Y=\\{l_{1},l_{2},\dots,l_{n}\\}$ includes $n$ labels, with one label assigned
to each time series sample. Let $M(ts,\theta)=y^{\prime}$ be a time series
classification model that predicts the label $y^{\prime}$ based on the input
time series $ts$, using the parameters $\theta$ and for all time series in a
dataset: $M(X,\theta)=Y^{\prime}$. Additionally, we introduce $A(X,M,\theta)$
as an XAI technique designed to generate attributions for the time series
data. The attributions for $ts$ produced by $A$ are represented as
$A(ts,M,\theta)=att$ and $att=\\{a_{1},a_{2},\dots,a_{m}\\}$. Here, $a_{i}$
corresponds to the attribution score assigned to the $i$th time point of $ts$.
For all time series in a dataset: $A(X,M,\theta)=Att$ with
$Att=\\{att_{1},att_{2},\dots,att_{n}\\}$. These attributions are generated
using the classification model $M$ and the parameters $\theta$ associated with
the technique.

We introduce a controlled perturbation function denoted as $g$, which operates
on and modifies the dataset $X$ to facilitate perturbation analysis.
Specifically, we define a perturbed time series dataset $X^{\prime}$ as
follows: $X^{\prime}=g(X,Att,\xi)$. The perturbation function $g$ alters the
dataset $X$ based on the attributions $att$ and a threshold value $\xi$. The
modification depends on the chosen function $g$, such as perturbing values to
zeros. The threshold $\xi$ can be manually specified or determined using
another function, for instance, by setting it to the 90th percentile of the
attributions. This way, attributions (e.g., $a_{i}$, representing the $i$th
element) exceeding the threshold will be modified to a predetermined value,
such as zero.

Figure 1: Starting from a time series $ts$, we use a selected attribution
technique $A$ to get attributions. Based on the attributions, we use a
selected perturbation function $g$ to set highly relevant time points, e.g.,
to zero. Further information in Schlegel et al. [25].

In Figure 1, we illustrate the approach using zero perturbations on
attributions with high values, showcasing the practical implementation of the
technique. To get more information on the perturbation strategies and the
influences of using different perturbation strategies on various datasets, we
suggest the work by Schlegel and Keim [24].

We evaluate the performance of the model $M$ on both the original dataset $X$
and the perturbed dataset $X^{\prime}$ by obtaining the predictions
$M(X)=Y^{\prime}$ and $M(X^{\prime})=Y^{\prime\prime}$, respectively. In line
with the findings of Schlegel et al. [25], we incorporate a quality metric
denoted as $qm$, e.g., accuracy or a similar measure. This metric enables us
to compare the performance of the model $M$ using the original dataset $X$ and
the perturbed dataset $X^{\prime}$. In the context of time series
classification, we assume that the quality metric $qm$ decreases as the
original data changes, implying that the previously assigned labels are no
longer accurate [22]. Moreover, we also assume that an effective attribution
technique, when applied to perturb the most relevant parts of the input data
[11], leads to a more significant decrease in performance.

Figure 2: Projection of the FordA training data attributions of DeepLIFTSHAP
on a CNN using UMAP. (A) is wanted after a perturbation, while (B) is not.

Hence, Schlegel et al. [25] posit the following relationship:

$qm(Y^{\prime},Y)\leq qm(Y^{\prime\prime},Y)$ (1)

Here, $qm(Y^{\prime},Y)$ represents the quality metric for the predictions
$Y^{\prime}$ compared to the actual labels $Y$, while $qm(Y^{\prime\prime},Y)$
corresponds to the quality for the predictions $Y^{\prime\prime}$ obtained
from the perturbed dataset $X^{\prime}$. Depending on the $qm$, the
performance drastically changes. Thus, a more independent measure is needed.

###  3.2 Requirements for a Measure on the Perturbation Analysis

Previous works, such as Schlegel et al. [25], demonstrate that a class change
is one of the most wanted properties to determine an attribution technique’s
fidelity. Thus, a class change in prediction is essential for working
attributions as these reveal the highly relevant parts of the time series for
the models’ prediction (R1). However, a binary decision on the class flip is
sometimes not enough as a measure, as attribution techniques can present part-
time working attributions that do not necessarily lead to a class flip. Thus,
we need to dig into the probabilities of the class prediction of the model and
compare the original to the perturbed instance to find more minor differences
in the perturbation changes (R2).

For a given dataset, we can calculate the attributions for every sample and
use a projection technique to visualize the attributions of the whole dataset.
Figure 2 presents the attributions extracted with Integrated Gradients for the
training data of the FordA dataset applied to a CNN using the projection
technique UMAP [16] (further information on the dataset and the model in
section 4). We have a two-class problem and want suitable attributions so that
these attributions differ quite heavily for each class and build two clusters
to generate a visual representation of the model. Figure 2 demonstrates such a
scenario. A working perturbation for a suitable attribution then changes the
attribution of the perturbed instance toward the other cluster, as in (A). Bad
attribution techniques create perturbed instances with similar attributions,
such as (B). Thus, a promising measure must include the original and the
perturbed instance attribution similarity (R3).

Figure 3: FordA time series with a change of a perturbation function $g$ with
attributions $att$ form $ts_{old}$ to $ts_{new}$ using zero as perturbation
value.

Contrary to the attributions, we want our perturbed instance to have a small
distance to the original time series to change as little as possible and still
get our prediction change. As seen in Figure 3, the more time points we change
using a perturbation function, the more jumps and cuts our novel time series
has. Depending on our black box, such a change can already lead to a change in
the prediction. However, we do not want to have many cuts and changes, so we
want our measure to include the distance from the original to the perturbed
instance to include a minimal change in our evaluation (R4).

Lastly, as we know that not every dataset is the same and some tasks have
different requirements, we need to be able to weigh the factors we identified
in our analysis. E.g., for some applications, the change in prediction is much
more critical to understanding the model’s weaknesses. Thus, we want to have
an attribution technique that reveals such flips. In further scenarios, we
want to investigate our attributions as visualizations with, e.g., a
projection such as in Figure 2, so we need to weigh our attributions heavier.
Thus, even with the previous requirements, we need to be able to include user
necessities (R5).

Summary – As we described before, a measure for the analysis of attribution
techniques either on a single sample or on a whole dataset needs the following
requirements: (1) change the predicted label (a class flip after the
perturbation analysis based on the attribution), (2) a drastic change in the
probabilities of the output of the model, (3) a significant distance between
the original and the perturbed samples attribution due to a change in the
prediction, (4) only minor changes between the original time series and the
perturbed instance are needed to decision boundaries, and (5) lastly,
weighting helps to let users steer the different requirements based on their
properties and needs.

###  3.3 The Attribution Stability Indicator as a Measure

On the perturbation, we propose the Attribution Stability Indicator ($ASI$) as
another measure to not only use one single quality metric as, e.g., Schlegel
et al. [22] or Simic et al.[29] to include the requirements we identified. To
structure and introduce the parts of $ASI$, we define our approach to tackle
the requirements and give some insights into the selected techniques.

Requirement 1 – We start with the most straightforward requirement we have
identified, targeting the classification change. To get such a change, we use
the selected label by the model with $x=ts$ to $M(ts)=y^{\prime}_{pre}$ and
the prediction after the perturbation $M(g(ts,att)=y^{\prime}_{new}$. Further,
we use the function $b_{f}$ as we want to have a low score for a working
technique:

$b_{f}(X,Y)=\begin{cases}1,&\text{if }\text{argmax}(X)=\text{argmax}(Y)\\\
0,&\text{otherwise}\end{cases}$ (2)

With $X$ and $Y$ being $y^{\prime}_{pre}$ and $y^{\prime}_{new}$ respectively
and argmax extracting the index of the largest value. Leading us in the end
to:

$b_{f}(M(ts),M(g(ts,att))$ (3)

Requirement 2 – Next, we target the change of the prediction probability
distribution with the Jensen-Shannon distance [6]. The Jensen-Shannon
divergence (JSD) can be defined as:

$JSD(P\parallel
Q)=\frac{1}{2}\left(D_{KL}\left(P\parallel\frac{P+Q}{2}\right)+D_{KL}\left(Q\parallel\frac{P+Q}{2}\right)\right)$
(4)

With $P$ and $Q$ as the two probability distributions and $D_{KL}$ as the
Kullback–Leibler divergence. By taking the square root, the JSD, Equation 4,
can be used as a distance (Endres and Schindelin [6]) reforming the JSD to:

$JS\\_{dist}(P,Q)=\sqrt{\frac{1}{2}\left(D_{KL}\left(P\parallel
m\right)+D_{KL}\left(Q\parallel m\right)\right)}$ (5)

With $m$ as the pointwise mean of $P$ and $Q$. We use the $JS_{dist}$,
Equation 5, for the similarity between the old $p$ predicted probabilities of
the model and the new ones $p_{pert}$. As the $JS_{dist}$ already uses our
wanted range from zero to one, we can use that value as it is. However, as a
one in the distance describes a working change in the distribution, we have to
change the one to a zero in the working case, leading us to:

$1-JS_{dist}(M(ts),M(g(ts,att)))$ (6)

We decided to use the Jensen-Shannon distance as we want to have a symmetric
measure to combine it with other symmetric measures in our approach, such as
the correlation for the time series and attributions. However, other symmetric
distribution measures, such as the Hellinger distance [10] ($H_{dist}$) or the
Bhattacharyya distance [4] (does not obey the triangle inequality), are also
possible. In a preliminary experiment between $JS_{dist}$ and $H_{dist}$, we
observed that the $JS_{dist}$ emphasizes differences in the tails of the
distributions, while the $H_{dist}$ is more sensitive to the central parts of
the distribution. So, the $JS_{dist}$ focuses more on outliers while
$H_{dist}$ focuses more on the shape. For our measure, the $JS_{dist}$ is more
advantageous as we are more interested in small changes and not the overall
dissimilarity of the shapes.

Requirement 3+4 – For the attribution and time series similarity, we decided
on the Pearson correlation coefficient from the original $ts$ to the perturbed
instance $ts_{pert}$ as the values range from minus one to plus one. We use
the base definition for the Pearson correlation coefficient:

$\rho(X,Y)=\frac{{\text{cov}(X,Y)}}{{\sigma_{X}\sigma_{Y}}}$ (7)

With $X$ and $Y$ being our $ts$ and $ts_{pert}$ respectively and
$\text{cov}(X,Y)$ the covariance between them as:
$\text{{cov}}(X,Y)=\frac{{\sum_{i=1}^{n}(X_{i}-\sigma_{X})(Y_{i}-\sigma_{Y})}}{{n-1}}$.
$\sigma_{X}$ and $\sigma_{Y}$ describe the means of $X$ and $Y$. As our
$\rho$, Equation 7, is always between minus one and plus one, we use these
borders to normalize the result to zero to one range to include it in the
formula as it is. Leading us to:

$pcc(X,Y)=\frac{\rho(X,Y)+1}{2}$ (8)

And respectively for the attributions to:

$pcc(att,A(g(ts,att),M))$ (9)

And the time series with a change to highlight large distances:

$1-pcc(ts,g(ts,att))$ (10)

We used the Pearson correlation coefficient due to its simplicity and
interpretability. The Pearson correlation is a straightforward and easy-to-
understand measure. The calculation of the Pearson correlation involves simple
mathematical operations, which makes it computationally efficient. Also, the
Pearson correlation provides a clear interpretation of the strength and
direction of the relationship between two time series. The correlation
coefficient allows us to quickly compare different pairs of variables and
assess their level of association. However, other correlation measures are
also possible with a bound between zero and one or some fixed bound and
normalization. Generally, any distance measure with bounds, which can be
modified to zero and one to enable uniform weighting and direction between the
different internal measures, can be incorporated into the measure. We still
advise using a correlation to include

Requirement 5 – To enable users to gain influence into the steering of the
measure, we introduce a weighting term $W$ with a weighting between zero and
one $w_{i}\in\\{0,1\\}$ to get to our weighting factor for our four other
requirements $W=(w_{1},w_{2},w_{3},w_{4})$. E.g. if a user wants to find
attribution techniques that can be interpreted using projections, the weight
for the attribution distance can be adjusted to a higher weight. Thus, the
case of (A) in Figure 2 is much more likely than case (B) to gain low scores
in a measure using the previous requirements. Also, through such a weighting,
we can mimic the raw flip amount using $W_{\text{raw}}=(1,0,0,0)$ for our
approach.

Attribution Stability Indicator ($ASI$) – To put everything together with our
weighting $W=(w_{1},w_{2},w_{3},w_{4})$, we get to:

$\begin{split}ASI(M,ts,att,W)&=\frac{1}{4}(w_{1}\times
b_{f}(M(ts),M(g(ts,att)))\\\ &+w_{2}\times(1-JS_{dist}(M(ts),M(g(ts,att))))\\\
&+w_{3}\times pcc(att,A(g(ts,att),M))\\\
&+w_{4}\times(1-pcc(ts,g(ts,att))))\end{split}$ (11)

First, we include our binary flip, next our probability distribution
similarity, then our correlation between attributions, and lastly, our time
series correlation. The weighting enables the focus on specific aspects of the
requirements heavier and users to steer the whole score in a desired
direction. The function ranges from zero to one, with a lower score presenting
better results. We also discussed incorporating the Euclidean distance or
dynamic time warping for the time series and similar distances. However, these
can have arbitrary values; in some cases, a comparison is rather not
understandable.

##  4 Experiment Setup and Results

Our experiment analyzes data characteristics using three of the most
exhaustive datasets (FordA, FordB, ElectricDevices) from the UCR benchmark
dataset collection [5]. While we specifically examine these datasets, it is
essential to note that our approach applies to any time series classification
dataset. The FordA and FordB datasets consist of sensor data and have a length
of 500 time points each. The data is a recording of the sound of a motor
engine running with abnormalities during the run-time. FordA and FordB are
utilized for binary classification tasks related to anomaly detection. The
FordA dataset comprises 3601 samples in the training and 1320 in the test. The
FordB dataset comprises 3636 samples in the training and 810 in the test. The
ElectricDevices dataset has a length of 96 time points and seven classes. The
training set has a size of 8926, and the test set is 7711 time series long.

We explore a CNN and a ResNet architecture in our investigation. The CNN
architecture is designed for the FordA dataset and the ResNet for the
ElectricDevices. However, both architectures are applied to all three
datasets. The CNN architecture comprises three 1D convolutional layers and two
fully connected layers. The Conv1D layers vary in kernel size and incorporate
batch normalization and max pooling. Rectified linear units (ReLU) are used as
activation functions. The ResNet architecture follows the proposed
architecture for time series classification by Fawaz et al. [13] with three 1D
convolutional ResNet blocks and residual skip connections. We use a batch size
of 120 to train the model and employ the Adam optimizer [14]. We further train
both networks for 500 epochs without an earlier stopping and use the cross-
entropy loss.

The performances of our models on the different datasets are as follows: For
the FordA dataset, our CNN (ResNet) achieves an accuracy of 0.99 (0.97) on the
train and 0.89 (0.94) on the test set. For FordB, it is for the CNN (ResNet)
0.99 (0.98) on the train and 0.72 (0.80) on the test set. For ElectricDevies,
the CNN (ResNet) has 0.97 (0.90) and 0.71 (0.74) respectievly. These results
indicate a clear case of overfitting, as the models excessively adapt to the
training data and struggle to generalize to unseen samples. Despite our
models’ simplicity and overfitting, it is worth noting, as demonstrated by
Ismail Fawaz et al. [13], that these still exhibit performance comparable to
state-of-the-art.

To illustrate the $ASI$ measure, we apply Saliency [30], DeepLIFT [28],
Integrated Gradients [31], GradientSHAP, and DeepLIFTShap [15] on our selected
datasets and our models and use the most suitable perturbation strategy for
the datasets based on the results of Schlegel and Keim [24] (OOD Low Sub for
FordA, Zero Sub for FordB, and Global Max for ElectricDevices) using the
implementation in Captum111https://captum.ai/. However, to mitigate some of
the overfittings of Schlegel and Keim [24], we changed the architecture of the
CNN by adding Batch Normalization [12] and observed fewer flips of all
training data. Thus, the most suitable perturbation strategy could have
changed, which needs further experiments. We excluded Occlusion,
ShapleyValueSampling, and KernelSHAP [15] as these need further tuning and
more runtime to process a whole dataset. We plan to run an experiment with
these in future works. As we want to focus on attribution and time series, we
use the weights $W=(0.5,1,0.5,3)$ to visualize the projections and create a
surface for the attributions. However, we can also use the metric to compare
the performance of the attributions on one sample..

Hypothesis – For our experiment, we collected the following hypothesis: The
number of flips can be the same or similar for different attribution
techniques [24], while $ASI$ can show a score more diverse than the flips or
the quality metric change from the perturbation analysis [22]. $ASI$ can
reveal highly relevant time points through a more diverse score and weighting.
Further, the $ASI$ score can reveal spurious correlations, e.g., also shown by
Schlegel and Keim [24] for a CNN on the FordA dataset when applied to the
whole dataset.

Figure 4: Different attribution techniques with the $ASI$ metric on a sample
of the FordA test dataset for our CNN. GradientSHAP wins by a small margin
against all others. The top row presents the attributions for the original
time series and the perturbed instance time series. The bottom row shows the
time series and the perturbed instances.

Results – Figure 4 illustrates the results of $ASI$ for our selected
attribution techniques and visualizes the time series and attributions. The
top row shows the attributions for the original time series instance in blue
and those after the perturbation in orange. Here, GradientSHAP and Integrated
Gradients perform best on the attributions from the original to the perturbed
instances by having a lower Pearson correlation coefficient. However,
inspecting the attributions based on a line plot can be tricky and misleading
[23]. The bottom row visualizes the time series instance in blue and the
perturbed instance in orange. Saliency does not change the time series much,
while the others do pretty heavily. The perturbation function here modifies
relevant time points to zero and also, on both sides, two more points so that
for every relevant variable, five time points are perturbed. The score of
Equation 10 for Saliency is much lower because of the low change of the time
series toward the perturbed instance.

To accept or reject our previously collected hypothesis, we need to focus not
only on one sample but on whole datasets and their attributions. Collecting
all possible scores for $ASI$ on both the training and test datasets can be
visualized in a distribution overview to gain insights into the attribution
techniques ( Figure 5 and Figure 6 top). The $ASI$ score and the number of
flips are also important (middle), and the projection of the attributions is
necessary for our assumption (bottom). At first glance, we can see that the
$ASI$ score is more diverse than the number of flips in the figures, Figure 5
and Figure 6 (static amount of flips and diverse $ASI$ score for the training
data). This result can be seen for all our datasets and confirms our first
hypothesis.

Figure 5: Top row: $ASI$ distribution scores for our CNN model on the FordA
train dataset using Saliency, GradientSHAP, DeepLIFT, DeepLIFTSHAP, and
Integrated Gradients. Bottom row: the projections of the attributions using
UMAP with the class ground truth as coloring. The more Gaussian the $ASI$
distribution looks, and the farther the classes are separated, the better the
projections are.

We want to find interesting points in the attribution projection ( Figure 2).
Thus, we set for our weighting $W=(0.5,1,0.5,3)$ to have a high distance
between the attributions before and after the perturbation. Based on such an
assumption, a projection of the attributions demonstrates two separate
clusters for a dataset with two classes. Our focus works quite well for
DeepLIFT and DeppLIFTSHAP on the FordA dataset, as seen in the training and
test data (bottom in Figure 5 and Figure 6). For the training data (Figure 5),
we can identify regions in the DeepLIFT projection where the attributions are
mixed from both classes. Samples in such regions are, in most cases, uncertain
predictions with probabilities not entirely deciding on one class and
containing interesting time points. Analyzing these samples can lead to
problems in the data and the model, e.g., wrong labels or overfitting. We
found that using the distribution of the $ASI$ score facilitates finding
projections for such tasks. The more Gaussian the $ASI$ distributions are, the
better the projections. If we further tune the weights, we can adjust the
$ASI$ score and get an accessible number for the selection of the projection
(e.g., increase attribution, decrease probability weights).

The ResNet has similar patterns for the FordA data and the distribution of the
$ASI$ score. The more Gaussian the $ASI$ distributions are, the better the
projections. However, for the ResNet, a more significant separation
demonstrates worse projections. Visualizing these projections also reveals
different structures for the gradient-based techniques than for the CNN. The
projections are not split in clusters between the ground truth labels but are
mixed heavily. While the SHAP techniques are separated more into clusters.
Such a finding indicates that the skip-connections support the model to learn
features more by heart than in the CNN, as attributions can be similar from
one class to another. Slight changes can lead to changes in the prediction and
the model fit; thus, similar gradients can lead to different predictions.
Adding SHAP helps to overcome the model fit and focuses on the prediction to
have more robust attributions.

Figure 6: Top row: $ASI$ distribution scores for our CNN model on the FordA
test dataset using Saliency, GradientSHAP, DeepLIFT, DeepLIFTSHAP, and
Integrated Gradients. Bottom row: the projections of the attributions using
UMAP with the class as coloring.

For both models, the FordA CNN and the ResNet, we see a flip only in one
direction from a prediction of one to minus one. Thus, the models only learn
specific features necessary for predicting the one class and classify
everything else as minus one. Such a behavior can be seen as a Shortcut
learning [8], which in some cases is undesirable if applied to real-world data
and deployed in a critical environment. Learning a more general pattern that
can separate the classes and flipping not only one class can increase the
robustness of the overall model.

The FordB data is a more complicated dataset for time series classification
models as seen in the test score compared to, e.g., FordA [13]. We found
similar patterns to those of FordA for the CNN applied to FordB. The more
Gaussian the $ASI$ score is distributed, the more separated the ground truth
clusters of the projections of the attributions are. In the FordB case, this
holds for the training and test data for nearly all attribution techniques
except for Saliency, with Saliency having the lowest $ASI$ score. For the
ResNet, we also experience a similar pattern as for FordA. SHAP techniques
have larger $ASI$ scores but create separated clusters, while the gradient-
based methods do not.

ElectricDevices captures seven classes and not only two, and thus is an even
more challenging dataset. The distribution of the $ASI$ score still separates
into a double-peaked distribution. However, the classes are not well divided
into these peaked distributions anymore. These build a more diverse pattern
with some classes distributed into both peaks. Here, our weighting seems quite
misleading as the flips are less influential than the attributions; other
methods are better than those identified by Schlegel and Keim [24]. If we
focus on the best working technique identified by the lowest $ASI$ score, we
get GradientSHAP. Further, we also get similar-looking projections for the
training and test data, which generally provides evidence that the method
works well on the dataset. However, inspecting the projections does not reveal
our wanted intention of clusters separated by the classes. Again, looking at
the distributions and comparing them to Gaussians can help to select working
projections. The more Gaussian and equal-looking the double-peaks look, the
better the projections. If we change the weighting for this dataset to be more
balanced, such as $W=(0.5,1,1,2)$, we get an $ASI$ score corresponding to
reasonable projections. In particular, DeepLIFTSHAP achieves the best $ASI$
score and creates clusters in the projection, capturing different classes in
these clusters.

We further compared our results to the results of Schlegel and Keim [24] and
found some central differences. We suggest investigating the Batch
Normalization and the generation of attributions, as these drastically changed
the projections. We also suggest working with other projection techniques,
such as PCA, and inspecting the visualizations, as UMAP can potentially
demonstrate local minima and non-optimal solutions. We also compared $ASI$ to
the infidelity of Yeh et al. [34] using the implementation of Captum on the
CNN of the FordA. The infidelity demonstrates more diverse scores between the
train and the test data. Thus, there is a change in the most suitable
technique. For the train data, these are Saliency and Integrated Gradients.
For the test data, these are DeepLIFTSHAP and Integrated Gradients. We
experienced Integrated Gradients as a less suitable technique with flawed
projections by using $ASI$. The complete results and further images can be
found in the GitHub repository, which also can be used to reproduce the
results and test other models and datasets.

##  5 Conclusion & Future Work

We presented $ASI$, the attribution stability indicator, as a new measure for
a perturbation analysis without using a quality metric on the whole dataset to
compare attribution techniques. $ASI$ incorporates five requirements we
identified: class flip, prediction probability distribution change, high
attribution distance, low time series distance, and user steering. These
requirements are incorporated using a binary function, the Jensen-Shannon
distance, and the Pearson correlation coefficient. As requirements can change
from dataset to dataset, we include a weighting into $ASI$ to let users steer
the factors. We demonstrated $ASI$ on the FordA, FordB, and ElectricDevices
datasets and discussed the influence of a weighting we selected for an $ASI$
analysis. Based on the distributions over the score for whole datasets, we
further presented how such an overview can lead to more information into the
dataset and model using a projection. While $ASI$ is quite general and can be
easily extended to other data types, we tackle time series as the correlation
between time series is widely used in literature [2], and thus the measure is
easier to interpret and to understand.

However, we also identified shortcomings of $ASI$ and want to improve the
measure with further additions. One of these additions includes an OOD (out-
of-distribution) check for the perturbed time series based on the
attributions. Through such a check, a plausibility check can be introduced
into the perturbation analysis approach to include domain knowledge and a
stronger focus on more real-looking time series. Also, other distance
functions can be added to incorporate more time series properties, such as the
shape of the time series, as long as these are bounded between zero and one.
Another option is using a transformation function and the transformed data for
a distance. Further, a more extensive benchmark with, e.g., TimeReise [18] can
help to find fitting parameters for users. Lastly, studying the attribution
technique, perturbation strategy, and $ASI$ can help to analyze models in more
depth.

#### Acknowledgements

This work has been partially supported by the Federal Ministry of Education
and Research (BMBF) in VIKING (13N16242).

## References

  * [1] Adadi, A., Berrada, M.: Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI). IEEE Access (2018)
  * [2] Aghabozorgi, S., Shirkhorshidi, A.S., Wah, T.Y.: Time-series clustering–a decade review. Information systems (2015)
  * [3] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation. PLOS ONE (2015)
  * [4] Bhattacharyya, A.: On a measure of divergence between two statistical populations defined by their probability distributions. Bull. Calcutta Math. Soc. (1943)
  * [5] Dau, H.A., Bagnall, A., Kamgar, K., Yeh, C.C.M., Zhu, Y., Gharghabi, S., Ratanamahatana, C.A., Keogh, E.: The UCR time series archive. IEEE/CAA Journal of Automatica Sinica (2019)
  * [6] Endres, D.M., Schindelin, J.E.: A new metric for probability distributions. IEEE Transactions on Information theory (2003)
  * [7] Galton, F.: Regression towards mediocrity in hereditary stature. The Journal of the Anthropological Institute of Great Britain and Ireland (1886)
  * [8] Geirhos, R., Jacobsen, J.H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., Wichmann, F.A.: Shortcut learning in deep neural networks. Nature Machine Intelligence (2020)
  * [9] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM computing surveys (CSUR) (2018)
  * [10] Hellinger, E.: Neue Begründung der Theorie quadratischer Formen von unendlichvielen Veränderlichen. Journal für die reine und angewandte Mathematik (1909)
  * [11] Hooker, S., Erhan, D., Kindermans, P.J., Kim, B.: A Benchmark for Interpretability Methods in Deep Neural Networks. In: Advances in Neural Information Processing Systems (2019)
  * [12] Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning (2015)
  * [13] Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., Muller, P.A.: Deep learning for time series classification: a review. Data Mining and Knowledge Discovery (2019)
  * [14] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [15] Lundberg, S., Lee, S.I.: A Unified Approach to Interpreting Model Predictions. In: Advances in Neural Information Processing Systems (2017)
  * [16] McInnes, L., Healy, J., Melville, J.: Umap: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv preprint arXiv:1802.03426 (2018)
  * [17] Mercier, D., Bhatt, J., Dengel, A., Ahmed, S.: Time to Focus: A Comprehensive Benchmark Using Time Series Attribution Methods. arXiv preprint arXiv:2202.03759 (2022)
  * [18] Mercier, D., Dengel, A., Ahmed, S.: TimeREISE: Time Series Randomized Evolving Input Sample Explanation. Sensors (2022)
  * [19] Mohseni, S., Zarei, N., Ragan, E.D.: A multidisciplinary survey and framework for design and evaluation of explainable AI systems. ACM Transactions on Interactive Intelligent Systems (TiiS) (2021)
  * [20] Ribeiro, M.T., Singh, S., Guestrin, C.: ”Why should i trust you?” Explaining the predictions of any classifier. In: International Conference on Knowledge Discovery and Data Mining (2016)
  * [21] Rudin, C.: Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence (2019)
  * [22] Schlegel, U., Arnout, H., El-Assady, M., Oelke, D., Keim, D.A.: Towards a Rigorous Evaluation of XAI Methods on Time Series. ICCV Workshop on Interpreting and Explaining Visual Artificial Intelligence Models (2019)
  * [23] Schlegel, U., Keim, D.A.: Time Series Model Attribution Visualizations as Explanations. TREX: Workshop on TRust and EXpertise in Visual Analytics (2021)
  * [24] Schlegel, U., Keim, D.A.: A Deep Dive into Perturbations as Evaluation Technique for Time Series XAI. International Conference on eXplainable Artificial Intelligence (xAI) (2023)
  * [25] Schlegel, U., Oelke, D., Keim, D.A., El-Assady, M.: An Empirical Study of Explainable AI Techniques on Deep Learning Models for Time Series Tasks. Pre-registration workshop NeurIPS (2020)
  * [26] Schlegel, U., Vo, D.L., Keim, D.A., Seebacher, D.: TS-MULE: Local Interpretable Model-Agnostic Explanations for Time Series Forecast Models. ECML-PKDD Workshop Advances in Interpretable Machine Learning and Artificial Intelligence (AIMLAI) (2021)
  * [27] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In: International Conference on Computer Vision (2017)
  * [28] Shrikumar, A., Greenside, P., Kundaje, A.: Learning Important Features Through Propagating Activation Differences. International Conference on Machine Learning (2017)
  * [29] Šimić, I., Sabol, V., Veas, E.: Perturbation Effect: A Metric to Counter Misleading Validation of Feature Attribution. In: Proceedings of the 31st ACM International Conference on Information & Knowledge Management (2022)
  * [30] Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: visualising image classification models and saliency maps. In: Proceedings of the International Conference on Learning Representations (ICLR) (2014)
  * [31] Sundararajan, M., Taly, A., Yan, Q.: Axiomatic Attribution for Deep Networks. In: International Conference on Machine Learning (2017)
  * [32] Theissler, A., Spinnato, F., Schlegel, U., Guidotti, R.: Explainable AI for Time Series Classification: A review, taxonomy and research directions. IEEE Access (2022)
  * [33] Turbé, H., Bjelogrlic, M., Lovis, C., Mengaldo, G.: Interprettime: a new approach for the systematic evaluation of neural-network interpretability in time series classification. arXiv preprint arXiv:2202.05656 (2022)
  * [34] Yeh, C.K., Hsieh, C.Y., Suggala, A., Inouye, D.I., Ravikumar, P.K.: On the (in) fidelity and sensitivity of explanations. In: Advances in Neural Information Processing Systems (2019)
  * [35] Zeiler, M.D., Fergus, R.: Visualizing and Understanding Convolutional Networks. In: European Conference on Computer Vision (2014)