# Explainable Predictive Process Monitoring: A User Evaluation

Williams Rizzi1,2∗, Marco Comuzzi4, Chiara Di Francescomarino1, Chiara
Ghidini1, Suhwan Lee5,
Fabrizio Maria Maggi2, Alexander Nolte3,6
1Fondazione Bruno Kessler, Via Sommarive, 18, 38123 Povo, Italy
2Free University of Bozen-Bolzano, Piazza Università, 5, 39100 Bolzano, Italy
3University of Tartu, Narva maantee 18, 51009 Tartu, Estonia
4Ulsan National Institute of Science and Technology, Ulsan, South Korea
5Utrecht University, Utrecht, The Netherlands
6Carnegie Mellon University, Pittsburgh, PA, USA

∗To whom correspondence should be addressed; E-mail<EMAIL_ADDRESS>
> Explainability is motivated by the lack of transparency of black-box Machine
> Learning approaches, which do not foster trust and acceptance of Machine
> Learning algorithms. This also happens in the Predictive Process Monitoring
> field, where predictions, obtained by applying Machine Learning techniques,
> need to be explained to users, so as to gain their trust and acceptance. In
> this work, we carry on a user evaluation on explanation approaches for
> Predictive Process Monitoring aiming at investigating whether and how the
> explanations provided (i) are understandable; (ii) are useful in decision
> making tasks; (iii) can be further improved for process analysts, with
> different Machine Learning expertise levels. The results of the user
> evaluation show that, although explanation plots are overall understandable
> and useful for decision making tasks for Business Process Management users —
> with and without experience in Machine Learning — differences exist in the
> comprehension and usage of different plots, as well as in the way users with
> different Machine Learning expertise understand and use them.

> Keywords - Predictive Process Monitoring, Process Mining, Explainable
> Artificial Intelligence, Explanation Plots, Qualitative Observational Study

##  1 Introduction

Predictive Process Monitoring (PPM) is a branch of Process Mining that aims at
providing predictions on the future of an ongoing process execution by
leveraging past historical execution traces. An increasing number of PPM
approaches leverage machine and deep learning techniques in order to learn
from past historical execution traces the outcome of an ongoing process
execution, the time remaining till the end of an ongoing execution, or the
next activities that will be performed.

In many of these applications, users are asked to trust a model that supports
them in making decisions. Therefore, understanding the rationale behind the
predictions would certainly help users decide when to trust or not to trust
these systems. In recent years, Explainable Artificial Intelligence (XAI) has
been investigating the problem of explaining machine learning models so as to
foster trust and acceptance of these models. Some of the recent XAI approaches
have also been applied and investigated in the field of PPM (Galanti ., 2020;
Velmurugan ., 20212; Rizzi ., 2020) in order to make a predictor returning,
besides predictions, also prediction explanations (for instance in the form of
explanation plots). None of these works has, however, investigated whether
users actually understand and use these plots.

In this paper, we focus on investigating whether the users actually understand
the explanation plots returned by XAI techniques in the context of PPM
problems, whether these plots are able to support users when they need to make
decisions, as well as how to improve them. Specifically, we aim at answering
the following research questions:

  1. RQ1.  _How do users make sense of explanation plots?_

  2. RQ2.  _How can explanation plots support users in decision making?_

  3. RQ3.  _How can explanation plots be improved?_

In order to answer these research questions, we carried out a user evaluation
with users working in the field of PPM and with different levels of Machine
Learning (ML) expertise. We provided them with a problem, with some
predictions and corresponding explanation plots. We hence asked the users (i)
some questions for checking their comprehension of the plots, (ii) to make a
decision by leveraging the plots, and (iii) how they would change the
explanation plots to make them more useful and understandable. The results of
the user evaluation show that, although explanation plots are overall
understandable and useful for decision making tasks for all participants,
differences exist in the comprehension and usage of different plots, as well
as in the way users with different ML expertise understand and use them. In
particular, the study reveals that, while on the one side for ML experts
understanding the explanation plots is easier than for participants without ML
experience, on the other hand, using the plots for making decisions is easier
for participants who do not have ML experience. ML experts, indeed, tend to be
more conservative in their decisions, feeling that explanation plots, mainly
showing correlations and not causalities, do not provide them with enough
evidence to make specific recommendations. Moreover, the evaluation carried
out revealed interesting suggestions and desiderata for explanation plots,
such as the need for interactive elements in the user interface as well as for
what-if analysis support.

In the next sections, we first provide some background knowledge useful for
understanding the remainder of the paper (Section 2) and summarize the related
work (Section 3). Then, Section 4 describes the methodology used for carrying
out the user evaluation and answering the research questions. Finally, Section
5 and Section 6 report and discuss the findings of the user evaluation.

##  2 Background

In this section, we report the main concepts discussed in this paper, i.e.,
Predictive Process Monitoring (Section 2.1), Explainability approaches
(Section 2.2), as well as Explainability applied to the Predictive Process
Monitoring field (Section 2.3).

###  2.1 Predictive Process Monitoring

PPM (Maggi ., 20141; Di Francescomarino, 2019) is a branch of process mining
(van der Aalst, Adriansyah ., 2011) that aims at exploiting event logs of past
historical process execution traces to predict how ongoing (uncompleted)
process executions will unfold up to their completion. Typical examples of
predictions on the future of an execution trace relate to its completion time,
to the fulfilment or violation of a certain predicate, or to the next sequence
of activities that will be executed.

PPM approaches are typically characterized by two phases: a training phase, in
which a predictive model is learned from historical traces, and a prediction
phase, in which the predictive model is queried for predicting the future
developments of an ongoing case. Recent works addressing PPM challenges mainly
leverage machine learning or statistical models, i.e., implicit models of the
process rather than explicit process models (Maggi ., 20142; Di
Francescomarino ., 2019; Tax ., 2017; Camargo ., 2019; Evermann ., 2017).

PPM approaches can be classified based on the predictions they provide
(Márquez-Chamorro ., 2018; Di Francescomarino ., 2018):

  1. (i)

numeric predictions, i.e., continuous measures. Typical examples of numeric
predictions are the remaining time of an ongoing execution, its duration or
cost (van der Aalst, Schonenberg Song, 2011; Polato ., 2018; BF. van Dongen .,
2008; Folino ., 2013; Verenich ., 2019).

  2. (ii)

outcome-based predictions, i.e., predictions related to categorical or boolean
outcomes. Typical examples of outcome-based predictions are the class of risk
of a given execution or to the fulfilment of a predicate along the lifecycle
of an execution trace (Maggi ., 20142; Di Francescomarino ., 2019; Leontjeva
., 2015; Teinemaa ., 2019);

  3. (iii)

next activity predictions, i.e., predictions related to the activities that
are going to be executed. For example, it is possible to predict the sequence
of activities of a process execution from the current time upon its completion
(Tax ., 2017; Camargo ., 2019; Evermann ., 2017).

Among the tools available that implement PPM techniques, the most used and
known ones are: ProM (BF. van Dongen ., 2005), Apromore (La Rosa ., 2011), and
Nirdizati (Rizzi ., 2019). While the first two are well-known general-purpose
tools covering different branches of the process mining field, the latter is
specifically focused on PPM.

###  2.2 Explainability Approaches

The lack of transparency of the black-box ML approaches, which do not foster
trust and acceptance of ML algorithms, motivates the need of explainability
approaches. In the literature, there are two main groups of techniques used to
develop explainable systems, a.k.a.  _explainers_ : post-hoc and ante-hoc
techniques. Post-hoc techniques allow models to be trained as usual, with
explainability only being incorporated at testing time. Ante-hoc techniques
entail integrating explainability into a model since from the training phase.
In this work, we mainly focus on post-hoc explainers since we want to use
these instruments to improve the state-of-the-art approaches for PPM available
in the literature without altering them but building new solutions on top of
them.

An example of post-hoc explainer is Local Interpretable Model-Agnostic
Explanations (LIME) (Ribeiro ., 2016), which explains the prediction of any
classifier in an interpretable manner. LIME learns an interpretable model
locally around the prediction and explains the predictive models providing
individual explanations for individual predictions. The explanations are
generated by approximating the underlying model with an interpretable one,
learned using perturbations of the original instance. In particular, each
feature is assigned with an _importance value_ that represents the influence
of that feature on a particular prediction. Another post-hoc explainer is
SHapley Additive exPlanations (SHAP) (Lundberg Lee, 2017). SHAP is a game-
theoretic approach explaining the output of any ML model. It connects optimal
credit allocation with local explanations using the classic Shapley Values
(Shapley, 2016) from game theory and their related extensions. SHAP provides
local explanations based on the outcome of other explainers thus representing
the only possible consistent and locally accurate additive feature attribution
method based on expectations. Other post-hoc explainers (Friedman, 2001) show
the marginal effect of some features using partial dependence plots. In
(Goldstein ., 2015), the authors refine the partial dependence plots
definition providing the visualisation of functional dependencies for
individual observations through Individual Conditional Expectation (ICE)
plots. (Ribeiro ., 2018) is also a model-agnostic system that explains complex
systems using rules called anchors. The anchors, when available, are local
conditions that can explain predictions.

###  2.3 Explainability in Predictive Process Monitoring

Recently, explainability approaches have also been applied and investigated in
the field of PPM (Harl ., 2020; Sindhgatta ., 2020; Weinzierl, Zilker ., 2020;
Galanti ., 2020; Velmurugan ., 20212; Rizzi ., 2020).

Some works focus on applying model-specific explainability approaches to
provide explanations for predictions obtained through neural network
predictive models, e.g., gated graph neural networks (Harl ., 2020),
attention-based LSTM models (Sindhgatta ., 2020), layer-wise relevance
propagation to LSTM (Weinzierl, Zilker ., 2020).

Other works focus on generic or model-agnostic post-hoc explanation approaches
(Galanti ., 2020; Velmurugan ., 20212; Rizzi ., 2020). For example, in (Rizzi
., 2020), explanations are used in order to identify the features leading to
wrong predictions in order to improve the accuracy of the predictive model. In
(Galanti ., 2020, 2021), shapley values (Shapley, 2016) are leveraged for
providing users with explanations in the form of tables explaining the
predictions related to a specific ongoing process execution. In addition, they
use plots relying on heatmaps to specify for each feature and at each point in
time the impact of the feature on a prediction. In Nirdizati (Rizzi ., 2019),
different types of explanation plots are implemented to provide explanations
in the context of binary classifications. In particular, explanation plots at
_event_ , _trace_ and _event log_ levels are provided. For event-based
explanations, LIME (Ribeiro ., 2016) and SHAP (Lundberg Lee, 2017) explanation
plots are adapted to the PPM scenario to measure the importance of each
feature for a prediction provided at a specific trace prefix (see Figure 1).
In order to provide trace-based explanations, a temporal stability (Velmurugan
., 20211; Teinemaa ., 2018) plot is used. This plot allows users to visualize
the importance of each feature at different trace prefixes (see Figure 2),
thus also revealing how stable the importance of the features is for
predictions returned at different prefix lengths. Finally, ICE (Goldstein .,
2015) explanation plots have been implemented in Nirdizati to provide log-
based explanations (see Figure 3). This type of plot reports, for a specific
feature, information about the average value of the predicted label (between 0
and 1 with 0 meaning _false_ and 1 meaning _true_) for the different feature
values, as well as the number of traces in the event log containing each
value.

##  3 Related Work

Due to the rapid growth of the field of XAI and to the key role of users for
these systems, several empirical studies with human subjects have been
conducted to investigate and evaluate XAI systems.

We can roughly classify the human-centered evaluation studies on
explainability based on the specific AI approach for which explanations have
been provided, e.g., neural networks, intelligent agents, random forest,
generic ML, as well as on the form in which explanations are provided, e.g.,
textual, numeric or visual explanations (Vilone Longo, 2021). We mainly focus
here on user evaluations dealing with explainability approaches developed for
machine and deep learning techniques and providing a visual support to the
users, which are the groups of works that are closer to our analysis.

We can classify evaluations for these XAI approaches based on the levels of
tests used (test of _satisfaction_ , test of _comprehension_ or test of
_performance_), as well as the type of tasks the participants carried out
(_verification_ , _forced choice_ , _forward simulation_ , _counterfactual
simulation_ , _system usage_ or _annotation_) (Chromik Schuessler, 2020). The
evaluation can indeed be focused on evaluating the satisfaction of the users
with the explanations or their subjective assessment of the understanding of
the system; their comprehension of the system in terms of the mental model
they have of the system; or the overall performance in terms of human-XAI
system performance. In addition, in verification tasks, participants are asked
about their satisfaction with the explanations; in forced choice tasks, the
user is asked to choose among different explanations; in forward simulation
tasks, participants are provided with data and explanations and asked to
predict the system’s output; in counterfactual simulation tasks, the
participants should predict what input changes are required for getting an
alternative output from the system; in system usage tasks, participants are
asked to use the system for its original purpose, e.g., for decision making
tasks; annotation tasks require participants to provide an explanation based
on the input provided to the system and the produced output.

Some of the works in the literature focus on the evaluation of the only
_satisfaction_ level. For instance, in (Krause ., 2016) an interactive
interface (Prospector) is proposed to the users to understand how the features
of a dataset affect the prediction of a model overall. A team of 5 data-
scientists was asked to interact with this tool for 4 months to debug a set of
models and, at the end of the experiment, the data scientists were interviewed
on whether they feel that the provided support was beneficial for their work.
In (Weitz ., 2021), the effects of incorporating virtual agents for
Explainable AI in a speech recognition system for keyword classification are
investigated. The 60 participants were split into 4 groups: one group received
only LIME explanation visualisations, while the other 3 groups were provided
with additional information through different modalities from a virtual agent
(text, voice and virtual presence). The results revealed a linear trend of the
user’s perceived trust: the visual presence of the agent combined with a voice
output resulted in greater trust than the output of text or the voice output
alone. In (Spinner ., 2020), _explAIner_ , a visual analytic system for
interactive XAI, is evaluated by 9 participants with different levels of
expertise. The users were asked to use the system and their feedback was then
collected. Results reveal that the system is intuitive and helpful in the
daily analysis routine.

Other works focus not only on the evaluation of the satisfaction level but
also of the users’ _comprehension_. For instance, in (Lundberg Lee, 2017),
users were asked to carry out an _annotation_ task, i.e., they were asked to
provide explanations on simple models. The human explanations were then
compared with LIME, DeepLIFT and SHAP explanations and their consistency with
human intuition evaluated. The results show that SHAP explanations are closer
to human ones than the ones of the other approaches. Also in (Tullio ., 2007),
participants were asked to interact with a predictive system for two weeks at
the end of which they were interviewed to collect their feedback and to check
whether they gained some insights on the logic of the model to be explained by
the explanatory system under analysis. Results show that participants were
able to improve their mental model of the system by reducing initial
misconceptions related to the information used by the system, and better
understanding how it works, also without technical knowledge.

Besides satisfaction and comprehension, a last group of works focuses instead
on evaluating the _performance_ of the user (and the system). For instance, in
(Huysmans ., 2011), the authors evaluate the levels of comprehensibility of
decision table, decision tree and rule-based predictive models with a user
study through _forward-simulation_ tasks. The user study involved 51 non-
expert users who were asked to carry on classification tasks and accuracy,
response time and confidence for each of the three types of predictive models
were then evaluated. The analysis of the results reveals that decision tables
outperform significantly all the other methods. Also in (Wang Yin, 2021),
users were asked to carry on _forward-simulation_ tasks. The work compares and
evaluates four common model-agnostic explainable AI methods on these tasks.
The results show that the effect of AI explanations depend on the level of
domain expertise of the users. Similarly, the empirical evaluation carried out
in Kulesza . (2015) focuses on comparing explanatory debugging and traditional
black-box learning systems. 77 participants were asked to make predictions as
close as possible to the ones provided by the system. The results show that
participants using the explanatory debugging were able to understand the
learning system 50% better than the black-box system and to correct its
mistakes.

Together with _forward-simulation_ tasks, _annotation_ tasks were also
proposed to 64 participants in (Alqaraawi ., 2020). The aim of the work was
evaluating the performance of saliency maps generated by layerwise relevance
propagation (LRP) (Bach ., 2015). The results show that although saliency maps
help users understand the features the system is sensitive to, they provide
limited support for the classification of new instances. Similarly, in (Cheng
., 2019), 199 participants used four different explanation interfaces to
understand a decision-making algorithm. Participants were asked to perform
some _annotation_ and _counterfactual_ tasks. The results show that
interactive explanations improve the understanding of the algorithm, although
it requires more time.

In Hase Bansal (2020), _counterfactual simulation_ tasks were given to
participants, besides _forward-simulation tasks_. In this work, the authors
propose a user study to evaluate the _simulatability_ of different explanation
approaches, where an approach is simulatable if a user is able to predict its
behavior on new instances.

Finally, in some works, participants are asked to carry on _system usage_ and,
more specifically, decision-making tasks. For instance, in (Ribeiro ., 2016),
an evaluation with human subjects is carried out with the two explanation
techniques proposed in the work, i.e., LIME and SP-LIME. Experts and non-
expert participants were required to carry on four decision making tasks,
i.e., (i) deciding whether to trust a prediction; (ii) choosing among
different classification models; (iii) carrying on feature engineering; (iv)
identifying why a classifier should not be trusted. Results show that
explanations are actually useful to support users in these tasks. Similarly,
in (Malhi ., 2020), 65 participants were asked to carry out a decision-making
task in order to evaluate the human understanding of the behavior of
explainable agents. Three user groups were considered: a group was provided
with an agent without explanations and the other two with explanations
generated by LIME and SHAP, respectively. The results show that notable
(though not statistically significant) differences exist between the groups
without and with explanations in terms of bias in human decision making.
Decision-making tasks are also used in (Krause ., 2018) to evaluate the effect
of using aggregated rather than single data point explanations. The results
show that using aggregated explanations has a positive impact on the detection
of biases in data.

To the best of our knowledge, however, so far, no user evaluation has been
conducted on how users understand and use explanation plots in the field of
PPM. To this aim, in this work, we evaluate explanation plots at
_satisfaction_ , _comprehension_ and _performance_ level.

##  4 Methodology

To study how individuals make sense of explanation plots (RQ${}_{\ref{RQ1}}$),
how they use them for decision making (RQ${}_{\ref{RQ2}}$) and identify means
to improve them (RQ${}_{\ref{RQ3}}$), we conducted a qualitative observational
study. This approach is suitable because it allows us to draw insights into
“how” individuals interact with prediction explanations in the context of PPM
(Lazar ., 2017).

Here, we first provide an overview of our study setting (Section 4.1) before
discussing our data collection strategy (Section 4.2) and analysis methodology
(Section 4.3).

###  4.1 Setting

For our study, we selected eight individuals with different backgrounds as
participants. We focused on individuals that had expertise in Business Process
Management (BPM) since such individuals would commonly make decisions related
to how processes are conducted. We selected four individuals that had
expertise in BPM and ML, and four individuals that had expertise in BPM only.
We made this differentiation since it can be expected that knowledge related
to ML would affect whether and how individuals understand explanation plots
and how they use them for decision making. Within these two groups of four
participants, we selected individuals from different domains to foster the
applicability of our findings beyond the confines of a specific use case.

For setting our study, we decided to create two separate scenarios: one from
the medical domain (Section 4.1.1) and one from the financial domain (Section
4.1.2). We selected these domains because they are common domains where PPM
techniques are applied (B. van Dongen, 2011, 2012, 2017).We opted for
developing scenarios based on real-life settings rather than conducting the
study _in situ_ for three main reasons. First, PPM methods are not widely used
in practice yet, which makes it difficult to find suitable test beds. Second,
studying such methods in a real setting would almost certainly lead to domain
knowledge of individuals to impact our findings. Third, using a real setting
instead of an artificial one would make it difficult to compare findings of
different study subjects. In the following, we describe the two scenarios we
used (Sections 4.1.1 and 4.1.2) before elaborating on the plots we used
(Section 4.1.3) and the tasks that we asked participants to complete within
the two scenarios (Section 4.1.4).

####  4.1.1 Domain A (medical)

We consider a process pertaining to the treatment of patients with fractures,
for which we want to predict patients who will recover soon or late. Every
process execution starts with examining the patient. If an X-ray is performed,
then the X-ray risk must be assessed before it. Treatments reposition, cast
application and surgery require that an X-ray is performed before. If a
surgery is performed, then a rehabilitation must be prescribed eventually
after it. Finally, after every cast application a cast removal must be
performed. Each process execution refers to a patient whose age is also known.
Moreover, for each patient the type of treatment prescribed is also known and
whether or not the patient should carry on a rehabilitation after the
treatment. We predict whether the patient will recover quickly (1) or not (0).

####  4.1.2 Domain B (financial)

We consider a process of a bank to handle the closing of a bank account. A
request is first created, by either the owner of the account or a 3rd-party,
such as an attorney. Then, the request is evaluated. As part of the
evaluation, a risk assessment may also occur, which involves checking for
abnormal transactions in the account history, or whether the account has been
involved in illicit or suspicious activities. Risk assessment is optional and
may also be executed later in the process. Then the outstanding balance of the
account is determined, including pending payments. Before the closing can be
finalized, an investigation of the account owner’s heirs has to be executed,
in order to understand to whom the outstanding balance should be transferred.
The outcome of the process can be either of the following: the request will be
executed (1) or the request will be sent to credit collection (0) for a more
thorough assessment. The latter may happen, for instance, when there are
irregularities in a request, pending payments, or the heirs are unreachable.
Since this procedure takes a long time and involves a high amount of
resources, the bank would like to minimise the number of requests sent for
credit collection.

####  4.1.3 Plots

We selected three plots for our study in order to investigate three levels of
explanations for predictions: (i) at _event_ , (ii) _trace_ and (iii) _event
log_ level.

Figure 1: SHAP values related to the correlation between the pair
(feature,value) and the returned prediction (_Plot P1_)

The first plot (_Plot P1_) - the plot at _event_ level - is an adaptation of
the SHAP explanation plots to the PPM field. Figure 1 shows an example of this
type of plot. For all features, which in the case of an encoding based on the
event position in the trace (index-based encoding (Leontjeva ., 2015))
correspond to the activities executed at different positions of the trace, the
plot shows the impact on the prediction returned by the predictive model,
i.e., the correlations in terms of SHAP values of each feature (and associated
value) with the prediction. For instance, in _Plot P1_ , reported in Figure 1,
the most impacting feature-value pair is the pair composed of the event at
position four (event_4) and activity High Insurance Check, i.e., activity High
Insurance Check occurring at position four has a high correlation with the
returned prediction (expressed as a SHAP value of $0.8$). Such a correlation
means that this feature-value pair strongly affects the returned prediction.

Figure 2: Temporal stability of the SHAP values related to the correlation
between each feature and the returned prediction (_Plot P2_)

The second plot (_Plot P2_) - the plot at _trace_ level - shows how the
correlation of each feature with the returned prediction (expressed in terms
of SHAP values) changes at different positions in the ongoing trace. The plot
shows hence how stable the importance of a feature for a prediction is as the
chosen prefix length at which the prediction is carried out increases. The
plot is an adaptation of the _temporal stability_ plot (Teinemaa ., 2018) used
in PPM to check the stability of a prediction at increasing prefix lengths to
the case of the prediction explanations. Differently from the original
temporal stability plots, in which the prediction values are plotted for
different prefix lengths, in these plots, the SHAP values of the features (and
corresponding values) are plotted for different prefix lengths. For instance,
in _Plot P2_ , reported in Figure 2, feature event_2 corresponding to the
event at the second position of the trace (green line) is rather unstable, as
it positively or negatively correlates with the returned prediction based on
the specific point of the trace in which the prediction is made. Instead,
feature event_4 corresponding to the event at position four of the trace (red
line) positively correlates with the prediction starting from the prediction
carried out at prefix length four in a stable way. This means that, starting
from prefix of length four, the returned prediction highly depends on the
event occurred at position four, which is actually known only from prefix
length four on. Note that, in the plot in Figure 2, the specific values of the
features are not reported, as they may change for different prefix lengths, as
in the case of event_4 at prefix four whose value for prefixes shorter than
four is set to null. The values of the features are however visible by
hovering the mouse on the dynamic version of the plot (see Figure 19(b)). The
plot allows the user to get an idea of the stability of the importance of a
certain feature as the trace evolves, thus increasing the confidence of the
user in making decisions based on explanations using that feature (Goldstein
., 2015).

Figure 3: Average target value through different values of a feature on the
whole event log (_Plot P3_)

The third plot (_Plot P3_) - the plot at the _log_ level - shows, for a
specific feature, the average value of the predicted label (between 0 and 1
with 0 meaning _false_ and 1 meaning _true_) for the different feature values,
as well as the number of traces in the event log containing each value. The
plot, which mainly collects some statistics on the training data, is based on
the ICE plots (Goldstein ., 2015). In a binary setting, in which two possible
values are associated to the predicted label, the average value of the label
is the average percentage of traces for which the label value is true. For
example, _Plot P3_ , reported in Figure 3, shows the average value of the
predicted label for the different values of feature event_4, as well as the
frequency of the values of that feature on the whole event log. For instance,
value High Insurance Check for feature event_4 has an average label value of
$0.7$, i.e., in $70\%$ of the traces of the training set having as value for
feature event_4 activity High Insurance Check, the associated label is true,
while, for the remaining $30\%$, the label is false. Moreover, the plot also
shows that such a value occurs in around $750$ traces in the training set.
Therefore, the plot allows the user to get an overall idea of the distribution
of the labels for a specific feature value, as well as of the frequency of
that specific value in the whole event log.

####  4.1.4 Tasks

For each of the aforementioned scenarios (Sections 4.1.1 and 4.1.1), we
developed a set of tasks that required participants to use the three types of
plots described in Section 4.1.3 for decision making.

In the following, we will describe the tasks we developed for domain A
(Section 4.1.1). The full list of tasks including the domain description,
plots and the questions the investigator asked can be found in Appendix A.2:

Task Description A1.a: Consider an incomplete trace of a patient who has
carried out one of treatments reposition, cast application, or surgery. For
this patient, the prediction is that she will not recover soon from the
fracture (0). The explanation of the prediction is reported in Figure 4(a).

Task Description A1.b: Consider now the incomplete trace of another patient
who has carried out one of treatments reposition, cast application, or
surgery. For this patient the prediction is that she will recover soon from
the fracture (1). The explanation of the prediction is reported in Figure
4(b).

(a) Predicted outcome for late recovery.

(b) Predicted outcome for quick recovery.

Figure 4: SHAP values related to the correlation between features, their value
and the predicted outcome (_Plot P1_).

Task Description A2: Consider an incomplete trace of a patient who has carried
out one of treatments reposition, cast application, or surgery. For this
patient, the prediction is that she will recover soon from the fracture (1).
The explanation of the predictions from the beginning of the trace up to the
current point is reported in Figure 5.

Figure 5: Temporal stability of the SHAP values related to the correlation
between each feature and the predicted outcome for a quick recovery (_Plot
P2_).

Task Description A3: Consider a set of process executions related to patients
who have carried out one of treatments reposition, cast application, or
surgery. For some of these patients, the prediction is that they will recover
soon from the fracture (1), for others the prediction is that it will take
time for them to recover (0). The explanation of the predictions for different
patients in the training set is reported in Figure 6.

Figure 6: Average recovery time from a fracture through different values of a
feature on the whole event log (_Plot P3_).

In addition, we also developed a comprehension task that we used at the
beginning of each individual observation. The aim of this task was for
participants to be able to familiarize themselves with the format of the study
as well as with the plots used.

###  4.2 Data collection

We conducted semi-structured observational interviews with individuals within
our study population which lasted between 68 and 109 minutes each. To guide
the participants through the study procedure, we created a web interface that
contained the description of the domains as well as the related plots and
tasks111The interface can be accessed at https://user-evaluation-
mock.web.app/.

The first part of the web interface contained explanation plots and tasks for
the comprehension task. The second and third parts contained explanation plots
and tasks for domains A and B. We presented the two scenarios in different
orders within the two groups of subjects to be able to mitigate the effect of
individuals learning how to interpret the plots (c.f. Table 1).

Expertise | Domain Order | Code
---|---|---
BPM | A then B | BA1
BA2
B then A | BB1
BB2
BPM+ML | A then B | MA1
MA2
B then A | MB1
MB2
Table 1: Participants, their expertise and provided order of domains

The interviews were conducted via Zoom by a team consisting of a _facilitator_
and an _observer_ , with the facilitator guiding the participant and the
observer serving in a supporting role. Participants were encouraged to think
aloud, to ask questions and to point out interesting aspects related to the
different plots during the interview. All interviews were video-recorded.

Each interview started with the facilitator introducing the study procedure
and the application to the participants who were then asked to open the link
to the comprehension task and share their screen. The facilitator then asked
the participants a set of predefined questions regarding their understanding
(e.g., “ _What do the different plots show?_ ”) and interpretation (e.g., “
_Which are the feature(s) influencing most the prediction related to the
visualized trace?_ ”) of the different plots (RQ${}_{\ref{RQ1}}$).

After the comprehension task the participants were asked to open the web
interface containing explanation plots and tasks for domains A and B. The
facilitator then introduced domains and tasks and asked the participants to
explain the plots (e.g., “ _How do you interpret the plot?_ ”,
RQ${}_{\ref{RQ1}}$) and to decide what to do next based on their
interpretation of the plot (e.g., “ _Based on the information presented in the
plot, what course of action would you suggest?_ ”, RQ${}_{\ref{RQ2}}$). In
addition, the facilitator also asked specific questions related to each plot
(e.g., “ _Would you recommend carrying on the rehabilitation? Why?_ ”, for
task A1.a) to further examine the decision making process leading the
participant to suggest a certain course of action (RQ${}_{\ref{RQ2}}$).

After finishing the tasks, the facilitator conducted a short post-interview.
The questions focused on the participant’s understanding of the different
plots (e.g., “ _Which visualization(s) was/were the hardest to interpret? Why
do you think you were struggling with these visualizations in particular?_ ”,
RQ${}_{\ref{RQ1}}$) and the perceived usefulness for decision making (e.g., “
_Which of these visualizations helped you to make more informed decisions?_ ”,
RQ${}_{\ref{RQ2}}$). In addition, the facilitator also asked each participant
suggestions on how to improve the plots to better support their ability of
making informed decisions based on them (e.g., “ _Is there any additional
information that would have helped you to understand the different
visualizations and make more informed decisions during the study?_ ”,
RQ${}_{\ref{RQ3}}$)

Finally, the participants were asked to answer a questionnaire after the
interview. The questionnaire focused on the perceived ease of use
(RQ${}_{\ref{RQ1}}$) and perceived usefulness of each plot
(RQ${}_{\ref{RQ2}}$). For this, we adapted common scales included in the
technology acceptance model (Davis, 1989). In addition, we asked participants
to rate their experience related to BPM, ML, PPM and Explainable AI on a scale
from 1 to 10.

###  4.3 Analysis procedure

Our qualitative analysis mainly focused on the video recordings, observations
and follow-up interviews. The collected questionnaire data served as an
additional qualitative data point.

For our analysis, we combined deductive and inductive coding. We started with
a set of predefined codes related to our research questions. We focused on
whether and how participants made sense of the different plots
(RQ${}_{\ref{RQ1}}$) using codes such as _correct interpretation_ , _wrong
interpretation_ and _required additional information_ and on how participants
used the plots for decision making using codes such as _correct decision_ ,
_wrong decision_ and _reasoning_. We also included codes focusing on the
improvement of the different plots such as _labeling_ and _interaction with
visualizations_.

A group of three researchers all of which are co-authors of this paper
collaboratively applied these codes to all interviews. The coding was done by
plot meaning that first all responses related to plot 1 for all interviewees
were coded before moving on the plot 2. Afterwards the same researchers
conducted a second round of inductive coding adding aspects that emerged
during the first round analysis including _confidence_ , _trust in
visualization_ and _what-if-analysis_. These codes were then again
collaboratively applied by the three researchers. The last step was that the
researchers clustered the findings into themes related to our three research
questions.

##  5 Findings

In this section, we discuss our findings related to how participants made
sense of explanation plots (Section 5.1, RQ${}_{\ref{RQ1}}$) and how they
utilized them to make decisions (Section 5.2, RQ${}_{\ref{RQ2}}$). We also
outline suggestions on how plots can be improved (Section 5.3,
RQ${}_{\ref{RQ3}}$).

###  5.1 RQ1: How do users make sense of explanation plots?

Our interviewees generally found the different plots to be reasonably easy to
use. This is evident by survey responses we received (Figure 7) and also by
some statements of different participants during the interviews (_“I found
[these] quite easy to interpret”_ , BA2, _“It’s easy to get”_ , BB2, _“the
interpretation is clear”_ , MB1). Out of the three plots we used, the
participants perceived P1 to be the easiest and P2 to be the hardest to use.
The BPM+ML experts we interviewed generally perceived the plots to be easier
to use than the BPM experts. This difference is minimal for plots P1 and P2
while albeit slightly larger for P3.

Figure 7: Perception of BPM and BPM+ML experts related to the ease of use of
the different explanation plots (1=very low to 5=very high scale). We report
in the plot also median and variance.

The fact that plots were generally easy to use, however, does not mean that
the experts did not have to climb a learning curve to fully master the use of
the plots. The main themes regarding the difficulties in understanding and
using the plots emerged during the interviews with both types of experts are
discussed next.

####  5.1.1 Complicated first impressions

Most users struggled at the beginning with understanding specific aspects of
each plot. This could have been expected, considering that many experts, in
particular the BPM ones, were seeing such plots for the first time.

In P1, the main misunderstanding from experts concerned the meaning of the
Y-axis. The Y-axis in this plot simply lists the features that are considered
by the model at a certain prefix and the order in which these are shown does
not matter. During the interviews, however, some experts associated a meaning
with the ordering of the features on the Y-axis. BA2 and BB2 associated this
with a notion of feature importance, i.e., the higher features on the Y-axis,
were considered somehow more important in the predictive model (_“So this is
from top to bottom, I’m assuming based on, okay, some kind of value for each
event of the event log”_ , BA2). MA2 interpreted this as a chronological
order, i.e., the higher features on the Y-axis, were assuming the
corresponding values earlier (_"I think that after event one, there is nothing
yet…. And then after event […] it seems that there is a relatively low, but
definitely some positive correlation"_ , MA2). Such misinterpretations may
also have been due to the fact that the comprehension task only used features
defined by event labels.

Regarding P2, two BPM+ML experts (MA2 and MB1) struggled to understand the
fact that the data shown are part of a time-series and that, therefore, they
have to be considered as a whole. That is, the correlation values are valid at
a given prefix length, and they may vary if the same plot is regenerated for a
different prefix length

Finally, the BPM experts considered P3 too complex. For BA1 and BB2, the plot
was hard to understand at the first glance because of its different graphical
appearance and different semantics compared to plots P1 and P2. The double
Y-axis, representing the average label value and the number of traces with
that feature value, also was deemed as confusing by the same experts (“
_Expert: So, trying to find out what for instance, the green value means it is
average labor value or contact hospitals which means all traces that had an
event contact hospitals. Interviewer: that have contact hospital at event 4 is
the average value of the label. Expert: Okay, and now I get it._ ”, BA1; “
_Expert: With the average label value, average label value. […] What is an
average label value? Interviewer: Labels can be one or zero. Expert: Yeah, the
outcome. Right._ ” BB2).

Except these issues, after struggling initially, the experts managed to
correctly interpret the information in the plot without explicit help from the
facilitator.

####  5.1.2 Additional explanations required

In many cases, the experts explicitly required additional explanations from
the facilitator to understand specific aspects of each plot.

Some experts in the BPM group required an explicit explanation of the meaning
of “correlation” in plot P1 (_"The prediction correlation to be honest, it
doesn’t tell me anything. So, there is a set of events and correlation with
respect to what?"_ BB1). Here, the lack of experience with explainable AI
techniques of the BPM experts plays a key role in their understanding of the
targets of the correlation mentioned on the X-axis of P1.

After having seen P1, all experts were able to understand P2. However, some
experts struggled to understand the meaning of what a “stable feature” in P2
is and required the explicit guidance of the facilitator. Intuitively, some
experts associated feature stability with a _flat_ line in the plot. However,
they also realized immediately that a flat line at 0 correlation identifies an
irrelevant feature that has no correlation with the trace outcome at any
prefix length (_"The stability suggest me [one feature] because it’s always
zero. But it also means that doesn’t affect at all the the prediction. It is
stable but it’s useless"_ BB1). After this initial hesitation, most experts
(BB1, BA2,MA2, MB1, and MA1) correctly considered as stable the feature(s) for
which the line in the plot was flat after having taken a positive value. Some
of the experts, however, considered other aspects when determining stable
features in P2, such as the trend of the correlation value (BA2, MA2), e.g.,
_"The correlation moving to negative value could mean the counterexample of
the possible alternative feature to outcome"_ (BA2) , and the relative change
of correlation value (MB1) to identify the stable features. Finally, one
expert (BA1) incorrectly identified as stable only the feature with highest
correlation value at the last prefix shown in P2.

####  5.1.3 Effect of data imbalance in P3

During the interviews, the experts generally understood correctly the
information provided in P3. For instance, BA1, who required explicit
explanations on both P1 and P2, could easily understand the content of P3,
even though the information conveyed by this plot is far more than the one
provided in P1 and P2 (“ _For me, it was clear that the correlation between
the values of a feature that we are evaluating […] There is like less noise,
so to say, like less information that I’m not using to make my assumptions._
”, BA1).

However, two experts (BB1 and MB1) assumed that P3 could point to a predictive
model that is unreliable or biased when the labels in the event log are not
balanced (“ _It could be more reliable whenever I have a set of traces that
has an equal number of true and false._ ”, BB1; “ _If you have a very skewed
data set where you have too high number of traces having the same label and
where the other label in a binary setting are fewer […] then you can have some
bias in the prediction._ ”, MB1). While there is no technical argument to
support this claim, we argue that these experts over-analyzed the information
shown in the plot.

###  5.2 RQ2: How can explanation plots support users in decision making?

Similar to the findings reported in the previous section (Section 5.1) our
interviewees generally found the different plots to be reasonably useful for
solving the decision making tasks we proposed as evident by the survey
responses we received (Figure 8) and by some participant statements (“ _I
clearly understood what I should recommend_ ”, BA1, “ _all of [the plots]
serve a purpose_ ”, BA2, “ _So it’s very intuitive the way to represent this
part here,_ ”, BB2). The general perception about the usefulness of the plots
is lower than the perception about their ease of use though. The survey also
showed that the BPM+ML experts who participated in our study found all three
plots equally useful. The BPM experts, however, perceived P2 to be
considerably less useful than the other plots.

Figure 8: Perception of BPM and BPM+ML experts related to the usefulness of
the different explanation plots (1=very low to 5=very high scale). We report
in the plot also median and variance.

To answer RQ2, we have identified several common themes regarding how the
experts have used the plots in the decision making tasks. The themes that have
emerged are, in most cases, common to both BPM and BPM+ML experts. The themes
specific to one particular expert group are discussed at the end of the
section.

####  5.2.1 Correct decision, but wrong reasoning

In some cases, experts ended up making the correct decision in a task, but
interpreting the information in the plots in the wrong way. A typical example
of this situation is the one of experts (BA1, BA2, MA2, and MB1) using only
the information associated with the last prefix shown in plot P2 in Task A2.
It is only by coincidence that this led the experts to make the correct
decision. The reason behind this behavior could be that experts anchored their
reasoning on the principle that the more information is given to a predictive
model, the more accurate it is likely to be ( “ _The latest you make the
prediction, the more accurate it is because then you have more information_ ”,
BA1). Hence, they considered reasonable to trust only the information
associated with the longer prefix in plot P2.

Another example concerns experts suggesting to prescribe the rehabilitation in
task A1 because it looked like the only option available, even though other
options were available, like proceeding without taking any action (“ _I
suggest doing [..] for the reason that it’s the only thing that the [doctor]
can change, or affect._ ”, BA2).

####  5.2.2 Varying level of confidence in the decision made

For different reasons, the experts were not always confident in the decisions
they made using the plots. This was particularly the case of tasks in which
plot P1 was used (Tasks A1 and B1). Right at the end of task A1, when asked
for general comments, MB2 was concerned by the possibility of making a wrong
choice (“ _What is the real business risk and setting? What kind of decisions?
[…] how do I need to act on the information I see?_ ”, MB2) and, in the
subsequent tasks, often refused to make decisions based on the information
shown by the plots.

MA2, in both task A1 and B1, remarked that his/her lack of confidence on the
decision made was due to the lack of clear options among which to choose and
to the lack of knowledge about the possible decisions that could be made (“
_There’s definitely no guarantee that [this decision] will then always have a
positive [outcome]._ ”, MA2)).

BA1, in task A1, lamented a low confidence in the decision made due to lack of
confidence in the effect that a specific intervention (to perform the
rehabilitation in this particular case) may have (“ _I don’t know what I can
assume […] now that rehabilitation is false, rehabilitation has -0.15
prediction correlation, then if rehabilitation is true, the prediction
correlation will be +0.85?_ ” BA1).

In other cases, however, the experts were more confident about the
interpretation of plot P1 and the decision. This happened when they could
easily identify the possible decisions and their effects on a trace execution
exploiting their own domain knowledge. For instance, in task A1.b, BB2 felt
confident to recommend no action because, according to plot P1, the trace was
already predicted to have a positive outcome (“ _I’m more positive towards
this interpretation, I feel more confident because age is nothing we have
under control._ ”, BB2) .

####  5.2.3 Further investigation required

A common theme emerging from the interaction with both groups of experts is
that often, even when declaring to be confident about the decision made,
experts still mentioned the need to do more investigation in order to fully
validate the decision. In some cases, experts (MA1, MA2, MB2, BA2, and BB2)
mentioned the necessity to further investigate a particular decision using
additional analysis or data gathering (e.g., “ _Yeah, prescribe [..] but
again, I would first investigate why_ ”, BB2). BPM+ML experts were more
explicit in this regard mentioning specific additional data analysis
techniques, requiring “ _to see more data_ ” (MA2) by performing additional “
_data collection, [..] full randomized trials, or bandit tests_ ” (MB2).

####  5.2.4 Confirming decisions combining information from different plots

Experts showed to be able to integrate the different perspectives offered by
the different plots and often used the information provided in plots shown
earlier to confirm the decision made in the current task (“ _If I forget about
the previous plot, I would say no. If I consider a combination between the
two, yes_ ”, BB1). In particular, BA1, BA2, BB1, BB2, MA1, MA2, and MB1 were
able to spot the different perspectives shown by the plots and to infer a more
complete overview of the situation presented in the decision making tasks
using more than one plot at the same time (“ _each single plot provides you
with a view of the reality._ ”, MB1). We can then infer that showing all the
plots at the same time would have been beneficial for the comprehension of the
situation at hand, “ _to make more informed decision._ ” (MB1), and could have
even improved the experts’ confidence in their decisions.

####  5.2.5 Enumerate the options, providing what-if analysis

Most of the experts (BA1, BA2, BB1, BB2, MB1, MA2, and MB2) felt the necessity
to receive additional information regarding the choices available when asked
to suggest a course of action using plot P1 or P2 ( “ _what are my degrees of
freedom?_ ”, MA2). In one case (BA1), the expert had also to be reminded that
taking no action was one of the possible options.

Once the possible options were enumerated, experts (BB2, MA1, MB1, and MB2)
also mentioned that tools supporting a prescriptive (what-if) analysis of the
different options could have been useful to make more informed decisions.
According to the experts, these could take the form of trace matching, i.e.,
matching the current trace to similar one(s) for which the outcome is known (“
_I don’t have a counter proof. So if I had another trace in which the third
event was [same value]_ ”, BB2, _you could do matching here and you could try
to see to match this to the to the closest case [..] to see what happened to
that._ ”, MB2), or actionable attributes, i.e., the possibility of changing at
run time the values of particular features and to see immediately the effect
on the plots and the predicted outcome (_we don’t know what happens after this
point, then you could add some actionable attribute that I can change in order
to change the course of the future._ ”, MB1)).

####  5.2.6 The type of domain influences the decisions

The domain in which tasks are situated has clearly influenced the experts
decisions (“ _So it’s not like in the previous domain, [..] here the domain
influences a lot the reasoning_ ”, BB1). Specifically, for tasks in Domain A
(medical), experts (MA1, MA2, BB1, and BB2) mentioned that depending on the
severity of the case at hand, which was not one of the available attributes in
the event log, their choice could change (“ _one might be tempted to say that
the best way to make [..] is to [..]. However, it’s hard to tell, because
there could be some confounding attribute aspect that is not recorded in the
log_ ”, MA1 or “ _assess the situation on a case by case basis_ ”, MA2). We
argue that this is due to the medical domain being perceived as one where
process analysis is particularly challenging because special cases are
frequent and process users (physicians, nurses) often have a higher amount of
freedom in deciding the course of action, e.g., on the treatments to
administer (Munoz-Gama ., 2022). The domain B (financial) was considered by
experts more driven by clear and standardized procedures and therefore less
challenging as far as predicting the outcome of cases is concerned.

####  5.2.7 Correlation is not causation

The experts, in particular the BPM+ML ones, mentioned multiple times that the
plots that they were using were showing a correlation, e.g., between features
values and predicted outcome, but that such a correlation should not be
directly interpreted as a causal relation (“ _Yes, there is a correlation, of
course, but it doesn’t mean there is a causation_ ” MA1). One of them in
particular (MB2) refused to carry out part of the decision making tasks for
this reason (“ _Obviously, we do see that the fact that the patient was young
contributed a lot to the fact that the positive outcome for the patient. I
guess that makes sense. Young people recover more easily. That’s a
correlation. But I would say just from domain knowledge that’s even likely to
be a causation. Even though we’re not sure about that. And so, that’s
something you can easily read in this plot. But we don’t have anything to make
recommendations here_ ”, MB2).

Other BPM+ML experts took a lighter stance in this regard, for instance,
suggesting, after having mentioned the correlation vs. causation theme, that “
_there’s definitely no guarantee_ ” (MA2) that the course of action that they
suggested was going to yield the desired outcome, or that their suggested
course of action was simply a “ _suggestion to the domain expert_ ” (MA1).

####  5.2.8 The cost of the suggested course of action should be a decision
variable

The experts often mentioned that understanding the cost of a suggested course
of action is a fundamental variable when making a decision (“ _I would
probably suggest not to do [..] because it would incur in an additional cost_
” MA1). In tasks that used plot P3 (Tasks A3 and/or B3), BA1, BA2, BB1, MA1,
MA2, and MB2 decided to avoid making a definite decision for the process “
_depending on the cost_ ” (BB1), even if they could make a decision that was
perceived as beneficial for the outcome of the process. Specifically, BA1,
BA2, BB1, MA1, MA2, and MB2 suggested to make the risk assessment a mandatory
task for the process in task B3 of Domain B (financial) only after having
considered the cost of this task. Similarly, MA1, MA2, and MB2 suggested to
make the rehabilitation mandatory in the process in task A3 of Domain A, only
after having considered its cost.

####  5.2.9 Distinguishing themes between expert groups

Finally, we summarize here the themes that have uniquely distinguished the BPM
from the BPM+ML experts in the decision making tasks.

First, we noticed that BPM+ML experts tended to be more conservative in their
decisions, often mentioning that the plots did not give them enough evidence
to make specific recommendations and that they would rather delegate these to
domain experts.

Regardless the task or plot, MB2 felt that the plots were not conveying enough
evidence to make any specific recommendation (“ _I can do a little bit of
diagnostics [..] but I can’t make any recommendations_ ”, MB2). This feeling
was partly echoed also by MA2 (“ _there’s definitely no guarantee that [this
decision] will always have a positive [outcome]_ ”, MA2), MA1, and MB1, who
consistently showed to have concerns regarding any choice they made, even
mentioning that they deemed necessary that someone else had the final word
(e.g., requiring a “ _suggestion to the domain expert_ ”, MA1; or “ _assessing
the situation on a case by case basis_ ”, MA2).

Conversely, BPM experts BA1, BA2, BB1, and BB2 were more willing to make clear
and specific recommendations based on the information available, often relying
on their domain knowledge (“ _I have a feeling that those cases are very rare
[therefore] rehabilitation should be a general practice_ ” BA2)).

The BPM experts generally experienced a linear interaction pattern throughout
the interview, answering the questions they were asked, without showing any
particular additional concern regarding how the plots were produced. The
questions they raised during the interviews were focused exclusively on
understanding how to perform the task at hand. For BPM experts, we also
witnessed a steep learning curve effect, which lead to the tasks on the second
domain taking a considerably lower amount of time than the ones on the first
domain, regardless of the order in which the domains were presented. These
experts perceived plot P3 as the easiest to understand and use. When asked,
the experts reported that this was due to it being a histogram, and them being
more used to interpreting histograms based on their own experience.

The BPM+ML experts instead showed a deeper understanding of the proposed
plots, but they also had a tendency to over-analyse the information showed in
them. This, at times, hindered the flow of the interview and led their
interviews to take on average longer than the ones with BPM experts. However,
the quality of the decisions made by BPM+ML expert appeared not to have been
affected by this tendency to over-scrutinize the information in the plots.

###  5.3 RQ3: How can explanation plots be improved?

Several suggestions for improving the plots have also emerged during the
interviews. These revolve around three main themes, which are discussed below.

####  5.3.1 Improving the interface

Several comments we received during the interviews were directed towards
improving the interface through which the plots were shown.

For example, the interface could be improved by showing all the plots together
in a dashboard setting. This would help giving the user multiple perspectives
regarding the process at the same time. We have drawn this conclusion by
analyzing the behavior of experts during the interviews. Both the BPM and
BPM+ML experts used multiple plots at the same time to attain a more precise
idea regarding their decisions. For instance, many experts used plot P3 to
find a confirmation, at the global event log level, of the decisions they made
based on the information found in P1 or P2. In alternative, experts mentioned
that it would be effective to show P1 and P2 together or, at least, P2 before
P1, since plot P2 represents a more general overview of the feature
contributions for different prefixes, whereas P1 is a snapshot of P2 at a
specific prefix length.

In addition, a number of specific improvements have emerged regarding the
elements of individual plots. Regarding P2, the wavy lines associated with
individual features were considered generally confusing and, more
specifically, hinting at the visualisation of a continuous variable (“
_Curving is great when the domain is continuous, here is discrete, there is no
1.5 event_ ”, BB2). In fact, the correlation values shown by P2 are discrete,
because they are defined only at discrete prefix length values. Therefore, the
continuous lines may be substituted by discrete plots, e.g., by markers or
bars. Furthermore, P2 does not show information about the prefix length at
which a feature takes on its value. For instance, feature “Rehabilitation
prescription”, in Domain A, does not have a value before the activity in which
the doctor decides whether to prescribe the rehabilitation or not is executed.
This information can be helpful for decision making tasks and can be captured
in the plot, for instance, by greying out the lines and/or labels associated
with a particular feature until the feature takes a known value. Regarding P3,
the green lines connecting the absolute number of traces in which the feature
takes each possible value has been considered to be misleading, since they
hint at a connection between the feature values, which is not meaningful for
decision making (“ _I ended up looking at data points […] what’s the point of
having a line?_ ”, BA2).

####  5.3.2 Lack of interactive elements

Experts have highlighted the lack of interactive elements in the plots they
were asked to use. A first level of missing interaction concerns the filtering
of specific attributes, traces, or events dynamically, instead of showing one
static view of them all. While, in P2, it is possible to hide the lines
associated with attributes that are not deemed important for a decision (for
instance, because they are associated with a constantly oscillating
correlation and therefore not helpful to make decisions regarding the case
outcome), a similar level of interactivity is not available in P3. Experts
suggested that, in P3, it should be possible to filter only traces with
certain characteristics, such as as traces similar to the one on which a
decision has to be made (“ _In this third plot […] I just see these numbers,
should I ask what is happening behind? […] I can divide [the data] in small
groups […] checking what it is happening behind the scene, how the data
varies, how [the data] depends on the events and the variables_ ”, BB1).

A second level of interaction concerns supporting what-if analysis. As
highlighted earlier, this is a requirement expressed in particular by the
BPM+ML experts, who have a more hands-on experience with predictive models and
explainability methods. What-if analysis represents a means to understand what
would happen to the outcome of a process when the decision regarding a certain
course of action is actually put in practice. Such an analysis requires,
first, to be able to enumerate the decision options available in a decision
making task. This should also include the option of “doing nothing”.

Then, the what-if analysis functionality can take two different forms. On the
one hand, it could look for similar cases that have already terminated,
inferring an expected outcome from them. On the other hand, it could take the
form of a simulation that yields an expected outcome associated with the
selected course of action. Plots P1 and P2 are particularly suited to provide
this sort of actionability, whereas this is not the case for Plot P3, which
does not represent the course of actions of a single trace, but a global
overview of the information available in the entire event log.

####  5.3.3 Adding information fields

In several cases, the experts mentioned, during the interviews, the need to
enrich the plots with more information. From a technical standpoint, experts
requested more information to gauge how much they could trust the performance
of the predictive model from which the plots were derived. This may involve,
for all plots, showing the accuracy measures of the predictive models, like
F-score or AUC. Regarding P3, experts mentioned that showing the relative
frequency of the feature values in the event log would have been more useful
for supporting the decision making tasks with respect to the absolute number
of traces including them. The BPM+ML experts also pointed out the need to show
additional statistical information in P3, such as box plots providing the
confidence interval of the label value for a given feature value.

More generally, the experts lamented the lack of information explaining the
plots, such as more text describing their semantics, explanations of the
acronyms used, different color codes for different types of features (e.g., to
distinguish features associated with the occurrence of activities in a trace
from other features) and a more clear explanations of the term “correlation”
in plots P1 and P2.

##  6 Discussion and Conclusion

In this section, we analyze and discuss the results obtained from the user
study by drawing the main implications for theory (Section 6.1), for practice
(Section 6.2), as well as by highlighting the limitations (Section 6.3) of the
evaluation carried out.

###  6.1 Implications for theory

The results presented in the previous section have highlighted that the
explanations plots need to be complemented by tools supporting the what-if
analysis to foster the experts’ confidence in the decisions made. Future
research should investigate the most effective way of implementing such tools,
e.g., whether focusing on finding similarities between historical cases and
the one for which a decision should be made, or implementing full-fledged
simulations tools. Reinforcement learning (Sutton Barto, 1998) and bandit
algorithms (Bubeck Cesa-Bianchi, 2012; Lattimore Szepesvári, 2020) may also be
considered to allow decision support tools to learn the best course of action
(policy) that maximizes an expected gain.

Future research should also address the issue of guaranteeing that the
explanation plots provided to experts convey the information required to
establish causality, between feature values and process outcomes in the case
of this paper, rather than simply correlation. In this direction, anchors
(Ribeiro ., 2018) are a post-hoc explanation technique that yields if-then
rules explaining the behavior of the underlying model, together with an
indication of the precision and coverage of the rules. As far as we are aware,
an approach customizing anchors to the case of PPM is missing in the
literature.

From a human factor standpoint, more research is needed to understand which
skills are required by decision makers to perform well in the decision making
tasks with the support of explanation plots. Counter-intuitively, our analysis
has highlighted that experts lacking deep ML knowledge felt more comfortable
using the plots. Future research should investigate to what extent other
variables such as decision making experience, process modeling expertise, or
process domain knowledge may influence the ability to make correct decisions
in PPM scenarios. Along this line, the opportunities for improving the current
plots that we have been identified in this work should be validated with
different types of decision makers and in different classes of decision making
tasks.

Finally, future research should investigate to what extent the vision of self-
optimizing business processes can be implemented with the support of AI tools
within the business process lifecycle, i.e., whether the decision making
tasks, possibly exploiting the feedback produced by explainable AI techniques,
can be fully delegated to automated tools, bypassing human decision makers. In
this direction, recent research on prescriptive monitoring of business
processes (Bozorgi ., 2021; Weinzierl, Dunzer ., 2020) has dealt with (semi-
automatically) incorporating the effects of what occurs during the execution
of a business process (for instance as the result of human actions) while
predicting the value of aspects of interest in the long term.

###  6.2 Implications for practice

Our findings revealed how different individuals utilize explanation plots for
decision making and can thus serve as a basis for suggestions on how to employ
them in the context of PPM.

While individuals appreciated the plots as a valuable source of information,
we also found that they often did not perceive the information presented to be
rich enough to make a decision confidently. When employing such plots, it is
thus important to be able to provide additional information to decision
makers. Our findings are in line with those of (Lim ., 2009) who proved that
employing explanation plots supports understanding.

One additional piece of information that the participants of our study deemed
to be particularly relevant was the causality of the presented predictions.
They did not perceive showing correlations to be sufficient to make a decision
especially in a medical context. Providing causality-related information can
thus be perceived to be important when using explanation plots for decision
making in the context of PPM. This need has also been reported by (Lipton,
2001) who found causality to be one of the four key features fostering
explanation.

Together with information about causality, participants also asked for domain-
related information, for details about the specific case that were not present
in the explanation plots, for options to filter the information that was
available in the plots, and for the possibility to perform additional analysis
like on-the-fly what-if analysis and statistical significance tests.Domain-
related information and case details can be easily added, as well as suitable
filters. The additional analysis suggested by the participants might be more
difficult to provide. The need for a what-if analysis has also been reported
in (Davis Kottemann, 1994). However, they also found that providing such
analysis can create an ‘illusion of control’ that causes the users to
overestimate its effectiveness. When providing more information, it is thus
necessary to carefully consider which information to provide and to whom.

We also found differences between the expectations of BPM experts and BPM+ML
experts regarding their need for information to make a decision. For BPM
experts, we suggest to pay particular attention to the wording of the plot
labels and captions. For instance, in our study, the BPM experts struggled
with the term ‘Correlation’ as it is too vague if not well explained and
contextualized. They also had issues in deciding which actions to take and
asked which actions would even be possible. To support their decision process,
we would thus suggest to provide examples of possible actions that were taken
in past process executions similar to the one of interest as explored in (Lu
Sadiq, 2007). Finally, the BPM experts also assumed that the information shown
was related to statistics on completed process executions and they did not
immediately realized that the plots serve as additional information
complementing the prediction tasks. We would thus propose to stress the goal
of the plots by also improving their captions.

On the other hand, BPM+ML experts asked for information regarding predictions
and the way they were computed including the data distribution of the original
data and measures representing the prediction accuracy of the predictive
model. This information can be easily provided, and would make the decision
making process more informed. We also observed that the BPM+ML experts avoided
reading the text in the captions. So we would make these elements of the
interface more prominent or use pop-ups when the user interacts with a given
element. Finally, the BPM+ML experts suffer from the information overload
coming from their expertise often resulting in getting lost in the details of
the plots, and the descriptions (Buchanan Kock, 2001). We therefore suggest to
complement these plots with plots that provide information at a higher level
of abstraction that focus more on the business value of the information
provided and on the decision that needs to be made rather than on the model-
specific information provided by the plots analyzed in this user evaluation.

###  6.3 Limitations

The goal of our study was to investigate how users make sense of explanation
plots and how they use them for decision making. Furthermore, we aimed at
exploring how explanation plots can be improved. It is thus reasonable to
conduct a qualitative observational study (Lazar ., 2017). There are, however,
innate limitations associated with this study design. We interviewed
individuals from different backgrounds, domain expertise and expertise related
to BPM and ML. Despite making a reasonable selection of the participants, it
is not possible to generalize findings beyond our study context since studying
different individuals with different backgrounds from different domains and
different levels of expertise might yield different results. Moreover, the
study was conducted by a team of researchers which poses a threat to validity
since different researchers might perceive the reactions of study participants
differently. To minimize this threat, we ensured that throughout the process
of the study, which included the preparation of the study materials, the
conduction of the study and the analysis of the study results, at least two
individuals from the research team collaborated on each step to avoid
depending on the perception of individual researchers. We also opted for
studying a specific artificial setting utilizing specific plots and asking
predefined questions. This can lead to observations and interpretations that
might not have happened or might have happened in a different way in a real-
life setting. To mitigate this threat, we made a state-of-the-art founded
selection of the explanation plots, study domains and decision tasks. We
acknowledge though that there is a remaining risk associated with studying an
artificial rather than a real setting. We are willing to accept this threat
because it allowed us to compare study findings across subjects which would
not be possible when studying a real case. Finally, we abstain from making
causal claims providing instead a rich description of the observed behavior
and reported perceptions of the study participants based on which we discuss
differences in how different individuals made sense of explanation plots and
used them for decision making.

##  Appendix A Appendix

This appendix contains the study material used by the investigators and
includes the different explanation plots, an introduction to each plot, the
questions that the investigators asked, the expected answers and the
interaction capabilities of the interface.

###  A.1 Plots and descriptions

Let us assume to have an insurance claim event log that collects claim
requests. We use this log to train a predictive model and predict whether the
claim will be accepted (TRUE) or rejected (FALSE). Let us assume that the log
is encoded using the simple-index encoding (an example of this encoding is
shown to the participant) for training the predictive model. Let us consider
the following explanation plots: the first two plots (Figures 9 and 10) refer
to trace 4633, while the third plot (Figure 11) refers to a set of incomplete
process instances.

Figure 9: Plot P1 for the comprehension task. Figure 10: Plot P2 for the
comprehension task. Figure 11: Plot P3 for the comprehension task.

Questions asked by the investigator:

  * •

What do the different plots show?

  * •

Which are the feature(s) influencing most the prediction related to trace
4633? Why?

  * •

How confident are you about your interpretation?

  * •

Which features are more stable over time with respect to their importance
towards the acceptance prediction for trace 4633? Why?

  * •

What is the value of the feature _Event 4_ influencing most the acceptance
prediction over all the traces in the log? Why?

  * •

When thinking about which answer to give, which information did you miss?

###  A.2 Two domains

####  A.2.1 Domain A

Let us assume to have an event log pertaining to the treatment of patients
with fractures. Every process instance in the log starts with activity Examine
patient. If activity Perform X-ray is performed, then Check X-ray risk must be
performed before it, without other executions of Perform X-ray in between.
Activities Perform reposition, Apply cast and Perform surgery require that
Perform X-ray and Make Prescriptions are executed before they are executed. If
Perform surgery is performed, then Prescribe rehabilitation is performed
eventually after it. Finally, after every execution of Apply cast, eventually
Remove cast is executed and, viceversa, before every execution of Remove cast,
Apply cast must be performed. Each process instance refers to a patient and
her age is also available in the event log. Moreover, the activity Make
Prescriptions includes the event attributes treatment type and rehabilitation
prescription, which record the type of treatment prescribed and whether or not
the patient should carry on the rehabilitation.

We aim at predicting whether the patient will recover quickly (TRUE) or not
(FALSE).

##### Task A1.a

Let us consider an incomplete trace of a patient who has carried out one of
the treatments, i.e., Perform Reposition, Apply cast or Perform Surgery. For
this patient, the prediction is that she will not recover soon from the
fracture. The explanation of the prediction is reported in Figure 12.

Figure 12: Plot P1 for the decision making task A1.a.

Questions asked by the investigator (Q) and projected answers based on the
information provided (A):

  * Q:

How do you interpret the plot?

  * A:

The patient does not recover because rehabilitation has not been prescribed.

  * Q:

Based on the information presented in the plot, what course of action would
you suggest?

  * Q:

Would you recommend carrying on the rehabilitation? Why?

  * A:

Yes, because the negative outcome, i.e., the fact that the patient will not
recover soon, is due to not carrying out the rehabilitation.

##### Task A1.b

Let us now consider the incomplete trace of another patient who has carried
out one of the treatments, i.e., Perform Reposition, Apply cast or Perform
Surgery. For this patient, the prediction is that she will recover soon from
the fracture. The explanation of the prediction is reported in Figure 13.

Figure 13: Plot P1 for the decision making task A1.b.

Questions asked by the investigator (Q) and projected answers based on the
information provided (A):

  * Q:

How do you interpret the plot?

  * A:

The patient will recover because of her age.

  * Q:

Based on the information presented in the plot, what course of action would
you suggest?

  * Q:

Would you recommend carrying on the rehabilitation? Why?

  * A:

No, because the positive outcome, i.e., the fact that the patient will recover
soon is due to the age of the patient.

##### Task A2

Let us consider an incomplete trace of a patient who has carried out one of
the treatments, i.e., Perform Reposition, Apply cast or Perform Surgery. For
this patient, the prediction is that she will recover soon from the fracture.
The explanation of the prediction is reported in Figure 14.

Figure 14: Plot P2 for the decision making task A2.

Questions asked by the investigator (Q) and projected answers based on the
information provided (A):

  * Q:

How do you interpret the plot?

  * A:

The explanation is initially (when looking only at the first two events) that
this is due to the age, then (when looking at the first three events) that
this is due to the rehabilitation prescription, then (when looking at the
first four events) that this is due again to the age and, finally (when
looking at the first five events) that this is due to the rehabilitation
prescription.

  * Q:

Based on the information presented in the plot, what course of action would
you suggest?

  * Q:

Would you recommend carrying on the rehabilitation? Why?

  * A:

No, because the importance of the rehabilitation prescription is rather
unstable.

##### Task A3

Let us consider a set of process executions related to patients who have
carried out one of the treatments, i.e., Perform Reposition, Apply cast or
Perform Surgery. For some of these patients, the prediction is that they will
recover soon from the fracture, for others, the prediction is that it will
take time for them to recover. The explanation of the predictions is reported
in Figure 15.

Figure 15: Plot P3 for the decision making task A3.

Questions asked by the investigator (Q) and projected answers based on the
information provided (A):

  * Q:

How do you interpret the plot?

  * A:

The explanation is that when the rehabilitation is prescribed, the prediction
goes towards a fast recovery, when the rehabilitation is not prescribed the
recovery is slow.

  * Q:

Based on the information presented in the plot, what course of action would
you suggest?

  * Q:

Would you recommend rehabilitation as a general practice for all patients?

  * A:

Yes, because cases in which the rehabilitation was performed are usually
experiencing a fast recovery.

####  A.2.2 Domain B

Let us consider a process at a bank to handle the closing of a bank account by
the finance operations department. A request is first created, by either the
owner of the account or a 3rd-party, such as their attorney. Then, the request
is evaluated. As part of the evaluation, a risk assessment may also occur,
which involves checking for abnormal transactions in the account history or
inquiring with the state fiscal authority to understand whether the account
has been involved in illicit or suspicious activities. Risk assessment is
optional and may also be executed later in the process. Then, the outstanding
balance of the account is determined, including pending payments; before the
closing request can be filed, an investigation of the account owner’s heirs
has to be executed in order to understand to whom the outstanding balance
should be transferred. After a final evaluation, the outcome of the case is
determined and the process terminates. The outcome of the process can be
either of the following: the request will be executed (TRUE) or the request
will be sent to the credit collection (FALSE) for a more thorough assessment.
The latter may happen, for instance, when there are irregularities in the
request, pending payments, or the heirs are unreachable. Since credit
collection takes long and involves a high amount of resources, the bank would
like to minimize the number of requests sent to credit collection.

##### Task B1

Let us consider an incomplete trace (trace 2) of a bank account closure. The
label of this trace is FALSE (meaning that it is sent to credit collection).
The explanation of this prediction is shown in Figure 16.

Figure 16: Plot P1 for the decision making task B1.

Questions asked by the investigator (Q) and projected answers based on the
information provided (A):

  * Q:

How do you interpret the plot?

  * A:

The request will be sent to credit collection because a risk assessment was
not executed.

  * Q:

Based on the information presented in the plot, what course of action would
you suggest?

  * Q:

Would you recommend running the risk assessment before the end of the case?
Why?

  * A:

Yes, the user should suggest to run the risk assessment because the feature
risk assessment = false is highly correlated to the negative outcome.

##### Task B2

Let us consider trace 3 at prefix 5. The label of this trace is FALSE (meaning
that it is sent to credit collection). The explanation plot is shown in Figure
17.

Figure 17: Plot P2 for the decision making task B2. Figure 18: Plot P3 for the
decision making task B3.

Questions asked by the investigator (Q) and projected answers based on the
information provided (A):

  * Q:

How do you interpret the plot?

  * A:

Initially, it seems like risk assessment is not necessary because the false
label is explained by the type of requester (attorney) and the type of account
(corporate), but then it becomes clear that the false label is explained by
the absence of risk assessment.

  * Q:

Based on the information presented in the plot, what course of action would
you suggest?

  * Q:

Would you recommend running the risk assessment before the end of the case?
Why?

  * A:

Yes, from prefix 3 onwards it becomes clear that the false label is explained
by the absence of risk assessment.

##### Task B3

Let us consider a set of process executions and let us focus on the feature
capturing whether the risk assessment was executed or not. Depending on the
value of this feature, for some requests, the prediction is that they can be
executed, for others, the prediction is that they will be sent to credit
collection. The explanation is shown in Figure 18.

(a) Detailed view the prediction correlation of Event 4.

(b) Detailed view showing the prediction correlation of all features when
Event 4 occurs.

(c) Detailed view for Plot P3 showing the number of traces containing the
feature value ’High Insurance Check’ and the average label value for this
subset of traces.

Figure 19: Examples of detailed views for Plot P1, Plot P2, and Plot P3.
Figure 20: Focused view retaining feature Event 2 and blurring out the others.

(a) Unfiltered view with five features.

(b) Filtered view retaining only features Event 2, Event 3, and Event 4.

(c) Unfiltered view with ten features.

(d) Filtered view retaining only features Event 1, Event 2, Age, and
Rehabilitation Prescription.

Figure 21: Example views before and after filtering some features.

Questions asked by the investigator (Q) and projected answers based on the
information provided (A):

  * Q:

How do you interpret the plot?

  * A:

When the risk assessment is required, the requests tend to be executed and not
sent to credit collection; when the risk assessment is not executed, the
requests tend to be sent to credit collection.

  * Q:

Based on the information presented in the plot, what course of action would
you suggest?

  * Q:

Would you recommend to change the process to include mandatory risk
assessment?

  * A:

Yes, because cases that executed the risk assessment usually are not sent to
credit collection.

###  A.3 Interaction Capabilities of the Interface

The plots of our study were provided with three interaction capabilities: (i)
Detailed view, (ii) Focused view, and (iii) Information Filter.

##### Detailed view

When the user hovers over an element in the plot the exact values of that
element are displayed, see Figure 19. This allows the user to quickly retrieve
specific information from the plots.

##### Focused view

Hovering does not only show additional information regarding a particular
point (see Section A.3). When hovering is done on the legend, it also leads to
the element to be highlighted. Figure 20 shows an example where the line for
Event 2 is highlighted while the others are not.

##### Information Filter

When the user clicks the labels in the legend they are temporarily removed
from the plot. This allows the user to lower the amount of information
displayed. This interaction capability is particularly useful for Plot P2
(Figure 21). Figures 21(a) and 21(c) show two examples of unfiltered views.
Figures 21(b) and 21(d) show the corresponding filtered views.

## References

  * Alqaraawi . (2020) DBLP:conf/iui/AlqaraawiSWCB20Alqaraawi, A., Schuessler, M., Weiß, P., Costanza, E. Berthouze, N.  2020\.  Evaluating saliency map explanations for convolutional neural networks: a user study Evaluating saliency map explanations for convolutional neural networks: a user study.  F. Paternò, N. Oliver, C. Conati, LD. Spano N. Tintarev (), IUI ’20: 25th International Conference on Intelligent User Interfaces, Cagliari, Italy, March 17-20, 2020 IUI ’20: 25th international conference on intelligent user interfaces, cagliari, italy, march 17-20, 2020 ( 275–285).  ACM.  https://doi.org/10.1145/3377325.3377519  10.1145/3377325.3377519
  * Bach . (2015) bach-plos15Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, KR. Samek, W.  201507\.  On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  PLoS ONE107e0130140.  http://dx.doi.org/10.1371%2Fjournal.pone.0130140  10.1371/journal.pone.0130140
  * Bozorgi . (2021) bozorgi2021prescriptiveBozorgi, ZD., Teinemaa, I., Dumas, M., La Rosa, M. Polyvyanyy, A.  2021\.  Prescriptive process monitoring for cost-aware cycle time reduction Prescriptive process monitoring for cost-aware cycle time reduction.  2021 3rd International Conference on Process Mining (ICPM) 2021 3rd international conference on process mining (icpm) ( 96–103).
  * Bubeck Cesa-Bianchi (2012) DBLP:journals/ftml/BubeckC12Bubeck, S. Cesa-Bianchi, N.  2012\.  Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems Regret analysis of stochastic and nonstochastic multi-armed bandit problems.  Found. Trends Mach. Learn.511–122.  https://doi.org/10.1561/2200000024  10.1561/2200000024
  * Buchanan Kock (2001) buchanan2001informationBuchanan, J. Kock, N.  2001\.  Information overload: A decision making perspective Information overload: A decision making perspective.  Multiple criteria decision making in the new millennium Multiple criteria decision making in the new millennium ( 49–58).  Springer.
  * Camargo . (2019) DBLP:conf/bpm/Camargo19Camargo, M., Dumas, M. Rojas, OG.  2019\.  Learning Accurate LSTM Models of Business Processes Learning accurate LSTM models of business processes.  TT. Hildebrandt, BF. van Dongen, M. Röglinger J. Mendling (), Business Process Management - 17th International Conference, BPM 2019, Vienna, Austria, September 1-6, 2019, Proceedings Business process management - 17th international conference, BPM 2019, vienna, austria, september 1-6, 2019, proceedings ( 11675, 286–302).  Springer.  https://doi.org/10.1007/978-3-030-26619-6_19  10.1007/978-3-030-26619-6_19
  * Cheng . (2019) Cheng2019Cheng, HF., Wang, R., Zhang, Z., O’Connell, F., Gray, T., Harper, FM. Zhu, H.  2019\.  Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders Explaining decision-making algorithms through ui: Strategies to help non-expert stakeholders.  Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems Proceedings of the 2019 chi conference on human factors in computing systems ( 1–12).  New York, NY, USAAssociation for Computing Machinery.  https://doi.org/10.1145/3290605.3300789  10.1145/3290605.3300789
  * Chromik Schuessler (2020) Chromik2020ATFChromik, M. Schuessler, M.  2020\.  A Taxonomy for Human Subject Evaluation of Black-Box Explanations in XAI A taxonomy for human subject evaluation of black-box explanations in xai.  ExSS-ATEC@IUI. Exss-atec@iui.
  * Davis (1989) davis1989perceivedDavis, FD.  1989\.  Perceived usefulness, perceived ease of use, and user acceptance of information technology Perceived usefulness, perceived ease of use, and user acceptance of information technology.  MIS quarterly319–340.
  * Davis Kottemann (1994) davis1994userDavis, FD. Kottemann, JE.  1994\.  User perceptions of decision support effectiveness: Two production planning experiments User perceptions of decision support effectiveness: Two production planning experiments.  Decision Sciences25157–76.
  * Di Francescomarino (2019) DBLP:reference/bdt/Francescomarino19Di Francescomarino, C.  2019\.  Predictive Business Process Monitoring Predictive business process monitoring.  S. Sakr AY. Zomaya (), Encyclopedia of Big Data Technologies. Encyclopedia of big data technologies.  Springer.  https://doi.org/10.1007/978-3-319-63962-8_105-1  10.1007/978-3-319-63962-8_105-1
  * Di Francescomarino . (2019) Di-Francescomarino:2016aaDi Francescomarino, C., Dumas, M., Maggi, FM. Teinemaa, I.  2019\.  Clustering-Based Predictive Process Monitoring Clustering-based predictive process monitoring.  IEEE Trans. Serv. Comput.126896–909.  10.1109/TSC.2016.2645153
  * Di Francescomarino . (2018) DBLP:conf/bpm/Francescomarino18Di Francescomarino, C., Ghidini, C., Maggi, FM. Milani, F.  2018\.  Predictive Process Monitoring Methods: Which One Suits Me Best? Predictive process monitoring methods: Which one suits me best?  M. Weske, M. Montali, I. Weber J. vom Brocke (), Business Process Management - 16th International Conference, BPM 2018, Sydney, NSW, Australia, September 9-14, 2018, Proceedings Business process management - 16th international conference, BPM 2018, sydney, nsw, australia, september 9-14, 2018, proceedings ( 11080, 462–479).  Springer.  https://doi.org/10.1007/978-3-319-98648-7_27  10.1007/978-3-319-98648-7_27
  * Evermann . (2017) Evermann2017Evermann, J., Rehse, JR. Fettke, P.  2017\.  Predicting process behaviour using deep learning Predicting process behaviour using deep learning.  Decision Support Systems.  https://doi.org/10.1016/j.dss.2017.04.003
  * Folino . (2013) Folino2013Folino, F., Guarascio, M. Pontieri, L.  2013\.  Discovering High-Level Performance Models for Ticket Resolution Processes Discovering high-level performance models for ticket resolution processes.  On the Move to Meaningful Internet Systems: OTM 2013 Conferences: Confederated International Conferences: CoopIS, DOA-Trusted Cloud, and ODBASE 2013, Graz, Austria, September 9-13, 2013. Proceedings On the move to meaningful internet systems: Otm 2013 conferences: Confederated international conferences: Coopis, doa-trusted cloud, and odbase 2013, graz, austria, september 9-13, 2013. proceedings ( 275–282).  Berlin, HeidelbergSpringer Berlin Heidelberg.
  * Friedman (2001) friedman2001greedyFriedman, JH.  2001\.  Greedy function approximation: a gradient boosting machine Greedy function approximation: a gradient boosting machine.  Annals of statistics1189–1232.
  * Galanti . (2020) DBLP:conf/icpm/GalantiCLCN20Galanti, R., Coma-Puig, B., de Leoni, M., Carmona, J. Navarin, N.  2020\.  Explainable Predictive Process Monitoring Explainable predictive process monitoring.  ICPM ICPM ( 1–8).  IEEE.
  * Galanti . (2021) GalantiICPM2021DemoGalanti, R., de Leoni, M., Marazzi, A., Bottazzi, G., Delsante, M. Folli, A.  2021\.  Integration of an Explainable Predictive Process Monitoring System into IBM Process Mining Suite Integration of an explainable predictive process monitoring system into ibm process mining suite.  Proceedings of the ICPM Doctoral Consortium and Tool Demonstration Track 2021 co-located with the 3rd International Conference on Process Mining. Proceedings of the icpm doctoral consortium and tool demonstration track 2021 co-located with the 3rd international conference on process mining.
  * Goldstein . (2015) goldstein2015peekingGoldstein, A., Kapelner, A., Bleich, J. Pitkin, E.  2015\.  Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation.  journal of Computational and Graphical Statistics24144–65.
  * Harl . (2020) harl2020explainableHarl, M., Weinzierl, S., Stierle, M. Matzner, M.  2020\.  Explainable predictive business process monitoring using gated graph neural networks Explainable predictive business process monitoring using gated graph neural networks.  Journal of Decision Systems1–16.
  * Hase Bansal (2020) DBLP:conf/acl/HaseB20Hase, P. Bansal, M.  2020\.  Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior? Evaluating explainable AI: which algorithmic explanations help users predict model behavior?  D. Jurafsky, J. Chai, N. Schluter JR. Tetreault (), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020 Proceedings of the 58th annual meeting of the association for computational linguistics, ACL 2020, online, july 5-10, 2020 ( 5540–5552).  Association for Computational Linguistics.  https://doi.org/10.18653/v1/2020.acl-main.491  10.18653/v1/2020.acl-main.491
  * Huysmans . (2011) DBLP:journals/dss/HuysmansDMVB11Huysmans, J., Dejaeger, K., Mues, C., Vanthienen, J. Baesens, B.  2011\.  An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models.  Decis. Support Syst.511141–154.  https://doi.org/10.1016/j.dss.2010.12.003  10.1016/j.dss.2010.12.003
  * Krause . (2018) krause2018userKrause, J., Perer, A. Bertini, E.  2018\.  A user study on the effect of aggregating explanations for interpreting machine learning models A user study on the effect of aggregating explanations for interpreting machine learning models.  ACM KDD Workshop on Interactive Data Exploration and Analytics. Acm kdd workshop on interactive data exploration and analytics.
  * Krause . (2016) DBLP:conf/chi/KrausePN16Krause, J., Perer, A. Ng, K.  2016\.  Interacting with Predictions: Visual Inspection of Black-box Machine Learning Models Interacting with predictions: Visual inspection of black-box machine learning models.  J. Kaye, A. Druin, C. Lampe, D. Morris JP. Hourcade (), Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, San Jose, CA, USA, May 7-12, 2016 Proceedings of the 2016 CHI conference on human factors in computing systems, san jose, ca, usa, may 7-12, 2016 ( 5686–5697).  ACM.  https://doi.org/10.1145/2858036.2858529  10.1145/2858036.2858529
  * Kulesza . (2015) Kulesza2015Kulesza, T., Burnett, M., Wong, WK. Stumpf, S.  2015\.  Principles of Explanatory Debugging to Personalize Interactive Machine Learning Principles of explanatory debugging to personalize interactive machine learning.  Proceedings of the 20th International Conference on Intelligent User Interfaces Proceedings of the 20th international conference on intelligent user interfaces ( 126–137).  New York, NY, USAAssociation for Computing Machinery.  https://doi.org/10.1145/2678025.2701399  10.1145/2678025.2701399
  * La Rosa . (2011) DBLP:journals/eswa/RosaRADMDG11La Rosa, M., Reijers, HA., van der Aalst, WMP., Dijkman, RM., Mendling, J., Dumas, M. García-Bañuelos, L.  2011\.  APROMORE: An advanced process model repository APROMORE: an advanced process model repository.  Expert Syst. Appl.3867029–7040.  10.1016/j.eswa.2010.12.012
  * Lattimore Szepesvári (2020) lattimore2020banditLattimore, T. Szepesvári, C.  2020\.  Bandit algorithms Bandit algorithms.  Cambridge University Press.
  * Lazar . (2017) lazar2017researchLazar, J., Feng, JH. Hochheiser, H.  2017\.  Research methods in human-computer interaction Research methods in human-computer interaction.  Morgan Kaufmann.
  * Leontjeva . (2015) Leontjeva2015Leontjeva, A., Conforti, R., Di Francescomarino, C., Dumas, M. Maggi, FM.  2015\.  Complex Symbolic Sequence Encodings for Predictive Monitoring of Business Processes Complex symbolic sequence encodings for predictive monitoring of business processes.  BPM 2015 Bpm 2015 ( 297–313).  Springer International Publishing.
  * Lim . (2009) DBLP:conf/chi/LimDA09Lim, BY., Dey, AK. Avrahami, D.  2009\.  _Why and why not_ explanations improve the intelligibility of context-aware intelligent systems _Why and why not_ explanations improve the intelligibility of context-aware intelligent systems.  DRO. Jr., RB. Arthur, K. Hinckley, MR. Morris, SE. Hudson S. Greenberg (), Proceedings of the 27th International Conference on Human Factors in Computing Systems, CHI 2009, Boston, MA, USA, April 4-9, 2009 Proceedings of the 27th international conference on human factors in computing systems, CHI 2009, boston, ma, usa, april 4-9, 2009 ( 2119–2128).  ACM.  https://doi.org/10.1145/1518701.1519023  10.1145/1518701.1519023
  * Lipton (2001) lipton2001goodLipton, P.  2001\.  What good is an explanation? What good is an explanation?  Explanation Explanation ( 43–59).  Springer.
  * Lu Sadiq (2007) DBLP:conf/er/LuS07Lu, R. Sadiq, SW.  2007\.  On the Discovery of Preferred Work Practice Through Business Process Variants On the discovery of preferred work practice through business process variants.  C. Parent, K. Schewe, VC. Storey B. Thalheim (), Conceptual Modeling - ER 2007, 26th International Conference on Conceptual Modeling, Auckland, New Zealand, November 5-9, 2007, Proceedings Conceptual modeling - ER 2007, 26th international conference on conceptual modeling, auckland, new zealand, november 5-9, 2007, proceedings ( 4801, 165–180).  Springer.  https://doi.org/10.1007/978-3-540-75563-0_13  10.1007/978-3-540-75563-0_13
  * Lundberg Lee (2017) DBLP:conf/nips/LundbergL17Lundberg, SM. Lee, S.  2017\.  A Unified Approach to Interpreting Model Predictions A unified approach to interpreting model predictions.  Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA Advances in neural information processing systems 30: Annual conference on neural information processing systems 2017, 4-9 december 2017, long beach, ca, USA ( 4765–4774).
  * Maggi . (20141) DBLP:conf/caise/MaggiFDG14Maggi, FM., Di Francescomarino, C., Dumas, M. Ghidini, C.  20141\.  Predictive Monitoring of Business Processes Predictive monitoring of business processes.  M. Jarke . (), Advanced Information Systems Engineering - 26th International Conference, CAiSE 2014, Thessaloniki, Greece, June 16-20, 2014. Proceedings Advanced information systems engineering - 26th international conference, caise 2014, thessaloniki, greece, june 16-20, 2014. proceedings ( 8484, 457–472).  Springer.  https://doi.org/10.1007/978-3-319-07881-6_31  10.1007/978-3-319-07881-6_31
  * Maggi . (20142) Maggi:CAiSE2014Maggi, FM., Di Francescomarino, C., Dumas, M. Ghidini, C.  20142\.  Predictive Monitoring of Business Processes Predictive monitoring of business processes.  Advanced Information Systems Engineering - 26th International Conference, CAiSE 2014, Thessaloniki, Greece, June 16-20, 2014. Proceedings Advanced information systems engineering - 26th international conference, caise 2014, thessaloniki, greece, june 16-20, 2014. proceedings ( 457–472).
  * Malhi . (2020) DBLP:conf/atal/MalhiKF20Malhi, A., Knapic, S. Främling, K.  2020\.  Explainable Agents for Less Bias in Human-Agent Decision Making Explainable agents for less bias in human-agent decision making.  D. Calvaresi, A. Najjar, M. Winikoff K. Främling (), Explainable, Transparent Autonomous Agents and Multi-Agent Systems - Second International Workshop, EXTRAAMAS 2020, Auckland, New Zealand, May 9-13, 2020, Revised Selected Papers Explainable, transparent autonomous agents and multi-agent systems - second international workshop, EXTRAAMAS 2020, auckland, new zealand, may 9-13, 2020, revised selected papers ( 12175, 129–146).  Springer.  https://doi.org/10.1007/978-3-030-51924-7_8  10.1007/978-3-030-51924-7_8
  * Márquez-Chamorro . (2018) marquez-chamorro_predictive_2018Márquez-Chamorro, AE., Resinas, M. Ruiz-Cortés, A.  2018\.  Predictive Monitoring of Business Processes: A Survey Predictive monitoring of business processes: A survey.  IEEE Transactions on Services Computing116962-977.  10.1109/TSC.2017.2772256
  * Munoz-Gama . (2022) MUNOZGAMA2022103994Munoz-Gama, J., Martin, N., Fernandez-Llatas, C., Johnson, OA., Sepúlveda, M., Helm, E.Zerbato, F.  2022\.  Process Mining for Healthcare: Characteristics and Challenges Process mining for healthcare: Characteristics and challenges.  Journal of Biomedical Informatics103994.  https://www.sciencedirect.com/science/article/pii/S1532046422000107  https://doi.org/10.1016/j.jbi.2022.103994
  * Polato . (2018) Polatoetal:2018Polato, M., Sperduti, A., Burattin, A. de Leoni, M.  2018Feb17.  Time and activity sequence prediction of business process instances Time and activity sequence prediction of business process instances.  Computing.  10.1007/s00607-018-0593-x
  * Ribeiro . (2016) DBLP:conf/kdd/Ribeiro0G16Ribeiro, MT., Singh, S. Guestrin, C.  2016\.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier "why should I trust you?": Explaining the predictions of any classifier.  KDD KDD ( 1135–1144).  ACM.
  * Ribeiro . (2018) DBLP:conf/aaai/Ribeiro0G18Ribeiro, MT., Singh, S. Guestrin, C.  2018\.  Anchors: High-Precision Model-Agnostic Explanations Anchors: High-precision model-agnostic explanations.  SA. McIlraith KQ. Weinberger (), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018 Proceedings of the thirty-second AAAI conference on artificial intelligence, (aaai-18), the 30th innovative applications of artificial intelligence (iaai-18), and the 8th AAAI symposium on educational advances in artificial intelligence (eaai-18), new orleans, louisiana, usa, february 2-7, 2018 ( 1527–1535).  AAAI Press.  https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16982
  * Rizzi . (2020) DBLP:conf/bpm/RizziFM20Rizzi, W., Di Francescomarino, C. Maggi, FM.  2020\.  Explainability in Predictive Process Monitoring: When Understanding Helps Improving Explainability in predictive process monitoring: When understanding helps improving.  D. Fahland, C. Ghidini, J. Becker M. Dumas (), Business Process Management Forum - BPM Forum 2020, Seville, Spain, September 13-18, 2020, Proceedings Business process management forum - BPM forum 2020, seville, spain, september 13-18, 2020, proceedings ( 392, 141–158).  Springer.  https://doi.org/10.1007/978-3-030-58638-6_9  10.1007/978-3-030-58638-6_9
  * Rizzi . (2019) DBLP:conf/bpm/RizziSFGKM19Rizzi, W., Simonetto, L., Di Francescomarino, C., Ghidini, C., Kasekamp, T. Maggi, FM.  2019\.  Nirdizati 2.0: New Features and Redesigned Backend Nirdizati 2.0: New features and redesigned backend.  B. Depaire . (), Proceedings of the Dissertation Award, Doctoral Consortium, and Demonstration Track at BPM 2019 co-located with 17th International Conference on Business Process Management, BPM 2019, Vienna, Austria, September 1-6, 2019 Proceedings of the dissertation award, doctoral consortium, and demonstration track at BPM 2019 co-located with 17th international conference on business process management, BPM 2019, vienna, austria, september 1-6, 2019 ( 2420, 154–158).  CEUR-WS.org.  http://ceur-ws.org/Vol-2420/paperDT8.pdf
  * Shapley (2016) ShapleyShapley, LS.  2016\.  17\. A Value for n-Person Games 17. a value for n-person games.  HW. Kuhn AW. Tucker (), Contributions to the Theory of Games (AM-28), Volume II Contributions to the theory of games (am-28), volume ii ( 307–318).  Princeton University Press.  https://doi.org/10.1515/9781400881970-018  doi:10.1515/9781400881970-018
  * Sindhgatta . (2020) DBLP:conf/bpm/SindhgattaMOB20Sindhgatta, R., Moreira, C., Ouyang, C. Barros, A.  2020\.  Exploring Interpretable Predictive Models for Business Processes Exploring interpretable predictive models for business processes.  D. Fahland, C. Ghidini, J. Becker M. Dumas (), Business Process Management - 18th International Conference, BPM 2020, Seville, Spain, September 13-18, 2020, Proceedings Business process management - 18th international conference, BPM 2020, seville, spain, september 13-18, 2020, proceedings ( 12168, 257–272).  Springer.  https://doi.org/10.1007/978-3-030-58666-9_15  10.1007/978-3-030-58666-9_15
  * Spinner . (2020) DBLP:journals/tvcg/SpinnerSSE20Spinner, T., Schlegel, U., Schäfer, H. El-Assady, M.  2020\.  explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning explainer: A visual analytics framework for interactive and explainable machine learning.  IEEE Trans. Vis. Comput. Graph.2611064–1074.  https://doi.org/10.1109/TVCG.2019.2934629  10.1109/TVCG.2019.2934629
  * Sutton Barto (1998) DBLP:books/lib/SuttonB98Sutton, RS. Barto, AG.  1998\.  Reinforcement learning - an introduction Reinforcement learning - an introduction.  MIT Press.  https://www.worldcat.org/oclc/37293240
  * Tax . (2017) DBLP:conf/caise/TaxVRD17Tax, N., Verenich, I., Rosa, ML. Dumas, M.  2017\.  Predictive Business Process Monitoring with LSTM Neural Networks Predictive business process monitoring with LSTM neural networks.  E. Dubois K. Pohl (), Advanced Information Systems Engineering - 29th International Conference, CAiSE 2017, Essen, Germany, June 12-16, 2017, Proceedings Advanced information systems engineering - 29th international conference, caise 2017, essen, germany, june 12-16, 2017, proceedings ( 10253, 477–492).  Springer.  https://doi.org/10.1007/978-3-319-59536-8_30  10.1007/978-3-319-59536-8_30
  * Teinemaa . (2018) DBLP:journals/datamine/TeinemaaDLM18Teinemaa, I., Dumas, M., Leontjeva, A. Maggi, FM.  2018\.  Temporal stability in predictive process monitoring Temporal stability in predictive process monitoring.  Data Min. Knowl. Discov.3251306–1338.  https://doi.org/10.1007/s10618-018-0575-9  10.1007/s10618-018-0575-9
  * Teinemaa . (2019) Teinemaa2019Teinemaa, I., Dumas, M., Rosa, ML. Maggi, FM.  201903\.  Outcome-Oriented Predictive Process Monitoring: Review and Benchmark Outcome-oriented predictive process monitoring: Review and benchmark.  ACM Trans. Knowl. Discov. Data132.  https://doi.org/10.1145/3301300  10.1145/3301300
  * Tullio . (2007) DBLP:conf/chi/TullioDCF07Tullio, J., Dey, AK., Chalecki, J. Fogarty, J.  2007\.  How it works: a field study of non-technical users interacting with an intelligent system How it works: a field study of non-technical users interacting with an intelligent system.  MB. Rosson DJ. Gilmore (), Proceedings of the 2007 Conference on Human Factors in Computing Systems, CHI 2007, San Jose, California, USA, April 28 - May 3, 2007 Proceedings of the 2007 conference on human factors in computing systems, CHI 2007, san jose, california, usa, april 28 - may 3, 2007 ( 31–40).  ACM.  https://doi.org/10.1145/1240624.1240630  10.1145/1240624.1240630
  * van der Aalst, Adriansyah . (2011) DBLP:conf/bpm/AalstAM11van der Aalst, WMP., Adriansyah, A., de Medeiros, AKA., Arcieri, F., Baier, T., Blickle, T.Wynn, MT.  2011\.  Process Mining Manifesto Process mining manifesto.  F. Daniel, K. Barkaoui S. Dustdar (), Business Process Management Workshops - BPM 2011 International Workshops, Clermont-Ferrand, France, August 29, 2011, Revised Selected Papers, Part I Business process management workshops - BPM 2011 international workshops, clermont-ferrand, france, august 29, 2011, revised selected papers, part I ( 99, 169–194).  Springer.  https://doi.org/10.1007/978-3-642-28108-2_19  10.1007/978-3-642-28108-2_19
  * van der Aalst, Schonenberg Song (2011) Aalstetal:2011van der Aalst, WMP., Schonenberg, MH. Song, M.  2011\.  Time prediction based on process mining Time prediction based on process mining.  Information Systems362450-475.
  * B. van Dongen (2011) vandongen_2011van Dongen, B.  2011Mar.  Real-life event logs - Hospital log. Real-life event logs - hospital log.  4TU.ResearchData.  10.4121/uuid:d9769f3d-0ab0-4fb8-803b-0d1120ffcf54
  * B. van Dongen (2012) vandongen_2012van Dongen, B.  2012Apr.  BPI Challenge 2012. Bpi challenge 2012.  4TU.ResearchData.  10.4121/uuid:3926db30-f712-4394-aebc-75976070e91f
  * B. van Dongen (2017) vandongen_2017van Dongen, B.  2017Feb.  BPI Challenge 2017. Bpi challenge 2017.  4TU.ResearchData.  10.4121/uuid:5f3067df-f10b-45da-b98b-86ae4c7a310b
  * BF. van Dongen . (2008) vanDongen2008van Dongen, BF., Crooy, RA. van der Aalst, WMP.  2008\.  Cycle Time Prediction: When Will This Case Finally Be Finished? Cycle time prediction: When will this case finally be finished?  On the Move to Meaningful Internet Systems: OTM 2008: OTM 2008 Confederated International Conferences, CoopIS, DOA, GADA, IS, and ODBASE 2008, Monterrey, Mexico, November 9-14, 2008, Proceedings, Part I On the move to meaningful internet systems: Otm 2008: Otm 2008 confederated international conferences, coopis, doa, gada, is, and odbase 2008, monterrey, mexico, november 9-14, 2008, proceedings, part i ( 319–336).  Berlin, HeidelbergSpringer Berlin Heidelberg.
  * BF. van Dongen . (2005) DBLP:conf/apn/DongenMVWA05van Dongen, BF., de Medeiros, AKA., Verbeek, HMW., Weijters, AJMM. van der Aalst, WMP.  2005\.  The ProM Framework: A New Era in Process Mining Tool Support The prom framework: A new era in process mining tool support.  G. Ciardo P. Darondeau (), Applications and Theory of Petri Nets 2005, 26th International Conference, ICATPN 2005, Miami, USA, June 20-25, 2005, Proceedings Applications and theory of petri nets 2005, 26th international conference, ICATPN 2005, miami, usa, june 20-25, 2005, proceedings ( 3536, 444–454).  Springer.  https://doi.org/10.1007/11494744_25  10.1007/11494744_25
  * Velmurugan . (20211) DBLP:conf/caise/Velmurugan0MS21Velmurugan, M., Ouyang, C., Moreira, C. Sindhgatta, R.  20211\.  Evaluating Fidelity of Explainable Methods for Predictive Process Analytics Evaluating fidelity of explainable methods for predictive process analytics.  S. Nurcan A. Korthaus (), Intelligent Information Systems - CAiSE Forum 2021, Melbourne, VIC, Australia, June 28 - July 2, 2021, Proceedings Intelligent information systems - caise forum 2021, melbourne, vic, australia, june 28 - july 2, 2021, proceedings ( 424, 64–72).  Springer.  https://doi.org/10.1007/978-3-030-79108-7_8  10.1007/978-3-030-79108-7_8
  * Velmurugan . (20212) DBLP:conf/icsoc/Velmurugan0MS21Velmurugan, M., Ouyang, C., Moreira, C. Sindhgatta, R.  20212\.  Evaluating Stability of Post-hoc Explanations for Business Process Predictions Evaluating stability of post-hoc explanations for business process predictions.  ICSOC ICSOC ( 13121, 49–64).  Springer.
  * Verenich . (2019) DBLP:journals/tist/VerenichDRMT19Verenich, I., Dumas, M., Rosa, ML., Maggi, FM. Teinemaa, I.  2019\.  Survey and Cross-benchmark Comparison of Remaining Time Prediction Methods in Business Process Monitoring Survey and cross-benchmark comparison of remaining time prediction methods in business process monitoring.  ACM Trans. Intell. Syst. Technol.10434:1–34:34.  https://doi.org/10.1145/3331449  10.1145/3331449
  * Vilone Longo (2021) VILONE202189Vilone, G. Longo, L.  2021\.  Notions of explainability and evaluation approaches for explainable artificial intelligence Notions of explainability and evaluation approaches for explainable artificial intelligence.  Information Fusion7689-106.  https://www.sciencedirect.com/science/article/pii/S1566253521001093  https://doi.org/10.1016/j.inffus.2021.05.009
  * Wang Yin (2021) DBLP:conf/iui/WangY21Wang, X. Yin, M.  2021\.  Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making Are explanations helpful? A comparative study of the effects of explanations in ai-assisted decision-making.  T. Hammond, K. Verbert, D. Parra, BP. Knijnenburg, J. O’Donovan P. Teale (), IUI ’21: 26th International Conference on Intelligent User Interfaces, College Station, TX, USA, April 13-17, 2021 IUI ’21: 26th international conference on intelligent user interfaces, college station, tx, usa, april 13-17, 2021 ( 318–328).  ACM.  https://doi.org/10.1145/3397481.3450650  10.1145/3397481.3450650
  * Weinzierl, Dunzer . (2020) weinzierl2020prescriptiveWeinzierl, S., Dunzer, S., Zilker, S. Matzner, M.  2020\.  Prescriptive business process monitoring for recommending next best actions Prescriptive business process monitoring for recommending next best actions.  International conference on business process management International conference on business process management ( 193–209).
  * Weinzierl, Zilker . (2020) DBLP:journals/corr/abs-2008-07993Weinzierl, S., Zilker, S., Brunk, J., Revoredo, K., Matzner, M. Becker, J.  2020\.  XNAP: Making LSTM-based Next Activity Predictions Explainable by Using LRP XNAP: making lstm-based next activity predictions explainable by using LRP.  CoRRabs/2008.07993.  https://arxiv.org/abs/2008.07993
  * Weitz . (2021) DBLP:journals/jmui/WeitzSSHA21Weitz, K., Schiller, D., Schlagowski, R., Huber, T. André, E.  2021\.  "Let me explain!": exploring the potential of virtual agents in explainable AI interaction design "let me explain!": exploring the potential of virtual agents in explainable AI interaction design.  J. Multimodal User Interfaces15287–98.  https://doi.org/10.1007/s12193-020-00332-0  10.1007/s12193-020-00332-0