# Recommending the optimal policy by learning to act from temporal data

Stefano Branchi1 Andrei Buliga1 Chiara Di Francescomarino2 Chiara Ghidini1
Francesca Meneghello1 Massimiliano Ronzani1 1Fondazione Bruno Kessler (FBK)
2University of Trento
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Prescriptive Process Monitoring is a prominent problem in Process Mining,
which consists in identifying a set of actions to be recommended with the goal
of optimising a target measure of interest or Key Performance Indicator (KPI).
One challenge that makes this problem difficult is the need to provide
Prescriptive Process Monitoring techniques only based on temporally annotated
(process) execution data, stored in, so-called execution logs, due to the lack
of well crafted and human validated explicit models. In this paper we aim at
proposing an AI based approach that learns, by means of Reinforcement Learning
(RL), an optimal policy (almost) only from the observation of past executions
and recommends the best activities to carry on for optimizing a KPI of
interest. This is achieved first by learning a Markov Decision Process for the
specific KPIs from data, and then by using RL training to learn the optimal
policy. The approach is validated on real and synthetic datasets and compared
with off-policy Deep RL approaches. The ability of our approach to compare
with, and often overcome, Deep RL approaches provides a contribution towards
the exploitation of white box RL techniques in scenarios where only temporal
execution data are available.

##  1 Introduction

Prescriptive Process Monitoring (PPM) is a prominent problem in Process
Mining, which consists in identifying a set of actions or interventions to be
recommended with the goal of optimising a target measure of interest or Key
Performance Indicator (KPI). In its simplest formulations it contains methods
for raising alarms or triggering interventions, to prevent or mitigate
undesired outcomes, as well as for recommending the best resource allocation.
Only recently, works have targeted the generation of recommendations of the
next activity(ies) to optimize a certain KPI of interest (?; ?; ?), such as,
the cycle time of the process execution in a single-actor setting (or ignoring
the actor dimension). Only one recent work (?) explicitly considers the
process execution in the context of a multi-actor complex environment that
depends upon exogenous factors, including how the other process actors behave.
In this setting, identifying the best strategy to follow for a target actor is
not straightforward and Artificial Intelligence (AI) techniques based on
Reinforcement Learning (RL) have shown promising results.

Indeed, Reinforcement Learning is increasingly used as one of the state-of-the
art solutions in several areas where strategic reasoning is needed in a multi-
actor setting: from gaming (?) to dialogue generation (?), from smart city
policy making (?), to planning (?), just to name a few. Two RL approaches are
used in the literature: one relies entirely on data and exploits techniques of
Deep RL; the other exploits explicit Markov Decision Processes (MDP), but
assumes the existence of a model (the MDP) for “playing the game”. The first
approach has the advantage of avoiding the construction of, often complex,
models, but has the drawback of producing black box decisions; while the
second is more transparent but requires a model (the MDP) of the scenario at
hand for the KPIs of interest. Unfortunately, the explicit knowledge required
to build an MDP is often not available in the Process Mining domain. Moreover,
it is a knowledge difficult to obtain by manually inspecting the process
executions stored in the, so-called, _event logs_ (as done in (?)), or by
using Process Discovery techniques (?), which are not tailored to discover
meaningful MDP states for the considered KPIs. Thus, an important challenge to
address in PPM is the ability to provide methods able to recommend next
activity(ies) in multi-actor processes only exploiting the information
contained in the temporally annotated (process) execution data, ideally
learning an MDP in a white box manner, thus avoiding black box Deep RL
approaches.

In this paper we aim at tackling this challenge by proposing an AI based
approach that learns, by means of Reinforcement Learning, an optimal policy
only from the observation of temporally annotated sequences of event data
i.e., the past process executions, and recommends the best activities to carry
on for optimizing a KPI of interest in a multi-actor setting. This is achieved
first by learning an explicit MDP for the specific KPIs from the event log
(Sec. 3.1), and then by using RL training to learn the optimal policy (Sec.
3.2). One of the challenges we have to tackle is the construction of a MDP
which must contain a fulfilling description of the system. This can be
technically difficult because a _state space_ which describes all possible
configurations of the system can become unbearably big, so some sort of
effective technique to avoid that must be put in place. We address this
challenge by using clusterization to group together similar process
executions, which — intuitively — represent similar configurations of the
system which can be grouped into a single state, thus greatly reducing the
state space dimension without affecting the quality of the result. To show the
validity of the approach we evaluate it on real and synthetic datasets, (i) by
adapting the estimation of the _state-action value function_ $q$ — via scaling
— to avoid problems related to overfitting, and (ii) by comparing it with off-
policy deep Reinforcement Learning approaches also taking into account the log
dimension and the popularity of successful traces in the event log (Sec. 4 and
5).

The ability of our approach to compare with, and often overcome, Deep RL
approaches shows how our technique contributes in a significant manner to
learn explicit Markov Decision Processes directly from temporal execution data
in an effective manner. Therefore, it opens the doors to the exploitation of
white box RL techniques in scenarios where only temporally annotated data are
available, which has the potential to be applied outside the Process Mining
area.

##  2 Background

In this section we provide the background knowledge necessary to understand
the rest of the paper.

###  2.1 Event log

An _event log_ $\mathcal{L}$ consists of traces representing executions of a
process (a.k.a. cases). A _trace_ $\sigma=\langle
e_{1},e_{2},\dots,e_{n}\rangle$ is an ordered finite sequence of events
$e_{i}$, each referring to the execution of an activity label $\mathfrak{a}\in
A$. The _event_ $e_{i}=(\mathfrak{a},t,p_{1},\ldots,p_{n})$ is, in turn, a
complex structure comprising the activity label
$\mathfrak{a}=\text{Act}(e_{i})$, its timestamp $t$, indicating the time in
which the event has occurred, and possibly _data payloads_
$p_{1},\ldots,p_{n}$, consisting of attributes, such as, the resource(s)
involved in the execution of an activity, or other data recorded during the
event. A partial trace execution, or _prefix_ , of a trace $\sigma=\langle
e_{1},e_{2},\dots,e_{n}\rangle$ is a sequence $\sigma_{k}$ containing the
first $k$ events of $\sigma$, with $k\leq n$. For instance, given a trace
$\sigma=\langle e_{1},e_{2},e_{3},e_{4}\rangle$, the prefix
$\sigma_{2}=\langle e_{1},e_{2}\rangle$.

We define the _frequency_ $f_{\mathfrak{a}}(\sigma)$ of a trace $\sigma$, the
number of times the activity label $\mathfrak{a}$ occurs in the trace
$\sigma$. We define the _position_ $p_{\mathfrak{a}}(\sigma)$ of a trace
$\sigma$, the last position in which the activity label $\mathfrak{a}$ occurs
in the trace $\sigma$, with the first event having position 1. Both frequency
and position are equal to zero if the corresponding activity is missing in the
trace. For instance, given a trace
$\sigma=\langle(\mathfrak{a}_{1},t_{1}),(\mathfrak{a}_{2},t_{2}),(\mathfrak{a}_{1},t_{3})\rangle$,
$f_{\mathfrak{a}_{1}}(\sigma)=2$ and $p_{\mathfrak{a}_{1}}(\sigma)=3$.

###  2.2 Reinforcement Learning

Reinforcement Learning (?) is a form of unsupervised learning where an agent
learns to act in an environment by maximizing the total amount of reward
received by its actions. At each time step $t$, the agent chooses and performs
an _action_ $a$ in response to the observation of the _state_ of the
environment $s$. Performing action $a$ causes, at the next time step $t+1$,
the environment to stochastically move to a new state $s^{\prime}$, and gives
the agent a _reward_ $r_{t+1}=\mathcal{R}(s,a,s^{\prime})$ that indicates how
well the agent has performed. The probability that, given the current state
$s$ and the action $a$, the environment moves into the new state $s^{\prime}$
is given by the state transition function $\mathcal{P}(s,a,s^{\prime})$.

The learning problem is therefore described as a discrete-time MDP $M$, which
is defined by a tuple
$M=(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma)$, where:

  * •

$\mathcal{S}$ is the set of states.

  * •

$\mathcal{A}$ is the set of agent’s actions.

  * •

$\mathcal{P}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}_{[0,1]}$
is the transition probability function.
$\mathcal{P}(s,a,s^{\prime})=Pr(s_{t+1}=s^{\prime}|s_{t}=s,a_{t}=a)$ is the
probability of transition (at the time step $t$) from state $s$ to state
$s^{\prime}$ under action $a\in\mathcal{A}$.

  * •

$\mathcal{R}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}$
is the reward function. $\mathcal{R}(s,a,s^{\prime})$ is the reward obtained
by going from $s$ to $s^{\prime}$ with action $a$.

  * •

$\gamma\in\mathbb{R}_{[0,1]}$ is the discount factor for the rewards. Value of
$\gamma<1$ models an agent that discounts future rewards.

An MDP satisfies the _Markov Property_ , that is, given $s_{t}$ and $a_{t}$,
the next state $s_{t+1}$ is conditionally independent of all prior states and
actions, and it only depends on the current state, i.e.,
$Pr(s_{t+1}|s_{t},a_{t})=Pr(s_{t+1}|s_{0},\cdots,s_{t},a_{0},\cdots,a_{t})$.

A policy $\pi:\mathcal{S}\rightarrow\mathcal{A}$ is a mapping from each state
$s\in\mathcal{S}$ to an action $a\in\mathcal{A}$, and the _cumulative reward_
is the (discounted) sum of the rewards obtained by the agent while acting at
the various time points.

The _state-action value function_ $q^{\pi}$ is the expected discounted future
reward obtained by taking action $a$ in state $s$ and then continuing to use
the policy $\pi$ thereafter

$q^{\pi}(s,a)=\mathbb{E}_{\pi}\bigg{(}\sum_{k=0}^{\infty}\gamma^{k}r_{k+t+1}\bigg{|}s=s_{t},a=a_{t}\bigg{)}$
(1)

where $r_{t+1}=\mathcal{R}(s_{t},a_{t},s_{t+1})$ is the reward obtained at
time $t+1$ and $\mathbb{E}$ means expectation value.

A policy is _optimal_ , usually denoted as $\pi^{*}$, if it maximizes the
state-action value (1), that is if $q^{*}(s,a)\coloneqq q^{\pi^{*}}(s,a)\geq
q^{\pi}(s,a)$ for all $\pi$, $s\in\mathcal{S}$ and $a\in\mathcal{A}$. Learning
the optimal policy is the goal of RL.

In RL, _on-policy_ methods improve the current policy, while _off-policy_
methods improve a target policy different from the policy used for generating
experiences.

Policy iteration is an on-policy method that iteratively evaluates
(_evaluation phase_) and improves (_optimization phase_) an arbitrary policy
$\pi$ until convergence. In the evaluation phase, _Monte Carlo methods_
estimate the action-value function $q^{\pi}(s,a)$ using simulations of
episodes generated by alternating between exploiting the policy and exploring
the action space. After each episode, the $q^{\pi}$ value is updated for every
visited state-action pair $(s,a)$ by estimating its value as the mean of the
sampled returns that originated from $(s,a)$ over time. With enough time, an
accurate estimate of $q^{\pi}$ can be obtained. Policy improvement computes a
greedy policy based on $q^{\pi}$, where given a state $s$, the new policy
returns an action that maximizes $q^{\pi}(s,\cdot)$.

In an _off-policy_ setting, the agent collects experiences by interacting with
the environment and stores them in a buffer. The actions are chosen from the
behavioural policy and for the policy improvement step, experiences are
sampled from the buffer to learn the target policy.

_Q-Learning_ is an off-policy method that evaluates and improves a separate
greedy policy while following the behavioural policy. It is a model-free value
iteration algorithm that defines an action-value function $Q$ by the following
update formula:

$Q(s,a)\leftarrow
Q(s,a)+\alpha\left[\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime})-Q(s,a)\right]$
(2)

where $\alpha$ is the learning rate parameter. The action-value function $Q$,
defined by (2), directly approximates the optimal action-value function
$q^{*}$, independently of the policy being followed in the updates. As opposed
to on-policy methods, Q-Learning does not wait until the end of the episode to
get the returns and update $q^{\pi}$, but can do so in an incremental fashion
at each timestep.

##  3 Learning to act from temporal data

In this section we describe the proposed RL solution, whose pipeline is
reported in Figure 1. The pipeline takes as input an event log related to a
multi-actor process, the definition of a relevant KPI to optimise, as well as
the ownership of the activities in the log and builds via RL a (recommender)
system that recommends the next best activity(ies) in order to optimise the
desired KPI. It is composed of three phases: in the _Preprocessing phase_ an
MDP is built starting from the three inputs of the pipeline (Sec. 3.1); in the
_Reinforcement Learning phase_ the MDP is trained by the RL algorithm to learn
the best policy, and, finally, in the _Runtime phase_ , the policy is used to
recommend the best next activity(ies) for an ongoing execution (Sec. 3.2).

Figure 1: Pipeline of the proposed RL solution.

###  3.1 MDP construction from log data

The aim of the _Preprocessing phase_ is building an MDP starting from: (i) the
event log; (ii) the list of activities carried out by the _agent_ , and the
ones by the _environment_ ; and (iii) the KPI to optimise.

At a high level, the following mapping between the MDP components (actions and
states) and the information extracted from the event log can be defined as
follows:

  * •

actions: the activities that the agent can carry out;

  * •

states: a comprehensive description of the system. In a fashion similar to (?)
it contains three components:

    * –

last activity selected by the agent or the environment;

    * –

the trace history, that is, the condensed information about the entire trace
execution up to that point;

    * –

information about the reward obtained up to that point.

Considering the definition of MDP action and state reported above, the steps
to follow for the construction of the MDP are the following:

  1. 1.

_Enriching the log._ In this step, based on the list of the agent’s
activities, each activity in the log is marked as an agent or as an
environment (all the other activities) activity. Moreover, the KPI of interest
is computed for each trace $\sigma$. Its value can depend on the executed
activities or on other attributes of the trace, and represents the reward
$r(\sigma)$ of the whole trace $\sigma$. Note that we define $r(\sigma_{k})=0$
for incomplete prefixes $\sigma_{k}$, $k<\text{len}(\sigma)$.

  2. 2.

_Encoding and clustering using k-means._ In this step, each prefix in the log
is encoded using three types of information: the frequency of the activities,
the last position in which an activity has occurred, and the reward obtained
up to that point. Namely, if the alphabet of all the activities is
$\\{\mathfrak{a}_{1},\dots,\mathfrak{a}_{n}\\}$ then the encoding of a prefix
$\sigma_{k}$ is:

$\bm{v}_{\sigma_{k}}=\left(\frac{f_{\mathfrak{a}_{1}}(\sigma_{k})}{f_{\text{max}}},\dots,\frac{f_{\mathfrak{a}_{n}}(\sigma_{k})}{f_{\text{max}}},\frac{p_{\mathfrak{a}_{1}}(\sigma_{k})}{p_{\text{max}}},\dots,\frac{p_{\mathfrak{a}_{n}}(\sigma_{k})}{p_{\text{max}}},\tilde{r}(\sigma_{k})\right)$
(3)

where $f_{\mathfrak{a}_{i}}$ and $p_{\mathfrak{a}_{i}}$ are defined in Sect.
2. $f_{\text{max}}$ and $p_{\text{max}}$ are respectively the highest
frequency and position for all activities and all prefixes in the log. Note
that the latter can also be seen as the max length of all traces in the log

$f_{\text{max}}=\max_{\begin{subarray}{c}\mathfrak{a}\in
A,\sigma\in\mathcal{L}\\\
0<k\leq\text{len}(\sigma)\end{subarray}}\big{(}f_{\mathfrak{a}}(\sigma_{k})\big{)},\qquad
p_{\text{max}}=\max_{\sigma\in\mathcal{L}}\big{(}\text{len}(\sigma)\big{)}$
(4)

Finally, $\tilde{r}(\sigma_{k})$ is the _prefix normalized reward_ : it is
equal to $0$ for proper trace prefixes, while it is the reward of the trace
normalized with respect to the rewards of all the traces for complete traces.
In other terms:

$\tilde{r}(\sigma_{k})=\begin{cases}0&k<\text{len}(\sigma)\\\
\frac{r(\sigma)-\min_{\sigma^{\prime}\in\mathcal{L}}r(\sigma^{\prime})}{\max_{\sigma^{\prime}\in\mathcal{L}}r(\sigma^{\prime})-\min_{\sigma^{\prime}\in\mathcal{L}}r(\sigma^{\prime})}&k=\text{len}(\sigma)\end{cases}$
(5)

The encoded prefixes are then used to train a k-means model (?) that will be
able to assign a new prefix to the cluster containing the most similar
prefixes.

  3. 3.

_Constructing the MDP._ Once the k-means model has been trained, the MDP can
be built. For each prefix $\sigma_{k}$, a state $s_{\sigma_{k}}$ is defined as
the pair of the last performed activity
$\mathfrak{a}_{i_{k}}=\text{Act}(e_{k})$ and the cluster $c_{\sigma_{k-1}}$
assigned to $\sigma_{k-1}$ by the k-means model:

$s_{\sigma_{k}}=\big{(}\mathfrak{a}_{i_{k}},\,c_{\sigma_{k-1}}\big{)}.$ (6)

This definition of state would allow us to include, in the state, information
about the history of the execution and its reward.

Once defined the states, the MDP is built by replaying the traces in the event
log: we build a directed graph, where states correspond to nodes and edges
correspond to actions moving from one node state to another. Moreover, for
each edge, the probability of reaching the target node (computed based on the
number of traces in the event log that reach the corresponding state) and the
value of the reward function are computed. Each edge is mapped to the tuple
$(s,a,s^{\prime},\mathcal{P},\mathcal{R})$ where $s$ is the state
corresponding to the source node of the edge, $a$ is the action used for
labelling the edge, $s^{\prime}$ is the state corresponding to the target node
of the edge, $\mathcal{P}(s,a,s^{\prime})$ is computed as the percentage of
the traces that reach the state $s^{\prime}$ among all the traces that reach
state $s$ and execute action $a$. The reward function
$\mathcal{R}(s,a,s^{\prime})$ is computed as the average of the rewards
$r(\sigma_{k})$ of those prefixes $\sigma_{k}$ corresponding to the edge
$(s,a,s^{\prime})$ 111A prefix $\sigma_{k}$ corresponds to an edge
$(s,a,s^{\prime})$ if $s^{\prime}=s_{\sigma_{k}}$, $a=\text{Act}(e_{l})$ and
$s=s_{\sigma_{l-1}}$, where $e_{l}$ is the last event with $l<k$ associated to
an agent’s activity in $\sigma_{k}$..

###  3.2 RL and runtime phases

In the _Reinforcement Learning_ phase, the MDP obtained in the previous phase
is trained by the RL algorithm to learn the optimal policy, as shown in Figure
1. We use Monte Carlo simulation: each simulation uses the MDP to generate an
episode, or trace, whose final reward is eventually used to properly update
the state-action value (1) and then the policy via policy iteration.

In (?) the authors noted that the MDPs mined from event data logs can have
some issues. Namely some action choices in the MDP guarantee to gain a high
final reward almost certainly during the training, however these choices are
not reliable, because they appear in too few traces, and so the very likely
high reward is accidental and does not rely on a robust statistical
correlation within the training data. This can be seen as an overfitting on
the training data used to construct the MDP.

To tackle this issue in (?) a recalibration of the reward was performed for
every transition with a multiplicative factor depending on the number of
occurrences of the given transition in the log. In this way, very convenient
but unreliable actions were discouraged during the training with respect to
convenient actions more popular in the log. In this work, instead of scaling
the reward of each transition, we scale the $q$-value function (1) as follows

$\tilde{q}(s,a)=q(s,a)\,h(s,a)$ (7)

where $h:\mathcal{S}\times\mathcal{A}\to\mathbb{R}_{[0,1]}$ is a monotonically
non-decreasing function of the number $n(s,a)$ of occurrences of the state-
action pair in the log. The trivial choice $h^{0}(s,a)=1$ defines the standard
$q$-value function. In this work we consider three non-trivial types for $h$
and we test their effect on the policy learned:

  * •

a linear function of the number of occurrences:

$h^{\text{lin}}(s,a)=\frac{n(s,a)-\min_{s^{\prime}\in\mathcal{S},a^{\prime}\in\mathcal{A}}{n(s^{\prime},a^{\prime})}}{\max_{s^{\prime}\in\mathcal{S},a^{\prime}\in\mathcal{A}}{n(s^{\prime},a^{\prime})}-\min_{s^{\prime}\in\mathcal{S},a^{\prime}\in\mathcal{A}}n(s^{\prime},a^{\prime})}$

  * •

a step function $h^{\text{step}}_{n_{t}}$ which is equal to zero for $n(s,a)$
smaller or equal than a certain threshold $n_{t}>0$ and is equal to one for
higher values of $n(s,a)$, that is:

$h^{\text{step}}_{n_{t}}(s,a)=\begin{cases}0&n(s,a)\leq n_{t}\\\
1&n(s,a)>n_{t}\end{cases}$

  * •

a smooth function

$h^{\text{smooth}}_{\lambda}(s,a)=1-2\frac{e^{-n(s,a)/\lambda}}{1+e^{-n(s,a)/\lambda}}$

parametrized by a real value $\lambda>0$. This choice for $h$ actually
interpolates between the previous two proposals: indeed
$h^{\text{smooth}}_{\lambda\to 0}\to h^{\text{step}}_{n_{0}=0}$ and
$h^{\text{smooth}}_{\lambda\to\infty}\to h^{\text{lin}}$.

For every choice of $h$ we obtain a different policy $\pi_{h}$ from the RL
training.

Finally, in the _Runtime_ phase, as shown in Figure 1, the policy is used to
recommend the best next activity(ies) for a ongoing process execution222 The
recommender system takes the ongoing prefix, maps it to a state following the
same steps as in Sect. 3.1 and uses the policy to recommend the best next
activity to maximize the reward (KPI)..

##  4 Evaluation setting

In the following we report the evaluation setting we designed in order to
measure the performance of the policies $\pi_{h}$ obtained by applying the
proposed pipeline, and to compare them with a state-of-the-art baseline.

###  4.1 Research questions

We aim at evaluating the capability of the proposed solution to conveniently
recommend the next activities so as to optimise the relevant KPI targeted. To
this aim, we assess the capability of the proposed solution: (i) when
different scaling functions $h$ are used; (ii) with respect to state-of-the-
art approaches; (iii) when applied to datasets with different characteristics
(e.g., size and success rate). More in detail, we are interested to answer the
following research questions:

  1. RQ1

How do the different instantiations of the scaling function $h$ perform in
terms of policy capability to optimise the KPI of interest?

  2. RQ2

How does the proposed approach perform in terms of policy capability to
optimise the KPI of interest with respect to state-of-the-art techniques
directly learning from temporal data (Deep RL approaches)?

  3. RQ3

How does the proposed approach perform in terms of robustness of the results
with datasets of different sizes and success rates with respect to Deep RL
approaches?

RQ1 aims at comparing the different instantiations of the $h$ function
presented and to evaluate whether they are able to recommend more effective
policies than the ones generated using the standard function $h^{0}=1$. The
aim of RQ2 is to compare the proposed pipeline with state-of-the-art RL
approaches that directly learn from temporal data without requiring further
domain knowledge (e.g., the MDP) as input and, in particular, with black-box
methods as Deep RL techniques. Finally, RQ3 aims at evaluating the impact of
different characteristics of the training dataset (dataset size and percentage
of successful traces, i.e., traces scoring well on the KPI of interest) on our
solution and Deep RL approaches to return a policy that optimises the KPI of
interest.

###  4.2 Datasets description

To evaluate the performance of the proposed solution we use the real-world
publicly-available BPI Challenge 2012 event log (BPIC2012) (?) and a set of
four synthetic event logs inspired by the BPIC2012 event log.

The BPIC2012 event log describes the execution of a loan application process
in a Dutch Financial Institute. Every process execution represents a single
application to the bank by a customer for a personal loan. The application can
be accepted or rejected by the bank, likewise the customer can withdraw the
application at any time. If the bank decides to grant a loan to a customer it
generates an offer and sends it to the customer, that in turn can decide to
accept the offer or to refuse it. This log is a good example of a multi-actor
process scenario, where we chose to take the perspective of the bank actor, so
that the agent is the _bank_ and the environment is the _customer_. The two
actors have different goals, the customer aims for a convenient loan offer,
whereas the bank seeks reliable customers to whom to send loan offers. The
overall goal of the bank is the acceptance of the offer by the customer.
Indeed, if the customer declines the offer, the time and the resources spent
by the bank in generating the offer are wasted. In this scenario we define the
KPI that the bank aims at optimising as the profit deriving from the loan,
excluding the costs spent in the process execution. The KPI of the bank is
hence composed of two parts: (i) a positive part that is the profit made by
the bank if the loan offer is accepted by the customer, namely the interest
rate of the loan, which we arbitrarily set as the $15\%$ of the amount
requested in the loan application; and (ii) a negative part proportional to
the working time spent in the process, which we set as a cost of 36 euros/h.

For the generation of the four synthetic logs we define a simulation model
$\mathcal{M}$ inspired by the BPIC2012 process. The simulation model consists
of the BPMN model in Figure 2, extended with additional information for the
simulation aspects333 The complete BPMN model and all the simulation
parameters used to define $\mathcal{M}$ are available in the evaluation
repository. (case inter-arrival time, activity durations, routing
probabilities, resource allocation and utilization, etc.). The main
differences between the four synthetic logs concern their sizes, that is the
number of traces they contain, and the probability that a loan application is
(pre)accepted by the bank, which, in turn, affects the probability that a loan
offer is accepted by a customer and hence the possibility for the bank to
obtain a high KPI. Table 1 shows the details of the different event logs
(including the real one). $\mathcal{L}_{rare}^{s}$ and
$\mathcal{L}_{rare}^{b}$ are the logs with a low success rate, that is with a
low percentage of traces achieving the acceptance of the offer by the customer
(activity accept in Figure 2), and hence with a low average KPI value. This is
achieved by imposing a low probability (less than 50%) in the model to have a
pre-accepted loan (gateway 1 in the model). After this first gateway, the
probability of obtaining the loan depends on different factors: the amount
required in the loan application, the number of offers created and calls made
by the bank during the process. The differences between these four synthetic
logs will be central for answering RQ3.

###  4.3 Baseline definition

With the aim of comparing the proposed pipeline with state-of-the-art
approaches directly learning from temporal data, that is Deep RL approaches
(RQ2 and RQ3), we trained an offline Deep-Q Network (DQN). Within a complex
environment, where there can be a large number of states, it is difficult to
compute the action-value function $Q(s,a)$ for every possible state-action
pair and thus model approximation methods have to be used to approximate
$q^{*}$. Following the architecture presented in (?), we employed a Long-Short
Term Memory (LSTM) to approximate $q^{*}$. Since LSTM-based neural networks
preserve the sequentiality of the input, when encoding the traces for the DQN
method, we encoded the frequency at each timestep, foregoing the last seen
position encoding used to construct the MDP in Section 3.

Q-learning updates at iteration $i$ are done through the use of the mean
squared error:

$L_{i}(\theta_{i})=\mathbb{E}_{s,a,r,s^{\prime}}\left[r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime},\theta^{-}_{i})-Q(s,a,\theta_{i}))\right]^{2}$
(8)

where $\gamma$ represents the discount factor applying penalties to future
rewards, $\theta_{i}$ represents the parameters of the LSTM model at iteration
$i$ and $\theta^{-}_{i}$ represents the parameters of the target network at
iteration $i$ used to approximate $\max_{a^{\prime}}q^{*}(s,a)$. In the
literature, DQNs are trained by randomly sampling batches of experiences from
the replay buffer during training to avoid overfitting. For trace executions,
by sampling a single transition from one timestep to the other, the
sequentiality of individual traces would not be picked up by the LSTM model.
To overcome this and avoid overfitting, we sampled traces to be trained
sequentially but randomize the order of the traces in the log to avoid
spurious correlations that might be learned by the order in which the traces
are fed into the model. Since some traces in the event log have long
executions in terms of the activities performed, a proper window size was used
when building the state input for the LSTM model to preserve long-term
dependencies. Since some actions are not available at specific timestamps,
action masking is utilised to constrain the possible action space based on the
last activity in the trace at the current timestep. For our baseline we
implemented a masking function that, during the $Q(s,a)$ value update,
restrict the $\max_{a^{\prime}}$ function in (8) to those actions that have
been seen in the event log. Since we cannot interact with the environment and
perform exploration, this allows us to restrict the space of possible choices
to the ones observed in the data.

###  4.4 Evaluation methods

We design two different evaluation approaches to answer the research questions
in Section 4.1: a simulation evaluation and a test log analysis.

#### Simulation evaluation

A well-known problem in evaluating recommender systems for a real-world
process scenario is the difficulty of testing the recommendations provided by
the recommender system (?). To circumvent this issue we leverage synthetic
logs. Indeed, this allows us to test the policies trained on the synthetic
logs, by simulating a testing situation via the model used to generate the
log.

The simulation evaluation uses a modified version of the simulation model
$\mathcal{M}$ employed to generate the synthetic dataset in Section 4.2, This
modified model $\mathcal{M}^{\prime}$ uses the policies obtained by our
method, or by the DQN baseline, to recommend the next activity for the bank.
Each step of the simulation is classified as a multiple decision point, i.e.
either the agent (bank) or the environment (customer) can act, or as a single
decision point, where only the agent can perform an activity.

For the multiple decision points, as _gateways_ 3, 5 and 6 in Figure 2, the
environment has priority over the agent, so it can act (or not act) according
to the probability distribution of the original model $\mathcal{M}$ 444As
described in Section 4.2 the probability distribution of the environment
response depends on the ongoing prefix $\sigma_{k}$.. The policy requires as
input the prefix $\sigma_{k}$ of the trace that is being simulated, and
returns the recommended next activity. The model $\mathcal{M}^{\prime}$ checks
if the recommended next activity is allowed, i.e. the activity satisfies the
BPMN model, otherwise the trace simulation ends with an exception and the
reward is the one accumulated until then. In addition, during the simulation,
it may happen that a trace never reaches the end of the process because the
policy keeps recommending activities that have negative effects on the
environment response, that is, it reduces to zero the probability of the
environment to respond. In this case, we stop the trace simulation as before.

For every synthetic log
($\mathcal{L}^{s},\mathcal{L}^{b},\mathcal{L}_{rare}^{s},\mathcal{L}_{rare}^{b}$)
we generate a new model $\mathcal{M}^{\prime}$ for each policy trained on it
($\pi_{0},\pi_{\text{lin}},\pi_{\text{step}},\pi_{\text{smooth}},\pi_{\text{DQN}}$).
Then we use these models to simulate the behaviour of the environment in
response to the agent’s action recommended by the policy for 5 000 traces and
compute their average reward, which is the metrics we use to evaluate and
compare the techniques. Although this type of setting offers a way to perform
an online evaluation of the recommender system, it can be used only for
synthetic scenarios in which the model is available. However, the behaviour of
synthetic logs can be less complex when compared to the behaviour of real-life
logs. We hence also have a test log analysis evaluation.

Figure 2: The BPMN model used by $\mathcal{M}$ to generate the synthetic logs
and by $\mathcal{M}^{\prime}$ to perform the simulation evaluation.

Dataset | Trace # | Variant # | Event # | Avg.trace | Application | Offers
---|---|---|---|---|---|---
length | preaccepted | accepted
BPI2012 | 13087 | 4366 | 262200 | 20 | 56% | 17%
$\mathcal{L}^{s}$ | 2000 | 704 | 29077 | 29 | 60% | 23%
$\mathcal{L}_{rare}^{s}$ | 2000 | 496 | 22165 | 22 | 40% | 16%
$\mathcal{L}^{b}$ | 10000 | 3083 | 149997 | 29 | 60% | 24%
$\mathcal{L}_{rare}^{b}$ | 10000 | 2070 | 109652 | 22 | 40% | 16%

Table 1: Datasets Description.

#### Test log analysis

We perform an analysis of the test event log, which is the same analysis
considered in the work (?). This analysis is important as it allows us to
evaluate the performance on the real-world log, for which we do not have the
exact process model. However, to test the trustworthiness of this type of
evaluation, we also apply it to the synthetic logs, so that for these datasets
the two type of evaluations (simulation and test log analysis) can be
compared.

We perform two types of log analysis. In the first one we compare the average
KPI value of the traces in the test log that follow the optimal policy with
the average KPI value of all the traces in the log. In the second log analysis
we focus on evaluating recommendations for ongoing executions, i.e., we
measure how much following the policy can improve the outcome of cases where
some activity has already been executed. To do this we consider, for each
trace in the test event log, all its prefixes and separately analyse each of
them, as a potential ongoing execution. For each prefix $\sigma_{k}$ of a
trace $\sigma$ in the test event log we compare the ground truth value of the
KPI of interest of the trace $\sigma$ once completed, against an estimation of
the value of the KPI obtained following the optimal policy from that execution
point forward. The estimation is obtained by averaging the KPI values of all
the traces $\tau$ in the log that have the same prefix $\tau_{k}=\sigma_{k}$
and follow the optimal policy from there on.

###  4.5 Training procedure

Each dataset described in Table 1 was split into two parts: $80\%$ was used to
train the agent of the RL model and 20% for testing the retrieved policy 555
For the test log analysis (Section 4.4) we excluded from the test event log
all those traces that end without the presence of an actual decision point for
the agent, since the outcome of these traces cannot be modified by the
decisions of the agent..

Each log was preprocessed as described in Section 3.1. In the kmeans
clusterization step we selected 100 clusters for every log, this choice was
made after performing a silhouette analysis on every dataset. We trained the
RL agent in order to obtain 4 different policies, each corresponding to one of
the instantiations of the scale function described in Section 3.2: $\pi_{0}$
to the trivial case $h^{0}=1$, $\pi_{\text{lin}}$ to the linear function
$h^{\text{lin}}$, $\pi_{\text{step}}$ to the step function
$h^{\text{step}}_{n_{t}=50}$ and $\pi_{\text{smooth}}$ to the smooth function
$h^{\text{smooth}}_{\lambda=50}$.666$n_{t}=50$ and $\lambda=50$ were selected
after an explorative analysis available at the evaluation repository.

##  5 Evaluation results

In this section we show the results of the evaluation and answer the research
questions of Section 4.1. All the code and material are available at
https://tinyurl.com/repOptimalPolicy.

To answer RQ1 we compared the four policies obtained with our method for each
dataset ($\pi_{0},\pi_{\text{lin}},\pi_{\text{step}},\pi_{\text{smooth}}$). We
compared them using the two methods described in Sect. 4.4. Figure 3 reports
the average rewards obtained for the synthetic logs in the simulation
evaluation. It can be noticed that all the policies provide a much higher
average reward than the standard average reward of the log $\mathcal{L}$
generated by the model $\mathcal{M}$. However, among the four policies there
is no definite winner, and their performances are actually close. To refine
this analysis we report in Table 2 the difference in terms of average reward
between pairs of policies, by highlighting in bold the statistically relevant
differences (p-value $\leq 0.05$). For example, only one statistically
relevant difference was found for the log $\mathcal{L}^{b}_{rare}$ between
$\pi_{\text{step}}$ and $\pi_{0}$.

Figure 4 shows the average rewards of the optimal traces obtained for the five
logs (synthetic and real) in the test log event analysis. In this case the
results are more varied and differences can be observed when compared to the
results in Figure 3. Indeed, there are few cases in which the optimal traces
with respect to a policy actually have an average reward lower than the
average reward of the test log. However, these latter results have to be
considered cautiously considering the low statistic of the samples used to
compute the average reward. Although the analyses carried out do not allow us
to identify the best scaling function instantiation, we can state that, by
applying a scaling function, we usually obtain better results and that
$\pi_{\text{smooth}}$ seems to have steady performance across all the logs
(RQ1).

Figure 3: Simulation evaluation results for the synthetic logs. The column
$\mathcal{L}$ corresponds to the average reward of the synthetic log generated
by the model $\mathcal{M}$, without the application of any policy.

$\pi_{1}$ vs $\pi_{2}$ | $\mathcal{L}^{s}$ | $\mathcal{L}_{rare}^{s}$ | $\mathcal{L}^{b}$ | $\mathcal{L}_{rare}^{b}$
---|---|---|---|---
$\pi_{\text{lin}}$ vs $\pi_{\text{smooth}}$ | 29 | -152 | -45 | 7
$\pi_{0}$ vs $\pi_{\text{smooth}}$ | -35 | -23 | -28 | -19
$\pi_{0}$ vs $\pi_{\text{lin}}$ | -64 | 129 | 17 | -26
$\pi_{\text{step}}$ vs $\pi_{\text{smooth}}$ | 10 | -85 | -120 | 33
$\pi_{\text{step}}$ vs $\pi_{\text{lin}}$ | -19 | 67 | 75 | 26
$\pi_{\text{step}}$ vs $\pi_{0}$ | 45 | -62 | 92 | 52

Table 2: Comparison between policies for different scaling functions $h$. Each
value is the difference of the average reward computed with the simulation
evaluation. The boldface values are statistically relevant differences, i.e.
with p-value $\leq 0.05$. Figure 4: Average reward of optimal traces in the
logs. The $\mathcal{L}$ column represents the average reward of the entire
test log. The numbers above each column correspond to the number of traces
used to compute the mean.

Focusing on RQ2, in Figure 3 we can notice that DQN usually performs worse
than our method in the simulation evaluation. More precisely, looking at Table
3 we can see that only for $\pi_{0}$ in $\mathcal{L}^{b}_{rare}$ and
$\pi_{\text{step}}$ in $\mathcal{L}^{b}$ the difference with DQN is not
statistically relevant. Instead, $\pi_{\text{lin}}$ and $\pi_{\text{smooth}}$
perform better than the baseline in every log.

The result of the test log analysis is more difficult to interpret. In fact,
in Figure 4, there are only two test logs ($\mathcal{L}^{s}$ and
$\mathcal{L}^{b}$) with traces following the DQN policy, and these traces have
a very low average reward. However this does not directly imply that the
baseline performs poorly, as can be seen from the simulation evaluation.

To understand better the case of the real-world log BPIC2012 we performed the
second type of event log analysis, which evaluates recommendations on ongoing
executions. The average gained reward for every prefix length is shown in
Figure 5. Notice that only at prefix length $11$ we get an estimated gain for
DQN, and this is a negative gain reward. It can also be noticed that for
longer prefixes, DQN results start improving and, for prefixes longer than
$20$, even outperform the proposed method. The negative results observed in
the two analyses could be attributed to the absence of optimal traces
suggested by the DQN agent in the test log. This issue is particularly evident
in the second analysis, where for shorter prefixes, the trace execution
consistently deviated from the recommended actions. We stress that this kind
of evaluation is limited, as it can not evaluate optimal traces which are not
present in the test log. The analyses carried out show that the proposed
pipeline allows us to significantly outperform (on synthetic logs) or in the
worst case compare existing approaches (RQ2).

Finally, concerning RQ3, by looking at Figure 3 it is clear that, with respect
to the size of the log used in training, our method is more robust than the
DQN. Indeed, the performance for $\mathcal{L}^{b}$ and
$\mathcal{L}^{b}_{rare}$ are close to the one for $\mathcal{L}^{s}$ and
$\mathcal{L}^{s}_{rare}$ respectively. On the contrary, the performance of DQN
greatly decreases for logs of small size. This effect is even more evident in
the case of the _rare_ logs, i.e. those with low success rate. Indeed in
$\mathcal{L}^{s}$, DQN has poor but positive performance, whereas in
$\mathcal{L}_{rare}^{s}$ its performance is entirely negative.

This suggests that our method is more robust than the offline DQN baseline
with respect to changes in the log size and success rate (RQ3). The reason is
probably two-fold: (i) deep learning algorithms require more data to provide
good results — especially many examples of successful traces; (ii) the offline
trait of our DQN baseline training does not allow the agent to explore traces
not present in the log, limiting its capability to act successfully in unseen
states.

Figure 5: Estimated average gained reward following the policy from a given
prefix length for the BPIC2012 test event log.

$\pi_{1}$ vs $\pi_{2}$ | $\mathcal{L}^{s}$ | $\mathcal{L}_{rare}^{s}$ | $\mathcal{L}^{b}$ | $\mathcal{L}_{rare}^{b}$
---|---|---|---|---
$\pi_{\text{0}}$ vs $\pi_{\text{DQN}}$ | 336 | 633 | 71 | 35
$\pi_{\text{lin}}$ vs $\pi_{\text{DQN}}$ | 400 | 504 | 54 | 61
$\pi_{\text{step}}$ vs $\pi_{\text{DQN}}$ | 381 | 571 | -21 | 87
$\pi_{\text{smooth}}$ vs $\pi_{\text{DQN}}$ | 371 | 656 | 99 | 54

Table 3: Comparison between our policies and the DQN baseline. Each value is
the difference of the average reward computed with the simulation evaluation.
The boldface values are statistically relevant differences, i.e. with p-value
$\leq 0.05$.

##  6 Related work

The state-of-the-art works related to this paper pertain to two fields:
Prescriptive Process Monitoring and Reinforcement Learning. The section is
hence structured by first presenting the PPM related work and then the RL
state-of-the-art works applied to process mining problems.

Several PPM techniques have been recently proposed in the literature (?).
Focusing on the type of recommended interventions, we can classify existing
work in three groups: (i) the ones that recommend interventions to prevent or
mitigate an undesired outcome (?; ?; ?; ?; ?); (ii) the ones that recommend a
resource allocation (?; ?); (iii) the ones that recommend next activities to
optimize a given KPI (?; ?; ?).

The approach presented in this paper falls under this small third family of
prescriptive process monitoring approaches. The work in (?) discusses how the
most likely behavior does not guarantee to achieve the desired business goal.
As a solution to this problem, they propose and evaluate a prescriptive
business process monitoring technique that recommends next best actions to
optimize a specific KPI, i.e., the time. The work in (?) presents a data-
mining driven concept of recommendation-based business process optimization
supporting adaptive and continuously optimized business processes. The work in
(?) discusses Process-aware Recommender (PAR) systems, in which a
prescriptive-analytics component, in case of executions with a negative
outcome prediction, recommends the next activities that minimize the risk to
complete the process execution with a negative outcome. Similarly to this
approach, the work in (?) takes the perspective of one of the actors of the
process and aim at optimizing a domain-specific KPI of interest for this actor
by leveraging an RL approach. However, in that work, the authors heavily rely
on a manual pre-processing of the event log in order to extract the background
knowledge needed to manually build the MDP. Differently from the works above,
we aim at taing a multi-actor perspective by building the RL pipeline
automatically from the event log using only the KPI of interest and the
activity ownership.

In the literature, only few RL approaches have been proposed for facing
problems in Process Mining. Silvander proposes using Q-Learning with function
approximation via a deep Q network (DQN) for the optimization of business
processes (?). He suggests defining a so called decay rate to reduce the
amount of exploration over time. The work in (?) instead apply a DQN for
Predictive Process Monitoring tasks, by learning to predict next activity and
execution times through the means of RL rather than classical supervised
learning solutions. Huang et al. employ RL for the dynamic optimization of
resource allocation in business process executions (?). Metzger et al. propose
an alarm-based approach to prevent and mitigate an undesired outcome (?). They
use online RL to learn when to trigger proactive process adaptations based on
the reliability of predictions. Although all these works use RL in the process
mining field, none of them use it for recommending the next actions to perform
in order to optimize a certain KPI of interest, as in this work. Finally, some
works have applied RL and Inverse RL approaches to recommend the next actions
on temporal data (?) or on data constrained by temporal constraints (?).

##  7 Conclusion

In this paper we provide an AI based approach that learns, by means of
Reinforcement Learning, an optimal policy only from the observation of
temporally annotated sequences of event data and recommends the best
activities to carry on for optimizing a Key Performance Indicator of interest
in a multi-actor setting. This is achieved first by learning an explicit
Markov Decision Process (MDP) for the specific KPIs from the event log using
clusterization techniques, and then by using RL training to learn the optimal
policy.

The evaluation shows that the usage of scaling for the state-action value
functions $q$ is useful to solve the problem of the over-fitting on the
training data from which the MDP is constructed (RQ1); it also shows that our
approach compares with, and often overcomes, off-policy deep RL approaches
especially when taking into account the log dimension and the popularity of
successful traces in the event log (RQ2 and RQ3). This can open the doors to
the exploitation of white box RL techniques in datasets where only temporal
execution data are available, which can be applied to temporally annotated
data outside the Process Mining area.

## References

  * Branchi et al. 2022 Branchi, S.; Di Francescomarino, C.; Ghidini, C.; Massimo, D.; Ricci, F.; and Ronzani, M.  2022\.  Learning to act: A reinforcement learning approach to recommend the best next activities.  In Business Process Management Forum - BPM 2022, Proc., volume 458 of LNBIP, 137–154.  Springer.
  * Chiorrini et al. 2020 Chiorrini, A.; Diamantini, C.; Mircoli, A.; and Potena, D.  2020\.  A preliminary study on the application of reinforcement learning for predictive process monitoring.  In Process Mining Workshops - ICPM 2020, Revised Selected Papers, volume 406 of LNBIP, 124–135.  Springer.
  * De Giacomo et al. 2019 De Giacomo, G.; Iocchi, L.; Favorito, M.; and Patrizi, F.  2019\.  Foundations for restraining bolts: Reinforcement learning with ltlf/ldlf restraining specifications.  In Proc. of the 29th Int. Conf. on Automated Planning and Scheduling, ICAPS 2018, 128–136.  AAAI Press.
  * de Leoni, Dees, and Reulink 2020 de Leoni, M.; Dees, M.; and Reulink, L.  2020\.  Design and evaluation of a process-aware recommender system based on prescriptive analytics.  In 2nd Int. Conf. on Process Mining (ICPM 2020), 9–16.  IEEE.
  * Dumas 2021 Dumas, M.  2021\.  Constructing digital twins for accurate and reliable what-if business process analysis.  In Proc. of the Int. Workshop on BPM Problems to Solve Before We Die (PROBLEMS 2021), volume 2938 of CEUR Workshop Proceedings, 23–27.  CEUR-WS.org.
  * Fahrenkrog-Petersen et al. 2019 Fahrenkrog-Petersen, S. A.; Tax, N.; Teinemaa, I.; Dumas, M.; de Leoni, M.; Maggi, F. M.; and Weidlich, M.  2019\.  Fire now, fire later: alarm-based systems for prescriptive process monitoring.  arXiv preprint arXiv:1905.09568.
  * Gröger, Schwarz, and Mitschang 2014 Gröger, C.; Schwarz, H.; and Mitschang, B.  2014\.  Prescriptive analytics for recommendation-based business process optimization.  In Int. Conf. on Business Information Systems, 25–37.  Springer.
  * Huang et al. 2011 Huang, Z.; van der Aalst, W.; Lu, X.; and Duan, H.  2011\.  Reinforcement learning based resource allocation in business process management.  Data & Know. Eng. 70(1):127–145.
  * Khaidem et al. 2020 Khaidem, L.; Luca, M.; Yang, F.; Anand, A.; Lepri, B.; and Dong, W.  2020\.  Optimizing transportation dynamics at a city-scale using a reinforcement learning framework.  IEEE Access 8:171528–171541.
  * Kubrak et al. 2022 Kubrak, K.; Milani, F.; Nolte, A.; and Dumas, M.  2022\.  Prescriptive process monitoring: _Quo vadis_?  PeerJ Comput. Sci. 8:e1097.
  * Li et al. 2016 Li, J.; Monroe, W.; Ritter, A.; Jurafsky, D.; Galley, M.; and Gao, J.  2016\.  Deep reinforcement learning for dialogue generation.  In Proc of the 2016 Conference on Empirical Methods in Natural Language Processing, 1192–1202.  ACL.
  * Lloyd 1957 Lloyd, S. P.  1957\.  Least squares quantization in pcm.  Technical Report RR-5497, Bell Lab.
  * Massimo and Ricci 2018 Massimo, D., and Ricci, F.  2018\.  Harnessing a generalised user behaviour model for next-poi recommendation.  In Proc. of the 12th ACM Conf. on Recommender Systems, RecSys 2018, 402–406.  ACM.
  * Metzger et al. 2019 Metzger, A.; Neubauer, A.; Bohn, P.; and Pohl, K.  2019\.  Proactive process adaptation using deep learning ensembles.  In Int. Conf. on Advanced Information Systems Engineering, 547–562.  Springer.
  * Metzger, Kley, and Palm 2020 Metzger, A.; Kley, T.; and Palm, A.  2020\.  Triggering proactive business process adaptations via online reinforcement learning.  In Int. Conf. on Business Process Management, 273–290.  Springer.
  * Micheli and Valentini 2021 Micheli, A., and Valentini, A.  2021\.  Synthesis of search heuristics for temporal planning via reinforcement learning.  In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021,, 11895–11902.  AAAI Press.
  * Park and Song 2019 Park, G., and Song, M.  2019\.  Prediction-based resource allocation using LSTM and minimum cost and maximum flow algorithm.  In Int. Conf. on Process Mining, ICPM 2019, 121–128.  IEEE.
  * Shoush and Dumas 2021 Shoush, M., and Dumas, M.  2021\.  Prescriptive process monitoring under resource constraints: A causal inference approach.  In Process Mining Workshops - ICPM 2021, Revised Sel. Papers, volume 433 of LNBIP, 180–193.  Springer.
  * Silvander 2019 Silvander, J.  2019\.  Business process optimization with reinforcement learning.  In International Symposium on Business Modeling and Software Design, 203–212.  Springer.
  * Silver et al. 2016 Silver, D.; Huang, A.; Maddison, C. J.; Guez, A.; Sifre, L.; van den Driessche, G.; Schrittwieser, J.; Antonoglou, I.; Panneershelvam, V.; Lanctot, M.; Dieleman, S.; Grewe, D.; Nham, J.; Kalchbrenner, N.; Sutskever, I.; Lillicrap, T.; Leach, M.; Kavukcuoglu, K.; Graepel, T.; and Hassabis, D.  2016\.  Mastering the game of go with deep neural networks and tree search.  Nature 529(7587):484–489.
  * Sindhgatta, Ghose, and Dam 2016 Sindhgatta, R.; Ghose, A.; and Dam, H. K.  2016\.  Context-aware analysis of past process executions to aid resource allocation decisions.  In Advanced Information Systems Engineering, 575–589.  Springer.
  * Sutton and Barto 2018 Sutton, R. S., and Barto, A. G.  2018\.  Reinforcement Learning: An Introduction.  The MIT Press, 2nd edition.
  * Teinemaa et al. 2018 Teinemaa, I.; Tax, N.; de Leoni, M.; Dumas, M.; and Maggi, F. M.  2018\.  Alarm-based prescriptive process monitoring.  In Business Process Management Forum, 91–107.  Springer.
  * van der Aalst 2022 van der Aalst, W. M. P.  2022\.  Foundations of process discovery.  In Process Mining Handbook, volume 448 of Lecture Notes in Business Information Processing. Springer.  37–75.
  * van Dongen 2012 van Dongen, B.  2012\.  Bpi challenge 2012.
  * Weinzierl et al. 2020 Weinzierl, S.; Zilker, S.; Stierle, M.; Matzner, M.; and Park, G.  2020\.  From predictive to prescriptive process monitoring: Recommending the next best actions instead of calculating the next most likely events.  In Wirtschaftsinformatik (Zentrale Tracks), 364–368.