11institutetext: Fraunhofer Institute for Applied Information Technology FIT,
Sankt Augustin, Germany 11email:
<EMAIL_ADDRESS>22institutetext: RWTH Aachen University, Aachen, Germany
22email<EMAIL_ADDRESS>
# Control-Flow-Based Querying of Process Executions from Partially Ordered
Event Data

Daniel Schuster 0000-0002-6512-9580 1122 Michael Martini 0000-0001-5376-4424
11 Sebastiaan J. van Zelst 0000-0003-0415-1036 1122 Wil M. P. van der Aalst
0000-0002-0955-6940 1122 Daniel Schuster 1122 0000-0002-6512-9580 Michael
Martini 11 0000-0001-5376-4424 Sebastiaan J. van Zelst 1122
0000-0003-0415-1036
Wil M. P. van der Aalst 1122 0000-0002-0955-6940

###### Abstract

Event logs, as viewed in process mining, contain event data describing the
execution of operational processes. Most process mining techniques take an
event log as input and generate insights about the underlying process by
analyzing the data provided. Consequently, handling large volumes of event
data is essential to apply process mining successfully. Traditionally,
individual process executions are considered sequentially ordered process
activities. However, process executions are increasingly viewed as partially
ordered activities to more accurately reflect process behavior observed in
reality, such as simultaneous execution of activities. Process executions
comprising partially ordered activities may contain more complex activity
patterns than sequence-based process executions. This paper presents a novel
query language to call up process executions from event logs containing
partially ordered activities. The query language allows users to specify
complex ordering relations over activities, i.e., control flow constraints.
Evaluating a query for a given log returns process executions satisfying the
specified constraints. We demonstrate the implementation of the query language
in a process mining tool and evaluate its performance on real-life event logs.

###### Keywords:

Process Mining Process querying Partial orders

##  1 Introduction

Executing operational processes generates large amounts of event data in
enterprise information systems. Analyzing these data provides great
opportunities for operational improvements, for example, reduced cycle times
and increased conformity with reference process models. Therefore, _process
mining_ [17] comprises data-driven techniques to analyze event data to gain
insights into the underlying processes; for example, automatically discovered
process models, conformance statistics, and performance analysis information.
Since service-oriented computing is concerned with orchestrating services to
form dynamic business processes [6], process mining can provide valuable
insights into the actual execution of processes within organizations [16].
These insights can then be used, for example, to define services and
ultimately construct service-oriented architectures. Further, process mining
provides valuable tools for service monitoring.

Most process mining techniques [17] define process executions, termed _traces_
, as a sequence, i.e., a _strict total order_ , of executed activities. In
reality, however, processes can exhibit parallel behavior, i.e., several
branches of a process are executed simultaneously. Consequently, the execution
of individual activities may overlap within a single trace. Thus, traces are
defined by _partially ordered_ executed activities. Considering traces as
partial orders, the complexity of observed control flow patterns, i.e.,
relations among executed activities, increases compared to sequential traces.
Thus, tools are needed that facilitate the handling, filtering, and exploring
of traces containing partially ordered process activities.

This paper introduces a novel query language for querying traces from an event
log containing partially ordered activities. The proposed language allows the
specification of six essential control flow constraints, which can be further
restricted via cardinality constraints and arbitrarily combined via Boolean
operators. The language design is based on standardized terms for control flow
patterns in process mining. We provide a formal specification of the
language’s syntax and semantics to facilitate reuse in other tools. Further,
we present its implementation in the process mining software tool Cortado
[14], which supports partially ordered event data. Query results are
visualized by Cortado using a novel trace variant visualization [13]. Finally,
we evaluate the performance of the query evaluation on real-life, publicly
available event logs.

The remainder of this paper is structured as follows. Sect. 2 presents related
work. Sect. 3 introduces preliminaries. In Sect. 4, we introduce the proposed
query language. We present an exemplary application use case of the query
language in Sect. 5. In Sect. 6, we present an evaluation focusing on
performance aspects of the proposed query language. Finally, Sect. 7 concludes
this paper.

##  2 Related Work

A framework for _process querying_ methods is presented in [10]. In short,
process query methods differ in the input used, for instance, event logs
(e.g., [3, 20]) or process model repositories (e.g., [2, 5]), and the goal or
capabilities of the query method. Overviews of process querying languages can
be found in [8, 9, 10, 19]; the majority of existing methods focuses on
querying process model repositories. Subsequently, we focus on methods that
operate on event logs.

Celonis PQL [18] is a multi-purpose, textual query language that works on
event logs and process models and provides a variety of query options.
However, traces are considered sequentially ordered activities compared to the
proposed query language in this paper. In [3], a query language is proposed
that operates on a single graph, i.e., a RDF, connecting all events in an
event log by user-defined correlations among events. The query language allows
to partition the events by specified constraints and to query paths that start
and end with events fulfilling certain requirements. Compared to our approach,
we do not initially transform the entire event log into a graph structure;
instead, we operate on individual traces composed of partially ordered event
data.

In [4], the authors propose a natural language interface for querying event
data. Similar to [3], a graph based search is used. The approach allows
specifying arbitrary queries like “Who was involved in processing case x” and
“For which cases is the case attribute y greater than z.” However, control
flow constraints over partially ordered event data are not supported, unlike
the query language proposed in this paper, which is designed exclusively for
control flow constraints. In [11], the authors propose an LTL-based query
language to query traces, consisting of sequentially aligned process
activities, fulfilling specified constraints from an event log. In [20], the
authors propose an approach to query trace fragments from various event logs
that are similar to a trace fragment surrounding a selected activity from a
process model using a notion of neighborhood context. Traces are, in this
approach, considered sequentially ordered activities.

In summary, various process querying methods exist, most of them operating
over process model repositories rather than event logs, cf. [8, 9, 10, 19]. In
short, the proposed query language differs in three main points from existing
work.

  1. 1.

First process querying language focusing on traces containing partially
ordered activities (to the best of our knowledge)

  2. 2.

Focus on traces rather than event data as a whole, i.e., executing a query
returns traces satisfying the specified constraints

  3. 3.

Specific focus on control flow patterns, i.e., extensive options for
specifying a wide range of control flow patterns

##  3 Preliminaries

This section introduces notations and concepts used throughout this paper.

We denote the natural numbers by $\mathbb{N}$ and the natural numbers
including $0$ by $\mathbb{N}_{0}$. We simplify by representing timestamps by
positive real numbers denoted by $\mathbb{R}^{+}$. We denote the universe of
activity labels by $\mathcal{L}$, activity instance identifier by
$\mathcal{I}^{A}$, and case identifier by $\mathcal{I}^{C}$. Further, we
denote a missing value by $\bot$.

######  Definition 1 (Activity instances)

An activity instance
$a=(i,c,l,t_{s},t_{c})\in\mathcal{I}^{A}\times\mathcal{I}^{C}\times\mathcal{L}\times\big{(}\mathbb{R}^{+}\cup\\{\bot\\}\big{)}\times\mathbb{R}^{+}$
uniquely identified by $i\in\mathcal{I}^{A}$ represents the execution of an
activity $l\in\mathcal{L}$ that was executed for the process instance
identified by $c\in\mathcal{I}^{C}$. The activity instance’s temporal
information is given by the optional start timestamp
$t_{s}\in\mathbb{R}^{+}\cup\\{\bot\\}$ and the complete timestamp
$t_{c}\in\mathbb{R}^{+}$. If $t_{s}\neq\bot\Rightarrow t_{s}\leq t_{c}$. We
denote the universe of activity instances by $\mathcal{A}$.

Let $a{=}(i,c,l,t_{s},t_{c})\in\mathcal{A}$ be an activity instance, we use
short forms to assess the different components of $a$; we write $a^{i}$,
$a^{c}$, $a^{l}$, $a^{t_{s}}$, and $a^{t_{c}}$.

Table 1: Example of an event log with partially ordered event data ID |  | Timestamp |
---|---|---|---
Activity Instance | Case |  Activity Label | Start | Completion | …
1 | 1 |  credit request received (CRR) | $\bot$ | 16.06.21 12:43:35 | …
2 | 1 |  document check (DC) | 17.06.21 08:32:23 | 18.06.21 12:01:11 | …
3 | 1 |  request info. from applicant (RIP) | 19.06.21 09:34:00 | 22.06.21 09:12:00 | …
4 | 1 |  request info. from third parties (RIT) | 19.06.21 14:54:00 | 25.06.21 08:57:12 | …
5 | 1 |  document check (DC) | $\bot$ | 28.06.21 14:23:59 | …
6 | 1 |  credit assessment (CA) | 30.06.21 13:02:11 | 04.07.21 08:11:32 | …
7 | 1 |  security risk assessment (SRA) | 01.07.21 17:23:11 | 06.07.21 18:51:43 | …
8 | 1 |  property inspection (PI) | $\bot$ | 05.07.21 00:00:00 | …
9 | 1 |  loan-to-value ratio determined (LTV) | $\bot$ | 05.07.21 00:00:00 | …
10 | 1 |  decision made (DM) | $\bot$ | 08.07.21 14:13:18 | …
11 | 2 |  credit request received (CRR) | $\bot$ | 17.06.21 23:21:31 | …
… | … |  … | … | … | …
Figure 1: Ordering of the activity instances within the trace describing case
1. Solid arcs depict the transitive reduction; solid and dotted arcs the
transitive closure.

An event log can be seen as a set of activity instances describing the same
process; Table 1 shows an example. Each row corresponds to an activity
instance describing the execution of an activity. For instance, the first row
describes the execution of the activity “credit request received” executed on
16.06.21 at 12:43:35 for the process instance identified by case-id 1.
Individual process executions within an event log are termed _traces_. Next,
we formally define traces as a partially ordered set of activity instances
belonging to the same case.

######  Definition 2 (Trace)

Let $T{\subseteq}\mathcal{A}$. We call $(T,\prec)$ a trace if:

  1. 1.

$\forall a_{i},a_{j}{\in}T(a_{i}^{c}=a_{j}^{c})$ and

  2. 2.

$\prec\subseteq T{\times}T$ and for arbitrary $a_{i},a_{j}{\in}T$ holds that
$a_{i}{\prec}a_{j}$ iff:

     * •

$a_{i}^{t_{c}}<a_{j}^{t_{s}}$ given that
$a_{i}^{t_{c}},a_{j}^{t_{s}}{\in}\mathbb{R}^{+}$, or

     * •

$a_{i}^{t_{c}}<a_{j}^{t_{c}}$ given that $a_{i}^{t_{c}}{\in}\mathbb{R}^{+}$
and $a_{j}^{t_{s}}{=}\bot$.

We denote the universe of traces by $\mathcal{T}$.

For a trace $(T,\prec){\in}\mathcal{T}$, note that the relation $\prec$ (cf.
2) is the _transitive closure_. We denote the _transitive reduction_ of
$\prec$ by $\prec^{R}$. For $\prec^{R}$ it holds that $\forall
a,b{\in}T\Big{[}a{\prec^{R}}b\leftrightarrow\Big{(}a{\prec}b\land\big{(}{\not\exists}\widetilde{a}{\in}T(a{\prec^{R}}\widetilde{a}\land\widetilde{a}{\prec^{R}}b\big{)}\Big{)}\Big{]}$.
Fig. 1 visualizes the ordering relations of the activity instances of the
trace describing case $1$ (cf. Table 1). Solid arcs show direct relationships
among activity instances. Thus, the solid arcs represent the transitive
reduction. Solid and dotted arcs represent all relations among activity
instances and thus, represent the transitive closure.

Finally, we define notation conventions regarding the existential quantifier.
Let $k{\in}\mathbb{N}$ and $X$ be an arbitrary set, we write $\exists^{=k}$,
$\exists^{\geq k}$, and $\exists^{\leq k}$ to denote that there exist
_exactly_ , _at least_ , and _at most_ $k$ distinct elements in set $X$
satisfying a given formula $P(\dots)$. Below we formally define the three
existential quantifier.

  * •

$\exists^{=k}x_{1},\dots,x_{k}{\in}X\ \big{(}\forall_{1\leq i\leq k}\
P(x_{i})\big{)}\equiv\exists
x_{1},\dots,x_{k}{\in}X\big{[}\big{(}\forall_{1\leq i<j\leq k}\
x_{i}{\neq}x_{j}\big{)}\land\big{(}\forall_{1\leq i\leq k}\
P(x_{i})\big{)}\land\big{(}\forall_{x{\in}X{\setminus}\\{x_{1},\dots,x_{k}\\}}\
\lnot P(x_{i})\big{)}\big{]}$

  * •

$\exists^{\geq k}x_{1},\dots,x_{k}{\in}X\ \big{(}\forall_{1\leq i\leq k}\
P(x_{i})\big{)}\equiv\exists
x_{1},\dots,x_{k}{\in}X\big{[}\big{(}\forall_{1\leq i<j\leq k}\
x_{i}{\neq}x_{j}\big{)}\land\big{(}\forall_{1\leq i\leq k}\
P(x_{i})\big{)}\big{]}$

  * •

$\exists^{\leq k}x_{1},\dots,x_{k}{\in}X\ \big{(}\forall_{1\leq i\leq k}\
P(x_{i})\big{)}\equiv\exists
x_{1},\dots,x_{k}{\in}X\big{[}\big{(}\forall_{1\leq i\leq k}\
P(x_{i})\big{)}\land\\\
\big{(}\forall_{x{\in}X{\setminus}\\{x_{1},\dots,x_{k}\\}}\ \lnot
P(x)\big{)}\big{]}$
Note that $x_{1},\dots,x_{k}$ must not be _different_ elements in the formula
above; it specifies that at most $k$ distinct elements in $X$ exist satisfying
$P(\dots)$.

##  4 Query Language

This section introduces the proposed query language. Sect. 4.1 introduces its
syntax, while Sect. 4.2 defines its semantics. Sect. 4.3 covers the evaluation
of queries. Finally, Sect. 4.4 presents the implementation in a process mining
tool.

###  4.1 Syntax

((’DC’ isC =2) OR ((’DC’ isC =1) AND (’CRR’ isDF ’DC’))) AND (NOT(’DC’ isDF
’DM’))

ANDOR(’DC’ isC =2)AND(’DC’ isC =1)(’CRR’ isDF ’DC’)NOT(’DC’ isDF ’DM’)

Figure 2: Example of a query. Leaves represent individual control flow
constraints (cf. Table 2) that are combined via Boolean operators.

This section introduces the syntax of the proposed query language. In total,
six operators exist, allowing to specify control flow constraints. Table 2
provides an overview of these six operators, three binary, (i.e., isContained
(isC), isStart (isS), and isEnd (isE)), and three unary operators (i.e.,
isDirectlyFollowed (isDF), isEventuallyFollowed (isEF), and isParallel (isP)).
Next to each operator, we list query examples, including the corresponding
operator, and present its semantics in natural language. As the examples show,
each operator can be additionally constrained by a cardinality. We call a
query a leaf query if only one operator is used, for instance, all examples
shown in Table 2 are query leaves. Query leaves can be arbitrarily combined
via Boolean operators, for instance, see Fig. 2. Next, we formally define the
query language’s syntax.

Table 2: Overview of the six control flow constraints and corresponding
examples

Type |  Syntax | Example
---|---|---
Nr. | Query |  Description of semantics
unary |  isContained (isC) | E1 | ’A’ isC |  activity A is contained in the trace
E2 | ’A’ isC $\geq$ 6 |  activity A is contained at least 6 times in the trace
E3 | ALL{’A’,’B’} isS $\geq$ 6 |  activity A and B are both contained at least 6 times each in the trace
isStart (isS) | E4 | ’A’ isS |  there exists a start activity A(a)
E5 | ’A’ isS = 1 |  exactly one start activity of the trace is an A activity(a)
E6 | ANY{’A’,’B’} isC = 1 |  trace starts with exactly one A activity or/and with exactly one B activity(a)
isEnd (isE) | E7 | ’A’ isE |  there exists an end activity A(a)
E8 | ’A’ isE $\geq$ 2 |  at least two end activities of the trace are an A activity(a)
E9 | ALL{’A’,’B’} isE |  trace ends with at least one A and one B activity(a)
binary |  isDirectly Followed (isDF) | E10 | ’A’ isDF ’B’ |  a B activity directly follows _each_ A activity in the trace
E11 | ’A’ isDF ’B’ = 1 |  trace contains exactly one A activity that is directly followed by B
E12 | ’A’ isDF ALL{’B’,’C’} |  every A activity is directly followed by a B and C activity
isEventually Followed (isEF) | E13 | ’A’ isEF ’B’ |  after _each_ A activity in the trace a B activity eventually follows
E14 | ’A’ isEF ’B’ $\geq$ 1 |  trace contains at least one A activity that is eventually followed by B
E15 | ALL{’A’,’B’} isEF ’C’ |  all A and B activities are eventually followed by a C activity
isParallel (isP) | E16 | ’A’ isP ’B’ |  each A activity in the trace is in parallel to some B activity
E17 | ’A’ isP ’B’ $\leq$ 4 |  trace contains at most four A activities that are in parallel to some B activity
E18 | ’A’ isP ANY{’B’,’C’} $\leq$ 2 |  trace contains at most two A activities that are parallel to a B or C activity

  * (a)

Trace may contain arbitrary further start respectively end activities.

######  Definition 3 (Query Syntax)

Let $l_{1},\dots,\allowbreak l_{n-1},\allowbreak l_{n}{\in}\mathcal{L}$ be
activity labels, $k{\in}\mathbb{N}_{0}$, $\square{\in}\\{\leq,\geq,=\\}$,
$\circ{\in}\\{\texttt{isDF},\allowbreak\texttt{isEF},\texttt{isP}\\}$,
$\bullet{\in}\\{\texttt{isC},\allowbreak\texttt{isS},\allowbreak\texttt{isE}\\}$,
and $\triangle{\in}\\{\texttt{ALL},\texttt{ANY}\\}$. We denote the universe of
queries by $\mathcal{Q}$ and recursively define a query $Q{\in}\mathcal{Q}$
below.

Leaf query with an unary operator (without/with cardinality constraint)

  * •

$Q{=}\texttt{'}l_{1}\texttt{'}\;\bullet$ |  $Q{=}\texttt{'}l_{1}\texttt{'}\;\bullet\;\square k$
---|---

  * •

$Q{=}\triangle\texttt{\\{'}l_{1}\texttt{'},\dots,\texttt{'}l_{n-1}\texttt{'\\}}\;\bullet$ |  $Q{=}\triangle\texttt{\\{'}l_{1}\texttt{'},\allowbreak\dots,\allowbreak\texttt{'}l_{n-1}\texttt{'\\}}\;\bullet\;\square k$
---|---

Leaf query with a binary operator (without/with cardinality constraint)

  * •

$Q{=}\texttt{'}l_{1}\texttt{'}\circ\texttt{'}l_{n}\texttt{'}$ |  $Q{=}\texttt{'}l_{1}\texttt{'}\circ\texttt{'}l_{n}\texttt{'}\;\square k$
---|---

  * •

$Q{=}\texttt{}\triangle\texttt{\\{'}l_{1}\texttt{'},\dots,\texttt{'}l_{n-1}\texttt{\\}'}\circ\texttt{'}l_{n}\texttt{'}$ |  $Q{=}\triangle\texttt{\\{'}l_{1}\texttt{'},\dots,\texttt{'}l_{n-1}\texttt{\\}'}\circ\texttt{'}l_{n}\texttt{'}\;\square k$
---|---

  * •

$Q{=}\texttt{}\texttt{'}l_{n}\texttt{'}\circ\triangle\texttt{\\{'}l_{1}\texttt{'},\dots,\allowbreak\texttt{'}l_{n-1}\texttt{\\}'}\texttt{}$ |  $Q{=}\texttt{'}l_{n}\texttt{'}\circ\triangle\texttt{\\{'}l_{1}\texttt{'},\dots,\texttt{'}l_{n-1}\texttt{\\}'}\;\square k$
---|---

Composed query using Boolean operators

  * •

If $Q_{1},Q_{2}{\in}\mathcal{Q}$ are two queries and
$\blacksquare{\in}\\{\texttt{AND},\texttt{OR}\\}$, then
$Q{=}\texttt{(}Q_{1}\blacksquare Q_{2}\texttt{)}$ is a query

  * •

If $Q_{1}{\in}\mathcal{Q}$ is a query, then
$Q{=}\texttt{NOT}\texttt{(}Q_{1}\texttt{)}$ is a query

###  4.2 Semantics

This section introduces the query language’s semantics. Table 2 presents query
examples with corresponding semantics. In short, the unary operators allow to
specify the existence of individual activities within a trace, for example, is
contained (isC), is a start activity (isS), or is an end activity (isE).
Optionally, operators can have cardinality constraints that extend the
existential semantics of unary operators by quantification constraints. Binary
operators allow to specify relationships between activities; for example, two
activities are parallel (isP), directly follow each other (isDF), or
eventually follow each other (isEF). In contrast to unary operators, binary
operators always have to hold globally when no cardinality constraint is
given. For example, E10 (cf. Table 2) specifies that a B activity must
directly follow each A activity, i.e., there is an arc in the transitive
reduction from each A activity to a B activity. In comparison, E11 specifies
that the trace contains precisely one A activity that is directly followed by
a B activity. ALL sets specify that a constraint must be fulfilled for all
activity labels within the set. Analogously, ANY sets specify that the
constraint must be fulfilled at least for one activity. Next, we formally
define the semantics.

######  Definition 4 (Query Semantics)

Let $Q,Q_{1},Q_{2}{\in}\mathcal{Q}$ be queries,
$T^{\prec}{=}(T,\prec)\in\mathcal{T}$ be a trace, and
$l_{1},\dots,l_{n}{\in}\mathcal{L}$ be activity labels. We recursively define
the function $eval:\mathcal{Q}{\times}\mathcal{T}\to\\{true,false\\}$
assigning a Boolean value, i.e., $eval(Q,T^{\prec})$, to query $Q$ and trace
$T^{\prec}$.

Unary operators:

  * •

If $Q{=}\texttt{'}l_{1}\texttt{' isC}\ \square k$, then
$eval\big{(}Q,T^{\prec}\big{)}\Leftrightarrow\\\ \exists^{\square
k}a_{1},\dots,a_{k}{\in}T\big{[}\forall_{1\leq i\leq
k}(a_{i}^{l}{=}l_{1})\big{]}$

  * •

If $Q{=}\texttt{'}l_{1}\texttt{' isS}\ \square k$, then
$eval\big{(}Q,T^{\prec}\big{)}\Leftrightarrow\\\ \exists^{\square
k}a_{1},\dots,a_{k}{\in}T\Big{[}\forall_{1\leq i\leq
k}\Big{(}a_{i}^{l}{=}l_{1}\land\lnot\exists\widetilde{a}{\in}T(\widetilde{a}{\prec}a_{i})\Big{)}\Big{]}$

  * •

If $Q{=}\texttt{'}l_{1}\texttt{' isE}\ \square k$, then
$eval\big{(}Q,T^{\prec}\big{)}\Leftrightarrow\\\ \exists^{\square
k}a_{1},\dots,a_{k}{\in}T\Big{[}\forall_{1\leq i\leq
k}\Big{(}a_{i}^{l}{=}l_{1}\land\lnot\exists\widetilde{a}{\in}T(a_{i}{\prec}\widetilde{a})\Big{)}\Big{]}$

Binary operators:

  * •

If $Q{=}\texttt{'}l_{1}\texttt{' isDF '}l_{2}\texttt{'}$, then
$eval\big{(}Q,T^{\prec}\big{)}\Leftrightarrow$
$\forall
a{\in}T\Big{[}a^{l}{=}l_{1}\rightarrow{\exists}\widetilde{a}{\in}T\left(\widetilde{a}^{l}{=}l_{2}\land
a{\prec^{R}}\widetilde{a}\right)\Big{]}$

  * •

If $Q{=}\texttt{'}l_{1}\texttt{' isDF '}l_{2}\texttt{'}\ \square k$, then
$eval\big{(}Q,T^{\prec}\big{)}\Leftrightarrow$
$\exists^{\square k}a_{1},\dots,a_{k}{\in}T\Big{[}\forall_{1\leq i\leq
k}\Big{(}a_{i}^{l}{=}l_{1}\land\exists\widetilde{a}{\in}T\left(\widetilde{a}^{l}{=}l_{2}\land
a_{i}{\prec^{R}}\widetilde{a}\right)\Big{)}\Big{]}$

  * •

If $Q{=}\texttt{'}l_{1}\texttt{' isDF
ANY\\{'}l_{2}\texttt{',}\dots\texttt{,'}l_{n}\texttt{'\\}}$, then
$eval\big{(}Q,T^{\prec}\big{)}\Leftrightarrow\\\ \forall
a{\in}T\Big{[}a^{l}{=}l_{1}\rightarrow\exists\widetilde{a}{\in}T\Big{(}a{\prec^{R}}\widetilde{a}\land\big{(}\bigvee_{j=2}^{n}\widetilde{a}^{l}=l_{j}\big{)}\Big{)}\Big{]}$

  * •

If $Q{=}\texttt{'}l_{1}\texttt{' isDF
ANY\\{'}l_{2}\texttt{',}\dots\texttt{,'}l_{n}\texttt{'\\}}\ \square k$, then
$eval\big{(}Q,T^{\prec}\big{)}\Leftrightarrow\\\ \exists^{\square
k}a_{1},\dots,a_{k}{\in}T\Big{[}\forall_{1\leq i\leq
k}\Big{(}a_{i}^{l}{=}l_{1}\land\exists\widetilde{a}{\in}T\big{(}a_{i}{\prec^{R}}\widetilde{a}\land\bigvee_{j=2}^{n}(\widetilde{a}^{l}{=}l_{j})\big{)}\Big{)}\Big{]}$

  * •

If $Q{=}\texttt{'}l_{1}\texttt{' isDF
ALL\\{'}l_{2}\texttt{',}\dots\texttt{,'}l_{n}\texttt{'\\}}\ \square k$, then
$eval\big{(}Q,T^{\prec}\big{)}\Leftrightarrow\\\ \exists^{\square
k}a_{1},\dots,a_{k}{\in}T\Big{[}\forall_{1\leq i\leq
k}\Big{(}a_{i}^{l}{=}l_{1}\land\allowbreak\exists\widetilde{a}_{2},\dots,\widetilde{a}_{n}{\in}T\allowbreak\big{(}\bigwedge_{j=2}^{n}\allowbreak\big{(}a_{i}{\prec^{R}}\widetilde{a}_{j}\land\widetilde{a}_{j}^{l}{=}l_{j}\big{)}\big{)}\Big{)}\Big{]}$

  * •

If $Q{=}\texttt{'}l_{1}\texttt{' isEF '}l_{2}\texttt{'}$, then
$eval\big{(}Q,T^{\prec}\big{)}\Leftrightarrow$
$\forall
a{\in}T\Big{[}a^{l}{=}l_{1}\rightarrow{\exists}\widetilde{a}{\in}T\left(\widetilde{a}^{l}{=}l_{2}\land
a{\prec}\widetilde{a}\right)\Big{]}$

  * •

If $Q{=}\texttt{'}l_{1}\texttt{' iEF '}l_{2}\texttt{'}\ \square k$, then
$eval\big{(}Q,T^{\prec}\big{)}\Leftrightarrow$
$\exists^{\square k}a_{1},\dots,a_{k}{\in}T\Big{[}\forall_{1\leq i\leq
k}\Big{(}a_{i}^{l}{=}l_{1}\land\exists\widetilde{a}{\in}T\left(\widetilde{a}^{l}{=}l_{2}\land
a_{i}{\prec}\widetilde{a}\right)\Big{)}\Big{]}$

  * •

If $Q{=}\texttt{'}l_{1}\texttt{' isEF
ANY\\{'}l_{2}\texttt{',}\dots\texttt{,'}l_{n}\texttt{'\\}}$, then
$eval\big{(}Q,T^{\prec}\big{)}\Leftrightarrow\\\ \forall
a{\in}T\Big{[}a^{l}{=}l_{1}\rightarrow\exists\widetilde{a}{\in}T\Big{(}a{\prec}\widetilde{a}\land\big{(}\bigvee_{i=2}^{n}\widetilde{a}^{l}=l_{i}\big{)}\Big{)}\Big{]}$

  * •

If $Q{=}\texttt{'}l_{1}\texttt{' isEF
ANY\\{'}l_{2}\texttt{',}\dots\texttt{,'}l_{n}\texttt{'\\}}\ \square k$, then
$eval\big{(}Q,T^{\prec}\big{)}\Leftrightarrow\\\ \exists^{\square
k}a_{1},\dots,a_{k}{\in}T\Big{[}\forall_{1\leq i\leq
k}\Big{(}a_{i}^{l}{=}l_{1}\land\exists\widetilde{a}{\in}T\big{(}a_{i}{\prec}\widetilde{a}\land\allowbreak\big{(}\bigvee_{j=2}^{n}\allowbreak\widetilde{a}^{l}{=}l_{j}\big{)}\big{)}\Big{)}\Big{]}$

  * •

If $Q{=}\texttt{'}l_{1}\texttt{' isEF
ALL\\{'}l_{2}\texttt{',}\dots\texttt{,'}l_{n}\texttt{'\\}}\ \square k$, then
$eval\big{(}Q,T^{\prec}\big{)}\Leftrightarrow\\\ \exists^{\square
k}a_{1},\dots,a_{k}{\in}T\Big{[}\forall_{1\leq i\leq
k}\Big{(}a_{i}^{l}{=}l_{1}\land\exists\widetilde{a}_{2},\dots,\widetilde{a}_{n}{\in}T\big{(}\bigwedge_{j=2}^{n}\allowbreak(a_{i}{\prec}\widetilde{a}_{j}\land\widetilde{a}_{j}^{l}{=}l_{j})\big{)}\Big{)}\Big{]}$

  * •

If $Q{=}\texttt{'}l_{1}\texttt{' isP '}l_{2}\texttt{'}$, then
$eval\big{(}Q,T^{\prec}\big{)}\Leftrightarrow$
$\forall
a{\in}T\Big{[}a^{l}{=}l_{1}\rightarrow{\exists}\widetilde{a}{\in}T\left(\widetilde{a}^{l}{=}l_{2}\land
a{\not\prec}\widetilde{a}\land\widetilde{a}{\not\prec}a\right)\Big{]}$

  * •

If $Q{=}\texttt{'}l_{1}\texttt{' isP '}l_{2}\texttt{'}\ \square k$, then
$eval\big{(}Q,T^{\prec}\big{)}\Leftrightarrow$
$\exists^{\square k}a_{1},\dots,a_{k}{\in}T\Big{[}\forall_{1\leq i\leq
k}\left(a_{i}^{l}{=}l_{1}\land{\exists}\widetilde{a}{\in}T\left(\widetilde{a}^{l}{=}l_{2}\land
a_{i}{\not\prec}\widetilde{a}\land\widetilde{a}{\not\prec}a_{i}\right)\right)\Big{]}$

  * •

If $Q{=}\texttt{'}l_{1}\texttt{' isP
ANY\\{'}l_{2}\texttt{',}\dots\texttt{,'}l_{n}\texttt{'\\}}$, then
$eval\big{(}Q,T^{\prec}\big{)}\Leftrightarrow$
$\forall
a{\in}T\Big{[}a^{l}{=}l_{1}\rightarrow\exists\widetilde{a}{\in}T\Big{(}a{\not\prec}\widetilde{a}\land\widetilde{a}{\not\prec}a\land\big{(}\bigvee_{j=2}^{n}\widetilde{a}^{l}=l_{j}\big{)}\Big{)}\Big{]}$

  * •

If $Q{=}\texttt{'}l_{1}\texttt{' isP
ANY\\{'}l_{2}\texttt{',}\dots\texttt{,'}l_{n}\texttt{'\\}}\ \square k$, then
$eval\big{(}Q,T^{\prec}\big{)}\Leftrightarrow$
$\exists^{\square k}a_{1},\dots,a_{k}{\in}T\Big{[}\forall_{1\leq i\leq
k}\Big{(}a_{i}^{l}{=}l_{1}\land\exists\widetilde{a}{\in}T\big{(}a_{i}{\not\prec}\widetilde{a}\land\widetilde{a}{\not\prec}a_{i}\land\big{(}\bigvee_{j=2}^{n}\widetilde{a}^{l}{=}l_{j}\big{)}\big{)}\Big{)}\Big{]}$

  * •

If $Q{=}\texttt{'}l_{1}\texttt{' isP
ALL\\{'}l_{2}\texttt{',}\dots\texttt{,'}l_{n}\texttt{'\\}}\ \square k$, then
$eval\big{(}Q,T^{\prec}\big{)}\Leftrightarrow$
$\exists^{\square k}a_{1},\dots,a_{k}{\in}T\Big{[}\forall_{1\leq i\leq
k}\Big{(}a_{i}^{l}{=}l_{1}\land\exists\widetilde{a}_{2},\dots,\widetilde{a}_{n}{\in}T\big{(}\bigwedge_{j=2}^{n}\allowbreak(a_{i}{\not\prec}\widetilde{a}_{j}\land\allowbreak\widetilde{a}_{j}{\not\prec}a_{i}\land\allowbreak\widetilde{a}_{j}^{l}{=}l_{j})\big{)}\Big{)}\Big{]}$

Boolean operators:

  * •

If $Q{=}\texttt{NOT(}Q_{1}\texttt{)}$, then
$eval\big{(}Q,T^{\prec}\big{)}\Leftrightarrow\lnot
eval\big{(}Q_{1},T^{\prec}\big{)}$

  * •

If $Q{=}\texttt{(}Q_{1}\texttt{ OR }Q_{2}\texttt{)}$, then
$eval\big{(}Q,T^{\prec}\big{)}\Leftrightarrow
eval\big{(}Q_{1},T^{\prec}\big{)}\lor eval\big{(}Q_{2},T^{\prec}\big{)}$

  * •

If $Q{=}\texttt{(}Q_{1}\texttt{ AND }Q_{2}\texttt{)}$, then
$eval\big{(}Q,T^{\prec}\big{)}\Leftrightarrow
eval\big{(}Q_{1},T^{\prec}\big{)}\land eval\big{(}Q_{2},T^{\prec}\big{)}$

Note that 4 does not cover all queries constructible using the syntax in 3.
However, any query can be rewritten into a _logically equivalent_ one covered
by 3. We call queries $Q_{1},Q_{2}{\in}\mathcal{Q}$ logically equivalent,
denoted $Q_{1}{\equiv}Q_{2}$, iff $\forall
T^{\prec}{\in}\mathcal{A}^{*}\big{(}eval(Q_{1},T^{\prec})\Leftrightarrow
eval(Q_{2},T^{\prec})\big{)}$. Below, we list query rewriting rules.

  * •

$\texttt{'}l_{1}\texttt{'}{\bullet}\equiv\texttt{'}l_{1}\texttt{'}{\bullet}{\geq}\texttt{1}$

  * •

$\texttt{ANY\\{'}l_{1}\texttt{',}\dots\texttt{,'}l_{n}\texttt{'\\}}{\bullet}\equiv\texttt{('}l_{1}\texttt{'}{\bullet}\texttt{)
OR}\dots\texttt{OR ('}l_{n}\texttt{'}{\bullet}\texttt{)}$

  * •

$\texttt{ALL\\{'}l_{1}\texttt{',}\dots\texttt{,'}l_{n}\texttt{'\\}}{\bullet}\equiv\texttt{('}l_{1}\texttt{'}{\bullet}\texttt{)
AND}\dots\texttt{AND ('}l_{n}\texttt{'}{\bullet}\texttt{)}$

  * •

$\texttt{ANY\\{'}l_{1}\texttt{',}\dots\texttt{,'}l_{n}\texttt{'\\}}{\bullet}\
\square\texttt{k}\equiv\texttt{('}l_{1}\texttt{'}{\bullet}{\square}\texttt{k}\texttt{)
OR}\dots\texttt{OR ('}l_{n}\texttt{'}{\bullet}\ \square\texttt{k}\texttt{)}$

  * •

$\texttt{ALL\\{'}l_{1}\texttt{',}\dots\texttt{,'}l_{n}\texttt{'\\}}{\bullet}\
\square\texttt{k}\equiv\texttt{('}l_{1}\texttt{'}{\bullet}\
\square\texttt{k}\texttt{) AND}\dots\texttt{AND ('}l_{n}\texttt{'}{\bullet}\
\square\texttt{k}\texttt{)}$

  * •

$\texttt{ANY\\{'}l_{1}\texttt{',}\dots\texttt{,'}l_{n-1}\texttt{'\\}}{\circ}\texttt{'}l_{n}\texttt{'}\equiv\texttt{('}l_{1}\texttt{'}{\circ}\texttt{'}l_{n}\texttt{')
OR}\dots\texttt{OR ('}l_{n-1}\texttt{'}{\circ}\texttt{'}l_{n}\texttt{')}$

  * •

$\texttt{ALL\\{'}l_{1}\texttt{',}\dots\texttt{,'}l_{n-1}\texttt{'\\}}{\circ}\texttt{'}l_{n}\texttt{'}\equiv\texttt{('}l_{1}\texttt{'}{\circ}\texttt{'}l_{n}\texttt{')
AND}\dots\texttt{AND ('}l_{n-1}\texttt{'}{\circ}\texttt{'}l_{n}\texttt{')}$

  * •

$\texttt{ANY\\{'}l_{1}\texttt{',}\dots\texttt{,'}l_{n-1}\texttt{'\\}}{\circ}\texttt{'}l_{n}\texttt{'}\square\texttt{k}\equiv\texttt{('}l_{1}\texttt{'}{\circ}\texttt{'}l_{n}\texttt{'}\square\texttt{k}\texttt{)
OR}\dots\texttt{OR
('}l_{n-1}\texttt{'}{\circ}\texttt{'}l_{n}\texttt{'}\square\texttt{k}\texttt{)}$

  * •

$\texttt{ALL\\{'}l_{1}\texttt{',}\dots\texttt{,'}l_{n-1}\texttt{'\\}}{\circ}\texttt{'}l_{n}\texttt{'}\square\texttt{k}\equiv\texttt{('}l_{1}\texttt{'}{\circ}\texttt{'}l_{n}\texttt{'}\square\texttt{k}\texttt{)
AND }\dots\texttt{ AND}\\\
\texttt{('}l_{n-1}\texttt{'}{\circ}\texttt{'}l_{n}\texttt{'}\square\texttt{k}\texttt{)}$

  * •

$\texttt{'}l_{1}\texttt{'}{\circ}\texttt{ALL\\{'}l_{2}\texttt{',}\dots\texttt{,'}l_{n}\texttt{'\\}}\equiv\texttt{('}l_{1}\texttt{'}{\circ}\texttt{'}l_{2}\texttt{')
AND}\dots\texttt{AND ('}l_{1}\texttt{'}{\circ}\texttt{'}l_{n}\texttt{')}$

Note that according to 4, the following queries are _not_ logically
equivalent. Thus, ANY and ALL sets are not syntactic sugar.

  * •

$\texttt{'}l_{1}\texttt{'}{\circ}\texttt{ANY\\{'}l_{2}\texttt{',}\dots\texttt{,'}l_{n}\texttt{'\\}}\not\equiv\texttt{('}l_{1}\texttt{'}{\circ}\texttt{'}l_{2}\texttt{')
OR}\dots\texttt{OR ('}l_{1}\texttt{'}{\circ}\texttt{'}l_{n}\texttt{')}$

  * •

$\texttt{'}l_{1}\texttt{'}{\circ}\texttt{ANY\\{'}l_{2}\texttt{',}\dots\texttt{,'}l_{n}\texttt{'\\}}\
\square\texttt{k}\not\equiv\texttt{('}l_{1}\texttt{'}{\circ}\texttt{'}l_{2}\texttt{'}\square\texttt{k)
OR}\dots\texttt{OR ('}l_{1}\texttt{'}{\circ}\texttt{'}l_{n}\texttt{'}\
\square\texttt{k)}$

  * •

$\texttt{'}l_{1}\texttt{'}{\circ}\texttt{ALL\\{'}l_{2}\texttt{',}\dots\texttt{,'}l_{n}\texttt{'\\}}\
\square\texttt{k}\not\equiv\texttt{('}l_{1}\texttt{'}{\circ}\texttt{'}l_{2}\texttt{'}\
\square\texttt{k) AND}\dots\texttt{AND
('}l_{1}\texttt{'}{\circ}\texttt{'}l_{n}\texttt{'}\ \square\texttt{k)}$

For example, consider E18 in Table 2. The query states that there exist at
most two A activities that are in parallel to B or C activities. Thus, a trace
containing four A activities, two parallel to an arbitrary number (greater
than zero) of B activities, and two parallel to C activities, does not fulfill
query E18. However, the described trace fulfills the query $Q=\texttt{('A' isP
'B'}\leq\texttt{2)}\texttt{ OR}\allowbreak\texttt{('A' isP
'C'}\leq\texttt{2)}$; hence, $E18=\texttt{'A' isP
ANY\\{'B','C'\\}}\leq\texttt{2}\not\equiv Q$.

###  4.3 Evaluating Queries

This section briefly discusses our approach to query evaluation. As shown in
Fig. 2, queries represent trees. Since each leaf represents a query, we
evaluate the queries composed of Boolean operators bottom-up. First, the
leaves are evaluated on a given trace, resulting in Boolean values per leaf.
Then, bottom-up, the given Boolean operators are applied recursively.

In many cases, however, a complete query evaluation is not needed to determine
its overall Boolean value for a given trace. For instance, if one leaf of a
logical AND parent evaluates to false, the other leaves do not need to be
further evaluated for the given trace. Similar applies to the logical OR.
Reconsider the query given in Fig. 2 and the trace depicted in Fig. 1. The
query consists of four leaves; however, only two must be evaluated. Following
a depth-first traversing strategy, we first evaluate the leaf (’DC’ isC =2)
satisfied by the given trace. Thus, we do not need to evaluate the right
subtree of the OR, i.e., leaves (’DC’ isC =1) and (’CRR’ isDF ’DC’). Finally,
we evaluate the leave (’DC’ isDF ’DM’). In short, by evaluating only two
leaves, we can evaluate the entire query.

###  4.4 Implementation

This section briefly demonstrates the implementation of the proposed query
language in the process mining tool Cortado [14]111Available at
https://cortado.fit.fraunhofer.de/. We refer to [14] for an introduction to
Cortado’s architecture and a feature overview.

Figure 3: Excerpt from a screenshot of Cortado showing a query editor (bottom
right), a trace variant explorer visualizing the matching trace variants of
the query, and a tabular overview of activities from the event log

Fig. 3 depicts a screenshot of Cortado. The shown chevron-based visualizations
represent _trace variants_ 222A trace variant summarizes traces that share
identical ordering relationships among the contained activities. from the
loaded event log that satisfies the displayed query. We refer to [13] for an
introduction to the trace variant visualization. As shown in Fig. 3, the query
editor offers syntax highlighting; colors of the activity labels in the query
editor correspond to the colors used in the variant explorer to improve
usability. Executing a query results in an updated list of trace variants
satisfying the query. In Fig. 3, the numbers at the top next to the blue
filter icon indicate that 109 out of 3,830 trace variants satisfy the
displayed query. In the backend, we use ANTLR [7] for generating a parser for
the query language. The language’s design ensures that every valid query, when
parsed with ANTLR, corresponds to a single parse tree that can be transformed
into a unique query tree (cf. Fig. 2).

##  5 Application Scenario Example

Figure 4: Example of an application scenario of the proposed query language,
i.e., trace variant selection in the context of incremental process discovery

This section presents an exemplary application scenario of the proposed query
language. _Process discovery_ is concerned with learning a process model from
an event log. Conventional discovery approaches [1] are fully automated, i.e.,
an event log is provided as input and the discovery algorithm returns a
process model describing the event data provided. Since automated process
discovery algorithms often return process models of low quality,
_incremental/interactive process discovery_ approaches have emerged [15] to
additionally utilize domain knowledge next to event data. Incremental process
discovery allows users to gradually add selected traces to a process model
that is considered under construction. By building a process model gradually,
users can control the discovery phase and intervene as needed, for example, by
selecting different traces or making manual changes to the model. In short,
gradually selecting traces from event data is the major form of interaction in
incremental process discovery, cf. Fig. 4.

With event logs containing numerous trace variants, user assistance in
exploring, finding, and selecting trace variants is critical for the success
of incremental process discovery. For instance, the log used in Fig. 3
contains 3,830 trace variants. Manual visual evaluation of all these variants
is inappropriate. In such a scenario, the proposed query language is a
valuable tool for users to cope with the variety, complexity, and amount of
trace variants. As most process discovery approaches [1], including
incremental ones, focus on learning the control flow of activities, a
specialized query language focusing on control flow constraints is a valuable
tool. To this end, we implemented the query language in Cortado, a tool for
incremental process discovery, cf. Fig. 4.

##  6 Evaluation

This section presents an evaluation focusing on performance aspects of the
query language. Sect. 6.1 presents the experimental setup and Sect. 6.2 the
results.

###  6.1 Experimental Setup

We used four publicly available, real-life event logs, cf. Table 3. For each
log, we automatically generated queries from which we pre-selected 1,000 such
that no finally selected query is satisfied by all or by no trace in the
corresponding log. With this approach, we have attempted to filter out trivial
queries to evaluate. We measured performance-related statistics given the
1,000 queries per log.

[b]  Event Log #Traces #Trace Variants(a) BPI Challenge 2012(b) $13,087$
$3,830$ BPI Challenge 2017(d) $31,509$ 5,937 BPI Challenge 2020, Prepaid
Travel Cost log(d) $2,099$ 213 Road Traffic Fine Management (RTFM)(e)
$150,370$ 350

Table 3: Statistics about the event logs used

  * (a)

Based on the variant definition presented in [13]

  * (b)

https://doi.org/10.4121/uuid:3926db30-f712-4394-aebc-75976070e91f

  * (d)

https://doi.org/10.4121/uuid:5f3067df-f10b-45da-b98b-86ae4c7a310b

  * (d)

https://doi.org/10.4121/uuid:52fb97d4-4588-43c9-9d04-3604d4613b51

  * (e)

https://doi.org/10.4121/uuid:270fd440-1057-4fb9-89a9-b699b47990f5

###  6.2 Results

(a) BPI Challenge 2012

(b) BPI Challenge 2017

(c) BPI Challenge 2020

(d) RTFM

Figure 5: Query evaluation time. Since the queries are applied to all traces,
they are ordered by the median number of leaves evaluated per trace

(a) BPI Challenge 2012

(b) BPI Challenge 2017

(c) BPI Challenge 2020

(d) RTFM

Figure 6: Query evaluation time distribution

Each query is applied to all traces from the given event log. Since not all
leaves of a query have to be evaluated, cf. Sect. 4.3, the number of leaves
evaluated may differ per trace. Thus, the actual trace determines how many
leaves of a given query must be evaluated. Fig. 5 shows the runtime (in
seconds) of the queries per event log for the median number of leaf nodes that
were evaluated. Thus, each boxplot is made up of 1,000 data points, i.e.,
1,000 queries each evaluated on all traces from the given log. Across all four
event logs, we clearly observe a linear trend of increasing runtime the more
query leaves are evaluated.

Fig. 6 depicts the distribution of queries according to their evaluation time.
Further, we can see the proportion of leaves evaluated at the median. As
before, each plot contains 1,000 data points, i.e., 1,000 queries. Similar to
Fig. 5, we observe that the number of evaluated leaves is the primary driver
of increased evaluation time. The observed behavior is similar for the
different logs.

(a) BPI Challenge 2012

(b) BPI Challenge 2017

(c) BPI Challenge 2020

(d) RTFM

Figure 7: Impact of early termination on the query evaluation time

Fig. 7 shows the impact of early termination, as introduced in Sect. 4.3. Note
that in the previous plots, i.e., Fig. 5 and Fig. 6, early termination was
always used. We clearly see from the plots in Fig. 7 that early termination
has a significant impact on the evaluation time of a query across all used
event logs. In conclusion, the results shown in this section indicate that the
time required to evaluate queries increases linearly with the number of leaves
evaluated.

##  7 Conclusion

We proposed a novel query language that can call up traces from event logs
containing partially ordered event data. The core of the language is the
control flow constraints, allowing users to specify complex ordering
relationships over executed activities. We formally defined the query
language’s syntax and semantics. Further, we showed its implementation in the
tool Cortado. We presented one potential application scenario of the language,
i.e., the trace selection within incremental process discovery. In short, the
proposed query language facilitates handling large event logs containing
numerous traces consisting of partially ordered activities. For future work,
we plan to conduct user studies exploring the query language’s ease of use
[12]. Further, we plan to extend the language with a graphical editor allowing
query specification in a no-code environment.

## References

  * [1] Augusto, A., Conforti, R., Dumas, M., La Rosa, M., Maggi, F.M., Marrella, A., Mecella, M., Soo, A.: Automated discovery of process models from event logs: Review and benchmark. IEEE Transactions on Knowledge and Data Engineering 31(4), 686–705 (2019). https://doi.org/10.1109/TKDE.2018.2841877
  * [2] Beeri, C., Eyal, A., Kamenkovich, S., Milo, T.: Querying business processes with bp-ql. Information Systems 33(6), 477–507 (2008). https://doi.org/10.1016/j.is.2008.02.005
  * [3] Beheshti, S.M.R., Benatallah, B., Motahari-Nezhad, H.R., Sakr, S.: A query language for analyzing business processes execution. In: Rinderle-Ma, S., Toumani, F., Wolf, K. (eds.) Business Process Management, Lecture Notes in Computer Science, vol. 6896, pp. 281–297. Springer (2011). https://doi.org/10.1007/978-3-642-23059-2_22
  * [4] Kobeissi, M., Assy, N., Gaaloul, W., Defude, B., Haidar, B.: An intent-based natural language interface for querying process execution data. In: 2021 3rd International Conference on Process Mining (ICPM). pp. 152–159. IEEE (2021). https://doi.org/10.1109/ICPM53251.2021.9576850
  * [5] Markovic, I., Costa Pereira, A., de Francisco, D., Muñoz, H.: Querying in business process modeling. In: Di Nitto, E., Ripeanu, M. (eds.) Service-Oriented Computing - ICSOC 2007 Workshops, Lecture Notes in Computer Science, vol. 4907, pp. 234–245. Springer (2009). https://doi.org/10.1007/978-3-540-93851-4_23
  * [6] Papazoglou, M.P., Traverso, P., Dustdar, S., Leymann, F.: Service-oriented computing: State of the art and research challenges. Computer 40(11), 38–45 (2007). https://doi.org/10.1109/MC.2007.400
  * [7] Parr, T.J., Quong, R.W.: Antlr: A predicated-ll(k) parser generator. Software: Practice and Experience 25(7), 789–810 (1995). https://doi.org/10.1002/spe.4380250705
  * [8] Polyvyanyy, A.: Business process querying. In: Sakr, S., Zomaya, A.Y. (eds.) Encyclopedia of Big Data Technologies, pp. 1–9. Springer, Cham (2019). https://doi.org/10.1007/978-3-319-77525-8_108
  * [9] Polyvyanyy, A.: Process Querying Methods. Springer (2022). https://doi.org/10.1007/978-3-030-92875-9
  * [10] Polyvyanyy, A., Ouyang, C., Barros, A., van der Aalst, W.M.P.: Process querying: Enabling business intelligence through query-based process analytics. Decision Support Systems 100, 41–56 (2017). https://doi.org/10.1016/j.dss.2017.04.011
  * [11] Räim, M., Di Ciccio, C., Maggi, F.M., Mecella, M., Mendling, J.: Log-based understanding of business processes through temporal logic query checking. In: Meersman, R., Panetto, H., Dillon, T., Missikoff, M., Liu, L., Pastor, O., Cuzzocrea, A., Sellis, T. (eds.) On the Move to Meaningful Internet Systems: OTM 2014 Conferences, Lecture Notes in Computer Science, vol. 8841, pp. 75–92. Springer (2014). https://doi.org/10.1007/978-3-662-45563-0_5
  * [12] Reisner, P.: Human factors studies of database query languages: A survey and assessment. ACM Computing Surveys 13(1), 13–31 (1981). https://doi.org/10.1145/356835.356837
  * [13] Schuster, D., Schade, L., van Zelst, S.J., van der Aalst, W.M.P.: Visualizing trace variants from partially ordered event data. In: Munoz-Gama, J., Lu, X. (eds.) Process Mining Workshops, Lecture Notes in Business Information Processing, vol. 433, pp. 34–46. Springer (2022). https://doi.org/10.1007/978-3-030-98581-3_3
  * [14] Schuster, D., van Zelst, S.J., van der Aalst, W.M.P.: Cortado—an interactive tool for data-driven process discovery and modeling. In: Buchs, D., Carmona, J. (eds.) Application and Theory of Petri Nets and Concurrency, Lecture Notes in Computer Science, vol. 12734, pp. 465–475. Springer (2021). https://doi.org/10.1007/978-3-030-76983-3_23
  * [15] Schuster, D., van Zelst, S.J., van der Aalst, W.M.P.: Utilizing domain knowledge in data-driven process discovery: A literature review. Computers in Industry 137 (2022). https://doi.org/10.1016/j.compind.2022.103612
  * [16] van der Aalst, W.M.P.: Service mining: Using process mining to discover, check, and improve service behavior. IEEE Transactions on Services Computing 6(4), 525–535 (2013). https://doi.org/10.1109/TSC.2012.25
  * [17] van der Aalst, W.M.P.: Process Mining: Data Science in Action. Springer (2016). https://doi.org/10.1007/978-3-662-49851-4
  * [18] Vogelgesang, T., Ambrosy, J., Becher, D., Seilbeck, R., Geyer-Klingeberg, J., Klenk, M.: Celonis pql: A query language for process mining. In: Polyvyanyy, A. (ed.) Process Querying Methods, pp. 377–408. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-92875-9_13
  * [19] Wang, J., Jin, T., Wong, R.K., Wen, L.: Querying business process model repositories. World Wide Web 17(3), 427–454 (2014). https://doi.org/10.1007/s11280-013-0210-z
  * [20] Yongsiriwit, K., Chan, N.N., Gaaloul, W.: Log-based process fragment querying to support process design. In: 2015 48th Hawaii International Conference on System Sciences. pp. 4109–4119. IEEE (2015). https://doi.org/10.1109/HICSS.2015.493


# Point Cloud Generation using Transformer Encoders and Normalising Flows

Benno Käch
Deutsches Elektronen-Synchrotron DESY
Germany
<EMAIL_ADDRESS>
Dirk Krücker
Deutsches Elektronen-Synchrotron DESY
Germany
<EMAIL_ADDRESS>
&Isabell-A. Melzer-Pellmann
Deutsches Elektronen-Synchrotron DESY
Germany
<EMAIL_ADDRESS>

###### Abstract

Data generation based on Machine Learning has become a major research topic in
particle physics. This is due to the current Monte Carlo simulation approach
being computationally challenging for future colliders, which will have a
significantly higher luminosity. The generation of collider data is similar to
point cloud generation, but arguably more difficult as there are complex
correlations between the points which need to be modelled correctly. A
refinement model consisting of normalising flows and transformer encoders is
presented. The normalising flow output is corrected by a transformer encoder,
which is adversarially trained against another transformer encoder
discriminator/critic. The model reaches state-of-the-art performance while
yielding a stable training.

##  1 Introduction

High Energy Physics (HEP) has benefited from the advances in Machine Learning
(ML) since the analysis of HEP data is a high-dimensional multivariate
problem. There have been multiple applications of ML to HEP [1], although most
of them were in the supervised approach to ML. In HEP, detailed simulations of
the physical processes, which almost perfectly describe the details of the
experimental measurement, are commonly available. These Monte Carlo
simulations (MC) provide labeled data and are needed in large numbers to cover
the extreme areas of the physical phase space. The simulations for the CMS
detector at the Large Hadron Collider, for example, require about $50\%$ [2]
of the current CMS computing budget. An even larger number of simulations will
be needed for the coming high-luminosity phase of the LHC [3]. Therefore,
generative modelling with Deep Learning (DL) has sparked great interest in the
HEP community.

In this study, the generation of jets is investigated using Normalising Flows
(NFs) [4; 5; 6; 7] and an adversarially trained refinement setup consisting of
two transformer [8; 9] encoder networks. The Jets in the dataset are described
as point clouds in momentum space
$\\{(\eta^{(i)},\phi^{(i)},p_{T}^{(i)})_{i\leq 30}\\}$, which motivates the
use of transformers as attention is permutation invariant. One of the
transformer encoders acts as a refinement model to enhance the NF output. The
use of NFs is motivated by their stable convergence thanks to Maximum
Likelihood training, however NFs often struggle to model all correlations
present in the training data correctly and are not particularly well suited
for point clouds. The use of the refinement network enhances the performance
in the modeling of the correlations significantly.

This extended abstract first gives an overview over the models used, then the
different training paradigms are briefly discussed. Finally, the results for
different models are presented and compared.

##  2 Dataset

In this first study, the _JetNet_ [10] top quark dataset [11] is used. We
restrict ourselves to top quark jets due to their more complex sub-structure.
The dataset contains $\sim 180,000$ samples and and a 70/30 train/test split
is applied. The data consists of top quark jets with an energy of about 1 TeV,
with each jet containing up to 30 particles. These jet constituents are
considered to be massless and can therefore be described by their 3-momenta or
equivalently by transverse momentum $p_{T}$, pseudorapidity $\eta$, and
azimuth angle $\phi$. In the _JetNet_ dataset, these variables are given
relative to the jet momentum:
$\eta_{i}^{rel}\coloneqq\eta_{i}^{particle}\\!-\eta^{jet}$,
$\,\phi_{i}^{rel}\coloneqq(\phi_{i}^{particle}\\!-\phi^{jet})\bmod 2\pi$, and
$\,p_{T,i}^{rel}\coloneqq p_{T.i}^{particle}\\!/p_{T}^{jet}$ , where $i$ runs
over the particles in a jet.

The invariant mass $m_{jet}$ of a jet is an essential high-level feature
containing important physics information. It is a global variable that depends
on the correlations between the single jet constituents and provides therefore
an important metric for the performance of the generative model. For the
relative quantities above, we can define the scaled jet mass as
$(m^{rel})^{2}=(\sum_{i}E^{~{}rel}_{i})^{2}-(\sum_{i}\vec{p}^{~{}rel}_{i})^{2}=m_{jet}^{2}/p_{T,jet}^{2}$.

An additional set of metrics is constructed in [10] by using the Energy Flow
Polynomials (EFP) [12], which form a complete set of jet substructure
observables.

The _JetNet_ paper additionally proposes a Message-Passing Generative
Adversarial Network (MPGAN) that outperforms other state-of-the-art models at
the time of publication and to which we compare our results.

##  3 Generative Modelling

Modeling an unknown complex probability function only from a set of data
points is challenging. In addition, it is often not straightforward to assess
the quality of generated data. New learning paradigms are needed to be able to
train such a model. GANs, as one of the most popular approaches, employ a
setup of two networks, a generator and a discriminator that tries to
distinguish generated and real data. Their training is notoriously difficult.
A more stable approach to generative modelling are NFs, allowing Maximum
Likelihood training, which is typically both fast and reproducible.

###  3.1 Normalising Flow Models

In NFs, the objective is to transform the training data distribution to a
Normal distribution by mapping the training data with a series of invertible
functions. This setup then allows for Maximum Likelihood optimization of the
parameters of the transformation. Once the NF can transform the training data
to a Normal distribution sufficiently well, new data is generated by sampling
from a Normal distribution and applying the inverted transformations. This
study used the nflows [13] library for NFs.

####  3.1.1 Rational Quadratic Spline Coupling Layers

In NFs with coupling layers, a clever trick is used to guarantee the
invertibility of the transformation: the input dataset is split into two
disjoint sets along its feature dimensions. The first set is mapped with the
identity function, whereas the second set is transformed with a usually
elementwise transformation. In the case of Rational Quadratic Splines (RQS)
[14], the quotient of two monotonic quadratic splines is used, which is by
construction invertible. To increase the expressivity of the transformation,
the parameters of the RQS for the second feature set are the output of a
neural network applied to the first set. In our model, multiple of such RQS
coupling layers are stacked, and after each layer a different split for the
two sets is chosen. The Neural Networks used for producing the parameters of
the RQS are fully connected feed-forward networks, using skip connections.

###  3.2 Transformers

Transformers were originally proposed as sequence-to-sequence models for
machine translation [9]. However, they have been widely adopted in various
fields like computer vision and speech processing. On an abstract level, the
main ingredients of a transformer are the encoder block and the decoder block.
In this study, similarly as for most vision transformers, only an encoder
block is used [15]. The attention architecture [8] is typically permutation
invariant, which is a useful feature for modelling particles in jets. The
encoder block consists of two main components, one being $1\times 1$
convolutions applied token-wise (here particle-wise, since in this case a
token corresponds to a particle) and the second being multi-headed self
attention. Important for our use case is that transformers can practically
take a variable number of particles as input. This is done by zero-padding
particles, which are then masked in the calculation of the attention: their
respective values are set to negative infinity in the softmax function of the
attention calculation such that they do not have any influence on the output
at all.

In this study the output of the NFs is corrected with a post-processing
transformer encoder network. An adversarial setup is used to train the
refinement network which provides an additive correction to the output of the
NF. A discriminator or critic is used to judge whether the corrected NF
generated jets resemble real data. This refinement setup is used instead of a
setup of only a generator and discriminator as the training of adversarial
architectures and transformers is already difficult enough without combining
them.

Figure 1: An overview of all components used in the model. The schematic on
the left describes the NF, made up of 15 coupling layers. The middle one shows
the refinement network, which is made up of 4 transformer encoder blocks and
the one on the right similarly illustrates the critic network. All three
models are used during the training, while only the first two are used during
generation. The number of layers are given for the model architecture that
gives the results presented in this study.

####  3.2.1 Post-processing network

The post-processing network is a transformer encoder, which takes the output
of the NF and reshapes it to $\mathbb{R}^{N\times D}$, where $N$ is the
maximum number particles per jet and $D$ is the features per particle, in the
case of this study $D=3$, and $N=30$. It uses Layer Normalization [16] and
residual connections to connect the pre-attention features with the post-
attention layer features (skip connections in fig. 1). The normalization is
applied after the addition (Add&Norm in fig. 1).

####  3.2.2 Discriminator/Critic Network

Depending on whether a more Vanilla GAN-like [17; 18] or a WGAN [19; 20] setup
is employed, a discriminator or critic is used. The latter is another
transformer encoder that uses a BERT style classification token [15] to assess
the realness of a jet. After the classification token passes through the
second transformer encoder network, a fully-connected network is applied,
which additionally takes the invariant mass as input. The rest of the
architecture is the same as for the post-processing network. A schematic of
all parts of the architecture is shown in Fig. 1.

###  3.3 Training

Different Training procedures are tested and compared, including the Vanilla
GAN [17] training, an LSGAN [18] and a WGAN training with gradient penalty
[19; 20]. Optimization is done with RMSprop [21], the learning rate was varied
with a two-cycle cosine scheduling after a linear warm-up that is commonly
used with transformers. The most stable results were obtained with the LSGAN
training. The presented model was trained on a NVIDIA P100 GPU after around 11
hours of training time. To produce results which can be compared to Ref. [10],
we perform the same (70/30) train/test split and evaluate with the functions
that are provided by the _JetNet_ library.

##  4 Results

The results on the top-quark dataset from Ref. [22] are shown in Table 1. The
first two entries are from the best performing models of Ref. [10], while VNF
is the evaluation of the NF output and TF is the transformer-refined results.
The best performing model used 15 RQS coupling layers and 4 transformer
encoder layers in the refinement network and the critic. The performance of
the proposed models is comparable or better with respect to all considered
metrics. Of particular interest is the FPND metric, where the proposed model
performs considerably better. In Fig. 2 the results are visualized by
comparing the generated particles with the holdout set. The marginal
distribution of all features of all particles and the distribution of the
invariant mass of the jet are shown. The model needs $\sim 7.6\mu s$ for the
generation of one jet, which is 4.6 times faster than the model in Ref. [10]
on a NVIDIA A100.

Model | $W_{1}^{M}(\times 10^{-3})$ | $W_{1}^{P}(\times 10^{-3})$ | $W_{1}^{EFP}(\times 10^{-5})$ | FPND | COV $\uparrow$ | MMD
---|---|---|---|---|---|---
MP-MP [10] | $\mathbf{0.6\pm 0.2}$ | $2.3\pm 0.3$ | $\mathbf{2\pm 1}$ | $0.37$ | $\mathbf{0.57}$ | $\mathbf{0.071}$
MP_LFC-MP [10] | $0.9\pm 0.3$ | $2.2\pm 0.7$ | $\mathbf{2\pm 1}$ | $0.93$ | $0.56$ | $0.073$
VNF | $5.0\pm 0.2$ | $2.5\pm 0.6$ | $14\pm 2$ | $5.61$ | $0.56$ | $0.072$
TF | $0.78\pm 0.09$ | $\mathbf{1.3\pm 0.3}$ | $\mathbf{2\pm 1}$ | $\mathbf{0.08}$ | $\mathbf{0.57}$ | $0.072$
Table 1: Comparison between the best performing models from Ref. [10], with
different models from this study on the top quark dataset. Scores written in
bold highlight the best value.  Figure 2: Distribution of all particles
$(\eta^{rel},\phi^{rel},p_{T}^{rel})$ and the invariant mass. The generated
data is compared to a hold out data set, and a ratio is shown below the plot.

##  5 Conclusion

In this study a particle cloud generation model consisting of normalising
flows and transformer encoders is proposed. The transformer encoder acts as a
refinement model on the output of the normalising flow, and is trained in an
adversarial fashion using another transformer encoder as critic. Intuitively
one would not expect that NFs are well suited for point cloud generation,
however they provide a stable starting point, as adversarial setups and
transformer models are notoriously difficult to optimize. The model reaches
state-of-the-art performance on the publicly available _JetNet_ [23] top quark
dataset. It is important to note that particle clouds are similar to point
clouds, possibly even more complex. As such it is expected that the model also
would perform well on point cloud generation tasks. For future studies it is
planned to use the model on the other _JetNet_ datasets [22] as well as on a
dataset with more particles, and to evaluate its scaleability. It is expected
that inputs with a higher dimensionality are difficult for the attention
calculation as it scales with the number particles squared. But as
transformers are an active field of research, there seem to be possible ways
to overcome this [24; 25].

##  6 Acknowledgements

Benno Käch is funded by Helmholtz Association’s Initiative and Networking Fund
through Helmholtz AI (grant number: ZT-I-PF-5-64). This research was supported
in part through the Maxwell computational resources operated at Deutsches
Elektronen-Synchrotron DESY (Hamburg, Germany). The authors acknowledge
support from Deutsches Elektronen-Synchrotron DESY (Hamburg, Germany), a
member of the Helmholtz Association HGF.

## References

  * mli [2022] A living review of machine learning for particle physics.  2022\.  URL https://iml-wg.github.io/HEPML-LivingReview/.
  * [2] CMS Collaboration.  CMS offline and computing public results.  URL https://twiki.cern.ch/twiki/bin/view/CMSPublic/CMSOfflineComputingResults.
  * Apollinari et al. [2015] G Apollinari, O Brüning, T Nakamoto, and L Rossi.  High luminosity large hadron collider hl-lhc, 2015.  URL https://cds.cern.ch/record/2120673.
  * [4] E. G. Tabak and Cristina V. Turner.  A family of nonparametric density estimation algorithms.  _Communications on Pure and Applied Mathematics_ , 66(2):145–164.  doi:https://doi.org/10.1002/cpa.21423.
  * Rezende and Mohamed [2015] Danilo Rezende and Shakir Mohamed.  Variational inference with normalizing flows.  In _Proceedings of the 32nd International Conference on Machine Learning_ , volume 37 of _Proceedings of Machine Learning Research_ , pages 1530–1538. PMLR, 2015.  URL https://proceedings.mlr.press/v37/rezende15.html.
  * Kobyzev et al. [2021] Ivan Kobyzev, Simon J.D. Prince, and Marcus A. Brubaker.  Normalizing flows: An introduction and review of current methods.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 43(11):3964–3979, nov 2021.  doi:10.1109/tpami.2020.2992934.  URL https://arxiv.org/abs/1908.09257.
  * Papamakarios et al. [2019] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan.  Normalizing flows for probabilistic modeling and inference.  2019\.  doi:10.48550/ARXIV.1912.02762.  URL https://arxiv.org/abs/1912.02762.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _CoRR_ , abs/1706.03762, 2017.  URL http://arxiv.org/abs/1706.03762.
  * Sutskever et al. [2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le.  Sequence to sequence learning with neural networks.  In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, _Advances in Neural Information Processing Systems_ , volume 27. Curran Associates, Inc., 2014.  URL https://proceedings.neurips.cc/paper/2014/file/a14ac55a4f27472c5d894ec1c3c743d2-Paper.pdf.
  * Kansal et al. [2021a] Raghav Kansal, Javier M. Duarte, Hao Su, Breno Orzari, Thiago Tomei, Maurizio Pierini, Mary Touranakou, Jean-Roch Vlimant, and Dimitrios Gunopoulos.  Particle cloud generation with message passing generative adversarial networks.  _CoRR_ , abs/2106.11535, 2021a.  URL https://arxiv.org/abs/2106.11535.
  * Kansal et al. [2021b] Raghav Kansal, Javier Duarte, Hao Su, Breno Orzari, Thiago Tomei, Maurizio Pierini, Mary Touranakou, Jean-Roch Vlimant, and Dimitrios Gunopulos.  Jetnet(1.1)[topquark], May 2021b.  URL https://doi.org/10.5281/zenodo.6302454.
  * Komiske et al. [2018] Patrick T. Komiske, Eric M. Metodiev, and Jesse Thaler.  Energy flow polynomials: a complete linear basis for jet substructure.  _Journal of High Energy Physics_ , 2018(4), apr 2018.  doi:10.1007/jhep04(2018)013.  URL https://doi.org/10.1007%2Fjhep04%282018%29013.
  * Durkan et al. [2020] Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios.  nflows: normalizing flows in PyTorch, November 2020.  URL https://doi.org/10.5281/zenodo.4296287.
  * Durkan et al. [2019] Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios.  Neural spline flows, 2019.  URL https://arxiv.org/abs/1906.04032.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2018.  URL https://arxiv.org/abs/1810.04805.
  * Ba et al. [2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.  Layer normalization, 2016.  URL https://arxiv.org/abs/1607.06450.
  * Goodfellow et al. [2014] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks, 2014.  URL https://arxiv.org/abs/1406.2661.
  * Mao et al. [2016] Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, and Zhen Wang.  Multi-class generative adversarial networks with the L2 loss function.  2016\.
  * Arjovsky et al. [2017] Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein GAN.  2017\.
  * Gulrajani et al. [2017] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville.  Improved training of Wasserstein GANs.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 30, page 5767. Curran Associates, Inc., 2017.  URL http://papers.nips.cc/paper/7159-improved-training-of-wasserstein-gans.pdf.
  * Tieleman and Hinton [2014] Tijmen Tieleman and Geoffery Hinton.  Rmsprop gradient optimization, 2014.  URL http://www.cs.toronto.edu/tijmen/csc321/slides/lecture_slides_lec6.pdf.
  * Kansal et al. [2021c] Raghav Kansal, Javier Duarte, Hao Su, Breno Orzari, Thiago Tomei, Maurizio Pierini, Mary Touranakou, Jean-Roch Vlimant, and Dimitrios Gunopulos.  JetNet, 05 2021c.
  * [23] Raghav Kansal.  jetnet library.  URL https://github.com/jet-net/JetNet.
  * Rabe and Staats [2021] Markus N. Rabe and Charles Staats.  Self-attention does not need $o(n^{2})$ memory, 2021.  URL https://arxiv.org/abs/2112.05682.
  * Keles et al. [2022] Feyza Duman Keles, Pruthuvi Mahesakya Wijewardena, and Chinmay Hegde.  On the computational complexity of self-attention, 2022.  URL https://arxiv.org/abs/2209.04881.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes]  Scalability is unclear

    3. (c)

Did you discuss any potential negative societal impacts of your work? [No]
The model brings the same negative impact as most generative models - but due
to the limited space in this extended abstract it has been left out.

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A]

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [No]

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [No]  Again, left out due to limited space

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [No]  Again, left out due to limited space

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes]  The
libraries that were used are nflows and JetNet

    2. (b)

Did you mention the license of the assets? [Yes]  Both libraries use the MIT
license

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [No]

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [N/A]

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [N/A]

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]