qtree/.style= baseline, for tree= circle, black, draw, parent anchor=south,
child anchor=north, align=center, inner sep=1pt,

11institutetext:  Institute of Information Security and Dependability,
Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany 11email:
<EMAIL_ADDRESS>
22institutetext: Group IT Solution & Enterprise Architecture,
Volkswagen AG, Wolfsburg, Germany
22email<EMAIL_ADDRESS>33institutetext: Institute of Software
Engineering and Automotive Informatics,
Technische Universität Braunschweig, Braunschweig, Germany

# A Query Language for
Software Architecture Information
(Extended version)

Joshua Ammermann 11 0000-0001-5533-7274 Sven Jordan 22 Lukas Linsbauer 33 Ina
Schaefer 11 0000-0002-7153-761X

###### Abstract

Software maintenance is an important part of a software system’s life cycle.
Maintenance tasks of existing software systems suffer from architecture
information that is diverging over time (architectural drift). The Digital
Architecture Twin (DArT) can support software maintenance by providing up-to-
date architecture information. For this, the DArT gathers such information and
co-evolves with a software system, enabling continuous reverse engineering.
But the crucial link for stakeholders to retrieve this information is missing.
To fill this gap, we contribute the Architecture Information Query Language
(AIQL), which enables stakeholders to access up-to-date and tailored
architecture information. We derived four application scenarios in the context
of continuous reverse engineering. We showed that the AIQL provides the
required functionality to formulate queries for the application scenarios and
that the language scales for use with real-world software systems. In a user
study, stakeholders agreed that the language is easy to understand and
assessed its value to the specific stakeholder for the application scenarios.

###### Keywords:

Software architecture Query languages Software architecture recovery.

##  1 Introduction

As software systems evolve over time, architecture documentation is getting
increasingly important. In many cases, the architecture documentation is not
kept up-to-date due to cost and time restrictions [8]. This divergence is
called architectural drift and leads to increased maintenance effort [18]. To
mitigate the problem of architectural drift, Software Architecture Recovery
(SAR) methods emerged [3, 5]. SAR methods enable the recovery of architecture
information but only target isolated views (e.g. structural or behavioral
views) of a software system from specific information sources (e.g. source
code). Further, architecture information is only recovered at a single point
in time.

To solve all of the aforementioned issues, Jordan et al. [7] recently
developed the Digital Architecture Twin (DArT), an architectural data model
that co-evolves with a software system. A DArT incorporates several SAR
methods to automatically generate up-to-date architecture information from
multiple information sources (e.g. source code, build and deployment scripts,
and views). The DArT enables integration into the development process in the
form of continuous reverse engineering [2]. Further, the DArT provides up-to-
date, tailored architecture information for stakeholders enabling design
decisions on the current software architecture of the software system.
However, the crucial link between stakeholders and extensive architecture data
models, i.e., a tailored Architecture Query Language (AQL) for the DArT, is
missing.

This ability to query an architecture information base is a challenge in
reverse engineering [2]. The term AQL was proposed by Sartipi [15] in the
context of their reverse engineering approach. Sartipi’s AQL is not suitable
for querying the DArT as it is limited in the supported views and constrained
to a specific recovery process [15], which is too restrictive to use for
extensive architecture data models. The problem that arises with limited views
is that accessing the vast amounts of information in architecture data models
becomes harder, the fewer views exist in the query language. In fact, some
information may be irretrievable if there is no suitable view to query it. The
AQL by Wang et al. [20] is tailored to information exchange between
Architecture Description Languages (ADLs), which document software
architectures [10, 18], so the query results are ADL instances themselves
[20]. This is inconvenient to stakeholders because for each view a suitable
ADL has to be known and used.

To counter those shortcomings of existing AQLs, we propose the Architecture
Information Query Language (AIQL)111This work briefly summarizes the author’s
master thesis[1] and provides further evaluation regarding the AIQL’s
scalability and usability. that is process-agnostic, typed and supports
querying architecture information of various software architecture views. The
AIQL is easily extensible to support additional views and has an easy-to-use
syntax. Architecture information can be queried by defining and restricting
query templates. Composition of query templates is a key element of the AIQL,
that strongly encourages reuse. The AIQL provides stakeholders with efficient
access to the architecture information in extensive architecture data models.
To demonstrate this, we derived four application scenarios in the context of
continuous reverse engineering, which now can be realized by accessing a DArT
using the AIQL. Utilizing these future applications will decrease the
maintenance caused by architectural drift. In a user study with eight experts,
they agreed that the AIQL is easy to understand and use. They further
confirmed that AIQL queries enable drilling down into the architecture
information in the DArT.

##  2 The Digital Architecture Twin (DArT)

A DArT [7] is an up-to-date digital representation of a software system. The
DArT automatically recovers and integrates architecture information of
multiple views and versions. Fig. 1 displays how the DArT is filled with
information. First, in an Architecture Recovery step (\raisebox{-.9pt} {1}⃝),
architecture data is continuously gathered from heterogeneous sources by Data
Collection Agents. Architecture Information Recovery Services apply different
architecture recovery approaches on the gathered data to recover various
architecture information. The recovered architecture information is then
integrated into the DArT as a virtual representation of a software system’s
architecture (step \raisebox{-.9pt} {2}⃝). The architecture information is
versioned and stored in a concise, unified and persistent Architecture
Information Model. In step \raisebox{-.9pt} {3}⃝ stakeholders want to access
tailored parts of this information to fulfill specific tasks. For this, an
easy-to-use language is required to make the information accessible to various
kinds of stakeholders, such as developers or enterprise architects.

Figure 1: Architecture Information Processing using the DArT [7].

##  3 Application Scenarios

Frequent challenges in the industry are understanding and evolving software
systems. This entails reusing of beneficial design patterns and the
harmonization of software systems. Another challenge is to ensure alignment of
the implemented software system with the initially planned architecture. This
implies making architecture decisions for existing and new software systems
using tailored and up-to-date architecture information, which supports the
design decision process. We identified four application scenarios addressing
these challenges, which define the requirements for the AIQL. The application
scenarios are refined into concrete use cases (UCs), which will be used later
to evaluate the language.

##### Sce. 1: Pattern and style identification for continuous analysis

The identification of reusable patterns and the exclusion of anti-patterns in
software systems is beneficial for a high-quality maintainable software
architecture. Therefore the language should support the exploitation of
patterns, anti-patterns, and design decisions, by enabling querying of
structural and architecture information. For example, a stakeholder may want
to get a list of design patterns to capture them in a design pattern catalog
(UC 1).

##### Sce. 2: Tailored documentation

Analyzing an architecture (or the available architecture information) is a
necessity to make design decisions. To ease this, the language should support
tailored architecture documentation in the form of different views for
specific stakeholders. Thus, querying for different views from an architecture
information base needs to be supported. The retrieved information may be used
to strengthen the understanding of the system by visualizing it. Examples for
this application scenario are an enterprise architect, who wants to list all
interfaces to another specific sub-system (UC 2.1), or a product owner, who
wants to create an overall structural diagram (UC 2.2), or a developer, who
wants to create a class diagram of a system component (UC 2.3).

##### Sce. 3: Automated continuous compliance checking

The divergence of the planned architectural information from the implemented
system can be detected at an early stage using automated techniques. A query
language should support the continuous comparison of the planned system
architecture with the actual system architecture. For instance, an architect
may want to run automated compliance checking against guidelines, which
requires access to the latest version of the system architecture (UC 3).

##### Sce. 4: Recommendation system for architectural design decisions

A recommendation system for architectural design decisions enables the reuse
of good design decisions of existing systems. Such a recommendation system
uses a knowledge base of previous design decisions to propose architectural
designs that may benefit another software system. For this, relevant system
properties as well as design decisions, have to be accessible via a query
language (UC 4.1). Furthermore, a system architect might want to browse the
knowledge-base using exploratory queries (UC 4.2).

##### Language requirements

To be applicable to the above application scenarios, the architectural
language to be developed has to provide information from various views. Also,
specific versions of a system should be queryable. Further, the syntax of the
language should be easy to use and understandable for a variety of
stakeholders. The architecture query language should capture the semantics of
the software architecture domain and use existing notations that stakeholders
are familiar with. The output of a query should be presented in a usable and
human-readable output format. The output format should also support automation
for further analysis and processing.

##  4 The Architecture Information Query Language (AIQL)

We designed the AIQL to satisfy the identified requirements. In this section,
we demonstrate the AIQL’s main language features by an example.222Detailed
description of the language’s syntax in Extended Backus–Naur Form (EBNF) and
its semantics are provided in appendix 0.A and 0.B respectively. Our running
example consists of a client and a server component connected via Remote
Procedure Call (RPC) (see Fig. 2(a)). After the architecture recovery step,
this system is represented in the DArT’s technical component view as depicted
in Fig. 2(b). For each component from the running example, a corresponding
TechnicalComponent was created in the DArT. Further, a SoftwareSystem
component, representing the complete system, was created.

((a))

((b))

Figure 2: Unified Modeling Language (UML) component diagram of a client-server
example system (a). The UML object diagram shows the system recovered in the
DArT (b). Highlighted are the objects that the AIQL query in Fig. 3 returns
for this example system.

⬇

1MODEL "system.model";

2VERSION LAST;

3

4LIST TechnicalComponent serverComponent RESTRICTIONS:

5 (Name ’server’);

6

7LIST SoftwareSystem system RESTRICTIONS:

8 (EXISTS Children serverComponent);

9

10OUTPUT serverComponent;

11OUTPUT system;

Figure 3: AIQL query example for client-server entities of the technical
component view. The last version of the model system.model is queried. Two
query templates (system and serverComponent) are matched. Restrictions on the
relation Children and the attribute Name are defined in the query templates.

###  4.1 Language Design

Imagine a software architect who wants to generate tailored documentation for
server components (see scenario 2). Fig. 3 shows an example AIQL query to
retrieve server components from the DArT (for simplicity, we assume that such
components could be identified by their name). Using this example, we
demonstrate the AIQL’s main language features.

In general, an AIQL query consists of three parts: header, body, and output.
The header contains information for the whole query so that the AIQL is
complete in the sense that no information has to be provided externally. In
the header (lines 1-2), the location of the MODEL from which the architecture
information will be queried, and the VERSION(s) of the model are defined. For
the version, static keywords for the FIRST and LAST versions are provided, and
multiple versions can be accessed through expressions. Versions are required,
e.g., for UC3, and are part of the header to ensure consistency in the query
results.

In the query body, query templates for elements of the architecture
information model can be defined, which a stakeholder either wants to output
or use to restrict further in the query. A query template is declared using
the LIST directive (line 4) followed by the unique name of the type of the
element, that should be queried (e.g. TechnicalComponent), and a user-defined
identifier (e.g. serverComponent). 333We refrained from reusing the Structured
Query Language’s (SQL) syntax (SELECT instead of LIST), as we want to
highlight the fundamental differences in the underlying semantics: instead of
selecting columns from a table, in AIQL query templates result in a list of
matched elements of a certain type. A query template matches the model
elements satisfying the specified RESTRICTIONS. Multiple restrictions can be
defined in one query and are implicit conjunctions except if the OR keyword is
used. Restrictions can be used on element attributes (e.g. Name in line 5) for
which a static expression language for primitive types is provided. For our
example the String attribute Name has to match the String ‘server’.

The composition of query templates is a key element of the AIQL, that strongly
encourages reuse. References to other model elements can be restricted through
a combination of a quantifier (such as EXISTS or FORALL), the name of the
reference, and the identifier of another query template. Quantifiers are only
required for one-to-many or many-to-many references. Entities of different
views may be connected through relations and, thus, a stakeholder’s queries
can span multiple views. In line 8, the one-to-many reference Children from
SoftwareSystem to TechnicalComponent is restricted by the EXISTS quantifier
and the serverComponent template. Only software systems that have a server
component are matched. The reference Children in this case is a shortcut
defined in the AIQL as the DArT in this example contains ComponentEdges (see
Fig. 2(b)) which are resolved internally. Shortcuts allow more concise queries
and can be user-defined.

Finally, architecture model elements can be output using multiple OUTPUT
directives with the identifier of the query template to be output. Output
directives support tailoring of the output format (e.g., to only output
specific attributes). The query in Fig. 3 returns all software systems that
have a server component and all server components contained in the latest
version of the model. For the example, the model elements returned by this
query are highlighted in Fig. 2(b).

###  4.2 Implementation

To show the real-world applicability of the AIQL, we implemented a prototype
for querying the DArT. Leveraging Model Driven Software Development (MDSD)
techniques, the DArT’s meta-model is a central artifact “considered equal to
code” [17] that incorporates domain-specific abstractions, in this case of the
software architecture domain. A concrete DArT instance has a concrete
architecture information model that is compliant with the meta-model and
filled with information of a concrete software system.

The concrete syntax of the AIQL is developed to match the DArT’s meta-model
using MDSD techniques and, thus, easily extensible444Due to confidentiality
reasons as required by the industry partner, the source code cannot be
provided, as the concrete language highly depends on the DArT’s meta-model..
The available query templates and restrictions (their attributes and
references) are derived directly from the DArT’s meta-model. The available
expression language is static and applied according to the attribute types
(e.g., in AIQL queries regular expressions can be used for String attributes
in the meta-model). Additionally, users can define concise shortcuts which are
accounted for in the AIQL’s interpreter.

The implementation builds upon the Epsilon Object Language [9] and the Xtext
language workbench [4]. An editor for manual use cases is provided as well as
a query builder Application Programming Interface (API) for automation.
Executing queries via the API returns lists of Java objects, while using the
provided graphical editor and interpreter the results are marshaled into
JavaScript Object Notation (JSON) format. Fig. 4 displays the JSON returned
for the example query on the running example. Leveraging code generation
through Xtext and a model-to-text transformation, the implementation can adapt
to changes in the model with low effort, resulting in higher maintainability.

string=[s]"", stringstyle=, comment=[l]:, commentstyle=,

⬇

[

[ {

"type" : "SoftwareSystem",

…

} ], [ {

"type" : "TechnicalComponent",

"Type" : "SUBSYSTEM",

"Name" : "server",

"Version": …,

"ComponentEdge": [],

…

} ]

]

Figure 4: Returned JSON for query of UC2.2 executed on the example system.

##  5 User study

We performed a user study at an industry partner to further evaluate the
AIQL’s usability. The eight participants (Enterprise-, Solution-, and Domain
Architects, IT Project managers, and developers) ranged from 7 to 21 years of
work experience. In the user study, we asked participants about the
comprehensibility of the AIQL (produced output and language itself), the
usability of the output and the complexity of example queries. In the user
study we presented the AIQL in-depth (i.e., theoretical and practical) to the
participants and allowed them to use the AIQL. Furthermore, we showed
participants the application scenarios and relevant example queries and asked
them to assess the value of the AIQL for the application scenarios 1, 2 and 4.

###  5.1 Research Questions

We identified four research questions to evaluate the AIQL in terms of
understanding, applicability and complexity:

RQ1

Is the AIQL easy to comprehend and to use for stakeholders?

RQ2

Is the output of the AIQL comprehensible and useful to stakeholders?

RQ3

How complex are (nested) AIQL queries to use for stakeholders?

RQ4

How valuable is the AIQL for the application scenarios?

Using these main research questions, we designed our user study questions to
ask the participants if the output of the AIQL is understandable and if the
output is useful to them. We further questioned participants if the syntax of
the AIQL is easy to understand. We also asked them for valuable feedback
regarding the power of the AIQL.

###  5.2 Results and Discussion

####  5.2.1 RQ1

All participants stated that the AIQL is easy to understand. In comparison to
other languages (e.g. the Structured Query Language (SQL)), the AIQL was
deemed equally easy. 62.5% of participants found the language intuitive and
the keywords meaningful. 37.5% of participants mentioned difficulties using
the AIQL, due to the short timeframe of the user study, they were not able to
memorize all keywords of the AIQL. More time to familiarize themselves with
the AIQL as well as with the DArT was needed. Three participants (solution and
enterprise architects) preferred the accessible language design, whereas one
participant did not immediately grasp the composition of sub-queries. It was
also noted that the comprehensibility is limited for non-technical experts and
that the language may be hard to understand for business users. Detailed
responses by the participants mentioned missing an “Aggregate”-keyword to
automatically filter the highest-level sub-systems. The current editor support
was praised to ease getting started with the AIQL.

####  5.2.2 RQ2

All participants stated that the AIQL is useful for extracting architecture
information about a software system, but it is restricted by the underlying
data model. 50% of participants noted that the AIQL, if not used to full power
is not necessarily more useful than a static view, but the AIQL is more useful
when used in an exploratory approach. It was debated that the AIQL is more
useful, when adding information about frequency of class calls and when the
stakeholder knows the system to some degree. A supplementing static/dynamic
view would be help for an initial overview of the software system. 37,5% of
participants found the AIQL more useful for drilling down into the
architecture information. It was noted that the AIQL does not provide more
information than a static visualisation as it only displays recovered
architecture information. Participants mentioned that the output format (JSON)
is hard for humans to read, but useful for further use and automation.

####  5.2.3 RQ3

Participants stated that the complexity of shown example queries varied from
trivial to complex. An easy example query was perceived as very simple. An
example query in the complexity of the running example was still perceived as
easy. Participants mentioned that linking queries using the sub-query
templates was perceived as doable. A complex query using four nested sub-
queries was deemed as complex, leading participants to express concerns about
the complexity of queries containing more nested sub-queries. Some
participants also criticized the lack of self-contained queries (i.e. no sub-
query composition).

####  5.2.4 RQ4

50% of participants stated that the AIQL is useful to extract tailored
architecture information. Depending on the participant’s background, some find
the AIQL to be potentially more useful. Developers or solution architects
assessed the AIQL as more useful for extracting tailored architecture
information for specific use cases than enterprise architects. The
participants believed that the AIQL can be helpful in understanding the source
code, from an unknown software system. All participants stated the use of a
recommendation system or wizard-based queries for finding similar software
systems based on requirements and architecture is very interesting and
"incredibly useful". They also discussed the potential benefits of the AIQL by
understanding functional requirements and how it can be useful for finding
software components and libraries used for these requirements. One participant
stated that the use of AIQL in combination of the DArT is potentially helpful
in finding an appropriate point of contact for further insights. The
participants mentioned two main aspects which the language was lacking:
comprehensibility of the output and aggregation of architectural information.
Future work is needed to provide other output formats or to provide
aggregation in the AIQL. Whereas the former depends on the output format and
requires changes to the AIQL’s interpreter, the latter requires enhancement of
the underlying DArT/data model, which has to provide the necessary
information, so that the AIQL is able to aggregate architecture information.

##  6 Empirical Evaluation

The AIQL is evaluated quantitatively in this section regarding its
applicability in the defined application scenarios (cf. 3), its usability, and
its scalability.

###  6.1 Research Questions and Methodology

We derived the following three research questions and conducted the evaluation
process depicted in Fig. 5.

RQ5

Can all defined application scenarios be realized appropriately using the
AIQL?

RQ6

How many inputs by a user are necessary to formulate AIQL queries?

RQ7

Does the AIQL scale for real-world systems?

RQ5 investigates if the AIQL contains all necessary capabilities and checks if
it is possible to map the requirements of the application scenarios to the
syntax of the AIQL. RQ6 is concerned with usability in a quantitative fashion
(for a qualitative user study see Sect. 5). Having fewer mandatory inputs
increases the likelihood of acceptance by the user. In this case, we envision
the AIQL as a tool to drill into the existing architecture information which
is a manual process. A stakeholder should be supported with the manual process
as much as possible. RQ7 is concerned with the scalability of the AIQL in
terms of query execution times and asks if the AIQL scales for large systems.
This is important as large systems can contain many different architecture
elements and providing architecture information to the stakeholder has to be
in a timely manner.

Figure 5: The quantitative evaluation process of the AIQL. AIQL queries for
the application scenarios are defined and their characteristics are measured.
The measurements are analyzed to answer the research questions.

###  6.2 Results and Discussion

####  6.2.1 RQ5

We defined queries that retrieve the information required to realize the use
cases corresponding to application scenarios. Even though the diverse use
cases required queries to retrieve information from multiple architectural
views and a range of complexities, a single AIQL example query was sufficient
for each use case. We implemented a recommendation system using AIQL queries
to identify similar software systems based on (non)-functional requirements.
Also possible is the support of the tailored documentation, as the AIQL can be
used to gather the needed architecture information before processing them in a
manual process to create the specific diagrams. The output could be customized
and was presented in JSON format (Fig. 4). Hence, we can answer the RQ
positively that the AIQL is able to handle the defined application scenarios.

Table 1: Measurements of query characteristics for example AIQL queries. Use case id |  1 |  2.1 |  2.2 |  2.3 |  3 |  4.1 |  4.2 | Avg.
---|---|---|---|---|---|---|---|---
Number of queries |  1 |  1 |  1 |  1 |  1 |  1 |  1 | 1
Number of query outputs |  1 |  2 |  1 |  1 |  3 |  1 |  2 | 1.57
Number of query characters |  53 |  112 |  58 |  154 |  103 |  297 |  165 | 134.57
Min. number of input keystrokes |  33 |  65 |  33 |  87 |  68 |  148 |  119 | 79
Number of keywords in total |  6 |  11 |  6 |  15 |  12 |  27 |  17 | 13.43
Number of unique keywords |  6 |  9 |  6 |  12 |  8 |  14 |  13 | 9.71

####  6.2.2 RQ6

Table 1 depicts the quantitative measurements conducted on the use case
example queries. The number of queries for each use case was counted by the
number of interpreter executions required in the editor. For the number of
query characters (the query length), we counted dynamic-sized elements such as
identifiers and strings as constant 1. Additionally, only syntactically
relevant white spaces (e.g. between keywords) were counted and comments were
ignored. The minimal number of input keystrokes was measured using the
provided editor utilities of the realization, where key combinations (e.g.
CTRL + enter) were counted as multiple keystrokes. Elements, white spaces, and
comments were counted similarly to the query length. For the number of query
outputs, the output directives of each query were counted. (Unique) keywords
(that are highlighted as bold in the queries) were counted for the last two
measurements.

From these measurements, we conclude the following regarding the AIQL’s
usability: Using multiple output directives, only one query is needed for
every use case. The query length varies depending on the query’s complexity,
averaging 135 characters for the examples. Making full use of provided
language smarts (auto-complete and proposals), significantly reduces the
number of input keystrokes in the examples: only 79 keystrokes on average
instead of 135 when typing every character. The AIQL supports 18 syntactic
constructs that the user has to remember. The total number of keywords used
increases depending on the concrete underlying architecture information model.
For the example queries, the average query made use of 13 keywords in total
and 10 unique keywords. By using language smarts provided in the AIQL editor,
the user does not have to remember the keywords.

####  6.2.3 RQ7

To evaluate the AIQL’s scalability, open-source real-world software systems
are transformed into the DArT. We used the mid-size system fastjson (44 kLOC)
and the two large-size systems Apache Hadoop (183 kLOC) and Apache Struts2
(161 kLOC). These systems were chosen because they differ in size (mid to
large) as well as structuredness (grown to well structured). We defined seven
AIQL queries555The queries are provided in appendix 0.C. that address only the
structural view, as this information is recovered directly from the code base
and assumed to be the largest view in the DArT. The queries were selected to
differ only in specific language elements, such as restrictions and template
nesting, to show how these elements influence runtime. The AIQL queries are
executed for the recovered systems and their run times are measured and
analyzed. For each query, its description and characteristics are depicted on
the left side of Table 2. The measured execution time of the query (including
interpretation) and the output marshaling times are depicted on the right side
of Table 2. The measurements were conducted on a Windows Desktop Computer with
Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz and 16 GB of RAM. Fig. 6 visualizes
the overall query execution and marshaling time.

Table 2: AIQL queries and scalability measurements using real-world systems. |  | Number of | Execution + marshaling
---|---|---|---
|  |  out- | tem- | refer- | expres- | time in ms
Id | Description |  puts | plates | ences | sions | fastjson | Hadoop | Struts2
1 | technical components |  1 | 1 | 0 | 0 | $52+16$ | $143+53$ | $208+55$
2 | classes |  1 | 1 | 0 | 1 | $54+7$ | $165+26$ | $210+18$
3 | classes named handler |  1 | 1 | 0 | 2 | $61+<1$ | $184+2$ | $238+1$
4 | edges to handler classes |  1 | 2 | 1 | 2 | $86+<1$ | $292+1$ | $522+2$
5 | edges AND handler classes |  2 | 2 | 0 | 2 | $132+23$ | $397+32$ | $463+47$
6 | parents of handler classes |  1 | 3 | 2 | 2 | $147+3$ | $657+31$ | $934+35$
7 | edges AND handler classes |  3 | 3 | 0 | 2 | $220+47$ | $466+82$ | $568+87$
| AND components |  |  |  |  |  |  |
$0$$500$$1{,}000$fastjsonApache HadoopApache Struts2Execution time in msQuery
1 (1 Template)Query 2 (1 Template)Query 3 (1 Template)Query 4 (2
Templates)Query 5 (2 Templates)Query 6 (3 Templates)Query 7 (3 Templates)
Figure 6: Overall AIQL query execution and marshaling time in ms for each
system and query (see Table 2). The AIQL scales well for real-world systems.
But, as expected, introducing new query templates increases query time.
Restrictions on references (crosshatched queries) greatly impact query time
for larger systems.

From the results, we can see that the queries are executed in a reasonable
time for real-world systems, below 1 $second$ for the example queries. A small
number of simple restrictions does not affect query execution time
significantly for all considered systems (queries 1 to 3). Introducing
reference restrictions (crosshatched queries) significantly increases
execution time, roughly doubling for large systems. This implies that nested
reference restrictions may become a bottleneck for large systems. Simply
outputting all templates instead of using reference restrictions (not
crosshatched vs crosshatched queries with the same amount of templates)
perform better for large systems and queries and may be an alternative.

###  6.3 Threats to Validity

####  6.3.1 Internal Threats

Scalability measurements of the queries were executed and measured on a
standard computer, thus its operating system’s scheduler influenced the
measured time. To address an introduced bias, the queries were executed 5
times and the average was taken, nevertheless, replicability of the exact same
measurements can not be assured. These measurements were conducted using our
own implementation of the AIQL’s interpreter and API, which we tested for
small samples, but cannot assure that no errors occurred.

####  6.3.2 External Threats

The evaluation is based on predefined application scenarios and use cases that
should be enabled through the AIQL. Thus, there is the possibility that these
scenarios and use cases are not representative. Furthermore, the defined use
case queries may also not represent actual user queries correctly, e.g., many
concrete restrictions may be larger, and the queries may be biased, as they
were formulated by the languages designers. Regarding this bias, another user
study could mitigate such uncertainties in the future. For the scalability
evaluation, we defined queries to assess the influence of language constructs
towards run time performance in the context of real-world systems. We
carefully selected queries to isolate the effects of certain language
constructs, but still, the evaluation may be biased by the selection of
queries as well as by the example systems.

##  7 Related Work

Querying of software systems is frequently used in the context of Software
Architecture Recovery (SAR). The following approaches stated in [3] are
closest to our approach, as they use queries to present architectural
information: ARM by Guo et al. (1999) [6], Alborz by Sartipi (2001) [15] and
Revealer by Pinzger et al. (2002) [14] focus on architectural patterns.
ArchView also by Pinzger (2005) [13] provides architectural views. Relational
queries are the most common query technologies and are used by tools such as
ArchView, ARM, and Kazman and Carrière’s Dali (1999) [8] [3]. Dali and
ArchView directly use SQL on an architectural database [3]. Both approaches
persist the recovered architecture information in a SQL database, which is
queried to retrieve the recovered architecture information, however both
approaches do not correlate the architecture information in the database with
each other. ARM allows the definition of pattern descriptions and rules in a
commonly used format (Rigi Standard Format) that are translated into pattern
queries exploitable in Dali via SQL [6]. Lexical queries in Revealer can be
defined using a simple pattern language based on regular expressions [3]. All
these tools except Alborz use generic query mechanisms that are not tailored
to the domain of software architecture. However, the architecture information
in the database is not correlated with other architecture information. Our
proposed AIQL is a Domain-Specific Language (DSL) instead of a general-purpose
query language. DSLs leverage the restriction to a particular problem domain -
the software architecture domain in our case - to gain expressive power by
providing domain-tailored notations and abstraction.

##### Architectural Query Languages (AQLs)

In contrast, AQLs are tailored for architecture information retrieval. The
term AQL was proposed by Sartipi in the context of an interactive reverse
engineering environment for pattern-based recovery of software architecture
information [16], that is implemented in the Alborz toolkit. The goal of this
environment was to address the lack of a reflective and uniform model for
pattern-based software architecture recovery. Achitecture information is
incrementally recovered by defining and refining patterns defined in a
proposed AQL, which are matched against a database of architectural
information, that is generated by the environment in an offline process. A
query in the language consists of abstract connectors and components inspired
by ADLs. The language itself supports entities on file and function level,
typed import, export, and containment relations, and constraints on the
composition and interconnection size and type between subsystems or modules.
Although Sartipi’s AQL originates from an architecture recovery background,
the tight coupling of Sartipi’s AQL to the manual recovery process using
special syntactic elements makes it difficult to reuse aspects for a language
that is not driven by the proposed process, whereas the AIQL is independent of
the architecture recovery process. Wang and Gupta (2004) defined the query
language ADLQL [20]. They aim to enable information exchange between ADLs,
which are not expressive enough by themselves. Thus, a framework enabling
integration of ADL is introduced. ADLQL allows the definition of queries that
can be executed against predicates, representing architectural information,
using denotational semantics. As ADLQL is inspired by SQL, common language
constructs are reused and an OCL-like expression syntax is provided. ADLQL is
limited to ADLs, which have to be integrated and do not account for other
relevant artifacts, such as a system’s source code, whereas the AIQL is
capable of handling source code. Michalik (2010) proposed a framework
supporting tailored architecture views of a software product line’s
architectures using an AQL in [11]. It was realized as an extensible API
querying an architecture knowledge repository and not as a language [21].
Monroy et al. (2021) introduced a query mechanism for recovered architecture
models using natural language (Spanish) in the ARCo framework [12]. An
advantage of the natural language is its richness benefiting inexperienced
stakeholders, for example, students, but on the other side, it is ambiguous
[12, 18]. The use of more semantically narrow and formal languages is more
suitable, as we expect our stakeholders to have a decent amount of domain
knowledge.

##### Detailed Comparison

Tab. 3 compares the proposed AIQL to the state-of-the-art languages AQL and
ADLQL. All three languages have different purposes leading to differences in
language design. The AQL incorporates an iterative architecture recovery
process through query refinement, thus the final output is the refined AQL
query. As the AQL is limited in the supported views and constrained to a
specific recovery process, it is too restrictive to use in another context.
For extensive architecture data models, some information may be irretrievable
if there is no suitable view to query it. ADLQL and AIQL focus on querying
architecture information, so the languages are process agnostic and support
the reuse of queries for different target systems. The AIQL provides a typed
expression language in contrast to ADLQL’s OCL-like expressions. ADLQL focuses
on ADL documents such that the language is extensible by adding support for
further ADLs and the output of a query is an ADL document. This is
inconvenient to stakeholders because for each view a suitable ADL has to be
learned and used. In contrast, the AIQL addresses the research gap of querying
recovered architecture information and outputs lists of instances from the
domain model. The AIQL is extensible through MDSD techniques.

Table 3: Comparison of AQL, ADLQL, and AIQL. | AQL by Sartipi [16] | ADLQL by Wang [19] | AIQL
---|---|---|---
Purpose | iterative architecture | query ADL | query recovered
| recovery | documents | architecture information
Process agnostic | no | yes | yes
Query reusability | no | yes | yes
Extensible | no | through ADLs | through meta-model
Output format | AQL | ADL | domain model instances
Expressions | no | OCL-like | typed expression language

##  8 Conclusion and Future Work

This work aimed to provide an architecture query language to access the
information present in extensive architecture data models such as DArTs. Based
on application scenarios of such a language in the reverse engineering
context, we proposed the AIQL and demonstrated its language features in this
paper. The language encourages reuse through the composition of templates and
supports querying information of different views and levels of abstraction
along with different versions of a system’s architecture over time. In the
empirical evaluation, we showed that the AIQL provides the required
functionality to formulate queries for application scenarios in continuous
reverse engineering and that the language scales for use with real-world
software systems. In a user study, we showed that the AIQL is easy to
understand, but also identified two aspects to improve our language: providing
additional language constructs and providing other output formats. The user
study also highlighted the demand for a supporting static view. Future work is
concerned with the realization of a continuous reverse engineering application
scenario using the AIQL and leveraging the API for automation purposes.

## References

  * [1] Ammermann, J.: An architecture information query language for the digital architecture twin. Master Thesis, TU Braunschweig (2022). https://doi.org/10.24355/dbbs.084-202209301202-0, https://leopard.tu-braunschweig.de/receive/dbbs_mods_00071363
  * [2] Canfora, G., Di Penta, M., Cerulo, L.: Achievements and challenges in software reverse engineering. Commun. ACM 54(4), 142–151 (apr 2011). https://doi.org/10.1145/1924421.1924451, https://doi.org/10.1145/1924421.1924451
  * [3] Ducasse, S., Pollet, D.: Software architecture reconstruction: A process-oriented taxonomy. IEEE Trans. Software Eng. 35(4), 573–591 (2009). https://doi.org/10.1109/TSE.2009.19, https://doi.org/10.1109/TSE.2009.19
  * [4] Eysholdt, M., Behrens, H.: Xtext: Implement your language faster than the quick and dirty way. In: Proceedings of the ACM International Conference Companion on Object Oriented Programming Systems Languages and Applications Companion. p. 307–309. OOPSLA ’10, Association for Computing Machinery, New York, NY, USA (2010). https://doi.org/10.1145/1869542.1869625, https://doi.org/10.1145/1869542.1869625
  * [5] Garcia, J., Ivkovic, I., Medvidovic, N.: A comparative analysis of software architecture recovery techniques. In: Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering. p. 486–496. ASE’13, IEEE Press (2013). https://doi.org/10.1109/ASE.2013.6693106, https://doi.org/10.1109/ASE.2013.6693106
  * [6] Guo, G.Y., Atlee, J.M., Kazman, R.: A software architecture reconstruction method. In: Donohoe, P. (ed.) Software Architecture, TC2 First Working IFIP Conference on Software Architecture (WICSA1), 22-24 February 1999, San Antonio, Texas, USA. IFIP Conference Proceedings, vol. 140, pp. 15–34. Kluwer (1999)
  * [7] Jordan, S., Linsbauer, L., Schaefer, I.: Autoarx: Digital twins of living architectures. In: Software Architecture - 16th European Conference, ECSA 2022, Prague, Czech Republic, September 19-23, 2022, Proceedings. Lecture Notes in Computer Science, vol. 13444, pp. 205–212. Springer (2022)
  * [8] Kazman, R., Carrière, S.J.: Playing detective: Reconstructing software architecture from available evidence. Autom. Softw. Eng. 6(2), 107–138 (1999). https://doi.org/10.1023/A:1008781513258, https://doi.org/10.1023/A:1008781513258
  * [9] Kolovos, D.S., Paige, R.F., Polack, F.: The epsilon object language (EOL). In: Rensink, A., Warmer, J. (eds.) Model Driven Architecture - Foundations and Applications, 2nd European Conference, ECMDA-FA 2006, Bilbao, Spain, July 10-13, 2006, Proceedings. Lecture Notes in Computer Science, vol. 4066, pp. 128–142. Springer (2006). https://doi.org/10.1007/11787044_11, https://doi.org/10.1007/11787044_11
  * [10] Medvidovic, N., Taylor, R.N.: A framework for classifying and comparing architecture description languages. SIGSOFT Softw. Eng. Notes 22(6), 60–76 (Nov 1997). https://doi.org/10.1145/267896.267903, https://doi.org/10.1145/267896.267903
  * [11] Michalik, B.: Architecture query language framework. In: Botterweck, G., Jarzabek, S., Kishi, T., Lee, J., Livengood, S. (eds.) Software Product Lines \- 14th International Conference, SPLC 2010, Jeju Island, South Korea, September 13-17, 2010. Workshop Proceedings (Volume 2 : Workshops, Industrial Track, Doctoral Symposium, Demonstrations and Tools). pp. 279–286. Lancaster University (2010), http://splc2010.postech.ac.kr/SPLC2010_second_volume.pdf
  * [12] Monroy, M., Pinzger, M., Arciniegas, J.: Arco: Architecture recovery in context. J. Xi’an Univ. Archit. Technol 13(2), 128–143 (2021)
  * [13] Pinzger, M.: ArchView - analyzing evolutionary aspects of complex software systems. Ph.D. thesis, TU Wien (2005)
  * [14] Pinzger, M., Fischer, M., Gall, H.C., Jazayeri, M.: Revealer: A lexical pattern matcher for architecture recovery. In: van Deursen, A., Burd, E. (eds.) 9th Working Conference on Reverse Engineering (WCRE 2002), 28 October - 1 November 2002, Richmond, VA, USA. p. 170. IEEE Computer Society (2002). https://doi.org/10.1109/WCRE.2002.1173075, https://doi.org/10.1109/WCRE.2002.1173075
  * [15] Sartipi, K.: Alborz: A query-based tool for software architecture recovery. In: 9th International Workshop on Program Comprehension (IWPC 2001), 12-13 May 2001, Toronto, Canada. pp. 115–116. IEEE Computer Society (2001). https://doi.org/10.1109/WPC.2001.921721, https://doi.org/10.1109/WPC.2001.921721
  * [16] Sartipi, K.: Software Architecture Recovery based on Pattern Matching. Ph.D. thesis, University of Waterloo, Waterloo, ON, Canada (2003)
  * [17] Stahl, T., Völter, M., Bettin, J., Haase, A., Helsen, S.: Model-driven software development - technology, engineering, management. Pitman (2006), http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470025700.html
  * [18] Taylor, R.N., Medvidovic, N., Dashofy, E.M.: Software Architecture - Foundations, Theory, and Practice. Wiley (2010), http://eu.wiley.com/WileyCDA/WileyTitle/productCd-EHEP000180.html
  * [19] Wang, Q.: A Semantics-Based Approach to Processing Formal Languages. Ph.D. thesis, University of Texas at Dalla, USA (2007)
  * [20] Wang, Q., Gupta, G.: A logic programming-based environment for integrating architecture description languages. In: Muñoz-Hernández, S., Gómez-Pérez, J.M. (eds.) Proceedings of the 14th International Workshop on Logic Programming Environments, WLPE 2004, Saint-Malo, France, September 6-7, 2004. pp. 67–86 (2004)
  * [21] Weyns, D., Michalik, B., Helleboogh, A., Boucké, N.: An architectural approach to support online updates of software product lines. In: 9th Working IEEE/IFIP Conference on Software Architecture, WICSA 2011, Boulder, Colorado, USA, June 20-24, 2011. pp. 204–213. IEEE Computer Society (2011). https://doi.org/10.1109/WICSA.2011.34, https://doi.org/10.1109/WICSA.2011.34

##  Appendix 0.A AIQL syntax in EBNF

<query> = ‘MODEL’, <pathToModel>,
‘VERSION’, (‘FIRST’ | ‘LAST’ <intExpr>),

{‘LIST’, <template>, ‘;’},
{‘OUTPUT’, <output>, ‘;’};

<template> = <typeName>, <templateIdentifier>,
[‘RESTRICTIONS:’, <disjunction>];

<disjunction> = ‘(’, <conjuntion>, ‘)’
{‘OR’, ‘(’, <conjuntion>, ‘)’};

<conjuntion> = {[‘NOT’] (<attribute> | <reference>)};

<attribute> = <attributeName>, <attributeExpr>;

<reference> = [quantifier], <referenceName>, <templateIdentifier>;

<attributeExpr> = <booleanExpr> | <stringExpr> | <intOrFloatExpr> | <dateExpr>;

<quantifier> = ‘EXISTS’ ‘FOR_ALL’ (‘COUNT’, ‘(’, <intLiteral>, ‘)’) (‘RANGE’,
‘(’, <intLiteral>, ‘,’, <intLiteral> ‘)’);

<output> = <templateIdentifier>,
[‘ORDER_BY’, <orderBy>, {‘,’, <orderBy>}],
[‘ATTRIBUTE’, <attributeName>, {‘,’, <attributeName>}],
‘;’;

<orderBy> = <attributeName>, (‘ASC’ | ‘DESC’);

Figure 7: AIQL main grammar rules in EBNF.

<booleanExpr> = ’true’ | ’false’;

<stringExpr> = <string>;

<comparativeOperator> = ‘<=’ | ‘<’ | ‘=’ | ‘>’ | ‘>=’;

<intExpr> = <comparativeOperator>, <intArithmeticExpr>;

<intArithmeticExpr> = <intTerm>, { ‘+’ | ‘-’, <intTerm>};

<intTerm> = <intFactor>, { ‘*’ | ‘/’, <intFactor>};

<intFactor> = <intLiteral>, ‘(’, <intArithmeticExpr>, ‘)’;

<intLiteral> = [‘-’] <digit>, {<digit>};

<floatExpr> = <comparativeOperator>, <floatArithmeticExpr>;

<floatArithmeticExpr> = <floatTerm>, { ‘+’ | ‘-’, <floatTerm>};

<floatTerm> = <floatFactor>, { ‘*’ | ‘/’, <floatFactor>};

<floatFactor> = <floatLiteral>, ‘(’, <floatArithmeticExpr>, ‘)’;

<floatLiteral> = [‘-’] <digit>, {<digit>}, ‘.’, <digit>, {<digit>};

<dateExpr> = <comparativeOperator>, <date>;

<pathToModel> = <string>;

<typeName> = <string>;

<attributeName> = <string>;

<referenceName> = <string>;

<templateIdentifier> = [‘^’], (<character>|‘_’), {<character>|‘_’|<digit>};

<string> = ? a string supporting regular expressions ?;

<character> = ? uppercase or lowercase character (a-z, A-Z) ?;

<digit> = ? 0-9 ?;

<date> = ? date and time in YYYY-MM-DDThh:mm:ss format ?

Figure 8: AIQL expression grammar rules in EBNF.

##  Appendix 0.B AIQL semantics

Let $C$ be the set of all query-able classes defined in the domain model with
the set $R$ of all directed relations $r\in R:c_{1}\rightarrow c_{2}$ where
$c_{1},c_{2}\in C$. The model also includes a set of all attributes $A$ of
query-able classes. Now for each class $c\in C$ two sets containing only the
attributes $A_{c}$ and references $R_{c}:c\rightarrow c_{ref}$ can be derived,
where the referenced class $c_{ref}$ can be any class so that $c_{ref}\in C$.
Also, $A_{c}\subset A$ and $R_{c}\subset R$ have to hold.

For a template $t$ the typeName has to correspond to the name of a class $c$
in $C$. Templates have a unique identifier assigned in the LIST directive, by
which they can be referenced in the OUTPUT directive and in references.
Attributes $A_{t}$ of the template $t$ that are restricted in the template’s
conjunctions have to be a subset of the available attributes in that class
$A_{c}$, so that $A_{t}\subset A_{c}$ and they are identified by their
attrName. The same holds for referenced attributes by their attrName in the
OUTPUT and ORDER_BY directives, which have to correspond to the class of the
referenced template. Attribute expressions, e.g., for float and int
attributes, should have correctly typed literals. Similar to attributes,
references $R_{t}$ are identified by their refName, and only outgoing
relations from the current class $R_{c}$ can be restricted on $t$ as
$R_{t}\subset R_{c}$. Quantifiers for references have to be provided if the
maximal cardinality of the referenced element class $c_{ref}$ in the relation
is greater than one: $|c_{ref}|>1$, thus multiple referenced elements are
possible. Otherwise, no quantifier should be provided.

Furthermore, we allow the definition of additional shortcut relations $s\in S$
mapping classes $c_{1}\rightarrow c_{2}$ where $c_{1},c_{2}\in C$ that have
not already been defined in $R$, so that $R\cap S=\emptyset$. For
quantification, the same constraints as for relations have to hold for
shortcuts. These shortcuts can be used to enable shorter queries for common
references over multiple edges.

##  Appendix 0.C Queries for scalability evaluation

⬇

MODEL "{corresponding model}";

VERSION LAST;

LIST TechnicalComponent comp;

OUTPUT comp;

Figure 9: AIQL query 1 for scalability evaluation: Output all technical
components.

⬇

MODEL "{corresponding model}";

VERSION LAST;

LIST TechnicalComponent comp RESTRICTIONS:

(

Type CLASS

);

OUTPUT comp;

Figure 10: AIQL query 2 for scalability evaluation: Output all technical
components that are classes.

⬇

MODEL "{corresponding model}";

VERSION LAST;

LIST TechnicalComponent comp RESTRICTIONS:

(

Type CLASS

Name ’.*Handler’

);

OUTPUT comp;

Figure 11: AIQL query 3 for scalability evaluation: Output all technical
components that are classes and that are named ’Handler’.

⬇

MODEL "{corresponding model}";

VERSION LAST;

LIST ComponentEdge edge RESTRICTIONS:

(

Child handler

);

LIST TechnicalComponent handler RESTRICTIONS:

(

Type CLASS

Name ’.*Handler’

);

OUTPUT edge;

Figure 12: AIQL query 4 for scalability evaluation: Output all edges to
handler technical components.

⬇

MODEL "{corresponding model}";

VERSION LAST;

LIST ComponentEdge edge;

LIST TechnicalComponent handler RESTRICTIONS:

(

Type CLASS

Name ’.*Handler’

);

OUTPUT handler;

OUTPUT edge;

Figure 13: AIQL query 5 for scalability evaluation: Output all edges and also
output all handler technical components.

⬇

MODEL "{corresponding model}";

VERSION LAST;

LIST Component handlerParent RESTRICTIONS:

(

EXISTS ComponentEdge edge

);

LIST ComponentEdge edge RESTRICTIONS:

(

Child handler

);

LIST TechnicalComponent handler RESTRICTIONS:

(

Type CLASS

Name ’.*Handler’

);

OUTPUT handlerParent;

Figure 14: AIQL query 6 for scalability evaluation: Output all Components that
have a handler technical component child.

⬇

MODEL "{corresponding model}";

VERSION LAST;

LIST Component component;

LIST ComponentEdge edge;

LIST TechnicalComponent handler RESTRICTIONS:

(

Type CLASS

Name ’.*Handler’

);

OUTPUT component;

OUTPUT edge;

OUTPUT handler;

Figure 15: AIQL query 7 for scalability evaluation: Output all components,
edges, and handler technical components.


# Symbolic Explanation of Affinity-Based Reinforcement Learning Agents with
Markov Models

Charl Maree and Christian W. Omlin
Center for Artificial Intelligence Research
University of Agder
Grimstad, Norway
<EMAIL_ADDRESS>
Second affiliation: Chief Technology Office, Sparebank 1 SR-Bank, Stavanger,
Norway.

###### Abstract

The proliferation of artificial intelligence is increasingly dependent on
model understanding. Understanding demands both an interpretation—a human
reasoning about a model’s behavior—and an explanation—a symbolic
representation of the functioning of the model. Notwithstanding the imperative
of transparency for safety, trust, and acceptance, the opacity of state-of-
the-art reinforcement learning algorithms conceals the rudiments of their
learned strategies. We have developed a policy regularization method that
asserts the global intrinsic affinities of learned strategies. These
affinities provide a means of reasoning about a policy’s behavior, thus making
it inherently interpretable. We have demonstrated our method in personalized
prosperity management where individuals’ spending behavior in time dictate
their investment strategies, i.e. distinct spending personalities may have
dissimilar associations with different investment classes. We now explain our
model by reproducing the underlying prototypical policies with discretized
Markov models. These global surrogates are symbolic representations of the
prototypical policies.

_K_ eywords Explainable AI, personalized financial services, policy
regularization, affinity-based learning, Markov models

##  1 Introduction

The ultimate goal of explainable AI is understanding. It builds trust,
improves safety, and improves predictive performance by facilitating precise
model improvements (Barredo Arrieta et al., 2020). For instance, feature
saliency analyses can improve feature selection and consequently the
predictive performance in stock trading (Carta et al., 2022), and rule
extraction can enhance trust in an AI system for loan approvals Sachan et al.
(2020). Despite considerable advancement in fields such as explainable
reinforcement learning (RL) (Heuillet et al., 2021; Wells and Bednarz, 2021;
Puiutta and Veith, 2020), the explainability of their underlying models has
not yet been fully addressed (Ramon et al., 2021; Cao, 2021).

Reinforcement learning has become omnipotent in finance, for example, multi-
agent RL for algorithmic trading (Shavandi and Khedmati, 2022). Methods such
as probabilistic argumentation (Riveret et al., 2019), structural causal
modeling (Madumal et al., 2019), and introspection through interesting
elements (Sequeira et al., 2019) exemplify the pursuit of post-hoc
explainability. We, however, propose an alternative approach: rather than
attempting to extract the learned strategy post hoc, ours is an intrinsic
method that instills a desirable behavior during training (Maree and Omlin,
2022a). Through regularization of the objective function, our method
encourages global action affinities and thus exercises control over what
agents learn. We have demonstrated the value of our method in personal
prosperity management, where individual spending behaviors dictate investment
strategies (Maree and Omlin, 2022b). We instilled affinities for certain asset
classes into the policies of a set of prototypical agents, each associating
with a given personality trait. For example, a conscientiousness agent prefers
asset classes typically associated with reduced risk.

Understanding ensues from a model explanation and an interpretation of its
behavior. We distinguish between these two concepts: an explanation is a
symbolic representation of a model’s predictions, while an interpretation is a
human reasoning about its behavior. While our agent’s policies are inherently
interpretable, they lacked a symbolic explanation. Using discretized Markov
models, we now provide that explanation and thus gain insight into previously
unanswered questions, such as why do the agents invest according to
conventional wisdom: exploiting the benefits of compound growth and reducing
risk with increasing customer age. These previously unanswered questions
demonstrate the need for both explanations and interpretations: the lack of a
symbolic representation of agents’ policies inhibited our complete
understanding.

Our contributions are therefore: (1) we demonstrate how to instill global
action affinities, thus affecting how RL agents learn, which we argue is a
useful paradigm shift over the current approach of either post hoc rule
extraction or constrained learning, (2) we distinguish between model
explainability and interpretability, and in an empirical example demonstrate
the difference and the utility of both, and (3) we propose a method of using
Markov models to extract symbolic explanations of RL agents’ policies. In the
next section, we provide an overview of the current state of the art in
explainable RL and identify limitations in the field. We then describe our
data and empirical methodology, discuss our results, and conclude with
insights and future work.

##  2 Related Work

RL agents learn to solve problems by maximizing the total expected reward
awarded by the environment in which they act. They are particularly adept at
learning in the presence of sparse and delayed rewards (Sutton and Barto,
2018). The environment is a discrete-time process where the current state
depends only on the previous state and the action taken by the agent: a Markov
decision process (MDP), described by the tuple $(S,A,R,P)$ where $S$ is a set
of states, $A$ a set of actions, $R(s,a)$ the reward for taking action $a\in
A$ in the state $s\in S$, and $P(s,a)=P(s^{\prime}|s,a)$ the probability that
action $a$ in the state $s$ leads to the state $s^{\prime}$ (Bellman, 1957).
Deep deterministic policy gradients (DDPG) is a model-free RL algorithm for
learning policies in a continuous action space (Lillicrap et al., 2019). A
DDPG agent consists of four neural networks: an actor $\mu(\theta)$
representing the policy, a critic $Q(\theta)$ representing the state action
value function, and for numerical stability, a target actor
$\mu^{\prime}(\theta^{\prime})$ and a target critic
$Q^{\prime}(\theta^{\prime})$. During learning, the target network parameters
are typically updated slowly given a soft update parameter $\tau\in[0,1]$ with
a small value:
$\theta^{\prime}_{i}=\tau\theta_{i}+(1-\tau)\theta^{\prime}_{i},\
i\in\\{\mu,Q\\}$.

Explainable RL has traditionally employed generic methods that explain the
underlying models of agents (Barredo Arrieta et al., 2020). More recently,
however, bespoke methods have emerged that consider the state-action space and
/ or the behavior of the learned policy (Puiutta and Veith, 2020; Heuillet et
al., 2021; Wells and Bednarz, 2021). Most, if not all, of these approaches
extract explanations after training; they generalize the learned policy
through observation or statistical analyses. Few of these extracted
explanations match our definition of explainability, and most are more
accurately described as interpretations. _State representation learning_
connects the state space with information from actions, rewards, or expert
knowledge when extracting representations that are useful for reasoning about
policies (Lesort et al., 2018). Under certain restrictions, e.g., linearity,
it learns models that either predict states form state-action pairs, or
actions from states, thus simplifying the state-action space and improving
interpretability. _Introspection_ analyzes an agent’s experience through
statistics such as the frequency of occurrences of states, state-actions, and
transitions, the transition probabilities, and estimated rewards compared to
the learned state-action value function (Sequeira et al., 2019). It uses
interesting elements from this analysis, such as outliers, mean values, etc.
to reason about agents’ behaviors. _Structural causal modeling_ learns causal
relationships between states, actions, and rewards by defining action
influence graphs that map the action transitions for all possible paths from
an initial state to a set of terminal states (Madumal et al., 2019). It
defines the causal chain as the one path in the action influence graph that
matches the learned policy, and a reward chain as the vector of rewards along
this causal chain. Its interpretation of the policy is the comparison between
the reward chain and all other possible reward vectors that do not follow the
causal chain. _Probabilistic argumentation_ uses argumentation graphs—sets of
attacking and supporting arguments for each action in a finite action space—to
learn interpretations in a RL setting (Riveret et al., 2019). The state is the
intersection of the argumentation graph and the policy to be explained, the
actions form a probabilistic distribution across the arguments, and the
rewards depend on whether an argument attacks or supports the current action.
The learned policy provides probabilistic interpretations of agents’ actions
in human understandable terms: supporting and attacking arguments for each
action. _Reward decomposition_ replaces the scalar reward with a vector of
more meaningful rewards, where the total reward is the sum of the vector (van
Seijen et al., 2017; Juozapaitis et al., 2019; Marzari et al., 2021). Although
evaluating the reward vector for a given action might enable reasoning about
that action in meaningful terms, it does not take into account expected future
rewards and can be insufficient in environments with delayed or sparse
rewards. _Reward redistribution_ addresses this problem by redistributing
delayed rewards in time; it assigns credit to previous actions, thus reducing
the delay of the reward (Dinu et al., 2022). The immediate reward for each
time step in a sequence of state-action transitions is equal to the change in
the total expected reward. It defines key interpretable events in the policy
and, through sequence alignment, redistributes rewards to those events given a
set of transition sequences. _Hierarchical RL_ divides complex tasks into
smaller and simpler tasks that are solved by correspondingly simpler RL agents
(Beyret et al., 2019; Marzari et al., 2021). An orchestration agent learns to
sequentially combine these prototypical agents to solve complex tasks. If
tasks are sufficiently subdivided, the interpretation, or human reasoning
about agents’ decisions, follows from their simplicity.

The complexity of RL models exacerbates the issue of fidelity and validation
of any post hoc explanation. We, instead, encourage agents to adapt their
behavior during learning, thus instilling an inherent probabilistic action
affinity that is also an interpretation of their behavior (Maree and Omlin,
2022a). Contrary to constrained RL, which avoids certain conditions
(Miryoosefi et al., 2019; Chow et al., 2015), affinity-based learning promotes
certain behaviors. This paradigm shift allows the developer to define a
desired behavior that an agent must follow during learning, thus instilling a
characterization and interpretation during learning; it decouples learned
strategies from the reward expectation (Aubret et al., 2019). Affinity-based
RL is not to be confused with preference-based RL that completely eliminates
the reward function and instead learns state-action trajectories that maximize
the preferences of the expert between pairs of state-action combinations
(Wirth et al., 2017). Affinity-based RL uses policy regularization that
aids—and is never detrimental to—learning convergence by encouraging
exploration in environments with complex dynamics or particularly sparse
rewards (Andres et al., 2022; Vieillard et al., 2020). It adds a term to the
objective function that penalizes any divergence between the current policy
and a given prior, for example, Kullback-Leibler (KL) regularization, which
uses KL divergence as the distance measure (Galashov et al., 2019). Entropy
regularization is a specific case of KL-regularization, where the prior is a
uniform action distribution that increases the entropy of the policy and thus
encourages general exploration of the state-action space(Haarnoja et al.,
2017). Our method instead encourages exploration of a predefined subset of the
state-action space, which describes the desired behavior (Maree and Omlin,
2022a). We define our objective function as follows:

$\displaystyle J(\theta)$
$\displaystyle=\mathbb{E}_{s,a\sim\mathcal{D}}\left[R(s,a)\right]-\lambda L$
(1) $\displaystyle L$
$\displaystyle=\frac{1}{M}\sum_{j=0}^{M}\left[\mathbb{E}_{a\sim\pi_{\theta}}(a_{j})-(a_{j}|\pi_{0}(a))\right]^{2}$

where $\mathcal{D}$ is the replay buffer, $\lambda$ is a hyperparameter that
scales the regularization term $L$, $M$ is the number of actions, and
$\pi_{0}$ is a specific prior action distribution that represents the desired
behavior. Instilling an interpretable behavior is sufficient for online policy
interpretation (Persiani and Hellström, 2022). Unlike KL-regularization, our
prior $\pi_{0}$ is independent of the state and therefore instills a global
action affinity in the learned policy. We have demonstrated this in Maree and
Omlin (2022a) where agents navigated a grid towards a destination; they
learned to prefer, for example, only right turns and followed optimal paths
given their global affinities. In a more elaborate example, we trained a set
of prototypical agents with global affinities to invest in certain asset
classes (Maree and Omlin, 2022b). We observed the emergence of interesting
investment strategies, such as capitalizing on compound growth and reducing
risk with portfolio maturity. Although consistent with conventional wisdom,
these strategies were absent from the objective function. To complete our
understanding of this behavior, we now provide a symbolic representation—an
explanation—of these policies using Markov models.

A hidden Markov model (HMM) models an unobservable Markov process $X$ from its
relation to an observable Markov process $Y$; it learns about $X$ by observing
$Y$ under the key assumptions that $Y_{t}$ is solely dependent on $X_{t}$, and
$X_{t}$ is solely dependent on $X_{t-1}$ (the Markovian property) (Rabiner and
Juang, 1986). For a finite hidden state space $X$, there exists a Markov
matrix $F$—the sum of the rows add up to one—of state transition probabilities
where $F_{ij}=P(X_{n+1}=j\ |\ X_{n}=i)$. Similarly, for a finite observed
state space $Y$, there exists a Markov matrix $E$ that describes emission
probabilities: $E_{ij}=P(Y_{t}=j\ |\ X_{t}=i)$. We illustrate this process in
Figure 1. Given a series of observed states $\\{Y_{t}\\}_{t=0}^{T}$, the
transition and emission probabilities can be estimated using the Baum-Welch
algorithm—a special case of the expectation-maximization algorithm (Yang et
al., 2017).

Unobservable Markov
process:${X_{t_{0}}}$${X_{t_{1}}}$${X_{t_{2}}}$${\cdots}$${X_{t_{T}}}$Observable
Markov
process:${Y_{t_{0}}}$${Y_{t_{1}}}$${Y_{t_{2}}}$${\cdots}$${Y_{t_{T}}}$$F$$E$$F$$E$$F$$E$$F$$E$

Figure 1: A trellis diagram representing a hidden Markov model with an
unobservable Markov process $X$, and observable Markov process $Y$, transition
probability matrix $F$, and emission probability matrix $E$.

##  3 Methodology

In Maree and Omlin (2022c), we defined a set of prototypical agents with
intrinsic investment behaviors associated with each of five personality
traits: openness, conscientiousness, extraversion, agreeableness, and
neuroticism. We used affinity-based RL to learn investment strategies for each
of the prototypical agents. Their actions were monthly investment
distributions across five different asset classes: savings accounts, property
funds, stocks, mortgage curtailment, and luxury items. While stocks, savings,
and property investments are self-explanatory, we defined mortgage curtailment
as additional payments that reduce the principal debt of a loan, and luxury
items such as art, classic cars, fine wines, etc., that might appear in, e.g.,
the Knight Frank luxury investment index (Knight Frank Company, 2022). We also
learned linear combinations of these agents to best match the spending
personalities of individual customers which, for the sake of brevity, we do
not discuss here. However, to facilitate an understanding of our application,
we summarize this paradigm in Figure 2 and refer the reader to a comprehensive
account in (Maree and Omlin, 2022c). We now provide an explanation for the
prototypical agents’ policies using Markov models.

Asset prices Prototypical associations Transaction history Generalized
investment actions RL RNN RL RNN Prototypical investment agentsPersonal
investment strategiesEncoded spending behavior Time-variant generali- zation
Figure 2: A flow diagram illustrating our system of RL agents that predict
personalized investment strategies. There are five prototypical affinity-based
RL agents (enclosed in a red dashed rectangle), each associating with one of
five personality traits: openness, conscientiousness, extraversion,
agreeableness, and neuroticism. These are the agents that we explain using
Markov models. Their actions are combined to match the spending behaviors of
individual customers, and these combinations are continuously adjusted
according to their changing spending behavior using a recurrent neural network
(RNN). While these combinations are outside of the scope of this study, we
believe it is useful to illustrate how the agents are used in a complete
application.

To train our agents, we used pricing data for the S&P500 index, Norwegian
property index, and the Norwegian interest rate index between 1994 and 2022.
We used two common market indicators—the moving average convergence divergence
(MACD) and the relative strength index (RSI) (Chong et al., 2014)—to capture
market dynamics. These indicators are the state space features of the
environment in which our agents learned. We show these features in Figure 3.
There is an additional state variable that indicates the maturity of the
portfolio; its value is 0.0 in the first month (January 1994) and linearly
increases to 1.0 in the final month (December 2021).

Figure 3: The state data used to train the prototypical agents. We used two
common market indicators—the moving average convergence divergence (MACD) and
relative strength index (RSI)—to represent market dynamics of the S&P500
index, Norwegian property index, and Norwegian interest rate index. Our
learning time frame was between 1994 and 2022.

We show the resulting policies for the five prototypical agents in Figure 4.
The agents optimized a common reward function, i.e., monthly returns; they
maximized the portfolio value. Though they shared a common reward function,
the agents learned unique investment strategies: the conscientiousness agent,
for instance, prefers low-risk investment in property followed by resolute
mortgage curtailment, while the openness agent prefers investments that might
incite their curiosity, such as luxury items and stocks.

Figure 4: The monthly actions of the five prototypical agents, shown on an
x-axis ranging between 0 to 1, representing months between 1994 and 2022. The
y-axis represents the monthly investment in each of the asset classes. Note
that the actions strictly represent the purchase of assets, i.e. the
extraversion agent, for instance, consistently invests 100% of available
monthly funds into stocks, thus consistently increasing the portfolio holding
of stocks; assets are never sold. Though the agents optimised a common reward
function—monthly returns—, their distinct strategies were instilled through
affinity-based learning.

To train Markov models that match the predictions of the five prototypical
agents, we discretized the states and actions of the agents. We assigned three
bins to the RSI indicator based on the knowledge that values between 0 and 0.3
indicate oversold conditions, values between 0.7 and 1 indicate overbought
conditions, and values between 0.3 and 0.7 are inconclusive (Chong et al.,
2014). We similarly assigned two bins for the MACD indicator based on the
knowledge that positive values represent a buy signal, while negative values
represent a sell signal (Chong et al., 2014). We divided the maturity state
feature into 28 bins: one for each year of the investment period. We finally
assigned 5 equally sized bins for the agents actions, between 0 and 1. This
resulted in 168 potential states, of which only 102 states ever occurred. It
is reasonable that not all possible states occurred, since MACD and RSI are
related; it is not unexpected that whenever RSI indicates oversold conditions,
MACD could suggest a buy signal (Chong et al., 2014). We then estimated the
transition probabilities in the Markov matrices $F_{i}$ and the emission
probabilities $E_{i}$, where $i\in[1,5]$, for the five Markov models by
observing the state transitions and the corresponding actions for each of the
prototypical agents. Using the initial state and the five Markov models
defined by $F_{i}$ and $E_{i}$, we can reproduce the policies of the five
prototypical agents with high fidelity.

##  4 Results

We trained five distinct Markov models as global surrogates to reproduce the
predictions of five affinity-based RL agents. We show the discretized actions
of the agents and the corresponding predictions of the Markov models in Figure
5.

Figure 5: A visual comparison between the discretized predictions of five RL
agents (on the left) and the five corresponding Markov models (on the right).
The single input to the Markov models is the initial state, from which they
predict the transition to the next state and the corresponding action by the
agent. The Markov models clearly predict the actions with high fidelity.

Using only the initial state as input, the Markov models predict the agents’
actions with high fidelity, with some uncertainly when action values change
due to the probabilistic nature of Markov models.

Figure 6 shows the state transitions for a non-exhaustive subset of states:
the first 16 states visited including the initial state.

Figure 6: A non-exhaustive illustration of the trained Markov model showing
state transitions for a subset of states. States are shown as blue circles,
and state transitions and their probabilities are shown in black. We show the
first 16 states, as visualizing all 102 states is not feasible. Each state
represents a set of features with discretized values for MACD and RSI
indicators of the property and stock indices, respectively, as well as the
maturity of the portfolio. Note that not all state transitions are shown,
since the origin or destination state might not be included in this subset.

We observe that not all states are visited, which is expected since the market
indicators MACD and RSI are not entirely independent, nor are the stock and
property markets in general. For example, during macroeconomic downturns we
often observe a decline in both these markets: refer to Figure 3 and observe,
for example, the decline in both the property and S&P500 indices during the
2008 recession. Property and stock markets can also demonstrate an inverse
correlation: in Figure 3 the RSI curves for property and stocks can have
reversed slopes, while the MACD curve can exist on opposite sides of zero. By
perturbing the sizes and number of bins, we observed that portfolio maturity
holds the most salient information. This is an important observation; it
suggests that the values of the market indicators have a lesser influence on
investment strategies compared to the maturity of the portfolio. This is in
line with conventional wisdom that long-term investment should not be overly
concerned with short-term market volatility; property and stock indices have
typically followed an upward trend in the long run. The reduced dependence on
market conditions increases confidence in model robustness when trading on
unseen data: the unseen market conditions are less important than investor
age; the basic principle that younger investors can afford increased risk in
return for higher reward, and mature investors should seek to reduce portfolio
risk, is common across a wide range of market conditions.

##  5 Conclusions

Understanding deep AI models requires an interpretation of their behavior and
a symbolic representation, or explanation, of their functioning. These two
elements facilitate reasoning about a model and, thus, enhance trust in its
decisions. We have proposed a novel affinity-based approach to interpretable
reinforcement learning; it encourages exploration of a predefined subset of
the state-action space. This prior action distribution describes the agent’s
desired behavior and is the interpretation of its policy. However, our
solution lacked a symbolic explanation, resulting in unanswered questions
about why they make certain decisions. A concrete example is why a set of
agents, that learned to invest according to the preferences of prototypical
personality traits, invest in more risky assets for younger investors and
reduce risk with investor age. We now provide a symbolic representation of the
agents’ policies, using Markov models, that answer such questions. Our Markov
models recreate, with high fidelity, the discretized investment strategies of
five prototypical investment agents using only the initial state. By
perturbing the bin sizes of the discretized state features, we are able to
determine the most salient feature: portfolio maturity. The fact that market
conditions play a diminutive role in model prediction is significant: it
enhances trust in out-of-sample predictions and suggests that investment
timing is more important than market conditions. The agents make use of
compounding growth by investing in higher reward—but more risky—assets early
on, and fulfill their prescribed action distributions towards the end of the
investment period; they learned how to maximize rewards. This use case
demonstrates the need for both interpretations and explanations to fully
comprehend the functioning and characterization of deep RL systems. The Markov
model is a valuable tool for extracting a symbolic representation of an
otherwise opaque RL model, and affinity-based RL is a unique approach to
control what RL agents learn and thus interpret their behavior. It is a
paradigm shift from current approaches that either encourage general
exploration for the purpose of improved convergence or constrain the state
space to prevent the policy from visiting undesirable states. It is compelling
to apply affinity-based RL to virtuous agents, personalized learning and
teaching, chronic disease treatment, climate change, wind farm operations,
etc.

## Declaration of competing interest

The authors declare that they have no competing interests.

## Funding

This study was partially funded by a grant from the Norwegian Research
Council, project number 311465.

## References

  * Barredo Arrieta et al. [2020] Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera.  Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI.  _Information Fusion_ , 58:82–115, 2020.
  * Carta et al. [2022] Salvatore Carta, Sergio Consoli, Alessandro Sebastian Podda, Diego Reforgiato Recupero, and Maria Madalina Stanciu.  Statistical arbitrage powered by explainable artificial intelligence.  _Expert Systems with Applications_ , 206:117763, 2022.
  * Sachan et al. [2020] Swati Sachan, Jian-Bo Yang, Dong-Ling Xu, David Eraso Benavides, and Yang Li.  An explainable ai decision-support-system to automate loan underwriting.  _Expert Systems with Applications_ , 144:113100, 2020.
  * Heuillet et al. [2021] A. Heuillet, F. Couthouis, and N. Díaz-Rodríguez.  Explainability in deep reinforcement learning.  _Knowledge-Based Systems_ , 214(106685):1–24, 2021.
  * Wells and Bednarz [2021] Lindsay Wells and Tomasz Bednarz.  Explainable AI and reinforcement learning: A systematic review of current approaches and trends.  _Frontiers in Artificial Intelligence_ , 4:1–48, 2021.
  * Puiutta and Veith [2020] Erika Puiutta and Eric M.S.P. Veith.  Explainable reinforcement learning: A survey.  _Machine Learning and Knowledge Extraction. CD-MAKE 2020. Lecture Notes in Computer Science_ , 12279:77–95, 2020.
  * Ramon et al. [2021] Yanou Ramon, R.A. Farrokhnia, Sandra C. Matz, and David Martens.  Explainable AI for psychological profiling from behavioral data: An application to big five personality predictions from financial transaction records.  _Information_ , 12(12):1–28, 2021.
  * Cao [2021] Longbing Cao.  Ai in finance: Challenges, techniques and opportunities.  _Banking & Insurance eJournal_, 14:1–40, 2021.
  * Shavandi and Khedmati [2022] Ali Shavandi and Majid Khedmati.  A multi-agent deep reinforcement learning framework for algorithmic trading in financial markets.  _Expert Systems with Applications_ , 208:118124, 2022.
  * Riveret et al. [2019] Régis Riveret, Yang Gao, Guido Governatori, Antonino Rotolo, Jeremy V. Pitt, and Giovanni Sartor.  A probabilistic argumentation framework for reinforcement learning agents.  _Autonomous Agents and Multi-Agent Systems_ , 33:216–274, 2019.
  * Madumal et al. [2019] P. Madumal, T. Miller, L. Sonenberg, and F. Vetere.  Explainable reinforcement learning through a causal lens.  _arXiv_ , 1905.10958v2, 2019.
  * Sequeira et al. [2019] P. Sequeira, E. Yeh, and M.T. Gervasio.  Interestingness elements for explainable reinforcement learning through introspection.  _Joint Proceedings of the ACM IUI Workshops_ , 2327:1–7, 2019.
  * Maree and Omlin [2022a] Charl Maree and Christian W. Omlin.  Reinforcement learning your way: Agent characterization through policy regularization.  _AI_ , 3(2):250–259, 2022a.
  * Maree and Omlin [2022b] Charl Maree and Christian W. Omlin.  Can interpretable reinforcement learning manage prosperity your way?  _AI_ , 3(2):526–537, 2022b.
  * Sutton and Barto [2018] Richard S. Sutton and Andrew G. Barto.  _Reinforcement Learning: An Introduction_.  The MIT Press, second edition, 2018.
  * Bellman [1957] Richard Bellman.  A markovian decision process.  _Journal of mathematics and mechanics_ , pages 679–684, 1957.
  * Lillicrap et al. [2019] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  _arXiv_ , 1509.02971, 2019.
  * Lesort et al. [2018] Timothée Lesort, Natalia Díaz-Rodríguez, Jean-François Goudou, and David Filliat.  State representation learning for control: An overview.  _Neural Networks_ , 108:379–392, 2018.
  * van Seijen et al. [2017] Harm van Seijen, Mehdi Fatemi, Joshua Romoff, Romain Laroche, Tavian Barnes, and Jeffrey Tsang.  Hybrid reward architecture for reinforcement learning.  _arXiv_ , 1706.04208, 2017.
  * Juozapaitis et al. [2019] Z. Juozapaitis, A. Koul, A. Fern, M. Erwig, and F. Doshi-Velez.  Explainable reinforcement learning via reward decomposition.  _International Joint Conference on Artificial Intelligence. A Workshop on Explainable Artificial Intelligence._ , 2019.
  * Marzari et al. [2021] Luca Marzari, Ameya Pore, Diego Dall’Alba, Gerardo Aragon-Camarasa, Alessandro Farinelli, and Paolo Fiorini.  Towards hierarchical task decomposition using deep reinforcement learning for pick and place subtasks.  _arXiv_ , 2102.04022, 2021.
  * Dinu et al. [2022] Marius-Constantin Dinu, Markus Hofmarcher, Vihang P. Patil, Matthias Dorfer, Patrick M. Blies, Johannes Brandstetter, Jose A. Arjona-Medina, and Sepp Hochreiter.  _XAI and Strategy Extraction via Reward Redistribution_ , chapter 2, pages 177–205.  Springer International Publishing, 2022.  ISBN 978-3-031-04083-2.
  * Beyret et al. [2019] B. Beyret, A. Shafti, and A.A. Faisal.  Dot-to-dot: Explainable hierarchical reinforcement learning for robotic manipulation.  In _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS_ , page 5014–5019, 2019.
  * Miryoosefi et al. [2019] Sobhan Miryoosefi, Kianté Brantley, Hal Daume III, Miro Dudik, and Robert E Schapire.  Reinforcement learning with convex constraints.  In _Advances in Neural Information Processing Systems_ , volume 32, pages 1–10, 2019.
  * Chow et al. [2015] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone.  Risk-constrained reinforcement learning with percentile risk criteria.  _Journal of Machine Learning Research_ , 18:1–51, 2015\.
  * Aubret et al. [2019] Arthur Aubret, Laetitia Matignon, and Salima Hassas.  A survey on intrinsic motivation in reinforcement learning.  _arXiv_ , 1908.06976, 2019.
  * Wirth et al. [2017] Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz.  A survey of preference-based reinforcement learning methods.  _Journal of Machine Learning Research_ , 18(136):1–46, 2017.
  * Andres et al. [2022] Alain Andres, Esther Villar-Rodriguez, and Javier Del Ser.  Collaborative training of heterogeneous reinforcement learning agents in environments with sparse rewards: What and when to share?  _arXiv_ , 2202.12174, 2022.
  * Vieillard et al. [2020] Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Remi Munos, and Matthieu Geist.  Leverage the average: An analysis of KL regularization in reinforcement learning.  In _Advances in Neural Information Processing Systems (NIPS)_ , volume 33, pages 12163–12174. Curran Associates, 2020.
  * Galashov et al. [2019] Alexandre Galashov, Siddhant Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojtek M. Czarnecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess.  Information asymmetry in KL-regularized RL.  In _International Conference on Learning Representations (ICLR)_ , pages 1–25, New Orleans, Louisiana, United States, 2019.
  * Haarnoja et al. [2017] Tuomas Haarnoja, Haoran Tang, P. Abbeel, and Sergey Levine.  Reinforcement learning with deep energy-based policies.  In _Proceedings of the 34th International Conference on Machine Learning_ , pages 1352–1361, 2017.
  * Persiani and Hellström [2022] Michele Persiani and Thomas Hellström.  Policy regularization for legible behavior.  _arXiv_ , 2203.04303:1–16, 2022.
  * Rabiner and Juang [1986] L. Rabiner and B. Juang.  An introduction to hidden markov models.  _IEEE ASSP Magazine_ , 3(1):4–16, 1986.
  * Yang et al. [2017] Fanny Yang, Sivaraman Balakrishnan, and Martin J. Wainwright.  Statistical and computational guarantees for the baum-welch algorithm.  _Journal of Machine Learning Research_ , 18(125):1–53, 2017.
  * Maree and Omlin [2022c] Charl Maree and Christian W. Omlin.  Reinforcement learning with intrinsic affinity for personalized prosperity management.  _arXiv_ , 2204.09218:1–12, 2022c.
  * Knight Frank Company [2022] Knight Frank Company.  Knight Frank luxury investment index, 2022.  https://www.knightfrank.com/wealthreport/luxury-investment-trends-predictions/, Accessed on 2022-05-27.
  * Chong et al. [2014] Terence Tai-Leung Chong, Wing-Kam Ng, and Venus Khim-Sen Liew.  Revisiting the performance of MACD and RSI oscillators.  _Journal of Risk and Financial Management_ , 7(1):1–12, 2014.