# A Short Comparison of Classical Complex Dynamics and Holomorphic Semigroup
Dynamics

Bishnu Hari Subedi Central Department of Mathematics
Institute of Science and Technology
Tribhuvan University
Kirtipur, Kathmandu
Nepal<EMAIL_ADDRESS>/<EMAIL_ADDRESS>and  Ajaya Singh
Central Department of Mathematics, Institute of Science and Technology,
Tribhuvan University, Kirtipur, Kathmandu, Nepal<EMAIL_ADDRESS>/
<EMAIL_ADDRESS>
###### Abstract.

This is an expository plus research paper which mainly exposes preliminary
connection and contrast between classical complex dynamics and semigroup
dynamics of holomorphic functions. Classically, we expose some existing
results of rational and transcendental dynamics and we see how far these
results generalized to holomorphic semigroup dynamics as well as we also see
what new phenomena occur.

###### Key words and phrases:

Holomorphic semigroup, Fatou set, Julia set, escaping set.

###### 2010 Mathematics Subject Classification:

37F10, 30D05

This research work of the first author is supported by PhD faculty fellowship
from University Grants Commission, Nepal.

##  1\. Introduction

It is quite natural to extend the Fatou-Julia-Eremenko theory of the iteration
of single holomorphic map in complex plane $\mathbb{C}$ or extended complex
plane $\mathbb{C}_{\infty}$ to composite of the family of holomorphic maps.
So, the purpose of this paper is to expose the theory of complex dynamics not
only for the iteration of single holomorphic map on $\mathbb{C}$ or
$\mathbb{C_{\infty}}$ but also for the composite of the family $\mathscr{F}$
of such maps. Let $\mathscr{F}$ be a space of holomorphic maps on $\mathbb{C}$
or $\mathbb{C}_{\infty}$. For any map $\phi\in\mathscr{F}$, $\mathbb{C}$ or
$\mathbb{C}_{\infty}$ is naturally partitioned into two subsets: the set of
normality and its complement. We say that a family $\mathscr{F}$ is normal if
each sequence from the family has a subsequence which either converges
uniformly on compact subsets or diverges uniformly to $\infty$. The set of
normality or Fatou set $F(\phi)$ of the map $\phi\in\mathscr{F}$ is the
largest open set on which the iterates
$\phi^{n}=\phi\circ\phi\circ\ldots\circ\phi$ (n-fold composition of $\phi$
with itself) is a normal family. The complement $J(\phi)$ is the Julia set. A
maximally connected subset of the Fatou set $F(\phi)$ is a Fatou component.
The main concern of such an iteration theory is to describe the nature of the
components of Fatou set and the structure and properties of the Julia set.

In our study, classical complex dynamics refers the iteration theory of single
holomorphic map and holomorphic semigroup dynamics refers the dynamical theory
generated by various classes of holomorphic maps. In holomorphic semigroup
dynamics, algebraic structure of semigroup naturally attached to the dynamics
and hence the situation is largely complicated. The principal aim of this
paper is to see how far classical complex dynamics applies to holomorphic
semigroup dynamics and what new phenomena appear in holomorphic semigroup
settings.

##  2\. The notion of holomorphic semigroup

Semigroup $S$ is a very classical algebraic structure with a binary
composition that satisfies associative law. It naturally arose from the
general mapping of a set into itself. So a set of holomorphic maps on
$\mathbb{C}$ or $\mathbb{C}_{\infty}$ naturally forms a semigroup. Here, we
take a set $A$ of holomorphic maps and construct a semigroup $S$ consists of
all elements that can be expressed as a finite composition of elements in $A$.
We say such a semigroup $S$ by holomorphic semigroup generated by set $A$. For
our simplicity, we denote the class of all rational maps on
$\mathbb{C_{\infty}}$ by $\mathscr{R}$ and class of all transcendental entire
maps on $\mathbb{C}$ by $\mathscr{E}$. Our particular interest is to study of
the dynamics of the families of above two classes of holomorphic maps. For a
collection $\mathscr{F}=\\{f_{\alpha}\\}_{\alpha\in\Delta}$ of such maps, let

$S=\langle f_{\alpha}\rangle$

be a holomorphic semigroup generated by them. Here $\mathscr{F}$ is either a
collection $\mathscr{R}$ of rational maps or a collection $\mathscr{E}$ of
transcendental entire maps. The index set $\Delta$ to which $\alpha$ belongs
is allowed to be infinite in general unless otherwise stated. Here, each $f\in
S$ is a holomorphic function and $S$ is closed under functional composition.
Thus, $f\in S$ is constructed through the composition of finite number of
functions $f_{\alpha_{k}},\;(k=1,2,3,\ldots,m)$. That is,
$f=f_{\alpha_{1}}\circ f_{\alpha_{2}}\circ f_{\alpha_{3}}\circ\cdots\circ
f_{\alpha_{m}}$. In particular, if $f_{\alpha}\in\mathscr{R}$, we say
$S=\langle f_{\alpha}\rangle$ a rational semigroup and if
$f_{\alpha}\in\mathscr{E}$, we say $S=\langle f_{\alpha}\rangle$ a
transcendental semigroup.

A semigroup generated by finitely many holomorphic functions
$f_{i},(i=1,2,\ldots,\\\ n)$ is called finitely generated holomorphic
semigroup. We write $S=\langle f_{1},f_{2},\ldots,f_{n}\rangle$. If $S$ is
generated by only one holomorphic function $f$, then $S$ is cyclic semigroup.
We write $S=\langle f\rangle$. In this case, each $g\in S$ can be written as
$g=f^{n}$, where $f^{n}$ is the nth iterates of $f$ with itself. Note that in
our study of semigroup dynamics, we say $S=\langle f\rangle$ a trivial
semigroup.

The following result will be clear from the definition of holomorphic
semigroup. It shows that every element of holomorphic semigroup can be written
as finite composition of the sequence of $f_{\alpha}$

######  Proposition 2.1.

Let $S=\langle f_{\alpha}\rangle$ be an arbitrary holomorphic semigroup. Then
for every $f\in S$, $f^{m}$(for all $m\in\mathbb{N}$) can be written as
$f^{m}=f_{\alpha_{1}}\circ f_{\alpha_{2}}\circ f_{\alpha_{3}}\circ\cdots\circ
f_{\alpha_{p}}$ for some $p\in\mathbb{N}$.

This proposition 2.1 tells us that it need not to apply same map over and over
again in holomorphic semigroup dynamics. But instead, we may start with family
of maps and we consider dynamics over iteratively defined composition of
sequence of maps.

Next, we define and discuss some special collection and sequences of
holomorphic functions. Note that all notions of convergence that we deal in
this paper will be with respect to the Euclidean metric on the complex plane
$\mathbb{C}$ or spherical metric on the Riemann sphere $\mathbb{C}_{\infty}$.

The family $\mathscr{F}$ of complex analytic maps forms a normal family in a
domain $D$ if given any composition sequence $(f_{\alpha})$ generated by the
member of $\mathscr{F}$, there is a subsequence $(f_{\alpha_{k}})$ which is
uniformly convergent or divergent on all compact subsets of $D$. If there is a
neighborhood $U$ of the point $z\in\mathbb{C}$ such that $\mathscr{F}$ is
normal family in $U$, then we say $\mathscr{F}$ is normal at $z$. If
$\mathscr{F}$ is a family of members from the semigroup $S$, then we simply
say that $S$ is normal in the neighborhood of $z$ or $S$ is normal at $z$.

Let $f$ be a holomorphic map. We say that $f$ iteratively divergent at
$z\in\mathbb{C}$ if
$f^{n}(z)\rightarrow\alpha\;\textrm{as}\;n\rightarrow\infty$, where $\alpha$
is an essential singularity of $f$. A sequence $(f_{k})_{k\in\mathbb{N}}$ of
holomorphic maps is said to be iteratively divergent at $z$ if
$f_{k}^{n}(z)\to\alpha_{k}\;\ \text{as}\;\ n\to\infty$ for all
$k\in\mathbb{N}$, where $\alpha_{k}$ is an essential singularity of $f_{k}$
for each $k$. Semigroup $S$ is iteratively divergent at $z$ if
$f^{n}(z)\rightarrow\alpha_{f}\;\textrm{as}\;n\rightarrow\infty$, where
$\alpha_{f}$ is an essential singularity of each $f\in S$. Otherwise, a
function $f$, sequence $(f_{k})_{k\in\mathbb{N}}$ and semigroup $S$ are said
to be iteratively bounded at $z$.

##  3\. Fatou set, Julia set and Escaping set

In classical complex dynamics, each of Fatou set, Julia set and escaping set
are defined in two different but equivalent ways. In first definition, Fatou
set is defined as the set of normality of the iterates of given function,
Julia set is defined as the complement of the Fatou set and escaping set is
defined as the set of points that goes to essential singularity under the
iterates of given function. The second definition of Fatou set is given as a
largest completely invariant open set and Julia set is given as a smallest
completely invariant close set whereas escaping set is a completely invariant
non-empty neither open nor close set in $\mathbb{C}$. Each of these
definitions can be naturally extended to the settings of holomorphic semigroup
$S$ but extension definitions are not equivalent. Based on above first
definition (that is, on the Fatou-Julia-Eremenko theory of a complex analytic
function), the Fatou set, Julia set and escaping set in the settings of
holomorphic semigroup are defined as follows.

######  Definition 3.1 (Fatou set, Julia set and escaping set).

Fatou set of the holomorphic semigroup $S$ is defined by

$F(S)=\\{z\in\mathbb{C}:S\;\ \textrm{is normal in a neighborhood of}\;\ z\\}$

and the Julia set $J(S)$ of $S$ is the compliment of $F(S)$. If $S$ is a
transcendental semigroup, the escaping set of $S$ is defined by

$I(S)=\\{z\in\mathbb{C}:S\;\text{is iteratively divergent at}\;z\\}$

We call each point of the set $I(S)$ by escaping point.

It is obvious that $F(S)$ is the largest open subset (of $\mathbb{C}$ or
$\mathbb{C}_{\infty}$) on which the family $\mathscr{F}$ in $S$ (or semigroup
$S$ itself) is normal. Hence its compliment $J(S)$ is a smallest closed set
for any semigroup $S$. Whereas the escaping set $I(S)$ is neither an open nor
a closed set (if it is non-empty) for any semigroup $S$. Any maximally
connected subset $U$ of the Fatou set $F(S)$ is called a Fatou component.

If $S=\langle f\rangle$, then $F(S),J(S)$ and $I(S)$ are respectively the
Fatou set, Julia set and escaping set in classical complex dynamics. In this
situation we simply write: $F(f),J(f)$ and $I(f)$.

The main motivation of this paper comes from seminal work of Hinkkanen and
Martin [5] on the dynamics of rational semigroup and the extension study of K.
K. Poon [8] to the dynamics of transcendental semigroup. Both of them
naturally generalized classical complex dynamics to the dynamics of the
sequence of different functions by means of composition. Another motivation of
studying escaping set of transcendental semigroup comes from the work of
Dinesh Kumar and Sanjay Kumar[6, 7] where they defined escaping set and
discussed how far escaping set of classical transcendental dynamics can be
generalized to semigroup dynamics.

The fundamental contrast between classical complex dynamics and semigroup
dynamics appears by different algebraic structure of corresponding semigroups.
In fact, non-trivial semigroup (rational or transcendental) need not be, and
most often will not be abelian. However, trivial semigroup is cyclic and
therefore abelian. As we discussed before, classical complex dynamics is a
dynamical study of trivial (cyclic) semigroup whereas semigroup dynamics is a
dynamical study of non-trivial holomorphic semigroup.

The following immediate result holds good from definition 3.1 of escaping set.

######  Theorem 3.1.

Let $S$ be a transcendental semigroup and let $z\in\mathbb{C}$ is an escaping
point under $S$. Then every sequence $(g_{k})_{k\in\mathbb{N}}$ in $S$ is
iteratively divergent at $z$ and every subsequence of
$(g_{k})_{k\in\mathbb{N}}$ is also iteratively divergent at $z$.

The following characterization of escaping set will be clear from the
definition 3.1 of escaping set and proposition 2.1, which can be an
alternative definition.

######  Theorem 3.2.

If a complex number $z\in\mathbb{C}$ is escaping point of any transcendental
semigroup $S$, then every sequence in $S$ has a subsequence which diverges to
$\infty$ at $z$.

On the basis of the theorem 3.2, we can say that our definition 3.1 of
escaping set is more general than that of the definition of Dinesh Kumar and
Sanjay Kumar [6, Definition 2.1]. That is, our definition of escaping set
implies the definition of Dinesh Kumar and Sanjay Kumar.

##  4\. Basic Comparison of classical and holomorphic semigroup dynamics

The following immediate relations hold for any $f\in S$ from the definition
3.1. Indeed, it shows certain connection between classical complex dynamics
and semigroup dynamics.

######  Theorem 4.1.

Let $S$ be a semigroup. Then

  1. (1)

$F(S)\subset F(f)$ for all $f\in S$ and hence $F(S)\subset\bigcap_{f\in
S}F(f)$.

  2. (2)

$J(f)\subset J(S)$ for all $f\in S$.

  3. (3)

$I(S)\subset I(f)$ for all $f\in S$ and hence $I(S)\subset\bigcap_{f\in
S}I(f)$ in the case of transcendental semigroup $S$.

Hinkkanen and Martin proved the following results ([5, Lemma 3.1 and Corollary
3.1]).

######  Theorem 4.2.

Let $S$ be a rational semigroup. Then Julia set $J(S)$ is perfect and
$J(S)=\overline{\bigcup_{f\in S}J(f)}$

K. K. Poon proved the following results ([8, Theorems 4.1 and 4.2]).

######  Theorem 4.3.

Let $S$ be a transcendental semigroup. Then Julia set $J(S)$ is perfect and
$J(S)=\overline{\bigcup_{f\in S}J(f)}$

From the theorem 4.1 ((1) and (3)), we can say that the Fatou set and the
escaping set may be empty. For example, the escaping set of semigroup
$S=\langle f,g\rangle$ generated by functions $f(z)=e^{z}$ and $g(z)=e^{-z}$
is empty (the particular function $h=g\circ f^{k}\in S$ (say) is iteratively
bounded at any $z\in I(f)$). We know that Fatou set may be empty but escaping
set is non-empty in classical complex dynamics. This is a contrast feature of
escaping set in classical complex dynamics and semigroup dynamics. From the
same theorem part (2), we can say that in classical and semigroup dynamics,
Julia set is non-empty.

Dinesh Kumar and Sanjay Kumar [6, Theorem 2.5] have identified the following
transcendental semigroup $S$, where $I(S)$ is an empty set.

######  Theorem 4.4.

The transcendental semigroup $S=\langle f_{1},\;f_{2}\rangle$ generated by two
functions $f_{1}$ and $f_{2}$ from respectively two parameter family
$\\{e^{-z+\gamma}+c\;\text{where}\;\gamma,c\in\mathbb{C}\;\text{and}\;Re(\gamma)<0,\;Re(c)\geqslant
1\\}$ and
$\\{e^{z+\mu}+d,\;\text{where}\;\mu,d\in\mathbb{C}\;\text{and}\;Re(\mu)<0,\;Re(d)\leqslant-1\\}$
of functions has empty escaping set $I(S)$

###### Proof.

Under the conditions stated in the theorem, for any function $f$ from the
first family we have
$I(f)\subset\\{z\in\mathbb{C}:Rez<0,(4k-3)\frac{\pi}{2}<Imz<(4k-1)\frac{\pi}{2},k\in\mathbb{Z}\\}$
(see for instance [6, Lemma 2.2]) and for any function $g$ from the second
family we have
$I(g)\subset\\{z\in\mathbb{C}:Rez>0,(4k-1)\frac{\pi}{2}<Imz<(4k+1)\frac{\pi}{2},k\in\mathbb{Z}\\}$(see
for instance [6, Lemma 2.3]). From theorem 4.1, for any $f,\;g\in S=\langle
f_{1},\;f_{2}\rangle$, we have $I(S)\subset I(f)\cap I(g)=\emptyset$. ∎

There are several transcendental semigroups where escaping set
$I(S)\neq\emptyset$. The following examples of Dinesh Kumar and Sanjay Kumar
[6, Examples 2.6 and 2.7] are better to mention here.

######  Example 4.1.

Let $S=\langle e^{\lambda z},e^{s\lambda z}+2\pi i/\lambda\rangle$ for all
$\lambda\in\mathbb{C}-\\{0\\}$ and $s\in\mathbb{N}$. Then
$I(S)=I(f)\neq\emptyset$.

######  Example 4.2.

Let $S=\langle f,g\rangle$, where $f(z)=\lambda\sin z$
($\lambda\in\mathbb{C}-\\{0\\}$) and $g(z)=f^{n}+2\pi$ for all
$n\in\mathbb{N}$. Then $I(S)=I(f)\neq\emptyset$.

If escaping set $I(S)\neq\emptyset$, then Eremenko’s result $\partial
I(f)=J(f)$ [4] of classical transcendental dynamics can be generalized to
semigroup settings. The following results is due to Dinesh Kumar and Sanjay
Kumar [6, Lemma 4.2 and Theorem 4.3] which yield the generalized answer in
semigroup settings.

######  Theorem 4.5.

Let $S$ be a transcendental semigroup such that $I(S)\neq\emptyset$. Then

  1. (1)

$int(I(S))\subset F(S)\;\ \text{and}\;\ ext(I(S))\subset F(S)$, where $int$
and $ext$ respectively denote the interior and exterior of $I(S)$.

  2. (2)

$\partial I(S)=J(S)$, where $\partial I(S)$ denotes the boundary of $I(S)$.

###### Proof.

  1. (1)

We refer for instance lemma 4.2 of [6].

  2. (2)

The facts $int(I(S))\subset F(S)\;\ \text{and}\;\ ext(I(S))\subset F(S)$ yield
$J(S)\subset\partial I(S)$. The fact $\partial I(S)\subset J(S)$ is obvious.

∎

From this theorem 4.5, the fact $J(S)\subset\overline{I(S)}$ follows
trivially. If $I(S)\neq\emptyset$, then we prove the following result which is
a generalization of Eremenko’s result $I(f)\cap J(f)\neq\emptyset$ [4, Theorem
2] of classical transcendental dynamics to holomorphic semigroup dynamics.

######  Theorem 4.6.

Let $S$ be a transcendental semigroup such that $F(S)$ has a multiply
connected component. Then $I(S)\cap J(S)\neq\emptyset$

Following result of Baker [1, Theorem 3.1] is better to worth mention.

######  Lemma 4.1.

Let $f$ be a transcendental entire function and $U$ be a multiply connected
component of $F(f)$. Then $f^{n}(z)\to\infty$ locally uniformly on $U$.

###### Proof of the Theorem 4.6.

Suppose $F(S)$ has a multiply connected component $U$. Then by theorem 4.1
(1), $U$ is also multiply connected component of $F(f)$ for each $f\in S$ and
by lemma 4.1, for each $f\in S$, $f^{n}(z)\to\infty$ locally uniformly on $U$
and also that $f^{n}(z)\to\infty$ on $\partial U$. It follows by normality
(that is, by theorem 3.2) that every sequence in $S$ has a subsequence which
diverges to $\infty$ locally uniformly on $U$ and $\partial U$. This proves
that $f^{n}(z)\to\infty$ for all $z\in U$ and $z\in\partial U$ for each $f\in
S$ and hence by theorem 4.1 (3) , $U\subset I(S)$. Since $\partial U\subset
J(f)$ for all $f\in S$. By theorem 4.1 (2), $\partial U\subset J(S)$. This
proves that $I(S)\cap J(S)\neq\emptyset$.

∎

The one of the most important result of classical complex dynamics is either
$J(f)=\mathbb{C}\;\text{or}\;\mathbb{C}_{\infty}$ or $J(f)$ has empty interior
for any holomorphic map $f$ on $\mathbb{C}\;\text{or}\;\mathbb{C}_{\infty}$
(see [3, Lemma 3]). There are lot of examples of transcendental entire
functions and rational functions whose Julia set is entire complex plane or
extended complex plane. Note that while
$J(f)=\mathbb{C}\;\text{or}\;\mathbb{C}_{\infty}$ is possible for some
holomorphic map $f$, we always have
$F(f)\neq\mathbb{C}\;\text{or}\;\mathbb{C}_{\infty}$. On the other hand, the
analogous result is not hold in semigroup dynamics. Hinkkanen and Martin [5,
Example-1] provided the following example that shows that Julia set of a
rational semigroup $S$ may have non-empty interior even if
$J(S)\neq\mathbb{C}_{\infty}$.

######  Example 4.3.

Rational semigroup $S=\langle z^{2},z^{2}/a\rangle$, where
$a\in\mathbb{C},|a|>1$ has Fatou set $F(S)=\\{z:|z|<1\;\text{or}\;|z|>|a|\\}$
and Julia set $J(S)=\\{z:1\leqslant|z|\leqslant|a|\\}$.

Let $U$ be a component of Fatou set $F(f)$. Then $f(U)$ is contained in some
component $V$ of $F(f)$. Note that if $f$ is rational map then $V=f(V)$. If
$f$ is transcendental, then it is possible that $V\neq f(U)$. Let us recall
the following result of Bergweiler and Rohde [3] of classical complex
dynamics.

######  Theorem 4.7.

If $f$ is entire, then $V-f(U)$ contains at most one point which an asymptotic
value of $f$.

The following example of Huang Zhigang [10, Example 2] shows that above result
(theorem 4.7) can not be preserved for general semigroup dynamics. This is a
contrast between classical complex dynamics and semigroup dynamics.

######  Example 4.4.

Let $S=\langle z^{n},az^{n}\rangle$, where $n>2$ and $|a|>1$. The Fatou set
$F(S)$ contains following components

$U=\Bigg{\\{}\sqrt[n]{\dfrac{1}{|a|}}<|z|<\sqrt[n]{\sqrt[n-1]{\dfrac{1}{|a|}}}\Bigg{\\}}\;\;\text{and}\;\;V=\\{|z|>1\\}.$

For a function $f(z)=az^{n}$ in semigroup $S$, $f(U)\subset V$ and $V-f(U)$ is
an unbounded domain.

######  Definition 4.1.

Let $S$ be a holomorphic semigroup. We define the backward orbit of any
$z\in\mathbb{C}$ (or $\mathbb{C}_{\infty}$) by

$O^{-}(z)=\\{w\in\mathbb{C}_{\infty}:\text{there exists}\;f\in S\;\text{such
that}\;f(w)=z\\}$

and the exceptional set of $S$ is defined by

$E(S)=\\{z\in\mathbb{C}_{{}_{\infty}}:O^{-}(z)\;\text{is finite}\\}$

Note that if $S$ finitely generated rational semigroup, then $E(S)\subset
F(S)$, otherwise we can not assert it. For example ([10, example 1]),
semigroup $S=\langle f_{m}\rangle$, where
$f_{m}(z)=a^{m}z^{n},m\in\mathbb{N},n\geqslant 2$ and $|a|>1$, is an
infinitely generated polynomial semigroup. Then, $E(S)=\\{0,\infty\\}$. It is
easy to see that 0 is a limit point of
$J(f_{m})=\\{|z|=|a|^{\frac{-m}{n-1}}\\}$, and hence $0\in J(S)$. So, in the
case of finitely generated rational semigroup $S$, we always have $E(S)\subset
F(S)\subset F(f)$ for any $f\in S$. Hence $E(S)$ contains at most two points.
However, if $S$ finitely generated transcendental semigroup, then we can not
assert $E(S)\subset F(S)$ in general because for a transcendental function, it
is difficult to determine whether Fatou exceptional value belongs Fatou set or
Julia set. For example, 0 is the Fatou exceptional value of $f(z)=e^{\lambda
z}$. It is known in classical complex dynamics that $0\in J(f)$ if
$\lambda>1/e$ and $0\in F(f)$ if $\lambda<1/e$. Poon and Yang [9] gave the
following characterization whether a Fatou exceptional value belongs to Fatou
set or Julia set.

######  Theorem 4.8.

Let $f$ is transcendental entire function. If $F(f)$ has no unbounded
component, then Fatou exceptional value always belongs to Julia set.

So, in the case of finitely generated transcendental semigroup $S$, if
$E(f)\subset F(f)$ for all $f\in S$, then we can say $E(S)\subset F(S)\subset
F(f)$ for any $f\in S$. Hence $E(S)$ contains at most one point. This fact is
a generalization of classical complex dynamics to semigroup dynamics and so it
is a nice connection between these two types of dynamics. Huang Zhigang [10,
Proposition 1] proved the following result which also shows a connection
between classical complex dynamics and semigroup dynamics.

######  Theorem 4.9.

If $z\notin E(S)$, then $J(S)\subseteq\overline{O^{-}(z)}$.

## References

  * [1] Baker, I. N.: Wandering domains in the iteration of entire functions, Proc. London Math. Soc., 49 (1984), 563-576.
  * [2] Bergweiler, W.: Iteration of meromorphic functions, Bull. (New Series) Amer. Math. Soc. Vol. 29, Number 2, (1993), 131-188.
  * [3] Bergweiler, W. and Rohde, S.: Omitted values in the domains of normality, Proc. Amer. Math. Soc. 123(1995), 1857-1858.
  * [4] Eremenko, A.: On the iterations of entire functions, Dynamical System and Ergodic Theory, Banach Center Publication Volume 23, Warsaw, Poland, (1989).
  * [5] Hinkkanen, A. and Martin, G.J.: The dynamics of semigroups of rational functions- I, Proc. London Math. Soc. (3) 73, 358-384, (1996).
  * [6] Kumar, D. and Kumar, S.: The dynamics of semigroups of transcendental entire functions-II, arXiv: 1401.0425 v3 (math.DS), May 22, 2014.
  * [7] Kumar, D. and Kumar, S.: Escaping set and Julia set of transcendental semigroups, arXiv:141.2747 v3 (math. DS) October 10, 2014.
  * [8] Poon, K.K.: Fatou-Julia theory on transcendental semigroups, Bull. Austral. Math. Soc. Vol- 58(1998) PP 403-410.
  * [9] Poon, K.K. and Yang, C. C.: Relations between Fatou exceptional value and component of the Fatou Set, Preprint.
  * [10] Zhigang, H.: The dynamics of semigroup of transcendental meromorphic functions, Tsinghua Science and Technology, ISSN 1007-0214 18/21, Vol 9, Number 4 (2004), 472-474.


# ACTER: Diverse and Actionable Counterfactual Sequences for Explaining and
Diagnosing RL Policies

Jasmina Gajcin<EMAIL_ADDRESS>0000-0002-8731-1236 and  Ivana Dusparic
<EMAIL_ADDRESS>0000-0003-0621-5400 Trinity College DublinDublinIreland

(2024)

###### Abstract.

Understanding how failure occurs and how it can be prevented in reinforcement
learning (RL) is necessary to enable debugging, maintain user trust, and
develop personalized policies. Counterfactual reasoning has often been used to
assign blame and understand failure by searching for the closest possible
world in which the failure is avoided. However, current counterfactual state
explanations in RL can only explain an outcome using just the current state
features and offer no actionable recourse on how a negative outcome could have
been prevented. In this work, we propose ACTER (Actionable CounTerfactual
Sequences for Explaining Reinforcement Learning Outcomes), an algorithm for
generating counterfactual sequences that provides actionable advice on how
failure can be avoided. ACTER investigates actions leading to a failure and
uses the evolutionary algorithm NSGA-II to generate counterfactual sequences
of actions that prevent it with minimal changes and high certainty even in
stochastic environments. Additionally, ACTER generates a set of multiple
diverse counterfactual sequences that enable users to correct failure in the
way that best fits their preferences. We also introduce three diversity
metrics that can be used for evaluating the diversity of counterfactual
sequences. We evaluate ACTER in two RL environments, with both discrete and
continuous actions, and show that it can generate actionable and diverse
counterfactual sequences. We conduct a user study to explore how explanations
generated by ACTER help users identify and correct failure.

Reinforcement Learning, Counterfactual Explanations, Actionable Explanations

††copyright: acmcopyright††journalyear: 2024††ccs: Human-centered computing
HCI design and evaluation methods††ccs: Human-centered computing User
studies††ccs: Computing methodologies Reinforcement learning

##  1\. Introduction

Reinforcement learning (RL) algorithms are being developed for many sequential
decision-making tasks in various fields such as robotics, medicine, and
autonomous driving (Li, 2017; Kiran et al., 2021; Coronato et al., 2020).
However, these algorithms are often opaque, making their decisions difficult
to understand and explain (Puiutta and Veith, 2020; Wells and Bednarz, 2021).
The lack of algorithm transparency makes such applications less trustworthy
and stands in the way of their successful integration with real-life high-
stakes applications.

Of particular importance is understanding the reasons behind the failure of RL
agents, as a necessary step for debugging, personalization, and developing
trustworthy RL algorithms. Explanations about why failure has occurred have
been shown to help recover user trust in the system (Tolmeijer et al., 2020;
Edmonds et al., 2019; Lyons et al., 2023). Additionally, detecting and
correcting failure is necessary for debugging and correcting mistakes in the
model. From the perspective of personalization, it is important to note that
the definition of failure can depend on the end-user. For example, for an
energy-conscious user of an autonomous vehicle, a failure is not only crashing
the vehicle but also wasting fuel. Understanding why unwanted outcomes occur
and how they can be avoided can help expert users know when to take over the
control from the black-box model and manually control execution. Similarly, to
adjust a model to their preferences, non-expert users need to understand how
unwanted outcomes occurred and how they could have been prevented.

Figure 1. Lack of actionability of current counterfactual explanations. Left:
ego vehicle (green) crashes with another vehicle (blue). Center and right: two
possible counterfactuals stating that had the agent been in a different lane
it would not have crashed. However, only the right counterfactual is
actionable, as an agent could have only switched to the right lane without
crashing with another car. Without examining the agent’s execution history it
is impossible to see how the crash could have been prevented.

Numerous explainability techniques have been developed to understand the
outcomes of RL policies. An outcome of a policy can be attributed to different
factors, such as state features, previous states and actions or the agent’s
goals and objectives. Saliency maps (Greydanus et al., 2018; Yang et al.,
2018; Atrey et al., 2019; Itaya et al., 2021) and feature attribution methods
(Puri et al., 2019; Pálsson and Björnsson, 2023) assign importance to state
features based on their contribution to choosing an action. Previous work also
offers methods for identifying important states in the agent’s execution
history, such as states in which the agent is uncertain of its actions, or
states in which taking different actions leads to substantially different
outcomes (Amir and Amir, 2018; Sequeira and Gervasio, 2020). Furthermore, an
outcome can be interpreted by inspecting the agent’s reward function
(Juozapaitis et al., 2019) or its future objectives (Madumal et al., 2020a,
b). While the majority of these approaches focus on identifying factors
responsible for a particular outcome, few works offer actionable recourse on
how to change these factors to avoid a negative outcome. Counterfactual state
explanations (Olson et al., 2021; Huber et al., 2023) address this shortcoming
by generating an alternative state in which a negative outcome would have been
avoided. Ideally, they should offer users insight into not only which features
are responsible for the outcome but also how they can be changed to avoid a
negative outcome. However, these explanations only consider state features and
are inherently not actionable in the RL context as they do not show how those
features can be changed using sequences of RL actions. For example, consider
two counterfactual state explanations for a car crash in Figure 1. While both
counterfactual states show that the crash could have been avoided had the
agent changed lanes, agent could not have switched to the left lane due to
another vehicle occupying it, making the counterfactual non-actionable.
Without considering the agent’s previous actions it is impossible to
distinguish between actionable and non-actionable counterfactuals. Lack of
actionability can render the counterfactual useless and decrease user trust.

In this paper, we propose ACTER (Actionable CounTerfactual Sequences for
Explaining Reinforcement Learning Outcomes), an approach that explores agent’s
past decisions to generate actionable counterfactual explanations for
diagnosing and correcting undesirable outcomes in RL. Given a sequence of
actions that failed, our goal is to find a counterfactual sequence of actions,
as similar as possible to the original one, but which does not fail. Our
approach can be used not only to explain and correct failure but also any
other outcome of interest that users might want to better understand or
prevent. We start by defining five counterfactual properties that describe the
desiderata the counterfactual sequence should satisfy. Specifically, we ensure
that the counterfactual sequence is as similar as possible to the original one
and that it avoids the negative outcome with high certainty. To find the most
suitable counterfactual according to the defined properties we use an
evolutionary algorithm. Additionally, ACTER can generate a set of diverse
counterfactual sequences that can prevent a negative outcome. Diverse
counterfactuals are necessary for enabling personalization, as different users
might have different preferences on how to best correct failure (Mothilal et
al., 2020; Dandl et al., 2020). For example, while all users can agree that a
car crash is an unwanted outcome, some of them might prefer slowing down while
others might want to change lanes to prevent it. Offering diverse
counterfactuals enables users to better understand all possible ways to
prevent an unwanted outcome, and choose the one that best suits their needs.

We evaluate ACTER in two environments, both with discrete and continuous
action spaces and compare it to HIGHLIGHTS (Amir and Amir, 2018) and
interestingness elements algorithms (Sequeira and Gervasio, 2020), which offer
metrics for identifying the most important elements in the agent’s execution.
Firstly, we evaluate ACTER and the baselines on counterfactual properties to
ensure that generated counterfactual sequences are similar to the original
ones and successfully avoid failure even in stochastic environments. Secondly,
we evaluate the diversity of generated counterfactual sequences to ensure that
ACTER offers versatile ways of preventing failure. To measure diversity, we
introduce three diversity metrics that can also be used to benchmark future
approaches for generating counterfactual sequences. Finally, we evaluate ACTER
in a user study. While previous work emphasizes the importance of actionable
recourse (Karimi et al., 2021; Upadhyay et al., 2021; Poyiadzi et al., 2020),
user studies have reported ambiguous results, finding that counterfactuals
might lead to an increase in false confidence in supervised learning tasks
compared to non-actionable explanations (Celar and Byrne, 2023). We explore
this question in RL and conduct a user study to explore the effect of
counterfactual sequences generated by ACTER on users’ ability to diagnose and
correct failure. We conduct the study using a highway driving task and compare
counterfactual sequences generated by ACTER to non-actionable explanations.
The main contributions of our work are as follows:

  * •

We define five desired properties for evaluating different counterfactual
sequences that ensure that they are easy to obtain and avoid the negative
outcome with high certainty.

  * •

We propose ACTER, an algorithm for generating actionable, diverse
counterfactual sequences for RL tasks

  * •

We define three benchmark metrics for evaluating the diversity of different
counterfactual sequences in RL.

  * •

We compare ACTER to state-of-the-art explanation techniques in their ability
to robustly correct unwanted outcomes with minimal changes, the diversity of
generated explanations, as well as in a user study to show the effect of
counterfactual explanations on users’ ability to identify and correct failure.

The full code for ACTER, baselines, and evaluation is available at:
https://github.com/jas97/temporalSimCFs/tree/acter.

##  2\. Related Work

Understanding why an unwanted outcome occurred is of utmost importance for
developing, debugging and personalizing RL policies. An unwanted outcome can
have many definitions, such as an unwanted action (e.g. sudden breaking) or a
function of state features (e.g high speed) that a user would like to avoid.
Various factors, such as state features, actions, agent’s goals or objectives,
can be used to explain an unwanted outcome.

The majority of current research in explainable RL uses current state features
to explain an outcome. For example, saliency maps identify parts of images
that influenced the agent’s decision in visual tasks (Greydanus et al., 2018;
Yang et al., 2018; Atrey et al., 2019; Itaya et al., 2021). Similarly, feature
attribution methods assign importance to individual features based on their
contribution to action choice in a state (Puri et al., 2019; Pálsson and
Björnsson, 2023). Previous actions and states can also explain the agent’s
outcome. EDGE (Guo et al., 2021) provides global explanations that identify
specific states in the episode that contributed to its outcome. EDGE trains an
additional self-interpretable model to approximate the black-box policy and
extract explanations for visual tasks. HIGHLIGHTS (Amir and Amir, 2018)
identifies critical states as those where taking different actions can lead to
substantially different outcomes. Similarly, “interestingness metrics” can be
used to identify important states as those that are (in)frequently visited, in
which the agent is (un)certain of its decision, or those where the agent’s
estimate of Q function reaches a local extremum (Sequeira and Gervasio, 2020).
The agent’s goal and objectives have been used to interpret outcomes.
Juozapaitis et al. (2019) propose a reward decomposition approach that can
interpret a decision by identifying which reward components contributed to it.
Yau et al. (2020) explains the agent’s decision by providing the intended
outcome as a motivation for the agent’s action. Similarly, Madumal et al.
(2020a, b) envision a causal chain of future events to explain a decision.
While these approaches can attribute the outcome to a factor in an agent’s
execution, they do not provide an actionable way to change them to avoid
negative outcomes. In contrast, ACTER provides actionable advice on how to
change an agent’s past actions and prevent unwanted outcomes.

Understanding how an outcome could have been prevented often requires
counterfactual reasoning (Byrne, 2019). Counterfactual state explanations
employ this reasoning to explain why a certain action was chosen by showing an
alternative state where a different action would be taken. Olson et al. (2021)
propose an approach based on generative modeling and generate realistic
counterfactuals for Atari environments. Similarly, GANterfactual-RL is a
generative model-agnostic approach that uses domain transfer architecture to
generate counterfactual states (Huber et al., 2023). Domains are determined by
actions and contain states in which a specific action would be chosen by the
agent. Understanding what is required for the action to change requires
translating the state from its original to an alternative domain. While these
explanations show what the state features should look like to prevent a
negative outcome, they do not offer actionable recourse in terms of RL actions
on how to actualize these changes. In contrast, our work provides actionable
recourse as sequences of actions that can be used to understand and prevent
failure in RL tasks.

Actionable recourse has been explored in previous RL work. RACCER offers
actionable advice on how to change features to elicit an action change (Gajcin
and Dusparic, 2023). However, RACCER generates the counterfactual sequence by
exploring the agent’s future goals and not its previous behavior. In contrast,
ACTER explores the agent’s past and proposes actionable changes to prevent a
negative outcome. Previous work also offers algorithms for finding
counterfactual sequences of actions based on causal inference (Tsirtsis et
al., 2021; Tsirtsis and Gomez-Rodriguez, 2023). However, these algorithms are
limited to discrete actions and do not offer diverse explanations.
Additionally, they can only define failure when it is represented by a low
reward and require a Lipschitz continuity assumption for the reward function
in continuous state spaces (Tsirtsis and Gomez-Rodriguez, 2023). In contrast,
ACTER applies to tasks with both discrete and continuous action spaces, allows
a wide range of failure definitions, and generates a set of diverse
counterfactual sequences. While previous work (Tsirtsis and Gomez-Rodriguez,
2023; Tsirtsis et al., 2021) explores algorithmic challenges of generating
counterfactual sequences, we focus on defining criteria that make these
sequences useful for the users.

##  3\. ACTER

In this section, we propose ACTER, our approach to explaining RL policies.
ACTER can be used for diagnosing and correcting failure in the broadest
context, either defined as an unwanted action or a function of state features.
ACTER is model-agnostic and can be used to explain the behavior of any RL
policy. In the remainder of this section, we define the problem of
counterfactual sequence generation (Section 3.1). We identify the desiderata a
counterfactual sequence should satisfy in Section 3.2 and provide an algorithm
for finding the optimal counterfactual sequence in Section 3.3.

###  3.1. Definitions

####  3.1.1. Failure Function

We define failure as a binary function over states $F(s):S\rightarrow[0,1]$:

(1) $F(s)=\begin{cases}1&\text{if failure occurs in state s}\\\
0&\text{otherwise}\end{cases}$

We assume $F(s)$ is defined beforehand. This can be done by the developer who
would like to debug unwanted behavior in a policy, such as an autonomous car
crash. Failure function can also be defined by a non-expert user trying to
obtain a personalized policy by correcting what they perceive as negative
outcomes, such as a car driving too fast for their preferences. Similarly,
failure states can also be automatically inferred by isolating states that
achieve local minima in the reward function. We leave different
interpretations of failure for future work, and in this work consider more
obvious failure cases, such as a car crash in a highway driving environment.

####  3.1.2. Counterfactual Sequence

Given a failure function $F(s)$ and a state $x_{n}$ where $F(x_{n})=1$ we
explore the sequence of factual actions $A=[a_{n-k},\dots,a_{a_{n-1}}]$
starting in $x_{n-k}$ and leading to $x_{n}$. We limit our search to previous
$k$ actions, where horizon $k$ depends on the specific task. To explain a car
crash, we might focus only on a small number of previous steps, while to
diagnose failure in a medical scenario we might need to look back months or
even years. ACTER searches for a counterfactual sequence of actions
$A^{*}=[a^{*}_{n-k},\dots,a^{*}_{n-1}]$ that starts in $x_{n-k}$ and
successfully avoids failure.

####  3.1.3. Stochastic Environment Configurations

Counterfactual thinking requires rewriting the past. While we can identify why
a car crash occurred and how it could have been prevented in a specific
scenario, due to the stochasticity of the world, it is unlikely that the exact
same conditions will be repeated. Counterfactual sequences should be useful to
users not only in singular scenarios but in a wider range of similar
situations. Thus, we need to distinguish between different types of stochastic
conditions. Firstly, we define $\mathcal{P}$ as the set of stochastic
processes $P_{i}$ outside of the agent’s control:

(2) $\mathcal{P}=\cup_{i\in I}{P^{i}}$

For example, these can describe weather conditions or the behavior of other
vehicles in an autonomous driving scenario. We define a stochastic
configuration $P_{T}$ for a trajectory $T$ as an instantiation of all
stochastic processes along $T$:

(3) $P_{T}=\cup_{i\in I}P^{i}_{T}$

Intuitively, $P_{T}$ gathers the specific values that the stochastic processes
took along the trajectory $T$. These can be specific weather conditions and
temperature on the days of the trajectory or the specific behavior of all
other cars on the road in an autonomous driving scenario. Knowing $P_{T}$ for
any trajectory would allow one to effectively look into the past and repeat
the same trajectory with all stochastic processes fixed. In this work, we
assume access to RL environment, meaning we can observe the stochastic
processes $P_{T}$ that shape the environment. However, if the full environment
is not present, it can be approximated from an environment model, constructed
using simulations or offline data. We leave the offline scenario for future
work and explore its further implications in Section 6.

###  3.2. Counterfactual Properties

While there are often multiple ways to prevent a failure, only some of them
might be useful for the user. For example, while technically correct, telling
the user that the car would not have crashed if it had never moved does not
help them prevent this issue in the future. In this section, we introduce five
counterfactual properties that ensure counterfactual sequences are actionable,
require minimal effort to change actions, and avoid failure even in stochastic
environments.

####  3.2.1. Validity

Validity ensures that taking counterfactual actions avoids failure. For a
state $x_{n}$ where the failure occurred ($F(x_{n})=1$), we observe trajectory
$T=[(x_{n-k},a_{n-k}),\dots,(x_{n-1},a_{n-1})]$ with a stochastic
configuration $P_{T}$ which leads to failure. Validity for a counterfactual
sequence of actions $A^{*}=[a^{*}_{n-k},\dots,a^{*}_{n-1}]$ is defined as:

(4) $V(A^{*})=F(A^{*}(x_{n-k})|_{P_{T}})=0$

where $A^{*}(x_{n-k})|_{P_{T}}$ is the state reached by following $A^{*}$ from
$x_{n-k}$ under the stochastic configuration $P_{T}$. This way, we ensure that
in the exact same episode with the same stochastic properties, choosing
$A^{*}$ would have prevented failure.

Figure 2. Stochastic uncertainty and validity evaluation in ACTER in a farming
scenario: In an episode that ends in a plant dying, an alternative sequence of
actions that increases water in the first action is examined. We run
simulations of the episode with an alternative sequence of actions under
different stochastic conditions represented by the weather. Validity indicates
whether the plant would die after the alternative sequence of actions under
the sunny weather conditions present in the original episode. Stochastic
uncertainty is calculated as the probability of the plant surviving over all
simulated weather conditions.

####  3.2.2. Proximity

To minimize user effort in preventing failure we want to ensure that sequences
A and A* are as similar as possible. For example, a counterfactual should not
instruct the user to reduce their speed by a lot to avoid a crash when even a
smaller speed reduction would have been enough. To that end, we define
proximity between the original sequence $A$ and counterfactual sequence
$A^{*}$ as:

(5) $P(A,A^{*})=\begin{cases}\frac{\sum_{i=1}^{k}(a_{n-i}\neq
a^{*}_{n-i})}{k}&\text{if A is discrete}\\\
\frac{\sum_{i=1}^{k}|a_{n-i}-a^{*}_{n-i}|}{k}&\text{if A is
continuous}\end{cases}$

####  3.2.3. Sparsity

Research has shown that users prefer selective explanations, focusing only on
a few most important elements (Miller, 2019). For this reason, we define
sparsity as a metric that minimizes the number of actions that need to be
changed to avoid failure. Given an original sequence of actions
$A=[a_{n-k},\dots,a_{n-1}]$ and a counterfactual sequence
$A^{*}=[a^{*}_{n-k},\dots,a^{*}_{n-1}]$, we define sparsity as:

(6) $S(A,A^{*})=\sum_{i=1}^{k}(a_{n-i}\neq a_{n-i})$

####  3.2.4. Stochastic uncertainty

With validity, we ensure that the counterfactual sequence $A^{*}$ successfully
avoids failure in the specific episode. However, due to the stochasticity in
the environment, the same sequence of actions might not avoid failure if the
stochastic configuration was slightly different. Providing users with
counterfactuals that are valid only for one stochastic configuration and might
fail as a result of a slight change in the environment can lead the users to
lose trust and discourage interaction with the system. Our goal is to provide
counterfactuals that are robust to stochastic processes in the environment and
can avoid the negative outcome with high certainty. For a counterfactual
sequence of actions $A^{*}$ executed from state $x_{n-k}$, we define
stochastic uncertainty as:

(7) $S_{t}(A^{*})=P[F(A^{*}(x_{n-k})|_{W})=0\quad\forall W\in\mathcal{P}]$

where $\mathcal{P}$ is a set of all possible stochastic configurations.
Intuitively, stochastic uncertainty ensures that $A^{*}$ avoids failure with
high certainty regardless of stochastic processes in the environment, such as
the behavior of other actors. A visualization of how stochastic uncertainty is
calculated is shown in Figure 2.

####  3.2.5. Recency

When thinking about counterfactuals, humans prefer to change the most recent
events to change the outcome (Byrne, 2019). Telling the user that the car
crash could have been avoided if agent never entered the street of the crash
is not as useful as showing that a small reduction in speed a few moments
earlier could have prevented failure. For this reason, we introduce recency as
an important metric for diagnosing and correcting failure. For a sequence of
action $A$ leading to failure and a counterfactual sequence $A^{*}$ we define
recency as:

(8) $R(A,A^{*})=\sum_{i=1}^{k}(a_{n-i}\neq a^{*}_{n-i})*\frac{2}{k(k+1)}$

The coefficient $\frac{2}{k(k+1)}$ is chosen so that recency always takes
values between 0 and 1.

###  3.3. Multi-objective Counterfactual Search

Our goal is to generate counterfactual sequences that optimize the defined
counterfactual properties. However, some of these properties contradict each
other and tradeoffs between them have to be made. Additionally, different
users might deem different properties as more important. For example, while
one user might only be interested in highly certain counterfactual sequences
regardless of the number of actions that have to be changed, another might
prefer changing fewer actions even if it comes with a greater risk of failure.
By receiving diverse counterfactual sequences, users can get a better
understanding of the different ways to prevent failure and choose the one that
best fits their preferences.

Previous work often conflates the counterfactual properties into a single
objective function that can be optimized (Huber et al., 2023; Olson et al.,
2021). However, such approaches require fine-tuning of weights corresponding
to different counterfactual properties. Additionally, these approaches only
produce one counterfactual and have to be run multiple times with different
sets of weights to obtain a set of diverse solutions for the counterfactual
properties. In contrast, we use multiobjective optimization which
simultaneously optimizes different objectives and generates a set of
solutions. Specifically, given a sequence of actions
$A=[a_{n-k},\dots,a_{n-1}]$ failing, ACTER generates an alternative sequence
of actions $A^{*}=[a_{n-k}^{*},\dots,a_{n-1}^{*}]$ such that it minimizes the
counterfactual properties defined in Section 3.2:

(9)
$\operatorname*{argmin}_{A^{*}}[1-V(A^{*}),P(A,A^{*}),S(A,A^{*}),1-S_{t}(A^{*}),R(A,A^{*})]$

Algorithm 1 Non-dominated Sorting Genetic Algorithm (NSGA-II) – Main loop

1: Input: initial population $P_{0}$ {Counterfactual sequences sampled from
the neighborhood of failure sequence}

2: constraint set $\mathcal{C}=[V(A,A^{*})]$ {Set of constraints includes only
validity}

3: objective set
$\mathcal{O}=[P(A,A^{*}),S(A,A^{*}),S_{t}(A,A^{*}),R(A,A^{*})]$ {Remaining
counterfactual properties}

4: Parameters: population size $N$, number of generations $G$

5: $Q_{0}$ = generate_child_population($P_{0}$) {Generates child population
using selection, recombination and mutation}

6: $i=0$

7: $P_{i},Q_{i}=P_{0},Q_{0}$

8: while i ¡ G do

9: $R_{i}=P_{i}+Q_{i}$ {Combine parent and child population}

10: $\mathcal{F}_{1},\dots,\mathcal{F}_{t}$ =
fast_nondominated_sorting($R_{i},\mathcal{C},\mathcal{O}$) {Generate sets of
nondominated solutions}

11: $P_{i+1}=\\{\\}$ {Initialize next parent population}

12: $l=1$

13: while —$P_{i+1}|+|F_{l}|<N$ do

14: $P_{i+1}=P_{i+1}\cup\mathcal{F}_{l}$ {Fill $P_{i+1}$ with solutions
starting with the best ones}

15: end while

16: if $|P_{i+1}|<N$ then

17: $C_{dist}$ = calculate_crowding_distance($\mathcal{F}_{i}$) {Calculate
crowding distance for the last nondominated set}

18: $\mathcal{F}^{*}_{l}$ = sort_descending($\mathcal{F}_{l},C_{dist}$) {Sort
descending based on the crowding distance}

19: for all $f\in\mathcal{F}^{*}_{l}$ do

20: $P_{i+1}=P_{i+1}\cup f$ {Include the best solutions from the final front
until size $N$ is reached}

21: if $|P_{i+1}|>=N$ then

22: break

23: end if

24: end for

25: end if

26: end while

We employ NSGA-II (Non-dominated Sorting Genetic Algorithm) (Deb et al.,
2002), an evolutionary algorithm that provides a set of non-dominated
solutions on the Pareto front. This way we can generate a set of diverse
counterfactuals that optimize different counterfactual objectives. We use
validity as a constraint for the algorithm and only consider solutions that
satisfy it. The other four counterfactual properties – proximity, sparsity,
stochastic uncertainty, and recency are used as objectives in the algorithm
and are simultaneously optimized. Additionally, we initialize the search with
a dataset of sequences similar to the original one. We generate this dataset
by applying noise to elements of the original sequence. This way search is
initialized to the neighborhood of the initial sequence. NSGA-II modifies the
initial population through generations while maintaining a front of the most
promising solutions based on the constraints and objectives. In each iteration
$i$, NSGA-II uses selection, recombination and mutation to create a child
population $Q_{i}$ from a parent population $P_{i}$. The parent and child
populations are combined into $R_{i}=P_{i}\cup Q_{i}$ and filtered together to
obtain the next parent population $P_{i+1}$. Firstly, a fast nondominated
sorting algorithm is used to split $R_{i}$ into different fronts of
nondominated solutions $\mathcal{F_{1}},\dots,\mathcal{F}_{t}$, where
$\mathcal{F}_{1}$ contains the best solutions and so on. $P_{t+1}$ is then
populated using fronts starting with the best $\mathcal{F}_{1}$ until size $N$
is reached. Finally, if $F_{l}$ is the last front that can be included, it is
sorted according to the crowding distance. This ensures that in case of
equally good solutions, the one from a less populated area of solution space
is selected providing a diversity of solutions. NSGA-II algorithm, as applied
to ACTER is described in more detail in Algorithm 1.

##  4\. Experimental Evaluation

In this section, we introduce evaluation tasks (Section 4.1) and baselines
(Section 4.2). In Section 4.3 we define evaluation hypotheses and in Section
4.4 we define diversity metrics for sets of counterfactual explanations.

###  4.1. Evaluation Environments

We evaluate ACTER in two complex, multi-objective and stochastic RL tasks –
highway driving and Farm environment.

####  4.1.1. Highway Environment

In the highway environment (Leurent, 2018), the agent is tasked with driving
on a multi-lane highway. The agent can observe the coordinates and speed of
neighboring vehicles and at each step choose one of five discrete actions –
change lane to the left, change lane to the right, speed up, slow down or do
nothing (i.e., continue driving without changing the lane or speed). During
training, agent has to optimize for multiple criteria such as avoiding
collisions, maintaining good speed according to the speed limit, and remaining
in the rightmost lane. Stochasticity in this environment comes from the
behavior of other vehicles on the road which operate independently of the
agent. As a failure in this environment, we choose the event of colliding with
another vehicle. Alternatively, ACTER could be used to explain several
different unwanted outcomes, such as changing a lane from the rightmost one,
exhibiting too high or too slow speed or failing to maintain an appropriate
distance from other vehicles.

####  4.1.2. Farm

Farm-gym (Maillard et al., 2023) is a collection of environments that simulate
diverse agricultural tasks. We evaluate ACTER in a simple Farm environment
that involves growing a tomato plant. At each time step, the agent can choose
to either apply between 1L and 10L of water or to harvest the plant. Positive
rewards are obtained by harvesting a ripe plant, and every time a plant
transitions to a different growth stage. Stochasticity of the environment is
exhibited in the weather conditions, such as temperature, humidity, and rain.
As a failure, we consider the event when the plant dies, which can happen
after being watered too much or too little or after failure to harvest a ripe
plant. However, ACTER could be used to explain various other outcomes of
interest, such as applying too much water or harvesting before the optimal
time.

Table 1. Parameters used for training a DQN black-box model $M$ in highway and
Farm environments.

Parameter Task Highway driving Farm Number of layers $2$ $2$ Nodes in each
layer $[128,128]$ $[128,128]$ Learning rate $5\cdot 10^{-4}$ $9\cdot 10^{-5}$
Training time steps ($N$) $2\cdot 10^{5}$ $2\cdot 10^{4}$

Table 2. Parameters used for generating counterfactual explanations ACTER in
highway and Farm environments.

Parameter Task Highway driving Farm Number of generations ($G$) $5$ $10$
Population size ($N$) $50$ $100$ Horizon ($k$) $5$ $10$ Evaluation dataset
size ($|D|$) $247$ $62$

###  4.2. Baselines

The goal of ACTER is to identify the changes in past decisions that could have
helped agents prevent an undesirable outcome. To the best of our knowledge,
ACTER is the first approach that can offer a counterfactual sequence of
actions that prevents an unwanted outcome. Previous approaches in
counterfactual state explanations can indicate which state features can be
changed to illicit a different outcome, but cannot attribute failure to a past
action, or provide an alternative action sequence to avoid failure (Olson et
al., 2021; Huber et al., 2023). For this reason, we compare ACTER to
approaches that can examine an agent’s path and attribute outcomes to previous
actions, as follows:

  1. (1)

HIGHLIGHTS: based on the HIGHLIGHTS algorithm (Amir and Amir, 2018) for
detecting important moments in the agent’s execution. HIGHLIGHTS identifies
important states as those in which there is a substantial difference in
outcomes when different actions are taken. Specifically, HIGHLIGHTS selects
states that maximize the difference in estimated Q values between the most and
least promising actions $max(Q(s,\mathcal{A}))-min(Q(s,\mathcal{A}))$.

  2. (2)

CERTAIN: identifies important execution moments based on the agent’s certainty
when making a decision based on a metric proposed by Sequeira and Gervasio
(2020). A state is considered important if the agent is certain of its
decision in the state. We measure the certainty using the entropy of Q values
in state $s$. Intuitively, the agent could be failing because it is overly
confident in certain incorrect behaviors, and changing the action the agent
takes in these states could prevent failure.

  3. (3)

UNCERTAIN: based on a metric opposite to the one used in CERTAIN (Sequeira and
Gervasio, 2020), it identifies important execution moments as states where the
agent is uncertain. We calculate the uncertainty of a state as the entropy of
the Q values. Intuitively, the agent could be failing because it is not fully
trained and still uncertain of correct behavior in certain states. Changing
which action the agent took in these uncertain states could prevent the
failure.

  4. (4)

LOCAL_MIN: based on the algorithm introduced by Sequeira and Gervasio (2020),
this baseline considers important the states in which the agent reaches a
local minimum in terms of the reward function. Local minimum states could
indicate why the agent failed, and preventing the agent from arriving to these
states could prevent failure.

  5. (5)

LOCAL_MAX: based on a metric opposed to LOCAL_MIN, this algorithm considers a
state important if it represents a local maximum in terms of the agent’s state
value. Local maxima states show states in which the agent considers itself to
be in a favorable position. However, if an agent is not correct in its
valuation of what a favorable position is, changing its action leading to a
local maximum could prevent failure. For example, if an autonomous vehicle
favors high speed, changing its decision that speed up the vehicle to a local
maximum could be useful for a user who values maintaining a lower speed.

These baseline approaches can only detect important actions but cannot
recommend alternative actions that could prevent failure. To enable a fair
comparison, we extend these approaches to enable this additional
functionality. Specifically, we try different random actions instead of the
one selected by the algorithm as important and choose the ones that satisfy
the validity constraint. In this way, we get an alternative sequence of
actions that can avoid failure and can be compared to ACTER. To increase
diversity we consider all sequences of actions that satisfy validity.

###  4.3. Evaluation Hypotheses

To evaluate ACTER we compare it to baselines described in Section 4.2 in
highway driving and Farm environments. We establish four evaluation hypotheses
for evaluating ACTER:

  * •

H1: ACTER can produce counterfactual sequences that prevent failure with lower
effort and higher certainty in stochastic environments compared to the
baselines.

  * •

H2: ACTER can produce a set of counterfactual sequences that offer more
diverse ways of preventing failure compared to the baselines.

  * •

H3: Actionable counterfactual sequences generated by ACTER help users better
understand, diagnose, and correct failure compared to non-actionable
explanations that only identify actions responsible for the failure.

  * •

H4: Explanations generated by ACTER will be perceived as more satisfactory by
users compared to non-actionable explanations that only identify actions
responsible for the failure.

Table 3. The average values of counterfactual properties for counterfactual
sequence generated using ACTER and the baseline approaches in highway driving
and Farm environments.

Task Highway driving Farm Metric Approach HIGHLIGHTS CERTAIN UNCERTAIN
LOCAL_MIN LOCAL_MAX ACTER HIGHLIGHTS CERTAIN UNCERTAIN LOCAL_MIN LOCAL_MAX
ACTER Generated counterfactuals (%) 23 15 53 20 44 89 92 90 0 0 21 100
Validity ($\uparrow$) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 - - 1.00 1.00
Proximity ($\downarrow$) 0.20 0.20 0.20 0.20 0.20 0.30 0.10 0.10 - - 0.10 0.14
Sparsity ($\downarrow$) 0.20 0.20 0.20 0.20 0.20 0.30 0.10 0.10 - - 0.10 0.42
Stochastic uncertainty ($\uparrow$) 0.66 0.62 0.75 0.61 0.76 0.85 0.0 0.0 - -
0.0 0.0 Recency ($\downarrow$) 0.18 0.09 0.25 0.12 0.31 0.25 0.07 0.04 - -
0.03 35

To evaluate H1, we evaluate the counterfactual properties of counterfactual
sequences generated by ACTER and the baseline approaches. The results are
presented in Section 5.1. Hypothesis H2 is presented in more detail in Section
5.2. Hypotheses H3 and H4 are evaluated through a user study and the results
are presented in Section 5.3.

###  4.4. Diversity Metrics

While previous work offered metrics to estimate diversity for a set of
counterfactual states (Mothilal et al., 2020), these do not apply to sequences
of actions. For that reason, we introduce three metrics for evaluating the
diversity of counterfactual sequences. Given a set of failure states $X_{f}$,
a set of counterfactual actions $A^{*}(x^{i})$ is generated for each $x^{i}\in
X_{f}$.

  1. (1)

Coverage (C): measures the average number of generated counterfactual
sequences per each failure:

(10) $C=\frac{1}{|X_{f}|}\sum\limits_{x^{i}\in X_{f}}|A^{*}(x^{i})|$

  2. (2)

Action diversity (AD): measures the average distance between generated action
sequences $A^{*}$:

(11) $AD=\frac{1}{|X_{f}|}\sum\limits_{x^{i}\in
X_{f}}\sum\limits_{\begin{subarray}{c}A^{*}_{i},A^{*}_{j}\in A^{*}(x^{i})\\\
A^{*}_{i}\neq A^{*}_{j}\end{subarray}}Dist(A^{*}_{i},A^{*}_{j})$

where the distance between two action sequences $Dist(A_{i},A_{j})$ is
calculated as:

(12) $Dist(A_{i},A_{j})=\begin{cases}\frac{1}{k}\sum_{a\in A_{i},b\in
A_{j}}a\neq b&\text{if $\mathcal{A}$ is discrete}\\\ \frac{1}{k}\sum_{a\in
A_{i},b\in A_{j}}|a-b|&\text{if $\mathcal{A}$ is continuous}\end{cases}$

  3. (3)

Counterfactual property diversity (CPD): measures the average difference in
counterfactual properties for generated counterfactual sequences $A^{*}$.
Given a set of counterfactual properties $\mathcal{C}$:

(13) $CDP(A^{*})=\frac{1}{|X_{f}|}\sum\limits_{x^{i}\in
X_{f}}\sum\limits_{\begin{subarray}{c}A^{*}_{i},A^{*}_{j}\in A^{*}(x^{i})\\\
A^{*}_{i}\neq A^{*}_{j}\end{subarray}}CFDist(A^{*}_{i},A^{*}_{j})$

where $CFDist(A_{i},A_{j})$ is defined as:

(14) $CFDist(A_{i},A_{j})=\sum_{c\in\mathcal{C}}(c(A_{i})-c(A_{j}))^{2}$

Coverage ensures that several options are offered to the users. Action
diversity allows the user to choose an action sequence that best fits their
preference. For example, if a car crash could have been prevented by either
slowing down or changing lanes, a user who is in a rush might prefer to change
lanes instead of sacrificing speed. Counterfactual property diversity ensures
that the set of solutions exhibits different trade-offs of the counterfactual
properties. For example, some users might prefer to change more actions but
ensure that failure is prevented with higher certainty, while others might be
more inclined to take a risk while changing fewer features.

Table 4. The diversity evaluation results for counterfactual sequences
generated using ACTER and the baseline approaches in highway driving and
farming environments.

Task Highway driving Farm Metric Approach HIGHLIGHTS CERTAIN UNCERTAIN
LOCAL_MIN LOCAL_MAX ACTER HIGHLIGHTS CERTAIN UNCERTAIN LOCAL_MIN LOCAL_MAX
ACTER C ($\uparrow$) 1.30 1.72 1.46 1.53 1.23 2.36 1.0 1.0 0.0 0.0 1.0 2.04 AD
($\uparrow$) 0.10 0.15 0.13 0.14 0.09 0.37 0.0 0.0 0.0 0.0 0.0 0.09 CPD
($\uparrow$) 0.06 0.08 0.04 0.13 0.08 0.19 0.0 0.0 0.0 0.0 0.0 0.01

##  5\. Results

We evaluate ACTER against five baselines in highway and Farm tasks. Firstly,
we obtain a black-box policy $M$ by training a DQN for $T$ steps. The training
parameters for policy $M$ are presented in Table 2. While this policy exhibits
good behavior in the environment, due to its complexity there are still
situations in which it can fail. In the highway task, we run $M$ for $1000$
and record failure in $247$ episodes. In the Farm task, we execute $M$ for
$2000$ and record failure in $62$ episodes. For each of the failures, we use
ACTER and baselines described in Section 4.2 to generate counterfactual
sequences of actions that could prevent the failure. The parameters used by
these approaches are shown in Table 2.

###  5.1. Evaluating Counterfactual Properties

To evaluate H1 we evaluate ACTER and the baselines on counterfactual
properties validity, proximity, sparsity, stochastic uncertainty, and recency,
as defined in Section 3.2. For each of the examined algorithms, we generate
counterfactuals for failure trajectories in highway and Farm environments. The
results are presented in Table 3.

ACTER successfully generates counterfactuals for most failures in both tasks
and performs the best on stochastic uncertainty property, partially confirming
H1. In the highway environment ACTER finds a counterfactual sequence of
actions that avoids failure in $89\%$ of trajectories. Among the baseline
approaches, UNCERTAIN generated the most counterfactuals in the highway
environment at $53\%$, while CERTAIN generated the least with only $15\%$.
This indicates that actions responsible for failure in the highway environment
are the ones the agent was not confident about. Similarly, in the Farm
environment, ACTER generated a counterfactual for each failure. CERTAIN and
HIGHLIGHTS were the most successful among the baselines, generating
counterfactuals for over $90\%$ of failures, while UNCERTAIN and LOCAL_MIN do
not generate counterfactuals for any failures in Farm task. This indicates
that in the Farm task, failure can be attributed to actions in which the agent
is falsely confident. ACTER shows the best results in stochastic uncertainty
in both environments, generating counterfactuals that are most robust to
changes in the stochastic conditions of the environment.

In both tasks, each counterfactual satisfies validity. This is due to ACTER
and baselines seeing validity as a hard constraint and generating only
counterfactuals that avoid failure. For proximity, sparsity, and recency
metrics, ACTER creates counterfactuals with worse values compared to
baselines. As baselines are limited to changing only one action, they always
generate counterfactuals with favorable proximity and sparsity values. ACTER,
however, generates multiple solutions that optimize different metrics, leading
to worse average proximity, sparsity, and recency. This corresponds to similar
results in supervised learning where the trade-off between diversity and
proximity has been shown (Mothilal et al., 2020).

###  5.2. Evaluating Diversity

To evaluate hypothesis H2, we compare the counterfactuals generated by ACTER
to baselines on the three diversity metrics defined in Section 4.4. The
results are presented in Table 4.

In both environments, ACTER performs better according to all three metrics,
confirming H2. It is important to note that ACTER was not trained to maximize
any of the diversity metrics. In fact, ACTER generates diverse counterfactuals
by design, due to its multi-objective optimization which simultaneously
minimizes counterfactual properties generating a Pareto set of solutions.

In the highway environment, ACTER generates on average $2.36$ counterfactual
sequences. ACTER also achieves almost double values for action diversity
compared to other approaches. Similarly, ACTER provides counterfactuals with
the highest counterfactual property diversity scores.

In the Farm task, all baselines generate no more than one counterfactual per
failure. In contrast, ACTER generates an average of $2.04$ counterfactual
sequences per failure. On the other two metrics, both ACTER and the baselines
show low values. This is because, in this task, the plant often dies because
it was not harvested in time. Failure is often prevented simply by executing a
harvest action sooner in the episode. For this reason, there is little variety
in explored counterfactual scenarios, resulting in lower values of action and
counterfactual property diversity metrics.

###  5.3. User Study

Ultimately, counterfactual explanations are developed to help humans
understand and engage with RL systems. While previous work on counterfactual
state explanations has explored how users engage with state-based
explanations, no user studies have explored how users perceive counterfactual
explanations that examine and change previous actions. In this work, we make
the first steps on the path of using counterfactual sequences with end-users.
The goal of the study is to evaluate how explanations help users identify and
correct failures as well as how they are perceived by the users. We compare
explanations generated by ACTER to non-actionable explanations that can only
identify the actions responsible for an outcome but do not offer alternative
actions that could change the outcome. Non-actionable explanations relay only
a portion of the information that counterfactuals do but might ease the
cognitive load of users. Previous work has found that in supervised learning
counterfactual explanations can lead to an increase in false confidence
compared to non-actionable explanations (Celar and Byrne, 2023). The goal of
this study is to examine this issue in RL. We obtain non-actionable
explanations by running ACTER but showing which action in the past can be
changed while omitting which alternative action could prevent failure. We
conducted the study in the highway task and showed participants examples of
car crashes and explanations for this outcome. The link to the study is
available at: https://qrxhyre44mt.typeform.com/to/B2CvHr6a.

We sourced 60 participants from English-speaking countries (UK, USA,
Australia, New Zealand, Canada, and Ireland) through the Prolific platform and
divided them into two groups. The first group received explanations generated
by ACTER and the other non-actionable explanations. Participants were
remunerated for their time according to the Prolific payment policy. The
consisted of 3 parts – training, testing, and subjective evaluation. Before
the study, the participants were introduced to the task and given information
about a specific type of explanation they would see during the study. In the
training part, participants were shown examples of car crashes along with
appropriate explanations. Each participant was presented with ten training
examples. In the testing phase, participants were then shown examples of
crashes without explanations and were asked to identify which previous action
could be changed and how to prevent failure. Participants were shown 10 test
questions. Due to visualization errors, one question had to be removed from
the study, resulting in 9 test questions. The questions were selected such
that there is only one alternative action sequence that would prevent failure.
Finally, the last part of the study involved users rating explanations on a
1-5 Likert scale based on the explanation goodness metrics (Hoffman et al.,
2018). The metrics measured how satisfactory, useful, detailed, complete,
actionable, trustworthy, and reliable users found the explanations. Users were
also asked whether explanations made them more confident in their
understanding of the agent’s behavior similar to the work of (Celar and Byrne,
2023).

The participants to whom explanations generated by ACTER were given could
identify the correct action in $42.59\%$, while those who have seen non-
actionable explanations succeeded in $42.22\%$ of test cases. Furthermore, the
former group successfully corrected the failure in $17.03\%$ and the latter
group for $18.5\%$ of cases. This indicates that there is no significant
difference in users’ ability to identify and correct failure between
explanations generated by ACTER and non-actionable explanations. These results
reject hypothesis H3.

On the other hand, users perceived explanations generated by ACTER as better
on all explanation goodness metrics. We performed a non-parametric one-tailed
Mann-Whitney U test and found that participants perceived counterfactual
explanations generated by ACTER to be significantly more useful ($0.0348$),
detailed ($0.0085$) and actionable ($0.0465$) compared to the non-actionable
explanations. Additionally, participants reported significantly higher
confidence after seeing explanations generated by ACTER ($p=0.0342$). However,
no significant difference was detected between participants’ ratings for
satisfaction ($p=0.0502$), completeness ($p=0.2478$), reliability ($p=0.0747$)
and trustworthiness ($p=0.1115$) of explanations. As a result, H4 is only
partially confirmed by our experiments.

Our study finds that counterfactual explanations do not help non-expert users
better diagnose and correct failure. One reason for this could be the
complexity of the task. Previous work on counterfactual state explanations
focuses on state-based explanations and shows users one state and its
explanation at a time (Olson et al., 2021; Huber et al., 2023). Our study,
however, required reasoning about longer chains of events and learning
connections between multiple states and actions and the outcome. We speculate
that counterfactuals on their own might not be enough to fully explain the
behavior of an agent in such a complex scenario to non-expert users. In future
work we hope to explore if these results hold for experts that are familiar
with RL systems and if counterfactuals can help them in debugging suboptimal
policies. Additionally, we hope to explore how counterfactuals can be combined
with other types of explanations to be best applied to explain policies to
non-expert users.

The outcomes of our study correspond to similar work in supervised learning
where counterfactuals have been found to provide false confidence when
predicting outcomes (Celar and Byrne, 2023). That counterfactuals can lead to
false confidence in RL is a worrying discovery as it can lead users to
confidently engage with the RL system without truly understanding it. On the
other hand, counterfactuals are perceived as more detailed, useful, and
actionable, indicating the potentially wider range of information that they
convey. Further research is needed to understand how counterfactuals can be
best utilized in RL.

##  6\. Future Work

In this work, we proposed ACTER, an approach for generating actionable and
diverse counterfactuals for diagnosing and correcting outcomes in RL. We
defined and implemented five counterfactual properties and provided an
evolutionary algorithm for the multi-objective optimization. Moreover, we
proposed three diversity metrics to benchmark the diversity of counterfactual
explanations. Finally, we conducted a user study and showed that while users
prefer counterfactual explanations, they are not more helpful in learning to
diagnose and correct failure than non-actionable explanations.

In future work, we hope to use encoders to better approximate proximity in
discrete action spaces. Additionally, in this work, we use simulations to
estimate the stochastic uncertainty of an action sequence. This is only
suitable for online scenarios where access to the environment is granted. In
future work, we hope to explore how uncertainty can be estimated even from
offline data. Finally, in this work we have shown that despite being preferred
by the users, counterfactual explanations do not help them better diagnose and
correct failure compare to simpler explanation types. In future work we plan
to examine further the scenarios in which counterfactual explanations can be
used both for explanation purposes as well as policy improvement and whether
they would be more useful for expert users.

## Acknowledgements

This publication has emanated from research supported in part by grants from
Science Foundation Ireland under grant number 18/CRT/6223 and SFI Frontiers
for the Future grant number 21/FFP-A/8957. For the purpose of Open Access, the
author has applied a CC BY public copyright licence to any Author Accepted
Manuscript version arising from this submission.

## References

  * (1)
  * Amir and Amir (2018) Dan Amir and Ofra Amir. 2018.  Highlights: Summarizing agent behavior to people. In _Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems_. 1168–1176.
  * Atrey et al. (2019) Akanksha Atrey, Kaleigh Clary, and David Jensen. 2019.  Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning. In _International Conference on Learning Representations_.
  * Byrne (2019) Ruth MJ Byrne. 2019.  Counterfactuals in Explainable Artificial Intelligence (XAI): Evidence from Human Reasoning.. In _IJCAI_. 6276–6282.
  * Celar and Byrne (2023) Lenart Celar and Ruth MJ Byrne. 2023.  How people reason with counterfactual and causal explanations for Artificial Intelligence decisions in familiar and unfamiliar domains.  _Memory & Cognition_ (2023), 1–16.
  * Coronato et al. (2020) Antonio Coronato, Muddasar Naeem, Giuseppe De Pietro, and Giovanni Paragliola. 2020.  Reinforcement learning for intelligent healthcare applications: A survey.  _Artificial Intelligence in Medicine_ 109 (2020), 101964.
  * Dandl et al. (2020) Susanne Dandl, Christoph Molnar, Martin Binder, and Bernd Bischl. 2020.  Multi-objective counterfactual explanations. In _International Conference on Parallel Problem Solving from Nature_. Springer, 448–469.
  * Deb et al. (2002) Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. 2002.  A fast and elitist multiobjective genetic algorithm: NSGA-II.  _IEEE transactions on evolutionary computation_ 6, 2 (2002), 182–197.
  * Edmonds et al. (2019) Mark Edmonds, Feng Gao, Hangxin Liu, Xu Xie, Siyuan Qi, Brandon Rothrock, Yixin Zhu, Ying Nian Wu, Hongjing Lu, and Song-Chun Zhu. 2019.  A tale of two explanations: Enhancing human trust by explaining robot behavior.  _Science Robotics_ 4, 37 (2019), eaay4663.
  * Gajcin and Dusparic (2023) Jasmina Gajcin and Ivana Dusparic. 2023.  RACCER: Towards Reachable and Certain Counterfactual Explanations for Reinforcement Learning.  _arXiv preprint arXiv:2303.04475_ (2023).
  * Greydanus et al. (2018) Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. 2018.  Visualizing and understanding atari agents. In _International conference on machine learning_. PMLR, 1792–1801.
  * Guo et al. (2021) Wenbo Guo, Xian Wu, Usmann Khan, and Xinyu Xing. 2021.  Edge: Explaining deep reinforcement learning policies.  _Advances in Neural Information Processing Systems_ 34 (2021), 12222–12236.
  * Hoffman et al. (2018) Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman. 2018.  Metrics for explainable AI: Challenges and prospects.  _arXiv preprint arXiv:1812.04608_ (2018).
  * Huber et al. (2023) Tobias Huber, Maximilian Demmler, Silvan Mertes, Matthew L Olson, and Elisabeth André. 2023.  GANterfactual-RL: Understanding Reinforcement Learning Agents’ Strategies through Visual Counterfactual Explanations. In _Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems_. 1097–1106.
  * Itaya et al. (2021) Hidenori Itaya, Tsubasa Hirakawa, Takayoshi Yamashita, Hironobu Fujiyoshi, and Komei Sugiura. 2021.  Visual explanation using attention mechanism in actor-critic-based deep reinforcement learning. In _2021 International Joint Conference On Neural Networks (IJCNN)_. IEEE, 1–10.
  * Juozapaitis et al. (2019) Zoe Juozapaitis, Anurag Koul, Alan Fern, Martin Erwig, and Finale Doshi-Velez. 2019.  Explainable reinforcement learning via reward decomposition. In _IJCAI/ECAI Workshop on explainable artificial intelligence_.
  * Karimi et al. (2021) Amir-Hossein Karimi, Bernhard Schölkopf, and Isabel Valera. 2021.  Algorithmic recourse: from counterfactual explanations to interventions. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_. 353–362.
  * Kiran et al. (2021) B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. 2021.  Deep reinforcement learning for autonomous driving: A survey.  _IEEE Transactions on Intelligent Transportation Systems_ 23, 6 (2021), 4909–4926.
  * Leurent (2018) Edouard Leurent. 2018.  An Environment for Autonomous Driving Decision-Making.  https://github.com/eleurent/highway-env.
  * Li (2017) Yuxi Li. 2017.  Deep reinforcement learning: An overview.  _arXiv preprint arXiv:1701.07274_ (2017).
  * Lyons et al. (2023) Joseph B Lyons, Izz aldin Hamdan, and Thy Q Vo. 2023.  Explanations and trust: What happens to trust when a robot partner does something unexpected?  _Computers in Human Behavior_ 138 (2023), 107473.
  * Madumal et al. (2020a) Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. 2020a.  Distal explanations for explainable reinforcement learning agents.  _arXiv preprint arXiv:2001.10284_ (2020).
  * Madumal et al. (2020b) Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. 2020b.  Explainable reinforcement learning through a causal lens. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 34. 2493–2500.
  * Maillard et al. (2023) Odalric-Ambrym Maillard, Timothée Mathieu, and Debabrota Basu. 2023.  Farm-gym: A modular reinforcement learning platform for stochastic agronomic games. In _Artificial Intelligence for Agriculture and Food Systems (AIAFS)_.
  * Miller (2019) Tim Miller. 2019.  Explanation in artificial intelligence: Insights from the social sciences.  _Artificial intelligence_ 267 (2019), 1–38.
  * Mothilal et al. (2020) Ramaravind K Mothilal, Amit Sharma, and Chenhao Tan. 2020.  Explaining machine learning classifiers through diverse counterfactual explanations. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_. 607–617.
  * Olson et al. (2021) Matthew L Olson, Roli Khanna, Lawrence Neal, Fuxin Li, and Weng-Keen Wong. 2021.  Counterfactual state explanations for reinforcement learning agents via generative deep learning.  _Artificial Intelligence_ 295 (2021), 103455.
  * Pálsson and Björnsson (2023) Aðalsteinn Pálsson and Yngvi Björnsson. 2023.  Unveiling concepts learned by a world-class chess-playing agent. In _Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence_. 4864–4872.
  * Poyiadzi et al. (2020) Rafael Poyiadzi, Kacper Sokol, Raul Santos-Rodriguez, Tijl De Bie, and Peter Flach. 2020.  FACE: feasible and actionable counterfactual explanations. In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_. 344–350.
  * Puiutta and Veith (2020) Erika Puiutta and Eric MSP Veith. 2020.  Explainable reinforcement learning: A survey. In _International cross-domain conference for machine learning and knowledge extraction_. Springer, 77–95.
  * Puri et al. (2019) Nikaash Puri, Sukriti Verma, Piyush Gupta, Dhruv Kayastha, Shripad Deshmukh, Balaji Krishnamurthy, and Sameer Singh. 2019.  Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attribution. In _International Conference on Learning Representations_.
  * Sequeira and Gervasio (2020) Pedro Sequeira and Melinda Gervasio. 2020.  Interestingness elements for explainable reinforcement learning: Understanding agents’ capabilities and limitations.  _Artificial Intelligence_ 288 (2020), 103367.
  * Tolmeijer et al. (2020) Suzanne Tolmeijer, Astrid Weiss, Marc Hanheide, Felix Lindner, Thomas M Powers, Clare Dixon, and Myrthe L Tielman. 2020.  Taxonomy of trust-relevant failures and mitigation strategies. In _Proceedings of the 2020 acm/ieee international conference on human-robot interaction_. 3–12.
  * Tsirtsis et al. (2021) Stratis Tsirtsis, Abir De, and Manuel Rodriguez. 2021.  Counterfactual explanations in sequential decision making under uncertainty.  _Advances in Neural Information Processing Systems_ 34 (2021), 30127–30139.
  * Tsirtsis and Gomez-Rodriguez (2023) Stratis Tsirtsis and Manuel Gomez-Rodriguez. 2023.  Finding Counterfactually Optimal Action Sequences in Continuous State Spaces.  _arXiv preprint arXiv:2306.03929_ (2023).
  * Upadhyay et al. (2021) Sohini Upadhyay, Shalmali Joshi, and Himabindu Lakkaraju. 2021.  Towards robust and reliable algorithmic recourse.  _Advances in Neural Information Processing Systems_ 34 (2021), 16926–16937.
  * Wells and Bednarz (2021) Lindsay Wells and Tomasz Bednarz. 2021.  Explainable ai and reinforcement learning—a systematic review of current approaches and trends.  _Frontiers in artificial intelligence_ 4 (2021), 550030.
  * Yang et al. (2018) Zhao Yang, Song Bai, Li Zhang, and Philip HS Torr. 2018.  Learn to interpret atari agents.  _arXiv preprint arXiv:1812.11276_ (2018).
  * Yau et al. (2020) Herman Yau, Chris Russell, and Simon Hadfield. 2020.  What did you think would happen? explaining agent behaviour through intended outcomes.  _Advances in Neural Information Processing Systems_ 33 (2020), 18375–18386.

## Appendix

##  7\. User Study Details

In this section, we provide some examples of user study questions and
templates used in this work. In Figure 3 we show examples of a counterfactual
explanation and a non-actionable explanation of a car crash presented to the
users during the training phase of the study. Figure 4 shows an example of a
test question that the users were asked.

Figure 3. Examples of user study explanations shown to the users. Left:
Counterfactual explanation. Right: Non-actionable explanation.

Figure 4. Examples of user study questions. Left: Question about identifying
action responsible for failure. Right: Question about alternative action to
prevent failure.