# A Black Swan Hypothesis in Markov Decision Process via Irrationality

Hyunin Lee<EMAIL_ADDRESS>
University of California, Berkeley
Berkeley, CA 94720-1776, USA David Abel<EMAIL_ADDRESS>
Google Deepmind
London, EC4A 3TW, United Kingdom
Ming Jin<EMAIL_ADDRESS>
Virginia Tech
Blacksburg, VA 24061, USA
Javad Lavaei<EMAIL_ADDRESS>
University of California, Berkeley
Berkeley, CA 94720-1776, USA Somayeh Sojoudi<EMAIL_ADDRESS>
University of California, Berkeley
Berkeley, CA 94720-1776, USA

###### Abstract

Black swan events are statistically rare occurrences that carry extremely high
risks. A typical view of defining black swan events is heavily assumed to
originate from an unpredictable time-varying environments; however, the
community lacks a comprehensive definition of black swan events. To this end,
this paper challenges that the standard view is _incomplete_ and claims that
high-risk, statistically rare events can also occur in unchanging environments
due to human misperception of their value and likelihood, which we call as
spatial black swan event. We first carefully categorize black swan events,
focusing on spatial black swan events, and mathematically formalize the
definition of black swan events. We hope these definitions can pave the way
for the development of algorithms to prevent such events by rationally
correcting human perception.

Keywords: Risk, Reinforcement learning, Irrationality, Cumulative prospect
theorem.

##  1 Introduction

> Life is the cumulative effect of a handful of significant shocks.
>
> The Black Swan: The Impact of the Highly Improbable, _Nassim Nicholas Taleb_

The reason behind the Lehman Brothers bankruptcy, an unexpected event with
extremely negative impacts on the world economy, remains controversial.
However, a strong explanation points to the irrationality of human decision-
making. The firm declared bankruptcy within 72 hours without any precursor
(McDonald and Robinson (2009)), and the only factor that changed during those
three days was investors’ _faith_ in the company (Housel (2023); Mawutor
(2014); Fleming and Sarkar (2014)). _Faith_ was intrinsically believed by
investors as an axiom. They made optimal decisions (being rational) based on
this faith, which turned out to be suboptimal (being irrational) once the
faith was revealed to be false during those 72 hours . Referring to the
unexpected bankruptcy event mentioned previously, we call such a rare and
high-risk event, often rationalized retrospectively with the benefit of
hindsight, a _black swan_ (Taleb (2010)).

Black swan events remain one of the unsolved problems in machine learning
safety (Hendrycks et al. (2021)) , and this work focuses on the origin of
black swan events from the perspective of _human perception_ , our main
messages being Hypothesis 2 and Definition 7, rather than on how to design
robust algorithms against them . We expect that providing a novel perspective
to understand black swan events can offer new insights for designing safe
machine learning algorithms . Supported by extensive documentation of black
swan events, such as the dissolution of the Soviet Union , the terrorist
attacks of September 11, 2001, and the Brexit vote (Taleb (2010)), we focus on
specific types of black swans that occur even in a stationary environment . We
refer to these as _spatial_ black swans. Based on the above example , we
deduce that certain types of black swan occur due to _misperception_ in the
way humans perceive the world (Hypothesis 2). Executing an optimal policy
based on misperception inevitably causes the agent to encounter unavoidable
risks, which we define as spatial black swans (Definition 7) . From a broad
perspective, our work proposes the following informal hypothesis regarding the
origin of black swan events and provides an informal definition of spatial
black swans built upon this hypothesis as follows.

######  Hypothesis 1 (Black swan origin (informal)).

Black swan events occur due to temporal and spatial misperception, and spatial
black swans occur due to human misperception of the real world.

######  Definition 1 (Spatial black swan (informal)).

A spatial black swan event is a state and action pair that humans perceive as
an infeasible event and assess its reward in a pessimistic manner.

Paper structure. Starting from the informal hypothesis and definition of
spatial black swans as mentioned above (Hypothesis 1, Definition 1), we
structure the paper to provide concrete foundations for our main message:
Hypothesis 2, and Definition 7 as follows. First, in Section 2, we define
spatial and temporal black swans, then focus on _spatial black swans_ for the
rest of the paper. Section 3 provides evidence for the soundness of Hypothesis
1. Specifically, in Subsection 3.1, we emphasize the necessity of a novel
perspective to understand black swans by demonstrating that the existing
decision-making frameworks under risk are insufficient, and in Subsection 3.2,
we explain how misperception can be related to black swans. Section 4
introduces Cumulative Prospect Theorem (CPT), a well-known theorem for
explaining irrational human behavior in the real world, to mathematically
formalize the term _misperception_ in Hypothesis 1 and Definition 1. In
Section 5, we introduce three types of Markov Decision Processes: Universe
MDP, Human MDP, and Human-Estimation MDP, which we denote as $\mathcal{M}$,
$\mathcal{M}^{\dagger}$, and $\widehat{\mathcal{M}}^{\dagger}$, respectively.
Before presenting the concrete definition and hypothesis of spatial black
swans, we provide case studies in Section 6 to illustrate how human
misperception can lead to suboptimal policy (Theorems 1, 2, and 3).
Subsequently, in Section 7, we present our main message, proposing the spatial
black swan hypothesis (Hypothesis 2) and a definition of spatial black swan
events (Definition 7) utilizing $\mathcal{M}$, $\mathcal{M}^{\dagger}$,
$\widehat{\mathcal{M}}^{\dagger}$, and CPT. It is worth noting that Definition
7 employs notations from $\mathcal{M}$, $\mathcal{M}^{\dagger}$, and
$\widehat{\mathcal{M}}^{\dagger}$ and is mathematically characterized by
concepts from CPT. Finally, in Section 8, we explore the properties of spatial
black swan events, particularly how their presence establishes a lower bound
on achieving true policy performance (Theorem 4) and affects the timing of
black swan event occurrences (Theorem 5).

Notations.  The sets of natural, real, nonnegative, and nonpositive real
numbers are denoted by $\mathbb{N}$, $\mathbb{R}$, $\mathbb{R}_{\geq 0}$, and
$\mathbb{R}_{\leq 0}$ respectively. For a finite set $Z$, the notation $|Z|$
represents its cardinality, and $\Delta(Z)$ denotes the probability simplex on
$Z$. Given $X,Y\in\mathbb{N}$ with $X<Y$, we define $[X]:=\\{1,2,\ldots,X\\}$,
the closed interval $[X,Y]:=\\{X,X+1,\ldots,Y\\}$. For $x\in\mathbb{R}_{+}$,
the floor function $\lfloor x\rfloor$ is defined as
$\max\\{n\in\mathbb{N}\cup\\{0\\}\mid n\leq x\\}$.

Markov Decision Process. We consider a finite horizon non-stationary Markov
Decision Process (MDP) denoted as
$\mathcal{M}=\langle\mathcal{S},\mathcal{A},\\{P_{t}\\}_{t=0}^{T},\\{R_{t}\\}_{t=0}^{T},\gamma\rangle$,
where $\mathcal{S}$ represents the state space, $\mathcal{A}$ denotes the
action space, $P_{t}:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S})$ is
the transition probability function at time $t$,
$R_{t}:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$ is a reward function as time
$t$, $\gamma$ is the discount factor, and $T\in\mathbb{N}$ is the horizon
length. We denote a policy as $\pi\in\Pi$, where
$\Pi:{\mathcal{S}}\to\Delta({\mathcal{A}})$ is a set of policies, and its
performance as $J(\pi)=\mathbb{E}_{\pi,{\mathcal{M}}}[G]$ where
$G=\sum_{t=0}^{T-1}R_{t}(s_{t},a_{t})$ is a scalar return. We denote $T$
length trajectory from ${\mathcal{M}}$ with policy $\pi$ as
$\\{s_{0},a_{0},r_{0},s_{1},a_{1},r_{1},\cdots,s_{T}\\}$. Note that based on
the non-stationary finite horizon MDP , we restrict our analysis to a
stationary MDP to elaborate on _spatial_ black swans in Sections 4, 5, 7, and
8, but fully utilized non-stationarity for the case study in Section 6, and
black swan clarification in Section 2.

Problem setting.  Our problem setting is that humans perceive the world
$\mathcal{M}$ as $\mathcal{M}^{\dagger}$ and collect misperceived trajectory
data
$\\{s^{\dagger}_{0},a_{0},r^{\dagger}_{0},s^{\dagger}_{1},a_{1},r^{\dagger}_{1},\cdots,s^{\dagger}_{T}\\}$,
where misperception occurs on the state and reward through a function
$g:\mathcal{S}\to\mathcal{S}$ and a function $u:\mathbb{R}\to\mathbb{R}$,
respectively. This means that the state and reward are distorted as
$s^{\dagger}_{t}=g(s_{t})$ and $r^{\dagger}_{t}=u(r_{t})$ for all $t\in[T]$.
Humans then estimate the reward and visitation probability (or transition
probability) of $\mathcal{M}^{\dagger}$ from the misperceived trajectory data
to form their estimation model $\widehat{\mathcal{M}}^{\dagger}$.

Main message and supporting evidence. Our main message (Hypothesis 2,
Definition 7) starts with the observation that even though humans can decrease
the _estimation_ gap to make $\widehat{\mathcal{M}}^{\dagger}$ converge to
$\mathcal{M}^{\dagger}$ by rolling out a longer horizon $T$ or better learning
algorithms to reconstruct visitation probability and reward function. However,
the uncertainty of estimating $\mathcal{M}$ has a lower bound due to the
_perception_ gap between $\mathcal{M}$ and $\mathcal{M}^{\dagger}$. In this
sense, we convey that spatial black swans can occur due to the perception gap,
even if the agent has zero estimation error and computes its optimal policy
from $\widehat{\mathcal{M}}^{\dagger}$.

Our main message is supported by our primary theorem, Theorem 4, which
provides two key insights. First, it demonstrates that the policy performance
gap between $\mathcal{M}$ and $\widehat{\mathcal{M}}^{\dagger}$ has a lower
bound due to the perception gap between $\mathcal{M}$ and
$\mathcal{M}^{\dagger}$, even though the estimation error between
$\mathcal{M}^{\dagger}$ and $\widehat{\mathcal{M}}^{\dagger}$ asymptotically
converges to zero as the horizon length $T$ increases. Second, Theorem 4
quantifies this lower bound in terms of three factors: the number of spatial
black swans, the minimum probability of spatial black swans, and the magnitude
of misperception between $\mathcal{M}$ and $\mathcal{M}^{\dagger}$. It also
highlights that the effective number of spatial black swans (number $\times$
probability) is more significant than considering sole probability or the
number of occurrences for future algorithm design. Additionally, Theorem 5
provides a hitting time of spatial black swans.

Contributions. This work offers three pivotal contributions:

  * •

A novel perspective on black swan events: We propose that these events arise
from human misperception of space and time. This idea, introduced as
Hypothesis 1, is further developed into Hypothesis 2 in Section 7, building on
foundations laid in Sections 3, 4, and 5.

  * •

A novel definition of spatial black swans: We introduce and define a specific
type of black swan event (initially presented in Definition 1 and later
formalized as Definition 7 in Section 7). This definition builds upon our core
hypothesis 2, providing a new framework for understanding these rare but
impactful events.

  * •

Theoretical analysis of spatial black swans: We conduct a rigorous examination
of how these events affect optimal policy performance (Theorem 4) and their
occurrence probability over time (Theorem 5). This analysis is supported by
case studies (Theorems 1, 2, and 3) demonstrating the impact of human
misperception on decision-making in various scenarios.

##  2 Spatial and temporal blackswans

Based on Hypothesis 1, this prompts us to investigate the concept of
_misperception_. Initially, we must clearly define what constitutes
_perception_. According to the definition of an agent by Barandiaran et al.
(2009) and Orseau and Ring (2012), an agent views its environment through the
lens of so-called _spatio-temporal_ dimensions. Consequently, if a black swan
event arises from misperception, the conceptual framework of perception leads
us to question whether the misperception originates from spatial or temporal
dimensions. This first leads us to define the black swan event dimension as
follows.

######  Definition 2 (Black swan dimension).

In a Markov Decision Process, a dimension of a black swan event is defined on
${\mathcal{S}}\times{\mathcal{A}}\times[T]$ with $\mathcal{S}$ representing
the state space and $\mathcal{A}$ representing the action space and $T$ is
time (horizon length). We collect black swan events as a set ${\mathcal{B}}$
(${\mathcal{B}}$ will be more elaborated on Hypothesis 2).

In Definition 2, unit time refers to any variable that represents its time
heterogeneity such as discrete step or discrete episode (Lee et al. (2024) or
discrete real-time (Abel et al. (2024); Dong et al. (2022)) in general Markov
Decision Process setting.

######  Assumption 1.

For fixed time $t^{\prime}\in[T]$, black swan events at time $t^{\prime}$,
i.e., the set $\\{(s,a)|(s,a,t=t^{\prime})\in{\mathcal{B}}\\}$ is invariant
set under static agent’s perception of MDP at time $t^{\prime}$.

Assumption 1 ensures that if the environment and the agent’s perception of the
environment are fixed at time $t$, then the black swan events are fully
determined. This assumption is also used to classify black swans in
Proposition 1, Example 1, and Remark 1.

The definition 2 leads us to first classify whether the black swan comes from
a misconception within the space $(s,a)$, termed _spatial mispercpetion_ , or
a misperception along the unit time $(t)$, termed _temporal misperception_. We
define _temporal misperception_ as the inherent inaccuracies in time-series
data prediction algorithms from agent, exemplified by black swans such as
COVID-19 or earthquakes, and define _temporal black swans_ as those
originating from temporal misperception. This type of misperception inevitably
results from the non-stationarity of the environment, which impacts the
algorithms’ ability to predict future events accurately. In contrast, we
define _spatial misperception_ as a misperception that occurs in a stationary
environment, often due to incorrect assessments or misunderstandings of the
spatial aspects of the data. We define _spatial black swans_ as those
originating from spatial misperception, such as the Russia-Ukraine war, the
Lehman Brothers bankruptcy, the Brexit vote or the September 11, 2001
terrorist attacks. Intuitively, temporal black swans occur due to the non-
stationarity of the environment, while spatial black swans occur even in
stationary environments. Based on the above description, we provide a
definition of spatial and temporal black swans as follows.

######  Definition 3 (Spatial black swans).

For a given MDP ${\mathcal{M}}$ for a time interval $[T]$, suppose
$(s,a,t_{bs})$ is a black swan where $t_{bs}\in[T]$. If $(s,a,t)$ is a black
swan event for $\forall t\in[T]$, then we define $(s,a,t_{bs})$ as a spatial
black swan.

######  Definition 4 (Temporal black swans).

For a given MDP ${\mathcal{M}}$ for a time interval $[T]$, suppose
$(s,a,t_{bs})$ is a black swan where $t_{bs}\in[T]$. If there exists $t\in[T]$
such that $(s,a,t)$ is not a black swan event, then we define $(s,a,t_{bs})$
as a temporal black swan.

Based on Definitions 3, 4 and Assumption 1, we can always identify a unit time
interval that classifies any black swan events as a spatial black swan as
following Proposition 1 and Example 1.

######  Proposition 1.

If $(s,a,t_{bs})$ is a black swan event, then there exists a time interval
$[t^{\prime}_{1},t^{\prime}_{2}]\subseteq[T]$ that classifies $(s,a,t_{bs})$
as a spatial black swan within $[t_{1},t_{2}]$.

We provide an intuitive interpretation for Proposition 1 as the following
example.

######  Example 1.

Suppose $(s,a,t_{bs})$ is a black swan event.

  1. case 1.

${\mathcal{M}}$ is a non-stationary MDP where $P_{t},R_{t}$ changes for every
unit time, i.e. $P_{t}\neq P_{t+1},R_{t}\neq R_{t+1}$. If
$t_{1}=t_{2}=t_{bs}$, then $(s,a,t_{bs})$ is a spatial black swan. If
$t_{1}\neq t_{2}$ where $t_{bs}\in[t_{1},t_{2}]$, then $(s,a,t_{bs})$ could
not be identified as spatial black swan or temporal black swan.

  2. case 2.

${\mathcal{M}}$ is a piecewise non-stationary MDP where $P_{t},R_{t}$ changes
for $\lfloor T/k\rfloor$ times, i.e. $P_{t}=P_{t+1},R_{t}=R_{t+1}$ for
$t\in[kj,kj+(k-1)],j=0,1,\cdots,\lfloor T/k\rfloor$. If
$t_{bs}\in[kj_{bs},kj_{bs}+(k-1)]$, then $(s,a,t_{bs})$ is a spatial black
swans when $t_{1}=kj_{bs},t_{2}=kj_{bs}+(k-1)$.

  3. case 3.

${\mathcal{M}}$ is stationary MDP where $P_{t}=P_{t+1},R_{t}=R_{t+1}$ for
$\forall t\in[T-1]$, then $(s,a,t_{bs})$ is a spatial black swan regardless of
interval $[t_{1},t_{2}]$.

We formulate case 3 of Example 1 as the following Remark.

######  Remark 1.

If ${\mathcal{M}}$ is stationary, then any black swan events $(s,a,t)$ are
spatial black swans, denoted as $(s,a)$.

This work investigates Remark 1. For the remainder of the paper, we focus on
spatial black swans by concretizing Hypothesis 1 and Definition 1 in the
context of spatial black swans. Based on Definition 3 and Remark 1 within the
framework of Markov Decision Processes, we provide an informal proposition of
spatial black swans by adapting the preliminary definition proposed by Taleb
(2010).

######  Proposition 2 (Spatial black swan (informal)).

For given MDP
$\mathcal{M}=\langle{\mathcal{S}},{\mathcal{A}},P,R,\gamma\rangle$. We define
an event $\sigma$ as a state and action
$(s,a)\in{\mathcal{S}}\times{\mathcal{A}}$. If $\sigma$ is a spatial black
swan event, then it satisfies that

  1. 1.

$\sigma$ is a high-risk event.

  2. 2.

$\sigma$ is a rare event.

  3. 3.

After $\sigma$ is first observed at time $t$, it is rationalized by hindsight,
as if it could have been expected.

Note that the first and second properties of Proposition 2 characterize the
appearance and nature of such events, while the third property elucidates
strategies for avoiding black swans, specifically addressing their post-event
properties. Therefore, we elaborate on the first and second properties of
Proposition 2 through three different MDPs ($\mathcal{M}$,
$\mathcal{M}^{\dagger}$, $\widehat{\mathcal{M}}^{\dagger}$) introduced in
Section 5 and CPT, which will be introduced in Section 4, then complete it as
Definition 7.

##  3 Necessity of a new perspective to understand black swans and evidence
for Hypothesis 1

In this section, we focus not only on addressing the necessity of a new
perspective to understand black swan events but also on providing evidence for
the proposed perspective of black swan origin (Hypothesis 1). This is
concretized by examining the following two questions. First, in Subsection
3.1, we discuss the insufficiency of existing decision-making rules under risk
by exploring related works, which support the need for a new perspective to
understand black swans. Specifically, we address _why existing safe
reinforcement learning strategies for solving Markov Decision Processes are
insufficient to handle black swan events?_. If this premise is validated, then
in Subsection 3.2, we elaborate on the motivation and related works that
support our informal hypothesis of black swan origin (Hypothesis 1).
Specifically, we explore _how irrationality relates to misperception and how
irrationality could bring about black swan events_.

###  3.1 Decision Making Under Risk

Based on the comprehensive survey on safe reinforcement learning in Garcıa and
Fernández (2015), the algorithms can be classified into threefold: worst case
criterion, risk-sensitive criterion and constraint criterion. We elaborate on
why the existence of black swans in the environment renders these three
approaches insufficient.

Worst case criterion. Learning algorithms of the worst case criterion focus on
devising a control policy that maximizes policy performance under the least
favorable scenario encountered during the learning process, defined as
$\max_{\pi\in\Pi}\min_{w\in{\mathcal{W}}}J(\pi;w)$, where ${\mathcal{W}}$
represents the set of uncertainties. This criterion can be categorized based
on whether ${\mathcal{W}}$ is defined in the environment or in the estimation
of the model. The presence of black swan events in the worst case, where
${\mathcal{W}}$ represents aleatoric uncertainty of the environment (Heger
(1994); Coraluppi (1997); Coraluppi and Marcus (1999, 2000)), results in
overly conservative, and thus potentially ineffective, policies. This occurs
because the significant impact of black swan events inflates the size of
${\mathcal{W}}$, even though such events are rare. In practical terms, this
could manifest itself as abstaining from any economic activity ($\pi$), such
as not investing in stocks or not depositing a check against future potential
bankruptcies ($\min_{w\in{\mathcal{W}}}J(\pi;w)$) in order to maximize its
income ($\max_{\pi\in\Pi}(\cdot)$), or maintaining constant health precautions
such as wearing mask or maintaining distance with groups ($\pi$) to prepare
for a possible pandemic ($\cdot=\min_{w\in{\mathcal{W}}}J(\pi;w)$) in order to
maintain its health ($\max_{\pi\in\Pi}$). Similarly, when ${\mathcal{W}}$
encompasses the uncertainty of the model parameter Bagnell et al. (2001);
Iyengar (2005); Nilim and El Ghaoui (2005); Wiesemann et al. (2013); Xu and
Mannor (2010) \- as seen in robust MDP or distributionally robust MDP - this
aligns closely with our black swan hypothesis, where misperception of the
world model is similar to uncertainty in model estimation. However, the need
to accommodate black swan events requires enlarging the possible set of models
($|{\mathcal{W}}|$), leading to extremely conservative policies. This can be
likened to performing an overly pessimistic portfolio optimization ($\pi$),
where every bank is assumed to have a minimal but possible risk of bankruptcy
$(\min_{win{\mathcal{W}}}J(\pi;w))$, thus influencing asset allocation
strategies ($\max_{\pi\in\Pi}\min_{w\in{\mathcal{W}}}J(\pi;w)$) to be
extremely conservative in asset investing.

Risk sensitive criterion.  Risk-sensitive algorithms strike a balance between
maximizing reinforcement and mitigating risk events by incorporating a
sensitivity factor $\beta<0$ (Howard and Matheson (1972); Chung and Sobel
(1987); Patek (2001)). These algorithms optimize an alternative value function
$J(\pi)=\beta^{-1}\log\mathbb{E}_{\pi}[\exp^{\beta G}]$, where $\beta$
controls the desired level of risk. However, it is recognized that associating
risk with the variance of the return is practical, as in
$J(\pi)=\beta^{-1}\log\mathbb{E}_{\pi}[\exp^{\beta
G}]=\max_{\pi\in\Pi}\mathbb{E}_{\pi}[G]+\frac{\beta}{2}\text{var}(G)+\mathcal{O}(\beta^{2})$,
and the existence of black swan events does not significantly affect the
returns of variance $(\text{var}(G))$ due to their rare nature. It should be
noted that risk-sensitive approaches are not well suited for handling black
swan events, as the same policy performance with small variance can entail
substantial risks (Geibel and Wysotzki (2005)). More generally, the objective
of the exponential utility function is one example of risk-sensitive learning
based on a trade-off between return and risk, i.e.,
$\max_{\pi\in\Pi}(\mathbb{E}_{\pi}[G]-\beta w)$ (Zhang et al. (2018)), where
$w$ is replaced by $\text{Var}(G)$. This approach is known in the literature
as the variance-penalized criterion (Gosavi (2009)), the expected value-
variance criterion (Taha (2007); Heger (1994)), and the expected-value-minus-
variance criterion (Geibel and Wysotzki (2005)). However, a fundamental
limitation of using return variance as a risk measure is that it does not
account for the fat tails of the distribution (Huisman et al. (1998); Bradley
and Taqqu (2003); Bubeck et al. (2013); Agrawal et al. (2021)). Consequently,
risk can be underestimated due to the oversight of low probability but highly
severe events (black swans).

Furthermore, a critical question arises regarding whether the log-exponential
function belongs to _appropriate utility function class_ for defining _real-
world risk_. Risk-sensitive MDPs have been shown to be equivalent to robust
MDPs that focus on maximizing the worst-case criterion, indicating that the
log-exponential utility function may not be beneficial in the presence of
black swans (Osogami (2012); Moldovan and Abbeel (2012); Leqi et al. (2019)).
This issue was first raised by Leqi et al. (2019) and led to the proposal of a
more realistic risk definition called ‘Human-aligned risk’, which also
incorporates human misperception akin to our informal black swan hypothesis
(Hypothesis 1).

Constrained Criterion.  The constrained criterion is applied in the literature
to constrained Markov processes where the goal is to maximize the expected
return while maintaining other types of expected utilities below certain
thresholds. This can be formulated as $\max_{\pi\in\Pi}\mathbb{E}_{\pi}[G]$
subject to $N$ multiple constraints $h_{i}(G)\leq\alpha_{i}$, for $i\in[N]$,
where $h_{i}:\mathbb{R}\to\mathbb{R}$ is a function of return $G$ (Geibel
(2006)). Typical constraints include ensuring the expectation of return
exceeds a specific minimum threshold ($\alpha$), such as
$\mathbb{E}[G]\geq\alpha$, or softening these hard constraints by allowing a
permissible probability of violation ($\epsilon$), such as
$\mathbb{P}(\mathbb{E}[G]\geq\alpha)\geq 1-\epsilon$, known as chance-
constraint (Delage and Mannor (2010); Ponda et al. (2013)). Constraints might
also limit the return variance, such as $\text{Var}(G)\leq\alpha$ (Di Castro
et al. (2012)). However, the presence of black swans highlights one of the
challenges with the Constrained Criterion, specifically the appropriate
selection of $\alpha$. The presence of black swans necessitates a lower
$\alpha$, which in turn leads to more conservative policies. Furthermore, a
black swan event is determined at least by the environment’s state and its
action, rather than its full return. Therefore, constraints should be
redefined over more fine-grained inputs—not merely returns, but in terms of
state and action—which leads to our definition of black swan dimensions
(Definition 2).

###  3.2 How irrationality relates with spatial black swans.

Before starting Subsection 3.2, we clarify that the term _irrationality_ is
used here to denote rational behavior based on a false belief. In this
subsection, we first review existing work on the four rational axioms and then
claim how two of these axioms should be modified to account for
_irrationality_ in human decision-making.

Rationality in decision making.  In the foundation of decision theory,
rationality is understood as internal consistency (Sugden (1991); Savage
(1972)). A prerequisite for achieving rationality in decision making is the
ability to compare outcomes, denoted as set ${\mathcal{O}}$, through a
_preference_ relation in a _rational_ manner. In Neummann and Morgenstern
(1944), it is demonstrated that preferences, combined with _rationality
axioms_ and probabilities for possible outcomes, denoted as $p_{i}$ which is a
probability of outcome $o_{i}\in{\mathcal{O}}$, imply the existence of utility
values for those outcomes that express a preference relation as the
expectation of a scalar-valued function of outcomes. Define the choice (or
lotteries) as set ${\mathcal{C}}$, which is a combination of selecting total
$n$ outcomes, that is, $\sum_{i=1}^{n}p_{i}o_{i}$. The essential rationality
axioms are as follows.

  1. 1.

Completeness: Given two choices, either one is preferred over the other or
they are considered equally preferable.

  2. 2.

Transitivity: If $A$ is preferred to $B$ and $B$ is preferred to $C$, then $A$
must be preferred to $C$.

  3. 3.

Independence: If $A$ is preferred to $B$, and a event probability $p\in[0,1]$,
then $pA+(1-p)C$ should be preferred to $pB+(1-p)C$.

  4. 4.

Continuity: If $A$ is preferred to $B$ and $B$ is preferred to $C$, there
exists a event probability $p\in[0,1]$ such that $B$ is considered equally
preferable to $pA+(1-p)C$.

Expanding on these axioms, Sunehag and Hutter (2015) extends rational choice
theory to encompass the full reinforcement learning problem, further
axiomatizing the concept in Sunehag and Hutter (2011) to establish a rational
reinforcement learning framework that facilitates optimism, crucial for
systematic explorative behavior. Subsequent studies focusing on defining
rationality in reinforcement learning, such as Shakerinava and Ravanbakhsh
(2022); Bowling et al. (2023), concentrate on the axioms of assigning
utilities to all finite trajectories of a Markov Decision Process.
Specifically, Shakerinava and Ravanbakhsh (2022); Bowling et al. (2023)
clarify the reward hypothesis Sutton  that underpins the design of rational
agents by introducing an additional axiom to existing rationality axioms.
Furthermore, Pitis (2024) explores the design of multi-objective rational
agents, and Carr et al. (2024) explores and defines rational feedback in Large
Language Models (LLMs) by investigating the existence of optimal policies
within a framework of learning from rational preference feedback (LRPF).

Irrationality due to subjective probability.  The definition of irrationality
and its origins has been extensively investigated through case studies in
various fields such as psychology, education, and particularly economics.
Simon (1993) defined irrationality as being poorly adapted to human goals,
diverging from the norm of human’s object, influenced by emotional or
psychological factors in decision-making. Subsequently, Martino et al. (2006);
Gilovich et al. (2002) further concretized what exactly these _emotional or
psychological factors_ entail by describing them as information loss during
human perception of the real-world. More specifically, Martino et al. (2006)
pointed out that in a world filled with symbolic artifacts, where optimal
decision-making often requires skills of abstraction and decontextualization,
such mechanisms may render human choices irrational. Further studies, such as
Opaluch and Segerson (1989), scrutinize more deeply and classify the
_irrationality_ of human behavior into five factors: subjective probability,
regret/disappointment, reference points, complexity, and ambivalence.

In this paper, we focus on the _subjective probability_ factor to elucidate
the relationship between irrationality and spatial black swans. Opaluch and
Segerson (1989) explores subjective probabilities as an early modification to
the expected utility model from Neummann and Morgenstern (1944), focusing on
decision makers who rely on _personal beliefs_ about probabilities rather than
objective truths. This minor conceptual shift can lead to significant
behavioral changes due to the imperfect information and processing abilities
of individuals. Especially, Opaluch and Segerson (1989) highlights the
difficulty in accurately estimating the probability of _rare events_ \- such
as black swans - which often leads to critical errors in judgment. These
errors occur because rare events provide insufficient data for accurate
probability estimation or are misunderstood due to their infrequency, leading
to perceptions that such events are either less likely or virtually
impossible. This misperception is exemplified in various scenarios, such as:

  1. 1.

An individual working in a dangerous job who has never personally observed an
accident may underestimate the probability of an accident occurring
Drakopoulos and Theodossiou (2016); Pandit et al. (2019).

  2. 2.

Media coverage of events such as plane crashes may cause an overestimation of
the probability of a crash, since the public is aware of all crashes but not
of all safe trips Wahlberg and Sjoberg (2000); Vasterman et al. (2005); van
der Meer et al. (2022).

  3. 3.

The popularity of purchasing lottery tickets may be explainable in terms of
people’s inability to comprehend the true probability of winning, influenced
instead by news accounts of ‘real’ people who win multi-million dollar prizes
(Rogers (1998); Wheeler and Wheeler (2007); BetterUp (2022)).

Therefore, in this work, we mathematically define spatial black swan events by
_subjective probability_ in Sections 5 and 7. We further assert that among the
four rational axioms, independence and continuity should be modified with a
subjective probability function $w:[0,1]\rightarrow[0,1]$ to fully address
human-decision making, which will be elaborated as _human MDP_ in Subsection
5.2. The function $w$ could be interpreted as humans’ _implicit probability
belief_ (Simon (1993)). We specify the property of function $w$ at Assumption
3 in Section 5 inspired by some case studies of human behavior from Kahneman
and Tversky (2013).

  1. 1.

_Subjective_ independence: If $A$ is preferred to $B$ and an event probability
$p\in[0,1]$, then humans perceive that $w(p)A+(1-w(p))C$ should be preferred
to $w(p)B+(1-w(p))C$, while in _real-world_ , $pA+(1-p)C$ should be preferred
to $pB+(1-p)C$.

  2. 2.

_Subjective_ continuity: If $A$ is preferred to $B$ and $B$ is preferred to
$C$, there exists an event probability $p\in[0,1]$ such that $B$ is considered
equally preferable to $pA+(1-p)C$ in the real-world. However, humans perceive
that there exists $B^{\prime}\neq B$ that is equally preferable to
$w(p)A+(1-w(p))C$.

We elaborate on how subjective independence and subjective continuity could
yield different optimal policies in the human-perceived world (which will be
termed as Human MDP) and in the real-world (which will be termed as Universe
MDP), as demonstrated through Examples 2 and 3, respectively. In particular,
we further describe the emergence of spatial black swan events due to
subjective probability in Example 2. First, Example 2 shows an example of
modified $3^{rd}$ rational axiom.

######  Example 2 (Spatial black swan due to underestimation of low event
probability).

Suppose that humans can access the true utility function. The utilities of the
outcomes $a,b,c\in\mathcal{O}$ are given as $u(a)=-1,u(b)=-1000,u(c)=1000$,
where $b$ is an extremely risky outcome. Suppose that the choices $A,B,$ and
$C$ return outcomes $a,b,$ and $c$ with $100\%$ certainty, respectively. Then,
humans choose two compound choices $L$ and $M$ where $L$ returns $0.001\%$ of
$A$ and $99.999\%$ of $C$, and compound choice $M$ returns $0.001\%$ of $B$
and $99.999\%$ of $C$. Through several trials, due to the extreme rarity of
occurrence of choices $A$ or $B$, regardless of choosing $L$ or $M$, humans
perceive that $L$ and $M$ return the same utility value of $1000$, which can
be interpreted as $w(0.001\%)=0\%$ and $w(99.999\%)=100\%$, and decide their
optimal policy is random sampling among choices $\\{L,M\\}$. However, over
time, humans have a probability of encountering choice $B$ with a probability
of $0.0005\%$, which is a black swan.

The modified $4^{th}$ rational axiom, irrational continuity, does not
intuitively have a direct relationship with the emergence of black swans, but
it provides how preference could be reversed and leads to an irrational
decision.

######  Example 3 (How subjective probability leads to irrational risky
decision).

Continuing from Example 2, suppose that the compound choice $N$ returns $B$
with $50.05\%$ and $C$ with $49.95\%$. Then, in the real world, choice $N$ and
choice $A$ return the same utility as $u(b)\cdot p(b)+u(c)\cdot
p(c)=-1000\cdot 0.5005+1000\cdot 0.4995=-1$. However, if humans perceive that
choice $N$ returns $B$ with $50.01\%$ and $C$ with $49.99\%$, they consider
choice $N$ to return the value $u(b)\cdot p(b)+u(c)\cdot p(c)=-1000\cdot
0.5001+1000\cdot 0.4999=-0.2$, and thus determine their optimal policy as
choosing the choice $N$.

In reality, choosing between choice $N$ and $A$ does not matter since both
provide the same utility value, but choosing choice $N$ is actually more risky
than choice $A$ since choice $N$ contains the high-risk outcome $B$.

Note that the result $B$ in Example 3 is not a black swan event, since it is
not a rare event. However, through these examples 2 and 3, it is clear that
human misperception (subjective probabilities) could yield different optimal
policies in the real-world.

##  4 Cumulative prospect theorem

In this section, we provide a preliminary overview of Cumulative Prospect
Theory (CPT), which offers a framework for understanding human decision making
under risk and uncertainty. We utilize the principles of CPT to elaborate the
concept of misperception and incorporate it into the MDP (Section 5) and
further to define spatial black swan events (Section 7).

For a random variable $X$, let $p_{i}$ where $i=1,\dots,K$ denote the indices
for the probability of incurring a gain or loss $x_{i}$ for each
$i=1,\dots,K$. Given a utility function $u$ and a weighting function $w$, the
_Prospect Theory (PT)_ value is defined as
$V(X)=\sum_{i=1}^{K}u(x_{i})w(p_{i})$, and the _Cumulative Prospect Theory
(CPT)_ value is defined as
$V(X)=\sum_{i=1}^{K}u(x_{i})\left(w(\sum_{j=1}^{i}p_{j})-w(\sum_{j=1}^{i-1}p_{j})\right)$.
Contrary to expected utility theory, which models decisions that perfectly
rational agents would make (Rabin (2013)), i.e.
$V(x)=\sum_{i=1}^{K}x_{i}p_{i}$, PT seeks to describe the actual behavior of
humans, attempting to encompass their irrational decision-making processes.
Specifically, PT introduces the concept of _probability distortion_ , where
individuals overestimate the likelihood of rare events and underestimate the
likelihood of moderate to highly probable events (Figure 1(b)). _Value
distortion_ refers to the way individuals assess gains and losses ($x$-axis of
Figure 1(a)), often valuing losses more heavily than equivalent gains, which
reflects a behavior known as loss aversion (Figure 1(a)) Kahneman and Tversky
(2013); Fennema and Wakker (1997). How PT explains human decision-making is
well-described in the following example.

######  Example 4 (Insurance policies).

Consider an example where the probability of an insured risk is $1$%, the
potential loss is $1,000$, and the insurance premium is $15$. According to
CPT, most would opt to pay the $15$ premium to avoid the larger loss.

Example 4 illustrates how a seemingly straightforward decision can be analyzed
as a sequential decision-making problem within a two-step Markov Decision
Process framework, where ${\mathcal{S}}=\\{s_{base},s_{premium},s_{risk}\\}$
and ${\mathcal{A}}=\\{a_{p},a_{np}\\}$. The states $s_{base},s_{premium},$ and
$s_{risk}$ represent receiving a loss of $0,-15$, and $-1000$, respectively.
The actions $a_{p}$ and $a_{np}$ denote paying and not paying the premium,
respectively. At time $t=0$, humans make a choice between $a_{p}$ and $a_{np}$
based on a policy $\pi:{\mathcal{S}}\rightarrow\Delta({\mathcal{A}})$ stating
at initial state $s_{0}=s_{base}$. Choosing $a_{p}$ results in a guaranteed
transition to state $s_{1}=s_{preimium}$. Otherwise, choosing $a_{np}$
potentially leads to state $s_{base}$ with a reward of $0$ with a $0.99$
probability, or to state $s_{risk}$ with a reward of $-1000$ with a $0.01$
probability. According to the expected utility theorem, which assumes
rationality, the estimated value of choosing $a_{p}$ is calculated as
$V(s_{base})=1\times r(s_{premium})=1\times(-15)=-15$, and the value of
choosing $a_{np}$ as $V(s_{base})=0.99\times r(s_{base})+0.01\times
r(s_{risk})=0.99\times 0+0.01\times(-1000)=-10$. Rationality would lead humans
to prefer $a_{np}$ since its expected cost is lower than that of $a_{p}$, i.e.
$a_{np}=\operatorname*{arg\,max}_{a\in{\mathcal{A}}}V(s_{base})$. However,
this choice is counterintuitive and often does not align with real-world human
decision making.

CPT uses a similar measure as PT, except that the $w$ is a function of
cumulative probabilities. The concept involves using an S-shaped utility
function, which adheres to the diminishing sensitivity property. If we set the
weighting function $w$ or utility function $u$ to be the identity function,
then we retrieve the classical expected utility mode (Rabin (2013)).

##  5 Agent-Environment intersects as perception

Thus far, we have informally introduced the black swan hypothesis (Hypothesis
1) and spatial black swan definition (Definition 1) in Section 1 and
elaborated on its necessity (Subsection 3.1) and supporting evidence
(Subsection 3.2) in Section 3. Then, we introduce one existing work to
concretize misperception, CPT, in Section 4. In Section 5, we utilize CPT to
elaborate the Hypothesis 1 by introducing universe, human, and human-
estimation MDP.

###  5.1 Misperception is information loss

Based on Hypothesis 1, this prompts us to investigate the concept of
_misperception_. Initially, we must clearly define what constitutes
_perception_. In _The Quest for a Common Model of the Intelligent Decision
Maker_ , Sutton defines perception as one of four principal components of
agents, stating: “The perception component processes the stream of
observations and actions to produce the subjective state, a summary of the
agent-world interaction so far that is useful for selecting action (the
reactive policy), for predicting future reward (the value function), and for
predicting future subjective states (the transition model)” Sutton (2022).
This definition leads us to consider misperception as the _information loss_
occurring when processing observations into the subjective state, such that
the reward and transition model are not equivalent to those from the
environment. The interpretation of misperception as _information loss during
processing_ is somewhat ambiguous, depending on how the boundary between the
agent and the environment is defined. The concept of a boundary between the
agent and environment was first proposed by Turing as a ‘skin of an onion’
Turing (2009), and later, Jiang (2019) suggested that algorithms are not
boundary-invariant.

Therefore, we propose a new agent-environment framework that incorporates the
notion that _misperception is the information loss from an agent’s
processing_. This framework positions perception at the intersection between
the agent and the environment. We provide a detailed description of our agent-
environment framework in Figure 2 and further elaborate in the following
subsection.

###  5.2 Universe, Human, and Human-Estimation MDP

Restricting the Markov Decision Process introduced in Section 3 in a
stationary environment, we consider a single episode finite horizon stationary
Markov Decision Process (MDP) denoted as
$\mathcal{M}=\langle\mathcal{S},\mathcal{A},P,R,\gamma,T\rangle$, where
$\mathcal{S}$ represents the state space, $\mathcal{A}$ denotes the action
space, $P:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S})$ is the
transition probability function, $R:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$
is a reward function, $\gamma$ is the discount factor, and $T\geq 0$ is a
Horizon. We define $\mathcal{M}$ as the universe MDP and operate under the
assumption that the abstraction from the universe to the model $\mathcal{M}$
is lossless, preserving all relevant information. Given a policy $\pi$, the
agent collects data $\\{s_{0},a_{0},r_{0},s_{1},\cdots\\}$ as it interacts
with the environment at discrete time steps $t$. The process starts from a
fixed initial state $s_{0}$, and we define the value function as follows:

$V(s_{0}):=\mathbb{E}\left[\sum_{t=0}^{T-1}R(s_{t},a_{t})\mid\pi,P\right]$ (1)

To fully leverage Hypothesis 1, and ‘perception as information loss from
agent’s processing’ from subsection 5.1, we define the human MDP
$\mathcal{M}^{\dagger}=\langle\mathcal{S}^{\dagger},\mathcal{A}^{\dagger},P^{\dagger},R^{\dagger},\gamma\rangle$
where the agent experiences distortions in cumulative distribution of
normalized visitation probability
$P^{\pi}(s,a):=\frac{1-\gamma^{T}}{1-\gamma}\sum_{t=0}^{T-1}\gamma^{t}\mathbb{P}((s_{t},a_{t})=(s,a)|s_{0},\pi,P)$
and reward function $R$ by functions $w^{+},w^{-},u^{+},w^{-}$ (function
characteristics are explained in Assumptions 2 and 3). Note that
$\mathbb{P}(s,a|s_{0},\pi,P)$ is the probability of visiting $(s,a)$ at time
$t$ by policy $\pi$ starting from $s_{0}$.

One internal assumption in the human MDP is that its state and action spaces
are the same as those of the universe MDP, i.e.
${\mathcal{S}}^{\dagger}={\mathcal{S}}$ and
${\mathcal{A}}^{\dagger}={\mathcal{A}}$. This assumption is significant, as
insufficient exploration might result in a limited understanding of the entire
state space by the human (agent), and the discrepancy between the human MDP
and the universe MDP could be considerable, especially in a large discrete
state and action space. However, we can augment the human MDP to match with
universe MDP by using the following method.

######  Remark 2.

If humans cannot perceive some state $s\in{\mathcal{S}}$, we can augment the
human state space with
$\left(S^{\dagger}\right)^{\prime}=S^{\dagger}\cup\\{s\\}$, define
$R^{\dagger}(s,a)=R(s,a)$, and set $P^{\dagger}\left(s^{\prime}\mid
s,a\right)=P\left(s^{\prime}\mid s,a\right)$ while $P\left(s\mid
s^{\prime},a\right)=0$ for all
$s\in\mathcal{S}^{\dagger},a\in\mathcal{A}^{\dagger}$. Thus, the new state $s$
does not affect decision-making in a human MDP since the probability that the
trajectory visits the state $s$ is zero.

From now on, we denote human MDP
$\mathcal{M}^{\dagger}=\langle\mathcal{S},\mathcal{A},P^{\dagger},R^{\dagger},\gamma\rangle$
For a finite state and action space, we can define the order statistics of
$P^{\pi}(s,a)$ for $\forall(s,a)$ and define the cumulative distribution. For
short notation, we use $\int P^{\pi}(s,a)$ as its cumulative distribution.
These distortions are defined by the relationships:

$\displaystyle\int P^{\dagger,\pi}(s,a)=$
$\displaystyle\begin{cases}w^{+}(\int P^{\pi}(s,a))\leavevmode\nobreak\
\text{if}\leavevmode\nobreak\ R(s,a)\geq 0\\\ w^{-}(\int
P^{\pi}(s,a))\leavevmode\nobreak\ \text{if}\leavevmode\nobreak\
R(s,a)<0\end{cases}\leavevmode\nobreak\
,\forall(s,a)\in{\mathcal{S}}\times{\mathcal{A}}$ (2) $\displaystyle
R^{\dagger}(s,a)=$
$\displaystyle\begin{cases}u^{+}(R(s,a))\leavevmode\nobreak\
\text{if}\leavevmode\nobreak\ R(s,a)\geq 0\\\
u^{-}(R(s,a))\leavevmode\nobreak\ \text{if}\leavevmode\nobreak\
R(s,a)<0\end{cases}\leavevmode\nobreak\
,\forall(s,a)\in{\mathcal{S}}\times{\mathcal{A}}$ (3)

We also define the value function of human MDP as follows.

$V^{\dagger}(s_{0}):=\mathbb{E}\left[\sum_{t=0}^{T-1}R^{\dagger}(s_{t},a_{t})\mid\pi,P^{\dagger}\right]$
(4)

In Equation (2), note that misperceptions are applied to the visitation
probability $P^{\pi}$ rather than the transition probability $P$ itself. This
approach is more reasonable, as humans distort the probability of _events_ ,
and we have defined the dimension of spatial black swan _events_ as $(s,a)$
(Definition 2). We explore case studies where humans misperceive $P$ itself in
Section 6. However, we address the following definition and lemma to bridge
the gap between these two types of misperceptions, implying that theoretical
analyses on one are also interchangeably applicable to the other.

######  Definition 5 (Biased and perceived reward and visitation).

For given constant $\kappa_{r},\kappa_{d}\in\mathbb{R}_{+}$, if
$\max_{(s,a)}|R^{\dagger}(s,a)-\widehat{R}^{\dagger}(s,a)|\leq\kappa_{r}$
holds, then $\widehat{R}^{\dagger}(s,a)$ is $\kappa_{r}$-biased reward. Also,
if
$\max_{(s,a)}|P^{\pi,\dagger}(s,a)-\widehat{P}^{\pi,\dagger}(s,a)|\leq\kappa_{d}$
holds, then $\widehat{P}^{\pi,\dagger}(s,a)$ is $\kappa_{d}$-biased visitation
probability. Also, for given constant
$\epsilon_{r},\epsilon_{d}\in\mathbb{R}_{+}$, if
$\max_{(s,a)}|R(s,a)-R^{\dagger}(s,a)|<\epsilon_{r}$ holds, then
$R^{\dagger}(s,a)$ is $\epsilon_{r}$-perceived reward. Also, if
$\max_{(s,a)}|P^{\pi}(s,a)-P^{\pi,\dagger}(s,a)|<\epsilon_{d}$ holds, then
$P^{\dagger,\pi}(s,a)$ is $\epsilon_{d}$-perceived visitation probability.

######  Lemma 1.

If
$\max_{s,a}||P(\cdot|s,a)-P^{\dagger}(\cdot|s,a)||_{1}\leq\frac{(1-\gamma)^{2}}{\gamma}\epsilon_{d}$
where $\epsilon_{d}>0$, then the agent can guarantee $\epsilon_{d}$-perceived
visitation probability (See Definition 5 for the definition of
$\epsilon_{d}$-perceived).

Then, the human MDP $\mathcal{M}^{\dagger}$ is fully characterized by the
utility functions $u^{+},u^{-}$ and the weight functions $w^{+},w^{-}$ derived
from the universe MDP $\mathcal{M}$. We proceed to delineate the properties of
these utility and weight functions as described by (Kahneman and Tversky
(2013)), under the following assumptions:

######  Assumption 2 (Utility function).

A function $u^{+}:\mathbb{R}_{\geq 0}\to\mathbb{R}_{\geq 0}$ is a non-
decreasing concave function that satisfies $\lim_{h\to
0+}(u^{+})^{\prime}(h)\leq 1$. A function $u^{-}:\mathbb{R}_{\leq
0}\to\mathbb{R}_{\leq 0}$ is a nondecreasing convex function that satisfies
$\lim_{h\to 0-}(u^{-})^{\prime}(h)>1$.

######  Assumption 3 (Weight function).

Let $w^{+},w^{-}:[0,1]\to[0,1]$ be a differentiable function, then those
satisfy

  1. 1.

$w^{+}(0)=0,w^{+}(1)=1$ and $w^{-}(0)=0,w^{-}(1)=1$.

  2. 2.

There exists $a,b\in(0,1)$ such that $w^{+}(a)=a,w^{-}(b)=b$.

  3. 3.

$(w^{+})^{\prime}(x)$ is monotonically decreasing on $x\in[0,a)$ and
monotonically increasing on $x\in(a,1]$. Meanwhile, $(w^{-})^{\prime}(x)$ is
monotonically increasing on $x\in[0,b)$ and monotonically decreasing on
$x\in(b,1]$.

Figure 1 illustrates the geometric properties of the utility and weight
functions, as defined under Assumptions 2 and 3. Note that the assumptions
concerning the weight functions (Assumption 3) stipulate that $w^{+}$ and
$w^{-}$ are Lipschitz continuous, with constants $(w^{+})^{\prime}(a)$ and
$(w^{-})^{\prime}(b)$, as derived from the mean value theorem. We denote these
constants as $L^{+}$ and $L^{-}$, respectively.

(a) Utility function $u$

(b) Weight function $w$

Figure 1: Utility and weight functions. A line with a gray color denotes
$y=x$.

Based on the perceptions of the agent, it executes a trajectory using the
policy $\pi$ within the human MDP, $\mathcal{M}^{\dagger}$, and _estimates_
the perceived reward $R^{\dagger}(s,a)$ and visitation probability
$P^{\dagger,\pi}(s,a)$ as $\widehat{R}^{\dagger}(s,a)$ and
$\widehat{P}^{\dagger,\pi}(s,a)$, respectively, from its trajectory. The
estimation error bounds are influenced by the complexity of the MDP, including
tabular MDPs, linear MDPs, and low-rank MDPs. This configuration is defined as
the human-estimation MDP
$\widehat{\mathcal{M}}^{\dagger}=\langle\mathcal{S},\mathcal{A},\widehat{P}^{\dagger},\widehat{R}^{\dagger},\gamma\rangle$.
In a similar way of value function definition in universe MDP as follows.

$\widehat{V}^{\dagger}(s_{0}):=\mathbb{E}\left[\sum_{t=0}^{T-1}\widehat{R}^{\dagger}(s_{t},a_{t})\mid\pi,\widehat{P}^{\dagger}\right].$

Based on the definition of Universe MDP, Human MDP, and Human-Estimation MDP,
we define their gap as follows, and its relationship could be described as
Figure 2.

$\hbox
to0.0pt{$\displaystyle\overbrace{\phantom{{\mathcal{M}}\xLongleftrightarrow[{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\epsilon_{r},\epsilon_{d}}]{\text{perception}}{\mathcal{M}}^{\dagger}}}^{\text{Environment}}$\hss}{\mathcal{M}}\xLongleftrightarrow[\epsilon_{r},\epsilon_{d}]{\text{perception}}\underbrace{{\mathcal{M}}^{\dagger}\xLongleftrightarrow[\kappa_{r},\kappa_{d}]{\text{estimation}}\widehat{{\mathcal{M}}}^{\dagger}}_{\text{Agent}}$
Figure 2: Agent-environment intersects with perception

Since CPT models irrational decision-making, our work begins by defining
modified value functions in an infinite-horizon Markov Decision Process (MDP)
inspired by CPT. As a preliminary step, we start by defining the CPT-value
function in a discrete state and action space.

First, in a discrete state and action space, the value function (Equation (1))
could be expressed as an inner product of reward function $R$ and normalized
occupancy measure $P^{\pi}$ as follows,

$V(s_{0})=\frac{1-\gamma^{H}}{1-\gamma}\sum_{s,a\in{\mathcal{S}}\times{\mathcal{A}}}R(s,a)P^{\pi}(s,a)$
(5)

Based on Definition 5, the CPT distorts the reward and its visitation
probability as follows,

$V^{\dagger}(s_{0})=\frac{1-\gamma^{H}}{1-\gamma}\sum_{s,a\in{\mathcal{S}}\times{\mathcal{A}}}u(R(s,a))\frac{d}{dsda}w\left(\int
P^{\pi}(s,a)\right).$ (6)

where $\dagger$ denotes the value was distorted due to misperception. As one
property of CPT is that human perception exhibits distinct distortions of
events based on whether the associated rewards are positive or negative, we
divide the functions $u(R(s,a))$ and $w(P^{\pi}(s,a))$ into
$u^{-}(R(s,a)),w^{-}(P^{\pi}(s,a))$ where $R(s,a)<0$, and
$u^{+}(R(s,a)),w^{+}(P^{\pi}(s,a))$ where $R(s,a)\geq 0$. Assume that the
rewards from all state-action pairs $R(s,a)$ are ordered as
$R_{[1]}\leq\dots\leq R_{[l]}\leq 0\leq R_{[l+1]}\leq\cdots\leq
R_{[|\mathcal{S}||\mathcal{A}|]}$, and the visitation probability as
$P^{\pi}_{[1]}\leq P^{\pi}_{[2]}\leq\cdots\leq
P^{\pi}_{[|\mathcal{S}||\mathcal{A}|]}$. Then, CPT-value function is defined
as follows:

$\displaystyle V^{\dagger}(s_{0})$
$\displaystyle=\sum_{i=1}^{|{\mathcal{S}}||{\mathcal{A}}|}u(R_{[i]})\left(w\left(\sum_{j=1}^{i}P^{\pi}_{[j]}\right)-w\left(\sum_{j=1}^{i-1}P^{\pi}_{[j]}\right)\right)$
$\displaystyle=\sum_{i=1}^{l}u^{-}(R_{[i]})\left(w^{-}\left(\sum_{j=1}^{i}P^{\pi}_{[j]}\right)-w^{-}\left(\sum_{j=1}^{i-1}P^{\pi}_{[j]}\right)\right)$
$\displaystyle\quad+\sum_{i=l+1}^{|{\mathcal{S}}||{\mathcal{A}}|}u^{+}(R_{[i]})\left(w^{+}\left(\sum_{j=i}^{|{\mathcal{S}}||{\mathcal{A}}|}P^{\pi}_{[j]}\right)-w^{+}\left(\sum_{j=i+1}^{|{\mathcal{S}}||{\mathcal{A}}|}P^{\pi}_{[j]}\right)\right)$
(7)

If we define the reward as the random variable $X$, then we can regard its
instance as $R_{[i]}$ and its probability as $P^{\pi}_{[i]}$ where the
probability is dependent on the policy $\pi$. Suppose that reward function
$R:{\mathcal{S}}\times{\mathcal{A}}\to\mathbb{R}$ is one to one function. Then
the probability $R^{-1}\circ P^{\pi}:\mathbb{R}\to[0,1]$ denotes the
probability of reward and we denote it as $\mathbb{P}_{r}$. Then, for a reward
random variable ${\mathcal{R}}\sim\mathbb{P}_{r}$, expanding the how CPT-
applied value function look like in Equation (4), we can define the value
function based on continuous state and actions space as follows.

$V^{\dagger}(s_{0})=\int_{0}^{\infty}w^{+}\left(\mathbb{P}_{r}(u^{+}({\mathcal{R}})>r)\right)dr-\int_{0}^{\infty}w^{-}\left(\mathbb{P}_{r}(u^{-}({\mathcal{R}})>r)\right)dr$
(8)

We use the fact that for real-value function $g$, it holds that
$\mathbb{E}[g({\mathcal{R}})]=\int_{0}^{\infty}\Pr(g({\mathcal{R}})>r))dr$. In
this sense, we define the black swan event in the continuous state and action
space.

###  5.3 Problem setting

Based on three different MDPs,
${\mathcal{M}},{\mathcal{M}}^{\dagger},\widehat{{\mathcal{M}}}^{\dagger}$, we
consider the following problem setting. The agent rolls out in a single
episode with a finite horizon $T$. If the agent has an unbiased perception,
then the agent collects a trajectory
$\\{s_{0},a_{0},r_{0},s_{1},a_{1},\cdots,s_{T-1},a_{T-1},s_{T}\\}$. However,
the agent perceives ${\mathcal{M}}$ as ${\mathcal{M}}^{\dagger}$ and now
observe distorted state and reward as
$\\{g(s_{0}),a_{0},u(r_{0}),g(s_{1}),a_{1},\cdots,g(s_{T-1}),a_{T-1},g(s_{T})\\}$
where function $g:{\mathcal{S}}\to{\mathcal{S}}$ distorts the state. Now, we
can claim the following

######  Lemma 2.

Suppose the ${\mathcal{M}}$ is given. Then, for any function $w$ that
satisfies the Assumption 3, one can always find the function
$g:{\mathcal{S}}\to{\mathcal{S}}$ that satisfies the following equation.

$w(\int P^{\pi}(s,a))=\int P^{\pi}(g(s),a)$

Note that Lemma 2 allows us to design the problem such that the agent distorts
the visitation probability by receiving a distorted state induced by the
function $g$. Lemma 2 justifies how the agent distorts the visitation
probability from the trajectory data.

###  5.4 Utilizing CPT for black swan

We note that existing work on incorporating cumulative prospect theory (CPT)
into reinforcement learning, such as (Prashanth et al. (2016); Jie et al.
(2018); Danis et al. (2023)), primarily focus on estimating the CPT-based
value function and optimizing it to derive an optimal policy. Specifically,
(Prashanth et al. (2016); Jie et al. (2018)) demonstrate how to estimate the
CPT value function using the Simultaneous Perturbation Stochastic
Approximation method and how to compute its gradient for policy optimization
algorithms. Additionally, (Shen et al. (2014); Ratliff and Mazumdar (2019))
proposed a novel Q-learning algorithm that applies a utility function to
Temporal Difference (TD) errors and demonstrated its convergence. However,
these studies (Prashanth et al. (2016); Jie et al. (2018); Danis et al.
(2023); Shen et al. (2014); Ratliff and Mazumdar (2019)) do not focus on
learning the utility and weight functions, $u$ and $w$, but rather assume
these as simple functions and focus on how to _estimate_ these functions.

However, this study aims to elucidate the mechanisms by which black swan
events arise from the discrepancies between $\mathcal{M}^{\dagger}$ and
$\mathcal{M}$, despite the agent having perfect estimation, i.e.,
$\kappa_{r}=0,\kappa_{p}=0$. As future work, concentrating on devising
strategies to _reweight_ the functions $u_{+},u_{-}$, and $w$ to mitigate the
divergence between the Human MDP $\mathcal{M}^{\dagger}$ and the universe MDP
$\mathcal{M}$ is suggested as a way to achieve antifragility.

##  6 Case study: how optimal decision deviates under irrationality

So far, we have informally introduced the black swan hypothesis (Hypothesis 1)
and spatial black swans (Definition 1) in Section 1, then elaborated on its
necessity (Subsection 3.1) and provided evidence (Subsection 3.2) in Section
3. Subsequently, in Section 4, we introduced the Cumulative Prospect Theorem
(CPT) to model human irrationality. In Section 5, we incorporated CPT into the
Markov Decision Process (Subsection 5.2), viewed through the lens of existing
work on defining perception (Subsection 5.1) by introducing universe, human,
and human-estimation MDPs (Figure 2).

Given our main hypothesis that black swan events occur due to human
misperception of the real world, we further investigate whether optimal policy
also deviates due to misperceptions of _value or probability_. This is
critical as overestimating or underestimating all values of probability does
not necessarily lead to changes in optimal policy. For example, revisiting
Example 4, overestimating or underestimating
$\\{r(s_{base}),r(s_{premium}),r(s_{risk})\\}$ to the same magnitude does not
alter the optimal policy for a human to choose $a_{p}$. More fundamentally, a
crucial question that this paper addresses is how misperception influences the
deviation of optimal policy. Therefore, in Section 6, we first present some
case studies in MDPs with small complexity to demonstrate how optimal policy
varies under misperception.

Before proceeding, we need to establish the core event of _subjective
probability_. For example, while agents (humans) might distort transition
probabilities and perceive them subjectively at a low level which is an
intuitive way to model misperception. It is essential to recall that in the
Markov Decision Process, the quality of an event is revealed through rewards
defined over specific states and actions. This suggests that it is more
reasonable to define the minimal object (event) as the state and action, and
conduct modeling as distortion on the visitation probability of state and
action rather than on the transition probability itself. Since this approach
has not been investigated in existing work, we examine both cases in Section 6
and Section 7. Specifically, in Section 6, we assume distortion of transition
probability within a non-stationary Markov decision process and investigate
how optimal policy deviates due to misperception of transition probability.
Then, in Section 7, we explore the distortion of the visitation probability
within a stationary Markov decision process.

###  6.1 Problem setup for Section 6

In this section, we consider discrete state and action stationary Markov
Decision Process. Build upon value function (Equation 4), we define value
function and state value function of time $t$ as

$\displaystyle V^{\pi,\dagger}_{t}(s)$
$\displaystyle:=\mathbb{E}\left[\sum_{t^{\prime}=t}^{T-1}\gamma^{T-1-t^{\prime}}u(R(s_{t^{\prime}},a_{t^{\prime}}))\mid
P^{\dagger},\pi,s_{t}=s\right]$ (9) $\displaystyle Q^{\pi,\dagger}_{t}(s,a)$
$\displaystyle:=\mathbb{E}\left[\sum_{t^{\prime}=t}^{T-1}\gamma^{T-1-t^{\prime}}u(R(s_{t^{\prime}},a_{t^{\prime}}))\mid
P^{\dagger},\pi,s_{t}=s,a_{t}=a\right]$

and define the optimal policy as time $t$ as
$\pi_{t}^{\star,\dagger}=\operatorname*{arg\,max}_{\pi}V_{t}^{\pi,\dagger}$.
Then the following bellman equation holds,

$\displaystyle V^{\pi,\dagger}_{t}(s)$
$\displaystyle=\sum_{a\in{\mathcal{A}}}\pi(a|s)Q^{\pi,\dagger}_{t}(s,a)$
$\displaystyle Q^{\pi,\dagger}_{t}(s,a)$
$\displaystyle=u(R(s,a))+\sum_{i\in|{\mathcal{S}}|}\gamma\left(w\left(\sum_{j=1}^{i}P(s^{\prime}_{[i]}\mid
s,a)\right)-w\left(\sum_{j=1}^{i-1}P(s^{\prime}_{[i]}\mid
s,a)\right)\right)V_{t+1}(s_{[i]})$

where $P(s_{[1]}|s,a)\leq P(s_{[2]}|s,a)\leq\cdots\leq P(s_{[|S|]}|s,a)$
holds. In addition, we assume that $\forall t\in[T-1],\forall s\in\mathcal{S}$
and $\forall a\in\mathcal{A},R(s_{t},a_{t})=0$. We only consider the reward
function at the final stage $R(s_{T},a_{T})$. Although this assumption may
appear unconventional, it aligns with standard practices in reinforcement
learning, especially when focusing on terminal rewards. Especially, for each
trajectory $\tau\in\mathcal{S}^{\prime}$, where
$\tau=\left(s_{0},a_{0},\ldots,s_{H},a_{H}\right)$ represents a $T$-step
decision sequence, set the terminal reward as
$r_{T}^{\prime}(\tau):=\sum_{t\in[T]}r_{t}\left(s_{t},a_{t}\right)$, and
$r_{t}^{\prime}(\tau)=0$ for all $t\in[T-1]$

###  6.2 Case 1. Contextual bandit ($T=1$)

We begin with a simple case where the decision horizon is $T=1$, commonly
referred to as a contextual bandit (Lattimore and Szepesvári (2020)).
Surprisingly, in this setting, the human optimal policy coincides with the
real-world optimal policy. This is somewhat counterintuitive, as several
significant examples (Examples 2, 3, and 4) suggest that human decision-making
often exhibits irrationality.

######  Theorem 1 (One-step Human Optimal Policy).

If $T=1$, then the optimal policy from the universe MDP aligns with the
optimal policy of the human MDP, i.e. $\pi^{\star}=\pi^{\star,\dagger}$.

An important insight from Theorem 1 is that when decisions are not sequential,
the typical distortions in human perception do not affect the alignment with
the optimal policy of the real-world, suggesting that human irrationality is
less influential in a single-step decision-making setting. This is further
explained in the following remark.

######  Remark 3.

Continuing from Example 4, the order of perceived rewards does not change as
$u^{-}(r(s_{loss}))<u^{-}(r(s_{premium}))<u^{-}(r(s_{base}))$ since $u^{-}$ is
a nondecreasing convex function.

###  6.3 Case 2. $|\mathcal{S}|=2$ when $T>1$

Now, let us consider the simplest case where $H>1$ where $|\mathcal{S}|=2$.
Surprisingly, we find results similar to those presented in Section

######  Theorem 2 (Multi-step human optimal policy).

If $|\mathcal{S}|=2$, then the optimal policy from the universe MDP also
aligns with the optimal policy of the human MDP, that is,
$\pi^{\star,\dagger}_{t}=\pi^{\star}_{t}$ for $\forall t\in[T]$.

The proof of Theorem 2 is based mainly on the assumption that
$|\mathcal{S}|=2$, notably using the property that $w(1)=1$. However, this
technique cannot be applied directly if $|\mathcal{S}|\geq 3$. For example, in
Example 4, where $|\mathcal{S}|=3$, it is demonstrated that human decision
making results in a suboptimal policy in the real-world. This outcome may seem
counterintuitive. However, a heuristic analysis suggests that having only two
states in the state space indicates that the actions do not introduce a varied
randomness. Essentially, if randomness is introduced by any action, it would
likely affect both states $s_{0}$ and $s_{1}$ if the action provides a
nondeterministic next state. Thus, every action provides the same next-state
set, which makes a mere comparison between two states. This observation
implies that in scenarios with a smaller state space, being irrational
(believing on false belief) does not affect to deviate from optimal policy. We
can also interpret this result as if $|{\mathcal{S}}|$ is small, sequential
decision-making problems are _easy_ so that humans can always provide the
optimal action in the real world.

###  6.4 Case 3: $|S|=3$ with unbiased reward perception

We consider the hypothetical scenario where $u(r)=r$, indicating that humans
have an unbiased perception of their reward.

######  Theorem 3 (Two step optimal decision when $|{\mathcal{S}}|=3$).

Given any state space ${\mathcal{S}}$ where $|\mathcal{S}|=3$ and a decision
horizon $T=2$, there exists a transition probability $P$ and reward $R$ of the
real world such that the optimal policy of human MDP differs from that of
universe MDP.

Theorems 1, 2, and 3 illustrate that the discrepancy between human and optimal
decision making increases as the complexity of the environment increases
(${\mathcal{S}}$) or as the decision sequence ($T$) lengthens for any $w$
function.

##  7 A definition of spatial black swans

Now, we introduce the definition of black swan inspired by CPT.

###  7.1 Black swan hypothesis

First, based on the newly proposed human model ${\mathcal{M}}^{\dagger}$, we
concretize the informal hypothesis 1 as follows.

######  Hypothesis 2 (Spatial black swan origin).

For a given universe MDP $\mathcal{M}$, the agent initially perceives the
universe as $\mathcal{M}^{\dagger}$. Then it estimates $\mathcal{M}^{\dagger}$
as $\widehat{\mathcal{M}}^{\dagger}$. Subsequently, the agent rolls out
trajectories in $\widehat{\mathcal{M}}^{\dagger}$ to learn its optimal policy,
$\widehat{\pi}^{\dagger,\star}$. Then, all spatial black swan events $(s,a)$
are functions of $\mathcal{M}$, $\mathcal{M}^{\dagger}$, and
$\pi^{\dagger,\star}$.

We define all spatial black swan events as a set
$\mathcal{B}(\mathcal{M},\mathcal{M}^{\dagger},\pi^{\dagger,\star})$. It is
important to note that $\mathcal{B}$ is a function of the agent’s perception
$\mathcal{M}^{\dagger}$, rather than its estimation
$\widehat{\mathcal{M}}^{\dagger}$, since estimation is what the agent can
improve over time. Specifically, all state and action events from
$\mathcal{B}(\mathcal{M},\widehat{\mathcal{M}}^{\dagger},\widehat{\pi}^{\dagger,\star})$
can be regarded as broad ‘risks’ that encompass spatial black swan events.
Among these risks, there are some that the agent can ‘be averse to’ and others
that it ‘cannot be averse to’ in the process of improving model estimation
$\widehat{\mathcal{M}}^{\dagger}$ to the human model
${\mathcal{M}}^{\dagger}$. The crux of spatial black swan events lies in their
unexpectedness, which occurs even when the agent has prepared the optimal
policy with zero estimation error. The model estimation process discussed in
this paper aligns with existing work on imperfect model-based reinforcement
learning (Jiang (2018); Gheshlaghi Azar et al. (2013); Fard and Pineau (2010);
Agarwal et al. (2020); Lecarpentier and Rachelson (2019)). For future work, we
leave room to define the ‘risks’ due to irrational human behavior that goes
beyond spatial black swans. Specifically, events from
$\mathcal{B}(\mathcal{M},\widehat{\mathcal{M}}^{\dagger},\widehat{\pi}^{\dagger,\star})\setminus\mathcal{B}(\mathcal{M},\mathcal{M}^{\dagger},\pi^{\dagger,\star})$
are risk events that can be avoided as time goes by, and events from
$\mathcal{B}(\mathcal{M},\mathcal{M}^{\dagger},\pi^{\dagger,\star})$ are non-
avoidable risk, i.e. spatial black swans. This conceptual framework can be
outlined by the following equation:

$\lim_{\kappa_{r},\kappa_{d}\to
0}\underbrace{\mathcal{B}(\mathcal{M},\widehat{\mathcal{M}}^{\dagger},\widehat{\pi}^{\dagger,\star})}_{\text{Risks}}=\underbrace{\mathcal{B}(\mathcal{M},\mathcal{M}^{\dagger},\pi^{\dagger,\star})}_{\text{Spatial
black swans}}.$ (10)

Equation (10) is supported by Definition 5, where $\kappa_{r},\kappa_{d}\to 0$
leads to $|R(s,a)-\widehat{R}^{\dagger}(s,a)|\to 0$ and
$|P^{\pi}(s,a)-\widehat{P}^{\pi,\dagger}(s,a)|\to 0$. This implies that
$\widehat{\mathcal{M}}^{\dagger}$ converges to $\mathcal{M}^{\dagger}$ by the
definitions of the Human MDP and the Human-Estimation MDP, and consequently,
$\widehat{\pi}^{\dagger,\star}$ converges to $\pi^{\dagger,\star}$.

###  7.2 A definition of spatial black swans

As a preliminary step, we define spatial black swan events within a discrete
state and action space. Note that we continue the problem setting that is
discussed in deriving Equation (7) in Subsection 5.2.

######  Definition 6 (Spatial black swan - discrete state and action space).

Among the indices of order statistics, if index $i<l$ meets the following
criteria:

  1. 1.

$R_{[i]}-u^{-}(R_{[i]})<-C_{bs}$.

  2. 2.

$w^{-}\left(\sum_{j=1}^{i}P^{\pi}_{[j]}\right)=w^{-}\left(\sum_{j=1}^{i-1}P^{\pi}_{[j]}\right)$,
yet $0<P^{\pi}_{[i]}<\epsilon_{bs}$.

where $C_{bs}\gg 0$ and $\epsilon_{bs}>0$ is given a constant, we define $i$
as a spatial black swan event.

The $1^{\text{st}}$ property of Definition 6 identifies a ‘high-risk event’
through the function gap between the universe reward $R$ from $\mathcal{M}$
and the perceived reward $u^{-}(R)$ from $\mathcal{M}^{\dagger}$.
Specifically, if the agent perceives $R$ in a pessimistic way such that $R\ll
u^{-}(R)<0$, this is considered a high-risk event resulting from
misperception. The $2^{\text{nd}}$ property of Definition 6 also characterizes
a ‘rare event’ through a lens of misperception. Technically, it describes that
a spatial black swan event feasibly occurs in the universe
($0<P^{\pi}_{[i]}$), but the agent perceives it as nonoccurring
($w^{-}\left(\sum_{j=1}^{i}P^{\pi}_{[j]}\right)=w^{-}\left(\sum_{j=1}^{i-1}P^{\pi}_{[j]}\right)$),
i.e. an infeasible event.

In addition, we denote two parameters, $C_{bs}$ and $\epsilon_{bs}$, to
represent the extent of distortion in the reward and the cumulative
probability of visitation. Intuitively, the magnitude of $C_{bs}$ and
$\epsilon_{bs}$ is related to the extent of the misperception gap between
$\mathcal{M}$ and $\mathcal{M}^{\dagger}$,i.e. $\epsilon_{r},\epsilon_{p}$. We
elaborate on this conjecture in Theorem 4 of Section 8. We now extend the
definition of black swan events from discrete state and action spaces to
continuous spaces as follows.

######  Definition 7 (Spatial black swan).

Given $w^{-},u^{-}$, if the state-action pair $(s,a)$ satisfies the following
conditions:

  1. 1.

$R(s,a)-u^{-}(R(s,a))<-C_{bs}$.

  2. 2.

$\frac{d\left(w^{-}(F(r))\right)}{dr}\Bigg{|}_{r=R(s,a)}=0$ but
$0<\frac{dF(r)}{dr}\Bigg{|}_{r=R(s,a)}<\epsilon_{bs}$.

where $F(r):=\int_{-\infty}^{r}d\mathbb{P}_{r}$ represents the cumulative
distribution function of $\mathbb{P}_{r}$, then we define that state and
action pair as a black swan event.

If the weight function $w^{-}(x)$ is differentiable, then the $2^{\text{nd}}$
property of spatial black swan events can be further elaborated as follows:
_The derivative
$\frac{dw^{-}(x)}{dx}\big{|}_{x=F(R(s,a))}\cdot\mathbb{P}_{r}(r=R(s,a))=0$,
yet the probability density distribution remains non-zero and bounded,
specifically $0<\mathbb{P}_{r}(r=R(s,a))<\epsilon_{bs}$_. Consequently, this
implies $\frac{dw^{-}(x)}{dx}\big{|}_{x=F(R(s,a))}=0$. This refined second
property will be utilized to extend the definition of spatial black swan
events to Proposition 3 in Section 8, facilitating further analysis (see
Figure 3(b)).

Also, for all functions $w^{-},u^{-}$ that prevent the existence of spatial
black swan events, i.e., $\mathcal{B}=\emptyset$, we refer to such
environments as safe perception, denoted by $w^{-}_{\star},u^{-}_{\star}$. In
contrast, environments characterized by $w^{-},u^{-}$ that inherently
encompass spatial black swan events, $\mathcal{B}\neq\emptyset$, are termed
risk perception environments. Specifically, if an agent perceives the world
through $w^{-}_{\star},u^{-}_{\star}$, then spatial black swan events are
absent; however, it is crucial to recognize that $w^{-}_{\star},u^{-}_{\star}$
are not unique functions (see Figure 3(b)).

##  8 Theoretical analysis of spatial black swans

Thus far, we have elaborated on the definition of spatial black swan events
(Definition 7) from its informal definition (Definition 1) through the lens of
our black swan hypothesis (Hypothesis 2). In this section, we provide
theoretical analysis that quantifies the impact of spatial black swan events.
Specifically, we demonstrate how the existence of spatial black swans
establishes a lower bound on policy performance (Theorem 4), and assesses the
probability of encountering spatial black swan events over time (Theorem 5).
It is important to note that Theorem 5 serves as a critical precursor to
characterizing the third property of the informal proposition (Proposition 2),
since the timing of learning to improve agent’s perception is triggered by the
statement ‘after a black swan is first observed’. Subsequently, we posit that
perception improvement learning exhibits antifragile behavior in agents.

###  8.1 Problem setting

As a preliminary step, we first establish the agent learning setting and
provide some assumptions for the theoretical analysis. We utilize the problem
setting elaborated in Subsection 5.3. For the theoretical analysis, we assume
the following assumptions:

######  Assumption 4 (Bounded reward).

The $R$ is bounded as $R\in[-R_{\max},R_{\max}]$ where $R_{\max}>0$

######  Assumption 5 (Relatively strong convexity).

With Assumption 2, $u^{-}_{\star}(r)\leq u^{-}(r)$ holds for $r<0$.

Before starting the theoretical analysis, the formal definition of spatial
black swan events based on state and action pairs (Definition 7) imposes
restrictions on further analysis due to the _openness_ of
$(s,a)\in\mathcal{B}$. Specifically, Definition 7 regarding the support of
$\mathcal{S}\times\mathcal{A}$ does not ensure the existence of a closed
subset $\mathcal{C}\subseteq{\mathcal{S}}\times{\mathcal{A}}$ such that
$\forall(s,a)\in\mathcal{C}\implies(s,a)\in\mathcal{B}$. Therefore, we propose
modifications to the support of the spatial black swan event definition
concerning the reward. To this end, we further assume that the reward function
$R:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$ is bijective. Consequently, we
introduce the following proposition of a spatial black swan event, an
alternative to Definition 7, which is defined over reward values in closed
intervals.

######  Proposition 3 (Spatial black swan).

Under Assumptions 1,2, and 3, if $w^{-},u^{-}$ satisfies spatial black swan
definition (Definition 7), then $\forall r\in[-R_{\max},-R_{bs}]$ satisfies

  1. 1.

$r-u^{-}(r)<-C_{bs}$

  2. 2.

$w^{-}(F(r))=0$ but $0<F(r)<\epsilon_{bs}$

where $R_{bs}-u^{-}(R_{bs})=-C_{bs}$ holds.

The Proposition 3 is well elaborated in Figure 3. Proposition 3 facilitates
the Definition 7 within the closed set $[-R_{\max},-R_{bs}]$. This approach to
defining spatial black swan events based on the support of the reward is more
intuitive to understand the black swan origin hypothesis (Hypothesis 2). That
is, Proposition 3 shows how decreasing the misperception gap
(${\mathcal{M}}^{\dagger}\to{\mathcal{M}}$) relates to reducing
$|\mathcal{B}|$. This is because a reduction in the misperception gap, where
$u^{-}(r)\to u^{-}_{\star}(r)$, leads to a decrease in the range of spatial
black swan rewards, moving from $-R_{bs}$ to $-R_{\max}$, thus reducing the
frequency of spatial black swan occurrences (see Figure 3).

Within the above problem setting, the agent’s goal is to estimate the value
function under safe perception ${\mathcal{M}}$ as follows:

$V(s_{0})=\int_{0}^{\infty}w^{+}\left(\mathbb{P}_{r}(u^{+}(X)>r)\right)dr-\int_{0}^{\infty}w^{-}_{\star}\left(\mathbb{P}_{r}(u^{-}_{\star}(X)>r)\right)dr$
(11)

However, the agent possesses its own perceptions ${\mathcal{M}}^{\dagger}$,
for which we assume the risk perception is represented as:

$V^{\dagger}(s_{0})=\int_{0}^{\infty}w^{+}\left(\mathbb{P}_{r}(u^{+}(X)>r)\right)dr-\int_{0}^{\infty}w^{-}\left(\mathbb{P}_{r}(u^{-}(X)>r)\right)dr$
(12)

As time goes by, the agent’s goal is learning the weight functions and utility
functions such as $w^{-}\to w^{-}_{\star}$ and $u^{-}\to u^{-}_{\star}$. Then,
by the single trajectory data up to time $t$, i.e.
$\\{s_{i},a_{i},u(r_{i}),s_{i+1}\\}_{i=0}^{t}$ where the reward value itself
and its sampling distribution are distorted due to the functions $u$ and $w$,
respectively. Let $r_{i},i=1,..,t$ denote $n$ samples of the reward random
variable $X$. We define the empirical distribution function for $u^{+}(X)$ and
$u^{-}(X)$ as follows

$\hat{F}^{+}_{t}(r)=\frac{1}{t}\sum_{i=1}^{n}\boldsymbol{1}_{(u^{+}(r_{i})\leq
r)},\quad\text{and}\quad\hat{F}^{-}_{t}(r)=\frac{1}{t}\sum_{i=1}^{n}\boldsymbol{1}_{(u^{-}(r_{i})\leq
r)}$

. using the EDFs, the CPT value up to time $t$ can be estimated as follows,

$\widehat{V}^{\dagger}_{t}(s_{0})=\int_{0}^{\infty}w^{+}\left(1-\hat{F}^{+}_{t}(r)\right)dr-\int_{0}^{\infty}w^{-}\left(1-\hat{F}^{-}_{t}(r)\right)dr$
(13)

Again, we note that the gap between ${\mathcal{M}}$ and
${\mathcal{M}}^{\dagger}$ is defined over a gap between $(u^{-},w^{-})$ and
$(u^{-}_{\star},w^{-}_{\star})$ that is proportional to the existence of
spatial black swan events.

(a) Utility function $u$

(b) Weight function $w$

Figure 3: Utility and weight functions with spatial black swans.

###  8.2 Theortical analysis

Based on the perspective of the blackswan event as a reward (Proposition 3),
the natural question would be how does the perception gap create a gap between
the value function?

######  Theorem 4 (Convergence of estimation value but lower bound on
perceived value gap).

Under Assumptions 2, 3, and 4, the asymptotic convergence of the value
function estimation holds as follows,

$\widehat{V}^{\dagger}_{t}(s_{0})\to
V^{\dagger}_{t}(s_{0})\quad\text{a.s.}\quad\text{as}\quad t\to\infty.$ (14)

where upper bound of the estimation error (sample complexity) holds as:

$P(|\widehat{V}^{\dagger}_{t}-V^{\dagger}_{t}|<\epsilon)\leq
1-4e^{-t\frac{\epsilon^{2}}{2c^{2}}}.$ (15)

where $c=\max\\{|L^{+}u^{+}(R_{\max})|,|L^{-}u^{-}(-R_{\max})|\\}$.

However, under specific conditions on
$\epsilon_{bs},\epsilon^{\min}_{bs},R_{bs}$, the lower bound of human value
function gap is expressed as:

$|V^{\dagger}_{t}(s_{0})-V_{t}(s_{0})|=\Omega\left(\frac{\left(\left(R_{\max}-R_{bs}\right)\epsilon^{\min}_{bs}-R_{bs}\epsilon_{bs}\right)(R_{\max}-R_{bs})C_{bs}}{R^{2}_{\max}}\right)$
(16)

due to misperception.

Insights of Theorem 4. The message of Theorem 4 establishes a support for our
hypothesis 2 and Defintion 7. Firstly, Equation (15) establishes that the
estimation error converges to zero as the agent rolls out a longer trajectory.
However, Equation (16) offers the insight that the value function gap between
${\mathcal{M}}$ and ${\mathcal{M}}^{\dagger}$ has a lower bound, regardless of
the trajectory length. It is straightforward to verify in Equation (16) that
if $u^{-}(x)\to u^{-}_{\star}(x)$ and $w^{-}(x)\to w^{-}_{\star}(x)$, which
implies $R_{bs}\to R_{\max}$ and $\epsilon_{bs}\to 0$, then the lower bound
also converges to zero. Furthermore, Equation (16) supports our intuition that
a higher distortion in reward perception (a large $C_{bs}$ value) and a
greater number of spatial black swan events (large $(R_{\max}-R_{bs})$) and a
large minimum probability of spatial black swan event occurrence (large
$\epsilon^{\min}_{bs}$) lead to a higher lower bound. It is important to note
that the gap $R_{\max}-R_{bs}$ is associated with the number of spatial black
swans, since we have assumed that the reward function maps uniquely, therefore
there exists a unique $(s,a)$ such that $R(s,a)=r$, where $r$ is in the range
$[-R_{\max},R_{\max}+R_{bs}]$. Therefore, through Theorem 4, we conclude that
even though the agent has a perfect estimation of what it has perceived, there
still exists a lower bound to obtaining the value function of the universe,
which increases as the parameter effect of spatial black swan events becomes
more dominant.

Now, based on Theorem 4, the next natural question is _how to decrease the
lower bound_ , that is, how the agent learns to nudge $u^{-}_{\star}\to
u^{-}_{\star}$ and $w^{-}_{\star}\to w^{-}_{\star}$. Before we ask when the
agent encounters the black swan, so the perception correction would happen.
Since black swan events are defined over probability, this question could be
more concretized as _If the agent takes a step $h>h_{bs}$, the probability of
encountering black swan events are at least $\delta_{bs}$._ This question also
further enables to naturally learn how to obtain antifragile behavior, which
makes $\delta_{bs}$ to decrease as every time the agent updates its
perception. We first start our analysis based on the non-zero one-step
reachability assumption.

######  Theorem 5 (Spatial black swan hitting time).

Assume $\mathbb{P}_{\pi^{\star}}(s^{\prime}|s)>0$ for any two states
$s,s^{\prime}\in\mathcal{S}$, signifying that the one-step state reachability
is non-zero, and consider that one step corresponds to a unit time. Then, if
the agent takes $t$ steps such that

$t\geq\frac{\log\left(\frac{\delta}{p_{\min}}\right)}{\log(1-p_{\max})}+1,$

where $p_{\min}=\frac{R_{\max}-R_{bs}}{2R_{\max}}\epsilon^{\min}_{bs}$ and
$p_{\max}=\frac{R_{\max}-R_{bs}}{2R_{\max}}\epsilon_{bs}$, it will encounter
spatial black swan events at least with probability $\delta\in[0,1]$.

##  9 Conclusion

In this paper, we propose a new perspective to understand the black swan
events by utilizing human misperception and CPT. We have divided black swans
into temporal and spatial black swans and focus on spatial black swans, where
misperception occurs in the state and action space. To define spatial black
swans, we propose three different MDPs: universe MDP (${\mathcal{M}}$), human
MDP (${\mathcal{M}}^{\dagger}$), and human-estimation MDP
($\widehat{{\mathcal{M}}}^{\dagger}$) and also introduce the well-known
irrational human behavior theorem, CPT. The main message is Hypothesis 2 and
Definition 7. They highlight that despite humans being able to reduce the
estimation gap, leading $\widehat{\mathcal{M}}^{\dagger}$ to converge to
$\mathcal{M}^{\dagger}$ through longer horizons or better algorithms, there
remains an inherent uncertainty due to the perception gap between
$\mathcal{M}$ and $\mathcal{M}^{\dagger}$. This implies that spatial black
swans can still occur because of this perception gap, even when the agent has
zero estimation error and computes its optimal policy from
$\widehat{\mathcal{M}}^{\dagger}$.

Acknowledgments and Disclosure of Funding

We thank Jason Jangho Choi for providing instructive insights on the early
draft, and Donghao Ying for discussions on mathematical notation. The authors
also thank Theophane Weber and Csaba Szepesvari and the rest of the Google
DeepMind team who reviewed the paper.

##  Appendix A Proofs

We first start from the following Lemma 3 to prove Lemma 1.

######  Lemma 3 (Bounding visitation probability of step $h$ when well-
perceived transition holds).

If for all $(s,a)$ holds $\epsilon_{d}$-perceived transition probability, then
we have

$\max_{\pi}\left(\sum_{(s,a)\in{\mathcal{S}}\times{\mathcal{A}}}\left|\mathbb{P}^{\pi}_{h}(s,a)-\mathbb{P}^{\pi,\dagger}_{h}(s,a)\right|\right)\leq
h\epsilon_{p}$

that holds for all $h\in\mathbb{N}$

###### Proof of Lemma 3.

Proof by induction. We use short notation for $P(s_{h}=s\mid
s_{h-1}=s^{\prime},a_{h-1}=a^{\prime})$ as $P_{h}(s\mid
s^{\prime},a^{\prime})$ and $P^{\dagger}(s_{h}=s\mid
s_{h-1}=s^{\prime},a_{h-1}=a^{\prime})$ as $P^{\dagger}_{h}(s\mid
s^{\prime},a^{\prime})$. By the definition of rational transition probability
the statement holds at $h=1$ for any policy $\pi$. Now, suppose the statement
holds for $h-1$ for any policy $\pi$. Then, we have

$\displaystyle\sum_{(s,a)\in{\mathcal{S}}\times{\mathcal{A}}}$
$\displaystyle\left|\mathbb{P}^{\pi}_{h}(s,a)-\mathbb{P}^{\pi,\dagger}_{h}(s,a)\right|$
$\displaystyle=$
$\displaystyle\sum_{(s,a)\in{\mathcal{S}}\times{\mathcal{A}}}\Big{|}\pi(a_{h}=a\mid
s_{h}=s)\sum_{s^{\prime},a^{\prime}}\left(P_{h}(s\mid
s^{\prime},a^{\prime})\mathbb{P}^{\pi}_{h-1}(s^{\prime},a^{\prime})\right)$
$\displaystyle-\pi(a_{h}=a\mid
s_{h}=s)\sum_{s^{\prime},a^{\prime}}\left(P^{\dagger}_{h}(s\mid
s^{\prime},a^{\prime})\mathbb{P}^{\pi,\dagger}_{h-1}(s^{\prime},a^{\prime})\right)\Big{|}$
$\displaystyle\leq$
$\displaystyle\sum_{(s,a)\in{\mathcal{S}}\times{\mathcal{A}}}\pi(a_{h}=a\mid
s_{h}=s)\Big{|}\sum_{s^{\prime},a^{\prime}}\left(P_{h}(s\mid
s^{\prime},a^{\prime})\mathbb{P}^{\pi}_{h-1}(s^{\prime},a^{\prime})\right)-\sum_{s^{\prime},a^{\prime}}\left(P^{\dagger}_{h}(s\mid
s^{\prime},a^{\prime})\mathbb{P}^{\pi,\dagger}_{h-1}(s^{\prime},a^{\prime})\right)\Big{|}$
$\displaystyle=$
$\displaystyle\sum_{s\in{\mathcal{S}}}\Big{|}\sum_{s^{\prime},a^{\prime}}\left(P_{h}(s\mid
s^{\prime},a^{\prime})\mathbb{P}^{\pi}_{h-1}(s^{\prime},a^{\prime})\right)-\sum_{s^{\prime},a^{\prime}}\left(P^{\dagger}_{h}(s\mid
s^{\prime},a^{\prime})\mathbb{P}^{\pi,\dagger}_{h-1}(s^{\prime},a^{\prime})\right)\Big{|}$
$\displaystyle=$
$\displaystyle\sum_{s\in{\mathcal{S}}}\Big{|}\sum_{s^{\prime},a^{\prime}}\left(P_{h}-P_{h}^{\dagger}\right)\mathbb{P}_{h-1}^{\pi}(s^{\prime},a^{\prime})+\sum_{s^{\prime},a^{\prime}}P_{h}^{\dagger}(s\mid
s^{\prime},a^{\prime})\left(\mathbb{P}_{h-1}^{\pi}(s^{\prime},a^{\prime})-\mathbb{P}^{\pi,\dagger}_{h-1}(s^{\prime},a^{\prime})\right)\Big{|}$
$\displaystyle\leq$
$\displaystyle\sum_{s^{\prime},a^{\prime}}\Big{|}\sum_{s\in{\mathcal{S}}}\left(P_{h}-P_{h}^{\dagger}\right)\mathbb{P}_{h-1}^{\pi}(s^{\prime},a^{\prime})\big{|}+\sum_{s^{\prime},a^{\prime}}\big{|}\sum_{s\in{\mathcal{S}}}P_{h}^{\dagger}(s\mid
s^{\prime},a^{\prime})\left(\mathbb{P}_{h-1}^{\pi}(s^{\prime},a^{\prime})-\mathbb{P}^{\pi,\dagger}_{h-1}(s^{\prime},a^{\prime})\right)\Big{|}$
$\displaystyle\leq$
$\displaystyle\epsilon_{p}\sum_{s^{\prime},a^{\prime}}\mathbb{P}_{h-1}^{\pi}(s^{\prime},a^{\prime})+1\cdot(h-1)\epsilon_{p}$
$\displaystyle=$ $\displaystyle\epsilon_{p}\cdot 1+(h-1)\epsilon_{p}$
$\displaystyle\leq$ $\displaystyle h\epsilon_{p}$

The all of above inequalities holds for all $\pi$. Therefore, the statement
holds for all $h\in\mathbb{N}$. ∎

Now, we state the proof of Lemma 1.

###### Proof of Lemma 1.

Lemma 1 is almost a corollary that stems from Lemma 3. By the definition of
visitation probability, we have

$\displaystyle\sum_{(s,a)\in{\mathcal{S}}\times{\mathcal{A}}}\left|P^{\pi}(s,a)-P^{\pi,\dagger}(s,a)\right|$
$\displaystyle=\sum_{(s,a)\in{\mathcal{S}}\times{\mathcal{A}}}\left|\sum_{h=0}^{\infty}\gamma^{h}\left(\mathbb{P}^{\pi}_{h}(s,a)-\mathbb{P}^{\dagger,\pi}_{h}(s,a)\right)\right|$
$\displaystyle\leq\sum_{(s,a)\in{\mathcal{S}}\times{\mathcal{A}}}\sum_{h=0}^{\infty}\gamma^{h}\left|\left(\mathbb{P}^{\pi}_{h}(s,a)-\mathbb{P}^{\dagger,\pi}_{h}(s,a)\right)\right|$
$\displaystyle=\sum_{h=0}^{\infty}\gamma^{h}\sum_{(s,a)\in{\mathcal{S}}\times{\mathcal{A}}}\left|\left(\mathbb{P}^{\pi}_{h}(s,a)-\mathbb{P}^{\dagger,\pi}_{h}(s,a)\right)\right|$
$\displaystyle\leq\sum_{h=0}^{\infty}\gamma^{h}h\frac{(1-\gamma)^{2}}{\gamma}\epsilon_{p}$

Let $S=\sum_{h=0}^{\infty}\gamma^{h}h$, then $\gamma
S=\sum_{h=0}^{\infty}\gamma^{h+1}h=\sum_{h=1}^{\infty}\gamma^{h}(h-1)$. Then
by subtracing those two equations, we have
$(1-\gamma)S=\sum_{h=1}^{\infty}\gamma^{h}=\frac{\gamma}{1-\gamma}$. Therefore
we have $S=\frac{\gamma}{(1-\gamma)^{2}}$. Finally, we have the following
inequality

$\displaystyle\sum_{(s,a)\in{\mathcal{S}}\times{\mathcal{A}}}\left|P^{\pi}(s,a)-P^{\pi,\dagger}(s,a)\right|\leq\frac{\gamma}{(1-\gamma)^{2}}\cdot\frac{(1-\gamma)^{2}}{\gamma}\epsilon_{p}=\epsilon_{p}$

∎

###### Proof of Lemma 2.

First, note that we have assumed the image of the function $R$ is closed and
dense as $[-R_{\max},R_{\max}]$. Then, in the progress of projecting all
$(s,a)$ into the reward, we define the probability of reward as
$\mathbb{P}({\mathcal{R}}=r)=\sum_{\forall(s,a)\in{\mathcal{S}}\times{\mathcal{A}}}d^{\pi}(s,a)\boldsymbol{1}[R(s,a)=r]$.
we use short notation for $\mathbb{P}({\mathcal{R}}=r)$ as
$\mathbb{P}_{\mathcal{R}}$. Now, since $d^{\pi}(s,a)$ is the visitation
probability of visiting $(s,a)$, then this could be converted to
$\mathbb{P}({\mathcal{R}}=r)$ by $d^{\pi}({\mathcal{R}}=R^{-1}(s,a))$ where
$R^{-1}$ is many to one function.

Now, since $\mathbb{R}$ is the many to one function, we can define independent
block the ${\mathcal{S}},{\mathcal{A}}$ as the set
$Z(r):=\\{(s,a)\in{\mathcal{S}}\times{\mathcal{A}}|R(s,a)=r\\}$. Note that if
$r_{1}\neq r_{2}$, then $Z(r_{1})\cap Z(r_{2})=0$. Then, if $g$ satisfies the
set $Z$ to in be permutation- invariant. Namely, if $R(s_{1},a)=R(s_{2},a)$,
then $R(g(s_{1}))=R(g(s_{2}),a)$ holds then there exists a one-to-one mapping
function $h:[-R_{\max},R_{\max}]\to[-R_{\max},R_{\max}]$ such that

$R(s,a)=h(R(g(s),a))$

holds. The proof can be divided into two folds. The existence of such function
and its a one-to-one mapping function is exists. We first prove the existence
of such function $h$. This is because for any state and action $s,a$, suppose
its reward value as $r$. Then suppose $g(s)=s^{\prime}$. Then since image of
function $R$ is closed and dense, there exists
$r^{\prime}\in[-R_{\max},R_{\max}]$ such that $R(s^{\prime},a)=r^{\prime}$
holds. Then, one can say the function $r=h(r^{\prime})$ exists. Now, we prove
the one-to-one mapping property. suppose for two state and action pair
$(s_{1},a_{1})$ and $(s_{2},a_{2})$ and let $s^{\prime}_{1}=g(s^{\prime}_{1})$
and $s^{\prime}_{2}=g(s^{\prime}_{2})$. Now, suppose $R(s_{1}^{\prime},a)\neq
R(s_{2}^{\prime},a)$ holds. Then, due to the property of $g$, then it should
also satisfies $R(s_{1},a)\neq R(s_{2},a)$. Therefore, this concludes that $h$
is the one-to-one mapping, and the following holds

$\displaystyle d^{\pi}(R(g(s),a)=r)$
$\displaystyle=d^{\pi}(h(R(g(s),a))=h(r))$
$\displaystyle=d^{\pi}(R(s,a)=h(r))$
$\displaystyle=\mathbb{P}\left({\mathcal{R}}=h(r)\right)$

holds. we denote $\mathbb{P}\left({\mathcal{R}}=h(r)\right)$ as
$\mathbb{P}_{h({\mathcal{R}})}$. Then, let’s define two different functions
$h^{+}$ and $h^{-}$ such that we want to claim that

$w^{-}\left(\int_{-R_{max}}^{r}d\mathbb{P}_{\mathcal{R}}\right)=\int_{-R_{max}}^{r}d\mathbb{P}_{h^{-}({\mathcal{R}})},\quad\text{and}\quad
w^{+}\left(\int_{-R_{max}}^{r}d\mathbb{P}_{\mathcal{R}}\right)=\int_{-R_{max}}^{r}d\mathbb{P}_{h^{+}({\mathcal{R}})}$
(17)

holds for any $w^{-},w^{+}$. Since the proof for either is similar, we prove
the case for the existence of $h^{-}$ under $w^{-}$ distortion.

Now, recall that for $0<x<b$, $w^{-}(x)<x$ holds and for $b<x<1$, $w^{-}(x)>x$
holds and $w^{-}(x)$ is monotically increasing function. Define
$r_{b}\in[-R_{\max},0]$ such that
$b:=\int_{-R_{\max}}^{r_{b}}d\mathbb{P}_{\mathcal{R}}$ holds, and for notation
simplicity we deonte
$F^{-}(r)=\int_{-R_{\max}}^{r_{b}}d\mathbb{P}_{\mathcal{R}}$. Then, one can
say $-R_{\max}<r<r_{b}$, $w(F(r))<F(r)$ holds and. Then we can always find a
unique ratio $0<\gamma(r)<1$ that depends on $r$ such that
$w^{-}(F(r))=\int_{-R_{\max}}^{\gamma(r)r}d\mathbb{P}_{r}$ holds where

$\gamma(r)=\frac{w^{-}(F(r))}{r}.$

This leads to set $h(r)=\gamma(r)r=w^{-}(F(r))$ that satisfies (17) and also
one-to-one mapping. In the same manner, we can also identify
$h(r)=\gamma(r)r=w^{-}(F(r))$ where $r_{b}<r<0$ holds for $\gamma(r)>1$. Then,
this completes that the function $h:r\to w^{-}(F(r))$ satisfies a one-to-one
function and Equation (17). This completes the proof. ∎∎

###### Proof of Theorem 1.

By the definition of optimal policy and the value function definition at the
time $T=1$, we have the optimal policy at time $0$ as follows.

$\displaystyle\pi^{\star}$
$\displaystyle=\operatorname*{arg\,max}_{\pi}V_{0}(s)$
$\displaystyle=\operatorname*{arg\,max}_{a\in\mathcal{A}}Q_{0}(s,a)$
$\displaystyle=\operatorname*{arg\,max}_{a\in{\mathcal{A}}}R(s,a)$
$\displaystyle\pi^{\star,\dagger}$
$\displaystyle=\operatorname*{arg\,max}_{a\in\mathcal{A}}V_{0}^{\dagger}(s)$
$\displaystyle=\operatorname*{arg\,max}_{a\in{\mathcal{A}}}Q^{\dagger}_{0}(s,a)$
$\displaystyle=\operatorname*{arg\,max}_{a\in{\mathcal{A}}}u(R(s,a))$

for any fixed $s\in{\mathcal{S}}$, let’s assume $a^{*}$ is the argument that
maximizes the $R(s,a)$. Since $u$ is the non-decreasing convex function,
$a^{\star}$ is still the same argument that maximizes the $u(R(s,a))$.
Therefore, $\pi^{\star}=\pi^{\star,\dagger}$ holds. ∎∎

###### Proof of Theorem 2.

We prove by backward induction. First by theorem 1,
$\pi^{\star}_{T}=\pi^{\star,\dagger}_{T}$ holds. Now suppose that
$\pi^{\star}_{t^{\prime}+1}=\pi^{\star,\dagger}_{t^{\prime}+1}$ holds for all
$t^{\prime}=t+1,\cdots,T$. Now, we prove the statement holds for $t$. To prove
$\pi^{\star}_{t}=\pi^{\star,\dagger}_{t}$, it is sufficient to show if
$Q^{\pi^{\star}}_{t}(s,a)\geq Q^{\pi}_{t}(s,a^{\prime})$, then
$Q^{\dagger,\pi^{\star}}_{t}(s,a)\geq
Q^{\dagger,\pi^{\star}}_{t}(s,a^{\prime})$ also holds for any actions
$a,a^{\prime}\in{\mathcal{A}}$. First, the gap
$Q^{\pi^{\star}}_{t}(s,a)-Q^{\pi^{\star}}_{t}(s,a)$ could be expressed as

$\displaystyle Q^{\pi}_{t}(s,a)-Q^{\pi}_{t}(s,a)$
$\displaystyle=R_{t}(s,a)-R_{t}(s,a^{\prime})+\left\\{\left(P(s_{1}|s,a)-P(s_{2}|s,a^{\prime})\right)\left(V_{t+1}^{\pi^{\star}}(s_{1})-V_{t+1}^{\pi^{\star}}(s_{2})\right)\right\\}$
$\displaystyle=\left(P(s_{1}|s,a)-P(s_{2}|s,a^{\prime})\right)\left(V_{t+1}^{\pi^{\star}}(s_{1})-V_{t+1}^{\pi^{\star}}(s_{2})\right)$

and $Q^{\dagger,\pi^{\star}}_{t}(s,a)-Q^{\dagger,\pi^{\star}}_{t}(s,a)$ as

$\displaystyle
Q^{\dagger,\pi^{\star}}_{t}(s,a)-Q^{\dagger,\pi^{\star}}_{t}(s,a)$
$\displaystyle=R^{\dagger}_{t}(s,a)-R^{\dagger}_{t}(s,a^{\prime})+\left\\{\left(P^{\dagger}(s_{1}|s,a)-P^{\dagger}(s_{2}|s,a^{\prime})\right)\left(V_{t+1}^{\pi^{\star}}(s_{1})-V_{t+1}^{\pi^{\star}}(s_{2})\right)\right\\}$
$\displaystyle=\left(P^{\dagger}(s_{1}|s,a)-P^{\dagger}(s_{2}|s,a^{\prime})\right)\left(V_{t+1}^{\dagger,\pi^{\star}}(s_{1})-V_{t+1}^{\dagger,\pi^{\star}}(s_{2})\right)$
$\displaystyle=\left(w(P^{\dagger}(s_{1}|s,a))-w(P^{\dagger}(s_{2}|s,a^{\prime}))\right)\left(V_{t+1}^{\dagger,\pi^{\star}}(s_{1})-V_{t+1}^{\dagger,\pi^{\star}}(s_{2})\right)$

the reward during $t\in[1,T-1]$ is zero by our problem formulation assumption
in section 6.1. Now, without loss of generality, we assume
$V_{t+1}^{\pi^{\star}}(s_{1})>V_{t+1}^{\pi^{\star}}(s_{2})$. Then, due to our
assumption that $\pi^{\star}_{t^{\prime}}=\pi^{\star,\dagger}_{t\prime}$ holds
for $t^{\prime}=t+1,\cdots,T$, we also have
$V_{t+1}^{\dagger,\pi^{\star}}(s_{1})>V_{t+1}^{\dagger,\pi^{\star}}(s_{2})$.
Also, noticing that weight function $w$ is also increasing function, then
$P(s_{1}|s,a)>P(s_{2}|s,a)$ also guarantees $w(P(s_{1}|s,a))>w(P(s_{2}|s,a))$
holds. Therefore, we can claim if $Q^{\pi}_{t}(s,a)-Q^{\pi}_{t}(s,a)>0$ holds,
then $Q^{\dagger,\pi^{\star}}_{t}(s,a)-Q^{\dagger,\pi^{\star}}_{t}(s,a)>0$
also holds. Then, this leads to claim that
$\operatorname*{arg\,max}Q^{\pi}_{t}(s,a)=\operatorname*{arg\,max}Q^{\dagger\pi}_{t}(s,a)$,
which implies $\pi^{\star}_{t}=\pi^{\star,\dagger}_{t}$. This completes the
proof. ∎∎

###### Proof of Theorem 3.

Assume that Theorem 3 does not hold. Given $T=2$, we have
$V_{2}^{\pi}(s)=\max_{a\in\mathcal{A}}R_{2}(s,a)=$ $R_{2}(s)$ for each state
$s$. At time $t=1$, assume $R_{2}\left(s_{1}\right)\leq
R_{2}\left(s_{2}\right)\leq R_{2}\left(s_{3}\right)$. The condition
$Q_{1}^{\dagger,\pi}\left(s,a_{1}\right)\geq$
$Q_{1}^{\dagger,\pi}\left(s,a_{2}\right)$ is then expressed as:

$\displaystyle w\left(P\left(s_{1}\mid
s,a_{1}\right)\right)r_{2}\left(s_{1}\right)+\left(w\left(P\left(s_{2}\mid
s,a_{1}\right)+P\left(s_{1}\mid s,a_{1}\right)\right)-w\left(P\left(s_{1}\mid
s,a_{1}\right)\right)\right)R_{2}\left(s_{2}\right)$
$\displaystyle\quad+\left(1-w\left(P\left(s_{2}\mid
s,a_{1}\right)+P\left(s_{1}\mid
s,a_{1}\right)\right)\right)R_{3}\left(s_{3}\right)$ $\displaystyle\geq
w\left(P\left(s_{1}\mid
s,a_{2}\right)\right)R_{2}\left(s_{1}\right)+\left(w\left(P\left(s_{2}\mid
s,a_{2}\right)+P\left(s_{1}\mid s,a_{2}\right)\right)-w\left(P\left(s_{1}\mid
s,a_{2}\right)\right)\right)R_{2}\left(s_{2}\right)$
$\displaystyle\quad+\left(1-w\left(P\left(s_{2}\mid
s,a_{2}\right)+P\left(s_{1}\mid
s,a_{2}\right)\right)\right)R_{3}\left(s_{3}\right)$

which simplifies to:

$\displaystyle\left(w\left(P\left(s_{1}\mid
s,a_{1}\right)\right)-w\left(P\left(s_{1}\mid
s,a_{2}\right)\right)\right)\left(R_{2}\left(s_{1}\right)-R_{3}\left(s_{3}\right)\right)$
$\displaystyle\quad+\left(\left(w\left(P\left(s_{2}\mid
s,a_{1}\right)+P\left(s_{1}\mid s,a_{1}\right)\right)-w\left(P\left(s_{1}\mid
s,a_{1}\right)\right)\right)\right.$
$\displaystyle\left.\quad-\left(w\left(P\left(s_{2}\mid
s,a_{2}\right)+P\left(s_{1}\mid s,a_{2}\right)\right)-w\left(P\left(s_{1}\mid
s,a_{2}\right)\right)\right)\right)\left(R_{2}\left(s_{2}\right)-R_{3}\left(s_{3}\right)\right)\geq
0$

For the non-distorted case, the analogous expression is:

$\displaystyle\left(P\left(s_{1}\mid s,a_{1}\right)-P\left(s_{1}\mid
s,a_{2}\right)\right)\left(R_{2}\left(s_{1}\right)-R_{3}\left(s_{3}\right)\right)$
$\displaystyle\quad+\left(P\left(s_{2}\mid s,a_{1}\right)-P\left(s_{2}\mid
s,a_{2}\right)\right)\left(R_{2}\left(s_{2}\right)-R_{3}\left(s_{3}\right)\right)\geq
0$

For arbitrary reward functions, $R_{2}$, the equality of the two cases under
any weighting function $w$ leads to:

$\displaystyle\frac{w\left(P\left(s_{1}\mid
s,a_{1}\right)\right)-w\left(P\left(s_{1}\mid
s,a_{2}\right)\right)}{w\left(P\left(s_{2}\mid s,a_{1}\right)+P\left(s_{1}\mid
s,a_{1}\right)\right)-w\left(P\left(s_{1}\mid
s,a_{1}\right)\right)-\left(w\left(P\left(s_{2}\mid
s,a_{2}\right)+P\left(s_{1}\mid s,a_{2}\right)\right)-w\left(P\left(s_{1}\mid
s,a_{2}\right)\right)\right.}$ $\displaystyle=$
$\displaystyle\frac{P\left(s_{1}\mid s,a_{1}\right)-P\left(s_{1}\mid
s,a_{2}\right)}{P\left(s_{2}\mid s,a_{1}\right)-P\left(s_{2}\mid
s,a_{2}\right)}$

where $w(p)=p$ is the only solution, contradicting the distortion required by
Assumption 3. ∎

###### Proof of Theorem 4.

The proof of Theorem 4 is divided into three-fold.

1\. Proof of asymptotic convergence

We first prove asymptotic convergence (Equation (14)), then we prove Equation
(15) in part 3 of this proof. Note that the empirical distribution function
$\widehat{F}_{n}(r)$ generate Stielgies measure which takes mass $\frac{1}{t}$
each of the sample points on $U^{+}(R_{i})$.

or equivalently, show that

$\displaystyle\lim_{n\rightarrow+\infty}\sum_{i=1}^{n-1}u^{+}(R_{[i]})(w^{+}(\frac{n-i+1}{n})-w^{+}(\frac{n-i}{n}))$
$\displaystyle\xrightarrow{n\rightarrow\infty}\int_{0}^{+\infty}w^{+}(P(U>t))dt,\text{w.p.
}1$ (18)

where $n$ denotes the number of positive reward among
$|{\mathcal{S}}||{\mathcal{A}}|$. Let $\xi^{+}_{\frac{i}{n}}$ and
$\xi^{-}_{\frac{i}{n}}$ denote the $\frac{i}{n}$th quantile of $u^{+}(X)$ and
$u^{-}(X)$, respectively.

For the convergence proof, we first concentrate on finding the following
probability,

$\displaystyle
P(\left|\sum_{i=1}^{n-1}u^{+}(R_{[i]})\cdot(w^{+}(\frac{n-i}{n})-w^{+}(\frac{n-i-1}{n}))-\sum_{i=1}^{n-1}\xi^{+}_{\frac{i}{n}}\cdot(w^{+}(\frac{n-i}{n})-w^{+}(\frac{n-i-1}{n}))\right|>\epsilon),$
(19)

for any given $\epsilon>0$. It is easy to check that

$\displaystyle
P(\left|\sum_{i=1}^{n-1}u^{+}(R_{[i]})\cdot(w^{+}(\frac{n-i}{n})-w^{+}(\frac{n-i-1}{n}))-\sum_{i=1}^{n-1}\xi^{+}_{\frac{i}{n}}\cdot(w^{+}(\frac{n-i}{n})-w^{+}(\frac{n-i-1}{n}))\right|>\epsilon)$
$\displaystyle\leq
P(\bigcup_{i=1}^{n-1}\left\\{\left|u^{+}(R_{[i]})\cdot(w^{+}_{(}\frac{n-i}{n})-w^{+}{(\frac{n-i-1}{n})})-\xi^{+}_{\frac{i}{n}}\cdot(w^{+}(\frac{n-i}{n})-w^{+}(\frac{n-i-1}{n}))\right|>\frac{\epsilon}{n}\right\\})$
$\displaystyle\leq\sum_{i=1}^{n-1}P(\left|u^{+}(R_{[i]})\cdot(w^{+}(\frac{n-i}{n})-w^{+}(\frac{n-i-1}{n}))-\xi^{+}_{\frac{i}{n}}\cdot(w^{+}_{(}\frac{n-i}{n})-w^{+}_{(}\frac{n-i-1}{n}))\right|>\frac{\epsilon}{n})$
(20)
$\displaystyle=\sum_{i=1}^{n-1}P(\left|(u^{+}(R_{[i]})-\xi^{+}_{\frac{i}{n}})\cdot(w^{+}(\frac{n-i}{n})-w^{+}(\frac{n-i-1}{n}))\right|>\frac{\epsilon}{n})$
$\displaystyle\leq\sum_{i=1}^{n-1}P(\left|(u^{+}(R_{[i]})-\xi^{+}_{\frac{i}{n}})\cdot\cdot(\frac{1}{n})^{\alpha}\right|>\frac{\epsilon}{n})$
$\displaystyle=\sum_{i=1}^{n-1}P(\left|(u^{+}(R_{[i]})-\xi^{+}_{\frac{i}{n}})\right|>\frac{\epsilon}{\cdot
n^{1-\alpha}}).$ (21)

The Right hand side of Inequality (21) could be expressed as follows.

$\displaystyle
P(\left|u^{+}(R_{[i]})-\xi^{+}_{\frac{i}{n}}\right|>\frac{\epsilon}{n^{(1-\alpha)}})$
$\displaystyle=P(u^{+}(R_{[i]})-\xi^{+}_{\frac{i}{n}}>\frac{\epsilon}{n^{(1-\alpha)}})+P(u^{+}(R_{[i]})-\xi^{+}_{\frac{i}{n}}<-\frac{\epsilon}{n^{(1-\alpha)}}).$

We focus on the term
$P(u^{+}(R_{[i]})-\xi^{+}_{\frac{i}{n}}>\frac{\epsilon}{n^{1-\alpha}})$. Now,
let us define an event
$A_{t}=I_{(u^{+}(X_{t})>\xi^{+}_{\frac{i}{n}}+\frac{\epsilon}{n^{(1-\alpha)}})}$
where $t=1,\ldots,n$. Since the Cumulative distribution is non-decrasing
function, we have the following,

$\displaystyle
P(u^{+}(R_{[i]})-\xi^{+}_{\frac{i}{n}}>\frac{\epsilon}{1-\alpha})$
$\displaystyle=P(\sum_{t=1}^{n}A_{t}>n\cdot(1-\frac{i}{n^{(1-\alpha)}}))$
$\displaystyle=P(\sum_{t=1}^{n}A_{t}-n\cdot[1-F^{+}(\xi^{+}_{\frac{i}{n}}+\frac{\epsilon}{n^{(1-\alpha)}})]>n\cdot[F^{+}(\xi^{+}_{\frac{i}{n}}+\frac{\epsilon}{n^{(1-\alpha)}})-\frac{i}{n}]).$

Using the fact that
$\mathbb{E}A_{t}=1-F^{+}(\xi^{+}_{\frac{i}{n}}+\frac{\epsilon}{n^{(1-\alpha)}})$
in conjunction with Hoeffding’s inequality, we obtain

$\displaystyle
P(\sum_{i=1}^{n}A_{t}-n\cdot[1-F^{+}(\xi^{+}_{\frac{i}{n}}+\frac{\epsilon}{n^{(1-\alpha)}})]>n\cdot[F^{+}(\xi^{+}_{\frac{i}{n}}+\frac{\epsilon}{n^{(1-\alpha)}})-\frac{i}{n}])<e^{-2n\cdot\delta^{{}^{\prime}}_{t}},$
(22)

where
$\delta^{{}^{\prime}}_{i}=F^{+}(\xi^{+}_{\frac{i}{n}}+\frac{\epsilon}{n^{(1-\alpha)}})-\frac{i}{n}$.
Since $F^{+}(x)$ is Lipschitz, we have that $\delta^{{}^{\prime}}_{i}\leq
L_{F^{+}}\cdot(\frac{\epsilon}{1-\alpha})$. Hence, we obtain

$\displaystyle
P(u^{+}(R_{[i]})-\xi^{+}_{\frac{i}{n}}>\frac{\epsilon}{1-\alpha})<e^{-2n\cdot
L_{F^{+}}\frac{\epsilon}{1-\alpha}}=e^{-2n^{\alpha}\cdot L^{+}\epsilon}$ (23)

In a similar fashion, one can show that

$\displaystyle
P(u^{+}(R_{[i]})-\xi^{+}_{\frac{i}{n}}<-\frac{\epsilon}{1-\alpha})\leq
e^{-2n^{\alpha}\cdot L_{F^{+}}\epsilon}$ (24)

Combining (23) and (24), we obtain

$\displaystyle
P(\left|u^{+}(R_{[i]})-\xi^{+}_{\frac{i}{n}}\right|>\frac{\epsilon}{1-\alpha})\leq
2\cdot e^{-2n^{\alpha}\cdot L_{F^{+}}\epsilon},\text{ }\forall
i\in\mathbb{N}\cap(0,1)$

Plugging the above in (21), we obtain

$\displaystyle
P(\left|\sum_{i=1}^{n-1}u^{+}(R_{[i]})\cdot(w^{+}(\frac{n-i}{n})-w^{+}(\frac{n-i-1}{n}))-\sum_{i=1}^{n-1}\xi^{+}_{\frac{i}{n}}\cdot(w^{+}(\frac{n-i}{n})-w^{+}(\frac{n-i-1}{n}))\right|>\epsilon)$
$\displaystyle\leq 2n\cdot e^{-2n^{\alpha}\cdot L_{F^{+}}}.$ (25)

Notice that $\sum_{n=1}^{+\infty}2n\cdot e^{-2n^{\alpha}\cdot
L_{F^{+}}\epsilon}<\infty$ since the sequence $2n\cdot e^{-2n^{\alpha}\cdot
L_{F^{+}}}$ will decrease more rapidly than the sequence $\frac{1}{n^{k}}$,
$\forall k>1$.

By applying the Borel Cantelli lemma, we have that $\forall\epsilon>0$

$P(\left|\sum_{i=1}^{n-1}u^{+}(R_{[i]})\cdot(w^{+}(\frac{n-i}{n})-w^{+}(\frac{n-i-1}{n}))-\sum_{i=1}^{n-1}\xi^{+}_{\frac{i}{n}}\cdot(w^{+}(\frac{n-i}{n})-w^{+}(\frac{n-i-1}{n}))\right|>\epsilon)=0,$

which implies

$\sum_{i=1}^{n-1}u^{+}(R_{[i]})\cdot(w^{+}(\frac{n-i}{n})-w^{+}(\frac{n-i-1}{n}))-\sum_{i=1}^{n-1}\xi^{+}_{\frac{i}{n}}\cdot(w^{+}(\frac{n-i}{n})-w^{+}(\frac{n-i-1}{n}))\xrightarrow{n\rightarrow+\infty}0\text{
w.p }1,$

which proves (18).

Also, the remaining part, conducting the proof of convergence of $w^{-}$ and
$u^{-}$,i.e.

$\displaystyle\lim_{n\rightarrow+\infty}\sum_{i=1}^{n-1}u^{-}(R_{[i]})(w^{-}(\frac{n-i+1}{n})-w^{-}(\frac{n-i}{n}))$
$\displaystyle\xrightarrow{n\rightarrow\infty}\int_{0}^{+\infty}w^{-}(P(U>t))dt,\text{w.p.
}1$ (26)

also follows simliar manner. we omit the proof for this.

2\. Proof of value function lower bound

By the definition, we have the following

$\displaystyle|V(s_{0})-V^{\dagger}(s_{0})|$
$\displaystyle=\left|\int_{-\infty}^{0}w^{-}_{\star}(\mathbb{P}_{r}(u^{-}_{\star}({\mathcal{R}}>r)))dr-\int_{\infty}^{0}w^{-}(\mathbb{P}_{r}(u^{-}({\mathcal{R}}>r)))dr\right|$
$\displaystyle=\bigg{|}\int_{-\infty}^{0}w^{-}_{\star}(\mathbb{P}_{r}(u^{-}_{\star}({\mathcal{R}}>r)))dr-\int_{\infty}^{0}w^{-}_{\star}(\mathbb{P}_{r}(u^{-}({\mathcal{R}}>r)))dr$
$\displaystyle\quad-\left(\int_{\infty}^{0}w^{-}(\mathbb{P}_{r}(u^{-}({\mathcal{R}}>r)))dr-\int_{\infty}^{0}w^{-}_{\star}(\mathbb{P}_{r}(u^{-}({\mathcal{R}}>r)))dr\right)\bigg{|}$
$\displaystyle\geq\underbrace{\bigg{|}\int_{-\infty}^{0}w^{-}_{\star}(\mathbb{P}_{r}(u^{-}_{\star}({\mathcal{R}}>r)))dr-\int_{-\infty}^{0}w^{-}_{\star}(\mathbb{P}_{r}(u^{-}({\mathcal{R}}>r)))dr\bigg{|}}_{\text{Term
(I)}}$
$\displaystyle\quad-\underbrace{\bigg{|}\int_{-\infty}^{0}w^{-}_{\star}(\mathbb{P}_{r}(u^{-}_{\star}({\mathcal{R}}>r)))dr-\int_{\infty}^{0}w^{-}_{\star}(\mathbb{P}_{r}(u^{-}({\mathcal{R}}>r)))dr\bigg{|}}_{\text{Term
(II)}}$

We first under bound the term (I). For notation simplicity, we let
$g(r)=\mathbb{P}_{r}(u^{-}({\mathcal{R}}>r)))$ and
$g_{\star}(r)=\mathbb{P}_{r}(u^{-}_{\star}({\mathcal{R}}>r)))$. Then we have
the following

Term (I)
$\displaystyle=\left|\int_{-R_{\max}}^{0}w^{-}_{\star}(g_{\star}(r))-w^{-}_{\star}(g(r))\right|$

Now, since $w^{-}_{\star}(x)$ is monotonically increasing in $x\in[0,a]$ and
monotonically decreasing in $x\in[a,1]$, we could say for any
$x,y\in[0,1],x\neq y$ that

$\frac{w^{-}_{*}(x)-w^{-}_{*}(y)}{x-y}=(w^{-}_{\star})^{\prime}(z)\geq\min_{z\in[0,1]}(w^{-}_{\star})^{\prime}(z)=\min\left\\{(w^{-}_{\star})^{\prime}(0),(w^{-}_{\star})^{\prime}(1)\right\\},$

where $z\in(x,y)$. The first equality holds due to the mean value theorem.
Therfore it holds that

Term (I)
$\displaystyle=\left|\int_{-R_{\max}}^{0}w^{-}_{\star}(g_{\star}(r))-w^{-}_{\star}(g(r))\right|$
$\displaystyle\geq\left|\int_{-R_{\max}}^{0}\min\left\\{(w^{-}_{\star})^{\prime}(0),(w^{-}_{\star})^{\prime}(1)\right\\}\left(g_{\star}(r)-g(r)\right)\right|$
$\displaystyle=\min\left\\{(w^{-}_{\star})^{\prime}(0),(w^{-}_{\star})^{\prime}(1)\right\\}\left|\int_{-R_{\max}}^{0}\left(g_{\star}(r)-g(r)\right)\right|$

Now, recall the definition of $g_{\star}(r)$ and $g(r)$, then we have the
following

$\left|\int_{-R_{\max}}^{0}\left(g_{\star}(r)-g(r)\right)dr\right|=\left|\mathbb{E}_{{\mathcal{R}}\sim\mathbb{P}_{\pi}}\left[u^{-}_{\star}({\mathcal{R}})-u^{-}({\mathcal{R}})\right]\right|$

Now, let us denote the intersection of $u^{-}(R)$ and $y=R+C_{bs}$ as
$R=-R_{bs}$. We can say if the blackswan happens, then its reward is bounded
between $[-R_{\max},-R_{bs}]$. Then we have the following,

$\displaystyle\left|\int_{-R_{\max}}^{0}\left(g_{\star}(r)-g(r)\right)\right|$
$\displaystyle=\left|\mathbb{E}_{{\mathcal{R}}\sim\mathbb{P}_{\pi}}\left[u^{-}_{\star}({\mathcal{R}})-u^{-}({\mathcal{R}})\right]\right|$
$\displaystyle=\bigg{|}\mathbb{E}_{{\mathcal{R}}\sim\mathbb{P}_{\pi}}\left[\boldsymbol{1}\left[{\mathcal{R}}<-R_{bs}\right]\left(u^{-}_{\star}({\mathcal{R}})-u^{-}({\mathcal{R}})\right)\right]$
$\displaystyle\quad-\mathbb{E}_{{\mathcal{R}}\sim\mathbb{P}_{\pi}}\left[\boldsymbol{1}\left[{\mathcal{R}}\geq-
R_{bs}\right]\left(-u^{-}_{\star}({\mathcal{R}})+u^{-}({\mathcal{R}})\right)\right]\bigg{|}$
$\displaystyle\geq\underbrace{\bigg{|}\mathbb{E}_{{\mathcal{R}}\sim\mathbb{P}_{\pi}}\left[\boldsymbol{1}\left[{\mathcal{R}}<-R_{bs}\right]\left(u^{-}_{\star}({\mathcal{R}})-u^{-}({\mathcal{R}})\right)\right]\bigg{|}}_{\text{Term
I-1}}$
$\displaystyle\quad-\underbrace{\bigg{|}\mathbb{E}_{{\mathcal{R}}\sim\mathbb{P}_{\pi}}\left[\boldsymbol{1}\left[{\mathcal{R}}\geq-
R_{bs}\right]\left(-u^{-}_{\star}({\mathcal{R}})+u^{-}({\mathcal{R}})\right)\right]\bigg{|}}_{\text{Term
I-2}}$
$\displaystyle\geq\bigg{|}\mathbb{E}_{{\mathcal{R}}\sim\mathbb{P}_{\pi}}\left[\boldsymbol{1}\left[{\mathcal{R}}<-R_{bs}\right]\left(u^{-}_{\star}({\mathcal{R}})-u^{-}({\mathcal{R}})\right)\right]\bigg{|}$

To lower bound the Term I-1, let’s denote the minimum reachability of
blackswan events as $\epsilon^{\min}_{bs}\neq 0$. Then we have

Term I-1
$\displaystyle\geq\frac{R_{\max}-R_{bs}}{R_{\max}}\epsilon^{\min}_{bs}\min_{R\in[-R_{\max},-R_{bs}]}\left|u^{-}(R)-u^{-}_{\star}(R)\right|$
$\displaystyle\geq\frac{R_{\max}-R_{bs}}{R_{\max}}\epsilon^{\min}_{bs}\left|u^{-}(-R_{bs})-u_{\star}^{-}(-R_{bs})\right|$
(27)

Term I-2
$\displaystyle\leq\frac{R_{bs}}{R_{\max}}\epsilon_{bs}\max_{R\in[-R_{bs},0]}\left|u^{-}(R)-u^{-}_{\star}(R)\right|$
$\displaystyle\leq\frac{R_{bs}}{R_{\max}}\epsilon_{bs}\left|u^{-}(-R_{bs})-u_{\star}^{-}(-R_{bs})\right|$
(28)

Therefore, we have the following equation,

$\text{Term
I}\geq\frac{\left(R_{\max}-R_{bs}\right)\epsilon^{\min}_{bs}-R_{bs}\epsilon_{bs}}{R_{\max}}\left|u^{-}(-R_{bs})-u^{-}_{\star}(-R_{bs})\right|$

Also, since the function $u^{-}_{\star}(r)$ is convex, and
$u^{-}_{\star}(-R_{\max})<-R_{\max}+C_{bs}$ holds. Therefore, we could say
$u^{-}_{\star}(r)<\frac{R_{\max}-C_{bs}}{R_{\max}}r$. This leads us to come up
with $u^{-}_{\star}(-R_{b}s)<\frac{R_{\max}-C_{bs}}{R_{\max}}(-R_{bs})$.
Therefore, we have a gap lowerbound as

$\displaystyle\left|u^{-}(-R_{bs})-u_{\star}^{-}(-R_{bs})\right|$
$\displaystyle\geq(R_{\max}-C_{bs})\frac{R_{bs}}{R_{\max}}-(R_{bs}-C_{bs})$
$\displaystyle=\frac{(R_{\max}-R_{bs})C_{bs}}{R_{\max}}$

The above inequality could be minimized as

Term I
$\displaystyle\geq\frac{\left(R_{\max}-R_{bs}\right)\epsilon^{\min}_{bs}-R_{bs}\epsilon_{bs}}{R_{\max}}\left(\frac{(R_{\max}-R_{bs})C_{bs}}{R_{\max}}\right)$
$\displaystyle=\frac{\left(\left(R_{\max}-R_{bs}\right)\epsilon^{\min}_{bs}-R_{bs}\epsilon_{bs}\right)(R_{\max}-R_{bs})C_{bs}}{R^{2}_{\max}}$

Now, let’s upper bound Term 2. Before, recall that the definition of
$g(r)=\mathbb{P}_{r}(u^{-}({\mathcal{R}})>r))$ and note that by the definition
of black swans, we have $u^{-}({\mathcal{R}})>{\mathcal{R}}+C_{bs}$ holds for
$R\in[-R_{\max},-R_{bs})$. Therefore, we can say for all
$r\in[-R_{\max},-R_{bs}),g(r)=1$ holds. Therefore, for all
$r\in[-R_{\max},-R_{bs}]$, we have
$w^{-}_{\star}(g(r))-w^{-}(g(r))=w^{-}_{\star}(1)-w^{-}(1)=1-1=0$

$\displaystyle\left|\int_{-R_{\max}}^{0}w^{-}_{\star}(g(r))-w^{-}(g(r))dr\right|$
$\displaystyle=\left|\int_{-R_{\max}+C_{bs}}^{0}w^{-}_{\star}(g(r))-w^{-}(g(r))dr\right|$
$\displaystyle=\left|\int_{-R_{\max}+C_{bs}}^{0}w^{-}(g(r))-w^{-}_{\star}(g(r))dr\right|$
$\displaystyle\leq\left|\int_{-R_{\max}+C_{bs}}^{0}L^{-}g(r)-g(r)dr\right|$
$\displaystyle=(L^{-}-1)\left|\int_{-R_{\max}+C_{bs}}^{0}g(r)dr\right|$
$\displaystyle\leq(L^{-}-1)\cdot\frac{R_{\max}-C_{bs}}{2R_{\max}}\epsilon_{bs}$
$\displaystyle=(L^{-}-1)\left|\int_{-R_{\max}+C_{bs}}^{0}1-\mathbb{P}_{r}(U^{-}({\mathcal{R}})<r)dr\right|$
$\displaystyle=(L^{-}-1)\left|\int_{-R_{\max}+C_{bs}}^{0}1-\mathbb{P}_{r}(U^{-}({\mathcal{R}})<r)dr\right|$
$\displaystyle=(L^{-}-1)\left|\left((R_{\max}-C_{bs})-\int_{-R_{\max}+C_{bs}}^{0}\mathbb{P}_{r}(u^{-}({\mathcal{R}})<r)dr\right)\right|$
$\displaystyle=(L^{-}-1)\left|\left((R_{\max}-C_{bs})-\mathbb{E}_{{\mathcal{R}}\sim\mathbb{P}_{r}}\left[u^{-}({\mathcal{R}})\boldsymbol{1}[-R_{\max}+C_{bs}<{\mathcal{R}}<0]\right]\right)\right|$
(29)

Note that if $-R_{\max}+C_{bs}<-R_{bs}$, then

$\boldsymbol{1}[-R_{\max}+C_{bs}<{\mathcal{R}}<0]\cdot\mathbb{E}_{{\mathcal{R}}\sim\mathbb{P}_{r}}\left[u^{-}({\mathcal{R}})\right]\geq\left(\frac{R_{\max}-C_{bs}-R_{bs}}{2R_{\max}}\epsilon^{\min}_{bs}+\frac{R_{bs}}{2R_{\max}}\epsilon_{bs}\right)u^{-}(-R_{\max}+C_{bs})$
(30)

and if $-R_{\max}+C_{bs}<-R_{bs}$, then

$\boldsymbol{1}[-R_{\max}+C_{bs}<{\mathcal{R}}<0]\cdot\mathbb{E}_{{\mathcal{R}}\sim\mathbb{P}_{r}}\left[u^{-}({\mathcal{R}})\right]\geq\left(\frac{R_{\max}-C_{bs}}{2R_{\max}}\epsilon_{bs}\right)u^{-}(-R_{\max}+C_{bs})$
(31)

Therefore, combining the Equations (29), (30), (31), we conclude that

$\text{Term II}\leq
C\cdot\frac{\left(\left(R_{\max}-R_{bs}\right)\epsilon^{\min}_{bs}-R_{bs}\epsilon_{bs}\right)(R_{\max}-R_{bs})C_{bs}}{R^{2}_{\max}}$

where $C\in[0,1]$ is a constant. This completes the proof.

3\. Value function upper bound

For the proof of Equation (15) of Theorem 4, we utilized the following Lemma 4
which provides a concentration inequality on the distance between empirical
distribution and true distribution.

Since $u^{+}({\mathcal{R}})$ is bounded above by $u^{+}(R_{\max})$ and
$w^{+}(p)$ is Lipschitz with constant $L^{+}(=(w^{+})^{\prime}(a))$, we have
the following inequality,

$\displaystyle\left|\int^{\infty}_{0}w^{+}(P(u^{+}(X))>x)dx-\int^{\infty}_{0}w^{+}(1-{\hat{F}_{t}}^{+}(x))dx\right|$
$\displaystyle=$
$\displaystyle\left|\int_{0}^{u^{+}(R_{\max})}w^{+}(P(u^{+}(X))>x)dx-\int_{0}^{u^{+}(R_{\max})}w^{+}(1-{\hat{F}_{t}}^{+}(x))dx\right|$
$\displaystyle\leq$
$\displaystyle\left|\int_{0}^{u^{+}(R_{\max})}L^{+}\cdot|P(u^{+}(X)<x)-{\hat{F}_{t}}^{+}(x)|dx\right|$
$\displaystyle\leq$ $\displaystyle
L^{+}u^{+}(R_{\max})\sup_{x\in\mathbb{R}}\left|P(u^{+}(X)<x)-{\hat{F}_{t}}^{+}(x)\right|.$

Now, plugging in the DKW inequality, we obtain

$\displaystyle
P\left(\left|\int^{\infty}_{0}w^{+}(P(u^{+}(X))>x)dx-\int^{\infty}_{0}w^{+}(1-{\hat{F}_{t}}^{+}(x))dx\right|>\epsilon/2\right)$
$\displaystyle\leq
P\left(L^{+}u^{+}(R_{\max})\sup_{x\in\mathbb{R}}\left|(P(u^{+}(X)<x)-{\hat{F}_{t}}^{+}(x)\right|>\epsilon/2\right)\leq
2e^{-t\frac{\epsilon^{2}}{2(L^{+}u^{+}(R_{\max}))^{2}}}.$ (32)

Along similar manner, we have

$P\left(\left|\int^{\infty}_{0}w^{-}(P(u^{-}(X))>x)dx-\int^{\infty}_{0}w^{-}(1-{\hat{F}_{t}}^{-}(x))dx\right|>\epsilon/2\right)\leq
2e^{-t\frac{\epsilon^{2}}{2(L^{-}u^{-}(-R_{\max}))^{2})}}.$ (33)

Combining (32) and (33), we obtain

$\displaystyle P(|\widehat{V}^{\dagger}_{t}-V^{\dagger}_{t}|>\epsilon)$
$\displaystyle\leq
P\left(\left|\int^{\infty}_{0}w^{+}(P(u^{+}(X))>x)dx-\int^{\infty}_{0}w^{+}(1-{\hat{F}_{t}}^{+}(x))dx\right|>\epsilon/2\right)$
$\displaystyle+P\left(\left|\int^{\infty}_{0}w^{-}(P(u^{-}(X))>x)dx-\int^{\infty}_{0}w^{-}(1-{\hat{F}_{t}}^{-}(x))dx\right|>\epsilon/2\right)$
$\displaystyle\leq 4e^{-t\frac{\epsilon^{2}}{2c^{2}}}.$

where $c=\max\\{|L^{+}u^{+}(R_{\max})|,|L^{-}u^{-}(-R_{\max})|\\}$

∎∎

###### Proof of Theorem 5.

For a given optimal policy $\pi_{\star}$, define the normalized occupancy
measure as
$d_{\pi_{\star}}=(1-\gamma)\sum_{t=0}^{\infty}\gamma^{t}\mathbb{P}_{\pi}((s_{t},a_{t})=(s,a))$.
Note that $d_{\pi_{\star}}$ represents the stationary distribution.
Additionally, given the assumption that the reward function
$R:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is a bijection, it
follows that the distribution $d_{\pi_{\star}}(R^{-1}(s,a))$ and
$\mathbb{P}_{r}$ are identical. This indicates that the occurrence of black
swan events can be entirely characterized by the reward values, rather than
the specific state-action pairs.

Now, we define the event $E_{bs}:=\\{{\mathcal{R}}\in[-R_{\max},-R_{bs}]\\}$
where ${\mathcal{R}}\sim\mathbb{P}_{r}$. The probability of event $E_{bs}$
happens is bounded as follows

$\displaystyle\mathbb{P}(E_{bs})$ $\displaystyle=F(-R_{bs})-F(-R_{\max})$
$\displaystyle=F(-R_{bs})$
$\displaystyle\in\left(\Big{(}\frac{R_{\max}-R_{bs}}{2R_{\max}}\Big{)}\epsilon^{\min}_{bs},\Big{(}\frac{R_{\max}-R_{bs}}{2R_{\max}}\Big{)}\epsilon^{\max}_{bs}\right)$
$\displaystyle:=[p^{\min}_{bs},p^{\max}_{bs}]$

Note that we have assumed the $0<\mathbb{P}_{r}(r=R(s,a))<\epsilon_{bs}$ and
its minimum reachable probability as $\epsilon^{\min}_{bs}$ for all reward.
now, for given trajectory, the reward instance is given as
$(r_{1},r_{2},...r_{h},...)$ where $r_{h}\sim\mathbb{P}_{r}$, the probability
that the agent first visit the black swan event at step $h$ would be defined
as

$\displaystyle\mathbb{P}\left(r_{1},\cdots,r_{h-1}\notin E_{bs},r_{h}\in
E_{bs}\right)$ $\displaystyle=(1-\mathbb{P}(E_{bs}))^{h-1}\mathbb{P}(E_{bs})$
$\displaystyle\leq(1-p_{\min})^{h-1}p_{\max}$

Therefore, its probability is bounded as follows,

$(1-p_{\max})^{h-1}p_{\min}\leq\mathbb{P}\left(r_{1},\cdots,r_{h-1}\notin
E_{bs},r_{h}\in E_{bs}\right)\leq(1-p_{\min})^{h-1}p_{\max}$

Now, to ensure that the blackswan probability to be lower bounded than
$\delta$, we need the following conditions,

$\displaystyle\delta$ $\displaystyle\leq(1-p_{\max})^{h-1}p_{\min}$
$\displaystyle\log{\delta}$
$\displaystyle\leq(h-1)\log{(1-p_{\max})}+\log{p_{\min}}$

Therefore, we have

$h\geq\log{\left(\delta/p_{\min}\right)}/\log(1-p_{max})+1.$

Therefore, we can conclude that if
$h=\Omega(\log{\left(\delta/p_{\min}\right)}/\log(1-p_{max}))$, then the
agent’s probability to meet the black swan is at least $\delta$.

∎∎

##  Appendix B Helpful Lemmas

######  Lemma 4.

(Dvoretzky-Kiefer-Wolfowitz (DKW) inequality)
Let ${\hat{F}_{n}}(u)=\frac{1}{n}\sum_{i=1}^{n}1_{((u(X_{i}))\leq u)}$ denote
the empirical distribution of a r.v. $U$, with $u(X_{1}),\ldots,u(X_{n})$
being sampled from the r.v $u(X)$. The, for any $n$ and $\epsilon>0$, we have

$P(\sup_{x\in\mathbb{R}}|\hat{F_{n}}(x)-F(x)|>\epsilon)\leq
2e^{-2n\epsilon^{2}}.$

## References

  * Abel et al. (2024) D. Abel, A. Barreto, B. Van Roy, D. Precup, H. P. van Hasselt, and S. Singh.  A definition of continual reinforcement learning.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Agarwal et al. (2020) A. Agarwal, S. Kakade, and L. F. Yang.  Model-based reinforcement learning with a generative model is minimax optimal.  In _Conference on Learning Theory_ , pages 67–83. PMLR, 2020.
  * Agrawal et al. (2021) S. Agrawal, S. K. Juneja, and W. M. Koolen.  Regret minimization in heavy-tailed bandits.  In _Conference on Learning Theory_ , pages 26–62. PMLR, 2021.
  * Bagnell et al. (2001) J. A. Bagnell, A. Y. Ng, and J. G. Schneider.  Solving uncertain markov decision processes.  2001\.
  * Barandiaran et al. (2009) X. Barandiaran, E. D. Paolo, and M. Rohde.  Defining agency: Individuality, normativity, asymmetry, and spatio-temporality in action.  _Adaptive Behavior_ , 17(5):367–386, 2009.
  * BetterUp (2022) BetterUp.  The availability heuristic.  https://www.betterup.com/blog/the-availability-heuristic, 2022.  Accessed: 2024-05-12.
  * Bowling et al. (2023) M. Bowling, J. D. Martin, D. Abel, and W. Dabney.  Settling the reward hypothesis.  In _International Conference on Machine Learning_ , pages 3003–3020. PMLR, 2023.
  * Bradley and Taqqu (2003) B. O. Bradley and M. S. Taqqu.  Financial risk and heavy tails.  In _Handbook of heavy tailed distributions in finance_ , pages 35–103. Elsevier, 2003.
  * Bubeck et al. (2013) S. Bubeck, N. Cesa-Bianchi, and G. Lugosi.  Bandits with heavy tail.  _IEEE Transactions on Information Theory_ , 59(11):7711–7717, 2013.
  * Carr et al. (2024) J. C. Carr, P. Panangaden, and D. Precup.  Conditions on preference relations that guarantee the existence of optimal policies.  In _International Conference on Artificial Intelligence and Statistics_ , pages 3916–3924. PMLR, 2024.
  * Chung and Sobel (1987) K.-J. Chung and M. J. Sobel.  Discounted mdp’s: distribution functions and exponential utility maximization.  _Siam Journal on Control and Optimization_ , 25:49–62, 1987.  URL https://api.semanticscholar.org/CorpusID:119760011.
  * Coraluppi (1997) S. P. Coraluppi.  _Optimal control of Markov decision processes for performance and robustness_.  University of Maryland, College Park, 1997.
  * Coraluppi and Marcus (1999) S. P. Coraluppi and S. I. Marcus.  Risk-sensitive and minimax control of discrete-time, finite-state markov decision processes.  _Automatica_ , 35(2):301–309, 1999.
  * Coraluppi and Marcus (2000) S. P. Coraluppi and S. I. Marcus.  Mixed risk-neutral/minimax control of discrete-time, finite-state markov decision processes.  _IEEE Transactions on Automatic Control_ , 45(3):528–532, 2000.
  * Danis et al. (2023) D. Danis, P. Parmacek, D. Dunajsky, and B. Ramasubramanian.  Multi-agent reinforcement learning with prospect theory.  In _2023 Proceedings of the Conference on Control and its Applications (CT)_ , pages 9–16. SIAM, 2023.
  * Delage and Mannor (2010) E. Delage and S. Mannor.  Percentile optimization for markov decision processes with parameter uncertainty.  _Operations research_ , 58(1):203–213, 2010.
  * Di Castro et al. (2012) D. Di Castro, A. Tamar, and S. Mannor.  Policy gradients with variance related risk criteria.  _arXiv preprint arXiv:1206.6404_ , 2012.
  * Dong et al. (2022) S. Dong, B. Van Roy, and Z. Zhou.  Simple agent, complex environment: Efficient reinforcement learning with agent states.  _Journal of Machine Learning Research_ , 23(255):1–54, 2022.
  * Drakopoulos and Theodossiou (2016) S. A. Drakopoulos and I. Theodossiou.  Workers’ risk underestimation and occupational health and safety regulation.  _European Journal of Law and Economics_ , 41:641–656, 2016.
  * Fard and Pineau (2010) M. Fard and J. Pineau.  Pac-bayesian model selection for reinforcement learning.  _Advances in Neural Information Processing Systems_ , 23, 2010.
  * Fennema and Wakker (1997) H. Fennema and P. Wakker.  Original and cumulative prospect theory: A discussion of empirical differences.  _Journal of Behavioral Decision Making_ , 10(1):53–64, 1997.
  * Fleming and Sarkar (2014) M. J. Fleming and A. Sarkar.  The failure resolution of lehman brothers.  _Economic Policy Review, Forthcoming_ , 2014.
  * Garcıa and Fernández (2015) J. Garcıa and F. Fernández.  A comprehensive survey on safe reinforcement learning.  _Journal of Machine Learning Research_ , 16(1):1437–1480, 2015.
  * Geibel (2006) P. Geibel.  Reinforcement learning for mdps with constraints.  In _Machine Learning: ECML 2006: 17th European Conference on Machine Learning Berlin, Germany, September 18-22, 2006 Proceedings 17_ , pages 646–653. Springer, 2006.
  * Geibel and Wysotzki (2005) P. Geibel and F. Wysotzki.  Risk-sensitive reinforcement learning applied to control under constraints.  _Journal of Artificial Intelligence Research_ , 24:81–108, 2005.
  * Gheshlaghi Azar et al. (2013) M. Gheshlaghi Azar, R. Munos, and H. J. Kappen.  Minimax pac bounds on the sample complexity of reinforcement learning with a generative model.  _Machine learning_ , 91:325–349, 2013.
  * Gilovich et al. (2002) T. Gilovich, D. Griffin, and D. Kahneman.  _Heuristics and biases: The psychology of intuitive judgment_.  Cambridge university press, 2002.
  * Gosavi (2009) A. Gosavi.  Reinforcement learning for model building and variance-penalized control.  In _Proceedings of the 2009 winter simulation conference (wsc)_ , pages 373–379. IEEE, 2009.
  * Heger (1994) M. Heger.  Consideration of risk in reinforcement learning.  In _Machine Learning Proceedings 1994_ , pages 105–111. Elsevier, 1994.
  * Hendrycks et al. (2021) D. Hendrycks, N. Carlini, J. Schulman, and J. Steinhardt.  Unsolved problems in ml safety.  _arXiv preprint arXiv:2109.13916_ , 2021.
  * Housel (2023) M. Housel.  Penguin, 2023.
  * Howard and Matheson (1972) R. A. Howard and J. E. Matheson.  Risk-sensitive markov decision processes.  _Management science_ , 18(7):356–369, 1972.
  * Huisman et al. (1998) R. Huisman, K. G. Koedijk, and R. A. Pownall.  Var-x: Fat tails in financial risk management.  _Journal of risk_ , 1(1):47–61, 1998.
  * Iyengar (2005) G. N. Iyengar.  Robust dynamic programming.  _Mathematics of Operations Research_ , 30(2):257–280, 2005.
  * Jiang (2018) N. Jiang.  Pac reinforcement learning with an imperfect model.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Jiang (2019) N. Jiang.  On value functions and the agent-environment boundary.  _arXiv preprint arXiv:1905.13341_ , 2019.
  * Jie et al. (2018) C. Jie, L. Prashanth, M. Fu, S. Marcus, and C. Szepesvári.  Stochastic optimization in a cumulative prospect theory framework.  _IEEE Transactions on Automatic Control_ , 63(9):2867–2882, 2018.
  * Kahneman and Tversky (2013) D. Kahneman and A. Tversky.  Prospect theory: An analysis of decision under risk.  In _Handbook of the fundamentals of financial decision making: Part I_ , pages 99–127. World Scientific, 2013.
  * Lattimore and Szepesvári (2020) T. Lattimore and C. Szepesvári.  _Bandit algorithms_.  Cambridge University Press, 2020.
  * Lecarpentier and Rachelson (2019) E. Lecarpentier and E. Rachelson.  Non-stationary markov decision processes, a worst-case approach using model-based reinforcement learning.  _Advances in neural information processing systems_ , 32, 2019.
  * Lee et al. (2024) H. Lee, Y. Ding, J. Lee, M. Jin, J. Lavaei, and S. Sojoudi.  Tempo adaptation in non-stationary reinforcement learning.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Leqi et al. (2019) L. Leqi, A. Prasad, and P. K. Ravikumar.  On human-aligned risk minimization.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Martino et al. (2006) B. D. Martino, D. Kumaran, B. Seymour, and R. J. Dolan.  Frames, biases, and rational decision-making in the human brain.  _Science_ , 313:684 – 687, 2006.
  * Mawutor (2014) J. K. M. Mawutor.  The failure of lehman brothers: causes, preventive measures and recommendations.  _Research Journal of Finance and Accounting_ , 5(4), 2014.
  * McDonald and Robinson (2009) L. McDonald and P. Robinson.  _A colossal failure of common sense: The incredible inside story of the collapse of Lehman Brothers_.  Random House, 2009.
  * Moldovan and Abbeel (2012) T. Moldovan and P. Abbeel.  Risk aversion in markov decision processes via near optimal chernoff bounds.  _Advances in neural information processing systems_ , 25, 2012.
  * Neummann and Morgenstern (1944) J. Neummann and O. Morgenstern.  Theory of games and economic behaviour, 1944.
  * Nilim and El Ghaoui (2005) A. Nilim and L. El Ghaoui.  Robust control of markov decision processes with uncertain transition matrices.  _Operations Research_ , 53(5):780–798, 2005.
  * Opaluch and Segerson (1989) J. J. Opaluch and K. Segerson.  Rational roots of “irrational” behavior: new theories of economic decision-making.  _Northeastern Journal of Agricultural and Resource Economics_ , 18(2):81–95, 1989.
  * Orseau and Ring (2012) L. Orseau and M. Ring.  Space-time embedded intelligence.  In _Artificial General Intelligence: 5th International Conference, AGI 2012, Oxford, UK, December 8-11, 2012. Proceedings 5_ , pages 209–218. Springer, 2012.
  * Osogami (2012) T. Osogami.  Robustness and risk-sensitivity in markov decision processes.  _Advances in Neural Information Processing Systems_ , 25, 2012.
  * Pandit et al. (2019) B. Pandit, A. Albert, Y. Patil, and A. J. Al-Bayati.  Impact of safety climate on hazard recognition and safety risk perception.  _Safety science_ , 113:44–53, 2019.
  * Patek (2001) S. D. Patek.  On terminating markov decision processes with a risk-averse objective function.  _Automatica_ , 37(9):1379–1386, 2001.