# Reinforcement Learning
with Non-Exponential Discounting

Matthias Schultheis
Centre for Cognitive Science
Technische Universität Darmstadt
<EMAIL_ADDRESS>
&Constantin A. Rothkopf
Centre for Cognitive Science
Technische Universität Darmstadt
<EMAIL_ADDRESS>
&Heinz Koeppl
Centre for Cognitive Science
Technische Universität Darmstadt
<EMAIL_ADDRESS>

###### Abstract

Commonly in reinforcement learning (RL), rewards are discounted over time
using an exponential function to model time preference, thereby bounding the
expected long-term reward. In contrast, in economics and psychology, it has
been shown that humans often adopt a hyperbolic discounting scheme, which is
optimal when a specific task termination time distribution is assumed. In this
work, we propose a theory for continuous-time model-based reinforcement
learning generalized to arbitrary discount functions. This formulation covers
the case in which there is a non-exponential random termination time. We
derive a Hamilton–Jacobi–Bellman (HJB) equation characterizing the optimal
policy and describe how it can be solved using a collocation method, which
uses deep learning for function approximation. Further, we show how the
inverse RL problem can be approached, in which one tries to recover properties
of the discount function given decision data. We validate the applicability of
our proposed approach on two simulated problems. Our approach opens the way
for the analysis of human discounting in sequential decision-making tasks.

##  1 Introduction

An often observed phenomenon in humans and animals is that they prefer rewards
rather sooner than later [1]. It comes with no surprise that an animal
searching for food aims to find it as soon as possible and employees working
in a company prefer to be paid after each month instead of after each year. In
behavioral experiments, it was observed that people are even willing to pay a
price to receive rewards earlier [2]. It can be concluded that rewards become
of less value in the future, a phenomenon which is known as discounting [2].

Modeling and experimentally inferring discounting functions, which describe
how values are discounted over time, already have a long history in economics
and psychology, where many different functional forms have been proposed [3,
4, 5]. While from an economic perspective, a fixed interest rate seems
reasonable, leading to an exponential discount function, human behavior is
oftentimes better described by a hyperbolic curve [6]. The reason for this is
that human decisions are typically not consistent regarding shifting rewards
in time, a phenomenon named preference reversal: We might find that a subject
prefers a smaller reward on the same day over a larger reward one day after.
However, when given the choice between the same smaller reward in 365 days,
compared to the same larger reward in 366 days, the subject is more likely to
be willing to wait one day more for the larger reward.

While some literature has branded the observed discounting behavior as not
being rational [7], there has been an increasing amount of work identifying
circumstances under which this behavior is indeed optimal [8, 9, 10]. One
widely-adopted theory rationalizing hyperbolic discounting is to assume a
constant risk for the reward to become unavailable but with the risk being
uncertain [11]. Under this condition, one should adapt the preference over
time, as with time the expected risk decreases.

Discounting is also widely applied in the field of reinforcement learning (RL)
and optimal control [12, 13, 14]. First, when modeling infinite horizon time
objectives, a discount function is needed to make the expected long-term
reward objective well-defined, as otherwise, it would become infinite. Second,
for autonomous agents, it also makes sense to model a preference for earlier
rewards in order to find shortest paths to save time and energy. Third, the
discount function can be interpreted as the probability of termination
inducing a specific end-time distribution [15, 12].

Despite these obvious connections, discounting models of psychology and
reinforcement learning have remained mostly independent with few exceptions
[16, 17]. Generalizing reinforcement learning to a broader range of discount
functions would enable solutions for applications in which the end time
follows a specific distribution. On the other hand, methods for determining
optimal decisions in sequential decision-making tasks with general end-time
distributions would provide tools which can help to explain human decision-
making under uncertainty.

In this work, we present a theory for model-based reinforcement learning in
continuous time based on non-exponential discount functions. First, we
investigate the conditions under which the objective of maximizing the long-
term reward formulated with hyperbolic discounting is well-defined. Second, we
derive a HJB-type equation for a general discount function and describe how to
solve it to obtain the optimal policy. Third, we provide an approach to tackle
the inverse reinforcement learning (IRL) problem, to estimate parameters of
the discount function given decision data. Finally, we show the applicability
of our proposed method on two simulated problems.

##  2 Related work

Optimal control in continuous time and space has a long history with many
classical works [18, 19, 20, 21]. Continuous-time reinforcement learning
formulations have been developed [14] and various solution methods have been
proposed [22, 23]. Solution approaches that solve the HJB equation directly
include linearization techniques [24, 25], path integral formulations [26, 27,
12], and collocation-based methods [28]. In recent years, it has become
increasingly popular to use neural networks for function approximation to
solve the HJB equation [29, 30, 31, 32, 33].

Non-exponential discounting was considered in literature in behavioral
economics, psychology, and neuroscience [4, 34, 2]. For humans and animals,
preference reversal behavior has been described [35, 36, 5] and different
functional forms for the discount function have been proposed [3, 37]. In most
works, only non-sequential decision-making tasks have been studied, i.e.,
situations with a single decision for each independent trial. In this line of
work, the method of limits has been used to elicit discount functions based on
binary decisions [38, 39, 40] or an adjustment method has been applied [41].
Some methods have been proposed that specifically aim to efficiently estimate
the discount function for binary choice responses [42, 43]. There have been a
few exceptions for which decision trials were actually dependent but could
still be considered independently for the analysis [44, 45]. Further, there
has been research aiming to find rational explanations for the encountered
discounting behavior [8, 9, 10], e.g., by assuming uncertainty about a
constant hazard [11].

Decision processes with non-exponential time distributions have been studied
in the field of semi-Markov Decision Processes [46, 47, 48, 49], where
transition times between discrete states can follow arbitrary distributions.
MDPs with quasi-hyperbolic discounting, for which all future rewards are
additionally discounted by a constant factor, have been addressed in [50, 51,
52]. They can be used to model preference reversal but are limited to the
specific form assumed. Stochastic processes with quasi-hyperbolic discounting
have been considered in financial economics for portfolio management [53, 54,
55, 56].

More closely related to our work, Fedus et al. [17] presented an approximate
method for solving MDPs with hyperbolic discounting. They approached the
problem by solving the corresponding exponentially-discounted problem for many
different discount factors and combining the results. Part of the
approximation, however, is that the value function and policy are static,
failing to model preference reversals over time. Another line of work
considers MDPs with discount factors that are coupled to the value function to
imitate hyperbolic discounting behavior [16, 57]. As time is only considered
indirectly through the magnitude of the value function, these approaches
cannot be used for finding the solution to a given specific discount function
or eliciting the discount function from data.

Finally, inverse reinforcement learning approaches have been mainly used to
learn reward functions given data [58, 59, 60, 61]. These methods were also
applied to learn properties of human behavior [62, 63, 64, 65]. Other inverse
approaches have focused on learning dynamics models [66] or learning rules
[67] from sequential decision-making data.

##  3 Background

###  3.1 Survival analysis

In survival analysis [68], one is interested in the duration until events
occur. In its classical form one considers a single event, for which the
duration can be described as a continuous random variable $T$ with cumulative
distribution function $F(t)=P(T\leq t)$ and probability density function
$f(t)$, where $t\in\operatorname{\mathbb{R}}^{+}_{0}$ denotes the elapsed
time. In survival analysis literature, $F(t)$ is known as the failure function
and the survival function is defined as $S(t)=1-F(t)=P(T>t)$. The survival
function is monotonically decreasing and has the properties $S(0)=1$ and
$\lim_{t\to\infty}S(t)=0$. By the conditioning rule, one finds
$P(T>t_{1}\>|\>T>t_{0})=S(t_{1})/S(t_{0})$ if $t_{1}>t_{0}$. The hazard rate
is defined as $\alpha(t)=\lim_{\Delta t\rightarrow 0}\frac{1}{\Delta t}P(t\leq
T<t+\Delta t\>|\>t\leq T)$, yielding the relations

$\displaystyle\alpha(t)=\lim_{\Delta t\rightarrow 0}\frac{1}{\Delta
t}\frac{S(t)-S(t+\Delta
t)}{S(t)}=-\frac{S^{\prime}(t)}{S(t)}\quad\text{and}\quad
S(t)=\exp\left(-\int_{0}^{t}\alpha(\tau)\,\mathrm{d}\tau\right).$ (1)

For a constant hazard rate $\alpha(t)=\lambda$, one finds $S(t)=\exp(-\lambda
t)$, which can be shown to be the unique memory-less survival function [68],
i.e., $P(T>t+\Delta t\>|\>T>t)=P(T>\Delta t)$ for all $t,\Delta
t\in\operatorname{\mathbb{R}}^{+}_{0}$.

###  3.2 Discounting and preference reversal

We consider the setting in which a subject collects a single reward and shows
a form of time preference, i.e., rewards are desired rather sooner than later.
We model the value of a reward $r\in\operatorname{\mathbb{R}}$ as a function
$L:\operatorname{\mathbb{R}}\times\operatorname{\mathbb{R}}^{+}_{0}\to\operatorname{\mathbb{R}}$
with $L(r,t)=S(t)\cdot r$, where $S(t)$ is the discount function decreasing
with time $t\in\operatorname{\mathbb{R}}^{+}_{0}$. With the convention that
$S(t_{0})=1$ and $\lim_{t\to\infty}S(t)=0$, we can regard $S(t)$ as the
survival function (Section 3.1), with the interpretation that the reward
becomes unavailable with hazard rate $\alpha(t)$ [11].

When assuming a constant hazard rate $\alpha(t)=\lambda$, we say that the
subject discounts exponentially, as $S(t)=\exp(-\lambda t)$. By the memory-
less property, we have that if the subject prefers reward $r_{1}$ after
$t_{1}$ over $r_{2}$ after $t_{2}$, she or he would remain consistent with the
election if we presented the choice again later in time. On the other hand, if
we assume a constant but unknown hazard rate $\lambda$ with belief
$p(\lambda)=\operatorname{\mathrm{Gamma}}(\lambda;\alpha_{0},\beta_{0})$, we
obtain a hyperbolic form for the expected survival function:

$\displaystyle S(t;\alpha_{0},\beta_{0})$
$\displaystyle=\int_{\lambda}\exp(-\lambda
t)\,p(\lambda)\,\mathrm{d}\lambda=\frac{1}{(\frac{t}{\beta_{0}}+1)^{\alpha_{0}}}$
(2)

The posterior belief over $\lambda$ at a later point in time can be derived
using Bayes rule and is given by
$p(\lambda\>|\>t)=\operatorname{\mathrm{Gamma}}(\lambda;\alpha_{0},\beta_{0}+t)$.
The expected hazard rate $\alpha(t)$ is given by the posterior mean,

$\displaystyle\alpha(t)=\int_{\lambda}\lambda\,p(\lambda\>|\>t)\,\mathrm{d}\lambda=\frac{\alpha_{0}}{\beta_{0}+t}.$
(3)

For discount functions other than the exponential, such as the hyperbolic
discount function in Eq. 2, the hazard rate varies over time and preferences
among options may change.

###  3.3 Optimal control

In stochastic optimal control [69], we consider a Markovian system with
continuous state $\mathbf{x}(t)\in\operatorname{\mathbb{R}}^{n}$ evolving
according to the stochastic differential equation (SDE)
$\mathrm{d}{\mathbf{X}}(t)=f(\mathbf{X}(t),\mathbf{u}(t))\,\mathrm{d}t+G(\mathbf{X}(t),\mathbf{u}(t))\,\mathrm{d}\mathbf{W}(t)$
where $t\in\operatorname{\mathbb{R}}^{+}_{0}$ denotes time,
$f:\mathcal{X}\times\mathcal{U}\to\mathcal{X}$ the drift function,
$G:\mathcal{X}\times\mathcal{U}\to\mathcal{X}\times\operatorname{\mathbb{R}}^{m}$
the dispersion matrix, and $\mathbf{W}(t)\in\operatorname{\mathbb{R}}^{m}$
$m$-dimensional Brownian motion. The goal of optimal control is to determine
the control inputs $\mathbf{u}(t)\in\mathcal{U}$ given the current state
$\mathbf{x}$ at time $t$, in order to maximize the expected long term
discounted reward with reward function
$R:\mathcal{X}\times\mathcal{U}\to\operatorname{\mathbb{R}}$. The solution is
characterized by the optimal value function,

$\displaystyle
V^{*}(\mathbf{x})=\max_{\mathbf{u}_{[t,\infty)}}\operatorname{\mathbb{E}\,}\left[\int_{t}^{\infty}\exp({-\lambda(\tau-t)})\,R(\mathbf{X}(\tau),\mathbf{u}(\tau))\,\mathrm{d}\tau\>\Big{|}\>\mathbf{X}(t)=\mathbf{x}\right],$
(4)

where maximization is carried out over all trajectories
$\mathbf{u}_{[t,\infty)}\mathrel{\mathop{:}}=\\{\mathbf{u}(\tau)\\}_{\tau\in[t,\infty]}$.
The quantity $\exp(-\lambda t)$ is the discount factor, which ensures
convergence of the integral and models a preference for earlier rewards.
$\lambda$ can be interpreted in terms of survival analysis as the hazard rate
for termination (cf. Section 3.1). According to the principle of optimality,
the stochastic Hamilton-Jacobi-Bellman (HJB) equation, given by

$\displaystyle\lambda
V^{*}(\mathbf{x})=\max_{\mathbf{u}\in\mathcal{U}}\left[R(\mathbf{x},u)+V_{\mathbf{x}}^{*}(\mathbf{x})^{T}f(\mathbf{x},\mathbf{u})+\frac{1}{2}\operatorname{\mathrm{tr}\,}{\left\\{V_{\mathbf{x}\mathbf{x}}^{*}(\mathbf{x})G(\mathbf{x},\mathbf{u})G(\mathbf{x},\mathbf{u})^{T}\right\\}}\right],$

provides a condition for the optimal value function. Here, partial derivatives
are denoted by the index notation, i.e., $V_{\mathbf{x}}$ denotes the partial
derivative of $V$ w.r.t. $\mathbf{x}$ and $V_{\mathbf{x}\mathbf{x}}$ the
respective Hessian. Note that the optimal value function depends on time only
indirectly through the state $\mathbf{x}$ due to the Markov property and the
memory-less discount function. This dependence also applies to the optimal
policy $\mathcal{\pi}^{*}:\mathcal{X}\to\mathcal{U}$, given by the maximizer
of the right-hand side of the HJB equation, i.e.,

$\displaystyle\pi(\mathbf{x})=\operatorname*{\arg\max}_{\mathbf{u}\in\mathcal{U}}\left[R(\mathbf{x},u)+V^{*}_{\mathbf{x}}(\mathbf{x})^{T}f(\mathbf{x},\mathbf{u})+\frac{1}{2}V^{*}_{\mathbf{x}\mathbf{x}}(\mathbf{x})G(\mathbf{x},\mathbf{u})G(\mathbf{x},\mathbf{u})^{T}\right].$

##  4 Reinforcement learning with general discount function

We consider a system as in stochastic optimal control (cf. Section 3.3) with
continuous state space $\operatorname{\mathbb{R}}^{n}$ and finite set of
controls $\mathcal{U}$. Instead of an exponential discount function, we allow
for general survival functions $S(t)$ based on a time-dependent hazard rate
$\alpha(t)$. This setting generalizes the discounting formulation of Section
3.2 to sequential decisions. As we will see later, the resulting value
function will become time-dependent in contrast to Section 3.3 and we can
further allow for time-dependent dynamics and reward without increasing
complexity. We therefore assume for the state evolution the SDE
$\mathrm{d}{\mathbf{X}}(t)=f(\mathbf{X}(t),\mathbf{u}(t),t)\,\mathrm{d}t+G(\mathbf{X}(t),\mathbf{u}(t),t)\,\mathrm{d}\mathbf{W}(t)$
with
$f:\mathcal{X}\times\mathcal{U}\times\operatorname{\mathbb{R}}^{+}_{0}\to\mathcal{X}$
and
$G:\mathcal{X}\times\mathcal{U}\times\operatorname{\mathbb{R}}^{+}_{0}\to\mathcal{X}\times\operatorname{\mathbb{R}}^{m}$.
The objective function measuring the total expected discounted reward is given
by

$\displaystyle
J\left(\mathbf{u}_{[0,\infty)}\right)=\operatorname{\mathbb{E}\,}\left[\int_{0}^{\infty}S(\tau)\,R(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)\,\mathrm{d}\tau\right],$
(5)

with time-dependent reward function
$R:\mathcal{X}\times\mathcal{U}\times\operatorname{\mathbb{R}}^{+}_{0}\to\operatorname{\mathbb{R}}$.
Analogous to Section 3.3, we define the expected reward-to-go as the value
function

$\displaystyle
V^{*}(\mathbf{x},t)=\max_{\mathbf{u}_{[t,\infty)}}\operatorname{\mathbb{E}\,}\left[\int_{t}^{\infty}\frac{S(\tau)}{S(t)}R(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)\,\mathrm{d}\tau\>\Big{|}\>\mathbf{X}(t)=\mathbf{x}\right],$
(6)

where $S(\tau)/S(t)$ is the probability of survival until time $\tau$,
conditioned on the fact that one already has survived until time $t$ (cf.
Section 3.1). In contrast to Eq. 4, the value function becomes time-dependent
through the general survival function and also the optimal policy depends on
time.

###  4.1 Technical requirements of model and discount function

First, the stochastic process defined in Section 4 must have a strongly unique
solution, which is the case if $f$ and $G$ grow at most linearly in
$\mathbf{X}$ and are Lipschitz continuous in the same variable [70]. Further,
the optimal value function is only in rare cases smooth enough to be a
solution in the “classical” sense. Instead, one considers a viscosity solution
[20], which satisfies the HJB equation in an appropriate generalized sense. A
sufficient condition for the existence of such a solution is that $f$ and $G$
are continuous with bounded continuous first derivatives w.r.t. $\mathbf{X}$
and $t$, and bounded. $R$ and $S$ need to be continuous and grow at most
polynomially in $\mathbf{X}$ and $t$ in an absolute sense. More details on the
existence and uniqueness of viscosity solutions can be found in [20]. A third
requirement is that the integral in the definition of the value function Eq. 5
converges. For the case of a hyperbolic discount function as in Eq. 2, we find
the following theorem:

######  Theorem 1.

Consider the hyperbolic discount function in Eq. 2. If the reward function
$R(\mathbf{x},\mathbf{u},t)$ is bounded above for all
$(\mathbf{x},\mathbf{u},t)\in\mathcal{X}\times\mathcal{U}\times\operatorname{\mathbb{R}}^{+}_{0}$,
and $\alpha_{0}>1$, the value function defined in equation Eq. 6 is well-
defined. If $R(\mathbf{x},\mathbf{u},t)$ is bounded below for all
$(\mathbf{x},\mathbf{u},t)\in\mathcal{X}\times\mathcal{U}\times\operatorname{\mathbb{R}}^{+}_{0}$,
and $\alpha_{0}\leq 1$, the value function is not well-defined as it becomes
infinite.

###### Proof.

See Appendix A. ∎

In the examples considered later, we will assume a bounded reward function and
a hyperbolic discount function with $\alpha_{0}>1$, for which Eq. 5 and Eq. 6
are well-defined.

###  4.2 HJB equation for a general discount function

In the following, we give a brief overview of the derivation of the HJB
equation for a general discount function. A more detailed derivation is
provided in Appendix B. First, we split the integral in Eq. 6 into two terms
such that we obtain a recursive formulation of the value function:

$\displaystyle V^{*}(\mathbf{x},t)=\max_{\mathbf{u}_{[t,t+\Delta
t]}}\operatorname{\mathbb{E}\,}$ $\displaystyle\left[\int_{t}^{t+\Delta
t}\frac{S(\tau)}{S(t)}R(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)\,\mathrm{d}\tau\right.$
$\displaystyle+\Biggl{.}\frac{S(t+\Delta t)}{S(t)}V^{*}(\mathbf{X}(t+\Delta
t),t+\Delta t)\>\Big{|}\>\mathbf{X}(t)=\mathbf{x}\Biggr{]}$

For the second term in the expectation, we apply a Taylor expansion and Itô’s
formula [69] and obtain

$\displaystyle V^{*}(\mathbf{X}(t+\Delta t),t+\Delta
t)=V^{*}(\mathbf{X}(t),t)+\int_{t}^{t+\Delta
t}V_{\mathbf{x}}^{*}(\mathbf{X}(\tau),\tau)\,f(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)\,\mathrm{d}\tau$
$\displaystyle\qquad+\int_{t}^{t+\Delta
t}V_{\mathbf{x}}^{*}(\mathbf{X}(\tau),\tau)\,G(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)\,\mathrm{d}\mathbf{W}(\tau)+\int_{t}^{t+\Delta
t}V_{t}^{*}(\mathbf{X}(\tau),\tau)\,\mathrm{d}\tau$
$\displaystyle\qquad+\int_{t}^{t+\Delta
t}\frac{1}{2}\operatorname{\mathrm{tr}\,}{\left\\{V_{\mathbf{x}\mathbf{x}}^{*}(\mathbf{X}(\tau),\tau)\,G(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)\,G(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)^{T}\right\\}}\,\mathrm{d}\tau+o(\Delta
t).$

Plugging this result into the equation above, dividing both sides by $\Delta
t$, and taking the limit $\Delta t\to 0$, as well as calculating the
expectation w.r.t. $\mathbf{W}(t)$ leads to the desired HJB equation

$\displaystyle\begin{split}\alpha(t)V^{*}(\mathbf{x},t)=\max_{\mathbf{u}}&\biggl{[}R(\mathbf{x},\mathbf{u},t)+V_{t}^{*}(\mathbf{x},t)+V_{\mathbf{x}}^{*}(\mathbf{x},t)\,f(\mathbf{x},\mathbf{u},t)\biggr{.}\\\
&\quad+\left.\frac{1}{2}\operatorname{\mathrm{tr}\,}{\left\\{V_{\mathbf{x}\mathbf{x}}^{*}(\mathbf{x},t)\,G(\mathbf{x},\mathbf{u},t)\,G(\mathbf{x},\mathbf{u},t)^{T}\right\\}}\right],\end{split}$
(7)

where $\alpha(t)$ can be recognized to be the hazard rate corresponding to the
survival function $S(t)$. We define the r.h.s. of the HJB equation in Eq. 7
without the maximization w.r.t. the action as

$\displaystyle\begin{split}Q(\mathbf{x},\mathbf{u},t)\mathrel{\mathop{:}}=\
&R(\mathbf{x},\mathbf{u},t)+V_{t}^{*}(\mathbf{x},t)+V_{\mathbf{x}}^{*}(\mathbf{x},t)\,f(\mathbf{x},\mathbf{u},t)\\\
&+\frac{1}{2}\operatorname{\mathrm{tr}\,}{\left\\{V_{\mathbf{x}\mathbf{x}}^{*}(\mathbf{x},t)\,G(\mathbf{x},\mathbf{u},t)\,G(\mathbf{x},\mathbf{u},t)^{T}\right\\}},\end{split}$

so that the optimal policy is given by
$\pi^{*}(\mathbf{x},t)=\operatorname*{\arg\max}_{\mathbf{u}}Q(\mathbf{x},\mathbf{u},t)$.
Later on, we will consider hyperbolic discounting, for which the hazard rate
$\alpha(t)$ is given by Eq. 3, i.e., $\alpha(t)=\alpha_{0}/(\beta_{0}+t)$.

###  4.3 Solving the HJB equation

In order to solve the HJB equation in Eq. 7, which is a PDE, we apply a
collocation-based method [28, 29, 31]. To do so, we first reformulate the HJB
equation as

$\displaystyle\begin{split}E(V,\mathbf{x},t)\mathrel{\mathop{:}}=-\alpha(t)V(\mathbf{x},t)+\max_{\mathbf{u}}&\biggl{[}R(\mathbf{x},\mathbf{u},t)+V_{t}(\mathbf{x},t)+V_{\mathbf{x}}(\mathbf{x},t)\,f(\mathbf{x},\mathbf{u},t)\biggr{.}\\\
&+\left.\frac{1}{2}\operatorname{\mathrm{tr}\,}{\left\\{V_{\mathbf{x}\mathbf{x}}(\mathbf{x},t)\,G(\mathbf{x},\mathbf{u},t)\,G(\mathbf{x},\mathbf{u},t)^{T}\right\\}}\right]=0,\end{split}$
(8)

and use a function approximator $V^{\psi}(\mathbf{x},t)$ for
$V^{*}(\mathbf{x},t)$. The parameters $\psi$ of the approximator can be
determined by sampling random states $\hat{\mathbf{x}}_{i}$ and time points
$\hat{t}_{i}$ and minimizing
$\sum_{i}E(V^{\psi},\hat{\mathbf{x}}_{i},\hat{t}_{i})^{2}$ w.r.t. $\psi$.
Calculating the derivatives $V_{\mathbf{x}}^{\psi}(\mathbf{x},t)$,
$V_{t}^{\psi}(\mathbf{x},t)$, $V_{\mathbf{x}\mathbf{x}}^{\psi}(\mathbf{x},t)$
and differentiating the objective function w.r.t. $\psi$ is straightforward
via automatic differentiation when choosing a neural network as function
approximator [71]. As $t$ is not bounded, we need to choose a
reparametrization of $t$ which maps all $t$ to the interval $[0,1)$ before
feeding the values into the network. More details about the implemented
parametrization and application of the collocation method are provided in
Appendix D. The complete algorithm to learn the value function and policy is
provided in Algorithm 1.

Result: Optimal value function $V^{\psi}(\mathbf{x},t)$, optimal policy
$\pi^{\psi}(\mathbf{x},t)$

Input: Parameters ${\bm{\theta}}$ of the discount function, system model,
number of iterations $K$

Initialize parameters $\psi$ of neural network for modeling
$V^{\psi}(\mathbf{x},t)$;

for  _$k\leftarrow 0$ to $K-1$_ do

Sample a batch of states and time points
$(\hat{\mathbf{x}},\hat{t})_{i=\\{1,\dots,N\\}}$;

Push $(\hat{\mathbf{x}},\hat{t})_{i=\\{1,\dots,N\\}}$ through the network to
obtain $\hat{V}^{\psi}_{i=\\{1,\dots,N\\}}$;

Use back-propagation to compute $\hat{V}_{\mathbf{x}i}^{\psi}$,
$\hat{V}_{ti}^{\psi}$, $\hat{V}_{\mathbf{x}\mathbf{x}i}^{\psi}$;

Evaluate $E(\hat{V}_{i}^{\psi},\hat{\mathbf{x}}_{i},\hat{t}_{i})$ in Eq. 8 and
determine maximizing action $\hat{\mathbf{u}}_{i}$ for $i=1,\dots,N$;

Use back-propagation to compute the gradient of
$\sum_{i}E(\hat{V}_{i}^{\psi},\hat{\mathbf{x}}_{i},\hat{t}_{i})^{2}$ w.r.t.
$\psi$ ;

Update $\psi$ using the gradient ;

end for

return $V^{\psi}(\mathbf{x},t)$,
$\pi^{\psi}(\mathbf{x},t)=\operatorname*{\arg\max}_{\mathbf{u}}Q(\mathbf{x},\mathbf{u},t)$

Algorithm 1 Computation of the optimal value function and policy for non-exp.
discounting

###  4.4 Inverse reinforcement learning for inferring the discount function

When analyzing human behavior, one might be interested in learning the
underlying discount function that led to a set of observed choices. In
contrast to standard inverse reinforcement learning settings, where the goal
is to learn the underlying reward function $R$, here we assume that the reward
function is given and the discount function $S(t)$ is unknown and needs to be
inferred. To learn the function, we assume a parametric form with parameters
${\bm{\theta}}$.

As given data, we assume the states and time points at which the subject
switches from one action to another one, i.e.,
$\mathcal{D}=\left\\{\mathbf{x},\mathbf{u}^{-},\mathbf{u}^{+},t\right\\}_{i=1\dots
N}$, describing that in state $\mathbf{x}$ at time $t$ the action $u^{-}$ is
switched to $u^{+}$. The observed decision maker is assumed to use the optimal
policy
$\pi(\mathbf{x},t)=\operatorname*{\arg\max}_{\mathbf{u}}Q(\mathbf{x},\mathbf{u},t)$.
Shortly before and after switching time $t$, we have $Q(\mathbf{x}(t-\Delta
t),\mathbf{u}^{-},t-\Delta t)>Q(\mathbf{x}(t-\Delta t),\mathbf{u}^{+},t-\Delta
t)$ and $Q(\mathbf{x}(t+\Delta t),\mathbf{u}^{-},t+\Delta
t)<Q(\mathbf{x}(t+\Delta t),\mathbf{u}^{+},t+\Delta t)$, respectively,
indicating that before $t$ action $\mathbf{u}^{-}$ is preferred over
$\mathbf{u}^{+}$ and vice versa afterwards. By letting $\Delta t\to 0$, one
finds $Q(\mathbf{x}(t),\mathbf{u}^{-},t)=Q(\mathbf{x}(t),\mathbf{u}^{+},t)$. A
sensible objective for the inverse problem is therefore to minimize
$\sum_{i}F(\mathbf{x}_{i},\mathbf{u}^{-}_{i},\mathbf{u}^{+}_{i},t_{i})$,
defined as the squared difference of both Q values, i.e.,

$\displaystyle F(\mathbf{x},\mathbf{u}^{-},\mathbf{u}^{+},t)$
$\displaystyle=\left[Q(\mathbf{x},\mathbf{u}^{-},t)-Q(\mathbf{x},\mathbf{u}^{+},t)\right]^{2}$
$\displaystyle=\biggl{[}R(\mathbf{x},\mathbf{u}^{+},t)-R(\mathbf{x},\mathbf{u}^{-},t)+V_{\mathbf{x}}(\mathbf{x},t)\left(f(\mathbf{x},\mathbf{u}^{+},t)-f(\mathbf{x},\mathbf{u}^{-},t)\right)\biggr{.}$
(9)
$\displaystyle+\left.\frac{1}{2}\operatorname{\mathrm{tr}\,}{\left\\{V_{\mathbf{x}\mathbf{x}}(\mathbf{x},t)\left(G(\mathbf{x},\mathbf{u}^{+},t)\,G(\mathbf{x},\mathbf{u}^{+},t)^{T}-G(\mathbf{x},\mathbf{u}^{-},t)\,G(\mathbf{x},\mathbf{u}^{-},t)^{T}\right)\right\\}}\right]^{2},$

w.r.t. the parameters ${\bm{\theta}}$ of the discount function. The
optimization needs to take Eq. 8 as a constraint into account to ensure that
the HJB equation is fulfilled. Note that Eq. 9 depends indirectly on the
parameters ${\bm{\theta}}$ through the definition of the value function.
However, the objective function $F$ could also directly depend on
${\bm{\theta}}$, we consider the general case in the following. In principle,
the state at which the switching occurs also depends on ${\bm{\theta}}$.
Nevertheless, we will assume that the parameters have only a minor influence
on the states of switching, so that terms including the variation of
$\mathbf{x}$ w.r.t. ${\bm{\theta}}$ can be neglected.

#### Gradient computation

For the minimization, it is useful to determine the gradient of the objective
function $F$ w.r.t. ${\bm{\theta}}$, which can be formulated using the chain
rule for total derivatives as

$\displaystyle\frac{\mathrm{d}F}{\mathrm{d}{\bm{\theta}}}$
$\displaystyle=F_{\bm{\theta}}+F_{V}V_{\bm{\theta}}+F_{V_{\mathbf{x}}}\left(V_{\mathbf{x}}\right)_{\bm{\theta}}+F_{V_{\mathbf{x}\mathbf{x}}}\left(V_{\mathbf{x}\mathbf{x}}\right)_{\bm{\theta}}$
$\displaystyle=F_{\bm{\theta}}+F_{V}V_{\bm{\theta}}+F_{V_{\mathbf{x}}}\left(V_{\bm{\theta}}\right)_{\mathbf{x}}+F_{V_{\mathbf{x}\mathbf{x}}}\left(V_{\bm{\theta}}\right)_{\mathbf{x}\mathbf{x}}.$
(10)

By switching the order of the partial derivatives in the last step, we have
obtained an expression for the gradient depending on $V_{\bm{\theta}}$. The
partial derivative $V_{\bm{\theta}}$ measures the influence of the discounting
parameters on the value function and is not straightforward to evaluate. We
can get a PDE for this quantity by observing that the constraint
$E(V^{*},\mathbf{x},t)$ in Eq. 8 is zero everywhere for the optimal value
function $V^{*}$ and therefore its derivative also needs to be zero. This
method is known as the forward sensitivity method [72] and provides an
additional PDE for the desired gradient $V_{\bm{\theta}}$:

$\displaystyle 0=\frac{\mathrm{d}E}{\mathrm{d}{\bm{\theta}}}$
$\displaystyle=E_{\bm{\theta}}+E_{V}V_{\bm{\theta}}+E_{V_{t}}\left(V_{t}\right)_{\bm{\theta}}+E_{V_{\mathbf{x}}}\left(V_{\mathbf{x}}\right)_{\bm{\theta}}+E_{V_{\mathbf{x}\mathbf{x}}}\left(V_{\mathbf{x}\mathbf{x}}\right)_{\bm{\theta}}$
$\displaystyle=E_{\bm{\theta}}+E_{V}V_{\bm{\theta}}+E_{V_{t}}\left(V_{\bm{\theta}}\right)_{t}+E_{V_{\mathbf{x}}}\left(V_{\bm{\theta}}\right)_{\mathbf{x}}+E_{V_{\mathbf{x}\mathbf{x}}}\left(V_{\bm{\theta}}\right)_{\mathbf{x}\mathbf{x}}=\mathrel{\mathop{:}}H(V,V_{\bm{\theta}},\mathbf{x},t)$
(11)

To solve the PDE and obtain $V_{\bm{\theta}}$, we can use the same procedure
as in Section 4.3. To obtain the gradient, first one has to solve the HJB
equation Eq. 7 to obtain $V^{\psi}$, then Section 4.4 to determine
$V^{\phi}_{{\bm{\theta}}}$. Afterwards, Section 4.4 gives the gradient to
update the parameters ${\bm{\theta}}$ of the discount function. The quantities
$V_{\bm{\theta}}$, $\left(V_{\bm{\theta}}\right)_{\mathbf{x}}$,
$\left(V_{\bm{\theta}}\right)_{\mathbf{x}\mathbf{x}}$ for the derivative be
computed via automatic differentiation as in Section 4.3. The complete
algorithm for computing the gradient is presented in Algorithm 2.

Result: Gradient of $F$ (Eq. 9) w.r.t. ${\bm{\theta}}$ Section 4.4

Input: Parameters ${\bm{\theta}}$ of the discount function, system model,
number of iterations $K$

Determine $V^{\psi}(\mathbf{x},\mathbf{u},t)$ using Algorithm 1 ;

Initialize parameters $\phi$ of neural network for modeling
$V_{{\bm{\theta}}}^{\phi}(\mathbf{x},t)$;

for  _$k\leftarrow 0$ to $K-1$_ do

Sample a batch of states and time points
$(\hat{\mathbf{x}},\hat{t})_{i=\\{1,\dots,N\\}}$;

Push $(\hat{\mathbf{x}},\hat{t})_{i=\\{1,\dots,N\\}}$ through the networks to
obtain $\hat{V}^{\phi}_{{\bm{\theta}}i=\\{1,\dots,N\\}}$ and
$\hat{V}^{\psi}_{i=\\{1,\dots,N\\}}$;

Use back-propagation to compute
$(\hat{V}^{\psi}_{{\bm{\theta}}})_{\mathbf{x}i}$,
$(\hat{V}^{\psi}_{{\bm{\theta}}})_{ti}$,
$(\hat{V}^{\psi}_{{\bm{\theta}}})_{\mathbf{x}\mathbf{x}i}$;

Evaluate
$\sum_{i}H(\hat{V}^{\psi}_{i},\hat{V}^{\phi}_{{\bm{\theta}}i},\hat{\mathbf{x}}_{i},\hat{t}_{i})^{2}$
in Section 4.4 for $i=1,\dots,N$;

Use back-propagation to compute the gradient of
$\sum_{i}H(\hat{V}^{\psi}_{i},\hat{V}^{\phi}_{{\bm{\theta}}i},\hat{\mathbf{x}}_{i},\hat{t}_{i})^{2}$
w.r.t. $\phi$ ;

Update $\phi$ using the gradient ;

end for

return $\mathrm{d}F/\mathrm{d}{\bm{\theta}}$ by evaluating Section 4.4 using
$V_{{\bm{\theta}}}^{\phi}(\mathbf{x},t)$

Algorithm 2 Computation of the gradient of $F$ w.r.t. ${\bm{\theta}}$ for
inferring the discount function Figure 1: Results for the investment problem.
A Learned value function for account balances and time points for hyperbolic
discounting. B Learned Q function for both actions over time (median and
quantiles for 10 runs). C Learned policy showing preference reversals for all
account balances except for $b=1$. D Simulation with account balance (top),
interest rate (middle), and action (bottom) over time. E Learned value
function for an exponential discount function, which is constant over time. G
Learned Q function for an exponential discount function. G Gradients obtained
for parameter pairs $[\alpha_{0},\beta_{0}]$ of the discount function given
simulated data on a $11\times 11$ grid. The parameters used to generate the
simulated trajectories are indicated by a red cross. H Values of the objective
function $F$ (Eq. 9) for parameter pairs on a $11\times 11$ grid when applying
the IRL method.

##  5 Experiments

We tested our derived method on two simulated problems with a random
termination time following a hyperbolic survival function. First, we solved
the HJB equation in Eq. 7 using the collocation method (Algorithm 1) and
computed the optimal policy. Then, for applying the inverse reinforcement
learning method, we evaluated the objective function in Eq. 9 and computed
gradients for different parameter sets ${\bm{\theta}}=[\alpha_{0},\beta_{0}]$
of the hyperbolic discount function in Eq. 2 using Algorithm 2. For generating
data, we randomly sampled starting states and determined the time points at
which subjects would switch their action. Afterward, the determined time
points were distorted by Gaussian noise.

All proposed methods were implemented in Python using the PyTorch framework
[71] and are available online111https://git.rwth-aachen.de/bcs/nonexp-rl. The
used hyperparameters are listed in Appendix F. As tasks, we considered an
investment problem and a problem controlling a point on a line. In the
following we provide a brief overview of the considered problems, more details
can be found in Appendix E.

###  5.1 Experimental tasks

In the investment problem, we model a subject having to decide whether to
invest her or his income in the bank account leading to future interests
(rewards) or to spend the money for immediate reward. We model the state as
the current balance of the bank account as well as the current interest rate.
When the money is spent (action spend), the subject receives rewards with a
rate of 0.1 but the balance of the account remains unchanged. When the income
is invested (action invest), the balance of the bank account increases with a
rate of 0.1 but there is no additional reward. In both cases the subject
receives rewards through interests, being proportional to the current balance
on the bank account. We assume that the interest rate varies over time
following a Gaussian diffusion model. To keep the state bounded, we model a
maximum balance for the account.

In the line problem, the task is to control a point along a line. The state
consists of the current position of the point. Possible actions are left and
right, which move the point to the respective direction, as well as stay,
which does not move the point at all. When moving the point, there is a
Gaussian diffusion on the position and one has to pay a small action cost
(negative reward). We model a state-dependent reward modeling a high reward
for distant states on one side and a low reward for close states on the other
side. This task can be seen as a continuous extension of the classical
discounting tasks. A more detailed description of the functional form is
provided in Appendix E.

Figure 2: Results for the line problem. A Learned value function for
hyperbolic discount function. B Learned Q function for each action over time
(median and quantiles for 10 runs) C Learned policy showing preference
reversals. D Simulation showing the state (top) and action (bottom) over time.
E Learned value function for an exponential discount function. G Learned Q
function for an exponential discount function. G Gradients obtained for
parameter pairs $[\alpha_{0},\beta_{0}]$ on a $15\times 15$ grid given
simulated data. The parameter used to generate the simulated trajectories is
indicated by a red cross. H Values of the objective function of the IRL
methods for parameter pairs on a $15\times 15$ grid.

###  5.2 Results

The proposed method in Algorithm 1 produces plausible value functions for the
considered problems. Figure 1 A shows the learned value function for the
investment problem. The values are increasing with account balance (b), as
expected rewards through interests become higher. Further in time, the value
also increases, as the hazard rate can be assumed to be lower and one is
expected to collect rewards for a longer time. Figure 1 B and C shows the
learned Q function and policy, respectively. While it is advantageous to spend
the income in the beginning, preference reversal occurs when the risk of
termination is assumed to be relatively low, and investing becomes more
attractive. The simulation in Fig. 1 D also reflects this behavior. For
comparison, when assuming exponential discounting (Fig. 1 E and F), the value
and Q function do not show preference reversal and remain constant over time.
With regard to the inverse approach, Fig. 1 G shows the computed gradients of
the IRL objective function $F$ in Eq. 9 for different parameter pairs. One can
observe that the gradients mostly point in the direction of the parameters
used for simulation, indicated by a red cross. Figure 1 H shows the evaluated
IRL objective function $F$ for the parameter pairs, correctly displaying low
values in the area close to the true parameters.

For the line problem, the computed value function is depicted in Fig. 2 A.
Also in this task, the learned value increases over time and is high in areas
close to highly rewarding states. Figure 2 B and C shows the learned Q
function and policy respectively. In the beginning when the risk could be
potentially high, if not close enough to the large reward, it is best to move
right to collect the smaller sooner reward. With time when the risk of
termination decreases, it becomes increasingly advantageous to move left to
collect the larger later reward. In between, there is a time span, during
which it is best to stay: The moving cost is too high for moving right and
shortly turning afterward, but the risk is not low enough for being worth it
to move left to the higher reward. The simulation in Fig. 2 D presents a
sampled trajectory, showing that the subject first moves right until the
maximum smaller sooner reward at $x=0.5$ is reached and stays there for some
time. After the risk has decreased over time, the simulated subject finally
moves right to collect the larger later reward – her or his preferences have
reversed. As for the investment problem, the value and Q function (Fig. 2 E
and F) do not model any time-varying behavior. With regard to the IRL problem,
the gradients and function values of $F$ of Eq. 9 in Fig. 2 G and H also lead
closely to parameter values used for the simulations.

##  6 Conclusion

In this work, we have proposed a method for reinforcement learning with non-
exponential discount functions. The approach can be used to solve decision-
making problems with an arbitrary end-time distribution and to model human
discounting behavior. First, we have shown the conditions for which the
problem is well-defined when using a hyperbolic discount function. Then, we
derived a HJB-type equation providing conditions for the optimal time-
dependent value function. We presented how the obtained PDE can be
approximately solved using a collocation method, leading to the optimal
policy. Further, we introduced an approach for the inverse problem, in which
the discount function needs to be inferred given behavioral data. The
application of our methods on two simulated problems led to plausible
solutions, opening the way for further applications such as the use in human
experiments.

##### Limitations and future work

In our proposed methods, we assume a finite action space. While this
assumption applies to many behavioral experiments, it can be a limitation for
the application to classical optimal control. To extend the method to
continuous control, one has to determine how the maximization problem in the
HJB equation is solved. For strictly convex action costs, there have been
approaches proposed that efficiently solve the maximization in the HJB
equation [31], under some conditions even in closed-form [29]. Further, for
Lipschitz-continuous controls, it is possible to approximate the control
dynamics to avoid solving the optimization problem [73]. Another requirement
of our method is that the model needs to be known. Model-free approaches, such
as TD-error learning for general discount functions could be explored, in line
with work such as [16]. Further, when approaching problems with large state
spaces, the proposed collocation method is likely to converge slowly. For
these cases, adapted collocation methods [30] or advantage updating approaches
[23] could be considered instead. To apply our method to more advanced
problems in financial engineering, such as modeling stocks with discontinuous
returns [74], one has to consider general jump-diffusion processes. While an
extension to these models is straightforward from a theoretical point, we here
left it out to keep the focus on the handling of the discount function.
Regarding our inverse reinforcement learning approach, we have assumed the
states and time points of the action switches to be given. While this
assumption is reasonable for many human behavior experiments, it might be
interesting to learn discount functions for given discretized trajectories
instead. Also, incorporating an extended timing model of human decisions [75]
instead of fixed-variance Gaussian diffusion would be an interesting
extension.

In the future, we are planning to apply our proposed methods in human
experiments to get new insights to human discounting behavior. Characterizing
individual human subjects by analyzing their behavior comes with the risk to
be used with negative social impact. This matter can be counteracted by
collecting only anonymized data for the application of our method.

## Acknowledgments and Disclosure of Funding

We thank Bastian Alt for insightful discussions and for providing feedback on
the first draft. This work was supported by the high-performance computer
Lichtenberg at the NHR Centers NHR4CES at TU Darmstadt and the project
“Whitebox” funded by the Priority Program LOEWE of the Hessian Ministry of
Higher Education, Science, Research and Art.

## References

  * O’Donoghue and Rabin [2000] Ted O’Donoghue and Matthew Rabin.  The economics of immediate gratification.  _Journal of Behavioral Decision Making_ , 13(2):233–250, 2000.
  * Frederick et al. [2002] Shane Frederick, George Loewenstein, and Ted O’donoghue.  Time discounting and time preference: A critical review.  _Journal of Economic Literature_ , 40(2):351–401, 2002.
  * McKerchar et al. [2009] Todd L McKerchar, Leonard Green, Joel Myerson, T Stephen Pickford, Jade C Hill, and Steven C Stout.  A comparison of four models of delay discounting in humans.  _Behavioural Processes_ , 81(2):256–259, 2009\.
  * Strotz [1955] Robert Henry Strotz.  Myopia and inconsistency in dynamic utility maximization.  _The Review of Economic Studies_ , 23(3):165–180, 1955.
  * Thaler [1981] Richard Thaler.  Some empirical evidence on dynamic inconsistency.  _Economics Letters_ , 8(3):201–207, 1981.
  * Mazur [1987] James E Mazur.  An adjusting procedure for studying delayed reinforcement.  _Quantitative Analyses of Behavior_ , 5:55–73, 1987.
  * Fisher [1930] Irving Fisher.  _Theory of interest: as determined by impatience to spend income and opportunity to invest it_.  Augustusm Kelly Publishers, 1930.
  * Dasgupta and Maskin [2005] Partha Dasgupta and Eric Maskin.  Uncertainty and hyperbolic discounting.  _American Economic Review_ , 95(4):1290–1299, 2005.
  * Takahashi [2005] Taiki Takahashi.  Loss of self-control in intertemporal choice may be attributable to logarithmic time-perception.  _Medical Hypotheses_ , 65(4):691–693, 2005\.
  * Ray and Bossaerts [2011] Debajyoti Ray and Peter Bossaerts.  Positive temporal dependence of the biological clock implies hyperbolic discounting.  _Frontiers in Neuroscience_ , 5:2, 2011.
  * Sozou [1998] Peter D Sozou.  On hyperbolic discounting and uncertain hazard rates.  _Proceedings of the Royal Society of London. Series B: Biological Sciences_ , 265(1409):2015–2020, 1998.
  * Sutton and Barto [2018] Richard S Sutton and Andrew G Barto.  _Reinforcement learning: An introduction_.  MIT Press, 2018.
  * Bertsekas [2019] Dimitri Bertsekas.  _Reinforcement learning and optimal control_.  Athena Scientific, 2019.
  * Doya [2000] Kenji Doya.  Reinforcement learning in continuous time and space.  _Neural Computation_ , 12(1):219–245, 2000\.
  * Puterman [1990] Martin L Puterman.  Markov decision processes.  _Handbooks in operations research and management science_ , 2:331–434, 1990.
  * Alexander and Brown [2010] William H Alexander and Joshua W Brown.  Hyperbolically discounted temporal difference learning.  _Neural Computation_ , 22(6):1511–1527, 2010\.
  * Fedus et al. [2019] William Fedus, Carles Gelada, Yoshua Bengio, Marc G Bellemare, and Hugo Larochelle.  Hyperbolic discounting and learning over multiple horizons.  _arXiv preprint arXiv:1902.06865_ , 2019.
  * Stratonovich [1968] Ruslan L Stratonovich.  Conditional Markov processes and their application to the theory of optimal control.  1968\.
  * Kushner and Dupuis [2001] Harold J Kushner and Paul G Dupuis.  _Numerical methods for stochastic control problems in continuous time_ , volume 24.  Springer Science & Business Media, 2001.
  * Fleming and Soner [2006] Wendell H Fleming and Halil Mete Soner.  _Controlled Markov processes and viscosity solutions_ , volume 25.  Springer Science & Business Media, 2006.
  * Pontryagin [1987] Lev Semenovich Pontryagin.  _Mathematical theory of optimal processes_.  CRC Press, 1987.
  * Vamvoudakis and Lewis [2010] Kyriakos G Vamvoudakis and Frank L Lewis.  Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem.  _Automatica_ , 46(5):878–888, 2010.
  * Baird [1994] Leemon C Baird.  Reinforcement learning in continuous time: Advantage updating.  In _IEEE International Conference on Neural Networks_ , volume 4, pages 2448–2453. IEEE, 1994.
  * Jacobson [1968] David H Jacobson.  New second-order and first-order algorithms for determining optimal control: A differential dynamic programming approach.  _Journal of Optimization Theory and Applications_ , 2(6):411–440, 1968.
  * Tassa et al. [2007] Yuval Tassa, Tom Erez, and William Smart.  Receding horizon differential dynamic programming.  In _Advances in Neural Information Processing Systems_ , volume 20. Curran Associates, Inc., 2007.
  * Kappen [2005] Hilbert J Kappen.  Path integrals and symmetry breaking for optimal control theory.  _Journal of Statistical Mechanics: Theory and Experiment_ , 2005(11):P11011, 2005.
  * Theodorou et al. [2010] Evangelos Theodorou, Jonas Buchli, and Stefan Schaal.  A generalized path integral control approach to reinforcement learning.  _Journal of Machine Learning Research_ , 11:3137–3181, 2010\.
  * Simpkins and Todorov [2009] Alex Simpkins and Emanuel Todorov.  Practical numerical methods for stochastic optimal control of biological systems in continuous time and space.  In _IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning_ , pages 212–218. IEEE, 2009.
  * Tassa and Erez [2007] Yuval Tassa and Tom Erez.  Least squares solutions of the HJB equation with neural network value-function approximators.  _IEEE Transactions on Neural Networks_ , 18(4):1031–1041, 2007.
  * Sirignano and Spiliopoulos [2018] Justin Sirignano and Konstantinos Spiliopoulos.  DGM: A deep learning algorithm for solving partial differential equations.  _Journal of Computational Physics_ , 375:1339–1364, 2018\.
  * Lutter et al. [2020] Michael Lutter, Boris Belousov, Kim Listmann, Debora Clever, and Jan Peters.  HJB optimal feedback control with deep differential value functions and action constraints.  In _Conference on Robot Learning_ , pages 640–650. PMLR, 2020.
  * Han et al. [2018] Jiequn Han, Arnulf Jentzen, and E Weinan.  Solving high-dimensional partial differential equations using deep learning.  volume 115, pages 8505–8510. National Academy of Sciences, 2018.
  * Alt et al. [2020] Bastian Alt, Matthias Schultheis, and Heinz Koeppl.  POMDPs in continuous time and discrete spaces.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 13151–13162. Curran Associates, Inc., 2020.
  * Thaler and Shefrin [1981] Richard H Thaler and Hersh M Shefrin.  An economic theory of self-control.  _Journal of Political Economy_ , 89(2):392–406, 1981.
  * Ainslie and Herrnstein [1981] George Ainslie and Richard J Herrnstein.  Preference reversal and delayed reinforcement.  _Animal Learning & Behavior_, 9(4):476–482, 1981.
  * Green et al. [1994] Leonard Green, Nathanael Fristoe, and Joel Myerson.  Temporal discounting and preference reversals in choice between delayed outcomes.  _Psychonomic Bulletin & Review_, 1(3):383–389, 1994.
  * Andersen et al. [2014] Steffen Andersen, Glenn W Harrison, Morten I Lau, and E Elisabet Rutström.  Discounting behavior: A reconsideration.  _European Economic Review_ , 71:15–33, 2014.
  * Richards et al. [1997] Jerry B Richards, Suzanne H Mitchell, Harriet De Wit, and Lewis S Seiden.  Determination of discount functions in rats with an adjusting-amount procedure.  _Journal of the Experimental Analysis of Behavior_ , 67(3):353–366, 1997.
  * Coller and Williams [1999] Maribeth Coller and Melonie B Williams.  Eliciting individual discount rates.  _Experimental Economics_ , 2(2):107–127, 1999\.
  * Andersen et al. [2008] Steffen Andersen, Glenn W Harrison, Morten I Lau, and E Elisabet Rutström.  Eliciting risk and time preferences.  _Econometrica_ , 76(3):583–618, 2008.
  * Noor [2009] Jawwad Noor.  Hyperbolic discounting and the standard model: eliciting discount functions.  _Journal of Economic Theory_ , 144(5):2077–2083, 2009.
  * Cavagnaro et al. [2016] Daniel R Cavagnaro, Gabriel J Aranovich, Samuel M McClure, Mark A Pitt, and Jay I Myung.  On the functional form of temporal discounting: an optimized adaptive test.  _Journal of Risk and Uncertainty_ , 52(3):233–254, 2016.
  * Chang et al. [2019] Jorge Chang, Jiseob Kim, Byoung-Tak Zhang, Mark A Pitt, and Jay I Myung.  Modeling delay discounting using gaussian process with active learning.  In _CogSci_ , pages 1479–1485, 2019.
  * Schweighofer et al. [2006] Nicolas Schweighofer, Kazuhiro Shishida, Cheol E Han, Yasumasa Okamoto, Saori C Tanaka, Shigeto Yamawaki, and Kenji Doya.  Humans can adopt optimal discounting strategy under real-time constraints.  _PLoS Computational Biology_ , 2(11):e152, 2006\.
  * Seinstra et al. [2018] Maayke Suzanne Seinstra, Manuela Sellitto, and Tobias Kalenscher.  Rate maximization and hyperbolic discounting in human experiential intertemporal decision making.  _Behavioral Ecology_ , 29(1):193–203, 2018.
  * Howard [1963] Ronald A Howard.  Semi-Markovian decision processes.  _Bulletin of the International Statistical Institute_ , 40(2):625–652, 1963.
  * Korolyuk et al. [1975] V S Korolyuk, S M Brodi, and A F Turbin.  Semi-Markov processes and their applications.  _Journal of Soviet Mathematics_ , 4(3):244–280, 1975.
  * Ross [1970] Sheldon M Ross.  Average cost semi-Markov decision processes.  _Journal of Applied Probability_ , 7(3):649–656, 1970.
  * Bradtke and Duff [1994] Steven Bradtke and Michael Duff.  Reinforcement learning methods for continuous-time Markov decision problems.  In _Advances in Neural Information Processing Systems_ , volume 7. MIT Press, 1994.
  * Jaśkiewicz and Nowak [2021] Anna Jaśkiewicz and Andrzej S Nowak.  Markov decision processes with quasi-hyperbolic discounting.  _Finance and Stochastics_ , 25(2):189–229, 2021\.
  * Bjork and Murgoci [2010] Tomas Bjork and Agatha Murgoci.  A general theory of Markovian time inconsistent stochastic control problems.  _SSRN_ , 2010.
  * Björk and Murgoci [2014] Tomas Björk and Agatha Murgoci.  A theory of Markovian time-inconsistent stochastic control in discrete time.  _Finance and Stochastics_ , 18(3):545–592, 2014\.
  * Grenadier and Wang [2007] Steven R Grenadier and Neng Wang.  Investment under uncertainty and time-inconsistent preferences.  _Journal of Financial Economics_ , 84(1):2–39, 2007.
  * Zou et al. [2014] Ziran Zou, Shou Chen, and Lei Wedge.  Finite horizon consumption and portfolio decisions with stochastic hyperbolic discounting.  _Journal of Mathematical Economics_ , 52:70–80, 2014.
  * Chunxiang et al. [2016] A Chunxiang, Zhongfei Li, and Fan Wang.  Optimal investment strategy under time-inconsistent preferences and high-water mark contract.  _Operations Research Letters_ , 44(2):212–218, 2016.
  * Chen et al. [2018] Shumin Chen, Zhongfei Li, and Yan Zeng.  Optimal dividend strategy for a general diffusion process with time-inconsistent preferences and ruin penalty.  _Journal on Financial Mathematics_ , 9(1):274–314, 2018.
  * van Hasselt et al. [2019] Hado van Hasselt, John Quan, Matteo Hessel, Zhongwen Xu, Diana Borsa, and André Barreto.  General non-linear Bellman equations.  _arXiv preprint arXiv:1907.03687_ , 2019.
  * Ng et al. [2000] Andrew Y Ng, Stuart J Russell, et al.  Algorithms for inverse reinforcement learning.  In _International Conference on Machine Learning_ , volume 1, 2000\.
  * Abbeel and Ng [2004] Pieter Abbeel and Andrew Y Ng.  Apprenticeship learning via inverse reinforcement learning.  In _International Conference on Machine Learning_ , volume 5, 2004\.
  * Ziebart et al. [2008] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al.  Maximum entropy inverse reinforcement learning.  In _AAAI_ , volume 23, pages 1433–1438, 2008.
  * Garg et al. [2021] Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon.  IQ-Learn: Inverse soft-q learning for imitation.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 4028–4039. Curran Associates, Inc., 2021.
  * Mombaur et al. [2010] Katja Mombaur, Anh Truong, and Jean-Paul Laumond.  From human to humanoid locomotion – an inverse optimal control approach.  _Autonomous Robots_ , 28(3):369–383, 2010.
  * Rothkopf and Ballard [2013] Constantin A Rothkopf and Dana H Ballard.  Modular inverse reinforcement learning for visuomotor behavior.  _Biological cybernetics_ , 107(4):477–490, 2013\.
  * Muelling et al. [2014] Katharina Muelling, Abdeslam Boularias, Betty Mohler, Bernhard Schölkopf, and Jan Peters.  Learning strategies in table tennis using inverse reinforcement learning.  _Biological Cybernetics_ , 108(5):603–619, 2014\.
  * Schultheis et al. [2021] Matthias Schultheis, Dominik Straub, and Constantin A Rothkopf.  Inverse optimal control adapted to the noise characteristics of the human sensorimotor system.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 9429–9442. Curran Associates, Inc., 2021.
  * Golub et al. [2013] Matthew Golub, Steven Chase, and Byron Yu.  Learning an internal dynamics model from control demonstration.  In _International Conference on Machine Learning_ , pages 606–614. PMLR, 2013.
  * Ashwood et al. [2020] Zoe Ashwood, Nicholas A. Roy, Ji Hyun Bak, and Jonathan W Pillow.  Inferring learning rules from animal decision-making.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 3442–3453. Curran Associates, Inc., 2020.
  * Aalen et al. [2008] Odd Aalen, Ornulf Borgan, and Hakon Gjessing.  _Survival and event history analysis: a process point of view_.  Springer Science & Business Media, 2008.
  * Hanson [2007] Floyd B Hanson.  _Applied stochastic processes and control for jump-diffusions: modeling, analysis and computation_.  SIAM, 2007.
  * Särkkä and Solin [2019] Simo Särkkä and Arno Solin.  _Applied stochastic differential equations_ , volume 10.  Cambridge University Press, 2019.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.
  * Rabitz et al. [1983] Herschel Rabitz, Mark Kramer, and D Dacol.  Sensitivity analysis in chemical kinetics.  _Annual Review of Physical Chemistry_ , 34(1):419–461, 1983.
  * Kim et al. [2021] Jeongho Kim, Jaeuk Shin, and Insoon Yang.  Hamilton-Jacobi deep Q-learning for deterministic continuous-time systems with Lipschitz continuous controls.  _Journal of Machine Learning Research_ , 22:206–1, 2021\.
  * Merton [1976] Robert C Merton.  Option pricing when underlying stock returns are discontinuous.  _Journal of Financial Economics_ , 3(1-2):125–144, 1976.
  * K Namboodiri et al. [2014] Vijay Mohan K Namboodiri, Stefan Mihalas, Tanya Marton, and Marshall Gilmer Hussain Shuler.  A general theory of intertemporal decision-making and the perception of time.  _Frontiers in Behavioral Neuroscience_ , 8:61, 2014.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes] All contributions are provided in
Section 4 with applications to experiments in Section 5.

    2. (b)

Did you describe the limitations of your work? [Yes] See Section 6.

    3. (c)

Did you discuss any potential negative societal impacts of your work? [Yes]
See Section 6.

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [Yes] In
the intro text in Section 4.

    2. (b)

Did you include complete proofs of all theoretical results? [Yes] Complete
proofs are provided in Appendix A and Appendix B.

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
See supplementary material.

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes] see Appendix F

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [Yes] See error bars for learned Q functions in
Section 5.2

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix
F.

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes] Pytorch
was used and referenced.

    2. (b)

Did you mention the license of the assets? [Yes] see Appendix F.

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [Yes] The code is provided in the supplementary material.

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [N/A] We have not used data of other people.

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [N/A] We have not used data of
other people.

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]

Appendix
Reinforcement Learning
with Non-Exponential Discounting

##  Appendix A Convergence proof for the value function under hyperbolic
discounting

In the following, we assume a hyperbolic survival function as presented in Eq.
2, i.e.,

$\displaystyle S(t;\alpha,\beta)=\frac{1}{(\frac{t}{\beta}+1)^{\alpha}}.$

##### Part I

If the reward function $R(\mathbf{x},\mathbf{u},t)$ is bounded above for all
$(\mathbf{x},\mathbf{u},t)\in\mathcal{X}\times\mathcal{U}\times\operatorname{\mathbb{R}}^{+}_{0}$,
and $\alpha_{0}>1$, the value function defined in equation Eq. 6 is well-
defined.

We denote the supremum of the reward function $R(\mathbf{x},\mathbf{u},t)$ for
all
$(\mathbf{x},\mathbf{u},t)\in\mathcal{X}\times\mathcal{U}\times\operatorname{\mathbb{R}}^{+}_{0}$
by $r_{\textrm{sup}}$. We find

$\displaystyle V^{*}(\mathbf{x},t)$
$\displaystyle=\max_{\mathbf{u}_{[t,\infty)}}\operatorname{\mathbb{E}\,}\left[\int_{t}^{\infty}\frac{S(\tau)}{S(t)}\,R(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)\,\mathrm{d}\tau\>\Big{|}\>\mathbf{X}(t)=\mathbf{x}\right]$
$\displaystyle\leq\int_{t}^{\infty}\frac{S(\tau)}{S(t)}\,r_{\textrm{sup}}\,\mathrm{d}\tau$
$\displaystyle=\frac{r_{\textrm{sup}}}{S(t)}\int_{t}^{\infty}S(\tau)\,\mathrm{d}\tau$
$\displaystyle=\frac{r_{\textrm{sup}}}{S(t)}\int_{t}^{\infty}\frac{1}{\left(\frac{\tau}{\beta}+1\right)^{\alpha}}\,\mathrm{d}\tau$
$\displaystyle\leq\frac{r_{\textrm{sup}}}{S(t)}\int_{t}^{\infty}\frac{1}{\left(\frac{\tau}{\beta}\right)^{\alpha}}\,\mathrm{d}\tau$
$\displaystyle=\frac{\beta^{\alpha}\,r_{\textrm{sup}}}{S(t)}\int_{t}^{\infty}\frac{1}{\tau^{\alpha}}\,\mathrm{d}\tau$
$\displaystyle=\frac{\beta^{\alpha}\,r_{\textrm{sup}}}{S(t)}\left[\frac{\tau^{1-\alpha}}{1-\alpha}\right]_{\tau=t}^{\infty}$
$\displaystyle=\frac{\beta^{\alpha}\,r_{\textrm{sup}}}{S(t)\,(1-\alpha)}\left[\tau^{1-\alpha}\right]_{\tau=t}^{\infty},$

which is finite for $\alpha>1$.

##### Part II

If $R(\mathbf{x},\mathbf{u},t)$ is bounded below for all
$(\mathbf{x},\mathbf{u},t)\in\mathcal{X}\times\mathcal{U}\times\operatorname{\mathbb{R}}^{+}_{0}$,
and $\alpha_{0}\leq 1$, the value function defined in equation Eq. 6 is not
well-defined.

We denote the infimum of the reward function $R(\mathbf{x},\mathbf{u},t)$ for
all
$(\mathbf{x},\mathbf{u},t)\in\mathcal{X}\times\mathcal{U}\times\operatorname{\mathbb{R}}^{+}_{0}$
by $r_{\textrm{inf}}$. We find

$\displaystyle V^{*}(\mathbf{x},t)$
$\displaystyle=\max_{\mathbf{u}_{[t,\infty)}}\operatorname{\mathbb{E}\,}\left[\int_{t}^{\infty}\frac{S(\tau)}{S(t)}\,R(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)\,\mathrm{d}\tau\>\Big{|}\>\mathbf{X}(t)=\mathbf{x}\right]$
$\displaystyle\geq\int_{t}^{\infty}\frac{S(\tau)}{S(t)}\,r_{\textrm{inf}}\,\mathrm{d}\tau$
$\displaystyle=\frac{r_{\textrm{inf}}}{S(t)}\int_{t}^{\infty}S(\tau)\,\mathrm{d}\tau$
$\displaystyle=\frac{r_{\textrm{inf}}}{S(t)}\int_{t}^{\infty}\frac{1}{\left(\frac{\tau}{\beta}+1\right)^{\alpha}}\,\mathrm{d}\tau$
$\displaystyle=\frac{r_{\textrm{inf}}}{S(t)}\int_{t}^{\infty}\frac{1}{\left(\frac{\tau+\beta}{\beta}\right)^{\alpha}}\,\mathrm{d}\tau$
$\displaystyle=\frac{\beta^{\alpha}\,r_{\textrm{inf}}}{S(t)}\int_{t}^{\infty}\frac{1}{(\tau+\beta)^{\alpha}}\,\mathrm{d}\tau$
$\displaystyle=\frac{\beta^{\alpha}\,r_{\textrm{inf}}}{S(t)}\int_{t+\beta}^{\infty}\frac{1}{\tau^{\alpha}}\,\mathrm{d}\tau$
$\displaystyle=\frac{\beta^{\alpha}\,r_{\textrm{inf}}}{S(t)}\left[\frac{1}{\tau^{\alpha}}\right]_{\tau=t+\beta}^{\infty}$
$\displaystyle=\frac{\beta^{\alpha}\,r_{\textrm{inf}}}{S(t)}\left[\frac{\tau^{1-\alpha}}{1-\alpha}\right]_{\tau=t+\beta}^{\infty}$
$\displaystyle=\frac{\beta^{\alpha}\,r_{\textrm{inf}}}{S(t)\,(1-\alpha)}\left[\tau^{1-\alpha}\right]_{\tau=t+\beta}^{\infty},$

in which the integral diverges for $\alpha\leq 1$.

##  Appendix B Full derivation of the HJB equation

In this section, we provide a full derivation for the HJB equation introduced
in Section 4.2. We start with the value function defined in Eq. 6, i.e.,

$\displaystyle
V^{*}(\mathbf{x},t)=\max_{\mathbf{u}_{[t,\infty)}}\operatorname{\mathbb{E}\,}\left[\int_{t}^{\infty}\frac{S(\tau)}{S(t)}R(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)\,\mathrm{d}\tau\>\Big{|}\>\mathbf{X}(t)=\mathbf{x}\right].$

First, we split the integral into two terms and obtain

$\displaystyle V^{*}(\mathbf{x},t)=\max_{\mathbf{u}_{[t,t+\Delta
t]}}\operatorname{\mathbb{E}\,}$ $\displaystyle\left[\int_{t}^{t+\Delta
t}\frac{S(\tau)}{S(t)}R(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)\,\mathrm{d}\tau\right.$
$\displaystyle+\left.\int_{t+\Delta
t}^{\infty}\frac{S(\tau)}{S(t)}R(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)\,\mathrm{d}\tau\>\Big{|}\>\mathbf{X}(t)=\mathbf{x}\right].$

By identifying the second term as the value function of state $x(t+\Delta t)$
at time $t+\Delta t$, we obtain the recursive formulation

$\displaystyle V^{*}(\mathbf{x},t)=\max_{\mathbf{u}_{[t,t+\Delta
t]}}\operatorname{\mathbb{E}\,}$ $\displaystyle\left[\int_{t}^{t+\Delta
t}\frac{S(\tau)}{S(t)}R(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)\,\mathrm{d}\tau\right.$
$\displaystyle+\left.\frac{S(t+\Delta t)}{S(t)}V^{*}(\mathbf{X}(t+\Delta
t),t+\Delta t)\>\Big{|}\>\mathbf{X}(t)=\mathbf{x}\right].$

Consider a small $\Delta t$, then the first term evaluates to

$\displaystyle\int_{t}^{t+\Delta
t}\frac{S(\tau)}{S(t)}R(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)\,\mathrm{d}\tau=R(\mathbf{X}(t),\mathbf{u}(t),t)\cdot\Delta
t+o(\Delta t).$

For the second term, we apply a Taylor expansion and get

$\displaystyle V^{*}(\mathbf{X}(t+\Delta t)$ $\displaystyle,t+\Delta
t)=V^{*}(\mathbf{X}(t),t)+\int_{t}^{t+\Delta
t}\frac{\mathrm{d}}{\mathrm{d}\tau}V^{*}(\mathbf{X}(\tau),\tau)\,\mathrm{d}\tau+o(\Delta
t)$ $\displaystyle=V^{*}(\mathbf{X}(t),t)+\int_{t}^{t+\Delta
t}V_{\mathbf{x}}^{*}(\mathbf{X}(\tau),\tau)\,\mathrm{d}\mathbf{X}(\tau)+\int_{t}^{t+\Delta
t}V_{t}^{*}(\mathbf{X}(\tau),\tau)\,\mathrm{d}\tau+o(\Delta t).$

Here, the second term can be evaluated using Itô’s formula as

$\displaystyle\int_{t}^{t+\Delta
t}V_{\mathbf{x}}^{*}(\mathbf{X}(\tau),\tau)\,$
$\displaystyle\mathrm{d}\mathbf{X}(\tau)=\int_{t}^{t+\Delta
t}V_{\mathbf{x}}^{*}(\mathbf{X}(\tau),\tau)\,f(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)\,\mathrm{d}\tau$
$\displaystyle+\int_{t}^{t+\Delta
t}\frac{1}{2}\operatorname{\mathrm{tr}\,}{\left\\{V_{\mathbf{x}\mathbf{x}}^{*}(\mathbf{X}(\tau),\tau)\,G(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)\,G(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)^{T}\right\\}}\,\mathrm{d}\tau$
$\displaystyle+\int_{t}^{t+\Delta
t}V_{\mathbf{x}}^{*}(\mathbf{X}(\tau),\tau)\,G(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)\,\mathrm{d}\mathbf{W}(\tau)+o(\Delta
t).$

Plugging in these terms into the equation above and dividing both sides by
$\Delta t$ yields

$\displaystyle\frac{1-\frac{S(t+\Delta t)}{S(t)}}{\Delta t}$ $\displaystyle
V^{*}(\mathbf{X}(t),t)=\max_{\mathbf{u}_{[t,t+\Delta
t]}}\operatorname{\mathbb{E}\,}\left[\frac{1}{\Delta t}\int_{t}^{t+\Delta
t}\frac{S(\tau)}{S(t)}R(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)\,\mathrm{d}\tau\right.$
$\displaystyle+\frac{1}{\Delta t}\int_{t}^{t+\Delta
t}V_{\mathbf{x}}^{*}(\mathbf{X}(\tau),\tau)\,f(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)\,\mathrm{d}\tau+\int_{t}^{t+\Delta
t}V_{t}^{*}(\mathbf{X}(\tau),\tau)\,\mathrm{d}\tau$
$\displaystyle+\frac{1}{\Delta t}\int_{t}^{t+\Delta
t}\frac{1}{2}\operatorname{\mathrm{tr}\,}{\left\\{V_{\mathbf{x}\mathbf{x}}^{*}(\mathbf{X}(\tau),\tau)\,G(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)\,G(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)^{T}\right\\}}\,\mathrm{d}\tau$
$\displaystyle+\frac{1}{\Delta t}\left.\int_{t}^{t+\Delta
t}V_{\mathbf{x}}^{*}(\mathbf{X}(\tau),\tau)\,G(\mathbf{X}(\tau),\mathbf{u}(\tau),\tau)\,\mathrm{d}\mathbf{W}(\tau)+\frac{o(\Delta
t)}{\Delta t}\>\Big{|}\>\mathbf{X}(t)=\mathbf{x}\right].$

The factor on the l.h.s. in the limit $\Delta t\to 0$ can be recognized to be
the hazard rate (cf. Eq. 1),

$\displaystyle\lim_{\Delta t\to 0}\frac{1-\frac{S(t+\Delta t)}{S(t)}}{\Delta
t}=\lim_{\Delta t\to 0}\frac{1}{\Delta t}\frac{S(t)-S(t+\Delta t)}{\Delta
t}=\alpha(t).$

Taking the limit $\Delta t\to 0$ on both sides and calculating the expectation
w.r.t. $\mathbf{W}(t)$, we obtain the HJB equation

$\displaystyle\alpha(t)V^{*}(\mathbf{x},t)=\max_{\mathbf{u}}\ $
$\displaystyle\left[R(\mathbf{x},\mathbf{u},t)+V_{t}^{*}(\mathbf{x},t)+V_{\mathbf{x}}^{*}(\mathbf{x},t)\,f(\mathbf{x},\mathbf{u},t)\right.$
$\displaystyle\quad+\frac{1}{2}\left.\operatorname{\mathrm{tr}\,}{\left\\{V_{\mathbf{x}\mathbf{x}}^{*}(\mathbf{x},t)\,G(\mathbf{x},\mathbf{u},t)\,G(\mathbf{x},\mathbf{u},t)^{T}\right\\}}\right].$

##  Appendix C Bellman equation for discrete time

We consider the discrete-time setting, in which the objective is given as

$\displaystyle
J\left(\mathbf{u}_{0},\mathbf{u}_{1},\dots\right)=\operatorname{\mathbb{E}\,}\left[\displaystyle\sum_{\tau=0}^{\infty}S(\tau)\,R(\mathbf{X}_{\tau},\mathbf{u}_{\tau},\tau)\right].$

As in the continuous-time case, we can define the value function as

$\displaystyle V(\mathbf{x},t)$
$\displaystyle=\max_{\mathbf{u}_{t},\mathbf{u}_{t+1},\dots}\operatorname{\mathbb{E}\,}\left[\displaystyle\sum_{\tau=t}^{\infty}\frac{S(\tau)}{S(t)}\,R(\mathbf{X}_{\tau},\mathbf{u}_{\tau},\tau)\>\Big{|}\>\mathbf{X}_{t}=\mathbf{x}\right].$

By identifying the recursive definition of the value function and evaluating
terms, we obtain the Bellman equation

$\displaystyle V(\mathbf{x},t)$
$\displaystyle=\max_{\mathbf{u}}\left\\{R(\mathbf{x},\mathbf{u},t)+\lambda(t)\operatorname{\mathbb{E}\,}\left[V(\mathbf{X}_{t+1},t+1)\>|\>\mathbf{X}_{t}=\mathbf{x}\right]\right\\},$

with $\lambda(t)=S(t+1)/S(t)$ being the hazard probability at time t.

##  Appendix D Value function approximation and collocation method

In the collocation method in Algorithm 1, we need to sample random states
$\hat{\mathbf{x}}_{i}$ and time points $\hat{t}_{i}$ for minimizing
$\sum_{i}E(V^{\psi},\hat{\mathbf{x}}_{i},\hat{t}_{i})^{2}$. If we assume a
bounded state space $\mathcal{X}\in\operatorname{\mathbb{R}}$, we can sample
$\hat{\mathbf{x}}_{i}$ uniformly from this space. The time points
$\hat{t}_{i}\in\operatorname{\mathbb{R}}^{+}_{0}$ can be sampled from an
exponential distribution. To do so, we first draw
$\hat{y}_{i}\sim\operatorname{\mathrm{Uniform}}(0,1)$ and compute
$\hat{t}_{i}=-\log(1-\hat{y}_{i})/\lambda$. To feed a normalized value of time
into the network, we use $\hat{y}_{i}$ instead of $\hat{t}_{i}$ as input to
the network. We denote the value function network depending on $y$ by
$\tilde{V}(\mathbf{x},y)$. Given a specific time value $t$, we can compute its
representation via $y(t)=1-\exp(-\lambda t)$.

When computing the partial derivative $V_{t}$, we have to take this
reparametrization into account. By the chain rule, we find

$\displaystyle V_{t}(\mathbf{x},t)=\tilde{V}_{y}(\mathbf{x},t)\,y_{t}(t),$

for which we have with the chosen parametrization

$\displaystyle y_{t}(t)=\lambda\exp(-\lambda t).$

In general, there are multiple solutions to the HJB equation and the
encountered solution depends on the initialization of the function
approximator [29, 31]. In other work, this problem has been dealt with by
omitting stochastic terms in the first episodes of training or annealing the
discount factor [29, 31, 33]. We adopt the second approach and move from short
to far-sighted discounting to converge to the desired solution. For hyperbolic
discounting, we initially add an offset to $\alpha_{0}$, leading to a high
expected hazard rate. Over time, we decrease the offset to converge to the
desired solution.

##  Appendix E Experiments

### Investment problem

  * •

State space $\mathcal{X}=[0,1]\times[0,1]$, modeling account balance and
interest rate, i.e., $\mathbf{x}=[x_{b},x_{i}]$

  * •

Action space $\mathcal{U}=\\{\textit{spend},\textit{invest}\\}$

  * •

Dynamics model

$\displaystyle f(\mathbf{x},\mathbf{u})$
$\displaystyle{}=\begin{cases}[0,0]^{T}&\quad\text{if }\mathbf{u}\text{ is
{spend}}\\\ [0.1,0]^{T}&\quad\text{if }\mathbf{u}\text{ is
{invest}}\end{cases}$ $\displaystyle G(\mathbf{x},\mathbf{u})$
$\displaystyle{}=\begin{pmatrix}0&0\\\ 0&0.01\end{pmatrix}$

  * •

Reward function

$\displaystyle R(\mathbf{x},\mathbf{u})$
$\displaystyle=R^{\mathbf{x}}(\mathbf{x})+R^{\mathbf{u}}(\mathbf{u})$
$\displaystyle R^{\mathbf{x}}([x_{b},x_{i}])$ $\displaystyle=x_{b}\cdot x_{i}$
$\displaystyle R^{\mathbf{u}}(\mathbf{u})$
$\displaystyle=\begin{cases}0.1&\quad\text{if }\mathbf{u}\text{ is {spend}}\\\
0&\quad\text{if }\mathbf{u}\text{ is {invest}}\end{cases}$

  * •

Initial belief of hazard rate $\alpha_{0}=3,\beta_{0}=1$ (visualized in Fig.
3)

### Line problem

  * •

State space $\mathcal{X}=[-1,1]$

  * •

Action space $\mathcal{U}=\\{\textit{left},\textit{stay},\textit{right}\\}$

  * •

Dynamics model

$\displaystyle f(\mathbf{x},\mathbf{u})$
$\displaystyle{}=\begin{cases}-1&\quad\text{if }\mathbf{u}\text{ is {left}}\\\
0&\quad\text{if }\mathbf{u}\text{ is {stay}}\\\ 1&\quad\text{if
}\mathbf{u}\text{ is {right}}\end{cases}$ $\displaystyle
G(\mathbf{x},\mathbf{u})$ $\displaystyle{}=\begin{cases}0.05&\quad\text{if
}\mathbf{u}\in\\{\textit{left},\textit{right}\\}\\\ 0&\quad\text{if
}\mathbf{u}\text{ is {stay}}\end{cases}$

  * •

Reward function

$\displaystyle R(\mathbf{x},\mathbf{u})$
$\displaystyle=R^{\mathbf{x}}(\mathbf{x})+R^{\mathbf{u}}(\mathbf{u})$
$\displaystyle R^{\mathbf{x}}(\mathbf{x})$
$\displaystyle=\begin{cases}0.5&\quad\text{if }\mathbf{x}\geq 0.5\\\
\mathbf{x}&\quad\text{if }0\leq\mathbf{x}<0.5\\\ 0&\quad\text{if
}-0.95\leq\mathbf{x}<0\\\ -60\mathbf{x}-57&\quad\text{if }\mathbf{x}<0.95\\\
\end{cases}$ $\displaystyle R^{\mathbf{u}}(\mathbf{u})$
$\displaystyle=\begin{cases}0.1&\quad\text{if
}\mathbf{u}\in\\{\textit{left},\textit{right}\\}\\\ 0&\quad\text{if
}\mathbf{u}\text{ is {stay}}\end{cases}$

  * •

Initial belief of hazard rate $\alpha_{0}=5,\beta_{0}=1$ (visualized in Fig.
3)

##  Appendix F Hyperparameters, implementation, and computing resources

Throughout the experiments, we have used the following hyperparameters:

  * •

The neural networks are parametrized as

            layers = (nn.Linear(input_dim, layer_size),
                  nn.Sigmoid(),
                  nn.Linear(layer_size, layer_size),
                  nn.Sigmoid(),
                  nn.Linear(layer_size, output_dim))
        model = nn.Sequential(*layers)

  * •

For the neural network representing $V$, we used

            input_dim = x_dim + 1
        output_dim = 1

  * •

For the neural network representing $V_{\theta}$, we used

            input_dim = x_dim + 1
        output_dim = theta_dim

  * •

We set $\lambda=0.2$.

  * •

For the collocation method, we used 10.000 samples in each iteration and
125.000 episodes for the investment problem and 100.000 episodes for the line
problem. The initial offset of $\alpha_{0}$ was set to 50 and linearly
decreased to zero over 50.000 episodes.

  * •

We used Adam optimizer with learning rate 0.003.

  * •

For the runs with exponential discounting, the mean of the initial belief over
the hazard rate was taken for $\lambda$, i.e., 3 for the investment problem
and 5 for the line problem.

Figure 3: Hazard rates and Discounting functions. A Expected hazard rate for
the investment problem. For hyperbolic discounting, the expected risk of
termination is decreasing over time, while for exponential discounting, the
hazard rate is constant. B Expected discount function for the investment
problem in comparison to an exponential discount function. C Expected hazard
rate for the line problem for hyperbolic discounting in comparison with the
constant hazard rate when applying exponential discounting D Expected discount
function for the line problem in comparison to an exponential discount
function.

More information about Implementation and computing resources:

  * •

Methods were implemented in Python using the PyTorch framework [71], which has
been published under a BSD license.

  * •

Resources used: Intel® Xeon® Platinum 9242 Processor, using 8 cores per run.

  * •

Network training took ~50 min. for the investment problem and ~30 min. for the
line problem.

##  Appendix G Derivation of the hyperbolic discount function as uncertainty
over the constant hazard rate

We assume $P(T>t\>|\>\lambda)=\exp(-\lambda t)$ and a belief
$\lambda\sim\operatorname{\mathrm{Gamma}}(\lambda;\alpha,\beta)$. For the
expected survival function, we calculate

$\displaystyle S(t)$ $\displaystyle=\int_{\lambda}\exp{(-\lambda
t)}p\left(\lambda\right)\,\mathrm{d}\lambda$
$\displaystyle=\int_{\lambda}\exp{(-\lambda
t)}\frac{\beta^{\alpha}\lambda^{\alpha-1}\exp(-\beta\lambda)}{\Gamma(\alpha)}\,\mathrm{d}\lambda$
$\displaystyle=\int_{\lambda}\frac{\beta^{\alpha}\lambda^{\alpha-1}\exp(-(\beta+t)\lambda)}{\Gamma(\alpha)}\,\mathrm{d}\lambda$
$\displaystyle=\frac{\beta^{\alpha}}{(\beta+t)^{\alpha}}\int_{\lambda}\mathrm{Gamma}(\lambda;\alpha,\beta+t)\,\mathrm{d}\lambda$
$\displaystyle=\frac{1}{\left(\frac{t}{\beta}+1\right)^{\alpha}}.$

##  Appendix H Interpretation of the discount factor as transition to terminal
state

A Markov decision process (MDP) with discounting can be converted to an MDP
without discounting by adding an additional terminal state $\Upsilon$ [13].
From each state with a certain probability $\gamma$, one transitions to the
terminal state, and the remaining transition probabilities are renormalized.
At the terminal state there is no possibility to transition to any other state
and a reward of zero is given. In continuous time, the same formalization can
be applied, but we consider a rate instead at which one transitions to the
terminal state. Further, we assume in the following that the rate depends on
time and denote it by $\lambda(t)$. The probability to be in the terminal
states at time $\Upsilon$ is given by the cumulative distribution function
(CDF),

$\displaystyle P(\mathbf{X}(t)=\Upsilon)$ $\displaystyle=P(T<t).$

The probability of not having terminated yet is given by the complementary
cumulative distribution function (CCDF),

$\displaystyle P(\mathbf{X}(t)\neq\Upsilon)$
$\displaystyle=1-P(\mathbf{X}(t)=\Upsilon)$ $\displaystyle=P(T\geq t)$
$\displaystyle=S(t),$

which is equal to the discount function.

For a constant termination rate, one obtains the CDF and CCDF of the
exponential distribution, respectively:

$\displaystyle P(\mathbf{X}(t)=\Upsilon)$
$\displaystyle=\lambda\int_{0}^{t}\exp(-\lambda\tau)\,\mathrm{d}\tau$
$\displaystyle=1-\exp(-\lambda t)$ $\displaystyle
P(\mathbf{X}(t)\neq\Upsilon)$ $\displaystyle=1-P(\mathbf{X}(t)=\Upsilon)$
$\displaystyle=\exp(-\lambda t)$

  *[HJB]: Hamilton–Jacobi–Bellman