# Risk-Sensitive Multi-Agent Reinforcement Learning in Network Aggregative
Markov Games

Hafez Ghaemi  University of Tehran, School of ECE<EMAIL_ADDRESS>,
Hamed Kebriaei  University of Tehran, School of ECE<EMAIL_ADDRESS>,
Alireza Ramezani Moghaddam  University of Tehran, School of ECE
<EMAIL_ADDRESS>and  Majid Nili Ahamdabadi  University of Tehran, School
of ECE<EMAIL_ADDRESS>
###### Abstract.

Classical multi-agent reinforcement learning (MARL) assumes risk neutrality
and complete objectivity for agents. However, in settings where agents need to
consider or model human economic or social preferences, a notion of risk must
be incorporated into the RL optimization problem. This will be of greater
importance in MARL where other human or non-human agents are involved,
possibly with their own risk-sensitive policies. In this work, we consider
risk-sensitive and non-cooperative MARL with cumulative prospect theory (CPT),
a non-convex risk measure and a generalization of coherent measures of risk.
CPT is capable of explaining loss aversion in humans and their tendency to
overestimate/underestimate small/large probabilities. We propose a distributed
sampling-based actor-critic (AC) algorithm with CPT risk for network
aggregative Markov games (NAMGs), which we call Distributed Nested CPT-AC.
Under a set of assumptions, we prove the convergence of the algorithm to a
subjective notion of Markov perfect Nash equilibrium in NAMGs. The
experimental results show that subjective CPT policies obtained by our
algorithm can be different from the risk-neutral ones, and agents with a
higher loss aversion are more inclined to socially isolate themselves in an
NAMG.111Code available at https://github.com/hafezgh/risk-sensitive-marl-namg

Multi-Agent Reinforcement Learning, Actor-Critic, Aggregative Games, Risk-
Sensitivity, Cumulative Prospect Theory

††copyright: none††conference: Preprint; Risk-Sensitive Multi-Agent
Reinforcement Learning in Network Aggregative Markov Games; ††price:
††submissionid: 1100

##  1\. Introduction

Markov game (MG) is a common framework for studying multi-agent systems (MAS),
and it is the main theoretical framework for multi-agent reinforcement
learning (MARL) (Shapley, 1953; Littman, 1994). In classical MARL, each agent
is assumed to have a risk-neutral objective, i.e., it tries to maximize a
notion of expected return without taking into account subjective preferences
of itself or of the other agents in the MAS. Risk-neutral MARL in MGs has seen
great advances in recent years, especially in specific types of MGs, such as
zero-sum MGs (Sayin et al., 2021; Zhang et al., 2020; Alacaoglu et al., 2022;
Perolat et al., 2015; Qiu et al., 2021b) and Markov potential games (Ding et
al., 2022; Fox et al., 2022; Maheshwari et al., 2022; Mguni et al., 2021;
Leonardos et al., 2021). However, the risk-neutral RL objective often falls
short when representing agents with distinct subjective preferences, such as
internal cognitive biases of themselves or of other agents. Thus, to address
these preferences, agents integrate a risk measure into their RL objective,
ushering into the realm of risk-sensitive reinforcement learning (RSRL). In
general, the literature on risk-sensitive MARL is more sparse compared to
single-agent RSRL. The majority of the works that consider risk-sensitive
multi-agent MDPs are concerned not with an RL setting but either with
theoretically proving the existence of Markov perfect Nash equilibria, or
finding these equilibria using iterative algorithms given complete information
of the game in a centralized setting for MDPs with specific constraints (Ghosh
et al., 2023; Zhong et al., 2022; Ghosh et al., 2022; Pal and Pradhan, 2021;
Başar, 2021; Zhang et al., 2021).

Risk in RL can be categorized into two main types based on the risk-sensitive
objective, as delineated by Prashanth and Fu (Prashanth et al., 2022). The
first category, explicit risks, involves directly incorporating the risk
measure into the objective function. In contrast, implicit risks are
integrated by imposing a constraint on the RL stochastic optimization problem.
Notably, in practice, implicit risk-sensitive objectives are often transformed
into explicit objectives. This is achieved by formulating a Lagrangian and
computing its gradient to employ algorithms founded on policy gradient (PG)
methods (Prashanth et al., 2022). Within the spectrum of implicit risk
measures in RL and MDPs, notable examples include variance as risk ((Tamar et
al., 2012, 2013; Prashanth and Ghavamzadeh, 2016) in single-agent RSRL, and
(Reddy et al., 2019) in risk-sensitive MARL), and chance constraints ((Chow et
al., 2017) in single-agent RSRL). On the other hand, explicit risk measures
encompass entropic risk measures predicated on exponential return ((Borkar,
2001; Fei et al., 2021; Moharrami et al., 2022) in single-agent RSRL and
(Noorani and Baras, 2022; Soorki et al., 2021) in risk-sensitive MARL),
coherent risk measures, and cumulative prospect theory (CPT).

Coherent risk measures (Artzner et al., 1999; Delbaen, 2002), such as the
well-known conditional value at risk (CVaR), mean semi-deviation (Shapiro et
al., 2021), and spectral risk (Acerbi, 2002), are widely used in the fields of
economy and operations research. Their application has also been explored
within MDPs as dynamic risk measures. Osogami et al. (Osogami, 2012) showed
that risk-sensitive MDPs governed by Markov coherent risk measures can be
classified under the domain of robust MDPs. Subsequently, dynamic programming
methodologies have been suggested for this type of MDPs (Ruszczyński, 2010;
Cavus and Ruszczynski, 2014). Building on these works, PG-based techniques and
actor-critic (AC) algorithms have also been developed for RSRL with coherent
risk measures, as detailed in (La and Ghavamzadeh, 2013; Tamar et al., 2014;
Chow and Ghavamzadeh, 2014; Tamar et al., 2015) for single-agent RSRL, and in
(Zhu et al., 2022; Munir et al., 2021; Qiu et al., 2021a) for risk-sensitive
MARL.

#### CPT Background.

The concept of Prospect Theory (PT) emerged as an alternative model to
expected utility theory, providing a more accurate model of human decision-
making under uncertainty (Kahneman and Tversky, 1979). To enhance the
applicability of PT, Cumulative Prospect Theory (CPT) was subsequently
introduced (Tversky and Kahneman, 1992). Unlike PT, CPT applies weighting
functions to cumulative probabilities, addressing them separately for positive
and negative outcomes. By integrating these probability weighting functions
and a non-linear utility function, CPT successfully illustrates varying human
attitudes towards potential gains and losses against a subjective reference
point. Central to CPT is the idea that humans typically exhibit aversion to
losses, i.e., they generally take more risks when facing potential gains and
take fewer risks when confronted with potential losses. Additionally, CPT’s
framework elucidates human inclinations to overestimate small probabilities
and underestimate large ones during uncertain decision-making. When we
consider CPT in the context of either static or dynamic Markov risk measures,
it meets only two of the four requirements that define a coherent risk
measure. A risk measure, when applied to a random variable (r.v.) representing
potential outcomes, is deemed coherent if it has the following four
characteristics: convexity, monotonicity, translation invariance, and positive
homogeneity (Artzner et al., 1999). Among these, the CPT risk measure only
possesses monotonicity and positive homogeneity, and is neither translation
invariant nor convex. The non-coherent nature of CPT makes it more challenging
to work with mathematically. CPT can be seen as a generalization of coherent
risk measures, i.e., by appropriate selection of CPT probability weighting
functions, one can derive various coherent risk measure formulations (Lin and
Marcus, 2013; Jie et al., 2018).

#### Contributions.

In this work, we consider risk-sensitive MARL with CPT risk measure in network
aggregative Markov games (NAMGs), and propose a distributed actor-critic
algorithm to find risk-sensitive policies for each agent. We derive a policy
gradient theorem for CPT MARL based on a subjective steady-state distribution
of the MDP from each agent’s prespective, and provide a sampling-based
approach to estimate the value functions with asymptotic consistency. Since
CPT is a generalization of coherent risk measures, our PG theorem generalizes
the previous PG works for static and dynamic coherent risk measures (Chow and
Ghavamzadeh, 2014; Tamar et al., 2015). Under a set of assumptions, we prove
the convergence of our algorithm to a subjective and risk-sensitive notion of
Markov perfect Nash equilibrium (MPNE) which we show is unique given the
aforementioned assumptions. Experimentally, we also demonstrate that a higher
loss aversion can make agents more conservative and increase their tendency
for social isolation in an NAMG.

###### Remark 1.

(_Application_) A potential application of the proposed framework is
calculating CPT risk-sensitive policies of human agents in real-world
settings, such as driving scenarios or financial markets, that can be modeled
by NAMGs. Subsequently, these policies can serve dual purposes: guiding agents
towards strategies optimized for their individual preferences or facilitating
social or economic changes in the environment to steer agents in a direction
that aligns with desired outcomes.

##  2\. Related Works

In the context of Markov risk measures in MDPs, CPT is articulated through two
distinct formulations. The first formulation is the nested structure, wherein
the CPT operator is applied to the cumulative return after each step (action
taken) (Lin and Marcus, 2013; Lin, 2013; Lin et al., 2018). An important
advantage of this formulation is that it ensures the existence of a Bellman
optimality equation. Recently, Tian et al. (Tian et al., 2021) extended this
nested formulation to a multi-agent setting with agents that are characterized
by bounded rationality and operating under quantal level-$k$ strategies
(Wright and Leyton-Brown, 2017). Restricting their approach to deterministic
policies, they propose a centralized value iteration algorithm to determine
optimal risk-sensitive policies given a complete model of the environment and
the reward functions.

In the second formulation, the CPT operator is applied solely to the agent’s
final cumulative return at the end of every episode (Prashanth et al., 2016;
Jie et al., 2018). Contrary to the nested formulation, this formulation does
not have a Bellman equation. However, it can be approached from a stochastic
optimization perspective, allowing policy optimization through a gradient-
based method akin to PG techniques (Jie et al., 2018). This PG method has also
been implemented by considering neural networks for policy approximation
(Markowitz et al., 2021). It is important to emphasize that the absence of the
Bellman equation in this context necessitates policy optimization exclusively
through offline Monte Carlo sampling constrained by a finite time horizon.

To date, no cognitive research has been conducted to ascertain which of the
two CPT RSRL formulations best represents the dynamic risk behavior exhibited
by humans, whether in single-agent or multi-agent environments. Nonetheless,
the following can be said about the two formulations:

  * •

The nested formulation benefits from the presence of a Bellman equation,
enabling the use of online actor-critic algorithms and a recursively defined
value function. This advantage is absent in the non-nested formulation, where
it is only plausible to use PG techniques using offline Monte Carlo sampling.

  * •

In both formulations, due to the substitution of the expectation operator with
the non-linear CPT operator, it is possible for the optimal policy to exhibit
non-deterministic characteristics (Lin et al., 2018; Jie et al., 2018) even in
single-agent RL.

  * •

The non-nested formulation aligns well with finite-horizon episodic tasks
where the agent is rewarded at the end of each episode. However, its
applicability is limited when considering infinite-horizon tasks. Conversely,
the nested formulation and its Bellman equation are suitable for tasks where
the agent is rewarded at every timestep.

  * •

In scenarios without complete information of the model, the reward function,
or the policies of other agents, both formulations necessitate a strategy for
estimating the CPT value given that we have access to a simulator of the MDP
or a large enough experience dictionary (replay buffer). Such an estimation
technique tailored for the non-nested formulation has been introduced by Jie
et al. (Jie et al., 2018).

###### Remark 2.

(_Motivation_) Given the above considerations, in risk-sensitive MARL with
CPT, in a setting where the agents interact in an online infinite-horizon MDP
with limited information about other agents’ policies, the nested CPT
formulation is the viable option to adopt. Due to the possibility of non-
deterministic optimal policies for each agent in MARL, we opt for actor-critic
style algorithms using parameterized policies. Furthermore, we consider NAMGs
as our MARL framework due to three reasons. First, because they are inherently
suited to distributed algorithms. Second, given a set of assumptions, NAMG,
and its risk-sensitive version can be shown to have a unique Markov perfect
Nash equilibrium which our algorithm converges to. And third, because NAMGs
are a suitable framework to show the tangible effect of loss aversion in
human-like agents and on their tendency for social isolation and conservatism.

##  3\. Preliminaries

###  3.1. Cumulative Prospect Theory

Given a real-valued r.v. $X$ with distribution $\operatorname{\mathbb{P}}(X)$,
a reference point $x_{0}$, two monotonically non-decreasing weighting
functions,
$\omega^{+}:[0,1]\rightarrow[0,1],\omega^{-}:[0,1]\rightarrow[0,1]$, utility
functions
$u^{+}:\mathbb{R}^{+}\rightarrow\mathbb{R}^{+},u^{-}:\mathbb{R}^{-}\rightarrow\mathbb{R}^{+}$,
and given appropriate integrability assumptions, we can define the CPT value
using Choquet integrals as

(1)
$\begin{split}\operatorname{\mathbb{C}\mathbb{P}\mathbb{T}}_{\operatorname{\mathbb{P}}}[X]&:=\int^{\infty}_{0}\omega^{+}(\operatorname{\mathbb{P}}(u^{+}((X-x_{0})_{+})>x))dx-\\\
&\int^{\infty}_{0}\omega^{-}(\operatorname{\mathbb{P}}(u^{-}((X-x_{0})_{-})>x))dx.,\end{split}$

where we denote $(.)_{+}=max(0,.)$ and $(.)_{+}=-min(0,.)$. For a discrete
r.v., we can define the CPT value similarly as

(2a)
$\begin{split}\operatorname{\mathbb{C}\mathbb{P}\mathbb{T}}_{\operatorname{\mathbb{P}}}[X]&:=\sum^{n}_{i=0}\phi^{+}(\operatorname{\mathbb{P}}{P}(X=x_{i}))u^{+}(x_{i}-x^{0})\\\
&-\sum^{-1}_{-m}\phi^{-}(\operatorname{\mathbb{P}}(X=x_{i}))u^{-}(x_{i}-x^{0}),\end{split}$
(2b)
$\begin{split}\phi^{+}(\operatorname{\mathbb{P}}(X=x_{i}))&=\omega^{+}\left(\sum_{j=i}^{n}\operatorname{\mathbb{P}}(X=x_{j})\right)\\\
&-\omega^{+}\left(\sum_{j=i+1}^{n}\operatorname{\mathbb{P}}(X=x_{j}))\right),\end{split}$
(2c)
$\begin{split}\phi^{-}(\operatorname{\mathbb{P}}(X=x_{i}))&=\omega^{-}\left(\sum_{j=-m}^{i}\operatorname{\mathbb{P}}(X=x_{j})\right)\\\
&-\omega^{-}\left(\sum_{j=-m}^{i-1}\operatorname{\mathbb{P}}(X=x_{j})\right),\end{split}$

where $x_{0}$ serves as a reference point that separates gains and losses.
Without loss of generality, we assume $x_{0}=0$ throughout this paper.
Conventional representations of CPT weighting functions include
$\omega^{+}(p)=\frac{p^{\gamma}}{(p^{\gamma}+(1-p)^{\gamma})^{(1/\gamma)}}$
and
$\omega^{-}(p)=\frac{p^{\delta}}{(p^{\delta}+(1-p)^{\delta})^{(1/\delta)}}$
(Tversky and Kahneman, 1992), or $\omega^{+}(p)=\exp(-(-lnp)^{\gamma})$ and
$\omega^{-}(p)=\exp(-(-lnp)^{\delta})$ (Prelec, 1998). Note that by setting
$\delta$ and $\gamma$ equal to $1$, the definition of expected utility
$E_{\operatorname{\mathbb{P}}}[u(X)]$ is recovered which shows that CPT is a
generalization of expected utility theory. Furthermore, $u^{+}$ and $u^{-}$
are usually concave functions ($-u^{-}$ is convex) to reflect the higher
sensitivity of humans towards losses compared to gains (Kahneman and Tversky,
1979). As a result, the utility function can have analytical representations
$u^{+}(x)=x^{\alpha}$ if $x\geq 0$, and $u^{-}(x)=\lambda(-x)^{\beta}$ if
$x<0$. The parameters $\gamma,\delta,\alpha,\beta,$ and $\lambda$ are
subjective model parameters that can differ from person to person based on
their level of risk-aversion and individual characteristics. The conventional
representations of weighting and utility functions given a set of subjective
parameters are plotted in Figures 1 and 2.

Figure 1. Conventional CPT weighting functions;
$\omega^{+}(p)=\frac{p^{\gamma}}{(p^{\gamma}+(1-p)^{\gamma})^{(1/\gamma)}}$
and
$\omega^{-}(p)=\frac{p^{\delta}}{(p^{\delta}+(1-p)^{\delta})^{(1/\delta)}}$
with $\gamma=\delta=0.69$. Figure 2. Conventional CPT utility functions; The
plot shows $u^{+}(x)=x^{\alpha}$ for $x\geq 0$, and
$-u^{-}(x)=-\lambda(-x)^{\beta})$ for $x<0$, with $\alpha=\beta=0.65$ and
$\lambda=2.6$.

###  3.2. Network Aggregative Markov Games

Throughout this paper, we assume that agents are interacting in an ergodic
network aggregative Markov game with a discounted infinite-horizon criterion.
An NAMG with $N$ players is an MG denoted by
$M=(S,N,A,R,P,G,\gamma,p_{s_{0}})$, where $S$ is the state space,
$A=A_{1}\times...\times A_{N}$ is the joint action space; $R:S\times A\times
S\rightarrow\mathbb{R}^{N}$ is a joint reward function bounded in
$[-R_{max},R_{max}]$ where $R_{max}>0$; $P(.|s,a)$ is the MDP transition
probability distribution; $\mathcal{G}(\mathcal{N},\mathcal{E})$ is a graph
with edge set $\mathcal{E}$ on which each agent interacts with its neighbors;
$\gamma$ is the MDP’s discount factor; and $p_{s_{0}}$ is the initial state
distribution. In NAMG, for each agent $n$, the reward function is a function
of its own action and an aggregative function of other agents’ actions,

(3) $R^{i}(s,a^{i},a^{-i})=R^{i}(s,a^{i},\sigma^{i}(a^{-i})),$

where we have,

(4) $\sigma^{i}(a^{-i})=\sum_{j\in\mathcal{N}\backslash i}\omega_{ij}a^{j},$

where $\omega$ are the edge weights of the communication graph $\mathcal{G}$,
with $w_{ij}$ denoting the weight of the edge from $j$ to $i$. Therefore,
given the graph, and by observing its neighbors’ actions, agent $i$ is able to
calculate $\sigma^{i}(a^{-i})$. Figure 3 shows a schematic of an NAMG.

Figure 3. A network aggregative Markov game

Previously, in various domains, such as resource allocation (Deng, 2019),
social networks (Zhu et al., 2013), electrical microgrids (Tan et al., 2021),
and power systems (Deng, 2021), either single-state network aggregative games
(NAGs), or dynamic network aggregative games have been studied in risk-neutral
setting. Furthermore, most of the theoretical works in this domain have
focused on studying convergence to the Markov perfect Nash or Stackelberg
equilibrium in single-state or dynamic NAGs with quadratic cost/reward
functions that ensure the uniqueness of the equilibrium (Parise et al., 2020;
Cenedese et al., 2020; Shokri and Kebriaei, 2020; Saffar et al., 2017). In
this paper, for the first time, we consider risk-sensitive NAMGs.

###  3.3. CPT Risk-Sensitive MARL Objective in NAMGs

Using the nested formulation, the objective of the risk-sensitive agent $i$ in
an NAMG $\max_{\pi^{i}}J^{\pi^{i},\pi^{-i}}$ will be equivalent to

(5) $\displaystyle\max_{\pi^{i}}V^{i}_{\pi}(s_{0})$
$\displaystyle=\max_{\pi^{i}}\operatorname{\mathbb{C}\mathbb{P}\mathbb{T}}_{\pi(s_{0},.)\times\operatorname{\mathbb{P}}(.|s_{0},a_{0})}\big{[}R^{i}(s_{\tau},a_{\tau})+...$
$\displaystyle+\gamma^{\tau}\operatorname{\mathbb{C}\mathbb{P}\mathbb{T}}_{\pi(s_{\tau},.)\times\operatorname{\mathbb{P}}(.|s_{\tau},a_{\tau})}\big{[}R^{i}(s_{\tau},a_{\tau})+...\big{]}\big{]}$
$\displaystyle=\max_{\pi^{i}}\operatorname{\mathbb{C}\mathbb{P}\mathbb{T}}_{\pi(s_{0},.)\times\operatorname{\mathbb{P}}(.|s_{0},a_{0})}\big{[}R^{i}(s_{0},a_{0})+\gamma
V^{i}_{\pi}(s_{1})\big{]},$

where $\pi(s_{t},.)=\pi^{i}(s_{t},.)\times\pi^{-i}(s_{t},.)$ and
$a_{t}=(a^{i}_{t},a^{-i}_{t})$, respectively. Using the properties of NAMGs,
and considering $\operatorname{\mathbb{P}}(\sigma^{-i}_{0}|s_{0})$ as the
probability that $\sigma^{-i}_{0}$ occurs at state $s_{0}$ for agent $i$, we
can rewrite the objective as

(6)
$\small\max_{\pi^{i}}V^{i}_{\pi}(s_{0})=\max_{\pi^{i}}\operatorname{\mathbb{C}\mathbb{P}\mathbb{T}}_{\pi^{i}(a^{i}_{0}|s_{0})\times\operatorname{\mathbb{P}}(\sigma^{-i}_{0}|s_{0})\times\operatorname{\mathbb{P}}(s_{1}|s_{0},a_{0})}\big{[}R^{i}(s_{0},a_{0})+\gamma
V^{i}_{\pi}(s_{1})\big{]}.$

Therefore, by observing the actions of neighboring agents and calculating the
aggregative term $\sigma^{-i}$, agent $i$ can treat
$\operatorname{\mathbb{P}}(\sigma^{-i}|s)$ as a probability distribution
similar to the transition probabilities for each state.

##  4\. Distributed Nested CPT Policy Gradient

In this section, we derive a gradient expression for the Markov dynamic CPT
risk measure in NAMGs, represented by the gradient of the initial state’s
value function in an ergodic CPT risk-sensitive NAMG, $\nabla
V^{i}_{\pi_{\theta}}(s_{0})$. Before presenting the PG theorem, we state the
following assumption,

###### Assumption 1.

The weight functions $w^{\pm}$ are double differentiable, and the derivatives
$w^{{}^{\prime}}_{\pm}$ are Lipschitz continuous with common constant $L$.
Furthermore, the utility functions $u^{\pm}$ are differentiable (denoted by
$u^{{}^{\prime}}_{\pm}$) for all agents.

The above assumption may seem strict at first. However, conventional forms of
the CPT utility functions, specifically $u^{+}(x)=x^{\alpha}$ and
$-u^{-}(x)=-\lambda(-x)^{\beta}$, along with the weighting functions
$\omega^{+}(p)=\frac{p^{\gamma}}{(p^{\gamma}+(1-p)^{\gamma})^{(1/\gamma)}}$
and
$\omega^{-}(p)=\frac{p^{\delta}}{(p^{\delta}+(1-p)^{\delta})^{(1/\delta)}}$,
depicted in Figures 1 and 2, satisfy this assumption.

###### Theorem 1.

(Nested CPT Policy Gradient)

Given Assumption 1, the gradient of the CPT return for agent $i$,
$V^{i}_{\pi_{\theta}}(s_{0})$, with respect to the policy parameter
$\theta^{i}$ is

(7) $\begin{split}\nabla
V^{i}_{\pi_{\theta}}(s_{0})\propto&\operatorname{\mathbb{E}}_{\mu_{cpt}^{i}(s)}\bigg{[}\sum_{a,s^{\prime}}\frac{\partial\phi}{\partial(\pi^{i}_{\theta}(a^{i}|s)\operatorname{\mathbb{P}}(\sigma^{-i}|s)\operatorname{\mathbb{P}}(s^{\prime}|s,a))}\\\
&\operatorname{\mathbb{P}}(\sigma^{-i}|s)\operatorname{\mathbb{P}}(s^{\prime}|s,a)(\nabla\pi_{\theta^{i}}(a^{i}|s))u(R^{i}(s,a^{i},\sigma^{-i},s^{\prime})+\gamma
V^{i}_{\pi_{\theta}}(s^{\prime}))\bigg{]},\end{split}$

where, $\phi$ and $u$ represent the CPT cumulative weighting and utility
functions of the agent from (2) (superscript $i$ is dropped). The distribution
$\mu_{cpt}^{i}$ is a _subjective_ steady-state probability distribution of the
MDP in which $\mu_{cpt}^{i}(s)=\frac{\eta_{cpt}^{i}(s)}{\sum_{s\in
S}\eta_{cpt}^{i}(s)}$, where
$\eta_{\operatorname{\mathbb{C}\mathbb{P}\mathbb{T}}}^{i}(s)$ is a subjective
measure of time spent in each state and can be obtained by solving the
following system of linear equations,

(8) $\eta_{cpt}^{i}(s)=p_{0}(s)+\sum_{\bar{s}\in
S}\eta_{cpt}^{i}(\bar{s})\sum_{a\in
A}\phi(\pi(a^{i}|\bar{s})\operatorname{\mathbb{P}}(\sigma^{-i}|\bar{s})\operatorname{\mathbb{P}}(s|\bar{s},a))\frac{\partial
u}{\partial V^{i}_{\pi_{\theta}}}(s),$

where $p_{0}(s)$ denotes the probability that the Markov chain starts in state
$s$, and $u$ and $\phi$ are the utility cumulative weighting functions of
agent $i$ from (2) ($u^{\pm}$ and $\phi^{\pm}$ are chosen according to the
sign of $R^{i}(\bar{s},a,s)+\gamma V^{i}_{\pi_{\theta}}(s)$).

###### Proof.

Considering agent $i$, we drop the subscript $i$ and denote
$R^{i}(s_{t},a_{t},s_{t+1})$ as $R_{t}$ and $V^{i}_{\pi_{\theta}}(s_{t})$ as
$V_{\pi_{\theta}}(s_{t})$. Furthermore, $\pi_{\theta}(s,a)$ used below, where
$a$ is equivalent to $(a^{i},a^{-i})$, is the more general case of joint
policies in Markov games, which encompasses
$\pi_{\theta}(a^{i}|s)\operatorname{\mathbb{P}}(\sigma^{-i}|s)$ in an NAMG.
The gradient of the CPT risk-sensitive return considering its recursive
definition can be written as

(9) $\small\begin{split}\nabla V_{\pi_{\theta}}(s_{0})&=\\\
&\nabla\left[\sum_{a_{0},s_{1}}\phi(\pi(s_{0},a_{0})\operatorname{\mathbb{P}}(s_{1}|s_{0},a_{0}))u(R_{0}+V_{\pi_{\theta}}(s_{1}))\right]\\\
&=\sum_{a_{0},s_{1}}\bigg{[}\nabla\phi(\pi(s_{0},a_{0})\operatorname{\mathbb{P}}(s_{1}|s_{0},a_{0}))u(R_{0}+V_{\pi_{\theta}}(s_{1}))\\\
&+\phi(\operatorname{\mathbb{P}}(s_{1}|s_{0},a_{0}))\nabla
u(R_{0}+V_{\pi_{\theta}}(s_{1}))\bigg{]}\\\
&=\sum_{a_{0},s_{1}}\Bigg{[}\nabla\phi(\pi(s_{0},a_{0})\operatorname{\mathbb{P}}(s_{1}|s_{0},a_{0}))u(R_{0}+V_{\pi_{\theta}}(s_{1}))\\\
&+\phi(\pi(s_{0},a_{0})\operatorname{\mathbb{P}}(s_{1}|s_{0},a_{0}))\frac{\partial
u}{\partial V_{\pi_{\theta}}(s_{1})}\\\
&\nabla\bigg{[}\sum_{a_{1},s_{2}}\phi(\pi(s_{1},a_{1})\operatorname{\mathbb{P}}(s_{2}|s_{1},a_{1}))u(R_{1}+V_{\pi_{\theta}}(s_{2}))\bigg{]}\Bigg{]}\\\
&=\sum_{a_{0},s_{1}}\Bigg{[}\nabla\phi(\pi(s_{0},a_{0})\operatorname{\mathbb{P}}(s_{1}|s_{0},a_{0}))u(R_{0}+V_{\pi_{\theta}}(s_{1}))\\\
&+\phi(\pi(s_{0},a_{0})\operatorname{\mathbb{P}}(s_{1}|s_{0},a_{0}))\frac{\partial
u}{\partial V_{\pi_{\theta}}(s_{1})}\\\
&\sum_{a_{1},s_{2}}\bigg{[}\nabla\phi(\pi(s_{1},a_{1})\operatorname{\mathbb{P}}(s_{2}|s_{1},a_{1}))u(R_{1}+V_{\pi_{\theta}}(s_{2}))\\\
&+\phi(\pi(s_{1},a_{1})\operatorname{\mathbb{P}}(s_{2}|s_{1},a_{1}))\nabla
u(R_{1}+V_{\pi_{\theta}}(s_{2}))\bigg{]}\Bigg{]}.\end{split}$

We define

(10) $\small DPr(s_{0}\rightarrow
s_{1},k=1,\pi_{\theta}):=\sum_{a_{0}}\phi(\pi(s_{0},a_{0})\operatorname{\mathbb{P}}(s_{1}|s_{0},a_{0}))\frac{\partial
u}{\partial V_{\pi_{\theta}}(s_{1})}$

as the _subjective (distorted)_ visitation probability of $s_{1}$ right after
$s_{0}$ following policy $\pi_{\theta}$. Note that since $u$ is a non-
decreasing function with positive derivatives everywhere and $\phi$ is a
function that maps $[0,1]$ to $[0,1]$, this term is always positive. By
defining $DPr(s_{0}\rightarrow s_{0},0,\pi_{\theta}):=1$, by recursion, we can
write the subjective probability of visiting state $s_{k+1}$ after $k+1$
steps, starting from $s_{0}$ and following policy $\pi_{\theta}$ as

(11) $\small\begin{split}&DPr(s_{0}\rightarrow s_{k+1},k+1,\pi_{\theta})=\\\
&\sum_{s_{k}}DPr(s_{0}\rightarrow s_{k},k,\pi_{\theta})DPr(s_{k}\rightarrow
s_{k+1},1,\pi_{\theta})\end{split}$

Therefore, after repeated unrolling, we can write (9) as

(12) $\begin{split}&\nabla
V_{\pi_{\theta}}(s_{0})=\sum_{s}\Bigg{(}\bigg{(}\sum_{k=0}^{\infty}DPr(s_{0}\rightarrow
s,k,\pi_{\theta})\bigg{)}\\\
&\sum_{a,s^{\prime}}\nabla\phi(\pi(s,a)\operatorname{\mathbb{P}}(s^{\prime}|s,a))u(R(s,a,s^{\prime})+V_{\pi_{\theta}}(s^{\prime}))\Bigg{)}.\end{split}$

Similar to a risk-neutral MDP, given $\pi_{\theta}$ and the state value
function corresponding to this policy, the function $DPr$ is an inherent
property of the CPT risk-sensitive MDP (this function can be compared with the
function $Pr$ in the proof of risk-neutral policy gradient theorem in (Sutton
and Barto, 2018), Section 13.2). Therefore, we let
$\eta_{cpt}(s):=\sum_{k=0}^{\infty}DPr(s_{0}\rightarrow s,k,\pi_{\theta})$,
which can be considered a _subjective (perceived)_ measure of time that the
CPT risk-sensitive agent spends in state $s$ when following policy
$\pi_{\theta}$ and starting from state $s_{0}$. In a similar fashion as in
risk-neutral ergodic MDPs (see (Sutton and Barto, 2018), Section 9.2),
$\eta(s)$ can be calculated by solving the following system of linear
equations,

(13) $\eta_{cpt}(s)=p_{0}(s)+\sum_{\bar{s}\in S}\eta_{cpt}(\bar{s})\sum_{a\in
A}\phi(\pi(\bar{s},a)\operatorname{\mathbb{P}}(s|\bar{s},a))\frac{\partial
u}{\partial V_{\pi_{\theta}}}(s),$

where $p_{0}$ is the probability distribution of the starting state.
Therefore, we can write (9) as

(14) $\begin{split}\nabla
V_{\pi_{\theta}}(s_{0})=\sum_{s}\eta_{cpt}(s)\sum_{a,s^{\prime}}&\nabla\phi(\pi(s,a)\operatorname{\mathbb{P}}(s^{\prime}|s,a))\\\
&u(R(s,a,s^{\prime})+V_{\pi_{\theta}}(s^{\prime})).\end{split}$

As $\eta_{cpt}(s)$ is positive for all $s$, we can define
$\mu_{cpt}(s)=\frac{\eta_{cpt}(s)}{\sum_{s\in S}\eta_{cpt}(s)}$ as the
_subjective_ limiting (steady-state) distribution of the CPT risk-sensitive
MDP, and therefore, we have

(15) $\begin{split}\nabla
V_{\pi_{\theta}}(s_{0})\propto\operatorname{\mathbb{E}}_{\mu_{cpt}(s)}\bigg{[}\sum_{a,s^{\prime}}&\nabla\phi(\pi(s,a)\operatorname{\mathbb{P}}(s^{\prime}|s,a))\\\
&u(R(s,a,s^{\prime})+V_{\pi_{\theta}}(s^{\prime}))\bigg{]}.\end{split}$

It is interesting to compare (15) with the similar expression in risk-neutral
policy gradient theorem,
$\operatorname{\mathbb{E}}_{\mu(s)}\bigg{[}\sum_{a}\nabla\pi(s,a)Q(s,a)\bigg{]}$.
Due to the non-linear CPT operator (compared to the linear expectation
operator), the policy is entangled with the transition probabilities inside
the gradient of the cumulative weighting function, and therefore, in the risk-
sensitive case, it is not possible to define a stand-alone Q-function as a
function of state and action to measure the quality of an action in a given
state. As noted by Lin (Lin, 2013), this complication has the consequence that
the optimal risk-sensitive policy even in the single-agent setting can be
stochastic. To further expand the above expression, we can use the chain rule
of calculus and write,

(16) $\begin{split}\nabla
V_{\pi_{\theta}}(s_{0})\propto&\operatorname{\mathbb{E}}_{\mu_{cpt}(s)}\bigg{[}\sum_{a,s^{\prime}}\frac{\partial\phi}{\partial(\pi_{\theta}(s,a)\operatorname{\mathbb{P}}(s^{\prime}|s,a))}\\\
&\operatorname{\mathbb{P}}(s^{\prime}|s,a)(\nabla\pi_{\theta}(s,a))u(R(s,a,s^{\prime})+\gamma
V_{\pi_{\theta}}(s^{\prime}))\bigg{]}.\end{split}$

This is the general case of PG in CPT risk-sensitive MARL. Given the
aggregative term $\sigma^{-i}$ in NAMGs, we can rewrite this equation as (7).

∎

We now provide an algorithm to estimate the above gradient using samples from
a simulator of the environment or a large enough experience dictionary. This
approximation scheme which is later used to also estimate the value function
is Algorithm 1 is proposed by Jie et al. (Jie et al., 2018) to estimate the
CPT value of an r.v., $X$, using samples from its distribution. The following
Assumption (A2 in (Jie et al., 2018)) is needed to guarantee the asymptotic
consistency of this estimation algorithm.

###### Assumption 2.

The utility functions $u^{+}$ and $u^{-}$ are continuous and non-decreasing on
their support $\mathbb{R^{+}}$ and $\mathbb{R^{-}}$, respectively.

The above assumption also holds for conventional forms of weighting and
utility functions in Figures 1 and 2 (Jie et al., 2018). Given Assumptions 1
and 2, Proposition 4 in (Jie et al., 2018) is verified and for a given r.v.
$X$, Algorithm 1 is guaranteed to have asymptotic consistency, i.e., it
converges to $\operatorname{\mathbb{C}\mathbb{P}\mathbb{T}}[X]$ asymptotically
as the number of samples, $n$, approaches infinity.

Algorithm 1 CPT Value Estimation

1: Require: Samples $X_{1}$,…, $X_{n}$ from the distribution of r.v. $X$,
sorted in ascending order.

2: Let
$\begin{split}\hat{\rho}_{cpt}^{+}&:=\sum^{n}_{i=1}u^{+}(X_{i})\bigg{(}\omega^{+}\bigg{(}\frac{n+1-i}{n}\bigg{)}-\omega^{+}\bigg{(}\frac{n-i}{n}\bigg{)}\bigg{)},\\\
\hat{\rho}_{cpt}^{-}&:=\sum^{n}_{i=1}u^{-}(X_{i})\bigg{(}\omega^{-}\bigg{(}\frac{i}{n}\bigg{)}-\omega^{-}\bigg{(}\frac{i-1}{n}\bigg{)}\bigg{)}.\end{split}$

3: Return $\hat{\rho}_{cpt}=\hat{\rho}_{cpt}^{+}-\hat{\rho}_{cpt}^{-}$.

#### Gradient estimation.

To have a estimate of the gradient in (7), we need estimates of CPT values
corresponding to
$\phi(\pi(a^{i}|\bar{s})\operatorname{\mathbb{P}}(\sigma^{-i}|\bar{s})\operatorname{\mathbb{P}}(s|\bar{s},a))$
and
$\frac{\partial\phi}{\partial(\pi_{\theta}(a^{i}|s)\operatorname{\mathbb{P}}(\sigma^{-i}|\bar{s})\operatorname{\mathbb{P}}(s^{\prime}|s,a))}$
(which we denote by
$\phi^{\prime}\left(\pi_{\theta}(a^{i}|s)\operatorname{\mathbb{P}}(\sigma^{-i}|\bar{s})\operatorname{\mathbb{P}}(s^{\prime}|s,a)\right)$).
Note that Assumption 1 states that $\omega^{\prime}_{\pm}$ are Lipschitz and
therefore, they can be used as independent CPT weighting functions with
corresponding cumulative weighting functions $\phi^{\prime}_{\pm}$. Given the
transition probabilities and repeated distributed sampling of rewards and
transitions by agents from the environment or the experience dictionaries, the
term in brackets corresponding to each state can be estimated using Algorithm
(1). Furthermore, using these samples and solving a linear system of equations
resulting from (13), the subjective steady-state distribution $\mu_{cpt}(s)$
can be found. We note that this estimation algorithm is model-based and
requires transition probabilities, however, it does not assume any knowledge
of the reward function or the policies of other agents, and is therefore
privacy-preserving. Having a policy gradient theorem and a corresponding
gradient approximation scheme, we can now develop our distributed actor-critic
algorithm.

##  5\. Distributed Nested CPT Actor-Critic

Algorithm (2) lays out the pseudocode for Distributed Nested CPT Actor-Critic.
As can be seen, the critic’s value function is estimated using the sampling
strategy in Algorithm 1, and we use the samples from the simulator for
bootstrapping (by adding them to the experience dictionary for later use).
Although sampling from the simulator for gradient and value function
approximation can be computationally intensive, it can become less so as we
build the experience dictionary and do away with the simulator. We now prove
the asymptotic convergence of the proposed algorithm.

Algorithm 2 Distributed Nested CPT Actor-Critic

1: Inputs: _shared among agents:_ Initial state $s^{0}$, number of samples
used for CPT estimation ($n_{max}$), learning rate sequences
($\\{\alpha_{cr,t}\\}_{t\geq 0},\\{\alpha_{ac,t}\\}_{t\geq 0}$), and
transition probabilities. _Local variables for agent $n$:_ Initial
$V_{\pi_{\theta_{0}}}^{n}$, initial policy parameters ($\theta^{n}_{0}$), and
an empty experience dictionary $ExpDict^{n}$.

2: For each agent $n$, do:

3: Sample action $a^{n}_{0}$ from policy $\pi_{\theta^{n}_{0}}(.|s_{0})$.

4: $t\leftarrow 0$.

5: Repeat

6: Sample $a^{n}_{t}$ from $\pi_{\theta^{n}_{t}}(.|s_{t})$.

7: Execute $a^{n}_{t}$ and observe $r^{n}_{t}$, $s_{t+1}$, and
$\sigma^{-n}_{t}$.

8: Push $(r_{t},s_{t+1},\sigma^{-n}_{t})$ to
$ExpDict^{n}(s_{t},a^{n}_{t},\sigma^{-n}_{t})$.

9: Critic value estimation:

10: Create empty array $X$ of size $n_{max}$.

11: for each $i=1,2,...,n_{max}$, do

12: Sample $\hat{a}^{n}_{t}$ from $\pi_{\theta^{n}_{t}}(.|s_{t})$ and
construct $\hat{\sigma}^{-n}_{t}$ by observing neighbors.

13: Sample $(\hat{r}^{n}_{t},\hat{s}_{t+1})$ from
$ExpDict(s_{t},\hat{a}^{n}_{t},\hat{\sigma}^{-n}_{t})$ if it is large enough,
and otherwise from a simulator of the environment.

14: Let $X_{i}=\hat{r}^{n}_{t}+\gamma V_{\pi_{\theta}}^{n}(\hat{s}_{t+1})$.

15: If the sample came from a simulator, push
$(\hat{r}^{n}_{t},\hat{s}_{t+1})$ to
$ExpDict(s_{t},\hat{a}^{n}_{t},\hat{\sigma}^{-n}_{t})$ for later use.

16: end for

17: Estimate $\hat{V}_{\pi_{\theta_{t}}}^{n}(s_{t})$ using Algorithm 1.

18: Critic step:

19: Calculate the TD-error:
$\delta_{t}:=\hat{V}_{\pi_{\theta_{t}}}^{n}(s_{t})-V_{\pi_{\theta_{t}}}^{n}(s_{t}).$
$V_{\pi_{\theta_{t}}}^{n}(s_{t})\leftarrow
V_{\pi_{\theta_{t}}}^{n}(s_{t})+\alpha_{cr,t}\delta_{t}.$

20: Actor step:

21: Compute $\nabla V^{n}_{\pi_{\theta_{t}}}(s_{0})$ using the gradient
estimation scheme described in Section 4.
$\theta_{t+1}^{n}:=\theta_{t}^{n}+\alpha_{ac,t}\nabla
V^{n}_{\pi_{\theta_{t}}}(s_{0}).$

22: $t\leftarrow t+1.$

23: Until convergence

###  5.1. Convergence of the Critic

In order to calculate the state value function corresponding to the current
policy, we define the following $TD(0)$ CPT operator (note that the agent’s
superscript $n$ has been dropped),

(17)
$T_{cpt}V_{\pi_{\theta}}(s)=\operatorname{\mathbb{C}\mathbb{P}\mathbb{T}}_{\pi_{\theta}(.|s)\times\operatorname{\mathbb{P}}(.|s,a)}\left[R(s,a,s^{\prime})+\gamma
V_{\pi_{\theta}}(s^{\prime})\right]$

The following assumption is needed to ensure that the operator in (17) is a
sup-norm contraction.

###### Assumption 3.

The utility functions $u^{+}$ and $u^{-}$ are invertible (denoted by
$u_{+}^{-1}$ and $u^{-1}_{-}$) and differentiable (denoted by
$u^{{}^{\prime}}_{+}$ and $u^{{}^{\prime}}_{-}$), and we have
$u^{+}(0)=u^{-}(0)=0$. Further, there exists $\beta\in(0,1)$ such that
$\int_{0}^{\gamma
c}\omega^{+}(\operatorname{\mathbb{P}}(X<x))u^{{}^{\prime}}_{+}(\gamma
c-x)dx+\int_{0}^{\gamma
c}\omega^{-}(\operatorname{\mathbb{P}}(X>x))u^{{}^{\prime}}_{-}(x)dx\leq\beta
c$ holds for any $c>0$ and any non-negative real-valued r.v. $X$, where
$\gamma$ is the discount factor of the MDP.

Similar to Assumptions 1 and 2, the above assumption can also be verified to
hold for typical analytical forms of $\omega^{\pm}$ and $u^{\pm}$ in Figures 2
and 1 as shown by Lin et al. (Lin et al., 2018). Under this assumption, based
on Theorem 6 in (Lin et al., 2018), the $TD$ operator (17) is a sup-norm
contraction on a Banach space defined over the MDP’s state space that includes
all possible state value functions $V_{\pi_{\theta}}$. Therefore, for every
$V_{\pi_{\theta}},\bar{V}_{\pi_{\theta}}\in B(S)$, there exists
$\alpha\in(0,1)$ such that

(18)
$\|T_{cpt}V_{\pi_{\theta}}-T_{cpt}\bar{V}_{\pi_{\theta}}\|_{\infty}\leq\alpha\|V_{\pi_{\theta}}-\bar{V}_{\pi_{\theta}}\|_{\infty}$

###### Remark 3.

(_Applicability of linear function approximation_) The contraction of the $TD$
operator (7) has only been validated for a tabular representation of the
state-value function (Lin et al., 2018). We also assessed the possibility of
approximating the state value function using linear functions for scaling up
the proposed actor-critic algorithm to large or continuous state spaces. The
traditional proof of convergence for a $TD$ critic with linear function
approximation requires the contraction of this operator with respect to the
$L^{2}$ norm defined over the steady-state distribution of the MDP (Lemma $4$
in Tsitsiklis and Van Roy (Tsitsiklis and Van Roy, 1997)). However, via
counterexample, it can be seen that this property does not necessarily hold
for the $TD$ operator (7).

###### Remark 4.

The previous remark and the fact that we were required to limit ourselves to
tabular representations implies that mathematical properties of CPT-sensitive
MDPs do not allow them to belong to the family of robust MDPs (Osogami, 2012)
and enjoy properties such as linear approximation for the state value function
and a convenient gradient estimation scheme as with coherent risk measures
(Tamar et al., 2015). It would be interesting to study and look at this
limitation from a cognitive perspective and to see whether dealing with
dynamic CPT risk-sensitive continuous control is cognitively cumbersome for
humans in behavioral experiments.

###  5.2. Convergence of the Actor

For notational simplicity, we denote $V^{i}_{\pi_{\theta}}(s)$ by
$V_{i}(\theta,s)$. We prove the convergence of our AC algorithm to a
subjective MPNE of the game if the following assumptions are satisfied.

###### Assumption 4.

For each agent $i$, $V_{i}(\theta,s)$ is convex with respect to $\theta^{i}$.
Also, the gradient is uniformly bounded, i.e., for each agent $i$ there exists
$\xi_{i}$ such that,

(19)
$\sup_{s\in\mathcal{S}}\left\|\nabla_{\theta_{i}}V_{i}(\theta,s)\right\|\leq\xi_{i}.$

###### Assumption 5.

The pseudo-gradient mapping value function,
$\nabla_{\theta}V(\theta,s)=\operatorname{col}\left(\nabla_{\theta_{i}}V_{1}(\theta,s),\ldots,\nabla_{\theta_{N}}V_{N}(\theta,s)\right)$,
is strongly monotone with respect to $\theta$, i.e., for every
$\theta,\theta^{\prime}\in\Theta,s\in\mathcal{S}$, there exists $\mu>0$ such
that

$\left(\nabla_{\theta}V(\theta,s)-\nabla_{\theta}V\left(\theta^{\prime},s\right)\right)^{\top}\left(\theta-\theta^{\prime}\right)\geq\mu\left\|\theta-\theta^{\prime}\right\|^{2}.$

Furthermore, this mapping is Lipschitz continuous, i.e.,

$\left\|\nabla_{\theta}V(\theta,s)-\nabla_{\theta}V\left(\theta^{\prime},s\right)\right\|\leq
L_{a}\left\|\theta-\theta^{\prime}\right\|.$

###### Theorem 1.

(Convergence of the actor) Given Assumptions 4 and 5 and a critic and an actor
with learning steps such that,

(20)
$\sum_{t=0}^{\infty}\alpha_{ac,t}=\infty,\quad\sum_{t=0}^{\infty}\alpha_{cr,t}=\infty,\\\
\quad\sum_{t=0}^{\infty}\alpha_{cr,t}^{2}<\infty,\quad\sum_{t=0}^{\infty}\alpha_{ac,t}^{2}<\infty,\quad\lim_{t\rightarrow\infty}\frac{\alpha_{ac,t}}{\alpha_{cr,t}}=0,$

algorithm (2) converges to the unique subjective Markov perfect Nash
equilibrium of the NAMG, asymptotically.

###### Proof.

We prove that the actor update will converge to the unique Nash policy of the
Markov game, which exists if Assumptions 4 and 5 are satisfied, starting from
any initial condition. We rewrite the actor update for agent $n$ below,

$\theta_{t+1}^{n}:=\theta_{t}^{n}+\alpha_{ac,t}\nabla_{\theta}V^{n}(\theta,s_{0}).$

Under Assumptions 4 and 5, based on Theorem 2 in Rosen (Rosen, 1965), there
exists a unique MPNE for the NAMG. Note that this CPT-sensitive (subjective)
MPNE can be different from the MPNE of the game when the agents are risk-
neutral. Consider the parameter vector $\theta^{*}$ as the vector that
constructs this unique Nash policy, for which we have
$\nabla_{\theta_{i}}J_{i}(\theta^{*})=0$ for all $i$. Per Assumption 4 and by
defining $\Delta\theta_{n}^{t}=\theta^{t}_{n}-\theta^{*}_{n}$, we have

(21) $\displaystyle\left\|\Delta\theta_{n}^{t+1}\right\|^{2}$
$\displaystyle=\left\|\Delta\theta_{n}^{t}-\alpha_{ac,t}\nabla_{\theta_{n}}V_{n}\left(\theta^{t}\right)\right\|^{2}$
$\displaystyle=\left\|\Delta\theta_{n}^{t}\right\|^{2}+\left(\alpha_{ac,t}\right)^{2}\left\|\nabla_{\theta_{n}}V_{n}\left(\theta^{t}\right)\right\|^{2}$
$\displaystyle-2\alpha_{ac,t}\nabla_{\theta_{n}}\left(V_{n}\left(\theta^{t}\right)\right)^{\top}\Delta\theta_{n}^{t}$
$\displaystyle\leq\left\|\Delta\theta_{n}^{t}\right\|^{2}+\left(\alpha_{ac,t}\right)^{2}\xi_{n}^{2}-2\alpha_{ac,t}\nabla_{\theta_{n}}V_{n}\left(\theta^{t}\right)^{\top}\Delta\theta_{n}^{t}.$

By summing the above equation over different $n\in\\{1,...,N\\}$ and defining
$\Delta\theta^{t}=\operatorname{col}\left(\Delta\theta_{1}^{t}\ldots,\Delta\theta_{N}^{t}\right)$,
we have

(22)
$\left\|\Delta\theta^{t+1}\right\|^{2}\leq\left\|\Delta\theta^{t}\right\|^{2}+\sum_{n\in\mathcal{N}}\xi_{n}^{2}\left(\alpha_{ac,t}\right)^{2}-2\alpha_{ac,t}\nabla_{\theta}V^{t^{\top}}\Delta\theta^{t},$

where
$\nabla_{\theta}V^{t}=\operatorname{col}\left(\nabla_{\theta_{1}}V_{1}\left(\theta^{t}\right),\ldots,\nabla_{\theta_{N}}V_{N}\left(\theta^{t}\right)\right)$.
We also know that
$\nabla_{\theta}V^{*}=\operatorname{col}\left(\nabla_{\theta_{1}}V_{1}\left(\theta^{*}\right),\ldots,\nabla_{\theta_{N}}V_{N}\left(\theta^{*}\right)\right)=0$.
Therefore, according to assumption 5,

(23) $\displaystyle-\nabla_{\theta}V^{t^{\top}}\Delta\theta^{t}$
$\displaystyle=-\left(\nabla_{\theta}J^{t}-\nabla_{\theta}V^{*}\right)^{\top}\Delta\theta^{t}$
$\displaystyle=-\left(\nabla_{\theta}V^{t}\left(\theta^{t}\right)-\nabla_{\theta}V^{t}\left(\theta^{*}\right)\right)^{\top}\Delta\theta^{t}$
$\displaystyle\leq-\mu\left\|\Delta\theta^{t}\right\|^{2}.$

Finally, with telescopic summation,

(24)
$\displaystyle\lim_{t\rightarrow\infty}\left\|\Delta\theta^{t}\right\|^{2}\leq\left\|\Delta\theta^{0}\right\|^{2}$
$\displaystyle+\underbrace{\sum_{n\in\mathcal{N}}\xi_{n}^{2}\sum_{t=0}^{\infty}\left(\alpha_{ac,t}\right)^{2}}_{T_{1}}$
$\displaystyle-\underbrace{\sum_{t=0}^{\infty}\mu\alpha_{ac,t}\left\|\Delta\theta^{t}\right\|^{2}}_{T_{2}}.$

Since $\mu>0$, $T_{2}\geq 0$. Therefore, as $T_{1}$ is bounded, $T_{2}$ is
bounded as well. Consequently, as $\sum_{t=0}^{\infty}\alpha_{ac,t}=\infty$,
$\lVert\Delta\theta^{t}\rVert^{2}$ converges to $0$ as $t\rightarrow\infty$
and as a result $\theta$ converges to $\theta^{*}$. ∎

Given the asymptotic proof of convergence for the actor and the critic and
considering the conditions of the learning step sequences in Theorem 1, we can
apply Theorem 1.1 of Borkar (Borkar, 1997), which shows the asymptotic
convergence of the AC algorithm to the unique MPNE of the NAMG. Note that if
Assumptions 4 and 5 do not hold, given the actor’s gradient expression, we can
only ensure the convergence of the AC algorithm to a locally optimal policy
parameter for each agent.

##  6\. Numerical Experiment

To examine the empirical convergence of Distributed Nested CPT AC, we
constructed a risk-sensitive NAMG with an interpretable design for measuring
the effect of loss aversion in risk-sensitive agents on the converged
policies. Note that due to the CPT operator, Assumptions 4 and 5 are hard to
verify in any experimental setup and we did not expect the algorithm to
converge to the subjective MPNE (which may not be unique if the aforementioned
assumptions are not satisfied). In the constructed NAMG, there are four
agents, five states, and three possible actions
$(\mathcal{N}=4,\mathcal{S}=\\{0,1,2,3,4\\},\mathcal{A}=\\{0,1,2\\})$, and the
communication graph is fully-connected. The reward function is defined as

(25)
$R^{i}(s,a^{i},\sigma^{i}(a^{-i}))=R_{self}^{i}(s)+\sigma^{i}(a^{-i})R^{i}_{com}(s)a^{i},$

where the first term is the reward solely affected by the agent’s action, and
the second term is the reward that is affected by the actions of the
neighboring $community$ of the agent. The aggregative term is

(26) $\sigma^{i}(a^{-i})=\frac{1}{N-1}(\sum_{j\in\mathcal{N}\backslash
i}a_{j}),$

which indicates a communication graph with equal weights among the neighbors.
The reward coefficient $R_{self}^{i}(s)$ for agent $i$ is randomly generated
from

(27) $R_{self}(s,a^{i})\sim Normal(0.5,0.1),\forall i\in{1,...,N}.$

Also, the reward coefficients $R_{com}^{i}(s)$ is randomly generated form

(28) $R_{com}^{i}(s)\sim 5\cdot uniform[-0.5,0.5].$

The above setup implies a high risk for the agent if it decides to take an
action greater than $a^{i}=0$ and become involved with its neighboring
community, e.g., take a financial risk in an interactive market. Thus, it can
be said that each agent in this non-cooperative environment chooses by its
action how much risk it wants to take and to what degree it wants to interact
with the community (other agents who could, for instance, be economic,
political, or social competitors), and ties its received reward to their
actions. If the agent chooses the action $a^{i}=0$, it will settle for a low,
but risk-free profit. However, when choosing another action, depending on the
actions of the other agents, it can make a significant profit or loss that is
also affected by stochasticity of the environment. Our objective is to study
the agents’ risk-aversion levels based on the parameter $\lambda$ in the
cumulative perspective theory utility function (Figure 2). The higher the
value of this parameter, the more loss-averse the agent is. We expect that in
the designed non-cooperative environment, a more loss-averse agent will choose
the action $a=0$ with a higher probability and have a tendency to become
socially isolated or conservative. To evaluate this hypothesis, we run the
proposed algorithm for this risk-sensitive NAMG and for four different loss-
aversion scenarios. In the first scenario, all agents are risk-neutral
(corresponding to a vanilla AC algorithm with linear function approximation).
In the second scenario, all agents have the same level of loss aversion
($\lambda=2.6$). In the third scenario, only Agent 1 is risk-sensitive
($\lambda=2.6$), and other agents are risk-neutral. Finally, in the last
scenario, all agents are risk-sensitive, but Agent one has a higher loss
aversion coefficient ($\lambda=3.2$), while others have $\lambda=2.6$. Figure
4 shows the convergence of the value functions corresponding to one of the
states in the second scenario. Figure 5 shows the probability of choosing
action $0$ (a quantitative indicator of social conservatism) in the converged
policies of agents in each scenario. As observed, the level of social
isolation and the probability of choosing the conservative action $a=0$ is
proportional to the risk-aversion level of the agents in the community.

Figure 4. Smoothed mean value function of a given state over eight independent
runs in Distributed Nested CPT-AC for scenario 2 (all agents are risk-
sensitive with $\lambda=2.6$). Figure 5. Mean converged policies over eight
independent runs for different loss aversion scenarios. Scenario 1: all agents
are risk-neutral, scenario 2: all agents are risk-sensitive ($\lambda=2.6$),
scenario 3: only Agent 1 is risk-sensitive ($\lambda=2.6$), scenario 4: Agent
1 has a higher loss aversion coefficient ($\lambda=3.2$) than others
($\lambda=2.6$).

##  7\. Conclusion and Future Works

In this work, we proposed a distributed risk-sensitive MARL algorithm in NAMGs
with theoretical convergence guarantees based on cumulative prospect theory, a
cognitive risk measure that broadens the scope of the traditionally adopted
coherent risk measure. We empirically showed the positive correlation between
loss aversion and social isolation of agents. We observed that scaling the
proposed algorithm to larger environments and continuous control is not
compatible with theoretical convergence gurantees. However, a plausible
direction of future work is the appropriate use of function approximation and
deep RL methods to tackle the curse of dimensionality for large state and
action spaces in CPT-sensitive RL, and in general risk-averse RL (Urpí et al.,
2021), in an empirical framework, albeit without theoretical convergence
gurantees.

## References

  * (1)
  * Acerbi (2002) Carlo Acerbi. 2002.  Spectral measures of risk: A coherent representation of subjective risk aversion.  _Journal of Banking & Finance_ 26, 7 (2002), 1505–1518.
  * Alacaoglu et al. (2022) Ahmet Alacaoglu, Luca Viano, Niao He, and Volkan Cevher. 2022.  A natural actor-critic framework for zero-sum Markov games. In _International Conference on Machine Learning_. PMLR, 307–366.
  * Artzner et al. (1999) Philippe Artzner, Freddy Delbaen, Jean-Marc Eber, and David Heath. 1999.  Coherent measures of risk.  _Mathematical finance_ 9, 3 (1999), 203–228.
  * Başar (2021) Tamer Başar. 2021.  Robust designs through risk sensitivity: An overview.  _Journal of Systems Science and Complexity_ 34 (2021), 1634–1665.
  * Borkar (1997) Vivek S Borkar. 1997.  Stochastic approximation with two time scales.  _Systems & Control Letters_ 29, 5 (1997), 291–294.
  * Borkar (2001) Vivek S Borkar. 2001.  A sensitivity formula for risk-sensitive cost and the actor–critic algorithm.  _Systems & Control Letters_ 44, 5 (2001), 339–346.
  * Cavus and Ruszczynski (2014) Ozlem Cavus and Andrzej Ruszczynski. 2014.  Risk-averse control of undiscounted transient Markov models.  _SIAM Journal on Control and Optimization_ 52, 6 (2014), 3935–3966.
  * Cenedese et al. (2020) Carlo Cenedese, Giuseppe Belgioioso, Sergio Grammatico, and Ming Cao. 2020.  Time-varying constrained proximal type dynamics in multi-agent network games. In _2020 European Control Conference (ECC)_. IEEE, 148–153.
  * Chow and Ghavamzadeh (2014) Yinlam Chow and Mohammad Ghavamzadeh. 2014.  Algorithms for CVaR optimization in MDPs.  _Advances in neural information processing systems_ 27 (2014).
  * Chow et al. (2017) Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. 2017.  Risk-constrained reinforcement learning with percentile risk criteria.  _The Journal of Machine Learning Research_ 18, 1 (2017), 6070–6120.
  * Delbaen (2002) Freddy Delbaen. 2002.  Coherent risk measures on general probability spaces.  _Advances in finance and stochastics: essays in honour of Dieter Sondermann_ (2002), 1–37.
  * Deng (2019) Zhenhua Deng. 2019.  Distributed algorithm design for resource allocation problems of second-order multiagent systems over weight-balanced digraphs.  _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ 51, 6 (2019), 3512–3521.
  * Deng (2021) Zhenhua Deng. 2021.  Distributed algorithm design for aggregative games of Euler–Lagrange systems and its application to smart grids.  _IEEE Transactions on Cybernetics_ 52, 8 (2021), 8315–8325.
  * Ding et al. (2022) Dongsheng Ding, Chen-Yu Wei, Kaiqing Zhang, and Mihailo Jovanovic. 2022.  Independent policy gradient for large-scale markov potential games: Sharper rates, function approximation, and game-agnostic convergence. In _International Conference on Machine Learning_. PMLR, 5166–5220.
  * Fei et al. (2021) Yingjie Fei, Zhuoran Yang, Yudong Chen, and Zhaoran Wang. 2021.  Exponential bellman equation and improved regret bounds for risk-sensitive reinforcement learning.  _Advances in Neural Information Processing Systems_ 34 (2021), 20436–20446.
  * Fox et al. (2022) Roy Fox, Stephen M Mcaleer, Will Overman, and Ioannis Panageas. 2022.  Independent natural policy gradient always converges in Markov potential games. In _International Conference on Artificial Intelligence and Statistics_. PMLR, 4414–4425.
  * Ghosh et al. (2022) Mrinal K Ghosh, Subrata Golui, Chandan Pal, and Somnath Pradhan. 2022.  Nonzero-sum risk-sensitive continuous-time stochastic games with ergodic costs.  _Applied Mathematics & Optimization_ 86, 1 (2022), 6.
  * Ghosh et al. (2023) Mrinal K Ghosh, Subrata Golui, Chandan Pal, and Somnath Pradhan. 2023.  Discrete-time zero-sum games for Markov chains with risk-sensitive average cost criterion.  _Stochastic Processes and their Applications_ 158 (2023), 40–74.
  * Jie et al. (2018) Cheng Jie, LA Prashanth, Michael Fu, Steve Marcus, and Csaba Szepesvári. 2018.  Stochastic optimization in a cumulative prospect theory framework.  _IEEE Trans. Automat. Control_ 63, 9 (2018), 2867–2882.
  * Kahneman and Tversky (1979) DANIEL Kahneman and Amos Tversky. 1979.  Prospect theory: An analysis of decision under risk.  _Econometrica_ 47, 2 (1979), 363–391.
  * La and Ghavamzadeh (2013) Prashanth La and Mohammad Ghavamzadeh. 2013.  Actor-critic algorithms for risk-sensitive MDPs.  _Advances in neural information processing systems_ 26 (2013).
  * Leonardos et al. (2021) Stefanos Leonardos, Will Overman, Ioannis Panageas, and Georgios Piliouras. 2021.  Global convergence of multi-agent policy gradient in markov potential games.  _arXiv preprint arXiv:2106.01969_ (2021).
  * Lin (2013) Kun Lin. 2013.  _Stochastic systems with cumulative prospect theory_.  Ph.D. Dissertation. University of Maryland, College Park.
  * Lin et al. (2018) Kun Lin, Cheng Jie, and Steven I Marcus. 2018.  Probabilistically distorted risk-sensitive infinite-horizon dynamic programming.  _Automatica_ 97 (2018), 1–6.
  * Lin and Marcus (2013) Kun Lin and Steven I Marcus. 2013.  Dynamic programming with non-convex risk-sensitive measures. In _2013 American Control Conference_. IEEE, 6778–6783.
  * Littman (1994) Michael L Littman. 1994.  Markov games as a framework for multi-agent reinforcement learning.  In _Machine learning proceedings 1994_. Elsevier, 157–163.
  * Maheshwari et al. (2022) Chinmay Maheshwari, Manxi Wu, Druv Pai, and Shankar Sastry. 2022.  Independent and decentralized learning in markov potential games.  _arXiv preprint arXiv:2205.14590_ (2022).
  * Markowitz et al. (2021) Jared Markowitz, Marie Chau, and I-Jeng Wang. 2021.  Deep CPT-RL: Imparting Human-Like Risk Sensitivity to Artificial Agents.. In _SafeAI@ AAAI_.
  * Mguni et al. (2021) David H Mguni, Yutong Wu, Yali Du, Yaodong Yang, Ziyi Wang, Minne Li, Ying Wen, Joel Jennings, and Jun Wang. 2021.  Learning in nonzero-sum stochastic games with potentials. In _International Conference on Machine Learning_. PMLR, 7688–7699.
  * Moharrami et al. (2022) Mehrdad Moharrami, Yashaswini Murthy, Arghyadip Roy, and Rayadurgam Srikant. 2022.  A Policy Gradient Algorithm for the Risk-Sensitive Exponential Cost MDP.  _arXiv preprint arXiv:2202.04157_ (2022).
  * Munir et al. (2021) Md Shirajum Munir, Sarder Fakhrul Abedin, Nguyen H Tran, Zhu Han, Eui-Nam Huh, and Choong Seon Hong. 2021.  Risk-aware energy scheduling for edge computing with microgrid: A multi-agent deep reinforcement learning approach.  _IEEE Transactions on Network and Service Management_ 18, 3 (2021), 3476–3497.
  * Noorani and Baras (2022) Erfaun Noorani and John S Baras. 2022.  Risk-attitudes, Trust, and Emergence of Coordination in Multi-agent Reinforcement Learning Systems: A Study of Independent Risk-sensitive REINFORCE. In _2022 European Control Conference (ECC)_. IEEE, 2266–2271.
  * Osogami (2012) Takayuki Osogami. 2012.  Robustness and risk-sensitivity in Markov decision processes.  _Advances in neural information processing systems_ 25 (2012).
  * Pal and Pradhan (2021) Chandan Pal and Somnath Pradhan. 2021.  Zero-sum games for pure jump processes with risk-sensitive discounted cost criteria.  _Journal of Dynamics and Games_ 9, 1 (2021), 13–25.
  * Parise et al. (2020) Francesca Parise, Sergio Grammatico, Basilio Gentile, and John Lygeros. 2020.  Distributed convergence to Nash equilibria in network and average aggregative games.  _Automatica_ 117 (2020), 108959.
  * Perolat et al. (2015) Julien Perolat, Bruno Scherrer, Bilal Piot, and Olivier Pietquin. 2015.  Approximate dynamic programming for two-player zero-sum Markov games. In _International Conference on Machine Learning_. PMLR, 1321–1329.
  * Prashanth et al. (2022) LA Prashanth, Michael C Fu, et al. 2022\.  Risk-Sensitive Reinforcement Learning via Policy Gradient Search.  _Foundations and Trends® in Machine Learning_ 15, 5 (2022), 537–693.
  * Prashanth and Ghavamzadeh (2016) LA Prashanth and Mohammad Ghavamzadeh. 2016.  Variance-constrained actor-critic algorithms for discounted and average reward MDPs.  _Machine Learning_ 105 (2016), 367–417.
  * Prashanth et al. (2016) LA Prashanth, Cheng Jie, Michael Fu, Steve Marcus, and Csaba Szepesvári. 2016.  Cumulative prospect theory meets reinforcement learning: Prediction and control. In _International Conference on Machine Learning_. PMLR, 1406–1415.
  * Prelec (1998) Drazen Prelec. 1998.  The probability weighting function.  _Econometrica_ (1998), 497–527.
  * Qiu et al. (2021b) Shuang Qiu, Xiaohan Wei, Jieping Ye, Zhaoran Wang, and Zhuoran Yang. 2021b.  Provably efficient fictitious play policy optimization for zero-sum Markov games with structured transitions. In _International Conference on Machine Learning_. PMLR, 8715–8725.
  * Qiu et al. (2021a) Wei Qiu, Xinrun Wang, Runsheng Yu, Rundong Wang, Xu He, Bo An, Svetlana Obraztsova, and Zinovi Rabinovich. 2021a.  RMIX: Learning risk-sensitive policies for cooperative reinforcement learning agents.  _Advances in Neural Information Processing Systems_ 34 (2021), 23049–23062.
  * Reddy et al. (2019) D Sai Koti Reddy, Amrita Saha, Srikanth G Tamilselvam, Priyanka Agrawal, and Pankaj Dayama. 2019.  Risk averse reinforcement learning for mixed multi-agent environments. In _Proceedings of the 18th international conference on autonomous agents and multiagent systems_. 2171–2173.
  * Rosen (1965) J Ben Rosen. 1965.  Existence and uniqueness of equilibrium points for concave n-person games.  _Econometrica: Journal of the Econometric Society_ (1965), 520–534.
  * Ruszczyński (2010) Andrzej Ruszczyński. 2010.  Risk-averse dynamic programming for Markov decision processes.  _Mathematical programming_ 125 (2010), 235–261.
  * Saffar et al. (2017) Mohsen Saffar, Hamed Kebriaei, and Dusit Niyato. 2017.  Pricing and rate optimization of cloud radio access network using robust hierarchical dynamic game.  _IEEE Transactions on Wireless Communications_ 16, 11 (2017), 7404–7418.
  * Sayin et al. (2021) Muhammed Sayin, Kaiqing Zhang, David Leslie, Tamer Basar, and Asuman Ozdaglar. 2021.  Decentralized Q-learning in zero-sum Markov games.  _Advances in Neural Information Processing Systems_ 34 (2021), 18320–18334.
  * Shapiro et al. (2021) Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszczynski. 2021.  _Lectures on stochastic programming: modeling and theory_.  SIAM.
  * Shapley (1953) Lloyd S Shapley. 1953.  Stochastic games.  _Proceedings of the national academy of sciences_ 39, 10 (1953), 1095–1100.
  * Shokri and Kebriaei (2020) Mohammad Shokri and Hamed Kebriaei. 2020.  Leader–follower network aggregative game with stochastic agents’ communication and activeness.  _IEEE Trans. Automat. Control_ 65, 12 (2020), 5496–5502.
  * Soorki et al. (2021) Mehdi Naderi Soorki, Walid Saad, Mehdi Bennis, and Choong Seon Hong. 2021.  Ultra-reliable indoor millimeter wave communications using multiple artificial intelligence-powered intelligent surfaces.  _IEEE Transactions on Communications_ 69, 11 (2021), 7444–7457.
  * Sutton and Barto (2018) Richard S Sutton and Andrew G Barto. 2018.  _Reinforcement learning: An introduction_.  MIT press.
  * Tamar et al. (2015) Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. 2015.  Policy gradient for coherent risk measures.  _Advances in neural information processing systems_ 28 (2015).
  * Tamar et al. (2012) Aviv Tamar, Dotan Di Castro, and Shie Mannor. 2012.  Policy gradients with variance related risk criteria. In _Proceedings of the twenty-ninth international conference on machine learning_. 387–396.
  * Tamar et al. (2013) Aviv Tamar, Dotan Di Castro, and Shie Mannor. 2013.  Temporal difference methods for the variance of the reward to go. In _International Conference on Machine Learning_. PMLR, 495–503.
  * Tamar et al. (2014) Aviv Tamar, Shie Mannor, and Huan Xu. 2014.  Scaling up robust MDPs using function approximation. In _International conference on machine learning_. PMLR, 181–189.
  * Tan et al. (2021) Shaolin Tan, Yaonan Wang, and Athanasios V Vasilakos. 2021.  Distributed population dynamics for searching generalized nash equilibria of population games with graphical strategy interactions.  _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ 52, 5 (2021), 3263–3272.
  * Tian et al. (2021) Ran Tian, Liting Sun, and Masayoshi Tomizuka. 2021.  Bounded risk-sensitive markov games: Forward policy design and inverse reward learning with iterative reasoning and cumulative prospect theory. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 6011–6020.
  * Tsitsiklis and Van Roy (1997) John Tsitsiklis and Benjamin Van Roy. 1997.  An analysis of temporal-difference learning with function approximation.  _IEEE Trans. Automat. Control_ 42, 5 (1997), 674–690.
  * Tversky and Kahneman (1992) Amos Tversky and Daniel Kahneman. 1992.  Advances in prospect theory: Cumulative representation of uncertainty.  _Journal of Risk and uncertainty_ 5 (1992), 297–323.
  * Urpí et al. (2021) Núria Armengol Urpí, Sebastian Curi, and Andreas Krause. 2021.  Risk-averse offline reinforcement learning.  _arXiv preprint arXiv:2102.05371_ (2021).
  * Wright and Leyton-Brown (2017) James R Wright and Kevin Leyton-Brown. 2017.  Predicting human behavior in unrepeated, simultaneous-move games.  _Games and Economic Behavior_ 106 (2017), 16–37.
  * Zhang et al. (2020) Kaiqing Zhang, Sham Kakade, Tamer Basar, and Lin Yang. 2020.  Model-based multi-agent rl in zero-sum markov games with near-optimal sample complexity.  _Advances in Neural Information Processing Systems_ 33 (2020), 1166–1178.
  * Zhang et al. (2021) Kaiqing Zhang, Xiangyuan Zhang, Bin Hu, and Tamer Basar. 2021.  Derivative-free policy optimization for linear risk-sensitive and robust control design: Implicit regularization and sample complexity.  _Advances in Neural Information Processing Systems_ 34 (2021), 2949–2964.
  * Zhong et al. (2022) Hai Zhong, Yutaka Shimizu, and Jianyu Chen. 2022.  Chance-Constrained Iterative Linear-Quadratic Stochastic Games.  _IEEE Robotics and Automation Letters_ 8, 1 (2022), 440–447.
  * Zhu et al. (2013) Kun Zhu, Ekram Hossain, and Dusit Niyato. 2013.  Pricing, spectrum sharing, and service selection in two-tier small cell networks: A hierarchical dynamic game approach.  _IEEE Transactions on Mobile Computing_ 13, 8 (2013), 1843–1856.
  * Zhu et al. (2022) Ziqing Zhu, Ka Wing Chan, Siqi Bu, Bin Zhou, and Shiwei Xia. 2022.  Nash Equilibrium Estimation and Analysis in Joint Peer-to-Peer Electricity and Carbon Emission Auction Market With Microgrid Prosumers.  _IEEE Transactions on Power Systems_ (2022).