,

# Kindness in Multi-Agent Reinforcement Learning

Farinaz Alamiyan-Harandi
Department of Electrical & Computer Engineering
Isfahan University of Technology
Isfahan 84156-83111, Iran
<EMAIL_ADDRESS>
&Mersad Hassanjani
Department of Electrical & Computer Engineering
Isfahan University of Technology
Isfahan 84156-83111, Iran
<EMAIL_ADDRESS>
&Pouria Ramazi
Department of Mathematics & Statistics
Brock University
St. Catharines, ON L2S 3A1, Canada
<EMAIL_ADDRESS>

###### Abstract

In human societies, people often incorporate fairness in their decisions and
treat reciprocally by being kind to those who act kindly. They evaluate the
kindness of others’ actions not only by monitoring the outcomes but also by
considering the intentions. This behavioral concept can be adapted to train
cooperative agents in Multi-Agent Reinforcement Learning (MARL). We propose
the KindMARL method, where agents’ intentions are measured by counterfactual
reasoning over the environmental impact of the actions that were available to
the agents. More specifically, the current environment state is compared with
the estimation of the current environment state provided that the agent had
chosen another action. The difference between each agent’s reward, as the
outcome of its action, with that of its fellow, multiplied by the intention of
the fellow is then taken as the fellow’s kindness. If the result of each
reward-comparison confirms the agent’s superiority, it perceives the fellow’s
kindness and reduces its own reward. Experimental results in the Cleanup and
Harvest environments show that training based on the KindMARL method enabled
the agents to earn $89\%$ (resp. $37\%$) and $44\%$ (resp. $43\%$) more total
rewards than training based on the Inequity Aversion and Social Influence
methods. The effectiveness of KindMARL is further supported by experiments in
a traffic light control problem.

##  1 Introduction

Reinforcement Learning (RL) is a promising framework of machine learning where
an intelligent agent interacts within a given environment, by applying actions
and receiving rewards, to learn an optimal policy, a probability distribution
over the available actions that maximizes the total obtained rewards for each
visited environment state (Sutton et al., 1998). RL tackles a wide range of
tasks by using a simple concept of rewards and punishments without requiring
prior knowledge about the given problem. Many real-world problems such as
resource allocation (Naderializadeh et al., 2021) and real-time traffic flow
control (Wei et al., 2019b) consider group activities in an environment with
shared resources where a conflict of interest exists between individual and
social preferences. To handle these problems, the single-agent RL algorithms
are extended as Multi-Agent RL (MARL) settings, where each agent learns a
policy from its own observations. However, partial observability and the
environmental impact of the actions applied by other agents raise uncertainty
in the environmental states and makes the training process unstable (Chu et
al., 2019). Learning communication protocols to exchange the agents’
information (Foerster et al., 2016) and enriching the extrinsic rewards
received from the environment by using some intrinsic rewards (Jaques et al.,
2019) are techniques that are introduced to stabilize training and improve the
learned policies in MARL approaches.

Intrinsic rewards encode some aspects of agents’ behavior that are not
represented by extrinsic rewards explicitly. They are scalar signals that an
agent internally computes and are typically defined based on the behavioral
practices and social concepts in human societies. Examples of these concepts
incorporated in MARL as intrinsic rewards are as follows: _(i) social
influence_ , the agent’s ability to select the action that influences its
fellows’ next action (Jaques et al., 2019), _(ii) joint attention_ , the
agent’s ability to manage its attention to the same object or event that one
of its fellows attends (Lee et al., 2021), _(iii) curiosity_ , the agent’s
capability to predict the outcomes of its own actions (Heemskerk, 2020), _(iv)
environmental impact_ , the agent’s criterion to measure the fellows’ role in
reaching the current rewarding environmental state (Anonymous, ), and _(v)
inequity aversion_ , the agent’s comparison-based criterion to judge its
fellows’ cooperation (Hughes et al., 2018).

Consider the Inequity Aversion (IA) method (Hughes et al., 2018), for example,
where each agent changes its own extrinsic reward based on the envy and guilt
concepts by comparing its own extrinsic reward with that of each of its
fellows. If the result of a pair-comparison confirms the agent’s advantageous
inequity, that it has earned more than its fellow, then the agent feels guilty
and if the result confirms the agent’s disadvantageous inequity, the agent
experiences envy. In both situations, the agent reduces its own reward as a
punishment.

We aim to enrich the IA model based on the theory of reciprocity (Falk and
Fischbacher, 2006) in social psychology, stating that consequences of an
action are not the only criterion to evaluate its kindness–the actor’s
intentions also influence others’ conclusions. In MARL, an agent’s intention
can be measured by its impact in reaching the current environment state that
resulted in the agents’ rewards (Anonymous, ). To this end, we propose the
KindMARL method. We equip each agent with the Extended Intrinsic Curiosity
Module (EICM) (Anonymous, ) to predict the role of every other agent’s actions
in reaching the current environment state. An agent considers the action set
of its fellow as the fellow’s alternatives and predicts the follow’s
counterfactual roles in achieving the current environment state. Then the
agent measures its fellow’s intention by dividing the impact of the fellow’s
action to the maximum impact obtained by any of the fellow’s available
actions. Multiplying every IA pair-comparison by the intention of its
corresponding fellow, the agent becomes sensitive to others’ intentions as
well as their rewards and reacts to other agents differently according to
their kindness. The agent, hence, can act fairer in dealing with itself as
well as the others. We test the KindMARL method on CLeanup and Harvest
environments (the _common pool resource problems_ (Leibo et al., 2017)) and
also the problem of traffic light control. We compare the results with those
of the state-of-the-art methods.

The remaining of this paper is organized as follows: Section 2 formalizes the
MARL problem and describes the proposed KindMARL method. The case study and
experimental results are given in Section 3 and are discussed in Section 4.

##  2 The KindMARL approach

We consider the following MARL problem: $N$ agents coexist in an environment
with the state space $\mathcal{S}$. At each time step $t$, each agent $k$
partially observes the environment’s global state $s_{t}$ as the local state
$s^{k}_{t}$, and selects an action $a^{k}_{t}$ from the action set
$\mathcal{A}^{k}=\\{a_{1},...,a_{m}\\}$ by using a policy $\pi^{k}$, that is a
probability distribution over the agent’s action set $\mathcal{A}^{k}$. By
applying the joint action $\bm{a_{t}}=[a^{1}_{t},...,a^{k}_{t},...,a^{N}_{t}]$
at the global state $s_{t}$, the environment transfers to the global state
$s_{t+1}$ according to a transition distribution
$\mathcal{T}(s_{t+1}|s_{t},\bm{a}_{t})$, resulting the reward $r^{k}_{t+1}$ to
each agent $k$. The reward is used it to compute a state-action value function
$Q^{\pi^{k}}(s^{k}_{t},a^{k}_{t})$ for all local states $s^{k}_{t}$ and agent
$k$’s action set $\mathcal{A}^{k}$, where $Q^{\pi^{k}}(s^{k}_{t},a^{k}_{t})$
approximates the return
$R^{k}_{t}=\sum_{i=0}^{\infty}\gamma^{i}r^{k}_{t+i+1}$, that is an estimation
of the cumulative $\gamma$\- discounted rewards over all local states visited
in future by applying action $a^{k}_{t}$ on local state $s^{k}_{t}$ and
following policy $\pi^{k}$. Agent $k$ selects the action with maximum
$Q^{\pi^{k}}(s^{k}_{t},a^{k}_{t})$ in each time step $t$.

To train cooperative agents, we use the linear reward function
$r^{k}_{t}=\alpha e^{k}_{t}+\beta i^{k}_{t}$ where $\alpha$ and $\beta$ are
constant scalars, $e^{k}_{t}$ is the extrinsic reward that agent $k$ receives
from the environment, and $i^{k}_{t}$ is the intrinsic reward that agent $k$
computes by incorporating its fellows’ kindness in the IA model (Hughes et
al., 2018) as follows:

$\displaystyle i_{t}^{k}=-\frac{\alpha_{k}}{N-1}\sum_{j\neq
k}d^{k,j}_{t}\max(w^{j}_{t}-w^{k}_{t},0)-\frac{\beta_{k}}{N-1}\sum_{j\neq
k}d^{k,j}_{t}\max(w^{k}_{t}-w^{j}_{t},0),$ (1)

where parameters $\alpha_{k}$,$\beta_{k}\in\mathbb{R}$ are adjustable
parameters of the IA model, $w^{j}_{t}$ as a temporary memory of the extrinsic
reward occurrence (Hughes et al., 2018) is computed as

$\displaystyle w^{j}_{t}=\gamma\lambda
w^{j}_{t-1}+e_{t}^{j}\;\;\;\;\;\;\;\forall t\geq 1,\;w^{j}_{0}=0,$ (2)

where $\lambda\in[0,1]$ is a trace-decay hyper-parameter, and $d^{k,j}_{t}$ is
agent $j$’s intention computed by agent $k$.

To measure the fellows’ intention as the main factor of evaluating kindness,
agent $k$ estimates the environment impact of each of its fellows considering
counterfactual reasoning. It estimates what would happen in the current state
if agent $j$ had acted differently. In other words, agent $k$ predicts its
current local state $s^{k}_{t}$ by supposing each agent $j$ applies one of its
other available actions. To this end, agent $k$ uses the forward model of EICM
structure (Anonymous, ), that estimates the current encoded local state of the
environment ($\hat{\phi}(\phi(s^{k}_{t-1}),\bm{a}_{t-1})$) by receiving the
previous encoded local state $\phi(s^{k}_{t-1})$ and the previous joint
action, $\bm{a}_{t-1}$, where $\phi$ is an encoding function. Agent $k$
estimates the current encoded local state by replacing agent $j$’s action with
one of its counterfactual actions ${\tilde{a}^{j}_{t-1}}=\\{b|b\neq
a^{j}_{t-1},b\in A\\}$. Then it measures the difference between the previous
encoded local state $\phi(s^{k}_{t-1})$ with the counterfactual prediction of
the current encoded local state
$\hat{\phi}_{\tilde{a}^{j}_{t-1}}(\phi(s^{k}_{t-1}),\bm{a}^{\tilde{j}}_{t-1})$
by using the mean square error function as follows:

$\displaystyle{L_{t}}=\frac{1}{2}\left\|\phi(s^{k}_{t-1})-\hat{\phi}(\phi(s^{k}_{t-1}),\bm{a}^{\tilde{j}}_{t-1}))\right\|_{2}^{2},$
(3)

where $\bm{a}^{\tilde{j}}_{t-1}$ indicates that agent $j$’s action in the
joint action $\bm{a}_{t-1}$ is replaced by one of its counterfactual actions.
Let $l$ be the difference that is computed by using the joint action
$\bm{a_{t-1}}$ and ${L}^{j}_{t}$ be the list of differences, that each of its
elements is calculated considering one counterfactual action of agent $j$.
Then agent $j$’s intention is computed as follows (Figure. 1):

$\displaystyle d^{k,j}_{t}=\frac{l}{\max(L^{j}_{t})}.$ (4)

$\phi(s^{k}_{t-1})$$\phi(s^{k}_{t-1})$$u^{k}_{t-1}$$\bm{a}^{\tilde{j}}_{t-1}$$\bm{a}^{\tilde{j}}_{t-1}$$\bm{a}^{\tilde{j}}_{t-1}$$\bm{a}^{\tilde{j}}_{t-1}$
Forward Model
$\hat{\phi}(\phi(s^{k}_{t-1}),\bm{a}^{\tilde{j}}_{t-1})$$\hat{\phi}(\phi(s^{k}_{t-1}),\bm{a}^{\tilde{j}}_{t-1})$$\hat{\phi}(\phi(s^{k}_{t-1}),\bm{a}^{\tilde{j}}_{t-1})$$\hat{\phi}(\phi(s^{k}_{t-1}),\bm{a}^{\tilde{j}}_{t-1})$$\frac{1}{2}\left\|\phi(s^{k}_{t-1})-\hat{\phi}(\phi(s^{k}_{t-1}),\bm{a}^{\tilde{j}}_{t-1}))\right\|_{2}^{2}$$\frac{1}{2}\left\|\phi(s^{k}_{t-1})-\hat{\phi}(\phi(s^{k}_{t-1}),\bm{a}^{\tilde{j}}_{t-1}))\right\|_{2}^{2}$$\frac{1}{2}\left\|\phi(s^{k}_{t-1})-\hat{\phi}(\phi(s^{k}_{t-1}),\bm{a}^{\tilde{j}}_{t-1}))\right\|_{2}^{2}$$\frac{1}{2}\left\|\phi(s^{k}_{t-1})-\hat{\phi}(\phi(s^{k}_{t-1}),\bm{a}^{\tilde{j}}_{t-1}))\right\|_{2}^{2}$$L^{j}_{t}$$d^{k,j}_{t}=\frac{l}{\max(L^{j}_{t})}$

Figure 1: The process of computing agent $j$’s intention by agent $k$.

##  3 Experiments

###  3.1 The problem of Cleanup and Harvest

Cleanup is a public goods game wherein agents must clean a river before apples
can develop, but aren’t able to harvest apples at the same time as cleaning.
In Cleanup, agents should effectively time harvesting apples and cleaning the
river, and allow agents cleaning the river a chance to consume apples. Harvest
is a public pool resource game where players must cooperate to harvest the
most apples while being careful not to kill the trees by gathering all the
apples they contain. If harvesting is done too quickly, the apples will not
grow back. Therefore, players must strategically distribute their harvests in
order to avoid over-harvesting in any one area and hurting the trees’ health.
In both games, the agents receive $+1$ extrinsic reward when they collect an
apple and are punished by $-1$ extrinsic reward when they use their punishment
beam as an action to fire others. The hit agent also receives $-50$ extrinsic
reward. The Cleanup and Harvest environments are open-source python codes
developed by Vinitsky et.al (Vinitsky et al., 2019).

####  3.1.1 Experimental setup

For Cleanup and Harvest environments, the EICM network structure used in the
KindMARL method is taken from Anonymous  where an actor-critic structure is
used to learn the policy and value functions. Since the input of this
structure is a $15\times 15$ pixels image, a single-layer convolutional
network is utilized as the feature extraction network. To learn the parameters
of the actor-critic structure, Proximal Policy Optimization (PPO) (Schulman et
al., 2017) and the Asynchronous Advantage Actor-Critic (A3C) (Mnih et al.,
2016) algorithms are used in Cleanup and Harvest environments, respectively. A
Linux server with $3$ CPUs, a $P100$ Pascal GPU, and $100$G RAM was used to
execute $7$ (resp. $5$) experiments with random seeds in Cleanup (resp.
Harvest) environment. The number of agents operating in each run was $5$
(resp. $4$) in Cleanup (resp. Harvest) environment and they were the
advantageous IA types with $\alpha=0.05$ and $\beta=0$ for both IA and
KindMARL methods.

We compared the KindMARL method with three methods: _(i)_ the baseline method
utilizing only the extrinsic rewards, _(ii)_ the IA method (Hughes et al.,
2018), and _(iii)_ the Social Influence (SI) method (Jaques et al., 2019) that
added the intrinsic rewards derived from social empowerment to the extrinsic
rewards.

####  3.1.2 Experimental result

We utilized The total reward received by all agents as a performance measure.
According to numerical comparisons (Table 3.1.2), in Cleanup environment,
training based on the KindMARL method enabled the agents to earn $60.8\%$,
$88.6\%$ and $44.2\%$ more total rewards than training based on the baseline,
IA model, and SI method, respectively. The KindMARL agents could also earn
$9.1\%$, $37.2\%$ and $42.7\%$ more total rewards compared to baseline, IA,
and SI method in Harvest environment. The superiority of the KindMARL method
is obvious in Figure. 2 and Figure. 3 for the Cleanup and Harvest
environments, respectively.

Table 1: Collected rewards in Cleanup and Harvest environments. The collective
reward obtained by agents in the last $\num{192000}$ and $\num{1600000}$ steps
is averaged over $7$ and $5$ experiments in Cleanup and Harvest environments,
respectively. The results of IA and KindMARL methods are reported for the
advantageous-IA agents.

=_3pt^3pt  o|c|c|[2pt gray]c|c| Environment (Algorithm) Cleanup (PPO) Harvest
(A3C) Methods Baseline 470.9 622.2 IA 401.6 494.7 SI 525.3 475.5 KindMARL
757.4 678.5

=_3pt^3pt  occc  (a) Baseline (b) IA (c) SI (d) KindMARL (e) Comparison

Figure 2: The results for the Cleanup environment using PPO algorithm. The
opaque curve is the mean of the results of $7$ experiments.

=_3pt^3pt  occc  (a) Baseline (b) IA (c) SI (d) KindMARL (e) Comparison

Figure 3: The results for the Harvest environment using A3C algorithm. The
opaque curve is the mean of the results of $5$ experiments.

###  3.2 The problem of traffic light control

We consider the problem of traffic light control to cooperate between
intersections, the traffic generating situations where a large number of
vehicles have to wait for traffic permits to reach their destinations while
avoiding accidents. Many researchers took into account this challenge and
introduced various methods of controlling traffic signals to improve
transportation efficiency.

Besides many conventional methods, the problem was recently tackled with RL
methods. Some examples are as follows: _(i)_ IntelliLight (Wei et al., 2018)
used real data collected from surveillance cameras and trained a DQN method to
control the traffic lights of one intersection in SUMO simulator (Lopez et
al., 2018). It evaluated the learned policy not only by obtained rewards but
also by considering the quality of the pattern learned for light switching.
_(ii)_ DemoLight (Xiong et al., 2019) utilized the Self-Organizing Traffic
Light Control (Cools et al., 2013) to make the agents learn from an expert
using CityFlow simulator, that is an open-source traffic simulator designed
for large-scale traffic scenarios (Zhang et al., 2019). To speed up the
learning process, DemoLight applied the Advantage Actor-Critic (A2C) algorithm
(Mnih et al., 2016). _(iii)_ PressLight (Wei et al., 2019a) defined the reward
function according to Max Pressure method (Lioris et al., 2016) and trained a
DQN structure to control the traffic lights of several intersections in
CityFlow simulator. _(iv)_ MetaLight (Zang et al., 2020) applied the gradient-
based meta-learning algorithm (Finn et al., 2017) in CityFlow simulator to
speed up the learning process and improve generalization so that the knowledge
gained in previous traffic scenarios can be used in new ones. This method also
improved the Flipping and Rotation and considers All Phase (FRAP) model (Zheng
et al., 2019), that is based on the DQN structure to train the agents. _(v)_
MPLight method (Chen et al., 2020) is another DQN based method that employed a
shared DQN structure to control all intersections and also used the concept of
pressure in traffic. It conducted the experiments on an environment with more
than $1,000$ traffic lights using CityFlow simulator. _(vi)_ Colight (Wei et
al., 2019b) utilized the graph attention network (Veličković et al., 2017),
that represents the neighboring intersections as an overall summary and
enables the agents to learn the model of neighboring influence on their under-
control intersection. Each agent managed an intersection in CityFlow simulator
and had an instance of a DQN structure.

####  3.2.1 Experimental setup

We utilized the Colight method (Wei et al., 2019b) as a baseline to implement
the KindMARL method. Colight creates some environments on the CityFlow
simulator to implement traffic light signal control methods by collecting data
from Chinese cities such as Hangzhou and Jinan and New York City in the United
States, as well as artificially generated data. In this environment, several
vehicles move from various origins to their destinations while encountering
traffic lights. The green signal of each traffic light is accompanied by a
yellow three-second signal and a two-second red signal.

We conducted our experiments on the environment that has been created by using
Hangzhou dataset, which consists of a $4\times 4$ grid ($16$ intersections).
Each agent controls an intersection and sends its information to its four
neighbors that located at its four main directions. To learn appropriate
polices in this environment, each experiment includes several episodes. Each
episode contains $1440$ samples, that is the maximum amount of data that can
be created in an intersection during a day.

Here in EICM Structure, each agent learns the parameters of a Deep Q-Network
(DQN) structure (Mnih et al., 2015) as $Q^{\pi^{k}}(s^{k}_{t},a^{k}_{t})$,
that receives the visited local state $s^{k}_{t}$ as input and estimates the
state-action value of applying each available action of agent $k$ in the local
state $s^{k}_{t}$ as output (Figure. 4).

$Observations$GAT 1$...$GAT L$s^{k}_{t-1}$Graph Attentional
Network$\bm{a}_{t-1}$ FC $\texttt{u}=32$ FC $\texttt{u}=32$ LSTM
$\texttt{u}=128$$u^{k}_{t-1}$ FC
$\texttt{u}=|A|(N-1)$$P(\bm{a}_{t}|s^{k}_{t-1},\bm{a}_{t-1})$ MOA FC
$\texttt{u}=32$ FC $\texttt{u}=32$ FC
$\texttt{u}=|\mathcal{A}^{k}|$$Q(s^{k}_{t-1},a^{k}_{t-1})$DQN FC
$\texttt{u}=128$ FC $\texttt{u}=128$$\phi(s^{k}_{t-1})$$s^{k}_{t}$ FC
$\texttt{u}=128$ FC $\texttt{u}=128$$\phi(s^{k}_{t})$Feature
Encoding$u^{k}_{t-1}$ FC $\texttt{u}=32$ FC
$\texttt{u}=\texttt{q}$$\hat{\phi}(s^{k}_{t-1},\bm{a}_{t-1})$Forward
Model$u^{k}_{t-1}$ FC $\texttt{u}=32$ FC
$\texttt{u}=|A|(N)$$\hat{\bm{a}}_{t-1}$Inverse Model

Figure 4: The EICM network structure used in the Traffic light control
problem. The inputs of each agent $k$’s EICM are the previous joint action
$\bm{a}_{t-1}$ and the previous local state $s^{k}_{t-1}$ that is composed of
the representing features provided by the output of the graph attentional
network that is utilized for graph-structured environment where a summary of
each graph node’s neighborhood is prepared by embedding information from the
neighbors and their importance to the central node. The $L$ graph attention
layers of this network are illustrated by GAT. The EICM uses the DQN structure
to learn the state-action value function, the MOA network to predict the next
action of other agents, the feature encoding network to extract q features
from the local state $s^{k}_{t-1}$, the forward model to learn $\hat{\phi}$,
and the inverse model to predict the applied actions. In each time step $t$,
feature encoding is done for the local states $s^{k}_{t-1}$ and $s^{k}_{t}$.
The layers represented by the transparent gray rectangles inside the purple
dashed rectangle just indicates the same operation done for $s^{k}_{t}$. Fully
connected layers are indicated by FC, and the parameter u determines the
number of their neurons. To predict each agent $j$’s action $a^{j}_{t}$, agent
$k$’s MOA captures the state transition in its internal LTSM state
$u^{k}_{t-1}$. The graph attentional network and DQN structures are taken from
Wei et al. (2019b). The MOA structure is taken from Jaques et al. (2019). The
forward and inverse model are based on the structures presented by Heemskerk
(2020); Anonymous .

We compared the KindMARL method with two methods: _(i)_ the CoLight method
(Wei et al., 2019b), and _(ii)_ the IA method (Hughes et al., 2018). Since the
goal of the traffic light control problem is minimizing the vehicles’ travel
times, the extrinsic reward received by each agent is defined by the quarter
of the collective time of all vehicles stopped at its associated intersection.
To evaluate the performance of each method, the time distance between the
entering and leaving the grid for each vehicle was measured and their average
was used as the vehicle travel time metric. We executed each method $3$ times
and took an average between the obtained evaluation metric. For the CoLight
method, we have set the number of the heads in the attention mechanism to $5$.
For the IA, and KindMARL methods, first we used the typical advantageous type
of IA model where the $\alpha$ is zero and $\beta$ is set to $0.05$ and
executed experiments including a length of $200$ episodes. Then, we searched
the range of $-1$ to $1$ for the $\alpha$ and $\beta$ hyperparameters in the
IA model to investigate the effect of agents’ aversion to each of the
advantageous and disadvantageous inequities and to find the best combination
of these coefficients. To reduce the execution time, this search was done for
IA and KindMARL methods separately by executing experiments including a length
of $100$ episodes. Each experiment took around $8$ hours to perform on a Linux
server with $16$ CPUs and $120$G RAM.

####  3.2.2 Experimental results

According to the results obtained by the typical advantageous IA agents in IA
and KindMARL methods (Figure 5), the convergence speed of the KindMARL method
is better than others, but the IA and CoLight methods have a lower distance
between the two convergence indexes. The average travel time computed for the
last $20$ episodes demonstrates that the KindMARL outperforms other methods.
The exact values of the average travel times reported for the methods also
confirm the superiority of the KindMARL method (Table. 2). It is observed that
the KindMARL method had the best average travel time of the last $20$ steps of
each episode as the final learned performance and it is better than the
CoLight and IA methods by $8.8\%$, and $5.9\%$, respectively. The $\alpha$ and
$\beta$ search for the IA and KindMARL methods (Figure. 3.2.2) resulted in
some combinations of these parameters that improve the travel time. The
results of the IA and KindMARL methods with the best $\alpha$ and $\beta$
combination (Figure. 6, Table. 3) also demonstrate the IA and KindMARL methods
obtain lower travel times and better convergence speed than the CoLight
method.

Table 2: Statistical results for the Hangzhou dataset using the typical
advantageous IA agents in the IA and KindMARL methods.

Methods | Episode | Performance | Convergence
---|---|---|---
| (From $20$ to $200$) | (Second) | (Episode numbers)
CoLight | $366.8$ | $347.4$ | $187,188$
IA |  |  |
$\alpha=0,\beta=0.05$ | $373.4$ | $336.7$ | $182,182$
KindMARL |  |  |
$\alpha=0,\beta=0.05$ | 364.9 | 316.9 | $\textbf{170},199$

00footnotetext: Note: The first numerical column displays the average travel
time obtained by whole vehicles in episodes from $20$ to $200$ over $3$
individual experiments of CoLight, IA and KindMARL for the Hangzhou dataset.
The second and third numerical columns indicate the average of the last $20$
steps of each episode as the final learned performance and the convergence
thresholds, respectively. The $\alpha$ and $\beta$ values are reported for the
advantageous type of IA model typically used in IA method.

=_3pt^3pt  oc

Figure 5: The comparison between the mean travel times of all methods using
the typical advantageous IA agents in the IA and KindMARL methods. The
reported delay time at each point of these curves is computed by considering
all $16$ agents and over $3$ individual experiments. The vertical dashed lines
illustrated with same color as each method’s curve represent the position
where the method converges. The closer this line is to the vertical axis, the
faster is the method’s convergence. To draw these lines, we calculated the
average travel time obtained in the last $20$ episodes of each method as a
threshold. Then we checked if there is an episode in range of $50$ and $200$
that the maximum travel time computed for each episode from that episode to
end is less that $1.1$ and $1.2$ times of the threshold. If the episodes with
the first and second conditions were found, we determined the first and second
vertical lines at those episode numbers, respectively. The horizontal lines
indicate the average of the last $20$ steps of each episode, that the lower
line means the less travel time of the method. The result of the KindMARL
method is the best considering the above evaluation metrics.

=_3pt^3pt  occ  (a) (b)

The results of the average travel time over the last 20 episodes achieved by
the 3 experiments done for each IA model’s $\bm{\alpha}$ and $\bm{\beta}$
combination using the Hangzhou dataset. (a) The search result for the IA
method, (b) The search result for the KindMARL method. In each figure, the
result of CoLight is obtained by $\alpha=0,\beta=0$. Table 3: Statistical
results for the Hangzhou dataset using the best combination of IA model’s
hyperparameters in the IA and KindMARL methods.

Methods | Episode | Performance | Convergence
---|---|---|---
| (From $20$ to $100$) | (Second) | (Episode numbers)
CoLight | $407.9$ | $363.4$ | $89,94$
IA (Exhaustive search) |  |  |
$\alpha=0.6,\beta=-0.2$ | $398.43$ | 309.1 | $\textbf{78},\textbf{88}$
KindMARL (Exhaustive search) |  |  |
$\alpha=0,\beta=-1$ | 360.9 | $319.9$ | $85,85$

00footnotetext: Note: The numerical columns display the same information as
described in Figure. 2 except the first column showing the average travel time
obtained by whole vehicles in episodes from $20$ to $100$ over $3$ individual
experiments of CoLight, IA, and KindMARL for the Hangzhou dataset. The
$\alpha$ and $\beta$ values are reported for the best results of different
types of IA model considering the positive and negative ranges of these
coefficients.

=_3pt^3pt  oc

Figure 6: The comparison between the mean travel times of all methods using
the best combination of IA model’s hyperparameters in the IA and KindMARL
methods. To draw vertical dashed lines, we did the same operation as described
in Figure. 5 except reducing the range of episodes from $200$ to $100$. The
result of CoLight is obtained by $\alpha=0,\beta=0$.

##  4 Conclusion

We introduced a method of modeling the kindness concept in human societies to
improve the cooperativeness of agents trained in MARL problems. To determine
the kindness of the other agents, each agent evaluates the intentions of its
fellows. Then it uses the intentions as the weights of the agents’ rewards
comparison to improve the intrinsic reward functions of the IA method to
encourage agents to cooperate effectively in the MARL environments. To
demonstrate the superiority of the proposed KindMARL method, we conducted the
experiments on real data from Hangzhou city and compared the result of the
KindMARL with three state-of-the-art methods IA, EMuReL, and Colight. The
comparison results showed that the KindMARL method reduces the average travel
delay of the vehicles more than the other methods.

These experiments can be performed considering more episodes on data in larger
volumes to evaluate the KindMARL method for more realistic environments. An
exhaustive search over the IA method parameters can also improve the final
results. These cases are postponed to future work.

## Acknowledgments and Disclosure of Funding

We would like to thank Digital Research Alliance of Canada for providing
computational resources that facilitated our experiments.

## References

  * (1) Anonymous , A.  Environmental-impact based multi-agent reinforcement learning.
  * Chen et al. (2020) Chen , C., Wei , H., Xu , N., Zheng , G., Yang , M., Xiong , Y., Xu , K., and Li , Z.  Toward a thousand lights: Decentralized deep reinforcement learning for large-scale traffic signal control.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 3414–3421, 2020.
  * Chu et al. (2019) Chu , T., Chinchali , S., and Katti , S.  Multi-agent reinforcement learning for networked system control.  In _International Conference on Learning Representations_ , 2019.
  * Cools et al. (2013) Cools , S.-B., Gershenson , C., and D’Hooghe , B.  Self-organizing traffic lights: A realistic simulation.  In _Advances in applied self-organizing systems_ , pages 45–55. Springer, 2013.
  * Falk and Fischbacher (2006) Falk , A. and Fischbacher , U.  A theory of reciprocity.  _Games and economic behavior_ , 54(2):293–315, 2006.
  * Finn et al. (2017) Finn , C., Abbeel , P., and Levine , S.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _International conference on machine learning_ , pages 1126–1135. PMLR, 2017.
  * Foerster et al. (2016) Foerster , J., Assael , I. A., De Freitas , N., and Whiteson , S.  Learning to communicate with deep multi-agent reinforcement learning.  _Advances in neural information processing systems_ , 29, 2016.
  * Heemskerk (2020) Heemskerk , H.  Social curiosity in deep multi-agent reinforcement learning.  Master’s thesis, 2020.
  * Hughes et al. (2018) Hughes , E., Leibo , J. Z., Phillips , M., Tuyls , K., Dueñez-Guzman , E., Castañeda , A. G., Dunning , I., Zhu , T., McKee , K., Koster , R., and others .  Inequity aversion improves cooperation in intertemporal social dilemmas.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , pages 3330–3340, 2018.
  * Jaques et al. (2019) Jaques , N., Lazaridou , A., Hughes , E., Gulcehre , C., Ortega , P., Strouse , D., Leibo , J. Z., and De Freitas , N.  Social influence as intrinsic motivation for multi-agent deep reinforcement learning.  In _International Conference on Machine Learning_ , pages 3040–3049. PMLR, 2019.
  * Lee et al. (2021) Lee , D., Jaques , N., Kew , C., Wu , J., Eck , D., Schuurmans , D., and Faust , A.  Joint attention for multi-agent coordination and social learning.  In _ICRA Workshop on Social Intelligence in Humans and Robots_ , 2021\.  URL https://arxiv.org/abs/2104.07750.
  * Leibo et al. (2017) Leibo , J. Z., Zambaldi , V., Lanctot , M., Marecki , J., and Graepel , T.  Multi-agent reinforcement learning in sequential social dilemmas.  _In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, International Foundation for Autonomous Agents and Multiagent Systems_ , page 464?473, 2017.
  * Lioris et al. (2016) Lioris , J., Kurzhanskiy , A., and Varaiya , P.  Adaptive max pressure control of network of signalized intersections.  _IFAC-PapersOnLine_ , 49(22):19–24, 2016.
  * Lopez et al. (2018) Lopez , P. A., Behrisch , M., Bieker-Walz , L., Erdmann , J., Flötteröd , Y.-P., Hilbrich , R., Lücken , L., Rummel , J., Wagner , P., and Wießner , E.  Microscopic traffic simulation using sumo.  In _The 21st IEEE International Conference on Intelligent Transportation Systems_. IEEE, 2018.  URL https://elib.dlr.de/124092/.
  * Mnih et al. (2015) Mnih , V., Kavukcuoglu , K., Silver , D., Rusu , A. A., Veness , J., Bellemare , M. G., Graves , A., Riedmiller , M., Fidjeland , A. K., Ostrovski , G., and others .  Human-level control through deep reinforcement learning.  _nature_ , 518(7540):529–533, 2015.
  * Mnih et al. (2016) Mnih , V., Badia , A. P., Mirza , M., Graves , A., Lillicrap , T., Harley , T., Silver , D., and Kavukcuoglu , K.  Asynchronous methods for deep reinforcement learning.  In _International conference on machine learning_ , pages 1928–1937. PMLR, 2016.
  * Naderializadeh et al. (2021) Naderializadeh , N., Sydir , J. J., Simsek , M., and Nikopour , H.  Resource management in wireless networks via multi-agent deep reinforcement learning.  _IEEE Transactions on Wireless Communications_ , 20(6):3507–3523, 2021.
  * Schulman et al. (2017) Schulman , J., Wolski , F., Dhariwal , P., Radford , A., and Klimov , O.  Proximal policy optimization algorithms.  _CoRR_ , abs/1707.06347, 2017.  URL http://arxiv.org/abs/1707.06347.
  * Sutton et al. (1998) Sutton , R. S., Barto , A. G., and others .  _Introduction to reinforcement learning_ , volume 135.  MIT press Cambridge, 1998.
  * Veličković et al. (2017) Veličković , P., Cucurull , G., Casanova , A., Romero , A., Lio , P., and Bengio , Y.  Graph attention networks.  _arXiv preprint arXiv:1710.10903_ , 2017.
  * Vinitsky et al. (2019) Vinitsky , E., Jaques , N., Leibo , J., Castenada , A., and Hughes , E.  An open source implementation of sequential social dilemma games.  https://github.com/eugenevinitsky/sequential_social_dilemma_games/issues/182, 2019\.  GitHub repository.
  * Wei et al. (2018) Wei , H., Zheng , G., Yao , H., and Li , Z.  Intellilight: A reinforcement learning approach for intelligent traffic light control.  In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 2496–2505, 2018.
  * Wei et al. (2019a) Wei , H., Chen , C., Zheng , G., Wu , K., Gayah , V., Xu , K., and Li , Z.  Presslight: Learning max pressure control to coordinate traffic signals in arterial network.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 1290–1298, 2019a.
  * Wei et al. (2019b) Wei , H., Xu , N., Zhang , H., Zheng , G., Zang , X., Chen , C., Zhang , W., Zhu , Y., Xu , K., and Li , Z.  Colight: Learning network-level cooperation for traffic signal control.  In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_ , pages 1913–1922, 2019b.
  * Xiong et al. (2019) Xiong , Y., Zheng , G., Xu , K., and Li , Z.  Learning traffic signal control from demonstrations.  In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_ , pages 2289–2292, 2019.
  * Zang et al. (2020) Zang , X., Yao , H., Zheng , G., Xu , N., Xu , K., and Li , Z.  Metalight: Value-based meta-reinforcement learning for traffic signal control.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 1153–1160, 2020.
  * Zhang et al. (2019) Zhang , H., Feng , S., Liu , C., Ding , Y., Zhu , Y., Zhou , Z., Zhang , W., Yu , Y., Jin , H., and Li , Z.  Cityflow: A multi-agent reinforcement learning environment for large scale city traffic scenario.  In _The world wide web conference_ , pages 3620–3624, 2019.
  * Zheng et al. (2019) Zheng , G., Xiong , Y., Zang , X., Feng , J., Wei , H., Zhang , H., Li , Y., Xu , K., and Li , Z.  Learning phase competition for traffic signal control.  In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_ , pages 1963–1972, 2019.