# Learning-based social coordination to improve safety and robustness of
cooperative autonomous vehicles in mixed traffic

Rodolfo Valiente  Behrad Toghi  Mahdi Razzaghpour  Ramtin Pedarsani  Yaser P.
Fallah

###### Abstract

It is expected that autonomous vehicles(AVs) and heterogeneous human-driven
vehicles(HVs) will coexist on the same road. The safety and reliability of AVs
will depend on their social awareness and their ability to engage in complex
social interactions in a socially accepted manner. However, AVs are still
inefficient in terms of cooperating with HVs and struggle to understand and
adapt to human behavior, which is particularly challenging in mixed autonomy.
In a road shared by AVs and HVs, the social preferences or individual traits
of HVs are unknown to the AVs and different from AVs, which are expected to
follow a policy, HVs are particularly difficult to forecast since they do not
necessarily follow a stationary policy. To address these challenges, we frame
the mixed-autonomy problem as a multi-agent reinforcement learning (MARL)
problem and propose an approach that allows AVs to learn the decision-making
of HVs implicitly from experience, account for all vehicles’ interests, and
safely adapt to other traffic situations. In contrast with existing works, we
quantify AVs’ social preferences and propose a distributed reward structure
that introduces altruism into their decision-making process, allowing the
altruistic AVs to learn to establish coalitions and influence the behavior of
HVs. Keywords: Cooperative Driving, Mixed-autonomy, Reinforcement Learning,
Social coordination.

##  0.1 Introduction

Autonomous vehicles (AVs) have been an attractive research area for decades,
as it offers the potential to generate more efficient and safer road networks
cosgun2017towards . The adoption of AVs will not become a reality until they
can co-exist with humans, as part of a complex social system. In order to
maximize the potential of AVs and optimize for safety and traffic efficiency
of all the vehicles on the road, AVs have to coordinate and influence the
other agents cosgun2017towards ; schwarting2019social ; sagberg2015review .

We recognize the importance of social interaction and behavior in safety and
reliability and identify two important research directions. First, AVs must be
social actors and behave predictably and safely. Driver behavior is shaped by
habits and expectations in the traffic environment. The vehicle’s interaction
will be influenced by the way AV decisions are perceived. Therefore, the
ability of AVs to drive in a socially obedient manner is critical for the
safety of passengers and other vehicles because predictable behavior allows
humans to comprehend and respond appropriately to the AV’s actions. Second,
AVs must be social-aware and learn to identify social cues of egoism or
altruism, understand the behavior of human drivers and learn how to interact
and coordinate with all agents in a mixed traffic environment, adapting and
influencing the HVs behaviors to optimize for a social utility that improves
traffic flow and safety.

Figure 1:  (a) Interaction of AV-HV to benefit a HV: Altruistic agents create
alliances and direct the behavior of HVs to improve traffic flow and prevent
dangerous circumstances. AV1 and AV2 can create a formation to guide HV2 and
provide a route for HV1, allowing the HV to change lanes and navigate to the
exit ramp. (b) Interaction of AV-AV to benefit a HV: The goal of HV1 is to
integrate onto the highway. Egoistic AVs disregard the merging vehicle and do
not make room for it, possibly resulting in dangerous situations, however, if
they exhibit sympathy for the merging HV, they can compromise on their own
interest to create a safe path for HV1 to merge into the highway. (c)
Interaction of AV-AV to benefit another AV: The goal of AV1 is to exit the
highway. If AV2 acts selfishly, AV1 may miss the exit and be unable to
complete its task. However, if AV2 and AV3 consider AV1’s mission and act
altruistically, they can free up space in the platoon by AV2 decelerating and
AV3 accelerating, allowing AV1 to safely take the exit.

In this chapter, we focus on social awareness challenges and seek a solution
that can ensure the safety and robustness of AVs in the presence of human
drivers with heterogeneous behavioral traits. Vehicle-to-vehicle (V2V)
communication allows connected and autonomous vehicles (CAVs) to interact
directly with their neighbors toghi2019spatio ; shah2019real . By using V2V
communication CAVs can create an extended perception that facilitates explicit
cooperation among vehicles to overcome the limits of a non-cooperative agent
valiente2019controlling ; razzaghpour2021impact ; valiente2020dynamic . While
planning in a fully AV scenario is relatively easy to achieve, coordination in
the presence of HVs is a significantly more challenging task, as the AVs not
only need to react to road objects but also need to consider the behaviors of
HVs aoki2020cooperative ; toghi2021cooperative ; Jami2022AugmentedDB .

In contrast to the individual non-cooperative approaches, we investigate the
mixed-autonomy decision-making challenge from a multi-agent and social
perspective. Rewarding AVs for adopting an altruistic behavior and taking into
consideration the interests of other vehicles allows them to see the broad
picture and find solutions that maximize the utility of the group. In addition
to the potential benefits of altruistic decision-making in terms of safety and
efficiency, altruism results in more societally advantageous outcomes
schwarting2019social . Figure 1(a) demonstrates how a group of AVs can guide
HV to increase safety and efficiency, while Figure 1(b) and 1(c) show how AVs
can collaborate to accomplish a social objective that benefits another HV or
AV.

Currently, AVs lack an understanding of human behavior and frequently act
extremely cautiously to avoid collisions. This conservative behavior not only
leaves AVs unprotected from aggressive HVs, but also results in unexpected
reactions that confuse HVs, creating bottlenecks in traffic flow and causing
accidents. It’s critical to distinguish between a human driver’s individual
traits, such as aggressiveness, conservativeness, and risk tolerance, and
their social preferences, such as egoism and altruism. Even though the two
categories are related, they have distinctive natures and so behave
differently in mixed traffic. An aggressive driver, for example, is not
inherently egoistic or selfish, but their aggression may hinder their ability
to collaborate with other drivers and participate in a socially desirable
coexistence with AVs sagberg2015review ; harris2014prosocial ;
vallieres2014intentionality . In the field of behavior planning for AVs in
mixed-autonomy traffic, we identify two fundamental problems. First, human
drivers differ considerably in their individual traits and social preferences,
making AV behavior planning exceedingly difficult because it is difficult for
the AV to foresee the type of behavior it would encounter when dealing with a
human driver. Furthermore, relying on a real-time inference of HV behaviors is
not always feasible because vehicle interactions can be brief, such as when
two vehicles meet at an intersection. Second, driving requires complex
interactions of agents in a partially observable and non-stationary
environment, as HVs do not follow a fixed policy and modify their policies in
real-time in response to the actions of other vehicles.

The integration of AVs into the real world requires them to address those
challenges. Due to the differences in maneuverability and reaction time
between AVs and HVs, a road shared by both becomes a competitive situation. In
contrast with the full-autonomy case, here the coordination between HVs and
AVs is not as straightforward since AVs do not have an explicit means of
harmonizing with humans and are therefore required to locally account for the
other HVs and AVs in their proximity. This dilemma intensifies if AVs act
egoistically and optimize solely for their local utility. As an illustration,
Figure 2 and Figure 3 demonstrate a highway exiting and merging scenario in
mixed traffic. We consider a general setup where AVs and HVs with different
behaviors coexist. Vehicles need to efficiently merge onto the lane or exit
the highway without collisions. In an ideal cooperative environment, AVs
should proactively decelerate or accelerate to provide sufficient room for
vehicles to safely exit/merge and prevent hazardous situations, while also
being resilient to various situations and behaviors and assuring safety in
decision-making bouton2019cooperation . For instance, in Figure 2 (merging
scenario) if AVs act egotistically, the merging vehicle must rely on the HV to
slow down to allow it to merge. However, due to the unpredictability of HVs,
relying solely on HVs might result in suboptimal or even dangerous
circumstances. Therefore, if all AVs act egotistically, the merging vehicle
would either be unable to join the highway or it will wait for an HV and risk
cutting into the highway without knowing whether the HV will slow down.
Nevertheless, if AVs act altruistically, they can coordinate to guide the
traffic on the highway to allow for a seamless and safe merging. In
particularly challenging driving scenarios, such altruistic AVs can achieve
societally beneficial results without relying on or making assumptions about
HVs behaviors.

Figure 2: For a seamless and safe highway merging, all AVs must coordinate and
account for the utility of HVs. _(top)_ Egoistic AVs optimize only for their
own utility, _(bottom)_ Altruistic AVs consider also the HV’s utility. Figure
3: Highway exiting and merging scenarios with AVs (green) and aggressive HVs
(red) sharing the road. Altruistic AVs must learn to cooperate to exit/merge
successfully and safely while being adaptable to a variety of scenarios and HV
behaviors.

To address these challenges, existing literature either depend on models of
human behavior generated from pre-recorded driving datasets sadigh2016planning
; wu2017emergent  or define social utilities that can impose cooperative
behavior among AVs and HVs toghi2021social . Other works focus on rule-based
methods that use heuristics and hand-coded rules to guide the AVs
rios2016survey  or probabilistic driver modeling mahjoub2019representing
learned from human driving data. While this is feasible for simple situations,
these methods become impractical in complex scenarios. Additionally, the human
driver models learned in the absence of AVs, are not necessarily valid when
humans confront AVs. This limits the application of the generated solutions,
as they are frequently limited to the human behaviors with which AVs
interacted during training. To account for this, several works in the
literature adopt an excessively cautious approach when interacting with humans
li2018safe . This strategy not only leaves the AVs vulnerable to other
aggressive drivers, particularly in competitive situations, but it also causes
traffic congestion and significant safety risks cosgun2017towards ;
schwarting2019social .

On the other hand, data-driven methods such as reinforcement learning (RL)
have received increased attention lin2020anti  as RL-based methods can learn
decision-making and driving behaviors that are hard for traditional rule-based
designs. However, the majority of the RL approaches are designed for a single
AV, or try to handle the interaction between AVs and HVs either by predicting
human behavior or by relying on the fact that humans are willing to
collaborate or can be influenced to do so sadigh2016planning ;
sadigh2018planning , which could compromise safety or lead to sub-optimal
performance. Recent works consider social interactions of AVs and train
altruistic AVs that learn from experience and influence HVs to optimize a
social utility function that benefits all vehicles on the road
toghi2021altruistic ; toghi2021cooperative .

In contrast, we consider a data-driven multi-agent reinforcement learning
(MARL) approach and let the autonomous agents implicitly learn the decision-
making process of human drivers only from experience, while optimizing for a
social utility. By incorporating a cooperative reward structure into our MARL
framework, we can train AVs that coordinate with each other, sympathize with
HVs, and, as a result, demonstrate enhanced performance in competitive driving
scenarios, such as highway exiting and merging. Despite not having access to
an explicit model of the human drivers, the trained autonomous agents learn to
implicitly model the environment dynamics, including the behavior of human
drivers, which enables them to interact with HVs and guide their behavior.

This research aims to create a safe and robust training regimen that allows
AVs to collaborate and influence the behavior of human drivers to achieve
socially desirable outcomes, regardless of HV individual traits and social
preferences. We based our work on the following insights. First, we rely on a
decentralized reinforcement learning architecture that optimizes for a social
utility that learns from experience and exposes the learning agents to a wide
range of driving. As a result, the agents become more resistant to human
driver behavior and can handle cooperative-competitive behaviors regardless of
HV’s hostility or social preference. Second, a safety prioritizer is presented
to minimize high-risk actions that could jeopardize driving safety. The safety
prioritizer constrains the policy of cooperative AVs to ensure the safety of
their behavior via masking the Q-states that lead to high-risk outcomes.
Figure 4 shows an overview of our process.

Our main contributions are summarized as follows:

  * •

We formulate the mixed-autonomy problem as a decentralized MARL problem and
present an approach to training altruistic agents which utilizes a
decentralized reward mechanism for achieving socially advantageous behaviors
and takes advantage of a 3D convolutional deep reinforcement learning
architecture to capture the temporal information in driving data.

  * •

A training algorithm is proposed to make AVs robust to different drivers’
behavior and situations while producing socially desirable outcomes. We
investigate the effect of HVs behaviors on our altruistic AVs agents and
especially conclude that the higher the traffic aggressiveness, the higher the
importance of social coordination.

  * •

We investigate the scenarios in which altruistic AVs can learn cooperative
policies that are robust to diverse traffic scenarios and HV behaviors without
compromising efficiency and safety, and present the results on transfer
learning and domain adaptation in mixed-autonomy traffic.

The purpose of this chapter is to study the challenges of robust and safe AVs
in mixed-autonomy traffic, especially in intrinsically competitive driving
scenarios like those shown in Figure 2, in which coordination is essential for
safety and efficiency. The intention is to utilize the autonomous driving
challenge as a case study to examine the use of social theories from
psychology literature in the MARL domain. To apply these theories to real-
world roads, more study is required. Nonetheless, the research on altruistic
AVs that are robust, safe, and capable of learning to influence HVs in
desirable ways, without the limitations of current solutions are promising.

Figure 4: An overview of our approach to leverage social awareness and
coordination to improve the safety and reliability of CAVs. Our social-aware
AVs learn from scratch not only to drive but also to understand the behavior
of HVs and coordinate with them, they learn to adapt and influence HVs in a
robust and safe manner.

##  0.2 Related Work

###  0.2.1 Multi-agent Reinforcement Learning

The intrinsic non-stationarity of the environment is a key problem for MARL.
To address those limitations a MARL derivation of importance sampling is
proposed and used to remove the outdated samples from the replay buffer
foerster2017learning . In xie2020learning  is presented another solution to
address this issue by including latent representations of partner strategies
allowing partner modeling and more scalable MARL.

To mitigate the problem of credit assignment in multi-agent systems,
foerster2018counterfactual  proposed the counterfactual multi-agent (COMA)
algorithm, which employs a centralized critic and decentralized actors. In
egorov2016multi  is proposed a deep RL algorithm with full environment
observability and a centralized controller to govern the joint-actions of all
the agents. Other current research on mixed-autonomy focuses on addressing
cooperative and competitive challenges by assuming the nature of interactions
between autonomous agents omidshafiei2017deep . In lowe2017multi  a variation
of an actor-critic approach with a centralized q-function is proposed. The
algorithm has access to local observations and the actions of all agents. In
our work, in contrast, we consider a decentralized controller with partial
observability, and train altruistic agents that optimize for a social utility.

###  0.2.2 Driver Behavior and Social Coordination

Existing works on driver behavior and social navigation approach agents
coordination by either modeling driver behaviors brown2020taxonomy ;
ivanovicgenerative ; mahjoub2019representing  or simplifying and making
assumptions about the nature of agent interactions lauer2000algorithm ;
omidshafiei2017deep . In toghi2020maneuver  is presented a maneuver-based
dataset and a model for classifying driving maneuvers is proposed. Other works
on driver behavior modeling consider graph theory chandra2020cmetric , data
mining constantinescu2010driving , driver attributes beck2014distress  or game
theory schwarting2019social . In ivanovicgenerative  is proposed a method for
modeling and forecasting human behavior in circumstances that involve multi-
human interactions in highly multi-modal situations.

Current research in social navigation has demonstrated the importance of AVs
as social actors and the advantages of coordination between AVs and HVs
pokle2019deep . Human driving patterns are learned from demonstration using
inverse RL in kuderer2015learning  and sadigh2018planning . Similarly, in
hadfield2016cooperative  is presented a centralized game-theory model for
cooperative inverse RL. The authors in trautman2010unfreezing  and
nikolaidis2015efficient  proposed a shared reward function to enable
cooperative trajectory planning for robots and humans. Sadigh  _et al._
presents a strategy based on imitation learning to learn a reward function for
human drivers, demonstrating how AVs can influence human actors
sadigh2016planning . The importance of coordination and the advantage of using
AVs to guide the traffic has been also investigated at the traffic level. Wu
_et al._ wu2018stabilizing  analyzes the capability of AVs to stabilize a
system of HVs and presents the conditions in which when concurrently enforcing
safety constraints on the AVs while stabilizing traffic improves traffic
performance. Similar works have highlighted the potential of influencing HVs
and how AVs can be used to stabilize and guide the traffic flow
wu2018stabilizing ; lazar2019learning . Recent works focus on optimizing
traffic networks in mixed autonomy to reduce traffic congestion and improve
safety. In biyik2021incentivizing  is presented a model of vehicle flow and a
model of how AV makes decisions among routes with various prices and
latencies. The planner optimizes for a social objective and shows improvement
in traffic efficiency. The vehicle routing problem is studied in
li2021learning  that proposed an innovative learning-augmented local search
system to mitigate the problem by using a Transformer architecture. Cameron
_et al._ explores how humans can supervise agents in order to attain an
acceptable degree of safety hickert2021cooperation . In contrast to previous
works, we do not rely on human cooperation and our AVs learn cooperative
behaviors directly from experience, our focus is on the emerging altruistic
behavior that allows agents to coordinate and optimize for a social utility.

###  0.2.3 Safe and Robust Driving

Safety is critical for AVs Safety_ECC , and it is especially important for AVs
that have been trained via RL. We must prioritize safety; because coordination
is frequently associated with risk. In cooperative driving, there are often
safe actions that have low rewards and riskier actions with higher rewards
wang2021emergent ; however, the risky action increases the likelihood of
crashes when cooperation fails. Especially, AVs utilizing trained RL
algorithms, may not always operate safely since the trained models may pick
dangerous actions li2018safe . Several attempts in this direction use pure
reward shaping to avoid collisions. While this is a frequent technique in RL,
safety is not implicitly emphasized, and AVs implementing such RL methods may
not behave properly in some cases due to function approximation.

To overcome this problem, the concept of safe RL is proposed in li2018safe ,
which aims to increase safety in unobserved driving conditions when the RL
algorithm performs dangerously. wang2019lane  proposes a rule-based decision-
making system that evaluates the controller’s decisions and substitutes
collision-causing actions. A short-horizon safety supervisor is included in
Nageshrao  _et al._ nageshrao2019autonomous  to replace unsafe actions with
safer ones. A Q-masking strategy is presented in
mohammadhasani2021reinforcement  to prevent collisions by deleting actions
that might lead to a crash. Chen  _et al._ proposes a novel priority-based
safety supervisor that reduces collisions considerably chen2021deep .

We leverage these approaches in this work using a decentralized reward
function, local actions, and assuming partial observability, to increase the
altruistic agents’ safety while also being adaptable to varied driver
behaviors and circumstances. As shown in Figure 2, we analyze a particular
situation in which AVs and HVs with various characteristics coexist. The
picture depicts two frequent traffic situations in which vehicles must either
merge into a lane effectively or depart the highway without colliding with
other vehicles. In an ideal cooperative context, vehicles should proactively
decelerate or accelerate to provide enough room for vehicles to safely
exit/merge and prevent stalemate situations, while also being resilient to
various conditions and behaviors and assuring safety in decision-making.

##  0.3 Preliminaries and Formalism

We study safety and robustness in the maneuver-level decision-making problem
for AVs to see what kinds of behaviors might lead to socially desirable
results. We’re interested in the question of how AVs can be trained from
scratch to drive safely and reliably, while also taking into account the
social aspects of their mission, i.e., optimizing for a social utility that
takes into account the interests of other vehicles in the vicinity. Social
awareness and coordination are essential to improve safety and reliability on
the roads. In this work, we explore that insight. Thus, we continue this
section by providing a quantitative description of an agent’s level of
altruism and formally defined our problem.

It is possible to define the MARL problem as a centralized or decentralized
problem. It’s simple to create a centralized controller that provides a
central joint reward and joint action. However, in the real world, such
assumptions are unfeasible. In this chapter, we focus on a decentralized
controller with partial observability and formulate the problem as a partially
observable stochastic game (POSG) defined by
$\langle\mathcal{I},\mathcal{S},P,\gamma,\\{\mathcal{A}_{i}\\}_{i\in\\{1,...,N\\}},\\{\mathcal{O}_{i}\\}_{i\in\\{1,...,N\\}},\\{R_{i}\\}_{i\in\\{1,...,N\\}}\rangle$
where

  * •

$\mathcal{I}$: a finite set of agents $N\geq 2$.

  * •

$\mathcal{S}$ : a set of possible states that contains all configurations that
$N$ AVs can take (probably infinite).

  * •

$P$: a state transition probability function from state $s\in\mathcal{S}$ to
state $s^{\prime}\in\mathcal{S}$, $P(S=s^{\prime}|S=s,A=a)$.

  * •

$\gamma$: a discount factor, $\gamma\in[0,1]$.

  * •

$\mathcal{A}_{i}$: a set of possible actions for agent $i$.

  * •

$\mathcal{O}_{i}$: a set of observations for agent $i$.

  * •

$R_{i}$: a reward function for the $i^{th}$ agent, $R_{i}(s,a)$.

At a given time $t$ the agent senses the environment and receives a local
observation $o_{i}:\mathcal{S}\rightarrow\mathcal{O}_{i}$, based on the
observation $o_{i}$ and its stochastic policy
$\pi_{i}:\mathcal{O}_{i}\times\mathcal{A}_{i}\rightarrow[0,1]$, the agent
takes an action within the action-space $a_{i}\in\mathcal{A}_{i}$.
Consequently, the agent transits to the next state $s^{\prime}$ which is
determined based on the state transition probability function
$P(s^{\prime}|s,a):\mathcal{S}\times\mathcal{A}_{1}\times...\times\mathcal{A}_{N}\rightarrow\mathcal{S}$
and receives a decentralized reward
$r_{i}:\mathcal{S}\times\mathcal{A}_{i}\rightarrow\mathbb{R}$. The goal of
each agent $i$ is to optimally solve the POSG by deriving a probability
distribution over actions in $\mathcal{A}$ at a given state, that maximizes
its cumulative discounted sum of future rewards over an infinite time horizon
and find the corresponding optimal policy
$\pi^{*}:\mathcal{S}\rightarrow\mathcal{A}$.

An optimal policy maximizes the action-value function, i.e.,

$\pi^{*}(s)=\arg\max_{a}Q^{*}(s,a)$ (1)

where,

$Q^{\pi}(s,a)\coloneqq\mathbb{E}_{\pi}[\sum_{k=0}^{\infty}\gamma^{k}R_{k}(s,a)|s_{0}=s,a_{0}=a].$
(2)

The optimal action-value function is determined by solving the Bellman
equation,

$Q^{*}(s,a)=\mathbb{E}\left[R(s,a)+\gamma\max_{a^{\prime}}Q^{*}(s^{\prime},a^{\prime})|s_{0}=s,a_{0}=a\right]$
(3)

###  0.3.1 Double Deep Q-Network

Deep Q-network (DQN) has been widely used in RL problems. DQN uses a deep
neural network (NN) with weights w as the function approximator to estimate
the state-action value function, i.e., $\tilde{Q}(.;\textbf{w})\cong Q(.)$.
DDQN improves DQN by decomposing the max operation in the target into action
selection and action evaluation, mitigating the over-estimation problem. The
idea is to periodically sample data from a buffer and compute an estimate of
the Bellman error or loss function, written as

$\mathcal{L}(\textbf{w})=\mathbb{E}_{s,a,r,s^{\prime}\sim\mathcal{RM}}[(Target-\tilde{Q}(s,a;\textbf{w}))^{2}]$
(4)
$Target=R(s,a)+\gamma\tilde{Q}(s^{\prime},\underset{a^{\prime}}{\arg\max}\tilde{Q}(s^{\prime},a^{\prime};\textbf{w});\hat{\textbf{w}}))$
(5)

The DDQN algorithm then performs mini-batch gradient descent steps as
$\textbf{w}_{i+1}=\textbf{w}_{i}-\alpha_{i}\hat{\nabla}_{\textbf{w}}\mathcal{L}(\textbf{w})$,
on the loss $\mathcal{L}$ to learn the approximation of the value function
($\tilde{Q}(.)$). The $\hat{\nabla}_{\textbf{w}}$ operator denotes an estimate
of the gradient at $\textbf{w}_{k}$, w are the weights of the online network
and $\hat{\textbf{w}}$ are the weights of the target network which are updated
at a lower frequency ($Target_{update}$) to stabilize training. The experience
replay buffer ($RM$) is used to generate training samples
$(s,a,r,s^{\prime})$, which are randomly drawn to protect from correlated
observations and non-stationary data distribution.

###  0.3.2 Driving Scenarios

Our objective is to investigate driving scenarios in which the lack of AV
coordination hinders safety and efficiency. We also study adaptability among
scenarios and driver behaviors. For this, we design a set of scenarios
$\mathcal{F}$ with highway exiting and merging ramps as the main scenarios, as
shown in Figure 2, where a mission vehicle (in our case an exiting/merging
vehicle) attempts to accomplish its task in a mixed-traffic environment.

The exiting and merging scenarios are designed in such a way that coordination
is necessary for safety. AVs must coordinate, and neither can achieve a safe
and smooth traffic flow on its own, i.e., exiting/merging will not be feasible
without the coordination of the other AVs. To facilitate safe exiting/merging
while also responding to varied traffic scenarios, altruistic AVs must learn
to account for the interests of all vehicles, coordinate, make compromises,
and influence human behavior. In Figure 2, for example, the AV1 has to
compromise its own utility and reduce speed to guide the traffic of aggressive
HVs, creating space for the exiting/merging vehicle, while the other AVs have
to increase speed to create room for the mission vehicle. The exiting and
merging scenarios are defined as $f_{e},f_{m}\in\mathcal{F}$ correspondingly.
We particularly chose those scenarios as a case of study because of their
intrinsic similarity and the need for coordination, as the exiting/merging
vehicle’s utility contrast with that of the HV highway vehicles.

###  0.3.3 Social Value Orientation for AVs

In this section, we introduce Social Value Orientation (SVO) to formally
investigate the social conflicts between humans and agents in diverse
environments. It is critical to quantify an individual’s social preference to
understand whether they would cooperate or not in a particular scenario, such
as opening a gap in our highway merging example. For that purpose, SVO is a
commonly used concept in the social psychology literature that has lately been
applied in robotics research schwarting2019social . In our context, SVO
defines the degree of an agent’s egoism or altruism toward others. Based on
the value placed on the utility of others, an HV or an AV’s behavior can range
from egoistic to completely altruistic. We rely on AVs to guide traffic toward
more socially advantageous outcomes since the SVO of HV is unknown. In formal
terms, an AV’s SVO angle $\phi$ determines how the AV balances its own reward
against that of others toghi2021cooperative ; toghi2021social ;
le2022cooperative . In terms of rewards, an AV’s total reward $R_{i}$ is
defined as:

$R_{i}=r_{i}\cos\phi_{i}+r^{-}_{i}\sin\phi_{i}$ (6)

where $r_{i}$ is the agent’s individual utility, $r^{-}_{i}$ is the total
utility of other agents from the perspective of the $i$th agent which in
general is a function $f(.)$ of their individual utilities,

$r^{-}_{i}=f(r_{j}),\quad\text{where }j\neq i$ (7)

The SVO angle can varied from $\phi=0$ (entirely selfish) to $\phi=\pi/2$
(entirely altruistic). Nonetheless, none of the limits are optimal, and a
point in the middle, known as the optimal SVO angle $\phi^{*}$ gives the most
socially favorable outcome. SVO allows us to understand the behaviors that
make possible the socially desirable outcomes in Figure 2.

Autonomous agents must be aware of human drivers’ social preferences as well
as their desire to collaborate. Humans, on the other hand, are known to be
diverse in SVO, and so their preferences are uncertain murphy2015social .
Figure 5 depicts a range of altruism across individuals with varying SVO. As a
result of the wide range of altruistic behavior seen in humans, is not safe to
rely on humans to guide the traffic, instead, we should rely on AV to guide
the traffic toward more socially advantageous goals. Therefore, our objective
is that the AVs learn to create alliances and influence HV behavior to improve
the global utility of the group.

Figure 5: The SVO angle $\phi$ quantifies the level of altruism of an agent.
In the figure the diameter of the circles, represents the size of the human
population that holds the associated SVO garapin2015does .

###  0.3.4 Autonomous Vehicles as Social Actors

AVs in a mixed environment will be social actors in the traffic road that will
react to HVs and influence and adapt to their behaviors. The traffic
environment is rich with habits and expectations, that determine driver
behaviors. The vehicle’s interaction will be influenced by the way AV
decisions are perceived socialmuller ; schwarting2019social . For instance,
some human drivers may be grateful if the AV stops for them but frustrated if
it does not perform as expected. Also, they might behave aggressively if
they’re stuck behind an overly cautious AV, which reduces speed constantly.
Another example is the case that when crossing a street while a vehicle is
waiting, pedestrians move faster (a gesture of respect for the driver). On the
other hand, will pedestrians speed up for an AV, or will they behave
differently? If an AV is understood as a social actor, the HVs will learn the
individual and social traits of AVs and behave accordingly in mutual
interactions. This would fit with current preconceptions that make assumptions
about drivers based on the brand and type of vehicle they drive. Current AVs’
driving is as conservative as possible to ensure safety. They will slow down
in front of a crossing because they believe the other vehicle will want to go
first, even though this is against the law. They wait for pedestrians when in
doubt. It’s not difficult to see how other agents and HVs could take advantage
of and exploit these over-conservative behaviors. As AVs are going to be
social actors in mixed autonomy traffic, the safety and reliability of AVs
will be coupled with their social awareness and their ability to engage in
complex social interactions. We consider risk awareness and social behavior as
fundamental traits for decision-making.

Failure to identify social cues of selfishness or collaboration by an AV has
ramifications for the general flow of the traffic network, as well as the
safety of traffic participants. Current AVs ignore social signs and driver
personality in favor of explicit communication or driver modeling. Because
these methods can’t handle complicated interactions, they tend to be
conservative, restricting autonomy solutions to simple road interactions
socialmuller ; schwarting2019social . The ability of AVs to drive in a
socially obedient manner is critical for the safety of passengers and other
vehicles because predictable behavior allows humans to comprehend and respond
appropriately to the AV’s actions.

###  0.3.5 Driving Behaviors

The problem of simulating varied behaviors may be defined as determining the
appropriate range of parameters to produce heterogeneous behaviors within the
simulator. Some works in social traffic psychology show that driving behavior
falls between conservative and aggressive. Nevertheless, the specific
definition is still under discussion and fluctuates across works
sagberg2015review . The phrase ”aggressive driving” refers to a wide range of
unsafe driving practices, including running red lights and speeding. The root
of aggressive driving has a variety of factors that aren’t necessarily clear.
Some are caused by hazardous road conditions, while others are caused by
personal characteristics or mental states. Moreover, there is a correlation
between aggressiveness and egoism, as egoistic drivers are less likely to
yield and have a tendency to over-speeding and engage in unsafe actions. While
there is a correlation between these concepts harris2014prosocial ;
vallieres2014intentionality , we distinguish aggressiveness from egoism in
this study by describing individual traits and social preferences.

In this work, we discriminate between individual traits and social preferences
because they result in different behaviors. We define altruism and egoism as
social preferences; in that sense, an egoistic driver is a selfish driver who
accounts for his personal utility irrespective of his aggression. We define
conservatism and aggressiveness as individual traits, and we describe an
aggressive driver as someone whose actions result in aggressive behavior.
Individual traits such as aggressiveness are characterized by the outcomes of
their actions, but social preferences such as egoism are distinguished by
their social objectives and purposes. In this direction, an egoistic driver is
a self-centered driver who lacks social motive, a driver who believes he
controls the road and disregards the other drivers. Egoist drivers frequently
engage in violent actions, and while ego defensiveness is not the primary
source of aggression, it is a major contributor to aggressive driving
harris2014prosocial ; vallieres2014intentionality . Despite their
similarities, the two groups have different origins and result in different
behaviors. A driver, for example, could be egoistic and conservative. We may
envision a driver who drives cautiously to protect himself (selfish
motivation/preference) and, as a result, is conservative in his behavior
(outcome of his actions).

Properly, we described social preferences (altruism or egoism) by the AV’s SVO
angular phase $\phi$; and individual traits (conservativeness and
aggressiveness) by the HV driver model parameters ($\mathcal{P}$) as described
in section 0.5.5. Based on the values of these parameters, a driver will
behave conservative or aggressive. In the simulations, the AVs have no access
to HVs’ SVO, we consider the SVO of HVs to be undetermined as they cannot
communicate that directly. Finally, we define a set of behaviors
$\mathcal{B}$, i.e, aggressive, moderate and conservative,
$b_{a},b_{m},b_{c}\in\mathcal{B}$ based on the parameters ($\mathcal{P}$)
obtained in section 0.5.5.

##  0.4 Problem Formulation

We investigate the safety and robustness of the scenarios described in Figure
2, an exiting/merging vehicle, which can be either HV or AV. This
configuration contains a group of AVs that hold the same SVO, as well as a
group of HVs which are heterogeneous in their SVO, making it unclear whether
they are allies or opponents. Formally, the road is shared by a set of HVs
$h_{k}\in\mathcal{H}$, with an undetermined SVO $\phi_{k}$ and heterogeneous
behaviors $b_{k}\in\mathcal{B}$; a set of AVs $i_{i}\in\mathcal{I}$, that are
connected together using V2V communication, controlled by a decentralized
policy and sharing the same SVO, and a _mission vehicle_ ,
$M\in\mathcal{I}\cup\mathcal{H}$ that is aiming to accomplish its mission
(highway exiting/merging) and it can be either AV or HV. We focus on the
multi-agent maneuver-level decision-making problem for AVs in mixed-autonomy
environments and study the following problems: how AVs can learn in a mixed-
autonomy environment optimal cooperative policies $\pi^{*}(s)$ that are robust
to different scenarios $f\in\mathcal{F}$ and behaviors $b\in\mathcal{B}$ while
ensuring safety on the decision-making, and how sensitive is the performance
of the altruistic AVs to the HVs’ behaviors.

As AVs are connected, we assume that they receive an accurate local
observation of the environment
$\tilde{\textbf{o}}_{i}\in\widetilde{\mathcal{O}}_{i}$, sensing all the
vehicles within their perception range, i.e, a subgroup of HVs
$\widetilde{\mathcal{H}}\subset\mathcal{H}$ and a subgroup of AVs
$\widetilde{\mathcal{I}}\subset\mathcal{I}$. Nevertheless, AVs are unable to
share their actions or rewards, and they take individual actions from a set of
high-level actions $a_{i}\in\mathcal{A}_{i}(|\mathcal{A}_{i}|=5)$. The goal of
this work is to train social-aware AVs that learn how to drive in a mixed-
autonomy scenario in a robust, efficient, and safe manner. We are interested
in how to obtain a utility function that enables AVs to handle competitive
driving scenarios (such as those in Figure 2) and leads them into socially-
desirable decisions that improve traffic efficiency, safety, and robustness.

##  0.5 Safe and Robust Social Driving

In this section, we present the safe and robust MARL approach. Our approach
uses a general decentralized reward function that optimizes for social utility
and induces altruism in the AVs; the general reward function accounts for any
anticipated vehicle’s mission, allowing it to be applied to a variety of
environments; and collisions are reduced by the safety prioritizer. What we
define as ”driving” is the outcome of decades of human learning from
experience. Consequently, we take the same approach and train AVs that learn
from experience and define the optimization problem as the eventual desirable
social outcome with adaptability, expecting AVs to learn how to drive safely
during the process. We carefully design a decentralized general reward
function, a suitable architecture, and a safety prioritizer to promote the
desired safe altruistic behavior in AVs’ decision-making process. The overview
of our approach as presented in Figure 4 and Figure 2 helps us to create
intuition on these points, by introducing driving scenarios in which
altruistic AVs lead to socially advantageous results while adapting to
different traffic scenarios.

Action Space: The goal of this research is to look at inter-agent and agent-
human interactions, as well as behavioral elements of mixed-autonomy driving.
Thus, we choose a more abstract level and define the action-space as a set of
discrete meta-actions $a_{i}\in\mathcal{A}_{i}$. In particular, we select a
set of five high-level actions $a_{i}$ as,

$a_{i}\in\mathcal{A}_{i}=\begin{bmatrix}\texttt{Lane Left}\\\ \texttt{Idle}\\\
\texttt{Lane Right}\\\ \texttt{Accelerate}\\\
\texttt{Decelerate}\end{bmatrix}$ (8)

These meta-actions are then converted into trajectories and low-level control
signals, which ultimately control the vehicle’s movement.

Observation Space: We use a multi-channel VelocityMap observation ($o_{i}$)
that embeds the relative speed of the vehicle with respect to the ego vehicle
in pixel values toghi2021social . We represent the information in multiple
semantic channels that embed: 1) an attention map to highlight the position of
the ego vehicle, 2) the HVs, 3) the AVs, 4) the mission vehicle, and 5) the
road layout. Figure 6 illustrates an example of this multi-channel
representation. In order to map the relative speed of the vehicles into
pixels, we use a clipped logarithmic function, which improves dynamic range
and yields better results than a linear map, i.e.,

$Z_{j}=1-\beta\log(\alpha|v_{j}^{(l)}|)\mathds{1}(|v_{j}^{(l)}|-v_{0})$ (9)

where $Z_{j}$ is the pixel value of the $j$th vehicle in the state
representation, $v^{(l)}$ is its relative Frenet longitudinal speed from the
$k$th vehicle’s point-of-view, i.e., $\dot{l_{j}}-\dot{l_{k}}$, $v_{0}$ is
speed threshold, $\alpha$ and $\beta$ are dimensionless coefficients, and
$\mathds{1}(.)$ is the Heaviside step function. Such non-linear mapping gives
more importance to neighboring vehicles with smaller $|v^{(l)}|$ and almost
disregards the ones that are moving either much faster or much slower than the
ego vehicle. As temporal information is necessary for safe decision-making, we
use a history of successive VelocityMaps observations to create the input
state to the Q-network.

Figure 6: Multi-channel VelocityMap state representation embeds the speed of
the vehicle in pixel values.

###  0.5.1 Distinguishing Sympathy from Cooperation

In our mixed-autonomy problem, we divide inter-agent relations into
interactions between autonomous agents (AV-AV interactions) and interactions
between autonomous agents and human drivers (HV-AV interactions). By
decoupling the two, we can analyze the interactions between human drivers with
unclear SVO and our autonomous agents in a methodical way. In that sense, we
define sympathy as the autonomous agent’s altruism toward a human, and
cooperation as the altruistic behavior among autonomous agents. The fact that
the components of altruism differ in nature is our reasoning for separating
them. Sympathy, for example, may not be reciprocated since agents differ in
their SVO, whereas cooperation among autonomous entities is fundamentally
homogeneous if they share the same SVO. Following this concept, we can rewrite
the AV reward in Eq. (6) as,

$\displaystyle R_{i}={}$ $\displaystyle
r_{i}\cos\phi_{i}+(\sin\theta_{i}R_{i}^{\mathrm{AV}}+\cos\theta_{i}R_{i}^{\mathrm{HV}})\sin\phi_{i}$
(10) $\displaystyle=$
$\displaystyle\underbrace{r_{i}\cos\phi_{i}}_{\mathrm{egoistic\,\,term}}+$
$\displaystyle\underbrace{\sin\theta_{i}\sin\phi_{i}R_{i}^{\mathrm{AV}}}_{\mathrm{cooperation\,\,term}}+\underbrace{\cos\theta_{i}\sin\phi_{i}R_{i}^{\mathrm{HV}}}_{\mathrm{sympathy\,\,term}}$

where $\theta$ is the sympathy angular phase determining the cooperation-to-
sympathy ratio. Parameters $R_{i}^{\mathrm{AV}}$ and $R_{i}^{\mathrm{HV}}$
denote the total utility of other AVs and HVs, respectively, as perceived from
the $i$th agent’s perspective. We expand on this topic in Section 0.5.2 where
we introduce the distributed reward structure.

###  0.5.2 Decentralized Social Reward

The AVs are trained using the partial local observations and the decentralized
reward function, and we anticipate them to learn how to drive in a variety of
settings while taking into consideration the individual diver’s missions. As a
result, we create a well-engineered general reward function that considers
social utility, traffic metrics, and individual diver’s missions. Following
the definition of sympathy and cooperation in equation (10) we decompose the
decentralized reward received by agent $I_{i}\in\mathcal{I}$ as,

$\displaystyle R_{i}(s,a)={}$ $\displaystyle
R^{\mathrm{ego}}+R^{\mathrm{social}}$ (11) $\displaystyle R^{\mathrm{ego}}={}$
$\displaystyle\cos\phi_{i}r_{i}(s,a)$ $\displaystyle R^{\mathrm{social}}={}$
$\displaystyle R^{\mathrm{coop}}+R^{\mathrm{symp}}$ $\displaystyle
R^{\mathrm{coop}}={}$
$\displaystyle\sin\theta_{i}\sin\phi_{i}\Big{[}\sum_{j}r^{\mathrm{AV}}_{i,j}(s,a)+\sum_{j}r_{i,j}^{M}(s,a)\Big{]}$
$\displaystyle R^{\mathrm{symp}}={}$
$\displaystyle\cos\theta_{i}\sin\phi_{i}\Big{[}\sum_{k}r^{\mathrm{HV}}_{i,k}(s,a)+\sum_{k}r_{i,k}^{M}(s,a)\Big{]}$

in which $R^{\mathrm{ego}}$, $R^{\mathrm{social}}$ represents the egoistic and
social reward, $i\in\mathcal{I}$,
$j\in(\widetilde{\mathcal{I}}\setminus\\{I_{i}\\})$,
$k\in\widetilde{\mathcal{H}}$. The term $r_{i}$ represents the ego vehicle’s
reward obtained from traffic metrics and the angle $\phi$ allows to adjust the
level of egoism or altruism. $R^{\mathrm{coop}}$ is the cooperation term (the
altruistic behavior among AVs, i.e, AV’s altruism toward others AVs) and
$R^{\mathrm{symp}}$ is the sympathy term (AV’s altruism toward HVs). The
sympathy reward term, $r^{\mathrm{HV}}_{i,k}$ considers the individual reward
of the HVs, while the cooperation reward term, $r^{\mathrm{AV}}_{i,j}$
considers the individual reward of the other AVs, and are defined as

$r^{\mathrm{HV}}_{i,k}=\frac{\mathcal{W}_{k}}{d_{i,k}^{\lambda}}\sum_{m}\omega_{m}x_{m}\quad
r^{\mathrm{AV}}_{i,j}=\frac{\mathcal{W}_{j}}{d_{i,j}^{\lambda}}\sum_{m}\omega_{m}x_{m}$
(12)

in which $d_{i,k}/d_{i,j}$ represents the distance between the agent and the
corresponding HV/AV, $\lambda$ is a dimensionless coefficient,
$\mathcal{W}_{k}$ is a weight value for individual vehicle’s importance, $m$
is the set of traffic metrics that have been considered in the vehicle’s
utilities (speed, crashes, etc.), in which $x_{m}$ is the $m$ metric
normalized value and $w_{m}$ is the weight associated to that metric. The term
$r^{\mathrm{M}}$ accounts for the reward of the vehicle’s mission. A mission
is defined as any desired specific outcome for a particular vehicle, as
merging, exiting, etc.

$r^{\mathrm{M}}_{i,j}=\begin{cases}\frac{w_{j}}{(d_{i,j})^{\mu}},&\mathrm{if}g(j)\\\
0,&\mathrm{o.w.}\end{cases}\quad
r^{\mathrm{M}}_{i,k}=\begin{cases}\frac{w_{k}}{(d_{i,k})^{\mu}},&\mathrm{if}g(k)\\\
0,&\mathrm{o.w.}\end{cases}$ (13)

The function $g(v)$ is an independent function to evaluate the mission; $g(v)$
returns true if the vehicle $v$ has a mission defined and the mission has been
accomplished in the recent time window. $\mu$ is a dimensionless coefficient,
$w_{j}/w_{k}$ are weights for an individual vehicle’s mission (importance of
the mission). This allows defining a general reward independent of the driving
scenario and mission goals for different vehicles. In the experiments, a HV
can be assigned a merging mission or a highway exiting mission, as referred to
in Figure 2.

###  0.5.3 Deep MARL architecture for Social driving

As shown in Figure 8, we leverage a 3D Convolutional Neural Network (CNN) with
a safety prioritizer for our MARL architecture. To account for the temporal
information, the 3D CNN operates as a feature extractor and leverages a
history of VelocityMap observations. The network receives a stack of 10
VelocityMap observations, i.e., a $10\times(4\times 512\times 64)$ tensor that
captures the latest 10 time-steps episodes. To mitigate the non-stationarity
issue in MARL, agents are trained in a semi-sequential manner, as illustrated
in Figure 7. The agents are trained independently for $N_{iterations}$
iterations while freezing the policies of the remaining AVs, $\textbf{w}^{-}$.
Subsequently, the other agents’ policies are updated with the new policy,
$\textbf{w}^{+}$.

Figure 7: The multi-agent training and policy dissemination process.

To improve safety we train our agents using a safety prioritizer that, in the
cases where the action selected by the agent policy is unsafe, selects a safe
action and stores the unsafe action ($a_{t}$) and the related state in the
$RM$ with a suitable penalty on the reward ($r_{unsafe}$) for the unsafe
state-action pair. The safety prioritizer reduces episode resets due to
imminent collisions improving sample efficiency. The unsafe state-action pairs
are not removed so the agent can also learn from unsafe experiences. The
experience $(\psi(s_{t}),a_{t},r_{unsafe},\emptyset$) is stored in $RM$ with a
terminal next state $\emptyset$, the target for this unsafe pair
$(s_{t},a_{t})$ is $Target(s_{t},a_{t})^{DDQN}=r_{unsafe}$. The details of the
safety prioritizer are given in the next section 0.5.4.

The proposed deep MARL architecture is described in Algorithm 1. As part of
the implementation, we start the learning process after the replay buffer has
been filled with a sufficient number of sample simulations. Furthermore, we
update the experience replay buffer to adjust for the extremely skewed
training data toghi2021social . Balancing skewed data is a frequent practice
in machine learning, and it was effective in our MARL problem.

Algorithm 1 Safety Prioritized Multi-agent DDQN

Initialize experience replay buffer $RM$.

Initialize $\tilde{Q}(.;\textbf{w}^{-})$ with random weights
$\textbf{w}^{-}=\textbf{w}_{ini}$

Initialize target network $\tilde{Q}(.;\hat{\textbf{w}})$ with weights
$\hat{\textbf{w}}=\textbf{w}^{-}$

Pre-store experience of first’s 50 episodes in $RM$

for $\mathrm{e}=50$ to $N_{\mathrm{episode}}$ do

Initialize $s_{1}=\\{\tilde{\textbf{o}}_{1}\\}$ and compute
$\psi_{1}=\psi(s_{1})$

for t = 1 to T do

for $I_{i}$ in $\mathcal{I}$ do

Freeze $\textbf{w}^{-}$ for all $I_{j}$, $j\neq i$

for $m=1$ to $N_{iterations}$ do

With probability $\epsilon$ select a random action $a_{t}$ ,

otherwise select $a_{t}=\max_{a^{\prime}\in
A}Q(\psi(s_{t}),a^{\prime};\textbf{w}^{+})$

if $a_{t}$ is unsafe (Algorithm 2) then

Store $(\psi_{t},a_{t},r_{unsafe},\emptyset$) in $RM$

$a_{t}$ = Compute a safe action (Algorithm 3)

end if

Execute safe action $a_{t}$ , and observe $r_{t},\tilde{\textbf{o}}_{t}$

Set $s_{t+1}=\\{s_{t},\tilde{\textbf{o}}_{t+1}\\}$ and
$\psi_{t+1}=\psi(s_{t+1})$

Store experience $(\psi_{t},a_{t},r_{t},\psi_{t+1})$ in $RM$

Sample a mini-batch of size $M$ from $RM$

Compute $\mathcal{L}(\textbf{w}^{+})$

Performs gradient descent

$\textbf{w}^{+}_{k+1}\leftarrow\textbf{w}^{+}_{k}-\alpha\hat{\nabla}_{\textbf{w}}\mathcal{L}(\textbf{w}^{+})$

end for

$\textbf{w}^{-}=\textbf{w}^{+}$ for all $I_{i}\in\mathcal{I}$

end for

Every $\quad Target_{update}\quad$ reset
$\hat{\textbf{w}}\leftarrow\textbf{w}^{-}$

end for

end for

###  0.5.4 Safety Prioritizer

We include a safety prioritizer to the MARL algorithm that penalizes and
reduce imminent crashes. This helps the agent to increase sample efficiency
during training and avoid collisions when in deployment. If the agent comes
into an unexpected situation and decides to perform a risky action, that
action will be prevented. The safety prioritizer enhances simulation results
and is crucial in real-world scenarios. The safety prioritizer included
Algorithm 2 and Algorithm 3.

Algorithm 2: During action selection of the agent $I_{i}$, once an action
$a_{t}$ is chosen, the safety prioritizer checks if the action is safe by
computing a safety score for $N_{steps}$ of planning. We utilize the time-to-
collision ($ttc$) as a safety score. If $safety_{score}<safe_{th}$ the action
is unsafe and we need to select a safe action. The selection of a safe action
is presented in Algorithm 3.

Algorithm 3: The safe action selection is different in training and testing.
During training, to encourage exploration, we remove the unsafe actions and
keep the random action selection following the current exploration policy on
the remaining actions. During testing, we follow the greedy policy in the
subset of safe actions
$a_{t}=\max_{a^{\prime}\in\widetilde{\mathcal{A}}_{safe}}Q(\psi(s_{t}),a^{\prime};\textbf{w})$.
It should be noted that the algorithm does not choose the safest of all
possible actions, as that action may lead to particularly conservative
behaviors that can compromise traffic efficiency; we instead remove the
imminent unsafe actions and follow the priority given by the learned
altruistic policy. If it happens that all possible actions are unsafe, we
return the action $a_{t}\in\mathcal{A}$ with the highest safety score. In that
way during training the constrained exploration will keep the agent from
taking unsafe actions which will lead to efficient sampling and more stable
learning; and during testing, the decision-making is based on the prosocial
learned policy with minimum intervention from the safety prioritizer,
achieving higher traveled distance while avoiding collisions.

Algorithm 2 Safety score

Simulate $I_{i}$ taking the action $a_{t}$

for $v$ in
$(\widetilde{\mathcal{I}}\cup\widetilde{\mathcal{V}})\setminus\\{I_{i}\\}$ do

Compute safety score of $I_{i}$, $v$ for $N_{steps}$ planning

if $safe_{score}$ $<safe_{th}$ then

Return unsafe

end if

end for

Return safe

Algorithm 3 Safe action

Initialize $\widetilde{\mathcal{A}}$ = $\mathcal{A}$

while $\widetilde{\mathcal{A}}$ is not empty do

if during training then

Select $a_{t}$ following the exploration policy on set
$\widetilde{\mathcal{A}}$

else if during test then

Select$a_{t}=\max_{a^{\prime}\in\widetilde{\mathcal{A}}_{safe}}Q(\psi(s_{t}),a^{\prime};\textbf{w})$

end if

if $a_{t}$ is safe (Algorithm 2) then

Return $a_{t}$

end if

end while

Return $a_{t}$ with highest safe score in $\mathcal{A}$

Figure 8: Deep MARL architecture with the safety prioritizer.

###  0.5.5 Modeling Driver Behaviors

We model the longitudinal movements of HVs using the Intelligent Driver Model
(IDM) treiber2000congested , while the lateral actions of HVs are based on the
MOBIL model kesting2007general . The MOBIL model considers two main criteria,

The safety criterion ensures that after the lane change, the deceleration of
the new follower $\mathrm{a}_{n}$ in the target lane does not exceed a safe
limit, i.e, $\mathrm{a}_{n}>-b_{\mathrm{safe}}$.

The incentive criterion determines the advantage of HV after the lane change,
quantified by the total acceleration gain, given by

$\mathrm{a}^{\prime}_{ego}-\mathrm{a}_{ego}+\sin\phi_{ego}\Big{(}(\mathrm{a}^{\prime}_{n}-\mathrm{a}_{n})+(\mathrm{a}^{\prime}_{o}-\mathrm{a}_{o})\Big{)}>\Delta
a_{th}$ (14)

where $\mathrm{a}_{o}$, $\mathrm{a}_{n}$ and $\mathrm{a}_{ego}$ represent the
acceleration of the original follower in the current lane, the new follower in
the target lane and the ego HV, correspondingly, and
$\mathrm{a}^{\prime}_{o}$, $\mathrm{a}^{\prime}_{n}$, and
$\mathrm{a}^{\prime}_{ego}$ are the equivalent accelerations considering that
the ego HV has changed the lane, $\sin\phi_{ego}$ is the politeness factor.
Finally, the lane change is performed if the safety and incentive criteria are
mutually satisfied.

The IDM Model determines the longitudinal acceleration of a HV
$\dot{v}_{\mathrm{k}}$ as follows,

$\dot{v}_{\mathrm{k}}=\mathrm{a}_{\mathrm{max}}\Big{[}1-\Big{(}\frac{v_{k}}{v_{\mathrm{k}}^{0}}\Big{)}^{\delta}-\Big{(}\frac{d^{*}(v_{k},\Delta
v_{k})}{d_{k}}\Big{)}^{2}\Big{]}$ (15)

in which $v_{k}$, $d_{k}$, $\delta$, $\Delta v_{k}$, $v_{\mathrm{0}}^{k}$
denote the speed, the actual gap, the acceleration exponent, the approach
rate, and the desired speed of the $k^{th}$ HV, respectively.

The desired minimum gap of the $k^{th}$ HV is given by,

$d^{*}(v_{k},\Delta v_{k})=d_{k}^{0}+v_{k}T_{\mathrm{k}}^{0}+\frac{v_{k}\Delta
v_{k}}{(2\sqrt{\mathrm{a}_{\mathrm{max}}.\mathrm{a}_{\mathrm{des}}})}$ (16)

where $T_{k}^{0}$, $d_{k}^{0}$, $\mathrm{a}_{\mathrm{max}}$, and
$\mathrm{a}_{\mathrm{des}}$ are the safe time gap, the minimum distance, the
comfortable maximum acceleration, and deceleration, correspondingly.

The typical parameters for the MOBIL model are $\sin\phi_{ego}=0.5$, $\Delta
a_{th}=0.1\frac{m}{s^{2}}$ and $b_{\mathrm{safe}}=4\frac{m}{s^{2}}$. Table 1
shows typically used parameters of the IDM model treiber2000congested .

Table 1: Common used parameters for the IDM model Parameter | $v^{0}$ | $T^{0}$ | $\mathrm{a}_{\mathrm{max}}$ | $\mathrm{a}_{\mathrm{des}}$ | $\delta$ | $d^{0}$
---|---|---|---|---|---|---
Value | 30 m/s | 1.5 s | 1 m/s2 | 1.5 m/s2 | 4 | 2 m

Heterogeneous Driver Behaviors. Although those parameters are typically used
for IDM and MOBIL models, they simulate just one behavior. In order to
generate diverse behaviors $\mathcal{B}$, we frame the task of simulating
diverse behaviors as the problem of obtaining the appropriate range of
parameters ($\mathcal{P}$) that can generate those behaviors. To achieve that,
we leverage a behavior classifier and iteratively simulate the parameters and
classify the behaviors, mapping parameters to behaviors. To classify the
behaviors we represent traffic using a traffic-graph at each time step $t$,
$\mathcal{G}_{t}$, with a set of edges $\mathcal{E}(t)$ and a set of vertices
$\mathcal{V}(t)$ as functions of time, i.e, the positions of vehicles
($\widetilde{\mathcal{H}}\cup\widetilde{\mathcal{I}}$) represent the vertices.
The adjacency matrix $A_{t}$ is given by $A(k,m)=d(v_{k},v_{m}),k\neq m$ , in
which $d(v_{k},v_{m})$ is the shortest travel distance between vertices $k$ to
$m$. Then we use centrality functions chandra2020cmetric  to classify the
behavior (level of aggressiveness) resulting from $\mathcal{P}$, and then use
those simulation parameters $\mathcal{P}$ to model behaviors within the
simulator with varying levels of aggressiveness. The centrality functions are
defined as,

Closeness Centrality: the discrete closeness centrality of the
$k^{\textrm{th}}$ vehicle at time $t$ is defined as,

$\mathcal{C}^{k}_{C}[t]=\frac{{N-1}}{\sum_{v_{m}\in\mathcal{V}(t)\setminus\\{v_{k}\\}}d_{t}(v_{k},v_{m})},$
(17)

where $N=|\widetilde{\mathcal{H}}\cup\widetilde{\mathcal{I}}|$. The more
central the vehicle is located, the higher $\mathcal{C}^{k}_{C}[t]$ and the
closer it is to all other vehicles.

Degree Centrality: the discrete degree centrality of the $k^{\textrm{th}}$
vehicle at time $t$ is defined as,

$\displaystyle\mathcal{C}^{k}_{D}[t]=\bigl{|}\\{v_{m}\in\mathcal{N}_{k}(t)\\}\bigr{|}+\mathcal{C}^{k}_{D}[t-1]$
(18) $\displaystyle\textrm{such that}\
(v_{k},v_{m})\not\in\mathcal{E(\tau)},\tau=0,\ldots,t-1$

in which $\mathcal{N}_{k}(t)=\\{v_{m}\in\mathcal{V}(t),\ A_{t}(k,m)\neq
0,\nu_{m}\leq\nu_{k}\\}$ represents the set of vehicles in the proximity of
the $k^{\textrm{th}}$ vehicle, given that $\nu_{m}\leq\nu_{k}$; and
$\nu_{m},\nu_{k}$ denote the velocities of the $m^{\textrm{th}}$ and
$k^{\textrm{th}}$ vehicles, $A_{t}(k,m)$ is the adjacency matrix. The more new
vehicles seen by vehicle $k$ that meet this condition, the higher
$\mathcal{C}^{k}_{D}[t]$.

With the centrality functions, we can measure the Style Likelihood Estimate
(SLE) for different driver styles chandra2020cmetric . We consider two SLE
measures. The SLE of overtaking and sudden lane changes ($SLE_{l}$) and the
SLE of overspeeding ($SLE_{o}$). The $SLE_{l}$ and $SLE_{o}$ can be computed
by measuring the first derivative of the centrality functions as,

$\textrm{SLE}_{l}(t)=\left\lvert\frac{\partial\mathcal{C}_{C}(t)}{\partial
t}\right\rvert\quad\textrm{SLE}_{o}(t)=\left\lvert\frac{\partial\mathcal{C}_{D}(t)}{\partial
t}\right\rvert$ (19)

The maximum likelihood $\textrm{SLE}_{\textrm{max}}$ is calculated as
$\textrm{SLE}_{\textrm{max}}=\max_{t\in\Delta t}{\textrm{SLE}}(t)$.

Using those functions, we can approximately quantify and classify driver
behaviors in our simulation. The intuition behind that is that an aggressive
driver may frequently overspeed or perform sudden lane changes; while
overspeeding the $\mathcal{C}_{D}(t)$ monotonically increases (higher
$\textrm{SLE}_{o}(t)$) and during sudden lane changes the slope and the
extrema of $\mathcal{C}_{C}(t)$ changes values. Thus higher values of
$\textrm{SLE}_{\textrm{max}}$ are related to increased levels of
aggressiveness. Conversely, conservative drivers are not inclined toward those
aggressive maneuvers, and the degree of centrality will be relatively flat,
thus $\textrm{SLE}_{o}(t)\approx 0$ for conservative drivers.

We use these metrics as approximations of the driver’s level of
aggressiveness. In order to compute the suitable values for our simulation, we
iteratively simulate the parameters from IDM and MOBIL models, and for each
set of parameters, we quantify the resulting behavior in the simulation (using
those metrics). A mapping of the parameters $\mathcal{P}$ to behaviors
(quantified in the simulation for those parameters). The estimated simulation
parameters that simulate conservative, moderate and aggressive behavior in our
scenarios are presented in Table 2.

Table 2: Estimated simulation parameters for conservative, moderate, and aggressive behaviors. Model | Parameter | Aggressive | Moderate | Conservative
---|---|---|---|---
MOBIL | $\sin\phi_{e}$ | 0 | 0.3 | 1
| $\Delta a_{th}$ | 0 $m/s^{2}$ | 0.1 $m/s^{2}$ | 0.4 $m/s^{2}$
| $b_{\mathrm{safe}}$ | 12.0 $m/s^{2}$ | 6.0 $m/s^{2}$ | 2.0 $m/s^{2}$
IDM | $T^{0}$ | 0.5s | 1s | 3s
| $d^{0}$ | 1 $m$ | 2 $m$ | 6.0 $m$
| $\mathrm{acc}_{\mathrm{max}}$ | 7.0 $m/s^{2}$ | 3.0 $m/s^{2}$ | 1.0 $m/s^{2}$
| $\mathrm{acc}_{\mathrm{des}}$ | 12.0 $m/s^{2}$ | 7.0 $m/s^{2}$ | 2.0 $m/s^{2}$

The desired velocity $v^{0}$ is set to $30m/s$ and the acceleration exponent
$\delta=4$.

###  0.5.6 Implementation and Computational Details

We customize the OpenAI Gym environment in leurent2019approximate  to suit our
particular driving situation and MARL problem. We design a merging ramp and
exiting highway scenario for our simulation running in python and used Pytorch
for the implementation of our safety prioritized MARL DDQN algorithm. Our
implementation on average uses 3.1GB of memory for 4 agents and 18 HVs using a
GPU NVIDIA Tesla V100. The training process is repeated several times to
ensure convergence of the experiments to a similar policy. The network is
trained for $N_{episodes}=10,000$ taking on average 8 hours. While each round
of $10,000$ training episodes in the Tesla V100 GPU takes around 8h, a full
forward pass during deployment for 4 simulated agents takes 15ms
(approximately 4ms per agent). We utilize 3,200 GPU hours for all our
simulation experiments. Table 4 lists our simulation and training hyper-
parameters.

In a real AV platform, each agent will receive a local observation of the
environment that will be used by our algorithm to compute the safe optimal
action based on the trained Q-network. The decision-making will take place on
each AV’s onboard computer; therefore, to verify the feasibility of the real-
time operation of our decentralized algorithm we tested a forward pass of the
Q-network during deployment in multiple hardware platforms. The results for
the different platforms are presented in Table 3, for instance, an online
forward pass of the network in the deployment phase using commodity GPU
hardware, i.e, an NVIDIA Jetson AGX platform will be around 32.9ms for each
agent.

Table 3: Computation time for each agent. Computing platform | Online forward pass time
---|---
NVIDIA Tesla V100 GPU | 3.7 ms
OnLogic Karbo 700 x2 | 65.2 ms
NVIDIA Jetson AGX Xavier GPU | 32.9 ms
NVIDIA Jetson TX2 GPU | 112.5ms

Table 4: List of hyper-parameters. Parameter | Value | Parameter | Value
---|---|---|---
$N_{\mathrm{episode}}$ | 10,000 |  $\epsilon$ decay | Linear
$RM$ buffer size | 8,000 | Initial exploration $\epsilon_{0}$ | 1.0
Batch size | 32 | Final exploration | 0.05
Learning rate $\alpha_{0}$ | 0.0005 | Optimizer | ADAM
$Target_{update}$ | 300 | Discount factor $\gamma$ | 0.95
$|\mathcal{H}|$ | $18$ | $|\mathcal{I}|$ | $4$

##  0.6 Experiments and Results

###  0.6.1 Manipulated Variables

We study how the $safe_{th}$, the _level of aggressiveness_ , the _traffic
scenarios_ ($f_{j}$) and the _HVs’ behaviors_ ($b_{k}$) impact the performance
of AVs. We consider the case in which the mission vehicle (exiting/merging) in
Figure 2 is _human-driven_ , $M\in\mathcal{H}$, and define the following
terms:

  * •

$AV_{S}$. Social AV ($\phi_{i}=\phi^{*}$) that act _altruistically_ in the
presence of diverse HVs behaviors $b\in\mathcal{B}$.

  * •

$AV_{E}$. Egoistic AV ($\phi_{i}=0$) that act _egoistically_ in the presence
of diverse HVs behaviors $b\in\mathcal{B}$.

with $\phi^{*}$ to be the optimal SVO angle tuned to reach the optimal level
of altruism as in toghi2021social .

###  0.6.2 Performance Metrics

The performance of our system is measured based on safety, efficiency,
altruistic performance gain ($PG$), and adaptation error
$\mathrm{A}_{\mathrm{error}}$. To measure safety, we compute the percentage of
episodes that encountered a crash ($C(\%)$). For efficiency, the average
traveled distance ($DT(m)$) of the vehicles and the number of missions
accomplished by the mission vehicle is used. The altruistic performance gain
is measured by computing the difference in the safety/efficiency performance
of $AV_{E}$ and $AV_{S}$, as

$PG_{safety}(\%)=\frac{(AV_{E})_{C(\%)}-(AV_{S})_{C(\%)}}{N_{Episodes}}$ (20)

$PG_{efficiency}(\%)=\frac{(AV_{S})_{DT(m)}-(AV_{E})_{DT(m)}}{(AV_{E})_{DT(m)}}$
(21)

Finally, the adaptation error is a weighted sum function of the safety
($C(\%)$) and efficiency ($DT(m)$) performance of the $AV_{S}$ when trained
and tested in different scenarios/behaviors. Defined as

$A_{error}(\%)=w_{s}\times(C(\%))+w_{e}\times 100(1-\frac{DT}{DT_{max}})$ (22)

such that an adaptation between different situations that result in $0\%$
crash and $DT=DT_{max}$ will have $\mathrm{A}_{\mathrm{error}}=0\%$.

###  0.6.3 Hypotheses

In this section we examine the following hypotheses

  * •

H1. _In a mixed-autonomy scenario, the higher the level of aggressiveness, the
bigger the impact of cooperation. We expect a higher performance gain ( $PG$)
when altruistic AVs face more aggressive environments_.

  * •

H2. _Altruistic AVs agents using the decentralized framework can adapt to
different driver behaviors and traffic scenarios without compromising the
overall traffic metrics. However, the higher the similarity of testing
scenarios to the ones seen during training (
$(f_{test},b_{test})\approx(f_{train},b_{train})$), the lowest adaption error
($\mathrm{A}_{\mathrm{error}}$)_.

  * •

H3. _We anticipate an improvement in both safety and efficiency with the
addition of the safety prioritizer. In the absence of a safety prioritizer (
$safe_{th}=0$) we expect that AVs will cause more crashes_.

###  0.6.4 Analysis and Results

Based on the hypotheses, we explore their correctness through the experiments
in this section.

#### Sensitivity Analyses

To study the hypothesis H1 we investigate the effect of HV behaviors on the
altruistic AV agents. We focus on scenarios with a HV mission vehicle, with
safe AVs that act _altruistically_ ($AV_{S}$) or _egoistic_ ($AV_{E}$), in
environments with increasing levels of HVs aggressiveness. Figure 9
illustrates the altruistic performance gain for increasing levels of HVs’
aggressiveness for 2 AVs (left) and 4 AVs (right). It demonstrates that the
more aggressive the HVs are, the higher the impact of cooperation and thus
confirms the H1. This is also observed in Figure 10 where the level of
aggressiveness is decomposed into lateral and longitudinal aggressiveness.
Lateral and longitudinal aggressiveness is varied by changing the MOBIL and
IDM parameters (Table 2) from aggressive to conservative. Figure 10 shows that
the altruistic gain increases in both directions, but is more pronounced in
the longitudinal direction. That is probably due to the simulated scenarios
having more longitudinal maneuvers.

Figure 9: Sensitivity analyses measured by altruistic performance gains (PGs)
of AVs show that the more aggressive the HVs are, the more the impact/gain of
cooperation. Figure 10: Both lateral and longitudinal sensitivity analyses
indicate an increase in altruistic performance gain (PG).

#### Domain Adaptation

Following the sensitivity analysis, we investigate the domain adaptation of
the AVs to validate the H2. Figure 11, Figure 12 and Figure 13 show how the
altruistic AVs learn to adapt to different scenarios and behaviors by
different performance metrics, i.e, crashed (a), distance traveled (b) and
adaptation error (c). For the experiments, $AV_{S}$ are trained in different
scenarios $f_{i}\in\mathcal{F}$ in the presence of HVs with different
behaviors $b_{k}\in\mathcal{B}$ and tested in other scenarios
$f_{j}\in\mathcal{F}$ and behaviors $b_{l}\in\mathcal{B}$. In our experiments,
we consider two case study scenarios $f_{e},f_{m}\in\mathcal{F}$
(exiting/merging) in environments with three different HVs behaviors
$b_{a},b_{m},b_{c}\in\mathcal{B}$ (aggressive, moderate, conservative) see
Table 2; and a mixed behavior environment, in which HVs are created randomly
and their behaviors are selected based on a uniform distribution over the
behaviors in $\mathcal{B}$, given equal probability to the defined behaviors.
In total, we have eight combinations of scenarios and behaviors, namely:
($f_{m},b_{mix}$), ($f_{m},b_{a}$), ($f_{m},b_{m}$), ($f_{m},b_{c}$),
($f_{e},b_{mix}$), ($f_{e},b_{a}$), ($f_{e},b_{m}$), ($f_{e},b_{c}$).

The results are presented in Figure 13 (c) as an adaptation matrix, showing
the $\mathrm{A}_{\mathrm{error}}$ for different domains, the
$\mathrm{A}_{\mathrm{error}}$ is in percentage ($\%$) and color-map in
logarithmic scale to increase the perceived dynamic range for visualization.
In our analyses, the weights used for $\mathrm{A}_{\mathrm{error}}(\%)$ are
$w_{s}=\frac{2}{3}$ and $w_{e}=\frac{1}{3}$, which weighs the safety
performance higher. $DT_{max}$ is computed based on the maximum distance for
each situation. Additionally, Figure 11 (a) and Figure 12 b) illustrate how
the AVs adapt in terms of safety (measured by $C(\%)$) and efficiency
(measured by $DT(m)$), separately.

The matrix shows the best performances in its diagonal; where agents are
trained and tested in the same environment (($f_{i},b_{k}$); ($f_{j},b_{l}$)
with $i=j$ and $k=l$); due to the fact that agents experience similar
situations during testing as they do during training. The vehicles trained in
the merging environment can perform the exiting mission for different
behaviors, and vice-versa. Interestingly, the performance of AVs trained in a
conservative environment ($b_{c}$) is poor when tested in an aggressive
environment ($b_{a}$). We believe that the reason is that in conservative
environments, the HVs yield the mission vehicle, and the AVs learn to rely on
HVs to guide the traffic. This learned policy is valid in a conservative
environment where one can expect the HVs to always create a safe space for the
mission vehicle. However, the same is not valid in more aggressive
environments, in which AVs have to guide the traffic to avoid dangerous
situations. As a result, the performance of vehicles trained in a conservative
environment and tested in an aggressive one is the worse.

On the other hand, an adequate performance adaptation (lower
$\mathrm{A}_{\mathrm{error}}$) is obtained when agents are trained in the
presence of all moderate HVs ($b_{m}$) or a mixed behavior environment
($b_{mix}$), in which AVs face situations where the HVs yield, but also
situations that require learning how to guide the traffic to optimize for the
social utility. The results from the domain adaptation matrix indicate that a
moderate or mixed environment is the most suitable for training robust AVs and
show the adaptability of AVs to different situations, thereby confirming the
H2 hypothesis.

It can be concluded that the adaptation between the environments is not
reciprocal and environment and situations selection should be considered
during training, based on the application needs and target situations. The
Domain adaptation matrix identifies the settings in which altruistic AVs can
best learn cooperative policies that are robust to different traffic scenarios
and human behaviors.

Figure 11: (a) The domain adaptation matrix with crash percentage ($C(\%)$)
between different traffic scenarios and behaviors. The lower $C(\%)$ the most
suitable the adaptability in terms of safety (measured by $C(\%)$) between
those domains. $\mathrm{AV}_{\mathrm{S}}$ are trained (rows of the matrix) in
different scenarios $f_{i}\in\mathcal{F}$ in the presence of HVs with
different behaviors $b_{k}\in\mathcal{B}$ and tested (columns of the matrix)
in other scenarios $f_{j}\in\mathcal{F}$ and behaviors $b_{l}\in\mathcal{B}$.
Each pair ($f_{i},b_{k}$) is a combination of scenario and behavior. Figure
12: (b) The domain adaptation matrix with distance traveled ($DT(m)$).
Illustrating how the AVs adapt to other situations in terms of efficiency
(measured by $DT(m)$). Figure 13: (c) The domain adaptation matrix with
adaptation error ($\mathrm{A}_{\mathrm{error}}$) between different traffic
scenarios and behaviors. The lower $\mathrm{A}_{\mathrm{error}}$ the most
suitable the adaptability between those domains.

#### Transfer Learning

Through domain adaptation and transfer learning, we promote generalization
while learning harder tasks efficiently from trained models and accelerate the
learning process. We study how the policies learned during merging can be
transferred to the exiting environment. For that, we train AVs agents from
scratch for the mission/task of merging $\mathrm{AV}_{\mathrm{merging}}$ (T1),
train AVs agents to drive on a highway, and then use that model as the
starting point to learn the merging task $\mathrm{AV}_{\mathrm{drive-to-
merging}}$ (T2), train AVs agents for the exiting task and then use that model
as the starting point to learn the merging task $\mathrm{AV}_{\mathrm{exiting-
to-merging}}$ (T3); and apply the same procedure for the exiting task,
learning to exit from scratch $\mathrm{AV}_{\mathrm{exiting}}$ (T4), after
learned how to drive $\mathrm{AV}_{\mathrm{drive-to-exiting}}$ (T5) and after
learned how to merge $\mathrm{AV}_{\mathrm{merging-to-exiting}}$ (T6). The
results of the experiments are presented in Figure 14 and show that our
transfer learning approach speeds up the learning process while archiving
similar performance as when learning the task from scratch.

Figure 14: The figure demonstrates how policies learned from merging can be
transferred to the exiting environment to speed up the learning process while
archiving similar performance to learning the task from scratch.

#### Safety

Finally, we compared state of the art architectures related to our approach
van2016deep ; toghi2021social ; toghi2021cooperative ; toghi2021altruistic  in
terms of safety and efficiency to validate H3. We trained the different
architectures in the same situations and examined their performance under
different levels of HVs behaviors. As noted in Table 5 our safe altruistic
agents consistently outperformed the other approaches, and the results are
more notable when the level of aggressiveness is higher. We conclude that when
using the safety prioritizer, immediate collisions are avoided reducing the
overall number of crashes in the episodes. Our agents can learn from scratch
not only how to drive, but also to understand the behavior of HVs and
coordinate with them.

Table 5: A comparison of the performance of related architectures. Our safe
altruistic AVs outperform the other solutions, and performance improvements
become more noticeable as the level of aggressiveness increases.

| Aggressive HVs | Moderate HVs | Conservative HVs
---|---|---|---
Approaches | C (%) | MF (%) | DT (m) | C (%) | MF (%) | DT (m) | C (%) | MF (%) | DT (m)
Conv2D+DQN van2016deep  | 31.2 | 28.9 | 316 | 25.4 | 20.3 | 302 | 14.0 | 7.9 | 274
Toghi  _et al._ toghi2021social  | 21.3 | 16.4 | 339 | 12.7 | 10.1 | 333 | 1.6 | 0.6 | 269
Conv3D+A2C toghi2021altruistic  | 14.8 | 12.6 | 341 | 9.4 | 8.8 | 328 | 1.1 | 0.1 | 267
Conv3D+DQN toghi2021cooperative  | 3.1 | 2.8 | 359 | 2.6 | 2.4 | 341 | 0.3 | 0 | 284
Ours | 0.2 | 0.1 | 397 | 0.1 | 0.1 | 354 | 0 | 0 |  281

C: _Crashed_ , MF: _Mission Failed_ , DT: _Distance Traveled_

#### Importance of Social Coordination

We demonstrate that social awareness and coordination are essential to improve
safety and reliability on the roads. Particularly in our sensitivity analyses
(Figure 9) we have shown that altruistic agents have a significant performance
gain when compared to egoistic agents and the gain is more notable as the road
becomes more aggressive. Additionally, to show that the performance gain vs.
driver behaviors is not just because of a single altruistic agent but as a
consequence of coordination among agents, we complement our results and
conducted an experiment with the difference that only $\mathrm{AV1}$ is
altruistic and the others are egoistic AVs, we label this scenario as single
altruistic agent SAA. Table 6 demonstrates the necessity of multi-agent
coordination and the fact that a single altruistic AV, i.e., the Guide AV, is
not able to achieve the mission of safe and seamless merging without help from
the other AVs. Our results show that a non-cooperative SAA is not enough to
guide the traffic and successful completion of the missions, as coordination
is not guaranteed in a single-agent setting. All the AVs have to coordinate
collectively to allow safe and efficient traffic, and this is unfeasible if
the others do not collaborate. Table 6 complements our results in Figure 9 and
support the hypothesis H1.

Table 6: Importance of Social Coordination: AVs require to coordinate to enable a safe and seamless merging /exiting and none of them can achieve this goal if the others do not cooperate. | Aggressive HVs $C(\%)$ | Moderate HVs $C(\%)$ | Conservative HVs $C(\%)$
---|---|---|---
MAA multi-agent altruistic | $0.2\%$ | $0.1\%$ | $0\%$
SAA single altruistic agent | $24.1\%$ | $17.4\%$ | $2.3\%$

###  0.6.5 Qualitative Analyses

We show a qualitative analysis of our altruistic AVs in the exit and merging
scenarios. Figure 15 provides further intuition about the policies learned by
altruistic AVs (green) in different situations, Figure 15 and Figure 16 show a
set of snapshots for different policies learned in an exit/merging environment
in the presence of HVs (blue) with different behaviors. In the presence of
aggressive HVs, the guide AV has to slow down and guide the HVs in the platoon
to allow a safe merging/exit of the mission vehicle; in this case, by slowing
down the AV learn to compromise on their own utility for a more desirable
social outcome. In the presence of moderate HVs behaviors, the guide AV slows
down (slowing down the vehicles in the platooning) to open a safe space for
the mission vehicle and then quickly accelerates, the space created by the
quick AV intervention is safe enough to allow the mission vehicle to
exit/merge the road; in this case, the AV compromise in their own utility but
does not need to compromise as much as in the aggressive traffic scenario, it
learns to take sequences of actions to not only enable the mission vehicle to
merge (by quick decelerating), but also manages to make the minimum compromise
on its individual utility. Finally, in the conservative environment, the HVs
are cautious enough to allow the mission vehicle to exit/merge safely, so the
AVs learn to accelerate in those scenarios as the mission vehicle has enough
space to merge, optimizing for their own utility (higher speed and longer
distance travel), while also considering other vehicles utilities and safety;
in this case the AV doe not need to compromise their own utility, it learns
that HV will allow the exiting/merging so AVs does not need to guide the
traffic. It is important to notice that the policies are learned by AVs from
experience to optimize the social utility, AVs learn to adapt to different
scenarios and behaviors. Is interesting to observe that our AVs develop some
form of social awareness and learn the HVs’ behaviors from experience, acting
accordingly to optimize traffic efficiency while prioritizing safety.

Figure 15: Mission vehicle exiting the road under different HV behaviors (from
left to right: aggressive, moderate and conservative HVs). AVs are shown in
green and HVs are shown in blue. Figure 16: Mission vehicle merging into the
highway under different HV behaviors (from left to right: aggressive,
moderate, and conservative HVs). AVs are shown in green and HVs are shown in
blue. The diameter of the circles on the trajectory plot (first-row plot)
shows the vehicles’ speed.

##  0.7 Concluding Remarks

AVs need to learn to co-exist with HVs vehicles as deploying egoistic AVs that
solely account for their individual interests on the road leads to sub-optimal
and non-desirable social outcomes. Social awareness and coordination are
essential to improve safety and reliability on the roads. We demonstrate how
altruistic AVs learn the decision-making process from experience, considering
the interests of all vehicles while prioritizing safety and optimizing a
general decentralized social utility function. We expose the settings for our
MARL problem in which transfer learning and domain adaptation are more
feasible, and conducted a sensitivity analysis under different HVs’ behaviors.
Our experiments reveal that altruistic AVs learn to leverage social
coordination to improve safety and reliability. Our social-aware AVs are
robust to heterogeneous driver behaviors and can form alliances and affect the
behavior of HVs to create socially-desirable outcomes that benefit the group
of the vehicles.

Future Work: Although we explored various elements of social navigation in a
variety of settings and the presence of diverse HV behaviors, the HV models
used are not from real human driver data, and the traffic scenarios are
limited to merging and exiting. However, we believe that by leveraging and
learning from actual human data and traffic circumstances, our approach might
be beneficial in practical traffic conditions. For this strategy to be used in
real-world circumstances, more attention to safety is necessary. We intend to
investigate more sophisticated architectures and state representations in
future work, as well as develop a more realistic simulation environment that
incorporates data from real-world traffic and can handle more complex
interactions between HVs and AVs, as well as diverse traffic agents like
bicycles and pedestrians. Despite the drawbacks, we are excited to see safe
and reliable social-aware AVs on the road that learns from experience. Beyond
driving, we expect these principles to be applied to general multi-agent
human-robot interactions in which agents influence humans and collaborate
safely for a socially beneficial result.

## References

  * (1) A. Cosgun, L. Ma, J. Chiu, J. Huang, M. Demir, A. M. Anon, T. Lian, H. Tafish, and S. Al-Stouhi, “Towards full automated drive in urban environments: A demonstration in gomentum station, california,” in _2017 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2017, pp. 1811–1818.
  * (2) W. Schwarting, A. Pierson, J. Alonso-Mora, S. Karaman, and D. Rus, “Social behavior for autonomous vehicles,” _Proceedings of the National Academy of Sciences_ , vol. 116, no. 50, pp. 24 972–24 978, 2019.
  * (3) F. Sagberg, Selpi, G. F. Bianchi Piccinini, and J. Engström, “A review of research on driving styles and road safety,” _Human factors_ , vol. 57, no. 7, pp. 1248–1275, 2015.
  * (4) B. Toghi, M. Saifuddin, M. Mughal, and Y. P. Fallah, “Spatio-temporal dynamics of cellular v2x communication in dense vehicular networks,” in _2019 IEEE 2nd Connected and Automated Vehicles Symposium (CAVS)_. IEEE, 2019, pp. 1–5.
  * (5) G. Shah, R. Valiente, N. Gupta, S. O. Gani, B. Toghi, Y. P. Fallah, and S. D. Gupta, “Real-time hardware-in-the-loop emulation framework for dsrc-based connected vehicle applications,” in _2019 IEEE 2nd Connected and Automated Vehicles Symposium (CAVS)_. IEEE, 2019, pp. 1–6.
  * (6) R. Valiente, M. Zaman, S. Ozer, and Y. P. Fallah, “Controlling steering angle for cooperative self-driving vehicles utilizing cnn and lstm-based deep networks,” in _2019 IEEE intelligent vehicles symposium (IV)_. IEEE, 2019, pp. 2423–2428.
  * (7) M. Razzaghpour, S. Shahram, R. Valiente, and Y. P. Fallah, “Impact of communication loss on mpc based cooperative adaptive cruise control and platooning,” _arXiv preprint arXiv:2106.09094_ , 2021.
  * (8) R. Valiente, A. Raftari, M. Zaman, Y. P. Fallah, and S. Mahmud, “Dynamic object map based architecture for robust cvs systems,” SAE Technical Paper, Tech. Rep., 2020.
  * (9) S. Aoki, T. Higuchi, and O. Altintas, “Cooperative perception with deep reinforcement learning for connected vehicles,” in _2020 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2020, pp. 328–334.
  * (10) B. Toghi, R. Valiente, D. Sadigh, R. Pedarsani, and Y. P. Fallah, “Cooperative autonomous vehicles that sympathize with human drivers,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2021.
  * (11) A. Jami, M. Razzaghpour, H. M. Alnuweiri, and Y. P. Fallah, “Augmented driver behavior models for high-fidelity simulation study of crash detection algorithms,” 2022.
  * (12) P. B. Harris, J. M. Houston, J. A. Vazquez, J. A. Smither, A. Harms, J. A. Dahlke, and D. A. Sachau, “The prosocial and aggressive driving inventory (padi): A self-report measure of safe and unsafe driving behaviors,” _Accident Analysis & Prevention_, vol. 72, pp. 1–8, 2014.
  * (13) E. F. Vallières, R. J. Vallerand, J. Bergeron, and P. McDuff, “Intentionality, anger, coping, and ego defensiveness in reactive aggressive driving,” _Journal of Applied Social Psychology_ , vol. 44, no. 5, pp. 354–363, 2014.
  * (14) M. Bouton, A. Nakhaei, K. Fujimura, and M. J. Kochenderfer, “Cooperation-aware reinforcement learning for merging in dense traffic,” in _2019 IEEE Intelligent Transportation Systems Conference (ITSC)_. IEEE, 2019, pp. 3441–3447.
  * (15) D. Sadigh, S. Sastry, S. A. Seshia, and A. D. Dragan, “Planning for autonomous cars that leverage effects on human actions.” in _Robotics: Science and Systems_ , vol. 2. Ann Arbor, MI, USA, 2016\.
  * (16) C. Wu, A. Kreidieh, E. Vinitsky, and A. M. Bayen, “Emergent behaviors in mixed-autonomy traffic,” in _Conference on Robot Learning_. PMLR, 2017, pp. 398–407.
  * (17) B. Toghi, R. Valiente, D. Sadigh, R. Pedarsani, and Y. P. Fallah, “Social coordination and altruism in autonomous driving,” _arXiv preprint arXiv:2107.00200_ , 2021.
  * (18) J. Rios-Torres and A. A. Malikopoulos, “A survey on the coordination of connected and automated vehicles at intersections and merging at highway on-ramps,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 18, no. 5, pp. 1066–1077, 2016.
  * (19) H. N. Mahjoub, A. Raftari, R. Valiente, Y. P. Fallah, and S. K. Mahmud, “Representing realistic human driver behaviors using a finite size gaussian process kernel bank,” in _2019 IEEE Vehicular Networking Conference (VNC)_. IEEE, 2019, pp. 1–8.
  * (20) Z. Li, U. Kalabić, and T. Chu, “Safe reinforcement learning: Learning with supervision using a constraint-admissible set,” in _2018 Annual American Control Conference (ACC)_. IEEE, 2018, pp. 6390–6395.
  * (21) Y. Lin, J. McPhee, and N. L. Azad, “Anti-jerk on-ramp merging using deep reinforcement learning,” in _2020 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2020, pp. 7–14.
  * (22) D. Sadigh, N. Landolfi, S. S. Sastry, S. A. Seshia, and A. D. Dragan, “Planning for cars that coordinate with people: leveraging effects on human actions for planning and active information gathering over human internal state,” _Autonomous Robots_ , vol. 42, no. 7, pp. 1405–1426, 2018.
  * (23) B. Toghi, R. Valiente, D. Sadigh, R. Pedarsani, and Y. P. Fallah, “Altruistic maneuver planning for cooperative autonomous vehicles using multi-agent advantage actor-critic,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , 2021.
  * (24) J. N. Foerster, R. Y. Chen, M. Al-Shedivat, S. Whiteson, P. Abbeel, and I. Mordatch, “Learning with opponent-learning awareness,” _arXiv preprint arXiv:1709.04326_ , 2017.
  * (25) A. Xie, D. Losey, R. Tolsma, C. Finn, and D. Sadigh, “Learning latent representations to influence multi-agent interaction,” in _Proceedings of the 4th Conference on Robot Learning (CoRL)_ , November 2020.
  * (26) J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, “Counterfactual multi-agent policy gradients,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 32, no. 1, 2018.
  * (27) M. Egorov, “Multi-agent deep reinforcement learning,” _CS231n: convolutional neural networks for visual recognition_ , pp. 1–8, 2016.
  * (28) S. Omidshafiei, J. Pazis, C. Amato, J. P. How, and J. Vian, “Deep decentralized multi-task multi-agent reinforcement learning under partial observability,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 2681–2690.
  * (29) R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” _arXiv preprint arXiv:1706.02275_ , 2017.
  * (30) K. Brown, K. Driggs-Campbell, and M. J. Kochenderfer, “A taxonomy and review of algorithms for modeling and predicting human driver behavior. arxiv e-prints, article,” _arXiv preprint arXiv:2006.08832_ , 2020.
  * (31) B. Ivanovic, E. Schmerling, K. Leung, and M. Pavone, “Generative modeling of multimodal multi-human behavior. in 2018 ieee,” in _RSJ International Conference on Intelligent Robots and Systems_ , 2018, pp. 3088–3095.
  * (32) M. Lauer and M. Riedmiller, “An algorithm for distributed reinforcement learning in cooperative multi-agent systems,” in _In Proceedings of the Seventeenth International Conference on Machine Learning_. Citeseer, 2000.
  * (33) B. Toghi, D. Grover, M. Razzaghpour, R. Jain, R. Valiente, M. Zaman, G. Shah, and Y. P. Fallah, “A maneuver-based urban driving dataset and model for cooperative vehicle applications,” 2020.
  * (34) R. Chandra, U. Bhattacharya, T. Mittal, A. Bera, and D. Manocha, “Cmetric: A driving behavior measure using centrality functions,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 2035–2042.
  * (35) Z. Constantinescu, C. Marinoiu, and M. Vladoiu, “Driving style analysis using data mining techniques,” _International Journal of Computers Communications & Control_, vol. 5, no. 5, pp. 654–663, 2010.
  * (36) K. H. Beck, B. Ali, and S. B. Daughters, “Distress tolerance as a predictor of risky and aggressive driving,” _Traffic injury prevention_ , vol. 15, no. 4, pp. 349–354, 2014.
  * (37) A. Pokle, R. Martín-Martín, P. Goebel, V. Chow, H. M. Ewald, J. Yang, Z. Wang, A. Sadeghian, D. Sadigh, S. Savarese _et al._ , “Deep local trajectory replanning and control for robot navigation,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 5815–5822.
  * (38) M. Kuderer, S. Gulati, and W. Burgard, “Learning driving styles for autonomous vehicles from demonstration,” in _2015 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2015, pp. 2641–2646.
  * (39) D. Hadfield-Menell, S. J. Russell, P. Abbeel, and A. Dragan, “Cooperative inverse reinforcement learning,” _Advances in neural information processing systems_ , vol. 29, pp. 3909–3917, 2016.
  * (40) P. Trautman and A. Krause, “Unfreezing the robot: Navigation in dense, interacting crowds,” in _2010 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2010, pp. 797–803.
  * (41) S. Nikolaidis, R. Ramakrishnan, K. Gu, and J. Shah, “Efficient model learning from joint-action demonstrations for human-robot collaborative tasks,” in _2015 10th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_. IEEE, 2015, pp. 189–196.
  * (42) C. Wu, A. M. Bayen, and A. Mehta, “Stabilizing traffic with autonomous vehicles,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 6012–6018.
  * (43) D. A. Lazar, E. Bıyık, D. Sadigh, and R. Pedarsani, “Learning how to dynamically route autonomous vehicles on shared roads,” _arXiv preprint arXiv:1909.03664_ , 2019.
  * (44) E. Bıyık, D. A. Lazar, R. Pedarsani, and D. Sadigh, “Incentivizing efficient equilibria in traffic networks with mixed autonomy,” _IEEE Transactions on Control of Network Systems_ , vol. 8, no. 4, pp. 1717–1729, 2021\.
  * (45) S. Li, Z. Yan, and C. Wu, “Learning to delegate for large-scale vehicle routing,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021\.
  * (46) C. Hickert, S. Li, and C. Wu, “Cooperation for scalable supervision of autonomy in mixed traffic,” _arXiv e-prints_ , pp. arXiv–2112, 2021.
  * (47) M. Razzaghpour, S. Mosharafian, A. Raftari, J. Mohammadpour Velni, and Y. P. Fallah, “Impact of information flow topology on safety of tightly-coupled connected and automated vehicle platoons utilizing stochastic control,” in _(ECC 2022)_.
  * (48) W. Z. Wang, M. Beliaev, E. Biyik, D. A. Lazar, R. Pedarsani, and D. Sadigh, “Emergent prosociality in multi-agent games through gifting,” in _30th International Joint Conference on Artificial Intelligence (IJCAI)_ , 2021.
  * (49) J. Wang, Q. Zhang, D. Zhao, and Y. Chen, “Lane change decision-making through deep reinforcement learning with rule-based constraints,” in _2019 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2019, pp. 1–6.
  * (50) S. Nageshrao, H. E. Tseng, and D. Filev, “Autonomous highway driving using deep reinforcement learning,” in _2019 IEEE International Conference on Systems, Man and Cybernetics (SMC)_. IEEE, 2019, pp. 2326–2331.
  * (51) A. Mohammadhasani, H. Mehrivash, A. Lynch, and Z. Shu, “Reinforcement learning based safe decision making for highway autonomous driving,” _arXiv preprint arXiv:2105.06517_ , 2021.
  * (52) D. Chen, Z. Li, Y. Wang, L. Jiang, and Y. Wang, “Deep multi-agent reinforcement learning for highway on-ramp merging in mixed traffic,” _arXiv preprint arXiv:2105.05701_ , 2021.
  * (53) V.-A. Le and A. A. Malikopoulos, “A cooperative optimal control framework for connected and automated vehicles in mixed traffic using social value orientation,” _arXiv preprint arXiv:2203.17106_ , 2022.
  * (54) R. O. Murphy and K. A. Ackermann, “Social preferences, positive expectations, and trust based cooperation,” _Journal of Mathematical Psychology_ , vol. 67, pp. 45–50, 2015.
  * (55) A. Garapin, L. Muller, and B. Rahali, “Does trust mean giving and not risking? experimental evidence from the trust game,” _Revue d’économie politique_ , vol. 125, no. 5, pp. 701–716, 2015.
  * (56) L. Müller, M. Risto, and C. Emmenegger, “The social behavior of autonomous vehicles,” in _Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct_ , ser. UbiComp ’16. New York, NY, USA: Association for Computing Machinery, 2016, p. 686–689. [Online]. Available: https://doi.org/10.1145/2968219.2968561
  * (57) M. Treiber, A. Hennecke, and D. Helbing, “Congested traffic states in empirical observations and microscopic simulations,” _Physical review E_ , vol. 62, no. 2, p. 1805, 2000.
  * (58) A. Kesting, M. Treiber, and D. Helbing, “General lane-changing model mobil for car-following models,” _Transportation Research Record_ , vol. 1999, no. 1, pp. 86–94, 2007.
  * (59) E. Leurent, Y. Blanco, D. Efimov, and O.-A. Maillard, “Approximate robust control of uncertain dynamical systems,” _arXiv preprint arXiv:1903.00220_ , 2019.
  * (60) H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 30, no. 1, 2016.