ifaamas [AAMAS ’22]Proc. of the 21st International Conference on Autonomous
Agents and Multiagent Systems (AAMAS 2022)May 9–13, 2022 OnlineP. Faliszewski,
V. Mascardi, C. Pelachaud, M.E. Taylor (eds.) 2022 2022 542 Cornell University
Ithaca New York USA Cornell Tech New York New York USA Cornell University
Ithaca New York USA

# Multi-Agent Curricula and Emergent Implicit Signaling

Niko A. Grupen  ,  Daniel D. Lee  and  Bart Selman

###### Abstract.

Emergent communication has made strides towards learning communication from
scratch, but has focused primarily on protocols that resemble human language.
In nature, multi-agent cooperation gives rise to a wide range of communication
that varies in structure and complexity. In this work, we recognize the full
spectrum of communication that exists in nature and propose studying lower-
level communication. Specifically, we study emergent implicit signaling in the
context of decentralized multi-agent learning in difficult, sparse reward
environments. However, learning to coordinate in such environments is
challenging. We propose a curriculum-driven strategy that combines: (i)
velocity-based environment shaping, tailored to the skill level of the multi-
agent team; and (ii) a behavioral curriculum that helps agents learn
successful single-agent behaviors as a precursor to learning multi-agent
behaviors. Pursuit-evasion experiments show that our approach learns effective
coordination, significantly outperforming sophisticated analytical and learned
policies. Our method completes the pursuit-evasion task even when pursuers
move at half of the evader’s speed, whereas the highest-performing baseline
fails at $80\%$ of the evader’s speed. Moreover, we examine the use of
implicit signals in coordination through position-based social influence. We
show that pursuers trained with our strategy exchange more than twice as much
information (in bits) than baseline methods, indicating that our method has
learned, and relies heavily on, the exchange of implicit signals.

###### Key words and phrases:

Multi-Agent Systems, Reinforcement Learning, Communication

##  1\. Introduction

Communication is a critical scaffolding for coordination. It enables humans
and animals alike to coordinate on complex tasks, synchronize plans, allocate
team resources, and share missing state. Understanding the process through
which communication emerges has long been a goal of philosophy, linguistics,
cognitive science, and AI. Recently, advances in multi-agent reinforcement
learning (MARL) have propelled computational studies of emergent communication
that examine the representations and social conditions necessary for
communication to emerge in situated multi-agent populations Lazaridou and
Baroni (2020). Existing approaches, targeting language-like communication,
have shown that it is possible for agents to learn protocols that exhibit
language-like properties such as compositionality Chaabouni et al. (2020);
Resnick et al. (2019) and Zipf’s Law Chaabouni et al. (2019) when given
additional learning biases Eccles et al. (2019).

Figure 1. Left: Animal communication spans a complete spectrum from explicit
to implicit communication. Right: A pursuit-evasion game with two pursuers
$p_{1}$ and $p_{2}$ and an evader $e$ (with positions $q_{1,t}$, $q_{2,t}$,
and $q_{e,t}$ at time $t$). Emergent communication only considers language-
like communication (top), where agents share information over specialized
sender-receiver architectures—e.g. $p_{1}$ receives the extended observation
$o_{1,t}=[q_{1,t}\ q_{2,t}\ q_{e,t}\ m_{2,t-1}]^{T}$ and selects actions using
the extra information $m_{2,t-1}$ that $p_{2}$ chose to share at the previous
time-step (and vice versa for $p_{2}$). With implicit signals (bottom),
$p_{1}$ selects a movement action $a_{1,t}$ based only on the observation
$o_{1,t}=[q_{1,t}\ q_{2,t}\ q_{e,t}]^{T}$. The only information $p_{1}$ gets
from $p_{2}$ is generated by $p_{2}$’s physical position (and vice versa for
$p_{2}$). We posit that implicit signals are an important step towards
learning richer communication.

Though emergent communication is fundamentally an ab initio approach compared
to top-down language learning Vaswani et al. (2017), most recent methods at
the intersection of emergent communication and multi-agent reinforcement
learning have targeted protocols with complex structure and representational
capacity, like that of human language Lazaridou and Baroni (2020); Lowe et al.
(2019). Such techniques equip agents with specialized sender-receiver
architectures and allow them to exchange bits of information (or continuous
signals) through these channels. One notable exception is the work of Jaques
et al. (2019), which considers how influential agents can pass information to
other agents through their actions in the environment. We build on these
ideas, recognizing the full spectrum of communication that exists in nature
and proposing the study of lower-level forms of communication that do not
require specialized architectures as a first step towards more sophisticated
communication (see Figure 1).

Multi-agent cooperation in nature yields a wide range of communication
protocols that vary in structure and complexity. In animal communication
Bradbury et al. (1998), for example, reef-dwelling fish use body shakes Vail
et al. (2013); Bshary et al. (2006) and octopuses punch collaborators Sampaio
et al. (2020)—forms of non-verbal communication—whereas chimps Boesch and
Boesch (1989), macaques Mason and Hollis (1962), and gentoo penguins Choi et
al. (2017) each maintain a diverse vocal repertoire. Importantly, animals also
exchange information through non-explicit channels. In the Serengeti, for
example, lions use the posture of fellow pride members to stalk prey covertly
Schaller (2009). Moreover, wolves Peterson and Ciucci (2003) and dolphins
Quick and Janik (2012), both frequent vocal communicators, communicate
implicitly during foraging—adjusting their group formation based on the
position and orientation of other pack members Herbert-Read (2016). Breazeal
et al. (2005) define such forms of communication, called implicit
communication, as ”conveying information that is inherent in behavior but
which is not deliberately communicated”. Studies have shown that implicit cues
are used frequently by humans as well. Human teams rely on gaze, facial
expressions, and non-symbolic movement during cooperative tasks Entin and
Serfaty (1999) including collaborative design Wuertz et al. (2018),
crowdsourcing Zhou et al. (2017), and multiplayer games (e.g. Hanabi Liang et
al. (2019); Bard et al. (2020)).

The role of implicit communication in teamwork has been studied extensively in
the human-computer Liang et al. (2019); Schmidt (2000) and human-robot
interaction Che et al. (2020); Breazeal et al. (2005); Knepper et al. (2017);
Butchibabu et al. (2016) communities, and even in multi-agent systems Pagello
et al. (1999); Gildert et al. (2018); but is less well-understood in the
context of emergent agent-agent interactions De Greeff and Nolfi (2010). This
is due in large part to the anthropomorphized nature of standard definitions
of implicit communication. Note that the key difference between explicit
communication (a deliberate information exchange) and implicit communication
(a non-deliberate exchange) in the definition above is intent Breazeal et al.
(2005); Gildert et al. (2018). Understanding implicit communication in the
context of artificial agents requires (i) ascribing human-like mental states
to artificial agents and (ii) giving agents the ability to reason about each
other’s mental states with respect to a common goal (à la Cohen and Levesque
(1991)); which is a highly non-trivial undertaking.

In this work, we take a first step in this direction by examining the extent
to which implicit communication aides coordination in multi-agent populations.
Following the work of De Greeff and Nolfi (2010) in evolutionary settings, we
study implicit signals: signals generated by the physical position of an agent
as it interacts with its environment. Implicit signals are observed passively
by an agent’s teammates, and so require neither special sender-receiver
architectures nor reasoning about an agent’s mental states. For RL agents, we
distinguish implicit signals from explicit communication through each agent’s
actions and observations. As a simple example, consider the environment in
Figure 1 where, at time $t$, agents $p_{1}$, $p_{2}$, and $e$ are described by
their positions $q_{1,t}$, $q_{2,t}$, and $q_{e,t}$ respectively. With
implicit signals, $p_{1}$ selects a movement action $a_{1,t}$ based on the
observation $o_{1,t}=[q_{1,t}\ q_{2,t}\ q_{e,t}]^{T}$—i.e. the only
information $p_{1}$ gets from $p_{2}$ is the information contained in
$p_{2}$’s physical position; and vice versa. With explicit communication,
$p_{1}$ emits a communicative action $m_{1,t}$ in addition to $a_{1,t}$ (and
same for $p_{2}$). In this case, $p_{1}$ receives the extended observation
$o_{1,t}=[q_{1,t}\ q_{2,t}\ q_{e,t}\ m_{2,t-1}]^{T}$ and can select actions
using the extra information $m_{2,t-1}$ that $p_{2}$ chose to share at the
previous time-step.

Experimentally, we study coordination and the role of implicit communication
with pursuit-evasion games Isaacs (1999). Pursuit-evasion simulates an
important coordination task, foraging, where communication is especially
impactful. To accurately model the conditions under which coordination and
communication emerges in the animal kingdom, we prioritize decentralized
multi-agent learning. Despite the challenges posed by decentralized approaches
Foerster et al. (2016), this feature is critical in the emergence of animal
communications and so we focus on this important class of multi-agent learning
frameworks to exploit inspiration from the natural world. Moreover, to
encourage teamwork, we target pursuit-evasion games in which the pursuers are
slower than the prey (coordination is not needed with the pursuers have a
velocity advantage). Similar to Lowe et al. (2017), we study pursuit-evasion
as a MARL problem.

We show that, under these conditions, naive decentralized learning fails to
solve the pursuit-evasion task. There are a number of challenges that lead to
this outcome. First, decentralized learning is non-stationary Lowe et al.
(2017)—with multiple agents learning simultaneously, the learning problem is
less stable for each agent individually. Next, since we prioritize tasks that
require teamwork (e.g. pursuer speed less than evader speed), learning from
scratch is difficult, as agents are forced to learn how to interact with the
environment individually while simultaneously learning to coordinate with
their teammates. Finally, pursuit-evasion is a sparse reward task—pursuers
receive reward only when capturing the evader. During the early stages of
training, when each agent is following a randomly initialized policy, it is
virtually impossible for the agents to coordinate effectively enough to
experience positive reward.

To address these challenges, we introduce a curriculum-driven approach for
solving difficult, sparse reward coordination tasks. Our strategy, combining
ideas from the ecological reinforcement learning Co-Reyes et al. (2020) and
automatic curriculum learning Portelas et al. (2020b) literature, is motivated
by the following insights: (i) we can create a curriculum over task difficulty
by shaping the environment to the current skill level of the cooperative
agents; (ii) we can aide agents in learning multi-agent team behaviors by
forcing them to learn single-agent behaviors first. More specifically, we
first adopt an environment shaping curriculum using velocity bounds, which
allows pursuers to gather initial experience at a velocity greater than that
of the evader, and then tune their strategies over time as velocity decreases.
Then, we introduce a curriculum over agent behavior that warm-starts
cooperative multi-agent learning by first seeding each agent with experience
of successful single-agent behaviors.

Our first result confirms the importance of curriculum-driven multi-agent
learning. We compare the performance of our strategy to ablations over the
multi-agent curricula individually and show that, using our strategy,
decentralized agents learn to solve difficult coordination tasks, such as the
pursuit-evasion game.

Next, we stress-test both the strength of coordination and capacity for
implicit communication of policies learned with our method. We compare the
performance of multi-agent coordination learned with our method against a set
of analytical and learned strategies that represent ablations over both
coordination and implicit communication—i.e. outperforming one of these
methods is equivalent to surpassing that method’s level of sophistication in
coordination, capacity for implicit communication, or both.

Empirical results show that our method significantly outperforms highly
sophisticated multi-agent strategies. Coordination strategies learned with our
method successfully complete the pursuit-evasion task at a speed ratio of 0.5
(i.e. the pursuers are moving at half the evader’s speed), whereas each of the
baseline methods fails to complete the pursuit-evasion task when pursuer speed
is only slightly below evader speed (speed ratio of 0.8).

Further, we hone in on the nature of each strategy’s coordination more
directly by examining agent behavior during the most important subsets of the
trajectory—those that immediately precede the accumulation of reward. In the
context of pursuit-evasion, we examine the distribution of pursuer locations
relative the evader at the time of capture. By comparing rotational symmetry
and rotational invariance in the agents’ capture distributions, we find that
our strategy learns structured coordination while simultaneously allowing
pursuers to make dynamic adjustments to their position relative the evader to
successfully achieve capture.

Finally, we examine the role of implicit signals (as introduced by De Greeff
and Nolfi (2010)) more directly using measures of social influence. Influence
measures such as Instantaneous Coordination Jaques et al. (2019) quantify the
extent to which one agent’s behavior influences its teammates. We repurpose
this method to measure the exchange of implicit signals as position-based
social influence. Results of this study show that pursuers trained with our
strategy exchange up to 0.375 bits of information per time-step on average,
compared to a maximum of only 0.15 bits on average from the baseline methods.
This indicates that our method learns, and relies heavily on, the exchange of
implicit signals.

Preview of contributions Altogether, our work is summarized by the following
contributions:

  * •

We highlight the importance of studying implicit signals as a form of emergent
communication.

  * •

We introduce a curriculum-driven learning strategy that enables cooperative
agents to solve difficult coordination tasks with sparse reward.

  * •

We show that, using our strategy, pursuers learn to coordinate to capture a
superior evader, significantly outperforming sophisticated analytical and
learned methods. Coordination strategies learned with our method complete the
pursuit-evasion task at a speed ratio of $0.5$ (i.e. half the evader’s speed),
whereas each of the competing methods fails to complete the pursuit-evasion
task at any speed ratio below $0.8$.

  * •

We examine the use of implicit signals in emergent multi-agent coordination
through position-based social influence. We find that pursuers trained with
our strategy exchange up to 0.375 bits of information per time-step on
average, compared to a maximum of only 0.15 bits on average from competing
methods, indicating that our method has learned, and relies heavily on, the
exchange of implicit signals.

##  2\. Related work

###  2.1. Emergent Communication

Emergent communication examines the process by which cooperative agents learn
communication protocols as a means to completing difficult multi-agent tasks
Lazaridou and Baroni (2020). Recent work has shown that MARL agents converge
upon useful communication in referential games Havrylov and Titov (2017) and
can even develop language-like properties such as compositionality Chaabouni
et al. (2020); Resnick et al. (2019) and Zipf’s Law Chaabouni et al. (2019)
when exposed to additional structural learning biases. More recently, this
class of techniques has expanded to include complex situated environments Das
et al. (2019), high-dimensional observations Cowen-Rivers and Naradowsky
(2020), and the negotiation of belief states Foerster et al. (2018b). Further
work has shown that influential communication can be incentivized through
additional learning objectives (i.e. inductive biases) Eccles et al. (2019)
and reward shaping Jaques et al. (2019).

Implicit interactions are less well-studied in the emergent communication
literature, despite their importance to teamwork, as outlined in the human-
computer Liang et al. (2019); Schmidt (2000) and human-robot interaction Che
et al. (2020); Breazeal et al. (2005); Knepper et al. (2017); Butchibabu et
al. (2016) communities. Implicit communication has been studied for
cooperative multi-agent tasks Gildert et al. (2018); Grover and Sahai (2010);
Pagello et al. (1999), though not in the context of emergent behavior. Though
some studies have shown that agents can learn to communicate non-verbally
through actions and gestures Mordatch and Abbeel (2017), such forms of action-
space communication Baker et al. (2019) are examples of non-verbal explicit
communication, whereas our goal is to study completely implicit signals. Most
similar to our work is that of De Greeff and Nolfi (2010), which examines both
implicit and explicit communication in evolutionary settings. Our work can be
interpreted as bridging these ideas with the MARL literature.

###  2.2. Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning (MARL) encompasses a large body of
literature extending RL techniques to multi-agent settings. In general, MARL
algorithms subscribe to either decentralized or centralized learning. With
decentralized learning (or independent learning), each agent is responsible
for updating its own policy (or Q-network) individually Tan (1993). Though
recent work has shown that decentralized learning is feasible in complex
environments de Witt et al. (2020), it is often challenging, due to the non-
stationary (and therefore unstable) nature of the learning problem Laurent et
al. (2011); Matignon et al. (2012). Centralized techniques stabilize multi-
agent learning by allowing agents to share a joint Q-network during training,
then act independently at test-time Lowe et al. (2017); Foerster et al.
(2018a). Centralization has also proven useful for emergent communication, as
agents can share gradients directly through their communication channels
Foerster et al. (2016). Despite these benefits, we prioritize decentralized
learning because it more accurately represents the learning problem that
humans and animals face in the real-world.

Recently, significant effort has been aimed at connecting ideas from the
curriculum learning literature Bengio et al. (2009) to RL Portelas et al.
(2020b). Such methods have derived curricula from virtually every aspect of
the RL problem; including reward shaping Bellemare et al. (2016); Pathak et
al. (2017), modifying initial state distributions Florensa et al. (2017), and
procedurally-generating sub-tasks Portelas et al. (2020a); Risi and Togelius
(2019). Some multi-agent curricula have been shown to lead to more generally-
capable RL agents DeepMind (2021). Further work has examined the environment’s
role in generating curricula, leading to new methods such as environment
shaping Co-Reyes et al. (2020) and unsupervised environment design Dennis et
al. (2020). Of particular relevance is the work of Co-Reyes et al. (2020),
which considers curricula that manipulate the dynamics of the RL environment
to the benefit of learning agents. Our work combines a velocity-based variant
of environment shaping with an additional curriculum for bootstrapping multi-
agent learning with single-agent experience, which itself is inspired by the
strategy employed by Yang et al. (2018) for multi-goal learning.

###  2.3. Pursuit-Evasion

Pursuit-evasion is a classic game setting for studying multi-agent
coordination Isaacs (1999). Though often played on a graph Parsons (1978),
work on continuous-space pursuit-evasion has enabled real-world applications
such as unmanned aerial vehicles Vidal et al. (2002) and mobile robots Chung
et al. (2011). Further work has shown that optimal control strategies can be
derived from value functions Jang and Tomlin (2005) or even learned from
scratch in MARL setting Lowe et al. (2017). A relevant class of pursuit-
evasion games define the evader to be of equal or greater speed than the
pursuers. This setting highlights the need for coordinated motion (e.g.
encircling) Vicsek (2010) and communication Wang et al. (2020) by the pursuers
and is subject to theoretical performance bounds under these conditions Ramana
and Kothari (2017). Our work uses this setting to study the emergence of
animalistic coordination strategies and is similar to Janosov et al. (2017) in
that regard, but with an emphasis on learned, rather than hand-defined,
strategies.

##  3\. Preliminaries

###  3.1. Markov Game

A Markov game extends the Markov Decision Process (MDP) to multi-agent
settings Littman (1994). For $n$ agents, the formalism consists of a joint
action-space
$\boldsymbol{\mathcal{A}}=\\{\mathcal{A}_{1},...,\mathcal{A}_{n}\\}$ and joint
observation space
$\boldsymbol{\mathcal{O}}=\\{\mathcal{O}_{1},...,\mathcal{O}_{n}\\}$ that
define the actions and observations, respectively. The environment defines a
state space $\mathcal{S}$ from which a state $s_{t}$ is drawn each time-step.
The environment transitions from its current state $s_{t}$ to a new state
$s_{t+1}$ when actions are selected by each of the agents, as dictated by the
transition function
$T:\mathcal{S}\times\boldsymbol{\mathcal{A}}\rightarrow\mathcal{S}$. Finally,
a reward function
$r:\mathcal{S}\times\boldsymbol{\mathcal{A}}\rightarrow\mathbb{R}$ scores the
quality of the composite action of the agents. The goal of each agent is to
maximize the total reward it receives over time.

###  3.2. Deep Deterministic Policy Gradients

Deep Deterministic Policy Gradients (DDPG) is an off-policy actor-critic
algorithm for policy gradient learning in continuous action spaces Lillicrap
et al. (2015). DDPG learns an optimal deterministic policy $\mu_{\phi}$ with
respect to the RL objective:

$J(\phi)=\mathbb{E}_{s}[Q_{\omega}(s,a)|_{s=s_{t},a=\mu_{\phi}(s_{t})}]$ (1)

by performing gradient ascent over the following gradient:

$\nabla_{\phi}J(\phi)=\mathbb{E}_{s}[\nabla_{a}Q_{\omega}(s,a)|_{s=s_{t},a=\mu(s_{t})}\nabla_{\phi}\mu(s)|_{s=s_{t}}]$
(2)

where $\phi$ and $\omega$ are parameters associated with $\mu$ and $Q$,
respectively. Equation (2) is a consequence of the deterministic policy
gradient theorem Silver et al. (2014). For critic updates, DDPG minimizes the
loss function:

$L(\omega)=\underset{s,a,r,s^{\prime}}{\mathbb{E}}\big{[}\big{(}Q_{\omega}(s,a)-(r(s,a)+\gamma
Q_{\omega}(s^{\prime},\mu_{\phi}(s^{\prime})))\big{)}^{2}\big{]}$ (3)

where ($s,a,r,s^{\prime}$) are transition tuples sampled from a replay buffer
and $\gamma{\in}[0,1]$ is a scalar discount factor. In this work, agents learn
in a decentralized manner, each performing DDPG updates individually.

###  3.3. Potential Field Navigation

Given an agent $i$ with position $q_{i}$, we can define a potential function
$U(q_{i},q_{\textrm{goal}})$ between $q_{i}$ and a target point
$q_{\textrm{goal}}$ such that the negative gradient
$F(q_{i},q_{\textrm{goal}})=-\nabla U(q_{i},q_{\textrm{goal}})$ specifies a
control law for $i$’s motion. For example, let
$U_{\textrm{att}}(q_{i},q_{\textrm{goal}})$ be a quadratic function of
distance between $q_{i}$ and a target point $q_{\textrm{goal}}$:

$U_{\textrm{att}}(q_{i},q_{\textrm{goal}})=\frac{1}{2}k_{\textrm{att}}\,d(q_{i},q_{\textrm{goal}})^{2}$
(4)

where $k_{\textrm{att}}$ is an attraction coefficient and $d(,)$ is a measure
of distance. The resulting force exerted on agent $i$ is:

$F_{\textrm{att}}=-\nabla
U_{\textrm{att}}(q_{i},q_{\textrm{goal}})=-k_{\textrm{att}}(q_{i}-q_{\textrm{goal}})$
(5)

In this work, the agent’s action-space is defined in terms of headings, so
only the direction of this force impacts our agents.

###  3.4. Pursuit-evasion

Pursuit-evasion (i.e. predator-prey) is a classic setting for studying multi-
agent coordination Isaacs (1999). A pursuit-evasion game is defined between
$n$ pursuers $\\{p_{1},...,p_{n}\\}$ and a single evader $e$. The goal of the
pursuers is to catch the evader as quickly as possible and, conversely, the
goal of the evader is to remain uncaught. Each agent $i$ is described by its
current position and heading $q_{i}$ and is subject to planar motion
$\dot{q}_{i}$:

$\displaystyle q_{i}=\begin{bmatrix}x_{i}\\\ y_{i}\\\ \theta_{i}\end{bmatrix}$
$\displaystyle\dot{q}_{i}=\begin{bmatrix}\dot{x}_{i}\\\ \dot{y}_{i}\\\
\dot{\theta}_{i}\end{bmatrix}=\begin{bmatrix}\lvert\@vec{v}_{i}\rvert\,\textrm{cos}(\theta_{i})\\\
\lvert\@vec{v}_{i}\rvert\,\textrm{sin}(\theta_{i})\\\
\textrm{atan2}(\dot{y}_{i},\dot{x}_{i})\end{bmatrix}$

where $\@vec{v}_{i}$ is the agent $i$’s velocity. The environment state
$s_{t}$ is described by the position and heading of all agents
$s_{t}=\\{q_{p_{1}},...,q_{p_{n}},q_{e}\\}$. Upon observing $s_{t}$, each
agent selects its next heading $\theta_{i}$ as an action. The chosen heading
is pursued at the maximum allowed speed for each agent
($\lvert\@vec{v}_{p}\rvert$ for the pursuers, $\lvert\@vec{v}_{e}\rvert$ for
the evader); with orientation changes being instantaneous. To encourage
teamwork, we set $\lvert\@vec{v}_{p}\rvert\leq\lvert\@vec{v}_{e}\rvert$. We
assume the evader to be part of the environment, as defined by the potential
function:

$U_{\textrm{evade}}(\theta_{e})=\sum_{i}\bigg{(}\frac{1}{r_{i}}\bigg{)}\cos(\theta_{e}-\tilde{\theta}_{i})$
(6)

where $r_{i}$ and $\tilde{\theta}_{i}$ are the L2-distance and relative angle
between the evader and the $i$-th pursuer, respectively, and $\theta_{e}$ is
the heading of the evader. This objective is inspired by theoretical analysis
of escape strategies in the pursuit-evasion literature Ramana and Kothari
(2017). Intuitively, Equation (6) pushes the evader away from pursuers, taking
the largest bisector between any two when possible. The goal of the
pursuers—to capture the evader as quickly as possible—is mirrored in the
reward function, where $r(s_{t},a_{t})=50.0$ if the evader is captured and
$r(s_{t},a_{t})=-0.1$ otherwise. A derivation of Equation (6) and additional
environmental details are provided in Appendix A.

###  3.5. Implicit Communication

For clarity, we restate the definitions of implicit vs. explicit communication
from Breazeal et al. (2005) and implicit signals from De Greeff and Nolfi
(2010), which we adopt throughout this work.

######  Definition 3.0 (Explicit Communication).

Communication that is “deliberate where the sender has the goal of sharing
specific information with the collocutor” Breazeal et al. (2005).

######  Definition 3.0 (Implicit Communication).

Communication that conveys “information that [is] inherent in behavior but
which is not deliberately communicated” Breazeal et al. (2005).

######  Definition 3.0 (Implicit Signal).

The “signal that is generated by the actual physical position of the [agents]
and that is detected by the other [agents]” De Greeff and Nolfi (2010).

In the context of RL agents, an implicit signal refers to positional
information observed by an agent’s teammates as part of the environmental
state $s_{t}$ or observation $o_{t}$ space, whereas explicit communication
involves sending/receiving messages over a dedicated communication channel. We
note that Definitions 3.5 and 3.5 rely on deliberation (or intention), whereas
Definition 3.5 does not. For the purposes of our study—exploring the first-
step in a bottom-up approach to emergent communication—we therefore focus on
implicit signals as a form of emergent implicit communication.

###  3.6. Instantaneous Coordination

Instantaneous Coordination (IC) is a measure of social influence between
agents Jaques et al. (2019). IC is defined for two agents $i$ and $j$ as the
mutual information $\textrm{I}(a_{i}^{t};a_{j}^{t+1})$ between $i$’s action at
time $t$ and $j$’s action at the next time-step. Formally, assuming agent
$i$’s actions are drawn from the random variable $A_{i}$ with marginal
distribution $P_{A_{i}}$ (and similarly for agent $j$), we can rewrite IC
using the standard definition of mutual information as the Kullback-Leibler
divergence between the joint distribution and the product of the marginals:

$\displaystyle I(A_{i};A_{j})$
$\displaystyle=D_{\textrm{KL}}(P_{A_{i}A_{j}}||P_{A_{i}}\times P_{A_{j}})$
$\displaystyle=\sum_{\begin{subarray}{c}a_{i}\in\mathcal{A}_{i},\\\
a_{j}\in\mathcal{A}_{j}\end{subarray}}P_{A_{i}A_{j}}(a_{i},a_{j})\log\bigg{(}\frac{P_{A_{i}A_{j}}(a_{i},a_{j})}{P_{A_{i}}(a_{i})\times
P_{A_{j}}(a_{j})}\bigg{)}$

where $\mathcal{A}_{i}$ and $\mathcal{A}_{j}$ are the spaces over $A_{i}$ and
$A_{j}$, respectively. Intuitively, high IC is indicative of influential
behavior, while low IC indicates that agents are acting independently. We
highlight that, in the absence of explicit communicative actions, IC is a
measure of implicit signals (as defined in the previous section). This is
because agent $i$’s action at time $t$ inherently dictates $i$’s position
$q_{i}$ at time $t+1$, which is observed by $i$’s teammates. IC in this
context is therefore a measure of position-based social influence. Since this
work considers deterministic policies and uses IC at test-time, IC is computed
over Monte-Carlo estimates of the relevant distributions.

##  4\. Method

Our goal is to learn multi-agent coordination with decentralized training and,
in doing so, explore the role of implicit communication in teamwork. However,
there are a number of issues with decentralized learning in difficult, sparse
reward environments, such as our pursuit-evasion game. First, though setting
$\lvert\@vec{v}_{p}\rvert\leq\lvert\@vec{v}_{e}\rvert$ is important for
studying teamwork, it places the pursuers at a severe deficit. In the early
stages of training—since each pursuer’s action selection is determined by the
randomly initialized weights of its policy network—the chance of slower
pursuers capturing the evader defined in Equation (6) is extremely low. The
pursuers are unlikely to obtain a positive reward signal, which is vital to
improving their policies. This issue is exacerbated by non-stationarity. In
the case of decentralized DDPG, multiple agents learning in the same
environment causes the value of state-action pairs for any one agent (as
judged by its Q-function) to change as a result of policy updates of other
agents. This non-stationarity leads to higher-variance gradient estimates and
unstable learning. Though recent advances in “centralized training,
decentralized execution” help in such cases Lowe et al. (2017), they violate
our goal of decentralized learning.

To address these challenges, we introduce a curriculum-driven method for
decentralized multi-agent learning in sparse reward environments. Our strategy
is motivated by the following principles: (i) we can create a curriculum over
task difficulty by shaping the environment to the current skill level of the
cooperative agents; (ii) we can aide agents in learning multi-agent team
behaviors by forcing them to learn single-agent behaviors first. More
specifically, we combine an environment shaping curriculum over agent
velocities with a behavioral curriculum for bootstrapping cooperative multi-
agent learning with successful single-agent experience. We refer to this
curriculum-driven variant of DDPG as CD-DDPG throughout the rest of this
paper.

###  4.1. Velocity Ratio Curriculum

Curriculum learning Bengio et al. (2009) is a popular technique for solving
complex learning problems by breaking them down into smaller, easier to
accomplish tasks. Recent advances in ecological reinforcement learning Co-
Reyes et al. (2020) have introduced environment shaping—in which properties,
initial states, or the dynamics of an environment are modified gradually—as a
useful and more natural instantiation of curriculum learning for RL agents
(than, say, traditional reward shaping). With this in mind, we construct a
sequence of increasingly difficult pursuit-evasion environments by
incrementally lowering the ratio of pursuer speed to evader speed
($\lvert\@vec{v}_{p}\rvert/\lvert\@vec{v}_{e}\rvert$).

More formally, we define a curriculum over velocity bounds. Let $\@vec{v}_{0}$
be an initial setting of the environment’s velocity ratio
$\lvert\@vec{v}_{p}\rvert/\lvert\@vec{v}_{e}\rvert$. We anneal
$\lvert\@vec{v}_{p}\rvert/\lvert\@vec{v}_{e}\rvert$ to a target ratio
$\@vec{v}_{\textrm{target}}$ over $v_{\textrm{decay}}$ epochs as:

$\@vec{v}_{i}\leftarrow\@vec{v}_{\textrm{target}}+(\@vec{v}_{0}-\@vec{v}_{\textrm{target}})*\max\bigg{(}\frac{(v_{\textrm{decay}}-i)}{v_{\textrm{decay}}},0.0\bigg{)}$
(7)

where $i$ represents the current training epoch and, in turn, $\@vec{v}_{i}$
the current velocity ratio. In practice, we initialize the environment such
that<EMAIL_ADDRESS>(i.e. pursuers are faster than the evader), then
anneal this ratio slowly as training progresses. This gives the pursuers an
opportunity enjoy a velocity advantage early on in training, then develop
increasingly coordinated strategies to capture the evader as
$\lvert\@vec{v}_{p}\rvert/\lvert\@vec{v}_{e}\rvert$ decays.

###  4.2. Behavioral Curriculum

We combine the aforementioned velocity curriculum with a behavioral curriculum
that extends off-policy learning to allow for targeted single-agent
exploration early in the training process. As a consequence of the
deterministic policy gradient and importance sampling Silver et al. (2014),
the policy gradient can be estimated in an off-policy manner—i.e. using
trajectories sampled from a separate behavior policy $\beta(a|s)$ where
$\beta(a|s)\neq\mu_{\phi}$. Formally, this means Equation (2) can be
represented equivalently as:

$\displaystyle J_{\beta}(\phi)$
$\displaystyle=\int_{\mathcal{S}}p^{\beta}Q_{\omega}(s,a)|_{s=s_{t},a=\mu_{\phi}(s_{t})}\textrm{d}s$
$\displaystyle=\mathbb{E}_{s\sim
p^{\beta}}[Q_{\omega}(s,a)|_{s=s_{t},a=\mu_{\phi}(s_{t})}]$

with the corresponding gradient:

$\nabla_{\phi}J_{\beta}(\phi)=\mathbb{E}_{s\sim
p^{\beta}}[\nabla_{a}Q_{\omega}(s,a)|_{s=s_{t},a=\mu(s_{t})}\nabla_{\phi}\mu(s)|_{s=s_{t}}]$

where $p^{\beta}$ is the state distribution of the behavior policy $\beta$.

Our behavioral curriculum takes advantage of off-policy learning by splitting
training into two exploration phases that use distinct behavior policies
$\beta_{0}$ and $\beta_{\mu}$, respectively. The key is that we define
$\beta_{0}$ strategically to be a supervisory policy that collects successful
single-agent experience and $\beta_{\mu}$ to be a standard exploration policy
($\epsilon-$greedy for discrete actions, random noise for continuous actions).
In multi-agent settings, defining $\beta_{0}$ this way allows each agent to
learn how to interact with the environment first, before learning to
coordinate with teammates. Without this curriculum, agents are forced to learn
both simultaneously.

In our pursuit-evasion environment, we define $\beta_{0}$ as follows:

$\beta_{0}=-\nabla U_{\textrm{att}}(q_{p_{i}},q_{e})$ (8)

where $U_{\textrm{att}}$ is the attractive potential-field defined in Equation
(4). Note that $\beta_{0}$ is a greedy policy that runs directly towards the
evader. This strategy is obviously sub-optimal when
$\lvert\@vec{v}_{p}\rvert/\lvert\@vec{v}_{e}\rvert\leq 1.0$, but helps
pursuers learn to move in the direction of the evader when
$\lvert\<EMAIL_ADDRESS>In our experiments,
we use $\beta_{0}$ only during the first phase of the velocity curriculum.
After this phase, agents follow the standard DDPG behavior policy
$\beta_{\mu}=\mu_{\phi}(s_{t})+\mathcal{N}$, where $\mathcal{N}$ is the
Ornstein-Uhlenbeck noise process.

##  5\. Results

Our evaluation addresses two primary questions: (i) Does our curriculum-driven
strategy enable decentralized agents to learn to coordinate in difficult
sparse reward environments? (ii) To what extent does implicit communication
emerge in the learned strategy? To answer these questions, we first perform an
ablation study over each of the multi-agent curricula that comprise CD-DDPG.
Then, we measure the performance of CD-DDPG against a set of analytical and
learned strategies of increasing sophistication. We intentionally select these
strategies to represent ablations over both coordination and implicit
communication.

###  5.1. Multi-Agent Curricula

Figure 2. An ablation study of curriculum-driven learning early in the
training process. Note that
$\lvert\@vec{v}_{p}\rvert/\lvert\@vec{v}_{e}\rvert$ is decreasing over time
(increasing the task difficultly), so learning is characterized by sustained
reward rather than improved reward. Only CD-DDPG successfully learns team
coordination.

To isolate our method’s performance, we run ablations over each of CD-DDPG’s
constituent parts. We compare CD-DDPG to the following alternatives: No
Behavioral, which follows the velocity curriculum, but not the behavioral one;
No Velocity, which follows the behavioral curriculum, but not the velocity
one; and No Curriculum, which uses neither curriculum during training (i.e. it
is vanilla DDPG trained at a constant ratio
$\lvert\@vec{v}_{p}\rvert/\lvert\@vec{v}_{e}\rvert=0.7$). We trained each
method for $15,000$ epochs in our pursuit-evasion environment across 10
different random seeds. Results are shown in Figure 2.

No Curriculum, unsurprisingly, flat-lines throughout the training process. It
never experiences a positive reward signal, reflecting the difficulty of
pursuit-evasion when $\lvert\<EMAIL_ADDRESS>No Behavioral, trained from a velocity ratio of
$\lvert\<EMAIL_ADDRESS>downwards, experiences
small spikes in reward, but not consistently enough to improve the pursuers’
policies. Finally, No Velocity also fails to gain any traction (it’s curve is
behind No Curriculum in the figure), highlighting the importance of both
curricula working together. This validates our intuition from Section 4. Even
with a velocity advantage, the agents struggle to learn individual interaction
with the environment and coordination with their teammates simultaneously. CD-
DDPG is able to capture valuable experience even in the earliest stages of
training. This bootstraps each pursuer’s learning process, allowing them to
maintain a high level of performance even after the warm-up is over. We
therefore find that the combination of both curricula is crucial to warm-
starting policy learning.

###  5.2. Properties of Emergent Teamwork

Table 1. A summary of the strategies used to evaluate CD-DDPG and their capacity for coordination and implicit communication. Outperforming one of these methods is equivalent to surpassing that method’s level of sophistication in coordination, capacity for implicit communication, or both. Name | Coordination | Implicit Communication
---|---|---
Greedy |  No |  No
CD-DDPG (Partial) |  Yes |  No
Pincer |  Yes |  Yes

To study the strength of CD-DDPG’s coordination further and identify the role
of implicit communication in the agents’ emergent teamwork, we compare the
performance of CD-DDPG to a set of analytical and learned strategies of
increasing sophistication. These strategies represent ablations over both
coordination and implicit communication—i.e. outperforming one of these
methods is equivalent to surpassing that method’s level of sophistication in
coordination, capacity for implicit communication, or both. With this in mind,
we evaluate CD-DDPG against the following policies, which are also summarized
in Table 1111We do not evaluate against centralized methods such as MADDPG
Lowe et al. (2017) because they violate our requirement of decentralized
learning.:

##### Greedy

Each pursuer follows the greedy control strategy in Equation (4). In Greedy
pursuit, each pursuer ignores the positions of its teammates. Greedy pursuit
therefore represents independent action (i.e. no coordination, no
communication).

##### CD-DDPG (Partial)

We train CD-DDPG under partial observability. Instead of the complete
environment state $s_{t}$, each pursuer $p_{i}$ receives a private observation
$o_{t}=\\{q_{p_{i}},q_{e}\\}$ consisting of its own location $q_{p_{i}}$ and
the location of the evader $q_{e}$. Despite not observing each other, CD-DDPG
(Partial) pursuers are capable of coordinating through static role assignment.
This is equivalent to assigning roles before each trajectory—i.e. $p_{1}$
always approaches from the left, $p_{2}$ from the right, etc.—and coordinating
through these roles during pursuit. CD-DDPG (Partial) pursuers are therefore
coordinated, but with no ability to communicate implicitly to modify their
behavior extemporaneously.

##### Pincer

We define the Pincer strategy as an adversarial function that exploits
knowledge of the evader’s objective in Equation (6):

$\displaystyle F(\boldsymbol{\tilde{\theta}_{i},r_{i}})$
$\displaystyle=\underset{\boldsymbol{\tilde{\theta}_{i},r_{i}}}{\max}\big{[}\underset{\theta_{e}}{\min}\big{[}U_{\textrm{evade}}(\theta_{e})\big{]}\big{]}$
(9)
$\displaystyle=\underset{\boldsymbol{\tilde{\theta}_{i},r_{i}}}{\max}\bigg{[}\underset{\theta_{e}}{\min}\bigg{[}\sum_{i}\bigg{(}\frac{1}{r_{i}}\bigg{)}\cos(\theta_{e}-\tilde{\theta}_{i})\bigg{]}\bigg{]}$
(10)

where $\boldsymbol{\tilde{\theta}_{i}}$ and $\boldsymbol{r_{i}}$ are the polar
coordinates of each pursuer relative the evader. Intuitively, Pincer encircles
the evader and cuts off potential bisector escape paths while enclosing the
circle. It therefore supports both coordination—pursuers uphold a circular
formation—and implicit communication—every time-step, pursuers use information
from the locations of their teammates to adjust their own position on the
circular formation. We provide additional details on the Pincer strategy in
Appendix A.3.

All experiments involve $n=3$ pursers. For CD-DDPG and CD-DDPG (Partial),
agents are trained for $50,000$ epochs. The velocity ratio is decreased from
$\lvert\<EMAIL_ADDRESS>to
$\lvert\<EMAIL_ADDRESS>in decrements of $0.1$
(i.e. eight separate training sessions each). Velocity ratio decay occurs over
$v_{\textrm{decay}}=15000$ epochs, during which the behavioral curriculum
$\beta_{0}$ is used (but only during the first training session). After
training, we test the resulting policies at each velocity step. Test-time
performance is measured over 100 independent trajectories (averaged over five
random seeds each).

####  5.2.1. Capture Success

Figure 3. Capture success as a function of velocity. CD-DDPG succeeds at
$\lvert\<EMAIL_ADDRESS>whereas each competing
method fails below $\lvert\<EMAIL_ADDRESS>
First, we evaluate capture success as a function of the velocity advantage of
the evader. The results are shown in Figure 3. Unsurprisingly, each method has
a high capture success rate when
$\lvert\<EMAIL_ADDRESS>The Greedy strategy
drops off at $\lvert\<EMAIL_ADDRESS>which is
also expected—a straight-line chase only works when
$\lvert\<EMAIL_ADDRESS>The Pincer and CD-
DDPG (Partial) strategies are able to coordinate successfully at lower speeds,
but eventually fail to capture the evader at
$\boldsymbol{\lvert\<EMAIL_ADDRESS>and
below. As $\lvert\@vec{v}_{p}\rvert/\lvert\@vec{v}_{e}\rvert$ decreases
further, CD-DDPG significantly outperforms the other strategies. Specifically,
CD-DDPG successfully completes the pursuit-evasion task at a speed ratio of
0.5 (i.e. pursuers moving at half the evader’s speed). These results show that
CD-DDPG learns to coordinate significantly more effectively than the other
strategies.

####  5.2.2. Relative position during capture

Figure 4. Distribution of relative angle for each pursuer during capture,
binned in the range $[0,2\pi]$ and displayed as a heatmap. Each row represents
a pursuit strategy.

Next, we study the nature of each strategy’s coordination with finer
granularity. In sparse reward tasks, the most important time-steps are those
that immediately precede reward. For this reason, we look at the distribution
of pursuer locations at the time of capture. We collect 100 successful
trajectories from each strategy and compute the distribution of pursuer
positions relative the evader. In general, we expect coordinated pursuit to
exhibit rotational symmetry during capture. Rotational symmetry suggests that
pursuers have learned strategies which lead them to well-defined “capture
points” around the evader. Conversely, rotational invariance is indicative of
independent pursuit—i.e. pursuers do not follow concrete patterns of attack.
Results for this study are shown in Figure 4.

We find that Greedy and Pincer both yield uniform capture distributions. This
is unsurprising for Greedy pursuers, whose pursuit paths are not effected by
their teammates. The Pincer strategy encircles the evader, but does not
constrain pursuers to specific locations on the circle. This leads to less-
structured capture(i.e. weaker role assignment) by the Pincer pursuers. In
contrast, CD-DDPG (Partial) pursuers demonstrate very strong role assignment,
with each pursuer capturing the evader from the same relative angle each time.
Taken into context with the results from Figure 3, it is clear that this level
of rotational symmetry impacts success. In fact, it is an example of over-
commitment to role assignment. The pursuers adopt very constrained roles—e.g.
“$p_{1}$ always move left”, “$p_{2}$ always move right”—which works when
$\lvert\@vec{v}_{p}\rvert/\lvert\@vec{v}_{e}\rvert\geq 1.0$, but fails at
lower velocities. CD-DDPG balances rotational symmetry and invariance. Each
pursuer follows a unique angle towards the evader, but does not commit to that
angle completely. CD-DDPG therefore learns structured coordination while
allowing pursuers to make dynamic adjustments to their position relative the
evader to achieve capture.

####  5.2.3. Position-based Social influence

Figure 5. Instantaneous Coordination (IC) results as a function of velocity
ratios. Agents trained with CD-DDPG exchange up to 0.375 bits of information
per time-step on average, increasing as
$\lvert\@vec{v}_{p}\rvert/\lvert\@vec{v}_{e}\rvert$ drops, whereas competing
strategies peaks remains constant.

To further study the role of implicit communication in pursuer performance, we
compute the IC score for each strategy. As noted in, section 3, by measuring
the amount that one agent’s actions (and therefore its next position)
influences the actions of its teammates, IC quantifies the exchange of
implicit signals (see Definition 3.5) amongst teammates. Following Jaques et
al. (2019), we compute IC empirically as a Monte-Carlo approximation over
multi-agent trajectories. We average influence across all trajectory steps and
for each agent-agent pair. We also evaluate pursuers that act randomly, which
provides a baseline for independent action. The results are shown in Figure 5.

We find that, as $\lvert\@vec{v}_{p}\rvert/\lvert\@vec{v}_{e}\rvert$
decreases, the IC levels attained by CD-DDPG increases significantly, whereas
it remains stagnant for other methods. In fact, across all
$\lvert\@vec{v}_{p}\rvert/\lvert\@vec{v}_{e}\rvert$ levels, we find that
pursuers trained with CD-DDPG exchange up to 0.375 bits of information per
time-step on average, compared to a maximum of only 0.15 bits on average from
the baseline methods. This indicates that CD-DDPG achieves increasingly
complex coordination as task difficulty increases and is a promising sign that
the CD-DDPG team is exchanging implicit signals—i.e. each pursuer is
responding to positional information from its teammates. Finally, we note a
minor (though surprising) increase in coordination for the greedy pursuers at
low velocities. This is an artifact we dub “phantom coordination” and discuss
in the Appendix B.2.

####  5.2.4. High-influence moments

Figure 6. Percentage of high-influence moments as a function of velocity
ratios. Agents trained with CD-DDPG become more influential as
$\lvert\@vec{v}_{p}\rvert/\lvert\@vec{v}_{e}\rvert$ decreases.

We also report on the percentage of high-influence moments that occur between
pairs of agents (see Figure 6). A high-influence moment is a time-step in
which IC is above the mean IC for the entire trajectory. Similar to previous
work Jaques et al. (2019), we find that influence is sparse in most cases—only
between 10-15% of trajectory steps exhibit high-influence across all
$\lvert\@vec{v}_{p}\rvert/\lvert\@vec{v}_{e}\rvert$ levels. The exception,
notably, occurs for CD-DDPG. At low speeds (i.e.
$\lvert\@vec{v}_{p}\rvert/\lvert\@vec{v}_{e}\rvert\leq 0.7$), we see a
significant increase in the percentage of high-influence moments between the
CD-DDPG pursuers, reaching a maximum near 30%. This is further evidence that,
as $\lvert\@vec{v}_{p}\rvert/\lvert\@vec{v}_{e}\rvert$ decreases, CD-DDPG
pursuers form increasingly highly-coordinated formations and make split-second
decisions based on the movements of their teammates. This points more
concretely to the use of implicit signals between CD-DDPG pursuers.

Interestingly, the behavior of CD-DDPG closely matches the documented
behaviors of social predators such as dolphins and wolves—i.e. sudden changes
of position/orientation as a response to the movements of other teammates
Herbert-Read (2016). We elaborate on these and other qualitative findings in
Appendix B.

##  6\. Conclusion

This work connects emergent communication to the spectrum of communication
that exists in nature, highlighting the importance of interpreting
communication as a spectrum from implicit to explicit communication. We
proposed a curriculum-driven strategy for policy learning in difficult multi-
agent environments. Experimentally, we showed that our curriculum-driven
strategy enables pursuers to coordinate and capture a superior evader,
outperforming other highly-sophisticated analytic pursuit strategies. We also
provided evidence suggesting that the emergence of implicit communication is a
key contributor to the success of this strategy. There are a number of
extensions of this work that study how common principles contribute to
integrated, communicative behavior; including: imperfect state information,
increased environmental complexity, and nuanced social dynamics between
agents.

## References

  * (1)
  * Baker et al. (2019) B Baker, I Kanitscheider, T Markov, et al. 2019\.  Emergent tool use from multi-agent autocurricula.  _arXiv preprint:1909.07528_ (2019).
  * Bard et al. (2020) N Bard, J N Foerster, et al. 2020\.  The hanabi challenge: A new frontier for ai research.  _Artificial Intelligence_ (2020).
  * Bellemare et al. (2016) M Bellemare, S Srinivasan, G Ostrovski, T Schaul, D Saxton, and R Munos. 2016\.  Unifying count-based exploration and intrinsic motivation.  _NeurIPS_ (2016).
  * Bengio et al. (2009) Y Bengio, J Louradour, R Collobert, et al. 2009\.  Curriculum learning. In _ICML_.
  * Boesch and Boesch (1989) C Boesch and H Boesch. 1989\.  Hunting behavior of wild chimpanzees in the Tai National Park.  _AJPA_ (1989).
  * Bradbury et al. (1998) J Bradbury, S Vehrencamp, et al. 1998\.  Principles of animal communication.  (1998).
  * Breazeal et al. (2005) C Breazeal, C Kidd, A Thomaz, G Hoffman, and M Berlin. 2005.  Effects of nonverbal communication on efficiency and robustness in human-robot teamwork. In _2005 IEEE/RSJ_. IEEE, 708–713.
  * Bshary et al. (2006) R Bshary, A Hohner, K Ait-el Djoudi, et al. 2006\.  Interspecific communicative and coordinated hunting between groupers and giant moray eels in the Red Sea.  _PLoS Biol_ (2006).
  * Butchibabu et al. (2016) A Butchibabu, C Sparano-Huiban, L Sonenberg, and J Shah. 2016.  Implicit coordination strategies for effective team communication.  _Human factors_ 58, 4 (2016), 595–610.
  * Chaabouni et al. (2020) R Chaabouni, E Kharitonov, D Bouchacourt, et al. 2020\.  Compositionality and generalization in emergent languages.  _arXiv preprint:2004.09124_ (2020).
  * Chaabouni et al. (2019) R Chaabouni, E Kharitonov, E Dupoux, et al. 2019\.  Anti-efficient encoding in emergent communication. In _NeurIPS_.
  * Che et al. (2020) Y Che, A Okamura, and D Sadigh. 2020.  Efficient and trustworthy social navigation via explicit and implicit robot–human communication.  _IEEE Transactions on Robotics_ 36, 3 (2020), 692–707.
  * Choi et al. (2017) N Choi, J Kim, N Kokubun, et al. 2017\.  Group association and vocal behaviour during foraging trips in Gentoo penguins.  _Scientific Reports_ (2017).
  * Chung et al. (2011) T Chung, G Hollinger, and V Isler. 2011.  Search and pursuit-evasion in mobile robotics.  _Autonomous robots_ (2011).
  * Co-Reyes et al. (2020) J Co-Reyes, S Sanjeev, G Berseth, A Gupta, and S Levine. 2020.  Ecological Reinforcement Learning.  _arXiv preprint arXiv:2006.12478_ (2020).
  * Cohen and Levesque (1991) P Cohen and H J Levesque. 1991.  Teamwork.  _Nous_ 25, 4 (1991), 487–512.
  * Cowen-Rivers and Naradowsky (2020) A Cowen-Rivers and J Naradowsky. 2020.  Emergent Communication with World Models.  _arXiv preprint:2002.09604_ (2020).
  * Das et al. (2019) A Das, T Gervet, J Romoff, et al. 2019\.  Tarmac: Targeted multi-agent communication. In _ICML_.
  * De Greeff and Nolfi (2010) J De Greeff and S Nolfi. 2010.  Evolution of implicit and explicit communication in mobile robots.  In _Evolution of communication and language in embodied agents_. Springer, 179–214.
  * de Witt et al. (2020) C de Witt, T Gupta, D Makoviichuk, V Makoviychuk, P Torr, M Sun, and S Whiteson. 2020.  Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?  _arXiv preprint arXiv:2011.09533_ (2020).
  * DeepMind (2021) Open Ended Learning Team DeepMind. 2021\.  Open-ended learning leads to generally capable agents.  _arXiv preprint arXiv:2107.12808_ (2021).
  * Dennis et al. (2020) M Dennis, N Jaques, E Vinitsky, et al. 2020\.  Emergent complexity and zero-shot transfer via unsupervised environment design.  _NeurIPS_ (2020).
  * Eccles et al. (2019) T Eccles, Y Bachrach, G Lever, et al. 2019\.  Biases for emergent communication in multi-agent reinforcement learning. In _NeurIPS_.
  * Entin and Serfaty (1999) E Entin and D Serfaty. 1999\.  Adaptive team coordination.  _Human factors_ (1999).
  * Florensa et al. (2017) C Florensa, D Held, M Wulfmeier, M Zhang, and P Abbeel. 2017.  Reverse curriculum generation for reinforcement learning. In _CoRL_. PMLR, 482–495.
  * Foerster et al. (2016) J Foerster, I Assael, N De Freitas, et al. 2016\.  Learning to communicate with deep multi-agent reinforcement learning. In _NeurIPS_.
  * Foerster et al. (2018a) J Foerster, G Farquhar, T Afouras, N Nardelli, and S Whiteson. 2018a.  Counterfactual multi-agent policy gradients. In _AAAI_ , Vol. 32.
  * Foerster et al. (2018b) J Foerster, F Song, E Hughes, et al. 2018b.  Bayesian action decoding for deep multi-agent reinforcement learning.  _arXiv preprint:1811.01458_ (2018).
  * Gildert et al. (2018) N Gildert, A Millard, A Pomfret, and J Timmis. 2018\.  The need for combining implicit and explicit communication in cooperative robotic systems.  _Frontiers in Robotics and AI_ 5 (2018), 65.
  * Grover and Sahai (2010) P Grover and A Sahai. 2010\.  Implicit and explicit communication in decentralized control. In _Allerton Conference on Communication, Control, and Computing_. IEEE, 278–285.
  * Havrylov and Titov (2017) S Havrylov and I Titov. 2017.  Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In _NeurIPS_.
  * Herbert-Read (2016) J Herbert-Read. 2016\.  Understanding how animal groups achieve coordinated movement.  _JEB_ (2016).
  * Isaacs (1999) R Isaacs. 1999.  _Differential games: a mathematical theory with applications to warfare and pursuit, control and optimization_.  Courier Corporation.
  * Jang and Tomlin (2005) J Jang and C Tomlin. 2005\.  Control strategies in multi-player pursuit and evasion game. In _AIAA ARC_.
  * Janosov et al. (2017) M Janosov, C Virágh, G Vásárhelyi, et al. 2017\.  Group chasing tactics: how to catch a faster prey.  _NJP_ (2017).
  * Jaques et al. (2019) N Jaques, A Lazaridou, E Hughes, et al. 2019\.  Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In _ICML_.
  * Knepper et al. (2017) R Knepper, C Mavrogiannis, J Proft, and C Liang. 2017\.  Implicit communication in a joint action. In _HRI_. 283–292.
  * Laurent et al. (2011) G J Laurent, L Matignon, L Fort-Piat, et al. 2011\.  The world of independent learners is not Markovian.  _International Journal of Knowledge-based and Intelligent Engineering Systems_ 15, 1 (2011), 55–64.
  * Lazaridou and Baroni (2020) A Lazaridou and M Baroni. 2020.  Emergent Multi-Agent Communication in the Deep Learning Era.  _arXiv preprint:2006.02419_ (2020).
  * Liang et al. (2019) C Liang, J Proft, E Andersen, and R Knepper. 2019\.  Implicit communication of actionable information in human-ai teams. In _CHI_.
  * Lillicrap et al. (2015) T Lillicrap, J Hunt, A Pritzel, et al. 2015\.  Continuous control with deep reinforcement learning.  _arXiv preprint:1509.02971_ (2015).
  * Littman (1994) M Littman. 1994\.  Markov games as a framework for multi-agent reinforcement learning.  In _ICML_.
  * Lowe et al. (2019) R Lowe, J Foerster, et al. 2019\.  On the pitfalls of measuring emergent communication.  _arXiv preprint:1903.05168_ (2019).
  * Lowe et al. (2017) R Lowe, Yi I Wu, A Tamar, et al. 2017\.  Multi-agent actor-critic for mixed cooperative-competitive environments. In _NeurIPS_.
  * Mason and Hollis (1962) W Mason and J Hollis. 1962\.  Communication between young rhesus monkeys.  _Animal Behaviour_ (1962).
  * Matignon et al. (2012) L Matignon, G J Laurent, and N Le Fort-Piat. 2012.  Independent reinforcement learners in cooperative markov games: a survey regarding coordination problems.  _The Knowledge Engineering Review_ (2012).
  * Mordatch and Abbeel (2017) I Mordatch and P Abbeel. 2017.  Emergence of grounded compositional language in multi-agent populations.  _arXiv preprint:1703.04908_ (2017).
  * Muro et al. (2011) C Muro, R Escobedo, L Spector, et al. 2011\.  Wolf-pack hunting strategies emerge from simple rules in computational simulations.  _Behavioural processes_ (2011).
  * Neumann and Orams (2003) D Neumann and M Orams. 2003\.  Feeding behaviours of short-beaked common dolphins, Delphinus delphis, in New Zealand.  _Aquatic Mammals_ (2003).
  * Pagello et al. (1999) E Pagello, A D’Angelo, et al. 1999\.  Cooperative behaviors in multi-robot systems through implicit communication.  _Robotics and Autonomous Systems_ (1999).
  * Parsons (1978) T Parsons. 1978\.  Pursuit-evasion in a graph.  In _TAG_.
  * Pathak et al. (2017) D Pathak, P Agrawal, A Efros, and T Darrell. 2017\.  Curiosity-driven exploration by self-supervised prediction. In _ICML_.
  * Peterson and Ciucci (2003) R Peterson and P Ciucci. 2003.  The wolf as carnivore.  _Wolves: Behavior, Ecology, and Conservation_ (2003).
  * Portelas et al. (2020a) R Portelas, C Colas, K Hofmann, and P Oudeyer. 2020a.  Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In _CoRL_. PMLR, 835–853.
  * Portelas et al. (2020b) R Portelas, C Colas, L Weng, K Hofmann, and P Oudeyer. 2020b.  Automatic curriculum learning for deep rl: A short survey.  _arXiv preprint arXiv:2003.04664_ (2020).
  * Quick and Janik (2012) N Quick and V Janik. 2012\.  Bottlenose dolphins exchange signature whistles when meeting at sea.  _Biological Sciences_ (2012).
  * Ramana and Kothari (2017) M Ramana and M Kothari. 2017.  Pursuit-evasion games of high speed evader.  _JINT_ (2017).
  * Resnick et al. (2019) C Resnick, A Gupta, J Foerster, et al. 2019\.  Capacity, bandwidth, and compositionality in emergent language learning.  _arXiv preprint:1910.11424_ (2019).
  * Risi and Togelius (2019) S Risi and J Togelius. 2019\.  Procedural content generation: From automatically generating game levels to increasing generality in machine learning.  (2019).
  * Sampaio et al. (2020) E Sampaio, M Seco, R Rosa, et al. 2020\.  Octopuses punch fishes during collaborative interspecific hunting events.  _Ecology_ (2020).
  * Schaller (2009) G Schaller. 2009\.  _The Serengeti lion: a study of predator-prey relations_.  UChicago Press.
  * Schmidt (2000) A Schmidt. 2000\.  Implicit human computer interaction through context.  _Personal technologies_ (2000).
  * Silver et al. (2014) D Silver, G Lever, N Heess, T Degris, D Wierstra, and M Riedmiller. 2014\.  Deterministic policy gradient algorithms.
  * Tan (1993) M Tan. 1993.  Multi-agent reinforcement learning: Independent vs. cooperative agents. In _ICML_.
  * Vail et al. (2013) A Vail, A Manica, and R Bshary. 2013.  Referential gestures in fish collaborative hunting.  _Nature Communications_ (2013).
  * Vaswani et al. (2017) A Vaswani, N Shazeer, N Parmar, et al. 2017\.  Attention is all you need. In _NeurIPS_.
  * Vicsek (2010) T Vicsek. 2010.  Closing in on evaders.  _Nature_ (2010).
  * Vidal et al. (2002) R Vidal, O Shakernia, H Kim, et al. 2002\.  Probabilistic pursuit-evasion games: theory, implementation, and experimental evaluation.  _IEEE TRA_ (2002).
  * Wang et al. (2020) Y Wang, L Dong, and C Sun. 2020.  Cooperative control for multi-player pursuit-evasion games with reinforcement learning.  _Neurocomputing_ (2020).
  * Wuertz et al. (2018) J Wuertz, S Alharthi, W Hamilton, S Bateman, C Gutwin, A Tang, Z Toups, and J Hammer. 2018.  A design framework for awareness cues in distributed multiplayer games. In _CHI_.
  * Yang et al. (2018) J Yang, A Nakhaei, D Isele, K Fujimura, and H Zha. 2018.  Cm3: Cooperative multi-goal multi-stage multi-agent reinforcement learning.  _arXiv preprint arXiv:1809.05188_ (2018).
  * Zhou et al. (2017) W Zhou, W Yan, and X Zhang. 2017.  Collaboration for success in crowdsourced innovation projects: Knowledge creation, team diversity, and tacit coordination. In _HICSS_.

##  Appendix A Pursuit-Evasion Details

###  A.1. Pursuit-Evasion Environment

Figure 7. A planar pursuit-evasion game with periodic boundary conditions
interpreted as a toroidal pursuit-evasion environment.

Our environment is a toroidal extension of the planar pursuit-evasion game
proposed by Lowe et al. (2017). In general, unbounded planar pursuit-evasion
can be described by two cases:

  * •

Case 1: $\lvert\@vec{v}_{p}\rvert>\lvert\@vec{v}_{e}\rvert$. The game is
solved by a straight-line chase towards the evader and is not interesting from
the perspective of coordination.

  * •

Case 2: $\lvert\@vec{v}_{p}\rvert\leq\lvert\@vec{v}_{e}\rvert$. The evader has
a significant advantage. Pursuers have at most one opportunity to capture the
evader and are usually only successful under strict initialization conditions
Ramana and Kothari (2017).

Lowe et al. (2017) addressed this by penalizing agents for leaving the
immediate area defined by the camera with negative reward. The evader defined
by Equation (6), however, will run away indefinitely in the
$\lvert\@vec{v}_{p}\rvert\leq\lvert\@vec{v}_{e}\rvert$ case. To provoke
consistent interaction between agents, we extend the planar environment with
periodic boundary conditions. One can think of this as playing the pursuit-
evasion game on a torus (see Figure 7).

We argue that toroidal pursuit-evasion is better suited for our problem
setting. First, it does not require strict initialization conditions or
special rewards. We can initialize pursuers randomly and allow them to
construct ad-hoc formations. Second, pursuit is no longer a one-and-done
proposition. This reflects the notion that, in nature, predators often do not
give up after a single attempt at a prey—they regroup and pursue it again.

###  A.2. Evader Strategy Explained

####  A.2.1. Derivation

The goal of the evader strategy in Equation (6) is to run from pursuers along
the maximum bisector between two pursuers. Given pursuer positions
$\\{q_{p_{1}},...,q_{p_{n}}\\}$, we compute polar coordinates:

$\displaystyle r_{i}$ $\displaystyle=d(q_{e},q_{p_{i}})$
$\displaystyle\tilde{\theta}_{i}$
$\displaystyle=\textrm{atan2}(y_{p_{i}},x_{p_{i}})$

for each pursuer $p_{i}$ relative the evader. Next, we define a potential
field that will push the evader towards a bisector:

$U(\theta_{e})=\sum_{i}\cos(\theta_{e}-\tilde{\theta}_{i})$

Using Ptolemy’s difference formula, we can expand the potential field as:

$\displaystyle U(\theta_{e})$
$\displaystyle=\sum_{i}\cos(\theta_{e}-\tilde{\theta}_{i})$
$\displaystyle=\sum_{i}\cos(\theta_{e})\cos(\tilde{\theta}_{i})+\sin(\theta_{e})\sin(\tilde{\theta}_{i})$
$\displaystyle=A\cos(\theta_{e})+B\sin(\theta_{e})$

when we plug-in the known $\tilde{\theta}_{i}$ values. The function
$U(\theta_{e})$ is maximized/minimized for values of $A$ and $B$ such that:

$\nabla U(\theta_{e})=-A\sin(\theta_{e})+B\cos(\theta_{e})=0$

which simplifies to:

$\tan(\theta_{e})=\frac{B}{A}$

The evader follows the direction of the negative gradient ($-\nabla
U(\theta_{e})$) and pursues it at maximum speed. Modulating the cost function
by $r_{i}$:

$U(\theta_{e})=\sum_{i}\bigg{(}\frac{1}{r_{i}}\bigg{)}\cos(\theta_{e}-\tilde{\theta}_{i})$

allows the evader to modify its bisector based on the distance to each
pursuer. This helps significantly when the evader is stuck in symmetric
formations.

####  A.2.2. Unit Tests

Figure 8. Unit tests simulating one time-step of action selection as dictated
by the evader’s cost function. (a) Given pursuers at relative angles
$\tilde{\theta}_{1}=0,\tilde{\theta}_{2}=\pi/2,\tilde{\theta}_{3}=\pi$ the
evader will select the heading $\theta_{e}=-\pi/2$, which minimizes its cost.
(b) Similarly, the evader will select the heading $\theta_{e}=-\pi=\pi$ when
pursuers are located at relative angles
$\tilde{\theta}_{1}=0,\tilde{\theta}_{2}=\pi/2,\tilde{\theta}_{3}=-\pi/2$.

We include a set of “unit tests” that shed light on the evader’s decision-
making behavior. We assume $n=3$ pursuers are stationed around the evader at
relative angles $\tilde{\theta}_{1}$, $\tilde{\theta}_{2}$, and
$\tilde{\theta}_{3}$. For simplicity, we initialize the pursuers such that
$\forall{i},r_{i}=1$ to negate the effects of radius modulation.

  * •

Case 1:
$\tilde{\theta}_{1}=0,\tilde{\theta}_{2}=\pi/2,\tilde{\theta}_{3}=\pi$.
Pursuers are spaced equally around the upper-half of the unit circle. In this
case, the cost minimizer occurs for $\theta_{e}=-\pi/2$ (see Figure 8a).

  * •

Case 2:
$\tilde{\theta}_{1}=0,\tilde{\theta}_{2}=\pi/2,\tilde{\theta}_{3}=-\pi/2$.
Pursuers are spaced equally around the right-half of the unit circle. In this
case, the cost minimizer occurs for $\theta_{e}=-\pi$ and $\theta_{e}=\pi$(see
Figure 8b). Either solution can be selected to move the evader towards the
largest opening.

In general, the cosine function imposes structure on the evader’s objective—it
will oscillate between $[-1,1]$ over a period of $\pi$, taking on a maximum
value of $U(\theta_{e})=1$ when the difference between the evader’s heading
$\theta_{e}$ and the relative angle of a pursuer $\tilde{\theta}_{i}$ is zero
and a minimum $U(\theta_{e})=-1$ when $\theta_{e}-\tilde{\theta}_{i}=\pi$.
Summing over all $\tilde{\theta}_{i}$’s incentivizes the evader to follow the
heading that splits the largest bisector of the pursuers, as shown in the
examples.

###  A.3. Pincer Pursuit Explained

The Pincer strategy described by Equation (9) is inspired by prior work on
theoretical pursuit-evasion Ramana and Kothari (2017). Solving Equation (9)
requires optimizing over both $\boldsymbol{\tilde{\theta}_{i}}$ and
$\boldsymbol{r_{i}}$. Fortunately, we can exploit the toroidal structure of
the environment to construct an optimization routine that solves for
$\boldsymbol{\tilde{\theta}_{i}}$ and $\boldsymbol{r_{i}}$ discretely. In
particular, we can unroll the torus $k$ steps in each direction to generate
$(2k+1)^{2}$ replications of the current environment state. Rather than
solving for optimal $\boldsymbol{\tilde{\theta}_{i}}$ and $\boldsymbol{r_{i}}$
values directly, we find the set $\boldsymbol{P}$ of pursuers that maximize
Equation (9) across all replications of the environment. We constrain the
problem by limiting selections of each pursuer $p_{i}$ to replications of
itself only. This dramatically cuts down the number of possible sets
$\boldsymbol{P}$ from $\binom{(2k+1)^{2}n}{n}$ to
$\binom{(2k+1)^{2}}{1}\cdot\binom{(2k+1)^{2}}{1}\cdot\binom{(2k+1)^{2}}{1}$,
where $n$ is the number of pursuers in the environment. Thus, we solve
Equation (9) via a discrete optimization over each of the $((2k+1)^{2})^{3}$
possible pursuer selections.

Figure 9. The torus environment unrolled $k=1$ times in each direction. The
filled in red circles denote the “active” pursuers that are pursuing the
evader at the current time-step, whereas the empty circles represent
“inactive” pursuers. We consider only a single evader, located in the center
tile.

The resulting set $\boldsymbol{P}$ defines the set of “active” pursuers that
will pursue the evader directly at the next time-step. Due to the nature of
the evader’s objective function—it is attracted to bisectors and repulsed from
pursuers—the maximum $\boldsymbol{P}$ tends to favor symmetric triangular
formations. Though this method obviously does not scale well with $n$ and $k$,
we found that we are able to find a sufficient maximizer with low values of
$k$ (i.e. $k=1$ in our experiments). The replication process is shown for the
$k=1$ case in Figure 9. Note that we discriminate between “active”
pursuers—i.e. those $p_{i}\in P$ pursuing the evader at the current time-
step—from “inactive” pursuers.

##  Appendix B Qualitative Results

Figure 10. Qualitative results from the pursuit-evasion experiment. (a) The
pursuers coordinate to capture the evader, displaying positional shifts
indicative of implicit communication. (b) Top: A diagram of dolphin foraging
strategies documented in Neumann and Orams (2003). Middle: Photos of wolves
coordinating while hunting, as shown in Muro et al. (2011). Bottom: The
learned behavior of our multi-agent system.

###  B.1. Animalistic Coordination

We perform post-hoc qualitative analysis of CD-DDPG trajectories. In the
trajectories, the pursuers appear to adjust their position slightly in
response to the movements of fellow pursuers as they close in on the evader
(Figure 10a). Moreover, it seems that the pursuers occasionally move away from
the evader—something a less coordinated strategy would not do—to maintain the
integrity of the group formation. This explains the performance difference
between CD-DDPG and the competing analytical strategies, as the potential-
field pursuers have no basis for making small-scale adaptive movements. We
interpret these results as encouraging evidence that implicit communication
has emerged amongst CD-DDPG pursuers.

###  B.2. Phantom Coordination

In Section 5, we describe “phantom coordination” as independent action that is
falsely perceived as coordination from the perspective of IC. Phantom
coordination appears in the greedy pursuit strategy, where we see the IC score
for greedy pursuers increase slightly as the velocity of the pursuers
decreases. This is counter-intuitive because each greedy pursuer ignores the
behavior of its teammates. We would expect the IC score for greedy pursuers to
remain flat, mirroring the IC score of the random pursuers.

To diagnose phantom coordination in our environment, we perform qualitative
analysis of greedy pursuit at low velocities. In particular, we examine $n=3$
pursuers as they chase an evader at a speed of
$\lvert\<EMAIL_ADDRESS>The pursuers have no
chance of successfully capturing the evader, as evidenced by their capture
success performance at this velocity in Figure 3. However, we find that the
straight-line chase patterns of greedy pursuers form temporary triangular
patterns around the evader. In Figure 11, the greedy pursuers form an ad-hoc
triangular formation around the evader for a duration of 60 time-steps. The
leftmost pursuer lies equidistant from the evader around the periodic
boundaries and iterates between moving leftward and rightward, depending on
which direction creates a shorter line to the evader. The other two pursuers
approach the evader from above and below, respectively. This behavior causes
the evader to move in a zig-zag pattern in the center of the triangle until a
large opening appears, through which the evader can escape.

This behavior leads to phantom coordination because IC is computed between two
consecutive time-steps and averaged over whole trajectories. This means that,
in the case of the greedy pursuers, IC scores for independent actions are
averaged together with subsets of each trajectory that consist of seemingly
highly coordinated behavior.

Figure 11. Snapshots from a 60 time-step trajectory in which phantom
coordination appears. Though the pursuers are following independent greedy
strategies, their actions produce a triangular formation that is perceived as
coordination by the IC performance measure.

##  Appendix C Experiment details

###  C.1. Potential-Field Hyperparameters

The Greedy potential-field defined by Equation (4) is subject to a single
hyperparameter $k_{\textrm{att}}$, which defines the strength of the
attractive pull of the agent towards its goal. We set $k_{\textrm{att}}=1.5$
in all of our experiments.

###  C.2. Policy Learning Hyperparameters

Actors $\mu_{\phi}$ are trained with two hidden layers of size 128. Critics
$Q_{\omega}$ are trained with three hidden layers of size 128. We use learning
rates of $1\textrm{e}^{-4}$ and $1\textrm{e}^{-3}$ for the actor and critic,
respectively, and a gradient clip of 0.5. Target networks are updated with
Polyak averaging with $\tau=0.001$. We used a buffer $\mathcal{D}$ of length
$500000$ and sample batches of size $512$. We used a discount factor
$\gamma=0.99$. All values are the result of standard hyperparameter sweeps.

###  C.3. Experiments

The models for all experiments were trained for $50000$ epochs of $500$ steps.
We reserve $W=1000$ epochs for warm-up, during which the behavioral policy
$\beta_{0}$ is followed. Pursuer velocity is decayed over
$v_{\textrm{decay}}=15000$ epochs. Test-time performance, as in Figures 3, 5,
and 6 is averaged across 100 independent trajectories with separate random
seeds. For the ablation in Figure 2, each method was trained for $7500$ epochs
across 10 random seeds. All experiments leveraged an Nvidia GeForce GTX 1070
GPU with 8GB of memory.

###  C.4. Assets

The toroidal pursuit-evasion environment used in this work is an extension of
the pursuit-evasion environment introduced by Lowe et al. (2017). The original
environment is open-sourced on Github under the MIT license. We cited the
authors accordingly in the main text. New assets (i.e. the new pursuit-evasion
environment) are included in the supplementary material. None of the assets
used in this work contain personally identifiable information or offensive
content.