# A Visual Communication Map for Multi-Agent Deep Reinforcement Learning

Ngoc Duy Nguyen, Thanh Thi Nguyen, Doug Creighton, Saeid Nahavandi  Ngoc Duy
Nguyen, Doug Creighton and Saeid Nahavandi are with the Institute for
Intelligent Systems Research and Innovation, Deakin University, Waurn Ponds
Campus, Geelong, Victoria, Australia (e-mails<EMAIL_ADDRESS><EMAIL_ADDRESS>and saeid.nahavandi@deakin.edu.au). Thanh Thi
Nguyen is with the School of Information Technology, Deakin University,
Burwood Campus, Melbourne, Victoria, Australia (e-mail:
thanh.nguyen@deakin.edu.au).

###### Abstract

Deep reinforcement learning has been applied successfully to solve various
real-world problems and the number of its applications in the multi-agent
settings has been increasing. Multi-agent learning distinctly poses
significant challenges in the effort to allocate a concealed communication
medium. Agents receive thorough knowledge from the medium to determine
subsequent actions in a distributed nature. Apparently, the goal is to
leverage the cooperation of multiple agents to achieve a designated objective
efficiently. Recent studies typically combine a specialized neural network
with reinforcement learning to enable communication between agents. This
approach, however, limits the number of agents or necessitates the homogeneity
of the system. In this paper, we have proposed a more scalable approach that
not only deals with a great number of agents but also enables collaboration
between dissimilar functional agents and compatibly combined with any deep
reinforcement learning methods. Specifically, we create a global communication
map to represent the status of each agent in the system visually. The visual
map and the environmental state are fed to a shared-parameter network to train
multiple agents concurrently. Finally, we select the _Asynchronous Advantage
Actor-Critic_ (A3C) algorithm to demonstrate our proposed scheme, namely
_Visual communication map for Multi-agent A3C_ (VMA3C). Simulation results
show that the use of visual communication map improves the performance of A3C
regarding learning speed, reward achievement, and robustness in multi-agent
problems.

###### Index Terms:

reinforcement learning, deep learning, learning systems, multi-agent systems.

##  I Introduction

Applications of deep reinforcement learning (RL) methods in the multi-agent
environments have attracted much attention recently because of their
capability of solving various complex problems. Multi-agent learning is to
learn the interaction between multiple distributed self-operated agents to
balance a shared interest or collectively achieve a designated objective [1,
2]. A vital form of such interaction involves the ability to communicate among
agents, which basically ensures agents to function as a team rather than as
individuals [3]. Examples of communication activities in multi-agent systems
include the scheduling problem in wireless sensor networks [4], strategic
planning in robot soccer [5], traffic junction problem [6], elevator control
[7], and multi-agent games [8]. Therefore, learning to communicate in multi-
agent systems has drawn a great deal of attention from the research community
in recent years [9, 10].

Figure 1: A traffic light can be used as a visual indicator to control the
level of cooperation between vehicles in a road intersection.

Deep RL [11] is known as a normative approach to deal with the _curse of
dimensionality_ in domains where the observation space and action space are
highly dimensional [12]. It is possible because deep RL approximately
estimates the value function by the use of neural networks, which basically
include convolutional layers to process raw graphical data directly. In this
paper, we leverage the potential of _Convolutional Neural Networks_ [13]
(ConvNets) to examine the effect of graphical features to the gradual
cooperation between interacting agents during training. To make it feasible,
we define a set of visual indicators so that each indicator represents the
current status of an agent. The visual indicators are globally visible to all
agents and plays as a virtual communication medium. For this reason, we call a
set of visual indicators a visual communication map.

To clearly present the motivation of the study, we consider the following
example. Fig. 1 describes a scenario in a road intersection. Vehicles in the
intersection must follow the traffic light to make subsequent decisions and
avoid possible collisions. When the light turns red, the vehicles are notified
to stop and when the light turns green, the vehicles are allowed to move
forward. In this case, the traffic light plays as a virtual communication
medium among vehicles. Because the traffic light is visible to every vehicle
in the intersection, it can be used as visual indicators to enable cooperation
between vehicles. Therefore, the drivers are only required to learn the
traffic light rules. The example implies that visual indicators can, somehow,
help cooperation in a multi-agent system.

Figure 2: Visual indicators are used to enable cooperation between multiple
agents.

By analyzing the previous example meticulously, we design a visual
communication map for multi-agent environments, as shown in Fig. 2. Initially,
we create a visual map that includes a set of visual indicators. Each
indicator represents the current status of an agent. The visual map is
combined with the environmental state to feed into a ConvNet. The output of
the ConvNet is connected with a fully-connected network followed by a control
layer where subsequent actions are predicted. Finally, each agent uses the
augmented knowledge of other agents via the visual communication map to learn
the optimal distributed policy on its own and, at the same time, maintain a
certain level of cooperation with other agents.

Deep RL brings in a great deal of success in many complex real-world problems
such as Atari games [12, 14, 15, 16], the game of Go [17], robotics control
tasks [18, 19, 20, 21], autonomous driving [22], and surgical robotics [23,
24]. As opposed to previous studies, we examine deep RL in multi-agent
domains. Multi-agent learning is more sophisticated than the single-agent
counterpart as the action space exponentially increases with the number of
agents. Furthermore, we face the challenge of _moving target problem_ [26,
27], which is a primary difficulty in finding the optimal policy of every
agent. In other words, the rewarding signal of an agent depends on the
decisions of other agents and causes the seeking policies to become _non-
stationary_ [27, 28]. The use of _experience replay_ memory even makes the
situation worse as a large portion of transitions in the memory could become
deprecated [26, 29, 30]. For this reason, we choose the A3C [31] method, which
is a policy-based algorithm, to demonstrate our proposed scheme. There are
different variants of A3C such as [32, 33, 34], but we use the A3C version
based on GPU [32] due to various beneficial factors: high performance with
multiple concurrent actor-learners, efficiency within a short period of
training, robustness with a wide range of learning rate, and easy to implement
due to many open-source codes available in Python.

Figure 3: A gameplay interface of Milk Factory.

Last but not least, we develop a game named _Milk Factory_ that is used to
benchmark our proposed scheme. Milk Factory simulates a task in a warehouse
where robots are used to automate the production chain. Fig. 3 describes the
gameplay interface of Milk Factory. There are two kinds of robots in the
factory: pick-up robots and mechanic robots. At first, a pick-up robot waits
for the milk bottle that is on the running conveyor belt and picks it when the
bottle is in front of the robot. The robot can only pick one milk bottle at a
time. It then brings the bottle to the box that is placed far away from the
conveyor belt. After placing the milk bottle in the box, the robot is free and
can go back to the conveyor belt to wait for another milk bottle. However, the
pick-up robot may stop to work (e.g. due to battery depletion) during the
operation period. In this case, the mechanic robot goes to the pick-up robot’s
position and fixes the error. After fixing, the pick-up robot can continue to
operate. A pick-up robot is rewarded a score of 10 whenever it picks a milk
bottle or puts the bottle in the box. The mechanic robot is also given a score
of 10 whenever it fixes the pick-up robot successfully. Therefore, the game
requires the cooperation of two robots to maximize reward achievement.

Finally, the paper contributes to the following highlights:

  * •

The study conducts an interesting investigation of the relationship between
visual indicators and interacting agents. Experimental results show that
visual indicators play as a communication bridge to enable the cooperation
between multiple agents. Therefore, the study provides a practical method for
the development of multi-agent systems.

  * •

We develop a multi-agent environment named Milk Factory. Milk Factory is
developed in Python, which follows the standard interface of _Arcade Learning
Environment_ (ALE) [35, 36] and _OpenAI Gym_ [37]. Therefore, previous deep RL
methods, which work with ALE and OpenAI Gym, adapt well with Milk Factory
without significant modifications. Furthermore, Milk Factory is a configurable
game that provides unlimited potentials to suit with any research purposes.

  * •

A3C is not a well-known method for multi-agent learning [8]. In this study,
however, we show that the combination of visual communication map and A3C can
work properly with multi-agent problems.

  * •

Although we use A3C to demonstrate our proposed scheme, it can be used with
any deep RL algorithms. Furthermore, the method is scalable in terms of the
number of agents and compatible with heterogeneous environments where agents
can function different tasks.

The paper is organized as follows. Section II examines recent studies in
multi-agent domains. Section III presents the concept of a visual
communication map and the proposed scheme VMA3C. The performance evaluation is
conducted based on Milk Factory in Section IV. Finally, to conclude the paper,
we present potential extension directions and the future work in Section V.

##  II Related Work

Since the first development of deep RL, namely _Deep Q-Network_ (DQN) [12],
there have been numerous variants of DQN such as _Double Q-Learning_ [30],
_Dueling Network Architecture_ [38], _Recurrent Deep Q-Learning_ [39], _Deep
Attention Recurrent Q-Network_ [40], and _Prioritized Experience Replay_ [29].
However, these approaches concentrate on approximating the value function,
which requires a large amount of memory to store historical transitions.
Furthermore, the experience replay is known to amplify the non-stationary
problem in multi-agent systems [26]. Therefore, a policy-based approach such
as A3C [31] and its variants, e.g. _UNsupervised REinforcement and Auxiliary
Learning_ (UNREAL) [41], are developed to hasten the learning speed by
allowing multiple actors-learners to be trained at the same time. These
methods are more efficient than the value-based methods as they do not require
an experience replay. Simulation results show that the policy-based methods
outperform the value-based ones concerning learning speed and total reward
achievement in the Atari domain [31]. Besides, _Deep Deterministic Policy
Gradient_ (DDPG) [42] and _Trust Region Policy Optimization_ (TRPO) [43] are
proposed to deal with continuous action domains. In the first case, DDPG
combines DQN with the actor-critic architecture to find the optimal policy.
Whereas, TRPO is a policy gradient method that allows controlling the policy
improvement in every training step. In this paper, we select A3C as the
baseline method to demonstrate our proposed scheme because Milk Factory has a
discrete action space.

There are notable methods that use communication to control multiple agents
such as Foerster _et al._ [3] formulate two approaches _Reinforced Inter-Agent
Learning_ (RIAL) and _Differentiable Inter-Agent Learning_ (DIAL). In the
first scheme, RIAL combines DQN with a recurrent network to independently
train every agent through a shared-parameter network. Whereas, DIAL
additionally sends messages across agents during the centralized learning.
These real-valued messages are mapped to a limited number of communication
actions. In this paper, we also adopt the centralized learning by training
multiple agents at the same time but operating in a distributed nature when
deployed. It is feasible because the visual indicators can be used as a
virtual communication medium. Agents are trained to learn the cooperation
through the visual indicators. Because the channel is virtual, there is no
cost for exchanging messages between agents and hence energy efficiency.

Sukhbaatar _et al._ [6] propose a communication model that allows multiple
agents to communicate before deciding subsequent actions. The authors create a
deep neural network that has an open access to a shared communication channel.
Agents receive aggregated transmissions from other agents via a continuous
vector embedded in the channel. Because the communication channel carries
continuous data, it is possible to combine the approach with a standard
single-agent RL method. However, the scheme is not scalable due to the
complication of the neural network structure. In this study, we use the visual
communication map, which is basically separated from the network
configuration. Therefore, our proposed method is more robust and scalable.

Finally, Gupta _et al._ [8] examine a set of cooperative control tasks and
compare the performance of three different deep RL approaches policy gradient,
temporal-difference error, and actor-critic method. The authors combine
various disciplines to efficiently train multiple agents to learn a wide range
of control tasks from discrete to continuous action domains. Initially, the
agents are trained to learn a simple task and move to the harder ones
afterward. The knowledge of the previous task is accumulated over time rather
than acquiring new knowledge from scratch. This approach is called _curriculum
learning_ [44, 45]. The authors also show that the use of a shared-parameter
network is scalable in terms of the number of agents. However, the paper
examines only homogeneous domains where agents function similar jobs. In our
study, we extend the investigation to heterogeneous systems where agents can
be in different task domains. Last but not least, we though evaluate our
proposed framework in Milk Factory, a fully observation problem, the presented
approach can be applied to partially observation domains such as [46]
similarly.

##  III Proposed Schemes

In this section, we have first presented preliminary terminologies of RL and
the A3C method. We then introduce the use of a visual communication map with
A3C as well as the network architecture for multi-agent problems. Finally, we
propose two implementation approaches for the visual communication map.

###  III-A Preliminary

####  III-A1 RL

Figure 4: The relationship between an agent and the environment in RL.

RL involves the interaction between an agent and the environment, as shown in
Fig. 4. At time-step $t$, the agent perceives an environmental state $s_{t}$.
It reacts with an action $a_{t}$ and the environment returns a reward $r_{t}$.
From time to time, the agent obtains a set of transitions:
$S^{T}=\\{s_{0},a_{0},r_{1},s_{1},a_{1},r_{2},...,s_{T},a_{T},r_{T+1}\\}$,
where $T$ is the terminal state of the episode. The initial state $s_{0}$ is
generated randomly from the observation space $S$ and the terminal reward
$r_{T+1}$ is assigned to 0.

On the agent’s side, a policy $\pi$ is defined as a mapping function $f$ from
the observation space $S$ to the action space $A$, _i.e._ ,
$\pi=f:S\longrightarrow A$. In stochastic environments, it is convenient to
define the policy $\pi$ as a conditional probability distribution of action
$a_{t}$, given the state $s_{t}$, as follows:

$\pi(a_{t}|s_{t})=\left\\{\ p(a_{t}|s_{t})\ \bigg{|}\ \forall a_{t}\in
A,\forall s_{t}\in S\right\\}.$

In this definition, the next state $s_{t+1}$ is assumed to depend only on the
previous state $s_{t}$ and the corresponding action $a_{t}$. A problem that
satisfies this condition is called a _Markovian decision process_ (MDP). Milk
Factory is an MDP. In real-world problems, the agent only senses a limited
view of the surrounding environment. Therefore, it is more practical to
formulate the problem as a _partially observable MDP_ (POMDP) [47]. In this
paper, for concise, we only consider the MDP, but the same procedure can be
inferred for the POMDP.

On the other hand, we define the discounted return at time-step $t$ as
$R_{t}=r_{t+1}+\gamma r_{t+2}+\gamma^{2}r_{t+3}+...$, where $\gamma$ is a
discounted factor so that $\gamma\in[0,1]$. The goal of RL is to maximize the
discounted return $R_{t}$.

Finally, to evaluate the _value_ of a state, we define the value function $V$
of a state $s_{t}$ under a policy $\pi$ as the expected value of the
discounted return $R_{t}$, _i.e._ ,

$V(s_{t})|_{\pi}=\mathbf{E}\left\\{R_{t}\middle|\pi\right\\}=\mathbf{E}\left\\{\sum_{i=0}^{T-t-1}\gamma^{i}r_{t+i+1}\middle|\pi\right\\}.$
(1)

####  III-A2 A3C

A3C is a policy-based method that uses the actor-critic architecture [48] and
the advantage function [49] to estimate the optimal policy $\pi*$.
Furthermore, it enables concurrent learning by creating multiple actors-
learners to update the parameters of the neural network asynchronously. In
particular, as shown in Fig. 5, the A3C’s network structure consists of two
essential sub-networks: one actor network $N^{a}(\theta)$ and one critic
network $N^{c}(\theta^{\prime})$, where $\theta$ and $\theta^{\prime}$ denote
$N^{a}$’s weight parameters and $N^{c}$’s weight parameters respectively. The
actor network estimates the policy $\pi(a_{t}|s_{t},\theta)$ of the agent
while the critic network evaluates the value function
$V(s_{t},\theta^{\prime})$ (1). Therefore, the A3C method optimizes two loss
functions: the actor loss function $L^{a}(\theta)$ and the critic loss
function $L^{c}(\theta^{\prime})$:

Figure 5: Actor-critic architecture in A3C.

$L^{a}(\theta)\simeq\log(\pi(a_{t}|s_{t},\theta))\left(A_{t}(\theta^{\prime})-V(s_{t},\theta^{\prime})\right)$
(2)

and

$L^{c}(\theta^{\prime})\simeq\frac{1}{2}\left(R_{t}(s_{t},\theta^{\prime})-V(s_{t},\theta^{\prime})\right)^{2},$
(3)

where $A_{t}$ is calculated by the following equation:

$A_{t}(\theta^{\prime})=\sum_{k=0}^{T_{max}-t-1}\gamma^{k}r_{t+k+1}+\gamma^{T_{max}-t}V(s_{T_{max}},\theta^{\prime}),$
(4)

where $T_{max}$ denotes the maximum number of observation steps. If
$T_{max}=T$, we have $V(s_{T_{max}},\theta^{\prime})=V(s_{T})=0$. Finally, to
enable the agent’s exploration during the training process, the entropy
regularization factor $H(s_{t},\theta)$ is added to the total loss.

Finally, the total loss is the summation of the actor loss, the critic loss,
and the entropy regularization:

$L^{total}=L^{a}+L^{c}+\beta H,$

where $\beta$ denotes the control parameter so that $\beta\in[0,1]$.

###  III-B VMA3C

####  III-B1 Multi-agent A3C

Figure 6: Multiple-actor-single-critic architecture in A3C.

In this section, we extend the use of A3C to multi-agent learning, as shown in
Fig. 6. The procedure can be applied to any standard deep RL methods. It is
even more straightforward to derive from a value-based method such as DQN
because its network has a single output while the A3C’s network includes two
outputs (actor-critic), as explained in the previous subsection.

We consider an MDP (or a POMDP) that consists of $N$ agents $\\{1,2,...,N\\}$.
Because A3C has two factors in the network configuration (actor and critic),
we create $N$ actor networks $N^{a}_{i}(\theta_{i})$ for $N$ agents ($i=1..N$)
while using a single critic network $N^{c}(\theta^{\prime})$. To efficiently
train multiple agents at the same time, we use a shared-parameter network for
$N$ actors, _i.e._ , $\theta_{1}=\theta_{2}=...=\theta_{N}=\theta$. Let
$\pi_{i}(a_{i,t}|s_{t})$ be a policy of the agent $i$, where $a_{i,t}\in
A_{i}$, $s_{t}\in S$, and $A_{i}$ is the action space of the agent $i$. We
consider that the intermediate reward $r_{t}$, given $s_{t}$ and
$a_{i,t}(i=1..N)$, is the summation of individual reward, _i.e._ ,
$r_{t}=\sum_{i=1}^{N}r_{i,t}$. Because the state space $S$ is shared among all
agents and because we use a single critic network, the critic loss function
for a multi-agent scenario is the same as in equation (3), _i.e._ ,
$L^{c}(\theta^{\prime})\simeq\frac{1}{2}\left(R_{t}(s_{t},\theta^{\prime})-V(s_{t},\theta^{\prime})\right)^{2}$.
The actor loss $L^{a}_{i}(\theta)$ of the agent $i$ is also calculated as in
equation (2) as below:

$L^{a}_{i}(\theta)\simeq\log(\pi_{i}(a_{i,t}|s_{t},\theta))\left(A_{t}(\theta^{\prime})-V(s_{t},\theta^{\prime})\right).$

However, the entropy regularization $H$ in this case is the summation of $N$
individual factors, _i.e._ , $H=\sum_{i=1}^{N}H_{i}(s_{t},\theta)$.

Finally, we have the total loss for multi-agent problems using A3C as follows:

$L^{total}=\sum_{i=1}^{N}L_{i}^{a}+\alpha L^{c}+\beta H,$ (5)

where $\alpha$ ($\alpha\geq 1$) is added to balance the importance of the
critic among multiple actors.

####  III-B2 Visual communication map

For each agent, we define a set of status
$S^{stat}_{i}=\\{c_{i1},c_{i2},...,c_{ij}\\}$, where $i=1..N$, $j\geq 0$, and
$c_{il}\neq c_{ip}(l\neq p)$. We also define a mapping function $F_{i}$ that
maps $S^{stat}_{i}$ to a set of graphical representations
$G=\\{g_{1},g_{2},...,g_{m}\\}$ so that

  * •

$\forall x\in s_{t}\Rightarrow x\not\in g_{i}$,

  * •

$\forall x\in g_{i}\Rightarrow x\not\in g_{j}(i\neq j$ and
$i,j\in\\{1,...,m\\})$,

where $m=\sum_{i=1}^{N}||S^{stat}_{i}||$. In other words, we have
$F_{i}:S^{stat}_{i}\rightarrow G$ $(i=1..N)$.

Let $C_{i}^{t}$ be the status of the agent $i$ at time-step $t$. We have
$C_{i}^{t}\in S^{stat}_{i}$. A visual communication map at time-step $t$ is
defined as a set of graphical representations of $N$ agents’ status at time-
step $t$:
$M_{t}=\\{F_{1}(C_{1}^{t}),F_{2}(C_{2}^{t}),...,F_{N}(C_{N}^{t})\\}$.

Finally, we define an operator $\oplus$ that combines the environmental state
$s_{t}$ with $M_{t}$. We assume that $S_{t}=s_{t}\oplus M_{t}$. Then, $S_{t}$
is used to feed to the neural network (instead of $s_{t}$). To maximize the
learning efficiency, we define the operator $\oplus$ so as to satisfy the
following conditions:

######  Premise 1.

The environmental state $s_{t}$ is a strict subset of $S_{t}$, i.e.,
$s_{t}\subset S_{t}$.

######  Premise 2.

Every element of the visual communication map $M_{t}$ is a strict subset of
$S_{t}$, i.e., $F_{i}(C_{i}^{t})\subset S_{t}(i=1..N)$.

######  Premise 3.

Given $E$=$\\{s_{t},F_{1}(C_{1}^{t}),F_{2}(C_{2}^{t}),...,F_{N}(C_{N}^{t})\\}$
and an arbitrary $x$ so that $x\in S_{t}$, we have that $x$ exclusively
belongs to one element of $E$.

Premises 1 and 2 make sure that $S_{t}$ maintains the integrity information of
the environmental state and the current status of all agents in the system.
Premise 3 is defined to maximize learning efficiency. We can infer the
following rules based on these conditions:

######  Lemma 1.

Let $s_{t}\neq\emptyset$ be the environmental state at time-step $t$ and
$G_{i}^{t}$ be the graphical representation of the status of agent $i$ at
time-step $t$, i.e., $G_{i}^{t}=F_{i}(C_{i}^{t})(i=1..N)$ so that $\exists
k\in\\{1,...,N\\}$ and $G_{k}^{t}\neq\emptyset$. The following operator
$S_{t}=s_{t}\oplus M_{t}$ $=$ $s_{t}\cup\left(\cup_{i=1}^{N}G_{i}^{t}\right)$
satisfies the three premises.

_Proof_ : Because $S_{t}=s_{t}\cup\left(\cup_{i}^{N}G_{i}^{t}\right)$, we have
$s_{t}\subseteq S_{t}$ and $G_{i}^{t}\subseteq S_{t}(i=1..N)$. However, the
_equal_ sign does not occur because $s_{t}\neq\emptyset$ and $\exists
k\in\\{1,...,N\\}$ so that $G_{k}^{t}\neq\emptyset$ and $s_{t}\neq
G_{i}^{t}(i=1..N)$. This implies that the operator satisfies premises 1 and 2.

Secondly, we get an element $x\in S_{t}$. We assume that $x$ belongs to at
least two elements of $E$. There are two possible cases:

  * •

If $x\in s_{t}$, we have $\exists k\in\\{1,...,N\\}$ so that $x\in G_{k}^{t}$
and $G_{k}^{t}\neq\emptyset$. This contradicts with the definition of the
visual communication map, _i.e._ , $x\not\in G_{k}^{t}(\forall x\in s_{t})$.

  * •

If $x\not\in s_{t}$, we have $\exists i,j\in\\{1,...,N\\}$ so that $i\neq
j,x\in G_{i}^{t}$ and $x\in G_{j}^{t}$, which also contradicts with the
definition of the visual communication map.

The lemma is completely proved. $\hfill\blacksquare$

######  Lemma 2.

Let $s_{t}\neq\emptyset$ be the environmental state at time-step $t$,
$G_{i}^{t}$ be the graphical representation of the status of agent $i$ at
time-step $t$, and $\Phi$ be an arbitrary 1-to-1 mapping function from set to
set. We assume that $G_{i}^{t}\neq\emptyset(i=1..N)$. The following operator
$S_{t}=s_{t}\oplus M_{t}$ $=$
$\Phi(s_{t})\cup\left(\cup_{i=1}^{N}\Phi(G_{i})\right)$ satisfies the three
premises.

_Proof_ : Because $\Phi:A\rightarrow B$ is a 1-to-1 mapping function from set
$A$ to set $B$, _i.e._ , $\forall x\in A,\exists!y\in B\Rightarrow y=\Phi(x)$
and vice versa. Therefore, the problem can be converted to Lemma 1, which is
completely proved. $\hfill\blacksquare$

Lemma 1 and Lemma 2 are important as they can be used to suggest different
implementation approaches for the visual communication map. In this paper, we
suggest the following approaches:

Figure 7: Visual aggregation by creating a holder.

  1. 1.

We create a graphical holder (a mask) that embeds both the environmental state
and the visual communication map, as shown in Fig. 7. The communication map is
a graphical representation that contains the current status information of $N$
agents. The holder is used as $S_{t}$ and is fed to the neural network. This
approach is inferred from Lemma 1.

  2. 2.

We create $N+1$ ConvNets: one for the environmental state and the other $N$
ConvNets for $N$ agents, as shown in Fig. 8. The output features of $N+1$
ConvNets are concatenated to form $S_{t}$. This approach is inferred from
Lemma 2.

Figure 8: Concatenating output features of $N+1$ ConvNets.

In this paper, we use the first method to benchmark our proposed scheme
because it is more efficient (using a single shared-parameter ConvNet). The
detailed algorithm is described in Algorithm 1.

Algorithm 1 Visual communication Map for Multi-agent A3C (VMA3C)

1:$N\leftarrow 0$ $\triangleright$ Global shared counter

2:procedure VMA3C $\triangleright$ A learner procedure

3: $t\leftarrow 0$

4: repeat

5: $s\leftarrow s_{t}\oplus M_{t}$

6: $c\leftarrow 0$

7: Reset experience memory $E\leftarrow\\{\\}$

8: repeat

9: $a\leftarrow\\{a_{1,t},a_{2,t},...,a_{N,t}\\}$ $\triangleright$ Using
$\pi_{1},\pi_{2},...,\pi_{N}$

10: $r\leftarrow r_{t+1}=\sum_{i}^{N}r_{i,t+1}$

11: $s^{\prime}\leftarrow s_{t+1}$

12: Save transition $(s,a,r,s^{\prime})$ to $E$

13: $s\leftarrow s^{\prime}$

14: $t\leftarrow t+1,N\leftarrow N+1,c\leftarrow c+1$

15: until $s=s_{T}\text{ or }c=T_{\text{max}}$

16: Retrieve all $(s_{j},a_{j},r_{j},s^{\prime}_{j})$ from $E$

17: Calculate $A_{t}$ based on (4)

18: Calculate gradients $\frac{\partial L^{total}}{\partial\theta}$,
$\frac{\partial L^{total}}{\partial\theta^{\prime}}$ based on (5)

19: Perform asynchronous update of $\theta$ and $\theta^{\prime}$

20: if $s=s_{T}$ then

21: $t\leftarrow 0$

22: until $N>N_{\text{end}}$

In a real-world application, it is not scalable if the number of visual
representations increases with the number of agents. Therefore, it is more
efficient to use the following _condensed_ rule to reduce the number of visual
representations.

Condensed rule: Let $c_{ij}$ be the status $j$-th of an agent $i$. If there
exists $K=\\{k_{1},k_{2},...,k_{N-1}\\}$ so that the conditional probability
$P(c_{ij}|c_{mk_{p}})>0$ $(\forall m\in\\{1,2,...,N\\}-\\{i\\},k_{p}\in K)$
then $F_{i}(c_{ij})=\emptyset$.

For example, in the road intersection problem, we only need three visual
indicators (red, green, orange) to control all vehicles. However, this rule
requires extra information about the problem to estimate the conditional
probability $P(c_{ij}|c_{mk_{p}})$.

##  IV Performance Evaluation

###  IV-A Parameter settings

In this section, we describe the parameter settings that are used in Milk
Factory and the proposed scheme (VMA3C) for performance evaluation. The
algorithm is run in a computer with an octa-core processor and a GTX 1080Ti
graphics card. In Milk Factory, an episode terminates when the number of steps
reaches 200. The pick-up robot is rewarded a score of 10 whenever it
successfully picks a milk bottle or puts it into the box. Whereas, the
mechanic robot is rewarded a score of 10 if it successfully fixes the pick-up
robot. The error rate of the pick-up robot is represented as a random variable
_ER_ , _i.e._ , the probability of an error per action of the pick-up robot
equals to $1\%$. Moreover, the pick-up robot has 5 decision actions: moving
up, moving down, moving left, moving right, and picking/dropping a milk
bottle. The mechanic robot also has 5 decision actions: moving up, moving
down, moving left, moving right, and fixing the pick-up robot. A history of 4
game frames is used as a single state to feed into the neural network.

Finally, based on the _condensed rule_ , we only map the status of the pick-up
robot to visual representations because the state of the mechanic robot
depends on the pick-up robot. The pick-up robot has two states: busy state
(bringing the milk bottle) and failed state (because of errors). A milk bottle
represents the visual indicator of a busy state and a question mark represents
the visual indicator of a failed state.

We compare the performance of the VMA3C method with the A3C method. We keep
the parameter settings of A3C same as the previous study [31] except the
following changes. The learning rate starts from 0.001 and anneals to 0 during
the course of training. The discounted factor $\gamma$ equals to 0.99. The
maximum observation step $T_{max}$ equals to 5. The encouraging factor $\beta$
equals to 0.01. Moreover, we perform the gradient clipping by using the
following formula [50]:

$\delta_{i}^{G}=\delta_{i}^{G}*\frac{\varpi}{\max(\omega,\varpi)},$

where $\varpi=40$, $\forall i:\delta_{i}^{G}\in\Delta_{G}$, and $\forall
j:\delta_{i,j}^{G}\in\delta_{i}^{G}$ we have:

$\omega=\sqrt{\sum_{i,j}|\delta_{i,j}^{G}|^{2}}.$

Finally, we use the RMSProp [51] optimizer with a decay of 0.99 and an epsilon
of 0.1. The network of the VMA3C method includes 4 layers: 2 convolutional
layers (one has 16 filters of 8 $\times$ 8 with a stride of 4 followed by
another one that has 32 filters of 4 $\times$ 4 with a stride of 2), one
fully-connected layer with a size of 256 and ReLU activation functions, and
one output layer that includes a single critic and and set of $N$ actors
($N=2$ for two-agent setting and $N=3$ for three-agent setting). The critic is
connected into a linear activation function while each actor is connected into
a _softmax_ activation function.

Figure 9: The reward distribution of VMA3C in the two-agent setting of Milk
Factory.

We use A3C and VMA3C to train agents in 4 million steps. It takes
approximately 12 training hours for each variant. During the training process,
we create 40 checkpoints in different training steps. For each checkpoint, we
perform a 10,000-step evaluation and record the mean of total reward and its
standard error. We divide the performance evaluation into three stages.
Firstly, we compare the performance of VMA3C with A3C in the two-agent setting
of Milk Factory (one pickup robot and one mechanic robot). Subsequently, we
add an additional pick-up robot to examine the scalability of the proposed
scheme. Finally, we re-evaluate the previous two stages with different values
of _ER_ to examine the robustness of VMA3C in stochastic environments.

###  IV-B Two-agent setting

In this setting, two robots are operated in Milk Factory: one pick-up robot
and one mechanic robot. We examine the cooperation between the robots to
maximize the total reward. Because the maximum number of episode steps is set
to 200, it is feasible to calculate the optimal score that the two robots can
achieve. In an ideal case, it takes 8 steps to pick a milk bottle, go to the
box, and put the bottle into the box. The robot optimally achieves a reward of
20 in each round. Therefore, the maximum reward that the robot can achieve is
$200*20/8=500$ without any errors. If an error occurs during the operation,
the total reward must be smaller because the robot misses the current milk on
the conveyor belt. Therefore, the reward of 500 is also the optimal total
reward of two robots. Fig. 9 describes the reward distribution of VMA3C during
the training. We can see that VMA3C takes only three training hours to
establish an optimal policy that approximately reaches a reward of 500. It is
possible due to the visual map aids the mechanic robot to fix the pick-up
robot immediately when an error occurs.

Figure 10: Comparison of the mean of total rewards between two methods (A3C
and VMA3C) in the two-agent setting of Milk Factory.

Fig. 10 shows the performance of two methods, A3C and VMA3C, regarding the
mean of total rewards. In the figure, A3C performs poorly without any
improvements after 12 hours of training. Furthermore, the pick-up robot gets
stuck after picking a milk bottle. The mechanic robot is located next to it,
as shown in the following video _https://youtu.be/J0qusnfyrr0_. As opposed to
A3C, VMA3C achieves an optimal policy after 3 hours of training. This marks
the importance of visual communication map in multi-agent problems. The pick-
up robot in VMA3C is operated as expected: it picks a milk bottle, goes to the
box by the shortest path, puts the milk bottle into the box, and goes back to
the conveyor belt. Moreover, the mechanic robot fixes the pick-up robot if
necessary.

###  IV-C Three-agent setting

Figure 11: The reward distribution of VMA3C in the three-agent setting of Milk
Factory.

To examine the scalability of VMA3C, we add a pick-up robot into the gameplay.
Therefore, we have two pick-up robots and one mechanic robot. We also add an
additional box and a milk bottle on the conveyor to increase the productivity.
Fig. 11 shows the reward distribution of the VMA3C method in the three-agent
setting. It can estimate an optimal policy that reaches a reward of 900.
However, the results vary wildly because it is impossible to fix two pick-up
robots at the same time. In this case, the environment is more stochastic than
the previous one.

Figure 12: Comparison of the mean of total rewards between two methods (A3C
and VMA3C) in the three-agent setting of Milk Factory.

Fig. 12 compares the performance of VMA3C and A3C in the three-agent setting
of Milk Factory. The A3C method obtains a maximum reward of 300 after 12 hours
of training while the VMA3C has a 200% higher performance than the A3C method.
As shown in the following video, two pick-up robots in VMA3C operate
concurrently while the mechanic robot is located in the middle of the screen,
as shown in the following video _https://youtu.be/eoud2D0nW1k_.

###  IV-D Robustness

Figure 13: The performance of A3C in Milk Factory with four different error
rates: $2\%$, $3\%$, $4\%$, and $5\%$. The left figure is conducted in the
two-agent setting and the right figure is conducted in the three-agent setting
of Milk Factory.

Figure 14: The performance of VMA3C in Milk Factory with four different error
rates: $2\%$, $3\%$, $4\%$, and $5\%$. The left figure is conducted in the
two-agent setting and the right figure is conducted in the three-agent setting
of Milk Factory.

In this subsection, we vary the values of _ER_ to examine the robustness of
the proposed method. We record the performance of both approaches by using
four different error rates: $2\%$, $3\%$, $4\%$, and $5\%$. Fig. 13 shows the
performance of A3C in the two-agent setting and the three-agent setting of
Milk Factory. In the two-agent setting, A3C performs poorly, as explained in
the previous subsection. We notice that the mean of total reward increases
with the error rate. This implies that the total reward is calculated by the
mechanic robot. Finally, Fig. 14 presents the performance of VMA3C in the two
and three-agent setting of Milk Factory. We conclude that VMA3C is robust in
the stochastic environment because it still achieves a high reward regardless
of the error rate.

##  V Conclusions

Communications between multiple agents are difficult to implement, especially
when agents are characterized by deep RL algorithms. The paper conducts an
interesting investigation of using virtual communication channel via visual
indicators to establish a cooperative policy for multi-agent problems. The
method has practical meaning and can be applied widely in real-world
applications due to many beneficial factors. Firstly, it can be used with any
standard RL methods. Secondly, the proposed method is scalable in terms of the
number of agents and the type of agents. Finally, the method is robust in
stochastic environments and plays a vital role to solve the non-stationary
problem in multi-agent domains. By using visual communication map, the agents
learn a cooperative policy on their own to maximize the reward in a short
period of training time. We continue to work on the effect of visual
indicators to multi-agent domains in different directions:

  * •

The method requires human knowledge to reduce the number of visual indicators,
which prevents the automation of the approach. We suggest using a Bayesian
model to predict the dependency between agents to completely automate the
proposed method.

  * •

The scheme is constructed from the A3C method and evaluated in a discrete
action environment. We will extend the use of visual communication map with
different single-agent deep RL algorithms and examine it in continuous action
domains.

## Acknowledgement

The authors wish to thank our colleagues in Institute for Intelligent Systems
Research and Innovation for their comments and helpful discussions.

## References

  * [1] L. Panait and S. Luke, “Cooperative multi-agent learning: The state of the art,” _Autonomous Agents and Multi-Agent Systems_ , vol. 11, no. 3, pp. 387–434, 2005.
  * [2] D. Bloembergen, K. Tuyls, D. Hennes, and M. Kaisers, “Evolutionary dynamics of multi-agent learning: A survey,” _Journal of Artificial Intelligence Research_ , vol. 53, pp. 659–697, 2015.
  * [3] J. Foerster, I.A. Assael, N. de Freitas, and S. Whiteson, “Learning to communicate with deep multi-agent reinforcement learning,” In _Advances in Neural Information Processing Systems_ , pp. 2137–2145, 2016.
  * [4] D. Fox, W. Burgard, H. Kruppa, and S. Thrun, “A probabilistic approach to collaborative multi-robot localization,” _Autonomous Robots_ , vol. 8, no. 3, pp. 325–344, 2000.
  * [5] M. Riedmiller, T. Gabel, R. Hafner, and S. Lange, “Reinforcement learning for robot soccer,” _Autonomous Robots_ , vol. 27, no. 1, pp. 55–73, 2009.
  * [6] S. Sukhbaatar and R. Fergus, “Learning multiagent communication with backpropagation,” In _Advances in Neural Information Processing Systems_ , pp. 2244–2252, 2016.
  * [7] R. H. Crites and A. G. Barto, “Elevator group control using multiple reinforcement learning agents,” _Machine Learning_ , vol. 33, no. 2–3, pp. 235–262, 1998.
  * [8] J. K. Gupta, M. Egorov, and M. Kochenderfer, “Cooperative multi-agent control using deep reinforcement learning,” In _International Conference on Autonomous Agents and Multiagent Systems_ , pp. 66–83, 2017.
  * [9] T. Nguyen, N. D. Nguyen, and S. Nahavandi, “Deep reinforcement learning for multi-agent systems: A review of challenges, solutions and applications,” _arXiv preprint arXiv:1812.11794_ , 2018.
  * [10] N. D. Nguyen, T. Nguyen, and S. Nahavandi, “System design perspective for human-level agents using deep reinforcement learning: A survey,” _IEEE Access_ , vol. 5, pp. 27091–27102, 2017.
  * [11] R. S. Sutton and G. B. Andrew, _Introduction to Reinforcement Learning_. Cambridge: MIT press, 1998.
  * [12] V. Mnih et al., “Human-level control through deep reinforcement learning,” _Nature_ , 2015.
  * [13] A. Krizhevsky, S. Ilya, and E. H. Geoffrey, “Imagenet classification with deep convolutional neural networks.” In _Advances in Neural Information Processing Systems_ , 2012.
  * [14] V. Mnih et al., “Playing atari with deep reinforcement learning,” _arXiv preprint arXiv:1312.5602_ , 2013.
  * [15] T. Nguyen, N. D. Nguyen, and S. Nahavandi, “Multi-agent deep reinforcement learning with human strategies,” In _2019 IEEE International Conference on Industrial Technology (ICIT)_ , pp. 1357-1362. IEEE, 2019.
  * [16] N. D. Nguyen, S. Nahavandi, and T. Nguyen. “A human mixed strategy approach to deep reinforcement learning,” In _2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC)_ , pp. 4023-4028. IEEE, 2018.
  * [17] D. Silver et al., “Mastering the game of Go with deep neural networks and tree search,” _Nature_ , vol. 529, no 7587, 2016.
  * [18] T. de Bruin et al., “The importance of experience replay database composition in deep reinforcement learning,” _Deep Reinforcement Learning Workshop (NIPS)_ , 2015.
  * [19] S. Gu et al., “Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates,” In _International Conference on Robotics and Automation_ , 2017.
  * [20] B. Thananjeyan et al., “Multilateral surgical pattern cutting in 2d orthotropic gauze with deep reinforcement learning policies for tensioning,” In _International Conference on Robotics and Automation_ , 2017.
  * [21] T. G. Thuruthel et al., “Model-based reinforcement learning for closed-loop dynamic control of soft robotic manipulators,” _IEEE Transactions on Robotics_ , vol. 35, pp.124–134, 2018.
  * [22] S. S.-Shwartz, S. Shaked, and S. Amnon, “Safe, multi-agent, reinforcement learning for autonomous driving,” _arXiv preprint arXiv:1610.03295_ , 2016.
  * [23] T. Nguyen, N. D. Nguyen, F. Bello, and S. Nahavandi, “A new tensioning method using deep reinforcement learning for surgical pattern cutting,” In _2019 IEEE International Conference on Industrial Technology (ICIT)_ , pp. 1339-1344. IEEE, 2019.
  * [24] N. D. Nguyen, T. Nguyen, S. Nahavandi, A. Bhatti, and G. Guest, “Manipulating soft tissues by deep reinforcement learning for autonomous robotic surgery,” In _2019 IEEE International Systems Conference (SysCon)_ , pp. 1-7. IEEE, 2019.
  * [25] M. Egorov, “Multi-agent deep reinforcement learning,” _CS231n: Convolutional Neural Networks for Visual Recognition_ , 2016.
  * [26] G. Palmer et al., “Lenient multi-agent deep reinforcement learning,” In _International Conference on Autonomous Agents and MultiAgent Systems_ , 2018.
  * [27] L. Bu, B. Robert, and D.S. Bart, “A comprehensive survey of multiagent reinforcement learning,” _IEEE Transactions on Systems, Man, and Cybernetics_ , vol. 38, pp. 156–172, 2008.
  * [28] K. Tuyls and W. Gerhar, “Multiagent learning: Basics, challenges, and prospects,” _AI Magazine_ , vol. 33, 2012.
  * [29] T. Schaul et al., “Prioritized experience replay,” _arXiv preprint arXiv:1511.05952_ , 2015.
  * [30] H. V. Hasselt, G. Arthur, and S. David, “Deep reinforcement learning with double q-learning,” In _Conference on Artificial Intelligence_ , 2016.
  * [31] V. Mnih et al., “Asynchronous methods for deep reinforcement learning,” _International Conference on Machine Learning_ , 2016.
  * [32] M. Babaeizadeh et al., “Reinforcement learning through asynchronous advantage actor-critic on a gpu,” _arXiv preprint arXiv:1611.06256_ , 2016.
  * [33] Y. Wu et al., “Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation,” In _Advances in Neural Information Processing Systems_ , 2017.
  * [34] B. Peng et al., “Adversarial advantage actor-critic model for task-completion dialogue policy learning,” In _International Conference on Acoustics, Speech and Signal Processing_ , 2018.
  * [35] M. G Bellemare et al., “The arcade learning environment: An evaluation platform for general agents,” _Journal of Artificial Intelligence Research_ , vol. 47, pp. 253–279, 2013.
  * [36] M. C. Machado et al., “Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents,” _Journal of Artificial Intelligence Research_ , vol 61, pp. 523–562, 2018.
  * [37] G. Brockman et al., “OpenAI Gym,” _arXiv preprint arXiv:1606.01540_ , 2016.
  * [38] Z. Wang et al., “Dueling network architectures for deep reinforcement learning,” _arXiv preprint arXiv:1511.06581_ , 2015.
  * [39] M. Hausknecht and S. Peter, “Deep recurrent q-learning for partially observable mdps,” In _AAAI Fall Symposium Series_ , 2015.
  * [40] I. Sorokin et al., “Deep attention recurrent Q-network,” _arXiv preprint arXiv:1512.01693_ , 2015.
  * [41] M. Jaderberg et al., “Reinforcement learning with unsupervised auxiliary tasks,” _arXiv preprint arXiv:1611.05397_ , 2016.
  * [42] T. P. Lillicrap et al., “Continuous control with deep reinforcement learning,” _arXiv preprint arXiv:1509.02971_ , 2015.
  * [43] J. Schulman et al., “Trust region policy optimization,” In _International Conference on Machine Learning_ , 2015.
  * [44] Y. Bengio et al., “Curriculum learning,” _Proceedings of the 26th annual international conference on machine learning_ , 2009.
  * [45] L. Jiang et al., “Self-paced curriculum learning,” In _AAAI Conference on Artificial Intelligence_ , 2015.
  * [46] J. Z. Leibo et al., “Multi-agent reinforcement learning in sequential social dilemmas,” In _Conference on Autonomous Agents and MultiAgent Systems_ , 2017.
  * [47] T. Jaakkola, P.S. Satinder, and I. J. Michael, “Reinforcement learning algorithm for partially observable Markov decision problems,” In _Advances in Neural Information Processing Systems_ , 1995.
  * [48] J. Peters and S. Stefan, “Natural actor-critic,” _Neurocomputing_ , vol. 71, no 7–9, pp. 1180-1190, 2008.
  * [49] R. S. Sutton et al., “Policy gradient methods for reinforcement learning with function approximation,” In _Advances in Neural Information Processing Systems_ , 2000.
  * [50] R. Pascanu, M. Tomas, and B. Yoshua, “On the difficulty of training recurrent neural networks,” In _International Conference on Machine Learning_ , 2013.
  * [51] T. Tieleman and H. Geoffrey, “Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude,” _COURSERA: Neural Networks for Machine Learning_ , pp. 26–31, 2012.