# Multi-Agent Reinforcement Learning for Distributed Joint Communication and
Computing Resource Allocation over Cell-Free Massive MIMO-enabled Mobile Edge
Computing Network

Fitsum Debebe Tilahun, Ameha Tsegaye Abebe, and Chung G. Kang,

###### Abstract

To support the newly introduced multimedia services with ultra-low latency and
extensive computation requirements, resource-constrained end user devices
should utilize the ubiquitous computing resources available at network edge
for augmenting on-board (local) processing with edge computing. In this
regard, the capability of cell-free massive MIMO to provide reliable access
links by guaranteeing uniform quality of service without cell edge can be
exploited for seamless parallel processing. Taking this into account, we
consider a cell-free massive MIMO-enabled mobile edge network to meet the
stringent requirements of the advanced services. For the considered mobile
edge network, we formulate a joint communication and computing resource
allocation (JCCRA) problem with the objective of minimizing energy consumption
of the users while meeting the tight delay constraints. We then propose a
fully distributed cooperative solution approach based on multi-agent deep
deterministic policy gradient (MADDPG) algorithm. The simulation results
demonstrate that the performance of the proposed distributed approach has
converged to that of a centralized deep deterministic policy gradient
(DDPG)-based target benchmark, while alleviating the large overhead associated
with the latter. Furthermore, it has been shown that our approach
significantly outperforms heuristic baselines in terms of energy efficiency,
roughly up to 5 times less total energy consumption.

###### Index Terms:

Cell-free massive MIMO, joint communication and computing resource allocation
(JCCRA), multi-agent reinforcement learning (MARL), deep determinstic policy
gradient (DDPG), edge intelligence, mobile edge computing (MEC).

##  I Introduction

The past few years have seen a rapid increase in the computationally intensive
applications such as face recognition, autonomous driving, traffic control,
and so on [1-2]. One key aspect of the services is their tight delay
requirement which poses a serious challenge to user equipment (UE) with
limited battery power, computational and storage capabilities. To handle these
bottlenecks at the UEs, computation offloading to powerful cloud computing
platforms had been implemented for various services [3-5]. Recently, a
paradigm that has brought computing resources to the edge of the network,
dubbed as mobile edge computing (MEC), has been introduced to further reduce
the experienced latency. As we move towards the next decade, however, the
current cellular MEC systems cannot keep up with the diverse and yet, even
more stringent requirements of the envisaged next generation advanced
services, and the evolution of the existing ones. For instance, expected use
cases of the sixth generation (6G) communication network, such as multi-
sensory extended reality (augmented, virtual, and mixed reality), and
holographic displays, require extremely high data rate, improved coverage and
spectral efficiency, deterministic ultra-low latency, and extensive
computational capability [6-9].

To attain the requirements of the envisioned applications, however, a
functionality of reliable _virtual bus_ might be required for parallel and
cooperative processing, i.e., to connect the users with ubiquitous computing
resources at the edge, such as nearby mobile users (with idle or more powerful
processors), and edge/cloud servers (with full-blown computing resources). In
this regard, we consider a cell-free massive MIMO system, one of the potential
network infrastructures for beyond-5G and 6G networks, for enabling a seamless
computation offloading in a mobile edge network. Cell-free network can provide
sufficiently fast and reliable access throughout the coverage of the network,
virtually eliminating cell-edge users. This is accomplished by serving a
relatively small number of users simultaneously from several geographically
distributed access points (APs) that are connected to a central processing
unit (CPU) [10-12]. In the mobile edge network under consideration, the CPU is
equipped with an edge server of finite capacity to render computing services
for resource-constrained users with time-critical and computationally
intensive tasks. As opposed to the unreliable wireless links in cellular MEC
systems, cell-free access links in our framework can play the key
functionality of realizing virtual buses for parallel computing, i.e. allowing
for instant access to the edge server in the CPU, irrespective of users’
location, thus energy-efficient and consistently low latency computational
task offloading can be provided.

In this paper, we formulate a joint communication and computing resource
allocation (JCCRA) problem for the cell-free massive MIMO-enabled mobile edge
network. Specifically, with the objective of minimizing energy consumption of
all users subject to the ultra-low delay requirements, we intend to design
joint allocation of local processor clock speed and uplink transmission power
for each user. While solving the problem centrally at the CPU can allow for
efficient joint resource allocation, global knowledge of the entire network
state is required for decision making. For instance, information originating
from users’ side such as computing demands, maximum tolerable task execution
deadlines, and channel conditions should be collected, processed and then the
allocation decisions need to be communicated back to the users within the
tight delay tolerance, incurring prohibitively significant signaling and
communication overheads. On the other hand, distributed approaches based on
traditional optimization methods may lead to suboptimal performance. In
general, applying conventional optimization methods or other heuristic
algorithms to solve the JCCRA problem in a dynamic mobile edge network
necessitates frequent re-evaluation of the optimal allocation strategy,
following the non-deterministic task arrivals and stochastic wireless channel
conditions. Furthermore, the algorithms must converge within the ultra-low
delay tolerance. It is also important to note that the joint resource
allocation decisions are inter-coupled among the users, which poses additional
challenges. Consequently, the application of these methods to support time-
critical and computation-intensive services is critically limited in practice.
The JCCRA problem, therefore, calls for an adaptive and robust solution
approaches with reasonable complexity and signaling overheads.

Recently, integrating the capabilities of artificial intelligence (AI) in the
mobile edge network, referred to as edge intelligence, is considered as one of
the key enablers to realize the highly delay-intolerant cloud services for
advanced multimedia applications, e.g., extended reality (XR) [13]. More
specifically, machine learning and reinforcement learning algorithms can be
applied at different entities in the mobile edge network for efficient
resource allocation strategies. Motivated by this trend, we propose a fully
distributed solution approach based on cooperative multi-agent reinforcement
learning (MARL) for the formulated JCCRA problem, wherein each user is
implemented as a learning agent that makes joint resource allocation relying
on local observations only. The agents are trained with the state-of-the-art
multi-agent deep deterministic policy gradient (MADDPG) algorithm under the
framework of centralized training and decentralized execution. Through
continuous interactions with the mobile edge network, each agent tries to
capture dynamics of the environment and makes use of the learned knowledge for
an adaptive and robust joint resource allocation during the online execution
phase. Our simulation results show that the proposed distributed JCCRA
approach has achieved comparable performance with a centralized deep
deterministic policy gradient (DDPG)-based target JCCRA scheme implemented at
the CPU, without the need for central processing and resorting to large
overhead. Moreover, a significant performance improvement in terms of
providing energy-efficient and consistently low latency task offloading has
been demonstrated as compared to conventional benchmarks. To the best of our
knowledge, this is the very first attempt to solve JCCRA problem in a fully
distributed fashion for mobile edge computing in a cell-free massive MIMO
network. The fully distributed and intelligent JCCRA scheme coupled with a
reliable performance of its cell-free architecture in our framework can be a
promising means of handling the stringent requirements involved with the
envisaged multimedia applications.

The rest of the paper is organized as follows: In Section II, we review
related works on JCCRA allocation in different MEC system models. In Section
III, we discuss the system model for the considered mobile edge network and
then, present problem formulation for JCCRA. Section IV discusses the proposed
distributed cooperative multi-agent reinforcement learning-based solution
approach for the JCCRA problem. Simulation results are presented in Section V,
followed by some concluding remarks in Section VI. For convenience, we
summarized the key notations to be used throughout the paper in Table I.

Table I: Summary of important notations Notation |  Description
---|---
$\mathcal{M}=\left\\{1,2,...,M\right\\}$ |  Set of access point (AP) indices
$\mathcal{K}=\left\\{1,2,...,K\right\\}$ |  Set of user indices
${{{\cal C}}_{k}}$ |  Cluster of APs to serve the $k$-th user
${N_{k}}$ |  The number of APs in a cluster for user $k$
$AP_{n}^{(k)}$ |  The $n$-th AP in ${{{\cal C}}_{k}}$ to serve the $k$-th user, $n=1,2,...,{{N}_{k}}$
$\Delta t$ |  Duration of a time step
$t_{k}^{d}$ |  Application deadline for user $k$
${{\alpha}_{k}}$ |  Proportion of local processor clock speed allocation for user $k$
${{\eta}_{k}}$ |  Uplink transmit power control coefficient of user $k$
$p_{k}^{\max}$ / $p_{k}$ |  Maximum / allocated uplink transmit power by user $k$
$W$ |  System bandwidth
${{R}_{k}}$ |  Uplink rate of user $k$
${{N}_{cpb}}$ |  The number of CPU cycles required to process a one-bit task
${{\cal T}_{k}}$ / ${\cal T}_{k}^{local}$ / ${\cal T}_{k}^{offload}$ |  Incoming / locally computed / offloaded task size (in bits) by user $k$
$f_{k}^{\max}$ / $f_{k}^{local}$ |  Maximum / allocated local processor clock speed (in Hz) by user $k$
${f^{CPU}}$ |  Computing clock speed of the edge server in the CPU (in Hz)
$f_{k}^{CPU}$ |  Allocated computational resource at the CPU for user $k$
$t_{k}^{tr}$ / $t_{k}^{comp}$ |  Transmission / computing delay experienced by user $k$
$t_{k}^{local}$ / $t_{k}^{offload}$ / ${{t}_{k}}$ |  Local execution / offloading / total delay experienced by user $k$
$E_{k}^{local}$ /$E_{k}^{offload}$ / ${{E}_{k}}$ |  Local / offloading / total energy consumption incurred by user $k$

##  II Related work

Recently, several centralized schemes of computation offloading and resource
allocation have been proposed for different MEC system models to minimize
latency [14-15], energy consumption [16-18], or to balance delay-energy
tradeoff [19-20]. Particularly, in [14] and [16], a joint computation
offloading and resource allocation problem to minimize latency is studied in
vehicular networks, and energy harvesting devices, respectively. Minimization
of energy consumption with partial offloading in a single-user MEC system is
investigated in [16], which was later extended to a multi-user MEC system by
jointly optimizing computation and communication resources [17]. Therein, to
adjust the processor frequency, dynamic voltage scaling (DVS) technique was
adopted. Similarly, energy-delay tradeoff is investigated in [19] for a binary
offloading MEC system, in which the users decide to offload or not. Meanwhile,
power-delay tradeoff for partial offloading MEC system based on Lyapunov
optimization is studied in [20]. We note that the above centralized schemes
are based on conventional optimization methods that need to gather information
originating at the mobile devices, such as application requirements, task
size, energy level of battery, offloading requests, and channel conditions of
the users [21]. Therefore, their application to support computation-hungry
time-sensitive services might be hindered due to communication and signaling
overheads associated with collecting users’ information and sharing the
resource allocation decisions back to the users within the tight delay
constraint. Moreover, the centralized algorithms are not scalable in the sense
that their computational complexity increases with system size. It is even
more challenging from a cell-free massive MIMO point of view as we have to
deal with a large number of access points (APs) in a joint manner, in contrast
to the above cell-centric MEC models, where a user is connected to only a
single AP in each cell.

To relieve the signaling overhead and address scalability issues of the
centralized resource allocation schemes, plethora of distributed approaches
have been proposed. A design of distributed offloading decision is formulated
as a multi-user offloading game among the users for mobile cloud computing
[22], wireless-powered MEC networks [23], and MEC-empowered small-cell
networks [24]. In [25], offloading decision and resource allocation problem is
solved sequentially through decomposition technique within proximate clouds in
a distributed fashion. However, the focus of the majority distributed JCCRA
approaches is confined to the design of offloading decisions. In addition to
this, due to the lack of global knowledge of the network environment, the
algorithms are likely to converge to suboptimal solutions. It is worth
mentioning that all these schemes are based on conventional optimization
methods that are limited to quasi-static systems with deterministic task
arrival and invariant channel conditions, failing to capture the randomness in
practical systems. Consequently, in response to the dynamics in the
environment, the JCCRA problem should be solved frequently, and the algorithms
are expected to converge swiftly within the task execution deadline. Such
approaches are, therefore, not suited for handling time-sensitive
applications.

JCCRA in a dynamic mobile edge network with time-varying channel conditions
and stochastic task arrivals require robust solutions that can not only adapt
to the diversities in the network conditions and application-specific
requirements (e.g., delay constraint), but also perform well with constrained
radio and computational resources. In this regard, owing to their adaptability
in dynamic settings, reinforcement learning-based frameworks have recently
been proposed to solve joint resource allocation problems in various MEC
systems. For instance, a Q-learning based joint design of offloading decisions
and resource allocation schemes have been proposed for a single-cell MEC to
minimize energy consumption [26], and weighted sum cost of energy and task
delay [27]. Furthermore, a deep Q-network (DQN) algorithm has been proposed to
solve binary offloading decisions in different MEC system models [29-31]. In
all of these works, however, the decisions are made by central entities which
need to gather system-wide information; thus, the schemes are prone to the
limitations of centralized resource allocations discussed above. Moreover,
training a centralized super-agent can potentially be ineffective as the
convergence of the training is not guaranteed. Furthermore, it is not a
scalable approach since the performance degrades due to the linear input and
output dimensions increase with the number of users in the system. [32]
investigates a distributed JCCRA problem in a single-cell MEC system in which
each user tries to minimize the individual cost, defined as a weighted sum of
power and delay, in a non-cooperative fashion, possibly resulting in
suboptimal performance. Moreover, the adopted computational model is not
applicable to handle computation-intensive tasks, requiring extreme
reliability and ultra-low hard deadline.

Recently, there are few works on the cell-free massive MIMO-based MEC systems.
In a MEC system with both CPU and APs equipped with computing servers, several
performance analyses are presented in [33], while considering coverage radius
of the APs and probabilities of offloading to the servers. From a massive
access perspective, the issues of active user detection and channel
estimations are discussed for a cell-free massive MIMO-based computing
framework in [34]. In contrast to our work, both [33] and [34] do not deal
with the JCCRA problem.

##  III System Model and Problem Formulation

###  III-A User-centric Cell-free Massive MIMO: Overview

A cell-free massive MIMO system is considered as a potential beyond-5G and 6G
network infrastructure as it provides a uniform user experience by eliminating
the cell edge. It exploits joint signal processing from a large number of
distributed access points (APs), which simultaneously serve a much smaller
number of users using the same radio resources. To attain the tight
requirements of the envisaged advanced services, the capability of cell-free
access links to provide reliable performance without cell edge is essential to
play the key functionality of virtual bus while offloading an intensive task
for parallel computation at the edge server.

We consider a user-centric mobile edge network with $M$ single-antenna access
points (APs) and $K$ single-antenna users, where $M\gg K$ , entailing to the
widely known model for cell-free massive MIMO system [10]. The APs are all
connected to a central processing unit (CPU) via error-free backhaul links.
Let $\mathcal{M}=\left\\{1,2,...,M\right\\}$ and
$\mathcal{K}=\left\\{1,2,...,K\right\\}$ denote sets of AP and user indices,
respectively. We assume that all APs and UEs are distributed uniformly
throughout the network. Fig. 1 illustrates a system model for cell-free
massive MIMO-enabled mobile edge network. Therein, the CPU is equipped with a
computing server of finite computational capability to provide computing
resources for users with computationally intensive and latency stringent
applications. Each user is also equipped with limited local processing
capability.

Figure 1: User-centric cell-free massive MIMO-enabled mobile edge network:
_Illustrative system model_

Let the channel between $m$-th AP and $k$-th user is given as
${g_{mk}}=\beta_{mk}^{{}^{1}/{}_{2}}{h_{mk}}$ where ${{\beta}_{mk}}$ is a
large-scale channel gain coefficient and
${{h}_{mk}}\sim\mathrm{}\left(0,1\right)$ represents small-scale channel
fading. Let ${{\tau}_{c}}$ denote a channel coherence period in which
${{h}_{mk}}$ remains the same, while ${{\beta}_{mk}}$ remains constant for a
time interval which lapses multiple channel coherence times. The coherence
period ${{\tau}_{c}}$ is divided into pilot transmission time of
${{\tau}_{p}}$ samples and uplink data transmission time of
(${{\tau}_{c}}-{{\tau}_{p}}$) samples.

All users simultaneously transmit pilot sequences to the APs for uplink
channel estimation. Let
${{\mathbf{\psi}}_{k}}\in{{\mathsf{\mathbb{C}}}^{{{\tau}_{p}}\times\,1}}$
denote a pilot sequence of user $k$ where
${{\left\|{{\mathbf{\psi}}_{k}}\right\|}^{2}}=1$. We assume that pilot
contamination can be ignored by picking pairwise orthogonal sequences
${{\mathbf{\psi}}_{1}},{{\mathbf{\psi}}_{2}},.....,{{\mathbf{\psi}}_{K}}$,
i.e., ${{\tau}_{p}}=K$. Then, the received pilot vector at the $m$-th AP,
$\mathbf{y}_{m}^{p}\in{{\mathsf{\mathbb{C}}}^{{{\tau}_{p}}\times\,1}}$ , can
be represented as follows:

${\bf{y}}_{m}^{p}=\sqrt{{\tau_{p}}}\sum\limits_{k=1}^{K}{\sqrt{p_{k}^{p}}{g_{mk}}{{\bf{\psi}}_{k}}}+{\bf{\omega}}_{m}^{p},$
(1)

where $p_{k}^{p}$ denotes the pilot transmit power, and
$\mathbf{\omega}_{m}^{p}$ denotes a $\,{{\tau}_{p}}$-dimensional additive
noise vector with independent and identically distributed entries of
${\bf{\omega}}_{m}^{p}\sim{{\cal C}{\cal N}}\left({0,\sigma_{m}^{2}}\right)$.
Based on the received vector ${\bf{y}}_{m}^{p}$, the least-square (LS) channel
estimate ${\hat{g}_{mk}}$ can be expressed as

${\hat{g}_{mk}}=\frac{1}{{\sqrt{{\tau_{p}}p_{k}^{p}}}}{\bf{\psi}}_{k}^{H}{\bf{y}}_{m}^{p}.$
(2)

The channel estimates are used to decode the uplink transmitted data of the
users. After pilot transmission, the users transmit offloaded data to the APs.
Let ${{x}_{k}}$ denote the uplink transmission data of user $k$. Then, the
received signal $y_{m}^{u}$ at the $m$-th AP is given as

$y_{m}^{u}=\sum\limits_{k=1}^{K}{\sqrt{{p_{k}}}{g_{mk}}{x_{k}}}+{\omega_{m}},$
(3)

where ${{p}_{k}}$ is the uplink data transmit power. Let
${{p}_{k}}={{\eta}_{k}}\,p_{k}^{\max}$, where ${{\eta}_{k}}$, and
$p_{k}^{\max}$ represent power control coefficient and maximum uplink transmit
power of UE $k$, respectively.

In order to ensure scalability of cell-free massive MIMO system in terms of
complexity for pilot detection and data processing, each UE is served by a
limited number of APs, which forms a cluster. Let ${N_{k}}$ denote a total
number of APs in a cluster of user $k$. To this end, it is essential to form a
cluster of APs serving each user in a user-centric manner. Let ${\cal C}_{k}$
denote a cluster of the $k$-th user, i.e., ${{{\cal
C}}_{k}}=\left\\{{AP_{1}^{(k)},AP_{2}^{(k)},\cdots,AP_{{N_{k}}}^{(k)}}\right\\}$,
where $AP_{n}^{(k)}$ denote the $n$-th AP in the cluster. For simplicity of
exposition in the current discussion, assume that all users have the same size
of cluster, given by ${{C}^{\max}}$, i.e., ${{N}_{k}}={{C}^{\max}},\text{
}k=1,2,...,K$. In order to construct a cluster, we employ a greedy approach
which orders the large-scale channel coefficients of user $k$ with the
respective APs in a descending order and then, includes all APs with the
largest until ${{N}_{k}}={{C}^{\max}}$. Thus, the transmitted data by user $k$
is only decoded by the APs in ${{{\cal C}}_{k}}$. Then each AP $m\in{{{\cal
C}}_{k}}$ transmits the quantity $\hat{g}_{mk}^{*}y_{m}^{u}$ to the CPU via a
fronthaul link. The received soft estimates are combined at the CPU to decode
the data transmitted by user $k$ as follows:

${\hat{x}_{k}}=\sum\limits_{m=1}^{N_{k}}{\hat{g}_{mk}^{*}y_{m}^{u}}.$ (4)

Then, the uplink SINR ${{\gamma}_{k}}$ for user $k$ can be expressed as

${\gamma_{k}}=\frac{{{p_{k}}\left\\{{{{\left|{\sum\limits_{m\in{{{\cal
C}}_{k}}}{\hat{g}_{mk}^{*}{g_{mk}}}}\right|}^{2}}}\right\\}}}{{\sum\limits_{k^{\prime}\neq
k}{{p_{k^{\prime}}}}\left\\{{{{\left|{\sum\limits_{m\in{{{\cal
C}}_{k}}}{\hat{g}_{mk}^{*}{g_{mk^{\prime}}}}}\right|}^{2}}}\right\\}+\sigma_{m}^{2}\left\\{{\sum\limits_{m\in{{{\cal
C}}_{k}}}{{{\left|{{{\hat{g}}_{mk}}}\right|}^{2}}}}\right\\}}}$ (5)

The uplink rate of user $k$ is then given as
${R_{k}}\left({{p_{k}}}\right)=W{\log_{2}}\left({1+{\gamma_{k}}}\left({{p_{k}}}\right)\right)$,
where $W$ is the system bandwidth. More specifically, we adopt the closed-form
uplink spectral efficiency from [12].

###  III-B Parallel Computation Model

Without loss of generality, we assume each user $k$ has a computationally
intensive task with ${{{\cal T}}_{k}}\left(t\right)$ bits at the beginning of
every discrete time step $t=1,2,...$, whose duration is set to a coherence
period, i.e., $\Delta t={{\tau}_{c}}$. The task sizes in different time steps
are independent and uniformly distributed over $\left[{{{\cal
T}}_{\min}},{{{\cal T}}_{\max}}\right]$, for every user $k\in{{\cal K}}$. Let
$t_{k}^{d}$ denote the maximum tolerable delay of user $k$ to complete
execution of the time-sensitive application. Furthermore, we assume the tasks
are independent and fine grained, i.e., can be broken into arbitrary portions
so that they can be computed at the user device and the edge server in
parallel. At time step $t$, considering the delay constraint and energy
consumption, the $k$-th user processes ${{\cal T}}_{k}^{local}\left(t\right)$
bits locally and offloads the remaining ${{\cal
T}}_{k}^{offload}\left(t\right)$ bits to the edge server at the CPU. In the
sequel, we describe the models adopted for local computation and computation
offloading.

####  III-B1 Local computation

Let us denote the maximum local computing clock speed of user $k$ by
$f_{k}^{\max}$ (in cycle per second). Furthermore, let ${{N}_{cpb}}$ denote
the number of CPU cycles to process a one-bit task. Taking energy consumption
and delay requirement of the application into account, the user decides the
proportion ${{\alpha}_{k}}\in\left[0,1\right]$ for achieving the local clock
speed of
$f_{k}^{local}\left(t\right)={{\alpha}_{k}}\left(t\right)f_{k}^{\max}$, which
in turn determines the size of locally computed task. The entire task ${{{\cal
T}}_{k}}\left(t\right)$ is offloaded to the edge server if
${\alpha_{k}}\left(t\right)=0$. Meanwhile, ${\alpha_{k}}\left(t\right)=1$ if
the whole local processing capability of $f_{k}^{\max}$ is fully utilized
while offloading the remaining task bits to the edge server. Note that
conservative local processing may lead to high energy consumption, while
aggressive computation offloading by all users may subject some users to
service outage, since offloading all tasks cannot be supported with finite
radio and computation resources. In general, ${\alpha_{k}}$ is one of the
decision variables for efficient JCCRA, which is governed by the channel and
computational constraints in the local and edge systems, subject to the energy
consumption and delay requirement.

Given the application deadline $t_{k}^{d}$ to process ${{{\cal
T}}_{k}}\left(t\right)$ task bits subject to the local processing clock speed
of $f_{k}^{local}\left(t\right)$, then the size of locally computed task can
be expressed as ${{\cal T}}_{k}^{local}\left(t\right)=\min\left({{{\cal
T}}_{k}}\left(t\right),\frac{t_{k}^{d}f_{k}^{local}\left(t\right)}{{{N}_{cpb}}}\right)$
(in bits). Let $t_{k}^{local}\left(t\right)$ denote the time taken for local
execution at time step $t$, which is given as

$t_{k}^{local}\left(t\right)=\min\left({\frac{{{{\cal
T}}_{k}^{local}\left(t\right){N_{cpb}}}}{{f_{k}^{local}\left(t\right)}},t_{k}^{d}}\right).$
(6)

Then, the energy consumed for the local execution is expressed as

$E_{k}^{local}\left(t\right)=\varsigma{{\cal
T}}_{k}^{local}\left(t\right){N_{cpb}}{\left({f_{k}^{local}\left(t\right)}\right)^{2}},$
(7)

where $\zeta$ corresponds to the effective switched capacitance depending on
the chip architecture.

####  III-B2 Computation offloading

The $k$-th user offloads the remaining task bits ${{\cal
T}}_{k}^{offload}\left(t\right)=\max\left(0,\,\,{{{\cal
T}}_{k}}\left(t\right)-{{\cal T}}_{k}^{local}\left(t\right)\right)$ to the
edge server at the CPU for parallel computation. While offloading the
computational task to the edge server, the experienced latency can be broken
down into transmission delay for offloaded data, processing delay in edge
server, and transmission delay for retrieving the result. Since the retrieved
data size after computation in the server is much smaller as compared to the
offloaded data size, we ignore the retrieving delay in our formulation. Let
$t_{k}^{tr}\left(t\right)$ and $t_{k}^{comp}\left(t\right)$ denote the
transmission delay and computing delay, respectively, for the $k$-th user. To
offload ${{\cal T}}_{k}^{offload}\left(t\right)$ data bits to the edge server,
the transmission delay $t_{k}^{tr}\left(t\right)$ is given as

$t_{k}^{tr}\left(t\right)=\frac{{{{\cal
T}}_{k}^{offload}\left(t\right)}}{{{R_{k}}({\eta_{k}},t)}},$ (8)

where ${{R}_{k}}({{\eta}_{k}},t)$ is the uplink rate of user $k$ with a power
control factor of ${{\eta}_{k}}$ at time step $t$, which is expressed
according to the discussion in the previous subsection. It should be noted
that power control plays a critical role as we must consider co-channel
interference among the different clusters while offloading computation. To
that end, ${\eta_{k}}$ is another variable to consider for efficient
utilization of limited radio and computing resources.

Let ${{f}^{CPU}}$(in cycle per second) denote the computing clock speed of the
edge server in the CPU which is shared among the users in proportion to the
offloaded task size so that each user can experience uniform computation
delay. In other words, the allocated computational resource at the CPU for
user $k$, denoted as ${{f}_{k}}^{CPU}\left(t\right)$, can be expressed as

$f_{k}^{CPU}\left(t\right)=\frac{{{{\cal
T}}_{k}^{offload}\left(t\right)}}{{\sum\limits_{k=1}^{K}{{{\cal
T}}_{k}^{offload}\left(t\right)}}}{f^{CPU}}.$ (9)

Then the computing time $t_{k}^{comp}\left(t\right)$ required to execute bits
${{\cal T}}_{k}^{offload}\left(t\right)$ is given as

$t_{k}^{comp}\left(t\right)=\frac{{{{\cal
T}}_{k}^{offload}\left(t\right){N_{cpb}}}}{{f_{k}^{CPU}\left(t\right)}}.$ (10)

Therefore, the total edge-computing delay for executing ${{\cal
T}}_{k}^{offload}\left(t\right)$ bits is given as the sum of the delays for
uplink transmission and computation at the CPU, i.e.,
$t_{k}^{offload}\left(t\right)=t_{k}^{comp}\left(t\right)+t_{k}^{tr}\left(t\right)$.
The corresponding energy consumption is given as

$E_{k}^{offload}\left(t\right)={p_{k}}\left(t\right)t_{k}^{tr}\left(t\right),$
(11)

where the transmission power ${{p}_{k}}\left(t\right)$ is expressed as
${{p}_{k}}\left(t\right)={{\eta}_{k}}\,\left(t\right)p_{k}^{\max}$, and
$p_{k}^{\max}$ corresponds to the maximum uplink transmission power of UE $k$.

According to the communication and parallel computation models discussed
above, the overall experienced latency ${{t}_{k}}\left(t\right)$ by user $k$,
to execute ${{{\cal T}}_{k}}\left(t\right)$ bits locally and at the edge is
given by

${t_{k}}\left(t\right)=\max(t_{k}^{local}\left(t\right),t_{k}^{offload}\left(t\right)).$
(12)

The corresponding energy consumption ${{E}_{k}}\left(t\right)$ of user $k$,
can be expressed as

${E_{k}}\left(t\right)=E_{k}^{local}\left(t\right)+E_{k}^{offload}\left(t\right).$
(13)

###  III-C JCCRA Problem Formulation

We now present JCCRA problem formulation in our framework. Specifically, our
objective is to minimize the total energy consumption of all users while
meeting the respective user-specific delay requirements by jointly optimizing
the local processor speed
$f_{k}^{local}\left(t\right)={{\alpha}_{k}}\left(t\right)f_{k}^{\max}$, and
uplink transmission power
${{p}_{k}}\left(t\right)={{\eta}_{k}}\,\left(t\right)p_{k}^{\max}$, for every
user $k\in{{\cal K}}$ at each time step $t$, without the assumption of any
prior knowledge of future realizations of the incoming task size and channel
conditions of the users. The JCCRA problem to jointly determine
$\left({{\alpha}_{k}}\left(t\right),{{\eta}_{k}}\left(t\right)\right),\forall
k\in{{\cal K}}$ can mathematically be formulated by (14), shown at the top of
the next page. Therein, the first constraint ensures that the task execution
delay ${{t}_{k}}\left(t\right)$ should not exceed the user-specific delay
requirement $t_{k}^{d}$.

$\begin{array}[]{l}\,\mathop{\min}\limits_{\\{{\alpha_{k}}(t),{\eta_{k}}(t)|\forall
k\\}}\,\,\,\,\,\,\,\,\,\sum\limits_{k=1}^{K}{{E_{k}}(t)=\sum\limits_{k=1}^{K}{\varsigma\min\left({{{{\cal
T}}_{k}}(t),\frac{{t_{k}^{d}{\alpha_{k}}(t)f_{k}^{\max}}}{{{N_{cpb}}}}}\right){N_{cpb}}{{\left\\{{{\alpha_{k}}(t)f_{k}^{\max}}\right\\}}^{2}}+{\eta_{k}}(t)\,p_{k}^{\max}\frac{{\max\left({0,\,\,{{{\cal
T}}_{k}}(t)-\min\left({{{{\cal
T}}_{k}}(t),\frac{{t_{k}^{d}\,{\alpha_{k}}(t)f_{k}^{\max}}}{{{N_{cpb}}}}}\right)}\right)}}{{{R_{k}}({\eta_{k}}(t))}}}}\\\
\,\,\,\,\,\,\,\,\,\,{\rm{subject}}\,{\rm{to}}\,\,\,\,\,\,\,\,\,\,\,\,\,\,\max\left({\min\left({\frac{{\min\left({{{{\cal
T}}_{k}}(t),\frac{{t_{k}^{d}\,{\alpha_{k}}(t)f_{k}^{\max}}}{{{N_{cpb}}}}}\right){N_{cpb}}}}{{{\alpha_{k}}(t)f_{k}^{\max}}},t_{k}^{d}}\right),{{\cal
T}}_{k}^{offload}\left(t\right)\left[{\frac{{{N_{cpb}}}}{{f_{k}^{CPU}\left(t\right)}}+\frac{1}{{{R_{k}}({\eta_{k}},t)}}}\right]}\right)\leq
t_{k}^{d},\,\,\forall k\\\
\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,0\leq{\alpha_{k}}\left(t\right)\leq
1,\,\,\forall k\\\
\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,0\leq{\eta_{k}}\left(t\right)\leq
1,\,\,\,\forall k\end{array}$ (14)

The JCCRA problem in (14) is a stochastic optimization problem in which the
objective function and the first condition involve constantly changing random
variables, following the dynamics caused by the random incoming task size at
each user, and wireless channel conditions. Hence, the time-varying
optimization variables
$\left({{\alpha}_{k}}\left(t\right),{{\eta}_{k}}\left(t\right)\right),\forall
k\in{{\cal K}}$, should be determined frequently, i.e., at every step $t$, and
rapidly within the ultra-low deadline. Further, the joint resource allocation
decisions are inter-coupled among the users, which poses additional challenge.
It is, therefore, challenging to solve the problem using conventional
optimization algorithms. Moreover, from the cell-free massive MIMO
perspective, solving the JCCRA problem centrally at the CPU might allow for
efficient management of the available radio and computing resources due to
globally processing of network-wide information. However, the associated
signaling and communication overheads are forbiddingly significant, especially
given the ultra-low delay constraints. Thus, to cope up with the dynamics in
the mobile edge network and derive a flexible and efficient joint resource
allocation for every user, we propose a fully distributed JCCRA based on
cooperative multi-agent reinforcement learning. We note that this is the very
first attempt to solve JCCRA problem in a fully distributed manner for a cell-
free massive MIMO-enabled mobile edge computing network. While alleviating the
large overheads associated with the centralized implementation, the proposed
distributed approach enables users to entertain energy-efficient and
consistently low end-to-end delay computational task offloading.

##  IV The Proposed Multi-Agent Reinforcement Learning-based Distributed JCCRA

In this section, we present a fully distributed JCCRA solution approach based
on cooperative multi-agent reinforcement learning framework, in which each
user device is implemented as an agent for joint resource allocation relying
on local observation only. In particular, the agents learn to map their local
observation into JCCRA actions that minimize the total energy consumption
while meeting the respective application deadlines. As opposed to the
centralized schemes, such a distributed approach significantly alleviates
signaling and communication overhead while making resource allocation
decisions.

Figure 2: Multi-agent reinforcement learning framework: _Illustration_

###  IV-A Distributed DRL Formulation for JCCRA

The DRL agents sequentially interact with the mobile edge network in discrete-
time steps to learn optimal joint resource allocation policies. Let ${{{\cal
O}}_{k}}$, and ${{\cal S}}$ denote the local observation space of agent
$k\in{{\cal K}}$, and the complete environment state space, respectively. As
shown in Fig. 2, at time step $t$, each agent relies on local observation of
the environment state ${o_{k}}\left(t\right):{{\cal S}}\mapsto{{{\cal
O}}_{k}}$ to determine an action ${a_{k}}\left(t\right)\in{{{\cal A}}_{k}}$,
from its action space ${{{\cal A}}_{k}}$ according to the current JCCRA policy
${{\mu}_{k}}$. The shared environment collects the joint action of the agents
$a\left(t\right)=\left({{a}_{1}}\left(t\right),...,{{a}_{K}}\left(t\right)\right)$,
and emits the next observations ${o_{k}}\left({t+1}\right)\in{{{\cal O}}_{k}}$
and real-valued scalar rewards ${r_{k}}\left(t\right):{{\cal S}}\times{{{\cal
A}}_{k}}\mapsto\mathsf{\mathbb{R}}$ for all $k\in{{\cal K}}$. The goal of the
agents is, therefore, to constantly improve their respective policy until it
converges to the optimal JCCRA policy $\mu_{k}^{*}$ that maximizes the
expected long-term discounted cumulative reward, defined as
${{J}_{k}}\left({{\mu}_{k}}\right)=\mathbb{E}\left[\sum\limits_{t=1}^{T}{{{\varepsilon}^{t-1}}{{r}_{k}}}\left(t\right)\right]$,
where $\varepsilon\in\left[0,\,1\right]$ is the discount factor and $T$ is the
total number of total time steps (horizon). The optimal JCCRA policy
$\mu_{k}^{*}$ is then given as

$\mu_{k}^{*}=\mathop{\arg\,}\limits_{{\mu_{k}}}\max{J_{k}}\left({{\mu_{k}}}\right)\,.$
(15)

In the sequel, we define the _local observation_ , _action_ , and _reward_ of
each agent $k\in{{\cal K}}$ for JCRRA at a given time $t$.

####  IV-A1 Local observation

As discussed in the previous section, the compute-intensive tasks are
subjected to user-specific application deadline. Specifically, at time step
$t$, the maximum delay tolerance of the $k$-th agent to execute incoming task
of ${{{\cal T}}_{k}}(t)$ bits is given by $t_{k}^{d}$. Meanwhile, we assume
the agent has a rate assignment result from the previous time step, which is
given as uplink data rate ${{R}_{k}}(t-1)$. At the beginning of each time step
$t$, the local observation of agent $k$ is defined as

${o_{k}}\left(t\right)\buildrel\Delta\over{=}\left[{{{{\cal
T}}_{k}}(t),t_{k}^{d},{R_{k}}(t-1)}\right].$ (16)

Note that the local observation is only a partial view of the environment,
i.e., it does not include information about the other agents.

####  IV-A2 Action

Based on the local observation of the environment, each agent $k$ jointly
determines how much local computing resources and uplink transmit power must
be allocated to execute ${{{\cal T}}_{k}}(t)$ within $t_{k}^{d}$. Accordingly,
the action of the $k$-th agent at time step $t$ can be expressed as

${a_{k}}\left(t\right)\buildrel\Delta\over{=}\left[{{\alpha_{k}}\left(t\right),{\eta_{k}}\left(t\right)}\right],$
(17)

where ${\alpha_{k}}(t)\in\left[{0,1}\right]$, and
${\eta_{k}}(t)\in\left[{0,1}\right]$ are continuous values that govern the
local processor clock speed
$f_{k}^{local}\left(t\right)={{\alpha}_{k}}\left(t\right)f_{k}^{\max}$, and
uplink transmit power ${{p}_{k}}={{\eta}_{k}}\,p_{k}^{\max}$, respectively.

####  IV-A3 Reward

The agents must learn a cooperative JCCRA policies that minimize the total
energy consumption of all users while meeting the respective delay
constraints. The immediate reward, therefore, should encapsulate both aspects
of the design objective. Accordingly, to encourage the agents to maximize the
common goal and enforce cooperation among them, we define the joint reward as

${r_{k}}(t)=-\sum\limits_{k=1}^{K}{{\xi_{k}}{E_{k}}\left(t\right),\,\,}\forall
k\in{{\cal K}},$ (18)

where ${\xi_{k}}=1$ if ${t_{k}}\left(t\right)\leq t_{k}^{d}$, otherwise
${\xi_{k}}=10$ to punish potentially selfish behavior of the agents that lead
to failure in meeting the delay constraint.

Figure 3: Centralized training and decentralized execution framework in MADDPG
algorithm

We note that the decision variables, i.e., ${{\alpha}_{k}}$ and
${{\eta}_{k}}$, for the JCCRA, are continuous-valued. However, a direct
extension of single-agent reinforcement learning methods for continuous action
spaces, such as deep deterministic policy gradient (DDPG) algorithm [35],
i.e., training each agent independently, faces several problems when applied
to learn coordinated policies. Learning in a multi-agent setup is more
challenging and complex than in single-agent case, since the environment is no
longer stationary from the agent’s perspective as the other agents update
their policies concurrently. In other words, the agents face a moving target
problem, which may lead to learning instability. Moreover, the non-
stationarity of the environment compounded on agents’ partial observations of
the dynamic environment can cause shadowed equilibria, a phenomenon in which
agents’ local optimal actions result in globally sub-optimal joint action
[36]. In the next subsection, we discuss a multi-agent version of DDPG
algorithm, referred to as multi-agent deep deterministic policy gradient
(MADDPG) [37], based on the centralized training and decentralized execution
framework, to train all agents for JCCRA decision making.

###  IV-B MADDPG Algorithm for Distributed JCCRA

Figure 4: Illustration of the training procedure in MADDPG algorithm for the
$k$-th agent

1

2for  _each agent $k\in{{\cal K}}$_ do

3 Initialize replay buffer ${{{\cal D}}_{k}}$.

4 Initialize the actor network
${\mu_{k}}\left({{o_{k}}|\theta_{k}^{\mu}}\right)$ and critic network
${Q_{k}}\left({{s_{k}},a|\theta_{k}^{Q}}\right)$ with weights
$\theta_{k}^{\mu}$ and $\theta_{k}^{Q}$, respectively.

5 Initialize the target networks, ${\mu^{\prime}_{k}}$ and ${Q^{\prime}_{k}}$
with weights $\theta_{k}^{\mu^{\prime}}$ and $\theta_{k}^{Q^{\prime}}$,
respectively.

6

7 end for

8

9for  _each episode $e=1,2,...$_ do

10 for  _each agent $k\in{{\cal K}}$_ do

11 Initialize random process ${{{\cal N}}_{k}}$ for exploration.

12 Generate initial local observation from the environment simulator.

13 end for

14

15 for  _each step $t=1,2,...$_ do

16 for  _each agent $k\in{{\cal K}}$_ do

17 Select action
${a_{k}}\left(t\right)={\mu_{k}}\left({{o_{k}}\left(t\right)|\theta_{k}^{\mu}}\right)+{{{\cal
N}}_{k}}\left(t\right)$

18 end for

19 Execute joint action
$a\left(t\right)=\left({{a_{1}}\left(t\right),....,{a_{K}}\left(t\right)}\right)$
for  _each agent $k\in{{\cal K}}$_ do

20

21 Collect reward ${r_{k}}\left(t\right)$ and observe
${s_{k}}\left({t+1}\right)$.

22 Store the transition
$\left({{s_{k}}\left(t\right),a\left(t\right),{r_{k}}\left(t\right),{s_{k}}\left({t+1}\right)}\right)$
into ${{{\cal D}}_{k}}$.

23 Sample random minibatch of _B_ transitions
$\left({s_{k}^{i},{a^{i}},r_{k}^{i},s_{k}^{i+1}}\right)$ from ${{{\cal
D}}_{k}}$.

24 Update the critic network by minimizing the loss given by (19).

25 Update the actor policy using the sampled policy gradient given by (20).

26 Update the target networks according to (21).

27

28 end for

29

30 end for

31

32 end for

33

Algorithm 1 MADDPG Algorithm for JCCRA

Similar to its single agent counterpart, MADDPG is an actor-critic policy
gradient algorithm. However, in the course of the offline training phase, the
multi-agents are trained at a central unit which presents an opportunity for
sharing extra information so as to ease the training process. As depicted in
Fig. 3, in addition to the local observations of the environment, the agents
are provided with additional information (shown as dotted lines), such as
observations and actions of the other agents. Specifically, agent $k$ has now
access to the joint action
$a\left(t\right)=\left({{a}_{1}}\left(t\right),...,{{a}_{K}}\left(t\right)\right)$,
and full observation of the environment state
${{s}_{k}}\left(t\right)=\left({{o}_{k}}\left(t\right),{{o}_{-k}}\left(t\right)\right)$,
where ${{o}_{-k}}\left(t\right)$ is the local observations of other agents at
time step $t$. The additional information, in particular the actions taken by
other agents, can enable the multi-agents to overcome the challenges posed by
the non-stationarity of the environment, thereby successfully capturing the
dynamics of the mobile edge network. The extra information endowed to the
agents during the centralized training phase, however, is discarded during the
execution phase, meaning that the agents fully rely on their local
observations to make JCCRA decisions in a fully distributed manner.

The MADDPG agent $k$ employs two main deep neural networks: actor network with
parameters $\theta_{k}^{\mu}$ to approximate a joint resource allocation
policy ${{\mu}_{k}}\left({{o}_{k}}|\theta_{k}^{\mu}\right)$ and a critic
network, parametrized by $\theta_{k}^{Q}$, to approximate a state-value
function ${{Q}_{k}}\left({{s}_{k}},a|\theta_{k}^{Q}\right)$, along with their
respective time-delayed copies, $\theta_{k}^{{{\mu}^{\prime}}}$ and
$\theta_{k}^{Q^{\prime}}$ which serve as targets. At time step $t$, the actor
deterministically maps local observation ${{o}_{k}}\left(t\right)$ to a
specific continuous action
${{\mu}_{k}}\left({{o}_{k}}\left(t\right)|\theta_{k}^{\mu}\right)$ and then, a
random noise process ${{{\cal N}}_{k}}$ is added to generate an exploratory
policy such that
${{a}_{k}}\left(t\right)={{\mu}_{k}}\left({{o}_{k}}\left(t\right)|\theta_{k}^{\mu}\right)+{{{\cal
N}}_{k}}\left(t\right)$. The environment, which is shared among the agents,
collects the joint action
$a\left(t\right)=\left\\{{{a_{k}}\left(t\right),\forall k\in{{\cal
K}}}\right\\}$ and returns the immediate reward ${{r}_{k}}\left(t\right)$ and
the next observation ${{o}_{k}}\left(t+1\right)$ to the respective agents. To
make use of the experience in later decision-making steps, i.e., to improve
sample efficiency, and stability of the training, the agent’s transition along
with the extra information
${{e}_{k}}\left(t\right)=\left({{s}_{k}}\left(t\right),a\left(t\right),{{r}_{k}}\left(t\right),{{s}_{k}}\left(t+1\right)\right)$
is saved in the replay buffer ${\cal D}_{k}$ of the agent $k$.

To train the main networks, mini batch of $B$ samples
$\left.{\left({s_{k}^{i},a^{i},r_{k}^{i},s_{k}^{i+1}}\right)}\right|_{i=1}^{B}$,
is randomly drawn from the replay buffer ${\cal D}_{k}$, denoting a sample
index by $i$. The critic network is updated to minimize the following loss
function:

${{\cal
L}_{k}}\left({\theta_{k}^{Q}}\right)=\frac{1}{B}\,\,\sum\nolimits_{i}{\,{{\left({y_{k}^{i}-{Q_{k}}\left({s_{k}^{i},{a^{i}}|\theta_{k}^{Q}}\right)}\right)}^{2}}}\,\,,$
(19)

where $y_{k}^{i}$ is the target value expressed as
$y_{k}^{i}={{\left.r_{k}^{i}+\varepsilon\,\,{{{{Q}^{\prime}}}_{k}}\left(s_{k}^{i+1},{{a}_{i+1}}|\theta_{k}^{Q^{\prime}}\right)\right|}_{{{a}_{{}^{i+1}}}=\left\\{{{{{\mu}^{\prime}}}_{k}}\left(o_{k}^{i+1}\right),\forall
k\in\cal{K}\right\\}}}$, with a discount factor $\varepsilon\,$. More
specifically, the parameters of the critic network, $\theta_{k}^{Q}$, are
adjusted by following the gradient of (19) such that
$\theta_{k}^{Q}\leftarrow\theta_{k}^{Q}-{\beta_{Q}}\,\,{\nabla_{\theta_{k}^{Q}}}{{\cal
L}_{k}}\left({\theta_{k}^{Q}}\right)$, with a learning rate of ${\beta_{Q}}$
for the critic network. On the other hand, the policy network updates its
parameters to maximize the expected long-term discounted reward
$J\left({{\mu}_{k}}|\theta_{k}^{\mu}\right)$ according to
$\theta_{k}^{\mu}\leftarrow\theta_{k}^{\mu}-{{\beta}_{\mu}}\,\,{{\nabla}_{\theta_{k}^{\mu}}}J\left({{\mu}_{k}}|\theta_{k}^{\mu}\right)$,
where ${{\nabla}_{\theta_{k}^{Q}}}J\left({{\mu}_{k}}|\theta_{k}^{\mu}\right)$
is the deterministic policy gradient expressed as

${\nabla_{\theta_{k}^{\mu}}}J\left({{\mu_{k}}|\theta_{k}^{\mu}}\right)\approx\frac{1}{B}\,\Biggl{[}\sum\nolimits_{i}{{\nabla_{{a_{k}}}}}{Q_{k}}\left({s_{k}^{i},{a_{i}}|\theta_{k}^{Q}}\right)\times\\\
\nabla_{\theta_{k}^{\mu}}\,{\mu_{k}}\left({o_{k}^{i}|\theta_{k}^{\mu}}\right)\Biggl{]}\Biggl{|}_{{a_{i}}=\left\\{{{\mu_{k}}\left({o_{k}^{i}}\right),\,\forall
k\in{{\cal K}}}\right\\}}\,,$ (20)

and ${{\beta}_{\mu}}$ is the learning rate of the actor network. The target
parameters in both actor and critic networks are updated with soft updates as
follows:

$\displaystyle\theta_{k}^{{{\mu}^{\prime}}}\leftarrow\tau\theta_{k}^{\mu}+\left(1-\tau\right)\theta_{k}^{{{\mu}^{\prime}}}$
$\displaystyle\theta_{k}^{{{Q}^{\prime}}}\leftarrow\tau\theta_{k}^{Q}+\left(1-\tau\right)\theta_{k}^{{{Q}^{\prime}}},$
(21)

where $\tau$ is a constant close to zero.

Fig. 4 depicts the interactions of the four networks while training the $k$-th
MADDPG agent. The learning agents are trained with MADDPG algorithm for
several episodes, each consisting of many time steps, to derive effective
joint resource allocation. The details of the MADDPG algorithm are summarized
in Algorithm 1.

##  V PERFORMANCE ANALYSIS

###  V-A Simulation Setup

We start by uniformly distributing $M=100$ APs and $K=10$ active users over an
area of 1km2. The APs are connected to the CPU via ideal fronthaul links.
Unless stated otherwise, 30% of the entire APs are clustered to serve each
user, i.e., ${N_{k}}=0.3M$, $k=1,2,\cdots,10$. The system bandwidth is set to
5 MHz, which is shared among all users without channelization. A channel
coefficient of the small-scale fading between the $k$-th user and $m$-th AP is
set as ${{h}_{mk}}\sim\mathrm{}\left(0,1\right)$, which is independent across
all APs and users. Furthermore, a channel gain of large-scale fading between
the $k$-th user and $m$-th AP is given as

${\beta_{mk}}={10^{\frac{{P{L_{mk}}}}{{10}}}}{10^{\frac{{{\sigma_{sh}}{z_{mk}}}}{{10}}}},$
(22)

where ${\sigma_{sh}}$ represents the standard deviation of the shadow fading,
and ${z_{mk}}\sim{{\cal N}}\left({0,1}\right)$. According to the three-slope
model [38], with fixed distance of ${{d}_{0}}$, and ${{d}_{1}}$, the path loss
$P{{L}_{mk}}$ between the $k$-th user and $m$-th AP with a distance of
${d_{mk}}$ apart at the carrier frequency of $f$ is given as

$P{L_{mk}}=\begin{cases}-L-35{\log_{10}}\left({{d_{mk}}}\right)&\text{if
${d_{mk}}>d_{1}$}\\\
-L-10{\log_{10}}\left({d_{mk}^{2}\,d_{1}^{1.5}}\right)&\text{if
${d_{0}}<{d_{mk}}\leq{d_{1}}$}\\\
-L-10{\log_{10}}\left({d_{0}^{2}\,d_{1}^{1.5}}\right)&\text{if
${d_{mk}}\leq{d_{0}}$}\end{cases},$ (23)

where

$L=46.3+33.9\,{\log_{10}}\left(f\right)-13.82\,{\log_{10}}\left({{h_{AP}}}\right)\\\
-\left({1.1\,{{\log}_{10}}\left(f\right)-0.7}\right){h_{u}}+{1.56\,{{\log}_{10}}\left(f\right)-0.8}\,,$
(24)

with ${h_{AP}}$ and ${{h}_{u}}$ to denote the antenna height (in meters) of AP
and user, respectively. Further, there is no shadowing unless
${d_{mk}}>{d_{1}}$. The values of ${{d}_{0}}$, and ${{d}_{1}}$ are set to 10
m, and 50m, respectively, similar to [10]. The edge server at the CPU has
computation capacity of ${f^{CPU}}=100\,{\rm{GHz}}$, while all users are
equipped with the same computation capacity of $f_{k}^{\max}=1\,\text{GHz}$.
Furthermore, each bit requires 500 clock cycles to be processed, i.e.,
${{N}_{cpb}}=500$. A time step $\Delta t$ is set to 1ms, and each user
generates a task with a random size that is uniformly distributed in the range
of 2.5 to 7.5 kbits, corresponding to a rate of 2.5 Mbps to 7.5 Mbps, at the
beginning of every time step $t$. Without loss of generality, each user
demands the task to be processed within the same interval, i.e.,
$t_{k}^{d}\leq\Delta t=1ms$, for every time step. Meanwhile, we assume that a
channel does not change over each time step, i.e., ${\tau_{c}}=1ms$. The
simulation parameters are summarized in Table II.

Table II: Simulation parameters Notation |  Parameter |  Value
---|---|---
${f^{CPU}}$ |  Computation capacity of the server in the CPU |  $100\,\,{\rm{GHz}}$
$f_{k}^{\max}$ |  Maximum local computation capacity |  $1\,\,{\rm{GHz}}$
${N_{cpb}}$ |  Number of cycles to process one bit task |  $500\,{\rm{cycles}}/{\rm{bit}}$
$\varsigma$ |  Effective switched capacitance of the devices |  ${10^{-27}}$
$t_{k}^{d}$ |  Application deadline for user $k$ |  $1\,\,{\rm{msec}}$
$W$ |  System bandwidth |  $5\,\,{\rm{MHz}}$
$p_{k}^{\max}$ |  Maximum uplink transmit power |  $0.1\,\,{\rm{W}}$
$f$ |  Carrier frequency |  $1.9\,\,{\rm{GHz}}$
${h_{AP}}$ |  AP antenna height |  $15\,\,{\rm{m}}$
${h_{u}}$ |  User antenna height |  $1.65\,\,{\rm{m}}$
${\sigma_{sh}}$ |  Standard deviation of the shadow fading |  $10\,{\rm{dB}}$
${T_{o}}$ |  Noise temperature |  $290\,\,{\rm{K}}$
${\eta_{f}}$ |  Noise figure |  $9\,\,{\rm{dB}}$
${K_{B}}$ |  Boltzmann constant |  $1.381\times{10^{-23}}\,\rm{J/K}$
Table III: Simulation hyperparameters for MADDPG training Hyperparameter | Value
---|---
Discount factor $\varepsilon$ | 0.99
Soft update rate $\tau$ | 0.005
Critic learning rate | 0.001
Actor learning rate | 0.0001
Mini batch size | 128
Replay buffer size | 10000

For the proposed MADDPG-based JCCRA, both actor and critic networks of each
agent are implemented with fully connected neural networks, which consist of
three hidden layers, each with 128, 64, and 64 nodes, respectively. All the
hidden layers are activated by ReLu function, while the outputs of the actors
are activated by sigmoid function. The target networks at each agent are
copies of the respective actor and critic networks. The parameters of critic
and actor networks are updated with adaptive moment (Adam) optimizer with
learning rate 0.001 and 0.0001. Further, the discount factor and target
networks update parameter are set to ${{\gamma}_{k}}=0.99$ and $\tau=0.005$,
respectively, while the size of a mini batch is set to 128. The agents are
trained with MADDPG algorithm for several episodes, each consisting of 100
steps. Table III summarizes the hyperparameter values used in our simulations.

###  V-B Performance Evaluation

In this section, we present the performance of the proposed distributed
MADDPG-based JCCRA. For performance comparison, we consider the following
three benchmarks:

  * •

Centralized DDPG-based JCCRA scheme: This approach refers to DDPG-based
centralized resource allocation scheme at the CPU. We adopt the same neural
network structure and other hyperparameters as the MADDPG scheme to train the
actor and critic in the single DDPG agent. However, the agent has full
observation of the environment state, given in dimension of $K\times 3$,
during training and execution. The action has dimension of $K\times 2$, with
each row corresponding to joint resource allocation for a particular user.
Since global information of the entire network is processed centrally at the
CPU to make JCCRA decisions for all users, we can potentially obtain the most
efficient resource allocation. Therefore, it can potentially serve as a target
performance benchmark. However, as discussed in the previous sections, due to
the significant signaling and communication overhead, this scheme is
infeasible to support time-sensitive applications.

  * •

Offloading-first with an uplink fractional power control (FPC) scheme: This
approach preferably offloads the computation to the edge server with the aim
of aggressively exploiting the reliable access link provided by cell-free
massive MIMO while saving the local processing energy consumption. The uplink
transmit power for the $k$-th user is given by the standard fractional power
control (FPC) [38,39] as follows:

${p_{k}}=\min\left({p_{k}^{\max},{p_{0}}\lambda_{k}^{-\nu}}\right),$ (25)

where ${{p}_{0}}=-35\,\text{dBm}$,
${{\lambda}_{k}}=\sum\limits_{m=1}^{{{N}_{k}}}{{{\beta}_{mk}}}$, and
$\nu=0.5$.

  * •

Local execution-first with an uplink fractional power control (FPC) scheme:
The entire local processing capability is fully utilized, i.e.,
$f_{k}^{local}=f_{k}^{\max}$, and the remaining task bits are offloaded to the
edge server with uplink transmit power given according to (25).

In Fig. 5, we investigate the learning convergence of the proposed scheme by
evaluating the total reward achieved while comparing it against other
benchmark schemes. In contrast to the two heuristic approaches, it can clearly
be observed that the learning-based schemes, i.e., centralized DDPG and the
proposed MADDPG-based JCCRA, entertain higher reward values, which reflects
lower total energy consumption while meeting the respective application
deadlines of the users. The rewards per episode have smoothly increased in
both learning schemes until they finally converge. A closer look at the
convergence rate, however, reveals that the centralized scheme has converged
faster than the distributed variant. This is attributed to the advantage of
globally processing system wide information at the CPU to derive JCCRA policy
for all users. Nevertheless, as the training episodes increase, the proposed
algorithm has closed the gap and converged to the target benchmark, i.e., the
centralized counterpart. This implies that by relying on local observation and
additional information provided during the offline centralized training phase,
the agents manage to learn efficient joint resource allocation, which is also
adaptive with respect to the dynamics in the mobile edge network. It is also
evident from Fig. 5 that conservative use of local computation resources leads
to high energy consumption, while aggressively offloading results in
ineffective utilization of the limited communication and computing resources.

Figure 5: Total reward with training process Figure 6: Average success rate
with training process  Figure 7: Comparison of the proposed JCCRA approach
against variants with computing resource allocation optimization only

In Fig. 6, we compare the average success rate of the users in meeting the
delay constraint of the time-sensitive applications. For the proposed MADDPG-
based and centralized single-agent DDPG-based JCCRA schemes, the success rate
under the simulation setup has gradually increased with episode indices to
eventually reach more than 99%, showing only a marginal gap from _local
execution-first_ approach. The 100% success rate in the latter scheme is,
however, at the cost of unreasonably high energy consumption, which is 5 times
more compared to the learning-based schemes, as observed in Fig 5. Moreover,
the fact that the learning-based schemes outperform the conventional
_offloading-first_ approach implies that even if cell-free massive MIMO can
provide reliable access links to the users, aggressive computation offloading
can result in performance degradation due to the ineffective use of the finite
communication and computing resources, thereby subjecting users to service
outage due to the failure to attain the tight delay constraints. It is,
therefore, evident that an adaptive joint resource optimization is very
critical to fully unleash the potential of the available communication and
computing resources in the mobile edge network.

Figure 8: Per user average energy consumption at the testing stage

To further investigate a gain obtained by the joint resource allocation, we
then compare the proposed JCCRA scheme with two variants that adopt MADDPG
algorithm for optimizing computing resource allocation (CRA) only, i.e.
without taking power control into account. Additionally, we consider the
heuristic fractional power control mechanism given in (25) for the first
variant, referred to as MADDPG-CRA (FPC), while in the other variant, users
offload computation with the maximum transmit power of
${p_{k}}=p_{k}^{\max},\forall k\in{{\cal K}}$, referred to as MADDPG-CRA
(MAX). From Fig. 7, the performance degradation is readily observed in the
variant schemes, which is reflected by 33% and 42% rise in total energy
consumption of the user devices in MADDPG-CRA (FPC), and MADDPG-CRA (MAX),
respectively. This can be attributed to two factors. The first one is due to
the critical role of power control in handling co-channel interference among
different clusters during computation offloading. Thus, the adaptive MADDPG-
based power control gives the proposed approach an edge over the variant
schemes. The second factor is that computing and communication resource
management in multi-user mobile edge network are inherently inter-dependent.
Consequently, the proposed joint communication and computing resource
allocation (JCCRA) optimization approach in the proposed scheme yields
additional gain as compared with the separate optimization of computing
resources in the other two variants.

After demonstrating the learning convergence and performance matching to the
target centralized DDPG-based JCCRA scheme during the offline training stage,
we then test the performance of the proposed JCCRA scheme in the dynamic
mobile edge environment. Note that the execution phase is fully decentralized
in the sense that the agents perform inference based on their local
observations only, alleviating the burden associated with signaling in the
centralized scheme. Specifically, in accordance with the learned policy, each
MADDPG agent relies on its own actor network to map their local observation of
the environment into efficient joint resource allocation in real time. Fig. 8
presents the per-user average energy consumption of the schemes during the
execution stage. As it can be seen, the proposed approach achieves the lowest
energy consumption, which is roughly 2.5 and 5 times lower than the
_offloading-first with FPC_ and _local execution-first with FPC_ baseline
schemes, respectively. We can deduce that the offline centralized training has
successfully addressed the learning challenges of the multi-agent setting. In
other words, the training has enabled the agents to effectively capture the
complicated dynamics of the mobile edge network and perform well during the
fully decentralized testing phase, as shown in Fig. 8. Such a distributed
approach can be a useful means to handle time-sensitive applications while
reducing energy consumption in individual mobile device.

Figure 9: Effect of cluster size

Next, we investigate the effect of cluster size on the performance of the
proposed framework. As discussed in Section III.A, we form a user-centric
cluster of APs ${{{\cal C}}_{k}}$ to serve a given user by including
${{\rm{N}}_{k}}$ APs with the largest ${{\beta}_{mk}}$ (large-scale channel
fading). In Fig. 9, we evaluate the average energy consumption by the
different sizes of cluster, i.e., setting ${N_{k}}=0.1M,0.2M,0.3M,0.5M,M$ for
$M=100$. It can be observed that the case of ${N_{k}}=0.1M,\,\forall k\in{\cal
K}$ has the highest energy consumption since the cell-free access rate is
highly constrained to support computation offloading. The agents, therefore,
have to either predominantly depend on local computation or increase the
transmission power to meet the requirements of the multimedia applications,
thus incurring high energy cost. On the other hand, as ${{\rm{N}}_{k}}$
increases to $0.2M$ and $0.3M$, the performance drop becomes more and more
marginal compared to the canonical case where all APs serve every user, i.e.,
${N_{k}}=100$. In fact, the case of ${N_{k}}=50$ has negligible performance
drop in contrast to ${N_{k}}=100$. This indicates that with a sufficient
number of APs in a cluster, a user-centric cell-free massive MIMO can
effectively establish a reliable link for computation offloading with
negligible performance loss. It should be noted that the user-centric approach
requires a limited computational complexity for pilot detection and data
processing, in contrast to the canonical counterpart. In practice, therefore,
a legitimate cluster size must be set to deal with complexity and deployment
cost while solving the JCCRA problem.

(a) Average success rate with different number of users

(b) Per user average reward with different number of users

Figure 10: Effect of the number of users in the mobile edge network

The effect of the number of users on the performance of the proposed JCCRA
scheme is investigated in Fig. 10, fixing $M=100$ and ${N_{k}}=0.3M$. One can
observe that the proposed algorithm has converged in all cases. Since there
are abundant communication and computing resources for a small number of users
in the system, e.g., $K=5$, the agents entertain almost 100% success rate in
accomplishing the intensive tasks within the deadline, while collecting the
highest average reward, implying the lowest energy consumption compared to the
other two cases. However, as the number of users increases, the competition
among the users for limited communication and computing resources becomes more
severe. In other words, the per-user resource reduces with an increasing
number of users, e.g., the link rate decreases as $K$ increases. In
particular, the case $K=15$ represents a very constrained scenario that
suffers from a significant outage. Herein, the available resources are not
sufficient enough or the performance requirements are too strict to support
all the devices within the stringent delay constraints, subjecting some users
to service outage. This implies for critically constrained scenarios, either
relaxing the performance requirements, for instance with soft delay
constraints, or injection of more resources are required to support more users
without outage.

##  VI Conclusion and future works

In this paper, motivated by its capability of realizing a reliable access link
without cell edge, we presented a cell-free massive MIMO-enabled mobile edge
network to address the stringent requirements of the next generation advanced
services. We formulated a user-centric joint communication and computing
resource allocation (JCCRA) problem to minimize energy consumption of the user
devices while satisfying the corresponding delay constraints. We then proposed
a fully distributed solution approach based on cooperative multi-agent
reinforcement learning framework, wherein each user is implemented as a
learning agent to make joint resource allocation relying on local information
only. The simulation results demonstrate that our distributed approach matches
the performance of the target centralized DDPG-based benchmark, significantly
alleviating the signaling and communication overhead. Such a fully distributed
and adaptive framework can be a promising tool to realize edge intelligence
for supporting the envisaged applications in dynamic mobile edge network. In
the future, it would be interesting to extend this mobile edge network to
handle efficient and robust 6G-based sophisticated scenarios, e.g., user-
centric QoS/mobility management with AP clustering and location-based
optimization along with soft application delay consideration. In particular,
the current user-centric joint formulation can be extended further to
determine a set of APs that are adaptive to dynamic user mobility and channel
condition.

## References

  * [1] A. Al-Shuwaili and O. Simeone, “Energy-efficient resource allocation for mobile edge computing-based augmented reality applications,” _Wireless Commun. Lett._ , vol. 6, no. 3, pp. 398–401, Jun. 2017.
  * [2] Z. Q. Jaber and M. I. Younis, “Design and implementation of real time face recognition system (RTFRS),” _Int. J. Comput. Appl._ , vol. 94, no. 12, pp. 15–22, May 2014.
  * [3] K. Kumar and Y. Lu, “Cloud Computing for Mobile Users: Can Offloading Computation Save Energy?,” in _Computer_ , vol. 43, no. 4, pp. 51-56, April 2010.
  * [4] S. Barbarossa, S. Sardellitti and P. Di Lorenzo, “Communicating While Computing: Distributed mobile cloud computing over 5G heterogeneous networks,” in _IEEE Signal Processing Magazine_ , vol. 31, no. 6, pp. 45-55, Nov. 2014.
  * [5] A. U. R. Khan, M. Othman, S. A. Madani, and S. U. Khan, “A survey of mobile cloud computing application models,” _IEEE Commun. Surveys Tuts._ , vol. 16, no. 1, pp. 393–413, 1st Quart., 2014.
  * [6] Samsung Research, “6G: The Next Hyper–Connected Experience for All”, White Paper, July, 2020.
  * [7] N. Rajatheva _et al._ , “White paper on broadband connectivity in 6G” , arXiv preprint arXiv:2004.14247, 2020..
  * [8] H. Tataria _et al._ , “6G Wireless Systems: Vision, Requirements, challenges, Insights, and Opportunities”, arXiv preprint arXiv:2008.03213, 2020. .
  * [9] ITU–T, “Focus group on technologies for network 2030: Representative use cases and key network requirements,” February 2020.
  * [10] H. Q. Ngo, A. Ashikhmin, H. Yang, E. G. Larsson and T. L. Marzetta, ”
Cell-Free Massive MIMO Versus Small Cells,“ in _IEEE Transactions on Wireless
Communications_ , vol. 16, no. 3, pp. 1834-1850, March 2017.

  * [11] S. Buzzi and C. D’Andrea, ”
Cell-Free Massive MIMO: User-Centric Approach,“ in _Wireless Communications
Letters_ , vol. 6, no. 6, pp. 706-709, Dec. 2017.

  * [12] W. Fan, J. Zhang, E. Bjornson, S. Chen and Z. Zhong, ”
Performance Analysis of Cell-Free Massive MIMO Over Spatially Correlated
Fading Channels,“ ICC 2019 - 2019 _International Conference on Communications
(ICC)_ , Shanghai, China, 2019, pp. 1-6.

  * [13] E. Peltonen, M. Bennis, M. Capobianco, M. Debbah _et al._ , “6G White Paper on Edge Intelligence” , arXiv preprint arXiv:2004.14850, Apri 2020.
  * [14] J. Zhao, Q. Li, Y. Gong and K. Zhang, ”
Computation Offloading and Resource Allocation For Cloud Assisted Mobile Edge
Computing in Vehicular Networks,“ in _Transactions on Vehicular Technology_ ,
vol. 68, no. 8, pp. 7944-7956, Aug. 2019.

  * [15] Y. Mao, J. Zhang and K. B. Letaief, ”
Dynamic Computation Offloading for Mobile-Edge Computing With Energy
Harvesting Devices,“ in _Journal on Selected Areas in Communications_ , vol.
34, no. 12, pp. 3590-3605, Dec. 2016.

  * [16] Y. Wang, M. Sheng, X. Wang, L. Wang and J. Li, ”
Mobile-Edge Computing: Partial Computation Offloading Using Dynamic Voltage
Scaling,“ in _IEEE Transactions on Communications_ , vol. 64, no. 10, pp.
4268-4282, Oct. 2016.

  * [17] M. Sheng, Y. Wang, X. Wang and J. Li, ”
Energy-Efficient Multiuser Partial Computation Offloading With Collaboration
of Terminals, Radio Access Network, and Edge Server,“ in _IEEE Transactions on
Communications_ , vol. 68, no. 3, pp. 1524-1537, March 2020.

  * [18] X. Cao, F. Wang, J. Xu, R. Zhang and S. Cui, ”
Joint Computation and Communication Cooperation for Energy-Efficient Mobile
Edge Computing,“ in _IEEE Internet of Things Journal_ , vol. 6, no. 3, pp.
4188-4200, June 2019.

  * [19] J. Zhang _et al._ , ”
Energy-Latency Tradeoff for Energy-Aware Offloading in Mobile Edge Computing
Networks,“ in _IEEE Internet of Things Journal_ , vol. 5, no. 4, pp.
2633-2645, Aug. 2018.

  * [20] Y. Mao, J. Zhang, S. H. Song and K. B. Letaief, ”
Power-Delay Tradeoff in Multi-User Mobile-Edge Computing Systems,“ 2016 _IEEE
Global Communications Conference (GLOBECOM)_ , Washington, DC, 2016.

  * [21] Y. Mao, C. You, J. Zhang, K. Huang and K. B. Letaief, ”
A Survey on Mobile Edge Computing: The Communication Perspective,“ in _IEEE
Communications Surveys & Tutorials_, vol. 19, no. 4, pp. 2322-2358,
Fourthquarter 2017.

  * [22] X. Chen, L. Jiao, W. Li and X. Fu, ”
Efficient Multi-User Computation Offloading for Mobile-Edge Cloud Computing,“
in _IEEE/ACM Transactions on Networking_ , vol. 24, no. 5, pp. 2795-2808,
October 2016.

  * [23] Y. Zhang, X. Dong and Y. Zhao, ”
Decentralized Computation Offloading over Wireless-Powered Mobile-Edge
Computing Networks,“ 2020 _IEEE International Conference on Artificial
Intelligence and Information Systems (ICAIIS)_ , Dalian, China, 2020.

  * [24] J. Guo, H. Zhang, L. Yang, H. Ji and X. Li, ”
Decentralized Computation Offloading in Mobile Edge Computing Empowered Small-
Cell Networks,“ 2017 _IEEE Globecom Workshops (GC Wkshps)_ , Singapore, 2017,
pp. 1-6.

  * [25] X. Lyu, H. Tian, C. Sengul and P. Zhang, ”
Multiuser Joint Task Offloading and Resource Optimization in Proximate
Clouds,“ in _IEEE Transactions on Vehicular Technology_ , vol. 66, no. 4, pp.
3435-3447, April 2017.

  * [26] K. Jiang, H. Zhou, D. Li, X. Liu and S. Xu, ”
A Q-learning based Method for Energy-Efficient Computation Offloading in
Mobile Edge Computing,“ 2020 29th _International Conference on Computer
Communications and Networks (ICCCN)_ , Honolulu, HI, USA, 2020.

  * [27] J. Li, H. Gao, T. Lv and Y. Lu, ”
Deep reinforcement learning based computation offloading and resource
allocation for MEC,“ 2018 _IEEE Wireless Communications and Networking
Conference (WCNC)_ , Barcelona, 2018.

  * [28] Y. Dai, K. Zhang, S. Maharjan and Y. Zhang, ”
Edge Intelligence for Energy-Efficient Computation Offloading and Resource
Allocation in 5G Beyond,“ in _IEEE Transactions on Vehicular Technology_ ,
vol. 69, no. 10, pp. 12175-12186, Oct. 2020.

  * [29] L. Huang, X. Feng, C. Zhang, L. Qian, and Y. Wu, “Deep reinforcement learning-based joint task offloading and bandwidth allocation for multiuser mobile edge computing,” _Digit. Commun. Netw._ , vol. 5, no. 1, pp. 10–17, 2019.
  * [30] M. Min, L. Xiao, Y. Chen, P. Cheng, D. Wu and W. Zhuang, ”
Learning-Based Computation Offloading for IoT Devices With Energy Harvesting,“
in _IEEE Transactions on Vehicular Technology_ , vol. 68, no. 2, pp.
1930-1941, Feb. 2019.

  * [31] Z. Ning _et al._ , ”
Deep Reinforcement Learning for Intelligent Internet of Vehicles: An Energy-
Efficient Computational Offloading Scheme,“ in _IEEE Transactions on Cognitive
Communications and Networking_ , vol. 5, no. 4, pp. 1060-1072, Dec. 2019.

  * [32] Chen Z., Wang, X. “Decentralized computation offloading for multi-user mobile edge computing: a deep reinforcement learning approach”. _J Wireless Com Network_ 2020, 188 (2020).
  * [33] S. Mukherjee and J. Lee, ”
Edge Computing-Enabled Cell-Free Massive MIMO Systems,“ in _IEEE Transactions
on Wireless Communications_ , vol. 19, no. 4, pp. 2884-2899, April 2020.

  * [34] M. Ke, Z. Gao, Y. Wu, X. Gao and K. -K. Wong, ”
Massive Access in Cell-Free Massive MIMO-Based Internet of Things: Cloud
Computing and Edge Computing Paradigms,“ in _IEEE Journal on Selected Areas in
Communications_ , vol. 39, no. 3, pp. 756-772, March 2021.

  * [35] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” in Proc. _International Conference on Learning Representations (ICLR)_ , 2016.
  * [36] Laëtitia Matignon, Guillaume J. Laurent, Nadine Le Fort-Piat., Independent reinforcement learners in cooperative Markov games: a survey regarding coordination problems. _Knowledge Engineering Review, Cambridge University Press (CUP)_ , 2012, 27 (1), pp.1-31.
  * [37] R. Lowe et al., “Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments” , arXiv preprint arXiv:1706.02275, 2020.
  * [38] Ao Tang, JiXian Sun and Ke Gong, ”
Mobile propagation loss with a low base station antenna for NLOS street
microcells in urban area,“ _IEEE VTS 53rd Vehicular Technology Conference,
Spring 2001. Proceedings (Cat. No.01CH37202)_ , Rhodes, Greece, 2001, pp.
333-336 vol.1.

  * [39] R. Nikbakht, R. Mosayebi and A. Lozano, ”
Uplink Fractional Power Control and Downlink Power Allocation for Cell-Free
Networks,“ _IEEE Wireless Communications Letters_ , vol. 9, no. 6, pp.
774-777, June 2020.

  * [40] C. D’Andrea, A. Garcia-Rodriguez, G. Geraci, L. G. Giordano and S. Buzzi, ”
Cell-Free Massive MIMO for UAV Communications,“ _2019 IEEE International
Conference on Communications Workshops (ICC Workshops)_ , 2019, pp. 1-6.