# Joint Path planning and Power Allocation of a Cellular-Connected UAV using
Apprenticeship Learning via Deep Inverse Reinforcement Learning

Alireza Shamsoshoara<EMAIL_ADDRESS>Fatemeh Lotfi
<EMAIL_ADDRESS>Sajad Mousavi<EMAIL_ADDRESS>Fatemeh
Afghah<EMAIL_ADDRESS>İsmail Güvenç<EMAIL_ADDRESS>School of
Informatics, Computing and Cyber Systems at Northern Arizona University,
Flagstaff, AZ, USA.  Department of Electrical and Computer Engineering,
Clemson University, Clemson, SC, USA.  Harvard Medical School, Boston, MA,
USA.  North Carolina State University, Raleigh, NC, USA.

###### Abstract

This paper investigates an interference-aware joint path planning and power
allocation mechanism for a cellular-connected unmanned aerial vehicle (UAV) in
a sparse suburban environment. The UAV’s goal is to fly from an initial point
and reach a destination point by moving along the cells to guarantee the
required quality of service (QoS). In particular, the UAV aims to maximize its
uplink throughput and minimize the level of interference to the ground user
equipment (UEs) connected to the neighbor cellular BSs, considering the
shortest path and flight resource limitation. Expert knowledge is used to
experience the scenario and define the desired behavior for the sake of the
agent (i.e., UAV) training. To solve the problem, an apprenticeship learning
method is utilized via inverse reinforcement learning (IRL) based on both
Q-learning and deep reinforcement learning (DRL). The performance of this
method is compared to learning from a demonstration technique called
behavioral cloning (BC) using a supervised learning approach. Simulation and
numerical results show that the proposed approach can achieve expert-level
performance. We also demonstrate that, unlike the BC technique, the
performance of our proposed approach does not degrade in unseen situations.

###### keywords:

Apprenticeship learning, cellular-connected drones, inverse reinforcement
learning, path planning, UAV communication.

[xcolor] [caption] [hyperref]

##  1 Introduction

The fast growth of various unmanned aerial vehicle (UAV)- use cases and
applications recently attracted extensive research studies in both academia
and industry. Different application domains utilize drones in different ways,
such as cargo delivery, search and rescue (SAR), aerial imaging, disaster
monitoring, surveillance and public safety, drone-assisted communication,
among others [1, 2, 3, 4, 5, 6]. Unique features such as a wide ground field
of view (FoV), 3-dimensional movement, fast deployment and agile response make
the UAVs suitable for use in various applications. A recent report published
by the Federal Aviation Administration (FAA) predicts that the number of UASs
will be expected to rise from 448,000 in 2021 to 1.8 million in 2024 [7, 8].

In drone-assisted communications, the UAVs appear as aerial base stations
(BSs) or flying access points (APs) to serve the terrestrial User Equipment
(UEs) in disaster relief scenarios. In particular, they can be used to extend
the cellular coverage and improve the connectivity, or the UAVs relay the
information between the UEs and the neighbor BSs [9, 10, 11, 12, 13, 14].
While using drones as aerial BSs has been well studied, an emerging path
focuses on cellular-connected UAVs, called aerial UEs. In this case, the
terrestrial BSs can serve the UAVs to handle both the command and control
(C&C) communication and payload communication.

Drones can be utilized as aerial UEs in LTE systems as well as 5G and beyond
networks [15, 16]. However, noting the fast-growing number of unmanned aerial
systems (UASs), cellular-connected drones are expected to cause various
challenges to cellular systems including interference caused by UAVs to base
stations and terrestrial users, spectrum management, Quality-of-Service (QoS)
management, frequent handovers, robustness, security, reliability, etc [17,
18, 19, 20, 21]. Moreover, traditional cellular infrastructures such as
terrestrial BSs are designed to service terrestrial UEs in uplink (UL) and
downlink (DL) communications and several modifications need to be applied to
these systems to serve the aerial UEs [22, 23, 24, 25]. Furthermore, [26, 27]
recommend technical specifications to ensure reliable and efficient
communication for aerial UE in 5G systems. For instance, path planning,
beamforming, and initial cell selection are essential for enabling reliable
communication in 5G-connected drones. The use of massive multiple-input
multiple-output (mMIMO) technology and beamforming algorithms can improve
interference management by directing the signal towards the intended receiver
and minimizing interference with other users [28]. Furthermore, an AI-based
path planning model can be used at the UAVs to determine the optimal
trajectory, altitude, and speed taking into account factors such as the
received signal strength, coverage and level of interference.

In this paper, we develop an interference-aware joint path planning and power
allocation scheme for a cellular-connected UAV to minimize the interference it
causes to the existing ground UEs in adjacent cells. The UAV is set to start
its task from an initial point, and the goal is for it to reach a certain
destination point to finish its task. The UAV’s task can be considered as
remote-sensing or aerial monitoring, where the collected information should be
delivered to a fusion center using a UL communication with the desired
terrestrial BS. The UAV’s goal is to maximize its UL throughput and signal-to-
noise ratio (SNR) while this may increase the interference on the terrestrial
UEs as well. The UAV’s operation field is assumed to be covered by non-
overlapping cells with different UE densities, which means some cells have a
higher UE density. The UAV’s flight path from the source to the destination
affects both the UAV’s UL throughput and the terrestrial UEs’ interference.

The problem formulation in this study accounts for several factors including
the UAV’s transmission power, path planning considering the terrestrial UEs
density, the imposed interference to the terrestrial UEs, and the UAV’s
performance in terms of task completion. The performance of common
reinforcement learning (RL) methods to equip the UAV with autonomous decision-
making capabilities highly depends on the definition of the reward function.
Defining a reward function that takes into account all the aforementioned
factors is not intuitive. As a result, we proposed a novel approach based on
apprenticeship learning via inverse reinforcement learning (IRL) in this
paper, employing both Q-learning and deep Q-networks (DQN). We assume that an
expert is available to generate some optimal datasets based on the simulator
and that these datasets can be used to reconstruct the optimal reward
function. The obtained reward function is then applied to find the optimal
policy. To the best of our knowledge, this work is one of the leading and
primary works that address the joint path planning and power allocation
problem in a cellular-connected UAV using apprenticeship learning and IRL. The
contributions of this study are as follows:

  * 1.

Formulating the problem of joint path planning and power allocation to
minimize the UAV’s UL transmission interference to the neighbor and adjacent
terrestrial UEs.

  * 2.

Designing an open-source simulator to collect experts’ information and
trajectory based on optimal behavior. The simulator is entirely object-
oriented, with different classes for BSs, UEs, and UAVs. The user can change
the number of cells, the density of UEs, etc. A graphical user interface is
developed to show the drone’s interaction with the simulation environment.

  * 3.

Devising an apprenticeship learning approach using both IRL and deep IRL to
analyze the optimal behavior obtained by the expert’s behavior and obtain the
optimal reward function. This optimal reward function can generate the optimal
policy based on the user’s defined threshold.

  * 4.

Saving more time during the training process by using approaches such as
imitation learning and apprenticeship learning which help the agent to reduce
the visiting states since the optimal behavior can guide the agent on which
states are necessary to visit.

  * 5.

Comparing the performance of the proposed apprenticeship learning via IRL
based on Q-learning and Deep Q-network with other alternative approaches such
as behavioral cloning, shortest path, and random path for various scenarios
including the situations where an error happens or the agent visits a state
due to environmental conditions such as wind and that state has never seen
before by the expert.

The rest of this paper is organized as follows. Section 1.1 reviews the
related works regarding the cellular-connected UAV for similar problems and
approaches. The problem definition and the system model are discussed in
Section 2 with all related assumptions. Section 3 introduces the
apprenticeship learning via IRL using both Q-learning with linear function
approximation and Deep Q-Network. Section 4 evaluates the convergence and
other metrics and parameters using our designed simulator. Conclusions and
discussions are summarized in Section 5.

###  1.1 Related works

This section reviews previous works regarding cellular-connected UAVs by
considering different models such as interference management, power
allocation, task allocation, path planning, etc. Different centralized and
distributed methodologies were utilized to solve these challenges.

The authors in recent studies such as [29] considered a scenario where a
cellular-connected UAV aims to finish its task and be covered by the ground
cellular network; however, there are coverage gaps in the area because of
antennas and infrastructure limitations such that the UAV cannot be
continuously connected to the BSs. The UAV’s goal was to avoid those areas
during the flight to minimize the timeout probability ($P_{out}$). The authors
performed the UAV trajectory/navigation optimization using a dueling double
deep Q-network and RL as a model-free approach to solving this problem. The
approach was considered for simultaneous navigation and radio mapping to
enhance the UAV’s aerial coverage while it is flying toward its destination.

In another study [30], the authors considered an inter-cell interference and
power coordination scenario for a cellular-connected UAV. First, the authors
tried to solve the problem of inter-interference minimization using a
centralized approach. The authors assumed that a central point exists to
collect global information for the optimization problem. The challenge is
defined as a non-convex optimization problem. Hence, the authors used a
successive convex approximation to solve the problem. They obtained a sub-
optimal solution for this approach. Later, the authors proposed a
decentralized approach by only using local information.

In a similar study [31], the authors defined a path planning problem for a
cellular-connected UAV to minimize its mission completion time and maximize
the connectivity probability to the cellular BSs. The problem is defined in a
Markov decision processing (MDP) environment, and they used a temporal
difference approach, as an RL, to find the optimal policy for path planning
optimization.

Most authors in this domain used a model-free approach such as the temporal
difference, as an RL, to solve these kinds of interference and path planning
optimization problems. However, problems like the one formulated in this paper
are hard to model for dynamic environments, and even if modeled, the problem
may not be convex, resulting in a suboptimal solution. Also, solving such non-
convex problems needs additional resources and time to be handled. RL usually
handles these complex problems. Several studies such as [32, 33, 34, 35] used
different types of RL approaches such as distributed Q-learning, team
Q-learning, multi-agents Q-learning to tackle the problem of relaying and
delivering information for terrestrial UEs in disaster scenarios. One of the
critical steps in developing a good RL solution is defining the reward
function. The reward function maps a relation between the MDP environment,
Q-action values, and optimal policy learning. As the problem becomes more
complex, defining a meaningful reward that considers all aspects of the
problem may not be trivial. In that case, the reward function is less likely
to be optimal, and as a result, the obtained policy may not be optimal.
Moreover, since most of the evaluated methods are implemented based on a
single reward function, the comparison with other methods is less meaningful.
In addition, in complex problems in a real-world scenario, it is not always
possible to define a reward function that can capture all features of the
problem.

With the rise of open-source software, simulators, and emulators such as
network simulator-3 (NS3) [36], srsLTE [37], OpenAirInterface [38], and other
implemented simulators, it is possible to ask for experts to generate an
optimal behavior for the agent in a similar simulated environment. These
generated data and trajectories by the expert can be used later for the agent
to learn that behavior. We like to note that in this study, the trajectory
refers to the sets of states and actions in the MDP environment. The expert
data can be used to define the optimal behavior, policy, or reward function.
This approach is called imitation learning (IL), learning from demonstration
(LfD), or apprenticeship learning (AL). The authors of [39] considered a
disaster relief scenario where a UAV helps the terrestrial UEs to schedule a
packet delivery to the neighbor BSs using an IL solution. The authors
implemented a supervised learning solution as behavioral cloning (BC) approach
to mimic the expert’s behavior in a similar situation. BC usually does not
consider any reward functions and it is only based on visited states and
actions where the agent blindly follows the expert. It means that if the
expert made any wrong decisions, then the agent may make the same error.
Furthermore, if the visited state by the agent has never been experienced by
the expert, then the agent cannot make a proper action since that state was
not considered in the optimal learned policy. These challenges motivate us to
utilize an approach called apprenticeship learning via IRL [40].

###  1.2 Table of Notations

Table 1 shows all the parameters and notations used in this study in both the
system model and methodology.

Table 1: Notation descriptions

Notation | Description | Notation | Description
---|---|---|---
$\mathcal{U}$ | Set of all ground users (UEs) | $\boldsymbol{\varpi}$ | Path association vector
$\mathcal{U}_{M}$ | Set of all ground users associated to the main BSs | $\boldsymbol{\varrho}$ | Cell association vector
$\mathcal{U}_{N}$ | Set of all UEs associated to the neighbor BSs | $\bar{T}$ | Throughput threshold
$\mathcal{B}_{M}$ | Set of main BSs | $S$ | State vector in the RL approach
$\mathcal{B}_{N}$ | Set of neighbor BSs | $A$ | Action vector in the RL approach
$B$ | Bandwidth | $P_{r}$ | Transition probability
$u_{l}=(x_{l},y_{l},z_{l})$ | UAV’s longitude, latitude, and altitude | $\gamma$ | Discount factor
$V$ | UAV’s pitch speed | $R(s)$ | Reward function
$\boldsymbol{p}$ | UAV’s taken path based on locations, i.e., $l_{i}$ | $w$ | Reward function’s weights
$\mathcal{L}$ | Set of all locations s.t. $l_{i}\in\mathcal{L}$ | $\phi(s)$ | Features vector
$d_{a,b}$ | Distance between locations $a$ and $b$ | $\pi(S)$ | Next action policy
$\varsigma_{b,l}$ | SNR received at BS $b$ at location $l$ | $V^{\pi}(s)$ | Policy value at state $s$
$f$ | Center frequency | $\mu(s)$ | Feature expectation value
$\Phi_{b_{m},l}$ | Received throughput at BS $b_{m}$ | $D$ | Hyper distance
$P_{b_{m},l}$ | UAV’s transmission power | $Q(s,a)$ | Q-Action value at state $s$
$h_{b_{m},l}$ | Channel gain between the UAV and the BS $b_{m}$ | $\theta$ | Stochastic Gradient Descent weights
$G_{b_{m},l}$ | Antennas gain between the UAV and BS $b_{m}$ | $MSE$ | Mean Squared Error
$N_{0}$ | Noise at the receiver | $\tau$ | Tuple of states-actions trajectories

##  2 System Model

Figure 1: A sample system model showing two different possible paths of the
UAV from the source to destination.

Consider a single cellular-connected UAV equipped with different payloads such
as sensors, GPS, and cameras. The equipment collects data and sends it to a
fusion center for further processing. We consider an orthogonal frequency-
division multiple access (OFDMA) scheme for the wireless communication link,
which divides the total cell bandwidth $B$ into $K$ resource blocks (RBs). The
UAV utilizes an uplink (UL) to transfer its data to the ground base station
(BS). Fig. 1 depicts the system model. As seen in this figure, the UAV’s
operation field is divided into multiple non-overlapping hexagonal cells, each
served by one BS which services several terrestrial UE with random movement.
The UAV can fly over these cells to reach its destination. The BSs are
categorized into two groups: 1) the main network to which the UAV is linked,
and the BS provides the required bandwidth for the UAV’s UL. UAV’s
transmission does not interfere with the communication of other terrestrial
UEs in the same cell, 2) the neighbor BSs that the UAV is not communicating
with them. However, the UAV’s transmission causes interference to the neighbor
BSs and their UEs when the UAV uses the same RB with them. We assumed that the
neighbors’ UEs cannot cause interference to the UAV’s transmission since the
transmission power is not strong enough. Hence, the number of terrestrial UEs
in neighbor cells is critical because they are affected by the UAV’s
interference. Here, we consider the UAV’s interference value on individual UEs
in our design. The set $\mathcal{U}$ of U ground UEs consists of two groups of
$\mathcal{U}_{\rm M}\subseteq\mathcal{U}$ and $\mathcal{U}_{\rm
N}\subseteq\mathcal{U}$ which are the sets of UEs in the main and neighbor
networks, accordingly.

The base stations in the main and neighbor networks are shown as
$b_{m}\in\mathcal{B}_{\rm M}$ and $b_{n}\in\mathcal{B}_{\rm N}$, respectively.
The UAV has a predefined initial source and destination. Its task is to reach
the destination while keeping its communication with a legitimate base
station, $b_{m}$. The UAV and other base stations are defined in a hexagon
grid area where the ground BSs are fixed, and the UAV can move in six
directions (North, North East, South East, South, South West, North West).
While near the zone’s edge, the UAV cannot fly off the grid. For the sake of
simplicity, the BS is assumed to be in the center of each hexagon. The UAV’s
location is known based on the on-board GPS sensor,
$u_{l}=(x_{l},y_{l},z_{l})$. The UAV changes its location with a constant
pitch speed velocity, $V$, $0<V\leq\bar{V}$, where $\bar{V}$ is the maximum
available pitch speed for the UAV. Also, it is assumed that the UAV changes
its direction using the yaw action not rolling. The UAV’s path from the source
to destination is shown as $\boldsymbol{p}=(l_{\rm 1},l_{\rm 2},\dots,l_{\rm
f})$, which the first location is the source cell, $l_{\rm 1}=S$ and the last
one is the destination cell, $l_{f}=D$. All locations are subsets of the
global location set $\mathcal{L}$, $(l_{\rm 1},l_{\rm 2},\dots,l_{\rm
L})\in\mathcal{L}$, where $L$ is the grid size or the number of hexagons. The
UAV’s path length is not fixed, but if it cannot reach the destination in a
certain number of movements, the UAV’s operation may fail due to battery
limitations. Based on the literature [41], the grid area is assumed small
enough such that the UAV’s location can be assumed constant inside each
hexagon.

###  2.1 Wireless Communication Model

Buildings and obstructions have an impact on signal propagation from the air
to the ground. Depending on the environment, this connection may be a line of
sight (LoS) or a non-line of sight (NLoS) link. Accordingly, the path loss of
LoS and NLoS links of the UAV at location $l$ and BS $b_{m}$ from the primary
network on a sub-6 GHz band can be described as follows [42]:

$\displaystyle\varsigma_{b_{m},l}^{\rm LoS}(dB)=20\log_{\rm
10}(d_{b_{m},l})+20\log_{\rm 10}(f)-147.55+\eta_{\rm LoS},$ (1)
$\displaystyle\varsigma_{b_{m},l}^{\rm NLoS}(dB)=20\log_{\rm
10}(d_{b_{m},l})+20\log_{\rm 10}(f)-147.55+\eta_{\rm NLoS},$ (2)

where $l$ is the UAV’s location or the center of the cell in which the UAV is
currently located. $f$ is the center frequency and $d_{b_{m},l}$ is the
distance between BS $b_{m}$ and the UAV located at location $l$. Also,
$\eta_{\rm LoS}$ and $\eta_{\rm NLoS}$ are LoS and NLoS links attenuation
factors, respectively. The probability of having LoS link depends on
environmental factors. In our model, for UAV to the ground link, the LoS
probability is given by the proposed model in [42]:

$\displaystyle\mathcal{P}_{\rm
LoS}(d_{b_{m},l})=\frac{1}{1+c_{1}\exp{\bigg{(}-c_{2}\bigg{(}\frac{180}{\pi}\tan^{-1}(\frac{H}{d_{b_{m},l}})-c_{1}\bigg{)}\bigg{)}}},$
(3)

where $H$ represents the UAV height, and $c_{1}$ and $c_{2}$ are environmental
dependent permanents. Accordingly, the total path loss of UAV at location $l$
and ground BS $b_{m}$ can be defined as

$\displaystyle\varsigma_{b_{m},l}=\mathcal{P}_{\rm
LoS}\,\,\varsigma_{b_{m},l}^{\rm LoS}+\big{(}1-\mathcal{P}_{\rm
LoS}\big{)}\,\,\varsigma_{b_{m},l}^{\rm NLoS}.$ (4)

The SNR for the UAV UL data transmission to the main BS $b_{m}$ is written as
$\Phi_{b_{m},l}$:

$\displaystyle\Phi_{b_{m},l,k}=\frac{P_{b_{m},l,k}h_{b_{m},l,k}}{N_{0}},$ (5)

where $P_{b_{m},l,k}$ is the transmission power that the UAV allocates to each
RB and sends its data to BS $b_{m}$ at location $l$.
$h_{b_{m},l,k}=G_{b_{m},l,k}10^{-\varsigma_{b_{m},l}/10}$ indicates the
channel gain between the UAV and the main BS $b_{m}$ based on the path loss
defined in 4, where $G_{b_{m},l,k}$ denotes the antenna gain values of the UAV
and the BS ($b_{m}$). While we assumed a simplified model in this study to
calculate the gain value, in the real-world scenario, the antennas of the BSs
are down-tilted to have better coverage for the ground UEs and reduce the
effect of inter-cell interference. Hence, the UAV is usually served by the
side-lobe antenna of the ground BS. $N_{\rm 0}$ is the noise power spectral
density at the main receiver BS $b_{m}$. Based on the SNR definition in (5),
and by considering $B_{k}$ as the bandwidth of RB $k$, the throughput rate for
the UAV at location $l$ and based on the connected BS $b_{m}$ can be written
as:

$\displaystyle T_{b_{m},l}=\sum_{k=0}^{K}B_{k}\log_{\rm
2}(1+\Phi_{b_{m},l,k}).$ (6)

The UAV may interfere with the terrestrial UEs transmission in the neighbor
networks. This interference depends on the UAV’s transmission power, quality
of UAV’s channel to the neighbor BS $b_{n}$ and the probability of allocating
the same resource block in the UAV and terrestrial UEs. Here, due to minimum
knowledge sharing the UAV is blind to the knowledge of UEs in the neighbor
networks. Therefore, we assume that there is at least one UE in neighbor cells
that is assigned with the same resource block. Accordingly, the UAV’s
interference in uplink of a UE $u\in\mathcal{U}_{\rm N}\subseteq\mathcal{U}$
at BS $b_{n}$, and by considering $\mathcal{K}_{u}$ as a set of allocated RBs
to UE $u$, will be [35]:

$\displaystyle I_{b_{n},l}^{u}=\sum\limits_{m=1}^{\mathcal{B}_{\rm
M}}\sum\limits_{k=0}^{|\mathcal{K}_{u}|}P_{b_{m},l,k}h_{b_{n},l,k},$ (7)

where $P_{b_{m},l,k}$ is the UAV’s transmission power in RB
$k\in\mathcal{K}_{u}$ when it was transmitting to the main BS
$b_{m}\in\mathcal{B}_{\rm M}$ at location $l$ on the same RBs of UE $u$. Also,
$h_{b_{n},l,k}$ indicates the channel gain between the UAV and the neighbor BS
$b_{n}$. This interference rate is an argument that affects the terrestrial UE
$u$ SINR and throughput rate:

$\displaystyle\Phi_{b_{n}}^{u}=\frac{{P_{b_{n}}^{u}h_{b_{n}}^{u}}}{I_{b_{n},l}^{u}+N_{\rm
0}},$ (8) $\displaystyle T_{b_{n}}^{u}=B\log_{2}(1+\Phi_{b_{n}}^{u}),$ (9)

where $P_{b_{n}}^{u}$ is the transmission power for UE $u$, for the sake of
simplicity, all UEs’ transmission powers are assumed to be constant. Also,
$h_{b_{n}}^{u}$ indicates the channel gain between the UE $u$ and the BS
$b_{n}$.

###  2.2 Objective Definition

Based on the defined system model, the UAV’s objective is to fly from the
source point and reach the destination point based on the BSs’ locations and
the density of the UEs in neighbor cells to minimize the interference level it
causes on the neighbor terrestrial UEs, maximize its UL throughput by finding
an optimal transmission power value, and the shortest path between the source
and destination.

The path association vector based on the hexagons is shown as
$\boldsymbol{\varpi}$ that consists of elements $\varpi_{a,b}\in\\{0,1\\}$.
The value 1 means that the UAV has moved from location $a$ to $b$ and 0 means
otherwise. The UAV has to find the optimized path association vector from the
source to destination $\boldsymbol{\varpi^{*}}$. To meet these requirements,
the UAV has to find the optimal transmission power $P_{b_{m},l}^{*}$ at each
location $l$ where the UAV is associated with tower $b_{m}$ which is the main
base station. The optimal transmission power $P_{b_{m},l}^{*}$ for
transmission to base station $b_{m}$ can be chosen from the transmission power
range $P_{b_{m},l}^{*}\in[P_{\rm min},P_{\rm max}]$ which includes a vector of
discrete values. $P_{b_{m},l}^{*}$ should meet the satisfactory throughput
rate $T_{b_{m},l}>\bar{T}$, where $\bar{T}$ is the minimum acceptable
throughput. The optimal power vector during the UAV’s flight is shown as
$\boldsymbol{P^{*}}$. To find $P_{b_{m},l}^{*}$, the UAV has to find its BS-
connectivity association vector. This connectivity vector,
$\varrho_{b_{m},l}\in\boldsymbol{\varrho}$, shows whether the UAV connects to
BS $b_{m}$ at location $l$ or not. The values for the connectivity vector are
$\\{\rm 0,1\\}$.

The UAV has to find the best BS connectivity vector $\boldsymbol{\varrho^{*}}$
based on other optimization variables in this problem. Since the UAV does not
cause any interference on the UEs in its current cell and it can interfere
with other neighbor UEs in neighbor cells, the optimal variables such as
$\boldsymbol{\varpi^{*}}$, $\boldsymbol{P^{*}}$, and
$\boldsymbol{\varrho^{*}}$ are chosen to minimize the interference to
neighboring terrestrial UEs. In a nutshell, this optimization problem can be
written as:

$\displaystyle\min_{\boldsymbol{P},\boldsymbol{\varpi},\boldsymbol{\varrho}}\hskip
14.22636pt$
$\displaystyle\alpha\,\sum_{l=l_{1}}^{l_{L}}\sum_{n=1}^{|\mathcal{B}_{\rm
N}|}I_{b_{n},l}^{u}+\beta\sum_{a=l_{1}}^{l_{L}}\sum_{b=l_{1}}^{l_{L}}\varpi_{a,b},$
$\displaystyle+\delta\sum_{l=l_{1}}^{l_{L}}\sum_{m=1}^{|\mathcal{B}_{\rm
M}|}\varrho_{b_{m},l},$ (10a) s.t., $\displaystyle\hskip 8.5359ptP_{\rm
min}\leq P_{b_{m},l}\leq P_{\rm max},\,\,b_{m}\in\mathcal{B}_{\rm M},$ (10b)
$\displaystyle\hskip
8.5359pt\varpi_{a,b}\in\\{0,1\\},\,\,\varrho_{b_{m},l}\in\\{0,1\\},\,\,l,a,b\in\mathcal{L},$
(10c) $\displaystyle\hskip
8.5359pt\sum_{l=l_{1}}^{l_{L}}T_{b_{m},l}\geq\varrho_{b_{m},l}\bar{T},$ (10d)
$\displaystyle\hskip 8.5359pt\sum_{m=1}^{|\mathcal{B}_{\rm
M}|}\varrho_{b_{m},l}-\sum_{b=l_{1},b\neq a}^{l_{L}}\varpi_{b,a}=0,$ (10e)
$\displaystyle\hskip 8.5359pt\sum_{a=l_{1}}^{l_{L}}\varpi_{S,a}=1,\quad\quad
a\neq S,$ (10f) $\displaystyle\hskip
8.5359pt\sum_{b=l_{1}}^{l_{L}}\varpi_{b,D}=1,\quad\quad b\neq D,\quad b\neq
S,$ (10g) $\displaystyle\hskip 8.5359pt\sum_{a=l_{1},a\neq
b}^{l_{L}}\varpi_{a,b}-\sum_{k=l_{1},k\neq b}^{l_{L}}\varpi_{b,k}=0,$ (10h)
$\displaystyle\hskip 8.5359pt\sum_{a=l_{1},a\neq b}^{l_{L}}\varpi_{a,b}\leq
1.$ (10i)

The optimization problem in (10a) aims to minimize the interference level on
neighbor network UEs and BSs, also it minimizes the UAV’s path length from the
source to the destination and selects the main BSs for the communication which
imposes less interference to neighbor networks. Equations (10b) and (10c) are
the optimization problem constraints regarding the UAV’s transmission power,
the UAV’s path between different hexagon cells, and the UAV-BS connectivity
vector. Equation (10d) guarantees that the throughput rate for the UAV-BS UL
at each location $l$ when it is associated with BS $b_{m}$ is greater than a
predefined threshold $\bar{T}$, equation (10e) guarantees that the UAV is only
connected to one main BS $b_{m}$ at each location $l$ on its path to the
destination. Equation (10f) claims that the UAV’s path starts with the source
point $S$ and the UAV only visits the source one time, (10g) claims that each
UAV’s path is ended at the destination $D$, (10h) guarantees that if the UAV
visits any midpoint such as $b$ between the source $S$ and destination $D$,
then it will leave that point and fly to another cell. (10i) claims that each
hexagon $b$ on the UAV’s path is going to be visited at most one time.

Fig. 1 shows a sample scenario of this system model. The UAV chooses two paths
from the source to the destination, the green one is the optimal path
considering the density of the terrestrial UEs since the neighbor cells have
lower densities and the UAV’s UL communication has less interference on the
terrestrial UEs. We should note that the UAV can allocate an optimal value for
the transmission power in each cell to control the trade-off between the
throughput satisfaction threshold and the level of interference. On the other
hand, the orange path shows a route in which the neighbor cells carry a larger
density regarding the number of UEs, and as a result, the UAV faces a more
strict restriction on its transmission power.

This optimization problem is categorized as a mixed integer non-linear one
which is hard to solve. This study aims to consider an autonomous approach as
a solution for the UAV to find the optimal values for the UAV’s transmission
power, path, and cellular connectivity vector. This multi-parameter problem
involves a huge state-space set which is overwhelming to solve and find the
optimal values in a reasonable amount of time. To investigate the possible
solution for this problem, we utilize an apprenticeship learning approach and
use the expert’s knowledge to expedite the solving problem and have a better
understanding of the optimal reward function.

##  3 UAV’s Self-Organizing Approach

(a) Reinforcement learning approach

(b) Apprenticeship learning using Inverse RL

Figure 2: Difference between reinforcement learning and Inverse RL for
apprenticeship learning (SARSA: State–action–reward–state–action, SVM: support
vector machine).

Many artificial intelligence and robotic applications consist of autonomous
agents such as ground robots, UAVs, and self-driving cars. These agents
usually operate in an uncertain and complex environment in which the decision-
making task is cumbersome and necessary.

In multi-objective reinforcement learning problems, where the agent deals with
various and possibly conflicting objective functions to perform tasks of
different natures, it is not always possible to define a reward function that
optimally captures all the objectives ((10a) - (10i)). The problem studied in
this paper also involves multiple actions such as the movement and level of
transmission power with different natures and behavior. This problem makes it
more intense for the agent to have a meaningful understanding of the reward
function. Usually, in these situations, an expert who has enough experience in
the domain and field can demonstrate appropriate behavior to the agent. These
demonstrations may save some training time while also assisting the agent in
determining the best reward function based on the most optimal presented
behavior. This type of learning is referred to as Learning from Demonstration
(LfD) or Programming by Demonstration (PbD), and it is also known as imitation
learning (IL). IL techniques are categorized into two branches, i) behavioral
cloning (BC) [43] and ii) apprenticeship learning via inverse reinforcement
learning (IRL) [44, 45].

In BC, the agent blindly mimics the expert or teacher without considering the
expert’s optimality which means if the expert makes a wrong decision, the
expert may follow that. Usually, IL problems are solved using supervised
learning approaches. However, if the observed states by the agent have not
been experienced by the expert, the learned policy may not achieve an optimal
outcome. This issue can be solved by an approach called Data Aggregation
(DAGGER) [46]. DAGGER aims to collect expert demonstrations based on the state
distribution observed by the agent. In this case, the expert should be
available at all times, which is not possible in all scenarios and
applications.

On the other hand, apprenticeship learning via IRL tries to find the expert’s
hidden desire (reward function) from the demonstrations to find the optimal
policy [47, 40]. This paper utilizes apprenticeship learning via IRL to solve
the problem proposed in Section 2. Similar approaches such as BC, Q-Learning,
and Deep Q-Network are also implemented to evaluate the numerical performance.

###  3.1 Apprenticeship Learning via Inverse Reinforcement Learning

The problem definition of this study can be shown as a sequential decision-
making process that can be posed in the (finite-state) Markov decision process
(MDP). Typically, the reward function is provided in MDP problems, allowing
the agent to determine the optimal transition probabilities, value function,
and optimal policy. The agent can utilize the optimal policy for decision-
making in every state. However, the assumption that the reward function is
present in every complex system model is overly optimistic, and manually
obtaining the reward function is challenging.

The existence of an expert may make it possible to have a set of trajectories
and demonstrations to learn the optimal behavior. The expert could be a
virtual agent that operates on a high-performance computing machine but cannot
be utilized onboard a UAV. In this study, the term “trajectory” refers to a
set of states and actions rather than the UAV’s trajectory movement or path.
The IRL aims to re-construct the unknown reward function $R(\tau)$ from the
expert demonstrations. Later, this obtained reward function can be used in an
RL approach to find the optimal policy. But, this problem is “ill-posed” since
there exist multiple reward functions for a unique and optimal policy.

Here, we assumed that the expert is trying to optimize an unknown reward
function which this unknown function can be represented as a linear mixture of
known “features”. Fig. 2 shows the main difference between a reinforcement
learning problem and an apprenticeship learning via IRL (AL-IRL) problem. The
reward function is available in Fig. 2(a), and based on that the optimal
policy is gained. However, in Fig. 2(b), the reward function is found based on
the IRL and the expert data, and then the optimal policy is obtained. Expert
trajectories are not the same as optimal policies; however, they describe
optimal behavior.

We assumed that the problem is MDP\R which shows the MDP problem without a
reward function. This MDP\R consists of a tuple of
$\langle\mathcal{S},\mathcal{A},Pr,\gamma,T\rangle$, where $\mathcal{S}$,
$\mathcal{A}$, and $Pr$ represent the state space, action space, and
transition probability from current state $s\in\mathcal{S}$ to the next state
$s^{\prime}\in\mathcal{S}$, respectively. Also, $\gamma$ is a discount factor,
and $T$ is a set of trajectories demonstrated by an expert. The MDP\R tuples
are described as follows:
1) State

To define the state $s\in\mathcal{S}$, we assume that a vector of features
$\boldsymbol{\phi}$ : s $\rightarrow$ [0, 1]k, where $k$ is the number of
features. $\mathcal{S}$ represents a finite set of feature states,
$\mathcal{S}=\\{\phi_{i}|1\leq i\leq 5\\}$. Here, each state $s\in\mathcal{S}$
is displayed with five feature functions.

  * 1.

$\phi_{1}(s)$ is the feature that keeps track of the UAV’s distance to the
destination. This feature helps the agent to have a better understanding of
the shortest path.

  * 2.

$\phi_{2}(s)$ is the feature that keeps track of the UAV’s hop counts as its
traveled distance. This feature helps the agent to perform its task in a
limited number of hop counts because of UAV’s battery life constraint.

  * 3.

$\phi_{3}(s)$ keeps track of the UAV’s successful task, where the term
”successful” refers to the cases in which the UAV reaches the destination
cell. $\phi_{3}(s)$ behaves like a binary variable, where it takes the value
of ”0” if the UAV doesn’t reach the destination and ”1” for reaching the
destination cell.

  * 4.

$\phi_{4}(s)$ is a feature for the amount of received throughput from the UAV
to the designated BS based on the allocated transmission power and SNR defined
in (5) and (6) at each state.

  * 5.

$\phi_{5}(s)$ keeps the value for the amount of interference imposed on the
neighbor UEs located in the neighbor cells.

All feature values are normalized to be in the range of [0, 1]. This condition
on the reward function is necessary for the IRL algorithm to converge. Hence,
each state can be expressed based on the feature value functions as,

$\displaystyle\boldsymbol{\phi}(s_{t})=\\{\phi_{i}(s_{t}),1\leq i\leq 5\\}.$
(11)

2) Action

$\mathcal{A}$ represents the action space which consists of the UAV’s movement
action and the transmission power allocation $\boldsymbol{P}$. The UAV’s
movement action somehow includes the defined movement variables
$\boldsymbol{\varpi},\boldsymbol{\varrho}$ in (10a). The UAV chooses both
actions autonomously at the same time.
3) Reward

Here, the unknown reward function is defined by considering the feature
functions in the state space. The reward function is defined as:

$\displaystyle R(s)=\boldsymbol{w}^{*}\ .\ \boldsymbol{\phi}(s),$ (12)

where, $\boldsymbol{w}^{*}$ are the optimal weights for the features to define
the desirable optimal reward function, $\boldsymbol{w}\in\mathbb{R}^{k}$. To
have the convergence for the IRL, it is necessary to bound the reward function
by $-1$ and $1$. Therefore, the weights are bounded as well
$||\boldsymbol{w}^{*}||\leq 1$. Based on the definition in (12) and the
defined features, the reward function can be re-written as:

$\displaystyle R(s,a,s^{\prime})$ $\displaystyle=\boldsymbol{w}^{T}\ .\
\boldsymbol{\phi}(s_{t}),$ (13)
$\displaystyle=\sum_{i=1}^{5}w_{i}\phi_{i}(s,a,s^{\prime}).$

As described in (13), the reward is a linear combination of the feature
values, $\boldsymbol{\phi}(s_{t})$. It is worth mentioning that not all
feature value functions behave positively for the reward function. For
instance, the throughput acts as an encouragement, and the interference plays
a punishment role. The goal of the IRL algorithm is to find the optimum value
for the weight vector, $\boldsymbol{w}^{*}$, to express the expert’s behavior
for this specific problem. Now, let’s define $\pi$ as a policy that maps the
visited states to probabilities on the action vectors for the decision-making
process. The value of each policy $\pi$ can be defined as an expected value of
the summation of discounted reward values based on the chosen policy:

$\displaystyle\mathbb{E}\ [V^{\pi}(s)]$ $\displaystyle=\mathbb{E}_{\pi}\
\left[\sum_{i=0}^{\infty}\gamma^{i}R(s_{i},a_{i},s^{\prime}_{i})\right],$
$\displaystyle=\mathbb{E}_{\pi}\ \left[\sum_{i=0}^{\infty}\gamma^{i}\
\boldsymbol{w}\ .\ \boldsymbol{\phi}(s_{i},a_{i},s^{\prime}_{i})\right],$
$\displaystyle=\boldsymbol{w}\ \mathbb{E}_{\pi}\
\left[\sum_{i=0}^{\infty}\gamma^{i}\boldsymbol{\phi}(s_{i},a_{i},s^{\prime}_{i})\right],$
(14) $\displaystyle=\boldsymbol{w}\ \mu(\pi),$

where $i$ is the step that the agent or the UAV takes in every episode.
$\mu(\pi)$ is defined as a feature expectation value for the policy $\pi$,
$\mu(\pi)\in\mathbb{R}^{k}$, and $k$ is the number of defined features.

To find the unknown parameters, $\boldsymbol{w}$, in the IRL algorithm,
different tools such as feature expectation matching [40, 44], maximum margin
planning [48, 49], and maximum causal entropy [50] have been proposed. In this
paper, we use the feature expectation matching between the expert and the
learner UAV. The reason is that the behavior of the agent and expert can be
expressed based on different explicit features, hence feature expectation
matching can be used in this scenario.

We assume that there exists an expert having access to the simulator
environment. The expert has a full understanding and knowledge of the problem
and scenario which means it considers the UEs’ density, shortest path,
throughput, and interference for the sake of the optimization problem defined
in (10a) - (10i). The expert simulates a finite number of trajectories to show
the desired behavior. These expert behaviors can be saved in terms of
trajectories as some vector of features state $\boldsymbol{\phi}(s)$ and
actions $a(s)\in\mathcal{A}$. Later, these trajectories can be used to define
the expert feature expectation value by getting an average over several
trajectories as follows:

$\displaystyle\bar{\mu}_{E}=\frac{1}{\rm N}\sum_{j=1}^{\rm N}\
\sum_{t=0}^{\infty}\gamma^{t}\phi(s_{t}^{j}),$ (15)

where $N$ is the number of trajectories and $j$ denotes the index for each
trajectory. This $\mu_{E}$ or $\mu(\pi_{E})$ carries the expert policy for the
demonstrated trajectories.

Claim: The goal is to find a policy $\pi^{*}$ such that the second-norm
distance between the expert feature expectation and the current policy feature
expectation is less than a threshold $\epsilon_{\textnormal{IRL}}$. The
mentioned policy will meet the criteria for the value of that policy as well
which means the first-norm distance between the two value functions for the
expert policy and learned policy based on the unknown reward function is less
than the same threshold $\epsilon_{\textnormal{IRL}}$.

###### Proof.

The first-norm distance between the expert’s and agent’s value function is
written as,

D $\displaystyle=\left\lvert\mathbb{E}\ \left[V^{\pi_{\rm
E}}\right]-\mathbb{E}\ \left[V^{\pi^{*}}\right]\right\rvert,$
$\displaystyle=\left\lvert\mathbb{E}_{\pi_{\rm
E}}\left[\sum_{i=0}^{\infty}\gamma^{i}R(s_{i},a_{i},s^{\prime}_{i})\right]-\mathbb{E}_{\pi^{*}}\left[\sum_{i=0}^{\infty}\gamma^{i}R(s_{i},a_{i},s^{\prime}_{i})\right]\right\rvert,$
$\displaystyle=\left\lvert\boldsymbol{w}^{T}\mu(\pi^{*})-\boldsymbol{w}^{T}\bar{\mu}(\pi_{\rm
E})\right\rvert,$
$\displaystyle\leq\left\lVert\boldsymbol{w}\right\rVert_{2}\left\lVert\mu(\pi^{*})-\bar{\mu}(\pi_{\rm
E})\right\rVert_{2},$ $\displaystyle\leq 1\ \ .\ \
\epsilon_{\textnormal{IRL}},$ (16) $\displaystyle\leq\
\epsilon_{\textnormal{IRL}}.$

This shows that finding the optimal weights for the reward function such that
$||\mu(\pi^{*})-\bar{\mu}(\pi_{E})||_{2}\leq\epsilon_{\textnormal{IRL}}$
guarantees that the distance between the value function for the expert and the
optimal policy is less than the same threshold. ∎

###  3.2 Support Vector Machine Problem Formulation

To find the weights for the reward function, the authors of [40] defined the
problem as a support vector machine (SVM) problem, where the aim is to
maximize the difference between the value function for the expert optimal
policy and other previous learned policies. In the SVM problem, this
maximization problem is mapped to maximize the distance between hyperplanes.
The goal of this SVM problem is to find a hyper-distance or margin ($D$) such
that the difference between the expert value function and all other previously
learned policies is $D$. This SVM problem can be written as

$\displaystyle\left\lvert\boldsymbol{w}^{*^{T}}\mu(\pi_{i})-\boldsymbol{w}^{*^{T}}\bar{\mu}(\pi_{E})\right\rvert\leq\Delta,$
(17)
$\displaystyle\boldsymbol{w}^{*^{T}}\left\lvert\mu(\pi_{i})-\bar{\mu}(\pi_{E})\right\rvert\leq\Delta,$
(18)

where, $i$ is index of the $i^{th}$ learned policy using the reinforcement
learning approach. To solve this SVM problem, the class for the expert policy
is labeled as “$+1$” and all other learned policies from the RL tool are
labeled as “$-1$”. Based on the defined SVM problem, the distance between the
origin and the expert policy class is
$\frac{1}{\left\lVert\boldsymbol{w}\right\rVert_{2}}$ and the distance between
the origin and all other learned policies is
$\frac{-1}{\left\lVert\boldsymbol{w}\right\rVert_{2}}$, hence, maximizing the
distance between hyper-planes is equal to minimizing
$\left\lVert\boldsymbol{w}\right\rVert^{2}_{2}$. As a result, (17) can be
written as:

$\displaystyle\min_{\boldsymbol{w}}\qquad\left\lVert\boldsymbol{w}\right\rVert_{2}^{2}\
,$ (19a) $\displaystyle\ s.t.\qquad\ \boldsymbol{w}^{T}\mu_{E}\geq 1,$ (19b)
$\displaystyle\qquad\qquad\boldsymbol{w}^{T}\mu_{\pi_{i}}\leq-1\leavevmode\nobreak\
.$ (19c)

Since this SVM problem looks like a quadratic programming (QP) problem, any
convex optimization tools or QP solvers such as CVXPY [51] or CVXOPT [52] can
solve this optimization problem. Here, CVXOPT is used in Python to find the
weights. The objective function of this optimization, (19a), is to minimize
the weights and the variables are the weights as well. Constraint (19b) is the
subject for the expert policy with the label of “+1” and constraint (19c) is
the subject for all the learned policies at different iterations with the
label of “-1”. It is worth mentioning that the algorithm of finding the
optimal weight such that the distance is less than a threshold
$\epsilon_{\textnormal{IRL}}$ is iterative and the number of learned policies
is increasing in the QP, hence the optimality of the given solution is crucial
in this problem. The standard format of any QP problem looks like the
following,

$\displaystyle\min_{x}\qquad\frac{1}{2}x^{T}Px+q^{T}x,$ (20a) $\displaystyle
s.t.\qquad\quad Gx\leq h,$ (20b) $\displaystyle\qquad\qquad\
Ax=b\leavevmode\nobreak\ .$ (20c)

To match this with our SVM problem (19a), matrices of $q$, $A$, and $b$ are
all zeros and matrix $P$ is $2\mathbf{I}_{K}$, which $\mathbf{I}$ is the
identity matrix and $K$ is the number of features (K=5 in our problem). The
dimension of matrices $G$ and $h$ changes and increases with each iteration.
Then in $n^{th}$ iteration for different weights and learned policies, the
formats of $G$ and $h$ are

$\displaystyle G_{(n+1,k)}=$
$\displaystyle\begin{bmatrix}[cccc]-\mu_{E}(1)&-\mu_{E}(2)&\ldots&-\mu_{E}(k)\\\
\mu_{\pi_{1}}(1)&\mu_{\pi_{1}}(2)&\ldots&\mu_{\pi_{1}}(k)\\\
\vdots&\vdots&\vdots\\\
\mu_{\pi_{n}}(1)&\mu_{\pi_{n}}(2)&\ldots&\mu_{\pi_{n}}(k)\end{bmatrix},$ (21)

$\displaystyle h_{(n+1,1)}=$ $\displaystyle\begin{bmatrix}[c]-1\\\ -1\\\
\vdots\\\ -1\end{bmatrix},$ (22)

where $\mu_{\pi_{n}}(k)$ is the $k^{th}$ UAV’s feature expectation based on
the $n^{th}$ learned policy. These matrices of $P$, $q$, $G$, $h$, $A$, and
$b$ map the QP problem to our scenario defined in (19a), (19b), and (19c).
Algorithm 1 summarizes the approach of the apprenticeship learning via IRL
inside the loop iteration to find the optimal weights.

1Initialization:

2 Load the simulation environment

3 Load the expert trajectories

4 Calculate the expert feature expectation
$\bar{\mu}_{E}=\frac{1}{N}\sum\limits_{j=1}^{N}\
\sum\limits_{t=0}^{\infty}\gamma^{t}\phi(s_{t}^{j})$;

5 Add some random values to the agent feature expectation for the first round
of optimization

6 while  _True_ do

7 Weights = SolveQP(Expert feature exp, Agent feature exp list);

8 Weights($w_{i}$) = Weights / norm;

9 TrainedPolicy($\pi_{i}$) = learner(Weights);

10 Reset the simulation environment;

11 $\mu_{i}(\pi_{i},w_{i})$ = RunSimulation($\pi_{i}$);

12 Add $\mu_{i}(\pi_{i},w_{i})$ to the agent feature expectation list;

13 Hyper distance = D = $\left\lvert
w^{T}\mu(\pi_{i})-w^{T}\bar{\mu}(\pi_{E})\right\rvert$;

14 if  _D $<$ $\epsilon_{\textnormal{IRL}}$_ then

15 Break;

16

17 end if

18 $i+=1$;

19

20 end while

21Return Weight($w_{i}$), $\mu_{i}$, $\pi_{i}$;

ALGORITHM 1 Apprenticeship learning via IRL algorithm

The main loop of Algorithm 1 continues until it reaches a point where the
hyper distance is less than a threshold and in that case, it breaks from the
main loop and returns the optimal weights $(w_{i})$, feature expectation
($\mu_{i}$), and the learned policy ($\pi_{i}$).

To train a model for the optimized weights of the reward function and find the
optimal policy to evaluate that for the hyper distance, it is required to
utilize learning methods like reinforcement learning (RL), especially a
temporal difference (TD) learning tool. These methods find the optimal policy
on line 9 of the Algorithm 1. In this paper, we use two TD methods, the first
one is Q-learning based on linear function approximation and the second one is
deep reinforcement learning (DRL) or deep Q-network (DQN). The next two
sections propose these two tools for the mentioned problem of apprenticeship
learning to learn the policy.

###  3.3 AL-IRL with Q-learning using Linear Function Approximation

After finding the normalized weights in apprenticeship learning, the reward
function is known for the specific iteration and the problem of MDP\R is
converted to a MDP problem with the tuple of
$\langle\mathcal{S},\mathcal{A},Pr,R,\gamma\rangle$ where $R$ is the reward
function defined based on the obtained weights, (12) and (13). The learning
policy aims to find a policy that can decide on an action for the agent at
each observed state to maximize the immediate reward with also considering the
future reward. Different value iteration approaches are available to find this
policy [53, 54]. We chose the Q-learning tool to solve this problem [55, 56].
In the standard Q-learning, the updating rule for the Q values is defined
based on

$\displaystyle
Q(s,a)\leftarrow(1-\alpha)Q(s,a)+\alpha[R(s,a,s^{\prime})+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime})],$
(23)

where $s\in\mathcal{S}$ is the current state, $a\in\mathcal{A}$ is the action
chosen based on the policy or random exploration which changes the agent’s
state from $s$ to $s^{\prime}$, $\alpha$ is the learning rate,
$R(s,a,s^{\prime})$ is the reward function defined based on the IRL approach
from the previous section. $\gamma$ is the discount factor considering the
future reward (value-iteration or $Q(s^{\prime})$). After updating the
Q-values based on different approaches such as temporal difference (TD) or
Monte Carlo (MC), the agent follows the policy rule for the exploitation
greedy behavior. Later, this policy is considered as a learned policy for the
apprenticeship learning.

$\displaystyle\pi(s)=\operatorname*{argmax}_{a}Q(s,a)\leavevmode\nobreak\ .$
(24)

It is worth mentioning that this Q updating approach works for problems with
discrete states and actions such as the grid-world cells or scenarios.
However, the state in this paper is based on the vector of features
$\boldsymbol{\phi}(s)$ based on (11). Because the states are defined as
continuous values from the feature vectors, this makes the Q-learning updating
rule more difficult. The solution to this problem is to find the Q-value from
the defined feature vectors using a linear function approximation (LFA) tool
such as linear regression. This means each function of $Q(s,a)$ can be defined
based on the values of features:

$\displaystyle Q(s_{t},a_{t})$ $\displaystyle=\theta_{0}+\theta_{1}\
\phi_{1}(s,a)+\theta_{2}\ \phi_{2}(s,a)+\dots$ (25) $\displaystyle\qquad\
+\theta_{\kappa}\ \phi_{\kappa}(s,a)=\boldsymbol{\theta}^{T}\phi(s,a),$

where $\kappa$ is the number of features and in this paper, there are five
features for each visited state. $\theta_{i}$ are the weight vectors to find
the value of $Q(s,a)$. To find the $\boldsymbol{\theta}$ vectors, we need to
use a batch of sample data to update the values. To solve this linear
regression problem, the stochastic gradient descent (SGD) approach is used to
minimize the loss function for the difference between the predicted Q-value
and the actual value. The true target value for the Q is obtained based on the
Q-value updating rule and this value is called $Q^{+}(s,a)$:

$\displaystyle Q^{+}(s,a)\approx
R(s,a,s^{\prime})+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime})\leavevmode\nobreak\
.$ (26)

This Q target is used in the loss function for the SGD:

$\displaystyle L(\theta^{(i)})$
$\displaystyle=\frac{1}{2}(Q^{+}(s,a)-Q(s,a))^{2}$ (27)
$\displaystyle=\frac{1}{2}(Q^{+}(s,a)-\phi(s,a)^{T}\theta^{(i)})^{2}\leavevmode\nobreak\
.$

Now considering the mentioned loss function, the values of the weights update
based on the gradient:

$\displaystyle\theta^{(i+1)}$
$\displaystyle=\theta^{(i)}-\alpha_{S}\partial(L(\theta^{(i)}))/\partial\theta$
(28)
$\displaystyle=\theta^{(i)}-\alpha_{S}\partial(\frac{1}{2}(Q^{+}(s,a)-\phi(s,a)^{T}\theta^{(i)})^{2})/\partial\theta$
$\displaystyle=\theta^{(i)}-\alpha_{S}[Q^{+}(s,a)-\phi(s,a)^{T}\theta^{(i)}]\
.\ \phi(s,a)\leavevmode\nobreak\ .$

The optimal values of the weights are obtained based on different episodes,
visits, and exploration of the agent. The agent’s exploration and exploitation
process is based on the $\epsilon$-greedy process with $\epsilon$ decay over
the episodes. It means that the agent behaves randomly in the early episodes,
and then acts more greedy based on exploitation of Q-values. At the end of
this process, the learned policy ($\pi_{i}$) will be used in the
apprenticeship learning to get the latest feature expectation values
($\mu_{i}$). Algorithm 2 summarizes the procedure for policy learning using
Q-learning via linear function approximation for a specific weight of the
reward function.

1 Initialization:

2 Load the simulation environment;

3 $\epsilon=1$;

4 while  _episode $<$ NUM_EPS_ do

5 $distance$ = 0;

6 Reset UAV();

7 $Done$ = FALSE;

8 while  _distance $<$ DIST_LIM & not DONE_ do

9 if  _random $<$ $\epsilon$_ then

10 $action$ = random Action;

11

12 else

13 $action$ = GetGreedyAction(SGD Model);

14

15 end if

16 Update $Location$ ;

17 Calculate SNR, Interference, Throughput;

18 Calculate Features
($\phi_{1}(s,a,s^{\prime}),\dots,\phi_{5}(s,a,s^{\prime})$);

19 Calculate immediate reward
$R(s)=\boldsymbol{w}^{T}.\boldsymbol{\phi}(s^{\prime})$;

20 $Q(s^{\prime})=$ predict(SGD Model, $\phi(s^{\prime})$);

21 if  _New location is DESTINATION_ then

22 $Done$ = TRUE;

23 $Q^{+}(s,a)=R(s)$;

24

25 else

26 $Q^{+}(s,a)=R(s)+\gamma\max\limits_{a^{\prime}}Q(s^{\prime},a^{\prime})$;

27

28 end if

29 Update SGD Model(SGD Model, $\boldsymbol{\phi}(s)$, $Q^{+}(s,a)$);

30 $distance+=1$;

31

32 end while

33 if  _$\epsilon$ $>$ 0.1 & episode $>$ NUM_EPS/10_ then

34 $\epsilon$ -= 1/NUM_EPS ;

35

36 end if

37 $episode+=1$;

38

39 end while

40Return SGD Model;

ALGORITHM 2 Q-learning algorithm with Linear Function Approximation

Algorithm 2 finds and returns the learned policy for the specific reward
function. DIST_LIM indicate the UAV’s hop counts limitation that based on
UAV’s battery life and flight time limitation is adopted. In this case, the
learned policy is the trained SGD model. The size of the SGD model depends on
the action vector. For instance, in this paper, the agent has 6 (Mobility)
$\times$ 6 (Power Allocation) actions to choose from, hence 36 SGD models are
the learned policy for this specific configuration. In next section, we
propose a deep Q-network instead of the stochastic gradient descent approach
for linear function approximation.

###  3.4 AL-IRL with Deep Q-Network

In the previous section, the solution to find the optimal policy was based on
the Q-learning algorithm using the linear function approximation with
stochastic gradient descent. In this section, the DQN approach is used to
predict the Q-value based on the current features vector (current state) and
the chosen action. Fig. 3 demonstrates the deep reinforcement learning
structure for the apprenticeship learning for the obtained reward weights
based on the inverse reinforcement learning algorithm.

Figure 3: Deep Q-Network approach for the optimal policy in the apprenticeship
learning.

The optimal policy learning block consists of a deep neural network with two
hidden layers, an input layer with the size of the number of features for each
state, and an output layer with the size of all actions. The numbers of
neurons in the hidden layers are 30 and 30, respectively. Both hidden layers
utilize the Rectified Linear Unit (RELU) [57] for the activation layer. The
last layer has the linear activation function since the output of the deep
neural networks is a continuous value for the Q-Action-Value. The structure of
this DNN is demonstrated in Fig 4.

Figure 4: Deep neural network structure used in the DQN approach to predict
the Q-Action value based on the features vector.

The concept of the training for the exploration and exploitation is similar to
Algorithm 2. However, the training process is different and in this approach,
training the model is based on the batch samples. The batch samples are picked
randomly from the replay memory. Also, a buffer length is considered for the
replay memory to keep the data fresh and pop the old data from the replay.
Algorithm 3 presents the random batch sample training and fitting those data
for the model. This algorithm is summarized since some structures appeared in
Algorithm 2, hence, a few lines are just removed.

1 while  _episode $<$ NUM_EPS_ do

2 …;

3 while  _distance $<$ DIST_LIM & not DONE_ do

4 …;

5 Collect features, actions, reward for the Replay memory;

6 if  _episode $>$ NUM_EPS/10_ then

7 batch = random.sample(replay, BATCH_SIZE) ;

8 for  _all data in batch_ do

9 x_train = batch[current feature];

10 if  _New location is DESTINATION_ then

11 $Q^{+}(s,a)=R(s)$;

12

13 else

14 $Q^{+}(s,a)=R(s)+\gamma\max\limits_{a^{\prime}}Q(s^{\prime},a^{\prime})$;

15

16 end if

17 y_train = batch[$Q^{+}(s,a)$];

18

19 end for

20 model_DQN.fit(x_train, y_train);

21

22 end if

23 $\textnormal{distance}+=1$;

24

25 end while

26 if  _$\epsilon$ $>$ 0.1 & episode $>$ NUM_EPS/10_ then

27 $\epsilon$ -= 1/NUM_EPS;

28

29 end if

30 $episode+=1$;

31

32 end while

33Return DQN Model;

ALGORITHM 3 Deep Q-Network to predict the Q-Action-Value on batch samples

To train the model and find the optimal weights for the neurons, the Mean
Squared Error (MSE) [58] is used as the loss function for the optimization.
And to perform the stochastic optimization on the MSE, an optimizer called
“Adam” is used to solve the problem for each batch of collected data [59]. The
MSE loss function for a batch with $n$ samples is shown below

$\displaystyle MSE=\frac{\sum\limits_{i=1}^{n}(Q^{+}(s,a)-Q(s,a))^{2}}{n},$
(29)

where $Q(s,a)$ is the predicted value based on the current features vector and
$Q^{+}(s,a)$ is the actual values (target value) for the Q-value updated based
on (26). $n$ is the number of samples in each batch samples for the training.

###  3.5 Imitation Learning: Behavioral Cloning

This section proposes the model used in this paper for the BC. Usually,
supervised learning tools can obtain the model for imitation learning to mimic
the expert’s behavior [60]. However, the BC approach only has the solution for
those states observed by the expert. If the agent or UAV examines a state that
the expert has never seen before, the outcome may not be optimal. All
trajectories from the expert are gathered for imitation learning and BC. The
same feature vectors of states with the respective actions are used to train a
deep neural network for task classification based on the observed state by the
UAV. This problem is categorized as a task classification problem since the
UAV wants to choose the right action at a right time based on the expert’s
experience.

To implement the supervised learning for the BC, we first collect the expert’s
data and trajectories. These trajectories provide a simulated dataset for the
training and test evaluation. BC does not need constant access to the expert;
however, DAGGER requires the expert’s presence in states that have never been
visited before. Implementing the DAGGER approach is more costly and not always
feasible; thus, the BC is applied here for imitation learning. We should note
that the collected dataset is based on states and actions, and there is no
meaning of the reward function, and the agent mimics the expert blindly.

The expert data, ($\mathcal{D}$), is collected in terms of
$\mathcal{D}=\\{(\mathbf{x}_{t},\mathbf{s}_{t},\mathbf{a}_{t})\\}$, where
$\mathbf{x}_{t}$ is the initial condition for the expert. In this paper, we
assume that $\mathbf{x}_{t}$ is the location where the agent or UAV starts its
initial location $(0,\ 0)$. $\mathbf{s}_{t}$ is the UAV’s state that consists
of the features vector defined in (11) in Section 3.1. Action
($\mathbf{a}_{t}$) is the mixed action based on the mobility and the
transmission power allocation, $\mathbf{a}\in\mathcal{A}$. The action space is
$\mathcal{A}=\\{a(i)|0\leq i\leq 35\\}$ which consists of 6 movement actions
and 6 transmission powers. Since the agent decides at the same time for both
the mobility and transmission power, there are 36 actions. If the actions were
independent, the number of actions would be 12 instead of 36. The collected
expert trajectories can be shown as,

$\displaystyle\tau=\\{\mathbf{\phi(s)}^{(0)},\mathbf{a}^{(0)},\mathbf{\phi(s)}^{(1)},\mathbf{a}^{(1)},\dots\\}\
.$ (30)

After collecting the expert trajectories, the problem is defined as multi-
class classification, and a supervised learning technique is utilized to
handle the problem. The supervised learning approach generates a model
predicting the appropriate class (action) based on the visited state. A
decision tree algorithm is chosen as a supervised learning technique. Next,
the UAV will use this model to decide on the next cell and the next
transmission power based on the expert’s collected information. Here, the
learned model is called $\pi_{\rm BC}$

$\displaystyle\mathbf{a}=\pi_{\rm BC}(\mathbf{x}_{t},\mathbf{s}_{t}),$ (31)

which BC stands for the BC. To train the model, the decision tree from the
SciKit-Learn Python package is used to solve the problem [61, 62]. Gini
impurity is used to determine the quality of split for various features. After
training, the decision tree model has a depth of five with seven leaves.

##  4 Numerical Results and Experiments

In this section, first, the designed simulator is explained, then, the
convergence of the apprenticeship learning for the IRL and the policy learner
using both the Q-learning and deep Q-network methods is analyzed. Afterward,
the performance comparison among various decision-making approaches at the UAV
including inverse reinforcement learning (Q-learning using linear function
approximation), inverse Deep Q-Network, BC, shortest path, and a random policy
is utilized to compare the proposed method with similar approaches. Through
all the experimental results a probabilistic channel according to 3 for UAV is
considered. However, in some cases, to show the performance comparison results
in different UAV’s channel conditions, we mentioned two different UAV’s
channel scenarios, probabilistic and LoS channels. All designs and simulations
are tested on a Ubuntu system with a Ryzen 9 3900X CPU and Nvidia RTX 2080 Ti
GPU. The training phase for the deep neural network approach was done based on
Tensorflow 2.3.0, Keras 2.4.0 API, CUDA 10.1, and cuDNN 7.6 with Python 3.6.

###  4.1 Designed Simulation Environment

In this study, we develop an Open Source simulation environment based on the
proposed problem in Sec. 2. The simulation environment is designed in Python
3.6 to consider the UAV in a pre-defined area with 25 cellular base stations
and multiple terrestrial UEs. The UEs’ distribution is in such a way that some
cells and areas have a higher density and others are less crowded.

The proposed solution should be able to find the optimal path regarding the
distance, throughput, number of adjacent UEs, and interference on UEs in a
dense urban environment with parameters $c_{1}=12.076$ and $c_{2}=0.114$. This
simulation is event-based, and the event in this case is the UAV’s action. The
UAV has six actions for mobility and six discrete actions for the transmission
power, which results in 36 actions together. The user can change the
transmission power range to consider a broader or narrower range to increase
or decrease the number of actions. More actions bring more complexity to
convergence.

Figure 5: Demonstration of the designed graphical interface for the user.

This simulator is designed based on an object-oriented approach. Different
classes are defined for the UAV, UEs, and BSs. All parameters for the learning
algorithms and environment are configurable in the simulator. Table 2
summarizes the main simulation parameters.

Table 2:  Simulation Parameters Parameter |  Value
---|---
Number of cellular BS |  $25$
Number of UEs |  $75$
UAV altitude ($H$) |  $50$ m
UAV max transmit power |  $200$ mW
UAV min transmit power |  $50$ mW
Dense urban ($\eta_{LoS}$, $\eta_{NLoS}$) |  $(1.6,23)$ [dB]
Dense urban ($c_{1}$, $c_{2}$) |  $(12.076,0.114)$
$N_{0}$ |  $-90$ dBm
Carrier frequency |  $2$ GHz
UE transmit power |  $2$ mW
UAV antenna gain |  $100$
Number of features |  $5$
Number of Epochs |  1e4
$\epsilon$ |  $0.1$
Batch size |  $24$
Replay buffer size |  $1e4$
$\gamma$ |  $0.99$
$\alpha$ |  $0.001$

In Fig. 5, a circle notation is considered to show the UAV’s coverage area as
a function of its transmission power for the sake of demonstrating the
interference level. The UAV’s directions are demonstrated by black arrows to
show the chosen path. The user can turn off the Graphical User Interface (GUI)
to speed up the simulation. All models and trajectories can be saved on a
drive for future evaluations. Fig. 5 shows a sample snapshot of the simulator
with the chosen path. In this sample, the UAV flies from the source cell (BS0)
toward the destination cell (BS24) with varying transmission power. The source
code of this simulator is available on the GitHub repository [63]. A sample
video of the designed simulator implementing the different approaches in this
study is available on YouTube [64]. In the video, we showed the simulation
based on the different approaches during the training phase and also after
training for the evaluation.

###  4.2 Convergence of the IRL

To investigate the convergence for Inverse Reinforcement Learning, the hyper
distance between the expert’s feature expectation and learner feature
expectation is considered as a metric to show the convergence behavior of the
IRL algorithm. Fig. 6 demonstrates this hyper distance for different
iterations. The learning process stops at the iteration where the hyper-
distance is getting lower than a predefined threshold
($\epsilon_{\textnormal{IRL}}$). The user can choose the desired threshold
based on the requirements. If the user chooses
$\epsilon_{\textnormal{IRL}}=0$, this means the user wants to achieve the same
behavior compared to the expert. Choosing $\epsilon_{\textnormal{IRL}}=0$ may
need an infinite number of iterations for the IRL algorithm to find the
optimal weight.

Figure 6: Hyper distance between the expert and agent feature expectation
based on different weights, reward functions, and optimal policy.

In Fig. 6, we assumed that the threshold ($\epsilon_{\textnormal{IRL}}$) for
the distance between the expert and agent feature expectation is $0.1$ and any
distance less than the threshold stops the algorithm. After ten iterations,
this algorithm stops when the distance is $0.057$.

###  4.3 Convergence of the Q-Learning and Deep Q-Network

Fig. 7 demonstrates a few iterations of the feature policy learning for each
unique reward function for the Q-learning with linear function approximation
using the SGD. Based on the exploration-greedy rate, the accumulative reward
function is converged to the best optimum value. It is worth mentioning that
the optimum value for the reward function depends on the obtained weight for
the reward function at each iteration based on the QP, and because of that,
each plot is converged to a different value. Also, Fig. 8 illustrates the same
convergence concept for the various iterations of the policy learning for the
Deep Q-Network.

(a) Accumulative reward in each episode at iteration 1

(b) Accumulative reward in each episode at iteration 2

(c) Accumulative reward in each episode at iteration 3

Figure 7: Convergence of the policy learning using Q-Learning in different
iterations for the IRL approach.

(a) Accumulative reward in each episode at iteration 1

(b) Accumulative reward in each episode at iteration 2

(c) Accumulative reward in each episode at iteration 3

Figure 8: Convergence of the policy learning using deep Q-Network in different
iterations for the IRL approach.

###  4.4 Apprenticeship Learning via Inverse RL performance - Training Phase

All results in this section are averaged over 25 runs to have a smooth and
clear demonstration for the sake of comparison. Figures 9(a), 9(b), 9(c)
represent results for UAV’s probabilistic channels, and Figures 9(d), 9(e),
9(f) show the results for UAV’s LoS channels. Fig. 9(a) and Fig. 9(d) show the
throughput result of the inverse RL using the Q-learning and deep Q-network
during the training phase of the optimal reward function after the final
optimization and termination of the IRL algorithm. In Fig. 9(d) both
approaches converge at the same number of epochs and at the same values after
the $\epsilon$-greedy exploration. It is worth mentioning that achieving the
maximum throughput is not always the optimum policy since it increases
interference as well.

Also, Fig. 9(b) and Fig. 9(e) show the summation of interference levels on the
neighbor UEs when the UAV uses the same resource block for its transmission
and the UAV’s uplink can interfere with the other UEs’ downlink as well.
Keeping the interference as low as possible is desired; however, it decreases
the throughput as well, and it is not in line with the problem objectives
defined in (10a) and (10d) in Section 2. As Fig. 9(e) represents Q-learning
and DQN algorithms converge to different values, and the reason is that
because these two algorithms utilized different paths for the problem, thus
the agent (UAV) senses a different number of UEs, and the effect of its UL
transmission is different in these models.

Fig. 9(c) and Fig. 9(f) demonstrate the distance between the location where
the UAV finishes its task and the destination cell where it was supposed to
finish and stop based on resource limitations such as the battery capacity and
flight time. In Fig. 9(c) DQN converges to the distance of zero which means
this algorithm finds the destination cell (in this scenario (BS24)). However,
the limited energy of UAVs for flight, coupled with the highly dynamic
wireless environment and the low convergence power of the Q-learning
algorithm, results in an inability to reach the destination in probabilistic
channel conditions. However, in Fig. 9(f) both Q-learning and DQN converges to
the distance of zero which means both algorithms find the destination cell.

(a) Transmission throughput rate for the UAV’s Up-Link in probabilistic
channel scenario

(b) UAV’s UL interference on neighbor UEs using the same RBs in probabilistic
channel scenario

(c) The distance between the last location where the UAV stops its task in
probabilistic channel scenario

(d) Transmission throughput rate for the UAV’s Up-Link in LoS channel scenario

(e) UAV’s UL interference on neighbor UEs using the same RBs in LoS channel
scenario

(f) The distance between the last location where the UAV stops its task in LoS
channel scenario

Figure 9: Evaluation of UAV’s UL throughput, interference value on neighbor
UEs, and the final distance of the UAV during the training phase of the
inverse RL for both Q-learning and DQN.

###  4.5 Apprenticeship Learning via Inverse RL Performance

Here, the goal is to compare the performance of different path planning
mechanisms including apprenticeship learning via IRL, inverse deep RL, BC,
shortest path with minimum yaw, pitch, and roll, and random action. The
previous section shows the result for the inverse RL using Q-learning and DQN
during the training phase. Here, based on the trained model and obtained
weights for the reward function, a sample scenario is considered to compare
the performance of these different approaches. The implemented BC model
defined in Section 3.5 predicts the desired action based on the visited
features vector state with 89.08% accuracy using the decision tree
classification approach.

Fig. 10(a) compares the throughput of the DQN, BC, shortest path, and random
policy. DQN has better behavior for the throughput than random and shortest
path algorithms, and the reason is the feature expectation for the throughput
metric has a positive role on the reward function in the features state vector
($\phi(s,t)$). In the shortest path, the UAV chooses the shortest path between
the source cell (BS0) and the destination with minimum yaw, pitch, and rolling
to consume less energy. The transmission power is chosen at random in the
shortest path. The movement action and transmission power are both random in
the random policy. Random policy has the worst performance in terms of
throughput in the same scenario. The main factor to affect the throughput
metrics is the agent’s (UAV) transmission power for its UL transmission.

(a) Throughput mean and standard deviation comparison for the UAV Up-link

(b) Average and standard deviation interference affecting the ground neighbor
UEs

Figure 10: Evaluation of the UAV’s UL throughput and the effect of
interference with neighbor UEs in a single scenario.

Another comparison is performed in Fig. 10(b) regarding the interference
management to compare all approaches. This interference is the summation of
the interference applied to all neighbor UEs if they all use the same resource
block. The level of interference in this figure depends on the UAV’s
transmission power for its UL, the density of neighbor UEs in the neighbor
cells, and the distance between the UAV and the affected UEs. BC has the
lowest interference level since it completely mimics the expert’s behavior
without knowing any understanding of it. Next, DQN approache has the next
level for applied interference. The shortest path chooses the path that has a
higher density of UEs density compared to other approaches, and because of
that, the interference is higher compared to other two techniques and almost
similar to random policy. If the user wants to train the DQN approache closer
to the expert’s behavior, then it is possible to choose a lower value for the
epsilon threshold ($\epsilon_{IRL}$). In that case, the algorithm’s
convergence process needs more time to meet the threshold; however, the agent
feature expectation vector ($\mu(\pi_{i})$) is closer to the expert feature
expectation vector ($\bar{\mu}(\pi_{E})$). Therefore, the threshold value can
be determined based on the user’s expectation for the learning and
optimization process of the reward function’s weights and the optimal policy.
Any changes in the epsilon threshold ($\epsilon_{IRL}$) may change the results
shown in Fig. 6, and more iterations may be required to meet the criteria.

Figure 11: The distance between the last location where the UAV stops its task
and the center of the destination cell

Fig. 11 shows the distance between the destination cell (BS24) and the cell
where the agent stops or finishes its tasks. Both the behavioral cloning and
shortest path visit the destination cell at the 6th step. DQN visits the
destination cell at the $7^{th}$ step. The random policy never finds the
destination cell because of battery life and flight time limitations.

###  4.6 Inverse RL and Behavioral Cloning in Unseen States by the Expert

Another scenario is studied in this section to compare the performance of the
Q-learning, DQN, and BC in a situation where the UAV is placed in a cell that
an expert has never seen or experienced. Since we assumed that the expert is
only available for a few trajectories or it is costly to have on-demand access
to the expert, it is not possible to ask the expert to experience the new
state. Hence, approaches like DAGGER are not practical in this situation. In
this case, an environmental error such as wind is applied to the drone and
placed the drone in the adjacent cell ($BS_{5}$) as the initial cell or the
cell of the source. This $BS_{5}$ has never been seen by an expert before, and
there is no data available for this point.

(a) Path of the behavioral cloning with the applied error

(b) Path of the Q-learning with the applied error

(c) Path of the deep Q-network with the applied error

Figure 12: Running Q-learning, DQN, and behavioral cloning on the defined
scenario with an environmental error where wind moves the UAV to from cell(0)
to the adjacent cell.

Fig. 12 shows the agent’s trajectories, states, and actions for the three
approaches mentioned. Based on Fig. 12(a) the UAV utilizing the BC approach
starts its location from BS5, and it chooses the path with a higher density of
UE, and it never reaches the destination cell (BS24) to finish its task. Also,
Q-learning in Fig. 12(b) utilizes the Q-learning approach and tried to avoid
the high UE density area but it fails to reach the destination because of a
complicated dynamic wireless situation. However, DQN found the destination
cell and tried to avoid the high UE density area considering the shortest path
as well in Fig. 12(c).

(a) Throughput mean and standard deviation for the UAV UL

(b) Interference mean and standard deviation on UEs DLs

(c) Distance between the UAV and the destination point

Figure 13: Performance of DQN, and BC with the applied error.

The performance evaluation of throughput, interference and the distance
between the drone and the destination cell is depicted in Fig. 13. DQN
approach has better average and standard deviation for the throughput in Fig.
13(a). The reason is that the BC chooses power values for its path, which is
incorrect, since the path has never been experienced before, and the data for
that was not available in the dataset. Fig. 13(b) demonstrates that DQN has
better interference management in the case that any errors happen to the
system. Also, Fig. 13(c) shows that DQN reached the destination and finished
the task at the $6^{th}$ step; however, the BC was not able to finish the task
by reaching the destination cell.

This evaluation shows that apprenticeship learning using inverse deep RL
performs better compared to imitation learning (behavioral cloning) for the
cases where the visited state by the agent was not experienced by the expert.

One future direction for this work is to study a more practical scenario for
modeling the antenna gains as the antennas of the ground BSs are usually down
tilted and optimized to increase the capacity for ground UEs and minimize the
inter-cell interference effect to the ground UEs. Hence, the cellular-
connected UAVs may be serviced by the sidelobes of the terrestrial BSs. In
this case, the UAV may connect to BSs which are located far away, and the
channel state information shows non-trivial behavior. Studies such as [65] and
[66] considered the antenna behavior for trajectory optimization and
observation in both simulation and experiments. Considering this challenge can
propose the future direction regarding the system model and methodology in
this paper.

##  5 Conclusions

The emerging applications of unmanned aerial vehicles call for new reliable
communication technologies to facilitate a low-delay high transmission rate
communication. Cellular networks, in particular 5G and beyond, offer various
advantages for drones including reliability, wide coverage, and security.
However, noting the fast-growing number of utilized drones as aerial cellular
users may raise several challenges such as interference to terrestrial users
and base stations. In this paper, a novel interference-aware joint path
planning and power allocation scheme are proposed for an autonomous cellular-
connected unmanned aerial vehicle.

The UAV’s object is to take a path from its initial point to its destination
subject to minimizing the interference imposed on the terrestrial User
Equipment (UEs) and maximizing its uplink throughput. To solve this problem,
we proposed apprenticeship learning via inverse RL using both Q-learning and
deep Q-network solutions to find the optimal path planning with power
allocation using expert knowledge. Another imitation learning method called
behavioral cloning is performed using the decision tree supervised learning to
compare the proposed method for the sake of comparison. In the numerical and
simulation results, the apprenticeship learning via IRL performs close to the
expert and the BC approach. Also, the user can define the threshold to
determine how close the agent should follow the desired expert behavior.
Moreover, we showed that inverse RL performs better in cases where an error
exists in the system or the supervised data for the visited state was never
collected by the expert.

## Acknowledgment

This material is based upon work supported by the Air Force Office of
Scientific Research under award number FA9550-20-1-0090 and the National
Science Foundation under Grant Numbers CNS-2232048 and ECCS 2030047.

## References

  * [1] A. Andreeva-Mori, D. Kubo, K. Kobayashi, Y. Okuno, J. R. Homola, M. Johnson, P. H. Kopardekar, Supporting disaster relief operations through utm: Operational concept and flight tests of unmanned and manned vehicles at a disaster drill, in: AIAA Scitech 2020 Forum, 2020, p. 2202.
  * [2] A. Pandey, P. K. Shukla, R. Agrawal, An adaptive flying ad-hoc network (FANET) for disaster response operations to improve quality of service (qos), Modern Physics Letters B 34 (10) (2020) 2050010.
  * [3] S. Javed, A. Hassan, R. Ahmad, W. Ahmed, M. M. Alam, J. J. Rodrigues, Uav trajectory planning for disaster scenarios, Vehicular Communications (2023) 100568\.
  * [4] F. Afghah, A. Razi, J. Chakareski, J. Ashdown, Wildfire monitoring in remote areas using autonomous unmanned aerial vehicles, in: IEEE INFOCOM 2019 \- IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), 2019, pp. 835–840.  doi:10.1109/INFCOMW.2019.8845309.
  * [5] A. Shamsoshoara, F. Afghah, A. Razi, L. Zheng, P. Z. Fulé, E. Blasch, Aerial imagery pile burn detection using deep learning: the flame dataset, Computer Networks (2021) 108001.
  * [6] M. Keshavarz, A. Shamsoshoara, F. Afghah, J. Ashdown, A real-time framework for trust monitoring in a network of unmanned aerial vehicles, in: IEEE INFOCOM 2020-IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), IEEE, 2020, pp. 677–682.
  * [7] Faa aerospace forecasts, https://www.faa.gov/data_research/aviation/aerospace_forecasts/, (Accessed on 03/06/2023).
  * [8] F. Forecast, Fiscal years 2022-2042, Federal Aviation Administration (2022).
  * [9] M. Mozaffari, W. Saad, M. Bennis, M. Debbah, Unmanned aerial vehicle with underlaid device-to-device communications: Performance and tradeoffs, IEEE Transactions on Wireless Communications 15 (6) (2016) 3949–3963.
  * [10] M. Mozaffari, W. Saad, M. Bennis, Y.-H. Nam, M. Debbah, A tutorial on uavs for wireless networks: Applications, challenges, and open problems, IEEE Communications Surveys & Tutorials 21 (3) (2019) 2334–2360.
  * [11] Q. Huang, A. Razi, F. Afghah, P. Fule, Wildfire spread modeling with aerial image processing, in: 2020 IEEE 21st International Symposium on ”A World of Wireless, Mobile and Multimedia Networks” (WoWMoM), 2020, pp. 335–340.  doi:10.1109/WoWMoM49955.2020.00063.
  * [12] Y. Zeng, R. Zhang, T. J. Lim, Wireless communications with unmanned aerial vehicles: Opportunities and challenges, IEEE Communications Magazine 54 (5) (2016) 36–42.
  * [13] N. Namvar, F. Afghah, Heterogeneous airborne mmwave cells: Optimal placement for power-efficient maximum coverage, in: IEEE INFOCOM 2022 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), 2022, pp. 1–6.  doi:10.1109/INFOCOMWKSHPS54753.2022.9798023.
  * [14] N. Namvar, F. Afghah, Joint 3d placement and interference management for drone small cells, in: 2021 55th Asilomar Conference on Signals, Systems, and Computers, 2021, pp. 780–784.  doi:10.1109/IEEECONF53345.2021.9723350.
  * [15] 3rd Generation Partnership Project (3GPP), 3rd generation partnership project (3gpp), release 15, https://www.3gpp.org/specifications/67-releases, (Accessed on 05/12/2020) (March 2020).
  * [16] 3rd Generation Partnership Project (3GPP), 3gpp ts 36.420 3gpp tsg ran evolved universal terrestrial radio access network (eutran), x2 general aspects and principles, version 11.0.0, release 11, https://www.3gpp.org/specifications/67-releases, (Accessed on 15/17/2020) (October 2012).
  * [17] A. Rovira-Sugranes, A. Razi, F. Afghah, J. Chakareski, A review of ai-enabled routing protocols for uav networks: Trends, challenges, and future outlook, Ad Hoc Networks 130 (2022) 102790.
  * [18] F. Lotfi, O. Semiari, W. Saad, Semantic-aware collaborative deep reinforcement learning over wireless cellular networks, in: ICC 2022 - IEEE International Conference on Communications, 2022, pp. 5256–5261.  doi:10.1109/ICC45855.2022.9839122.
  * [19] M.-A. Lahmeri, M. A. Kishk, M.-S. Alouini, Artificial intelligence for uav-enabled wireless networks: A survey, IEEE Open Journal of the Communications Society 2 (2021) 1015–1040.
  * [20] Y. Huang, Q. Wu, R. Lu, X. Peng, R. Zhang, Massive mimo for cellular-connected uav: Challenges and promising solutions, IEEE Communications Magazine 59 (2) (2021) 84–90.
  * [21] F. Lotfi, O. Semiari, F. Afghah, Evolutionary deep reinforcement learning for dynamic slice management in O-RAN, arXiv preprint arXiv:2208.14394 (2022).
  * [22] Y. Huo, X. Dong, W. Xu, 5g cellular user equipment: From theory to practical hardware design, IEEE Access 5 (2017) 13992–14010.
  * [23] A. Checko, H. L. Christiansen, Y. Yan, L. Scolari, G. Kardaras, M. S. Berger, L. Dittmann, Cloud ran for mobile networks—a technology overview, IEEE Communications surveys & tutorials 17 (1) (2014) 405–426.
  * [24] M. Gharib, S. Nandadapu, F. Afghah, An exhaustive study of using commercial lte network for uav communication in rural areas (2021).  arXiv:2105.03778.
  * [25] F. Lotfi, O. Semiari, Performance analysis and optimization of uplink cellular networks with flexible frame structure, in: 2021 IEEE 93rd Vehicular Technology Conference (VTC2021-Spring), 2021, pp. 1–5.  doi:10.1109/VTC2021-Spring51267.2021.9448665.
  * [26] 3rd Generation Partnership Project (3GPP), 5g;unmanned aerial system (uas) support in 3gpp, 3gpp ts 22.125 version 16.3.0 release 16, technical specification, https://www.3gpp.org/specifications/67-releases (November 2020).
  * [27] 3rd Generation Partnership Project (3GPP), Universal mobile telecommunications system (umts); lte; 5g; t8 reference point for northbound apis, 3gpp ts 29.122 version 17.5.0 release 17, technical specification, https://www.3gpp.org/specifications/67-releases (May 2022).
  * [28] D. Mishra, E. Natalizio, A survey on cellular-connected uavs: Design challenges, enabling 5g/b5g innovations, and experimental advancements, Computer Networks 182 (2020) 107451.
  * [29] Y. Zeng, X. Xu, S. Jin, R. Zhang, Simultaneous navigation and radio mapping for cellular-connected uav with deep reinforcement learning, IEEE Transactions on Wireless Communications (2021).
  * [30] W. Mei, Q. Wu, R. Zhang, Cellular-connected uav: Uplink association, power control and interference coordination, IEEE Transactions on Wireless Communications 18 (11) (2019) 5380–5393.
  * [31] Y. Zeng, X. Xu, Path design for cellular-connected uav with reinforcement learning, in: 2019 IEEE Global Communications Conference (GLOBECOM), IEEE, 2019, pp. 1–6.
  * [32] A. Shamsoshoara, M. Khaledi, F. Afghah, A. Razi, J. Ashdown, Distributed cooperative spectrum sharing in uav networks using multi-agent reinforcement learning, in: 2019 16th IEEE Annual Consumer Communications & Networking Conference (CCNC), IEEE, 2019, pp. 1–6.
  * [33] A. Shamsoshoara, M. Khaledi, F. Afghah, A. Razi, J. Ashdown, K. Turck, A solution for dynamic spectrum management in mission-critical uav networks, in: 2019 16th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON), IEEE, 2019, pp. 1–6.
  * [34] A. Shamsoshoara, F. Afghah, A. Razi, S. Mousavi, J. Ashdown, K. Turk, An autonomous spectrum management scheme for unmanned aerial vehicle networks in disaster relief operations, IEEE Access 8 (2020) 58064–58079.
  * [35] U. Challita, W. Saad, C. Bettstetter, Interference management for cellular-connected uavs: A deep reinforcement learning approach, IEEE Transactions on Wireless Communications 18 (4) (2019) 2125–2140.
  * [36] G. F. Riley, T. R. Henderson, The ns-3 network simulator, in: Modeling and tools for network simulation, Springer, 2010, pp. 15–34.
  * [37] I. Gomez-Miguelez, A. Garcia-Saavedra, P. D. Sutton, P. Serrano, C. Cano, D. J. Leith, srslte: An open-source platform for lte evolution and experimentation, in: Proceedings of the Tenth ACM International Workshop on Wireless Network Testbeds, Experimental Evaluation, and Characterization, 2016, pp. 25–32.
  * [38] N. Nikaein, M. K. Marina, S. Manickam, A. Dawson, R. Knopp, C. Bonnet, Openairinterface: A flexible platform for 5g research, ACM SIGCOMM Computer Communication Review 44 (5) (2014) 33–38.
  * [39] A. Shamsoshoara, F. Afghah, E. Blasch, J. Ashdown, M. Bennis, Uav-assisted communication in remote disaster areas using imitation learning, IEEE Open Journal of the Communications Society (2021).
  * [40] P. Abbeel, A. Y. Ng, Apprenticeship learning via inverse reinforcement learning, in: Proceedings of the twenty-first international conference on Machine learning, 2004, p. 1.
  * [41] Q. Wu, Y. Zeng, R. Zhang, Joint trajectory and communication design for multi-uav enabled wireless networks, IEEE Transactions on Wireless Communications 17 (3) (2018) 2109–2121.
  * [42] A. Al-Hourani, S. Kandeepan, S. Lardner, Optimal LAP altitude for maximum coverage, IEEE Wireless Communications Letters 3 (6) (2014) 569–572.
  * [43] M. Bain, C. Sammut, A framework for behavioural cloning., in: Machine Intelligence 15, 1995, pp. 103–129.
  * [44] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, J. Peters, et al., An algorithmic perspective on imitation learning, Foundations and Trends® in Robotics 7 (1-2) (2018) 1–179.
  * [45] S. Adams, T. Cody, P. A. Beling, A survey of inverse reinforcement learning, Artificial Intelligence Review (2022) 1–40.
  * [46] S. Ross, G. Gordon, D. Bagnell, A reduction of imitation learning and structured prediction to no-regret online learning, in: Proceedings of the fourteenth international conference on artificial intelligence and statistics, JMLR Workshop and Conference Proceedings, 2011, pp. 627–635.
  * [47] A. Y. Ng, S. J. Russell, et al., Algorithms for inverse reinforcement learning., in: Icml, Vol. 1, 2000, p. 2.
  * [48] N. D. Ratliff, J. A. Bagnell, M. A. Zinkevich, Maximum margin planning, in: Proceedings of the 23rd international conference on Machine learning, 2006, pp. 729–736.
  * [49] M. Wulfmeier, P. Ondruska, I. Posner, Maximum entropy deep inverse reinforcement learning, arXiv preprint arXiv:1507.04888 (2015).
  * [50] Z. Zhou, M. Bloem, N. Bambos, Infinite time horizon maximum causal entropy inverse reinforcement learning, IEEE Transactions on Automatic Control 63 (9) (2017) 2787–2802.
  * [51] S. Diamond, S. Boyd, CVXPY: A Python-embedded modeling language for convex optimization, Journal of Machine Learning Research 17 (83) (2016) 1–5.
  * [52] Cvxopt, convex optimization for python, https://cvxopt.org/, (Accessed on 04/07/2021).
  * [53] R. S. Sutton, A. G. Barto, Reinforcement learning: An introduction, MIT press, 2018\.
  * [54] S. Levine, A. Kumar, G. Tucker, J. Fu, Offline reinforcement learning: Tutorial, review, and perspectives on open problems, arXiv preprint arXiv:2005.01643 (2020).
  * [55] Y. Shoham, R. Powers, T. Grenager, Multi-agent reinforcement learning: a critical survey, Tech. rep., Technical report, Stanford University (2003).
  * [56] L. Busoniu, R. Babuska, B. De Schutter, Multi-agent reinforcement learning: A survey, in: 2006 9th International Conference on Control, Automation, Robotics and Vision, IEEE, 2006, pp. 1–6.
  * [57] Y. Li, Y. Yuan, Convergence analysis of two-layer neural networks with relu activation, in: Advances in neural information processing systems, 2017, pp. 597–607.
  * [58] D. M. Allen, Mean square error of prediction as a criterion for selecting variables, Technometrics 13 (3) (1971) 469–475.
  * [59] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014).
  * [60] F. Torabi, G. Warnell, P. Stone, Behavioral cloning from observation, arXiv preprint arXiv:1805.01954 (2018).
  * [61] L. Breiman, J. Friedman, C. J. Stone, R. A. Olshen, Classification and regression trees, CRC press, 1984.
  * [62] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825–2830.
  * [63] A. Shamsoshoara, Apprenticeship learning using deep inverse reinforcement learning code, https://github.com/AlirezaShamsoshoara/Inverse-RL-Apprenticeship-learning-UAV-Communication (2021).
  * [64] A. Shamsoshoara, Apprenticeship learning via inverse RL for a cellular-connected UAV, https://youtu.be/FGAlHaTQ_nc, wireless Networking & Information Processing (WINIP) LAB, accessed on 05/19/2021 (2021).
  * [65] M. M. U. Chowdhury, S. J. Maeng, E. Bulut, I. Güvenç, 3-d trajectory optimization in uav-assisted cellular networks considering antenna radiation pattern and backhaul constraint, IEEE Transactions on Aerospace and Electronic Systems 56 (5) (2020) 3735–3750.
  * [66] M. M. U. Chowdhury, C. K. Anjinappa, I. Guvenc, M. Sichitiu, O. Ozdemir, U. Bhattacherjee, R. Dutta, V. Marojevic, B. Floyd, A taxonomy and survey on experimentation scenarios for aerial advanced wireless testbed platforms, in: 2021 IEEE Aerospace Conference (50100), IEEE, 2021, pp. 1–20.