# Deep UAV Path Planning with Assured Connectivity in Dense Urban Setting
††thanks: This work was supported in part by IITP grant funded by the Korean
government (MSIT) under IITP-2024-2020-0-01821(40%), IITP-2021-0-02068(30%),
and IITP-2019-0-00421(30%).

Jiyong Oh  Dept. of Artificial Intelligence, Sungkyunkwan University, Suwon,
Korea. Syed M. Raza  Dept. of Electrical and Computer Engineering,
Sungkyunkwan University, Suwon, Korea. 0000-0001-6580-3232 Lusungu J. Mwasinga
Dept. of Computer Science and Engineering, Sungkyunkwan University, Suwon,
Korea. Moonseong Kim  Dept. of IT Convergence Software, Seoul Theological
University, Bucheon, Korea. Hyunseung Choo  0000-0002-6485-3155

###### Abstract

Unmanned Ariel Vehicle (UAV) services with 5G connectivity is an emerging
field with numerous applications. Operator-controlled UAV flights and manual
static flight configurations are major limitations for the wide adoption of
scalability of UAV services. Several services depend on excellent UAV
connectivity with a cellular network and maintaining it is challenging in
predetermined flight paths. This paper addresses these limitations by
proposing a Deep Reinforcement Learning (DRL) framework for UAV path planning
with assured connectivity (DUPAC). During UAV flight, DUPAC determines the
best route from a defined source to the destination in terms of distance and
signal quality. The viability and performance of DUPAC are evaluated under
simulated real-world urban scenarios using the Unity framework. The results
confirm that DUPAC achieves an autonomous UAV flight path similar to base
method with only 2% increment while maintaining an average 9% better
connection quality throughout the flight.

###### Index Terms:

UAV flight planning, signal quality, Deep Reinforcement Learning (DRL), Unity
framework

##  I Introduction

Unmanned aerial vehicles (UAVs) are emerging as a new service paradigm in
various areas including package delivery, rescue efforts, and agriculture [1].
This trend is expected to expand, encompassing a broader range of services
that utilize several UAVs [2]. It is not feasible in terms of service
flexibility, scalability, and maintenance to manually configure UAV flight
paths or hire a control operator for each UAV [3]. This necessitates
autonomous and effective UAV flight path planning and uninterrupted
connectivity as they depend on batteries and communications for operation and
control, respectively [4]. Effective flight path planning must account for the
surrounding complex 3D environment and obstacles to determine the optimal
flight path, whereas uninterrupted connectivity facilitates precise control of
UAVs, allowing for real-time flight optimization [5]. However, maintaining an
uninterrupted excellent connection conflicts with the ideal minimum flight
path, as there is no guarantee of consistent signal strength on the optimal
route. Consequently, there is a trade-off between these two critical UAV
requirements, making their balanced management a significant challenge.

Existing studies have proposed various techniques for addressing UAV path
planning and connectivity issues. The study in [6] employs graph theory to
design an optimal UAV path that minimizes flight distance and further uses
dynamic programming for UAV path planning. Dijkstra’s algorithm has been
employed to optimize the trajectories of cellular-connected UAVs [7]. These
methods face challenges when dealing with dynamic environment changes, as they
rely on predefined settings. Recently, Deep Reinforcement Learning (DRL) has
shown a capability to handle dynamic environmental changes through continuous
learning in complex settings. DRL methods like DQN and multi-step dueling DDQN
have been used to optimize UAV flight time and reduce expected communication
interruptions in [8, 9], respectively. However, these studies use discrete
grid points to determine the next position of a UAV during the flight which
sacrifices the precise maneuvering of a UAV in a dense urban environment and
increases risk of collisions. Moreover, it focuses only on maintaining
connectivity at UAV and does not take the quality of connection into account.
These substantial limitations underscore the necessity for the development of
a flexible and robust UAV path-planning approach while maintaining low delay
and high-quality network connectivity.

This study addresses limitations in existing studies by proposing a Deep
Reinforcement Learning Approach for UAV Path Planning with Assured
Connectivity (DUPAC). DUPAC is designed to achieve a balance between
maintaining stable communication links and optimizing flight efficiency. It
achieves excellent and seamless connectivity by executing handovers to the
most appropriate Ground Base Stations (GBSs) at opportune moments using
Reference Signal Received Power (RSRP). The flight path is determined by
measuring the distance to the destination in each step and if the distance has
increased compared to the previous step DUPAC adjusts the UAV direction and
learns not to repeat the action that caused the distance increment. The
evaluation of DUPAC is done in 3D dense urban environment implemented in the
Unity framework, where GBSs mimic real-world deployments with coverage area
defined using an antenna model [9], and a path loss model determines RSRP at
the UAV. The preliminary results confirm the efficacy of DUPAC as it achieves
5% better RSRP on average while keeping flight distance similar to a base
method where DRL focuses only on minimizing flight distance. In summary, the
main contributions of this study are:

  * •

DUPAC framework that integrates connectivity assurance into UAV path planning,
and achieves a delicate balance between stable communication at the UAV and
minimum flight distance to the given destination.

  * •

A transformation of action space from discrete to continuous, which not only
enhances the decision-making capabilities in DRL but also significantly
improves the precision and flexibility of UAV movements.

  * •

Implementation of the real-world 3D urban environment which enables DUPAC
evaluation under realistic settings and provides a foundation to extend this
study and develop complex UAV systems.

##  II System Model

An urban environment model consists of high-density arrangement of variable-
height buildings, where a flight for UAV $u$ starts from source to destination
position. The flight is mapped in 3D coordinate system and at time $t$ the
position of $u$ is $u^{p}_{t}=[x_{t},y_{t},z_{t}]$, where the altitude $z_{t}$
value is in the range $z_{t}\in[z_{\text{m}in},z_{\text{m}ax}]$ as defined by
Federal Aviation Administration (FAA) regulations [10]. From the source, $u$
determines its 3D flight path with an average speed of 75 Km/h and must reach
the destination within time limit $T$ such that ${u^{p}_{t}:0\leq t\leq T}$.
The network in the urban environment is modeled with $M$ GBSs, where each GBS
$g_{m}$ is deployed using data from CellMapper which is a source for global
mobile network GBS locations [11].

The coverage footprint of the GBSs is determined using the antenna radiation
model. Each GBS is divided into three sectors, each equipped with a uniform
linear array consisting of eight elements vertically arranged. These elements
are designed to provide high directional gain. The radiation pattern of each
element $E$ is a combination of horizontal $A_{(E,H)}$and vertical $A_{(E,V)}$
radiation patterns. By integrating the vertical and horizontal radiation
patterns, we can derive the 3D antenna element gain $A_{E}(\theta,\phi)$ for
each pair of angles.

$A_{E}(\theta,\phi)=G_{E,\text{max}}-\min\left\\{-\left[A_{(E,V)}(\theta)+A_{(E,H)}(\phi)\right],A_{m}\right\\}$
(1)

The maximum directional gain $G_{E,\text{max}}$ of each antenna element is
$8\,\text{dBi}$. This provides the dB increase experienced by a beam with a
pair of angles $(\theta,\phi)$ due to the effect of the element radiation
pattern. A comprehensive antenna array radiation pattern $A_{A}(\theta,\phi)$
is represented as a combination of the single-element $A_{E}(\theta,\phi)$
radiation pattern and array factor $AF(\theta,\phi,n)$ [9].

$A_{A}(\theta,\phi)=A_{E}(\theta,\phi)+AF(\theta,\phi,n),$ (2)

where $n$ is the number of antenna elements. Using (2), total antenna gain at
a location is calculated for the UAV.

Figure 1: Proximal Policy Optimization (PPO) framework for the proposed UAV
Path Planning with Assured Connectivity.

The RSRP at the UAV is calculated using path loss model that consists of Urban
Micro $\omega_{\text{micro}}$ and Urban Macro $\omega_{\text{macro}}$ models
specified by 3GPP [12]. In $\omega_{\text{micro}}$, GBS antenna is installed
below the rooftop height, whereas in $\omega_{\text{macro}}$ GBS antenna is
installed above the rooftop height of the surrounding buildings. The RSRP at
$u$ from GBS $g_{m}$ at time $t$ is a function of position $u^{p}_{t}$ and is
calculated as:

$\text{RSRP}_{g_{m}}(u^{p}_{t})=P_{g_{m}}+G_{g_{m}}(u^{p}_{t})-L_{g_{m}}(u^{p}_{t}),$
(3)

where $P_{g_{m}}$ is the constant reference signal power.
$G_{g_{m}}(u^{p}_{t})$ and $L_{g_{m}}(u^{p}_{t})$ are GBS antenna gain and the
path loss, respectively, and they are functions of location $u^{p}_{t}$. The
path loss $L_{g_{m}}(u^{p}_{t})$ is calculated for either line-of-sight (LoS)
or non-line-of-sight (NLoS) link between GBS $g_{m}$ and UAV position
$u^{p}_{t}$ at time $t$ as follows:

$L_{g_{m}}^{\text{LoS}}(u^{p}_{t})=\begin{cases}\omega_{1}&\text{if
}\omega_{\text{micro}},1.5\leq z_{t}\leq 22.5\\\
\max\left\\{L_{g_{m}}^{\text{FSPL}},\omega_{2}\right\\}&\text{if
}\omega_{\text{micro}},22.5<z_{t}\leq 300\\\ \omega_{3}&\text{if
}\omega_{\text{macro}},1.5\leq z_{t}\leq 22.5\\\ \omega_{4}&\text{if
}\omega_{\text{macro}},22.5<z_{t}\leq 300\end{cases}$ (4)

where $L_{g_{m}}^{\text{FSPL}}$ is a free-space path loss between $u^{p}_{t}$
and $g_{m}$, $\omega_{1}=32.4+21\log d_{g_{m}}(u^{p}_{t})+20\log f_{c}$,
$\omega_{2}=30.9+(22.25-0.5\log z_{t})\log d_{g_{m}}(u^{p}_{t})+20\log f_{c}$,
$\omega_{3}=32.4+20\log d_{g_{m}}(u^{p}_{t})+20\log f_{c}$, and
$\omega_{4}=28+22\log d_{g_{m}}(u^{p}_{t})+20\log f_{c}$. Here, $f_{c}$ is the
frequency, and 3D distance between $u^{p}_{t}$ and $g_{m}$ at $t$ is
represented by $d_{g_{m}}(u^{p}_{t})$ .

$L_{g_{m}}^{\text{NLoS}}(u^{p}_{t})=\begin{cases}\max\left\\{L_{g_{m}}^{\text{LoS}},\omega_{5}\right\\}&\text{if
}\omega_{\text{micro}},1.5\leq z_{t}\leq 22.5\\\
\max\left\\{L_{g_{m}}^{\text{LoS}},\omega_{6}\right\\}&\text{if
}\omega_{\text{micro}},22.5<z_{t}\leq 300\\\
\max\left\\{L_{g_{m}}^{\text{LoS}},\omega_{7}\right\\}&\text{if
}\omega_{\text{macro}},1.5\leq z_{t}\leq 22.5\\\
\left\\{\omega_{8}\right\\}&\text{if }\omega_{\text{macro}},22.5<z_{t}\leq
100\end{cases}$ (5)

where $\omega_{5}=22.4+35.3\log d_{g_{m}}(u^{p}_{t})+21.3\log
f_{c}-0.3(z_{t}-1.5)$, $\omega_{6}=32.4+(43.2-7.6\log z_{t})\log
d_{g_{m}}(u^{p}_{t})+20\log f_{c}$, $\omega_{7}=13.54+39.08\log
d_{g_{m}}(u^{p}_{t})+20\log f_{c}-0.6(z_{t}-1.5)$, and
$\omega_{8}=-17.5+(46-7\log z_{t})\log d_{g_{m}}(u^{p}_{t})+20\log(\frac{40\pi
f_{c}}{3})$.

Figure 2: Snapshot of the simulated 3D dense urban environment of New York
City, USA, and RSRP coverage heatmap for multiple altitudes caused by 30 base
Stations.

##  III DRL-Driven UAV Path Planning with Assured Connectivity (DUPAC)

The proposed DUPAC effectively designs a UAV flight path that ensures
uninterrupted connectivity between the UAV and GBSs throughout its flight. It
exploits DRL to design an optimal path for UAV by solving sequential decision-
making problems that are formulated within the context of a Markov Decision
Process (MDP). It consists of an agent that continuously interacts with its
environment by collecting states, applying actions, and calculating rewards to
learn an optimal decision policy. At time $t$, the agent observes environment
state $s_{t}$, decides to perform action $a_{t}$, transforming the environment
to next state $s_{t+1}$. The agent calculates the reward $r_{t}$ based on
$s_{t+1}$ resulted from $a_{t}$ performed in $s_{t}$. The objective of the
agent is to maximize the accumulative discounted reward over all time steps.

State: The environment state, $s_{t}$, at time $t$ consists of UAV 3D position
$u^{p}_{t}=(x_{t},y_{t},z_{t})$, velocity vector $\vec{v}_{t}$, the distance
$c_{t}$ between $u^{p}_{t}$ and obstacles, the direction $\theta_{t}$ formed
by the UAV and obstacles, and the ID of the UAV currently associated GBS
$n_{t}$.

$s_{t}=(u^{p}_{t},\vec{v}_{t},c_{t},\theta_{t},n_{t})$ (6)

Action: The action $a_{t}$ is composed of 3D movement of the UAV and ID of
associated GBS at $t+1$. For the 3D movement, new values for $x_{t+1}$,
$y_{t+1}$, and $z_{t+1}$ are calculated by adding values between [-1,1] to
$x_{t}$, $y_{t}$, and $z_{t}$. A GBS $g_{m}$ from total $M$ GBSs is selected
as the associated GBS $n_{t+1}$. If $n_{t}$ and $n_{t+1}$ are the same then
the UAV continues its association with $n_{t}$ otherwise it handovers to
$n_{t+1}$.

$a_{t}=(u^{p}_{t+1},n_{t+1})$ (7)

Reward: The reward $r_{t}$ is calculated as follows for action $a_{t}$
executed on the state $s_{t}$.

$r_{t}=\begin{cases}\mu_{1}(d_{t-1}-d_{t}),&\text{if }C1\\\
\mu_{1}(d_{t-1}-d_{t})+\mu_{2}w_{n_{t}}^{RSRP},&\text{if }C2,\\\
\mu_{3}w_{n_{t}}^{RSRP},&\text{otherwise}\end{cases}$ (8)

where $d_{t}$ is the distance from current UAV position $u^{p}_{t}$ to its
destination, $w_{n_{t}}^{RSRP}$ is the RSRP at $u$ at $t$. $\mu_{1}$,
$\mu_{2}$, and $\mu_{3}$ are the weights. Moreover, condition $C1$ represents
excellent RSRP as (-80$\leq$RSRP), condition $C2$ represents mediocre RSRP as
(-100$\leq$RSRP$<$-80), and otherwise RSRP at $u$ is poor (RSRP$<$-100).
Differentiated learning of DUPAC for varying RSRP ranges allows it to
distinctly control the UAV flight path for specific connectivity conditions.

This study utilizes Proximal Policy Optimization (PPO), an on-policy DRL
algorithm, to learn the aforementioned DUPAC framework. Trajectory memory in
PPO stores the UAV state and utilize them to train policy and value networks,
as shown in Fig. 1. An advantage value $v_{t}$ is calculated using calculated
reward $r_{t}$ and estimated reward from value network for the state $s_{t}$.
Mean Squared Error (MSE) over $v_{t}$ is calculated to train the value network
by reducing the error between estimated rewards at $s_{t}$ and $s_{t+1}$ with
discount rate $\gamma$ [13].

The training of policy network in PPO is conducted through a modified
advantage value function $v^{GAE}$ which operates as a Generalized Advantage
Estimator (GAE) to reduce the variance of estimated rewards. In addition to
discount rate $\gamma$ in $v_{t}$, a variable $\lambda$ is included in
$v^{GAE}$ for balancing a trade-off between bias and variance. In addition to
$v^{GAE}$, PPO uses parameter $\epsilon$ and clip function for curtailing the
variance of estimated rewards and gradually updating the policy network [13].
The clip function provides a probability ratio between $1-\epsilon$ and
$1+\epsilon$, where the probability ratio is for every action by new and old
stochastic policies in $s_{t}$. This enables the curtailment of gradient size
and training of policy network with gradual learning and higher convergence
rate.

Figure 3: DUPAC impact on UAV flight distance and RSRP trade-off compared to
base method.

##  IV Performance Evaluation

###  IV-A Experimental setup

The evaluation of the proposed DUPAC is conducted in a dense urban environment
of New York City, USA. For this purpose, a 3D simulation of $\sim$2km2 area of
New York City around Times Square is developed in the Unity framework, as
shown in Fig. 2(a). As per real-world network deployments indicated in [11],
30 GBS are deployed in the locations mentioned in [11]. Through the
implementation of antenna gain and path loss models with 30 GBS, an RSRP
coverage map is obtained for different altitudes shown in Fig. 2(b) where
signal strength is poor near the ground due to congested building structures.
However, the signal strength improves with the altitude as the environment is
more open due to only few high rise buildings. DUPAC determines the UAV path
during the flight through distance measurements to the destination and ensures
the best possible connectivity via the RSRP coverage map. The evaluation of
DUPAC is done for short as well as long flight distances to confirm its
generalization capability for a variety of flight distances.

The preliminary viability of DUPAC is established by comparing it with a base
method. In the base method state and reward functions are the same as DUPAC
but the action function only provides the next position of UAV and the
selection of GBS is done via 3GPP standardized rule-based mechanism.
Comparison with other state-of-the-art methods requires their implementation
in our 3D simulated environment and is in the process of being included in the
extended article. The values used for $P_{g_{m}}$ and $f_{c}$ in the path loss
calculation are 15.2dBm and 2GHz, respectively. To control the UAV behaviors
for different RSRP conditions, the weights $\mu_{1}$, $\mu_{2}$, and $\mu_{3}$
in DUPAC and base method are set as 10, 0.01, and 0.1, respectively. Moreover,
the learning rate of PPO framework in DUPAC is set as 0.0003, the discount
rate $\gamma$ is 0.95, and the cut-off threshold between the old and new
policies is 0.2.

Figure 4: DUPAC effect on RSRP distribution during UAV flight compared to base
method.

###  IV-B Results

The most important aspect of UAV flight planning is that UAV reaches the
destination within the shortest possible distance to minimize energy
consumption and elongate flight capability. To this end, UAV flight distances
with DUPAC and base method are evaluated when manually calculated shortest
distance between source and destination is increased from 200m to 800m. The
results in Fig. 3 show that DUPAC covers 14% more distance for the longest
flight (800m) which is only 2% more than the base method. On average DUPAC
cover 11% distance and base method covers 9% more distance compared to
shortest possible path to the destination. This is because DRL in the base
method optimizes its policy only to minimize the distance, whereas DUPAC
achieves a trade-off between the UAV flight distance and connection quality by
achieving on average 7.5% better RSRP during the UAV flight for all distances.
The trade-off of distance and signal quality in DUPAC can be tweaked by
adjusting the weights $\mu_{1}$, $\mu_{2}$, and $\mu_{3}$, and operators
benefit from this flexibility by altering the primary objective of DUPAC as
per service requirements.

Consistent excellent/good communication signal is critical for most of the UAV
services like surveillance, videography, and rescue operations. The results in
Fig. 4 evaluate this aspect by illustrating the distribution of RSRP at UAV
throughout its flight in DUPAC and base method. It is clear that DUPAC
achieves excellent RSRP for longer periods during the flight than base method.
In particular, DUPAC maintains excellent RSRP on UAV for maximum 11% and on
average 9% longer periods during the flight compared to the base method.
Consequently, UAV with DUPAC has fair RSRP for a very short period during the
flights for all distances, and it does not suffer with poor signal quality
during flights except for 0.5% flight duration in case of 800m distance. The
enhanced signal stability from DUPAC at extended ranges improves communication
reliability for UAV and reduces communication failures. The result also
reveals that as the distance between source and destination increases the
flight period with excellent RSRP decreases for both DUPAC and base method,
however, the rate of decrement is more for base method than DUPAC. From these
results it can be concluded that the distance and signal quality trade-off
policy in DUPAC successfully achieves a balance between flight distance and
RSRP as it maintains excellent RSRP at UAV for longer periods at the cost of
slight increase in flight distance compared to base method.

##  V Conclusion and Future Directions

This paper presents a DUPAC framework that uses on-policy DRL algorithm for
achieving a trade-off between UAV path planning and signal quality for
uninterrupted high quality communication. A generic 3D dense urban environment
implemented in the Unity framework with network modeling enables evaluation of
DUPAC in realistic settings and sets foundation for other UAV related studies.
The comparative analysis with the base method confirms the efficacy of DUPAC
in achieving the trade-off between UAV path planning and signal quality, as it
achieves on average 7.5% better signal quality with only 2% increment in
flight distance. The differentiated reward function for varying RSRP ranges
enables DUPAC to keep UAV in excellent RSRP regions for on average 9% longer
period during the flights than the base method. This leads to improved data
transmission quality and lower signal loss risk, thereby enhancing the
reliability for mission-critical UAV operations. The in-depth evaluation of
DUPAC for diverse urban environments is on-going to be included in the
extended article along with the comparison with the state-of-the-art.
Moreover, we are adding handover reduction functionality in DUPAC to minimize
connectivity disruptions.

## References

  * [1] F. Giones and A. Brem, “From toys to tools: The co-evolution of technological and entrepreneurial developments in the drone industry,” _Business Horizons_ , vol. 60, no. 6, pp. 875–884, 2017.
  * [2] N.-N. Dao, Q.-V. Pham, N. H. Tu, T. T. Thanh, V. N. Q. Bao, D. S. Lakew, and S. Cho, “Survey on aerial radio access networks: Toward a comprehensive 6g access infrastructure,” _IEEE Communications Surveys & Tutorials_, vol. 23, no. 2, pp. 1193–1225, 2021.
  * [3] G. Geraci, A. Garcia-Rodriguez, M. M. Azari, A. Lozano, M. Mezzavilla, S. Chatzinotas, Y. Chen, S. Rangan, and M. Di Renzo, “What will the future of uav cellular communications be? a flight from 5g to 6g,” _IEEE communications surveys & tutorials_, vol. 24, no. 3, pp. 1304–1335, 2022.
  * [4] L. Zhu, J. Zhang, Z. Xiao, X. Cao, X.-G. Xia, and R. Schober, “Millimeter-wave full-duplex uav relay: Joint positioning, beamforming, and power control,” _IEEE Journal on Selected Areas in Communications_ , vol. 38, no. 9, pp. 2057–2073, 2020.
  * [5] Y. Zeng, J. Lyu, and R. Zhang, “Cellular-connected uav: Potential, challenges, and promising technologies,” _IEEE Wireless Communications_ , vol. 26, no. 1, pp. 120–127, 2018.
  * [6] H. Yang, J. Zhang, S. Song, and K. B. Lataief, “Connectivity-aware uav path planning with aerial coverage maps,” in _IEEE Wireless Communications and Networking Conference (WCNC)_ , 2019, pp. 1–6.
  * [7] S. De Bast, E. Vinogradov, and S. Pollin, “Cellular coverage-aware path planning for uavs,” in _2019 IEEE 20th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC)_ , 2019, pp. 1–5.
  * [8] G. Fontanesi, A. Zhu, M. Arvaneh, and H. Ahmadi, “A transfer learning approach for uav path design with connectivity outage constraint,” _IEEE Internet of Things Journal_ , vol. 10, no. 6, pp. 4998–5012, 2022.
  * [9] H. Xie, D. Yang, L. Xiao, and J. Lyu, “Connectivity-aware 3d uav path design with deep reinforcement learning,” _IEEE Transactions on Vehicular Technology_ , vol. 70, no. 12, pp. 13 022–13 034, 2021.
  * [10] L. F. Dorr, “Federal aviation administration (faa) small unmanned aircraft rule part 107,” 2016.
  * [11] Cellmapper. (2023) Cellular network coverage maps. [Online]. Available: https://www.cellmapper.net/map?MCC=311&MNC=480&type=LTE&latitude=40.76375609047827&longitude=-73.97709096997261&zoom=17.798046841204
  * [12] 3GPP, “Radio access network: Study on enhanced lte support for aerial vehicles,” _Technical Specification (TS) 36.777_ , 2017.
  * [13] Y. Jang, S. M. Raza, M. Kim, and H. Choo, “Proactive handover decision for uavs with deep reinforcement learning,” _Sensors_ , vol. 22, no. 3, 2022.