Each of the UVs/relays multiplies its received signal with weight $w_{i}[k]$ and all UVs collectively transmit their signals to the destination using a collective power budget $P_{I}$. The channel from each UV to the destination, $\mathcal{H}_{iD}$, depends  on the UV's position and the time instance. The received signal at the destination $x_{D}$ is a function of all channels (from the source to the UV and from the UV to the destination) and all relay positions:

\begin{equation}
x_{D}[k] = \mathcal{F}\left(\{ q_{i}[k]\}_{i=1}^{I}, \{\mathcal{H}_{Si} \}_{i=1}^{I}, \{\mathcal{H}_{iD} \}_{i=1}^{I} ,k\right),
\end{equation}

where $\{\mathcal{H}_{Si} \}_{i=1}^{I}$ denotes the channels from the source to all deployed UVs/relays and $\{\mathcal{H}_{iD} \}_{i=1}^{I}$ the channels from all relays to the destination.  Noise is induced both at the signal reception at each UV and at the signal reception at the destination. Therefore, the destination signal can be decomposed as:

\begin{equation}
x_{D}[k] = x_{D}^{signal}[k] + x_{D}^{noise}[k],
\end{equation}

Under the assumption that the channels (both from the source to each UV and from each UV to the destination) exhibit correlations with respect to time and space, the problem can be formulated as:
\beginmaxi!|s|[2]
𝔼_𝒬[∑_k=0^∞𝒰 ( {ℋ_Si[k] }_i=1^I, {ℋ_iD[k] }_i=1^I)]

q_i[0]=q_i0  ∀i
||q_i[k+1] - q_i[k]||_2≤v_max Δ_k  ∀i,k
𝔼[||x_i [q_i[k], k]||^2] ≤P_I

\endmaxi!

The set of all infinite duration permissible trajectories of UVs is denoted as $\mathcal{Q}$. $\mathcal{U}$ is a general utility function of the system that depends on the realization of the channels and therefore on the corresponding relay positions for all time steps of operation. The most typical choice for the utility is the SINR at the destination, which is expressed as:

\begin{equation}
\mathcal{U}[k] \equiv \text{SINR}[k] = \frac{|x_{D}^{signal}[k]|^2}{|x_{D}^{noise}[k]|^2},
\end{equation}

Besides the SINR, there exists a plethora of other candidate utility metrics, such as the sum-rate (for multiple receivers) and the secrecy rate (refer to Section <ref>). The choice of the utility function introduces different nuances that are domain-specific and have to be explicitly addressed (e.g., additional interference when considering multiple destinations).
The above formulation paves the way for DRL approaches to be adopted. The works [152, 217, 218] consider multiple UVs as relays facilitating the communication of a single source-destination pair. The space over which the UVs can move is discretized into 2D grid cells, and various variations of the deep Q-learning approach [117, 129] are proposed to solve the problem. In those works, the authors noted the inability of Multilayer Perceptrons (MLPs) with ReLU activations to capture the high frequency spectral components of the optimal value function, which arise due to the pronounced variability of the underlying channels with respect to time and space. It was also noted that by preprocessing the input of the state of the corresponding MDP with a learnable Fourier kernel, those issues can be mitigated, leading to an increase in convergence speed and average SNR accumulation at the receiver.
In [153], the assumption for space discretization is relaxed and  a variation of the Soft Actor-Critic [124] algorithm for continuous motion control is proposed.

The works [75, 147] also examine the aforementioned problem and propose a stochastic programming approach to solve it. The approach is data-driven and optimizes the SNR in expectation for the subsequent time step of system operation. In [219], the authors perform a comparative analysis of the performance of the DRL approach of [152] and the stochastic programming approach of [75]. Finally, the work [128] considers the same problem and develops a multi-armed bandit method to tackle it.

The work [159] studies a scenario where a single UV serves multiple users as a relay. Multiple model-free RL algorithms are proposed for UV motion control with the goal of maximizing energy efficiency under quality of service constraints. The proposed RL approaches are shown to surpass the performance of a connected hybrid precoding approach. The work of [220] examines scenarios where a UAV serves as a mobile relay in order to maintain the communication between multiple destinations (end users) and a BS. Both model-free and model-based RL approaches are proposed for the same setup. In [155], the authors examine a scenario where one UAV acts as a relay and  propose an on-policy DRL approach for trajectory design with the goal being to optimize the average throughput of the collective system. The DRL approach is a proximal policy optimization method [121]. The proposed algorithm accounts for the molecular absorption effect that is prominent in THz communications [221].

The work of [222] examines a communication set up where multiple UVs serve as relays in order to facilitate the communication between a BS and multiple end users. A deep Q-learning approach with a dueling architecture of the Q-network [129] is proposed in order to optimize the UV locations over time. The goal is to design the trajectories of all deployed UVs in order to maximize the downlink capacity of the network under the constraint that all end users should be served by at least one relay at all times of system operation.

Both this subsection and the previous one offer insights and problem formulations concerning the intersection between DRL and UV motion control. The distinction between the two lies in the specific roles played by UVs in each scenario. In the first subsection, the UV operates as a provider, akin to a mobile BS, while in the second subsection, it acts as an intermediate mobile node facilitating communication between  source and destination points.

At a cursory glance, the formulations presented in Problem (<ref>) and the Problem of (<ref>) might appear similar. Both require the design of UV motion policies to maximize the expected sum of a QoS metric, contingent on the channel conditions, over an extended time horizon. Solving sequential decision-making problems using DRL hinges significantly on estimating the optimal value function of the corresponding MDP. In both cases, this value function depends on the underlying channels. When the UV serves as a provider, only the channel from the UV to the user impacts the value function. Conversely, when the UV acts as a relay or helper, the optimal value function is influenced by both the channels from the UV to the receiver and the channels from the source or BS to the UV. This additional dependency introduces localized variability in the true value function, making its accurate estimation through bootstrapping a challenging endeavor [127].

§.§.§ Unmanned Vehicles as Consumers

This subsection delves into the collaborative relationship between  DRL and UVs specifically when UVs serve as consumers of services within wireless networks. DRL is commonly used for UV  motion control or resource allocation, especially in “data harvesting" scenarios from IoT/sensor networks. The inherent mobility of UVs is advantageous for efficient data collection from dispersed sensors. However, challenges arise, requiring advanced algorithms to optimize UV motion for objectives like maximizing data throughput, operational time, data freshness, and minimizing energy consumption amidst environmental randomness.

A broad scenario that encompasses various settings of sequential decision-making for  UV-assisted data collection unfolds as follows: A single UV is tasked with collecting data from $M$ clusters of sensors, each cluster conveniently comprising the same number of $I$ sensors. The UV's position at each step is represented as $q[k] = \left[x[k], y[k], z[k]\right] \in \mathbb{R}^{3}$. The position of the $i$th sensor within the $m$th cluster at time step $t$ is denoted as $q_{i}^{m} = \left[ x_{i}^{m}[k], y_{i}^{m}[k]\right]$. The throughput from sensor $i$ within cluster $m$ depends on both the UV's position and the sensor's location, under particular assumptions for the underlying channel model: $R_{i,m}[k] = R_{i,m}[q[k], q_{i}^{m}[k]]$.

Considering the existence of a probability of link blockage between the UV and each sensor in every cluster, the sensor's throughput can be defined as $R_{i,m}[k] = P_{i,m}[k] R_{i,m}[q[k], q_{i}^{m}[k]]$, where $P_{i,m}[k] \in [0,1]$ signifies the probability of a lost connection between the UV and the $i$th user within the $m$th cluster at time step $k$. The overarching aim of the sequential decision-making framework is to construct the policy for the UV motion such that the overall throughput achieved throughout the mission is maximized while simultaneously the overall mission time is minimized.

On denoting the set of all permissible UV trajectories by $\mathcal{Q}$, the generic optimization problem can be defined as

\beginmaxi!|s|[2]
𝔼_Q [ ∑_k=0^K ∑_i^I ∑_m^M αR_i, m[q[k], q_i^m[k]] - βk]

\endmaxi!

where $\alpha$ and $\beta$ are scalars that manage the design tradeoff between the desire for overall sum-rate maximization and mission completion time minimization.

The work presented in [186] considers a scenario very similar to the one described above. The state is the position of the UV at each time step, the operational space is discretized into a fine grid and the action space is comprised by the displacement of the UV to one of the neighboring cells. The authors propose the adoption of the deep Q-learning algorithm to control the UV positions over time in order to optimize a linear combination of the overall sum-rate and the mission completion time. The study presented in [189] considers a scenario where multiple UVs perform both data collection and wireless energy transfer. In particular, each UV supplies energy to each one of the sensors. Subsequently, the sensors transmit their collected data to the UVs. However, when the same UV is used both for data collection and energy harvesting, both operations are negatively affected. The authors propose splitting the group of UVs into two categories, one that is solely responsible for energy harvesting and another that is solely responsible for data collection. The goal is to optimize the trajectories of both groups of UVs. Since the optimal policies of the two groups are correlated, the authors propose a multi-agent DRL algorithm in order to control the trajectories of the UVs of each category such that a linear combination of the overall throughput, the Age-of-Information, the overall harvested energy and the UV energy expenditure is optimized. The authors of [190] examine a challenge, particular to UV-assisted data collection in IoT systems, that pertains to the unsuccessful data transmission due to buffer overflow caused by the maneuvering of UVs. Since the state of the individual IoT nodes can be potentially obscure to the UV, the induced MDP is Partially Observable [223]. The authors propose a deep Q-learning-based approach for Partially Observable MDPs in order to control both the UV flight and the data scheduling with the goal of minimizing the overall packet loss.

A similar data collection problem is considered in [192], which additionally considers the data transfers from the UV to a remote station. The UV's trajectory is co-optimized with an adaptive transmit rate for wireless data transfer to a remote station, and an MDP is formulated, with a cost function which captures both the motion and communication energy of the UV, and a provably-optimal Monte-Carlo tree search (MCTS) method is proposed which treats the optimization of communication resources along the entire path as the final stage in the MDP, thus greatly reducing the size of the state space. In a related work, [188] examines the persistent dynamic relay scenario first studied in [187],  in which a single UV services a group of disparate source-destination pairs. The authors are then interested in finding the optimal robotic routing locations and dynamic routing policies. The authors show that the problem of minimizing the average wait time can be posed as an average-cost semi-Markov decision process, and propose using a variation of PPO to efficiently solve the problem, while showing how to handle a long-run average cost criterion.

§ ADDITIONAL CONSIDERATIONS

§.§ Security

The affordability of hardware and the high probability of LoS communication links make UVs appealing for wireless communications, yet they can also present vulnerabilities. This becomes particularly crucial in military applications, where UVs often carry sensitive information, making privacy and security paramount requirements of system design.

One challenge lies in the low-cost hardware mounted on UVs, which may struggle to run sophisticated encryption algorithms necessary for ensuring information security. As a result, UVs become susceptible to cybersecurity attacks. These threats may seek to disrupt access to critical services, as in Denial-of-Service (DoS) [224] and battery attacks [225], or they may compromise the privacy of sensitive information, as in spoofing attacks [226].

However, even if computational resources are available, traditional cryptographic approaches may be difficult to implement on a UV. This is because they rely on secret keys, which are difficult to distribute in a wireless network.
Physical layer security (PLS) approaches may thus be more applicable in such scenarios.
By exploiting the physical characteristics of the wireless channel, PLS design aims to enable the legitimate destination to obtain the source information successfully, while preventing an eavesdropper (ED), or unauthorized user from decoding the information [227]. PLS system design aims to maximize the secrecy rate, which is the rate the legitimate receiver receives minus the rate that the eavesdropper receives, subject to certain system constraints.
PLS design of  communication systems has been well investigated [228, 229, 230, 231, 232, 233, 234]. One approach to ensure PLS is cooperative jamming, where trusted relays act as helpers and beamform artificial noise (AN), aiming to degrade the ED's channel [228, 229, 230, 231, 232]. Another approach is for the source to beamform AN along with the intended information for users, in a way that the users do not experience interference [233, 234].
PLS design for DFRC systems has been considered in [235, 236], while the design of DFRC aided by an RIS is investigated in [237].

§.§ Edge Computing

UVs can benefit from [238] and contribute to edge computing and edge intelligence [239]. The effective functioning of UVs, both as consumers and providers of wireless connectivity, relies on the application of computationally intensive algorithms, such as vision algorithms for precise localization and mapping [240]. The traditional approach of offloading such operations to a central cloud server is impractical due to latency issues that exceed stringent requirements [238]. Therefore, leveraging edge servers near UVs allows for substantial storage and computation, striking a balance between latency and computational capabilities. Particularly in the realm of artificial intelligence on the edge, various research areas such as transfer learning [241], knowledge distillation [242], and neural network pruning/quantization [243] are emerging to facilitate processing on the network edge, in close network proximity to the UVs.

Conversely, UVs can serve as edge nodes where end users offload tasks [244]. The mobility of UVs introduces a dynamic edge computing environment, allowing them to approach users based on a flexible schedule. This dynamic interaction reduces latency and enhances throughput for end users in response to evolving network conditions. Additionally, edge computing brings indirect security benefits to UV operation. As UVs are equipped with onboard sensors such as cameras, they inherently collect data that is private to end users. Leveraging the paradigm of federated learning [245] alongside edge computing helps avoid the transfer and storage of sensitive information on central cloud servers. Instead, storage and computation are offloaded to edge nodes, which are less susceptible to information interception and, therefore, invasion of privacy.

§.§ Federated Learning

Undoubtedly, the integration of UVs into wireless networking is closely associated with the rise of deep learning methods [246]. These approaches prove valuable for optimizing components of the UV system configuration and harnessing the vast amount of data collected or generated by UVs. Deep learning paradigms, known for their data-intensive nature, demand substantial computational resources, typically provided by cloud servers. However, this gives rise to two significant concerns:
Firstly, transferring data from UVs to the central cloud introduces latency, which can be prohibitive for many applications. Secondly, the transmitted data may include sensitive information, posing privacy and security risks.

The federated learning paradigm [247] addresses these concerns by keeping data at the terminal device that generated or collected it [248]. In this approach, a local deep learning model is trained by the data, and only the parameters of the local model are transmitted to the central cloud center. This methodology helps mitigate privacy invasion for end-users. In addition, the transmission of model parameters induces less latency than the transmission of raw data. A plethora of different works leverage federated learning for UV-assisted applications. In [249], the authors examine distributed image classification for multi-UV exploration. The authors in [250]  propose a federated learning scheme that addresses nuanced privacy requirements in UV-assisted crowdsensing. Overall, utilizing federated learning for UV-assisted 6G networks remains an open and exciting future research direction.

§ CONCLUSIONS

The tech world eagerly anticipates the advent of the 6th Generation of wireless networking, promising transformative capabilities and applications. The success of this revolution, from closing of the digital divide to fueling the industrial sector, hinges on diverse technological advancements. Unmanned  Vehicles emerge as a pivotal technology, especially crucial for extended coverage and connectivity. While UVs enhance wireless infrastructure, they also introduce novel challenges requiring attention from academia and industry.

This paper offers a comprehensive view of settings and scenarios at the intersection of UVs and wireless networking, accompanied by an in-depth exploration of corresponding problem formulations and methods. To present this information effectively, the central thrust of the paper has been the unification of the problem space, providing a cohesive framework to understand the use cases, problem formulations, and necessary tools. Overall, the paper can equip researchers, engineers, and stakeholders with a clear understanding of both the opportunities and challenges inherent in the convergence of unmanned vehicles and 6G networks, paving the way towards a successful integration.

[1]
W. Jiang, B. Han, M. A. Habibi, and H. D. Schotten, “The road towards 6G: A
comprehensive survey,” IEEE Open Journal of the Communications
Society, vol. 2, pp. 334–366, 2021.

[2]
W. Saad, M. Bennis, and M. Chen, “A vision of 6G wireless systems:
Applications, trends, technologies, and open research problems,” IEEE
Network, vol. 34, no. 3, pp. 134–142, 2020.

[3]
M. Giordani, M. Polese, M. Mezzavilla, S. Rangan, and M. Zorzi, “Toward 6G
networks: Use cases and technologies,” IEEE Communications Magazine,
vol. 58, no. 3, pp. 55–61, 2020.

[4]
“Framework and overall objectives of the future development of IMT for 2030
and beyond,” ITU-R, Tech. Rep., 2023, draft New Recommendation.

[5]
A. Chaoub, M. Giordani, B. Lall, V. Bhatia, A. Kliks, L. Mendes, K. Rabie,
H. Saarnisaari, A. Singhal, N. Zhang, S. Dixit, and M. Zorzi, “6G for
bridging the digital divide: Wireless connectivity to remote areas,”
IEEE Wireless Communications, vol. 29, no. 1, pp. 160–168, 2022.

[6]
“6G: The Next Hyper-Connected Experience for All,” Samsung, Tech. Rep.,

[7]
T. S. Rappaport, Y. Xing, O. Kanhere, S. Ju, A. Madanayake, S. Mandal,
A. Alkhateeb, and G. C. Trichopoulos, “Wireless communications and
applications above 100 GHz: Opportunities and challenges for 6G and
beyond,” IEEE Access, vol. 7, pp. 78 729–78 757, 2019.

[8]
“The ITU-R Framework for IMT-2030,” ITU-R, Tech. Rep., 2023. [Online].

[9]
“Recommendation ITU-R M.2083-0: IMT Vision - Framework and overall objectives
of the future development of IMT for 2020 and byeond,” ITU-R, Tech. Rep.,

[10]
R. Liu, R. Y.-N. Li, M. D. Renzo, and L. Hanzo, “A vision and an evolutionary
framework for 6G: Scenarios, capabilities and enablers,” 2023.

[11]
C. de Alwis, Q.-V. Pham, and M. Liyanage, “Smart Cities and Society 5.0,”
in 6G Frontiers: Towards Future Wireless Systems, 2023, pp. 197–210.

[12]
S. Polymeni, S. Plastras, D. N. Skoutas, G. Kormentzas, and C. Skianis, “The
impact of 6G-IoT technologies on the development of agriculture 5.0: A
review,” Electronics, vol. 12, no. 12, 2023. [Online]. Available:

[13]
H. Shin, S. Park, L. Kim, J. Kim, T. Kim, Y. Song, and S. Lee, “The future
service scenarios of 6G telecommunications technology,”
Telecommunications Policy, p. 102678, 2023. [Online]. Available:

[14]
F. Liu, Y. Cui, C. Masouros, J. Xu, T. X. Han, Y. C. Eldar, and S. Buzzi,
“Integrated sensing and communications: Toward dual-functional wireless
networks for 6G and beyond,” IEEE Journal on Selected Areas in
Communications, vol. 40, no. 6, pp. 1728–1767, 2022.

[15]
P. R. Singh, V. K. Singh, R. Yadav, and S. N. Chaurasia, “6G networks for
artificial intelligence-enabled smart cities applications: A scoping
review,” Telematics and Informatics Reports, vol. 9, p. 100044, 2023.
[Online]. Available:

[16]
K. B. Letaief, W. Chen, Y. Shi, J. Zhang, and Y.-J. A. Zhang, “The roadmap to
6G: AI empowered wireless networks,” IEEE Communications
Magazine, vol. 57, no. 8, pp. 84–90, 2019.

[17]
H. Sun, C. Ng, Y. Huo, R. Q. Hu, N. Wang, C.-M. Chen, K. Vasudevan, J. Yang,
W. Montlouis, D. Ayanda, K. V. Mishra, K. Tekbıyık, N. Hussain, H. K.
Sahoo, and Y. Miao, “Massive MIMO,” in 2022 IEEE Future Networks
World Forum (FNWF), 2022, pp. 1–51.

[18]
A. Pallaprolu, W. Hurst, S. Paul, and Y. Mostofi, “I beg to diffract: RF
field programming with edges,” in Proceedings of the 29th Annual
International Conference on Mobile Computing and Networking.1em plus
0.5em minus 0.4emNew York, NY, USA: Association for Computing
Machinery, 2023. [Online]. Available:

[19]
E. Björnson, H. Wymeersch, B. Matthiesen, P. Popovski, L. Sanguinetti, and
E. de Carvalho, “Reconfigurable intelligent surfaces: A signal processing
perspective with wireless applications,” IEEE Signal Processing
Magazine, vol. 39, no. 2, pp. 135–158, 2022.

[20]
M. Di Renzo, M. Debbah, D.-T. Phan-Huy, A. Zappone, M.-S. Alouini, C. Yuen,
V. Sciancalepore, G. C. Alexandropoulos, J. Hoydis, H. Gacanin et al.,
“Smart radio environments empowered by reconfigurable AI meta-surfaces: An
idea whose time has come,” EURASIP Journal on Wireless Communications
and Networking, vol. 2019, no. 1, pp. 1–20, 2019.

[21]
M. Di Renzo and A. I. Aravanis, “Catching the 6G wave by using
metamaterials,” in Shaping Future 6G Networks: Needs, Impacts, and
Technologies, 2022, pp. 69–87.

[22]
U. Ghafoor, M. Ali, H. Z. Khan, A. M. Siddiqui, and M. Naeem, “Noma and future
5G & B5G wireless networks: A paradigm,” Journal of Network and
Computer Applications, vol. 204, p. 103413, 2022. [Online]. Available:

[23]
B. Clerckx, Y. Mao, E. A. Jorswieck, J. Yuan, D. J. Love, E. Erkip, and
D. Niyato, “A primer on rate-splitting multiple access: Tutorial, myths, and
frequently asked questions,” IEEE Journal on Selected Areas in
Communications, vol. 41, no. 5, pp. 1265–1308, 2023.

[24]
S. Stańczak, A. Keller, R. L. Cavalcante, N. Binder, and S. Velayutham,
“Long‐term Perspectives,” in Shaping Future 6G Networks: Needs,
Impacts, and Technologies, 2022, pp. 235–254.

[25]
Z. M. Miličević and Z. B. Bojković, “From the early days of
unmanned aerial vehicles (UAVS) to their integration into wireless
networks,” Vojnotehnički glasnik/Military Technical Courier,
vol. 69, no. 4, pp. 941–962, 2021.

[26]
A. Dogra, R. K. Jha, and S. Jain, “A survey on beyond 5g network with the
advent of 6G: Architecture and emerging technologies,” IEEE Access,
vol. 9, pp. 67 512–67 547, 2020.

[27]
R. Shrestha, R. Bajracharya, and S. Kim, “6G enabled unmanned aerial vehicle
traffic management: A perspective,” IEEE Access, vol. 9, pp.
91 119–91 136, 2021.

[28]
R. Bajracharya, R. Shrestha, S. Kim, and H. Jung, “6G nr-u based wireless
infrastructure UAV: Standardization, opportunities, challenges and future
scopes,” IEEE Access, vol. 10, pp. 30 536–30 555, 2022.

[29]
M. Mozaffari, X. Lin, and S. Hayes, “Toward 6G with connected sky: UAVs
and beyond,” IEEE Communications Magazine, vol. 59, no. 12, pp.
74–80, 2021.

[30]
E. B. Parker, “Closing the digital divide in rural america,”
Telecommunications Policy, vol. 24, no. 4, pp. 281–290, 2000.

[31]
G. Karabulut Kurt, M. G. Khoshkholgh, S. Alfattani, A. Ibrahim, T. S. J.
Darwish, M. S. Alam, H. Yanikomeroglu, and A. Yongacoglu, “A vision and
framework for the high altitude platform station (HAPS) networks of the
future,” IEEE Communications Surveys & Tutorials, vol. 23, no. 2,
pp. 729–779, 2021.

[32]
A. Weissberger, “AT&T introduces 5G flying COWs (Cell on Wings) drones,”
November 2023, [Online; posted 22-June-2022]. [Online]. Available:

[33]
S. Halawi, “Zephyr's next chapter,” Airbus HAPS Connectivity Solutions,
Tech. Rep., 2022,
https://mediaassets.airbus.com/pm_38_602_602492-ie1gtafyip.pdf [Online].

[34]
“3GPP Release 17 - Building Blocks for UAV Applications,” 3GPP, Tech.
Rep., 2022.

[35]
J. Stanczak, “NR Support for UAVs,” August 2023, [Online; posted
06-July-2023]. [Online]. Available:

[36]
“FAA unmanned aircraft systems integration pilot program,” U.S. DoT, FAA,
Tech. Rep., 2020.

[37]
“BEYOND,” August 2023, [Online]. [Online]. Available:

[38]
“Wing,” <https://wing.com>, accessed: 2023-11-30.

[39]
C. Chen, “11 photos of amazon’s new prime air drone that can fly in light
rain and deliver packages up to 5 pounds in under an hour,” 2023,

[40]
“Nuro,” <https://nuro.ai>, accessed: 2023-11-30.

[41]
S. Dresser, “Amazon announces 2 new ways it's using robots to assist employees
and deliver for customers,” 2022,

[42]
H. Shakhatreh, A. H. Sawalmeh, A. Al-Fuqaha, Z. Dou, E. Almaita, I. Khalil,
N. S. Othman, A. Khreishah, and M. Guizani, “Unmanned aerial vehicles
(UAVs): A survey on civil applications and key research challenges,”
IEEE Access, vol. 7, pp. 48 572–48 634, 2019.

[43]
“Autopilot and Full Self-Driving Capability,”
<https://www.tesla.com/support/autopilot>, accessed: 2023-11-30.

[44]
“Waymo,” <https://waymo.com>, accessed: 2023-11-30.

[45]
“Autonomous mobility and delivery at Uber,”
<https://www.uber.com/us/en/autonomous>, accessed: 2023-11-30.

[46]
Y. Mostofi, “Communication-aware motion planning in fading environments,” in
2008 IEEE International Conference on Robotics and Automation, 2008,
pp. 3169–3174.

[47]
Y. Mostofi, M. Malmirchegini, and A. Ghaffarkhah, “Estimation of communication
signal strength in robotic networks,” in Proceedings of IEEE
International Conference on Robotics and Automation (ICRA), Anchorage, AK,
USA, May 2010, pp. 1946 –1951.

[48]
A. Muralidharan and Y. Mostofi, “Communication-aware robotics: Exploiting
motion for communication,” Annual Review of Control, Robotics, and
Autonomous Systems, vol. 4, no. 1, pp. 115–139, 2021.

[49]
B. Li, Z. Fei, and Y. Zhang, “UAV communications for 5G and beyond: Recent
advances and future trends,” IEEE Internet of Things Journal, vol. 6,
no. 2, pp. 2241–2263, 2019.

[50]
Y. Zeng, Q. Wu, and R. Zhang, “Accessing from the sky: A tutorial on UAV
communications for 5G and beyond,” Proceedings of the IEEE, vol.
107, no. 12, pp. 2327–2375, 2019.

[51]
M. Mozaffari, W. Saad, M. Bennis, Y.-H. Nam, and M. Debbah, “A tutorial on
UAVs for wireless networks: Applications, challenges, and open problems,”
IEEE Communications Surveys & Tutorials, vol. 21, no. 3, pp.
2334–2360, 2019.

[52]
J. Du, C. Jiang, J. Wang, Y. Ren, and M. Debbah, “Machine learning for 6G
wireless networks: Carrying forward enhanced bandwidth, massive access, and
ultrareliable/low-latency service,” IEEE Vehicular Technology
Magazine, vol. 15, no. 4, pp. 122–134, 2020.

[53]
Q. Wu, J. Xu, Y. Zeng, D. W. K. Ng, N. Al-Dhahir, R. Schober, and A. L.
Swindlehurst, “A comprehensive overview on 5G-and-beyond networks with
UAVs: From communications to sensing and intelligence,” IEEE J. Sel.
A. Commun., vol. 39, no. 10, p. 2912–2945, oct 2021. [Online]. Available:

[54]
M. Mozaffari, X. Lin, and S. Hayes, “Toward 6G with connected sky: UAVs
and beyond,” IEEE Communications Magazine, vol. 59, no. 12, pp.
74–80, 2021.

[55]
G. Geraci, A. Garcia-Rodriguez, M. M. Azari, A. Lozano, M. Mezzavilla,
S. Chatzinotas, Y. Chen, S. Rangan, and M. D. Renzo, “What will the future
of UAV cellular communications be? A flight from 5G to 6G,”
IEEE Communications Surveys & Tutorials, vol. 24, no. 3, pp.
1304–1335, 2022.

[56]
J. Luo, Z. Wang, M. Xia, L. Wu, Y. Tian, and Y. Chen, “Path planning for UAV
communication networks: Related technologies, solutions, and opportunities,”
ACM Comput. Surv., vol. 55, no. 9, jan 2023. [Online]. Available:

[57]
C. E. Shannon, “A mathematical theory of communication,” Bell Syst.
Tech. J., vol. 27, pp. 623–656, 1948.

[58]
A. J. Goldsmith, Wireless Communications.1em plus 0.5em minus
0.4emCambridge Univ. Press, 2005.

[59]
M. Malmirchegini and Y. Mostofi, “On the spatial predictability of
communication channels,” IEEE Transactions on Wireless
Communications, vol. 11, no. 3, pp. 964–978, March 2012.

[60]
Qualcom, “Lte unmanned aircraft systems: Trial report (v1.0.1),” Study V
1.0.1, 2017.

[61]
ITU, “Propagation data and prediction methods required for the design of
terrestrial broadband radio access systems operating in a frequency range
from 3 to 60 GHz,” International Telecommunication Union, Recommendation
P.1410-5, 2013.

[62]
3GPP, “Technical specification group radio access network; study on enhanced
lte support for aerial vehicles (release 15),” Study TR 36.777, 2018.

[63]
——, “Technical specification group radio access network; study on 3d
channel model for lte (release 12),” Study TR 36.777, 2018.

[64]
A. Al-Hourani, S. Kandeepan, and S. Lardner, “Optimal lap altitude for maximum
coverage,” IEEE Wireless Communications Letters, vol. 3, no. 6, pp.
569–572, 2014.

[65]
A. Al-Hourani, S. Kandeepan, and A. Jamalipour, “Modeling air-to-ground path
loss for low altitude platforms in urban environments,” in 2014 IEEE
Global Communications Conference, 2014, pp. 2898–2904.

[66]
C. Yan, L. Fu, J. Zhang, and J. Wang, “A comprehensive survey on UAV
communication channel modeling,” IEEE Access, vol. 7, pp.
107 769–107 792, 2019.

[67]
W. Khawaja, I. Guvenc, D. W. Matolak, U.-C. Fiebig, and N. Schneckenburger, “A
survey of air-to-ground propagation channel modeling for unmanned aerial
vehicles,” IEEE Communications Surveys & Tutorials, vol. 21, no. 3,
pp. 2361–2391, 2019.

[68]
I. A. Hemadeh, K. Satyanarayana, M. El-Hajjar, and L. Hanzo, “Millimeter-wave
communications: Physical channel models, design considerations, antenna
constructions, and link-budget,” IEEE Communications Surveys &
Tutorials, vol. 20, no. 2, pp. 870–913, 2018.

[69]
C. Han, Y. Wang, Y. Li, Y. Chen, N. A. Abbasi, T. Kürner, and
A. F. Molisch, “Terahertz wireless channels: A holistic survey on
measurement, modeling, and analysis,” Commun. Surveys Tuts., vol. 24,
no. 3, p. 1670–1707, jul 2022. [Online]. Available:

[70]
H. Kim and J. Choi, “Channel estimation for spatially/temporally correlated
massive mimo systems with one-bit adcs,” J. Wireless Commun. Netw.,
vol. 59, no. 5, pp. 2253–2265, 2019.

[71]
C. Oestges, N. Czink, B. Bandemer, P. Castiglione, F. Kaltenberger, and A. J.
Paulraj, “Experimental characterization and modeling of outdoor-to-indoor
and indoor-to-indoor distributed channels,” IEEE Trans. Veh.
Technol., vol. 59, no. 5, pp. 2253–2265, 2010.

[72]
M. Torun, H. Cai, and Y. Mostofi, “Spatial prediction of channel signal
strength map using deep fully convolutional neural network,” in 2022
56th Asilomar Conference on Signals, Systems, and Computers, 2022, pp.

[73]
C. R. Karanam and Y. Mostofi, “A foundation for wireless channel prediction
and full ray makeup estimation using an unmanned vehicle,” IEEE
Sensors Journal, vol. 23, no. 18, pp. 21 452–21 462, 2023.

[74]
S. Evmorfos, K. I. Diamantaras, and A. P. Petropulu, “Reinforcement learning
for motion policies in mobile relaying networks,” IEEE Transactions on
Signal Processing, vol. 70, pp. 850–861, 2022.

[75]
D. S. Kalogerias and A. P. Petropulu, “Spatially controlled relay
beamforming,” IEEE Transactions on Signal Processing, vol. 66,
no. 24, pp. 6418–6433, 2018.

[76]
K. Diamantaras and A. Petropulu, “Optimal mobile relay beamforming via
reinforcement learning,” in 2019 IEEE 29th International Workshop on
Machine Learning for Signal Processing (MLSP), 2019, pp. 1–6.

[77]
Remcom, “Wireless InSite.” [Online]. Available:

[78]
Qualcomm, “Snapdragon 8 gen 2 mobile platform,” Product Brief, 2023.
[Online]. Available:

[79]
“Mostofi-lab wireless channel simulator,”

[80]
A. Gonzalez-Ruiz, A. Ghaffarkhah, , and Y. Mostofi, “A Comprehensive Overview
and Characterization of Wireless Channels for Networked Robotic and Control
Systems,” Journal of Robotics, vol. 5, p. 19, 2011.

[81]
“NYUSIM: Open-source mmwave and sub-THz wireless channel simulator in
matlab and ns-3,” https://wireless.engineering.nyu.edu/nyusim-5g-and-6g/.

[82]
D. S. Kalogerias and A. P. Petropulu, “Spatially controlled relay beamforming:
$2 $-stage optimal policies,” arXiv preprint arXiv:1705.07463, 2017.

[83]
S. Evmorfos, “Reinforcement learning for motion policies in mobile relaying
networks - code implementation,”
<https://github.com/SpiliosEv/DeepQLearningSIREN>, 2022, software
Implementation - Pytorch.

[84]
S. Evmorfos and A. P. Petropulu, “Deep actor-critic for continuous 3d motion
control in mobile relay beamforming networks,” in ICASSP 2022 - 2022
IEEE International Conference on Acoustics, Speech and Signal Processing
(ICASSP), 2022, pp. 5353–5357.

[85]
U. Challita, W. Saad, and C. Bettstetter, “Interference management for
cellular-connected UAVs: A deep reinforcement learning approach,”
IEEE Transactions on Wireless Communications, vol. 18, no. 4, pp.
2125–2140, 2019.

[86]
M. Bloch and J. Barros, “Secrecy capacity,” in Physical-Layer Security:
From Information Theory to Security Engineering.1em plus 0.5em minus
0.4emCambridge University Press, 2011, p. 49–111.

[87]
C. You and R. Zhang, “Hybrid offline-online design for UAV-enabled data
harvesting in probabilistic los channels,” IEEE Transactions on
Wireless Communications, vol. 19, no. 6, pp. 3753–3768, 2020.

[88]
DJI, “Phantom 4 rtk: Specs,” Technical Specifications, 2023. [Online].
Available: <https://www.dji.com/phantom-4/info>

[89]
L. Steven M., Planning Algorithms.1em plus 0.5em minus
0.4emCambridge University Press, 2006. [Online]. Available:
<https://search.ebscohost.com/login.aspx?direct=true   db=nlebk   AN=170093   site=ehost-live>

[90]
L. E. Dubins, “On curves of minimal length with a constraint on average
curvature, and with prescribed initial and terminal positions and tangents,”
American Journal of Mathematics, vol. 79, no. 3, pp. 497–516, 1957.
[Online]. Available: <http://www.jstor.org/stable/2372560>

[91]
D. Bonilla Licea, M. Bonilla E., M. Ghogho, and M. Saska, “Energy-efficient
fixed-wing UAV relay with considerations of airframe shadowing,”
IEEE Communications Letters, vol. 27, no. 6, pp. 1550–1554, 2023.

[92]
Y. Mei, Y.-H. Lu, Y. Hu, and C. Lee, “Deployment of mobile robots with energy
and timing constraints,” IEEE Transactions on Robotics, vol. 22,
no. 3, pp. 507–522, 2006.

[93]
Y. Zeng, J. Xu, and R. Zhang, “Energy minimization for wireless communication
with rotary-wing UAV,” IEEE Transactions on Wireless
Communications, vol. 18, no. 4, pp. 2329–2345, 2019.

[94]
Y. Zeng and R. Zhang, “Energy-efficient UAV communication with trajectory
optimization,” IEEE Transactions on Wireless Communications, vol. 16,
no. 6, pp. 3747–3760, 2017.

[95]
G. Scutari, F. Facchinei, P. Song, D. P. Palomar, and J.-S. Pang,
“Decomposition by partial linearization: Parallel optimization of
multi-agent systems,” IEEE Transactions on Signal Processing,
vol. 62, no. 3, pp. 641–656, 2014.

[96]
L. Peng and R. Vidal, “Block coordinate descent on smooth manifolds:
Convergence theory and twenty-one examples,” 2023.

[97]
J. Lee and S. Leyffer, Mixed Integer Nonlinear Programming, ser. The IMA
Volumes in Mathematics and its Applications.1em plus 0.5em minus
0.4emSpringer, 2012.

[98]
A. Del Pia, S. S. Dey, and M. Molinaro, “Mixed-integer quadratic programming
is in np,” Mathematical Programming, pp. 225–240, 2017.

[99]
H. Y. Benson and Ümit Sağlam, “Mixed-integer second-order cone programming:
A survey,” in Theory Driven by Influential Applications, 2013, ch.
Chapter 2, pp. 13–36. [Online]. Available:

[100]
IBM, “IBM ILOG CPLEX Optimization Studio,” 2023.

[101]
Moesk, “MOSEK 10.1,” 2023. [Online]. Available:

[102]
Gurobi Optimization, “Gurobi Optimizer,” 2023. [Online]. Available:

[103]
P. Toth and D. Vigo, The Vehicle Routing Problem, P. Toth and D. Vigo,
Eds.1em plus 0.5em minus 0.4emSociety for Industrial and
Applied Mathematics, 2002. [Online]. Available:

[104]
J. Pasha, Z. Elmi, S. Purkayastha, A. M. Fathollahi-Fard, Y.-E. Ge, Y.-Y. Lau,
and M. A. Dulebenets, “The drone scheduling problem: A systematic
state-of-the-art review,” IEEE Transactions on Intelligent
Transportation Systems, vol. 23, no. 9, pp. 14 224–14 247, 2022.

[105]
S. H. Chung, B. Sah, and J. Lee, “Optimization for drone and drone-truck
combined operations: A review of the state of the art and future
directions,” Computers & Operations Research, vol. 123, p. 105004,
2020. [Online]. Available:

[106]
N. A. H. Agatz, P. C. Bouman, and M. Schmidt, “Optimization approaches for the
traveling salesman problem with drone,” Econometrics: Mathematical
Methods & Programming eJournal, 2016.

[107]
P. E. Hart, N. J. Nilsson, and B. Raphael, “A formal basis for the heuristic
determination of minimum cost paths,” IEEE Transactions on Systems
Science and Cybernetics, vol. 4, no. 2, pp. 100–107, 1968.

[108]
S. Karaman and E. Frazzoli, “Sampling-based algorithms for optimal motion
planning,” The International Journal of Robotics Research, vol. 30,
pp. 846 – 894, 2011. [Online]. Available:

[109]
D. Liberzon, Calculus of Variations and Optimal Control Theory: A Concise
Introduction.1em plus 0.5em minus 0.4emPrinceton Univ. Press,

[110]
D. Bertsekas, Dynamic Programming and Optimal Control, 4th ed.1em plus 0.5em minus 0.4emAthena Scientific, 2017, vol. 1.

[111]
J. P. Hespanha, Linear Systems Theory.1em plus 0.5em minus
0.4emPrinceton Univ. Press, 2009.

[112]
Z. Shaikh, S. Baidya, and M. Levorato, “Robust multi-path communications for
UAVs in the urban iot,” in 2018 IEEE International Conference on
Sensing, Communication and Networking (SECON Workshops).1em plus
0.5em minus 0.4emIEEE, 2018, pp. 1–5.

[113]
A. Feriani, A. Mezghani, and E. Hossain, “On the robustness of deep
reinforcement learning in irs-aided wireless communications systems,”
arXiv preprint arXiv:2107.08293, 2021.

[114]
R. S. Sutton and A. G. Barto, Reinforcement learning: An
introduction.1em plus 0.5em minus 0.4emMIT press, 2018.

[115]
D. Bertsekas, Reinforcement learning and optimal control.1em plus
0.5em minus 0.4emAthena Scientific, 2019.

[116]
R. Bellman, “Dynamic programming,” Science, vol. 153, no. 3731, pp.
34–37, 1966.

[117]
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare,
A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al.,
“Human-level control through deep reinforcement learning,” nature,
vol. 518, no. 7540, pp. 529–533, 2015.

[118]
G. Dulac-Arnold, R. Evans, H. van Hasselt, P. Sunehag, T. Lillicrap, J. Hunt,
T. Mann, T. Weber, T. Degris, and B. Coppin, “Deep reinforcement learning in
large discrete action spaces,” arXiv preprint arXiv:1512.07679, 2015.

[119]
D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller,
“Deterministic policy gradient algorithms,” in International
conference on machine learning.1em plus 0.5em minus 0.4emPmlr,
2014, pp. 387–395.

[120]
M. Köppen, “The curse of dimensionality,” in 5th online world
conference on soft computing in industrial applications (WSC5), vol. 1,
2000, pp. 4–8.

[121]
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal
policy optimization algorithms,” arXiv preprint arXiv:1707.06347,

[122]
J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region
policy optimization,” in International conference on machine
learning.1em plus 0.5em minus 0.4emPMLR, 2015, pp. 1889–1897.

[123]
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal
policy optimization algorithms,” arXiv preprint arXiv:1707.06347,

[124]
T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy
maximum entropy deep reinforcement learning with a stochastic actor,” in
International conference on machine learning.1em plus 0.5em
minus 0.4emPMLR, 2018, pp. 1861–1870.

[125]
T. M. Moerland, J. Broekens, A. Plaat, C. M. Jonker et al.,
“Model-based reinforcement learning: A survey,” Foundations and
Trends® in Machine Learning, vol. 16, no. 1, pp. 1–118,

[126]
A. G. Barto, “Intrinsic motivation and reinforcement learning,”
Intrinsically motivated learning in natural and artificial systems,
pp. 17–47, 2013.

[127]
S. Evmorfos, K. I. Diamantaras, and A. P. Petropulu, “Reinforcement learning
for motion policies in mobile relaying networks,” IEEE Transactions on
Signal Processing, vol. 70, pp. 850–861, 2022.

[128]
K. Diamantaras and A. Petropulu, “Optimal mobile relay beamforming via
reinforcement learning,” in 2019 IEEE 29th International Workshop on
Machine Learning for Signal Processing (MLSP), 2019, pp. 1–6.

[129]
Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas,
“Dueling network architectures for deep reinforcement learning,” in
International conference on machine learning.1em plus 0.5em
minus 0.4emPMLR, 2016, pp. 1995–2003.

[130]
S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approximation error
in actor-critic methods,” in International conference on machine
learning.1em plus 0.5em minus 0.4emPMLR, 2018, pp. 1587–1596.

[131]
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa,
D. Silver, and D. Wierstra, “Continuous control with deep reinforcement
learning,” arXiv preprint arXiv:1509.02971, 2015.

[132]
H. Kumar, D. S. Kalogerias, G. J. Pappas, and A. Ribeiro, “Zeroth-order
deterministic policy gradient,” arXiv preprint arXiv:2006.07314,

[133]
A. Alidadi Shamsabadi, A. Yadav, O. Abbasi, and H. Yanikomeroglu, “Handling
interference in integrated HAPS-terrestrial networks through radio resource
management,” IEEE Wireless Communications Letters, vol. 11, no. 12,
pp. 2585–2589, 2022.

[134]
X. Fang, W. Feng, Y. Wang, Y. Chen, N. Ge, Z. Ding, and H. Zhu, “Noma-based
hybrid satellite-uav-terrestrial networks for 6G maritime coverage,”
IEEE Transactions on Wireless Communications, vol. 22, no. 1, pp.
138–152, 2023.

[135]
R. I. Bor-Yaliniz, A. El-Keyi, and H. Yanikomeroglu, “Efficient 3-d placement
of an aerial base station in next generation cellular networks,” in
2016 IEEE International Conference on Communications (ICC), 2016, pp.

[136]
Q. Ding, Y. Luo, R. Yang, J. Hu, and C. Luo, “Joint deployment, beamforming
and power allocation of mmwave full-duplex UAV-BS,” in Proceedings
of the 5th International ACM Mobicom Workshop on Drone Assisted Wireless
Communications for 5G and Beyond, ser. DroneCom '22.1em plus 0.5em
minus 0.4emNew York, NY, USA: Association for Computing Machinery,
2022, p. 127–132. [Online]. Available:

[137]
M. Mozaffari, A. Taleb Zadeh Kasgari, W. Saad, M. Bennis, and M. Debbah,
“Beyond 5G with UAVs: Foundations of a 3D wireless cellular network,”
IEEE Trans. Wireless Commun., vol. 18, no. 1, pp. 357–372, 2019.

[138]
Q. Wu, Y. Zeng, and R. Zhang, “Joint trajectory and communication design for
multi-uav enabled wireless networks,” IEEE Transactions on Wireless
Communications, vol. 17, no. 3, pp. 2109–2121, 2018.

[139]
H. Bayerlein, P. De Kerret, and D. Gesbert, “Trajectory optimization for
autonomous flying base station via reinforcement learning,” in 2018
IEEE 19th International Workshop on Signal Processing Advances in Wireless
Communications (SPAWC).1em plus 0.5em minus 0.4emIEEE, 2018,
pp. 1–5.

[140]
Z. Qin, Z. Liu, G. Han, C. Lin, L. Guo, and L. Xie, “Distributed UAV-BSs
trajectory optimization for user-level fair communication service with
multi-agent deep reinforcement learning,” IEEE Trans. Veh. Technol.,
vol. 70, no. 12, pp. 12 290–12 301, 2021.

[141]
C. Zhan and Y. Zeng, “Energy minimization for cellular-connected UAV: From
optimization to deep reinforcement learning,” IEEE Transactions on
Wireless Communications, vol. 21, no. 7, pp. 5541–5555, 2022.

[142]
Y. Zeng and X. Xu, “Path design for cellular-connected UAV with
reinforcement learning,” in 2019 IEEE Global Communications Conference
(GLOBECOM).1em plus 0.5em minus 0.4emIEEE, 2019, pp. 1–6.

[143]
S. Yin, S. Zhao, Y. Zhao, and F. R. Yu, “Intelligent trajectory design in
UAV-aided communications with reinforcement learning,” IEEE Trans.
Veh. Technol., vol. 68, no. 8, pp. 8227–8231, 2019.

[144]
H. Mohammadi, V. Marojevic, and B. Shang, “Analysis of reinforcement learning
schemes for trajectory optimization of an aerial radio unit,” arXiv
preprint arXiv:2211.10524, 2022.

[145]
Y. Li, X. Yuan, Y. Hu, J. Yang, and A. Schmeink, “Optimal UAV trajectory
design for moving users in integrated sensing and communications networks,”
IEEE Transactions on Intelligent Transportation Systems, pp. 1–18,

[146]
K. Meng, Q. Wu, S. Ma, W. Chen, K. Wang, and J. Li, “Throughput maximization
for UAV-enabled integrated periodic sensing and communication,” IEEE
Transactions on Wireless Communications, vol. 22, no. 1, pp. 671–687, 2023.

[147]
D. S. Kalogerias and A. P. Petropulu, “Mobile beamforming & spatially
controlled relay communications,” in 2016 IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP), 2016, pp.

[148]
Y. Yan and Y. Mostofi, “Robotic router formation in realistic communication
environments,” IEEE Transactions on Robotics, vol. 28, no. 4, pp.
810–827, July 2012.

[149]
J. Fink, A. Ribeiro, and V. Kumar, “Robust control of mobility and
communications in autonomous robot teams,” IEEE Access, vol. 1, pp.
290–309, 2013.

[150]
A. Muralidharan and Y. Mostofi, “Energy optimal distributed beamforming using
unmanned vehicles,” IEEE Transactions on Control of Network Systems,

[151]
——, “Distributed beamforming using mobile robots,” in 2016 IEEE
International Conference on Acoustics, Speech and Signal Processing
(ICASSP).1em plus 0.5em minus 0.4emIEEE, 2016, pp. 6385–6389.

[152]
S. Evmorfos, K. I. Diamantaras, and A. P. Petropulu, “Reinforcement learning
for motion policies in mobile relaying networks,” IEEE Transactions on
Signal Processing, vol. 70, pp. 850–861, 2022.

[153]
S. Evmorfos and A. P. Petropulu, “Deep actor-critic for continuous 3d motion
control in mobile relay beamforming networks,” in ICASSP 2022 - 2022
IEEE International Conference on Acoustics, Speech and Signal Processing
(ICASSP), 2022, pp. 5353–5357.

[154]
D. S. Kalogerias and A. P. Petropulu, “Mobi-cliques for improving ergodic
secrecy in fading wiretap channels under power constraints,” in 2014
IEEE International Conference on Acoustics, Speech and Signal Processing
(ICASSP).1em plus 0.5em minus 0.4emIEEE, 2014, pp. 1578–1591.

[155]
S. S. Hassan, Y. M. Park, Y. K. Tun, W. Saad, Z. Han, and C. S. Hong, “3TO:
THz-enabled throughput and trajectory optimization of UAVs in 6G
networks by proximal policy optimization deep reinforcement learning,” in
ICC 2022-IEEE International Conference on Communications.1em
plus 0.5em minus 0.4emIEEE, 2022, pp. 5712–5718.

[156]
M. Mozaffari, W. Saad, M. Bennis, and M. Debbah, “Drone small cells in the
clouds: Design, deployment and performance analysis,” in 2015 IEEE
global communications conference (GLOBECOM).1em plus 0.5em minus
0.4emIEEE, 2015, pp. 1–6.

[157]
M. J. Farooq and Q. Zhu, “A multi-layer feedback system approach to resilient
connectivity of remotely deployed mobile internet of things,” IEEE
Transactions on Cognitive Communications and Networking, vol. 4, no. 2, pp.
422–432, 2018.

[158]
Z. Kang, C. You, and R. Zhang, “3d placement for multi-uav relaying: An
iterative gibbs-sampling and block coordinate descent optimization
approach,” IEEE Transactions on Communications, vol. 69, no. 3, pp.
2047–2062, 2020.

[159]
S. Y. Shin et al., “Sub-connected hybrid precoding and trajectory
optimization using deep reinforcement learning for energy-efficient
millimeter-wave UAV communications,” IEEE Wireless Communications
Letters, 2023.

[160]
C. Villani, Optimal Transport: Old and New.1em plus 0.5em minus
0.4emSpringer, 2008.

[161]
F. Liu, C. Masouros, A. P. Petropulu, H. Griffiths, and L. Hanzo, “Joint radar
and communication design: Applications, state-of-the-art, and the road
ahead,” IEEE Trans. Commun., vol. 68, no. 6, pp. 3834–3862, 2020.

[162]
J. A. Zhang, F. Liu, C. Masouros, R. W. Heath, Z. Feng, L. Zheng, and
A. Petropulu, “An overview of signal processing techniques for joint
communication and radar sensing,” IEEE J. Sel. Topics Signal
Process., vol. 15, no. 6, pp. 1295–1315, 2021.

[163]
K. Meng, Q. Wu, J. Xu, W. Chen, Z. Feng, R. Schober, and A. L. Swindlehurst,
“UAV-enabled integrated sensing and communication: Opportunities and
challenges,” IEEE Wireless Communications, pp. 1–9, 2023.

[164]
J. Mu, R. Zhang, Y. Cui, N. Gao, and X. Jing, “UAV meets integrated sensing
and communication: Challenges and future directions,” IEEE
Communications Magazine, vol. 61, no. 5, pp. 62–67, 2023.

[165]
C. Chen, H. Song, Q. Li, F. Meneghello, F. Restuccia, and C. Cordeiro,
“Wi-Fi sensing based on IEEE 802.11bf,” IEEE Communications
Magazine, vol. 61, no. 1, pp. 121–127, 2023.

[166]
I. Griva and R. J. Vanderbei, “Case studies in optimization: Catenary
problem,” Optimization and Engineering, vol. 6, pp. 463–482, 2005.
[Online]. Available:

[167]
B. Zafar, S. Gherekhloo, and M. Haardt, “Analysis of multihop relaying
networks: Communication between range-limited and cooperative nodes,”
IEEE Vehicular Technology Magazine, vol. 7, no. 3, pp. 40–47, 2012.

[168]
S. Boyd and L. Vandenberghe, Convex Optimization.1em plus 0.5em
minus 0.4emCambrdige University Press, 2004.

[169]
A. D. Ames, S. Coogan, M. Egerstedt, G. Notomista, K. Sreenath, and P. Tabuada,
“Control barrier functions: Theory and applications,” in 2019 18th
European Control Conference (ECC), 2019, pp. 3420–3431.

[170]
S. J. Wright, “Coordinate descent algorithms,” Mathematical
programming, vol. 151, no. 1, pp. 3–34, 2015.

[171]
A. E. Gelfand, “Gibbs sampling,” Journal of the American statistical
Association, vol. 95, no. 452, pp. 1300–1304, 2000.

[172]
G. Zheng, K.-K. Wong, A. Paulraj, and B. Ottersten, “Collaborative-relay
beamforming with perfect csi: Optimum and distributed implementation,”
IEEE Signal Processing Letters, vol. 16, no. 4, pp. 257–260, 2009.

[173]
D. Pisinger, “A minimal algorithm for the multiple-choice knapsack problem,”
European Journal of Operational Research, vol. 83, no. 2, pp.
394–410, 1995, eURO Summer Institute Combinatorial Optimization. [Online].

[174]
E. Koyuncu, Y. Jing, and H. Jafarkhani, “Distributed beamforming in wireless
relay networks with quantized feedback,” IEEE Journal on Selected
Areas in Communications, vol. 26, no. 8, pp. 1429–1439, 2008.

[175]
M. Sabin and R. Gray, “Global convergence and empirical consistency of the
generalized lloyd algorithm,” IEEE Transactions on information
theory, vol. 32, no. 2, pp. 148–155, 1986.

[176]
V. Havary-Nassab, S. Shahbazpanahi, A. Grami, and Z.-Q. Luo, “Distributed
beamforming for relay networks based on second-order statistics of the
channel state information,” IEEE transactions on signal processing,
vol. 56, no. 9, pp. 4306–4316, 2008.

[177]
D. S. Kalogerias, N. Chatzipanagiotis, M. M. Zavlanos, and A. P. Petropulu,
“Mobile jammers for secrecy rate maximization in cooperative networks,” in
2013 IEEE International Conference on Acoustics, Speech and Signal
Processing.1em plus 0.5em minus 0.4emIEEE, 2013, pp.

[178]
W. Hurst, H. Cai, and Y. Mostofi, “Communication-aware RRT*: Path planning
for robotic communication operation in obstacle environments,” in ICC
2021 - IEEE International Conference on Communications, 2021, pp. 1–6.

[179]
S. Zhang, Y. Zeng, and R. Zhang, “Cellular-enabled UAV communication: A
connectivity-constrained trajectory optimization perspective,” IEEE
Transactions on Communications, vol. 67, no. 3, pp. 2580–2604, 2019.

[180]
N. Cherif, W. Jaafar, H. Yanikomeroglu, and A. Yongacoglu,
“Disconnectivity-aware energy-efficient cargo-uav trajectory planning with
minimum handoffs,” in ICC 2021 - IEEE International Conference on
Communications, 2021, pp. 1–6.

[181]
A. Muralidharan and Y. Mostofi, “Path planning for minimizing the expected
cost until success,” IEEE Transactions on Robotics, vol. 35, no. 2,
pp. 466–481, 2019.

[182]
——, “Statistics of the Distance Traveled until Connectivity for Unmanned
Vehicles,” Autonomous Robots, vol. 44, no. 1, pp. 25–42, 2020.

[183]
Y. Yan and Y. Mostofi, “Co-optimization of communication and motion planning
of a robotic operation under resource constraints and in fading
environments,” IEEE Transactions on Wireless Communications, vol. 12,
no. 4, pp. 1562–1572, April 2013.

[184]
U. Ali, Y. Yan, Y. Mostofi, and Y. Wardi, “An optimal control approach for
communication and motion co-optimization in realistic fading environment.”

[185]
U. Ali, H. Cai, Y. Mostofi, and Y. Wardi, “Motion-communication
co-optimization with cooperative load transfer in mobile robotics: An optimal
control perspective,” IEEE Transactions on Control of Network
Systems, vol. 6, no. 2, pp. 621–632, 2019.

[186]
K. K. Nguyen, T. Q. Duong, T. Do-Duy, H. Claussen, and L. Hanzo, “3d UAV
trajectory and data collection optimisation via deep reinforcement
learning,” IEEE Transactions on Communications, vol. 70, no. 4, pp.
2358–2371, 2022.

[187]
W. Hurst and Y. Mostofi, “Optimization of mobile robotic relay operation for
minimal average wait time,” IEEE Transactions on Wireless
Communications, vol. 22, no. 6, pp. 3733–3747, 2023.

[188]
——, “Optimal dynamic trajectories for UAVs in mobility-enabled relay
systems,” in Proceedings of the 62nd IEEE Conference on Decision and
Control (CDC), 2023, To appear.

[189]
O. S. Oubbati, M. Atiquzzaman, H. Lim, A. Rachedi, and A. Lakas,
“Synchronizing UAV teams for timely data collection and energy transfer by
deep reinforcement learning,” IEEE Trans. Veh. Technol., vol. 71,
no. 6, pp. 6682–6697, 2022.

[190]
K. Li, W. Ni, E. Tovar, and M. Guizani, “Joint flight cruise control and data
collection in UAV-aided internet of things: An onboard deep reinforcement
learning approach,” IEEE Internet of Things Journal, vol. 8, no. 12,
pp. 9787–9799, 2021.

[191]
Y. Yan and Y. Mostofi, “To go or not to go on energy-aware and
communication-aware robotic operation,” IEEE Transactions on Control
of Network Systems, vol. 1, no. 3, pp. 218 – 231, July 2014.

[192]
H. Cai and Y. Mostofi, “Co-optimization of motion, communication, and sensing
in real wireless channel environments via monte carlo tree search,”
IEEE Transactions on Control of Network Systems, vol. 9, no. 3, pp.
1493–1505, 2022.

[193]
M. Mozaffari, W. Saad, M. Bennis, and M. Debbah, “Mobile unmanned aerial
vehicles (UAVs) for energy-efficient internet of things communications,”
IEEE Transactions on Wireless Communications, vol. 16, no. 11, pp.
7574–7589, 2017.

[194]
M. Samir, S. Sharafeddine, C. M. Assi, T. M. Nguyen, and A. Ghrayeb, “UAV
trajectory planning for data collection from time-constrained IoT
devices,” IEEE Transactions on Wireless Communications, vol. 19,
no. 1, pp. 34–46, 2020.

[195]
O. Ghdiri, W. Jaafar, S. Alfattani, J. B. Abderrazak, and H. Yanikomeroglu,
“Energy-efficient multi-uav data collection for IoT networks with time
deadlines,” in Proc. IEEE IEEE Global Telecommun. Conf. (GLOBECOM),
2020, pp. 1–6.

[196]
A. Ghaffarkhah and Y. Mostofi, “Dynamic networked coverage of time-varying
environments in the presence of fading communication channels,” ACM
Transactions on Sensor Networks, vol. 10, no. 3, April 2014.

[197]
X. Gao, X. Zhu, and L. Zhai, “AoI-sensitive data collection in
multi-UAV-assisted wireless sensor networks,” IEEE Transactions on
Wireless Communications, vol. 22, no. 8, pp. 5185–5197, 2023.

[198]
M. A. Abd-Elmagid and H. S. Dhillon, “Average peak age-of-information
minimization in UAV-assisted IoT networks,” IEEE Transactions on
Vehicular Technology, vol. 68, no. 2, pp. 2003–2008, 2019.

[199]
R. J. Kerczewski and J. H. Griner, “Control and non-payload communications
links for integrated unmanned aircraft operations,” in 2012 Joint
Conference - 18th Ka and Broadband, 2012.

[200]
L. M. Ricciardi and S. Sato, “First-passage-time density and moments of the
Ornstein-Uhlenbeck process,” Journal of Applied Probability,
vol. 25, no. 1, pp. 43–57, 1988.

[201]
E. F. Beckenbach and R. Weller, “Problem of bolza,” in Modern
Mathematics for the Engineer - First Series.1em plus 0.5em minus
0.4emDover Publications, 2013, ch. 4.12. [Online]. Available:

[202]
R. Shah, S. Roy, S. Jain, and W. Brunette, “Data mules: modeling a three-tier
architecture for sparse sensor networks,” in Proceedings of the First
IEEE International Workshop on Sensor Network Protocols and Applications,
2003., 2003, pp. 30–41.

[203]
Y. Sun, E. Uysal-Biyikoglu, R. D. Yates, C. E. Koksal, and N. B. Shroff,
“Update or wait: How to keep your data fresh,” IEEE Transactions on
Information Theory, vol. 63, no. 11, pp. 7492–7508, 2017.

[204]
V. Vishnevsky and O. Semenova, “Polling systems and their application to
telecommunication networks,” Mathematics, vol. 9, no. 2, 2021.

[205]
M. Ahmed, R. Seraj, and S. M. S. Islam, “The k-means algorithm: A
comprehensive survey and performance evaluation,” Electronics,
vol. 9, no. 8, 2020. [Online]. Available:

[206]
C. Blum, “Ant colony optimization: Introduction and recent trends,”
Physics of Life Reviews, vol. 2, no. 4, pp. 353–373, 2005. [Online].

[207]
R. L. Disney and D. Konig, “Queueing networks: A survey of their random
processes,” SIAM Review, vol. 27, no. 3, pp. 335–403, 1985.
[Online]. Available: <http://www.jstor.org/stable/2031581>

[208]
C. J. Watkins and P. Dayan, “Q-learning,” Machine learning, vol. 8,
pp. 279–292, 1992.

[209]
S. K. Singh, R. Singh, and B. Kumbhani, “The evolution of radio access network
towards open-ran: Challenges and opportunities,” in 2020 IEEE Wireless
Communications and Networking Conference Workshops (WCNCW).1em plus
0.5em minus 0.4emIEEE, 2020, pp. 1–6.

[210]
D. Zhao, H. Wang, K. Shao, and Y. Zhu, “Deep reinforcement learning with
experience replay based on sarsa,” in 2016 IEEE symposium series on
computational intelligence (SSCI).1em plus 0.5em minus 0.4emIEEE, 2016, pp. 1–6.

[211]
I. Kostrikov, A. Nair, and S. Levine, “Offline reinforcement learning with
implicit q-learning,” arXiv preprint arXiv:2110.06169, 2021.

[212]
G. Tesauro et al., “Temporal difference learning and td-gammon,”
Communications of the ACM, vol. 38, no. 3, pp. 58–68, 1995.

[213]
A. A. Sherstov and P. Stone, “Function approximation via tile coding:
Automating parameter choice,” in International symposium on
abstraction, reformulation, and approximation.1em plus 0.5em minus
0.4emSpringer, 2005, pp. 194–205.

[214]
Y. Zeng, R. Zhang, and T. J. Lim, “Throughput maximization for UAV-enabled
mobile relaying systems,” IEEE Transactions on communications,
vol. 64, no. 12, pp. 4983–4996, 2016.

[215]
Q. Wang, A. Kakkavas, X. Gong, and R. A. Stirling-Gallacher, “Towards
integrated sensing and communications for 6G,” in 2022 2nd IEEE
International Symposium on Joint Communications & Sensing (JC&S).1em plus 0.5em minus 0.4emIEEE, 2022, pp. 1–6.

[216]
J. Tan and L. Dai, “Delay-phase precoding for THz massive MIMO with beam
split,” in 2019 IEEE Global Communications Conference
(GLOBECOM).1em plus 0.5em minus 0.4emIEEE, 2019, pp. 1–6.

[217]
S. Evmorfos, K. Diamantaras, and A. Petropulu, “Double deep q learning with
gradient biasing for mobile relay beamforming networks,” in 2021 55th
Asilomar Conference on Signals, Systems, and Computers, 2021, pp. 742–746.

[218]
——, “Deep q learning with fourier feature mapping for mobile relay
beamforming networks,” in 2021 IEEE 22nd International Workshop on
Signal Processing Advances in Wireless Communications (SPAWC), 2021, pp.

[219]
S. Evmorfos, D. Kalogerias, and A. Petropulu, “Adaptive discrete motion
control for mobile relay networks,” Frontiers in Signal Processing,
vol. 2, p. 867388, 2022.

[220]
S. M. Abohashish, R. Y. Rizk, and E. Elsedimy, “Trajectory optimization for
UAV-assisted relay over 5g networks based on reinforcement learning
framework,” EURASIP Journal on Wireless Communications and
Networking, vol. 2023, no. 1, p. 55, 2023.

[221]
C. Han, W. Gao, N. Yang, and J. M. Jornet, “Molecular absorption effect: A
double-edged sword of terahertz communications,” IEEE Wireless
Communications, 2022.

[222]
Q. Wang, W. Zhang, Y. Liu, and Y. Liu, “Multi-uav dynamic wireless networking
with deep reinforcement learning,” IEEE Communications Letters,
vol. 23, no. 12, pp. 2243–2246, 2019.

[223]
M. T. Spaan, “Partially observable markov decision processes,” in
Reinforcement learning: State-of-the-art.1em plus 0.5em minus
0.4emSpringer, 2012, pp. 387–414.

[224]
G. Vasconcelos, R. S. Miani, V. C. Guizilini, and J. R. Souza, “Evaluation of
dos attacks on commercial wi-fi-based UAVs,” International Journal
of Communication Networks and Information Security, vol. 11, no. 1, pp.
212–223, 2019.

[225]
V. Desnitsky and I. Kotenko, “Simulation and assessment of battery depletion
attacks on unmanned aerial vehicles for crisis management infrastructures,”
Simulation Modelling Practice and Theory, vol. 107, p. 102244, 2021.

[226]
D. He, S. Chan, and M. Guizani, “Communication security of unmanned aerial
vehicles,” IEEE Wireless Communications, vol. 24, no. 4, pp.
134–139, 2016.

[227]
A. D. Wyner, “The wire-tap channel,” Bell system technical journal,
vol. 54, no. 8, pp. 1355–1387, 1975.

[228]
S. A. A. Fakoorian and A. L. Swindlehurst, “Solutions for the mimo gaussian
wiretap channel with a cooperative jammer,” IEEE Transactions on
Signal Processing, vol. 59, no. 10, pp. 5013–5022, 2011.

[229]
L. Dong, Z. Han, A. P. Petropulu, and H. V. Poor, “Improving wireless physical
layer security via cooperating relays,” IEEE Transactions on Signal
Processing, vol. 58, no. 3, pp. 1875–1888, 2010.

[230]
G. Zheng, L.-C. Choo, and K.-K. Wong, “Optimal cooperative jamming to enhance
physical layer security using relays,” IEEE Transactions on Signal
Processing, vol. 59, no. 3, pp. 1317–1322, 2011.

[231]
J. Li, A. P. Petropulu, and S. Weber, “On cooperative relaying schemes for
wireless physical layer security,” IEEE Transactions on Signal
Processing, vol. 59, no. 10, pp. 4985–4997, 2011.

[232]
G. Zheng, I. Krikidis, J. Li, A. P. Petropulu, and B. Ottersten, “Improving
physical layer secrecy using full-duplex jamming receivers,” IEEE
Transactions on Signal Processing, vol. 61, no. 20, pp. 4962–4974, 2013.

[233]
A. Khisti and G. W. Wornell, “Secure transmission with multiple
antennas—part ii: The mimome wiretap channel,” IEEE Transactions on
Information Theory, vol. 56, no. 11, pp. 5515–5532, 2010.

[234]
S. Goel and R. Negi, “Guaranteeing secrecy using artificial noise,”
IEEE Transactions on Wireless Communications, vol. 7, no. 6, pp.
2180–2189, 2008.

[235]
N. Su, F. Liu, C. Masouros, T. Ratnarajah, and A. Petropulu, “Secure
dual-functional radar-communication transmission: Hardware-efficient
design,” in 2021 55th Asilomar Conference on Signals, Systems, and
Computers, 2021, pp. 629–633.

[236]
N. Su, F. Liu, Z. Wei, Y.-F. Liu, and C. Masouros, “Secure dual-functional
radar-communication transmission: Exploiting interference for resilience
against target eavesdropping,” IEEE Transactions on Wireless
Communications, vol. 21, no. 9, pp. 7238–7252, 2022.

[237]
Y.-K. Li and A. Petropulu, “An irs-assisted secure dual-function
radar-communication system,” arXiv preprint arXiv:2310.00555, 2023.

[238]
P. McEnroe, S. Wang, and M. Liyanage, “A survey on the convergence of edge
computing and ai for UAVs: Opportunities and challenges,” IEEE
Internet of Things Journal, vol. 9, no. 17, pp. 15 435–15 459, 2022.

[239]
Z. Liu, Y. Cao, P. Gao, X. Hua, D. Zhang, and T. Jiang, “Multi-uav network
assisted intelligent edge computing: Challenges and opportunities,”
China Communications, vol. 19, no. 3, pp. 258–278, 2022.

[240]
J. Li, Y. Bi, M. Lan, H. Qin, M. Shan, F. Lin, and B. M. Chen, “Real-time
simultaneous localization and mapping for UAV: A survey,” in Proc.
of International micro air vehicle competition and conference, vol. 2016,
2016, p. 237.

[241]
A. Sufian, C. You, and M. Dong, “A deep transfer learning-based edge computing
method for home health monitoring,” in 2021 55th annual conference on
information sciences and systems (CISS).1em plus 0.5em minus
0.4emIEEE, 2021, pp. 1–6.

[242]
I. Jang, H. Kim, D. Lee, Y.-S. Son, and S. Kim, “Knowledge transfer for
on-device deep reinforcement learning in resource constrained edge computing
systems,” IEEE Access, vol. 8, pp. 146 588–146 597, 2020.

[243]
P. S. Chandakkar, Y. Li, P. L. K. Ding, and B. Li, “Strategies for re-training
a pruned neural network in an edge computing paradigm,” in 2017 IEEE
International Conference on Edge Computing (EDGE).1em plus 0.5em
minus 0.4emIEEE, 2017, pp. 244–247.

[244]
S. Zhu, L. Gui, D. Zhao, N. Cheng, Q. Zhang, and X. Lang, “Learning-based
computation offloading approaches in UAVs-assisted edge computing,”
IEEE Transactions on Vehicular Technology, vol. 70, no. 1, pp.
928–944, 2021.

[245]
Y. Cheng, J. Lu, D. Niyato, B. Lyu, J. Kang, and S. Zhu, “Federated transfer
learning with client selection for intrusion detection in mobile edge
computing,” IEEE Communications Letters, vol. 26, no. 3, pp.
552–556, 2022.

[246]
Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” nature, vol. 521,
no. 7553, pp. 436–444, 2015.

[247]
C. Zhang, Y. Xie, H. Bai, B. Yu, W. Li, and Y. Gao, “A survey on federated
learning,” Knowledge-Based Systems, vol. 216, p. 106775, 2021.

[248]
Y. Qu, H. Dai, Y. Zhuang, J. Chen, C. Dong, F. Wu, and S. Guo, “Decentralized
federated learning for UAV networks: Architecture, challenges, and
opportunities,” IEEE Network, vol. 35, no. 6, pp. 156–162, 2021.

[249]
H. Zhang and L. Hanzo, “Federated learning assisted multi-uav networks,”
IEEE Transactions on Vehicular Technology, vol. 69, no. 11, pp.
14 104–14 109, 2020.

[250]
Y. Wang, Z. Su, N. Zhang, and A. Benslimane, “Learning in the air: Secure
federated learning for UAV-assisted crowdsensing,” IEEE Transactions
on network science and engineering, vol. 8, no. 2, pp. 1055–1069, 2020.


# A Machine Learning Approach to Optimal Inverse
Discrete Cosine Transform (IDCT) Design

Yifan Wang1, Zhanxuan Mei1, Chia-Yang Tsai2, Ioannis Katsavounidis2 and C.-C.
Jay Kuo1  1University of Southern California, Los Angeles, California, USA  2
Facebook, Inc., Menlo Park, California, USA

###### Abstract

The design of the optimal inverse discrete cosine transform (IDCT) to
compensate the quantization error is proposed for effective lossy image
compression in this work. The forward and inverse DCTs are designed in pair in
current image/video coding standards without taking the quantization effect
into account. Yet, the distribution of quantized DCT coefficients deviate from
that of original DCT coefficients. This is particularly obvious when the
quality factor of JPEG compressed images is small. To address this problem, we
first use a set of training images to learn the compound effect of forward
DCT, quantization and dequantization in cascade. Then, a new IDCT kernel is
learned to reverse the effect of such a pipeline. Experiments are conducted to
demonstrate that the advantage of the new method, which has a gain of
0.11-0.30dB over the standard JPEG over a wide range of quality factors.

##  I Introduction

Many image and video compression standards have been developed in the last
thirty years. Examples include JPEG [1], JPEG2000 [2], and BPG [3] for image
compression and MPEG-1 [4], MPEG-2 [5], MPEG-4 [6], H.264/AVC [7] and HEVC [8]
for video compression. Yet, we see few machine learning techniques adopted by
them. As machine learning becomes more popular in multimedia computing and
content understanding, it is interesting to see how machine learning can be
introduced to boost the performance of image/video coding performance. In this
work, we investigate how machine learning can be used in the optimal design of
the inverse discrete cosine transform (IDCT) targeting at quantization error
compensation.

Block transforms are widely used in image and video coding standards for
energy compaction in the spatial domain. Only a few leading DCT coefficients
have larger magnitudes after the transformation. Furthermore, a large number
of quantized coefficients become zeros after quantization. These two factors
contribute to file size reduction greatly. The 8x8 block DCT [9] is used in
JPEG. The integer cosine transform (ICT) is adopted by H.264/AVC for lower
computational complexity. All forward and inverse kernels are fixed in these
standards.

Research on IDCT has primarily focused on complexity reduction via new
computational algorithms and efficient software/hardware implementation. For
example, an adaptive algorithm was proposed in [10] to reduce the IDCT
complexity. A zero-coefficient-aware butterfly IDCT algorithm was investigated
in [11] for faster decoding. As to effective implementations, a fast
multiplier implementation was studied in [12], where the modified Loeffler
algorithm was implemented on the FPGA to optimize the speed and the area.
There is however little work on analyzing the difference of DCT coefficients
before and after quantization. In image/video coding, DCT coefficients go
through quantization and de-quantization before their inverse transform. When
the coding bit rate is high, quantized-and-then-dequantized DCT coefficients
are very close to the input DCT coefficients. Consequently, IDCT can
reconstruct input image patches reasonably well.

However, when the coding bit rate is low, this condition does not hold any
longer. The quantization effect is not negligible. The distribution of
quantized DCT coefficients deviates from that of original DCT coefficients.
The traditional IDCT kernel, which is derived from the forward DCT kernel, is
not optimal. In this work, we attempt to find the optimal IDCT kernel. This
new kernel is derived from the training set using machine learning approach.
Simply speaking, it transforms quantized DCT coefficients back to spatial-
domain pixel values, which is exactly what IDCT is supposed to do. Our
solution can account for the quantization effect through the training of real
world data so that the quantization error can be reduced in decoding. It can
improve the evaluation score and visual quality of reconstructed images
without any extra cost in decoding stage.

The rest of this paper is organized as follows. The impact of quantization on
the IDCT is analyzed in Sec. II. The new IDCT method is proposed in Sec. III.
Experiment results are presented in Sec. IV. Finally, concluding remarks and
future research directions are given in Sec. V.

##  II Impact of Quantization on Inverse DCT

The forward block DCT can be written as

${\bf X}=K{\bf x},$ (1)

where ${\bf x}\in R^{N}$, $N=n\times n$, is an N-dimensional vector that
represents an input image block of $n\times n$ pixels, $K\in R^{N\times N}$
denotes the DCT transform matrix, and ${\bf X}\in R^{N}$ is the vector of
transformed DCT coefficients. Rows of $K$ are mutually orthogonal.
Furthermore, they are normalized in principle. In the implementation of
H.264/AVC and HEVC, rows of $K$ may not be normalized to avoid floating point
computation, which could vary from one machine to the other. Mathematically,
we can still view $K$ as a normalized kernel for simplicity. Then, the
corresponding inverse DCT can be represented by

${\bf x}_{q}=K^{-1}{\bf X}_{q}=K^{T}{\bf X}_{q},$ (2)

where $K^{-1}=K^{T}$ since $K$ is an orthogonal transform and ${\bf X}_{q}$ is
the vector of quantized DCT coefficients. Mathematically, we have

${\bf X}_{q}=\mbox{DeQuan}(\mbox{Quan}({\bf X})),$ (3)

where $Quan(\cdot)$ represents the quantization operation and $DeQuan(\cdot)$
represents the de-quantization operation. If there is little information loss
in the quantization process (i.e., nearly lossless compression), we have ${\bf
X}_{q}\approx{\bf X}$. Consequently, the reconstructed image block ${\bf
x}_{q}$ will be close to the input image block, ${\bf x}$. Sometimes, a
smaller compressed file size at the expense of lower quality is used. Then,
the difference between ${\bf X}_{q}$ and ${\bf X}$ can be expressed as

${\bf e}_{q}={\bf X}-{\bf X}_{q}.$ (4)

In decoding, we have ${\bf X}_{q}$ rather than ${\bf X}$. If we apply the
traditional IDCT (i.e., $K^{-1}=K^{T}$) to ${\bf X}_{q}$, we can derive the
image block error based on Eqs. (2), (2) and (4) as

$\displaystyle{\bf e}_{q}={\bf x}-K^{-1}\mbox{DeQuan}[\mbox{Quan}(K{\bf x})].$
(5)

To minimize ${\bf e}_{q}$, we demand taht

$K_{q}^{-1}\approx\mbox{DeQuan}\odot\mbox{Quan}\odot K$ (6)

where $\odot$ denotes an element-wise operation. Since it is difficult to find
a mathematical model for the right-hand-side of Eq. (6), we adopt a machine
learning approach to learn the relationship between quantized DCT
coefficients, ${\bf X}_{q}$, and pixels of the original input block, ${\bf
x}$, from a large number of training samples. Then, given ${\bf X}_{q}$, we
can predict target ${\bf x}$.

##  III Proposed IDCT Method

We use image block samples of size $8\times 8$ to learn the optimal IDCT
matrix. The training procedure is stated below.

  1. 1.

Perform the forward DCT on each image block.

  2. 2.

Quantize DCT coefficients with respect to quality factor $QF$.

  3. 3.

Conduct linear regression between de-quantized DCT coefficients, which are the
input, and image pixels of input blocks, which are the output, to determine
optimal IDCT matrix $\hat{K}$.

Then, we will use $\hat{K}$, instead of $K^{-1}$ in Eq. (5), for the IDCT
task. Note that it is costly to train $\hat{K}$ at every QF value.

For Step 2, it is important to emphasize that there is no need to train and
store $\hat{K}$ for every QF value, which is too costly. Instead, we can train
and store several $\hat{K}$ for a small set of selected quality factors. Then,
according to the QF that is used to encode images, we can choose the K with
the closest QF to decode them. This point will be elaborated in Sec. IV.

For Step 3, we first reshape the 2D layout of image pixels and dequantized DCT
coefficients (both of dimension $8\times 8$) into 1D vectors of dimension
$64$. Notation ${\bf x}_{i}$ denotes the flattened pixel vector of the $i$th
image block and

${\bf X}_{q,i}=\mbox{DeQuan}[\mbox{Quan}(K{\bf x}_{i})]$ (7)

denotes the flattened dequantized DCT coefficients of the $i$th image block.
Next, we form data matrix $P\in R^{64\times N}$ of image pixels and data
matrix $D\in R^{64\times N}$ of dequantized DCT coefficients. Columns of
matrix $P$ are formed by ${\bf x}_{i}$ while columns of matrix $D$ are formed
by ${\bf X}_{q,i}$, $i=1,\cdots,N$. Then, we can set up the regression problem
by minimizing the following objective function

$\xi(\hat{K})=\sum_{i=1}^{N}||{\bf e}_{i}||^{2},$ (8)

where $||\cdot||$ is the Euclidean norm and ${\bf e}_{i}$ is the column vector
of error matrix

$E=P-\hat{K}D,$ (9)

and where learned kernel, $\hat{K}$, is the regression matrix of dimension
$64\times 64$. Typically, the sample number, $N$, is significantly larger than
$64^{2}=4,096$.

Figure 1: The $L_{2}$ norm of differences between kernels learned with
different QFs.

(a) Kodak PSNR versus QF

(b) DIV2K PSNR versus QF

(c) MBT PSNR versus QF

(d) Kodak SSIM versus QF

(e) DIV2K SSIM versus QF

(f) MBT SSIM versus QF

Figure 2: Comparison of quality of decoded test images using the standard IDCT
in JPEG and the proposed optimal IDCT for the Kodak, DIV2K and MBT datasets.

(a) Input image from Kodak

(b) Zoom-in

(c) JPEG

(d) Ours

Figure 3: Visualization of a zoom-in region of an input Kodak image, the
decoded region by JPEG and the proposed method with QF=70.

(a) Input image from DIV2K

(b) Zoom-in

(c) JPEG

(d) Ours

Figure 4: Visualization of a zoom-in region from an input DIV2K image, the
decoded region by JPEG and the proposed method with QF=70.

(a) Input image from MBT

(b) Zoom-in

(c) JPEG

(d) Ours

Figure 5: Visualization of a zoom-in region of an input image from MBT, the
decoded region by JPEG and the proposed method with QF=70.

##  IV Experiments

Experimental Setup. We perform experiments on three datasets with the libjpeg
software [13] on a Unix machine. The three datasets are: Kodak [14], DIV2K
[15], and Multiband Texture (MBT) [16]. The first two datasets contain generic
images. They are used to test the power of the proposed IDCT method in typical
images. The third dataset has specific content and it is desired to see
whether our solution can be tailored to it for more performance gain. For each
dataset, we split images into disjoint sets - one for training and the other
for testing. There are 24, 785 and 112 total images in Kodak, DIV2K, and MBT
datasets, and we choose 10, 100 and 20 from them as training images,
respectively. The DCT remains the same as that in libjpeg. The IDCT kernel is
learned at a certain QF based on discussion in Sec. III. Afterwards, the
learned kernel is used to decode images compressed by libjpeg. Both RGB-PSNR
and SSIM [17] metrics are used to evaluate the quality of decoded images.

Impact of QF Mismatch. It is impractical to train many kernels with different
QF values. To see the QF mismatch effect on kernel learning, we show the
$L_{2}$ norm of differences between learned kernels derived by different QFs
in Fig. 1. We see from the figure that the kernel learned from a certain QF is
close to those computed from its neighbouring QF values except for very small
QF values (say, less than 20). When QF is very small, quantized DCT
coefficients have a lot of zeros especially in high frequency region. This
makes linear regression poor. The IDCT kernel derived from these quantized
coefficients contains more zeros in each column which makes it different from
others. We plot the PSNR value and the SSIM value of decoded test images using
the standard IDCT in JPEG and the proposed optimal IDCT as a function of QF in
Fig. 2. We see a clear gain across all QFs and all datasets. Furthermore, we
show the averaged PSNR and SSIM gains offered by the proposed IDCT designed
with the fixed QF values over the standard DCT in Table I. The PSNR gain
ranges from 0.11-0.30dB.

Dataset QF |  | 50 | 70 | 90
---|---|---|---|---
Kodak | PSNR (dB) | +0.1930 | +0.2189 | +0.2057
SSIM | +0.0029 | +0.0025 | +0.0012
DIV2K | PSNR (dB) | +0.1229 | +0.1488 | +0.1144
SSIM | +0.0024 | +0.0020 | +0.0008
MBT | PSNR (dB) | +0.1603 | +0.2537 | +0.3038
SSIM | +0.0025 | +0.0022 | +0.0005
TABLE I: Evaluation results on test images in the Kodak, DIV2K, and MBT
datasets, where training QFs are set to 50, 70 and 90 for each column.

Quality Comparison via Visual Inspection. We show a zoom-in region of three
representative images, each of which is selected from Kodak, DIV2K and MBT
datasets using the standard IDCT and the proposed IDCT for visual inspection
in Figs. 3, 4 and 5, respectively. The proposed IDCT method gives better edge,
smooth and texture regions over the standard IDCT in these three examples.
Specifically, edge boundaries suffer less from the Gibbs phenomenon due to the
learned kernel. Similarly, texture regions are better preserved and the smooth
regions close to edge boundaries are smoother using the learned kernel. All
familiar quantization artifacts decrease by a certain degree.

(a) PSNR versus QF

(b) SSIM versus QF

Figure 6: Comparison of quality of decoded images using the standard IDCT and
the proposed IDCT trained by the Kodak dataset yet tested on the DIV2K
dataset.

Impact of Image Content. Another phenomenon of interest is the relationship
between image content and the performance of the proposed IDCT method. On one
hand, we would like to argue that the learned IDCT kernel is generally
applicable. It is not too sensitive to image content. To demonstrate this
point, we use 10 images from the Kodat dataset to train the IDCT kernel with
QF=70 and then apply it to all images in the DIV2K dataset. This kernel offers
a PSNR gain of 0.24dB over the standard IDCT kernel. On the other hand, it is
still advantageous if the training and testing image content matches well. As
shown in Table I, MBT has a higher PSNR gain than Kodak and DIV2K. It is well
known that texture images contain high-frequency components. When QF is
smaller, these components are quantized to zero and it is difficult to learn a
good kernel. Yet, when QF is larger, high-frequency components are retained
and the learned kernel can compensate quantization errors better for a larger
PSNR gain on the whole dataset.

##  V Conclusion and Future Work

An IDCT kernel learning method that compensates the quantization effect was
proposed in this work. The proposed method adopts a machine learning method to
estimate the optimal IDCT kernel based on the quantized DCT coefficients and
the desired output block of image pixels. Extensive experiments were conducted
to demonstrate a clear advantage of this new approach. The learned kernel is
not sensitive to the learning QF values neither to the image content. It
offers a robust PSNR gain from 0.1 to 0.3 dB over the standard JPEG. Since it
is used in the decoder, it does increase encoding time or the compressed file
size. The learned kernel can be transmitted offline as an overhead file or
simply implemented by the decoder alone.

It is interesting to consider region adaptive kernels. For example, we can
roughly categorize regions into smooth, edge and textured regions. The
distribution of DCT coefficients in these regions are quite different. Thus,
we can use a clustering technique to group similar DCT coefficient
distributions and conduct learning within a cluster. Furthermore, we may
consider separable IDCT kernels since they can be implemented more
effectively. Finally, it is desired to apply the same idea to video coding
such as H.264/AVC and HEVC for further performance improvement.

## References

  * [1] G. K. Wallace, “The jpeg still picture compression standard,” _IEEE transactions on consumer electronics_ , vol. 38, no. 1, pp. xviii–xxxiv, 1992\.
  * [2] M. Rabbani, “Jpeg2000: Image compression fundamentals, standards and practice,” _Journal of Electronic Imaging_ , vol. 11, no. 2, p. 286, 2002\.
  * [3] _Better Portable Graphics_. [Online]. Available: https://bellard.org/bpg/
  * [4] K. Brandenburg and G. Stoll, “Iso/mpeg-1 audio: A generic standard for coding of high-quality digital audio,” _Journal of the Audio Engineering Society_ , vol. 42, no. 10, pp. 780–792, 1994.
  * [5] B. G. Haskell, A. Puri, and A. N. Netravali, _Digital video: an introduction to MPEG-2_. Springer Science & Business Media, 1996.
  * [6] F. C. Pereira, F. M. B. Pereira, F. C. Pereira, F. Pereira, and T. Ebrahimi, _The MPEG-4 book_. Prentice Hall Professional, 2002.
  * [7] T. Wiegand, G. J. Sullivan, G. Bjontegaard, and A. Luthra, “Overview of the h. 264/avc video coding standard,” _IEEE Transactions on circuits and systems for video technology_ , vol. 13, no. 7, pp. 560–576, 2003.
  * [8] G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, “Overview of the high efficiency video coding (hevc) standard,” _IEEE Transactions on circuits and systems for video technology_ , vol. 22, no. 12, pp. 1649–1668, 2012\.
  * [9] N. Ahmed, T. Natarajan, and K. R. Rao, “Discrete cosine transform,” _IEEE transactions on Computers_ , vol. 100, no. 1, pp. 90–93, 1974.
  * [10] I.-M. Pao and M.-T. Sun, “Modeling dct coefficients for fast video encoding,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 9, no. 4, pp. 608–616, 1999.
  * [11] S.-h. Park, K. Choi, and E. S. Jang, “Zero coefficient-aware fast butterfly-based inverse discrete cosine transform algorithm,” _IET Image Processing_ , vol. 10, no. 2, pp. 89–100, 2016.
  * [12] A. B. Atitallah, P. Kadionik, F. Ghozzi, P. Nouel, N. Masmoudi, and P. Marchegay, “Optimization and implementation on fpga of the dct/idct algorithm,” in _2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings_ , vol. 3. IEEE, 2006, pp. III–III.
  * [13] _libjpeg_. [Online]. Available: http://libjpeg.sourceforge.net
  * [14] _Kodak images_. [Online]. Available: http://r0k.us/graphics/kodak/
  * [15] E. Agustsson and R. Timofte, “Ntire 2017 challenge on single image super-resolution: Dataset and study,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , July 2017.
  * [16] S. Abdelmounaime and H. Dong-Chen, “New brodatz-based image databases for grayscale color and multiband texture analysis,” _ISRN Machine Vision_ , vol. 2013, 2013.
  * [17] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE transactions on image processing_ , vol. 13, no. 4, pp. 600–612, 2004.