# UAV Immersive Video Streaming: A Comprehensive Survey, Benchmarking, and
Open Challenges

Mohit K. Sharma, Chen-Feng Liu, , Ibrahim Farhat, Nassim Sehad, Wassim
Hamidouche,
and Mérouane Debbah Mohit Sharma, Chen-Feng Liu, Ibrahim Farhat, and Wassim
Hamidouche are with the Technology Innovation Institute, Masdar City, Abu
Dhabi 9639, United Arab Emirates (e-mail: firstname.lastname@tii.ae).Nassim
Sehad is with Aalto University, 02150 Espoo, Finland (e-mail:
nassim.sehad@aalto.fi).Mérouane Debbah is with Khalifa University, Abu Dhabi,
United Arab Emirates (e-mail: merouane.debbah@ku.ac.ae).

###### Abstract

Over the past decade, the utilization of unmanned aerial vehicles has
witnessed significant growth, owing to their agility, rapid deployment, and
maneuverability. Among various applications, capturing videos using UAVs has
emerged as a prominent and promising use case, enabling diverse applications
such as remote surveillance and gaming. In particular, the use of UAV-mounted
360-degree cameras to capture omnidirectional videos has enabled truly
immersive viewing experiences with up to six degrees of freedom (6DoF).
However, achieving this immersive experience necessitates encoding
omnidirectional videos in high resolution, leading to increased bitrates.
Consequently, new challenges arise in terms of latency, throughput, perceived
quality, and energy consumption for real-time streaming of such content. This
paper presents a comprehensive survey of research efforts in UAV-based
immersive video streaming, benchmarks popular video encoding schemes, and
identifies open research challenges. Initially, we review the literature on
360-degree video coding, packaging, and streaming, with a particular focus on
standardization efforts to ensure interoperability of immersive video
streaming devices and services. Subsequently, we provide a comprehensive
review of research efforts focused on optimizing video streaming for time-
varying UAV wireless channels. Additionally, we introduce a high-resolution
360-degree video dataset captured from UAVs under different flying conditions.
This dataset facilitates the evaluation of complexity and coding efficiency of
software and hardware video encoders based on popular video coding standards
and formats, including AVC/H.264, HEVC/H.265, VVC/H.266, VP9, and AV1. Our
results demonstrate that HEVC achieves the best trade-off between coding
efficiency and complexity through its hardware implementation, while AV1
format excels in coding efficiency through its software implementation,
specifically using the libsvt-av1 encoder. Furthermore, we present a real
testbed showcasing 360-degree video streaming over a UAV, enabling remote
control of the drone via a 5G cellular network. Finally, we discuss open
challenges and outline future research directions for efficient and low-
latency immersive video streaming using UAV.

###### Index Terms:

UAV, 360∘, low latency, video coding.

##  I Introduction

Immersive video technology enables users to experience a quasi-realistic
virtual environment, fostering engagement and a sense of presence in a digital
space. Various visual media modalities, such as volumetric, light field, and
omnidirectional video (ODV), have emerged as viable options for delivering an
immersive viewing experience [1]. Among these, ODV, commonly known as
360-degree video, has gained widespread popularity due to the availability of
acquisition and display devices, as well as standardization efforts ensuring
interoperability. Integrating real-time transmission of 360-degree video using
a UAV-mounted camera enhances the immersive viewing experience by adding an
extra degree of freedom through UAV mobility. This advancement holds promise
for diverse applications like remote video surveillance, scientific
exploration, autonomous manufacturing assistance, agricultural monitoring, and
more. However, this acquisition system for 360-degree video presents new
challenges in delivering high quality of experience (QoE), primarily due to
the limited computational and energy resources of UAVs and the rapid
fluctuations of wireless channels. Additionally, the need for high-quality
video and ultra-low end-to-end (E2E) latency becomes crucial to ensure real-
time control of the UAV, especially in dynamic mobility conditions, further
amplifying these challenges.

Addressing the above challenges will require efforts to enhance the
communication for UAVs and develop adaptive and low-complexity schemes for
360∘ video encoding and streaming. In Table I, we present a summary of recent
efforts [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18] surveying
the state-of-the-art research on communication for UAVs and immersive
streaming. The literature in Table I can be broadly classified into two
categories: covering the communication aspects of UAVs or the streaming of
360∘ videos. The authors in [2, 3, 4] presented a comprehensive survey of
challenges and fundamental tradeoffs in designing wireless networks involving
the UAVs. In particular, Mozaffari _et al_. [2] described various analytical
frameworks and tools to address the design challenges, and Hayat _et al_. [4]
surveyed the quality of service, connectivity, safety, and other general
networking requirements for unmanned aircraft systems in civilian
applications. Similarly, Baltaci _et al_. [5] reviewed the connectivity
requirements for aerial vehicles, especially for piloting applications, and
advocated achieving these stringent connectivity requirements through multi-
technology heterogeneous networks. In [3, 6], the authors evaluated various
enabling 6G technologies highlighting their benefits and drawbacks regarding
their integration in a 6G wireless network with UAVs. They also discussed the
design issues associated with integrating the UAVs into the wireless networks
in the presence of these technologies. Further, authors in [7] surveyed the
channel models for air-to-ground and air-to-air channel models for UAV
communication.

TABLE I: Summary of the State of the Art

Category | Summary
---|---
Communication for UAVs |  |  Covered topics | References
---|---
|  • Fundamental design tradeoffs
---
• Connectivity requirements
[2, 5, 4]
|  Enabling technologies for UAV’s integration into 6G networks:
---
• Intelligent reflecting surfaces
• millimeter wave (mmWave) connectivity
• Short-packet communication
• Integrated communication and sensing
[3, 6]
Wireless channel model for UAV-to-ground channel and vice versa | [7]
mmWave-enabled UAV wireless networks | [8]
Joint communication, control, and computation design | [9, 14, 10]
Standardization, experimentation, and prototyping | [12, 13, 11]
Interference issues in UAV networks | [11, 9]
360∘ video streaming |  | Various aspects of 360∘ streaming: |
---|---
• 360∘ video delivery architecture |
• Viewport dependent, viewport independent, and tile based solutions | [15, 18]
Issues pertaining to networks for 360∘ streaming | [15, 16]
Compression and coding for 360∘ streaming | [16]
Video streaming ( two-dimensional (2D)) from aerial platforms | [17]

To meet high data rate requirements for UAVs, Xiao _et al_. [8] reviewed
relevant antenna structures and channel models for mmWave. Furthermore, the
technologies and solutions for UAV-connected mmWave cellular networks and
mmWave-UAV ad hoc networks were discussed. The authors in [9] and [10]
reviewed the methods for communication and trajectory co-design. In addition,
Zeng _et al_. [9] surveyed various techniques to deal with the air-to-ground
interference issues in cellular communication with UAVs. Fotouhi _et al_. [11]
investigated the interference issues and potential solutions addressed by
standardization bodies for serving aerial users with the existing terrestrial
base stations. In addition, they reviewed the ongoing prototyping, testbed
activities, and regulatory efforts to manage the commercial use of UAVs, along
with cyber-physical security of UAV-assisted cellular communication. In [12],
Marojevic _et al_. presented the architecture of aerial experimentation and
research platform for advanced wireless, which facilitates experimental
research in controlled yet production-like environments. In [13], Abdalla _et
al_. surveyed the ongoing Third Generation Partnership Project (3GPP)
standardization activities for enabling networked UAVs, requirements,
envisaged architecture, and services provided by UAVs. The authors in [14]
studied the UAV networks from the perspective of cyber-physical systems and
considered the joint design of communication, computation, and control to
improve the performance of UAV networks.

On the other hand, the work in [15, 16, 17] surveyed the adaptive streaming
techniques for 360∘ videos. Yaqoob _et al_. in [15] reviewed the adaptive 360∘
video approaches that dynamically adjust the size and quality of the viewport.
In addition, they surveyed the standardization efforts for 360∘ video
streaming, highlighting the main research challenges such as viewport
prediction, QoE assessment, and low latency streaming for both the on-demand
and live 360∘ video streaming. Further, [16] surveyed the field-of-view (FoV)
prediction methods, compression, and coding schemes for reducing the bandwidth
required for streaming immersive videos. In addition, they reviewed caching
strategies and datasets for immersive video streaming. The work in [17]
focused on the issues pertaining to 2D video streaming from an aerial
platform. In particular, they surveyed the works using artificial intelligence
(AI)-based techniques for enhancing video transmission performance.

We emphasize that none of the above papers have explicitly surveyed the issues
relevant to immersive video streaming from a UAV platform, which poses unique
challenges regarding computational complexity, quality, and E2E latency. In
this work, we present a survey of solutions related to state-of-the-art
schemes for immersive video streaming from a UAV and benchmark the existing
video encoding schemes. In this direction, our contributions are the
following:

  * •

Review exiting wireless communication techniques for video streaming using
UAV.

  * •

Build a new dataset of immersive 360∘ videos captured from UAV in different
acquisition conditions and scenes.

  * •

Assess the coding efficiency and complexity of software and hardware encoders
of five video standards and formats for immersive 360∘ video streaming.

  * •

Discuss open challenges related to ODV streaming over UAV.

The rest of this paper is organized as follows. Section II presents the main
components of the ODV streaming chain, including acquisition, encoding,
packaging, rendering, and optimization. Then, the key performance metrics and
wireless optimization techniques for UAV 360∘ video streaming are presented in
Sections III and IV, respectively. In Section V, first the proposed UAV 360∘
video dataset is presented, and then benchmark and analysis of software and
hardware encoders of five video standards are provided in Section VI. Next,
the challenges of ODV streaming over UAV are discussed in Section VII.
Finally, Section VIII concludes the paper.

##  II Omnidirectional Video Streaming

Figure 1 illustrates the E2E ODV streaming pipeline. In this section, we
briefly review the technology used at the different stages to deliver ODV to
the end user over the network.

###  II-A Acquisition and Preprocessing

An omnidirectional visual signal is presented in a spherical space with
angular coordinates: the azimuth angle $\phi\in[\pi,-\pi]$, and the elevation
or polar angle $\theta\in[-\frac{\pi}{2},\frac{\pi}{2}]$, assuming a unit
sphere (radius $r=1$) for acquisition and rendering. The sphere’s origin
represents the viewing reference that captures the light coming from all
directions. In practical applications, an omnidirectional visual signal is
captured using a multi-view wide-angle acquisition system, often utilizing
fish-eye lenses. While a single eye-fish camera can only capture a partial
sphere, combining multiple acquisitions from such cameras allows for complete
sphere coverage through the process of stitching the images together [19].
However, the stitching operation introduces two main challenges. The first
challenge involves blending and wrapping non-overlapping captured images,
while also addressing inconsistencies in illumination and color that may arise
after stitching. The second challenge arises when dealing with video signals,
as the camera sensors need to be perfectly synchronized. To facilitate further
processing, the omnidirectional visual signal in spherical representation is
then mapped to a 2D texture signal during the pre-processing stage, prior to
encoding using conventional 2D video coding standards. The most commonly used
mapping technique is known as equirectangular projection (ERP), which is
particularly well-suited for production and contribution purposes. However,
more advanced mapping techniques, such as equi-angular cubemap (EAC), cube map
projection (CMP), and truncated square pyramid (TSP), have been proposed in
the literature [20]. Notably, CMP and TSP offer enhanced coding efficiency,
achieving bitrate savings of 25% and 80%, respectively, compared to ERP,
making them more suitable for distribution purposes [21].

Figure 1: ODV E2E streaming pipeline.

###  II-B Encoding

After mapping the sphere in a 2D plane, ODV content is encoded in practice by
conventional 2D video standards such as advanced video coding (AVC)/H.264
[22], high-efficiency video coding (HEVC)/H.265 [23] versatile video coding
(VVC)/H.266 [24], as well as VP9 and AOMedia video 1 (AV1) video formats. In
particular, tailored coding tools are integrated into the HEVC/H.265,
VVC/H.266 standards to enhance the ODV coding efficiency and enable advanced
streaming features, improving the user’s QoE. Yet, some non-normative coding
techniques were proposed in the literature, encoding the ODV content in
spherical representation to prevent projection distortions, resulting in
higher coding efficiency.

####  II-B1 HEVC/H.265 Tools for ODV

The tile concept in HEVC/H.265 plays a crucial role in enabling independent
and parallel encoding/decoding of rectangular regions within the picture. By
breaking the dependency of context prediction in arithmetic encoding and intra
prediction, tiles allow for efficient processing and coding of specific
regions [25]. Additionally, the tile boundaries also enable the possibility of
disabling in-loop filters, further enhancing the flexibility of the encoding
process. Moreover, the introduction of the motion-constrained tile set (MCTS)
technique in HEVC/H.265, along with supplemental enhancement information (SEI)
messages, extends the tile concept to the sequence of frames. This technique
restricts the MV to a selected set of tiles in the reference picture, thereby
enabling the download and decoding of only the tiles within the displayed
viewport during ODV streaming. This approach significantly improves the user’s
QoE by delivering high-quality content while efficiently utilizing bandwidth.
However, the limitation of restricting motion vectors within a set of tiles in
the reference picture can have a negative impact on coding efficiency. To
overcome this, the literature proposes non-normative solutions that enhance
inter-prediction by utilizing the base layer as a reference in the scalable
HEVC extension [26]. Alternatively, Bidgoli et al. [27] propose an enhanced
intra-prediction technique with fine granularity random access capability,
allowing end-users to request specific parts of the stream while ensuring
efficient intra-coding. Furthermore, in the context of spherical bitrate
allocation, a new entropy equilibrium optimization (EEO) strategy is proposed
in [28]. This strategy derives the Lagrangian multiplier at the block level,
which is used in rate-distortion optimization. The proposed solution,
evaluated with ERP and CMP projection methods, demonstrates significant
bitrate gains when compared to the HEVC reference software encoder [28].

####  II-B2 VVC/H.266 Tools for ODV

The VVC/H.266 standard introduces several advancements for efficient encoding
of ODV content, including the ability to signal the used projection technique
and the definition of tailored coding tools [24]. In the case of 360-degree
representation and ERP mapping, objects can span across the left and right
picture boundaries continuously. Consequently, in VVC/H.266, inter-prediction
samples may wrap around from the opposite left or right boundary when MVs
point outside the coded area. Additionally, virtual boundaries are defined to
skip in-loop filters across edges. For CMP projection, where cube maps may
exhibit content discontinuities, virtual boundaries can be signaled to disable
in-loop filtering and prevent artifacts arising from non-homogeneous
boundaries. Furthermore, VVC/H.266 introduces the concept of subpictures,
which allows for the extraction of independent rectangular regions within the
picture, specifically designed for viewport-dependent VVC streaming
applications. Subpictures offer two critical improvements over the previous
MCTS concept. Firstly, subpictures enable MVs to refer to blocks outside the
subpicture, and padding at subpicture boundaries is permitted, similar to
picture boundaries. This new feature demonstrates higher coding efficiency
compared to the tight motion constraints applied in MCTS. Secondly, the need
to rewrite slice headers when extracting a sequence of subpictures to build a
new VVC/H.266 compliant bitstream is eliminated, streamlining the encoding
process [24].

####  II-B3 Learning-Based Coding for ODV

Machine learning techniques have been extensively investigated in the
literature to optimize and improve the coding efficiency of ODV content. In
[29] a convolution neural network (CNN) was trained to learn the rotation of
the sphere, resulting in the highest coding efficiency. This rotation is
applied as a pre-processing step along the spherical axis before projection,
leading to different rotations of the cube map. Experimental results
demonstrated that incorporating rotation prediction achieved a significant
coding gain of 8% to 10% with a prediction accuracy of 80%.

Similar to conventional video standards, learning-based video codecs can
encode ODV content after its projection onto a 2D plane. Initially, the 2D
representation is transformed into a compact latent space using an analysis
transform based on an artificial neural network (ANN). The resulting latent
representation is then encoded with a lossless entropy encoder to construct
the bitstream. At the decoder side, a synthesis transform, also based on an
ANN, reconstructs a version of the input 2D representation from the received
bitstream. Moreover, the hyperparameters of the latent space entropy
distribution, such as mean and variance, are encoded using an auto-encoder and
utilized by the encoder and decoder to enhance the performance of the entropy
encoder [30].

###  II-C Transport Protocols

Various packaging protocols can be employed for streaming ODV content,
allowing the selection of a suitable protocol based on the specific
application and end-user requirements concerning video quality, latency, and
advanced functionalities provided by the protocol [31]. In the followin, we
outline the key characteristics of two widely utilized streaming protocols:
omnidirectional media format (OMAF) and web real-time communication (WebRTC).
Specifically, we provide an overview of the primary features of these
protocols for ODV streaming. For a more comprehensive understanding, readers
are encouraged to consult the comprehensive overview papers on OMAF [32] and
WebRTC [33].

####  II-C1 OMAF

The ISO/IEC 23090-2 standard, also known as OMAF, is a system standard
developed by motion picture experts group (MPEG) with the objective of
ensuring device and service interoperability for storing and streaming
omnidirectional media content. This includes various forms of media such as
360∘ images and videos, spatial audio, and associated text. The initial
version of the standard, completed in October 2017, provides fundamental tools
for streaming 360∘ images and videos, enabling a three degrees of freedom
(3DoF) viewing experience. In the subsequent release of the standard in
October 2020, the second version introduced additional tools to support more
advanced features. These features include enhanced viewport-dependent
streaming, overlay capabilities, and the ability to stream multiple
viewpoints, marking the initial steps towards achieving a 6DoF viewing
experience. The specifications of OMAF are organized into three main modules:
content authoring, delivery, and player. Furthermore, these specifications
serve as extensions to the ISO base media file format (ISOBMFF) and dynamic
adaptive streaming over HTTP (DASH), ensuring backward compatibility with
conventional 2D media formats. OMAF supports three types of omnidirectional
visual signal representations: projected, mesh, and fish-eye. Each of these
formats requires specific pre-processing for encoding and post-processing for
rendering and display. Among the projected formats, OMAF includes support for
two widely used projection algorithms: ERP and CMP. Additionally, OMAF
incorporates a region-wise packing (RWP) operation, which allows for optional
pre-processing operations prior to encoding. These operations include
resizing, repositioning, rotation by 90∘, 180∘, and 270∘, as well as vertical
and horizontal mirroring of specific rectangular regions. rwp serves various
purposes, such as signaling the exact coverage of a partial spherical
representation, generating viewport-specific (VS) video, enhancing coding
efficiency, or compensating for over-sampling in the pole areas of ERP. The
RWP metadata indicates the applied operations to the player, which then
performs inverse operations to map the regions of the decoded picture back
into the projected picture. This ensures proper rendering and display of the
content, aligning with the intended transformations specified by the RWP.

In addition, the OMAF standard offers tools for viewport-dependent ODV
streaming, which enables the selection of segments covering the user’s
viewport at high quality and other segments at lower quality and bitrate. This
approach allows for more efficient utilization of network bandwidth, resulting
in an improved user experience. Viewport-dependent ODV streaming can be
achieved through two methods: viewport-specific and tile-based streaming. In
the viewport-specific approach, multiple VSs are created and signaled, each
encoding different viewports at high quality. Users can select the appropriate
VS stream based on their viewing orientation. The OMAF region-wise quality
ranking (RWQR) metadata can be used to signal the quality of different regions
in the sphere. In the tile-based configuration, the ODV is divided into
independent rectangular regions called tiles. Each tile only depends on the
co-located tile in the sequence and can be decoded independently from other
tiles. There are two alternatives for encoding video in independent regions.
The first method utilizes the HEVC tile concept, where tiles are grouped into
motion-constrained slices known as motion-constrained tile set. The second
method, applicable to AVC which does not support tiles, partitions the video
into sub-picture sequences, each representing a spatial subset of the original
sequence. These sub-picture sequences are then encoded with motion constraints
and merged into tiles in a single bitstream. Each tile or sub-picture sequence
is stored in its respective track. Additionally, tiles can be encoded in
different bitrates and resolutions, allowing users to select the optimal
combination of tiles based on their viewing orientation, available bandwidth,
and decoding capability.

The OMAF standard specifies six video media profiles that define the type of
video representation and the supported video standard with its associated
levels. For example, the ”HEVC-based viewport-independent” profile uses the
ERP projected representation and is constrained to HEVC Main 10 profile level
5.1. This level limits the spatial resolution to 4K (4096 $\times$ 2048).
However, the ”unconstrained HEVC-based viewport-independent” profile,
introduced in the second edition, supports all HEVC Main 10 profile levels,
thus increasing the decoding capacity and display resolution. Furthermore,
there are already several open-source implementations available that support
the first111NOKIA: https://github.com/nokiatech/omaf, Fraunhofer HHI:
https://github.com/fraunhoferhhi/omaf.js, Intel Open Visual Cloud:
https://github.com/OpenVisualCloud/Immersive-Video-Sample. edition of the OMAF
standard. Further, some tools of the OMAF second edition have been
demonstrated in [34, 35].

####  II-C2 WebRTC

The WebRTC framework is an open-source solution specifically designed to
facilitate real-time and low-latency video transmission. Within the WebRTC
transmitter, the ”video collector” module takes on the responsibility of video
encoding and encapsulating the encoded video frames into real-time transport
protocol (RTP) packets. These packets are subsequently transmitted using the
secure real-time transport protocol (SRTP) protocol. On the receiver side,
relevant information regarding the received RTP packets is collected, and this
information is relayed back to the ”video collector” through the transport-
wide feedback message of the real-time transport control protocol (RTCP)
protocol. The ”bandwidth controller” module, located within the ”video
collector,” utilizes these control messages to compute essential network
metrics such as inter-packet delay variation, queuing delay, and packet loss.
These metrics play a crucial role in determining the target bitrate, which is
then employed by the rate control module of the video encoder. The rate
control module dynamically adjusts the encoding parameters, such as the
quantization parameter and resolution, based on the target bitrate
requirements. Although the standard WebRTC implementation does not offer
explicit tools for transmitting immersive video, it has gained significant
popularity for real-time and ultra-low latency ODV transmission by treating
360∘ video representation as a conventional 2D video [36, 37]. Additionally,
viewport-dependent streaming can be effectively supported by incorporating a
combination of high-resolution and low-resolution tiles. This approach
optimizes bandwidth utilization while ensuring high quality within the field
of view, all while maintaining a low motion-to-photon latency [38].

###  II-D Rendering and Display

The human visual system has a limited field of view, which means that users
cannot directly perceive the entire 360∘ content in its spherical
representation. Instead, only a portion of the sphere, known as the
”viewport,” is displayed, which is an image tangent to the sphere. To enhance
immersion, interaction with the user is crucial. This interaction can involve
head movements (roll, yaw, and pitch), mouse/keyboard controls, or in the case
of viewing on a smartphone, the viewing angle can be controlled by moving the
device in space, providing a visual experience of up to 3DoF. However, one of
the main limitations of ODV is the absence of motion parallax, which refers to
the relative position of objects changing based on the viewer’s position
relative to the object. This limitation can lead to discomfort and sickness
for users. To address this limitation, a potential solution is to employ a
360∘ camera mounted on an UAV. This combination offers enhanced flexibility
and mobility, allowing users to explore the environment and move around
objects within the scene. By leveraging the mobility provided by the UAV along
with the 360∘ video, a viewing experience of up to 6DoF can be achieved.
However, to enable interactive control of the UAV and ensure a more natural
viewing experience with accurate control, it is essential to have ultra-low
E2E latency (below 100 ms) and high visual quality. These factors are crucial
in order to maintain a seamless and responsive interaction between the user
and the UAV.

###  II-E ODV Streaming Optimization

Several strategies have been proposed in the literature to enhance the QoE in
streaming ODV [39]. Depending on the specific application, different metrics
can be optimized, including perceived video quality, storage cost, bandwidth
usage, and various latency measures (e.g., end-to-end, motion-to-photon, or
motion-to-high-resolution/quality latency). Low motion-to-photon latency is
particularly important to minimize user discomfort when changing the displayed
viewport. Additionally, achieving low end-to-end latency is crucial for live
ODV streaming applications, such as UAVs, to enable accurate remote control,
especially during high-speed flying conditions. As shown in Figure 2, ODV
streaming strategies can be categorized as either viewport-dependent or
viewport-independent, depending on whether the field of view is considered in
the optimization process or not. The initial streaming approach, commonly used
in early literature, involved transmitting the entire 360-degree content at
high quality, allowing users to extract the desired viewport based on their
head position with ultra-low motion-to-high-resolution latency. This approach
aligns with the viewport-independent profiles defined in the OMAF standard.
However, it is a bandwidth-intensive solution, requiring over 100 Mbps to
transmit an 8K resolution video at high quality [40]. This is inefficient
since the end user only observes a small portion (15%) of the ODV. To address
this limitation, more advanced techniques have been proposed to provide users
with ODV services at high quality and low motion-to-photon latency. In this
context, viewport-dependent strategies have gained wide adoption at the
projection (projection-based) and encoding (tile-based) stages. The
projection-based approach employs dynamic projection methods, such as
pyramidal projection and its refined version, offset cubic projection [41].
Offset cubic projection allocates higher pixel density and better quality near
the offset direction, which corresponds to the user’s viewing direction.
Another solution proposed in [37] is oriented projection for real-time
360-degree video streaming using the WebRTC framework. Oriented projection
allocates more pixels in the projected frame to areas on the sphere that are
close to a target pixel-concentration orientation. This solution is jointly
optimized with adaptive resolution and bitrate allocation to accommodate
bandwidth variations.

The viewport-dependent OMAF profiles, as outlined in [41], enable independent
tile decoding by restricting motion vectors to refer only to the adjacent tile
area. This allows the end user to request and decode tiles independently.
Following the projection stage, the ODV is encoded into tiles representing
different quality representations. The end user can then request the tiles
covering the viewport at high quality while the remaining area tiles can be
requested at a lower quality. As a result, the tile-based ODV encoding using
viewport-dependent OMAF profiles significantly improves the user’s QoE and
reduces the required transmission bandwidth. However, storing and encoding ODV
tiles in multiple representations, each with different rate-quality
characteristics, necessitates a large storage capacity and incurs high encoder
computational complexity. To address these challenges, the VR Industry Forum
guidelines [42] introduced the HEVC-based FoV Enhanced Video Profile. This
profile employs HEVC encoding to achieve low-quality coverage of the entire
360-degree video, while high-quality sub-pictures are encoded to cover
specific regions of the video. Each bitstream is then encapsulated within a
track compliant with the HEVC-based viewport-dependent OMAF profile. The
player can subsequently request the bitstream covering the viewport in high
quality, along with the low-quality bitstream representing the entire
360-degree coverage. In live scenarios, the low-quality stream can be
transmitted via multicast, allowing for more efficient bandwidth utilization.
This approach ensures efficient bandwidth utilization while maintaining ultra-
low motion-to-photon latency.

The prediction of end user head movements can be leveraged to enhance the QoE
by assigning higher fetching priority to tiles within the predicted viewport.
This streaming approach, known as the ”human-centric” approach, focuses on
optimizing the user experience, in contrast to the ”system-centric” approach
that prioritizes overall system performance without considering user behavior.
The design can be categorized as single-user or cross-users, with the latter
considering the behavior of multiple users in predicting the viewport. These
techniques rely on accurate viewport prediction models, which are used to
optimize the streaming system. In [43], the potential of predicting head
movements for optimizing 360-degree video streaming over cellular networks was
demonstrated, resulting in up to 80% network bandwidth savings. This approach
has been followed by several research papers and commercial products, aiming
to optimize network and computational resources while providing users with a
highly immersive experience [44].

forked edges, for tree=draw,align=center,edge=-latex,fill=white,blur shadow,
where level=1 for descendants=grow’=0, folder, l sep’+=2.5pt, ,
[Omnidirectional video streaming, fill=gray!5 [Viewport-independent,
draw=green!80, fill=green!10 ] [Viewport-dependent [Tile-based, draw=red!80,
fill=red!10 [Client-based tile binding] [Author-based tile binding] ]
[Projection-based, draw=blue!80, fill=blue!10 ] ] ]

Figure 2: odv streaming strategies. TABLE II: Performance of streaming
approaches regarding storage capacity, bandwidth usage, motion-to-photon
latency, and encoding time.

Storage Bandwidth Latency Encoding time Viewport-indep. ••• ••• ••• •••
Projection-based ••• ••• ••• ••• Tile-based ••• ••• ••• •••

Performance metrics: High $\equiv$ •••, Average $\equiv$ •••, Low $\equiv$ •••

Finally, Table II depicts the performance of discussed 360∘ video streaming
strategies regarding storage cost, bandwidth usage, motion-to-photon latency,
and encoding time.

##  III UAV-Based Real-Time Video Streaming: Performance Metrics

This section focuses on the optimization of wireless networks for real-time
video streaming using UAVs, with a particular emphasis on key performance
metrics. The discussion encompasses three essential aspects: latency, video
quality, and UAV energy consumption.

###  III-A Latency

The latency in video transmission from UAVs significantly impact user’s QoE in
360-degree video streaming, including E2E latency. It is captured using
several metrics such as motion-to-photon latency, and motion-to-high
resolution latency.

End-to-end latency: In a point-to-point real-time video transmission, E2E
latency plays a vital role in ensuring a seamless and immersive experience. It
encompasses the total delay from event capture by the sensor to processing,
transmission, and actuator response. The E2E latency between camera and user’s
display is often referred to as glass-to-glass (G2G) latency. It measures the
time difference between when the photons of an event first pass through the
camera lens and when the event is displayed to the viewer. Additionally,
glass-to-algorithm (G2A) latency represents the time gap between the photon
corresponding to an event passing through the camera lens and the availability
of the first image corresponding to that event for processing before display.
G2A latency is crucial in applications utilizing computer vision algorithms
for tasks such as control, object detection, segmentation, viewport
prediction, and more. Figure 1 provides an overview of G2G latency and its
relationship to G2A latency by removing the latency introduced during the
display process.

The overall G2G latency can be expressed as the sum of delays incurred in
camera acquisition, encoding, network transmission, decoding, and display
processing. Table III presents a breakdown of G2G latency for a state-of-the-
art WebRTC-based implementation of an ODV E2E streaming pipeline [45]. This
pipeline transmits $8K$ resolution $360^{\circ}$ videos captured using an
Insta 360 camera to a Samsung S10 client. The latency breakdown in Table III
highlights that the acquisition and stitching process, along with the encoder,
contribute to approximately 80% of the total G2G latency. Notably, the latency
introduced at the transmitter scales proportionally with the video’s
resolution. At a high level, the total G2G latency comprises network latency
and latency stemming from the transmitter and client components.

TABLE III: Glass-to-Glass Latency Brakeup[45]

Block | Latency (ms)
---|---
Transmitter |  |  Live Streaming | 503
---|---
FFMpeg Decoder | 568
$360^{\circ}$ stitching | 28.5
HEVC encoder | 406
Video packetizer | 1.9
Total latency at transmitter | 1508
Client |  | RTP Packet | 79
---|---
Decoder | 34
Renderer | 14
Total latency at client | 127
Total G2G latency |  1745-1856

Based on the preceding discussion, it can be deduced that reducing latency
entails reducing the number of processed pixels throughout the ODV streaming
pipeline, which is primarily determined by the framerate and resolution.
Additionally, higher framerates and quality necessitate increased transmission
rates, resulting in heavier overheads in terms of transmission delay and
transmit power. Conversely, the E2E delay increases when the encoding bitrate
fails to adapt to wireless channel variations. As a result, efforts to
minimize latency have a direct impact on video quality. Hence, the design of
wireless communications for UAV-based ODV streaming predominantly revolves
around maximizing video quality while adhering to a latency constraint.
Typically, in wireless optimization problems, this latency constraint is
imposed as a _delay outage probability_ constraint, representing the
probability of packet delay exceeding a predefined delay budget. However, it
is important to note that the delay outage probability constraint only
encompasses queueing and transmission delays, which constitute only a portion
of the overall G2G delay.

Motion-to-photon latency: In the context of ODV streaming, particularly in
viewport-dependent scenarios, it is crucial to consider additional latency
metrics that can impact the user’s quality of service. Two such metrics are
motion-to-photon (M2P) latency and motion-to-high resolution latency. M2P
latency measures the delay required to display the new viewport corresponding
to the user’s updated viewing direction after head movement. It encompasses
the time needed to request and render the viewport aligned with the user’s
viewing direction. The specific streaming approach and the technology of the
head-mounted display (HMD) can influence the motion-to-photon latency.
Additionally, recent work presented in [46] demonstrates the potential of
utilizing head motion prediction algorithms at the end user’s side to
significantly reduce the motion-to-photon latency. These algorithms can
effectively anticipate the user’s head movements and optimize the rendering
process accordingly.

###  III-B odv quality

The quality of experience for end users is primarily determined by the
perceived video quality and latency. In the context of 2D video, widely used
full-reference objective quality metrics include peak signal-to-noise rate
(PSNR), structural similarity index measure (SSIM), and video multi-method
assessment fusion (VMAF). These metrics provide a comprehensive assessment of
the perceived quality by comparing the original and reconstructed videos.
However, for 360-degree video content, specialized quality metrics have been
proposed to account for the unique geometrical distortions introduced by the
spherical representation. Notable examples include Spherical PSNR (S-PSNR) and
weighted to spherically uniform PSNR (WS-PSNR), which are full-reference
objective quality metrics specifically developed for 360-degree video content
[47].

As mentioned earlier, video quality is influenced by various factors,
including encoding bitrate, frame resolution, frame rate, and the
characteristics of the air-to-ground wireless channel. Generally, higher
quality and lower distortion can be achieved by using a higher bitrate (or
resolution) and benefiting from favorable channel conditions. Consequently,
selecting a higher video bitrate is preferable for improved video
reconstruction quality. However, it is important to note that bitrate
selection not only affects video quality but also impacts latency. A higher
bitrate necessitates a more stringent throughput requirement, posing
challenges for efficient wireless resource allocation. Therefore, when
designing the system, a trade-off must be considered between reconstruction
quality/distortion and bitrate selection, while optimizing the provision of
wireless resources to meet the selected video bitrate.

###  III-C UAV Energy consumption/ Flight Time

uav are frequently required to maneuver in three-dimensional space to perform
various monitoring and video streaming missions. One crucial consideration in
UAV-based ODV streaming systems is the energy consumption of UAVs due to their
limited energy budget. The energy consumed by a UAV during movement is
referred to as ”propulsion energy,” which is influenced by the UAV’s velocity
and acceleration. Moreover, when the UAV hovers at a fixed position while
streaming the video, it consumes ”hovering energy” [48, 49]. Furthermore, as
discussed below, the air-to-ground channel between the UAV and the ground user
is implicitly affected by the UAV’s position in the 3D space. For example, the
small-scale fading component of the UAV-ground wireless channel can be modeled
as an ”angle-dependent Rician fading channel” with the Rician factors directly
proportional to the UAV-ground elevation angle [50]. This model captures the
fact that as the elevation angle increases, the UAV-ground link tends to
experience less scattering, resulting in a larger line-of-sight (LoS)
component. On the other hand, the large-scale fading component, which includes
path loss and shadowing, depends not only on the 3D locations of the UAV and
the ground user but also on the geographic distribution of buildings. In urban
areas, the signal propagation of a UAV flying at a lower altitude may be
obstructed by buildings, leading to the shadowing effect [51]. In contrast,
when the UAV transmits at a higher altitude, it only experiences path loss
without any shadowing. However, conducting a comprehensive path-loss
measurement for a wide geographic area is infeasible. Therefore, a generic
probabilistic aerial-to-ground channel model that statistically incorporates
both LoS and non-line-of-sight (NLoS) large-scale fading is considered in
[52]. In this model, the probability of experiencing LoS path loss increases
as the UAV raises its altitude or moves closer to the ground user
horizontally.

In conclusion, the trajectory and position of a UAV have an impact on its
energy consumption and the quality of the transmitted video. Hence, in the
deployment and trajectory design of UAVs for video streaming, the distinctive
features of the air-to-ground channel, as well as the propulsion and hovering
energy consumption, must be carefully taken into account. The following
section provides a more detailed exploration of these design challenges and
discusses the current state-of-the-art in each aspect.

##  IV Wireless Communications Design for UAV-Based Real-Time Video Streaming

This section provides a comprehensive survey of the ongoing research efforts
in the design and optimization of wireless communications for real-time video
streaming systems using UAVs. Additionally, we also review the relevant
standardization activities conducted by 3GPP.

###  IV-A Quality of Experience Maximization

To meet the demanding requirements of real-time transmission in high-
resolution video streaming systems with superior QoE, UAV-based ODV streaming
systems impose stringent criteria on both throughput and latency performance.
The latency of such systems is typically characterized by E2E latency or
photon-to-motion latency. The inherent randomness of wireless channels poses a
significant challenge in achieving desired QoE, as fluctuating channel
conditions result in unpredictable latency, leading to interrupted or choppy
video streaming. Maximizing QoE is generally approached as a problem of
maximizing PSNR through optimizing wireless resource allocation, including
transmit power, rate, or bandwidth, while adhering to various wireless network
and UAV-imposed constraints. In this section, we survey the state-of-the-art
advancements in this area. We note that most of the literature in this area
has focused only on transmission of 2D videos from UAVs.

One notable work by Xia et al. [53] utilized the internal sensor data of the
UAV for adaptive bitrate selection. They leveraged location, velocity, and
acceleration information to predict future throughput and proactively select
the video bitrate accordingly. The performance evaluation, conducted using a
laptop on the ground and the DJI Matrice 100 drone with an attached Android
smartphone in an outdoor environment, employed the IEEE 802.11n protocol. The
simulations demonstrated that the selected bitrates effectively adapted to
future throughput, maintaining relatively stable video bitrates over time,
resulting in a seamless video viewing experience despite channel fluctuations.
In another study, Muzaffar et al. [54] focused on a multicast video streaming
framework where a UAV delivers video to ground users. The proposed approach
incorporated feedback from the users to dynamically adjust the transmission
rate and video bitrate. The performance evaluation, conducted using the AscTec
Pelican drone equipped with a Logitech C920 camera and employing the IEEE
802.11a protocol and AVC/H.264 video format, investigated throughput, packet
loss, and delay. The rate-adaptation approach demonstrated improvements in
throughput, latency, and packet loss compared to a constant transmission rate
and bitrate baseline, resulting in up to 30% PSNR gain. These works represent
significant advancements in enhancing QoE through adaptive bitrate selection
and rate control mechanisms, showcasing the potential of optimizing wireless
communications in UAV-based video streaming systems.

In [55], the authors addressed the transmission rate allocation problem in a
UAV video streaming system, where multiple UAVs transmit their captured videos
to different users. The objective was to minimize the overall reconstruction
error of all users by optimizing the transmission rates, subject to the total
network channel capacity. The performance evaluation, conducted using PSNR as
a measure of video reconstruction quality, showed that the proposed rate
allocation approach achieved a 6 dB gain over the equal allocation baseline.
Extending the system model in [55], [56], considered a multi-UAV setup, where
UAVs competed for transmission rates by incurring a cost to obtain higher
rates. Each UAV aimed to maximize its utility, comprising PSNR and cost, by
selecting a transmission rate within the network capacity budget. The authors
designed a rate allocation algorithm using game theory to address the rate
competition among UAVs. Compared to the equal bandwidth allocation baseline,
the proposed algorithm increased network utility while considering video
quality requirements. In contrast to adapting the bitrate to match the channel
fluctuations, another stream of work [57, 58] attempts to maximize the PSNR by
using an scalable video coding (SVC) based video transmission. In SVC, the
video is encoded into a base layer and $N$ enhancement layers. If the $n$th
quality is selected for the streamed video, the base layer and all lower
enhancement layers, i.e., $1,\cdots,n-1$, have to be delivered along with the
$n$th layer [57]. Note that more enhancement layers give the better quality of
the received video, i.e., the higher PSNR, but require more transmit power at
the UAV. In [57], Zhang _et al_. considered a system where a UAV transmits
video to a terrestrial BS with SVC. The objective was to maximize the energy
efficiency maximization subject to the _delay outage probability_ constraint,
i.e., the probability that packet delay exceeds a predetermined delay budget.
Energy efficiency is defined as the ratio of the PSNR to the total power. The
optimal solution jointly determines the number of enhancement layers and
transmit power. In contrast with the baseline, which randomly selects the
number of layers and power, the proposed approach improved the energy
efficiency by 40% and decreased the delay outage probability from 0.3 to 0.05.
The work [58] studied a video streaming system in which the base and
enhancement layers of the SVC video are sent from a terrestrial BS and the UAV
BSs with the storage and computation capabilities to the ground users. Each
layer of the video can be served by either the terrestrial BS or a UAV BS,
i.e., the user obtains the layers of the video from various BSs. The
computation capabilities at the BSs can be used for video processing, e.g.,
encoding the video’s base layer and enhancement layers. In addition, the UAVs
without the storage and computation capabilities act as relays to help the
transmission from the terrestrial BS to the users. Since the number of
enhancement layers affects the video quality, the users desire more
enhancement layers. By optimizing over the transmit power and allocated
bandwidth of the BS and UAVs, the numbers of enhancement layers for the users,
the video layer assignment (i.e., from which BS), and the 2D deployment of the
UAVs, the objective in [58] was to maximize the sum of all users’ QoE metrics,
e.g., normalized PSNR, subject to the constraint on the transmission and
computation delays. The proposed approach achieved 15% better QoE, i.e.,
received video quality improvement, than a baseline, where the video layers
for the user originate from a single BS delivering the highest throughput. In
contrast with the other baselines in which the video layers for all users
originate from the terrestrial BS, and the video transmission is helped by the
UAV relays, the proposed approach could achieve 68% QoE further enhancement.
Nevertheless, due to its high computational complexity requirements and lack
of broad support by consumer devices, the SVC based approach is not preferable
for real-time video transmission.

###  IV-B UAV Deployment and Trajectory Design

In addition to wireless resource allocation, such as transmit power and
bandwidth, the maneuverability of UAVs offers an additional dimension for
enhancing video streaming performance, including throughput and latency. By
optimizing the UAV’s location or trajectory in three-dimensional (3D) space,
both energy consumption and wireless channel conditions can be improved.

Guo et al. [59] focused on the 3D trajectory design of a UAV deployed to
inspect multiple facilities and transmit real-time video to a control center.
The objective was to minimize the total energy consumption associated with
propulsion and hovering. The trajectory between successive facilities directly
impacted propulsion energy, while hovering energy depended on the inspection
time at each facility, determined by video bitrate and transmission latency.
Therefore, a trajectory planning algorithm was proposed in [59] to minimize
total energy consumption, assuming a fixed video bitrate. Simulation results
demonstrated that the proposed algorithm significantly reduced the UAV’s
energy consumption and flight time. Moreover, resource allocation in terms of
time slots, transmit power, and transmission rate was studied in [60], where a
UAV delivered videos to multiple ground users. The trajectory design took into
account the propulsion energy consumption. Building upon the work in [59], Bur
et al. [61] extended the research to collaborative inspection of a fire area
by multiple UAV-users, with the inspected videos sent to a UAV-BS. The
optimization involved the transmit power of all UAVs, 3D trajectories of UAV-
users, and dynamic bitrates of the users’ inspected videos. The focus was on
QoE maximization, which accounted for transmission delay violation and the
normalized transmission rate based on the selected video bitrate.
Additionally, the transmission rate needed to be sufficient to support the
selected video bitrate, considering the trajectories and transmit power of the
UAVs. The proposed approach enabled the support of 720p and 1080p videos with
an average delay of 0.05 ms, whereas a greedy approach relying on immediate
QoE decisions only supported 140p video with an average delay of 1.2 ms.
Overall, these studies highlight the importance of optimizing UAV trajectories
and resource allocation to enhance video streaming performance, achieving
energy efficiency, reduced delay, and improved QoE.

Furthermore, Khan et al. [62] investigated a UAV-to-UAV communication network
where UAVs collaboratively streamed video to a ground server. Their approach
involved utilizing dual paths for transmitting SVC video with one enhancement
layer. The base layer is sent directly from a UAV to the ground server via a
radio frequency link, while the enhancement layer is relayed to the server by
neighboring UAVs using free-space-optical links. The objective was to minimize
distortion in the received video by jointly optimizing the bitrates of the
base and enhancement layers, the routing path, and UAVs deployment. The
optimization was subject to a constraint on propulsion energy consumption and
the channel capacity’s bitrate limitations. The proposed approach achieved an
average PSNR gain of 6 dB compared to a baseline approach that used dual paths
with only radio frequency links, without optimizing the routing path and UAVs
deployment. In another study, Zhang et al. [60] formulated the user’s utility
as the normalized transmission rate relative to a predetermined bitrate
(considering fairness among users). They aimed to maximize the lowest time-
averaged utility among all users by jointly designing trajectories and
allocating wireless resources. The proposed approach outperformed three
baselines: trajectory optimization, wireless resource optimization, and no
optimization. It achieved up to a 3-fold increase in transmission rate. In
summary, Khan et al. explored UAV-to-UAV communication networks, demonstrating
the benefits of jointly optimizing routing paths, UAVs deployment, and bitrate
allocation for enhanced video streaming performance. Zhang et al. focused on
maximizing users’ utility through joint trajectory design and resource
allocation, achieving significant improvements in transmission rates compared
to various baselines.

###  IV-C Control Command and Video Data Coexistence

In the context of UAV teleportation, the operator at a remote location guides
the UAV to perform critical missions using the live video feed. This involves
simultaneous uplink streaming of real-time video and downlink delivery of
control commands. Achieving high-quality video streaming necessitates high
throughput and low latency, while delivering control commands requires ultra-
reliable and low-latency communication. Hence, in wireless systems designed
for UAV teleportation, ensuring reliable and low-latency control command
delivery, as well as optimal throughput and latency performance for video
streaming, becomes crucial. In the following sections, we examine recent
testbed setups that addressed the co-existence of control command and video
data, focusing on reliability and latency performance.

Stornig et al. [63] employed the ns-3 network simulator to study E2E delays
and video quality metrics (PSNR and SSIM) in video streaming over 4G networks.
They modeled the UAV’s 3D trajectory using a Gauss-Markov mobility model, and
the video traffic was simulated using the MPEG-4 format with the Evalvid
application. The impact of UAV mobility on latency performance was thoroughly
examined. Simulation results indicated that approximately two-thirds of frames
were received with good or excellent quality, while 27% of frames in regular
mobility and 30% of frames in high mobility exhibited inferior quality.
Moreover, the average PSNR and SSIM values for the received video were 33 dB
and 0.945, respectively, indicating good quality.

In the testbed presented in [64], a DJI Matrice 100 drone equipped with the
Quectel EC25 Long Term Evolution (LTE) module and a Raspberry Pi camera were
utilized. A computer with a USRP B210 radio frequency unit served as the BS,
connected to the remote controller via a wireline connection. The experiments
were conducted indoors using the AVC/H.264 video standard. Various metrics
were evaluated, including transmission delay, packet loss probability of
control commands, and video data throughput. The results demonstrated that
when the control command was updated less than 40 times per second, the
command delivery experienced a 20 ms transmission delay without any packet
loss. Furthermore, the average delay and throughput for 480p and 720p video
resolutions ranged from 1.5 s to 5.5 s and from 2 Mbps to 9 Mbps,
respectively. In [65], the authors evaluated the performance of a testbed
equipped with the Huawei MH5000 5G module, operating in an outdoor
environment. The transmission rates for streaming 1080p video in HEVC/H.265
format over 4G and 5G networks were measured at 16 Mbps and 97 Mbps,
respectively. The G2G delays were evaluated as 1.2 s and 3 s for the
respective networks. Additionally, the E2E delay of control command delivery
was measured to be 30 ms in the 5G network. In a different study, [66], an
immersive UAV control testbed was implemented using the Oculus Quest 2 HMD to
control UAV movement and FoV over 4G, 5G, and WiFi networks. The Insta360 One
X camera captured 360∘ video, and streaming rates of 2 Mbps to 8 Mbps were
considered, investigating various delays: G2G delay, glass-to-reaction-to-
execution delay, and sensor reaction delay. The G2G delay ranged from 0.595 s
to 0.985 s, the glass-to-reaction-to-execution delay ranged from 0.89 s to
1.38 s, and the sensor reaction delay ranged from 0.67 s to 1.12 s as the
streaming rate varied 2 Mbps to 8 Mbps. The control command transmission delay
was measured at 138 ms, 103 ms, and 88 ms for 4G, 5G, and WiFi networks,
respectively. Additionally, the PSNR quality of the received video for 720p
and 4K resolutions ranged from 30 to 47 dB.

###  IV-D Wireless Network Architecture

Based on the aforementioned quality factors, the design of wireless systems
for UAV-based video streaming can vary depending on the specific wireless
network architectures employed. Each network architecture comes with its own
restrictions, advantages, overheads, and hardware requirements, leading to
diverse performance outcomes. However, poor performance can significantly
hinder the feasibility of real-time UAV video streaming. Therefore, it is
crucial to incorporate practical and distinct features that evaluate video
streaming quality and UAV communication performance in the design of wireless
systems for UAV-based video streaming. The evaluation outcomes can also serve
as guidance for selecting the appropriate network, depending on the
application requirements of UAV-based video streaming.

In previous works, such as [67, 68], the network simulator ns-3 was utilized
to investigate the performance of UAV video streaming in 4G networks. The
Evalvid application was employed to simulate the video transmission from the
UAV to the BS using MP4 format videos. The study in [67] primarily focused on
throughput investigation in both outdoor and indoor environments. In the
outdoor scenario, the average throughput achieved by a static macrocell UAV
was found to be 60 kbps, while the throughput decreased to 20 kbps as the UAVs
moved at speeds ranging from 1 to 5 m/s. In the indoor environment, the
improvement in throughput was more significant for multi-story buildings with
an increased number of deployed femtocell BSs.

Naveed _et al_. [68] explored the relationship between the reference signal
received power (RSRP) and throughput. Their findings revealed that as the RSRP
varied from -110 dBm to -75 dBm, the UAV achieved video streaming throughputs
ranging between 2 kbps and 80 kbps. Additionally, the authors evaluated the
received video quality using PSNR and SSIM scores under various wireless
channel conditions. The PSNR scores were observed to be 49.41 dB, 35.42 dB,
and 24.31 dB in the best, good, and poor channel conditions, respectively.
Similarly, the SSIM scores were found to be 0.99, 0.63, and 0.35 in the
respective channel conditions. Furthermore, the impacts of different channel
conditions on video quality were visually highlighted through sample videos.

In another study by Sinha _et al_. [69], the network simulator ns-2.29 was
employed to evaluate the throughput, packet loss, packet retransmission, and
E2E delay performance of video streaming between UAVs and from the UAV to the
ground control station in different network configurations, including wireless
local area network (WLAN), WLAN router, WiFi hotspot, and WiFi Direct. Results
indicated that WiFi Direct achieved the best performance for all metrics,
followed by the WiFi hotspot, while the WLAN network exhibited the poorest
performance in all considered metrics.

The performance evaluation of multi-path video streaming in 4G networks was
conducted in the works by Liu & Jia [70] and Nihei et al. [71]. In the testbed
presented in [70], video data was transmitted from dual devices inside the UAV
to a smartphone. The dual-stream approach employed in this study demonstrated
the capability to reduce the E2E delay to approximately 50 ms, contrasting the
single-stream approach. In an independent study, Nihei et al. [71] tested the
multi-path video streaming method in 4G networks by distributing the video
data over two 4G mobile network operations in Indonesia. The objective of data
splitting was to minimize the average E2E delay. The experimental setup
involved the use of a DJI Spreading Wings S800 drone equipped with a Raspberry
Pi camera. Outdoor experiments were conducted using the AVC/H.264 format,
demonstrating the ability to adapt the video data to network throughput.
Visual illustrations provided in the study showcased the quality improvement
achieved with the multi-path method, highlighting its potential suitability
for forest fire surveillance. The performance of 60 GHz mmWave for video
transmission was evaluated by Yu et al. [72]. In their experiment conducted in
an outdoor environment, a 4K uncompressed video was transmitted from the UAV
to a nearby server to offload further computations. The testbed achieved a
throughput of 1.65 Gbps, and the results indicated that offloading
computations to the server enabled the UAV to save 271.8 watts in computations
at the expense of 4.1 watts for mmWave communication. In contrast to the
aforementioned studies, Hu et al. [73] conducted a numerical analysis of a
UAV-based ODV streaming system, where ground users requested specific video
tiles within their FoV from the UAVs. The UAV then transmitted the requested
tiles to the users via associated access points, which acted as decode-and-
forward relays. These APs collaboratively broadcasted the video data to the
corresponding users. The objective of their approach was to maximize the PSNR
by scheduling time slots to the UAVs and associating them with the APs. The
proposed approach yielded an enhancement in PSNR compared to baselines where
each AP worked independently or all APs worked together.

It is worth noting that while the majority of the studies discussed in this
section focused on non-real-time video streaming, they offer valuable insights
into the design of UAV-based real-time ODV streaming. For example, the work by
Yu et al. [72] emphasizes the importance of joint communications, computation,
and control design for UAV-based real-time video streaming. Similarly, the
results presented in [70, 71] demonstrate the effectiveness of multi-path
streaming in significantly reducing E2E delays. Furthermore, the
aforementioned studies shed light on the impact of various network settings on
video streaming performance.

###  IV-E Overview of Relevant 3GPP Standardization Activities

The standardization activities related to UAV-based immersive video streaming
within the 3GPP can be divided into two main categories. The first category
involves the integration of UAVs with cellular networks, while the second
category focuses on 5G support for media streaming applications, such as
augmented reality, virtual reality, and real-time communication. In the
following sections, we provide a comprehensive overview of the recent
advancements and state-of-the-art developments in these two areas.

####  IV-E1 Communication for UAVs

To evaluate the potential of LTE networks in supporting UAVs through cellular
connectivity, the 3GPP initiated the Release 15 study in March 2017 [74]. The
findings of this study are documented in TR 36.777 [75]. The study revealed
that the line-of-sight signal propagation in UAV communications increases the
likelihood of severe interference in both uplink and downlink scenarios.
Consequently, various interference detection and mitigation solutions were
proposed as study items and work items. Additionally, solutions related to
mobility information management and aerial user identification were put forth.
In Release 16, the focus shifted towards investigating the feasibility of
remotely identifying UAVs [76]. In Release 17, 3GPP further addressed the
operational 5G support of UAVs by providing functionalities for UAV
authentication, authorization, and tracking [77]. Moreover, it allows for
command and control authorization.

####  IV-E2 Support for Media Streaming over 5G

The support for virtual reality (VR) over wireless networks was initiated in
3GPP Release 15 with the publication of TR 26.918 [78]. This report aimed to
identify the potential gaps and use cases for facilitating virtual reality
(VR) services over wireless networks. In addition, Release 17 TS 26.118
introduced operation points, such as resolution and color mappings, and
defined media profiles for the distribution of VR content. To address the
challenges associated with real-time immersive media streaming, Release 18 of
3GPP is currently investigating several relevant issues. For a comprehensive
overview of the activities under Release 18, please refer to Table IV.

TABLE IV: Summary of 3GPP Release 18 activites for supporting media streaming on 5G networks 3GPP Document |  Focus
---|---
TS 26.501[79] |  5G Media Streaming (5GMS); General description and architecture
TS 26.506[80] |  5G real-time media communication architecture
TS 26.522[81] |  5G real-time media transport protocol configuration
TS 26.803[82] |  Study on 5G media streaming extensions for edge processing
TR 26.927[83] |  Artificial intelligence and machine learning in 5G media services

##  V Dataset Description

As mentioned earlier, the utilization of visual attention and saliency
information can provide valuable insights into human visual scene analysis
patterns. This knowledge can be harnessed to develop effective encoding and
streaming methods. Visual attention and saliency information can be derived by
analyzing viewers’ head movement (HM) and eye movement (EM) during video
playback. In this section, we present a comprehensive survey of existing EM
and HM datasets for ODV captured by UAVs. Additionally, we introduce a new
dataset that we have curated for this study.

TABLE V: Summary of Existing Datasets Dataset | Resolution | Frame rate | Dimension | Description
---|---|---|---|---
EyeTrackUAV2[84] |  $1280\times 720$ and $720\times 420$ |  $30$ fps |  $2$D | Eye tracking data
AVS1K[85] | $1280\times 720$ |  $30$ fps |  $2$D | Eye tracking data
WinesLab [86] | $1080\times 1920$ |  $30$ fps | 360∘ | Videos recorded using both handheld and UAV mounted camera
360 Track[87] | $3840\times 2160$ |  $30$ fps | 360∘ | Includes the ground truth for tracking
Proposed | $3840\times 2160$ |  $30-50$ fps | 360∘ | Table VI

In the existing literature, several works have introduced datasets focused on
ODV, encompassing EM and HM information of viewers [88]. For better
understanding the user behavior while watching ODVs, these datasets categorize
the ODVs, based on the number of moving objects and camera motion, and include
users’ feedback about viewing experience [89]. On the other hand, [90]
classified the videos based on their genre, such as documentary, movie, etc.
The majority of these datasets consists of videos with 3DoF which makes them
less suitable for learning the user viewing pattern for a UAV based ODV
streaming protocol. Indeed, inferences obtained using ODV with 3DoF may not be
applicable for video transmission platforms with 6DoF, such as UAV based ODV
transmission. This raises the need to develop novel datasets of ODVs captured
using UAVs. In the following, we briefly survey the existing datasets based on
the videos captured from UAVs.

While many datasets in the literature include images and 2D videos captured by
UAVs for applications such as remote sensing and navigation, only a limited
number of publicly available datasets capture EM and HM information for UAV-
recorded videos, with only one dataset currently accessible [86]. Similarly,
there is only one dataset available for UAV-based 360∘ videos. We summarize
these datasets in Table V. The EyeTrackUAV2 dataset [84] collects binocular
gaze information from 30 viewers watching 43 2D videos under both free viewing
and task conditions. The AVS1K dataset comprises ground truth salient object
regions for 1000 videos observed by 24 viewers in free viewing conditions. The
WinesLab dataset contains 11 360∘ videos, seven of which were recorded by a
pedestrian using a handheld camera, while the remaining four were captured
using a drone-mounted camera in various surroundings and lighting conditions.
The 360Track dataset consists of 9 360∘ videos with manually marked ground
truth positions of salient objects. Additionally, we describe in the following
our dataset of _aerial_ 360∘ videos presented in Table VI.

The dataset presented in Table VI comprises a total of ten 360-degree videos.
The resolution for all videos, except for “FreeStyleParaGliding,” is 3840
$\times$ 1920, while “FreeStyleParaGliding” has a resolution of $5120\times
2560$. Each video sequence in the dataset has a length of 40 seconds. The
majority of videos, except for “DubaiVertical” and “AbuDhabiCity,” have a
frame rate of 30 frames per second (fps), whereas “DubaiVertical” and
“AbuDhabiCity” consist of 50 fps. The dataset consists of five outdoor videos,
one sports video, and one video recorded in night time conditions. The
“NorthPoleTrip” video captures motion in the azimuth plane, while the
“DubaiVertical” video captures motion in the elevation. Lastly, all video
sequences are encoded and transmitted using the ERP representation.

TABLE VI: Summary of Our Dataset

Sequence Name | Spatial Resolution | #Frames | Frame rate (fps) | Scene Feature
---|---|---|---|---
PetraJordan | 3840 $\times$ 1920 | 1200 | 30 | Outdoor
CapeTownCityPenorama | 3840 $\times$ 1920 | 1200 | 30 | Outdoor
CapeTownCityBeach | 3840 $\times$ 1920 | 1200 | 30 | Outdoor
CapeTownCityGarden | 3840 $\times$ 1920 | 1200 | 30 | Outdoor
CapeTownCitySquare | 3840 $\times$ 1920 | 1200 | 30 | Outdoor
FreeStyleParaGliding | 5120 $\times$ 1920 | 1200 | 30 | Sports
StPetersBergMuseum | 3840 $\times$ 1920 | 1200 | 30 | Night
NorthPoleTrip | 3840 $\times$ 1920 | 1200 | 30 | Motion
DubaiVertical | 3840 $\times$ 1920 | 2000 | 50 | Vertical Motion
AbuDhabiCity | 3840 $\times$ 1920 | 2000 | 50 | City Panorama

##  VI Benchmark and Analysis

In this section, we first perform a comprehensive performance benchmarking of
five video coding standards and formats (i.e., AVC/H.264, HEVC/H.265,
VVC/H.266, AV1, and VP9) through their software implementations: libx264,
libx265, fraunhofer versatile video encoder (VVenC), libvpx-vp9, and
libsvtav1, respectively. We also considered two NVIDIA hardware encoder
designs hevc_nvenc and avc_nvenc, for the AVC/H.264 and HEVC/H.265 standards,
respectively. All encoders are configured in their fastest preset, targeting
live 360∘ video streaming applications. Table VIII gives the used hardware and
software encoder libraries for the five standards and formats. The encoding
process was deployed on a DELL precision 7820 tower workstation. This later is
equipped with an Intel Xeon CPU with 8 cores running at a maximum frequency of
3.9 Ghz and a NVIDIA RTX A5000 GPU. Furthermore, we present a real test-bed
for real-time drone ODV streaming using a hardware AVC/H.264 encoder and
WebRTC streaming protocol, enabling remote UAV control and navigation with
6DoF viewing experience.

TABLE VII: Parameters of used encoding workstation CPU | Intel Xeon Silver
---|---
#cores | 8
Max Freq (GHz) | 3.9
RAM | 32 GB
SSD | 256GB
GPU | NVIDIA Ampere RTX A5000
Operating System | Ubuntu 20.04

###  VI-A Coding and complexity performance

This section evaluates the coding performance and speedup of the considered
software and hardware encoders on video contents captured by a UAV. The
quality of decoded 360∘ videos is assessed using three objective quality
metrics: Spherical PSNR (S-PSNR), SSIM, and VMAF. The videos are encoded at
four practical UAV target bitrates of 1.5 Mbps, 3 Mbps, 4.5 Mbps, and 5.8
Mbps, enabling the computation of the BD-rate performance. The BD-rate gives
the average bitrate saving or loss compared to the anchor encoder over the
four considered bitrates.

Figures 3(a), 3(b), and 3(c) provide the average quality performance of the
studied software and hardware video encoders on the proposed dataset,
utilizing three distinct quality metrics: S-PSNR, SSIM, and VMAF,
respectively. From the results, it is evident that the AV1 software encoder
achieves the highest quality in terms of S-PSNR and VMAF across all four
bitrates. Following closely is the VVenC software encoder, which demonstrates
competitive performance with AV1, particularly at high bitrates, based on the
SSIM metric. On the other hand, the libx264 software encoder reports the
lowest quality among the tested encoders. It is worth noting that the hardware
design for the AVC/H.264 standard outperforms the libx264 software encoder
significantly across all quality metrics and bitrates. Interestingly, the
software implementation of the HEVC/H.265 standard exhibits slightly higher
quality than its hardware implementation. This can be attributed to the
increased complexity introduced by the new tools in the HEVC/H.265 standard,
making the design of a hardware encoder for HEVC more challenging compared to
the AVC/H.264 standard.

The associated BD-rate results with respect to the AVC/H.264 software encoder
for S-PSNR, SSIM, and VMAF are depicted respectively in Figures 4(a), 4(b) and
4(c), plotted versus the encoding time. These figures reveal that the hardware
encoders (h264_nvenc and h265_nvenc) and the AV1 software encoder offer the
best tradeoff between coding efficiency and encoding time. Notably, only the
hardware encoders can achieve real-time encoding at 30 frames per second. To
achieve real-time encoding, the AV1, AVC, and VP9 software encoders would
require a powerful processor with multiple cores operating at a higher
frequency. In contrast, the VVC/H.266 software encoder (VVenC) exhibits
significantly longer encoding times, taking more than one hour to encode a
10-second video. The new coding tools introduced in the VVC/H.266 standard
have expanded the search space for rate-distortion optimizations, leading to
increased encoding complexity. To enable real-time capability, advanced
algorithmic optimizations, along with more efficient low-level optimizations,
are necessary. Furthermore, the development of efficient hardware designs for
the VVC/H.266 standard becomes crucial, particularly for low-energy embedded
devices, to achieve real-time encoding and benefit from its high coding
efficiency and advanced features for ODV contents.

(a) S-PSNR

(b) SSIM

(c) VMAF

Figure 3: The average quality in S-PSNR (dB), SSIM and VMAF at different bit-
rate for the seven considered encoders.

(a) BD-PSNR

(b) BD-SSIM

(c) BD-VMAF

Figure 4: The BD-rate performance in S-PSNR (dB), SSIM and VMAF versus
encoding time for the seven considered encoders on 10 seconds video contents.
TABLE VIII: Video encoder SW/HW libraries

Video codec standard | Software | Version | Hardware
---|---|---|---
AVC/H.264 | libx264 [91] | 0.164.3106 | h264_nvenc [92]
HEVC/H.265 | libx265 [93] | 3.5+1-f0c1022b6 | hevc_nvenc [94]
VVC/H.266 |  VVenC [95] | 1.7.0 | -
AV1 | libsvtav1 [96] | 1.4.1 | -
VP9 | libvpx-vp9 [97] | 1.11.0-30-g888bafc78 | -

###  VI-B Testbed for UAV 360∘ Video Streaming

The proposed testbed comprises essential components, namely a UAV equipped
with a 360-degree camera, a 5G modem, and an edge server. The 360-degree
camera captures a comprehensive view of the surroundings, providing an
immersive 3DoF viewing experience that ensures no critical details are
overlooked during acquisition. The 5G modem enables real-time transmission of
high-resolution footage from the UAV to the edge server. Users can connect to
the edge server through a HMD at the command center, facilitating prompt
decision-making by providing immediate access to live 4K 360-degree video
footage.

Figure 5(a) depicts the field tests conducted with a First Person View (FPV)
UAV operator controlling the UAV in a desert environment. The operator sends
commands to the UAV through a central server located 100 km away from both the
UAV and the operator. Both the UAV and the operator are connected to a
consumer 5G network, as shown in Figure 5(b), with specific settings outlined
in Table IX. During the experiment, the operator flew the drone at a fixed
position while varying the altitude. Simultaneously, the onboard computer of
the UAV recorded information received from the 5G modem, including the Cell
ID, throughput, and network latency from the UAV to the central server.

Figures 6(a) and 6(b) provide insights into the flying conditions for
handovers and the instantaneous throughput as a function of altitude in the
scenario of vertical landing of the drone. In Figure 6(a), it can be observed
that the drone experienced a total of ten handovers, utilizing the four
available base stations that cover the flying area. Figure 6(b) shows that
most of the handovers resulted in improved instantaneous throughput. However,
the throughput exhibited significant fluctuations due to wireless
communication instability and interference.

At higher altitudes, the drone encounters interference from base stations
primarily designed for ground-based users. This interference introduces
latency and quality degradation in the transmission of video and control
signals, thereby posing challenges for effective drone navigation by the
operator. Our real field tests showcase the control of UAV through 5G using a
VR headset and 360-degree video feedback at altitudes of up to 600 meters.
These tests shed light on the potential challenges associated with interfering
base stations and suboptimal handover conditions in VR-based UAV control.

TABLE IX: Configuration of the 360∘ video streaming over UAV testbed. Parameter |  Value
---|---
5G Max(upload/download) |  50 Mbps/100 Mbps
Server CPU |  8 cors @ 2.5 GHz
Server memory |  16 Gb
Distance UAV to Server |  500m
Distance VR HMD/UAV to Server |  100km
UAV flight speed during tests |  25Km/h
UAV’s onboard computer |  Jetson nano
UAV’s weight |  2.5 Kg
360-degree camera |  Ricoh Theta Z1

##  VII Open Challenges

###  VII-A Adaptive Low-latency 360∘ Video Streaming

From Figure 6(b), it is evident that UAV communication, particularly at high
altitudes and during mobility, is susceptible to significant throughput
variation. This inherent issue raises fundamental concerns regarding the
attainment of high QoE with superior video quality and minimal G2G latency in
real-time 360-degree video streaming. To address these challenges, several
open research directions can be pursued. Firstly, leveraging the latest video
coding standards and efficient hardware encoders, such as hevc_nvenc, can
substantially enhance perceived video quality. The hevc_nvenc encoder enables
real-time encoding with low energy consumption, harnessing the coding
efficiency promised by the advanced video coding standard, HEVC/H.265. This,
in turn, extends the UAV’s battery life. At the cloud level, more efficient
software encoders like SVT-AV1 can be utilized for video transcoding,
leveraging the available cloud resources. Furthermore, advanced optimization
techniques, such as FoV prediction, can be employed to allocate higher quality
to the viewing viewport, resulting in improved bandwidth utilization and
perceived video quality for end-users.

Secondly, ensuring fast and accurate adaptation of the video bitrate to
channel throughput variations is crucial to prevent buffering at both the
transmitter and receiver sides, thereby minimizing G2G latency. In this
regard, leveraging information from the physical layer as well as the UAV
status, position and its environment can be valuable for predicting throughput
variations and facilitating proactive encoder adaptation. Additionally,
jointly considering other source video parameters, such as spatial resolution,
temporal frame rate, and projection, in the rate control mechanism can further
minimize G2G latency and maximize perceived quality. Advanced machine learning
techniques, including deep reinforcement learning, have demonstrated potential
in learning real-time prediction of pre-processing and encoder parameters to
achieve the target bitrate while maximizing perceived quality [98, 99].

Finally, these research directions pave the way for addressing open challenges
in UAV-based 360-degree video streaming, leading to improved QoE, minimized
latency, and enhanced video quality.

###  VII-B Cooperative Aerial Video Streaming

Cooperative immersive video streaming, exemplified by Intel’s Trueview [100],
has the potential to enable a truly immersive viewing experience [101]. This
approach allows users to independently select their preferred viewing angle by
streaming from multiple cameras or sources, leveraging spatial diversity in
terms of viewing angle, content, or geographic location. Moreover, employing
multiple UAVs to capture aerial views can enhance the immersive experience
with 6DoF capabilities [102, 103]. However, developing a multi-UAV cooperative
immersive video streaming system entails addressing a unique set of challenges
in joint communication, computation, and control design. Streaming a scene
captured by multiple UAVs requires effective coordination among the UAVs to
ensure comprehensive scene coverage without compromising QoE while minimizing
network bandwidth usage. Additionally, capturing more dynamic events, such as
sports or moving ground targets [101, 103], necessitates accurate motion
prediction, such as player or target movement, which, in turn, relies on
coordinated trajectory planning and 3D placement of all UAVs. Furthermore, the
trajectory and placement of UAVs must also consider their battery levels, in
addition to QoE considerations.

(a) Field test

(b) UAV

Figure 5: Illustration of the field test setting and the UAV configuration.

In multi-UAV applications, the individual UAV can collaboratively and
cohesively capture videos, which are then synthesized into a panoramic video.
However, streaming videos from all UAVs simultaneously poses a significant
resource burden. To address this challenge, bandwidth-saving streaming
techniques can be employed by leveraging user attention information [104].
Specifically, UAVs whose videos are deemed unnoticed by users can remain idle
during transmission. However, we argue that instead of staying idle, these
UAVs can contribute to real-time video streaming, thus enhancing communication
efficiency and throughput further. For instance, the UAV swarm can
collectively form a virtual multiple-input and multiple-output (MIMO) system
[105]. Nonetheless, this type of MIMO system exhibits distinct wireless
channel characteristics. Considering the unique channel model and the
requirements for throughput and latency, thus designing cooperative aerial
video streaming for real-time and interactive panoramic videos poses
considerable challenges.

Addressing these challenges in cooperative aerial video streaming requires
innovative solutions that account for coordination, resource optimization,
wireless channel characteristics, throughput, and latency requirements.
Meeting these objectives will contribute to the development of robust and
efficient systems for real-time and interactive panoramic video streaming.

(a) Handover vs. altitude.

(b) Throughput vs. handover.

Figure 6: Handovers and instantaneous throughput performance versus the drone
altitude in vertical landing flying conditions. The average throughput values
of the cells in green, orange, blue, and red are 14.55 Mbps, 17.19 Mbps, 11.21
Mbps, and 10.79 Mbps, respectively.

###  VII-C QoE-Aware Control and Communication of UAVs

Ensuring high-quality user experience in 360∘ video streaming systems is
primarily affected by stall time resulting from low transmission rates and
overall video quality perceived by users [106]. Experiencing a stall time
longer than the tolerance level can lead to VR sickness, making it crucial to
achieve high data rates and low-latency transmission for enhancing QoE [107].
Addressing these challenges requires leveraging video saliency to predict
users’ FoV and employing multicast transmission techniques based on users’
locations and FoV correlations, as grouping and multicasting can improve
network throughput and QoE [108, 109]. Additionally, adapting the encoding
bitrate of tiles based on channel quality, available resources, content
quality, and inaccurate FoV prediction can further enhance QoE [107].

In the context of VR streaming from aerial users, such as drones, additional
challenges arise due to their dynamic topology and limited energy resources
[110, 111]. The channel quality and network throughput of aerial users are
also influenced by their flight trajectory, necessitating the joint design of
QoE-aware resource allocation and drone route selection mechanisms [112, 113].
Moreover, the limited onboard energy availability of drones requires judicious
resource allocation strategies [114]. Furthermore, in a multi-UAV-based
streaming system for 360∘ videos [56], additional challenges arise in terms of
resource allocation among the UAVs. Each UAV can independently adjust its
encoding bitrate and position [115], while simultaneously competing for
resources with other UAVs in the swarm. Given these significant challenges,
designing a UAV-based 360∘ streaming system necessitates a thorough study of
joint communication and control design for these systems.

In critical missions involving the teleoperation of UAVs, such as fire
disaster monitoring [71] and suspicious vehicle tracking [116], the quality of
service relies on the interplay between control command delivery and video
data transmission. The latency experienced in one link can impact the latency
budget in the other link. Moreover, unreliable control command communication
can influence the UAV’s reaction and view angle, resulting in undesired
information for the remote operator. Therefore, the entanglement and mutual
influence between control command delivery and real-time UAVs video
transmission require dedicated consideration in the design process.

Figure 7: Use case scenario for LLMs control commands for UAV with 360∘
camera.

###  VII-D Tailored Design of UAV Communication for Video Streaming

The design and optimization of UAV wireless communication systems for real-
time video streaming pose several open challenges. While theoretical solutions
and algorithms have been proposed, many of them focus on incorporating
specific characteristics of video streaming and air-to-ground channels, as
well as addressing communication requirements. However, in practical testbed
implementations, the existing protocols are often used without tailored
optimization for video streaming, resulting in limited performance evaluation
in real-world scenarios. To enable more immersive services in the context of
6G networks, it is essential to dedicate effort towards implementing wireless
protocols specifically tailored to the demands of video streaming. Currently,
the wireless system primarily considers factors such as video capture rate or
bitrate, neglecting other important aspects.

One approach to achieve video streaming-tailored wireless systems is to
incorporate additional characteristics of video encoding in the design. For
example, considering the correlation among frames in video content can enhance
video encoding efficiency. Additionally, adopting a semantic communication
approach, as explored in [117], where the focus is on effectively conveying
the semantic meaning of information rather than solely delivering digital
bits, shows promise in realizing UAV communication systems tailored for video
streaming. Addressing these challenges and exploring novel techniques that
take into account video content correlation, semantic communication, and other
relevant factors will be crucial in developing efficient and optimized
wireless systems for UAV video streaming applications.

###  VII-E LLM for Immersive Video Streaming

The rapid advancement in natural language processing (NLP) has paved the way
for the development of LLMs like BERT [118], GPT-3/GPT-4, and FALCON. These
versatile models push the state-of-the-art on many down-stream tasks, finding
applications in various domains, including conversation, medicine,
telecommunications [119], and robotics [120].

In the context of streaming 360-degree video from one or multiple UAVs,
exploring these LLMs can greatly enhance performance. For example, end users
can provide task prompts to the LLM along with descriptions of the environment
captured by the 360-degree camera. The LLM can then generate commands for the
UAVs to carry out the tasks successfully while minimizing their energy
consumption and avoiding obstacles. In particular, the description of the
surrounding environment from the 360-degree camera can be performed by the end
used or automatically by exploiting vision-language models, such as SimVLM
[121], Flamingo [122], or BLIP-2 [123].

##  VIII Conclusion

In this paper, we have investigated omnidirectional video streaming over UAVs,
focusing on the benefits and challenges associated with live 360-degree video
streaming. By enabling immersive viewing with up to 6DoF, this technology
enhances the QoE for various applications such as surveillance, autonomous
driving, healthcare, and education. However, 360-degree video streaming poses
challenges in terms of high bandwidth and computing requirements, while the
UAV wireless channel exhibits interference and instability, leading to
significant bandwidth variations. To overcome these challenges, we first
reviewed the key components of 360-degree video streaming over wireless
channels and highlighted the technology used to achieve low-latency end-to-end
streaming. Additionally, we introduced a new dataset consisting of ten
360-degree videos captured by UAV in various flying conditions, enabling us to
evaluate the coding efficiency and complexity of different Software (SW) and
Hardware (HW) video encoders. Through our experiments, we found that only HW
AVC/H.264 and HEVC/H.265 encoders achieved real-time encoding, making them
suitable for UAV with limited computing and energy resources. Furthermore, the
SW AV1 encoder demonstrated the best tradeoff between coding performance and
complexity and thus be utilized for efficient video transcoding on more
powerful devices in the cloud. Moreover, we presented a real testbed of
360-degree video streaming over a drone with 5G communication, illustrating
the significant fluctuations in the wireless channel due to interference and
multiple handovers. Finally, we discussed open challenges and proposed future
research directions to enhance the key performance metrics of live immersive
video streaming over UAVs.

Overall, this paper provides valuable insights into the field of
omnidirectional video streaming over UAVs and contributes to the understanding
of how to improve the QoE in this context. The findings and recommendations
presented here pave the way for further advancements and innovations in the
area of live immersive video streaming over UAVs, ultimately benefiting a wide
range of applications and industries.

## References

  * [1] J. van der Hooft, H. Amirpour, M. T. Vega, Y. Sanchez, R. Schatz, T. Schierl, and C. Timmerer, “A tutorial on immersive video delivery: From omnidirectional video to holography,” _IEEE Communications Surveys & Tutorials_, vol. 25, no. 2, pp. 1336–1375, 2023.
  * [2] M. Mozaffari, W. Saad, M. Bennis, Y.-H. Nam, and M. Debbah, “A tutorial on UAVs for wireless networks: Applications, challenges, and open problems,” _IEEE Commun. Surveys Tuts._ , vol. 21, no. 3, pp. 2334–2360, 3rd Quart. 2019\.
  * [3] Q. Wu, J. Xu, Y. Zeng, D. W. K. Ng, N. Al-Dhahir, R. Schober, and A. L. Swindlehurst, “A comprehensive overview on 5G-and-beyond networks with UAVs: From communications to sensing and intelligence,” _IEEE J. Sel. Areas Commun._ , vol. 39, no. 10, pp. 2912–2945, Dec. 2021.
  * [4] S. Hayat, E. Yanmaz, and R. Muzaffar, “Survey on unmanned aerial vehicle networks for civil applications: A communications viewpoint,” _IEEE Commun. Surveys Tuts._ , vol. 18, no. 4, pp. 2624–2661, 4th Quart. 2016.
  * [5] A. Baltaci, E. Dinc, M. Ozger, A. Alabbasi, C. Cavdar, and D. Schupke, “A survey of wireless networks for future aerial communications (FACOM),” _IEEE Commun. Surveys Tuts._ , vol. 23, no. 4, pp. 2833–2884, 4th Quart. 2021\.
  * [6] D. Mishra, A. M. Vegni, V. Loscrí, and E. Natalizio, “Drone networking in the 6G era: A technology overview,” _IEEE Commun. Standards Mag._ , vol. 5, no. 4, pp. 88–95, Dec. 2021.
  * [7] C. Yan, L. Fu, J. Zhang, and J. Wang, “A comprehensive survey on UAV communication channel modeling,” _IEEE Access_ , vol. 7, pp. 107 769–107 792, 2019.
  * [8] Z. Xiao, L. Zhu, Y. Liu, P. Yi, R. Zhang, X.-G. Xia, and R. Schober, “A survey on millimeter-wave beamforming enabled UAV communications and networking,” _IEEE Commun. Surveys Tuts._ , vol. 24, no. 1, pp. 557–610, 1st Quart. 2022\.
  * [9] Y. Zeng, Q. Wu, and R. Zhang, “Accessing from the sky: A tutorial on UAV communications for 5G and beyond,” _Proc. IEEE_ , vol. 107, no. 12, pp. 2327–2375, Dec. 2019.
  * [10] T. Elmokadem and A. V. Savkin, “Towards fully autonomous UAVs: A survey,” _Sensors_ , vol. 21, no. 18, pp. 1–39, Sep. 2021.
  * [11] A. Fotouhi, H. Qiang, M. Ding, M. Hassan, L. G. Giordano, A. Garcia-Rodriguez, and J. Yuan, “Survey on UAV cellular communications: Practical aspects, standardization advancements, regulation, and security challenges,” _IEEE Commun. Surveys Tuts._ , vol. 21, no. 4, pp. 3417–3442, 4th Quart. 2019\.
  * [12] V. Marojevic, I. Guvenc, R. Dutta, M. L. Sichitiu, and B. A. Floyd, “Advanced wireless for unmanned aerial systems: 5G standardization, research challenges, and AERPAW architecture,” _IEEE Veh. Technol. Mag._ , vol. 15, no. 2, pp. 22–30, Jun. 2020.
  * [13] A. S. Abdalla and V. Marojevic, “Communications standards for unmanned aircraft systems: The 3GPP perspective and research drivers,” _IEEE Commun. Standards Mag._ , vol. 5, no. 1, pp. 70–77, Mar. 2021.
  * [14] H. Wang, H. Zhao, J. Zhang, D. Ma, J. Li, and J. Wei, “Survey on unmanned aerial vehicle networks: A cyber physical system perspective,” _IEEE Commun. Surveys Tuts._ , vol. 22, no. 2, pp. 1027–1070, 2nd Quart. 2020.
  * [15] A. Yaqoob, T. Bi, and G.-M. Muntean, “A survey on adaptive 360∘ video streaming: Solutions, challenges and opportunities,” _IEEE Commun. Surveys Tuts.s_ , vol. 22, no. 4, pp. 2801–2838, 4th Quart. 2020.
  * [16] D. He, C. Westphal, and J. J. Garcia-Luna-Aceves, “Network support for AR/VR and immersive video application: A survey,” in _Proc. 15th Int. Joint Conf. e-Business Telecommun._ , Jul. 2018, pp. 359–369.
  * [17] T.-V. Nguyen, N. P. Nguyen, C. Kim, and N.-N. Dao, “Intelligent aerial video streaming: Achievements and challenges,” _J. Netw. Comput. Appl._ , vol. 211, p. 103564, Feb. 2023.
  * [18] M. Zink, R. Sitaraman, and K. Nahrstedt, “Scalable 360° video stream delivery: Challenges, solutions, and opportunities,” _Proc. IEEE_ , vol. 107, no. 4, pp. 639–650, Apr. 2019.
  * [19] I.-C. Lo, K.-T. Shih, and H. H. Chen, “Efficient and accurate stitching for 360° dual-fisheye images and videos,” _IEEE Trans. Image Process._ , vol. 31, pp. 251–262, 2022.
  * [20] Y. Ye, J. M. Boyce, and P. Hanhart, “Omnidirectional 360° video coding technology in responses to the joint call for proposals on video compression with capability beyond HEVC,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 30, no. 5, pp. 1241–1252, 2020.
  * [21] E. Kuzyakov and D. Pio. (2016, Jan.) Next-generation video encoding techniques for 360 video and VR. Accessed on 1-July-2023. [Online]. Available: https://engineering.fb.com/2016/01/21/virtual-reality/next-generation-video-encoding-techniques-for-360-video-and-vr/
  * [22] T. Wiegand, G. Sullivan, G. Bjontegaard, and A. Luthra, “Overview of the H.264/AVC video coding standard,” _IEEE Trans. Circuits Syst. Video Technol._ , vol. 13, no. 7, pp. 560–576, Jul. 2003.
  * [23] G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, “Overview of the high efficiency video coding (HEVC) standard,” _IEEE Trans. Circuits Syst. Video Technol._ , vol. 22, no. 12, pp. 1649–1668, Dec. 2012.
  * [24] B. Bross, Y.-K. Wang, Y. Ye, S. Liu, J. Chen, G. J. Sullivan, and J.-R. Ohm, “Overview of the versatile video coding (VVC) standard and its applications,” _IEEE Trans. Circuits Syst. Video Technol._ , vol. 31, no. 10, pp. 3736–3764, Dec. 2021.
  * [25] K. Misra, A. Segall, M. Horowitz, S. Xu, A. Fuldseth, and M. Zhou, “An overview of tiles in HEVC,” _IEEE J. Sel. Topics Signal Process._ , vol. 7, no. 6, pp. 969–977, Dec. 2013.
  * [26] J. Son, D. Jang, and E.-S. Ryu, “Implementing motion-constrained tile and viewport extraction for VR streaming,” in _Proc. 28th ACM SIGMM Workshop Netw. Oper. Syst. Support Digit. Audio Video_ , Jun. 2018, pp. 61–66.
  * [27] N. Mahmoudian Bidgoli, T. Maugey, and A. Roumy, “Fine granularity access in interactive compression of 360-degree images based on rate-adaptive channel codes,” _IEEE Trans. Multimedia_ , vol. 23, pp. 2868–2882, 2021.
  * [28] Y. Zhou, L. Tian, C. Zhu, X. Jin, and Y. Sun, “Video coding optimization for virtual reality 360-degree source,” _IEEE J. Sel. Topics Signal Process._ , vol. 14, no. 1, pp. 118–129, Jan. 2020.
  * [29] Y.-C. Su and K. Grauman, “Learning compressible 360∘ video isomers,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 43, no. 8, pp. 2697–2709, Aug. 2021.
  * [30] D. Minnen, J. Ballé, and G. D. Toderici, “Joint autoregressive and hierarchical priors for learned image compression,” _Adv. Neural Inf. Process. Syst._ , vol. 31, 2018.
  * [31] J. Cao, X. Su, B. Finley, A. Pauanne, M. Ammar, and P. Hui, “Evaluating multimedia protocols on 5G edge for mobile augmented reality,” in _Proc. 17th Int. Conf. Mobility, Sens. Netw._ , Dec. 2021, pp. 199–206.
  * [32] M. M. Hannuksela and Y.-K. Wang, “An overview of omnidirectional media format (OMAF),” _Proc. IEEE_ , vol. 109, no. 9, pp. 1590–1606, Sep. 2021.
  * [33] B. Sredojev, D. Samardzija, and D. Posarac, “WebRTC technology overview and signaling solution design and implementation,” in _Proc. 38th Int. Convention Inf. Commun. Technol., Electron. Microelectron._ , May 2015, pp. 1006–1009.
  * [34] K. K. Sreedhar, I. D. D. Curcio, A. Hourunranta, and M. Lepistö, “Immersive media experience with MPEG OMAF multi-viewpoints and overlays,” in _Proc. 11th ACM Multimedia Syst. Conf._ , May 2020, pp. 333–336.
  * [35] TileMedia. (2022, Jan.) How ClearVR Drives and Leverages Standards. Accessed on 1-July-2023. [Online]. Available: https://www.tiledmedia.com/how-clearvr-drives-and-leverages-standards/
  * [36] N. Sehad, B. Cherif, I. Khadraoui, W. Hamidouche, F. Bader, R. Jäntti, and M. Debbah, “Locomotion-based UAV control toward the internet of senses,” _IEEE Transactions on Circuits and Systems II: Express Briefs_ , vol. 70, no. 5, pp. 1804–1808, 2023.
  * [37] S. Wang, X. Zhang, M. Xiao, K. Chiu, and Y. Liu, “SphericRTC: A system for content-adaptive real-time 360-degree video communication,” in _Proceedings of the 28th ACM International Conference on Multimedia_ , ser. MM ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 3595–3603.
  * [38] Intel. (2022, Dec.) Immersive video sample reference implementation. Accessed on 1-July-2023. [Online]. Available: https://www.intel.com/content/www/us/en/developer/articles/technical/immersive-video-sample-powered-by-ovc.html
  * [39] Q. Zhang, J. Wei, S. Wang, S. Ma, and W. Gao, “RealVR: Efficient, economical, and quality-of-experience-driven VR video system based on MPEG OMAF,” _IEEE Transactions on Multimedia_ , pp. 1–15, 2022.
  * [40] C. Bonnineau, W. Hamidouche, J. Fournier, N. Sidaty, J.-F. Travers, and O. Déforges, “Perceptual quality assessment of HEVC and VVC standards for 8K video,” _IEEE Trans. Broadcast._ , vol. 68, no. 1, pp. 246–253, Mar. 2022.
  * [41] C. Zhou, Z. Li, and Y. Liu, “A measurement study of oculus 360 degree video streaming,” in _Proceedings of the 8th ACM on Multimedia Systems Conference_ , ser. MMSys’17. New York, NY, USA: Association for Computing Machinery, 2017, p. 27–37.
  * [42] V. I. Forum. (2021, Jan.) VR Industry Forum Guidelines V2.3. Accessed on 1-July-2023. [Online]. Available: https://www.vr-if.org/wp-content/uploads/vrif2020.180.00-Guidelines-2.3_clean..pdf
  * [43] F. Qian, L. Ji, B. Han, and V. Gopalakrishnan, “Optimizing 360 video delivery over cellular networks,” in _Proc. 5th Workshop All Things Cellular: Oper., Appl. Challenges_ , Oct. 2016, pp. 1–6.
  * [44] TileMedia. (2022, Jan.) Tiled Media High Quality VR streaming. Accessed on 1-July-2023. [Online]. Available: https://www.tiledmedia.com/high-quality-vr-streaming/
  * [45] W. Cheung, Gang Shen and Dusty Robbins, “Solution implementation summary: Media 360-degree video distribution,” accessed: 19-january-2023. [Online]. Available: https://www.intel.it/content/dam/www/central-libraries/us/en/documents/advanced-360video-implementation-summary-final.pdf
  * [46] M. Warburton, M. Mon-Williams, F. Mushtaq, and J. R. Morehead, “Measuring motion-to-photon latency for sensorimotor experiments with virtual reality systems,” _Springer - Behavior Research Methods_ , vol. 10, Oct. 2022.
  * [47] H. T. T. Tran, N. P. Ngoc, C. M. Bui, M. H. Pham, and T. C. Thang, “An evaluation of quality metrics for 360 videos,” in _2017 Ninth International Conference on Ubiquitous and Future Networks (ICUFN)_ , 2017, pp. 7–11.
  * [48] N. Gao, Y. Zeng, J. Wang, D. Wu, C. Zhang, Q. Song, J. Qian, and S. Jin, “Energy model for UAV communications: Experimental validation and model generalization,” _China Commun._ , vol. 18, no. 7, pp. 253–264, Jul. 2021\.
  * [49] H. Yan, S.-H. Yang, Y. Ding, and Y. Chen, “Energy consumption models for UAV communications: A brief survey,” in _Proc. IEEE Int. Conf. Internet Things and IEEE Green Comput. Commun. and IEEE Cyber, Physical Social Comput. and IEEE Smart Data and IEEE Congr. Cybermatics_ , Aug. 2022, pp. 161–167.
  * [50] C. You and R. Zhang, “3D trajectory optimization in Rician fading for UAV-enabled data harvesting,” _IEEE Transactions on Wireless Communications_ , vol. 18, no. 6, pp. 3192–3207, 2019.
  * [51] ——, “Hybrid offline-online design for UAV-enabled data harvesting in probabilistic LoS channels,” _IEEE Trans. Wireless Commun._ , vol. 19, no. 6, pp. 3753–3768, Jun. 2020.
  * [52] A. Al-Hourani, S. Kandeepan, and S. Lardner, “Optimal LAP altitude for maximum coverage,” _IEEE Wireless Commun. Lett._ , vol. 3, no. 6, pp. 569–572, Dec. 2014.
  * [53] X. Xiao, W. Wang, T. Chen, Y. Cao, T. Jiang, and Q. Zhang, “Sensor-augmented neural adaptive bitrate video streaming on UAVs,” _IEEE Trans. Multimedia_ , vol. 22, no. 6, pp. 1567–1576, Jun. 2020.
  * [54] R. Muzaffar, E. Yanmaz, C. Raffelsberger, C. Bettstetter, and A. Cavallaro, “Live multicast video streaming from drones: An experimental study,” _Auton. Robot._ , vol. 44, pp. 75–91, 2020.
  * [55] J. Chakareski, “UAV-IoT for next generation virtual reality,” _IEEE Trans. Image Process._ , vol. 28, no. 12, pp. 5977–5990, Dec. 2019.
  * [56] C. He, Z. Xie, and C. Tian, “A QoE-oriented uplink allocation for multi-UAV video streaming,” _Sensors_ , vol. 19, no. 15, pp. 1–19, Aug. 2019.
  * [57] Q. Zhang, J. Miao, Z. Zhang, F. R. Yu, F. Fu, and T. Wu, “Energy-efficient video streaming in UAV-enabled wireless networks: A safe-DQN approach,” in _Proc. IEEE Global Commun. Conf._ , Dec. 2020, pp. 1–7.
  * [58] L. Zhang and J. Chakareski, “UAV-assisted edge computing and streaming for wireless virtual reality: Analysis, algorithm design, and performance guarantees,” _IEEE Trans. Veh. Technol._ , vol. 71, no. 3, pp. 3267–3275, Mar. 2022.
  * [59] Y. Guo, Y. Chen, J. Hu, and H. Zheng, “Trajectory planning of UAV with real-time video transmission based on genetic algorithm,” in _Proc. Cross Strait Radio Sci. Wireless Technol. Conf._ , Dec. 2020, pp. 1–3.
  * [60] C. Zhan, H. Hu, Z. Wang, R. Fan, and D. Niyato, “Unmanned aircraft system aided adaptive video streaming: A joint optimization approach,” _IEEE Trans. Multimedia_ , vol. 22, no. 3, pp. 795–807, Mar. 2020.
  * [61] L. A. b. Burhanuddin, X. Liu, Y. Deng, U. Challita, and A. Zahemszky, “QoE optimization for live video streaming in UAV-to-UAV communications via deep reinforcement learning,” _IEEE Trans. Veh. Technol._ , vol. 71, no. 5, pp. 5358–5370, May 2022.
  * [62] M. Khan, J. Chakareski, and S. Gupta, “RF-FSO dual-path UAV network for high fidelity multi-viewpoint scalable 360∘ video streaming,” in _Proc. IEEE 22nd Int. Workshop Multimedia Signal Process._ , Sep. 2020, pp. 1–6.
  * [63] A. Stornig, A. Fakhreddine, H. Hellwagner, P. Popovski, and C. Bettstetter, “Video quality and latency for UAV teleoperation over LTE: A study with ns3,” in _Proc. IEEE 93rd Veh. Technol. Conf._ , Apr. 2021, pp. 1–7.
  * [64] H. Zhou, F. Hu, M. Juras, A. B. Mehta, and Y. Deng, “Real-time video streaming and control of cellular-connected UAV system: Prototype and performance evaluation,” _IEEE Wireless Commun. Lett._ , vol. 10, no. 8, pp. 1657–1661, Aug. 2021.
  * [65] J. Jin, J. Ma, L. Liu, L. Lu, G. Wu, D. Huang, and N. Qin, “Design of UAV video and control signal real-time transmission system based on 5G network,” in _Proc. IEEE 16th Conf. Ind. Electron. Appl._ , Aug. 2021, pp. 533–537.
  * [66] T. Taleb, N. Sehad, Z. Nadir, and J. Song, “VR-based immersive service management in b5g mobile systems: A uav command and control use case,” _IEEE Internet of Things Journal_ , vol. 10, no. 6, pp. 5349–5363, 2023.
  * [67] S. Qazi, A. S. Siddiqui, and A. I. Wagan, “UAV based real time video surveillance over 4G LTE,” in _Proc. Int. Conf. Open Source Syst. Technol._ , Dec. 2015, pp. 141–145.
  * [68] M. Naveed, S. Qazi, B. A. Khawaja, and M. Mustaqim, “Evaluation of video streaming capacity of UAVs with respect to channel variation in 4G-LTE surveillance architecture,” in _Proc. 8th Int. Conf. Inf. Commun. Technol._ , Nov. 2019, pp. 149–154.
  * [69] C. Singhal and B. N. Chandana, “Aerial-SON: UAV-based self-organizing network for video streaming in dense urban scenario,” in _Proc. Int. Conf. Commun. Syst. Netw._ , Jan. 2021, pp. 7–12.
  * [70] Z. Liu and Y. Jiang, “Cross-layer design for UAV-based streaming media transmission,” _IEEE Trans. Circuits Syst. Video Technol._ , vol. 32, no. 7, pp. 4710–4723, Jul. 2022.
  * [71] K. Nihei, N. Kai, Y. Maruyama, T. Yamashita, D. Kanetomo, T. Kitahara, M. Maruyama, T. Ohki, K. Kusin, and H. Segah, “Forest fire surveillance using live video streaming from UAV via multiple LTE networks,” in _Proc. IEEE 19th Annu. Consum. Commun. Netw. Conf._ , Jan. 2022, pp. 465–468.
  * [72] T. Yu, Y. Takaku, Y. Kaieda, and K. Sakaguchi, “Design and PoC implementation of mmWave-based offloading-enabled UAV surveillance system,” _IEEE Open J. Veh. Technol._ , vol. 2, pp. 436–447, 2021.
  * [73] F. Hu, Y. Deng, and A. H. Aghvami, “Cooperative multigroup broadcast 360° video delivery network: A hierarchical federated deep reinforcement learning approach,” _IEEE Trans. Wireless Commun._ , vol. 21, no. 6, pp. 4009–4024, Jun. 2022.
  * [74] X. Lin, V. Yajnanarayana, S. D. Muruganathan, S. Gao, H. Asplund, H.-L. Määttänen, M. Bergstrom, S. Euler, and Y.-P. E. Wang, “The sky is not the limit: LTE for unmanned aerial vehicles,” _IEEE Commun. Mag._ , vol. 56, no. 4, pp. 204–210, Apr. 2018.
  * [75] 3GPP TR 36.777, “Enhanced LTE support for aerial vehicles.” [Online]. Available: https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?specificationId=3231
  * [76] 3GPP TS 22.125, “Unmanned aerial system (UAS) support in 3GPP,” accessed: 1-July-2023. [Online]. Available: https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?specificationId=3545
  * [77] ATIS-I-0000092, “3GPP Release 17 – building blocks for uav applications,” accessed: 1-July-2023. [Online]. Available: https://access.atis.org/apps/group_public/download.php/66824/ATIS-I-0000092.pdf
  * [78] 3GPP TS 26.918, “Virtual reality (VR) media services over 3gpp,” accessed: 1-July-2023. [Online]. Available: https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?specificationId=3053
  * [79] 3GPP TS 26.501, “5G media streaming (5GMS); general description and architecture,” accessed: 1-July-2023. [Online]. Available: https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?specificationId=3582
  * [80] 3GPP TS 26.506, “5G real-time media communication architecture,” accessed: 1-July-2023. [Online]. Available: https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?specificationId=4102
  * [81] 3GPP TS 26.522, “5G real-time media transport protocol configurations,” accessed: 1-July-2023. [Online]. Available: https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?specificationId=4114
  * [82] 3GPP TS 26.803, “Study on 5G media streaming extensions for edge processing,” accessed: 1-July-2023. [Online]. Available: https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?specificationId=3742
  * [83] 3GPP TR 26.927, “Study on Artificial Intelligence and Machine learning in 5G media services,” accessed: 1-July-2023. [Online]. Available: https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?specificationId=4040
  * [84] A.-F. Perrin, V. Krassanakis, L. Zhang, V. Ricordel, M. Perreira Da Silva, and O. Le Meur, “EyeTrackUAV2: A large-scale binocular eye-tracking dataset for UAV videos,” _Drones_ , vol. 4, no. 1, 2020.
  * [85] K. Fu, J. Li, Y. Zhang, H. Shen, and Y. Tian, “Model-guided multi-path knowledge aggregation for aerial saliency prediction,” _IEEE Trans. Image Process._ , vol. 29, pp. 7117–7127, 2020.
  * [86] S. Colonnese, F. Cuomo, L. Ferranti, and T. Melodia, “Efficient video streaming of 360° cameras in unmanned aerial vehicles: An analysis of real video sources,” in _Proc. 7th Eur. Workshop Visual Inf. Process._ , Nov. 2018, pp. 1–6.
  * [87] T.-W. Mi and M.-T. Yang, “Comparison of tracking techniques on 360-degree videos,” _Appl. Sci._ , vol. 9, no. 16, p. 3336, 2019.
  * [88] E. J. David, J. Gutiérrez, A. Coutrot, M. P. Da Silva, and P. L. Callet, “A dataset of head and eye movements for 360° videos,” in _Proc. 9th ACM Multimedia Syst. Conf._ , Jun. 2018, pp. 432–437.
  * [89] A. T. Nasrabadi, A. Samiei, A. Mahzari, R. P. McMahan, R. Prakash, M. C. Q. Farias, and M. M. Carvalho, “A taxonomy and dataset for 360° videos,” in _Proc. 10th ACM Multimedia Syst. Conf._ , Jun. 2019, pp. 273–278.
  * [90] C. Wu, Z. Tan, Z. Wang, and S. Yang, “A dataset for exploring user behaviors in VR spherical video streaming,” in _Proc. 8th ACM Multimedia Syst. Conf._ , Jun. 2017, pp. 193–198.
  * [91] VideoLAN, “libx264,” https://www.videolan.org/developers/x264.html, 2013, accessed: 1-July-2023.
  * [92] NVIDIA Corporation, “h264_nvenc,” https://developer.nvidia.com/nvidia-video-codec-sdk, 2018, accessed: 1-July-2023.
  * [93] MulticoreWare Inc., “x265,” https://www.videolan.org/developers/x265.html, 2021, version: 3.5.
  * [94] NVIDIA Corporation, “hevc_nvenc,” https://developer.nvidia.com/nvidia-video-codec-sdk, May 2021, sDK Version: 11.0.
  * [95] A. Wieckowski, J. Brandenburg, T. Hinz, C. Bartnik, V. George, G. Hege, C. Helmrich, A. Henkel, C. Lehmann, C. Stoffers, I. Zupancic, B. Bross, and D. Marpe, “Vvenc: An open and optimized vvc encoder implementation,” in _2021 IEEE International Conference on Multimedia & Expo Workshops (ICMEW)_, 2021, pp. 1–2.
  * [96] Open Visual Cloud, “Svt-av1,” https://www.openvisualcloud.org/, 2021, version: 0.8.7.
  * [97] WebM Project, “libvpx,” https://www.webmproject.org/code/, 2021, version: 1.10.0.
  * [98] T. Huang, R.-X. Zhang, and L. Sun, “Deep reinforced bitrate ladders for adaptive video streaming,” in _Proceedings of the 31st ACM Workshop on Network and Operating Systems Support for Digital Audio and Video_ , 2021, pp. 66–73.
  * [99] A. Mandhane, A. Zhernov, M. Rauh, C. Gu, M. Wang, F. Xue, W. Shang, D. Pang, R. Claus, C.-H. Chiang _et al._ , “Muzero with self-competition for rate control in vp9 video compression,” _arXiv preprint arXiv:2202.06626_ , 2022\.
  * [100] Intel, “True view intel sports,” 2021, accessed: 1-July-2023. [Online]. Available: https://www.intel.com/content/www/us/en/sports/technology/true-view.html
  * [101] X. Wang, A. Chowdhery, and M. Chiang, “Networked drone cameras for sports streaming,” in _Proc. IEEE 37th Int. Conf. Distrib. Comput. Syst._ , Jun. 2017, pp. 308–318.
  * [102] X. Meng, W. Wang, and B. Leong, “SkyStitch: A cooperative multi-UAV-based real-time video surveillance system with stitching,” in _Proc. 23rd ACM Int. Conf. Multimedia_ , Oct. 2015, pp. 261–270.
  * [103] Y. Wang and J. Farooq, “Zero touch coordinated uav network formation for $360^{\circ}$ views of a moving ground target in remote VR applications,” in _Proc. IEEE Military Commun. Conf._ , Nov. 2022, pp. 950–955.
  * [104] Z. Wang, J. Du, G. Li, and X. Song, “Control of panoramic video flow for UAV,” in _Proc. Int. Conf. Social Comput. Digit. Economy_ , Aug. 2021, pp. 105–107.
  * [105] J. Qian, J. Wang, and S. Jin, “Configurable virtual MIMO via UAV swarm: Channel modeling and spatial correlation analysis,” _China Commun._ , vol. 19, no. 9, pp. 133–145, Sep. 2022.
  * [106] M. S. Anwar, J. Wang, W. Khan, A. Ullah, S. Ahmad, and Z. Fei, “Subjective QoE of 360-degree virtual reality videos and machine learning predictions,” _IEEE Access_ , vol. 8, pp. 148 084–148 099, 2020.
  * [107] L. Teng, G. Zhai, Y. Wu, X. Min, W. Zhang, Z. Ding, and C. Xiao, “QoE driven VR 360° video massive MIMO transmission,” _IEEE Trans. Wireless Commun._ , vol. 21, no. 1, pp. 18–33, Jan. 2022.
  * [108] W. Huang, L. Ding, G. Zhai, X. Min, J.-N. Hwang, Y. Xu, and W. Zhang, “Utility-oriented resource allocation for 360-degree video transmission over heterogeneous networks,” _Digit. Signal Process._ , vol. 84, pp. 1–14, Jan. 2019.
  * [109] C. Perfecto, M. S. Elbamby, J. D. Ser, and M. Bennis, “Taming the latency in multi-user VR 360°: A QoE-aware deep learning-aided multicast framework,” _IEEE Trans. Commun._ , vol. 68, no. 4, pp. 2491–2508, Apr. 2020\.
  * [110] Y. Ding, D. Jiang, J. Huang, L. Xiao, S. Liu, Y. Tang, and H. Dai, “QoE-aware power control for UAV-aided media transmission with reinforcement learning,” in _Proc. IEEE Global Commun. Conf._ , Dec. 2019, pp. 1–6.
  * [111] R. Shirey, S. Rao, and S. Sundaram, “Optimizing quality of experience for long-range UAS video streaming,” in _Proc. IEEE/ACM 29th Int. Symp. Quality Service_ , Jun. 2021, pp. 1–10.
  * [112] S. Colonnese, A. Carlesimo, L. Brigato, and F. Cuomo, “QoE-aware UAV flight path design for mobile video streaming in HetNet,” in _Proc. IEEE 10th Sensor Array Multichannel Signal Process. Workshop_ , Jul. 2018, pp. 301–305.
  * [113] M. Tang and V. W. Wong, “Online bitrate selection for viewport adaptive 360-degree video streaming,” _IEEE Trans. Mobile Comput._ , vol. 21, no. 7, pp. 2506–2517, Jul. 2022.
  * [114] J. Yang, J. Luo, D. Meng, and J.-N. Hwang, “QoE-driven resource allocation optimized for uplink delivery of delay-sensitive VR video over cellular network,” _IEEE Access_ , vol. 7, pp. 60 672–60 683, 2019.
  * [115] L. A. b. Burhanuddin, X. Liu, Y. Deng, U. Challita, and A. Zahemszky, “QoE optimization for live video streaming in UAV-to-UAV communications via deep reinforcement learning,” _IEEE Trans. Veh. Technol._ , vol. 71, no. 5, pp. 5358–5370, May 2022.
  * [116] Y. Liu, C. Zhu, X. Deng, P. Guan, Z. Wan, J. Luo, E. Liu, and H. Zhang, “UAV-aided urban target tracking system based on edge computing,” _CoRR_ , vol. abs/1902.00837, pp. 1–6, Feb. 2019.
  * [117] E. Calvanese Strinati and S. Barbarossa, “6G networks: Beyond Shannon towards semantic and goal-oriented communications,” _Computer Networks_ , vol. 190, p. 107930, May 2021.
  * [118] J. Devlin, M.-W. Chang, K. Lee, and K. N. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” 2018. [Online]. Available: https://arxiv.org/abs/1810.04805
  * [119] L. Bariah, Q. Zhao, H. Zou, Y. Tian, F. Bader, and M. Debbah, “Large language models for telecom: The next big thing?” _arXiv preprint arXiv:2306.10249_ , 2023.
  * [120] S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor, “ChatGPT for robotics: Design principles and model abilities,” Microsoft, Tech. Rep. MSR-TR-2023-8, February 2023. [Online]. Available: https://www.microsoft.com/en-us/research/publication/chatgpt-for-robotics-design-principles-and-model-abilities/
  * [121] Z. Wang, J. Yu, A. W. Yu, Z. Dai, Y. Tsvetkov, and Y. Cao, “SimVLM: Simple visual language model pretraining with weak supervision,” in _International Conference on Learning Representations_ , 2022. [Online]. Available: https://openreview.net/forum?id=GUrhfTuf_3
  * [122] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisserman, and K. Simonyan, “Flamingo: a visual language model for few-shot learning,” in _Advances in Neural Information Processing Systems_ , A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, Eds., 2022. [Online]. Available: https://openreview.net/forum?id=EbMuimAbPbs
  * [123] J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” 2023.

  *[UAVs]: unmanned aerial vehicle
  *[UAV]: unmanned aerial vehicle
  *[6DoF]: six degrees of freedom
  *[AVC]: advanced video coding
  *[HEVC]: high-efficiency video coding
  *[VVC]: versatile video coding
  *[VP9]: VP9
  *[ODV]: omnidirectional video
  *[QoE]: quality of experience
  *[E2E]: end-to-end
  *[mmWave]: millimeter wave
  *[2D]: two-dimensional
  *[3GPP]: Third Generation Partnership Project
  *[FoV]: field-of-view
  *[AI]: artificial intelligence
  *[ERP]: equirectangular projection
  *[EAC]: equi-angular cubemap
  *[CMP]: cube map projection
  *[TSP]: truncated square pyramid
  *[AV1]: AOMedia video 1
  *[MCTS]: motion-constrained tile set
  *[SEI]: supplemental enhancement information
  *[MV]: motion vector
  *[EEO]: entropy equilibrium optimization
  *[MVs]: motion vector
  *[CNN]: convolution neural network
  *[ANN]: artificial neural network
  *[OMAF]: omnidirectional media format
  *[WebRTC]: web real-time communication
  *[MPEG]: motion picture experts group
  *[3DoF]: three degrees of freedom
  *[ISOBMFF]: ISO base media file format
  *[DASH]: dynamic adaptive streaming over HTTP
  *[RWP]: region-wise packing
  *[VS]: viewport-specific
  *[VSs]: viewport-specific
  *[RWQR]: region-wise quality ranking
  *[RTP]: real-time transport protocol
  *[SRTP]: secure real-time transport protocol
  *[RTCP]: real-time transport control protocol
  *[VR]: virtual reality
  *[G2G]: glass-to-glass
  *[G2A]: glass-to-algorithm
  *[M2P]: motion-to-photon
  *[HMD]: head-mounted display
  *[PSNR]: peak signal-to-noise rate
  *[SSIM]: structural similarity index measure
  *[VMAF]: video multi-method assessment fusion
  *[LoS]: line-of-sight
  *[NLoS]: non-line-of-sight
  *[SVC]: scalable video coding
  *[BS]: base station
  *[BSs]: base station
  *[3D]: three-dimensional
  *[LTE]: Long Term Evolution
  *[APs]: access point
  *[AP]: access point
  *[HM]: head movement
  *[EM]: eye movement
  *[ODVs]: omnidirectional video
  *[VVenC]: fraunhofer versatile video encoder
  *[BD-rate]: bjøntegaard delta rate
  *[SW]: Software
  *[HW]: Hardware
  *[FPV]: First Person View
  *[CPU]: central processing unit
  *[MIMO]: multiple-input and multiple-output
  *[LLMs]: large language model
  *[LLM]: large language model
  *[NLP]: natural language processing
  *[GPT]: generative pre-trained Transformer