# DCP and VarDis: An Ad-Hoc Protocol Stack for Dynamic Swarms and Formations
of Drones
Extended Version

Samuel Pell, Andreas Willig
Department of Computer Science
University of Canterbury
Christchurch, New Zealand
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recently, swarms or formations of drones have received increased interest both
in the literature and in applications. To dynamically adapt to their operating
environment, swarm members need to communicate wirelessly for control and
coordination tasks. One fundamental communication pattern required for basic
safety purposes, such as collision avoidance, is beaconing, where drones
frequently transmit information about their position, speed, heading, and
other operational data to a local neighbourhood, using a local broadcast
service. In this paper, we propose and analyse a protocol stack which allows
to use the recurring-beaconing primitive for additional purposes. In
particular, we propose the VarDis (Variable Dissemination) protocol, which
creates the abstraction of variables to which all members of a drone swarm
have (read) access, and which can naturally be used for centralized control of
a swarm, amongst other applications. We describe the involved protocols and
provide a mainly simulation-based performance analysis of VarDis.

###### Index Terms:

Drone formations; data disssemination; performance analysis; simulation;
Markov modeling.

##  I Introduction

Unmanned aerial vehicles (UAVs) or drones are nowadays used in a wide range of
commercial or scientific applications, including environmental monitoring and
remote sensing [1, 2], infrastructure monitoring, search and rescue [3, 4],
surveillance, telecommunications and public safety communications [5],
agriculture [6, 7] and forestry [8], delivery of small goods in urban areas
[9], and many others. Many of these applications focus on individual drones,
but there is also increasing interest in applications and communications /
networking support for swarms or formations of closely collaborating drones
[10, 11, 12, 13, 14, 2, 15]. Drone swarms naturally allow for parallelization
of a task. For example, in a search-and-rescue operation, to speed up the
overall search process, the search area can be partitioned and each drone is
assigned to one such partition. Beyond this, using multiple drones allows
observation of the same phenomenon from different angles or using different
sensors at the same time, or to form flying antenna arrays supporting
localization of transmitters [16, 17] or telecommunications.

Depending on the application, drone swarms can require significant amounts of
coordination and data exchange between drones. A very basic (and application-
independent) requirement is that drones are able to keep track of the
position, speed and heading of nearby drones to avoid collisions, particularly
in the presence of disturbances such as wind gusts. The knowledge of position
/ speed / heading of neighbours is also a pre-requisite for algorithms
controlling the movements and relative positions of drones in a swarm, e.g.
algorithms based on virtual spring models [18, 19]. When it comes to
determining the movements of the swarm as a whole, this can either be done in
a distributed fashion (e.g., in the virtual spring algorithm proposed in [18],
the drones closer to a location of interest “pull” the other drones towards
that location), or in a centralized “leader-follower” fashion, with one
“leader” drone disseminating instructions to the other swarm members [20, 21,
22].

These various safety and coordination tasks require wireless communication
between drones. In this context, a drone swarm can be regarded as a
(potentially) large-scale, multi-hop and connected wireless network.
Currently, one major research strand is to use modern cellular networks such
as 4G, 5G or the upcoming 6G standard to support communications between drones
[23, 5]. However, these are not always available, in particular in disaster
scenarios or sparsely populated areas. Therefore, we are interested in
employing ad-hoc networking techniques for UAV communications, which often
comes under the heading of flying ad-hoc networks or FANETs [10, 11, 24, 25].

The key goal of this paper is to provide networking support for coordination
and safety tasks in large-scale flying ad-hoc networks such as drone swarms
and formations, with only modest requirements on the underlying wireless
technology. The focus on coordination- and safety-related tasks has some
impact on the design of such protocols, as these tasks often only require
small packet sizes and many periodic transmissions. For example, to avoid
collisions with neighboured drones, it is natural to adopt a similar approach
as in ad-hoc vehicular communications, whereby each vehicle frequently
broadcasts its current position, speed, heading, and other operational data to
neighboured vehicles, using a local broadcast. The overall size of this
operational data is quite limited, e.g. the required part of a WAVE basic
safety message transmitted in DSRC is in the order of 40 bytes [26, Table 8].
Many road safety applications require that vehicles transmit such messages at
a rate of 10 Hz [27]. In this paper we refer to the recurring transmission of
such safety-related data to a local neighbourhood as _beaconing_ , and one
such locally broadcast packet is simply called a _beacon_. Another motivating
example for this work is centralized position and pose control of a swarm or
rigid drone formation [22]. A rigid formation can be described with only six
degrees of freedom, three position and three angular coordinates. If a drone
formation operates in such a way that each drone can derive its own trajectory
from the knowledge of the movements of the formation and its own position
relative within the formation, then it is sufficient for a leader drone to
disseminate timestamped records containing six floating point numbers into the
formation. These records should be disseminated quickly and reliably, to
achieve consistent trajectories and avoid collisions.

Different protocol mechanisms have been used in the literature to support this
kind of global dissemination of data, e.g. based on a spanning tree
construction, clustering or other topology-control methods, or on flooding
(see Section II). The approach adopted in this paper rests _entirely_ on the
periodic local beaconing process.111A note on terminology: When we use the
term periodic in this paper we do not mean strictly periodic, but allow for
some variation in inter-beacon transmission times. We assume that there exists
a well-defined average interval, and we refer to this average as the period.
The beacons are not only used to transmit safety-related information to
immediate neighbours, but we also piggyback global coordination data and make
sure that this data is disseminated through the entire network. Specifically,
we introduce the abstraction of a globally shared _variable_ , discussed
below. Piggybacking global coordination data onto beacons can reduce packet
overheads, as small pieces of coordination data are not wrapped into their own
packets anymore, but may come at the cost of increasing packet length and
delay. Furthermore, if the underlying medium access control (MAC) protocol is
based on CSMA (as is the case in WiFi [28]), then piggybacking reduces the
number of channel access cycles — in particular, the safety-critical beacon
packets do not have to compete anymore with other packets flooded into the
network for global dissemination. A further advantage of only relying on
locally broadcast beacons is that the dissemination does not require routing
or any other topological state (no parent-child or similar routing
relationships need to be maintained), and will therefore be quite immune to
any topology changes coming from mobility, as long as connectivity is
maintained. We also remark that by relying on only one service of the
underlying wireless technology, the proposed approach is largely agnostic of
this technology.

In this paper we describe and analyse a protocol stack built on these ideas,
the Drone Coordination Protocols (DCP) stack. At its base, DCP enables
periodic beaconing. To support global coordination and collaboration in a
swarm, one of the component protocols in our stack, the VarDis (Variable
Dissemination) protocol, offers the abstraction of a globally shared
_variable_ , which can be dynamically created, read, updated and deleted
(CRUD). The protocol disseminates information about any modifying operation on
a variable (create, update, delete) to the entire network within (repeated)
beacon payloads. It includes a mechanism by which nodes can recognize that
they do not have the most recent value of a variable, and it allows such nodes
to request that value from neighbours. Every node that has received a new
modifying instruction for a variable (create, update, delete) will repeat that
instruction in a number of distinct beacons. This approach combines repetition
coding (or time diversity) and, importantly, spatial diversity, as the same
information is repeated by several spatially distributed nodes. In more
detail, in this paper we make the following contributions:

  * •

We describe the DCP (Drone Coordination Protocols) protocol stack, which
includes at its base a simple protocol allowing to embed different payload
types into frequently transmitted beacons (multiplexing), and the VarDis
protocol, implementing the variable abstraction.

  * •

We present the results of a (mostly) simulation-based performance evaluation
of VarDis, focusing on its reliability and latency in a range of scenarios.
Our performance evaluation includes a sensitivity analysis, showing how
performance depends on key VarDis and beaconing parameters. We also include a
comparison of VarDis delay performance against numerical results obtained from
a discrete-time Markov chain model in a simplified setting, to validate the
simulation model. Furthermore, we also compare VarDis against a flooding
protocol, which does not rely on periodic beaconing but floods each variable
update separately.

Our results confirm that VarDis is a viable solution for global dissemination
of coordination data, particularly in more dense networks. It is possible to
tune its parameters to achieve quite competitive delay and reliability
performance, both in absolute numbers and compared to a flooding protocol.

The protocol specification of the DCP stack and the VarDis protocol in Version
1.0 is publicly available, as is the source code of the simulation model used
in this paper.222https://github.com/awillig/dcp-vardis.git

The remaining paper is structured as follows: we next discuss related work in
Section II. In Section III we describe the DCP protocol stack and the VarDis
protocol in particular. Our system model and setup for the simulation-based
performance evaluation are discussed in Section IV. In Section V we present
our performance results for a range of scenarios. Conclusions and a discussion
of future work are given in Section VI.

##  II Related Work

###  II-A Data Dissemination

Many different protocols have been used to support global data dissemination
in a wireless network.

In one broad class of protocols suitable routing structures are established.
For example, in [21] a spanning tree rooted in the leader is constructed based
on hop distances to the leader node. Each follower picks a suitable parent and
retrieves the required information from it. A similar example from the sensor
network domain is the CTP protocol [29], [30]. Other examples are based on
clustering [31, 32]. In the presence of significant relative mobility,
however, such protocols have to spend some effort and time on maintaining and
updating these routing structures, and there is always a risk of stale
entries.

Approaches based on opportunistic or epidemic data dissemination such as the
one proposed in [33] work naturally in intermittently connected networks, but
often only address the case of unicast delivery and do not guarantee that a
given piece of information is provided to _every_ node in the network. They
also differ from our semantics of a variable — usually they treat different
messages as entities to be disseminated individually and completely, and do
not allow to treat one message as an updated version of another one and
overwrite the old version. Finally, the context in which opportunistic or
epidemic protocols are analyzed is often very different from the context of
online control with tight latency targets relevant for swarm control. Another
approach is to rely on classical flooding or optimized flooding techniques
[34] — in a lightly loaded network these indeed can disseminate information to
everyone quickly and with very high reliability, but they have a lot of
overhead (particularly when the data items to be disseminated are small), they
are susceptible to the broadcast storm problem [35], and can become very
inefficient when several flooding operations triggered by different nodes run
in parallel.

Perhaps closest in spirit to our designs are gossip protocols as used in
consensus schemes [36, 37, 38, 39, 40, 41, 42]. In consensus problems, nodes
attempt to reach agreement on a particular value, for example the average
temperature. Each node starts with its own local temperature observation,
collects observations from neighbours and processes these together with its
own current value to calculate a new estimate, and then in turn transmit this
estimate to neighbours. If implemented correctly, the local estimates for
nodes will converge to the average. A range of link models and node failure
models are varied in these investigations, often the analysis focuses on (time
until) convergence, e.g. when calculating the average of the initial values or
their maximum [43, 44, 45]. In terms of protocols, consensus schemes often
either use global broadcasts (transmissions of nodes with sufficient transmit
power to reach every other node) or gossiping, where information is
disseminated through the network using local broadcasts. It is important to
note that the semantics of consensus problems differs from the semantics of a
variable in VarDis.

###  II-B Congestion Control

In our simulations, we use beaconing on top of IEEE 802.11g. The overall setup
is quite similar to the beaconing process known from vehicular networks. The
beacons are broadcast locally and unacknowledged, and since the most-used IEEE
802.11g MAC protocols (DCF or EDCA) are CSMA-based, they suffer from direct or
hidden-terminal collisions, leading to packet losses. This problem, which is
often referred to as congestion control problem, is well-studied [46, 47, 48,
49], and some of the results presented in this paper are affected by this
problem.

##  III The Drone Control Protocol (DCP) and the Variable Dissemination
(VarDis) Protocol

In this section we describe the overall DCP protocol stack and the variable
dissemination (VarDis) protocol.

###  III-A Overall Architecture

ApplicationsOthersSRPVarDisOthersBeaconing Protocol (BP)Underlying Wireless
Bearer (UWB) Figure 1: DCP Protocol Stack

The overall DCP protocol stack and its environment are shown in Figure 1. At
the lowest layer, and outside the DCP protocol stack, we have the _underlying
wireless bearer_ (UWB). We do not make strong assumptions about its abilities
and services, we only assume that it provides a local broadcast service, i.e.
it allows a node to broadcast a packet to a local single-hop neighbourhood
(generally determined by transmit power, antenna characteristics and
propagation environment), and to receive and process such broadcasts from
neighboured nodes. We also assume that it allows for a maximum packet size of
at least 200 – 300 bytes. The local broadcasts are generally unacknowledged,
but the packets are protected by checksums, and packets with bit errors are
dropped. Based on this and on the general absence of receiver feedback for
local broadcasts, we have chosen not to include separate error detection or
correction mechanisms into any of the DCP protocols. We also assume that the
UWB provides protocol multiplexing, so that it can run DCP in parallel to
other client protocols such as a TCP/UDP/IP stack. For our performance
evaluation the UWB is provided by WiFi [28], such that DCP packets are
directly encapsulated into WiFi payloads.

The lowest layer of DCP proper is provided by the _beaconing protocol_ (BP),
which operates on top of the UWB. The beaconing protocol is mainly responsible
for the frequent generation and transmission of _beacons_ including one or
more _client payloads_ , coming from BP client protocols such as the State
Reporting Protocol (SRP) or VarDis. The beacons are transmitted using the
local broadcast service of the UWB. The beacon generation rate is
configurable. In general, the protocol does not specify which payloads from
client protocols are included in any specific beacon, but for the purposes of
this paper we only include VarDis as a client protocol, and VarDis payloads
consequently are always included. In the receive direction, the BP can receive
beacons sent by neighboured nodes and hand over the included payloads to their
respective client protocols. Client protocols can dynamically register and de-
register with BP, and are being offered different interfacing methods (e.g.
queues vs. single-packet buffers). In other words, BP provides protocol
multiplexing amongst client protocols. The BP is described in more detail in
Section III-B.

The State-Reporting Protocol (SRP) is a client protocol of BP. Its main
responsibility is to periodically generate payloads containing the position,
speed, heading and other operational data of the sending drone. This data is
only sent to immediate neighbours and its main purpose is the avoidance of
drone collisions. In the receive direction, an SRP entity uses received SRP
payloads to build a _neighhbour table_ , which lists all the currently known
neighbours, their positions / speed / heading etc., and the time that has
passed since the last reception of a SRP payload. The main purpose of the
neighbour table is to allow a drone to assess the uncertainty in a neighbours
position and predict impending collisions. The entries of the neighbour table
use a soft-state mechanism to push out stale entries. We will not discuss the
SRP any further, as it is outside the scope of this paper.

The Variable Dissemination (VarDis) protocol also sits as a client protocol on
top of BP. It allows applications to create, read, update and delete (CRUD)
globally shared _variables_ , which VarDis disseminates into the entire
network, with the goal to achieve a consistent view after any modifying
variable operation (create, update, delete) as quickly and reliably as
possible. In the VarDis version reported in this paper (Version 1.0), update
and delete operations are restricted to the node that created the variable.
The protocol also contains mechanisms for nodes to detect missing variables or
updates, and to request this missing information from neighbours. VarDis is
described in more detail in Section III-C.

###  III-B Beaconing Protocol (BP)

As mentioned above, the key tasks of BP are:  (i) the periodic transmission of
beacons carrying client protocol payloads to the local neighbourhood, using
local broadcasts;  (ii) the reception of beacons from neighbours and
forwarding received client protocol payloads to their respective client
protocol; and  (iii) the dynamic registration, query and de-registration of
these client protocols.

In the transmit direction, the interface between a client protocol and BP
offers three different ways of buffering payloads:

  * •

Queueing: payloads handed over by the client protocols are stored in a first-
in-first-out queue of indefinite size. Once a payload has reached the head of
the queue, it can be inserted into an outgoing beacon. After its insertion the
payload is dropped. Hence, each payload is transmitted exactly once.

  * •

Buffered-Once: payloads are stored in a single buffer and can be overwritten
by new payloads if they have not been transferred into a beacon yet. Once the
buffer contents have been transferred into a beacon, the buffer is cleared.
Hence, each payload is transmitted at most once.

  * •

Buffered-Repeated: payloads are stored in a single buffer and can be
overwritten by new payloads. In contrast to Buffered-Once, the buffer is never
cleared, i.e. the same payload is inserted into subsequent beacons as long as
it is not overwritten. Hence, each payload can be transmitted an arbitrary
number of times, including zero.

The protocol specification leaves it to implementations to define at which
times the BP generates new beacons, it only stipulates that the long-term
average beacon transmission rate converges to a configurable value (this
configuration can be changed at runtime). For the purposes of this paper, we
assume that the inter-beacon generation times are independent and identically
distributed (iid) random variables, which has the benefit of de-synchronizing
the nodes. In our simulations, the beacon transmissions occur with an average
rate of $\beta$ beacons per second, where the inter-beacon times are either
taken from an exponential distribution with rate parameter $\beta$, or from
$U\left[\frac{0.9}{\beta},\frac{1.1}{\beta}\right]$, which we refer to as
“periodic with 10 % jitter” at rate $\beta$ in the remainder of this paper.

The protocol specification also does not prescribe in which order the buffers
or queues associated with the different client protocols are checked to see
whether they have a payload ready, this is left to implementations (these
should generally make sure that each beacon contains an SRP payload when the
SRP protocol is present). However, in this paper the only client protocol
considered is VarDis.

###  III-C Variable Dissemination Protocol (VarDis)

VarDis333The specification is available at https://github.com/awillig/dcp-
vardis.git. is a BP client protocol utilizing the Buffered-Once queueing mode.
On the highest level, VarDis maintains a _real-time database_ 444Our use of
the term “real-time” does not imply that we can guarantee any latency or
reliability, it only represents our aspiration to achieve low latency and high
reliability when disseminating changes to variable values., which is a
collection of globally shared _variables_. A variable refers to something
which has a unique identifier (represented by an integer value), a current
value generally varying over time (represented by a length value and a
contiguous block of bytes without any further discernible structure), and an
indication of how recent it is (represented through sequence numbers). The
protocol aims to disseminate all changes to the variable value to all drones
in as little time and as reliably as possible. The byte blocks are assumed to
be of small to moderate size (significantly smaller than the maximum beacon
size), VarDis does not support a fragmentation-and-reassembly type of
mechanism for individual variables. We support four basic operations on a
variable: creating it, reading its value, updating its value, and deleting it
(CRUD). Variables can be created and deleted dynamically, their variable
identifier must be unique.555Uniqueness must be ensured by the users, it is
not strongly enforced by the protocol. In particular, when two nodes multiple
hops away create a variable with the same variable identifier at about the
same time, some nodes in the middle will be able to detect this condition, but
the protocol does not contain a mechanism to repair it or to even signal this
condition.

Currently, each variable in VarDis has one _producer_ node: the producer node
is the unique node creating the variable, and this node is also the only node
allowed to update the value of the variable or deleting it — in other words,
our variables are single-writer. All other nodes are only allowed to read the
value of the variable, we occasionally also refer to them as
_consumers_.666Conceptually, VarDis is not a publish/subscribe scheme [50], as
every node consumes every variable, without there being a choice concerning
its subscription.

####  III-C1 Variable Attributes

A variable is described by its producer through a _variable specification_ ,
which includes its variable identifier (varId), the unique node identifier
(e.g. MAC address) of the variable producer, a textual description of the
variable, and a repetition counter (repCnt), which specifies how often each
node should repeat an instruction to create, update, or delete a variable in
distinct beacons.

The producer of a variable furthermore maintains a sequence number for it,
which is initialized to zero at the time of its creation, and incremented for
each new update operation. When a variable update is disseminated through the
network, the sequence number is included so that consumer nodes can check
whether a received update is truly more recent than what they currently have
stored in their local real-time database.

Any consumer node777Which includes the producer as well, which additionally is
a consumer of its own variable. maintains a _database entry_ for a variable,
with its varId as a key. This database entry contains the variable
specification, the current variable value (both length and byte array), the
sequence number of the last received update, a local timestamp representing
the time this last update was received, and counters telling how many further
repetitions of a variable create / update / delete instruction are to be
included in future distinct beacons. The operation of reading a variable
simply refers to retrieving its current value from its database entry. A
consumer can furthermore retrieve a list of current database entries from the
database.

####  III-C2 Variable Creation, Update and Deletion

When the application on the producer node of a variable decides to create that
variable, it hands over a variable specification and an initial value for the
variable. The producer wraps both into a data structure signifying an
instruction to create a variable to other nodes. It then includes this data
structure into a given number of distinct beacons (c.f. the repCnt parameter).
Each other network node receiving this instruction for the first time adds the
new variable to its real-time database, and repeats the create instruction in
as many future distinct beacons as given by the repCnt parameter.

The same approach of having all other network nodes repeat an instruction in
as many future distinct beacons as indicated in the repCnt parameter of a
variable is also followed when the producer disseminates the instruction to
delete a variable, or when it disseminates an update to a variable. Such an
update instruction carries the varId of the variable, its current sequence
number (seqno) as assigned by the producer, and its new length and value. When
a consumer receives an update instruction for an existing variable, it will:

  * •

Discard the update when it has the same sequence number as stored in the local
real-time database.

  * •

Update the local real-time database when the received update instruction has a
strictly more recent sequence number than stored in the real-time database.

  * •

Discard the update when it has a strictly older sequence number than stored in
the local real-time database, and additionally schedule repCnt transmissions
of the current variable value (in an update instruction) in distinct future
beacons. The purpose of this is to correct an outdated value in the original
sender of the update instruction.

In this paper we ignore issues around the representation of sequence numbers,
wrap-overs and re-start after node crashes.

####  III-C3 Summary Mechanism, Update and Create Requests

If there is sufficient space in a beacon, each node includes _summaries_ of
the variables in its internal real-time database into its beacons. Generally
not all variables can be summarized in a single beacon, so the summaries are
sent in a round-robin fashion. A summary for a variable includes its varId and
its seqno. The summaries sent by a node are not repeated by its neighbours.

When a node $A$ receives a variable summary from a neighbour, it first checks
the varId. When $A$ does not have that variable in its local real-time
database, it includes a special request instruction in one of its next
beacons. This request instruction (“create-request”) carries the varId of the
missing variable and any neighbour receiving this will include a variable-
creation instruction in as many future distinct beacons as the repCnt for this
variable indicates. This mechanism gives $A$ a way to learn the specification
and current value of a variable it did not previously have.

If on the other hand $A$ does have the variable in its real-time database, but
in a strictly outdated version (as recognizable from the sequence numbers),
then $A$ includes another special request instruction in a future beacon
(“update-request”). In this update-request instruction, $A$ includes the varId
and the sequence number from its real-time database. Any neighbour hearing
this update-request will include variable updates into repCnt future distinct
beacons, provided they have strictly more recent updates than the requesting
node.

It is important to note that VarDis does not guarantee that a node will
receive every update ever issued by the producer — if updates are generated
too quickly, newer values may “over-write” older ones during dissemination.
This leads to “update gaps”, which we investigate in our performance
evaluation.

####  III-C4 Packet Construction and Transmit Path

A VarDis beacon payload can have up to six different sections. These sections
are prioritized amongst each other, and a section is only added to a beacon
when there is sufficient remaining space (VarDis allows to limit its maximum
payload size). The highest-priority section are the instructions to create
variables, followed by instructions to delete variables. Third are variable
updates, then variable summaries. Fifth priority have create-request
instructions, and finally then update-request instructions are being included.
Within each section (except the section for variable summaries), entries for
variables are added as long as space is available. The number of summaries to
be included in the summary section is upper-bounded by a configurable protocol
parameter, referred to as maxSumCnt in the remainder of the paper.

VarDis provides payloads to the underlying BP for every outgoing beacon. If
all sections are empty, then no payload is being prepared.

##  IV System Model and Simulation Setup

We primarily use a simulation-based approach to explore the performance of
VarDis and BP in different scenarios and for different parameter combinations.
For our simulations we use WiFi (IEEE 802.11g) with the distributed
coordination function (DCF) as the UWB. All simulations were carried out using
OMNeT++ in version 6.0888https://www.omnetpp.org and the INET framework in
version 4.4999https://inet.omnetpp.org, which provides the models for the IEEE
802.11g radio and channel. Our simulation code and setup files are available
online101010https://github.com/awillig/dcp-vardis.git. It should be noted that
the simulation code does not fully conform to the specification, but the
differences are irrelevant for the purposes of this paper.

In this section we describe our system model in more detail. In Table I we
summarize some key parameters that we keep fixed throughout, whereas Table II
describes the parameters we vary. We also define a simple flooding protocol to
compare VarDis against.

Parameter |  Explanation | Value
---|---|---
Channel and PHY parameters
$\eta$ |  Path loss exponent | 2.25
|  PHY data rate | 12 Mbps
$PL_{0}$ |  Path loss at reference distance 1 m | 40 dB
$N_{0}$ |  Thermal noise intensity |  $-174$ dB/Hz
|  PHY Data rate (code rate 3/4, 16-QAM) | 36 Mb/s
|  Transmit power | 0 dBm
BP and VarDis parameters
|  Maximum beacon length | 200 B
|  Random inter-beacon jitter | 10 %
|  Variable length | 12 B
TABLE I: Fixed Parameters Parameter |  Explanation
---|---
Deployment Parameters
$P$ |  Packet error rate on links (%)
$K$ |  Controls number of nodes in deployment ($K$ in line deployments, $K^{2}$ in grid deployments)
BP and VarDis parameters
$\beta$ |  Beacon transmission rate (Hz)
$\lambda$ |  Average inter-arrival time for variable updates at a producer (s)
repCnt |  repetition counter for modifying operations
maxSumCnt |  Maximum number of summaries in a beacon
TABLE II: Varied parameters

###  IV-A Channel Model and Physical Layer Settings

To allow us to focus purely on the effect of VarDis protocol mechanisms, we
opt for a relatively simple air-to-air channel model that takes path loss into
account, but does not model shadowing, fading or external interference. While
idealized, these settings correspond to a situation where a drone swarm flies
high enough and uses a sufficiently small transmit power to inhibit multipath
propagation. In particular, we employ the log-distance path loss model [51]
with path loss exponent $\eta=2.25$ and path loss at reference distance
$PL_{0}=40$ dB (the reference distance is 1 m). The only noise source is
thermal noise at $-174$ dBm/Hz.

On the physical layer we use IEEE 802.11g settings, operating in the 2.4 GHz
band (channel 1, centre frequency of 2,412 MHz), using OFDM with a 36 Mb/s
modulation and coding scheme (16-QAM, 3/4 code rate), and a transmit power of
0 dBm. The simulated radio has a receiver sensitivity of $-96$ dBm. With these
settings the transmission range (interpreted as the distance within which
packet reception is still possible with a packet error rate of less than 10
%)111111All references to packet error rates made in this paper are with
reference to packets with a 50B MAC payload, corresponding to 77B bytes
transmitted in total. is 255 m, whereas the interference range (defined as the
distance at which the receiver judges the signal energy to just still be above
the carrier-sense threshold, but without being able to decode successfully) is
294 m.

###  IV-B Deployments and Traffic

Figure 2: Line deployment with fixed density and $K$ nodes. The half-filled
node is the sole producer of a variable, the fully filled node is the
reference consumer. Each link is $L_{P}$ meters, where $P$ is a given packet
error rate.

Figure 3: Line deployment with variable density and $K$ nodes. The half-filled
node is the sole producer of a variable, the fully filled node is the
reference consumer.

We investigate node deployment scenarios of different density and size, but in
all scenarios the nodes are stationary. We simulate two different types of
line deployments with $K$ nodes each. In _fixed-density_ line deployments, we
place nodes at a fixed distance $L_{P}$ between neighbours, where $L_{P}$ is
chosen to achieve a given packet error rate of $P$ % over one link — compare
Figure 2. In _variable-density_ line deployments (cf. Figure 3) we also place
nodes equidistantly, but we keep the position of the two end nodes fixed to
$D=1,120\,\mbox{m}$ as $K$ (and with it the inter-node distance) varies.

Figure 4: Grid deployment with $K$ nodes on a side and $K^{2}$ nodes in total,
with either fixed density (all vertical or horizontal links have common packet
error rate $L_{P}$) or variable density (total side length fixed to 1,120 m).
All nodes produce a variable, the producer and consumer we focus on are
highlighted as the half-filled node in the lower right corner (producer) and
the fully filled node in the upper left corner (consumer).

We also simulate two different types of regular grid deployments, with $K$
nodes on each side and $K^{2}$ nodes in the overall deployment (cf. Figure 4).
In fixed-density grid deployments the length of one vertical or horizontal
link is fixed to $L_{P}$, to achieve a given packet error rate $P$ % on that
link, whereas in a variable-density deployment the total side length (from one
corner node to a neighboured corner node) is kept fixed at 1,120 m as $K$
varies.

For our simulations, we created a dummy VarDis application and run one
instance on every node. The nodes chosen as producers update their variable
with an average inter-update time of $\lambda$ s, which is either strictly
periodic (in the line experiments) or follows an exponential distribution (in
the grid experiments). Each of these variables is 12 B long, with 8 B used to
encode the simulation time at which the variable was updated at the producer
(as a double float), and 4 B are used for an application sequence
number.121212Using a separate sequence number in an application variable
allows to make it longer than the sequence numbers we use in a VarDis variable
update, and removes the need to instrument the VarDis code as such. Each
producer initializes its local sequence number to zero and increments it
before each variable update. The timestamps can be used by a consumer node to
measure the end-to-end application-layer update delay (time between generation
of the update at the producer and its reception at the consumer), and the
sequence number can be used to identify whether the consumer has missed any
updates. The update delay and the difference in received sequence number
(“gap”) are the two metrics we use to judge the performance of VarDis in the
rest of the paper (cf. Section IV-E).

In the line deployments we have only one producer of a variable, and we
consider only one consumer — these two nodes are placed on either end of the
line. In the grid deployments, _all_ nodes produce a variable, but when
showing our results we focus on one particular producer (lower right node in
the grid, see Figure 4) and one particular consumer (upper left node). We
consider these two reference nodes to represent the “worst-case”.

###  IV-C MAC Protocol and Beaconing

On the MAC layer we use the IEEE 802.11 DCF with default contention window
parameters. The network runs in ad-hoc mode without any requirements for
authentication or association. We only generate beacons containing VarDis
payloads, there is no other traffic source present in the system. The beacons
are locally broadcast. Throughout all simulations the maximum beacon size is
set to 200 B (except where noted otherwise), SRP payload generation is
disabled. The beacon inter-arrival times are drawn independently from the
uniform distribution over $\left[\frac{0.9}{\beta},\frac{1.1}{\beta}\right]$
(we sometimes refer to this as “periodic with 10 % jitter”) or from an
exponential distribution with rate parameter $\beta$. In both cases, $\beta$
(in Hz) is the beacon generation rate. It is a variable parameter.

###  IV-D Variable Parameters

There is a number of parameters which we vary directly or indirectly in our
performance evaluation: the number of drones (through varying $K$), the drone
density (through varying $K$ in variable-density settings), the network
diameter (through $K$ in fixed-density settings), the beaconing rate $\beta$,
or the inter-arrival times $\lambda$ for variable updates. Furthermore, we
consider different settings for some key VarDis parameters: whether or not the
summarization mechanism is enabled, the value of the repetition count
parameter repCnt, and the maximum number of variable summaries included in a
beacon (maxSumCnt). The variable parameters are summarized in Table II.

###  IV-E Main Performance Metrics

In all simulations reported below we will have one selected producer and one
selected consumer of a variable as described in Section IV-B. We measure the
reception reliability for the consumer node by evaluating the average
application layer sequence number gap between successively received variable
updates (recall that the variable itself includes a sequence number and
timestamp, see Section IV-B). In the absence of update losses, that average
gap would be exactly one, whereas a higher value indicates losses of updates.
Similarly, in the consumer node we measure the end-to-end update delay by
subtracting the timestamp in the variable from the timestamp of reception at
the consumer.

Error bars — if shown — represent the 95% confidence intervals of the result.
Unless otherwise mentioned, for each parameter combination we collected at
least 100,000 variable updates at the chosen consumer across a number of
replications which use different random number seeds. The in-simulation-time
and replication count were dependent on the variable update rate and the
computing resources available to us.

###  IV-F Flooding Protocol

As part of our experiments we compare the performance of VarDis against a
flooding protocol, in which each variable update triggers a separate flooding
operation. More specifically, an update for a variable as described in Section
IV-B is handed down to the flooding protocol, which encapsulates it in its own
packet. The flooding protocol header is 14 B long and includes its own
sequence number for duplicate detection, time-to-live (irrelevant for the
purposes of this paper), and source address fields. The flooding protocol then
puts the packet into a broadcast queue for transmission.

The flooding protocol has a separate worker process which constantly pulls
packets from this broadcast queue, waits a random backoff period, hands over
the packet to the MAC for transmission, and then starts over. Such a backoff
helps to reduce the number of collisions between forwarder nodes that have
received this update at the same time. The backoff period is drawn from an
exponential distribution with an average back-off period of 10 ms. The
flooding protocol also repeats transmissions, putting the last transmitted
packet back at the front of the queue until repCnt transmissions have
occurred.

When a station receives a broadcast, it checks if it is the source of the
packet, or whether it has seen the same packet or one with a more recent
sequence number before. If either of these conditions are met, it drops the
packet. Otherwise, the station passes the message up to the application and
adds the packet to the broadcast queue for further dissemination.

It should be noted that this flooding protocol is somewhat adapted to (and
optimized for) the concept of variables that we use in this paper. In
particular, a forwarder in a fully generic flooding protocol would only drop
received packets if they are exactly the same as packets that have been
forwarded recently (as determined from the node address of the sender and a
sequence number), and not drop it already when it has seen a more recent
sequence number. In other words, a fully generic flooding protocol would not
be able to tell “younger” from “older”, it would only test for equality. Note
that this adaptation is beneficial for the flooding protocol, as it reduces
the number of packets transmitted. Furthermore, fully generic flooding
protocols would not repeat the same packet repCnt times, as happens in VarDis.

##  V Results

In this section we discuss our results. We first consider fixed-density
networks. For these, we demonstrate the effect of summarization for a line
network, then study a grid network to perform a sensitivity analysis for key
DCP and VarDis parameters, and finally compare VarDis against a flooding
protocol. We then turn our attention to variable-density networks. First, we
consider again the case of the line network. To at least partially validate
our simulation model, we compare simulation results against numerical results
obtained from a discrete-time Markov chain model. We then evaluate grid
networks with a substantially higher traffic load wherein each node produces
and frequently updates a variable, and present results showing how often nodes
can update their variables subject to certain delay and reliability (average
sequence number gap) constraints.

###  V-A Fixed-Density Networks: The Effect of Summarization

We consider the impact of summarization in a fixed-density line network
(Section IV-B, in particular Figure 2). We consider per-hop packet error rates
of $P\in\left\\{20\,\%,50\,\%,80\,\%\right\\}$ and choose the link distance
$L_{P}$ accordingly
($L_{P}\in\left\\{263\,\mbox{m},273\,\mbox{m},280\,\mbox{m}\right\\}$). In all
cases, the chosen settings imply that each drone will only be able to receive
packets from immediate neighbours, the distance to two- or more-hop neighbours
is too large to receive any packets. The sole producer of a variable generates
variable updates every 5 s. We only recorded delay and gap size results for
the last drone in the chain, below referred to as the consumer.

We have varied the following parameters: the status of VarDis summarization
(enabled or disabled), the VarDis repetition count
($\mathtt{repCnt}\in\left\\{1,2,3\right\\}$), the number of nodes in the line
($K\in\left\\{3,4,\ldots,17\right\\}$), and the beaconing rate
($\beta\in\left\\{10\,\mbox{Hz},20\,\mbox{Hz}\right\\}$).

(a)

(b)

Figure 5: Average sequence number gap for per-hop PER $P=20\,\%$, with and
without summaries, varying repCnt and beaconing rate.

(a)

(b)

Figure 6: Average sequence number gap for per-hop PER $P=50\,\%$ and
$P=80\,\%$ with summaries, varying repCnt and beaconing rate.

In Figure 5 we show the average sequence number gap for a per-link packet
error rate of $P=20\,\%$, both with and without summarization, and for varying
beaconing rates and repCnt values. In Figure 6 we show the average sequence
number gap for $P=50\,\%$ and $P=80\,\%$, in both cases with summarization
being enabled, as the results without summarization are very poor. For the
lower per-link PER of $P=20\,\%$ the summarization mechanism achieves
practically loss-free propagation between producer and consumer with virtually
no difference between the different beaconing rates and repCnt values, even
over the maximum number of 16 hops. The results with summarization for the
higher per-link PER values of $P=50\,\%$ and $P=80\,\%$ (shown in Figure 6)
show that the best-performing variants are consistently those with the higher
beaconing rate of 20 Hz and at least two repetitions, followed by the variant
with 10 Hz rate and three repetitions. This is likely due to the fact that
with a higher beaconing rate the average time for an update to make the next
hop (either directly or through the update-request mechanism) becomes smaller,
and therefore the probability that an update reaches the consumer node within
the allotted five seconds of lifetime becomes higher — if an update takes
longer than five seconds to propagate, it might be “over-written” by the
following update and never reach the consumer. It is also interesting to note
that even for a per-hop PER of 80 % and $K=17$ nodes the average gap size for
20 Hz beaconing rate and three repetitions is below five. Furthermore,
consistently the variants with only one repetition behave the worst.

We now consider the case without summarization for $P=20$ % (compare Figure
5). The average sequence number gap becomes very large for just one repetition
($\mathtt{repCnt}=1$), being around 405 for the 10 Hz beaconing rate and 525
for the 20 Hz beaconing rate for $K=17$ nodes (likely a result of higher
beacon collision rates for $\beta=20$ Hz, see below), whereas adding a second
and third repetition brings the gap size down drastically (to around 4.8 for
two repetitions and both beaconing rates, and 1.6 for three repetitions and
both beaconing rates). A related finding emerges when comparing the simulation
results against a very simple and idealized probabilistic model of propagation
without summarization: with a per-link packet error rate of $P$, a number
$\mathtt{repCnt}$ of independent repetitions of an update per hop, and $K-1$
hops, and furthermore assuming independent and statistically identical links,
the probability that an update actually reaches the consumer can be expressed
as

$Q=\left(1-P^{\mathtt{repCnt}}\right)^{K-1},$ (1)

and the average sequence number gap in this model emerges as the expectation
$1/Q$ of a geometric random variable for the number of trials required until
(and including) a success. We find that the expected sequence number gap from
this model is noticeably smaller than the average sequence number gap observed
in our simulations. For example, for $\mathtt{repCnt}=2$, $K=17$ and $P=0.2$
the expected sequence number gap is $\approx 1.92$, while in our simulations
we have observed an average gap of around 4.8. This loss in reliability is
likely the result of beacon collisions (direct or hidden-terminal collisions),
which are known to be a significant issue for local broadcasts (see Section
II), and which the simple analytical model given in Equation (1) does not
account for.

A key conclusion is that the summarization mechanism is indispensable from a
reliability perspective, so in the remaining paper we only consider VarDis
with summarization enabled.

(a)

(b)

Figure 7: Average update delay for per-hop PER $P=20\,\%$, with and without
summaries, varying repCnt and beaconing rate.

(a)

(b)

Figure 8: Average update delay for per-hop PER $P=50\,\%$ and $P=80\,\%$ with
summaries, varying repCnt and beaconing rate.

In Figure 7 we show the average update delay (travel time between producer and
consumer for a successful update) for a per-hop PER of 20 %, with and without
summarization and for varying beacon rate and repCnt. In Figure 8 we show the
average update delay for per-hop PER values of 50 % and 80 %, using summaries,
again for varying beacon rate and repCnt. Unsurprisingly, in all cases the
average update delay increases linearly with the number of hops, and the
update delay for 20 Hz beaconing rate is overall smaller than for their 10 Hz
counterparts. Furthermore, for the same parameters, the delay results without
summaries will always be at least as good as for the case with summaries —
recall that for the update delay we only consider those updates that actually
reach the destination, and these tend to travel faster when the summary and
update-request mechanism are not used.

When summarization is enabled, in all cases the update delay for 20 Hz and
three repetitions is the smallest, followed by 20 Hz and two repetitions. This
suggests that receiving updates via repetitions is faster than having to
request them via the request-update mechanism. Note that for $P=50\,\%$ and
$P=80\,\%$ the update delay for 10 Hz beaconing rate and three repetitions is
better than for 20 Hz and one repetition. A likely explanation is that for
these high per-link PER values a node is more likely to have to resort to the
summary and update-request mechanism, compared to the case where just one
repetition is being used.

###  V-B Sensitivity Analysis: Method

We perform a sensitivity analysis on a fixed-density grid network (compare
Section IV-B) to investigate the relative importance of key BP and VarDis
parameters.

We first very briefly summarize the methodology used for the sensitivity
analysis, then we present our findings in Section V-C. The method adopted here
is the same as in [52], and our presentation of this method in this paper
follows the one given in [52] closely.

We apply the response surface methodology (RSM) [53], [54]. In this approach,
the dependency of a response variable (here: the update delay or the average
gap size) on a number of input factors is investigated using a regression
model. Each factor takes two different values, interpreted as minimum and
maximum values.

We have adopted a second-order polynomial regression model. In general, the
response variable $Y$ under consideration (average delay or average gap size)
is assumed to depend on the $k$ factors $x_{i}$ as follows:

$Y=\alpha_{0}+\sum_{i=1}^{k}\alpha_{i}x_{i}+\sum_{i=1}^{k}\sum_{j<i}\alpha_{i,j}x_{i}x_{j}$
(2)

For each combination of factor values
$\mathbf{x}=(x_{1},\ldots,x_{k})\in\left\\{-1,1\right\\}^{k}$ we simulate to
obtain a response value $y_{x_{1},\ldots,x_{k}}$, then we determine the
parameters $\alpha_{\nu}$ to minimize the least-squares error between the
regression model and the observed responses. The coefficients $\alpha_{0}$,
$\alpha_{i}$ and $\alpha_{i,j}$ are known as the intercept (it represents the
average value of all responses observed for the various parameter
combinations), linear, and mixed coefficients (or interactions), respectively.
It is common to encode the minimum and maximum values for each factor not
directly, but rather as $-1$ and $1$, so all factors enter with the same
magnitude. Units are ignored. After computing the regression coefficients
$\alpha_{\nu}$, we determine the _sum-of-squares-total_ (SST)

$SST=\sum_{\mathbf{x}\in\left\\{-1,1\right\\}^{k}}(y_{x}-\bar{y})^{2}$ (3)

representing the total amount of variation between the simulated response
values and the average response $\bar{y}$ (taken over all parameter
combinations), and the _sum-of-squares-errors_

$\displaystyle SSE=$
$\displaystyle\sum_{\mathbf{x}\in\left\\{-1,1\right\\}^{k}}\left(y_{x}-\left(\alpha_{0}+\sum_{i=1}^{k}\alpha_{i}x_{i}+\sum_{i=1}^{k}\sum_{j<i}\alpha_{i,j}x_{i}x_{j}\right)\right)^{2}$

representing the total error between regression model and observed responses.
Exploiting that

$SST=2^{k}(\alpha_{1}^{2}+\ldots+\alpha_{k}^{2}+\alpha_{2,1}^{2}+\alpha_{3,1}^{2}+\alpha_{3,2}^{2}+\ldots+\alpha_{k,k-1}^{2})$
(5)

holds, the relative impact of factor $k$ or any interaction $k,j$ is given as:

$\frac{2^{k}\alpha_{k}^{2}}{SST}\qquad,\qquad\frac{2^{k}\alpha_{k,j}^{2}}{SST}$
(6)

These values express the contribution of individual factors with respect to
the total observed variation.

A key quantity to assess the fit of the regression model is the coefficient of
determination or $R^{2}$ value, given by:

$R^{2}=\frac{SST-SSE}{SST}$ (7)

where higher values are better. This quantity is a relative measure of the
total variation that can be explained by the regression model.

###  V-C Fixed-Density Networks: Sensitivity Analysis Results

We perform this experiment using fixed-density $K\times K$ grid networks with
horizontal and vertical links of length $L_{P}$ having packet error rates of
$P\,\%$ (see Section IV-B, compare also Figure 4). Data can only move along
the horizontal and vertical links, as the diagonal links are long enough to
have a PER of nearly 100%. Each of the $K^{2}$ drones in the network produces
a variable with update inter-arrival times drawn from an exponential
distribution with an average inter-arrival time of $\lambda$, which is varied.
We have used the following three factors:

  * •

beaconing rate $\beta\in\left\\{10\,\mbox{Hz},20\,\mbox{Hz}\right\\}$,

  * •

repetition count $\mathtt{repCnt}\in\left\\{1,3\right\\}$, and

  * •

maximum number of summaries $\mathtt{maxSumCnt}\in\left\\{10,20\right\\}$.

We have varied all these factors in twelve different settings: the packet
error rates $P$ along vertical / horizontal links are chosen from
$P\in\left\\{10\,\%,20\,\%\right\\}$ (with the distances $L_{P}$ being set
accordingly), the grid size parameter $K$ is chosen from
$K\in\left\\{9,13\right\\}$, and for the average update inter-arrival time
$\lambda$ we have used
$\lambda\in\left\\{300\,\mbox{ms},1000\,\mbox{ms},3000\,\mbox{ms}\right\\}$.
For each of these twelve settings we have simulated all $2^{3}=8$ combinations
of the above factor values. For these simulations we use a larger maximum
beacon size of 300 B to account for the larger number of nodes.

$K$ | Link PER (%) | Upd. period (ms) |  $\mathbf{R^{2}}$ (%) | Average | Min. | Max. | % Contr. repCnt | % Contr. $\beta$ | % Contr. Interactions
---|---|---|---|---|---|---|---|---|---
9 | 10 | 300 | 100.0 | 2345.71 | 521.03 | 5222.15 | 64.85 | 26.17 | 8.96
9 | 10 | 1000 | 100.0 | 1156.78 | 388.2 | 3013.43 | 26.71 | 47.68 | 25.61
9 | 10 | 2000 | 99.99 | 545.79 | 341.97 | 767.84 |  | 98.41 | 1.22
9 | 20 | 300 | 100.0 | 2702.51 | 1283.88 | 4923.47 | 34.86 | 55.3 | 9.75
9 | 20 | 1000 | 100.0 | 1635.7 | 678.19 | 2949.04 | 3.58 | 87.15 | 8.98
9 | 20 | 2000 | 99.91 | 962.63 | 462.72 | 1596.73 | 5.86 | 92.37 | 0
13 | 10 | 300 | 100.0 | 6663.97 | 1305.74 | 13640.6 | 75.83 | 20.86 | 3.3
13 | 10 | 1000 | 100.0 | 5769.56 | 767.22 | 15619.07 | 57.21 | 25.64 | 17.14
13 | 10 | 2000 | 100.0 | 3169.2 | 615.35 | 10089.21 | 30.75 | 39.34 | 29.91
13 | 20 | 300 | 100.0 | 7416.38 | 3422.17 | 13379.16 | 52.91 | 39.46 | 7.6
13 | 20 | 1000 | 100.0 | 6305.68 | 2624.26 | 13540.75 | 29.27 | 52.8 | 17.89
13 | 20 | 2000 | 100.0 | 3880.74 | 1163.29 | 8247.26 | 8.8 | 74.3 | 16.79
TABLE III: Results of sensitivity analysis for update delay.

The sensitivity analysis results for the average update delay are presented in
Table III. We have included the $R^{2}$ values, the average or intercept
value, the minimum and maximum observed values, the percentage contributions
of the factors repCnt and $\beta$, and the combined contributions of the
interactions between factors. We have not included contribution results for
maxSumCnt, since neither the percentage contribution of its corresponding
factor nor of any interaction involving this factor exceed 1 %. Stated
differently: the maxSumCnt parameter has very limited impact. Furthermore, in
the table we have also omitted entries for contributions below the 1 %
threshold. The following points are noteworthy:

  * •

The chosen regression model (Equation (2)) represents the data very well, as
all $R^{2}$ values are (very close to) 100 %.

  * •

Generally, as the update period $\lambda$ increases, the contribution of
repCnt decreases and the contribution of $\beta$ becomes (often vastly)
dominant.

  * •

The average delay values decrease with increasing update period. We attribute
this to a reduced update load, in particular to the resulting ability of nodes
to include a higher fraction of updates currently scheduled for transmission
into the next beacon, therefore reducing their waiting times.

  * •

When investigating which values for the factors achieve the minimum delay
observations in the twelve settings, we find that in all cases the beaconing
rate has to be 20 Hz and the value of maxSumCnt has to be 10 (though the
difference to maxSumCnt=20 is very small). For the larger network ($K=13$), in
all but one case the best value of repCnt is 1, only for $P=20\,\%$ and
$\lambda=2\,\mbox{s}$ is it 3, but in that case the contribution of the repCnt
factor is quite small ($\approx 8.8\,\%$). For the smaller network, we find
that repCnt=1 gives minimal delay for $\lambda=0.3\,\mbox{s}$, and for
$\lambda=1\,\mbox{s}$ at $P=10\,\%$, in the other cases repCnt=3 is minimal.
However, again in these other cases the contribution of the repCnt parameter
is quite small. Hence, the overall recommendation (particularly for larger
deployments) is to generally pick higher beaconing rates while keeping repCnt
at its lowest value of 1.

$K$ | Link PER (%) | Upd. period (ms) |  $\mathbf{R^{2}}$ (%) | Average | Min. | Max. | % Contr. repCnt | % Contr. $\beta$ | % Contr. Interactions
---|---|---|---|---|---|---|---|---|---
9 | 10 | 300 | 100.0 | 2.57 | 1.62 | 3.98 | 36.51 | 55.72 | 7.73
9 | 10 | 1000 | 100.0 | 1.2 | 1.08 | 1.41 | 10.46 | 72.15 | 17.34
9 | 10 | 2000 | 99.96 | 1.06 | 1.03 | 1.08 | 8.1 | 91.71 | 0
9 | 20 | 300 | 99.93 | 4.33 | 2.78 | 6.31 | 9.14 | 89.3 | 0
9 | 20 | 1000 | 99.9 | 1.56 | 1.18 | 2.02 | 14.09 | 85.13 | 0
9 | 20 | 2000 | 100.0 | 1.16 | 1.05 | 1.31 | 31.99 | 63.65 | 4.3
13 | 10 | 300 | 100.0 | 4.82 | 2.81 | 7.61 | 40.92 | 53.45 | 5.58
13 | 10 | 1000 | 100.0 | 1.72 | 1.23 | 2.55 | 36.6 | 50.11 | 13.21
13 | 10 | 2000 | 100.0 | 1.19 | 1.05 | 1.45 | 15.74 | 60.37 | 23.83
13 | 20 | 300 | 99.82 | 8.84 | 5.41 | 14.51 | 27.68 | 65.11 | 5.38
13 | 20 | 1000 | 99.88 | 2.75 | 1.77 | 3.95 | 11.1 | 87.52 | 0
13 | 20 | 2000 | 99.9 | 1.56 | 1.14 | 2.05 | 15.78 | 83.43 | 0
TABLE IV: Results of sensitivity analysis for average gap size.

The corresponding sensitivity analysis results for the average sequence number
gap size between the bottom-right corner node and the top-left corner node are
presented in Table IV. Again, information about the maxSumCnt parameter has
been omitted, as it never contributes more than $1\,\%$ to the observed
variation of results. The following points are noteworthy:

  * •

Again the chosen regression model explains the variation very well, with
$R^{2}$ values very close to 100 %.

  * •

Generally, the larger the average update period $\lambda$, the smaller the
average gap size, which again can be attributed to a reduced update load,
faster forwarding of updates and therefore reduced chances of an update being
over-written by the next update of the same variable while in transit.

  * •

The contribution of the beaconing rate $\beta$ is dominant or even very
dominant over repCnt in all cases, with the dominance mostly being more
pronounced for more error-prone links ($P=20\,\%$).

  * •

When investigating the factor values giving the minimum average gap size in
the twelve settings, we find that in all cases the beacon rate is
$\beta=20\,\mbox{Hz}$, the maxSumCnt parameter is always 20 (but recall that
it is largely inconsequential), and the repCnt parameter is three in most
settings, except for $K=13$, $P=10\,\%$ and
$\beta\in\left\\{0.3\,\mbox{s},1\,\mbox{s}\right\\}$, and for $K=9$,
$P=10\,\%$ and $\beta=0.3\,\mbox{s}$.

Finally, to give an additional impression on how the results depend on the
average update period $\lambda$, we show in Figures 9 and 10 the effect of
varying $\lambda$ for different values of repCnt and $K$ on the update delay
and sequence number difference. Note that we only show results for the 10 Hz
beacon rate, as the 20 Hz values show the same trends. In scenarios with an
already high update load (e.g., short average update periods or large
networks), increasing the repCnt value further reduces VarDis’ performance in
terms of both the average sequence number difference and the update delay.
However, for small networks and longer average update periods,
$\mathtt{repCnt}=2$ performs slightly better on both update delay and sequence
number difference than both other options.

(a)

(b)

(c)

Figure 9: Average update delay for varying repCnt with
$\mathtt{maxSumCnt}=10,P=10\%$, and $\beta=10\,\mbox{Hz}$.

(a)

(b)

(c)

Figure 10: Average sequence number difference for varying repCnt with
$\mathtt{maxSumCnt}=10,P=10\%$, and $\beta=10\,\mbox{Hz}$.

###  V-D Fixed-Density Networks: Comparison Against Flooding Protocol

We compare the performance of VarDis to the flooding protocol described in
Section IV-F, in which each variable update triggers a flooding operation
(with each node repeating a packet repCnt times). A flooding protocol like
that can be seen as a natural competitor for VarDis. We again use a fixed-
density grid network. Based on the results of our sensitivity analysis
(Section V-C), we only compare against the $\beta=20\,\mbox{Hz}$ and
$\texttt{maxSumCnt}=10$ results.

We perform the comparison on a fixed-density grid network with
$K\in\left\\{5,7,9,11\right\\}$ nodes on each side of the grid (compare
Section IV-B), and having packet error rates
$P\in\left\\{10\,\%,20\,\%\right\\}$ on the horizontal and vertical links
(corresponding to distances
$L_{P}\in\left\\{255\,\mbox{m},263\,\mbox{m}\right\\}$). We furthermore
considered two different values for the update inter-arrival times, namely
$\lambda\in\left\\{200\,\mbox{ms},1000\,\mbox{ms}\right\\}$. We restrict the
comparison to a metric we have not used so far, namely the percentage of
variable updates issued on the chosen producer that are received by the chosen
consumer (see Section IV-B) – note that for both protocols any received update
at the consumer is guaranteed to be strictly new, i.e. it is not a duplicate
of an earlier received update. The reason for this choice of metric is that
the flooding protocol involves queues and we have found these queues to be
unstable (i.e. continuously growing over time) for most of the considered
scenarios, which prohibits the calculations of average delay or gap size
results.

(a)

(b)

(c)

(d)

Figure 11: Comparing percentage of received updates between the flooding
protocol and VarDis

The results are shown in Figure 11. It can be observed that VarDis
consistently outperforms the flooding protocol, except for the smallest
network ($K=5$), where both protocols perform on similar, nearly perfect
levels. For both protocols their performance decreases as the network size
(and therefore network load) increases. The difference can likely be explained
by VarDis remaining stable (the VarDis protocol does not include any queues),
where flooding does not. This is supported by noting that for the flooding
protocol the variants with $\mathtt{repCnt}=1$ outperform those with
$\mathtt{repCnt}\in\left\\{2,3\right\\}$ for all but the smallest network size
— note that higher $\mathtt{repCnt}$ values lead to higher packet load and
longer queues. When focusing on VarDis, it can be observed that the variant
with $\mathtt{repCnt}=2$ shows consistently optimal or close-to-optimal
performance, whereas the relative order of $\mathtt{repCnt}=1$ and
$\mathtt{repCnt}=3$ depends on the scenario, with $\mathtt{repCnt}=1$ having
the advantage for higher link packet error rates.

###  V-E Variable-Density Networks: Comparison against Numerical Results and
Effect of Inter-Beacon Times

We again explore the impact of VarDis parameters on the reliability and update
delays, but now for a variable-density line network, where drones may be able
to reach more than one neighbour in each direction.

We use the variable-density line network introduced in Section IV-B, and in
particular Figure 3. There are $K$ drones in total, placed equidistantly on a
line, with a fixed distance of $D=1,120\,\mbox{m}$ between the first and last
drones. The first drone in the chain produces the chain’s sole variable,
generating updates every 5 s. The final drone in the chain is the consumer for
which we present average delay results. We do not show results for the average
sequence number gap, as it is consistently extremely close to one for all
considered parameter combinations.

As we have a fixed chain total length with only one producer, in our
simulations we have only varied three parameters: the number of drones
($K\in\left\\{6,7,\ldots,18\right\\}$), the VarDis repetition count
($\mathtt{repCnt}\in\left\\{1,2,3\right\\}$), and the beaconing rate
($\beta\in\left\\{10\,\mbox{Hz},20\,\mbox{Hz}\right\\}$).

####  V-E1 Line Network: Simulation Results

In Figure 12 we show the average update delay. As the number of drones
increases, the update delay increases as well, before suddenly dropping to
below the initial update delay and then beginning to increase again. We
hypothesize that this is due to two inter-related effects. As $K$ increases,
each node changes from having just one neighbour (a neighbour being a node
reachable in a single hop with reasonably high packet success rate) to having
two neighbours, then three neighbours and so on. When a beacon transmitted by
a node carrying a variable update is received by $n$ other nodes further down
the chain, and each of these $n$ other nodes waits a random time $T_{i}$
before forwarding the variable update further, then the time before _any_ of
the $n$ downward nodes forwards the beacon is
$\min\left\\{T_{i}:i=1,\ldots,n\right\\}$. In an example where each forwarder
transmits beacons without collisions and with an exponentially distributed and
independent inter-arrival time distribution of rate parameter $\beta$, the
average forwarding delay would become $\frac{1}{n\beta}$. Furthermore, with
reference to the specific topology shown in Figure 3, there is a possibility
that this next forwarder is not the node closest to the transmitter, but a
node two or more hops closer to the destination, so that the packet makes
quicker geometric progress — this in fact happens with probability
$\frac{n-1}{n}$. In Figure 12, each “bump” corresponds to one further
forwarder coming into reach of a transmitting node.

The key conclusion is that VarDis benefits significantly from increased node
densities, speeding up delivery substantially.

(a)

(b)

Figure 12: Average update delay for varying repCnt and beaconing rate.

####  V-E2 Line Network: Numerical Results

To at least partially validate our simulation model, we compare simulation
results against numerical results gained from a time-homogeneous Discrete-Time
Markov chain (DTMC) model of the propagation of an update in a network, under
a simplified protocol model. From our DTMC model we can calculate the expected
number of “steps” required until all nodes have received an update, with one
step corresponding to a single beacon transmission by a uniformly chosen node.
We then multiply this expected number of steps with the expected time between
steps to calculate the expected time until all nodes have received an update.

We assume one single variable. For simplicity, we ignore beacon collisions.
There are $K$ nodes in the network, numbered from $1$ to $K$. One of the nodes
is selected to be the producer of the variable, e.g. node 1. The model allows
for nodes to be arranged in a general network, but in this study will only be
evaluated for a line network. Packet losses on the channel between nodes $i$
and $j$ happen independently of packet losses on other channels with
probability $q_{i,j}$ — we assume that $q_{i,i}=0$ for all nodes $i$. To
determine the $q_{i,j}$ for the variable-density line network investigated in
Section V-E1, we have used a step-wise approximation of the PER-versus-
distance function obtained from our simulator. We assume that beacons with and
without a variable update have the same length and therefore the same packet
loss probability. The model captures broadcasting effects, where a single
transmission can reach multiple receivers. For each slot one node is randomly
selected (with uniform distribution) to be the beacon transmitter,
independently of whether the node has the update or not. Once a node has
received the update, it will include it in all subsequent beacons (and not
just in repCnt beacons). With this simplification, the state space of the
model is
$\mathcal{S}=\left\\{0,1\right\\}^{K}\setminus\left\\{(0,\ldots,0)\right\\}$,
where a particular state $\mathbf{s}=(s_{1},\ldots,s_{K})$ records for each of
the nodes $i$ whether it actually has received the variable update ($s_{i}=1$)
or not ($s_{i}=0$). Note that we have excluded the uninteresting state
$(0,0,\ldots,0)$ in which no node has the update. To write down the state
transition probabilities of moving from state $\mathbf{s}\in\mathcal{S}$ to
state $\mathbf{t}\in\mathcal{S}$ from one slot to the next (after one beacon
transmission), the following notations are convenient:

  * •

Let $\mathbf{s}=(s_{1},\ldots,s_{K})\in\mathcal{S}$ and
$\mathbf{t}=(t_{1},\ldots,t_{K})\in\mathcal{S}$ be the source and target
state, such that $\mathbf{s}\leq\mathbf{t}$ (i.e. for all
$i\in\left\\{1,\ldots,K\right\\}$ it holds that $s_{i}\leq t_{i}$). If the
condition $\mathbf{s}\leq\mathbf{t}$ is not fulfilled, then there must be one
index $i$ such that $s_{i}=1$ and $t_{i}=0$ — but such a transition will be
impossible, as nodes never drop the update. Impossible transitions are
assigned zero probability.

  * •

$\mathcal{R}(\mathbf{s},\mathbf{t})=\left\\{i\in\left\\{1,\ldots,K\right\\}:s_{i}=0\wedge
t_{i}=1\right\\}$ is the set of nodes which did not have the update before the
transmission but receive it through the transmission.

  * •

$\mathcal{N}(\mathbf{s},\mathbf{t})=\left\\{i\in\left\\{1,\ldots,K\right\\}:s_{i}=0\wedge
t_{i}=0\right\\}$ is the set of nodes which did not have the update before the
transmission and do not have it after the transmission.

  * •

$\mathcal{T}(\mathbf{s})=\left\\{i\in\left\\{1,\ldots,K\right\\}:s_{i}=1\right\\}$
is the set of nodes that already have the update at the start of the slot, and
which are the desirable transmitters to help further dissemination.

With these notations, the state transition probabilities are shown in Figure
13.

$p_{\mathbf{s},\mathbf{t}}=\left\\{\begin{array}[]{r@{\quad:\quad}l}0&\mathbf{s}\nleq\mathbf{t}\\\
\frac{K-|\mathcal{T}(\mathbf{s})|}{K}+\sum_{i\in\mathcal{T}(\mathbf{s})}\left(\frac{1}{K}\cdot\prod_{k\in\mathcal{N}(\mathbf{s},\mathbf{t})}q_{i,k}\right)&\mathbf{s}=\mathbf{t}\\\
\sum_{i\in\mathcal{T}(\mathbf{s})}\frac{1}{K}\left(\cdot\prod_{j\in\mathcal{R}(\mathbf{s},\mathbf{t})}(1-q_{i,j})\right)\cdot\left(\prod_{k\in\mathcal{N}(\mathbf{s},\mathbf{t})}q_{i,k}\right)&\mathbf{s}\neq\mathbf{t},\mathbf{s}\leq\mathbf{t}\\\
\end{array}\right.$ Figure 13: Transition probabilities

The first case is the case of an invalid state transition
($\mathbf{s}\nleq\mathbf{t}$), to which we assign probability zero. In the
third case, we consider a valid transition from $\mathbf{s}$ to
$\mathbf{t}\neq\mathbf{s}$, in which the set
$\mathcal{R}(\mathbf{s},\mathbf{t})$ of nodes newly receiving the beacon
transmission is non-empty. To achieve this transition, it is required that one
of the nodes currently having the beacon is chosen, i.e. a node from
$\mathcal{T}(\mathbf{s})$. Each member of this set is chosen with probability
$\frac{1}{K}$. Assume that node $i\in\mathcal{T}(\mathbf{s})$ is chosen as
sender. Node $i$’s transmission then must be successfully received by all $j$
in the set $\mathcal{R}(\mathbf{s},\mathbf{t})$ of receivers (which happens
independently with probability $1-q_{i,j}$ for receiver $j$), and it must be
missed by any node $k$ in the set of non-receivers
$\mathcal{N}(\mathbf{s},\mathbf{t})$, which happens independently with
probability $q_{i,k}$. As the channels are assumed to be independent, the
reception and non-reception events at the different nodes are independent as
well. The second case (where no node receives a new update) is similar, but
includes the option of choosing a beacon transmitter that does not yet have
the update, which happens with probability
$\frac{K-|\mathcal{T}(\mathbf{s})|}{K}$, and also is simplified by noting that
$\mathcal{R}(\mathbf{s},\mathbf{t})=\emptyset$ when $\mathbf{s}=\mathbf{t}$.

This Markov chain has one absorbing state: the state $(1,1,\ldots,1)$, in
which every node has the update. The desired average number of steps for an
update to reach _all_ nodes in the network can be calculated as the expected
number of steps required to reach state $(1,1,\ldots,1)$, which in the
literature is referred to as the average hitting time for the target set
$\mathcal{A}=\left\\{(1,1,\ldots,1)\right\\}$ [55]. In general, when state
$\mathbf{s}$ is the starting state of the chain, the average number of steps
$k_{\mathbf{s}}$ required to reach the target set
$\mathcal{A}\subset\mathcal{S}$ is given as the solution of the following
linear system (see [55, Sec. 1.3]):

$k_{\mathbf{s}}=\left\\{\begin{array}[]{r@{\quad:\quad}l}0&\mathbf{s}\in\mathcal{A}\\\
1+\sum_{\mathbf{t}\notin\mathcal{A}}p_{\mathbf{s},\mathbf{t}}\cdot
k_{\mathbf{t}}&\mathbf{s}\notin\mathcal{A}\end{array}\right.$ (8)

For our given starting state $\mathbf{s}_{0}=(1,0,\ldots,0)$ (a state with
just one node at the end of the line having the update), we use the solution
of Equation (8) to numerically determine the average number
$k_{\mathbf{s}_{0}}$ of steps — beacon transmissions — required.131313It
should be noted that the reach of our model is quite limited, as the number of
states for $K$ nodes is $2^{K}$, and the numerical solution of the system
given by Equation (8) hits singularities for $K>15$ when calculating with
double precision.

Figure 14: Average update delay for simulation and numerical model, using
exponentially distributed beacon inter-arrival times

For the comparison of numerical and simulation results, we consider the case
of exponentially distributed (with rate parameter $\beta$) beacon inter-
transmission times at the individual nodes. The beacon arrivals at one node
thus form a Poisson process of rate $\beta$ [56], and the overall beacon
arrival process then is the superposition of these Poisson processes. From the
well-known properties of Poisson processes, this superposition is again a
Poisson process of arrival rate $K\lambda$, and beacons indeed arrive
independently at the nodes with a uniform distribution. The numerical and
simulation results with exponentially distributed beacon inter-arrival times
are shown in Figure 14. It can be observed that simulation and numerical
results show excellent agreement, which confirms our trust into the simulation
results.

Figure 15: Average update delay for exponentially distributed interarrival
times and periodic interarrival times with 10 % jiiter, for $\beta=10$
(simulation).

We finally consider the impact of the choice of the distribution of inter-
beacon generation times at the nodes. In Figure 15 we show simulation results
for the delay, for varying $K$, $\mathtt{repCnt}=3$ and $\beta=10$, for
exponentially distributed inter-arrival times with rate parameter $\beta$, and
for periodic inter-arrivals of average period $1/\beta$ and 10 % jitter. It
can be seen that the choice of the inter-beacon generation time distribution
makes a substantial difference. We illustrate the reason for the difference
for the case of $K=6$ and $\beta=10$ Hz. For this value of $K$, a node in our
line network can practically only reach the next node in the line, but not
others, and the update hence has to take 5 hops. First consider the case where
the beacon inter-arrival times are exponentially distributed with parameter
$\beta$. As mentioned above, the beacon arrivals at one node form an
independent Poisson process of rate $\beta$. Now suppose that an update
arrives at node 1. From the memoryless property of the exponential
distribution, the waiting time until the _next_ beacon transmission of node 1
is again exponentially distributed with parameter $\beta$. Hence, with
$\beta=10$ Hz it takes on average 0.1 s before the beacon transmission takes
place and node 2 receives the update. By the same argument, node 2 has to wait
on average 0.1 s before its own next beacon transmission, and so on. This
means that with exponential inter-arrival times the beacon will need 0.5 s on
average to reach the final node in five hops distance to the producer. In the
second case we consider that at each node beacons arrive periodically with 10
% random jitter (i.e. the beacon inter-arrival times are drawn uniformly from
$U\left[\frac{0.9}{\beta},\frac{1.1}{\beta}\right]$). The time between update
arrivals at the producer is comparatively long with five seconds, and after
this time (corresponding to 50 beacons on average), the beacon transmissions
between different nodes have become unsynchronized. As a result, once any node
$i$ other than the last one has received the update from its predecessor, it
will have to wait on average half a period (i.e. $\frac{1}{2\beta}$ s) before
$i$ transmits the update to the next node. This means for $\beta=10$ Hz that
the update takes on average 0.05 s per hop, for a total average time of 0.25
s, twice as fast as in the case of exponential inter-arrival times.

###  V-F Variable-Density Networks: The Grid Case and Network Capacity

This experiment focuses on exploring the impact of drone density on the
performance of VarDis, but now in larger variable-density grid networks, and
focusing more on capacity questions, i.e. asking what update load a network
can carry subject to certain delay or gap size constraints. The grid network
has $K\times{}K$ nodes with a fixed total sidelength $D=1,120\,\mbox{m}$ (cf.
Section IV-B). Each node produces its own variable and generates updates with
iid exponentially distributed inter-update times, with an average inter-update
time of $\lambda$ seconds. Both $K$ and $\lambda$ are varied.

We define two different metrics to measure the VarDis protocols’ performance.
The _reliability update capacity_ is the smallest per-node average update
period $\lambda$ (or the highest average per-node update rate
$\frac{1}{\lambda}$) at which the average sequence number gap is still less
than 1.5, whereas the _delay update capacity_ is the smallest per-node update
period $\lambda$ at which the average update delays is less than 250 ms.
Recall that every node in the $K\times K$ network produces a variable, but the
delay and gap size are measured at the reference consumer for the variable
produced by the reference producer (compare Section IV-B and Figure 4),
representing a “worst-case” scenario. The two ‘capacity’ metrics introduced
here characterize the maximum amount of aggregated variable update traffic
that the network can carry subject to (somewhat arbitrary) delay and gap size
constraints.

Motivated by our findings in the sensitivity analysis (Section V-C, obtained
for a fixed-density network), we have fixed the beaconing period to
$\beta=20\,\mbox{Hz}$, and maxRepCnt to 1. We also selected maxSumCnt to be 10
and used the larger maximum beacon size of 300 B. We simulated all networks
for
$\lambda\in\left\\{0.15\,\mbox{s},0.175\,\mbox{s},\ldots,0.475\,\mbox{s},0.5\,\mbox{s},0.75\,\mbox{s},1.0\,\mbox{s},1.5\,\mbox{s},2.0\,\mbox{s}\right\\}$.

The reliability and delay update capacity results are shown in Figure 16. As
the network size increases, both the update and reliability capacity decrease
as expected (the minimal average periods $\lambda$ that can be supported
become larger), resulting from an increased network load due to having more
producers. However, it is quite remarkable that in a network with $13\times
13=169$ nodes it is still possible for each node to send an update
approximately every 500 ms before the gap size exceeds 1.5, or for every node
to send updates with a period of $\approx 350\,\mbox{ms}$ before the delay
between two nodes on opposite corners exceeds 250 ms. These results confirm
that VarDis can provide quite satisfying delay and reliability performance in
larger and denser networks.

(a)

(b)

Figure 16: The capacity of the VarDis protocol as the network density
increases.

##  VI Conclusions

In this paper we have presented the DCP / VarDis approach to global data
dissemination in networks of closely collaborating drones, for example in a
swarm or formation. VarDis offers the abstraction of a variable, which is
replicated into the entire network. It operates by piggybacking modifying
operations on a variable onto regularly transmitted beacons, which for reasons
of collision avoidance between drones are required in many settings. We have
analysed the performance of VarDis in a range of conditions, and our results
show that VarDis provides a simple solution that can operate on top of a
variety of underlying wireless technologies and medium access control
protocols.

There is substantial potential for future research. For example, the current
version of the VarDis summary mechanism is simple and straightforward and can
be optimized, e.g. by reducing the rate at which summaries are transmitted for
variables that have not changed in a long time, or by aggregating summaries
further. Another option is to not re-broadcast a variable update when the same
update has already been broadcast by a certain number of neighbours (like for
example in the PSFQ protocol [57]). A further optimization is to adapt the
beaconing rate depending on the neighbourhood density and the observed rate at
which variable updates arrive, with the goal to maintain a balance between
delay, reliability and medium utilization. Aside from such operational
matters, the functionality of VarDis can be extended for example by allowing
more than one writer for a variable, liveness detection and failure takeover
for variables, or variables with limited scope which are only disseminated in
a $k$-hop neighbourhood. Finally, one of the next steps is the implementation
of DCP and VarDis on real hardware and their experimental evaluation. A key
part of this implementation will be the efficient choice of data formats and
reduction of packet overheads.

## References

  * [1] C. M. Gevaert, J. Suomalainen, J. Tang, and L. Kooistra, “Generation of Spectral-Temporal Response Surfaces by Combining Multispectral Satellite and Hyperspectral UAV Imagery for Precision Agriculture Applications,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 8, no. 6, pp. 3140 – 3146, Jun. 2015.
  * [2] F. Saffre, H. Hildmann, H. Karvonen, and T. Lind, “Monitoring and Cordoning Wildfires with an Autonomous Swarm of Unmanned Aerial Vehicles,” _MDPI Drones_ , vol. 6, 2022.
  * [3] D. Barry, A. Willig, and G. Woodward, “An Information-Motivated Exploration Agent to Locate Stationary Persons with Wireless Transmitters in Unknown Environments,” _MDPI Sensors_ , 2021.
  * [4] M. A. A. Careem, J. Gomez, D. Saha, and A. Dutta, “RFEye in the Sky,” _IEEE Transactions on Mobile Computing_ , 2021.
  * [5] G. Geraci, A. Garcia-Rodriquez, M. M. Azari, A. Lozano, M. Mezzavilla, S. Chatzinotas, Y. Chen, S. Rangan, and M. D. Renzo, “What Will the Future of UAV Cellular Communications Be? A Flight from 5G to 6G,” _IEEE Communications Surveys and Tutorials_ , vol. 24, no. 3, pp. 1304 – 1335, 2022\.
  * [6] A. Caruso, S. Chessa, S. Escolar, J. Barba, and J. C. Lopez, “Collection of Data With Drones in Precision Agriculture: Analytical Model and LoRa Case Study,” _IEEE Internet of Things Journal_ , vol. 8, no. 22, pp. 16 692 – 16 704, 2021.
  * [7] J. Li, B. Long, H. Wu, X. Hu, X. Wei, Z. Zhang, L. Chai, J. Xie, and H. Mei, “Rapid Evaluation Model of Endurance Performance and Its Application for Agricultural UAVs,” _MDPI Drones_ , vol. 6, 2022.
  * [8] R. J. ap Lewis Hartley, I. L. Henderson, and C. L. Jackson, “BVLOS Unmanned Aircraft Operations in Forest Environments,” _MDPI Drones_ , vol. 6, 2022\.
  * [9] A. Valencia-Arias, P. A. Rodriguez-Correa, J. C. Patino-Vanegas, M. Benjumea-Arias, J. D. L. Cruz-Vargas, and G. Moreno-Lopez, “Factors Associated with the Adoption of Drones for Product Delivery in the Context of the COVID-19 Pandemic in Medellin, Colombia,” _MDPI Drones_ , vol. 6, 2022\.
  * [10] A. Baltaci, E. Dinc, M. Ozger, A. Alabbasi, C. Cavdar, and D. Schupke, “A Survey of Wireless Networks for Future Aerial Communications (FACOM),” _IEEE Communications Surveys and Tutorials_ , vol. 23, no. 4, pp. 2833 – 2884, 2021.
  * [11] S. Chandrasekharan, K. Gomez, A. Al-Hourani, S. Kandeepan, T. Rasheed, L. Goratti, L. Reynaud, D. Grace, I. Bucaille, T. Wirth, and S. Allsopp, “Designing and Implementing Future Aerial Communication Networks,” _IEEE Communications Magazine_ , vol. 54, no. 5, pp. 26 – 34, May 2016.
  * [12] K. A. Ghamry, M. A. Kamel, and Y. Zhang, “Cooperative Forest Monitoring and Fire Detection Using a Team of UAVs–UGVs,” in _Proc. 2016 International Conference on Unmanned Aircraft Systems (ICUAS)_ , Arlington, USA, Jun. 2016.
  * [13] A. Guillen-Perez and M.-D. Cano, “Flying Ad Hoc Networks: A New Domain for Network Communications,” _Sensors_ , vol. 18, no. 10, 2018.
  * [14] H. Gao, W. Li, and H. Cai, “Fully Distributed Robust Formation Flying Control of Drones Swarm Based on Minimal Virtual Leader Information,” _MDPI Drones_ , vol. 6, 2022.
  * [15] K. Zhang, P. Chermprayong, F. Xiao, D. Tzoumanikas, B. Dams, S. Kay, B. B. Kocer, A. Burns, L. Orr, C. Choi, D. D. Darekar, W. Li, S. Hirschmann, V. Soana, S. A. Ngah, S. Sareh, A. Choubey, L. Margheri, V. M. Pawar, R. J. Ball, C. Williams, P. Shepherd, S. Leutenegger, R. Stuart-Smith, and M. Kovac, “Aerial additive manufacturing with multiple autonomous robots,” _Nature_ , vol. 609, Sep. 2022.
  * [16] S. Pell and A. Willig, “Using a Drone Formation with Sectored Antennas in Search-And-Rescue: Heuristics for Orienting Drones and Moving the Formation,” in _Proc. 2022 IEEE 33rd Annual International Symposium on Personal, Indoor and Vehicular Networking Conference (PIMRC)_ , Virtual conference, Sep. 2022.
  * [17] J. Werner, J. Wang, A. Hakkarainen, N. Gulati, D. Patron, D. Pfeil, K. Dandekar, D. Cabric, and M. Valkama, “Sectorized Antenna-based DoA Estimation and Localization: Advanced Algorithms and Measurements,” _IEEE Journal on Selected Areas in Communications_ , vol. 33, no. 11, pp. 2272 – 2286, Nov. 2015.
  * [18] K. Derr and M. Manic, “Extended Virtual Spring Mesh (EVSM): The Distributed Self-Organizing Mobile Ad Hoc Network for Area Exploration,” _IEEE Transactions on Industrial Electronics_ , vol. 58, no. 12, pp. 5424 – 5437, Dec. 2011.
  * [19] A. Trotta, L. Montecchiari, M. D. Felice, and L. Bonioni, “A GPS-Free Flocking Model for Aerial Mesh Deployments in Disaster-Recovery Scenarios,” _IEEE Access_ , vol. 8, pp. 91 558 – 91 573, May 2020.
  * [20] G. Pantelimon, K. Tepe, R. Carriveau, and S. Ahmed, “Survey of Multi-Agent Communication Strategies for Information Exchange and Mission Control of Drone Deployments,” _Journal of Intelligent and Robotic Systems_ , vol. 95, 2019.
  * [21] W. Chen, J. Liu, and H. Guo, “Achieving Robust and Efficient Consensus for Large-Scale Drone Swarm,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 12, pp. 15 867 – 15 879, Dec. 2020.
  * [22] M. Aggravi, C. Pacchierotti, and P. R. Giordano, “Connectivity-Maintenance Teleoperation of a UAV Fleet with Wearable Haptic Feedback,” _IEEE Transactions on Automation Science and Engineering_ , vol. 18, no. 3, pp. 1243 – 1262, Jul. 2021.
  * [23] D. Mishra and E. Natlizio, “A survey on cellular-connected UAVs: Design challenges, enabling 5G/B5G innovations, and experimental advancements,” _Computer Networks_ , Aug. 2021.
  * [24] O. S. Oubbati, M. Atiquzzaman, P. Lorenz, M. H. Tareque, and M. S. Hossain, “Routing in Flying Ad Hoc Networks: Survey, Constraints, and Future Challenge Perspectives,” _IEEE Access_ , vol. 7, pp. 81 057 – 81 105, 2019.
  * [25] A. Chikri, H. Touati, H. Snoussi, and F. Kamoun, “FANET: Communication, mobility models and security issues,” _Computer Networks_ , Aug. 2019.
  * [26] J. B. Kenney, “Dedicated Short-Range Communications (DSRC) Standards in the United States,” _Proceedings of the IEEE_ , vol. 99, no. 7, pp. 1162–1182, Jul. 2011.
  * [27] G. Karagiannis, O. Altintas, E. Ekici, G. Heijenk, B. Jarupan, K. Lin, and T. Weil, “Vehicular Networking: A Survey and Tutorial on Requirements, Architectures, Challenges, Standards and Solutions,” _IEEE Communications Surveys and Tutorials_ , vol. 13, no. 4, pp. 584–616, 2011.
  * [28] IEEE Computer Society, sponsored by the LAN/MAN Standards Committee, _IEEE Standard for Information technology – Telecommunications and Information Exchange between Systems – Local and Metropolitan Area Networks – Specific Requirements – Part 11: Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) Specifications_ , 2016.
  * [29] R. Fonseca, O. Gnawali, K. Jamieson, S. Kim, P. Levis, and A. Woo, “The collection tree protocol (ctp),” TEP 123, TinyOS Network Working Group, Aug. 2006\.
  * [30] O. Gnawali, R. Fonseca, K. Jamieson, D. Moss, and P. Levis, “Collection tree protocol,” Stanford Information Networks Group, Stanford University, Technical Report SING-09-01, 2009.
  * [31] Seyhan Ucar and Sinem Coleri Ergen and Oznur Ozkasap, “Multihop-Cluster-Based IEEE 802.11p and LTE Hybrid Architecture for VANET Safety Message Dissemination,” _IEEE Transactions on Vehicular Technology_ , vol. 65, no. 4, pp. 2621 – 2636, Apr. 2016.
  * [32] Wenjun Li and Huaiyu Dai, “Cluster-Based Distributed Consensus,” _IEEE Transactions on Wireless Communications_ , vol. 8, no. 1, pp. 28 – 31, Jan. 2009.
  * [33] D. Wu, D. I. Arkhipov, M. Kim, C. L. Talcott, A. C. Regan, J. A. McCann, and N. Venkatasubramanian, “ADDSEN: Adaptive Data Processing and Dissemination for Drone Swarms in Urban Sensing,” _IEEE Transactions on Computers_ , vol. 66, no. 2, pp. 183–198, Feb. 2017.
  * [34] H. ALshaer and E. Horlait, “An optimized adaptive broadcast scheme for Inter-vehicle communication,” in _Proc. IEEE 61st Vehicular Technology Conference_ , Stockholm, Sweden, Jun. 2005.
  * [35] Y.-C. Tseng, S.-Y. Ni, Y.-S. Chen, and J.-P. Sheu, “The Broadcast Storm Problem in a Mobile Ad Hoc Network,” _Wireless Networks_ , vol. 8, pp. 153–167, 2002.
  * [36] R. Bakhshi, L. Cloth, W. Fokkink, and B. R. Haverkort, “Mean-Field Analysis for the Evaluation of Gossip Protocols,” _ACM SIGMETRICS Performance Evaluation Review_ , vol. 36, pp. 31–39, 2008.
  * [37] A. Chaintreau, J.-Y. L. Boudec, and N. Ristanovic, “The Age of Gossip: Spatial Mean Field Regime,” in _Proc. ACM SIGMETRICS_ , Jun. 2009.
  * [38] T. C. Aysal, M. E. Yildiz, A. D. Sarwate, and A. Scaglione, “Broadcast Gossip Algorithms for Consensus,” _IEEE Transactions on Signal Processing_ , vol. 57, no. 7, pp. 2748 – 2761, Jul. 2009.
  * [39] G. Shi, B. Li, M. Johansson, and K. H. Johansson, “Finite-Time Convergent Gossiping,” _IEEE/ACM Transactions on Networking_ , vol. 24, no. 5, pp. 2782–2794, Oct. 2016.
  * [40] Sateeshkrishna Dhuli and Kumar Gaurav and Yatinda Nath Singh, “Convergence Analysis for Regular Wireless Consensus Networks,” _IEEE Sensors Journal_ , vol. 15, no. 8, pp. 4522 – 4531, Aug. 2015.
  * [41] Zhiyuan Jiang and Zixu Cao and Bhaskar Krishnamachari and Sheng Zhou and Zhisheng Niu, “SENATE: A Permissionless Byzantine Consensus Protocol in Wireless Networks for Real-Time Internet-of-Things Applications,” _IEEE Internet of Things Journal_ , vol. 7, no. 7, pp. 6576 – 6588, Jul. 2020\.
  * [42] R. Olfati-Saber and R. M. Murray, “Consensus problems in networks of agents with switching topology and time-delays,” _IEEE Transactions on Automatic Control_ , vol. 49, no. 9, pp. 1520–1533, Sep. 2004.
  * [43] G. Muniraju, C. Tepedelenlioglu, and A. Spanias, “Analysis and Design of Robust Max Consensus for Wireless Sensor Networks,” _IEEE Transactions on Signal and Information Processing over Networks_ , vol. 5, no. 4, pp. 779 – 791, Dec. 2019.
  * [44] F. Iutzeler, P. Ciblat, and J. Jakubowicz, “Analysis of Max-Consensus Algorithms in Wireless Channels,” _IEEE Transactions on Signal Processing_ , vol. 60, no. 11, pp. 6103 – 6107, Nov. 2012.
  * [45] Fabio Molinari and Navneet Agrawal and Slawomir Stanczak and Joerg Raisch, “Max-Consensus Over Fading Wireless Channels,” _IEEE Transactions on Control of Network Systems_ , vol. 8, no. 2, pp. 791 – 802, Jun. 2021.
  * [46] X. Shen, X. Cheng, R. Zhang, B. Jiao, and Y. Yang, “Distributed Congestion Control Approaches for the IEEE 802.11p Vehicular Networks,” _IEEE Intelligent Transportation Systems Magazine_ , vol. 5, no. 4, pp. 50 – 61, 2013\.
  * [47] J. Sahoo, E. H.-K. Wu, P. K. Sahu, and M. Gerla, “Congestion-Controlled-Coordinator-Based MAC for Safety-Critical Message Transmission in VANETs,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 14, no. 3, pp. 1423–1437, Sep. 2013.
  * [48] M. I. Hassan, H. L. Vu, and T. Sakurei, “Performance Analysis of the IEEE 802.11 MAC Protocol for DSRC Safety Applications,” _IEEE Transactions on Vehicular Technology_ , vol. 60, no. 8, pp. 3882–3896, Oct. 2011.
  * [49] Y. Yao, L. Rao, and X. Liu, “Performance and Reliability Analysis of IEEE 802.11p Safety Communication in a Highway Environment,” _IEEE Transactions on Vehicular Technology_ , vol. 62, no. 9, pp. 4198 – 4212, Nov. 2013\.
  * [50] P. T. Eugster, P. A. Felber, R. Guerraoui, and A.-M. Kermarrec, “The Many Faces of Publish/Subscribe,” _ACM Computing Surveys_ , vol. 35, no. 2, pp. 114–131, Jun. 2003.
  * [51] T. S. Rappaport, _Wireless Communications – Principles and Practice_. Upper Saddle River, NJ, USA: Prentice Hall, 2002.
  * [52] A. Moravejosharieh and A. Willig, “Mutual Interference in Large Populations of Co-Located IEEE 802.15.4 Body Sensor Networks – A Sensitivity Analysis,” _Elsevier Computer Communications_ , vol. 81, pp. 86 – 96, May 2016.
  * [53] R. Myers and D. Montgomery, _Response Surface Methodology_. John Wiley and Sons,Inc, 2002.
  * [54] R. Jain, _The Art of Computer Systems Performance Analysis – Techniques for Experimental Design, Measurement, Simulation, and Modeling_ , ser. Wiley Professional Computing. New York, Chichester: John Wiley and Sons, 1991.
  * [55] J. R. Norris, _Markov Chains_. Cambridge, UK: Cambridge University Press, 1997.
  * [56] S. Karlin and H. M. Taylor, _A First Course in Stochastic Processes_ , 2nd ed. San Diego, California: Academic Press, 1975.
  * [57] C.-Y. Wan, A. T. Campbell, and L. Krishnamurthy, “Pump-slowly, fetch-quickly (psfq): a reliable transport protocol for sensor networks,” _IEEE Journal on Selected Areas in Communications_ , vol. 23, no. 4, pp. 862–872, Apr. 2005.