Learning Decentralized Flocking Controllers with Spatio-Temporal Graph
  Neural Network
Recently a line of researches has delved the use of graph neural networks
(GNNs) for decentralized control in swarm robotics. However, it has been
observed that relying solely on the states of immediate neighbors is
insufficient to imitate a centralized control policy. To address this
limitation, prior studies proposed incorporating $L$-hop delayed states into
the computation. While this approach shows promise, it can lead to a lack of
consensus among distant flock members and the formation of small clusters,
consequently resulting in the failure of cohesive flocking behaviors. Instead,
our approach leverages spatiotemporal GNN, named STGNN that encompasses both
spatial and temporal expansions. The spatial expansion collects delayed states
from distant neighbors, while the temporal expansion incorporates previous
states from immediate neighbors. The broader and more comprehensive information
gathered from both expansions results in more effective and accurate
predictions. We develop an expert algorithm for controlling a swarm of robots
and employ imitation learning to train our decentralized STGNN model based on
the expert algorithm. We simulate the proposed STGNN approach in various
settings, demonstrating its decentralized capacity to emulate the global expert
algorithm. Further, we implemented our approach to achieve cohesive flocking,
leader following and obstacle avoidance by a group of Crazyflie drones. The
performance of STGNN underscores its potential as an effective and reliable
approach for achieving cohesive flocking, leader following and obstacle
avoidance tasks.
Siji Chen, Yanshen Sun, Peihan Li, Lifeng Zhou, Chang-Tien Lu
# Learning Decentralized Flocking Controllers with

###### Abstract

Recently a line of researches has delved the use of graph neural networks (GNNs) for decentralized control in swarm robotics. However, it has been observed that relying solely on the states of immediate neighbors is insufficient to imitate a centralized control policy. To address this limitation, prior studies proposed incorporating \(L\)-hop delayed states into the computation. While this approach shows promise, it can lead to a lack of consensus among distant flock members and the formation of small clusters, consequently resulting in the failure of cohesive flocking behaviors. Instead, our approach leverages spatiotemporal GNN, named STGNN that encompasses both spatial and temporal expansions. The spatial expansion collects delayed states from distant neighbors, while the temporal expansion incorporates previous states from immediate neighbors. The broader and more comprehensive information gathered from both expansions results in more effective and accurate predictions. We develop an expert algorithm for controlling a swarm of robots and employ imitation learning to train our decentralized STGNN model based on the expert algorithm. We simulate the proposed STGNN approach in various settings, demonstrating its decentralized capacity to emulate the global expert algorithm. Further, we implemented our approach to achieve cohesive flocking, leader following and obstacle avoidance by a group of Crazyflie drones. The performance of STGNN underscores its potential as an effective and reliable approach for achieving cohesive flocking, leader following and obstacle avoidance tasks.

## I Introduction

Flocking is a collective behavior observed in groups of animals or autonomous agents, such as birds, fish, or artificial robots, where individuals move together in a coordinated manner. In a flock, each robot follows simple rules based on local information to achieve a common group objective [1]. Multi-robot systems based on flocking models exhibit self-organization and goal-directed behaviors, making them suitable for various applications, including automated parallel delivery, sensor network design, and search and rescue operations [2]. Flocking is typically modeled as a consensus or alignment problem, aiming to ensure that all robots in the group eventually agree on their states [3]. Classical methods such as those proposed by Tanner [4] and Olfati-Saber [3], define rules and constraints governing the position, speed, and acceleration of the robots. However, these methods heavily rely on parameter tuning and are limited to predefined scenarios. In contrast, learning-based methods spontaneously explore complex patterns and adapt their parameters through training, providing more flexibility and adaptability compared to classical approaches.

There are primarily two research directions in learning-based methods. One approach focuses on imitation learning, as demonstrated by Tolstaya et al. [5], Kortvelesy et al. [6], Zhou et al. [7], and Lee et al. [8]. The other approach involves multi-robot deep reinforcement learning (MADRL), as explored in the studies of Yan et al. [9] and Xiao et al. [10]. MADRL is particularly useful when labels are unavailable, but it presents its own set of challenges, including the demand for an extensive volume of training data and limitations in generalizing to new and unencountered scenarios [11]. In this work, we choose to utilize imitation learning due to the availability of an expert policy that has proven to be effective for our task [4, 5, 12]. Recent research in this direction adopts a graph-based approach to represent robots and their interactions and leverages Graph Neural Networks (GNNs) [13, 14] for modeling and analyzing flock dynamics. This approach shows promise in addressing flocking tasks by harnessing the power of graph-based representations and neural networks. Specifically, studies such as Tolstaya et al. [5], Kortvelesy et al. [6], Zhou et al. [7], and Lee et al. [8] utilize a technique called "delayed state" to incorporate

Fig. 1: The comparison of the information gathered by DGNN (top) and TGNN (bottom) in the case of four aerial robots. DGNN propagates the states of robots 2 and 3 at \(t-1\) to robot 0 through robot 1, along with robot 1’s current state at \(t\). TGNN propagates the states at the previous step (\(t-1\)) along with the current state of the robot 1 at \(t\) to robot 0. STGNN combines information from both DGNN and TGNN, thus processing superior predictive power than either method alone.
the information from the \(l\)-step-before states of a robot's \(l\)-hop neighbors, where \(l=1,2,\cdots,L\)[5]. Henceforth, we identify this type of model as "delayed graph neural network" (DGNN). DGNN enables the learning of spatially extended representations in the local network. However, it overlooks the influence of a robot's historical states and the historical states of its neighbors, thereby neglecting the temporal sequence of flock movement. Diffidently, the spatial and temporal expansion of STGNN enables it to gather information from both spatial and temporal dimensions. We illustrate the distinct information collected by DGNN and TGNN, as well as their combined information gathered by STGNN in Figure 1. With this insight, we make the following primary contributions in this paper.

* **Design a STGNN-based imitation learning framework for decentralized flocking control.** STGNN enables effective information fusion by integrating delayed states from distant neighbors and previous states from immediate neighbors. To the best of our knowledge, we are the first to design a STGNN-based learning framework for multi-robot flocking with leader following and obstacle avoidance.
* **Develop a centralized expert algorithm for flocking, with leader following and obstacle avoidance.** Finding an expert algorithm is the key to success in imitation learning. We develop an expert algorithm that provides effective control over a large swarm of robots based on Tanner's [4] and Olfati-Saber's work [3]. To the best of our knowledge, we are the first to offer a complete global expert algorithm, capable of handling the flocking with both leader following and obstacle avoidance.
* **Conduct an extensive evaluations of STGNN.** We comprehensively evaluate STGNN by comparing it to DGNN and TGNN and testing STGNN with varying history horizons. The results demonstrate STGNN's effectiveness and superior performance in completing complex flocking tasks. Additionally, we implement STGNN into a group of Crazyllie drones to achieve flocking with obstacle avoidance.

## II Problem Formulation

Consider a collection of \(N\) robots that are identical and possess the same capabilities such as maximum acceleration \(U_{\text{max}}\), maximum velocity \(V_{\text{max}}\), and communication range \(R_{c}\). Each robot \(i\) can be uniquely identified by its position \(\mathbf{p}_{i}\) and its velocity \(\mathbf{v}_{i}\). The task for our learning model is to compute the control input \(\mathbf{u}_{i}\), i.e., acceleration for each robot, based on the current state by itself and its neighbors, represented by the position and velocity \((\mathbf{p},\mathbf{v})\).

In this paper, we aim to solve three problems as a whole: robot flocking, leader following, and flocking with obstacle avoidance. Each robot is required to avoid collisions with other robots, follow and maintain proximity to a virtual leader, and navigate around obstacles. Specifically, flocking is to maintain a consistent distance from other robots while synchronizing their movements. The leader following asks the robot swarm to track one or more leader(s) during flocking. We opt for a single virtual leader for the entire swarm, as the primary objective of flocking is to achieve consensus, and introducing multiple leaders could potentially violate this objective. It is important to note that the virtual leader differs from other robots in that it does not need to avoid collisions. Instead, it represents a predefined trajectory known to all robots. Finally, the obstacle avoidance requests the robot swarm to achieve the two tasks above while avoiding crashing into obstacles.

## III Methodology

In this research, we employ imitation learning to train STGNN. Specifically, we train an expert model whose outputs serve as labels for training our STGNN model. The expert assumes that the strategy provider possesses real-time information of all robots (i.e., centralized communication), which is impracticable in reality. In contrast, STGNN considered real-world scenarios, where each robot makes decisions by itself with local communication and delayed information (i.e., decentralized communication). Due to the limited information access in decentralized scenarios, STGNN may not generate strategies as effectively as the expert. Nevertheless, after being trained with historical and neighboring data and the output of the expert model, STGNN demonstrates the capability to generate strategies that approach expert-level optimization, despite the limited input data.

### _Problem Statement_

Denote the robot network at timestamp \(t\) as \(\mathcal{G}^{(t)}(\mathbf{V}^{(t)},\mathbf{E}^{(t)},\mathbf{X}^{(t)})\), where \(\mathbf{V}^{(t)}\) is the set of robots, \(\mathbf{E}^{(t)}\) is the set of direct communications between nearby robots, and \(\mathbf{X}^{(t)}\) is the set of states for the robots. For an arbitrary node \(V_{i}\), its neighborhood \(\mathcal{N}_{i}^{(t)}\) at timestamp \(t\) is composed of other robots \(V_{j}\) within its communication range \(R_{c}\), i.e., \(\{V_{j}\in\mathcal{N}_{i}^{(t)}|\forall V_{j},r_{i,j}^{(t)}<R_{c}\}\). \(r_{i,j}^{(t)}\) is the distance between \(V_{i}\) and \(V_{j}\) at \(t\). The problem can then be summarized as below:

\[f_{\text{expert}}(X_{i}^{(t)},\mathbf{V}^{(t)},\mathbf{E}^{(t)})\to \mathbf{u}^{(t+1)}, \tag{1}\] \[f_{\text{STGNN}}(X_{i}^{(t-L)}...X_{i}^{(t)},\mathcal{N}_{i}^{(t -L)}...\mathcal{N}_{i}^{(t)},\] (2) \[\mathbf{E}_{i,j;V_{j}\in\mathcal{N}_{i}^{(t-L)}...\mathcal{N}_{ i}^{(t)}},\theta\}\to\mathbf{u}^{(t+1)},\] \[\theta^{*}=\operatorname*{arg\,min}_{\theta}(\sum_{V_{i}\in \mathbf{V}}(\mathbf{u}^{(t+1)}-\mathbf{u}^{(t+1)})^{2}), \tag{3}\]

where \(L\) is the number of historical states used in STGNN, \(\theta\) is the set of trainable parameters of STGNN, and \(\theta^{*}\) is the optimized parameter set for STGNN.

### _Expert algorithm_

Previous researchers recognized the challenge of communication delays in the early stages of robot flocking studies, leading them to focus on decentralized scenarios. Consequently, only a limited number of prior algorithms have been designed to address centralized scenarios in the context of robot flocking problems. Therefore, to formulate an expert
model for targeting our tasks and considering centralized scenarios, we propose a novel centralized model to generate labels for STGNN training.

As introduced in Section II, an expert algorithm should cover three crucial aspects--flocking, the leader following, and obstacle avoidance. In this case, the algorithm should be constrained by both the distances between robots and the distances between robots and obstacles. Similar to [4], the regularized update of speed is computed jointly from the collision avoidance potential and the velocity agreement. The collision avoidance potential ensures that the distances between robots exceed a predefined threshold, while the velocity agreement ensures that robots maintain consistent behavior in relation to the other robots.

Specifically, consider there exists a control \(u_{i}\) of robot \(i\) as the input of the expert algorithm. The algorithm then generates an update \(u_{i}\) following specific rules. \(u_{i}\) can be defined by the combination of these components as in Equation 4. In Equation 4, \(c_{\alpha}\), \(c_{\beta}\), and \(c_{\gamma}\) are positive weighting parameters. The \(\alpha\)- term specifies collision avoidance and velocity alignment among the robots, while the \(\beta\)- term specifies collision avoidance and velocity alignment between robots to obstacles. The \(\gamma\) term is peer-to-peer guidance from the virtual leader to each robot \(i\).

\[\mathbf{u}_{i}=c_{\alpha}\mathbf{u}_{i}^{\alpha}+c_{\beta}\mathbf{u}_{i}^{ \beta}+c_{\gamma}\mathbf{u}_{i}^{\gamma}. \tag{4}\]

The collision avoidance potential \(U\) increases significantly as the distance \(r_{i,j}\) between two robots decreases. This increment is governed by a reciprocal function that dominates when \(r_{i,j}\) approaches zero (Eq. 5). Consequently, \(U_{i,j}\) captures the fact that beyond a certain distance threshold (e.g., communication range \(R_{c}\)), no direct interaction exists between robots in terms of collision avoidance. The velocity agreement is the velocity difference between robot \(i\) and all the other robots (the first term in Eq. 6). The resulting control input \(u_{i}^{\alpha}\) (as defined in Eq. 6) is a centralized solution that takes into account the velocity mismatch among all robots and the local collision potential. Note that the collision avoidance implemented by Equation 5 does not guarantee a minimal distance between robots. Our results show that the minimal distance decreases when the number of robots increases.

\[U_{i,j}=\frac{1}{r_{i,j}^{2}}+\texttt{log}||r_{i,j}||^{2},||r_{i,j}||\leq R_{c}. \tag{5}\]

\[\mathbf{u}_{i}^{\alpha}=-\sum_{j=1}^{N}(\mathbf{v}_{i}-\mathbf{v}_{j})-\sum_{j =1}^{N}(\nabla_{r_{i,j}}U_{i,j}). \tag{6}\]

To incorporate obstacle avoidance in the model, we follow Olfati-Saber [3]'s work by introducing an imaginary robot, i.e., \(\beta\)-robot, which is defined by the projection of a robot \(i\) on the \(k\)-th obstacle \(O^{k}\) within its communication distance \(R_{c}\), to assist with the obstacle avoidance task. In practice, this can be achieved by a robot, equipped with sensors, measuring the relative position and velocity between the closest point on an obstacle and itself [3]. The control input \(\mathbf{u}_{i}^{\beta}\) follows the flocking control (Eq. 6) while focusing on the potential between robot \(i\) and its projection on obstacles. Let \(\mathbf{p}_{k}^{\delta}\) denote the position of the \(k\)-th obstacle. Then the position and velocity of the \(\beta\)-robot, created by projecting the \(i\)-th robot on the \(k\)-th obstacle, can be calculated by Equation 7 and Equation 8, respectively.

\[\mathbf{p}_{i,k}=\mu*\mathbf{p}_{i}+(1-\mu)\mathbf{p}_{k}^{o},\;\mu=\frac{r_{k }}{||\mathbf{p}_{i}-\mathbf{p}_{k}^{o}||}. \tag{7}\]

\[\mathbf{v}_{i,k}=\mu P\mathbf{v}_{i},\;P=I-\mathbf{a}_{k}\mathbf{a}_{k}^{T},\; a_{k}=\frac{(\mathbf{p}_{i}-\mathbf{p}_{k}^{o})}{||\mathbf{p}_{i}-\mathbf{p}_{k}^{o}||}. \tag{8}\]

Define the state of the virtual leader by its position \(p^{r}\) and velocity \(v^{r}\). The leader following control \(u^{r}\) can be defined in Equation 9, where \(c_{1}\), \(c_{2}\) are positive weighting parameters.

\[\mathbf{u}_{i}^{\gamma}=-c_{1}(\mathbf{p}_{i}-\mathbf{p}^{r})-c_{2}(\mathbf{v }_{i}-\mathbf{v}^{r}),c_{1}>0,c_{2}=\sqrt{c_{1}}. \tag{9}\]

### _STGNN-based imitation learning_

In this section, we present an overview of the STGNN-based learning model. We begin by describing the strategy used to construct the model input. Then we discuss the architecture of the STGNN model, which involves two levels of state expansion.

Local observationWe develop a decentralized solution that only requires local observations of each robot. Inspired by Tolstaya [5], we define the local state of \(i\)-th robot in Equation 10, which consists of the aggregation from neighboring robots (\(\alpha\) term), local observation of obstacles (\(\beta\) term), and local observation of the virtual leader (\(\gamma\) term). The local observation of obstacles is defined as \(k\in\mathcal{N}\) with \(r_{i,k}<R_{c}\). \(r_{i,k}\) is the distance between robot \(i\) and \(\beta\)-robot, the projection of robot \(i\) on the \(k\)-th obstacle.

\[X_{i}^{\alpha,\beta,\gamma}=[[\sum_{j\in\mathcal{N}}X_{i,j}^{\alpha}];[\sum_{k \in\mathcal{N}}X_{i,k}^{\beta}];[X_{i}^{\gamma}]]. \tag{10}\]

Instead of directly using the position \(\mathbf{p}_{i}\) and velocity \(\mathbf{v}_{i}\) of robot \(i\) as input for our model, we adopt the relative state to be consistent with the expert algorithm. We define the relative state for robot \(i\) to robot \(j\) as \(X_{i,j}\) (Eq. 11). Then we aggregate the local peer-to-peer relative states into a local state \(X_{i}\), which includes \(X_{i}^{\alpha}\) (Eq. 12) and \(X_{i}^{\beta}\) (Eq. 13).

\[X_{i,j}=[\mathbf{v}_{i}-\mathbf{v}_{j},\frac{\mathbf{p}_{i}-\mathbf{p}_{j}}{r_{ i,j}},\frac{\mathbf{p}_{i}-\mathbf{p}_{j}}{r_{i,j}^{2}}]. \tag{11}\]

\[X_{i}^{\alpha}=\frac{1}{\mathcal{N}}\sum_{j\in\mathcal{N}}X_{i,j}^{\alpha},\;j \in\mathcal{N}\;\text{if }r_{i,j}<R_{c}. \tag{12}\]

\[X_{i}^{\beta}=\sum_{k\in\mathcal{N}}X_{i,j}^{\beta},\;k\in\mathcal{N}\;\text{if }r_{i,o}<R_{c}. \tag{13}\]

Notably, our model only requires local state information defined by Equation 11, which makes our approach decentralized.
StgnnIn order to mimic the control input \(\mathbf{u}\) generated by the expert algorithm that uses the global information, we implement two levels of expansions based on local observation as illustrated in Figure 2. The first level is spatial expansion, aiming at extracting information from additional robots, while the second level is temporal expansion, which centers on local state evolution. For each expansion, we extract information on \(L\) timestamps. For the timestamp \(t\), consider an arbitrary node \(V_{i}\) with state \(X_{i}^{(t)}\) and \(1\)-hop neighborhood \(\mathcal{N}_{i}^{(t)}\). A node \(V_{j}\) is \(V_{i}\)'s \(l\)-hop neighbor if it is a \(1\)-hop neighbor of at least one \(V_{i}\)'s \(l-1\)-hop neighbor. Note that even though we use "\(l\)-hop neighbor" to account for state delays resulting from spatial distances, the state of one hop further is the same as one timestamp earlier in our scenario. In other words, the state of a \(l\)-hop neighbor \(j\), \(X_{j}^{(l)}\), is equivalent to \(X_{j}^{(t-l+1)}\).

In spatial expansion, we merge each node's current state \(X_{i}^{(t)}\) to its \(l\)-hop delayed neighbors, where \(l=1,2,...,L\). Specifically, we first aggregate the delayed states for each hop, as indicated in Equation 14. Then, \(X_{i}^{(t)}\) is updated with its neighborhood information as in Equation 15.

\[{X^{\prime}}_{i\_s}^{(l)}=\sum_{j\in\mathcal{N}_{i}^{(t)}}X_{j}^{(l)},l=1,2,...,L. \tag{14}\]

\[H_{i\_s}^{(t)}=[X_{i}^{(t)}||_{l=1,...,L}{X^{\prime}}_{i}^{(l)}]. \tag{15}\]

The same method is applied to other timestamps \(l=t-L,...,t-1\), but with \(t-l+1\)-hop neighbors. Then, \(H_{i\_s}=\{H_{i\_s}^{(l)},l=t-L,...,t\}\) are fed into a transformer [15] for temporal-wise information fusion. The transformer takes three copies of \(H_{i\_s}\) and considers them as query, key, and value separately. The query and key are used to compute relations between different timestamps and the value is used to generate temporal-wise fused embeddings:

\[{H^{\prime}}_{i\_s}=\sigma(\frac{WqH_{i\_s}(WkH_{i\_s})^{T}}{\sqrt{d}})WvH_{i\_s} \tag{16}\]

where \(\sigma\) is a softmax function and \(d\) is the number of features of \(H_{i\_s}\).

The temporal expansion considers the temporal evolution pattern for local spatial state fusion (i.e., considers only \(1\)-hop neighborhood). Specifically, the spatial data fusion method is shown in Equations 17 and 18, and the temporal feature fusion is achieved by a transformer as described in Equation 16. Consider the output of the temporal expansion module as \({H^{\prime}}_{i\_t}\), the concatenation of \({H^{\prime}}_{i\_s}\) and \({H^{\prime}}_{i\_t}\) is fed to a feed-forward network to generate the final flocking controls for the robots.

\[{X^{\prime}}_{i\_t}=\sum_{j\in\mathcal{N}_{i}^{(1)}}X_{j}^{(l=1)}. \tag{17}\]

\[H_{i\_t}=[X_{i}^{(t)}||{X^{\prime}}_{i}^{(l=1)}]. \tag{18}\]

## IV Simulation and experiment

In this section, we discuss the settings of the STGNN-based learning model, evaluation metrics, simulation results, and a real robot experiment. A video of our simulation and experiment is available online. 1

Footnote 1: [https://youtu.be/Hb70fe3Ivva](https://youtu.be/Hb70fe3Ivva)

### _Settings for STGNN-based learning model_

Local observationThe local observation (Eq. 10) denoted as \(X\in\mathbb{R}^{18}\), is passed through a two-layer multi-layer perceptron (MLP) with a hidden size of 128 to extract the local feature embedding \(H\in\mathbb{R}^{128}\) prior to undergoing spatial and temporal expansion. Each \(X\) from the historical data follows the same process.

STGNN spatial expansionA single SAGEConv layer with a hidden size of 128 is used to aggregate local neighbor information. Then the \(l\)-hop neighbor information is obtained using Equations 14 and 15. The result is a sequence of \(L\) delayed states. A transformer consisting of two encoder layers, each with a feedforward size of 16 and a head size of 4, is used to extract the spatial expansion results.

STGNN temporal expansionFor each local features from the history \(L\), a single SAGEConv layer with hidden size of 128 is used to aggregate neighbor information (Eq. 15). A transformer consisting of 2 encoder layers, each with a feedforward size of 16 and a head size of 4, is used to extract the temporal expansion result.

Action generationThe last output from both spatial and temporal expansions is concatenated to get the fusion embedding \(H\in\mathbb{R}^{256}\). Subsequently, this output undergoes a two-layer MLP to generate the predicted control signal \(\mathbf{\hat{u}}\). The ground truth of the training data set is derived by the expert algorithm (Sec. III-B). L2 loss with the Adam optimizer is used to train the model.

We trained the STGNN model with \(N=20\), which includes three spherical obstacles and one virtual leader. The size of the swarm is chosen to ensure that spatial expansion

Fig. 2: STGNN spatial and temporal expansion module with \(L=3\). Spatial expansion and temporal expansion operate in parallel. Both spatial and temporal expanded states are concatenated and passed to the next layer of the neural network to predict the control input.
could encompass a significant number of robots. The virtual leader goes through the obstacles but the robots must avoid collision with obstacles while following the virtual leader. The virtual leader moves continuously at a constant speed of 1m/s along the x-axis. The communication range is \(R_{c}=1\) m, and the sampling period is \(T_{s}=0.01\) s. The initial robot positions are randomized, following a uniform distribution within the range of \([0,0.5R_{c}\sqrt{N}]\). Initial velocity is \(0\) m/s. The maximum allowed acceleration is \(U_{\max}=10\) m/s\({}^{2}\) and the maximum velocity allowed is \(V_{\max}=10\) m/s on each axis. The safety distance is 0.15 m. If \(r_{i,j}\) falls below 0.15 m, the experiment is terminated early. To ensure a valid initial configuration, \(r_{i,j}\) must be greater than 0.15 m. The training episode has 1200 steps to allow all robots to pass obstacles and form flocking behavior on the other side. One trajectory of the expert algorithm which is used in training is shown in Figure 3.

We implement our model using PyTorch and the OpenAI gym framework in Python3.9. The server we use has Intel(R) Xeon(R) W-2133 CPU @ 3.60GHz, NVIDIA QUADRO P5000 GPU, and 32 GB RAM. We conduct a comprehensive evaluation of our proposed method, STGNN, with \(L\) set to \(1\), \(2\), \(3\). The compared algorithms include DGNN, TGNN, and Olfati-Saber's decentralized flocking algorithm denoted as Saber [3].

### _Metrics_

1. Completion Rate (\(C\%\)): the rate of successfully completed episodes. The episode can be terminated early if any robot hits obstacles or \(r_{i,j}\) falls below 0.15 m. _The higher value is better._
2. Mean Absolute Error (MAE): the mean absolute error between the expert control \(\mathbf{u}\) and model prediction \(\hat{\mathbf{u}}\). _The lower value is better._
3. Velocity alignment (\(V\)): the velocity variance of the swarm at the end of the episode. All robots should be velocity aligned at the end of the episode thus _the lower value is better._
4. Distance to the leader (\(\tau\), Eq. 19): the mean distance from any robot to the leader is an auxiliary measure. A large value indicates robots deviate from the leader and move in different direction, resulting in the failure of swarm formation. However a small value indicates a higher risk of collision within the swarm. _Close to the expert algorithm is better._ \[\tau=\frac{1}{nT}\sum_{i=1}^{n}\sum_{t=1}^{T}\tau_{i}^{t}.\] (19)

### _Experiment results_

We train one model for each setting described in Section IV-A. The training consists of 200 epochs with an initial learning rate of 1e-3. We implement early stopping and exponential learning rate decay to prevent overfitting.

Evaluation on swarm size of 20In the first set of experiments, the trained models are tested in the same environment as the training environment, as described in Section IV-A. The only difference lies in the initial positions of the robots, which vary due to random initialization. During the testing phase, the swarm's next state is determined by the model's prediction \(\hat{\mathbf{u}}\). If an episode is terminated early due to collision, the failed episode's metric (MAE, \(V\), \(\tau\)) are not included in the aggregation for reporting. Each model runs over 20 trials and the mean and standard deviation are reported in Table I. For STGNN L1, L2 and L3 models, the results demonstrate consistently improvement as spatial and temporal expansion increases. By increasing L from 1 to 3 in both spatial and temporal expansion, the completion rate increases from 0.85 to 1.0, the MAE decreases from 3.03 to 1.71, and the velocity variance also decreases from 0.14 to 0.01. The distance to the leader does not consistently decrease, but as explained in Section IV-B, a smaller value indicates a higher risk of collision, so both values of 1.03 and 1.06 are acceptable. When considering DGNN L3 and TGNN L3, both models exhibit improved performance compared to STGNN L1, which uses only local neighbor information. Furthermore, STGNN L3, which is the combination of DGNN L3 and TGNN L3, leads to further performance enhancement. Olfati-Saber's decentralized solution [3] achieves a similar success rate as STGNN L1, as both models utilize only local information. However, the Saber algorithm differs from our expert algorithm, and its formations tend to have larger minimal distances, resulting in a larger \(\tau\), which is not directly comparable to the expert algorithm.

In Figure 3, a test trajectory of the expert algorithm is plotted on the top and the test trajectory of STGNN L3 is plotted at the bottom. The plots illustrate that STGNN L3 possesses the capability to mimic the expert algorithm's control input \(\mathbf{u}\) and generate similar flocking trajectories for the robots.

Evaluation on varying swarm sizesIn the second set of experiments, we examine our model's transferability across various swarm sizes, specifically for \(N=\) 30, 40, and 50. We directly employ previously trained models from larger swarm sizes and conduct 20 trials for each scenario. The mean and standard deviation for each model and swarm size are reported in Table II. To facilitate swarm formation after obstacle avoidance, we increased the maximum episode steps from 1200 to 1500, 1800, and 2100 for swarm sizes \(N\) = 30, 40, and 50, respectively.

STGNN L3 successfully achieves flocking with leader
following and obstacle avoidance through all testing cases and attains the lowest MAE compared to STGNN models with shorter history. Velocity alignment is achieved in both STGNN L2 and STGNN L3. The large \(\tau\) in STGNN L1 indicates the failure in swarm formation. Our proposed expert algorithm consistently performs well in terms of success rate, velocity alignment, and distance to target, which demonstrates its suitability as the ground truth. Furthermore, the performance of STGNN L3 underscores its ability to provide accurate estimations of the global control inputs using only local information.

### _Real robot experiment_

We further demonstrate the effectiveness of STGNN by implementing it to achieve flocking behaviors of a group of Bitcraze Crazyllie 2.1 drones [16]. The drones are controlled using the Crazyswarm platform [17], which is based on The Robot Operating System (ROS) [18] and allows Crazyllie drones to fly as a swarm in a tight and synchronized formation. The positions of the drones are obtained using the LightHouse positioning system. The system utilizes the SteamVR base stations together with the positioning deck on the drone to estimate the position of the drones. As shown in Figure 4, the six drones start with random locations (top), navigate through and avoid obstacles (middle), and form a flock on the other side of the obstacle (right). Please refer to the online video for a more detailed real robot experiment with 4, 5, and 6 drones (see footnote 1).

## V Conclusion and Future Work

We demonstrate the effectiveness of STGNN as a decentralized solution for flocking with the leader following and obstacle avoidance tasks. STGNN overcomes the limitations of relying solely on local information by integrating prediction capabilities into the model, enabling it to capture and respond to global swarm dynamics. Our STGNN-based learning model consistently outperforms the existing decentralized algorithm introduced by Olfati-Saber. Moreover, the performance of the STGNN model improves as \(L\) increases. Furthermore, STGNN outperforms spatial-only models, demonstrating its ability to utilize both spatial and temporal information for enhanced flocking control. The design is flexible in terms of \(L\)-hop spatial and temporal expansion, which enables seamless adaptation to various swarm sizes and history lengths. In the future, we plan to investigate the capabilities of STGNN [19, 20, 21] in a broader

Fig. 4: The plots from top to bottom show an experiment of six drones starting from random locations, avoiding an obstacle, and achieving flocking. More experiments can be found in the video attachment.

Fig. 3: Simulation result of \(N=20\). Top: the expert model drives 20 robots. Bottom: STGNN with \(L3\) drives 20 robots. The two models are tested in the same environment with the same initialization.
range of multi-agent tasks, including target tracking [7, 22], path planning [23, 24] and coverage and exploration [25, 26].
