# Deep Learning-Based Vehicle Speed Prediction for Ecological Adaptive Cruise
Control in Urban and Highway Scenarios

###### Abstract

In a typical car-following scenario, target vehicle speed fluctuations act as
an external disturbance to the host vehicle and in turn affect its energy
consumption. To control a host vehicle in an energy-efficient manner using
model predictive control (MPC), and moreover, enhance the performance of an
ecological adaptive cruise control (EACC) strategy, forecasting the future
velocities of a target vehicle is essential. For this purpose, a deep
recurrent neural network-based vehicle speed prediction using long-short term
memory (LSTM) and gated recurrent units (GRU) is studied in this work. Besides
these, the physics-based constant velocity (CV) and constant acceleration (CA)
models are discussed. The sequential time series data for training (e.g. speed
trajectories of the target and its preceding vehicles obtained through
vehicle-to-vehicle (V2V) communication, road speed limits, traffic light
current and future phases collected using vehicle-to-infrastructure (V2I)
communication) is gathered from both urban and highway networks created in the
microscopic traffic simulator SUMO. The proposed speed prediction models are
evaluated for long-term predictions (up to $10\,\mathrm{s}$) of target vehicle
future velocities. Moreover, the results revealed that the LSTM-based speed
predictor outperformed other models in terms of achieving better prediction
accuracy on unseen test datasets, and thereby showcasing better generalization
ability. Furthermore, the performance of EACC-equipped host car on the
predicted velocities is evaluated, and its energy-saving benefits for
different prediction horizons are presented.

###### keywords:

Adaptive cruise control, Velocity prediction, Car-following, Recurrent neural
networks, Model predictive control, Intelligent transportation systems, V2V,
V2I

††thanks: This work is funded by the German Ministry for Education and
Research (BMBF) and partially supported by the Center of Commercial Vehicle
Technology (Zentrum für Nutzfahrzeugtechnologie, ZNT) at the University of
Kaiserslautern.

Sai Krishna Chada* Daniel Görges* Achim Ebert** Roman Teutsch***

* Institute of Electromobility ** Human Computer Interaction Group *** Institute for Mechanical and Automotive Design
University of Kaiserslautern, Germany
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>&
<EMAIL_ADDRESS>
##  1 Introduction

Vehicle speed prediction is regarded as a key aspect in intelligent
transportation systems (ITS) that fundamentally aim at improving the road
safety, traffic-efficiency and vehicular energy efficiency (Jiang and Fei,
2017). Past studies on speed forecasting in transportation systems focused
mainly in two directions, namely, network-wide traffic speed prediction (Cui
et al., 2018), and host vehicle velocity prediction (Sun et al., 2015; Gaikwad
et al., 2019). Forecasting the future velocities of the host vehicle for the
entire driving route for the purpose of efficient energy management in hybrid
electric vehicles (HEVs) is studied in (Sun et al., 2015). Moreover, the
vehicle speed prediction can largely benefit the newly developed advanced
driver assistance systems (ADAS) as well (Schmied et al., 2015). With a goal
to minimize the energy consumption in on-road vehicles, efforts are being made
towards developing advanced adaptive cruise control (ACC) concepts to control
the host vehicle in an automated fashion. In this regard, ecological adaptive
cruise control (EACC) concepts, that use an optimal controller to compute an
energy-optimal speed for the host vehicle while tracking a target (leading)
vehicle are becoming popular (Moser et al., 2015). In prior studies (Weißmann
et al., 2018; Chada et al., 2020), to explore the energy consumption reduction
benefits using EACC in a car-following scenario, a common assumption was made
that the future velocities of the target vehicle are perfectly available.
However, the perfect future velocities of the target vehicle in a real world
setting are not known a priori, but rather must be predicted through either
behavioral models or data-driven approaches.

Several methods for developing vehicle speed predictors for various
applications were proposed in the prior works. For instance, a comparative
analysis on the parametric and non-parametric approaches for speed prediction
in highway driving is presented in (Lefèvre et al., 2014). The authors
classified the prediction space into short-term prediction ($<4\,\mathrm{s}$)
and long-term prediction ($4-10\,\mathrm{s}$). Vehicle velocity prediction
until $10\,\mathrm{s}$ for the use case in energy management in HEVs is
studied in (Gaikwad et al., 2019). (Liu et al., 2019) investigated on the one
hand stochastic models such as Markov chain and conditional linear Gaussian
(CLG) for the host vehicle velocity prediction. On the other hand,
deterministic models such as auto-regressive moving average (ARMA), nonlinear
auto-regressive exogenous model (NARX) and recurrent neural networks such as
long-short term memory (LSTM) units were studied. The authors in (Wegener et
al., 2021) studied longitudinal vehicle speed prediction in urban environments
using CLG and deep neural networks (DNNs). Moreover, (Shin et al., 2019)
proposed a fuzzy markov chain model with speed constraints to perform host
vehicle speed prediction. In (Lin and Görges, 2018), the authors proposed a
cloud-based seasonal autoregressive integrated moving average (SARIMA)
framework for vehicle speed prediction purpose, for which a highway database
was used. To enhance the accuracy of the speed predictors, availing the
benefits of the surrounding information using vehicle-to-vehicle (V2V) and
vehicle-to-infrastructure (V2I) communication can be essential (Moser et al.,
2015).

Concerning the speed prediction for the ecological cruise control use case,
there exist only a handful of studies (Schmied et al., 2015; Moser et al.,
2015; Jia et al., 2020; Sankar et al., 2022). (Schmied et al., 2015) used a
simplified prediction model with sinusoidal functions to predict the preceding
vehicle behavior, and demonstrated a predictive cruise control approach. In
(Moser et al., 2015), a Bayesian network approach with CLG model was used to
predict the information of a target vehicle for the cooperative adaptive
cruise control (CACC) use case. In (Jia et al., 2020) long-short term memory
(LSTM)-based energy-optimal ACC for target vehicle speed prediction in an
urban environment is proposed. Furthermore, the authors in (Wegener et al.,
2021) implemented vector autoregressive (VAR) model to generate simultaneous
predictions of the target vehicle, and used receding horizon control to derive
optimal accelerations for the host vehicle.

In most of the previous works (Lin and Görges, 2018; Shin et al., 2019; Liu et
al., 2019; Jia et al., 2020; Wegener et al., 2021), the speed prediction
models were trained on datasets that were gathered from repeated trials in the
same driving route by considering either limited or no surrounding traffic.
Although the driving patterns are comparatively easy to predict in such an
approach, the limitation, however, remains that the prediction model may not
generalize well for unseen data from a different route. To address this, the
present work proposes a scalable and more generalizable method for preparing
the time series data, and develop a speed predictor that uses the historical
observations to predict the target vehicle future velocities in both urban and
highway environments.

The contributions made in this paper are: (1) A novel approach for time series
data preparation for urban and highway networks using the microscopic traffic
simulation tool SUMO is presented. (2) To predict the target vehicle future
velocities, both deep recurrent neural networks (LSTM and GRU) and physics-
based models (CV and CA) are studied. (3) The influence of various input
variables (e.g. preceding vehicle behavior, traffic light signal phase and
road speed limits) on the prediction accuracy are investigated. Moreover, the
impact of using additional V2V and V2I information on the accuracy of the
predicted outputs is explored. (4) Furthermore, the performance of the EACC on
the predicted target vehicle velocities is evaluated, and the energy-saving
potential for different prediction horizons are investigated.

##  2 Ecological Adaptive Cruise Control

###  2.1 System Dynamics

The longitudinal vehicle dynamics of the host car is described by

$\displaystyle\frac{dv_{\text{h}}}{dt}=\frac{1}{m_{\text{eq}}}(F_{\text{t}}-F_{\text{b}}-\underbrace{F_{\text{a}}-F_{\text{roll}}-F_{\text{g}}}_{F_{\text{r}}})$
(1)

where $v_{\text{h}}$ is the host car velocity, $m_{\text{eq}}$ is the
equivalent mass which is the sum of vehicle weight, rotational equivalent
masses, driver and cargo weight, $F_{\text{t}}$ is the traction force,
$F_{\text{b}}$ is the braking force and $F_{\text{r}}$ is the combination of
aerodynamic resistance $F_{\text{a}}=\frac{1}{2}\rho
A_{\text{f}}c_{\text{a}}v_{\text{h}}^{2}$, rolling resistance
$F_{\text{roll}}=c_{\text{r}}m_{\text{v}}g\cos\theta$ and gradient resistance
$F_{\text{g}}=m_{\text{v}}g\sin\theta$. To handle the nonlinearity occurring
due to the term $v_{\text{h}}^{2}$ in $F_{\text{a}}$, an approximation of the
aerodynamic resistance $F_{\text{a}}\approx\frac{1}{2}\rho
A_{\text{f}}c_{\text{a}}(p_{1}v_{\text{h}}+p_{\text{2}})$ is considered in
this work. Here, $c_{\text{a}}$ is the drag coefficient, $A_{\text{f}}$ is the
frontal cross-sectional area of the vehicle, $\rho$ is the density of the air,
$p_{\text{1}}$ and $p_{\text{2}}$ are the coefficients obtained through line
fitting. Furthermore, $m_{\text{v}}$ is the host vehicle weight, $g$ is the
gravitational acceleration, $\theta$ is the gradient angle and $c_{\text{r}}$
is the rolling resistance coefficient.

The chosen host vehicle in this work is a battery electric vehicle (BEV),
whose accurate vehicle and battery models are obtained from (Lin et al.,
2014). A half-map approximation of the BEV power consumption map (Chada et
al., 2020) is used in this study.

###  2.2 Model Predictive Control Problem Formulation

A typical car-following scenario is illustrated in Fig. 1, in which a host car
is tracking a target vehicle. The EACC optimization problem based on model
predictive control (MPC) framework is formulated in time-domain with a goal to
minimize the objective function (2). Here, $N$ is the length of the prediction
horizon and $k$ denotes the discrete time.

$\displaystyle\min_{\textit{{$F_{t,k}$}},\textbf{$F_{b,k}$},{\zeta}_{1,k},{\zeta}_{2,k}}\leavevmode\resizebox{258.36667pt}{}{$\sum_{k=0}^{N-1}P(v_{\text{h},k},F_{\text{t},k})+\varepsilon_{1}F_{\text{b},k}^{2}+$}$
(2a) $\displaystyle\leavevmode\resizebox{422.77661pt}{}{$\text{s.t.}\ \
v_{\text{h},k+1}=v_{\text{h},k}+\frac{\Delta
T}{m_{\text{eq}}}(F_{\text{t},k}-F_{\text{b},k}-c_{\text{r}}m_{\text{v}}g\cos\theta_{k}$}-$
(2b) (2c) (2d) (2e) (2f) (2g) (2h) $\displaystyle\ \ \ \ \ \ \ \ \ \ \ \ \ \ \
d_{\text{rel},k}\geq\underbrace{d_{\text{min}}+h_{\text{m}}v_{\text{h},k}}_{d_{\text{s},k}}$
(2i) $\displaystyle\ \ \ \ \ \ \ \ \ \
d_{\text{rel},k}\leq\underbrace{d_{\text{min}}+h_{\text{c}}v_{\text{h},k}}_{d_{\text{c},k}}+\zeta_{\text{1},k}$
(2j)

The first term in the cost function minimizes the BEV power consumption that
can be approximated as a function of $v_{h,k}$ and $F_{t,k}$ (Chada et al.,
2020). Using the second term in (2), an excessive braking force
$F_{\text{b},k}$ is penalized. In the third term, to motivate the host car to
stay within the desired region $d_{\text{c}}$ to a target vehicle, a slack
variable $\zeta_{1}$ is penalized. Furthermore, in the final term the
variation in the traction force at successive time steps is penalized using a
slack variable $\zeta_{2}$ if it exceeds a constant value $\Delta
F_{\text{t,max}}$, with the aim to minimize jerks and improve the driving
comfort, as given in (2g and 2h). $\varepsilon_{1}$, $\varepsilon_{2}$ and
$\varepsilon_{3}$ are the corresponding weighting factors for the
aforementioned terms in the cost function (2). The host vehicle velocity and
relative distance to the target vehicle in discretized form is given in (2)
and (2c), where $v_{\text{t}}$ is the velocity of the target vehicle.
Furthermore, the limits for the traction force $F_{\text{t}}$ are set using
(2e), in which $F_{\text{t,max}}$ is the maximum traction force. The physical
limitations of the vehicle with respect to velocity and braking force are
addressed in (2d) and (2f). In order to maintain a safe distance to the target
vehicle, a hard constraint is introduced in (2i). Here, the state variables
are $x_{k}=[v_{\text{h},k},d_{\text{rel},k}]^{\top}$ and the control variables
are
$u^{*}_{k}=[F_{\text{t},k},F_{\text{b},k},\zeta_{\text{1},k},\zeta_{\text{2},k}]^{\top}$.

Figure 1: Schematic of a typical car-following scenario

##  3 Data Preparation for Speed Prediction

In contrary to the other approaches which considered a fixed driving route for
data preparation, this work focuses on network-based data collection as it is
laborious and time-consuming to extract V2V and V2I information in real world.
The goal here is to gather rich driving datasets from the urban and highway
networks that are route-independent under varied traffic conditions. Moreover,
the data must enable developing scalable prediction models and promote
generalizability. Therefore, to extract time series information in this work,
the open source microscopic traffic simulation tool SUMO (Simulation for Urban
Mobility)(Lopez et al., 2018) was used. The road networks in SUMO are
generated from the open-source map database (OpenStreetMap contributors,
2017). To reproduce the real world traffic flow in the simulation, a random
traffic ($3000$ vehicles/network) is generated with each traffic object in the
network having a random starting and destination points (Chada et al., 2021).
The training datasets are chosen from the Landstuhl highway (Fig. 3a) and
Kaiserslautern city (Fig. 3b). To improve generalizability, five test datasets
are chosen from a different city and connecting highway (Nieder-Olm), as shown
in Fig. 3c.

In a highway driving scenario, the speed of the target vehicle is primarily
influenced by the regulatory road speed limits and the velocities of a
preceding vehicle. Besides these factors, the presence of traffic light
signals in the urban environments influence the target vehicle speed as well.
By leveraging the V2V and V2I information, additional inputs from multiple
vehicles ahead of the target vehicle can be obtained. Moreover, the future
signal phase and timing (SPaT) information can be used to improve the
predictions for the target vehicle.

In this work, two feature groups FG1 and FG2 are investigated. As listed in
Table 1 and illustrated in Fig. 2, the feature group FG1 consists of six input
features, namely, velocity of the target vehicle $v_{\text{t}}$, velocity of
the first preceding vehicle $v_{\text{p}_{1}}$, relative distance between the
target vehicle and the first preceding vehicle $d_{\text{rel}_{1}}$, traffic
light signal current state $s_{\text{TL}}$, relative distance between the
target vehicle and the traffic light signal $d_{\text{TL}}$, and the maximum
road speed limit $v_{\text{max}}$. In addition to the input features described
in FG1, the feature group FG2 considers the influence of the second preceding
vehicle with velocity $v_{\text{p}_{2}}$, and the relative distance between
the target vehicle and the second preceding vehicle $d_{\text{rel}_{2}}$ that
is calculated using,

$d_{\text{rel}_{2}}=\ d_{\text{rel}_{1}}+\ l_{\text{p}_{1}}+\
d_{\text{p}_{12}}$ (3)

where, $l_{\text{p}_{1}}$ is the length of the first preceding vehicle and
$d_{\text{p}_{12}}$ is the relative distance between the first and second
preceding vehicles. Besides, FG2 also considers the future traffic light
signal state information $s_{\text{TL},k+1},..., s_{\text{TL},k+H}$ until the
prediction horizon $H$ as input features. During the data collection process,
each traffic object in the network is considered to be a target vehicle and
the input variables as given in Table 1 are gathered. To access the above
mentioned variables from the simulation, a traffic control interface known as
TraCI4Matlab is used (Acosta et al., 2015). Furthermore, preprocessing
techniques such as data cleaning, normalization, and data splitting into
training, validation and testing are performed to handle the data in an
efficient manner.

Figure 2: Schematic of V2V and V2I enabled traffic scenario

(a) Landstuhl highway

(b) Kaiserslautern city

(c) Nieder-Olm

Figure 3: SUMO networks to generate training (a & b) and testing (c) datasets

##  4 Speed Prediction Methods

In this work, to develop a target vehicle speed predictor, both the physics-
based prediction methods and deep recurrent neural networks are studied.

###  4.1 Physics-Based Prediction Models

####  4.1.1 Constant Velocity:

The constant velocity (CV) model assumes that the future velocities of the
target car remain constant, and can be determined using

$v_{\text{t}}(t+\Delta T)=\overline{v}_{\text{t}}(t)$ (4)

where
$\overline{v}_{\text{t}}(t)=\frac{v_{\text{t}}(t)+v_{\text{t}}(t-1)}{2}$,
$\Delta T$ is the sample time.

####  4.1.2 Constant Acceleration:

The constant acceleration (CA) model makes the assumption that the future
velocities of the target car are incremented by a constant amount of
acceleration, i.e.

$v(t+\Delta T)=v(t)+\Delta Ta(t)$ (5)

where $a(t)=\frac{v_{\text{t}}(t)-v_{\text{t}}(t-1)}{\Delta T}$.

Table 1: Feature groups and their corresponding input variables Feature group | Input Features
---|---
FG1 | $v_{\text{t},k},v_{\text{p}_{1},k},d_{\text{rel}_{1},k},s_{\text{TL},k},d_{\text{TL},k},v_{\text{max},k}$
FG2 | $v_{\text{t},k},v_{\text{p}_{1},k},d_{\text{rel}_{1},k},s_{\text{TL},k},d_{\text{TL},k},v_{\text{max},k}$
$v_{\text{p}_{2},k},d_{\text{rel}_{2},k},s_{\text{TL},k+1},...,
s_{\text{TL},k+H}$

###  4.2 Recurrent Neural Networks

Recurrent neural networks (RNNs) are popularly known in the category of deep
neural networks as they are capable of using their internal state memory to
process sequential or time series data. Two variants in the RNN architecture,
namely, gated recurrent unit (GRU) and long short-term memory (LSTM) units are
investigated in this work. The internal cell structure of LSTM and GRU are
illustrated in the Fig. 4(a) and Fig. 4(b), respectively.

The internal mechanisms of an LSTM (Fig. 4(a)) consists of a cell state
$C_{\text{t}}$ and various gates such as a forget gate $f_{\text{t}}$, an
input gate ($i_{\text{t}}$, $\\!\widetilde{C}_{\text{t}}$) and an output gate
$o_{\text{t}}$. The cell states act as a transport highway to transfer the
information from previous intervals all the way down the entire sequence
chain. The gates can regulate the flow of information by either adding or
removing the information from the cell state. The inputs to the LSTM are the
previous hidden state $h_{\text{t-1}}$ and the current state $x_{\text{t}}$.
The forget gate uses a sigmoid activation function to decide on which
information to throw away from the cell state, and is described using (6a).

$\displaystyle
f_{\text{t}}=\sigma\left(W_{f}\left[h^{T}_{t-1},X^{T}_{t}\right]+b_{f}\right)$
(6a) $\displaystyle
i_{\text{t}}=\sigma\left(W_{i}\left[h^{T}_{t-1},X^{T}_{t}\right]+b_{i}\right)$
(6b)
$\displaystyle\\!\widetilde{C}_{\text{t}}=\text{tanh}\left(W_{C}\left[h^{T}_{t-1},X^{T}_{t}\right]^{T}+b_{C}\right)$
(6c) $\displaystyle
C_{\text{t}}=f_{t}*C_{t-1}+i_{t}*\\!\widetilde{C}_{\text{t}}$ (6d)
$\displaystyle
o_{\text{t}}=\sigma\left(W_{o}\left[h^{T}_{t-1},X^{T}_{t}\right]+b_{o}\right)$
(6e) $\displaystyle h_{\text{t}}=o_{t}*\text{tanh}(C_{t})$ (6f)

Figure 4: Internal cell structure of LSTM and GRU

The input gate is a combination of two layers. The first layer is referred to
as input layer gate $i_{\text{t}}$, which decides on which important
information to retain to update in the cell-state and the second layer is a
tanh layer which adds new candidate values $\\!\widetilde{C}_{\text{t}}$ to
the cell state as given in (6b) and (6c) respectively. Thus, the old cell
state $C_{\text{t-1}}$ is updated to a new cell state $C_{\text{t}}$ using the
forget gate and the input gate information according to (6d). Finally, the
cell state passes through the tanh activation function and the final output is
filtered using a sigmoid layer resulting in the next hidden state
$h_{\text{t}}$ as described in (6e) and (6f). In equation (6), $W_{\text{f}}$,
$W_{\text{i}}$,$W_{\text{c}}$, $W_{\text{o}}$ are the weights and
$b_{\text{f}}$, $b_{\text{i}}$,$b_{\text{c}}$, $b_{\text{o}}$ are the biases
of the forget, input and output gates respectively.

In contrary to the LSTM, the GRU has a simpler cell structure with only two
gates, namely, reset and update gates, as illustrated in Fig. 4(b). The reset
gate is used to determine how much historical information to forget and the
update gate decides which new information must be passed along to the future.
The equations for the GRU are given by

$\displaystyle y_{t}=\sigma(U_{y}x_{t}+W_{y}q_{t-1}+a_{y})$ (7a)
$\displaystyle s_{t}=\sigma(U_{s}x_{t}+W_{s}q_{t-1}+a_{s})$ (7b)
$\displaystyle\tilde{q}_{t}=tanh(U_{q}x_{t}+W_{q}(s_{t}\odot q_{t-1}+a_{q}))$
(7c) $\displaystyle q_{t}=(1-y_{t})\odot q_{t-1}+y_{t}\odot\tilde{q}_{t}$ (7d)

where $U_{*}$, $W_{*}$, and $a_{*}$ are the two weights and the bias, while
$\odot$ is the scalar product of the vectors and $\sigma$ refers to the
sigmoid activation function.

Figure 5: Architecture of Stacked RNN Figure 6: Prediction results of LSTM-FG2
on one of the test datasets for a prediction horizon of $5\,\mathrm{s}$ Figure
7: Comparison of target vehicle predictions for LSTM-FG1 and FG2 models for a
prediction horizon of $5\,\mathrm{s}$

####  4.2.1 Stacked RNN:

In order to obtain a greater level of abstraction, the hidden layers are
arranged together in stacked fashion as illustrated in Fig. 5. The
architecture consists of three segments, described from bottom to top: (i) an
input layer, (ii) stacked LSTM or GRU layers, (iii) an output layer. The input
layer consists of multiple input features (marked in green) as described in
Table 1. In Fig. 5 from left to right, the feature values of past, current and
future timesteps are provided as inputs to the stacked layers. On the right,
along with the current traffic light state, future state information up to a
prediction horizon $H$ is additionally used. The combined input sequence is
fed into the stacked hidden layers. Each layer has memory cells associated
with it, otherwise known as neurons. The lower layers are used to capture the
low level representations and the deeper layers can learn higher levels of
abstraction. The layer in yellow outputs multi-step sequence predictions up to
a prediction horizon $H$, in this case the future velocities of the target
vehicle.

To model the RNNs in this work, the keras deep learning library was used
(Brownlee, 2018) in python, and training of the deep learning prediction
models is performed on a GPU cluster. The hyperparameters for both the LSTM
and GRU are tuned using the Keras Bayesian optimization function. The tuned
hyperparameters for the $5\,\mathrm{s}$ speed predictors are illustrated in
the Table 2. Four stacked layers were implemented to model the non-linear
input data representations to outputs. Moreover, to reduce overfitting of the
data during the training process, a regularization technique called dropout is
used. Furthermore, a rectified linear activation function (reLU) is used to
train the network faster, and to estimate the loss of the model while
training, a mean squared error (MSE) is used as the loss function.
Additionally, the ADAM optimizer is used to efficiently update the network
weights while training. The length of the past sequence is maintained to be
equivalent to the prediction horizon $H$, and the learning rate is chosen as
$1e^{-}3$. The number of epochs was chosen as 25, in addition to which an
early stopping was implemented to be able to stop the training process when a
desired metric has stopped to improve.

Hyperparameters | FG1 | FG2
---|---|---
LSTM | GRU | LSTM | GRU
Batch size | 32 | 512
Input features | 6 | 13
Stacked layer 1 | 90 | 450 | 600 | 300
Dropout 1 | 0 | 0.3 | 0 | 0.2
Stacked layer 2 | 60 | 600 | 420 | 600
Dropout 2 | 0 | 0.3 | 0.25 | 0
Stacked layer 3 | 600 | 60 | 450 | 600
Dropout 3 | 0.3 | 0 | 0.3 | 0
Stacked layer 4 | 600 | 60 | 480 | 180
Dropout 4 | 0 | 0.3 | 0.3 | 0.1
Dense layer | 30 | 570 | 60 | 30
Table 2: Hyperparameters for LSTM and GRU models for the predicition horizon
of $5\,\mathrm{s}$

##  5 Results and discussion

Figure 8: Speed forecasting results of $\hat{v}_{\text{GRU}\text{,FG2}}$,
$\hat{v}_{\text{LSTM}\text{,FG2}}$, $\hat{v}_{\text{CA}}$ and
$\hat{v}_{\text{CV}}$ for a prediction horizon of $5\,\mathrm{s}$

###  5.1 Evaluation Metrics

To evaluate the performance of the prediction models on the test dataset, two
error metrics are used in this work. Firstly, the mean absolute error (MAE)
measures the average magnitude of the errors in a set of predictions and is
given by

$\text{MAE}=\sum\limits^{n}_{i=1}\frac{|\hat{v}_{t(i)}-v_{t(i)}|}{n}$ (8)

where $\hat{v}_{t(i)}$ is the predicted target vehicle speed at the $i^{th}$
prediction step, $v_{t(i)}$ is the original target vehicle observation and $n$
is the total number of observations. Secondly, the root mean squared error
(RMSE) is the square root of the average of squared differences between
predictions and actual observation and is described as

$\text{RMSE}=\sqrt{\sum\limits^{n}_{i=1}\frac{(\hat{v}_{t(i)}-v_{t(i)})^{2}}{n}}$
(9)

###  5.2 Performance Evaluation of the Prediction Methods

Prediction Models | MAE [m/s] | RMSE [m/s]
---|---|---
5s | 10s | 5s | 10s
FG1 | FG2 | FG1 | FG2 | FG1 | FG2 | FG1 | FG2
CV | 2.12 | 3.16 | 3.79 | 5.43
CA | 2.75 | 5.26 | 4.77 | 8.68
LSTM | 2.06 | 1.75 | 4.26 | 2.92 | 3.19 | 2.84 | 5.63 | 4.61
GRU | 2.02 | 2.51 | 3.19 | 3.3 | 3.29 | 3.68 | 4.58 | 4.7

Table 3: Comparison of MAE and RMSE for various prediction methods

The performance of the prediction methods discussed in Section 4 is evaluated
on the $5$ test datasets. The average MAE and RMSE for each prediction model
across different prediction horizons ($5\,\mathrm{s}$ and $10\,\mathrm{s}$)
with respect to the feature groups (FG1 and FG2) are summarized in Table 3. It
can be noticed that the prediction error increases for all the models as the
prediction horizon increases due to the increasing uncertainties. Moreover,
the LSTM model with input features FG2 has outperformed remaining methods, and
showcased lower average prediction error as compared to feature group FG1.

The prediction results for the LSTM-FG2 model with a prediction horizon of
$5\,\mathrm{s}$ after evaluating it on one of the test datasets that consists
of various traffic scenarios are shown in Fig. 6. Furthermore, the prediction
result of $\hat{v}_{\text{LSTM-FG1}}$ is compared against
$\hat{v}_{\text{LSTM-FG2}}$ in Fig. 7 and demonstrated for two scenarios. In
Fig. 7(a), the target vehicle is under the influence of two preceding vehicles
ahead (exemplary scenario in Fig. 2). As presented in Table 1, the FG1 uses
the inputs from the first preceding vehicle alone and the FG2 uses both the
preceding vehicle information as inputs with an assumption that this
information can be obtained from V2V communication. The results illustrate
that the $\hat{v}_{\text{LSTM-FG2}}$ has better tracking ability of the target
vehicle ${v}_{\text{t}}$ as compared to $\hat{v}_{\text{LSTM-FG1}}$. In the
second scenario as illustrated in Fig. 7(b), the target vehicle stopped at a
red phase of a traffic light signal. The predictions of $\hat{v}_{\text{LSTM-
FG2}}$ match the expected behavior of the target vehicle ${v}_{\text{t}}$,
however, as the feature group FG1 did not take the future traffic light phase
into account, the $\hat{v}_{\text{LSTM-FG1}}$ depict inaccurate predictions
that the target vehicle will move forward from standstill. Such a behavior may
in turn lead to traffic signal violations which is not desirable.

Furthermore, a comparison of the prediction results for the methods discussed
in Table 3 when evaluated on a few scenarios can be found in Fig. 8. It can be
observed that the LSTM-FG2 prediction model has demonstrated better prediction
accuracy in all the scenarios as compared to GRU-FG2, CV and CA models.
Although the prediction results of $\hat{v}_{\text{GRU,FG2}}$ matches with the
predictions of $\hat{v}_{\text{LSTM,FG2}}$ at a few timestamps, its accuracy
needs to be improved in a few scenarios (e.g. round-abouts). The predictions
based on CV and CA models were not able to accurately predict the target
vehicle behavior due to the presence of abrupt speed variations in the target
vehicle. It can be noticed from Fig. 8, that a prediction error exists with
the $\hat{v}_{\text{LSTM,FG2}}$ as well. For instance, while maneuvering a
right turn (Fig. 8(a)), at $89\,\mathrm{s}$ the model did not predict the slow
down at the curvature accurately. Similarly, in Fig. 8(d) the model is able to
predict the future speeds accurately in the round-about only until
$3\,\mathrm{s}$ into the future.

###  5.3 Performance Evaluation of EACC using Predicted Speeds

Figure 9: Speed profiles of EACC-equipped host car while tracking three target
vehicle velocity criteria Figure 10: Energy savings for EACC-equipped host car
while tracking three target vehicle velocity criteria

To analyze the performance of the EACC in a typical car-following scenario
while tracking a target vehicle, an evaluation is conducted based on three
criteria in which the target vehicle future velocities are (I) assumed to be
constant due to the lack of a robust speed predictor and V2V communications,
(II) predicted using the LSTM-based speed prediction model proposed in this
work, and (III) assumed to be perfectly available. The speed profiles of the
host vehicle for the three criteria $v_{\text{h,I}}$, $v_{\text{h,II}}$ and
$v_{\text{h,III}}$ are illustrated in Fig. 9 while tracking a target vehicle
velocity profile $v_{\text{t}}$. In the distance plot in the above part of
Fig. 9, the host vehicle with condition II is seen performing a robust car-
following and maintains a good inter-vehicle distance to the target vehicle
$d_{\text{rel,II}}$. It can be noticed from the two enlarged sections in the
below part of Fig. 9, that the $v_{\text{h,II}}$ is able to track the
$v_{\text{h,III}}$ better than the $v_{\text{h,I}}$. Moreover, considering a
constant speed (CS) model in criterion I has resulted in abrupt decelerations
and sharp accelerations as compared to the II and III.

In Fig. 10, a comparison of the energy savings of the EACC-equipped host
vehicle after evaluating on the three above mentioned criteria is shown. The
results demonstrate that the energy savings can increase with the increase in
the prediction horizon, i.e. the more the information about the target vehicle
is available into the future, the better that the host vehicle is able to plan
its optimal trajectories. Moreover, the proposed LSTM-based EACC (criterion
II) is able to achieve better energy savings as compared to the criterion I
and is close to criterion III.

Concerning the execution time of the proposed LSTM-based EACC, an evaluation
is performed for the prediction horizon of
$10\,\mathrm{s}~{}(50\,\mathrm{steps})$ using Matlab® R2021b profiler on a
Windows 10 PC equipped with an Intel® Core™ i7-7500U CPU processor with 2.70
GHz clock frequency and 12 GB RAM. No GPU for parallel computing was used. The
mean execution time for the proposed concept is found to be $60\,\mathrm{ms}$.
The sample time in this work is chosen as $\Delta T$=$200\,\mathrm{ms}$
including the time for inference of the neural networks. As with the current
system configuration, the LSTM-based EACC can be solved at each step below the
sample time, thus showcasing the real-time capability of the proposed
controller.

##  6 Conclusion

In this work, to enhance the efficiency of an ecological adaptive cruise
control (EACC) strategy, an LSTM-based target vehicle speed prediction model
for both urban and highway scenarios is proposed. In the speed prediction
task, the LSTM model outperformed GRU, CV and CA models, and was able to
capture the historical dependencies from several input features and perform
long-term predictions up to $10\,\mathrm{s}$. Moreover, considering the
additional input features, such as information about multiple preceding
vehicles in the driving route obtained through V2V and traffic light signal
future phases gathered through V2I has enhanced the prediction accuracy.
Furthermore, energy savings up to $26\%$ can be realizable for the EACC-
equipped host car while tracking the target vehicle predicted velocities. A
performance increment in terms of additional average energy savings of up to
$2.5\%$ with the proposed LSTM-based EACC can be achieved as compared to the
constant speed (CS) model. For further improvements in the prediction
accuracy, additional input features constituting traffic rules at non-priority
intersections, road topology and curvature must be further investigated.

## References

  * Acosta et al. (2015) Acosta, A.F., Espinosa, J.E., and Espinosa, J. (2015).  Traci4matlab: Enabling the integration of the SUMO road traffic simulator and Matlab® through a software re-engineering process.  _Lecture Notes in Control and Information Sciences_ , 13, 155–170.
  * Brownlee (2018) Brownlee, J. (2018).  Deep learning for time series forecasting.  _Machine Learning Mastery_.
  * Chada et al. (2021) Chada, S.K., Görges, D., Ebert, A., and Teutsch, R. (2021).  A driver-in-the-loop co-simulation framework for testing predictive EDAS for commercial vehicles in urban environments.  _In Proceedings of the 6th Commercial Vehicle Technology Symposium 2020/2021_ , 107–118.
  * Chada et al. (2020) Chada, S.K., Purbai, A., Görges, D., Ebert, A., and Teutsch, R. (2020).  Ecological adaptive cruise control for urban environments using SPaT information.  _In Proceedings of the IEEE Vehicle Power and Propulsion Conference_ , 2–7.
  * Cui et al. (2018) Cui, Z., Ke, R., Pu, Z., and Wang, Y. (2018).  Deep bidirectional and unidirectional LSTM recurrent neural network for network-wide traffic speed prediction.  URL http://arxiv.org/abs/1801.02143.
  * Gaikwad et al. (2019) Gaikwad, T.D., Asher, Z.D., Liu, K., Huang, M., and Kolmanovsky, I. (2019).  Vehicle velocity prediction and energy management strategy part 2: Integration of machine learning vehicle velocity prediction with optimal energy management to improve fuel economy.  _SAE Technical Paper_ , 2019-01-1212.
  * Jia et al. (2020) Jia, Y., Cai, C., and Görges, D. (2020).  An LSTM-based speed predictor based on traffic simulation data for improving the performance of energy-optimal adaptive cruise control.  _In Proceedings of the 23rd IEEE International Conference on Intelligent Transportation Systems_ , 1–7.
  * Jiang and Fei (2017) Jiang, B. and Fei, Y. (2017).  Vehicle speed prediction by two-level data driven models in vehicular networks.  _IEEE Transactions on Intelligent Transportation Systems_ , 18, 1793–1801.
  * Lefèvre et al. (2014) Lefèvre, S., Sun, C., Bajcsy, R., and Laugier, C. (2014).  Comparison of parametric and non-parametric approaches for vehicle speed prediction.  _In Proceedings of the American Control Conference_ , 3494–3499.
  * Lin and Görges (2018) Lin, X. and Görges, D. (2018).  Cloud-based vehicle velocity prediction based on seasonal autoregressive integrated moving average processes.  _SAE Technical Paper_ , 2018-01-1178, 1–9.
  * Lin et al. (2014) Lin, X., Görges, D., and Liu, S. (2014).  Eco-driving assistance system for electric vehicles based on speed profile optimization.  _In Proceedings of the IEEE Conference on Control Applications_ , 629–634.
  * Liu et al. (2019) Liu, K., Asher, Z., Gong, X., Huang, M., and Kolmanovsky, I. (2019).  Vehicle velocity prediction and energy management strategy part 1: Deterministic and stochastic vehicle velocity prediction using machine learning.  _SAE Technical Paper_ , 2019-01-1051.
  * Lopez et al. (2018) Lopez, P.A., Behrisch, M., Bieker-walz, L., Erdmann, J., Fl, Y.p., Hilbrich, R., Leonhard, L., Rummel, J., Wagner, P., and Wießner, E. (2018).  Microscopic traffic simulation using SUMO.  _In Proceedings of the 21st IEEE International Conference on Intelligent Transportation Systems_ , 2575–2582.
  * Moser et al. (2015) Moser, D., Waschl, H., Schmied, R., Efendic, H., and del Re, L. (2015).  Short term prediction of a vehicle’s velocity trajectory using ITS.  _SAE International Journal of Passenger Cars - Electronic and Electrical Systems_ , 8(2), 364–370.
  * OpenStreetMap contributors (2017) OpenStreetMap contributors (2017).  Planet dump retrieved from https://planet.osm.org .  https://www.openstreetmap.org .
  * Sankar et al. (2022) Sankar, G.S., Kim, M., and Han, K. (2022).  Data-driven leading vehicle speed forecast and its application to ecological predictive cruise control.  _IEEE Transactions on Vehicular Technology_ , 1–12.  10.1109/TVT.2022.3193091.
  * Schmied et al. (2015) Schmied, R., Waschl, H., and Del Re, L. (2015).  A simplified fuel efficient predictive cruise control approach.  _SAE Technical Paper_ , 2015-01-0296.
  * Shin et al. (2019) Shin, J., Kim, S., Sunwoo, M., and Han, M. (2019).  Ego-vehicle speed prediction using fuzzy markov chain with speed constraints.  _In Proceedings of the IEEE Intelligent Vehicles Symposium_ , 2106–2112.
  * Sun et al. (2015) Sun, C., Hu, X., Moura, S.J., and Sun, F. (2015).  Velocity predictors for predictive energy management in hybrid electric vehicles.  _IEEE Transactions on Control Systems Technology_ , 23(3), 1197–1204.
  * Wegener et al. (2021) Wegener, M., Herrmann, F., Koch, L., Savelsberg, R., and Andert, J. (2021).  Longitudinal vehicle motion prediction in urban settings with traffic light interaction.  _IEEE Transactions on Intelligent Vehicles_.  10.1109/TIV.2021.3114156.
  * Weißmann et al. (2018) Weißmann, A., Görges, D., and Lin, X. (2018).  Energy-optimal adaptive cruise control combining model predictive control and dynamic programming.  _Control Engineering Practice_ , 72, 125–137.


# Generative AI for Space-Air-Ground Integrated Networks (SAGIN)

Ruichen Zhang, Hongyang Du, Dusit Niyato, , Jiawen Kang, Zehui Xiong,
Abbas Jamalipour, , Ping Zhang, , and Dong In Kim, R. Zhang, H. Du, and D.
Niyato are with the School of Computer Science and Engineering, Nanyang
Technological University, Singapore (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>dniyato@ntu.edu.sg).J. Kang is with the School of
Automation, Guangdong University of Technology, China (e-mail:
kavinkang@gdut.edu.cn).Z. Xiong is with the Pillar of Information Systems
Technology and Design, Singapore University of Technology and Design,
Singapore (e-mail: zehui_xiong@sutd.edu.sg).Abbas Jamalipour is with The
University of Sydney, Sydney NSW 2006, Australia (e-mail:
a.jamalipour@ieee.org).P. Zhang is with the State Key Laboratory of Networking
and Switching Technology, Beijing University of Posts and Telecommunications,
China (e-mail: pzhang@bupt.edu.cn).D. I. Kim is with the Department of
Electrical and Computer Engineering, Sungkyunkwan University, Suwon 16419,
South Korea (email:dikim@skku.ac.kr).

###### Abstract

Recently, generative AI technologies have emerged as a significant advancement
in artificial intelligence field, renowned for their language and image
generation capabilities. Meantime, space-air-ground integrated network (SAGIN)
is an integral part of future B5G/6G for achieving ubiquitous connectivity.
Inspired by this, this article explores an integration of generative AI in
SAGIN, focusing on potential applications and case study. We first provide a
comprehensive review of SAGIN and generative AI models, highlighting their
capabilities and opportunities of their integration. Benefiting from
generative AI’s ability to generate useful data and facilitate advanced
decision-making processes, it can be applied to various scenarios of SAGIN.
Accordingly, we present a concise survey on their integration, including
channel modeling and channel state information (CSI) estimation, joint air-
space-ground resource allocation, intelligent network deployment, semantic
communications, image extraction and processing, security and privacy
enhancement. Next, we propose a framework that utilizes a Generative Diffusion
Model (GDM) to construct channel information map to enhance quality of service
for SAGIN. Simulation results demonstrate the effectiveness of the proposed
framework. Finally, we discuss potential research directions for generative
AI-enabled SAGIN.

###### Index Terms:

Generative AI, space-air-ground integrated network, generative diffusion
model, channel state information

##  I Introduction

With the development of wireless communications, the goal of future 6th
generation (6G) mobile networks is to achieve comprehensive connectivity with
high performance and reliability. To meet the requirements, the space-air-
ground integrated network (SAGIN) has been proposed, which combines ground,
air and space communications in which their functions efficiently complement
each other for seamless and high-speed connections [1]. Specifically, the
space-based network is composed of the geostationary orbit (GEO) satellites
and the low-earth orbit (LEO) satellites, which can provide global coverage,
high-speed data transmission, and ensure reliable communication even in remote
or under-served regions. For example, Starlink, a massive satellite network
developed by SpaceX, is a constellation of thousands of low-Earth orbit (LEO)
satellites designed to provide broadband internet service across the
globe111https://www.space.com/spacex-starlink-satellites.html. The air-based
network is composed of the aircraft, unmanned aerial vehicles (UAVs) and high
altitude platforms (HAPs), which can swiftly respond to fluctuating demand,
and provide coverage in areas where ground or space-based solutions might be
less effective. The ground-based network is composed of cellular towers, small
cells, and edge computing facilities. They enable the ultra-low latency
communication and support dense urban connectivity. Thanks to its integrated
multi-layered approach, SAGIN is able to provide a seamless, adaptive, and
resilient communication infrastructure that meets the diverse needs of users
across varied environments.

With the development of artificial intelligence (AI) technologies, generative
AI has attracted more and more attention in terms of research, development,
and applications. In particular, generative AI describes methods or instances
that can be used to create new content, including audio, codes, images, texts,
simulations, and videos [2]. For example, DALL-E
2222https://openai.com/dall-e-2, a generative AI system from open AI,
showcases the prowess of generative AI by synthesizing novel images based on
text descriptions (i.e., alled prompts). Regarding generative AI, key models
include generative adversarial networks (GANs), diffusion models, and
variational autoencoders (VAEs), all of which have found practical
applications in real-world scenarios [3]. For example,
GauGAN333https://blogs.nvidia.com/blog/2019/03/18/gaugan-photorealistic-
landscapes-nvidia-research/, an NVIDIA AI demonstration for generating photo-
realistic images, enables individuals to leverage GAN to craft landscapes.
Unlike traditional discriminative AI models that focus primarily on analyzing
and processing existing data, generative AI holds significant advantages:

  * •

Content Generation: Generative models are capable of creating entirely new,
high-quality content that is contextually relevant, thereby reducing the
necessity for exhaustive data collection and manual content creation.

  * •

Flexibility and Adaptability: Generative AI can adapt and learn from a diverse
array of unstructured data sources, rendering it more versatile for handling a
variety of applications.

  * •

Enhanced Simulation and Optimization Capabilities: Through simulating
intricate scenarios or systems, generative AI facilitates better forecasting,
planning, and decision-making processes.

The integration of the SAGIN with generative AI is expected to significantly
transform wireless communication networks and digital content creation. By
leveraging the comprehensive connectivity provided by SAGIN, generative AI
facilitates the seamless generation and dissemination of high-quality contents
across disparate networks [2]. For example, during emergencies, critical
information can be created and distributed in real time, ensuring that both
individuals and authorities are fully informed and prepared to respond.
Additionally, generative AI’s adaptability can be used for intelligent network
management across all layers of SAGIN. By analyzing network traffic, user
requirements and environmental conditions, generative AI can assist in
dynamically allocating resources, predict network congestion and optimize
various performance of space, air and ground networks. This convergence also
enhances user experience by delivering personalized services, real-time
information and interactive contents through SAGIN’s extensive network
infrastructure. Although the integration of generative AI and SAGIN has
several potential advantages, there are still the following issues that need
to be addressed:

  * •

Q1: Which issues in SAGIN can be solved with generative AI?

  * •

Q2: What kind of generative AI can be used for SAGIN?

  * •

Q3: How to apply generative AI to SAGIN?

Motivated by these, this article is an attempt to provide a forward-looking
research to answer the above questions, where the contributions of this
article are summarized as follows.

  * •

First: We overview the SAGIN and introduce different types of generative AI
techniques. To the best of the authors’ knowledge, the synergy between
generative AI and SAGIN has still been an open issue thus far.

  * •

Second: We explore the existing issues and solutions of generative AI-enabled
SAGIN. Next, we discuss the challenges associated with implementing generative
AI to improve various aspects of SAGIN.

  * •

Third: To address these challenges, we construct a novel generative AI-enabled
channel information map to enhance quality of service for SAGIN. Here, GDM is
adopted to perform channel classification. Simulation results validate the
effectiveness of the proposed framework.

##  II OVERVIEW OF SAGIN AND GENERATIVE AI

In this section, we first review SAGIN toward its openness to leverage the
potentials of generative AI. Then, typical generative AI models are
introduced.

###  II-A Overview of SAGIN

SAGIN network represents a cross-layer architecture in the B5G/6G
communication, and is committed to providing enlarged coverage, high speed and
seamless connection, strong security and reliability. The SAGIN combines
technologies from the ground, air and space networks, where the details are as
follows.

  * •

Ground-based network: The ground-based network is the foundation of SAGIN,
including elements such as ground-based Internet and mobile communication
networks. To meet the communication requirements from cloud centers to
individual mobile users, a variety of machine learning is adopted for
optimization. For example, in [4], a traditional machine learning method based
on two-dimensional long short-term memory (LSTM) is proposed to predict
network traffic and achieve resource allocation. However, due to the dynamic
nature of ground-based networks (i.e., user mobility and traffic
fluctuations), LSTM may encounter delays in adapting to network changes, thus
preventing timely and optimal resource allocation.

  * •

Air-based network: The air-based network establishes high-altitude
communication platforms and UAV self-organized networks. The high-altitude
platforms provide a wide range of communication services, in which UAV
networks composed of multiple UAVs offer autonomous connections to enhance
communication support for ground users. Several studies apply traditional
machine learning-based methods to air-based networks. For example, in [5], a
deep reinforcement learning (DRL) approach was proposed to optimize UAV
trajectories to minimize the weighted sum of execution time and expected
transmission interruption duration. However, traditional DRL requires a high
computing resource consumption, a large amount of calculation, and a long
convergence time, which bring challenges to real-time decision-making in
highly-dynamic aerial environments.

  * •

Space-based network: The space-based network is composed of a variety of
satellite systems as backbone networks that ensure global connectivity and
facilitate specialized communications. To achieve the goals of wide coverage,
high bandwidth and low latency, machine learning-based methods were developed
for the space-based networks. For example, in [6], a cooperative transmission
scheme for satellites and airborne base stations was developed using K-means
and K-medoids. However, due to the sensitivity of K-means methods to the
initial centroid location and their two-dimensional assumption of clusters,
they may have difficulty in fully covering complex channel data of three-
dimensional spatial communication. This is especially in situations such as
satellite signal interference or multi-path propagation.

Therefore, to achieve better design and optimization, there is an urgent need
to introduce entirely new machine learning algorithms, especially generative
AI.

TABLE I: Summary of typical generative AI models in SAGIN.

###  II-B Overview of generative AI

Generative AI is a distinguished subdomain of AI, exmphasizing on the
conceptualization and generation of content. It emerges from the aspiration to
enable machines to fabricate innovative and unprecedented data that faithfully
mirrors the inherent patterns, structures, and nuances encapsulated in the
training datasets. Generative AI integrates a plethora of sophisticated
models, including VAE, GAN, GDM, and TBM, each espousing unique methodologies
for assimilating and generating data.

  * •

Variational Autoencoders (VAEs): VAEs are generative models that use
variational inference to model high-dimensional data, allowing them to
generate new, consistent data instances. In SAGINs, VAEs facilitate the
modeling and generation of adaptive communication protocols and diverse
network configurations. For instance, in [7], a generative framework based on
VAE was proposed to generate synthetic data for LOS estimation in non-
terrestrial networks. In this case, the generated synthetic data does not
contain identification information from the original dataset, allowing for its
public use without violating privacy.

  * •

Generative Adversarial Networks (GANs): GANs are based on a game-theoretic
approach involving two parts, i.e., a generator and a discriminator， which are
effective in creating high-quality data. In SAGINs, GANs are used to mimic
real network traffic patterns, aiding in better network traffic management and
anomaly detection. For example, in [8], a network traffic prediction method
based on LSTM and GAN was proposed to address the significant variations in
service traffic access across different regions of satellite networks. Here,
GAN was utilized to augment the dataset and prevent the occurrence of training
overfitting issues.

  * •

Generative Diffusion Model (GDMs): GDMs replicate random diffusion processes
to map data structures, leading to the creation of realistic data instances.
In SAGINs, generative diffusion models are instrumental for simulating diverse
network states and communication channel conditions. For example, in [9], a
GDM detection algorithm was proposed that can capture fast-moving anomalies in
satellite data without any moving components. Simulation results showed that
the proposed GDM detection algorithm outperforms other baseline methods.

  * •

Transformer-Based Models (TBMs): TBMs are characterized by a self-attention
mechanism that can manage data, learn complex dependencies, and generate
coherent contextual instances. In SAGINs, TBM is used to simplify sequential
communication protocols and ensure consistent exchanges between different
network domains. For example, in [10], a model based on TBM was proposed for
semantic segmentation of remote sensing images in satellite networks, where an
adaptive fusion module was designed to blend semantic information between
features of different scales through a self-attention mechanism.

The features of generative AI models are summarized in Table I.

Figure 1: AI-Generated model enabled SAGIN. Different types of generative AI
technologies can be used to perform channel modeling and CSI estimation,
perform joint air-space-ground resource allocation, perform intelligent
network deployment, perform semantic communications, perform image extraction
and processing, and perform security and privacy enhancement.

##  III Generative AI-enabled SAGIN

In this section, we explore the issues and solutions of generative AI in
SAGIN. Next, we discuss the challenges associated with implementing generative
AI-enabled SAGIN.

###  III-A Research Issues and Solutions.

As shown in Fig. 1, by incorporating these advanced generative AI
technologies, generative AI-enabled SAGIN is capable of enhancing channel
estimation, optimizing resource allocation, and ensuring intelligent network.
We summarize major research issues and solutions of generative AI-enabled
SAGIN.

####  III-A1 Channel modeling and CSI Estimation

Traditional channel modeling and CSI estimation often rely on predefined
models based on historical data and empirical measurements, necessitating
manual updates and revisions to account for the dynamic SAGIN environment.
Iterative methods, such as the RL or Kalman filters, enhance estimation
accuracy through repetition and feedback. However, they lack an ability to
autonomously adapt to unmodeled channel conditions or recognize complex
patterns in data. In contrast, generative AI, specifically GAN and VAE,
provides an approach to enhance channel modeling and CSI estimation in SAGIN
through using real-time spatial information and analyzing radio wave
propagation. Specifically, GAN, with its two-part neural architecture,
effectively simulates channel perturbations by creating virtual channel
scenarios. Its generator introduces underlying channel conditions, while the
discriminator distinguishes between real and simulated conditions. This
iterative process enables GAN to develop reliable channel models that reflect
real-world SAGIN conditions [11]. Moreover, VAE can learn probabilistic
mappings and is used for CSI estimation from observed data. Through the
compression and reconstruction of wireless channel data, VAE is able to
provide accurate CSI predictions, aligning empirical data with theoretical
models.

####  III-A2 Joint Air-Space-Ground Resource Allocation

Traditional joint resource allocation often utilizes algorithms like linear
programming or convex optimization, which are based on well-defined
mathematical models. These methods, while precise for known scenarios, might
not adapt well to the dynamic changes in SAGIN environments. Techniques such
as fixed power allocation and static trajectory planning are employed, but
they may not guarantee optimal performance under varying conditions. In
contrast, generative AI, specifically GDM and GAN, provides an approach for
joint resource allocation in SAGIN. These generative AIs are capable of
optimizing trajectories of aerial devices and dynamically adjusting transmit
power to meet the various requirements of air and ground communications.
Specifically, GDM can capture patterns such as time series dependence and
spatial correlation to model channel distribution in the time and spatial
domains, respectively, which can provide a view of an entire communication
environment. With this capability, GDM can effectively identify suitable
trajectories and power levels, ensuring efficient communications and minimal
interference [12]. Moreover, in the GAN structure, the generator produces
hypothetical resource allocation scenarios, and the discriminator assesses and
refines the efficiency of these allocations, enhancing resource utilization
and network performance.

####  III-A3 Intelligent Network Deployment

SAGIN network deployment has been guided by grid-based methodologies, genetic
algorithm, and RL. These techniques, though effective in specific settings,
might not be optimal for the diverse and dynamic demands of SAGIN
environments. Moreover, they often require manual intervention or adjustment.
In contrast, generative AI, specifically VAE and TBM, provides an approach for
intelligent network deployment in SAGIN. Specifically, VAE can encode high-
dimensional data into a lower-dimensional representation and decode it. This
generative capability allows VAE to discern underlying patterns and
dependencies, leading to efficient and adaptive network deployment.
Furthermore, with its self-attention mechanism, TBM ensures that all SAGIN
users (i.e., ground users and air users) can interact with any other user.
This can improve the heterogeneity of the SAGIN network and enhance network
decisions about topology, node placement, and resource allocation[13].

TABLE II: Comparison between Traditional methods and Generative AI methods in
SAGIN applications.

####  III-A4 Semantic Communications

Traditional semantic communications rely on methods like ontology-based data
representation or convolutional neural network (CNN). While they provide
structured ways to represent and communicate data, they can become rigid and
unable to adapt to new or evolving data patterns in SAGIN. In contrast,
generative AI, specifically GDM and TBM, provides an effective semantic
communications approach for SAGIN based on the remote sensing. Specifically,
thanks to its ability to simulate data distribution, GDM is able to understand
the semantic structure of communication datasets. Therefore, GDM can obtain
valuable information from a wide range of datasets to facilitate semantic
communications. On the other hand, through its self-attention mechanism, TBM
is able to establish a context-aware communication environment, which ensures
that key information is always maintained in the SAGIN network. For example,
when satellites in SAGIN make observations about channel condition or remote
sensing, TBM helps identify relationships between geographic features such as
vegetation, water, and urban areas, which maintains the fidelity of the
relationships during data transmission and interpretation.

####  III-A5 Image Extraction and Processing

Traditional image extraction and processing in SAGIN have often been performed
using techniques like edge detection, fourier transforms, and CNN. While these
methods can process images, they might not be adept at handling diverse range
and complexity of images in SAGIN scenarios. In contrast, generative AI,
specifically VAE and GAN, provides an approach for image extraction and
processing in SAGIN, which paves the way for diverse applications such as
autonomous vehicle navigation with Internet of Vehicles (IoV). Specifically,
VAE can effectively compress and extract high-dimensional image data and
reconstruct it through generating data, making it applicable to a variety of
SAGIN datasets [12]. Alternatively, through the generator-discriminator
structure, GAN is able to improve image quality by generating enhanced
synthetic data. The generator generates high-quality versions of images while
the discriminator distinguishes them from the original dataset, thereby
iteratively improving the image output.

####  III-A6 Security and Privacy Enhancement

Traditional security measures in SAGIN primarily utilize intrusion detection
systems (IDS), and deep belief network (DBF). These rule-based systems, while
essential, may not always identify emerging or evolving threats, especially
those designed to exploit unknown vulnerabilities. In contrast, generative AI,
specifically GDM and GAN, provides an approach to enhance security and privacy
in SAGIN. Specifically, thanks to its ability to model data distribution, GDM
is able to identify inconsistencies in typical data flows, ultimately enabling
it to flag irregular network behavior or security vulnerabilities. Moreover,
GAN composed of dual neural entities operates within security vulnerabilities
assumed by the generator design, while the discriminator enhances its
detection algorithm based on these simulated vulnerabilities. This iterative
process of confrontation gradually enhances the network’s defense capabilities
and ensures improved resiliency against current and emerging cyber threats
[14].

For clarity, the comparison between Traditional methods and Generative AI
methods about SAGIN applications is summarized in Table II.

###  III-B Challenges

As generative AI is integrated into SAGIN, SAGIN’s intelligence and seamless
communication can be further improved. However, as SAGIN involves components
such as ground-based networks, air-based networks and space-based networks,
certain challenges still exist and need to be addressed.

####  III-B1 Real-time Responsiveness

The integration of generative AI into the SAGIN system demands immediate and
responsive processing and decision-making capabilities. Ensuring real-time
response is critical, as any delays can severely compromise the effectiveness
of the fast-changing network. This challenge emphasizes the need to design
systems and algorithms that can quickly process and respond to SAGIN’s diverse
and rapidly changing conditions.

####  III-B2 Integration Overhead

The integration of generative AI into SAGIN introduces additional
computational and communication complexities. The layer of generative AI
technologies, especially sophisticated ones like GANs and VAEs, could lead to
increased overhead, in terms of both computational resources and time.
Efficiently managing this integration overhead is essential to ensure that the
enhanced capabilities do not come at the cost of system performance or the
degradation of network efficiency.

####  III-B3 Scalability Issues

SAGIN’s architecture includes ground to air and space-based components,
presenting scalability challenges when integrating with generative AI. As the
network grows and more devices and nodes are added, the requirement for
generative AI technologies will increase. Ensuring that generative AI
technologies can scale effectively without compromising their performance or
network reliability is critical to the long-term success of generative AI-
enabled SAGIN.

##  IV Generative AI-enabled Channel Information Map Construction

In this section, we propose a framework that utilizes a GDM-based model for
construct channel information map to enhance quality of service.

###  IV-A Motivations

The SAGIN framework integrates different networks in a hierarchical structure.
However, in densely urban landscapes, a significant thing is regarding the
non-line-of-sight (NLOS) signals. In essence, while direct line-of-sight (LOS)
signals from satellites, UAVs, or ground-base station, NLOS signals get
disrupted by urban barriers like high-buildings and huge-bridges. This causes
these signals, especially those from satellites and UAVs, to reflect multiple
times before reaching their target. These multiple reflections introduce
“multipath errors", where the direct and reflected signals have different
arrival times and phases [15]. This discrepancy results in notable position
errors, making it challenging for applications that demand high accuracy of
signal.

Figure 2: A process on channel map architecture, where the GDM classification
method is proposed. In GDM classification, latent space is used for capturing
essential data features while ignoring non-essential variations； Feature
tensor denotes a multi-dimensional array encapsulating the significant
attributes of the data. Predicted Noise denotes the model estimation of
structured noise to be eliminated from the data. Tensor With Noise denotes
that data representation posts incremental noise addition over various time
slots.

###  IV-B Proposed Framework

Addressing the implications and seeking solutions to the challenges posed by
multipath errors in SAGIN are possible through harnessing the potential of
generative AI technologies. Accordingly, as shown in Fig. 2, a GDM-based
channel map framework is proposed, which consists of the following functions.

  * •

Step 1: Image Acquisition: Within the scope of SAGIN, data collection is a
continuous process. Satellites and UAVs are regularly deployed to capture
detailed ground information from both urban and rural landscapes. While
satellites play a major role in capturing overview ground information, UAVs
are capable of providing more detailed local images. Once acquired, the
information, e.g., captured images, are transferred to the processing centers,
providing a foundation for subsequent steps of the framework.

  * •

Step 2: Image Processing: At the processing center, attention should be paid
to the division of areas from the image where the LOS signal is inferred. From
these delineated areas, relevant data such as excess delay, signal paths, and
transmit power of the transmitters (i.e., UAVs or satellites) are extracted
through ground-based receivers. For a comprehensive representation of the
indicators, this data is structured into a graphical network.

  * •

Step 3: Elimination of Redundant Information: In SAGIN, especially under
complex multipath conditions, there is always potential for noise or
superfluous information. Specifically, during image processing, unintended
anomalies can emerge from data redundancy, leading to an environment
representation (EP) that might be cluttered with irrelevant details, hindering
the accurate identification of the target transmitters. To solve this, a
continuous noise reduction mechanism based on a GDM is initiated. This process
refines the diffusion network by removing unnecessary noise, ensuring that the
resulting classification is relevant to the current target.

  * •

Step 4: Zero-shot Classification: Once the data is in place, stable diffusion
density estimation is utilized. These estimates possess the capability to
drive zero-shot classification, removing the necessity for additional
training. The classification is accomplished by discerning patterns and
variations within the network and aligning them with density estimates.

  * •

Step 5: Channel Information Construction: Finally, the channel information map
is constructed to classify each potential LOS signal to improve the
communication and service quality of the entire system.

Figure 3: The simulation results, where the Part A shows about the attention
map, Part B shows the GDM classification process, Part C verifies the validity
of the classification, and the Part D shows the application of using the
channel map.

###  IV-C Case study

As mentioned above, using images taken by UAVs and satellites, ground
receivers are first used to extract corresponding picture information, and
then map construction and channel classification are performed. In order for
the ground receiver to more accurately extract relevant data such as the
direction angle of the transmitter (i.e., UAVs or satellites), the key
semantic information in the images is first extracted, as shown in part A of
Fig. 3. For the left results, the attention image reflects the importance of
image information through different brightness levels, where the more
brightness indicates the greater weight of the corresponding image
information. For instance, the information weight of the road area and roof
area is greater because there is no obstruction between them and the link
information is better. Moreover, the attention map depicted on the right uses
a color scale from blue to green. The dark blue represents the areas with the
least attention, while the bright green represents the areas with the most
attention. By comparing the two results, it is evident that areas with more
brightness in the original image usually correspond to areas with brighter
green in the attention map. This demonstrates the system’s ability to assign
higher weights to specific parts of the image and focus more on areas such as
roads and rooftops, as these areas provide a clearer LOS.

After obtaining the channel data information, it is required to be classified.
In this step, the GDM model is employed for classification, which utilizes the
principles of information diffusion. The process of diffusion classification
is shown in Part B of Fig. 3. Fig. 3 (B) shows the stage scatter plot as
training time changes. Each point is a set of training data, and the root mean
square error (RMSE) is used as the evaluation target. From the first to the
last image, one can clearly see the RMSE gradually decreasing, from the
starting value of 109.32 to only 0.58. This indicates that the accuracy of
model’s predictions significantly improves as the diffusion process proceeds.
Additionally, each point on the scatter plot represents the model prediction
error. At the beginning of the diffusion, these error points are widely
distributed and far away from the red baseline (i.e., 0 represents the LOS
signal). However, as the diffusion steps proceed, these points gradually
converge toward the red baseline, which further verifies the improvement in
model accuracy.

Next, to verify the effectiveness of the proposed GDM classification, we
compare it with several commonly used classification algorithms such as
K-nearest neighbors (KNN) method, extreme gradient boosting (XG-boost) method,
and CNN method, where the results are shown in part C of Fig. 3. The results
show that the proposed GDM classification method is superior to the other
benchmarks in terms of the accurate, F1-score and recall rate. This is because
for high-latitude or complex data, the GDM model can effectively capture the
distribution and complex patterns of the underlying data. Unlike traditional
classification methods, which may be susceptible to noise or data
irregularities, GDM utilizes a diffusion process to gradually transform the
data, ensuring a smoother, more robust representation. This inherent ability
to extract complex data structures allows it to perform better in situations
where other algorithms struggle with overfitting or misclassification. It is
worth noting the performance specifics of the three benchmark algorithms. The
KNN methods, although intuitive, sometimes have difficulty handling high-
dimensional data and are sensitive to the choice of distance metric. The XG-
boost method is a gradient enhancement method, but it is prone to overfitting.
The CNN method works on the principle of local receptive fields and
hierarchical feature extraction, which may not always be optimal for data sets
that lack spatial structure. While each of these methods has its merits and
applications, their limitations in certain scenarios make the GDM’s inherent
ability to extract complex data structures even more commendable. This makes
the GDM model especially valuable in situations where other algorithms might
grapple with overfitting or misclassification.

Finally, based on the classification results, the channel map is constructed.
To further verify the effectiveness of the constructed channel model, part D
of Fig. 3 shows the cumulative reward value obtained with the increase of the
number of iterations using the same PPO-based strategy for different types of
channel models. Here, the corresponding curves are smoothed via a sliding
window to provide a clearer overall trend of the raw results. The results show
that regardless of the types of channel models, the RL-based strategy can
converge with an increment of iterations. Nevertheless, the channel model
constructed using our method can achieve higher performance gains. This is
because the channel constructed by our method more accurately reflects the
channel conditions under better classification conditions, allowing related
resource optimization methods to find more reasonable and feasible solutions,
i.e., the suitable transmit power.

##  V Future Directions

In this section, we outline three main future directions on implementation and
enhancement for generative AI-enabled SAGIN scenario.

###  V-A Multi-modal Integration for Holistic Connectivity

SAGIN brings together various types of data sources (such as images, videos,
audio, and radar scans) from satellites to ground sensors, providing a wealth
of information. With generative AI, it is possible to seamlessly integrate
this multi-modal data to provide a more comprehensive view of SAGIN operating
environment. Through this multi-modal data integration, SAGIN can generate a
comprehensive environmental model, enhancing the network’s ability to make
informed decisions that are enriched with a multitude of data layers and
contextual intelligence.

###  V-B Sustainable and Green Operations

With growing concerns about energy consumption and environmental impact, there
is an urgent need for SAGIN to be both efficient and green. Generative AI
enables networks to autonomously generate configurations that prioritize
energy efficiency. By analyzing past data, current operations, and predicting
future needs, generative AI is able to recommend network architectures that
reduce power consumption, take advantage of renewable energy available in
satellites or ground stations, and minimize resource waste.

###  V-C Support for Generative AI Services in SAGIN

The incorporation of generative AI into SAGIN emphasizes the rich range of
innovative services. generative AI can revolutionize data interpretation
through dynamic visualizations, predictive simulations based on vast datasets,
and enhance user interactions by offering intuitive interfaces. By leveraging
these services, SAGIN’s operational prowess can be greatly magnified, driving
more precise and context-aware decisions across its platform.

##  VI Conclusion

In this article, we explored the integration of generative AI and SAGIN.
Specifically, we first reviewed SAIGN, including ground, air, and space-based
networks. Then, generative AI technologies, such as GAN, VAE, GDM, and TBM
were introduced. Next, certain issues and solutions of generative AI-enabled
SAGIN and potential challenges were presented. To address the challenges, we
proposed a framework that utilizes a GDM-based model for constructing channel
information map to enhance quality of service. The simulation results verified
the effectiveness of the proposed framework. Finally, future directions
regarding generative AI-enabled SAGIN scenarios were summarized.

## References

  * [1] J. Liu, Y. Shi, Z. M. Fadlullah, and N. Kato, “Space-air-ground integrated network: A survey,” _IEEE Commun. Sur. Tut._ , vol. 20, no. 4, pp. 2714–2741, 2018.
  * [2] H. Du, R. Zhang, Y. Liu, J. Wang, Y. Lin, Z. Li, D. Niyato, J. Kang, Z. Xiong, S. Cui, B. Ai, H. Zhou, and D. I. Kim, “Beyond deep reinforcement learning: A tutorial on generative diffusion models in network optimization,” _arXiv preprint 2308.05384_ , 2023.
  * [3] M. Xu, H. Du, D. Niyato, J. Kang, Z. Xiong, S. Mao, Z. Han, A. Jamalipour, D. I. Kim, Xuemin, Shen, V. C. M. Leung, and H. V. Poor, “Unleashing the power of edge-cloud generative ai in mobile networks: A survey of aigc services,” _arXiv 2303.16129_ , 2023.
  * [4] Y. Fu and X. Wang, “Traffic prediction-enabled energy-efficient dynamic computing resource allocation in cran based on deep learning,” _IEEE Open J. Commun. Soc._ , vol. 3, pp. 159–175, 2022.
  * [5] Y. Zeng, X. Xu, S. Jin, and R. Zhang, “Simultaneous navigation and radio mapping for cellular-connected UAV with deep reinforcement learning,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 7, pp. 4205–4220, 2021.
  * [6] M. Karavolos, L. Tsipi, P. S. Bithas, D. Vouyioukas, and P. T. Mathiopoulos, “Satellite aerial terrestrial hybrid NOMA scheme in 6G networks: An unsupervised learning approach,” in _Proc. IEEE 6GNet_ , 2022, pp. 1–5.
  * [7] S. Bano, A. Machumilane, P. Cassarà, and A. Gotta, “How generative models improve LOS estimation in 6G non-terrestrial networks,” _arXiv preprint arXiv:2305.18845_ , 2023.
  * [8] J. Cai, S. Song, H. Zhang, R. Song, B. Zhang, and X. Zheng, “Satellite network traffic prediction based on LSTM and GAN,” in _Proc. IEEE ICIBA_ , vol. 3. IEEE, 2023, pp. 175–178.
  * [9] A. Awasthi, S. Ly, J. Nizam, S. Zare, V. Mehta, S. Ahmed, K. Shah, R. Nemani, S. Prasad, and H. V. Nguyen, “Anomaly detection in satellite videos using diffusion models,” _arXiv 2306.05376_ , 2023.
  * [10] L. Ding, H. Guo, S. Liu, L. Mou, J. Zhang, and L. Bruzzone, “Bi-temporal semantic reasoning for the semantic change detection in HR remote sensing images,” _IEEE Trans. Geosci. Remote Sens._ , vol. 60, pp. 1–14, 2022.
  * [11] B. Tolba, M. Elsabrouty, M. G. Abdu-Aguye, H. Gacanin, and H. M. Kasem, “Massive MIMO CSI feedback based on generative adversarial network,” _IEEE Commun. Let._ , vol. 24, no. 12, pp. 2805–2808, 2020.
  * [12] H. Du, R. Zhang, D. Niyato, J. Kang, Z. Xiong, D. I. Kim, X. S. Shen, and H. V. Poor, “Exploring collaborative distributed diffusion-based ai-generated content (AIGC) in wireless networks,” _IEEE Netw._ , pp. 1–8, 2023.
  * [13] G. Fontanesi, A. Zhu, M. Arvaneh, and H. Ahmadi, “A transfer learning approach for UAV path design with connectivity outage constraint,” _IEEE Internet Things J._ , vol. 10, no. 6, pp. 4998–5012, 2023.
  * [14] I. K. Dutta, B. Ghosh, A. Carlson, M. Totaro, and M. Bayoumi, “Generative adversarial networks in security: A survey,” in _Proc. IEEE UEMCON_ , 2020, pp. 0399–0405.
  * [15] Z. Li, K. Zeng, L. Wang, X. Kan, R. Yuan, and S. Xie, “NLOS/LOS classification by constructing indirect environment interaction from GNSS receiver measurements using a transformer-based deep learning model,” in _in Proc. ITM_ , 2023, pp. 859–870.