Spatio-Temporal Graph Neural Networks for Predictive Learning in Urban
  Computing: A Survey
With recent advances in sensing technologies, a myriad of spatio-temporal
data has been generated and recorded in smart cities. Forecasting the evolution
patterns of spatio-temporal data is an important yet demanding aspect of urban
computing, which can enhance intelligent management decisions in various
fields, including transportation, environment, climate, public safety,
healthcare, and others. Traditional statistical and deep learning methods
struggle to capture complex correlations in urban spatio-temporal data. To this
end, Spatio-Temporal Graph Neural Networks (STGNN) have been proposed,
achieving great promise in recent years. STGNNs enable the extraction of
complex spatio-temporal dependencies by integrating graph neural networks
(GNNs) and various temporal learning methods. In this manuscript, we provide a
comprehensive survey on recent progress on STGNN technologies for predictive
learning in urban computing. Firstly, we provide a brief introduction to the
construction methods of spatio-temporal graph data and the prevalent
deep-learning architectures used in STGNNs. We then sort out the primary
application domains and specific predictive learning tasks based on existing
literature. Afterward, we scrutinize the design of STGNNs and their combination
with some advanced technologies in recent years. Finally, we conclude the
limitations of existing research and suggest potential directions for future
work.
Guangyin Jin, Yuxuan Liang, Yuchen Fang, Zezhi Shao, Jincai Huang, Junbo Zhang, Yu Zheng
# Spatio-Temporal Graph Neural Networks for Predictive Learning in Urban Computing: A Survey

###### Abstract

With recent advances in sensing technologies, a myriad of spatio-temporal data has been generated and recorded in smart cities. Forecasting the evolution patterns of spatio-temporal data is an important yet demanding aspect of urban computing, which can enhance intelligent management decisions in various fields, including transportation, environment, climate, public safety, healthcare, and others. Traditional statistical and deep learning methods struggle to capture complex correlations in urban spatio-temporal data. To this end, Spatio-Temporal Graph Neural Networks (STGNN) have been proposed, achieving great promise in recent years. STGNNs enable the extraction of complex spatio-temporal dependencies by integrating graph neural networks (GNNs) and various temporal learning methods. In this manuscript, we provide a comprehensive survey on recent progress on STGNN technologies for predictive learning in urban computing. Firstly, we provide a brief introduction to the construction methods of spatio-temporal graph data and the prevalent deep-learning architectures used in STGNNs. We then sort out the primary application domains and specific predictive learning tasks based on existing literature. Afterward, we scrutinize the design of STGNNs and their combination with some advanced technologies in recent years. Finally, we conclude the limitations of existing research and suggest potential directions for future work.

 Spatio-Temporal Data Mining, Graph Neural Networks, Urban Computing, Predictive Learning, Time Series

## 1 Introduction

With the rapid advancement of sensing and data stream processing technologies, vast amounts of data in urban systems have been efficiently collected and stored. This has laid the foundation for the era of urban computing, which aims to understand the urban patterns and dynamics from different application domains where the big data explodes, such as transportation, environment, climate, etc. Predictive learning is a typical supervised learning paradigm that learns from historical data to forecast future trends. According to urban computing theories [1], predictive learning based on massive urban data is the most important loop, forming the foundation for intelligent decision-making, scheduling, and management in smart cities. In addition, the predictability of urban big data can also provide the possibility for the development of some new technologies such as digital twin cities and metaverse [2].

The majority of urban data is spatio-temporal, representing that it pertains not only to spatial locations but also changes over time. Within urban systems, spatio-temporal data exhibits ubiquitous properties of _correlation_ and _heterogeneity_[3]. Correlation refers to the data being auto-correlated not only in the temporal dimension, but also in the spatial dimension. Heterogeneity is a property of spatio-temporal data wherein it displays varying patterns across different temporal or spatial ranges, as illustrated in Figure 1. The complex nature of the above characteristics has resulted in an increased difficulty in feature engineering. As a consequence, some methods that performed well in traditional time series forecasting, such as Support Vector Regression (SVR) [4], Random Forest (RF) [5], and Gradient Boosting Decision Tree (GBDT) [6], are less effective in achieving accurate prediction results. In the past decade, the rapid development of deep learning technologies has led to the emergence of hybrid neural networks based on Convolutional Neural Networks (CNN) [7] and Recurrent

Fig. 1: An example of spatio-temporal heterogeneity. (a) Districts with different functionality in an urban network. (b) Statistic of crowd flow data in different nodes from various districts. Despite all nodes exhibiting evident peak patterns, there are notable variations in crowd flow values across nodes located in different districts. Conversely, nodes situated within the same district display similar values even if they differ in their location, _e.g._, the case with node 3&4.
Neural Networks (RNN) [8]. These hybrid networks (_e.g._, ConvLSTM [9], PredRNN [10]) have been increasingly applied to predictive learning of urban spatio-temporal data and have shown significant advantages. However, the major limitation of these methods is their inability to learn directly from non-Euclidean data existing in urban systems, such as vehicle flows over road networks, traffic on route networks, and entities in urban knowledge graphs.

Over the past few years, there have been significant breakthroughs in representation learning of non-Euclidean data through deep learning techniques, particularly Graph Neural Networks (GNN) [11]. This has paved the way for predictive learning of diverse and intricate urban data. Given the spatio-temporal characteristics of urban data, such as traffic flows, a line of studies integrated GNNs with various temporal learning methods to capture dynamics in both space and time dimension [3]. This type of hybrid neural architecture is generally known as **Spatio-Temporal Graph Neural Network (STGNN)**. Recently, STGNNs have been widely used for predictive learning scenarios in urban computing, including transportation, environment, public safety, health, energy, economy, and other fields. Using the search engine of Google Scholar, we perform meticulous keyword searches and tally the relevant paper publications in the past five years. As depicted in Figure 2, we can witness a notable surge in the number of relevant papers on STGNN year by year. In 2018, there are fewer than 20 papers, while in 2022, the number reaches nearly 140. This trend of progress indicates that STGNN-related applications have emerged as a highly sought-after research area in recent years. It is worth noting that a majority of these publications concentrated on predictive learning tasks.

**Related Surveys.** In recent years, there have been a few related surveys on the applications of STGNN-based predictive learning techniques across different fields. Wang et al. [3] conducted a review of deep learning techniques for spatio-temporal data mining involving a series of STGNNs in predictive learning up to 2020. There were also several surveys [12, 13, 14] investigating the blossom of STGNNs in transportation domains. To be specific, [12] analyzed multiple practical problems and revisited related works about prediction, detection, and control problems in urban traffic systems. Bui et al. [13] and Jiang et al. [14] focused on the latest STGNN technologies in traffic forecasting tasks. Gao et al. [15] summarized a wide range of applications using Generative Adversarial Networks (GAN) for spatio-temporal data learning, including some approaches combined with spatio-temporal graph data.

**Our Contributions.** In contrast to prior surveys, the contributions of our survey lie in four aspects:

* To our knowledge, this is the first comprehensive survey to systematically review recent studies that use STGNNs for predictive learning in urban computing. We scrutinize the progress of STGNN from both application and methodology perspectives based on extensive literature.
* We categorize the primary application domains as well as particular predictive learning tasks of STGNNs in urban computing, and sort out a list of public datasets attached with the previous works on STGNNs.
* We provide an in-depth analysis of existing STGNN methods for temporal learning, spatial learning, and spatio-temporal fusion. We further examine some recently emerging approaches that integrated STGNN with other advanced learning frameworks.
* We summarize the challenges shared by STGNNs for predictive learning tasks in urban computing and suggest future directions for addressing these challenging issues.

**Organization.** The rest is organized as follows. Section 2 illustrates how to construct spatio-temporal graphs based on prior knowledge. In Section 3, a taxonomy of STGNNs for predictive learning in urban computing is presented. Section 4 overviews various predictive learning tasks from different domains that can be addressed by STGNNs. Section 5 delineates the fundamental deep learning architectures commonly used in STGNNs. Section 6 and Section 7 delve into an in-depth analysis of the neural architecture design methods of STGNNs and popular advanced techniques that can be combined, respectively. Section 8 further highlights the limitations of existing works and suggests future directions. Finally, we conclude this survey in Section 9.

## 2 Spatio-Temporal Graph Construction

Suppose we obtain some observations from sensors, denoted as \(\mathbf{X}=\{x_{t}\in\mathbb{R}^{N\times F}|t=0,\dots,T\}\), where \(N\) is the number of spatial vertices and \(F\) is the number of features. **Spatio-temporal Graph** is an efficient structure to characterize the relationships between different vertices in a certain spatial and temporal range. We can represent a spatio-temporal graph as \(\mathcal{G}_{t}=(\mathcal{V},\mathcal{E}_{t},\mathbf{A}_{t})\), where \(\mathcal{V}\) is the vertices set, \(\mathcal{E}_{t}\) is the edge set, and \(A_{t}\) denotes the adjacency matrix at time \(t\). In most scenarios, the size of \(\mathcal{V}\) is static, while the size of \(\mathcal{E}_{t}\) can be time-varying or constant, which indicates that \(\mathbf{A}_{t}\in\mathbb{R}^{N\times N}\) also changes with \(\mathcal{E}_{t}\). In terms of connectivity, spatio-temporal graphs can be either directed or undirected, as well as weighted or unweighted. From the perspective of evolution, the structure of spatio-temporal graphs can be either static or dynamic. Figure 3 illustrates the difference between static and dynamic spatio-temporal graphs. The appropriate type of spatio-temporal graph to construct depends on the task and the given data conditions.

Generally, the construction methods of predefined spatio-temporal graphs in urban computing systems can be divided into four categories: topology-based, distance-based, similarity-based, and interaction-based.

Fig. 2: The publication trend of STGNN-related papers in Google Scholar over the past five years. The blue bars represent the total number of relevant publications and the red bars denote those focusing on predictive learning tasks.
**Topology-based graph:** In the context of urban systems, topology-based graphs are usually constructed based on given topology structures, such as road networks [16, 17]. The adjacency matrix of a topology-based graph can be formulated as:

\[a_{ij}^{t}=\begin{cases}1,\text{if }v_{i}\text{ connects to }v_{j}\\ 0,\text{ otherwise}\end{cases}, \tag{1}\]

where \(a_{ij}^{t}\) denotes an element in adjacency matrix at time \(t\), \(v_{i}\) and \(v_{j}\) are different vertices in the graph. Since the connections in topology structures can be symmetrical or asymmetrical, the topology-based graphs can be directed or undirected. Topology only represents connections in non-Euclidean spaces, thus the topology-based graphs are unweighted. In addition, the topology structures in social systems are usually fixed for quite a long time, so we can treat them as static graphs.

**Distance-based graph:** According to first law of geography, _i.e._, "Everything is related to everything else, but near things are more related to each other", we can construct a distance-based graph when a predefined topology is absent. In most applications, the elements in the adjacency matrix are calculated using a kernel function that takes the distances into account [18, 19, 20]. Gaussian radial basis function and inverted function are two common kernel functions used in previous literature. For example, the adjacency matrix of a distance-based graph with gaussian radial basis can be computed as:

\[a_{ij}^{t}=\begin{cases}\dfrac{\exp(-\left\|d_{ij}^{t}\right\|_{2})}{\sigma}, \text{if }d_{ij}^{t}<\epsilon,\\ 0,\quad\text{ otherwise}\end{cases} \tag{2}\]

where \(d_{ij}^{t}\) denotes the distance between node \(i\) and node \(j\) at time \(t\); \(\epsilon\) is a predefined threshold to control the sparsity of the adjacency matrix; \(\sigma\) is a hyperparameter to control the distribution.

**Similarity-based graph:** Similarity can provide insights into the relations between different entities from a semantic perspective. Similarity-based graphs can be constructed based on either the proximity of time series [21, 22, 23] or similarity of the spatial attribute, _e.g._, Point of Interest (POI) [24]. In scenarios where additional data is unavailable, similarity-based graphs are typically constructed based on the similarity of time series. Pearson Correlation Coefficient (PCC) and Dynamic Time Wrapping (DTW) are two prevalent methods used to calculate the similarity between time series. For instance, the adjacency matrix of a similarity-based graph computed by PCC is defined as:

\[a_{ij}^{t}=\begin{cases}\dfrac{\sum_{i=1}^{n}\left(x_{i}^{0:t}-x_{i}^{\bar{0}: t}\right)\left(x_{j}^{0:t}-x_{j}^{\bar{0}:t}\right)}{\sqrt{\sum_{i=1}^{n} \left(x_{i}^{0:t}-x_{i}^{\bar{0}:t}\right)^{2}}}\sqrt{\sum_{i=1}^{n}\left(x_{ j}^{0:t}-x_{j}^{\bar{0}:t}\right)^{2}}},\\ 0,\quad\text{ otherwise}\end{cases} \tag{3}\]

where \(x_{i}^{0:t}\) and \(x_{j}^{0:t}\) represent the time series of node \(i\) and node \(j\) of a given time span \(t\), respectively; \(x_{i}^{0:t}\) and \(x_{j}^{\bar{0}:t}\) are the mean value of the time series of node \(i\) and node \(j\), and \(n\) denotes the number of samples over the time span \(t\).

**Interaction-based graph:** The interaction between different locations can express their connection from the perspective of information flow [18, 20]. This is especially important when representing the characteristics of mobility, as the proportion of flow between two nodes can indicate the strength of their connection. Hence, the adjacency matrix of an interaction-based graph can be written as:

\[a_{ij}^{t}=\begin{cases}\dfrac{F_{ij}^{t}}{\sum_{m\in N(i)}F_{im}^{t}},\;if \;F_{ij}^{t}>0\\ 0,\quad\text{ otherwise}\end{cases}, \tag{4}\]

where \(F_{ij}^{t}\) denotes the flow from node \(i\) to node \(j\) at time \(t\); \(N(i)\) indicates the set of nodes that interact with node \(i\); \(F_{im}^{t}\) is the flow from node \(i\) to other nodes (_e.g._, \(m\)) in a set \(N(i)\) at time \(t\).

In addition to the common predefined graph construction methods mentioned above, many relations in urban systems are implicit and difficult to be directly predefined. Therefore, spatio-temporal graphs based on adaptive learning have been proposed in some recent works. More details about these methods can be found in Section 6.1.2.

## 3 Taxonomy

This section provides a taxonomy of STGNNs for predictive learning in urban computing, which is also a generalization of our follow-up content. As shown in Figure 4, there are four main parts in our survey that need to be highlighted: main application domains, basic spatio-temporal learning neural architecture, improved spatio-temporal learning methods, and advanced methods combined with STGNNs. We present an overview of specific predictive learning tasks based on major application domains in Section 4. In Section 5, we review the fundamental neural architectures of STGNNs from three perspectives: spatial learning, temporal learning and spatio-temporal fusion. Subsequently, in Section 6, we examine the enhanced spatio-temporal dependency learning methods from the same perspectives as in Section 5. Finally, we discuss the advanced techniques combined with STGNNs in Section 7.

Fig. 3: The schematic diagram of static and dynamic spatio-temporal graphs. The color shades of the nodes represent different features.
## 4 Application Domains & Task Description

This section delves into the primary application domains and specific predictive learning tasks in urban computing. Based on the available literature in recent years, we conducted a statistical analysis of the various application domains of STGNN in urban computing. Figure 5 illustrates the main application domains of STGNN, which encompass transportation, safety, environment, and public health. Among these, transportation is the most widely studied application domain of STGNN, constituting over 60% of the existing literature.

### _Transportation_

Modern urban systems have numerous sensors distributed across traffic road networks and critical regions to monitor changing traffic states, such as flow and speed. The objective of traffic state prediction is to forecast future traffic states based on historical traffic states within a particular spatial range. As shown in Figure 6, traffic state prediction can be divided into two main categories:

* **Network-based prediction.** The object of network-wide prediction is usually the traffic flow or speed on the given road networks [25, 26, 27, 28, 16, 19]. The basic graph structures can be directly converted from road networks in most previous works.
* **Region-based prediction.** This task aims to forecast the traffic (_e.g._, crowd flow) in urban areas [29, 30, 31, 32]. In this case, the whole urban area is partitioned into irregular or regular regions, and a spatio-temporal graph can be constructed based on the distances, connectivity, semantic correlations between different regions, etc.

In general, traffic state prediction tasks can be summarized in the following form:

\[\left[\mathbf{X}^{\left(t-T^{\prime}+1\right)},\cdots,\mathbf{X}^{\left(t\right)}; \mathcal{G}\right]\overset{f(\cdot)}{\longrightarrow}\left[\mathbf{X}^{\left(t+1 \right)},\cdots,\mathbf{X}^{\left(t+T\right)}\right], \tag{5}\]

where \(\mathbf{X}^{\left(t\right)}\in\mathbb{R}^{N\times d}\) denotes the traffic states of \(N\) vertices at time step \(t\), \(\mathcal{G}\) is the constructed graph structure, \(f(\cdot)\) is the corresponding STGNN model for making predictions.

Fig. 4: The taxonomy for STGNN in our survey.

Fig. 5: The summary of the different application domains of STGNN in urban computing.

Fig. 6: The two categories of traffic state prediction.
#### 4.1.1 Traffic Demand Prediction

Accurately predicting changing urban traffic demand patterns (_e.g._, taxi order demands, rail transit passenger demands, and sharing bike demands) in various regions can facilitate traffic scheduling to alleviate congestion during peak hours. Traffic demands can be broadly categorized into three main types: origin demands, destination demands, and origin-destination (OD) demands. Predicting origin demands and destination demands are similar to region-based traffic state prediction, _i.e._, forecasting future demands based on historical demands in \(N\) regions [20, 24, 33, 34]. However, OD demand prediction is somewhat distinct, requiring prediction of future origin-destination matrices using historical OD matrices [35, 36, 37, 38, 39]. To be specific, the outputs of OD demand prediction are a series of matrices with size \(N\times N\), which can characterize the flow demand among these region pairs.

#### 4.1.2 Traffic Incident Prediction

With the dramatic increase in the number of vehicles, more and more traffic incidents such as congestion and accidents have occurred, placing significant pressure on urban traffic management. The aim of the traffic incident prediction task is to predict some important properties (_e.g._, occurrence probability, occurrence time) of these incidents that may occur on road networks [40, 41, 42, 43, 44]. In addition to differences in the objects being predicted, similar to the traffic state prediction task, accurate traffic incident prediction also requires capturing spatio-temporal dependencies on road networks by building STGNN models. Compared with the relatively macroscopic traffic state prediction, the incident-oriented prediction can more accurately respond to various emergencies in the traffic system for early warning.

#### 4.1.3 Travel Time Prediction

Travel time prediction is highly valued in industries, especially in online map navigation and ride-hailing software. Accurate travel time estimation can significantly enhance the user experience with these types of software. This task aims to predict the travel time of a given trajectory using historical traffic states on road networks. In order to predict travel time more accurately, not only the characteristics of the trajectory itself need to be considered, but also the spatio-temporal dynamics (_e.g._, flow, speed, etc.) attached to the road networks should be addressed. Under this circumstance, spatio-temporal graphs are established based on road networks for this task. So far, large technology companies such as Baidu [45, 46], Google [47], and DiDi [48] have developed practical travel time prediction functions on their online platforms. STGNN-based travel time prediction can be defined as follows:

\[\mathcal{F}(P_{t}|X_{t-w:t},\mathcal{G})\to T_{g},T_{l} \tag{6}\]

where \(P_{t}\) denotes the given trajectory with departure time \(t\), \(X_{t-w:t}\) denotes the spatio-temporal features in historical time window \(w\) attached to the given road network \(\mathcal{G}\). \(T_{g}\) and \(T_{l}\) respectively represent the global travel time of the entire trajectory and local travel time of the road segment.

#### 4.1.4 Trajectory Prediction

Trajectory prediction is a crucial task for comprehending the intricate group dynamics of humans and vehicles [49, 50, 51, 52, 53, 54], which can foster advancements in autonomous driving and urban monitoring technologies. As shown in Figure 7, there exists some correlations or interactions in the movement patterns of agents in the group, thus we can build spatio-temporal graphs based on the relations between different agents within a group. Upon creating these spatio-temporal graphs, STGNN models can be devised to predict the coordinates that agents may occupy in the future, considering their historical traversal coordinates, thereby facilitating the predictions of future trajectories.

#### 4.1.5 Other prediction tasks

Aside from the mainstream transportation application scenarios mentioned earlier, there are also some relatively niche scenarios that use STGNN techniques to improve prediction outcomes. For example, parking availability prediction [55, 56] and traffic delay prediction [57, 58] are two emerging tasks in transportation management that can be addressed with STGNN. Analogous to other prevalent research topics, these prediction tasks adopt STGNNs to better learn spatio-temporal contextual representations on traffic networks, yielding more accurate predictions.

### _Environment_

#### 4.2.1 Air Quality Prediction

Air quality has become a pressing issue that needs immediate attention and improvement. Accurate air quality prediction can not only assist governments in formulating energy-saving and emission-reduction policies but also provide guidance for residents' outdoor activities. Air quality index (AQI), PM2.5, and emissions are the indicators we are among the most significant indicators of concern. The related data are collected by city-level or national-level monitoring stations [59, 60]. Due to the fluidity of the air, monitoring stations that are geospatially close or sharing the same wind direction may collect correlated results [61, 62, 63]. Hence, utilizing spatio-temporal graph-based deep learning models can not only establish such spatial dependencies, but also capture the time-varying dynamics of air quality.

Fig. 7: An example of social interactions between different agents within a group [49].
#### 4.2.2 Meteorological Prediction

Meteorological forecasting is another research topic intimately connected to the environment and human society. Similar to air quality data, meteorological data are also collected by distributed monitoring stations. However, the correlations between different stations could be more complex and susceptible to a greater number of factors. In recent years, STGNN-based approaches have been progressively applied in various meteorological prediction scenarios such as temperature prediction [64, 65, 66], frost prediction [67] and wind prediction [68, 69, 70], showcasing their superior performance in practice.

### _Public Safety_

#### 4.3.1 Crime Frequency Prediction

Effectively combating and preventing crime is the foundation for ensuring urban safety. Accurate prediction of crime frequency can assist governments in understanding real-time crime dynamics and allocating police resources rationally. Most existing work in this research line focus on crime frequency prediction in urban areas. Given that different urban regions have distinct functions, POI, and other characteristics, these factors could contribute to varying crime types and trends. However, regions with similar characteristics or close distances may exhibit latent correlations in crime incidents [71, 72, 73]. Consequently, many previous studies [74, 75, 76, 77, 78, 79, 80, 71, 72, 71] have introduced a series of STGNN to capture these correlations to reduce the prediction errors.

#### 4.3.2 Disaster Situation Prediction

Natural disasters (_e.g._, earthquakes) have posed serious challenges to the safety of human society since ancient times. Accurate disaster situation prediction can enable governments to implement disaster prevention measures, allocate disaster relief materials, and evacuate residents in a timely manner. To model correlated and heterogeneous features across geographical locations, STGNN can be a fruitful approach in this task. Currently, some literature has introduced the STGNN models into scenarios such as flood prediction [82, 83], fire prediction [84, 85], typhoon forecasting [86, 87, 88] and earthquake prediction [89, 90, 91].

### _Public Health_

#### 4.4.1 Epidemic Prediction

Epidemics are one of the greatest challenges to the public health systems, especially the novel coronavirus that has been prevalent in recent years, which has caused more than six million deaths worldwide. Therefore, accurately predicting the spread of epidemics is an important but challenging task, which can provide data support for the strengthening strategy of the urban public health systems. Some recent existing works have employed STGNN models to address the national-level [92, 93, 94, 95, 96, 97] or international-level [98] epidemic prediction tasks. Many of them combine the mathematical formulations of epidemic dynamics and the modeling of spatio-temporal graphs, which have achieved better prediction results than traditional methods [97, 98, 99, 100, 101].

#### 4.4.2 Ambulance Demand Prediction

In today's aging society, the allocation of ambulance resources is a challenging task that needs careful consideration. Accurate ambulance demand prediction can effectively alleviate the burden on the urban healthcare systems. Since there could be time-varying correlations in public medical resources, traffic conditions, and demand patterns among different regions of the social systems, STGNN-based methods have increasingly been exploited to learn these multi-view spatial correlations in recent years [102, 103, 104].

### _Other Application Domains_

In addition to the four main application domains mentioned above, other scenarios where the spatio-temporal graph structures can be established based on the intrinsic relations of data are potential areas for the development of STGNN-based predictive learning models. In recent years, STGNN-based predictive learning models have also been promoted to other domains such as energy, economy, finance, and production. In the energy domain, STGNN models have been utilized in wind power prediction [105, 106] and photovoltaic power prediction [107, 108]. In the economy domain, a typical application is nation-level regional economy prediction, when researchers have explored the usage of STGNN models [109, 110]. In the finance domain, STGNN models have been widely applied for stock prediction [111, 112, 113]. In the production domain, Fan et al. first adopted STGNN model in predicting crop yield [114].

### _Open Datasets in Main Application Domains_

As depicted in Table I, we have compiled a list of some of the most frequently used public datasets from previous works in the primary application domains, including their details such as source links and related publications. For example, California-PEMS is a set of real-world traffic datasets used for traffic prediction tasks. These datasets are collected from the California Department of Transportation's Performance Measurement System (PeMS) and are commonly used to evaluate and benchmark various traffic prediction models. The PEMS datasets are named according to the number of sensor stations in each dataset, such as PEMS-04 and PEMS-08. These datasets are widely used in the research field of transportation and urban mobility, particularly for traffic state prediction. Due to their high granularity, realistic nature, and real-world applicability, they serve as valuable resources for researchers working on spatio-temporal forecasting and traffic modeling.

## 5 Basic Neural Architectures

Here we introduce basic neural architectures for STGNNs. As shown in Figure 8, the basic framework of STGNNs for predictive learning contains three main modules: Data Processing Module (DPM), Spatio-Temporal Graph Learning Module (STGLM) and Task-Aware Prediction Module (TPM). For predictive learning tasks in urban computing, DPM is responsible for constructing the spatio-temporal graph data from the raw data; STGLM seeks to capture hidden spatio-temporal dependencies from complex social systems, while TPM aims to map the spatio-temporal hidden
representations from STGLM into the space of downstream prediction tasks. STGLM serves as the most vital component of STGNNs, which usually combines spatial learning networks and temporal learning networks organically through a certain spatio-temporal fusion method. Spatial learning networks may utilize spectral graph convolutional networks (Spectral GCNs) [151], spatial graph convolutional networks (Spatial GCNs) [11, 152], and graph attention networks (GATs) [153] as potential options. Temporal learning networks, on the other hand, may incorporate recurrent neural networks (RNNs), temporal convolutional networks (TCNs), or temporal self-attention networks (TSANs). Compared with STGLM, TPM is a relatively simple neural network, thus the majority of existing research focuses on the design of the neural architectures in STGLM.

### _Graph Neural Networks_

Graph neural networks (GNNs) are fruitful tools for learning spatial dependencies in non-Euclidean space. In recent years, popular GNNs can be divided into three categories: spectral GCNs, spatial GCNs and GATs.

#### 5.1.1 Spectral Graph Convolutional Network

Initially, most GNNs were based on Fourier transform, which converts the graph signal in the spatial domain into the spectral domain to conduct convolution calculations [154]. In this approach, Graph Fourier transform and Inverse Graph Fourier transform are required to achieve the transformation between the spatial domain and spectral domain, and are defined as:

\[\mathcal{F}(\mathbf{x})=\mathbf{U}^{T}\mathbf{x}, \tag{7}\] \[\mathcal{F}^{-1}(\mathbf{x})=\mathbf{U}\mathbf{x},\]

where \(\mathbf{U}\) denotes the matrix of eigenvectors of the normalized graph Laplacian. Based on this, the graph convolution operation is defined as:

\[\mathbf{g}\star\mathbf{x}=\mathcal{F}^{-1}(\mathcal{F}(\mathbf{g})\odot\mathcal{F}(\mathbf{x} ))=\mathbf{U}\left(\mathbf{U}^{T}\mathbf{g}\odot\mathbf{U}^{T}\mathbf{x}\right), \tag{8}\]

where \(\odot\) is the convolution operator and \(\mathbf{U}^{T}\mathbf{g}\) denotes the filter in the spectral domain. Eq. 8 can be further simplified as:

\[\mathbf{g}_{w}\star\mathbf{x}=\mathbf{U}\mathbf{g}_{w}\mathbf{U}^{T}\mathbf{x}. \tag{9}\]

Most of the subsequent GNNs based on the spectral domain mainly improve the calculation method of \(\mathbf{g}_{w}\). For example, ChebNet is one of the most popular Spectral GNN methods. According to the theory that \(\mathbf{g}_{w}\) can be approximated by a truncated expansion of Chebyshev polynomials [155], Defferrard et al. [151] further propose ChebNet, which can be formulated as:

\[\begin{split}&\mathbf{\tilde{L}}=\frac{2}{\lambda_{\max}}\mathbf{L}- \mathbf{I}_{N},\\ &\mathbf{g}_{w}\star\mathbf{x}=\sum_{k=0}^{K}w_{k}\mathbf{T}_{k}(\mathbf{\tilde{ L}})\mathbf{x},\end{split} \tag{10}\]

where \(\mathbf{\tilde{L}}\) is the normalized graph Laplacian, \(\lambda_{\max}\) is the largest eigenvalue of \(\mathbf{L}\), \(\mathbf{T}_{k}(x)\) denotes the Chebyshev polynomials up to \(k_{th}\) order, \(w_{k}\) denotes a vector of Chebyshev

\begin{table}
\begin{tabular}{|l|l|l|l|} \hline Domain & Dataset & Link & Reference \\ \hline \multirow{10}{*}{Transportation} & California-PEMS & [http://pems.dot.ca.gov/](http://pems.dot.ca.gov/) & [16, 12, 115, 116, 117, 118, 119] \\ \cline{2-4}  & METR-LA & [https://www.metro.net/](https://www.metro.net/) & [120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 21, 21, 22, 22, 23, 24, 25, 26, 27, 28, 29, 22, 24, 27, 29, 25, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 41, 43, 45, 44, 46, 47, 49, 42, 43, 47, 48, 49, 40, 43, 49, 40, 45, 41, 42, 44, 45, 46, 47, 48, 49, 42, 45, 49, 43, 46, 47, 48, 49, 40, 41, 45, 42, 46, 49, 44, 45, 46, 47, 48, 49, 40, 41, 46, 49, 42, 45, 47, 49, 43, 48, 41, 42, 45, 46, 47, 49, 45, 48, 42, 46, 49, 40, 47, 49, 41, 48, 42, 49, 43, 44, 45, 46, 47, 48, 49, 41, 42, 49, 44, 45, 46, 47, 49, 42, 48, 43, 44, 49, 44, 45, 46, 47, 48, 49, 40, 49, 41, 42, 43, 44, 46, 49, 45, 47, 48, 49, 42, 49, 43, 44, 46, 49, 45, 48, 49, 46, 47, 49, 48, 49, 49, 40, 41, 42, 43, 44, 45, 46, 47, 49, 48, 49, 40, 42, 49, 41, 43, 44, 45, 46, 49, 42, 45, 47, 49, 43, 44, 46, 47, 48, 49, 45, 49, 46, 47, 49, 48, 49, 49, 40, 41, 42, 43, 44, 45, 46, 49, 41, 43, 45, 47, 49, 42, 46, 47, 48, 49, 45, 49, 46, 48, 49, 47, 49, 48, 49, 49, 40, 49, 41, 42, 49, 45, 49, 46, 49, 47, 48, 49, 49, 40, 42, 49, 41, 45, 49, 46, 47, 49, 48, 49, 49, 41, 45, 49, 42, 47, 49, 42, 48, 49, 43, 49, 45, 46, 47, 49, 48, 49, 49, 40, 49, 42, 49, 43, 44, 45, 49, 46, 47, 48, 49, 49, 40, 43, 41, 45, 49, 41, 46, 47, 49, 42, 48, 49, 45, 49, 46, 47, 49, 48, 49, 49, 42, 49, 45, 49, 46, 49, 47, 48, 49, 49, 40, 49, 41, 45, 49, 42, 49, 43, 46, 49, 47, 48, 49, 49, 40, 45, 49, 41, 42, 49, 45, 46, 47, 49, 48, 49, 49, 40, 49, 42, 49, 41, 43, 45, 49, 46, 47, 49, 49, 41, 45, 47, 49, 48, 49, 49, 40, 42, 49, 43, 41, 45, 49, 42, 49, 45, 46, 47, 48, 49, 41, 49, 42, 49, 45, 47, 49, 42, 48, 49, 43, 49, 45, 46, 49, 47, 49, 48, 49, 49, 40, 49, 45, 49, 41, 46, 49, 42, 49, 45, 47, 49, 46, 48, 49, 41, 49, 42, 49, 45, 49, 46, 49, 47, 48, 49, 49, 40, 49, 45, 49, 40, 46, 47, 48, 49, 49, 40, 47, 49, 48, 49, 49, 40, 49, 40, 49, 41, 45, 49, 40, 49, 41, 42, 49, 41, 45, 49, 42, 49, 43, 41, 46, 49, 41, 45, 49, 46, 47, 48, 49, 49, 40, 47, 49, 48, 49, 49, 40, 49, 41, 45, 49, 42, 49, 40, 49, 42, 49, 40, 49, 41, 42, 49, 43, 41, 45, 49, 42, 49, 45, 46, 47, 48, 49, 41, 47, 49, 48, 49, 40, 49, 49, 40
coefficients. ChebNet eliminates the necessity of computing the eigenvectors of the Laplacian by employing the K-localized graph convolution.

#### 5.1.2 Spatial Graph Convolutional Network

While spectral graph convolutional networks (GCNs) have made significant advancements, their primary limitation lies in their dependence on the graph Laplacian matrix. Whenever there is a change in the underlying graph structure, the graph Laplacian matrix must be recomputed, rendering spectral GCNs better suited to scenarios where the graph structure remains constant. To overcome the dependency on the graph Laplacian matrix, Kipf et al. simplify the graph convolution operation [11] by performing message passing in the spatial domain. We called this new form as spatial GCNs, which is defined as:

\[\mathbf{g}_{w}\star\mathbf{x}=w\left(\mathbf{I}_{N}+\mathbf{D}^{-\frac{1}{2}}\mathbf{A}\mathbf{D}^{- \frac{1}{2}}\right)\mathbf{x}, \tag{11}\]

where \(\mathbf{A}\) is the adjacency matrix; \(\mathbf{D}\) is the degree matrix; \(w\) is the learnable parameters in the spatial GCN.

However, the above spatial GCN has to take the full graph as input, which makes it hard to scale up to large graphs in practice. To address this problem, GraphSAGE [152] proposes a sampling aggregation approach to achieve flexible inductive learning on large graphs. The aggregation operator in GraphSAGE is formulated as:

\[\mathbf{h}_{\mathcal{N}(u)}^{k}\leftarrow\mathrm{Aggregate}_{k}\left( \left\{\mathbf{h}_{u^{\prime}}^{k-1},\forall u^{\prime}\in\mathcal{N}_{k}(u) \right\}\right), \tag{12}\] \[\mathbf{h}_{u}^{k}\leftarrow\sigma\left(\mathbf{W}^{k}\cdot\mathrm{Concat }\left(\mathbf{h}_{u}^{k-1},\mathbf{h}_{\mathcal{N}(u)}^{k}\right)\right),\]

where \(\mathcal{N}_{k}(u)\) denotes the set of neighbor nodes of \(u\); \(\mathbf{h}_{\mathcal{N}(u)}^{k}\) means the embedding of node \(u\) after aggregation operation.

#### 5.1.3 Graph Attention Network

To account for the importance of neighbor nodes in learning spatial dependencies, GAT [153] integrates the attention mechanism into the node aggregation operation as:

\[\mathbf{h}_{v}^{t+1}=\rho\left(\sum_{u\in\mathcal{N}_{v}}\alpha_{vu}\mathbf{W}\mathbf{h}_{ u}^{t}\right), \tag{13}\]

\[\alpha_{vu}=\frac{\exp\left(\mathrm{LeakyReLU}\left(\mathbf{a}^{T}\left[\mathbf{W} \mathbf{h}_{v}\|\mathbf{W}\mathbf{h}_{u}\right]\right)\right)}{\sum_{k\in\mathcal{N}_{u}} \exp\left(\mathrm{LeakyReLU}\left(\mathbf{a}^{T}\left[\mathbf{W}\mathbf{h}_{v}\|\mathbf{W} \mathbf{h}_{k}\right]\right)\right)},\]

where \(\alpha_{vu}\) denotes the attention scores of neighbor node \(u\) to the central node \(v\), \(W\) is the weight matrix associated with the linear transformation for each node, and \(\mathbf{\mathrm{a}}\) is the weight parameter for attention output. To further stabilize the computational process of attention, the multi-head trick form also be introduced in GAT:

\[\mathbf{h}_{v}^{t+1} =\|_{k=1}^{K}\sigma\left(\sum_{u\in\mathcal{N}_{v}}\alpha_{vu}^{k} \mathbf{W}_{k}\mathbf{h}_{u}^{t}\right), \tag{14}\] \[\mathbf{h}_{v}^{t+1} =\sigma\left(\frac{1}{K}\sum_{k=1}^{K}\sum_{u\in\mathcal{N}_{v}} \alpha_{vu}^{k}\mathbf{W}_{k}\mathbf{h}_{u}^{t}\right),\]

where \(\alpha_{vu}^{k}\) is the normalized attention score computed by the \(k_{th}\) attention head. The aggregation method for multiple attention heads can be achieved by concatenation or an averaging operation.

### _Recurrent Neural Networks_

Recurrent neural networks (RNNs) are a class of deep neural networks for sequential learning based on recursive computations, and have found extensive applications in time series modeling. However, the vanilla version of RNNs is subject to a significant limitation - the gradient vanishing or explosion problem during the training process [156]. In response to this challenge, two of the most prominent variants of RNNs, _i.e._, Long Short-Term Memory (LSTM) [157] and Gated Recurrent Units (GRU) [158], have been proposed.

#### 5.2.1 Long-Short Term Memory Network

To address the gradient vanishing/explosion problem, LSTM first introduces a new gated mechanism to control the information flow by selectively retaining and forgetting temporal information in each time step. The formulation of LSTM is defined as:

\[\mathbf{f}_{t} =\sigma\left(\mathbf{W}_{f}\cdot\left[\mathbf{h}_{t-1},\mathbf{x}_{t}\right] +b_{f}\right), \tag{15}\] \[\mathbf{i}_{t} =\sigma\left(\mathbf{W}_{i}\cdot\left[\mathbf{h}_{t-1},\mathbf{x}_{t}\right] +b_{i}\right),\] \[\mathbf{o}_{t} =\sigma\left(\mathbf{W}_{o}\cdot\left[\mathbf{h}_{t-1},\mathbf{x}_{t}\right] +b_{o}\right),\] \[\tilde{\mathbf{C}}_{t} =\tanh\left(\mathbf{W}_{C}\cdot\left[\mathbf{h}_{t-1},\mathbf{x}_{t}\right] +b_{C}\right),\] \[\mathbf{C}_{t} =\mathbf{f}_{t}\ast\mathbf{C}_{t-1}+\mathbf{i}_{t}\ast\mathbf{\tilde{C}}_{t},\] \[\mathbf{h}_{t} =\mathbf{o}_{t}\ast\tanh\left(\mathbf{C}_{t}\right),\]

where \(\mathbf{f}_{t}\) represents the forgetting gate, whose function is to discard historical information in a certain proportion. \(\mathbf{i}_{t}\) denotes the input gate, which is to update the information of the current time step. \(\mathbf{o}_{t}\) is the output gate, which is to control the current output information in a certain ratio. \(\sigma(\cdot)\) represents the sigmoid function, so the output values of the forget gate, input gate and output gate are all between 0 and 1. \(\mathbf{C}_{t}\) represents the candidate state of the current LSTM unit. \(\mathbf{C}_{t}\) represents the state of the LSTM unit after the calculation of the forget gate and the input gate, where the forget gate acts on the state of the LSTM unit of the previous time step and the input gate acts on the state of the current candidate LSTM unit. Finally, the hidden state of this time step is obtained by the output gate.

#### 5.2.2 Gated Recurrent Unit Network

Due to the integration of multiple gates at each time step, LSTM bears a comparatively high computational burden. To this end, GRU simplifies the gates of each LSTM unit into two: an update gate and a reset gate. \(\mathbf{u}_{t}\) represents the update gate, which determines how to combine the information of the new input time step with the memory of the previous time step. \(\mathbf{r}_{t}\) represents the reset gate, which defines the amount of memory reserved from the previous time step to the current time step. Although the learnable parameters of GRU are streamlined, its performance can be compared with LSTM in previous works, while improving the training and inference efficiency. The calculation process of GRU is defined as follows:

\[\mathbf{u}_{t} =\sigma(\mathbf{W}_{u}\cdot x_{t}+\mathbf{U}_{u}\cdot\mathbf{C}_{t-1}+b_{u}), \tag{16}\] \[\mathbf{r}_{t} =\sigma(\mathbf{W}_{r}\cdot x_{t}+\mathbf{U}_{r}\cdot\mathbf{C}_{t-1}+b_{r}),\] \[\tilde{\mathbf{C}}_{t} =\tanh(\mathbf{W}_{C}\cdot x_{t}+\mathbf{U}_{C}(\mathbf{r}_{t}\odot\mathbf{C}_{t-1} )+b_{C}),\] \[\mathbf{C}_{t} =\mathbf{u}_{t}\odot\mathbf{C}_{t-1}+(1-\mathbf{u}_{t})\odot\tilde{\mathbf{C}}_{t}.\]
### _Temporal Convolutional Networks_

RNNs have been extensively applied for temporal learning in many spatio-temporal tasks, but their disadvantage is readily apparent: the recurrent structures necessitate the computation of sequences at every time step, leading to a substantial increase in computational cost and a consequent decrease in model efficiency. In contrast, Temporal Convolutional Networks (TCN) with their parallel 1D-CNN structures can address this problem effectively. Similar to 2D-CNN applied in image recognition, 1D-CNN also operates and aggregates features through convolution kernels. The major difference is that the convolution kernel of 1D-CNN is one-dimensional and only slides on the time axis.

#### 5.3.1 Gated Temporal Convolutional Network

Inspired by the gated mechanism in LSTMs and GRUs, we can also integrate it with pure 1D-CNN architecture to enhance the capability of temporal learning. We called this hybrid neural architecture a gated temporal convolutional network (Gated-TCN) [159]. The calculation process of Gated-TCN is defined as follows:

\[F(x)=tanh(\mathbf{\Theta_{1}\star}x)\odot\sigma(\mathbf{\Theta_{2}\star}x), \tag{17}\]

where \(\mathbf{\Theta_{1}}\) and \(\mathbf{\Theta_{2}}\) represent the learnable parameters of the convolution kernel in two different 1D-CNNs, respectively; \(\star\) denotes the convolution operation; \(\odot\) is the element-wise multiplication mechanism; \(\sigma(\mathbf{\Theta_{2}\star}x)\) indicates the gating unit, which is utilized to control the utilization rate of historical information.

#### 5.3.2 Causal Temporal Convolutional Network

While TCN is an efficient parallel neural architecture for sequential learning, it violates the temporal order of spatio-temporal graph data. Compared to standard TCNs, Causal TCNs, which are proposed in Wavenet [160], offer the additional benefit of explicitly modeling the causal nature of temporal data. This is achieved by removing connections between future time steps and past time steps, which eliminates the possibility of data leakage from future time steps to past time steps, as depicted in Figure 9. Furthermore, in order to more effectively capture longer-range temporal dependencies, 1D-CNN with dilated factors [161] can be used. By increasing the dilated factors layer by layer, this model has the capacity to learn temporal dependencies from a short range to a long range. A Causal TCN with dilated factors can be expressed as:

\[F(s)=\left(\boldsymbol{x}*_{d}f\right)(s)=\sum_{i}^{k-1}f(i)\cdot x_{s-d\cdot i}, \tag{18}\]

where \(s\) denotes the time series input; \(d\) represents the dilation factor, and the ordinary convolution operator is a special case of the dilated convolution operator when \(d=1\); \(s-d\cdot i\) refers to the positioning of certain historical information.

### _Temporal Self-Attention Networks_

Self-attention networks represent a highly effective approach for capturing long-range temporal relationships among different time steps, with the most prominent example being the Transformer model [162]. The Transformer comprises three primary components: a scaled dot-product attention network, a feed-forward network, and position encodings, as illustrated in Figure 10.

The scaled dot-product network is the core part of Transformers, in which the attention calculation is formulated as:

\[\mathrm{Attention}(\boldsymbol{Q},\boldsymbol{K},\boldsymbol{V})=\mathrm{ Softmax}\left(\frac{\boldsymbol{Q}\boldsymbol{K}^{T}}{\sqrt{\boldsymbol{d}_{k}}} \right)\boldsymbol{V}, \tag{19}\]

where queries \(\boldsymbol{Q}\), keys \(\boldsymbol{K}\), values \(\boldsymbol{V}\) are three basic elements in the self-attention mechanism, which are obtained by non-shared linear transformations from the original input. \(\boldsymbol{d}_{k}\) denotes the scaling factor, whose value is equal to the dimension of the model. To stabilize the training process, this part can also employ the multi-head attention form, which is similar to Eq. 14.

Since Transformer contains no recurrence or convolution operator, we have to inject some positional information about the tokens in the sequence to consider the order of the sequence. Trigonometric function-based encoding is a common positional encoding approach, which is defined as:

\[\begin{split} PE_{(pos,2i)}&=\sin\left(pos/10000^{2i /d_{\text{model}}}\right),\\ PE_{(pos,2i+1)}&=\cos\left(pos/10000^{2i/d_{\text{ model}}}\right),\end{split} \tag{20}\]

where \(pos\) is the position and \(i\) is the dimension. This method performs sine encoding and cosine encoding on even and odd positions respectively, to distinguish different positions. In addition to Trigonometric functions, we can leverage fully learnable position encodings to encode the position information.

Fig. 10: The overview of Transformer [162]

Fig. 9: The overview of causal temporal convolutional network with exponentially increasing dilated factors [160].
### _Spatio-Temporal Fusion Neural Architecture_

In addition to spatial learning networks and temporal learning networks, spatio-temporal fusion neural architecture represents another critical area of cows, as it determines how spatial learning networks and temporal learning networks are integrated into the complete STGNN. Existing fusion neural architectures can be divided into two categories - factorized or coupled neural architecture.

#### 5.5.1 Factorized Neural Architecture

In factorized neural architectures, spatial learning networks and temporal learning networks are stacked in parallel or serially like building blocks layer by layer. There are two typical examples for factorized neural architectures in STGNN models, as shown in Figure 11 and Figure 12, respectively. The first example is STGCN [19], whose temporal learning network is TCN. In each ST-Conv block of STGCN, two TCNs and one GCN are stacked in series, forming a sandwich structure. Since this model learns temporal information through convolutional structures, its spatio-temporal learning method is parallelized, _i.e._, it receives all information of a given time window length as input at the same time. Mathematically, the calculation of each ST-Conv block in this model can be defined as follows::

\[v^{l+1}=\mathbf{\Gamma}_{1}^{l}*_{\mathcal{T}}\mathrm{ReLU}\left(\mathbf{ \Theta}^{l}*_{\mathcal{G}}\left(\Gamma_{0}^{l}*_{\mathcal{T}}v^{l}\right) \right), \tag{21}\]

where \(\Gamma_{0}^{l}\) and \(\Gamma_{1}^{l}\) denote the upper and lower temporal convolutional kernel within block \(l\), and \(\Theta^{l}\) is the spectral kernel of graph convolution.

The second one is T-GCN [25], which utilizes GRUs for temporal learning. This model captures the spatio-temporal dependencies in a recursive manner. For each time step, graph signals are sequentially processed by GCN and GRU to learn spatial and temporal dependencies separately. The whole process of each stacked GCN and GRU in this model can be expressed as:

\[\begin{split}&\boldsymbol{f}(\boldsymbol{X},\boldsymbol{A})= \sigma\left(\boldsymbol{AXW}_{0}\right),\\ &\boldsymbol{u}_{t}=\sigma\left(\boldsymbol{W}_{u}\left[f\left( \boldsymbol{A},\boldsymbol{X}_{t}\right),\boldsymbol{h}_{t-1}\right]+ \boldsymbol{b}_{u}\right),\\ &\boldsymbol{r}_{t}=\sigma\left(\boldsymbol{W}_{r}\left[\boldsymbol {f}\left(\boldsymbol{A},\boldsymbol{X}_{t}\right),\boldsymbol{h}_{t-1}\right]+ \boldsymbol{b}_{r}\right),\\ &\boldsymbol{c}_{t}=\tanh\left(\boldsymbol{W}_{c}\left[f\left( \boldsymbol{A},\boldsymbol{X}_{t}\right),\left(\boldsymbol{r}_{t}*\boldsymbol{ h}_{t-1}\right)\right]+\boldsymbol{b}_{c}\right),\\ &\boldsymbol{h}_{t}=\boldsymbol{u}_{t}*\boldsymbol{h}_{t-1}+(1- \boldsymbol{u}_{t})*\boldsymbol{c}_{t},\end{split} \tag{22}\]

where \(f(A,X_{t})\) denotes the output of spatial GCN at time step \(t\). Then \(f(A,X_{t})\) is put forward into GRU to obtain the hidden state at \(t\).

#### 5.5.2 Coupled Neural Architecture

In coupled neural architectures, spatial learning networks are usually integrated into the architecture of temporal learning networks as embedded components. In STGNN, this type of neural architecture occurs almost exclusively in combinations of GNN-based spatial learning networks and RNN-based temporal learning networks. One example of a coupled neural architecture in STGNNs is the DCRNN [27], which integrates GCN into the architecture of GRU, as illustrated in Figure 13. In this model, the original linear units in LSTM are replaced with a graph convolution operator, which can be written as:

\[\begin{split}\boldsymbol{r}^{(t)}&=\sigma\left( \boldsymbol{\Theta}_{r}*\mathcal{G}\left[\boldsymbol{X}^{(t)},\boldsymbol{H}^{(t -1)}\right]+\boldsymbol{b}_{r}\right),\\ \boldsymbol{u}^{(t)}&=\sigma\left(\boldsymbol{\Theta} _{u}+\mathcal{G}\left[\boldsymbol{X},\boldsymbol{H}^{(t-1)}\right]+ \boldsymbol{b}_{u}\right),\\ \boldsymbol{C}^{(t)}&=\tanh\left(\boldsymbol{\Theta} _{C}*\mathcal{G}\left[\boldsymbol{X}^{(t)},\left(\boldsymbol{r}^{(t)}\odot \boldsymbol{H}^{(t-1)}\right)\right]+\boldsymbol{b}_{c}\right),\\ \boldsymbol{H}^{(t)}&=\boldsymbol{u}^{(t)}\odot \boldsymbol{H}^{(t-1)}+\left(1-\boldsymbol{u}^{(t)}\right)\odot\boldsymbol{C }^{(t)},\end{split} \tag{23}\]

where \(\boldsymbol{\Theta}_{r}*\mathcal{G}\) denotes the graph convolution operator with parameter \(\boldsymbol{\Theta}_{r}\). Compared with the equation 16 of the original GRU, we can find that except for the internal graph convolution operator, the external calculation methods of the recurrent network are not much different. Similar to some neural translation models [163], DCRNN can also employ sequence-to-sequence structure to improve predictions.

In order to further have an explicit and in-depth understanding of spatio-temporal fusion neural architecture in STGNN, we sort out and analyze the neural architectures of some existing classical models, as shown in Table II.

Fig. 11: The overview1 of STGCN [19].

Fig. 12: The overview of T-GCN [25].

Fig. 13: The overview of DCRNN [27].
## 6 STGNN Variants

In Section 5, we have introduced the basic neural architectures of STGNNs, thereby augmenting the comprehension of the spatio-temporal learning paradigm within this research domain. However, in recent years, there have been numerous innovative methods devised to enhance the learning of spatio-temporal dependencies. In this section, we elaborate on some advanced STGNN variants that can better capture spatio-temporal dependencies for predictive learning in urban computing.

### _Spatial Learning Methods_

#### 6.1.1 Multi-Graph Convolution

In urban systems, there are often multiple types of spatial relations that exist simultaneously. For instance, in transportation systems, adjacent regions and regions with similar POIs may exhibit similar traffic patterns. Hence, jointly considering multiple spatial relations is necessary for spatio-temporal learning in STGNN. In recent years, a series of STGNN variants that integrate multi-graph convolutions have been proposed to address this challenge [166, 167, 168, 20, 21, 24, 33, 170]. Among them, STMGCN [24] is a typical model for urban ride-haling demand prediction, as shown in Figure 14. This model first constructs multi-graph based on neighborhood, function similarity, and connectivity to characterize multiple spatial correlations. For each graph, contextual gated RNN and ChebNet are respectively adopted to capture temporal and spatial dependencies. Finally, the final prediction result is obtained by fusing the parallelized multi-graph spatio-temporal hidden information.

#### 6.1.2 Adaptive Graph Learning

Despite its capability to capture multiple spatial correlations to some extent, multi-graph modeling still suffers from two limitations. Firstly, the graph construction process may be insufficient and fail to account for other implicit correlations. Secondly, the rationality of graph construction may be questioned, particularly in the absence of sufficient domain knowledge to support it. To overcome these challenges, adaptive graph learning methods have been developed gradually. According to existing literature, adaptive graph learning methods in STGNN can be broadly categorized into two main categories: random initialization-based and feature initialization-based approaches.

**Random initialization-based** methods perform adaptive graph structure learning via randomly initialized learnable matrices [171, 172, 173, 174, 34, 80, 122, 34]. Two prominent models in this category are Graph WaveNet [122] and MTGNN [172], which have been widely applied or improved upon in subsequent works. In Graph WaveNet, the adaptive graph is produced as follows:

\[\tilde{\mathbf{A}}_{adap}=\mathrm{SoftMax}\left(\mathrm{ReLU}\left(\mathbf{E}_{1}\mathbf{ E}_{2}^{T}\right)\right), \tag{24}\]

where \(\mathbf{E}_{1}\in\mathbb{R}^{N\times C}\) and \(\mathbf{E}_{2}\in\mathbb{R}^{N\times C}\) are source node embedding and target node embedding, respectively. They are two learnable matrices with the random initialization, where \(N\) denotes the number of nodes in the graph and \(C\) denotes the dimension of the embedding.

In contrast, the generation process of the adaptive graph in MTGNN is defined as:

\[\mathbf{M}_{1}=\tanh\left(\alpha\mathbf{E}_{1}\mathbf{\Theta}_{1}\right), \tag{25}\] \[\mathbf{M}_{2}=\tanh\left(\alpha\mathbf{E}_{2}\mathbf{\Theta}_{2}\right),\] (26) \[\tilde{\mathbf{A}}_{adap}=\mathrm{ReLU}\left(\tanh\left(\alpha\left( \mathbf{M}_{1}\mathbf{M}_{2}^{T}-\mathbf{M}_{2}\mathbf{M}_{1}^{T}\right)\right)\right), \tag{27}\]

\begin{table}
\begin{tabular}{|l|c|c|c|c|} \hline Model & Specific Task & Spatial learning & Temporal Learning & Fusion \\ \hline STGCN [19] & Traffic state prediction & Spectral GCN & Gated TCN & Factorized \\ \hline T-GCN [25] & Traffic state prediction & Spatial GCN & GRU & Factorized \\ \hline DCRNN [27] & Traffic state prediction & Spatial GCN & GRU & Coupled \\ \hline GMAN [120] & Traffic state prediction & Graph attention & Transformer & Factorized \\ \hline Graph WaveNet [122] & Traffic state prediction & Spatial GCN & Causal TCN & Factorized \\ \hline GSNet [40] & Traffic incident prediction & Spatial GCN & GRU & Factorized \\ \hline STGNN-TTE [141] & Travel time prediction & Spatial GCN & Gated TCN+Transformer & Factorized \\ \hline Social-STGCN [53] & Trajectory prediction & Spatial GCN & TCN & Factorized \\ \hline Social-STAGE [144] & Trajectory prediction & Spatial GCN & TCN+Attention & Factorized \\ \hline PM2-5-GNN [62] & Air quality prediction & Spatial GCN & GRU & Factorized \\ \hline SpAttRNN [164] & Air quality prediction & Graph attention & GRU & Coupled \\ \hline ATGCN [165] & Air quality prediction & Graph attention & GRU & Coupled \\ \hline MGWN [68] & Wind prediction & Spatial GCN & Causal TCN & Factorized \\ \hline AGL-STAN [80] & Crime frequency prediction & Spatial GCN & Transformer & Factorized \\ \hline AGCLSTM [82] & Flood situation prediction & Spatial GCN & LSTM+Attention & Factorized \\ \hline STEP [96] & Epidemic prediction & Spatial GCN+Attention & GRU & Factorized \\ \hline \end{tabular}
\end{table} TABLE II: The summary of basic neural architectures of some existing STGNN models.

Fig. 14: The overview of STMGCN [24].
where \(\mathbf{E}_{1}\in\mathbb{R}^{N\times C}\) and \(\mathbf{E}_{2}\in\mathbb{R}^{N\times C}\) represent two randomly initialized node embedding; \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\) are learnable parameters within the model; \(\alpha\) is a hyperparameter for controlling the saturation rate of the activation function. Numerous subsequent random initialization-based adaptive graph learning methods are proposed based on the two aforementioned methods. For example, CCRNN [34] introduced a layer-wise adaptive graph learning mechanism to adjust the graph structures layer by layer. DMSTGCN [124] presented an adaptive graph learning approach with tensor decomposition.

**Feature initialization-based** approaches aim to construct adaptive graph structure learning based on the given inputs or the hidden states [175, 176, 177, 123, 84, 129, 178]. These models usually adopt learnable matrices or attention mechanism to incorporate with the given features for generating the adaptive graph structures. For example, DGCRN [123] proposed a recurrent adaptive graph learning mechanism based on the hidden states to construct the graph structures for each time step. DSTAGNN [175] presented a self-attention-based adaptive graph learning method to establish the connection between the graph structures and the hidden states. BSTGCN [176] designed a Bayesian graph learning mechanism based on the pre-defined graphs and input features. GTS [178] presented a novel probabilistic graph structure learning method based on input features.

#### 6.1.3 Multi-Scale Spatial Learning

Due to the wide existence of spatial heterogeneity in urban systems, entities can be divided into communities with different functions. Entities in the same community may have inter-community correlations, while entities in different communities could also have cross-community correlations. In light of these facts, some recent methods have investigated multi-scale spatial learning based on community partitioning. These methods often leverage domain knowledge to guide the community partitioning process.

In this research line, some studies obtain partitioned communities by artificial division [133, 43, 179] or clustering algorithms [132, 20, 180] while others obtain them by neural networks [181, 71, 182]. For example, ST-SHN [71] and ST-HSL [147] learn the hyperedges, _i.e._ communities, of the hypergraph to capture global spatial dependencies for crime prediction. Besides, GAGNN [181] is a group-aware STGNN model for air quality prediction among hundreds of Chinese cities, as shown in Figure 15. This model first proposed a differentiable grouping network for learning the assignment matrix, which automatically computes the mapping relationships between cities and city groups. Then the spatial GCNs are respectively calculated for graph data at these two different scales to learn the inter-community and cross-community spatio-temporal dependencies. Another notable research line involves THINK [183] and DMGCRN [184], which utilize hyperbolic graph neural networks on the Poincare ball to capture multi-scale spatial dependencies more directly. The hyperbolic space is particularly suitable for modeling hierarchies, including local and global dependencies of spatio-temporal data, which makes it a promising approach for improving STGNN models.

#### 6.1.4 Heterogeneous spatial learning

As mentioned in the introduction section, heterogeneity is an essential property of spatio-temporal data in smart cities wherein it displays varying patterns across various temporal or spatial ranges. Different from the above multi-scale spatial learning methods, some works focused on the fine-grained node-to-node heterogeneous relationships in the spatio-temporal data. To distinguish between the influence of static undirected edges (_e.g._, distance-based edges) and the dynamic directed edges (_e.g._, vehicle's mobility-caused edges) in the spatio-temporal graph, HMGCN [102] performs heterogeneous aggregation on the spatial dimension. Similarly, MasterGNN [60] constructs a heterogeneous graph structure based on multiple relations between air quality and weather monitoring stations, while HT-GNN aggregates heterogeneous information from spatial-based intra-edges, temporal-based inter-edges, and spatio-temporal-based across-time-edges.

Another line of heterogeneous spatial learning utilizes transportation, time, and geographical information to capture intricate spatio-temporal message passing. For instance, HeGA [185] and MOHER [186] design multiple transportation mode-based heterogeneous graphs to receive information from multi-sources at the same time, _e.g._, bike, bus, vehicle, etc. The framework of MOHER is depicted in Figure 16, where the spatio-temporal heterogeneous graph is constructed by the region pair-wise relations and inter-mode multi-relations to characterize the correlations between different transportation modes. Then the heterogeneous graph convolution operator is incorporated with LSTM to capture the complex spatio-temporal dependencies. Additionally, DH-GEM [187] proposed the incorporation of node-position edges, while CAP [188] further expanded this concept by designing node-time and node-location edges within the heterogeneous graph, allowing for the derivation of time and geographical knowledge.

Fig. 16: The overview of MOHER [186].

Fig. 15: The overview of GAGNN [181].
### _Temporal Learning Methods_

#### 6.2.1 Multi-Scale Temporal Learning

Given the prevalence of short- and long-range correlations in spatio-temporal data, capturing multi-scale temporal correlations has emerged as a crucial direction for improving temporal learning. So far, there are two mainstream design directions for multi-scale temporal learning in STGNNs. The first direction utilizes TCNs with receptive fields of varying scales [172, 68]. A typical example is MTGNN [172] which employs multiple TCNs with various kernel sizes for learning temporal dependencies in different scales. The second direction involves integrating other temporal learning networks [17, 20, 117]. For example, DMVSTVGNN [20] jointly utilizes TCNs and Transformers for long-short range temporal learning. As depicted in Figure 17, Traffic STGNN [17] achieves multi-scale temporal learning by multi-networks integration. This model adopts GRU for short-range temporal learning and Transformer for long-range temporal learning.

#### 6.2.2 Multi-Granularity Temporal Learning

There are multiple types of temporal characteristics in spatio-temporal data. For instance, the traffic flow at a given time is not only related to the recent traffic flow but may also exhibit similarities to the traffic flow at the same time on the previous day or even the previous week. This reflects the closeness, periodicity, and trend, respectively. To consider the temporal characteristics at these three granularities, many previous works [30, 31, 32, 189, 190, 191] adopted a three-branch architecture to learn features from different temporal granularities separately, and then fuses the learned hidden states for predictions. As shown in Figure 18, ASTGCN [189] employs a typical three-branch architecture for multi-granularity temporal learning, where \(\mathcal{X}_{h}\), \(\mathcal{X}_{d}\) and \(\mathcal{X}_{w}\)represent the spatio-temporal data for the latest one hour, the data of the same hour from the previous day, and the data of the same hour from the previous week, respectively. After going through separate branches, they are finally fused by the learnable weight matrix.

#### 6.2.3 Decomposition Temporal Learning

Individual temporal patterns usually contain a variety of hidden components, such as inherent components, diffusion components, and periodic components. To better capture these complex temporal dependencies, decomposition-based temporal learning methods have been proposed, which can automatically decompose and integrate different temporal components through special neural designs [171, 192, 193, 194, 195]. FC-GAGA [192] is a noteworthy example of decomposition methods that adopt the subtraction residual from N-BEATS [196] to decompose different components in traffic data and model spatial correlations of each component. As shown in Figure 19, FC-GAGA consists of multiple stacked layers, each consisting of a time gate block, a graph gate block, and several fully connected blocks.The time gate block aims to eliminate node-specific multiplicative seasonality from the block's input and reuse it at the block's output, while the graph gate block aims to capture spatial correlations from various entities. The fully connected blocks are similar to those in N-BEATS, acting on the final model output and removing redundant temporal components for downstream blocks through the two branches of forecast projection and backcast projection.

In addition to FC-GAGA, other works also adopted decomposition-based ideas. For example, StemGNN [194] decomposed the temporal components by the subtraction residual from N-BEATS, but modeled spatial correlations in the spectral domain. D2STGNN [171] proposed a temporal residual decomposition method to incorporate with graph structure learning. STWave [193] directly utilized the discrete wavelet transform to disentangle the event and the trend from the spatio-temporal graph data.

Fig. 19: The overview of FC-GAGA [192].

Fig. 17: The overview of Traffic STGNN [17].

Fig. 18: The overview of ASTGCN [189].
### _Spatio-Temporal Fusion Methods_

#### 6.3.1 Spatio-Temporal Joint Modeling

In Section 5.5, we have discussed basic spatio-temporal fusion architectures of STGNNs, which are either factorized or coupled by spatial learning networks and temporal learning networks. Although these architectures can effectively learn spatial and temporal dependencies separately, they lack the ability to model the joint spatial-temporal dependencies, making it challenging to capture complex spatio-temporal relations across different time steps.

In recent years, some literature focused on jointly modeling spatial-temporal dependencies based on 3D GCN [197], Spatio-Temporal Joint GCN (STJGCN) [198] and Spatio-Temporal Synchronous GCN (STSGCN) [118]. Among them, STSGCN has become a mainstream method for spatio-temporal dependencies jointly fusion. This type of neural architecture enables the modeling of spatio-temporal dependencies in a unified graph structure, which can replace the separated spatial learning networks and temporal learning networks. The crucial part of STSGNN is the construction of the spatio-temporal synchronous graph, as shown in Figure 20. The original spatio-temporal synchronous graph is simple, whose nodes with the same location are connected to each other across adjacent time steps. This graph construction approach not only characterizes spatial neighbors, but also temporal neighbors, establishing unified spatio-temporal relations. After graph construction, STSGNN employs a simple GCN model to capture the spatio-temporal dependencies.

There are some follow-up works [119, 120, 116, 117, 118, 119, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 282, 286, 287, 288, 289, 291, 292, 293, 294, 295, 296, 297, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 323, 335, 336, 337, 341, 342, 343, 344, 345, 346, 347, 358, 360, 361, 362, 363, 364, 365, 366, 367, 371, 383, 384, 385, 386, 387, 388, 389, 390, 311, 323, 339, 340, 341, 342, 343, 344, 346, 347, 358, 369, 370, 371, 389, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426, 427, 428, 429, 430, 444, 446, 447, 445, 446, 447, 45, 46, 471, 472, 473, 474, 475, 476, 477, 488, 490, 411, 412, 413, 414, 415, 416, 417, 418, 419, 420, 421, 423, 424, 425, 426, 427, 428, 429, 431, 432, 433, 434, 435, 436, 437, 438, 439, 444, 448, 449, 450, 451, 452, 453, 454, 45
## 7 Advanced Learning Frameworks

I In recent years, more and more advanced learning frameworks have been developed to enhance the performance of STGNN in terms of deep representation and prediction accuracy. In this section, we review and discuss six typical advanced learning frameworks that are combined with STGNNs, including adversarial learning, meta learning, self-supervised learning, continuous spatio-temporal modeling, physics-informed learning, and transfer learning.

### _Adversarial Learning_

Considering that traditional loss functions, such as L1 and L2 norms, are commonly used in predictive learning tasks to measure data point errors, these optimization objectives may lack the ability to capture the distribution and correlation between predicted data and real data. This limitation could potentially lead to distorted prediction results. Hence, the adversarial loss can be introduced to incorporate with the traditional loss for addressing this problem to some extent, which has been widely applied in time series prediction. To incorporate the adversarial loss, Generative Adversarial Networks (GANs) have been proposed, with the neural predictors as the generators and the neural architecture of discriminators designed separately. We have witnessed a blossom of works [60, 224, 225, 226, 227, 228, 229, 230] to combine the adversarial loss with STGNNs for predictive learning tasks.

However, in certain predictive learning scenarios involving urban systems, it can be challenging to discriminate the prediction results from spatio-temporal scales. To address this, TFGAN proposed a STGNN model combined with adversarial loss for traffic flow prediction, where the discriminator is composed of a GCN and GRU, as illustrated in Figure 22. The combination of GCN and GRU allows for joint discrimination of the prediction results across both spatial and temporal dimensions, ensuring that the predicted results are distributed similarly to the real data at the spatio-temporal scale. TFGAN [225] is trained adversarially as a min-max game between the generator \(G\) and the discriminator \(D\). The generator \(G\) of this model is an STGNN with multi-graph convolution. The loss functions in TFGAN are expressed as follows:

\[\mathcal{L}_{\mathbf{G}} =\mathbb{E}_{\hat{\mathbf{z}}\sim\mathcal{P}_{(F)}}[\log(1- \mathbf{D}(\hat{z}))], \tag{29}\] \[\mathcal{L}_{\mathbf{M}} =\frac{1}{b}\sum_{i=1}^{b}\left\|Y^{i}-\mathcal{Y}^{i}\right\|^{2},\] \[\mathcal{L}_{\mathbf{D}} =\mathbb{E}_{\mathbf{z}\sim\mathcal{P}_{(R)}}[\log(\mathbf{D}(z))]\] \[\quad\quad+\mathbb{E}_{\hat{z}\sim\mathcal{P}_{(F)}}[\log(1- \mathbf{D}(\hat{z}))],\] \[\theta_{\mathbf{G}},\theta_{\mathbf{D}} =\min_{\theta_{\mathbf{G}}}\left[\lambda\mathcal{L}_{\mathbf{M}}+\max_{ \theta_{\mathbf{D}}}\mathcal{L}_{\mathbf{D}}\right],\]

where \(\mathcal{L}_{\mathbf{G}}\) is the generator loss, \(\mathcal{L}_{\mathbf{M}}\) is the mean squared error (MSE) loss between the prediction results and ground-truths, \(\mathcal{L}_{\mathbf{D}}\) is the discriminator loss, \(\mathbf{D}(\cdot)\) denotes the discriminator network. The parameter of generator network \(\theta_{\mathbf{G}}\) and discriminator network \(\theta_{\mathbf{D}}\) are optimized by the min-max target.

\begin{table}
\begin{tabular}{|l|l|l|l|} \hline Main Categories & Sub-Categories & Models & Source Codes \\ \hline \multirow{8}{*}{\begin{tabular}{l} Spatial Learning \\ Methods \\ \end{tabular} } & Multi-Graph Convolution & \begin{tabular}{l} PVGCN [21] \\ \end{tabular} & [https://github.com/HCRTLab-SYSU7/PVCGN](https://github.com/HCRTLab-SYSU7/PVCGN) \\ \cline{2-4}  & \multirow{4}{*}{Adaptive Graph Learning} & STGCN [24] & [https://github.com/mraden/Graph-WaveNet](https://github.com/mraden/Graph-WaveNet) \\ \cline{3-4}  & & Graph WaveNet [122] & [https://github.com/mnzhan/Grafph-WaveNet](https://github.com/mnzhan/Grafph-WaveNet) \\ \cline{3-4}  & & MTCNN [122] & [https://github.com/mnzhan/MTCNN](https://github.com/mnzhan/MTCNN) \\ \cline{3-4}  & & DCGRN [123] & [https://github.com/mspihua-fb-lab/](https://github.com/mspihua-fb-lab/) Traffic-Benchmark \\ \cline{2-4}  & \multirow{4}{*}{Multi-Scale Spatial Learning} & GACNN [181] & [https://github.com/Frigger/GACNN](https://github.com/Frigger/GACNN) \\ \cline{3-4}  & & HKNN [180] & [https://github.com/guokar98/THGCN.git](https://github.com/guokar98/THGCN.git) \\ \cline{3-4}  & & ST-SHN [71] & [https://github.com/kar3d1.ST-SHN](https://github.com/kar3d1.ST-SHN) \\ \cline{3-4}  & & ST-HSL [147] & [https://github.com/kar3d1.ST-SHN](https://github.com/kar3d1.ST-SHN) \\ \cline{3-4}  & & \begin{tabular}{l} Heterogeneous Spatial Learning \\ \end{tabular} & \begin{tabular}{l} HTCNN [211] \\ \end{tabular} & [https://github.com/TestLab-Code/HTGNN](https://github.com/TestLab-Code/HTGNN) \\ \hline \multirow{8}{*}{\begin{tabular}{l} Temporal Learning \\ Methods \\ \end{tabular} } & Multi-Scale Temporal Learning & STCNN [17] & [https://github.com/gmg00M17/DH-GEM](https://github.com/gmg00M17/DH-GEM) \\ \cline{2-4}  & \multirow{4}{*}{Decomposition Temporal Learning} & ASICNN [189] & [https://github.com/dosub1BT/ASTCCN-r-pytorch](https://github.com/dosub1BT/ASTCCN-r-pytorch) \\ \cline{3-4}  & & FC-GAGA [192] & [https://github.com/boreshshmail/f-gaga](https://github.com/boreshshmail/f-gaga) \\ \cline{3-4}  & & StemGNN [194] & [https://github.com/microsoft/SceNN](https://github.com/microsoft/SceNN) \\ \cline{3-4}  & & STWave [193] & [https://github.com/AlxMisher/STWave](https://github.com/AlxMisher/STWave) \\ \hline \multirow{8}{*}{\begin{tabular}{l} Spatio-Temporal \\ Fusion Methods \\ \end{tabular} } & Spatio-Temporal Synchronous & STGCNN [118] & [https://github.com/Davidham3/STSCN](https://github.com/Davidham3/STSCN) \\ \cline{3-4}  & Graph Modeling & STGCNN [117] & [https://github.com/MengZhang/TISTCNN](https://github.com/MengZhang/TISTCNN) \\ \cline{3-4}  & AutoTSG [204] & [https://github.com/AutoTSG/AutoTSG](https://github.com/AutoTSG/AutoTSG) \\ \cline{3-4}  & & Auto-DSTSCN [119] & [https://github.com/linguanghyin/Auto-DSTSCN](https://github.com/linguanghyin/Auto-DSTSCN) \\ \cline{3-4}  & & AutoTCS [208] & [https://github.com/VM520/AutoCTS](https://github.com/VM520/AutoCTS) \\ \hline \multirow{8}{*}{Advanced Methods  Combined with STGNNs} & Adversarial Learning & MasterGAN [80] & [https://github.com/hanjindong/MasterGAN](https://github.com/hanjindong/MasterGAN) \\ \cline{3-4}  & & MT-ASIN [212] & [https://github.com/MiaoHacSunny/MI-ASIN](https://github.com/MiaoHacSunny/MI-ASIN) \\ \cline{3-4}  & & Meta Learning & ST-MetaNet [213] & [https://github.com/panzhey1/ST-MetaNet](https://github.com/panzhey1/ST-MetaNet) \\ \cline{3-4}  & & MegaCN [214] & [https://github.com/deepshshmail/MegaCN](https://github.com/deepshshmail/MegaCN) \\ \cline{2-4}  & & STGCL [215] & [https://github.com/linux4v7/STGCL](https://github.com/linux4v7/STGCL) \\ \cline{3-4}  & & SFCL [216] & [https://github.com/RonghZ19/STGCL](https://github.com/RonghZ19/STGCL) \\ \cline{3-4}  & & ST-SSL [217] & [https://github.com/RonghZ19/STGCL](https://github.com/RonghZ19/STGCL) \\ \cline{3-4}  & & STGCN [218] & [https://github.com/RonghZ19/STGCL](https://github.com/RonghZ19/STGCL) \\ \cline{3-4}  & & ST-SSL [217] & [https://github.com/RonghZ19/STGCL](https://github.com/RonghZ19/STGCL) \\ \cline{3-4}  & & STGODE [218] & [https://github.com/square-coder/STGODE](https://github.com/square-coder/STGODE) \\ \cline{3-4}  & & MTGDGDc [219] & [https://github.com/GAKNAch-lab/MLGDc](https://github.com/GAKNAch-lab/MLGDc) \\ \cline{3-4}  & & ST-NCE [220] & [https://github.com/roughhZ19/STGCL-NCE](https://github.com/roughhZ19/STGCL-NCE) \\ \cline{3-4}  & & \begin{tabular}{l} Physics-Informed Learning \\ \end{tabular} & 
\begin{tabular}{l} STDEN [26] \\ \end{tabular} & [https://github.com/Echo-Ji/STDEN](https://github.com/Echo-Ji/STDEN) \\ \cline{3-4}  & & CoLa-GNN [98] & [https://github.com/github.com/github516/CoLa_GNN_review](https://github.com/github.com/github516/CoLa_GNN_review) \\ \cline{3-4}  & & ST-GPSL [221] & [https://github.com/RobinL120/97-GPSL](https://github.com/RobinL120/97-GPSL) \\ \cline{3-4}  & & TL-DCRNN [222] & [https://github.com/tanwimallick/TL-DCRNN](https://github.com/tanwimallick/TL-DCRNN) \\ \cline{3-4}  & & DASTNet [223] & [https://github.com/YhongT/DASTNet](https://github.com/YhongT/DASTNet) \\ \hline \end{tabular}
\end{table} TABLE III: The source codes for some typical STGNN models with improved spatio-temporal learning methods.
### _Meta Learning_

Meta learning is an advanced learning paradigm focused on the concept of "how to learn to learn". Incorporating meta learning techniques in STGNN models is important since they can capture high-dimensional heterogeneity and dynamic spatio-temporal dependencies from raw data, and teaching them how to learn can significantly improve their prediction performance. Typically, meta learning techniques in STGNN models involve extracting additional spatio-temporal attributes through a meta-learner. STMetaNet [213] is the pioneering study to introduce meta learning into STGNNs. The ST-MetaNet, as illustrated in Figure 23, is composed of RNN, Meta-GAT, and Meta-RNN, and utilizes two types of meta-knowledge learners, namely Node Meta-Knowledge (NMK) and Edge Meta-Knowledge (EMK) learners, to effectively incorporate additional spatio-temporal information. Both of the two different meta-knowledge learner use fully-connected network as the basic learning network. The NMK learner is designed to learn meta-knowledge from the node attributes, such as distances and road networks, while the EMK learner is designed to learn meta-knowledge from the edge attributes, such as locations and POIs. The learned meta-knowledge is then utilized to learn the weights of the Meta-RNN and Meta-GAT components, improving the overall learning capability of the model. Formally, the calculation process of Meta-RNN for any spatial node \(i\) is formulated as:

\[\begin{split} W_{\Omega}^{(i)}&=g_{W_{\Omega}} \left(\mathrm{NMK}\left(v^{(i)}\right)\right),\\ U_{\Omega}^{(i)}&=g_{U_{\Omega}}\left(\mathrm{ NMK}\left(v^{(i)}\right)\right),\\ b_{\Omega}^{(i)}&=g_{b_{\Omega}}\left(\mathrm{ NMK}\left(v^{(i)}\right)\right),\\ h_{t}^{(i)}&=\mathrm{GRU}\left(z_{t}^{(i)},h_{t-1 }^{(i)}\mid W_{\Omega}^{(i)},U_{\Omega}^{(i)},b_{\Omega}^{(i)}\right),\end{split} \tag{30}\]

where \(W_{\Omega}^{(i)},U_{\Omega}^{(i)}\) and \(b_{\Omega}^{(i)}\) are learnable parameters in GRUs and they are generated from node attributes \(v^{(i)}\) by node knowledge meta learner. The meta learner is composed of three different fully-connected networks \(g_{W_{\Omega^{\prime}}}\), \(g_{U_{\Omega}}\) and \(g_{b_{\Omega}}\).

In light of the success of ST-MetaNet, some other STGNN models have been proposed that incorporate meta learning. For example, ST-MetaNet+ [231] fuse the dynamic spatio-temporal state and meta-knowledge for weight generation of GAT and GRU. AutoSTG [204] also adopts a meta learning method similar to ST-MetaNet while introducing neural architecture search, using meta-knowledge to generate weight parameters for graph convolution and temporal convolution. MegaCRN [214] introduced an attention-based memory network, which stores the typical features in seen samples for further pattern matching, thus improving the capability of graph structure learning. In addition, meta learning can also be used for spatio-temporal graph knowledge transfer in predictive learning scenarios [221, 232].

### _Self-Supervised Learning_

Self-supervised learning is a type of method that transforms an unsupervised learning task into a supervised task by constructing its own labels. The goal of this learning paradigm is to learn better representations for downstream supervised tasks. By using self-supervised learning, a representation with strong generalization performance can be learned. Combining STGNN models with self-supervised learning can enhance the capability of spatio-temporal graph learning, which can improve the accuracy of downstream predictive learning tasks.

**Contrastive learning** is one of the most important self-supervised learning methods realized by constructing positive and negative samples, which has been introduced into STGNN models in recent years. One notable example is STGCL, which was introduced by Liu et al. [215] and was the first work to incorporate contrastive learning into STGNN architectures. As shown in Figure 24, the first step of STGCL is the data augmentation to construct the positive and negative samples, where positive and negative samples are constructed using techniques such as edge masking, input masking, and temporal shifting. After obtaining the positive and negative samples, the same STG encoder is employed to learn the spatio-temporal graph representation for both the original data and the augmented data. Then, STGCL splits into two branches - a predictive branch and a contrastive branch. In the predictive branch, the STG decoder directly outputs the prediction results and traditional data point errors, such as mean absolute error (MAE), can be used as the loss function. In the contrastive branch, the two types of representation \(H^{{}^{\prime}}\) and \(H^{{}^{\prime\prime}}\) are put forward into the

Fig. 23: The overview of ST-MetaNet [213].

Fig. 22: The overview of discriminator in TFGAN [225].
projection head to further obtain the latent representation \(z^{\prime}\) and \(z^{\prime\prime}\). For the two latent representation, the contrastive loss proposed in GraphCL [233] can also be adopted in this case, which is as follows:

\[\mathcal{L}_{cl}=\frac{1}{M}\sum_{i=1}^{M}-\log\frac{\exp\left(\sin\left(\mathbf{ z}_{i}^{\prime},\mathbf{z}_{i}^{\prime\prime}\right)/\tau\right)}{\sum_{j\in x_{i}} \exp\left(\sin\left(\mathbf{z}_{i}^{\prime},\mathbf{z}_{j}^{\prime\prime} \right)/\tau\right)}, \tag{31}\]

where \(\sin(\cdot)\) denotes the cosine similarity, \(\tau\) denotes the temperature parameter. Note that, STGCL also proposed to filter out unsuitable negatives based on the unique properties of spatio-temporal graph data such as first-order neighbors of each node, closeness and periodicity temporal patterns, due to their similarity in the latent space. Thus, \(chi_{i}\) denotes the set of acceptable negatives for the \(i_{th}\) object after negative filting.

Based on STGCL, several other contrastive learning methods have been proposed to enhance the learning capabilities of STGNN in recent years. For example, SPGCL [216] proposed to learn the informative relations by maximizing the distinguishing margin between positive and negative neighbors for generating an optimal graph structure. STSSL [217] proposed an adaptive augmentation method over the spatio-temporal graph data at both attribute and structure levels. START [142] presented a spatio-temporal graph-based contrastive learning method for trajectory representation learning. This model proposed multiple negative trajectories construction methods such as trajectory trimming and road segments mask, to aid the STGNN model in achieving better performance in travel time prediction tasks.

### _Continuous Spatio-Temporal modeling_

Most existing STGNN-based approaches capture the spatial and temporal dependencies in a discrete way, leading to discontinuous latent state trajectories and higher prediction errors. To address this problem, some research has focused on continuous spatio-temporal modeling. Motivated by the success of Neural Ordinary Differential Equation (Neural-ODE) [234], a well-known approach for continuous system modeling, STGNNs combined with Neural-ODE can improve the capability of spatio-temporal graph representation learning in a continuous manner. STGODE [218] was the first attempt to introduce Neural-ODE into STGNNs, however, it only considers integrating Neural-ODE with GCN and neglects continuous modeling for temporal patterns. To achieve a joint continuous modeling for spatio-temporal dependencies, MTGODE [219] introduced the integration of Neural-ODE with graph convolution operators and temporal convolution operators, as depicted in Figure 25, to enable continuous spatio-temporal encoding.

In this model, multi-layer GCN with residual connections is transformed as a continuous form, which is formulated by an ordinary differential equation:

\[\frac{\mathbf{d}\mathbf{H}^{G}(t)}{\mathrm{d}t}=\left(\widehat{\mathbf{A}}-\mathbf{I} _{N}\right)\mathbf{H}^{G}(t), \tag{32}\]

where \(\widehat{\mathbf{A}}\) is the adjacency matrix, \(\mathbf{H}^{G}(t)\) is the continuous hidden state. \(\widehat{\mathbf{A}}\) in MTGODE is an adaptive graph which can be computed by Eq. 27. To obtain the approximate solution of ODE, ODESolver \((\cdot)\) can be any black-box ODE solver introduced in [234] such as Euler, Euler-Cauchy, and Runge-Kuta fourth order. Given the initial hidden state \(\mathbf{H}^{G}(0)\), the continuous hidden state in GCN at \(t\) can be estimated as:

\[\mathbf{H}^{G}\left(t_{i}\right)=\text{ ODESolver }\left(\mathbf{H}^{G}(0),\frac{ \mathrm{d}\mathbf{H}^{G}(t)}{\mathrm{d}t},t_{i}\right), \tag{33}\]

Similar to a continuous GCN, a multi-layer TCN with residual connections can also be transformed to a continuous form, which is defined as:

\[\frac{\mathrm{d}\mathbf{H}^{T}(t)}{\mathrm{d}t}=\mathcal{P}\left(\text{TCN} \left(\mathbf{H}^{T}(t),t,\mathbf{\Theta}\right),R\right), \tag{34}\]

where \(\mathcal{P}\) denotes the padding operation; \(\mathbf{\Theta}\) denotes the parameter of convolutional kernel; \(R\) indicates the receptive field of TCN. Note that the temporal dimension needs to be consistent to ensure the continuity of the hidden state, a padding operation is necessary in this case. The continuous hidden state at \(t\) in TCN can be approximately computed by the ODE solver, which is as follows:

\[\widetilde{\mathbf{H}}_{\text{out}}^{T}=\text{ ODESolver }\left(\mathbf{H}^{T}(0),\frac{\mathrm{d}\mathbf{H}^{T}(t)}{ \mathrm{d}t},t_{i}\right), \tag{35}\]

In addition, Social ODE [235] extended the ODE-based STGNN to the scenario of multi-agent trajectory prediction. MixRNN+ [236] combined Neural-ODE and RNN for continuous recurrent hidden state modeling. STG-NCDE [220] developed a STGNN combined with the neural controlled differential equation (Neural-CDE) for better continuous modeling, compared with Neural-ODE-based methods.

Fig. 24: The overview of STGCL [215].

Fig. 25: The spatio-temporal encoding part in MTGODE [218]. The continuous graph propagation (CGP) module and continuous temporal aggregation (CTA) module are stacked in this part.
### _Physics-Informed Learning_

In the last few years, a new paradigm called Physics-Informed Neural Networks (PINNs) [237] have emerged for exploring and computing real-world dynamics integrating physical differential equations and neural networks with powerful fitting capabilities.The main advantage of PINNs is their ability to enforce physical constraints on the predictions, thereby ensuring that the model's outputs are consistent with the laws of physics. Inspired by PINNs that are based on simple neural networks, physical-informed learning methods can be also combined with STGNNs, especially in epidemic prediction tasks [97, 98, 99, 100, 101]. As shown in Figure 26, STAN first integrates the constraints of SIR differential equations into the STGNN architecture. This model used GAT and GRU to capture the spatial and temporal dependencies respectively and performed a multi-task prediction. There are four components in the output of this model: transmission rate \(\beta\), recovery rate \(\gamma\), time-varying number of infections \(\Delta\)I and recoveries \(\Delta\)R. These components need to satisfy physical constraints based on the SIR equation, which can be expressed as follows:

\[\frac{dR}{dt}=\gamma I, \tag{36}\] \[\frac{dI}{dt}=\beta S-\gamma I,\] \[S=N-I-R,\]

where \(S\) denotes the survivors and \(N\) means the total number of people. In STAN, a constraint loss was used to enforce that the predicted time-varying infections and recoveries are close to those calculated by the SIR equations.

In addition to epidemic prediction tasks, there are also a few works in other domains. For example, STDEN [26] proposed a unified framework that combines traffic potential energy field differential equations and neural networks for traffic flow prediction. The work [66] proposed a knowledge transfer approach from physics-based models to guide the learning of a recurrent graph convolution neural network for predicting flow and temperature in river networks.

### _Transfer Learning_

Due to the scarcity of some spatio-temporal graph data, transfer learning techniques have become a cost-effective approach to extend the same basic STGNN model to different data scenarios. However, there are two main limitations in conducting transfer learning for STGNNs. The first one is the heterogeneity of spatial structures and the other one is the heterogeneity of temporal patterns in different circumstances. To be specific, in different scenarios, the spatial topology, relations, etc. are completely different as well as the temporal patterns such as periodicity and trend.

The existing literature on spatio-temporal graph transfer learning can be roughly divided into three categories: clustering-based [222, 238, 239, 240], domain adaptation-based [223, 241] and meta-learning-based [221, 232]. For example, TL-DCRNN [222] proposed a graph partitioning method to divide the entire highway network into different sub-clusters and then used the DCRNN model to learn the spatio-temporal dependencies from source sub-clusters to target sub-clusters. DASTNet [223] combined the graph representation learning and multi-domains adversarial adaptation methods to obtain domain-invariant node embeddings, achieving the knowledge transfer among different scenarios with different spatial structures. ST-GFSL [221] is a spatio-temporal graph neural network that employs Model-Agnostic Meta Learning (MAML) method for cross-city knowledge transfer. The first step in this model is to meta-train a base model on multiple source datasets, which generates the parameters for adaptation. During the adaptation phase, the feature extractor of the basic STGNN is initialized by the generated parameters, and then the parameters of the feature extractor and predictor are jointly trained on the target dataset.

In order to facilitate future research in the field, we have compiled source codes of several representative STGNN models and categorized them according to the methods used to improve spatio-temporal representation learning. The models and corresponding categories are listed in Table III. This resource can serve as a valuable reference for researchers seeking to develop and compare new STGNN models.

## 8 Challenges and Future Directions

We have investigated the applications, basic neural architectures, and recent advancements of STGNN for predictive learning in social systems. Although STGNN models have achieved remarkable performance in recent years, there are still some challenging problems to be addressed, which point to potential future research directions. We summarize these challenges and suggest potentially feasible research directions as follows:

* **Lack of interpretability:** So far, the vast majority of STGNN-related work has focused on improving predictive performance through sophisticated model design. However, research on the interpretability of models has been relatively lacking, that is, we cannot clearly understand which spatio-temporal features take a leading role in improving predictive performance. In the most recent work, STNSCM [242] proposed to construct a causal graph to describe the bike flow prediction and analyze the causal relationship between the spatio-temporal features and prediction results. Causal-based spatio-temporal graph modeling could be a potential direction to enhance the interpretability of STGNN models.
* **Lack of calibration methods:** Uncertainty quantification is of great significance to practical industrial production, which reflects the degree of trust in the prediction results

Fig. 26: The overview of STAN [97].
of the model. In order to improve the credibility of the deep models, appropriate model calibration methods are necessary, which have been widely used in image recognition [243] and graph representation learning [244] in recent years. At present, only works [245, 246] have studied the uncertainty of STGNN models, and there is a lack of research on calibration methods. Calibration for the STGNN models need to take into account the characteristics of spatial and temporal simultaneously, thus it is more challenging than previous related works.
* **Lack of physical constraints:** In most of previous works, STGNN models capture the complex spatio-temporal dependencies through the integration of deep neural networks, while ignoring the consideration of physical constraints in different application domains, which makes the model less recognized in some professional fields. In recent years, although some STGNN models for epidemic prediction have combined professional differential equations as physical constraints [97, 98, 99, 100, 101], such work is still lacking and needs to be improved in other application fields.
* **Lack of pre-training techniques:** Pre-training techniques have been greatly developed in the fields of time series and graph representation learning in recent years, but they are relatively lacking in STGNN-related work. In the most recent work, STEP [247] proposed a pre-training model combined with the Mask Auto-Encoder (MAE) [248] architecture to efficiently learn temporal patterns from very long-term history spatio-temporal graph data. In the future, pre-training techniques for long-range spatial and long-term temporal learning are necessary, which are of great value to the scalability and deployability of the STGNN models.
* **Hurdle of distribution shifts:** Spatio-temporal data, such as traffic flows over road networks, are often collected from various locations and time periods, resulting in significant differences in the distribution of the training, validation, and test sets. For instance, the training set may span the first two years, while the validation and test sets come from the following two years. This can pose a challenge for STGNNs, as training a model on one dataset may not perform well on validation and test sets due to _distribution shifts_, which is similar to the distribution shift issue in domain adaptation (where the joint distribution of inputs and outputs differs between the training and test stages). Despite its importance, this problem has received less attention in the spatio-temporal research community. While several studies [249] investigated defeating distribution shifts in time series, they fail to encode the spatial correlations among locations.
* **Exploring new training strategies:** Previous studies have primarily focused on introducing novel STGNNs with sophisticated layers or modules to enhance human mobility analytics. However, another promising direction is to investigate new training strategies. For instance, in traffic prediction tasks, every location is treated equally, and the data belonging to these locations are jointly fed into neural networks. Nevertheless, the complexity of modeling the spatio-temporal correlations of each location can vary significantly, necessitating a new training strategy such as curriculum learning. Curriculum learning trains a machine learning model on increasingly difficult data, starting from simpler data, and may be effective in addressing this issue. In addition, other potential training strategies for STGNNs include multi-task learning, transfer learning, and continual learning. By exploring new training strategies, we can improve the performance and accuracy of STGNNs and enable them to tackle even more complex tasks.
* **Scalability issue:** One particularly challenging case for designing efficient STGNNs is when the number of locations in the sensor network is very large. For example, there are over ten thousand of loop detectors in PEMS systems. In this scenario, there is a need to develop STGNNs that can efficiently process and analyze the vast amounts of spatio-temporal data generated by the network while maintaining high prediction accuracy. Under this circumstance, more efficient AI solutions are appreciated, _e.g._, through model pruning/distillation, graph sampling techniques, or exploring the next-generation AI models with high efficiency. There are also a few studies probing into graph-free approaches [250] to reduce computational costs when scaling up to large-scale sensor networks.

## 9 Conclusion

In this paper, we present a systematic survey of spatio-temporal graph neural networks (STGNNs) for predictive learning in urban computing. We start with a basic form and construction method of spatio-temporal graph data, and then summarize the predictive learning tasks involving STGNNs from different application domains in urban computing. Next, Moving on, we delve into the fundamental neural network architectures that underpin STGNNs, including the spatial learning network and temporal learning network, which consist of graph neural networks (GNNs), recurrent neural networks (RNNs), temporal convolutional networks (TCNs), self-attention networks (SANs), and explore the basic fusion techniques used to integrate these spatio-temporal neural architectures. To stay up-to-date with the latest developments in STGNNs, we review notable recent works, focusing on spatial learning methods, temporal learning methods, spatio-temporal fusion methods, and other advanced techniques that can be combined. Finally, we summarize the challenges of current research and suggest some potential directions.
