Spatio-Temporal driven Attention Graph Neural Network with Block
  Adjacency matrix (STAG-NN-BA)
Despite the recent advances in deep neural networks, standard convolutional
kernels limit the applications of these networks to the Euclidean domain only.
Considering the geodesic nature of the measurement of the earth's surface,
remote sensing is one such area that can benefit from non-Euclidean and
spherical domains. For this purpose, we propose a novel Graph Neural Network
architecture for spatial and spatio-temporal classification using satellite
imagery. We propose a hybrid attention method to learn the relative importance
of irregular neighbors in remote sensing data. Instead of classifying each
pixel, we propose a method based on Simple Linear Iterative Clustering (SLIC)
image segmentation and Graph Attention GAT. The superpixels obtained from SLIC
become the nodes of our Graph Convolution Network (GCN). We then construct a
region adjacency graph (RAG) where each superpixel is connected to every other
adjacent superpixel in the image, enabling information to propagate globally.
Finally, we propose a Spatially driven Attention Graph Neural Network (SAG-NN)
to classify each RAG. We also propose an extension to our SAG-NN for
spatio-temporal data. Unlike regular grids of pixels in images, superpixels are
irregular in nature and cannot be used to create spatio-temporal graphs. We
introduce temporal bias by combining unconnected RAGs from each image into one
supergraph. This is achieved by introducing block adjacency matrices resulting
in novel Spatio-Temporal driven Attention Graph Neural Network with Block
Adjacency matrix (STAG-NN-BA). We evaluate our proposed methods on two remote
sensing datasets namely Asia14 and C2D2. In comparison with both non-graph and
graph-based approaches our SAG-NN and STAG-NN-BA achieved superior accuracy on
all the datasets while incurring less computation cost. The code and dataset
will be made public via our GitHub repository.
U. Nazir, W. Islam, M. Taj
# Spatio-Temporal driven Attention Graph Neural Network with Block Adjacency matrix (STAG-NN-BA)

###### Abstract.

Despite the recent advances in deep neural networks, standard convolutional kernels limit the applications of these networks to the Euclidean domain only. Considering the geodesic nature of the measurement of the earth's surface, remote sensing is one such area that can benefit from non-Euclidean and spherical domains. For this purpose, we propose a novel Graph Neural Network architecture for spatial and spatio-temporal classification using satellite imagery. We propose a hybrid attention method to learn the relative importance of irregular neighbors in remote sensing data. Instead of classifying each pixel, we propose a method based on Simple Linear Iterative Clustering (SLIC) image segmentation and Graph Attention GAT (Wang et al., 2017). The superpixels obtained from SLIC become the nodes of our Graph Convolution Network (GCN). We then construct a region adjacency graph (RAG) where each superpixel is connected to every other adjacent superpixel in the image, enabling information to propagate globally. Finally, we propose a Spatially driven Attention Graph Neural Network (SAG-NN) to classify each RAG. We also propose an extension to our SAG-NN for spatio-temporal data. Unlike regular grids of pixels in images, superpixels are irregular in nature and cannot be used to create spatio-temporal graphs. We introduce temporal bias by combining unconnected RAGs from each image into one supergraph. This is achieved by introducing block adjacency matrices resulting in novel Spatio-Temporal driven Attention Graph Neural Network with Block Adjacency matrix (STAG-NN-BA). We evaluate our proposed methods on two remote sensing datasets namely Asi14 and C2D2. In comparison with both non-graph and graph-based approaches our SAG-NN and STAG-NN-BA achieved superior accuracy on all the datasets while incurring less computation cost. The code and dataset will be made public via our GitHub repository.

Geometric deep learning, Euclidean and non-Euclidean domains, Superpixels, Region adjacency graph, Remote sensing +
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Journal of Physics A: Mathematical and Theoretical Physics (London) (1978) 2020 203 2003 20031 2020 2023 2023 2020 2023 2020 2023 2020 2023 2020 2023 2020 2023 2020 2023 2020 2023 2020 2023 2020 2023 2020 2023 2020 2023 202020 2023 2020 2023 202023 2020 2023 20202 2023 202020 2023 202020 2023 2020 202023 2020 2023 202020 2023 202020 2023 202020 202023 202020 2023 202020 202020 202020 2023 202020 202020 202020 202020 202020 20202 20202 202020 20202 202020 202020 202020 202020 202020 202020 2202020 202020 202020 202020 2022020 202020 202020 2022020 202020 2202020 2202020 202020 2020202 202020 2202020 2022020 202020 2020220 202020 2022020 2022020 202020 2202020 20202020 2020220 20202020 2022020 202020 2022020 202020 2020220 2020202 20202020 220202 2020202 20202020 2020202 2020202 20202020 202020 2020202 2020202 20202020 202020202 20202020 20202020 202020202 20202020 20202020 2020202 20202020 202020202 20202020 20220202 202020202 202020202 2022020202 2020202020 20220202 20220202 20202020 202020220 202022020 20220202 202020202 2020220202 20202202 2020202 2022020202 202020202 2022020202 2020202 20220202 20202202 20220202 20202202 20202022 20202022 20202202 20220202 20220202 20220202 20202202 20220202 20202202 2022022 20220202 2022022 20220202 202202202 202022202 20220202 20202202 2022202 20202202 202202202 2022022 20202202 2022022 20202202 2022022 2022022 202022202 2022022 202202022 2022022 20220222 20220222 2022022 2020222 2022022 2022022 20220222 2022022 2022202 2022022 2022022 2022022 2022022 202222 2022022 2022202 2202202 20222202 2202222 202222 2022222 202222 202222 2022222 202222 2022222 202222 2222222 22222222 222222 2222222 222222 22
While the utility of graph neural networks for emerging applications is promising, the complexity of graph data imposes significant challenges on many existing machine learning algorithms. For instance, in the area of image processing, the use of Graph Convolutional Networks (GCN) is still limited to a few examples only (Nguyen et al., 2017; Wang et al., 2018; Wang et al., 2018). By some carefully hand-crafted graph construction methods or other supervised approaches, images can be converted to structured graphs capable of processing by GCNs. In these GNNs, each pixel of an image is considered as a graph node (Garshan et al., 2016) which is cumbersome and in many cases unnecessary. Instead of learning from raw image pixels, the use of'superpixels' addresses this concern (Wang et al., 2018; Wang et al., 2018) and helps in reducing the graph size and thereby the computational complexity. The applications of Superpixels include saliency estimation (Wang et al., 2018), optical flow estimation (Wang et al., 2018), object detection (Wang et al., 2018), semantic segmentation (Wang et al., 2018), reduce input for subsequent algorithms (Wang et al., 2018) and explainable AI (Wang et al., 2018).

In this paper, we propose a hybrid attention method to incorporate these relational inductive biases in remote sensing data. Instead of classifying each pixel, we propose a method based on Simple Linear Iterative Clustering (SLIC) image segmentation and Graph Attention GAT (Yang et al., 2018) to detect socio-economic indicators from remote sensing data (see Table 1). We first over-segment the image into superpixels. These superpixels become the nodes of our Graph Convolution Network (GCN). We then construct a region adjacency graph (RAG) where each superpixel is connected to every other adjacent superpixel in the image, enabling information to propagate globally. Finally, we classify each RAG via Spatially driven Attention Graph Neural Network (SAG-NN). We also propose an extension to our SAG-NN for spatio-temporal data named as Spatio-temporal Attention driven GNN (STAG-NN). Unlike, pixels or objects, superpixels are prone to change over time, to address this problem we propose a STAG-NN with Block diagonal Adjacency matrix (STAG-NN-BA) which enables us to incorporate both the spatial as well as temporal information in a single time-varying graph. The main novelty of this paper is the SAG-NN and STAG-NN-BA architectures for the prediction of spatio-temporal transition classes (such as construction, destruction, cultivation, and harvesting) from remote sensing data. We also show this approach incurs less computational cost compared with other deep learning methods. The details of our proposed approach, which is derived from vanilla GAT (Yang et al., 2018), are presented in Section 4.

## 2. Related Work

**Image Classification**: Availability of high resolution satellite imagery paved a way for future planning and geographical studies for large-scale analysis across the globe (Wang et al., 2018; Wang et al., 2018; Wang et al., 2018; Wang et al., 2018; Wang et al., 2018). Automated large-scale surveys via remote sensing often make use of image classification (Wang et al., 2018) and segmentation (Wang et al., 2018; Wang et al., 2018). In this study, we are focusing on image classification as it plays an important role in land-use land-cover applications. The generic problem of image classification consists of distinguishing the images into object classes (usually, a set of predefined collection of labels). Traditional approaches followed the method of preprocessing images to extract image features (e.g., texture, color, etc.) and running a classifier on those features.

Krizhevsky et al.(Krizhevsky et al., 2014) published a seminal study that explored deep neural networks for image classification. They won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2012

Figure 1. Flow diagram of our proposed spatial attention graph neural network (SAG-NN). Images are first converted into superpixels using SLIC, region adjacency graph is then constructed using these superpixels, finally spatial attention graph neural network is applied. Graph embedding are then used for classification via MLP. (Satellite images courtesy Google Earth).
by a large margin and set a turning point for image classification research. In the following years networks like GoogLeNet (Shi and Malik, 2015) and Squeeze-and-Excitation (Shi and Malik, 2015) further helped in reducing the top-5 error rate from 15.3% to just 2.51%. More recent approaches, such as ViT-e (Shi and Malik, 2015) and CoAtNet-7 (Dong et al., 2016), combined convolution with attention /transformers (Shi and Malik, 2015) and achieved a top-1 accuracy of 90.45% and 90.88% respectively.

Despite the recent advances in datasets and network architectures, using standard convolutional kernels limits the applications of these networks in problems that do not present a domain based on rectangular grids. For example, panoramas capture a whole 360-degree field of view, similarry, measurement on earth's surface are geodesic in nature. To handle these issues, some researchers suggested networks designed to adapt to the spherical domain (Dong et al., 2016). In contrast, others proposed to learn how to adapt convolutional layers to the spherical domain (Dong et al., 2016). More recently, graph based methods have been introduced so that such non-Euclidean spaces can be modeled via geometric deep learning (Shi and Malik, 2015).

**Geometric Deep Learning**: Recently, there has been an increasing interest in geometric deep learning (Shi and Malik, 2015), attempting to generalize deep neural models to non-Euclidean structured domains such as graphs and manifolds. Graph-based representations can be used to model a variety of problems and domains. Some examples include social networks in computational social sciences, sensor networks in communications, functional networks in brain imaging, regulatory networks in genetics, and meshed surfaces in computer graphics (Garshan et al., 2016). In addition, they naturally allow to model multi-resolution representations of the same object. Furthermore, they naturally allow several "multi-resolution" representations of the same object. The same image can be converted to graphs using pixel-level or superpixel-level representations. Superpixel-based representations reduce the input size while also allowing domains such as pinhole and spherical images to be represented as graphs, reducing computation costs needed for classification. Furthermore, there are several recent advances toward the development of Graph Neural Networks (GNNs) (Shi and Malik, 2015; Shi and Malik, 2015; Shi and Malik, 2015), including Graph Attention Networks (Shi and Malik, 2015), which could bridge the gap between different domains.

**Image Classification via Graphs**: To the best of our knowledge, Monti et al.(Mori et al., 2015) proposed the first application of Graph Neural Networks(GNNs) to image classification and the MoNET framework for dealing with geometric data in general. Their framework works by weighting the neighborhood aggregation through a learned scaling factor based on geometric distances. Velickovic et al. (Velickovic et al., 2016) proposed a model using self-attention for weighting the neighborhood aggregation in GNNs. Although this model was a sub-model of the MoNET framework, it provided extraordinary results on other datasets, namely Cora and Citeseer, two famous citation networks (Shi and Malik, 2015), and on the FAUST humans dataset (Carparpes et al., 2016).

Although Shi and Malik's seminal paper applied graph-based methods directly to images by converting each pixel to a graph node for image segmentation, smaller graphs can be generated with lower-level representations. Each segmentation region can be the natural choice for nodes of a graph but generating accurate segmentation results is still an open problem. Superpixels might be the middle ground between pixel-based graphs and object-related region-based graphs. Superpixels group pixels similar in color near each other into meaningful representation units called segments (Shi and Malik, 2015). Several computer vision tasks can be performed on these over-segmented images, including depth estimation, segmentation, and object localization as in (Bai et al., 2016). The work mentioned above on using GNNs for images, alongside the work on adapting self-attention for GNNs and the works for generating superpixels of images, form the pillars on which we based our experiments.

SplineCNN (Shi and Malik, 2015), and Geo-GCN (Mori et al., 2015) are two other models which extend MoNET frameworks to weight neighborhood aggregation based on geometric information. SplineCNN leverages B-spline base properties in their neighborhood aggregation procedure, while Geo-GCN engineered a learned distance function to perform data augmentation using rotations and conformations. Semi-supervised augmentation for classification is another technique for using GNNs with image data as in (Velickovic et al., 2016). The main difference between their method is that they extract a feature vector for each image with a convolutional network and then build a graph on which they used their model. Although their technique is useful for semi-supervised learning. We use the vanilla GAT based classifier for a graph representing an image directly that is not comparable.

In this paper, we propose a unified framework allowing to generalize geometric deep learning to remote sensing data and learn spatial and spatio-temporal features using superpixels. We improve the GAT scoring function to overcome the following shortcomings in GATv1 (Shi and Malik, 2015) and GATv2 (Shi and Malik, 2015): 1) In GATv1, the learned layers \(\mathbf{W}\) and \(a\) are applied consecutively, and thus can be collapsed into the single linear layer. 2) GATv2 (Shi and Malik, 2015) performs best for a complete bipartite graph. We improved the graph attention scoring function by introducing the relational inductive bias in data using neighborhood features aggregation as well as the ranking of attended nodes. Our proposed approach achieves higher accuracy with less computing cost than state-of-the-art graph neural network architectures.

## 3. Challenges

### Heterogeneity in Remote Sensing Data

While considering a large geographic area, several inherent complexities in satellite imagery make automated detection of change in land-use a challenging task. This includes, but is not limited to, i) variations in imaging sensors, ii) differences in construction design across the countries, iii) dynamic surroundings and iv) variations in luminosity, seasonal changes, and pollution levels, etc.

\begin{table}
\begin{tabular}{|c|l|l|c|} \hline
**Formula** & **Indications** & **Ventilation** & **Method** \\ \hline Cora A.K., et al.(Shi and Malik, 2015) & Land Cross Mapping & Earth Surface Dry & Graph Graph CNN \\ \hline Basic, et al.(Shi and Malik, 2015) & Body Kitping & Memory & CNN \\ \hline Basic, et al.(Shi and Malik, 2015) & Land Cross Mapping & Land Cross Mapping & \(\mathcal{O}\)CNN \\ \hline Basic, et al.(Shi and Malik, 2015) & Land Cross Mapping & Land Cross Mapping & \(\mathcal{O}\)CNN \\ \hline Basic, et al.
The heterogeneity of land surface covers, in particular, poses a major challenge for the task of spatial and spatio-temporal analysis. High resolution satellite imagery is drawing much attention from researchers due to the fine spatial details of land surface covers. Pixel-based classification methods are hardly applicable for high-resolution remote sensing images due to the high interior heterogeneity of land surface covers. The separation between spectral signatures of different land surface covers is more difficult due to the abundant details in pixel-based classification (Sandhi, 2017). To deal with this challenge, we are using superpixel-based classification which reduces the redundancy of the spatial features of different ground objects. Details of other challenges can be found in this paper (Sandhi, 2017).

### Representation of Images as Graphs

GNNs on images are characterized by unique challenges with respect to their implementation. Most of the graph neural frameworks (Golovolovolov et al., 2013; Golovolovolov and LeCun, 2015; LeCun, 2016) are designed for dense representations such as pixel-based graphs. However, pixel based representation results in a large number of nodes which increases both the compute as well as memory costs. Since adjacent pixels are known to have similar information except at object boundaries, pixel based representation is not only cumbersome, but it is also highly redundant. To address this concern superpixel and object-based graphs have been extensively used in the literature (see Table 2). For subsequent processing, superpixels have been widely used as an effective way to reduce the number of image primitives.

The literature includes numerous methods for determining a superpixel based representation from an image, each with different strengths and weaknesses. Recently, many DNN-based methods to identify superpixels have been proposed (Sandhi, 2017; Sandhi, 2017). But the most popular of practices in the GNN literature (on account of generally good results and low compute complexity) are SLIC (Golovolovolov et al., 2013), Quickshift (Quickshift, 2017) and Felzenszwalb (2017). Details of these methods are presented in the following subsections.

#### 3.2.1. Slc

The SLIC (simple linear iterative clustering) (Golovolovolov et al., 2013) algorithm simply performs an iterative clustering approach in the 5D space of color information and image location. The algorithm quickly gained momentum and is now widely used due to its speed, storage efficiency, and successful segmentation in terms of color boundaries. However, the limitation of SLIC is that it often captures the background pixels as shown in Fig. 2 - Column 1, and therefore does not significantly help in data reduction for the graph generation. But it performs better in capturing built-up and grassy land from satellite imagery as shown in Fig. 5 - Column 2.

#### 3.2.2. Quickshift

Quickshift (Quickshift, 2017) is a relatively recent 2D algorithm that is based on an approximation of kernelized mean-shift (Golovolovolov et al., 2013). It segments an image based on the three parameters: \(\epsilon\) for the standard deviation of the Gaussian function, \(\alpha\) for the weighting of the color term, and \(S\) to limit the calculating a window size of \(S\times S\). Therefore, it belongs to the family of local mode-seeking algorithms and is applied to the 5D space consisting of color information and image location. One of the benefits of Quickshift is that it actually computes a hierarchical segmentation on multiple scales simultaneously. As shown in Fig. 2 - Column 2, it does not capture background pixels and also reduces 30% of input data for the graph generation. But it cannot segment built-up and grassy areas perfectly as shown in Fig. 5 - Column 3.

#### 3.2.3. Felzenszwalb

This fast 2D image segmentation algorithm, proposed in (Golovolovolovolov et al., 2013), has a single scale parameter that influences the segment size. The actual size and number of segments can vary greatly, depending on local contrast. This segmentation appeared to be less suitable in tests on a series of images, as its parameters require a special adjustment, and consequently, a static choice of this parameter leads to unusable results. As shown in Fig. 2 - Column 3 and Fig. 5 - Column 1, it only captures the pixels corresponding to the region of interest pixels but performs poorly in graph generation procedure as shown in Fig. 3 - Column 3.

Instead of grid-based placement as in images, superpixels usually result in irregular representation depending upon image content. Such irregular representation restricts the construction of graph on spatio-temporal data. This work has addressed this issue by proposing STAG-NN-BA which resolves the issue via a block adjacency matrix.

## 4. Proposed Methodology

The proposed methodology consist of following major steps:

* Generate a superpixel representation of the input images.
* Create a region adjacency graph (RAG) from the superpixel representation, by connecting neighbouring superpixels.

\begin{table}
\begin{tabular}{c c} \hline \hline Graph Type & Proposals \\ \hline Pixel-based Graph & (Golovolovolov et al., 2013; Golovolovolov et al., 2013; Golovolovolov and LeCun, 2015) \\ Superpixel-based Graph & (Golovolovolov et al., 2013; Golovolovolov and LeCun, 2015; LeCun, 2016; LeCun, 2016) \\ Object-based Graph & (Golovolovolovolov et al., 2013; Golovolovolov and LeCun, 2016; LeCun, 2016; LeCun, 2016) \\ \hline \hline \end{tabular}
\end{table}
Table 2. Classification of proposals for graph generation from images

Figure 3. Region Adjacency Graphs (RAG) generation from SLIC, Quickshift and Felzenszwalb superpixels respectively.

Figure 2. Superpixel segmentation techniques on MNIST digit: 9.
* Spatial Attention Graph Neural Network (SAG-NN) from region adjacency graph (RAG) for spatial classification.
* Spatio-temporal driven Graph Attention Neural Network with Block Adjacency matrix (STAG-NN-BA) for classification of transitions or changes in land-use over time.

The following subsections discuss the proposed architecture in detail.

### Superpixel Segmentation

When we apply segmenation techniques on satellite imagery, SLIC (Beng et al., 2015) performs better as compared to Quickshift (Quickshift et al., 2016), Felzenszwalb (2017) and Compact watershed (Wang et al., 2017). As shown in Fig. 4 - Column 2, SLIC captures the color boundaries, and segments perfectly the built-up area and agricultural land. It is more stable for satellite imagery as compared to other segmentation techniques. The superpixel segmentation technique using SLIC (Beng et al., 2015) provides an elegant way to divide the satellite image into homogeneous regions as shown in Fig. 4. We set the number of segments to 75 and compactness to 10. This resulted in approximately 75 superpixels per image and subsequently a graph of 75 nodes instead of 65536 nodes in case of using raw pixel values of remote sensing imagery.

### Graph generation from superpixels

After using a superpixel segmentation technique, a Region Adjacency Graph (RAG) is generated by treating each superpixel as a node and adding edges between all directly adjacent superpixels. Unlike MoNet (Wang et al., 2017), which use K-Nearest Neighbours to form a connection between nodes, in our graph \(G\) we formed connections based on immediate adjacency only. Thus ours is a more compact graph while the information from neighbours of neighbours can still be incorporated in our case by using K-hop messaging passing. Each graph node can have associated features, providing aggregate information based on the characteristics of the superpixel itself. The regions obtained in the segmentation stage are represented as vertices \(V\) and relations between neighboring regions are represented as edges \(E\). The search for the most similar pair of regions is repeated several times per iteration and every search requires \(\mathcal{O}(N)\) region similarity computations. The graph is utilized so that the search is limited only to the regions that are directly connected by the graph structure.

Figure 4. Superpixel segmentation techniques on image from Asia14 dataset. Felzenszwalb’s method and quickshift cannot segment perfectly built-up and barren land due to inherent complexities in satellite imagery. On the other hand, compact watershed poorly performed on grassy land. While SLIC works perfectly on satellite imagery.

Figure 5. RAG generation from SLIC superpixels on image from Asia14 dataset (Satellite images courtesy Google Earth).

Figure 8. The block adjacency matrix (Temporal RAG) for land-use classification used in proposed spatio-temporal models: STAG-NN-BA-GCP and STAG-NN-BA-GSP.

Figure 6. RAG Generation from a single geospatial image.

Figure 7. Generation of Temporal RAG from geospatial images of same geolocation from multiple years.
### Spatial Attention Graph Neural Network (SAG-NN)

We will start by describing a single message passing layer, as the sole layer utilized throughout all of the GCN (Srivastava et al., 2017) and GAT (Yang et al., 2017) architectures.

Consider a graph \(G(V,E)\), where \(V\) is set of \(n\) nodes and \(E\) is the set of \(m\) vertices. \(G\) is specified as a set of nodes' initial embeddings (input features): \((\overrightarrow{x_{1}},\overrightarrow{x_{2}},\ldots,\overrightarrow{x_{n}})\), and an adjacency matrix **ADJ**, such that \(\textbf{ADJ}_{i,j}=1\) if \(i\) and \(j\) are connected, and \(0\) otherwise. Consider node \(i\)'s initial embedding (for step \(k=0\)) is:

\[\overrightarrow{h}_{i}^{(0)}=\overrightarrow{x}_{i},\forall i\in V \tag{1}\]

A graph convolutional layer at step \(k=1,2,\ldots,K\) then computes a set of new node features \((\overrightarrow{h}_{1}^{k},\overrightarrow{h}_{2}^{k},\ldots,\overrightarrow {h}_{n}^{k})\), based on the input features as well as the graph structure. Every graph convolutional layer starts off with a shared feature transformation specified by a weight matrix \(\mathbf{W}\).

In general, to satisfy the localization property, we will define a graph convolutional operator as an aggregation of features across neighbourhoods; defining \(\mathcal{N}_{i}\) as the neighbourhood of node \(i\) (typically consisting of all first-order neighbours of \(i\), including \(i\) itself), we can define the output features of node \(i\) as

\[\overrightarrow{h}_{i}^{(k)}=f^{(k)}\Bigg{(}\mathbf{W}^{(k)}\cdot\Bigg{[}\sum _{j\in\mathcal{N}_{i}}C^{(k)}\overrightarrow{h}_{j}^{(k-1)}+C^{(k)} \overrightarrow{h}_{i}^{(k-1)}\Bigg{]}\Bigg{)} \tag{2}\]

where \(\forall i\in V\) and \(f^{(k)}\) is an activation function. Each neighbour can be assigned different importance as:

\[\overrightarrow{h}_{i}^{(k)}=f^{(k)}\Bigg{(}\mathbf{W}^{(k)}\cdot\Bigg{[}\sum _{j\in\mathcal{N}_{i}}\alpha_{ij}^{(k-1)}\overrightarrow{h}_{j}^{(k-1)}+ \alpha_{ii}^{(k-1)}\overrightarrow{h}_{i}^{(k-1)}\Bigg{]}\Bigg{)} \tag{3}\]

where \(\forall i\in V\) and \(\sum_{j\in\mathcal{N}_{i}}(.)\) is the weighted mean of i's neighbour's embedding at step \(k-1\) and the attention weights \(\alpha^{(k)}\) are generated by an attention mechanism \(\mathbf{A}^{(k)}\), normalized such that the sum over all neighbours of each node \(i\) is 1:

\[\alpha_{ij}^{(k)}=\frac{\mathbf{A}^{(k)}(\overrightarrow{h}_{i}^{(k)}, \overrightarrow{h}_{j}^{(k)})}{\sum_{w\in\mathcal{N}_{i}}\mathbf{A}^{(k)}( \overrightarrow{h}_{i}^{(k)},\overrightarrow{h}_{w}^{(k)})},\,\forall(i,j)\in E \tag{4}\]

In standard GAT (see eq. 3 & 4) \(\alpha_{ij}\) is implicitly defined, employing self-attention over the node features to do so. This choice was not without motivation, as self-attention has previously been shown to be self-sufficient for state-of-the-art-level results on machine translation, as demonstrated by the Transformer architecture (Yang et al., 2017).

Generally, we let \(\alpha_{ij}\) be computed as a byproduct of an attentional mechanism, \(a:\mathcal{R}^{N}\times\mathcal{R}^{N}\longrightarrow\mathcal{R}\) which computes normalized coefficients \(\alpha_{ij}\) across pairs of nodes \(i,j\), based on their features (see eq. 4).

In contrast, in GATv2, every node can attend to any other node using scoring function shown in eq. 5.

\[\overrightarrow{h}_{i}^{(k)}=\alpha_{ij}^{(k-1)}\Bigg{[}f^{(k)}\Bigg{(} \mathbf{W}^{(k)}\cdot\sum_{j\in\mathcal{N}_{i}}\overrightarrow{h}_{j}^{(k-1) }+\overrightarrow{h}_{i}^{(k-1)}\Bigg{)}\Bigg{]} \tag{5}\]

The main problem in the standard GAT scoring function (see eq. 3) is that the learned layers \(\mathbf{W}\) and \(\alpha\) are applied consecutively, and thus can be collapsed into single linear layer (Chen et al., 2017). To fix this limitation in our work, we then impose a relational inductive bias in data using neighborhood features aggregation (see eq. 6 & 7). In our proposed SAG-NN, the node \(i\)'s embedding at step k for \(k=1\) is:

\[\overrightarrow{h}_{i}^{(k)}=f^{(k)}\Bigg{(}\mathbf{W}^{(k)}\cdot\Bigg{[}AGG _{j\in\mathcal{N}_{i}}(\{\overrightarrow{h}_{j}^{(k-1)}\}),\overrightarrow{h }_{i}^{(k-1)}\Bigg{]}\Bigg{)}, \tag{6}\]

where \(\forall i\in V\) and \(AGG(.)\) is the aggregation of i's neighbour's embeddings at step \(k-1\) and \(h_{i}^{(k-1)}\) is the node i's embedding at step \(k-1\). And node i's embedding at step k for \(k=2,3,\ldots\) upto \(K\) is:

\[\overrightarrow{h}_{i}^{(k)}=f^{(k)}\Bigg{(}\mathbf{W}^{(k)}\cdot\Bigg{[}\sum _{j\in\mathcal{N}_{i}}\alpha_{ij}^{(k-1)}\overrightarrow{h}_{j}^{(k-1)}+ \alpha_{ii}^{(k-1)}\overrightarrow{h}_{i}^{(k-1)}\Bigg{]}\Bigg{)} \tag{7}\]

The proposed solution not only improves the aggregation of features from neighbouring nodes, it also improves the ranking of attended nodes (static attention) as shown in eq. 6 & 7.

#### 4.3.1. Spatio-temporal Classification via SAG-NN-E

Although the proposed SAG-NN architecture is developed to account for neighborhood features' aggregation to learn spatial land-use classes, we also extended it for spatio-temporal classification. Given \(T\) time steps, our resulting ensemble SAG-NN-E has \(T\) copies of SAG-NN, one for each time step, connected in parallel. The ensemble has a voting scheme that takes the spatial classification from each SAG-NN and generates the spatio-temporal classification (see Fig. 9). We used this ensemble as a baseline for evaluation of our proposed Spatio-temporal driven Graph Attention Neural Network which is discussed next.

### Spatio-temporal driven Graph Attention Neural Network with Block Adjacency matrix (STAG-NN-BA)

Images having multiple channels such as in case of color or multi-spectral images or sequence of multiple images are usually represented as a spatio-temporal volume. These patio-temporal volumes have fixed spatial dimension or pixels at each depth of the volume. However, when instead of pixels, superpixels of images are used this result in different dimension at each time step. Thus graph from superpixels of each image from a sequence cannot be stacked

Figure 9. Spatio-temporal Classification via SAG-NN-E.
together as in case of pixel based representation. Furthermore, in GNNs the structure of the graph remains unchanged over multiple layers, only the node representation changes (Wang et al., 2017). This restricts the use of GNNs for spatio-temporal classification problems having varying nodes over time.

We addressed this problem by proposing a novel temporal-RAG that connects the individual RAG from each image. To incorporate the temporal change in graphs, we add the fourth dimension in the node features of these RAGs which is basically a numeric index that indicates the chronological order of the image the superpixel belongs to. We then combine the RAGs of these separate images into a supergraph that has these RAGs as unconnected subgraphs, we call this supergraph _Temporal-RAGs_. Figure 7 depicts the creation of _Temporal-RAGs_ from Images of a geo-location from different years. Our proposed temporal-RAG is an extension of our SAG-NN architecture. The supergraph of SAG-NN's is generated by combining the adjacency matrices from each RAG into a single adjacency matrix (see Figs. 7 and 8). This results in a block diagonal adjacency matrix for Temporal-RAGs resulting in Spatio-temporal driven Graph Attention Neural Network with Block Adjacency matrix (STAG-NN-BA) defined as:

\[\overrightarrow{h}_{i}^{(k)} =ReLU\Bigg{(}\mathbf{W}^{(k)}\cdot\Bigg{[}\sum_{j\in\mathcal{N} _{i}}\alpha_{ij}^{(k-1)}\overrightarrow{h}_{j}^{(k-1)}+\sigma_{ii}^{(k-1)} \overrightarrow{h}_{i}^{(k-1)}\Bigg{]}\Bigg{)}\] \[+ReLU\Bigg{(}\mathbf{W}^{(k)}\cdot\Bigg{[}\sum_{j\in\mathcal{N} _{i}}\alpha_{ij}^{(k-1)}\overrightarrow{h}_{j}^{(k-1)}+\alpha_{ii}^{(k-1)} \overrightarrow{h}_{i}^{(k-1)}\Bigg{]}\Bigg{)}\] \[+ReLU\Bigg{(}\mathbf{W}^{(k)}\cdot\Bigg{[}\sum_{j\in\mathcal{N} _{i}}\alpha_{ij}^{(k-1)}\overrightarrow{h}_{j}^{(k-1)}+\alpha_{ii}^{(k-1)} \overrightarrow{h}_{i}^{(k-1)}\Bigg{]}\Bigg{)} \tag{8}\]

where \(\mathbf{+}\) symbol represent the concatenation of features.

In STAG-NN-BA we aggregate the node embedding from all the RAGs into one graph embedding \(X_{G}\) of length \(D\). Then, we feed that embedding to the Multi-Layer Perceptron (MLP) for assigning one of the final transition classes. Our proposed architecture allows to impose relational inductive bias in data using neighborhood features aggregation over space as well as time resulting in a single architecture for data with a varying number of nodes over time (see Fig. 10). Thus it can be used to classify the transitions or change in land-use over time in the remote sensing data. Since transitions are essentially temporal phenomena, the proposed STAG-NN-BA method can incorporate temporal information into regional adjacency graphs. We believe that this method can be extended to other geometric data.

We do not assign features to the edges, since our model uses an attention mechanism, and we believe that the edge features will be learned according to the features of the connecting nodes. STAG-NN-BA combine ideas of graph convolutions (Wang et al., 2017), which allows graph nodes to aggregate information from their irregular neighbourhoods, with self-attention mechanisms (Wang et al., 2017), which allows nodes to learn the relative importance of each neighbour during the aggregation process.

Although, there are many different models that try to incorporate weights in neighborhood aggregation such as SplineCNN (Krizhevsky et al., 2015) and GEO-GCN (Krizhevsky et al., 2015). We used three approaches to perform a land-use transition classification of temporal images namely SAG-NN-E (see section 4.3.1), Global Sum Pooling (STAG-NN-BA-GSP) and Global Concatenated Pooling (STAG-NN-BA-GCP). The last two are discussed as follows:

**Global Sum Pooling (STAG-NN-BA-GSP)**: There exist many different types of order-in-variant read-out layers in the literature, such as Global Average Pooling (Wang et al., 2017), Global Attention Pooling (Wang et al., 2017), Global Max Pooling (Wang et al., 2017), and Global Sum Pooling (Wang et al., 2017).

We use Global Sum Pooling (GSP) for it's simplicity as defined in the equation: \(\mathbf{x}_{\theta}=\sum_{\mathbf{x}\in\mathcal{V}}\mathbf{x}_{\mathbf{0}}^{(L)}\), where \(V\) is the set of vertices, \(\mathbf{x}_{\mathbf{0}}^{(L)}\) is the node embedding at the last layer of a graph neural network, and \(\mathbf{x}_{\mathcal{G}}\) is the embedding for the graph obtained as a result of the pooling operation.

**Global Concatenated Pooling (STAG-NN-BA-GCP)**: We are using RAGs of images from three different timestamps combined into one Temporal-RAGs for the transition classification. Taking the graph readout in the last layers of GAT using Global Sum Pooling (GSP) adding all the nodes of the Temporal-RAGs into one \(n\)-dimensional vector. This makes the embedding of a Temporal-RAG indistinguishable from the embedding of a Temporal-RAG in which the underlying RAGs were to swap places. To solve this problem, we introduced a variation of GSP which gives us separate embedding for each underlying RAG concatenated into one \(n\times D\) vector (see Fig. 10).

## 5. Results and Evaluation

### Datasets

We used three datasets for evaluation of our proposed approach namely MNIST (Krizhevsky et al., 2015), Asia14 (Krizhevsky et al., 2015) and C2D2 Dataset (Beng et al., 2016). Both Asia14 and C2D2 datasets are remote sensing datasets for spatial and spatio-temporal classification respectively. These datasets are graph signal classification tasks, where graphs are represented in mixed mode: one adjacency matrix, many instances of node features. Details of these datasets are discussed next.

#### 5.1.1. MNIST Pixel-based Dataset

The MNIST dataset (Krizhevsky et al., 2015) is an acronym that stands for the Modified National Institute of Standards and Technology dataset. It is a dataset of \(28\times 28\) pixel grayscale images of handwritten single digits between 0 and 9. MNIST dataset containing \(70,000\) pixel based region adjacency graphs as described by (Krizhevsky et al., 2015). Every graph is labeled by one of 10 classes.

#### 5.1.2. Asia14 pixel-based and Superpixels Dataset

_Asia14_ dataset contains samples under varying conditions as discussed in Section 3.1. Furthermore, unlike street imagery, land-use is subject to significant variations in satellite imagery. To cater for this, we used a subset of 14-class dataset named _Asia14_(Krizhevsky et al., 2015). This dataset consisting of Digital Globe RGB band images from 2016 and 2017 of resolution \(256\times 256\) at zoom level 20 (corresponding to 0.149 pixel per meter on the equator). We used 9 classes including brick kilns, houses, roads, tennis courts, grass, dense forest, parking lots, parks. The issue of sensor variations is handled by diversifying the training data across several spatial locations within the Indo-Pak region of South Asia. We have \(9,000\) pixel-based region adjacency graphs and we generated the superpixels using SLIC (Beng et al., 2016). Then \(9,000\) graphs with 75 nodes each, were generated using region adjacency graph method.
#### 5.1.3. C2D2 Dataset

This dataset contains Spatio-temporal data annotated for four fundamental land-use land-change transitions namely construction, destruction, cultivation, and de-cultivation. This dataset was originally collected and prepared by (Beng et al., 2017). They browsed Digital Globe imagery data for the years 2011, 2013, and 2017 and visited almost \(5,50,000\) random locations which make approximately \(5310\ km^{2}\). Along with lat-long, at each location, we cropped an image patch of resolution \(256\times 256\) at zoom level 20 (i.e. 0.149 pixel per meter on the equator). The provided dataset contained 3D volumes of Spatio-temporal images from different years. we had to reverse the process to separate out the individual images for a location into the directories of each year. We then generate regional adjacency graphs (RAGs) from the superpixels of these images that were generated using SLIC and use the same annotations as it was assigned to the 3D volumes.

### Implementation Details

All the graph neural networks are trained using PyTorch. Optimization method is Adam with an initial learning rate of \(1e^{-3}\). The learning rate increases by 0.1 if validation loss does not decline for 20 epochs. Instead of using fixed number of epochs, we used early stopping criteria and patience for early stopping is 200. We have kept the same train, validation and test splits for all the datasets, i.e. 70%, 15%, and 15% respectively.

### Evaluation of SAG-NN

We evaluated our Spatial Attention Graph Attention Network (SAG-NN) architecture on two datasets namely MNIST and Asia14. We performed two experiments, in the first experiment we generated pixel-based graphs and in the second experiment we used superpixel based graphs. We performed comparisons with two classical methods namely Inception-ResNet-v2 (Vinyals et al., 2015) and 2D-ResNet-50 (Vinyals et al., 2015) and seven graph based state-of-the-art methods namely MoNet (Vinyals et al., 2015), ChebNet (Chen et al., 2016), GATv1 (Wang et al., 2017), AGNN (Wang et al., 2017), GraphSAGE (Chen et al., 2017), Crystal GCN (Wang et al., 2017), GATv2 (Wang et al., 2017).

We first trained and validated our Spatial Attention Graph Attention as well as all the other methods on MNIST dataset. Our SAG-NN model achieved highest accuracy of 98.14% on MNIST dataset with 25.69 million number of parameters on pixel-based RAG. Then we trained and tested SAG-NN as well as all the other methods on Asia14 dataset. Here again our proposed SAG-NN achieved highest test accuracy of 77.00% and 80.98% on pixel-based graph and superpixel RAGs respectively (see Table 3 and 4).

In Table 3, the experiments show that the SAG-NN outperforms on pixel-based RAGs as compared to other classical or RAG-based GNN classifiers. In Table 4, SAG-NN has comparable training parameters and shows high accuracy when compared with GCN (Wang et al., 2017) and GraphSAGE (Chen et al., 2017). It shows comparable high accuracy when

\begin{table}
\begin{tabular}{|c|c|c|} \hline Architectures & \#Param (M) & Asia14 \\ \hline Classical models of neural network on image dataset \\ \hline Inception-ResNet-v2 (Vinyals et al., 2015) & 23.50 & 57.70 \% \\ \hline
2D-ResNet-50 (Vinyals et al., 2015) & 23.50 & 56.45 \% \\ \hline \multicolumn{3}{|c|}{Graph neural networks} \\ \hline GCN (Wang et al., 2017) & **0.015** & 9.78\% \\ \hline GraphSAGE (Chen et al., 2017) & **0.015** & 65.00\% \\ \hline GATv1 (Wang et al., 2017) & **0.030** & **80.30** \\ \hline GATv2 (Wang et al., 2017) & 0.055 & 72.04\% \\ \hline SAG-NN (our) & **0.030** & **80.98\%** \\ \hline \end{tabular}
\end{table}
Table 4. Spatial classification accuracy on SLIC superpixels based Region Adjacency Graph (RAG) of subset of Asia14 (Wang et al., 2017) datasets. Top-2 ranking methods are in bold and, in particular, red (1st) and violet (2nd).

Figure 10. Spatio-Temporal driven Attention Graph Neural Network with Block Adjacency matrix (STAG-NN-BA).

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline Architectures & \#Param (M) & MNIST & Asia14 \\ \hline \multicolumn{3}{|c|}{Classical models of neural network on image dataset} \\ \hline Inception-ResNet-v2 (Vinyals et al., 2015) & 23.50 & - & 57.70 \% \\ \hline \multicolumn{3}{|c|}{Classical models of neural network on image dataset} \\ \hline Inception-ResNet-v2 (Vinyals et al., 2015) & 23.50 & - & 57.70 \% \\ \hline \multicolumn{3}{|c|}{Classical models of neural network on image dataset} \\ \hline Inception-ResNet-v2 (Vinyals et al., 2015) & 23.50 & - & 56.45 \% \\ \hline \multicolumn{3}{|c|}{Graph neural networks} \\ \hline MoNet (Vinyals et al., 2015) & 2.12 & 91.11\% & 66.39\% \\ \hline ChebNet (Chen et al., 2016) & 12.85 & 75.62\% & 64.60 \% \\ \hline GATv1 (Wang et al., 2017) & 25.70 & 96.19\% & 69.85 \% \\ \hline AGNN (Wang et al., 2017) & **0.41** & 97.98\% & 47.80\% \\ \hline GraphSAGE (Chen et al., 2017) & 12.85 & 97.27\% & 70.00\% \\ \hline Crystal GCN (Wang et al., 2017) & **0.41** & **98.04\%** & 63.20\% \\ \hline GATv2 (Wang et al., 2017) & 25.70 & - & 71.10\% \\ \hline SAG-NN (our) & 25.69 & **98.14\%** & **77.00\%** \\ \hline \end{tabular}
\end{table}
Table 3. Spatial classification accuracy on pixel based Region Adjacency Graph (RAG) of MNIST (Wang et al., 2017) and subset of Asia14 (Wang et al., 2017) datasets. Top-2 ranking methods are in bold and, in particular, red (1st) and violet (2nd).
compared with GATv1 (Wang et al., 2017). GATv2 (Chen et al., 2017) is proposed for bipartite graphs that's why it shows low performance on pixel-based and superpixel-based region adjacency graphs as compared to our proposed model.

### Evaluation of STAG-NN-BA

We compared both the variants of our STAG-NN-BA with two other methods namely 3D-ResNet-34 (Chen et al., 2017) and SAG-NN-E. SAG-NN-E is our extension of SAG-NN for spatio-temporal data and serves as the baseline. 3D-ResNet-34 (Chen et al., 2017) on the other hand uses 3D convolution and is the only state-of-the-art method with published results on C2D2 dataset. In order to compare our results on C2D2 dataset, we used the same train/test split as in 3D-ResNet-34 (Chen et al., 2017). The ability of transition classification for SAG-NN-E approach is dependent on the performance of land-use classification and voting procedure (see section 4.3.1). Both STAG-NN-BA-GCP and STAG-NN-BA-GSP achieved significantly higher accuracies as compared to SAG-NN-E and 3D-ResNet-34 (Chen et al., 2017) in terms of accuracy and compute cost. STAG-NN-BA-GCP and STAG-NN-BA-GSP achieved approximately 7% and 20% higher accuracy as compared to 3D-ResNet-34. They also achieved 4.88% and 17.81% higher accuracy as compared to SAG-NN-E which indicates the effectiveness of our temporal model STAG-NN-BA as compared to spatial model via SAG-NN. Furthermore, STAG-NN-BA-GSP outperforms all the other methods which shows that the global sum pooling is a more suited method of aggregation as compared to global concatenated pooling.

Table 5 also compares the training parameters, forward pass time, and accuracy of our models used for spatio-temporal land-use classification. It can be seen that the forward pass time of STAG-NN-BA is almost 1ms lower as compared to SAG-NN and much lower as compared to 3D-ResNet-34.

In the land-use transition classification, the STAG-NN-BA-GSP approach is the most reliable. However, we also draw comparison of 3D-ResNet-34 (Chen et al., 2017) with SAG-NN-E and STAG-NN-BA-GCP (see Table 5). Both spatio-temporal proposed models (STAG-NN-BA-GCP and STAG-NN-BA-GSP) achieved higher performance with low computational cost on the C2D2 dataset.

### Qualitative Analysis

Fig. 11 shows the sample annotations for _Construction_ transition class. In Fig. 11 (Row 1), SAG-NN-E with voting mechanism classifies it as _Cultivation_ which is clearly wrong as it can be seen from the middle and last image that the land has undergone the _Construction_. This type of misclassification is expected from the model since there are two transitions in three images of geolocation. The voting mechanism tends to get confused when multiple transitions are present in an example. But our proposed model STAG-NN-BA-GSP' correctly classifies it as _Construction_. In Fig. 11 (Row 2) our all models: SAG-NN-E, STAG-NN-BA-GCP, and STAG-NN-BA-GSP classify it as _Construction_. In Fig. 11 (Row 3), SAG-NN-E and STAG-NN-BA-GSP correctly classify it but STAG-NN-BA-GCP confused it with _Destruction_ perhaps because in this example one building is removed while multiple others were added.

## 6. Conclusion and Future Work

This paper proposed two novel Graph Neural Network architectures for spatial and spatio-temporal classification of remote sensing imagery. We also proposed a novel method to represent temporal information in images using region adjacency graph called Temporal-RAG. We evaluated our approaches on two remote sensing datasets namely Asia14 and C2D2. The comparison with the previously existing classical and graph neural network methods showed that our approaches achieved higher performance and reduced the computation power greatly. There are two areas recognized while working on this paper that can serve as interesting problems for future works. Firstly, there is an issue of information loss during the generation of graphs from superpixel segmentation. Secondly, over-segmentation of an image to make superpixels causes information loss, which decreases the representation power of pixels-based graphs. The information about the shape of the underlying superpixel segment is lost. We can extract generic shape embedding using an auto-encoder into a single \(N\) dimensional vector. While assigning the

Figure 11. Examples showing the change in land-use between 2011 and 2017. In all three examples, more and more land was used for construction purposes over the years. See Section 5.5 for a discussion on results. (Satellite images courtesy Google Earth).

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline Model & \# Par. (M) & FPT (ms) & Acc. \\ \hline
3D-ResNet-34 (Chen et al., 2017) & 63.50 & \textgreater{} 3.6 & 57.72 \% \\ SAG-NN-E & **0.030** & 3.60 ms & 60.02 \% \\ STAG-NN-BA-GCP (ours) & **0.050** & **2.50 ms** & 64.90 \% \\ STAG-NN-BA-GSP (ours) & **0.030** & 2.62 ms & **77.83 \%** \\ \hline \end{tabular}
\end{table}
Table 5. Spatio-temporal comparative evaluation for land-use transition classification on C2D2 dataset respectively. (Key: Acc.: Accuracy, Par.: Parameters, M: Millions, FPT: Forward Pass Time in milliseconds for 100 forward passes). Top-2 ranking methods are in bold and, in particular, red (1st) and violet (2nd).
color values as features, this \(N\)-dimensional shape embedding vector can be concatenated into the initial features. This can help incorporate the shape into graph representations.
