A Multidimensional Graph Fourier Transformation Neural Network for
  Vehicle Trajectory Prediction
This work introduces the multidimensional Graph Fourier Transformation Neural
Network (GFTNN) for long-term trajectory predictions on highways. Similar to
Graph Neural Networks (GNNs), the GFTNN is a novel network architecture that
operates on graph structures. While several GNNs lack discriminative power due
to suboptimal aggregation schemes, the proposed model aggregates scenario
properties through a powerful operation: the multidimensional Graph Fourier
Transformation (GFT). The spatio-temporal vehicle interaction graph of a
scenario is converted into a spectral scenario representation using the GFT.
This beneficial representation is input to the prediction framework composed of
a neural network and a descriptive decoder. Even though the proposed GFTNN does
not include any recurrent element, it outperforms state-of-the-art models in
the task of highway trajectory prediction. For experiments and evaluation, the
publicly available datasets highD and NGSIM are used
Marion Neumeier, Andreas Tollkühn, Michael Botsch, Wolfgang Utschick
# A Multidimensional Graph Fourier Transformation Neural Network for Vehicle Trajectory Prediction

###### Abstract

This work introduces the multidimensional Graph Fourier Transformation Neural Network (GFTNN) for long-term trajectory predictions on highways. Similar to Graph Neural Networks (GNNs), the GFTNN is a novel network architecture that operates on graph structures. While several GNNs lack discriminative power due to suboptimal aggregation schemes, the proposed model aggregates scenario properties through a powerful operation: the multidimensional Graph Fourier Transformation (GFT). The spatio-temporal vehicle interaction graph of a scenario is converted into a spectral scenario representation using the GFT. This beneficial representation is input to the prediction framework composed of a neural network and a descriptive decoder. Even though the proposed GFTNN does not include any recurrent element, it outperforms state-of-the-art models in the task of highway trajectory prediction. For experiments and evaluation, the publicly available datasets highD and NGSIM are used.

## I Introduction

Predicting the motion intention of nearby road users in different traffic scenarios is crucial for autonomous driving systems to operate safely. However, the trajectory prediction of other participants is a challenging task since it highly depends on the scenario's cooperative context. The cooperative context describes the situational and social interactions between traffic participants, that influence the behavior of each individual. These interdependencies of the cooperative context mainly result from the temporal and spatial relations between all participants. Existing approaches therefore attempt to model or statistically learn causalities within these two key dimensions. Since trajectory prediction can be regarded as a sequence-to-sequence learning task, most machine learning based methods use Recurrent Neural Networks (RNNs) [1][7][19]. Recently, Graph Neural Networks (GNNs) have also gained increasing popularity in the field of autonomous driving. GNNs are deep learning networks that operate on graph structures. Therefore, this class of deep learning architectures is especially powerful when applied on data generated from non-Euclidean domains or information represented by graphs.

Although GNNs are successfully used in several applications [24][8][5], there has been limited amount of works that study their representational properties. Recent works with focus on mathematical properties revealed theoretical limitations of the representative power of GNNs [22][17][3]. It has been shown, that popular GNN variants like the Graph Convolutional Network (GCN) [13] and GraphSAGE [11] lack expressive power, since these networks are unable to distinguish different graph structures [29]. The expressive power of any GNN mainly depends on the discriminative power of the aggregation scheme. A maximally powerful GNN provides an _injective_ neighborhood aggregation scheme, which means that two different neighborhoods are never mapped to the same representation [29]. GNNs that regard this principle are able to effectively learn complex relationships and interdependencies.

In this work, the relational representation power of graphs is used to find an efficient scenario representation in order to predict a vehicle's trajectory. It introduces the multidimensional Graph Fourier Transformation Neural Network (GFTNN), which models the traffic scenario through a multidimensional graph. The aggregation of the scenario graph is done by applying the multidimensional Graph Fourier Transformation (GFT). This embedding is injective and preserves dimension properties. By forwarding the GFT embedding through a basic Feedforward Neural Network (FNN), the architecture outperforms current state-of-the-art deep neural networks in the task of trajectory prediction on highways. To the best of the authors' knowledge, this work is the first to simultaneously utilize spatial and temporal dependencies through the multidimensional GFT within a neural network setup for trajectory prediction in traffic.

**Contribution.** The work contributes towards the usage of multidimensional dependencies for trajectory prediction through the definition of graphs and dimension perceiving operations. The main contributions are summarized as follows:

* Introduction of the GFTNN for vehicle trajectory predictions on highways.
* Performance evaluation on the publicly available highway datasets highD and NGSIM.
* Interpretation of the prediction performance of the proposed GFTNN.

In this work, vectors are denoted as bold lowercase letters and matrices as bold capital letters.

## II Related Work

Being aware of the cooperative traffic context and the behaviour of individual traffic participants is essential for autonomous driving systems to safely plan their motion. There
fore, it is necessary to contrive architectures that are able to capture the interactive nature of traffic scenarios and predict the future motions of participants. Motion prediction of vehicles can be categorized into two main categories: short-term predictions (\(<1\,\mathrm{s}\)) and long-term predictions (\(\geq 1\,\mathrm{s}\)) [21]. Current state-of-the-art deep learning methods for long-term trajectory prediction of vehicles are mainly dominated by neural network architectures that are not based on graph structure. [18][19][6] An extensive survey on different approaches for motion prediction was done by Lefevre _et al._[16]. Important representatives are the Social LSTM [1] and Convolutional Social Pooling LSTM (CS-LSTM) [7]. These models use RNNs and Convolution Neural Networks (CNNs) to evaluate the sequential and interdependent nature of the trajectory prediction task. Messaoud _et al._[19] extended this idea by introducing the Multi-head Attention Social Pooling (MHA-LSTM) model. The core idea of the MHA-LSTM is to use the attention mechanism [26] to better evaluate the cooperative context of a scenario. Although such RNN-based models are widespread and perform good in sequence analysis, these methods suffer from time-consuming iterative propagation and gradient explosion/vanishing issues [23]. The proposed GFTNN does not contain recurrent network structures for the prediction and therefore prevents these issues.

Over the last years, the field of GNNs has experienced increasing attention and various models were introduced. For spatio-temporal sequence learning tasks, which relates to the trajectory prediction task, often spatio-temporal Graph Neural Networks (STGNN) are applied. The surveys [32] and [28] provide an introduction to the group of STGNN. In [30], a STGNN was applied on traffic forecasting. The STGNN model is based on a graph structure followed by operations of stacked sequential convolutions. In contrast to the proposed network in this work, their algorithm does not define a spatio-temporal graph but rather creates a stacked set of spatial graphs. In [20], a graph and RNN based vehicle trajectory prediction model for highway driving was introduced. The model includes a Time-Extrapolator Convolution Neural Network (TXPCNN) layer to setup a stateless system for the trajectory prediction. Zhou _et al._[31] propose an Attention-based Spatio-Temporal Graph Neural Network (AST-GNN) for interaction-aware pedestrian trajectory prediction. This model uses the attention mechanism in order to extract interactions and motion patterns within the spatial-temporal domain. Contrary to the architecture proposed in this work, the AST-GNN handles the spatio-temporal dimensions separately. It has a spatial GNN and an additional temporal GNN. In [9], the Repulsion and Attraction Graph Attention (RA-GAT) model for trajectory prediction is presented. The model is based on two stacked Graph Attention Networks [27], that address either free space or vehicle state information through distinct graph definition. Through this setup, the authors follow the idea of repulsive and attractive forces within a traffic scenario. To encode and decode the vehicle movements, LSTMs are used. VectorNet, introduced in [10], is a hierarchical GNN that initially aggregates the agents' trajectories and map features to polyline subgraphs. The information is then passed to a global interaction graph (GNN) to fuse the features among the subgraphs. The global graph aggregation function is implemented through a self-attention mechanism.

The approach most related to the one of this work is the Spectral Temporal Graph Neural Network (SpecTGNN) of Cao _et al._[4]. The SpecTGNN handles the environmental and agent modelling separately. Both matters are computed by defining a 1D spatial connectivity graph structure, where each node contains a feature vector stacking the information of all considered time steps. By separately applying a spectral graph convolution and a temporal convolution, the state information of the interactive agents (and the environment) are encoded. After combining the resulting spatial and temporal latent spaces, the inverse Fourier transform is applied. Following, the environmental and agent modelling latent spaces are added and a multi-head attention mechanism is applied. The trajectory is predicted through a temporal CNN. In contrast to the proposed model of this work, the SpecTGNN does not define a spatio-temporal graph, but rather addresses both dimensions separately and successively.

## III Method

The proposed GFTNN is novel due to the extraction of expressive representations of traffic scenarios through a global and injective aggregation function based on the graph properties. By defining a spatio-temporal graph based on the scenario and applying the multidimensional GFT, the graph's spectral features are computed. This spectrum holds the temporal and spatial dynamic characteristics of a scenario. The transformed scenario representation is input to a neural network. The GFTNN does not contain any computationally expensive components for the trajectory prediction and thus model complexity is reduced.

### _Preliminaries_

**Graph definition**: Let \(\mathcal{G}=(\mathcal{V},\mathcal{E},\mathbf{W})\) be an undirected weighted graph with \(\mathcal{V}\) being the set of nodes, \(\mathcal{E}\) the set of edges and the weight matrix \(\mathbf{W}\in\mathbb{R}^{N\times N}\). The weight matrix is symmetric and satisfies \(\mathbf{W}_{ij}=1\) for all connections \(i\to j\) if the graph is unweighted. The graph consists of \(N=|\mathcal{V}|\) nodes and \(E=|\mathcal{E}|\) edges. The connectivity of the graph is represented by the adjacency matrix \(\mathbf{A}\in\mathbb{R}^{N\times N}\) where \(\mathbf{A}_{ij}=1\) if there exists an edge between the nodes \(i\) and \(j\), otherwise \(\mathbf{A}_{ij}=0\). The degree matrix \(\mathbf{D}\in\mathbb{R}^{N\times N}\) is a diagonal matrix with the degrees on the diagonals such that \(\mathbf{D}_{ii}=\sum_{j}\mathbf{W}_{ij}\). The node feature of node \(i\) is denoted by \(f(i)\). Particularly important for the GFT is the Laplacian matrix \(\mathbf{L}=\mathbf{D}-\mathbf{W}\odot\mathbf{A}\), where \(\odot\) indicates the Hadamard product operation. The Laplacian matrix \(\mathbf{L}\in\mathbb{R}^{N\times N}\) is a real, symmetric and positive-semidefinite matrix.

**Cartesian product graph**: A Cartesian product \(\mathcal{G}_{1}\square\mathcal{G}_{2}\) is a type of graph multiplication that satisfies specific properties as described in [15]. Such a Cartesian product operation for the graphs \(\mathcal{G}_{1}\) and \(\mathcal{G}_{2}\) is qualitatively illustrated in Fig. 1. A beneficial property of the Cartesian product graph is that
solving the eigenproblem of the resulting graph \(\mathcal{G}_{1}\square\mathcal{G}_{2}\) can be broken down into addressing the eigenproblem of each factor graph \(\mathcal{G}_{1}\) and \(\mathcal{G}_{2}\) separately.

**Eigenvalue decomposition of graphs**: The eigenvalue problem of graphs can be solved by using the Laplacian matrix \(\mathbf{L}\)[15]. Due to the mathematical properties of the Laplacian matrix \(\mathbf{L}\in\mathbb{R}^{N\times N}\) its eigenvalue decomposition results in \(N\) real, non-negative eigenvalues \(\lambda_{0},\ldots,\lambda_{N-1}\) and the eigenvectors \(\mathbf{u}_{0},\ldots,\mathbf{u}_{N-1}\) such that

\[\mathbf{L}\mathbf{u}_{i}=\lambda_{i}\mathbf{u}_{i}, \tag{1}\]

where index \(i=0,\ldots,N-1\). The resulting set of eigenvalues (usually sorted by size) are denoted as the spectrum of a graph. The corresponding eigenvectors form the orthogonal basis \(\mathfrak{B}=\{\mathbf{u}_{0},\ldots,\mathbf{u}_{N-1}\}\) which can be used to perform frequency transformations.

**Multidimensional graph Fourier transformation**: Analogous to the classic Fourier transformation, the set of eigenvalues \(\mathbf{\Lambda}\in\mathbb{R}^{N}\) of a Laplacian matrix represent frequencies and the eigenvectors \(\mathbf{U}\in\mathbb{R}^{N\times N}\) form the basis \(\mathfrak{B}\). The multidimensional GFT of a Cartesian product graph \(\mathcal{G}_{1}\square\mathcal{G}_{2}\) of the signal \(f\) is described by [15]

\[\hat{f}(\lambda_{l_{1}}^{(1)},\lambda_{l_{2}}^{(2)})=\sum_{i_{1}=0}^{N_{1}-1} \sum_{i_{2}=0}^{N_{2}-1}f(i_{1},i_{2})\overline{u_{l_{1}}^{(1)}(i_{1})u_{l_{2} }^{(2)}(i_{2})}, \tag{2}\]

for \(l_{1}=0,\ldots,N_{1}-1\) and \(l_{2}=0,\ldots,N_{2}-1\). The superscript \({}^{(1)}\) denotes the eigenvalues/eigenvectors of graph \(\mathcal{G}_{1}\) and \({}^{(2)}\) of graph \(\mathcal{G}_{2}\), respectively. The expression \(\overline{u_{l_{1}}^{(1)}(i_{1})u_{l_{2}}^{(2)}(i_{2})}\) denotes the element-wise complex conjugate of the multiplication.

### _Problem Definition_

The task is to predict the trajectory of a single target vehicle in a traffic scenario, given motion observations of the target vehicle and its surrounding vehicles. In total, the \(N_{V}\) road users including the target vehicle itself are considered in the prediction process. At each timestep \(t\) the relative spatial coordinates of the surrounding road users with respect to the target vehicle are given. The motion information of the \(j\)-th participating road user is represented as \(\mathbf{\xi}_{j}=[\mathbf{x}_{j,\text{rel}},\mathbf{y}_{j,\text{rel}},\mathbf{y}_ {j,x,\text{rel}},\mathbf{v}_{j,y,\text{rel}}]\). The input matrix of the \(j\)-th road user contains the relative distances \((\mathbf{x}_{j,\text{rel}},\mathbf{y}_{j,\text{rel}})\) and velocities \((\mathbf{v}_{j,x,\text{rel}},\mathbf{v}_{j,y,\text{rel}})\) within a specified observation period \(t_{\text{obs}}\) up to the current time step \(t_{0}\). The scenario origin is fixed within the start position of the target vehicle. The input of the prediction module is given through \(\mathbf{X}=[\mathbf{\xi}_{1},\mathbf{\xi}_{2},\ldots,\mathbf{\xi}_{N_{V}}]^{\text{T}}\) where \(\mathbf{\xi}_{1}\) is the prediction vehicle's motion information. By processing the input relations, the model predicts the trajectory of the target vehicle \(\hat{\mathbf{\epsilon}}=[\hat{\mathbf{x}}^{\text{T}},\hat{\mathbf{y}}^{\text{T}}]^ {\text{T}}\) within the prediction period from \(t_{0}\) to \(t_{\text{pred}}\). The total number of time steps covered by the observation and prediction period are given by \(T_{\text{obs}}\) and \(T_{\text{pred}}\), respectively.

### _Model Structure_

The model is structured into three elements, namely the spectral feature generation, an encoder and a decoder. In Fig. 2, the complete architecture of the prediction model resulting from the sub-elements is shown. The proposed GFTNN represents a scenario as 2D graph. From this graph representation, spectral features are extracted by applying the multidimensional GFT and these features are used as input to a Multilayer Perceptron (MLP).

In a first step, a multidimensional graph structure is built through the Cartesian product graph considering the temporal and spatial interdependencies. The definition of graph \(\mathcal{G}_{\text{S}}\) is based on the spatial relation between the traffic participants of a scenario. Within the graph, each node represents an agent and the edges describe possible interdependencies due to cooperative interactions. The spatial graph construction is defined based on the interaction assumptions between the traffic participants, e. g.:

* _Spider graph_: All traffic participants primarily interact with the target vehicle, but not among each other,
* _Mesh graph_: All traffic participants interact with the target vehicle and also among each other, and
* _Scenario-dependent graph_: Dependent on pre-defined rules that define interactions, the graph can be build up accordingly, e. g., \(k\)-Nearest-Neighbor or \(\epsilon\)-Neighborhoods.

In the proposed GFTNN, a spider graph is applied and hence, the spatial graph definition is similar to graph \(\mathcal{G}_{2}\) in Fig. 1. Note, that there also might be interdependencies between neighboring agents. Despite not modelling these explicitly through the spider graph, these interdependencies are implicitly transformed into the spectral scenario since the prediction vehicle itself is connected to all participants. If the goal was a multi-modal prediction, it would be reasonable to explicitly model all connections by using a mesh graph. To represent the temporal dependencies of each vehicle, a line-graph is set up. Each vehicle has dependencies with itself as time evolves. Therefore, each step of the observed time sequence represents a node and each node is connected to its previous and following time step node. The resulting temporal graph \(\mathcal{G}_{\text{T}}\) is qualitatively similar to graph \(\mathcal{G}_{1}\) of
Fig. 1. Hence, the resulting Cartesian product graph of \(\mathcal{G}_{\mathrm{T}}\square\mathcal{O}_{\mathrm{S}}\cong\mathcal{G}_{1} \square\mathcal{G}_{2}\).

In the proposed GFTNN, the graph definition is kept constant and is not adopted depending on the scenario. The multidimensional GFT can be considered as the aggregation scheme of the graph structure. As stated initially, such graph aggregations are optimally chosen to be injective. Since the GFT is based on the eigendecomposition of the graph, which is a non-injective operation, also the aggregation would be non-injective if the graph was non-static. By keeping the graph definition constant the GFT represents an injective aggregation function. In appendix -A, a proof for the injective nature of the GFT based on a static graph is provided.

### _Spectral Feature Generation_

The spectral feature generation of the GFTNN is similar to a deterministic aggregation scheme within GNNs. Along with the graph definition, the Laplacian matrices \(\mathbf{L}_{\mathrm{T}}\in\mathbb{R}^{T_{\text{obs}}\times T_{\text{obs}}}\) and \(\mathbf{L}_{\mathrm{S}}\in\mathbb{R}^{N_{V}\times N_{V}}\) are computed, where \(T_{\text{obs}}\) is the dimension of the observation time steps and \(N_{V}\) the total number of traffic participants in a scenario. The eigenvalues \(\mathbf{\Lambda}^{(\mathrm{T})},\mathbf{\Lambda}^{(\mathrm{S})}\) and eigenvectors \(\mathbf{U}^{(\mathrm{T})},\mathbf{U}^{(\mathrm{S})}\) result from the individual eigenvalue decomposition (c. f. (1)) of the Laplacian matrices \(\mathbf{L}_{\mathrm{T}}\) and \(\mathbf{L}_{\mathrm{S}}\). Based on this information, the scenario graph is transformed into the spectral domain by a multidimensional GFT. The advantage of the multidimensional GFT is that the transformation retains the dimensional context of the graph signal. This differentiation of the spatial and temporal domain within the graph spectrum would be lost by using only an one-dimensional GFT. Since the dynamics along these two dimensions imply different scenario conditions, it is meaningful to keep the multidimensional nature. Semantically, the spectral feature space indicates how smooth a vehicle's movement is with respect to the neighboring vehicles and along time. Without loss of generality, the signal function is extended in a fashion that each node of the graph holds a feature vector information such that \(\mathbf{F}\in\mathbb{R}^{K\times N_{1}\times N_{2}}\). Features can be for example: velocity, acceleration, etc. Totally there are \(K\) features. The extended formula describing the GFT, considering the extended feature information, is

\[\mathbf{\hat{F}}(k,\lambda_{l_{1}}^{(1)},\lambda_{l_{2}}^{(2)})=\sum_{i_{1}=0}^{N_{ 1}-1}\sum_{i_{2}=0}^{N_{2}-1}\mathbf{F}(k,i_{1},i_{2})\overline{u_{l_{1}}^{(1)}(i_ {1})u_{l_{2}}^{(2)}(i_{2})}, \tag{3}\]

where \(k=1,\ldots,K\) represents the feature dimensions. The spectral representation maps the temporal and spatial dynamic within a traffic scenario under consideration of the context interdependencies. The transformation results in a GFT tensor \(\mathbf{\hat{F}}\in\mathbb{R}^{K\times N_{1}\times N_{2}}\) where \(N_{1}\equiv N_{T_{\text{obs}}}\) is the total amount of observed time steps and \(N_{2}\equiv N_{V}\) the considered traffic participants including the target vehicle. An exemplar of a spectral scenario representation for a one-dimensional feature information with \(K=1\) can be seen in Fig. 2(c). The spectral representation results from the GFT of the scenario illustrated in Fig. 2(a), where the velocity information for 5 different vehicles is given over a period of 30 time steps. The information of the resulting spectrum can be interpreted as classical frequencies: Small (eigen-)values indicate low frequencies and vice versa. For this application, the high frequencies can be interpreted as noise, hence, they do not carry important scenario information. Large eigenvalues within the temporal spectra can possibly be neglected for further computations, since the important scenario characteristic is represented through the small eigenvalues. By selecting the \(p\) most important eigenvalues \(\mathbf{\Lambda}_{[0:p-1]}^{(\mathrm{T})}\) irrelevant information for the trajectory prediction can be filtered out. The filtering process within the frequency domain results in a low-pass characteristics in the original time domain. The resulting low-pass characteristic can be seen in Fig. 2(b), where the inverse GFT is applied by only using the \(p=10\) smallest eigenvalues \(\mathbf{\Lambda}_{[0:p-1]}^{(\mathrm{T})}\) of the spectrum illustrated in Fig. 2(c). Since no inverse GFT is applied within the GFTNN, the hyper-parameter \(p\) defines the temporal eigenvalue information used for further computations. Hence, from the GFT representation \(\mathbf{\hat{F}}\) the constant subsection of \(\mathbf{s}=[s_{1},s_{2},\ldots,s_{Z}]\), where \(Z=|K\times N_{T_{\text{obs}}[0:p-1]}\times N_{V}|\) is chosen to be forwarded as input to the encoder.

### _Encoder_

The encoder is designed as lightweight neural network which does not contain any recurrence or computationally

Fig. 2: Architecture of the GFTNN. The observed traffic scenario is represented as a 2D graph and mapped into a frequency scenario representation through the GFT. The GFT tensor operates as the input to a MLP that computes the parameters of the descriptive decoder in order to predict the future trajectory of the target vehicle.
expensive component. Its core element is the MLP block as illustrated in Fig. 4. After an element-wise multiplication of the GFT scenario representation

\[\mathbf{h}_{s}=\mathbf{s}\odot\mathbf{w}_{s}, \tag{4}\]

where \(\mathbf{w}_{s}\in\mathbb{R}^{Z}\) are learnable parameters, the result is passed on to the MLP block. This block is stacked and computationally repeated \(N_{x}\) times. Note, that each of the \(K\) features is passed through a separate MLP block. The feature-selective information \(\mathbf{h}_{l}^{k}\) are concatenated afterwards. Mathematically, the MLP block is represented as

\[\mathbf{h}_{\text{norm}}^{k}=\frac{\mathbf{h}_{s}^{k}-\mathbb{E}[\mathbf{h}_{s}^{k}]}{ \sqrt{\text{VAR}[\mathbf{h}_{s}^{k}]+\epsilon}},\quad\epsilon=1e-05 \tag{5}\]

\[\mathbf{h}_{l}^{k}=\mathbf{W}_{l}^{k}\Phi(\mathbf{W}_{h}^{k}\mathbf{h}_{\text{norm}}^{k}+\mathbf{b} _{n}^{k})+\mathbf{b}_{l}^{k} \tag{6}\]

\[\mathbf{h}_{c}=concat(\mathbf{h}_{l}^{k},\dots,\mathbf{h}_{l}^{K}), \tag{7}\]

where \(\mathbf{h}_{s}^{k}\in\mathbb{R}^{|N_{x_{\text{ind}}(0:p-1)}\times N_{V}|}\) is the corresponding feature information from \(\mathbf{h}_{s}\) and \(\Phi\) is the non-linear activation through the GELU function [12]. Following [2], the layer normalization (5) computes the mean \(\mathbb{E}[\cdot]\) and variance \(\text{VAR}[\cdot]\) based on a single feature dimension and training case. The combined information vector \(\mathbf{h}_{c}\) is forwarded through a sigmoid function and another fully connected linear layer, that outputs the latent space representation \(\mathbf{h}_{z}\in\mathbb{R}^{3}\).

### _Decoder_

The decoder of the GFTNN is implemented as the descriptive decoder proposed in [21]. Instead of using neural networks for creating the future trajectory, a model-based approach is used. The usage of the descriptive decoder setup introduces interpretability in the latent space \(\mathbf{h}_{z}\). In this descriptive decoder the longitudinal (\(\hat{\mathbf{x}}\)) and lateral (\(\hat{\mathbf{y}}\)) trajectory are approximated through functions capable of representing the vehicle dynamics on highways. This simple vehicle dynamic model provides three parametriable variables, that define the trajectory characteristic. The parametrization is based on the computations of the encoder network: The variables of the latent space \(\mathbf{h}_{z}\) are used as variables in the model-based decoder where

\[\hat{\mathbf{x}}=v_{0}\mathbf{t}+0.5h_{z_{1}}\mathbf{t}^{2}, \tag{8}\]

\[\hat{\mathbf{y}}=\frac{h_{z_{2}}}{1+e^{h_{z_{3}}\mathbf{\tau}}}-\frac{h_{z_{2}}}{1+e^ {h_{z_{3}}\mathbf{\tau}_{0}}}. \tag{9}\]

The variables \(\mathbf{t}=0,\dots,T_{pred}\) and \(\mathbf{\tau}=\mathbf{t}-0.5T_{\text{pred}}\) define the temporal information of the trajectory, and \(v_{0}\) is the latest observed longitudinal velocity of the prediction vehicle.

## IV Experimental Evaluation

In order to evaluate the prediction performance of the GFTNN, it is compared to state-of-the-art approaches in the task of vehicle trajectory prediction. Thereby, prediction should only be based on traffic dynamics and no additional network to handle the infrastructural conditions is added. To yet allow a fair comparison with baseline models, datasets and models where infrastructural information is crucial for the prediction performance are excluded. The proposed GFTNN is hence benchmarked only upon highway datasets. For training and testing the publicly available datasets highD [14] and NGSIM [25] are used due to their extend of application-oriented scenarios. The datasets provide traffic observations of different German or US highways, respectively. The raw datasets, however, hold a naturally huge imbalance of scenarios since lane changes happen less often than lane keepings. When training on such a highly unbalanced data, a pretty high accuracy can be achieved by solely predicting the trajectory belonging to the majority class. Simultaneously, the model most probably fails to capture the minority class of lane change predictions. This problem was investigated in more depth by Ding _et al._[9]: As analyzed, about \(96.37\,\mathrm{\char 37}\) of the scenarios contained in NGSIM dataset are keep lane scenarios. While the performance of their introduced trajectory prediction model on the overall dataset is superior to other baseline models, it performs poorly on the lane change scenarios. In order to enable fair benchmarking, each dataset is pre-selected and pre-processed in a fashion, that the contained trajectory scenarios are balanced. Furthermore, the global description of the datasets is transformed

Fig. 4: Feature-selective MLP block architecture.

Fig. 3: The traffic scenario regarding the feature velocity in (a) is mapped into a frequency scenario representation (c) through a multidimensional GFT. By applying the inverse GFT that only considers the \(p\) smallest eigenvectors results in a low-pass filtered time signal as shown in (b).
into relative (target vehicle centered) descriptions. The resulting data format aligns with the problem definition explained beforehand. In total, \(N_{V}=9\) vehicles are considered. This definition allows the inclusion of the eight possible immediate positional neighbors and the target vehicle itself. When a scenario holds too many vehicles, the participants with the smallest Euclidean distance to the target vehicle are selected. When a scenario lacks participants, "ghost vehicles" are inserted. These ghost vehicles are attributed with the same motion features as the target vehicle itself, so that no artificially simulated dynamic is added. Each scenario consists of an observation period and a prediction period. The observation period is set to \(3\,\mathrm{s}\) so that \(T_{\text{obs}}\) is adaptively parametrized by the dataset frequency (\(fps_{\text{highD}}=25\,\mathrm{Hz}\), \(fps_{\text{NGSIM}}=10\,\mathrm{Hz}\)). The prediction horizon is set to \(t_{\text{pred}}=5\,\mathrm{s}\) and the number of prediction steps result from the respective dataset frequency. From the highD/NGSIM dataset 9000/1100 highway scenarios are extracted, that represent an equal distribution of the maneuvers _keep lane_, _lane change right_ and _lane change left_. The reduced amount of scenarios for the NGSIM results from limited lane change scenarios in the dataset itself.

### _Implementation Details_

With the scope of implementing a lightweight neural network, the MLP block is set to \(N_{x}=1\). The used encoder architecture hence holds one element-wise multiplication layer and three MLP layers with the dimensions \(T_{\text{obs}(0:p-1]}\times N_{V}-50-3\) for each feature \(K\). In the following, different graph and hyperparamter settings for the GFTNN, which are included into evaluation, are explained.

**gftnn**: The spatio-temporal graph is an undirected and unweighted Cartesian product graph. The node feature dimension is set to \(K=4\) so that all features of \(\mathbf{\xi}_{j}\) are considered in the GFT. The complete spectrum is forwarded to the encoder, hence \(p=T_{\text{obs}}\).

**gftnn-w**: The spatio-temporal graph is an undirected and _weighted_ Cartesian product graph. The weights are defined by the inverse Euclidean distance to the target vehicle. The node feature dimension is therefore reduced to \(K=2\), only considering the velocities of \(\mathbf{\xi}_{j}\) in the GFT. The complete spectrum is forwarded to the encoder, hence, \(p=T_{\text{obs}}\). Since the graph weighting is adapted according to the scenario, the aggregation scheme is non-injective.

**gftnn-rdcby5/15**: The spatio-temporal graph is an undirected and unweighted Cartesian product graph. The node feature dimension is set to \(K=4\) so that all features of \(\mathbf{\xi}_{j}\) are considered in the GFT. Only a reduced (rdc) fraction of the complete spectrum is forwarded to the encoder:

\[\mathcal{L}=\text{MSE}(\mathbf{x},\mathbf{\hat{x}})+\text{MSE}(\mathbf{y},\mathbf{\hat{y}}). \tag{10}\]

### _Evaluation_

The GFTNN is compared with state-of-the-art baselines including conventional recurrent models (CS-LSTM [7], MHA-LSTM(+f) [19]) and novel graph-based approaches (Two-Channel GNN [20], RA-GAT [9]). For evaluation the

Fig. 5: Average displacement error histogram on highD trajectory predictions. The GFTNN variants show an exponential decrease for error values greater than \(\sim$0.4\,\mathrm{m}$\). In contrast, the baseline models indicate a more frequent occurrence of greater prediction errors.

\begin{table}
\begin{tabular}{c|c||c|c|c|c} Architecture & \(n_{params}\) & \multicolumn{2}{c}{highD} & \multicolumn{2}{c}{NGSIM} \\  & & ADE[m]@5s & FDE[m]@5s & ADE[m]@5s & FDE[m]@5s & FDE[m]@5s \\ \hline \hline
**gftnn** & 567.253 & **1.04** & **2.41** & **3.08** & 7.31 \\ \hline
**gftnn-w** & 283.653 & 1.15 & 2.60 & 3.13 & **7.21** \\ \hline
**gftnn-rdcby5** & 113.653 & 1.10 & 2.58 & 3.56 & 8.75 \\ \hline
**gftnn-rdcby15** & 38.053 & 1.15 & 2.57 & 4.00 & 9.97 \\ \hline VAE [21] & 253.736 & 4.21 & 4.38 & 3.45 & 7.36 \\ \hline CS-LSTM & 556.362 & 3.27*1/2.88 & 5.71 & 8.65 & 16.93 \\ \hline MHA-LSTM(+f) & 673.305 & 1.18*1/2.58 & 5.44 & 13.10 & 27.45 \\ \hline Two-channel & 80.370 & 2.97 & 6.30 & 6.55 & 14.13 \\ \hline RA-GAT & 91.578 & 3.46 & 6.93 & 4.23*2/7.05 & 15.49 \\ \hline \hline \end{tabular}
\end{table} TABLE I: Trajectory prediction performance of different AI-models including state-of-the-art approaches. Where needed, the grid definitions of the baseline models were adopted, so that the same traffic scenario is provided to each model. Results marked with *1 are taken from [19] and *2 from [9], since their problem definition is equal to the one of this work.
widely used metrics _Average Displacement Error (ADE)_

\[\text{ADE}=\sqrt{\frac{\sum_{i=1}^{N}\frac{1}{\text{1}_{\text{pred}}}\left((\mathbf{x}^{( i)}-\hat{\mathbf{x}}^{(i)})^{2}+(\mathbf{y}^{(i)}-\hat{\mathbf{y}}^{(i)})^{2}\right)}{N}} \tag{11}\]

and _Final Displacement Error (FDE)_

\[\text{FDE}=\frac{\sum_{i=1}^{N}\sqrt{(\mathbf{x}_{\text{1}_{\text{pred}}}^{(i)}- \hat{\mathbf{x}}_{\text{1}_{\text{pred}}}^{(i)})^{2}+(\mathbf{y}_{\text{1}_{\text{pred} }}^{(i)}-\hat{\mathbf{y}}_{\text{1}_{\text{pred}}}^{(i)})^{2}}}{N} \tag{12}\]

are applied. As shown in Table I, GFTNN achieves the best results on both metrics and datasets. More specifically, applied on the highD dataset it achieves a performance improvement of \(\sim 13\,\mathrm{\char 37}\) on ADE with respect to the results provided by literature of the best performing model. Evaluation based on the balanced dataset used for this work emphasizes the prediction performance of GFTNN:the error metrics ADE and FDE indicate a performance improvement of \(\sim 60\,\mathrm{\char 37}\) and \(\sim 45\,\mathrm{\char 37}\), respectively, with respect to the best performing baseline model. Considering the results of NGSIM the performance of all models decreases. Yet, the prediction performance of the GFTNN without parameter reduction is still superior. On this dataset, the best performing baseline model is the LSTM-based VAE [21], which the vanilla gftnn outperforms by \(\sim 12\,\mathrm{\char 37}\). Since average values do not depict the error variance, the prediction performance is additionally evaluated through the error distribution. In Fig. 5 the distribution of the ADEs resulting from the best performing models on highD is illustrated. Both GFTNN implementations represent distributions that have their peaks at an ADE of \(0.4\,\mathrm{m}\) and exponentially decrease as the prediction error increases. The compared models Two-Channel and MHA-LSTM(+f) show a suboptimal error distributions that indicate a worse prediction accuracy in general. The distribution peaks of both models (\(1.7\,\mathrm{m}/1.2\,\mathrm{m}\)) indicate a greater universal prediction error. Evaluation of the different hyperparameter settings for the GFTNN shows, that the best performing model is the gftnn setup. For the highD dataset, however, performance does not drastically decrease even when the model complexity is reduced as in gftnn-rdcby15. This equals a parameter reduction to \(7\,\mathrm{\char 37}\) of the original number of parameters. Applied on the NGSIM dataset, the reduction of parameters affects the prediction performance substantially. Note here, that the frequency provided by the NGSIM dataset (\(10\,\mathrm{Hz}\)) is lower than the one of the highD (\(25\,\mathrm{Hz}\)). Consequently, also the embedded temporal spectra using the NGSIM dataset provide a reduced frequency range. By neglecting the high frequencies of the NGSIM spectra essential dynamics information for the motion prediction is removed. This character indicates that when data frequency is low, high frequency neglection leads to inaccurate predictions. Furthermore, prediction performance decreases when using a non-injective neighborhood aggregation scheme. The gftnn-w adapts its spectral transformation for each scenario by weighting the graph edges through distance information. Even though this approach is provided the same information as the non-weighted GFTNNs, prediction accuracy generally decreases.

### _Interpretation of Prediction Performance_

The GFT is a global aggregation scheme, which comprises the complete scenario graph instead of several local \(k\)-hop neighborhoods. The global characteristic is important for the understanding of the cooperative context of the scenario and ensures the sequence consideration without recurrent layers. By this transformation, the multidimensional context of the graph signal is retained. Since the Fourier transformation is an integral transform, the resulting spectrum holds the information of the complete time series. The spectral feature space indicates how smooth the prediction vehicle's movement is within the cooperative spatio-temporal context. By explicitly providing the network with relational information through the graph structure, a lot of additional knowledge is inserted that fortifies the learning process. The resulting relation-based spectral scenario representation is a beneficial mapping, which facilitates the neural network to capture the scenario context and extract the key information for trajectory predictions.

## V Conclusion

In this work, the GFTNN is proposed which combines the advantages of using graph structures and the expressive power of conventional FNNs. By representing a traffic scenario through a spatio-temporal graph and applying a multidimensional GFT, an efficient time-space-relation representation of the scenario is obtained. Even though the overall GFTNN architecture is kept computationally simple, it outperforms the baseline models in the task of vehicle trajectory prediction on highways. Quantitatively, the vanilla GFTNN setup surpasses the best performing state-of-the-art method by \(\sim 13\,\mathrm{\char 37}\) on the highD dataset and \(\sim 12\,\mathrm{\char 37}\) on the NGSIM. The architecture enables to create a lightweight GFTNN version with highly reduced complexity and parameters while still achieving competitive performance to state-of-the-art networks. In future work, the intuition behind the graph properties and the usage of non-static graphs will be addressed. Furthermore, the GFTNN will be extended to include infrastructural scenario information and compared to state-of-the-art models that consider infrastructure, like the SpecTGNN [4].

## Appendix A Appendix: GFT on static graphs

The 2D GFT is represented as a chain of matrix multiplications. By using \(N_{1}\times N_{2}\) matrices \(\mathbf{F}_{i_{1},i_{2}}=f(i_{1},i_{2})\) and \(\hat{\mathbf{F}}_{k_{1},k_{2}}=\hat{f}(\lambda_{k_{1}}^{(1)},\lambda_{k_{2}}^{(2)})\), the 2D GFT applied to the signal \(f\) is expressed as

\[\mathcal{F}(\mathbf{F})=\hat{\mathbf{F}}=\mathbf{U}_{1}^{*}\mathbf{F}\overline{\mathbf{U}}_{2} \tag{13}\]

where \(\mathbf{U}_{n}\) is an \(N_{n}\times N_{n}\) unitary matrix with (\(i,k\))-th element \(u_{k}^{(n)}(i)\) for \(n=1,2\). Let \(\mathbf{U}_{n}^{*}\) be the Hermitian transpose and \(\overline{\mathbf{U}}_{n}\) the element-wise complex conjugate matrix. The eigenvector matrices \(\mathbf{U}_{n}\in\mathbb{R}^{N_{n}\times N_{n}}\) result from the constant Laplacian matrices which are real, symmetric and positive-semidefinite.
_Proof of injective characteristic_

A function is injective if \(\forall a,b\in\mathcal{X},\mathcal{F}(a)=\mathcal{F}(b)\to a=b\). Assume \(\mathcal{F}(\mathbf{F}_{A})=\mathbf{U}_{1}^{*}\mathbf{F}_{A}\overline{\mathbf{U}}_{2}\) and \(\mathcal{F}(\mathbf{F}_{B})=\mathbf{U}_{1}^{*}\mathbf{F}_{B}\overline{\mathbf{U}}_{2}\) where \(\mathcal{F}:\mathbb{R}^{N_{1}\times N_{2}}\to\mathbb{R}^{N_{1}\times N_{2}}\).

\[\mathcal{F}(\mathbf{F}_{A}) =\mathcal{F}(\mathbf{F}_{B}) \tag{14}\] \[\mathbf{U}_{1}^{*}\mathbf{F}_{A}\overline{\mathbf{U}}_{2} =\mathbf{U}_{1}^{*}\mathbf{F}_{B}\overline{\mathbf{U}}_{2}\] (15) \[\mathbf{F}_{A} =\mathbf{F}_{B} \tag{16}\]
