# BAT: Behavior-Aware Human-Like Trajectory Prediction for Autonomous Driving

Haicheng Liao 1, Zhenning Li1∗†, Huanming Shen2, Wenxuan Zeng3, Dongping
Liao1, Guofa Li4, Shengbo Eben Li5, Chengzhong Xu1† Authors contributed
equally; †Corresponding author.

###### Abstract

The ability to accurately predict the trajectory of surrounding vehicles is a
critical hurdle to overcome on the journey to fully autonomous vehicles. To
address this challenge, we pioneer a novel behavior-aware trajectory
prediction model (BAT) that incorporates insights and findings from traffic
psychology, human behavior, and decision-making. Our model consists of
behavior-aware, interaction-aware, priority-aware, and position-aware modules
that perceive and understand the underlying interactions and account for
uncertainty and variability in prediction, enabling higher-level learning and
flexibility without rigid categorization of driving behavior. Importantly,
this approach eliminates the need for manual labeling in the training process
and addresses the challenges of non-continuous behavior labeling and the
selection of appropriate time windows. We evaluate BAT’s performance across
the Next Generation Simulation (NGSIM), Highway Drone (HighD), Roundabout
Drone (RounD), and Macao Connected Autonomous Driving (MoCAD) datasets,
showcasing its superiority over prevailing state-of-the-art (SOTA) benchmarks
in terms of prediction accuracy and efficiency. Remarkably, even when trained
on reduced portions of the training data (25%), our model outperforms most of
the baselines, demonstrating its robustness and efficiency in predicting
vehicle trajectories, and the potential to reduce the amount of data required
to train autonomous vehicles, especially in corner cases. In conclusion, the
behavior-aware model represents a significant advancement in the development
of autonomous vehicles capable of predicting trajectories with the same level
of proficiency as human drivers. The project page is available at our
Github111https://github.com/Petrichor625/BATraj-Behavior-aware-Model.

## Introduction

Figure 1: An overview of our proposed behavior-aware pooling mechanism and the
classical pooling mechanism. Left: Modeling the vehicle using polar
coordinates. Right: Modeling the vehicle using fixed-size grids and
representing the position in Cartesian coordinates.

Recent advancements in autonomous driving (AD) have been remarkable.
Nonetheless, as we move towards the commercialization of high-level AD
technology, challenges abound. One of the most significant barriers is
equipping autonomous vehicles (AVs) with the ability to anticipate the
trajectory of nearby vehicles in intricate situations as skillfully as humans.

Driving, for humans, necessitates continuous monitoring of the current states
of surrounding vehicles and forecasting their future states before actions
like acceleration or overtaking. These states, predominantly determined by
trajectories, form the bedrock of safe driving and collision prevention. This
demands a keen assessment of the interaction among vehicles and an unbiased
grasp of their behavior, in line with traffic regulations and accumulated
driving experience (Müller, Risto, and Emmenegger 2016).

In our quest to enhance the trajectory prediction capabilities of AVs,
mimicking human-like comprehension and response to surrounding scenarios might
be a breakthrough. As highlighted in (Schwarting et al. 2019; Wang et al.
2022a), accounting for the behaviors of other drivers in the decision-making
processes of AVs can potentially result in enhanced driving performance. With
this understanding, we advocate that a deeper dive into driver behavior can
significantly uplift trajectory prediction for AVs.

Previous investigations have posited that there exists a certain relationship
between different drivers’ behaviors and their driving performance (Toledo,
Musicant, and Lotan 2008; Chandra et al. 2020; Xie et al. 2020). When
confronted with the prospect of another vehicle attempting to overtake,
aggressive drivers may accelerate to impede the overtaking vehicle, while
cautious drivers may reduce their speed slightly to facilitate safe passing.
In addition, driver behavior on the road tends to exhibit a degree of
predictability, persistence, and consistency (Hang, Lv, and Chen 2022;
Schwarting et al. 2019). For example, an individual who has recently violated
the speed limit is likely to continue driving at high speeds as long as
circumstances allow, while cautious drivers maintain their conservative
driving strategy. These stability and repetition characteristics make it
possible to predict and anticipate the behavior of other drivers.

In addition, humans naturally perceive their surroundings in relative terms,
especially when it involves spatial understanding. This intrinsic way of
processing spatial data based on relative positioning and orientation often
does not align with the fixed Cartesian coordinates commonly used in many
predictive models. However, polar coordinates, which detail a point’s position
based on its distance from a reference and the angle from a reference
direction, echo this human-centric perception. When driving, humans think in
terms like ”slightly ahead and to the right” rather than specific Cartesian
coordinates. Adopting this perspective, our pioneering pooling mechanism, as
illustrated in Fig.1, captures vehicle positions using polar coordinates,
offering a more intuitive representation especially pertinent for trajectory
prediction in AVs.

Despite extensive research in AD trajectory prediction, significant gaps
remain. To bridge these, we’ve combined insights from human behavior and
decision-making to design an innovative behavior-aware trajectory prediction
model. In summary, our work’s principal contributions are:

  * •

We present a novel dynamic geometric graph approach that eliminates the need
for manual labeling during training. This method addresses the challenges of
labeling non-continuous behaviors and selecting appropriate time windows,
while effectively capturing continuous driving behavior. Inspired by traffic
psychology, decision theory, and driving dynamics, our model incorporates
centrality metrics and behavior-aware criteria to provide enhanced flexibility
and accuracy in representing driving behavior. To the best of our knowledge,
this is the first attempt to incorporate continuous representation of
behavioral knowledge in trajectory prediction for AVs.

  * •

We propose a novel pooling mechanism, aligned with human observational
instincts, that extracts vehicle positions in polar coordinates. It simplifies
the representation of direction and distance in Cartesian coordinates,
accounts for road curvature, and allows modeling in complex scenarios such as
roundabouts and intersections.

  * •

We introduce a new Macao Connected Autonomous Driving (MoCAD) dataset, sourced
from a L5 autonomous bus with over 300 hours across campus and busy urban
routes. Characterized by its unique right-hand-drive system, MoCAD, set to be
publicly available, is pivotal for research in right-hand-drive dynamics and
enhancing trajectory prediction models.

  * •

Our model significantly outperforms the SOTA baseline models when tested on
the NGSIM, HighD, RounD, and MoCAD datasets, respectively. Remarkably, it
maintains impressive performance even when trained on only 25.0% of the
dataset, demonstrating exceptional robustness and adaptability in various
traffic scenarios, including highways, roundabouts, and busy urban locales.

## Related Work

A plethora of research has been conducted in the realm of trajectory
prediction, with a diverse array of approaches being proposed. These
approaches can be broadly classified into three categories: physics-based,
statistics-based, and deep learning-based approaches.

Physics-based Approaches. These approaches are primarily divided into kinetic
and kinematic models (Lin, Ulsoy, and LeBlanc 2000). They use principles from
physics and mechanics, taking into account the current state of the vehicle,
such as speed and steering angle, to make predictions (Batz, Watson, and
Beyerer 2009; Wong et al. 2022). Despite their interpretability and
computational efficiency, these methods often exhibit lower prediction
accuracy compared to SOTA techniques (Huang et al. 2022).

Statistics-based Approaches. In contrast, statistical-based approaches, both
parametric and non-parametric, describe predicted trajectories using
predefined maneuver distributions, such as Gaussian processes, hidden Markov
models, dynamic Bayesian networks, and support vector machines (Wang et al.
2021; Li et al. 2020; Xie et al. 2017; Li et al. 2023b). These methods tend to
offer more refined and sophisticated model structures, resulting in better
prediction performance than physics-based approaches. Their experiments on
real-world data showed significant improvements over baselines.

Deep Learning-based Approaches. The surge in popularity of deep learning has
led to extensive research in trajectory prediction for AVs. Recurrent Neural
Networks (RNNs), Convolutional Neural Networks (CNNs), and Transformers
(Vaswani et al. 2017) are among the most widely used approaches, each offering
unique modeling considerations and focuses (Ye, Cao, and Chen 2021; Liang et
al. 2020). RNNs, such as Long Short-Term Memory (LSTM), are often used to
process time-series trajectory data, while CNNs excel at extracting spatial
features from inputs such as bird’s-eye or raster images. Some researchers
combine RNNs and CNNs to integrate both temporal and spatial features into
their models (Liao et al. 2023; Huang, Mo, and Lv 2022; Bhattacharyya, Huang,
and Czarnecki 2023; Zhang and Li 2022). Transformers, with their renowned
success in many domains, have also demonstrated superior performance in
trajectory prediction (Li et al. 2022; Zeng et al. 2023; Li et al. 2023a).
Compared to physics-based and statistics-based methods, these data-driven
approaches have generally demonstrated superior prediction performance,
especially for tasks requiring long-term predictions (beyond 3 seconds).

## Problem Formulation

The trajectory prediction task can be formulated as follows. At each time $t$,
we predict the multimodal trajectories of the ego vehicle, based on historical
observations of both the ego vehicle and its surrounding vehicles (agents).
Given the inputs of historical observations $\bm{X}$, the model aims to
predict a multi-modal distribution over future trajectories of the ego vehicle
$P(\bm{Y}|\bm{X})$.

### Inputs and Outputs

The inputs $\bm{X}$ to our model are the historical trajectories over a fixed
time horizon $t_{h}$ of both the ego vehicle (subscript $0$) and all the
surrounding vehicles (subscripts $1$ to $n$):

$\bm{X}_{i}^{t-t_{h}:t}=\left\\{p_{i}^{t-t_{h}:t}\right\\},\forall i\in[0,n]$
(1)

where $p_{0:n}^{t-t_{h}:t}$ denotes the 2D position coordinates.

The output of the model is a probability distribution over the future
trajectory of the ego vehicle during the prediction horizon $t_{f}$ :

$\bm{Y}=\bm{Y}_{0}^{t+1:t+t_{f}}={\\{y_{0}^{t+1},y_{0}^{t+2},\ldots,y_{0}^{t+t_{f}-1},y_{0}^{t+t_{f}}\\}}$
(2)

As aforementioned, we define the motion of the vehicles in Polar coordinates
(shown in Fig.1) rather than the Cartesian coordinates. In the Polar
coordinate, we assume the origin ${O}$ of the stationary frame of reference is
fixed at the center of the ego vehicle at time $t$. Our inputs and outputs can
be further written as (take the ego vehicle as an example, for convenience,
assume input at instant $t$ and output at instant $t+1$):

$x_{0}^{t}=\\{\rho_{0}^{t},\theta_{0}^{t}\\}$ (3)

and

$y_{0}^{t+1}=\\{\rho_{0}^{t+1},\theta_{0}^{t+1}\\}$ (4)

where $\rho$ and $\theta$ are the distance and angle of the vehicle.

The transformation relationship between Cartesian and Polar coordinate systems
is illustrated below. Given a vehicle’s position history in lateral coordinate
$x_{i}^{t_{k}}$ and longitudinal coordinate $y_{i}^{t_{k}}$ at time $t_{k}$,
the distance $\rho_{i}^{t_{k}}$ and vehicle orientation $\theta_{i}^{t_{k}}$
for Polar representation can be computed as the following formula:

$\left\\{\begin{array}[]{l}\rho_{i}^{t_{k}}=\sqrt{\left(x_{i}^{t_{k}}-x_{0}^{t}\right)^{2}+\left(y_{i}^{t_{k}}-y_{0}^{t}\right)^{2}}\\\
\theta_{i}^{t_{k}}=\arctan\left(\frac{y_{i}^{t_{k}}-y_{0}^{t}}{x_{i}^{t_{k}}-x_{0}^{t}}\right)\end{array}\right.$
(5)

where $x_{0}^{t}$ and $y_{0}^{t}$ are the lateral and longitudinal coordinates
of the ego vehicle (defined as the origin $\mathcal{O}$) at time $t$,
respectively. $\rho_{i}^{t_{k}}$ is the Polar diameter relative to the origin
$\mathcal{O}$, and $\theta_{i}^{t_{k}}$ is the orientation of the $i$th
vehicle at time $t_{k}$.

### Multi-modal Probabilistic Maneuver Prediction

To account for the uncertainty and variability in the prediction, the
multimodal prediction framework considers multiple potential maneuvers that
the ego vehicle could perform and estimates the probability of each maneuver
based on previous observations, as shown in Fig. 2. This not only provides
multiple predictions but also quantifies the confidence level associated with
each prediction. This is particularly beneficial for informed decision-making
in response to anticipated maneuvers, as it allows AVs to account for the
uncertainty inherent in the predictions.

Figure 2: Multi-modal maneuver prediction framework with corresponding
probability outputs.

To provide a formal description, the future trajectories can be hierarchically
predicted in a Bayesian framework at two hierarchical levels: (1) At each time
instant, the probability of different maneuvers $\bm{M}$ of the ego vehicle is
determined; (2) Subsequently, the detailed trajectories of the vehicle
conditional on each maneuver are generated within a predefined distributional
form. In accordance with the characteristics of the driver’s actions during
driving, the possible maneuvers of the vehicles are decomposed into a
combination of two distinctive sub-maneuvers, comprising the positioning-wise
sub-maneuver $\bm{M}_{p}$ and the velocity-wise sub-maneuver $\bm{M}_{v}$. In
addition, the position-wise sub-maneuver includes three discrete driver
choices regarding position changes, namely left lane change, right lane
change, and lane keeping. Meanwhile, the speed-wise sub-maneuver includes
three distinct decisions, including accelerating, braking, and maintaining
speed, which serve as branch categories.

Conditioned on the estimated maneuvers $\bm{M}$ in the first layer, the
probability distribution of multimodal trajectory predictions is assumed to
follow a Gaussian distribution:

$P_{\bm{\Omega}}(\bm{Y}|\bm{M},\bm{X})=N(\bm{Y}|\mu(X),\Sigma(X))$ (6)

where $\bm{\Omega}=\left[\Omega^{t+1},\ldots,\Omega^{t+t_{f}}\right]$ are the
estimable parameters of the distribution, and
$\Omega^{t}=[\mu^{t},\Sigma^{t}]$ is the mean and variance of the distribution
of predicted trajectory point at $t$. Correspondingly, in the second layer,
the multimodal predictions are formulated as a Gaussian Mixture Model, i.e.,

$P(\bm{Y}|\bm{X})=\sum_{\forall
i}P\left(M_{i}|\bm{X}\right)P_{\bm{\Omega}}\left(\bm{Y}|M_{i},\bm{X}\right)$
(7)

where $M_{i}$ is the $i$-th element in $\bm{M}$.

## Proposed Model

Figure 3: Architecture of behavior-aware trajectory prediction model

Fig. 3 shows the architecture of BAT, which is upon the encoder-decoder
framework with four modules to capture different aspects of behaviors and
interactions between different agents, including behavior-aware, interaction-
aware, priority-aware, and position-aware modules.

### Behavior-aware Module

The complex and dynamic nature of traffic scenarios presents significant
challenges in interpreting and categorizing driver behavior. Unlike previous
studies that categorize driver behavior into finite and human-defined
classifications, we present a more flexible and adaptable solution, namely the
behavior-aware module, by avoiding discrete behavior categories in favor of a
continuous representation of behavioral information. Our behavior-aware module
is motivated by the multi-policy decision-making (MPDM) framework for human
drivers (Markkula et al. 2020) and integrates traffic psychology (Toghi et al.
2022) using dynamic geometric graphs (DGGs) (Dall and Christensen 2002) to
model and evaluate human driving behavior.

#### Dynamic Geometric Graphs

At time $t$, the graph $G^{t}$ can be given as follow:

${G}^{t}=\\{V^{t},{E}^{t}\\}$ (8)

where $V^{t}=\\{{v}_{0}^{t},{v}_{1}^{t}\ldots,{v}_{n}^{t}\\}$ is the set of
nodes, ${v}_{i}^{t}$ is the $i$-th node representing the $i$-th vehicle,
${E^{t}}=\\{{e_{0}^{t}},{e_{1}^{t}}\ldots,{e_{n}^{t}}\\}$ is the set of
undirected edges, and ${e_{i}^{t}}$ is the edge between the node ${v}_{i}^{t}$
and other vehicles that have potential influences on it. It is assumed that
the interaction only exists when the nodes $v_{i}$ and $v_{j}$ are in close
proximity to one another, or formally, the shortest distance between them,
$d\left(v_{i}^{t},v_{j}^{t}\right)$, is less than or equal to the
predetermined distance threshold $r$. Therefore, we define

${e_{i}^{t}}=\\{{v_{i}^{t}}{v_{j}^{t}}\mid(j\in{N}_{i}^{t})\\}$ (9)

where ${N}_{i}^{t}=\left\\{v_{j}^{t}\in
V^{t}\backslash\left\\{v_{i}^{t}\right\\}\mid
d\left(v_{i}^{t},v_{j}^{t}\right)\leq r,i\neq j\right\\}$.

Correspondingly, the symmetrical adjacency matrix $A^{t}$ of $G^{t}$ can be
given as:

$A^{t}(i,j)=\begin{cases}d\left(v_{i}^{t},v_{j}^{t}\right)&\text{ if
}d\left(v_{i}^{t},v_{j}^{t}\right)\leq{r},i\neq j\\\
0&\text{otherwise}\end{cases}$ (10)

#### Centrality Measures

To more accurately capture the potential interactions between the observed
traffic agents, we use centrality measures (degree, closeness, and eigenvector
centrality measures) (Freeman 1978) as prior knowledge to portray and describe
driving behavior in DGGs, as shown in Table 1.

Degree Centrality. Degree centrality is characterized by the count of
immediate connections a node has with other nodes within the graph. This
concept intuitively suggests that a traffic agent with more connections is
both more susceptible to the influences of other agents and more influential
in shaping their actions. Formally,

$\mathcal{J}_{i}^{t}(D)=\left|\mathcal{N}_{i}^{t}\right|+\mathcal{J}_{i}^{t-1}(D)$
(11)

where $\left|\mathcal{N}_{i}^{t}\right|$ is the total number of elements in
$\mathcal{N}_{i}^{t}$ at time $t$.

Closeness Centrality. We propose that the closer a vehicle is to its
surroundings, the higher its likelihood of interacting with adjacent vehicles.
This idea is encapsulated by the closeness centrality metric, which gauges the
ease of interaction and accessibility between a vehicle and its neighboring
vehicles. Closeness centrality is determined using the shortest paths between
the vehicle (node) and other vehicles in the traffic graph. This is achieved
by summing the inverse of their distances. Formally,

$\mathcal{J}_{i}^{t}(C)=\frac{\left|\mathcal{N}_{i}^{t}\right|-1}{\sum_{\forall
v_{j}^{t}\in\mathcal{N}^{t}_{i}}d\left(v_{i}^{t},v_{j}^{t}\right)}$ (12)

Table 1: Centrality measures and their interpretations

Centrality Measures | Magnitude (Original Measure) | Gradient (1st Derivative)
---|---|---
Degree | Agent’s potential and capability for interaction in the traffic environment | Agent’s sensitivity to traffic density variations
Closeness | Agent’s significance in dynamic traffic scenarios | Variation in agent’s importance in dynamic traffic scenes
Eigenvector | Extent of influence an agent exerts on others via direct and indirect interactions at time t | Agent’s capacity to modify interactions in complex traffic scenarios

Eigenvector Centrality. In the context of understanding driver behavior, a
vehicle’s eigenvector centrality takes into account both its interactions with
nearby vehicles and the influence of those interactions. Specifically, this
metric integrates the vehicle’s number of connections and the weight of the
influence of connected vehicles. This helps to identify influential vehicles
in a traffic context and their potential impact on other drivers. Formally,

$\mathcal{J}_{i}^{t}(E)=\frac{\sum_{\forall
v_{j}^{t}\in\mathcal{N}^{t}_{i}}d\left(v_{i}^{t},v_{j}^{t}\right)}{\lambda}$
(13)

where $\lambda$ is the eigenvalue. In addition, the Perron-Frobenius theorem
states that for a non-negative matrix (such as the adjacency matrix in our
case), there exists a positive eigenvector solution for the greatest
eigenvalue of the matrix (Pillai, Suel, and Cha 2005). This means that the
eigenvector corresponding to the greatest eigenvalue of the adjacency matrix
can be used to compute the eigenvector centrality measure of the nodes in the
graph.

#### Behavior-aware Criterion

The behavior-aware criterion is devised to mirror human-like trajectory
predictions by leveraging the analytical properties of centrality measures.
This aids in detecting and comprehending human driving behavior. By doing so,
it removes the necessity for manual labeling, addressing issues like labeling
non-continuous behaviors and choosing optimal time frames. Furthermore, this
criterion effectively encapsulates continuous driving behaviors. Incorporating
this with the Behavior Likelihood Estimate (BLE) and Behavior Intensity
Estimate (BIE) refines prediction accuracy and dependability in fluctuating
and intricate traffic conditions.

Behavior Likelihood Estimate. The BLE criterion quantifies behavior
probabilities using time-based row derivatives, even without explicit behavior
classifications. A higher probability of a behavior is indicated by prominent
row derivatives and local extrema. For $v_{i}^{t}$ at time $t$, the BLE
considering all three centrality measures is as follows:

$\operatorname{\mathcal{I}}_{i}^{t}=\left[\left|\frac{\partial\mathcal{J}^{t}_{i}(D)}{\partial
t}\right|,\left|\frac{\partial\mathcal{J}^{t}_{i}(C)}{\partial
t}\right|,\left|\frac{\partial\mathcal{J}^{t}_{i}(E)}{\partial
t}\right|\right]^{T}$ (14)

where $\left|\cdot\right|$ denotes the absolute value operator.

Behavior Intensity Estimate. The BIE quantifies the potential impact intensity
of a driving behavior on surrounding vehicles. It takes into account the
duration of the behavior, with longer-lasting behaviors assumed to have a
greater impact than those that are brief. The BIE for the node $v_{i}^{t}$ at
time $t$ is build on top of BLE, and is defined as:

$\displaystyle\operatorname{{\mathcal{L}}}_{i}^{t}=\left|\frac{\partial\mathcal{\operatorname{{\mathcal{I}}}}_{i}^{t}}{\partial
t}\right|=\left[\left|\frac{\partial^{2}\mathcal{J}_{i}^{t}(D)}{\partial^{2}t}\right|,\left|\frac{\partial^{2}\mathcal{J}_{i}^{t}(C)}{\partial^{2}t}\right|,\left|\frac{\partial^{2}\mathcal{J}_{i}^{t}(E)}{\partial^{2}t}\right|\right]^{T}$
(15)

In summary, the BIE in conjunction with the BLE as prior knowledge provides a
comprehensive understanding of individual driver behavior. This is achieved by
segmenting each traffic scene into behavior-aware regions centered around
observed agents of the ego vehicle. For these regions, behavioral features are
extracted from the traffic agents, including contextual information (as shown
in Fig. 1). These features are then embedded and encoded frame by frame by an
LSTM network to generate high-dimensional behavior vectors. By combining
insights into the probability and intensity of the behavior, the overall
impact on the surrounding traffic is determined. This infusion of human-like
reasoning aligns with human perception and cognition, improving the accuracy
and efficiency of trajectory prediction for AVs.

### Interaction-aware Module

To capture and assemble interactions between the ego vehicle and its
surrounding agents, we introduce an innovative interaction-aware pooling
mechanism. Unlike conventional methods (Deo and Trivedi 2018a; Chen et al.
2022b; Wang et al. 2023), which treat agents as fixed-size grid cells with
Cartesian coordinates, our mechanism uses relative displacements in polar
coordinates. This better adapts to complex non-regularized scenarios such as
roundabouts and irregular intersections. Hence, we apply a hierarchical LSTM
encoder and a position encoding layer to enhance the effective and efficient
awareness and aggregation of interactions between the ego vehicle and its
surrounding vehicles in the context of highly interactive driving scenarios.
The LSTM encoder effectively captures interactions and dynamic motion between
surrounding agents. At each discrete time step, the LSTM encoder learns the
most recent $t_{h}$ frames of historical trajectory information for both the
ego vehicle and its observed agents, embedding their positions in a temporal
order. The hidden position states of these vehicles are then updated by the
LSTM on a frame-by-frame basis, with the weights of the LSTM shared across all
vehicles. The position data is then represented in polar coordinates and
mapped through the position encoding layer to capture higher-order interactive
information.

### Priority-aware Module

The priority-aware module uses an attention mechanism layer to compute dynamic
attention weight vectors for surrounding agents based on their higher-order
interactive information. This attention mechanism (Vaswani et al. 2017)
assigns weights that indicate their importance in predicting the ego vehicle’s
trajectory. These weight vectors express the relative importance of the agents
and are used to weight higher-order interaction data in later stages. They are
then fed into a multi-layer perceptron (MLP) to produce high-dimensional
aggregate pooling vectors through a max-pooling layer.

### Position-aware Module

To further enhance the modeling of temporal dependencies and spatial
relationships, this module employs a dedicated LSTM network to encode and
learn the dynamic position of the ego vehicle. The historical trajectory of
the ego vehicle is also represented in polar coordinates and subsequently
embedded using an LSTM. This refinement augments the model’s capability to
capture the details of the agent’s trajectory.

### Decoder

The position vector of the ego vehicle is integrated with additional
information about the hidden pooling vectors and the high-dimensional behavior
vector. This composite undergoes embedding by a softmax activation function
(behavior embedding), followed by processing by an MLP (behavior encoding).
Finally, the processed input is analyzed by the LSTM decoder, which generates
a probability distribution over the possible future trajectories of the ego
vehicle.

## Experiments

We evaluate the effectiveness of our model using four datasets: NGSIM (Deo and
Trivedi 2018a), HighD (Krajewski et al. 2018), RounD (Krajewski et al. 2020),
and MoCAD. These datasets, sourced from varied and intricate real-world
traffic situations like highways, roundabouts, and urban locales, serve as a
comprehensive testing ground. To gauge our model’s precision, we employed the
Root Mean Square Error (RMSE) metric.

### Experimental Setup

These data sets were partitioned into training, validation, and test sets
using standard sampling. We refer to the complete test set as the overall test
set. The trajectories for the NGSIM, HighD, and MoCAD datasets were divided
into 8-second intervals. The first 3 seconds served as the trajectory history
($t_{h}=3$) for input, and the following 5 seconds represented the ground
truth ($t_{f}=5$) for output. For the RounD dataset, the trajectories were
divided into 6-second chunks with $t_{h}=2$ and $t_{f}=4$.

To delve deeper into our model’s performance, the NGISM dataset was further
split based on distinct vehicular maneuvers, including no lane-change (keep),
on-ramp lane merging (merge), right lane-change (right), and left lane-change
(left). This subset, termed the maneuver-based test set, allowed for a more
granular examination of our model’s capabilities across different traffic
actions.

### Training and Implementation Details

Our model is trained to converge using an NVIDIA A100 40GB GPU. We introduce
the Negative Log-Likelihood criterion as a complement to the RMSE criterion in
the loss function. Our model is based on a Gaussian Mixture Model (GMM) with a
multimodal predictive structure. The encoder comprises behavior-aware,
interaction-aware, and position-aware LSTMs with dimensions of 32, 64, and 64,
respectively, while the LSTM decoder uses 128 state dimensions. Training is
performed for 12 epochs with the Adam optimizer (lr=0.001, batch size=256). To
enhance training stability, we employ the CosineAnnealingWarmRestarts
scheduler and set a threshold distance of 25 feet, which is the average
headway. Additionally, for maneuver-based models, evaluating the correctness
of predicted trajectories concerning intended maneuvers is crucial. The NLL
criterion proves valuable in assessing the model’s ability to capture the
underlying dynamics and constraints of the vehicle’s motion during different
maneuvers, ensuring reliable trajectory predictions. To address the
potentially detrimental effects of misclassifying maneuver types on trajectory
prediction accuracy and robustness, we propose complementing the RMSE
criterion with the NLL criterion in the loss function. This diversity loss
term incentivizes the model to generate trajectory predictions consistent with
intended maneuvers, promoting trajectory diversity.

Table 2: Evaluation results for BAT and the baselines in the overall test set
over a different horizon. Note: RMSE (m) is the evaluation metric, where lower
values indicate better performance, with some not specifying (’-’). Values in
bold represent the best performance in each category.

Dataset | Model | Prediction Horizon (s)
---|---|---
1 | 2 | 3 | 4 | 5
NGSIM | S-LSTM (Alahi et al. 2016) | 0.65 | 1.31 | 2.16 | 3.25 | 4.55
S-GAN (Gupta et al. 2018) | 0.57 | 1.32 | 2.22 | 3.26 | 4.40
CS-LSTM (Deo and Trivedi 2018a) | 0.61 | 1.27 | 2.09 | 3.10 | 4.37
MATF-GAN (Zhao et al. 2019) | 0.66 | 1.34 | 2.08 | 2.97 | 4.13
NLS-LSTM (Messaoud et al. 2019) | 0.56 | 1.22 | 2.02 | 3.03 | 4.30
M-LSTM (Deo and Trivedi 2018b) | 0.58 | 1.26 | 2.12 | 3.24 | 4.66
IMM-KF (Lefkopoulos et al. 2020) | 0.58 | 1.36 | 2.28 | 3.37 | 4.55
GAIL-GRU (Kuefler et al. 2017) | 0.69 | 1.51 | 2.55 | 3.65 | 4.71
MFP (Tang and Salakhutdinov 2019) | 0.54 | 1.16 | 1.89 | 2.75 | 3.78
DRBP(Gao et al. 2023) | 1.18 | 2.83 | 4.22 | 5.82 | -
DN-IRL (Fernando et al. 2019) | 0.54 | 1.02 | 1.91 | 2.43 | 3.76
WSiP (Wang et al. 2023) | 0.56 | 1.23 | 2.05 | 3.08 | 4.34
| CF-LSTM (Xie et al. 2021) | 0.55 | 1.10 | 1.78 | 2.73 | 3.82
| MHA-LSTM (Messaoud et al. 2021) | 0.41 | 1.01 | 1.74 | 2.67 | 3.83
| HMNet (Xue et al. 2021) | 0.50 | 1.13 | 1.89 | 2.85 | 4.04
| TS-GAN (Wang et al. 2022b) | 0.60 | 1.24 | 1.95 | 2.78 | 3.72
| STDAN (Chen et al. 2022b) | 0.39 | 0.96 | 1.61 | 2.56 | 3.67
| BAT (25%) | 0.31 | 0.85 | 1.65 | 2.69 | 3.87
| BAT | 0.23 | 0.81 | 1.54 | 2.52 | 3.62
HighD | S-LSTM (Alahi et al. 2016) | 0.22 | 0.62 | 1.27 | 2.15 | 3.41
S-GAN (Gupta et al. 2018) | 0.30 | 0.78 | 1.46 | 2.34 | 3.41
WSiP (Wang et al. 2023) | 0.20 | 0.60 | 1.21 | 2.07 | 3.14
CS-LSTM(M) (Deo and Trivedi 2018a) | 0.23 | 0.65 | 1.29 | 2.18 | 3.37
CS-LSTM (Deo and Trivedi 2018a) | 0.22 | 0.61 | 1.24 | 2.10 | 3.27
MHA-LSTM (Messaoud et al. 2021) | 0.19 | 0.55 | 1.10 | 1.84 | 2.78
MHA-LSTM(+f) (Messaoud et al. 2021) | 0.06 | 0.09 | 0.24 | 0.59 | 1.18
| NLS-LSTM (Messaoud et al. 2019) | 0.20 | 0.57 | 1.14 | 1.90 | 2.91
| DRBP(Gao et al. 2023) | 0.41 | 0.79 | 1.11 | 1.40 | -
| EA-Net (Cai et al. 2021) | 0.15 | 0.26 | 0.43 | 0.78 | 1.32
| CF-LSTM (Xie et al. 2021) | 0.18 | 0.42 | 1.07 | 1.72 | 2.44
| STDAN (Chen et al. 2022b) | 0.19 | 0.27 | 0.48 | 0.91 | 1.66
| iNATran (M) (Chen et al. 2022a) | 0.04 | 0.05 | 0.21 | 0.54 | 1.11
| iNATran (Chen et al. 2022a) | 0.04 | 0.05 | 0.21 | 0.54 | 1.10
| BAT (25%) | 0.14 | 0.34 | 0.65 | 0.89 | 1.27
| BAT | 0.08 | 0.14 | 0.20 | 0.44 | 0.62
RounD | S-LSTM (Alahi et al. 2016) | 0.94 | 1.82 | 3.43 | 5.21 | -
S-GAN (Gupta et al. 2018) | 0.72 | 1.57 | 3.01 | 4.78 | -
CS-LSTM(M) (Deo and Trivedi 2018a) | 0.74 | 1.43 | 2.44 | 4.21 | -
CS-LSTM (Deo and Trivedi 2018a) | 0.71 | 1.21 | 2.09 | 3.92 | -
MATH (Hasan et al. 2021) | 0.38 | 0.80 | 1.76 | 3.08 | -
MHA-LSTM (Messaoud et al. 2021) | 0.62 | 0.98 | 1.88 | 3.65 | -
MHA-LSTM(+f) (Messaoud et al. 2021) | 0.51 | 0.91 | 1.80 | 3.57 | -
| NLS-LSTM (Messaoud et al. 2019) | 0.62 | 0.96 | 1.91 | 3.48 | -
| WSiP (Wang et al. 2023) | 0.52 | 0.99 | 1.88 | 3.07 | -
| CF-LSTM (Xie et al. 2021) | 0.51 | 0.87 | 1.79 | 3.14 | -
| STDAN (Chen et al. 2022b) | 0.35 | 0.77 | 1.74 | 2.92 | -
| BAT (25%) | 0.32 | 0.72 | 1.99 | 3.12 | -
| BAT | 0.23 | 0.55 | 1.43 | 2.46 | -
MoCAD | S-LSTM (Alahi et al. 2016) | 1.73 | 2.46 | 3.39 | 4.01 | 4.93
S-GAN (Gupta et al. 2018) | 1.69 | 2.25 | 3.30 | 3.89 | 4.69
CS-LSTM(M) (Deo and Trivedi 2018a) | 1.49 | 2.07 | 3.02 | 3.62 | 4.53
CS-LSTM (Deo and Trivedi 2018a) | 1.45 | 1.98 | 2.94 | 3.56 | 4.49
MHA-LSTM (Messaoud et al. 2021) | 1.25 | 1.48 | 2.57 | 3.22 | 4.20
MHA-LSTM(+f) (Messaoud et al. 2021) | 1.05 | 1.39 | 2.48 | 3.11 | 4.12
NLS-LSTM (Messaoud et al. 2019) | 0.96 | 1.27 | 2.08 | 2.86 | 3.93
| WSiP (Wang et al. 2023) | 0.70 | 0.87 | 1.70 | 2.56 | 3.47
| CF-LSTM (Xie et al. 2021) | 0.72 | 0.91 | 1.73 | 2.59 | 3.44
| STDAN (Chen et al. 2022b) | 0.62 | 0.85 | 1.62 | 2.51 | 3.32
| BAT (25%) | 0.65 | 0.99 | 1.89 | 2.81 | 3.58
| BAT | 0.35 | 0.74 | 1.39 | 2.19 | 2.88

### Experimental Results

We evaluated our model against various SOTA trajectory prediction methods from
2016 to 2023. The results, displayed in Table 2, highlight our model’s
significant advancements in trajectory prediction over current SOTA baselines.
Using RMSE as the evaluation metric, our model surpasses recent baselines
(2021-2023) by 2.6% for short-term predictions (1s-3s) and reduces prediction
error by 56.7% for long-term predictions (4s-5s) on the NGSIM dataset. On the
HighD dataset, known for its superior data volume and precision, our model
significantly outperforms most baselines, showing improvements of 62.7% and
43.6% compared to STDAN and iNATran, respectively, over a 5-second horizon.

The strengths of BAT become more evident in complex scenarios, like urban
streets and unstructured roads (RounD and MoCAD datasets). Here, our model
consistently surpasses current SOTA baselines, with accuracy gains between
17.8%-75.5% on RounD and 12.7%-79.8% on MoCAD. Such improvements underscore
the significance of factoring in driving behavior and our relative distance
pooling mechanism, especially in dense traffic scenarios. For scalability
testing, even when our model was trained on just 25% of the training data, it
still managed to outperform most baselines, indicating a potential reduction
in data needs for training AVs in challenging contexts.

We also conducted tests on the maneuver-based test set, as detailed in Table
3. Specifically, in the merge and right test subsets, our model achieves
significantly lower RMSE values than the SOTA baselines, demonstrating an
improvement of at least 10.1% for a prediction horizon of 5 seconds, which
could significantly mitigate the risk of traffic accidents. Moreover, our
model shows remarkable improvement in the keep and left test subsets,
highlighting its robustness and effectiveness in accurately predicting future
vehicle trajectories in various driving scenarios and maneuvers.

Overall, our findings affirm our model’s capability and efficiency in
predicting vehicle trajectories for AVs.

### Ablation Studies

Table 4 presents an analysis of four critical components: polar coordinates,
behavior-aware, interaction-aware, and priority-aware modules. We tested five
models: Model A (using Cartesian coordinates), Model B (excluding the
behavior-aware module), Model C (excluding the interaction-aware module),
Model D (excluding the priority-aware module), and Model E (with all
components).

On evaluating against the NGSIM and RounD datasets, all stripped-down versions
(A-D) underperformed compared to the comprehensive Model E. Notably, the
integration of interaction-aware and priority-aware modules significantly
boosted performance, underlining their importance in enhancing prediction
accuracy.

Table 3: Evaluation results for the proposed model and the baselines in
maneuver-base test set for NGSIM dataset.

Dataset | keep | merge
---|---|---
Model | Horizon (s) | Horizon (s)
1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5
S-LSTM (Alahi et al. 2016) | 0.35 | 1.01 | 1.81 | 2.82 | 4.15 | 0.81 | 1.31 | 2.51 | 4.01 | 5.78
S-GAN (Gupta et al. 2018) | 0.36 | 1.01 | 1.81 | 2.83 | 4.15 | 0.71 | 1.32 | 2.53 | 4.11 | 5.97
CS-LSTM (Deo and Trivedi 2018a) | 0.34 | 0.98 | 1.75 | 2.77 | 4.06 | 0.61 | 1.34 | 2.58 | 4.12 | 5.94
MATF-GAN (Zhao et al. 2019) | 0.37 | 1.11 | 1.74 | 2.66 | 3.91 | 0.53 | 1.41 | 2.56 | 3.97 | 5.52
WSiP (Wang et al. 2023) | 0.32 | 0.89 | 1.58 | 2.51 | 3.59 | 0.40 | 1.18 | 2.41 | 3.72 | 5.16
HMNet (Xue et al. 2021) | 0.31 | 0.83 | 1.56 | 2.51 | 3.68 | 0.34 | 1.17 | 2.32 | 3.63 | 5.20
STDAN (Chen et al. 2022b) | 0.28 | 0.85 | 1.52 | 2.53 | 3.49 | 0.28 | 1.19 | 2.21 | 3.67 | 4.95
BAT (25%) | 0.28 | 0.86 | 1.54 | 2.52 | 3.73 | 0.31 | 0.95 | 1.95 | 3.31 | 4.98
BATaj | 0.23 | 0.81 | 1.49 | 2.44 | 3.66 | 0.25 | 0.89 | 1.83 | 3.04 | 4.45
|  |  |  |  |  |  |  |  |  |
Dataset | left | right
Model | Horizon (s) | Horizon (s)
1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5
S-LSTM (Alahi et al. 2016) | 0.77 | 1.68 | 3.04 | 4.67 | 6.59 | 0.69 | 1.97 | 3.81 | 6.17 | 9.09
S-GAN (Gupta et al. 2018) | 0.66 | 1.68 | 3.11 | 4.85 | 6.87 | 0.72 | 1.97 | 3.91 | 6.32 | 9.23
CS-LSTM (Deo and Trivedi 2018a) | 0.54 | 1.63 | 3.01 | 4.71 | 6.63 | 0.61 | 2.01 | 3.97 | 6.48 | 9.48
MATF-GAN (Zhao et al. 2019) | 0.61 | 1.72 | 3.02 | 4.62 | 6.34 | 0.56 | 1.88 | 3.90 | 6.07 | 9.01
WSiP (Wang et al. 2023) | 0.41 | 1.46 | 2.82 | 4.42 | 6.22 | 0.52 | 1.61 | 3.60 | 5.78 | 8.45
HMNet (Xue et al. 2021) | 0.41 | 1.31 | 2.87 | 4.47 | 6.33 | 0.49 | 1.62 | 3.47 | 5.87 | 8.59
STDAN (Chen et al. 2022b) | 0.35 | 1.33 | 2.84 | 4.51 | 5.97 | 0.38 | 1.49 | 3.46 | 5.87 | 7.93
BAT (25%) | 0.43 | 1.24 | 2.43 | 4.01 | 5.91 | 0.47 | 1.41 | 3.09 | 5.19 | 7.87
BAT | 0.33 | 1.07 | 2.24 | 3.73 | 5.51 | 0.31 | 1.36 | 2.96 | 5.15 | 6.78

Table 4: Ablation results for different models on NGSIM and RounD datasets
(Evaluation metric: RMSE (m))

Dataset |  Time (s) |  Model ($\Delta$Method E)
---|---|---
A |  B |  C |  D |  E
NGSIM |  1 |  0.27 |  0.30 |  0.27 |  0.28 |  0.23
2 |  0.86 |  0.89 |  0.85 |  0.87 |  0.81
3 |  1.63 |  1.68 |  1.60 |  1.63 |  1.54
4 |  2.65 |  2.68 |  2.62 |  2.64 |  2.52
5 |  4.02 |  4.08 |  3.97 |  3.97 |  3.62
RounD |  1 |  0.76 |  0.55 |  0.44 |  0.35 |  0.23
2 |  0.94 |  0.83 |  0.76 |  0.72 |  0.55
3 |  1.87 |  1.72 |  1.63 |  1.54 |  1.43
4 |  3.02 |  2.82 |  2.76 |  2.68 |  2.46

The behavior-aware module’s inclusion significantly enhanced performance by
capturing dynamic vehicular interactions, vital for accurate trajectory
prediction. By factoring in surrounding vehicles’ behavior, BAT predicts the
ego vehicle’s trajectory more insightfully. This mirrors human decision-
making, where actions and intentions of other agents, including vehicles,
shape trajectory predictions (Baron 2000).

Furthermore, adopting the polar coordinate system in Model C outperformed the
Cartesian approach, especially in roundabout environments like RounD. This
aligns with studies on human perception, suggesting people process goal-
relevant information distinctively (Todd and Gigerenzer 2000). The polar
system better reflects human cognition of spatial vehicular relationships,
emphasizing the significance of both behavioral and spatial considerations in
trajectory prediction.

### Ablation Study for Distance Threshold

We further explored the effect of the proper setting of the threshold distance
$d\left(v_{i}^{t},v_{j}^{t}\right)$ of two vehicles on model performance. To
this end, we conducted experiments using three different distance threshold
parameters: 0 feet (0 meters), 25 feet (7.62 meters), and 50 feet (15.24
meters). The results, depicted in Fig. 2, reveal that when the predefined
distance threshold parameter r is set to 0 feet (0 meters), the interaction
between vehicles is disregarded, leading to a significant deterioration in the
model’s ability to predict trajectories. Conversely, when $r=25$ feet, the
model’s performance for trajectory prediction is significantly improved, while
it declines at $r=50$ feet.

These findings align with research on human attention and decision-making,
which suggests that humans tend to prioritize information that is relevant to
their current goals and that is within their immediate visual field (Kahneman
1973). This is likely because the brain has limited processing resources and
must prioritize information in order to make efficient decisions. In the
context of driving, this would mean that human drivers are more likely to pay
attention to and be influenced by the movements of nearby vehicles. By setting
the distance threshold to 25 feet (7.62 meters), the model is able to capture
the complex and dynamic interactions between vehicles that are most likely to
impact the ego vehicle’s trajectory, in a manner that is consistent with human
attention and decision-making.

Figure 4: Ablation results for different distance thresholds Figure 5: Multi-
modal probabilistic prediction of the ego vehicle. The heat maps depict the
Gaussian Mixture Model of predicted outcomes at each time step, where brighter
colors indicate higher probabilities. Figure 6: Visualizations and heat maps
selected from the NGSIM (a-b) and RounD (c-d) datasets. The target vehicle is
depicted in orange, while its surrounding vehicles are shown in blue. The
darkness of the blue color indicates the higher importance weight of the
surrounding vehicle.

### Intuition and Interpretability Analysis

Figure 5 illustrates our model’s multi-modal probabilistic prediction
performance on the NGSIM dataset. To further underscore the prowess of BAT, we
visually dissect its prediction outcomes across diverse scenarios in Fig. 6.
For the sake of clarity, we spotlight solely the trajectories deemed most
probable for the ego vehicle in each context. We meticulously chose two
demanding driving situations: transitioning into the right lane (Fig.6 (a-b))
and maneuvering through a roundabout (Fig.6 (c-d)). Intriguingly, the heat
maps vividly unveil a direct relationship between the proximity of the ego to
neighboring vehicles and their respective significance. This exposes
pronounced social interplay among the nearby agents. In Fig. 6 (a), the
bottom-most vehicle with the red circle exhibits friendly driving behavior,
intuitively creating ample space for the ego vehicle’s lane transition. On the
flip side, the vehicle with the red circle in Fig. 6 (b) manifests aggressive
driving tendencies, potentially accelerating to impede the ego vehicle’s lane
merge. Herein lies the genius of BAT’s behavior-aware module: it discerns
driver personas, predicting the ego vehicle’s inability to seamlessly merge,
aligning impeccably with the ground truth. Conversely, models bereft of this
driving behavior consideration falter, such as Stdan and WSiP, deviating
significantly from the actual trajectory.

In addition, BAT captures the influence of agents even from non-adjacent
lanes, attributing this to their distinct driving behavior—a facet frequently
sidestepped in conventional studies. To sum up, BAT doesn’t just predict; it
observes, interprets, and decides like a human. By mirroring human decision-
making, BAT offers a promising leap toward autonomous driving that’s both
accurate and reliable.

## Conclusion

Predicting the trajectories of surrounding vehicles with a high degree of
accuracy is a fundamental challenge that must be addressed in the quest for
full AVs. To address this challenge, we propose a behavior-aware modular model
with four components: behavior-aware, interaction-aware, priority-aware, and
position-aware modules. Our model outperforms current SOTA baselines in terms
of prediction accuracy and efficiency on the NGSIM, HighD, RounD, and MoCAD
datasets, even when trained on 25% training set, demonstrating its robustness,
applicability, and potential to reduce training data requirements for AVs in
challenging or unusual situations such as corner cases, roundabouts.

## Acknowledgement

This research is supported by the Science and Technology Development Fund of
the Macau SAR (File No. 0021/2022/ITP, 0081/2022/A2, 0015/2019/AKP, SKL-
IoTSC(UM)-2021-2023/ORP/GA08/2022, SKL-IoTSC(UM)-2024-2026/ORP/GA06/2023) and
the University of Macau (SRG2023-00037-IOTSC). Part of this research is
carried out at SICC with support from SKL-IOTSC, University of Macau. For any
correspondence regarding this work, please contact Dr. Zhenning Li
<EMAIL_ADDRESS>and Dr. Chengzhong Xu (czxu@um.edu.mo).

## References

  * Alahi et al. (2016) Alahi, A.; Goel, K.; Ramanathan, V.; Robicquet, A.; Fei-Fei, L.; and Savarese, S. 2016.  Social lstm: Human trajectory prediction in crowded spaces.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 961–971.
  * Baron (2000) Baron, J. 2000.  _Thinking and deciding_.  Cambridge University Press.
  * Batz, Watson, and Beyerer (2009) Batz, T.; Watson, K.; and Beyerer, J. 2009.  Recognition of dangerous situations within a cooperative group of vehicles.  In _2009 IEEE Intelligent Vehicles Symposium_ , 907–912. IEEE.
  * Bhattacharyya, Huang, and Czarnecki (2023) Bhattacharyya, P.; Huang, C.; and Czarnecki, K. 2023.  Ssl-lanes: Self-supervised learning for motion forecasting in autonomous driving.  In _Conference on Robot Learning_ , 1793–1805. PMLR.
  * Cai et al. (2021) Cai, Y.; Wang, Z.; Wang, H.; Chen, L.; Li, Y.; Sotelo, M. A.; and Li, Z. 2021.  Environment-attention network for vehicle trajectory prediction.  _IEEE Transactions on Vehicular Technology_ , 70(11): 11216–11227.
  * Chandra et al. (2020) Chandra, R.; Bhattacharya, U.; Mittal, T.; Bera, A.; and Manocha, D. 2020.  CMetric: A Driving Behavior Measure Using Centrality Functions.  _arXiv preprint arXiv:2003.04424_.
  * Chen et al. (2022a) Chen, X.; Zhang, H.; Zhao, F.; Cai, Y.; Wang, H.; and Ye, Q. 2022a.  Vehicle trajectory prediction based on intention-aware non-autoregressive transformer with multi-attention learning for Internet of Vehicles.  _IEEE Transactions on Instrumentation and Measurement_ , 71: 1–12.
  * Chen et al. (2022b) Chen, X.; Zhang, H.; Zhao, F.; Hu, Y.; Tan, C.; and Yang, J. 2022b.  Intention-aware vehicle trajectory prediction based on spatial-temporal dynamic attention network for Internet of Vehicles.  _IEEE Transactions on Intelligent Transportation Systems_ , 23(10): 19471–19483.
  * Dall and Christensen (2002) Dall, J.; and Christensen, M. 2002.  Random geometric graphs.  _Physical review E_ , 66(1): 016121.
  * Deo and Trivedi (2018a) Deo, N.; and Trivedi, M. M. 2018a.  Convolutional social pooling for vehicle trajectory prediction.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 1468–1476.
  * Deo and Trivedi (2018b) Deo, N.; and Trivedi, M. M. 2018b.  Multi-modal trajectory prediction of surrounding vehicles with maneuver based lstms.  In _2018 IEEE intelligent vehicles symposium (IV)_ , 1179–1184. IEEE.
  * Fernando et al. (2019) Fernando, T.; Denman, S.; Sridharan, S.; and Fookes, C. 2019.  Neighbourhood context embeddings in deep inverse reinforcement learning for predicting pedestrian motion over long time horizons.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops_ , 0–0.
  * Freeman (1978) Freeman, L. C. 1978.  Centrality in social networks conceptual clarification.  _Social networks_ , 1(3): 215–239.
  * Gao et al. (2023) Gao, K.; Li, X.; Chen, B.; Hu, L.; Liu, J.; Du, R.; and Li, Y. 2023.  Dual Transformer Based Prediction for Lane Change Intentions and Trajectories in Mixed Traffic Environment.  _IEEE Transactions on Intelligent Transportation Systems_.
  * Gupta et al. (2018) Gupta, A.; Johnson, J.; Fei-Fei, L.; Savarese, S.; and Alahi, A. 2018.  Social gan: Socially acceptable trajectories with generative adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2255–2264.
  * Hang, Lv, and Chen (2022) Hang, P.; Lv, C.; and Chen, X. 2022.  _Human-Like Decision Making and Control for Autonomous Driving_.  CRC Press.
  * Hasan et al. (2021) Hasan, M.; Solernou, A.; Paschalidis, E.; Wang, H.; Markkula, G.; and Romano, R. 2021.  Maneuver-aware pooling for vehicle trajectory prediction.  _arXiv preprint arXiv:2104.14079_.
  * Huang et al. (2022) Huang, Y.; Du, J.; Yang, Z.; Zhou, Z.; Zhang, L.; and Chen, H. 2022.  A Survey on Trajectory-Prediction Methods for Autonomous Driving.  _IEEE Transactions on Intelligent Vehicles_.
  * Huang, Mo, and Lv (2022) Huang, Z.; Mo, X.; and Lv, C. 2022.  Multi-modal Motion Prediction with Transformer-based Neural Network for Autonomous Driving.  In _2022 International Conference on Robotics and Automation (ICRA)_ , 2605–2611.
  * Kahneman (1973) Kahneman, D. 1973.  _Attention and effort_ , volume 1063.  Citeseer.
  * Krajewski et al. (2018) Krajewski, R.; Bock, J.; Kloeker, L.; and Eckstein, L. 2018.  The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems.  In _2018 21st International Conference on Intelligent Transportation Systems (ITSC)_ , 2118–2125.
  * Krajewski et al. (2020) Krajewski, R.; Moers, T.; Bock, J.; Vater, L.; and Eckstein, L. 2020.  The rounD Dataset: A Drone Dataset of Road User Trajectories at Roundabouts in Germany.  In _2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)_ , 1–6.
  * Kuefler et al. (2017) Kuefler, A.; Morton, J.; Wheeler, T.; and Kochenderfer, M. 2017.  Imitating driver behavior with generative adversarial networks.  In _2017 IEEE Intelligent Vehicles Symposium (IV)_ , 204–211. IEEE.
  * Lefkopoulos et al. (2020) Lefkopoulos, V.; Menner, M.; Domahidi, A.; and Zeilinger, M. N. 2020.  Interaction-aware motion prediction for autonomous driving: A multiple model kalman filtering scheme.  _IEEE Robotics and Automation Letters_ , 6(1): 80–87.
  * Li et al. (2022) Li, G.; Qiu, Y.; Yang, Y.; Li, Z.; Li, S.; Chu, W.; Green, P.; and Li, S. E. 2022.  Lane change strategies for autonomous vehicles: a deep reinforcement learning approach based on transformer.  _IEEE Transactions on Intelligent Vehicles_.
  * Li et al. (2020) Li, Y.; Lu, X.-Y.; Wang, J.; and Li, K. 2020.  Pedestrian trajectory prediction combining probabilistic reasoning and sequence learning.  _IEEE Transactions on Intelligent Vehicles_ , 5(3): 461–474.
  * Li et al. (2023a) Li, Z.; Chen, Z.; Li, Y.; and Xu, C. 2023a.  Context-aware trajectory prediction for autonomous driving in heterogeneous environments.  _Computer-Aided Civil and Infrastructure Engineering_.
  * Li et al. (2023b) Li, Z.; Liao, H.; Tang, R.; Li, G.; Li, Y.; and Xu, C. 2023b.  Mitigating the impact of outliers in traffic crash analysis: A robust Bayesian regression approach with application to tunnel crash data.  _Accident Analysis & Prevention_, 185: 107019.
  * Liang et al. (2020) Liang, M.; Yang, B.; Hu, R.; Chen, Y.; Liao, R.; Feng, S.; and Urtasun, R. 2020.  Learning lane graph representations for motion forecasting.  In _European Conference on Computer Vision_ , 541–556. Springer.
  * Liao et al. (2023) Liao, H.; Shen, H.; Li, Z.; Wang, C.; Li, G.; Bie, Y.; and Xu, C. 2023.  GPT-4 Enhanced Multimodal Grounding for Autonomous Driving: Leveraging Cross-Modal Attention with Large Language Models.  _arXiv preprint arXiv:2312.03543_.
  * Lin, Ulsoy, and LeBlanc (2000) Lin, C.-F.; Ulsoy, A. G.; and LeBlanc, D. J. 2000.  Vehicle dynamics and external disturbance estimation for vehicle path prediction.  _IEEE Transactions on Control Systems Technology_ , 8(3): 508–518.
  * Markkula et al. (2020) Markkula, G.; Madigan, R.; Nathanael, D.; Portouli, E.; Lee, Y. M.; Dietrich, A.; Billington, J.; Schieben, A.; and Merat, N. 2020.  Defining interactions: A conceptual framework for understanding interactive behaviour in human and automated road traffic.  _Theoretical Issues in Ergonomics Science_ , 21(6): 728–752.
  * Messaoud et al. (2019) Messaoud, K.; Yahiaoui, I.; Verroust-Blondet, A.; and Nashashibi, F. 2019.  Non-local social pooling for vehicle trajectory prediction.  In _2019 IEEE Intelligent Vehicles Symposium (IV)_ , 975–980. IEEE.
  * Messaoud et al. (2021) Messaoud, K.; Yahiaoui, I.; Verroust-Blondet, A.; and Nashashibi, F. 2021.  Attention Based Vehicle Trajectory Prediction.  _IEEE Transactions on Intelligent Vehicles_ , 6(1): 175–185.
  * Müller, Risto, and Emmenegger (2016) Müller, L.; Risto, M.; and Emmenegger, C. 2016.  The social behavior of autonomous vehicles.  In _Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct_ , 686–689.
  * Pillai, Suel, and Cha (2005) Pillai, S. U.; Suel, T.; and Cha, S. 2005.  The Perron-Frobenius theorem: some of its applications.  _IEEE Signal Processing Magazine_ , 22(2): 62–75.
  * Schwarting et al. (2019) Schwarting, W.; Pierson, A.; Alonso-Mora, J.; Karaman, S.; and Rus, D. 2019.  Social behavior for autonomous vehicles.  _Proceedings of the National Academy of Sciences_ , 116(50): 24972–24978.
  * Tang and Salakhutdinov (2019) Tang, C.; and Salakhutdinov, R. R. 2019.  Multiple futures prediction.  _Advances in neural information processing systems_ , 32.
  * Todd and Gigerenzer (2000) Todd, P. M.; and Gigerenzer, G. 2000.  Précis of simple heuristics that make us smart.  _Behavioral and brain sciences_ , 23(5): 727–741.
  * Toghi et al. (2022) Toghi, B.; Valiente, R.; Sadigh, D.; Pedarsani, R.; and Fallah, Y. P. 2022.  Social coordination and altruism in autonomous driving.  _IEEE Transactions on Intelligent Transportation Systems_ , 23(12): 24791–24804.
  * Toledo, Musicant, and Lotan (2008) Toledo, T.; Musicant, O.; and Lotan, T. 2008.  In-vehicle data recorders for monitoring and feedback on drivers’ behavior.  _Transportation Research Part C: Emerging Technologies_ , 16(3): 320–331.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2023) Wang, R.; Wang, S.; Yan, H.; and Wang, X. 2023.  WSiP: Wave Superposition Inspired Pooling for Dynamic Interactions-Aware Trajectory Prediction.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, 4685–4692.
  * Wang et al. (2022a) Wang, W.; Wang, L.; Zhang, C.; Liu, C.; Sun, L.; et al. 2022a.  Social interactions for autonomous driving: A review and perspectives.  _Foundations and Trends® in Robotics_ , 10(3-4): 198–376.
  * Wang et al. (2021) Wang, Y.; Wang, C.; Zhao, W.; and Xu, C. 2021.  Decision-Making and planning method for autonomous vehicles based on motivation and risk assessment.  _IEEE Transactions on Vehicular Technology_ , 70(1): 107–120.
  * Wang et al. (2022b) Wang, Y.; Zhao, S.; Zhang, R.; Cheng, X.; and Yang, L. 2022b.  Multi-Vehicle Collaborative Learning for Trajectory Prediction With Spatio-Temporal Tensor Fusion.  _IEEE Transactions on Intelligent Transportation Systems_ , 23(1): 236–248.
  * Wong et al. (2022) Wong, C.; Xia, B.; Hong, Z.; Peng, Q.; Yuan, W.; Cao, Q.; Yang, Y.; and You, X. 2022.  View Vertically: A hierarchical network for trajectory prediction via fourier spectrums.  In _European Conference on Computer Vision_ , 682–700. Springer.
  * Xie et al. (2017) Xie, G.; Gao, H.; Qian, L.; Huang, B.; Li, K.; and Wang, J. 2017.  Vehicle trajectory prediction by integrating physics-and maneuver-based approaches using interactive multiple models.  _IEEE Transactions on Industrial Electronics_ , 65(7): 5999–6008.
  * Xie et al. (2020) Xie, G.; Shangguan, A.; Fei, R.; Ji, W.; Ma, W.; and Hei, X. 2020.  Motion trajectory prediction based on a CNN-LSTM sequential model.  _Science China Information Sciences_ , 63(11): 1–21.
  * Xie et al. (2021) Xie, X.; Zhang, C.; Zhu, Y.; Wu, Y. N.; and Zhu, S.-C. 2021.  Congestion-aware multi-agent trajectory prediction for collision avoidance.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , 13693–13700. IEEE.
  * Xue et al. (2021) Xue, Q.; Li, S.; Li, X.; Zhao, J.; and Zhang, W. 2021.  Hierarchical Motion Encoder-Decoder Network for Trajectory Forecasting.  _arXiv preprint arXiv:2111.13324_.
  * Ye, Cao, and Chen (2021) Ye, M.; Cao, T.; and Chen, Q. 2021.  Tpcn: Temporal point cloud networks for motion forecasting.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 11318–11327.
  * Zeng et al. (2023) Zeng, W.; Li, M.; Xiong, W.; Tong, T.; Lu, W.-j.; Tan, J.; Wang, R.; and Huang, R. 2023.  Mpcvit: Searching for accurate and efficient mpc-friendly vision transformer with heterogeneous attention.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 5052–5063.
  * Zhang and Li (2022) Zhang, K.; and Li, L. 2022.  Explainable multimodal trajectory prediction using attention models.  _Transportation Research Part C: Emerging Technologies_ , 143: 103829.
  * Zhao et al. (2019) Zhao, T.; Xu, Y.; Monfort, M.; Choi, W.; Baker, C.; Zhao, Y.; Wang, Y.; and Wu, Y. N. 2019.  Multi-agent tensor fusion for contextual trajectory prediction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 12126–12134.