TSGN: Temporal Scene Graph Neural Networks with Projected Vectorized
  Representation for Multi-Agent Motion Prediction
Predicting future motions of nearby agents is essential for an autonomous
vehicle to take safe and effective actions. In this paper, we propose TSGN, a
framework using Temporal Scene Graph Neural Networks with projected vectorized
representations for multi-agent trajectory prediction. Projected vectorized
representation models the traffic scene as a graph which is constructed by a
set of vectors. These vectors represent agents, road network, and their spatial
relative relationships. All relative features under this representation are
both translationand rotation-invariant. Based on this representation, TSGN
captures the spatial-temporal features across agents, road network,
interactions among them, and temporal dependencies of temporal traffic scenes.
TSGN can predict multimodal future trajectories for all agents simultaneously,
plausibly, and accurately. Meanwhile, we propose a Hierarchical Lane
Transformer for capturing interactions between agents and road network, which
filters the surrounding road network and only keeps the most probable lane
segments which could have an impact on the future behavior of the target agent.
Without sacrificing the prediction performance, this greatly reduces the
computational burden. Experiments show TSGN achieves state-of-the-art
performance on the Argoverse motion forecasting benchmar.
Yunong Wu, Thomas Gilles, Bogdan Stanciulescu, Fabien Moutarde
TSGN: Temporal Scene Graph Neural Networks with Projected Vectorized Representation for Multi-Agent Motion Prediction

###### Abstract

Predicting future motions of nearby agents is essential for an autonomous vehicle to take safe and effective actions. In this paper, we propose TSGN, a framework using Temporal Scene Graph Neural Networks with projected vectorized representations for multi-agent trajectory prediction. Projected vectorized representation models the traffic scene as a graph which is constructed by a set of vectors. These vectors represent agents, road network, and their spatial relative relationships. All relative features under this representation are both translation- and rotation-invariant. Based on this representation, TSGN captures the spatial-temporal features across agents, road network, interactions among them, and temporal dependencies of temporal traffic scenes. TSGN can predict multimodal future trajectories for all agents simultaneously, plausibly, and accurately. Meanwhile, we propose a Hierarchical Lane Transformer for capturing interactions between agents and road network, which filters the surrounding road network and only keeps the most probable lane segments which could have an impact on the future behavior of the target agent. Without sacrificing the prediction performance, this greatly reduces the computational burden. Experiments show TSGN achieves state-of-the-art performance on the Argoverse motion forecasting benchmark.

## I Introduction

Predicting the motion of multiple agents is an essential step in autonomous driving. It helps autonomous driving vehicles to plan their actions and prevent accidents. However, the traffic scene is highly complex. It contains agents, road network, and interactions among them. The prediction model needs to take these entities as inputs and outputs reasonable multimodal trajectories that the target agent could take in the future.

The traffic scene is viewed differently from the perspective of different agents. However, most existing methods select one target agent each time and take it as the central reference to perform the coordinate transformation on the entire scene, which is not symmetric for the other agents and results in only one prediction at a time. It is inefficient and not robust to the coordinate transformation. To alleviate these problems, HiVT [1] takes a symmetric way to encode the traffic scene. It divides the traffic scene into a set of local regions, each corresponding to one agent and centered on it, and uses a local-global mechanism to encode agents' features.

We adopt the symmetrical representation of HiVT and its local-global mechanisms. Based on them, we propose an extended scene representation method: Projected vectorized representation and an optimized prediction framework: TSGN. Our representation method introduces more features about agents and lane segments. It projects all relative spatial features between entities into the direction of the target entity for every timestamp, such that, all these relative features under this representation are also both translation- and rotation-invariant. TSGN consists of diverse interaction modules and temporal dependencies modules. These modules are constructed based on the transformer structure. TSGN treats the traffic scene in its entirety and encodes the agents' dynamics, surrounding lane segments, and interactions among them across time steps. In addition, we present a Hierarchical Lane Transformer module inside TSGN to capture the influence of the road network on the future motion of target agents. Unlike most approaches that use the full structural information of the surrounding road network, the Hierarchical Lane Transformer only selects the most probable lane segments which could have an impact on the future behavior of the target agent. It greatly reduces the computation burden and enables the model to do faster predictions without sacrificing the prediction performance.

Our contributions are summarized as follow:

* We extend the scene representation method of HiVT by adding new features about agents, lane segments, and their spatial relative relationships. We adopt a projected vectorized representation for all relative features of the traffic scene for every timestamp, which describes the relative spatial relationships in a more detailed way.
* We propose a Hierarchical Lane Transformer which only selects lane segments that affect the future behaviors of target agents the most for modeling the interactions, enabling much faster and equally accurate predictions.
* Our designed TSGN can make plausible and accurate predictions. We validate the performance of TSGN on Argoverse forecasting benchmark. It ranked \(1^{st}\) in terms of minADE on the leaderboard on September 13, 2022.

## II Related Work

Recently deep learning-based models have brought great progress to the motion prediction. According to the different representation methods, these learning-based models can be divided into rasterized-representation-based models and vectorized-representation-based models.