# Words in Motion: Representation Engineering for Motion Forecasting

Ömer Şahin Taş Royden Wagner∗
FZI Research Center for Information Technology
Karlsruhe Institute of Technology
Equal contribution. Corresponding author<EMAIL_ADDRESS>
###### Abstract

Motion forecasting transforms sequences of past movements and environment
context into future motion. Recent methods rely on learned representations,
resulting in hidden states that are difficult to interpret. In this work, we
use natural language to quantize motion features in a human-interpretable way,
and measure the degree to which they are embedded in hidden states. Our
experiments reveal that hidden states of motion sequences are arranged with
respect to our discrete sets of motion features. Following these insights, we
fit control vectors to motion features, which allow for controlling motion
forecasts at inference. Consequently, our method enables controlling
transformer-based motion forecasting models with textual inputs, providing a
unique interface to interact with and understand these models.111Our
implementation is available at https://github.com/kit-mrt/future-motion.

> Keywords: control vectors, linear probing, neural collapse, representation
> engineering, alignment

##  1 Introduction

Accurate and interpretable motion forecasting of road users is essential for
autonomous driving. These objectives are often in conflict, since high
accuracy frequently comes at the cost of lower interpretability. This trade-
off is primarily linked to the representational capacity of the underlying
model: methods achieving higher accuracy tend to rely on the increased
complexity of their underlying models. This, in turn, renders them difficult
to understand and interpret in terms of semantically meaningful patterns.

Deep learning models employ loss functions that encourage the clustering of
data samples based on their patterns [1]. Together with regularizers that
prevent overfitting, clusters become more distinct over the course of training
[2, 3]. Recent motion forecasting methods use transformer models [4, 5, 6].
They process features of past motion sequences (i.e., past positions,
orientation, acceleration, speed) and environment context (i.e., map data and
traffic light states), and transform them into future motion sequences. Recent
methods rely on learned representations of these features, resulting in hidden
states that are difficult to interpret.

Natural language is a medium to describe semantically meaningful patterns.
Previous works have leveraged language models to enable text conditioned
scenario generation and to decode essential textual information about the
underlying model’s actions and reasoning (e.g., [7, 8, 9]). However, analyzing
learned representations with respect to human-interpretable classes of motion
features remains an open challenge.

We use natural language to quantize motion features into discrete and
interpretable sets. During training, we use linear probing to measure the
degree to which these features are embedded in hidden states. Our experiments
demonstrate that hidden states of motion sequences are arranged with respect
to our discrete sets of motion features. In this way, we introduce a novel
method to interpret the hidden states of transformer-based motion forecasting
models. Building on these insights, we fit control vectors to opposing motion
features which allow for controlling motion forecasts at inference. As a
result, our method enables controlling transformer-based motion forecasting
models using textual inputs, offering a novel and intuitive interface for
interacting with and understanding these models.

##  2 Related Work

The concept of linking learned representations to natural language has gained
significant attention (e.g., [10, 11, 12]).

Conditioning. Numerous works leverage natural language to guide or condition
generative models [13, 14, 15]. Tan et al. [16] generate dynamic traffic
scenes that match user-specified descriptions expressed in natural language.

Prompting. Some works use language as an interface to prompt driving models.
Kuo et al. [8] generate linguistic descriptions of predicted trajectories
during decoding, capturing essential information about future maneuvers and
interactions. More recent works employ large language models (LLMs) to analyze
driving environments in a human-like manner, providing explanations of driving
actions and the underlying reasoning [17, 18, 9, 7]. This offers a human-
centric description of the driving environment and the model’s decision-making
capabilities. Although these works create a language interfaces to interact
with the underlying model, in contrast to our work, they do not analyze the
alignment of internal representations with human-interpretable motion
features.

Enriching. Another line of work aims to use LLMs generalization capacities to
enrich the context embeddings and provide additional information to improve
motion forecasting accuracy. Zheng et al. [19] demonstrate that by integrating
this enriched context information into motion forecasting models. Wang et al.
[20] use LLMs for data augmentation to improve out-of-distribution
generalization. Others use pre-trained LLMs for better generalization during
decision-making [21, 22, 23].

Instructing. Natural language can also serve as a powerful tool for issuing
instructions. Instructions are explicit commands given to a model to perform a
specific task and differ from language conditioned generation [24, 25, 26].
The challenge in instructing lies in connecting the abstractions and
generality of language to environment grounded actions. Huang et al. [27]
demonstrate the potential of using instructions to guide task execution in
simulation environments for autonomous driving.

##  3 Method

Figure 1: Words in Motion. We use natural language to quantize motion features
in an interpretable way. (b) The corresponding direction, speed, and
acceleration classes are highlighted in blue. (c) To reverse engineer motion
forecasting models, we measure the degree to which these features are embedded
in their hidden states $H_{i}$ with linear probes. Furthermore, we use our
discrete motion features to fit control vectors $V_{i}$ that allow for
controlling motion forecasts during inference.

We reverse engineer recent motion forecasting models by using natural language
to quantize motion features into discrete sets (i.e., subclasses). Textual
descriptions naturally represent human-interpretable classes of motion
features (e.g., for speed or direction). In this work, we focus on coarse-
grained direction (e.g. moving right in Figure 1), speed (e.g. moderate speed
in Figure 1), acceleration classes (e.g. decelerating in Figure 1) and fine-
grained motion classes (complete text representation in Figure 1).

###  3.1 Motion classification using natural language

We classify motion features using four primary classes of direction, following
insights from [28]. Initially, we determine these classes by the cumulative
sum of difference in yaw angles. Trajectories with a sum less than $None$
degrees are classified as straight. For sums exceeding this threshold, a
positive value labels the motion as right, while a negative value, exceeding
the threshold in absolute terms, indicates a left direction. Additionally, we
introduce a stationary class for stationary objects, where direction lacks
semantic significance. We further classify motion based on speed. Speeds
between $25\text{\,}\mathrm{km}\text{\,}{\mathrm{h}}^{-1}$ and
$50\text{\,}\mathrm{km}\text{\,}{\mathrm{h}}^{-1}$ are labeled moderate.
Speeds above this range are high, while those below are low. Negative speeds
are classified as backwards. Lastly, we analyze acceleration by comparing the
integral of speed over time to projected displacement with initial speed.
Trajectories are decelerating if under $0.9$ times, accelerating if over $1.1$
times the projected displacement, and constant otherwise. All threshold values
are selected by analyzing the distribution of the used dataset. These
direction, speed, and acceleration categories enable us to construct sentence
descriptions, as illustrated in Figure 1.

###  3.2 Neural collapse as a metric of interpretability

A recent line of work [1, 2, 3] introduces the term neural collapse to
describe a desirable learning behavior of deep neural networks for
classification.222Neural collapse is not to be confused with representation
collapse [29, 30], where learned representations across all classes collapse
to redundant or trivial solutions (e.g., zero vectors). It refers to the
phenomenon that learned top-layer representations form semantic clusters until
these collapse to their means. In addition, the cluster means transform
progressively into equidistant vectors when centered around the global mean.
Therefore, neural collapse facilitates classification tasks and is considered
a desirable learning behavior for both supervised [1] and self-supervised
learning [31].

We propose to measure neural collapse as a metric of interpretability.
Specifically, we focus on interpreting hidden states (i.e., latent
representations) of motion sequences in motion forecasting models. Our
approach is motivated by the assumption that hidden states of motion sequences
should embed human-interpretable motion features. Therefore, we propose to
evaluate if hidden states can be classified as human-interpretable motion
features by linear probes [32] (i.e., learned linear classifiers). Simple
linear probes ensure that the abstract hidden states are closely related to
interpretable semantics.333 Ben-Shaul et al. [31] show that linear probing
accuracy is consistent with the accuracy of nearest class center classifiers,
which are typically used to measure neural collapse.  Adapted to motion
forecasting, we choose the aforementioned language-based motion features. In
summary, we use natural language to quantize motion features in an
interpretable way, and measure the degree to which they are embedded in hidden
states using linear probing.

###  3.3 Natural language-based control vectors

We transfer the concept of control vectors from natural language processing
[33, 34, 35] to motion forecasting. Control vectors ($V_{i}$ in Figure 1) are
sets of vectors, which are added to the hidden states of transformer models to
modify their predictions, without fine-tuning or prompt engineering. In
general, control vectors represent the difference of hidden states from
opposite concepts or features. Therefore, adding temperature-scaled ($\tau$ in
Figure 1) control vectors to the hidden states of a given sample controls the
strength of the corresponding features. However, this requires that the hidden
states are arranged in a well-structured latent space, where samples are
clustered according to classes or features (e.g., a high degree of neural
collapse, see Section 3.2).

We use our natural language-based classes of motion features (see Section 3.1)
to build pairs of opposing features. Specifically, we generate speed control
vectors using the high and low speed classes, acceleration control vectors
using the accelerating and decelerating classes, and direction control vectors
using the left and right classes. For each pair, we build a dataset and
extract the corresponding hidden states $H_{i}^{[m]}$ of the module $m=2$.
Afterwards, we compute the element-wise difference between the hidden states
of opposing classes. We choose the hidden state of the last token per motion
sequence (with $i=-1$) as it is closest to the start of the prediction.
Finally, we follow [33] and use principal component analysis (PCA) with one
component to reduce the computed differences to one scalar per hidden
dimension.

##  4 Experimental Setup

###  4.1 Motion forecasting models

We study two recent transformer models for motion forecasting in self-driving
vehicles. Wayformer [4] and RedMotion [5] models employ attention-based scene
encoders to learn agent-centric embeddings of past motion, map, and traffic
light data. To efficiently process long token sequences, Wayformer uses latent
query attention [36] for subsampling, RedMotion lowers memory requirements via
local-attention [37] and redundancy reduction. For Wayformer, we use the
implementation by [38] and a custom hierarchical fusion setup with a modality-
specific encoder for past motion and a shared encoder for environment context.
For RedMotion, we use the publicly available implementation. Therefore, both
models contain specific motion encoders with three modules, which we combine
with linear probes and control vectors to interpret and control hidden states
with human-interpretable motion features (see Figure 1).

###  4.2 Datasets

The Argoverse 2 Forecasting Dataset [39] consists of $250,000$ scenarios of
urban or suburban driving, each covering a period of 11 seconds. These
scenarios include $5$ seconds of past and $6$ seconds of future data. The
dataset captures interactions along more than $2000\text{\,}\mathrm{km}$ of
roadways in six geographically diverse cities.

The Waymo Open Motion Dataset [40] consists of over $1.1$ million data points
extracted from $103,000$ urban or suburban driving scenarios, each spanning
$20$ seconds. The state of the traffic agents includes attributes such as
position, dimensions, speed, acceleration, orientation, and angular velocity.
Each data point captures $1$ second of past data followed by $8$ seconds of
future data.

###  4.3 Metrics

Linear probing accuracy. To measure the degree to which interpretable motion
features are embedded in latent representations, we use linear probing [32].
In this approach, we train linear probes (i.e., linear classifiers detached
from the overall gradient computation) on top of the hidden states ($H_{i}$ in
Figure 1). During training, we track their accuracy in classifying our
quantized motion features on validation sets.

Representation quality. Following [41], we use the mean of the standard
deviation of the $\ell_{2}$-normalized embedding to measure representation
collapse. Representation collapse refers to an undesirable learning behaviour
where learned embeddings collapse into redundant or trivial representations.
Redundant representations have a standard deviation close to zero. In a way,
representing the opposite of neural collapse. As shown in [41], rich
representations have a standard deviation close to $1/\sqrt{\text{dim}}$,
where dim is the hidden dimension.

###  4.4 Training details and hyperparameters

We add linear probes for our quantized motion features to each hidden state of
both models ($H_{i}^{[m]}$ in Figure 1, where $m\in\\{0,1,2\\}$ is the module
number and $i$ is the token index). These classifiers are learned during
training using regular cross-entropy loss to classify our fine-grained motion,
and coarse-grained speed, acceleration and direction classes from hidden
states. We decouple this objective from the overall gradient computation.
Therefore, these classifiers do not contribute to the alignment of the latent
representations, but exclusively measure the corresponding neural collapse
into interpretable clusters.

As the main loss, we use a common combination loss terms for motion
forecasting. Following [38], we use the unweighted sum of the negative log-
likelihood loss for positions, the cosine loss for the heading angle, and the
Huber loss for velocities as motion loss.

We provide Wayformer models with the nearest $512$ map polylines and RedMotion
models with the nearest 128 map polylines, and both models with a maximum of
$48$ surrounding traffic agents as environment context. For the Argoverse 2
Forecasting dataset, we use past motion sequences with $50$ timesteps
(representing $5\text{\,}\mathrm{s}$) as input. For the Waymo Open Motion
dataset, we use past motion sequences with $11$ steps (representing
$1.1\text{\,}\mathrm{s}$) as input. We use AdamW [42] in its default
configuration as optimizer and set the initial learning rate to $0.0002$. All
models are configured to forecast $k=6$ motion modes per agent. As post-
processing, we do not perform trajectory aggregation, but follow [38] and
modify only the predicted confidences of redundant forecasts.

##  5 Results

deceleratingconstantaccelerating$11.69$$65.32$$22.99$$31.17$$16.76$$52.08$ (a)
Acceleration

stationarystraightleftright$27.88$$48.62$$11.24$$12.26$$51.48$$26.09$$10.77$$11.65$
(b) Direction

backwardshighmoderatelow$4.47$$12.66$$20.68$$62.19$$24.17$$4.16$$18.49$$53.32$
(c) Speed

Figure 2: Distributions of our natural language-based motion subclasses for
the Argoverse 2 Forecasting and the Waymo Open Motion dataset. All numbers are
percentages.

Figure 2 presents the distribution of motion subclasses across the datasets.
The Waymo dataset predominantly features vehicles with constant acceleration
(65%) and traveling straight (49%), while the Argoverse 2 dataset showcases a
higher proportion of accelerating instances (52%). Both datasets predominantly
capture low-speed scenarios, with 62% of Waymo instances and 53% of Argoverse
2 instances falling into this category. Furthermore, a notable difference lies
in the proportion of stationary vehicles, with Waymo exhibiting a
significantly higher percentage (51%) compared to Argoverse 2 (28%).
Conversely, Argoverse 2 stands out with a much larger proportion of instances
involving backward motion (24%) compared to Waymo (4%). This disparity in
motion characteristics highlights that the two datasets capture different
driving environments and scenarios, with Waymo potentially focusing on highway
or structured urban driving, while Argoverse 2 encompasses more diverse
traffic situations.

$0$$1$$2$$3$$0.75$$0.80$$0.85$Step $\cdot 10^{4}$Top-1 linear acccuracy

$0$$1$$2$$3$$0.30$$0.35$$0.40$$0.45$Step $\cdot 10^{4}$Top-1 linear acccuracy

$0$$1$$2$$3$$0.86$$0.88$$0.90$$0.92$$0.94$Step $\cdot 10^{4}$Top-5 linear
acccuracy

Figure 3: Linear accuracies for RedMotion and Wayformer on the Argoverse 2
Forecasting dataset. Left plot: Direction validation top-1. Middle plot:
Motion validation top-1. Right plot: Motion validation top-5.

Figure 3 shows the top-1 linear probing accuracy for our coarse-grained
direction and the top-1 and top-5 linear accuracy for our fine-grained motion
classes on the Argoverse 2 Forecasting dataset. The scores are computed on the
validation split over the course of training.

Both models achieve similar direction top-1 and motion top-5 accuracy scores,
while the Wayformer model achieves a slightly higher motion top-1 accuracy.
Both models reach high direction top-1 accuracy and motion top-5 accuracy
scores, $>$80% and $>92$% respectively. This shows that both models likely
exhibit neural collapse towards interpretable motion features.

$0$$1$$2$$3$$0$$2$$4$$6$$8$$\frac{1}{\sqrt{\text{dim}}}$Step $\cdot
10^{4}$std-$\ell_{2}$-norm $\cdot 10^{-2}$ Figure 4: Normalized standard
deviation representation quality metric for RedMotion and Wayformer.

The representation quality metric normalized standard deviation of embeddings
is shown in Figure 4. Both RedMotion and Wayformer learn embeddings with a
normalized standard deviation close to the desired value of
$1/\sqrt{\text{dim}}$.

Figure 5 shows the linear probing accuracy scores for our coarse-grained
acceleration, direction, and speed features on the Waymo Open Motion dataset.
Here, we report the scores for each hidden state $H_{i}$ in the RedMotion
model (i.e., after each module in the motion encoder, see Figure 1). Similar
accuracy scores are reached for all features at all three hidden states. The
accuracy scores for the acceleration and speed features progressively improve,
while the scores for the direction class jump to a score of 80% early on.
Compared to the direction scores on the Argoverse 2 Forecasting dataset, the
scores on the Waymo Open Motion dataset “jump” earlier. We hypothesize that
this is linked to the shorter input motion sequence on Waymo
($1.1\text{\,}\mathrm{s}$ vs. $5\text{\,}\mathrm{s}$), which limits the amount
possible movements and thus simplifies classifying direction. Overall, the
highest scores are reached for the speed features, indicating that these are
most relevant for motion forecasting. Additionally, the rightmost plot in
Figure 5 shows the linear probing results for classifying agent types (i.e.,
vehicle, cyclist, and pedestrian). Across all hidden states, very high linear
accuracy scores are reached indicating that the representations for agent
types are fully collapsed and are particularly important for motion
forecasting.

$0$$1$$2$$0.76$$0.78$$0.80$Step $\cdot 10^{5}$Linear accuracy acceleration

$0$$1$$2$$0.70$$0.75$$0.80$Step $\cdot 10^{5}$Linear accuracy direction

$0$$1$$2$$0.91$$0.92$$0.93$$0.94$$0.95$Step $\cdot 10^{5}$Linear accuracy
speed

$0$$1$$2$$0.998$$0.999$$1.000$Step $\cdot 10^{5}$Linear accuracy agent

Figure 5: Top-1 linear accuracies at module 0, module 1 and module 2 for
acceleration, direction, and speed on the Waymo Open Motion dataset.
$0$$1$$2$$0.45$$0.50$$0.55$$0.60$Step $\cdot 10^{5}$Linear
accuracy$0.00$$0.02$$0.04$Difference Figure 6: Linear accuracy for our fine-
grained motion classes and its aggregated components, along with their
difference. The aggregated components’ accuracy, calculated as the product of
individual component accuracies, is consistently higher than the motion
accuracy. The difference between these accuracies varies from $0.00$ to $0.04$
throughout the training.

Figure 6 presents the linear accuracy measurements for both our fine-grained
motion classes and its aggregated components (i.e., our coarse-grained
direction, speed, and acceleration features and the agent types) across
various steps. The blue curve represents the accuracy of the aggregated
components, calculated as the product of individual component accuracies. The
green curve, on the other hand, depicts the linear accuracy of our fine-
grained motion classes. As expected, the linear accuracy of the aggregated
components consistently exceeds that of the motion curve. This outcome is
expected as it is generally less challenging to identify individual components
with distinct probes than to identify all components simultaneously.

##  6 Controlling motion forecasts at inference

Building on the insight that hidden states are likely collapsed towards our
quantized motion features, we fit control vectors to opposing motion features.
These control vectors allow for controlling motion forecasts at inference.
Specifically, we use our quantized motion features to build pairs of opposing
features for the Argoverse 2 Forecasting and the Waymo Open Motion dataset.
Afterwards, we fit sets of control vectors ($V_{i}$ in Figure 1) as described
in Section 3.3.

Figure 7 shows qualitative results on the Waymo Open Motion dataset for
controlling a motion forecast with the set of control vectors for speed and
different temperatures $\tau$. Subfigure (a) shows the default (i.e., non-
controlled) top-1 (i.e., most likely) motion forecast. In subfigure (b) and
(c), we apply our speed control vector to de- and increase the driven speed of
a vehicle. Both controls affect the future speed analogously, while increasing
the speed also changes the route to fit the given environment context (i.e.,
lanes and surrounding vehicles).

(a) Default motion forecast

(b) Speed controlled $\tau=-32$

(c) Speed controlled $\tau=100$

Figure 7: Controlling a vehicle before a predicted right turn. In subfigure
(b) and (c), we apply our speed control vector to de- and increase the driven
speed of a vehicle. Both controls affect the future speed analogously, while
increasing the speed also changes the route to fit the given environment
context (lanes and surrounding vehicles). The focal agent is highlighted in
orange, dynamic agents are blue, and static agents are grey. Lanes are black
lines and road markings are white lines.

Figure 8 shows a qualitative example from the Argoverse 2 Forecasting dataset.
In subfigure (b) and (c), we apply our acceleration control vector with
$\tau=-20$ and $\tau=100$ to enforce a strong deceleration and a moderate
acceleration. Moreover, we include an example for our direction control in the
Appendix. Overall, these qualitative results support the finding that the
hidden states of motion sequences are arranged with respect to our discrete
sets of motion features.

Possible applications in self-driving vehicles include aligning motion
forecasts to specific driving styles or areas. For example, assuming a more
aggressive driving style may be modeled by the speed or acceleration control
vectors applied to other road users.

(a) Default motion forecast

(b) Enforced strong deceleration

(c) Enforced acceleration

Figure 8: Controlling a vehicle at an intersection. In subfigure (b) and (c),
we apply our acceleration control vector with $\tau=-20$ and $\tau=100$ to
enforce a strong deceleration and a moderate acceleration. The focal agent is
highlighted in orange, dynamic agents are blue, and static agents are grey.
Lanes are black lines and road markings are white lines.

##  7 Limitations

We limit the scope of our work to motion forecasting models that use agent-
centric environment representations. Models that use scene-wide [43, 44] or
pairwise relative representations [38, 45] may be controlled differently.
Furthermore, we did not evaluate how downstream tasks such as motion planning
can benefit from aligned motion forecasts. These are promising directions for
future research.

##  8 Conclusion

In this work, we take a significant step towards interpreting and controlling
motion forecasting models. We use textual descriptions to quantize motion
features in an interpretable way. These textual descriptions offer a human-
interpretable way to classify motion sequences. Most importantly, we
demonstrate that neural collapse towards human-interpretable classes of motion
features occurs in recent motion forecasting models. Building on these
insights, we fit control vectors to opposing motion features, which allow for
controlling motion forecasts at inference.

We analyze “words in motion” and examine the representations associated with
motion features. Future work can explore detailed text descriptions by
incorporating both static and dynamic scene elements. Additionally,
investigating the impact of how multiple motion decoders affect embeddings
offers a promising direction for further research. Furthermore, natural
language-based motion classes could enable voice control. Adding voice control
to self-driving vehicles could significantly change how humans interact with
these machines, enhancing their accessibility and usability.

#### Acknowledgments

The research leading to these results is funded by the German Federal Ministry
for Economic Affairs and Climate Action within the project “NXT GEN AI
METHODS”.

## References

  * Papyan et al. [2020] V. Papyan, X. Y. Han, and D. L. Donoho.  Prevalence of neural collapse during the terminal phase of deep learning training.  _PNAS_ , 2020.
  * Galanti et al. [2022] T. Galanti, A. György, and M. Hutter.  On the Role of Neural Collapse in Transfer Learning.  In _ICLR_ , 2022.
  * Wu and Papyan [2024] R. Wu and V. Papyan.  Linguistic Collapse: Neural Collapse in (Large) Language Models.  _arXiv:2405.17767_ , 2024.
  * Nayakanti et al. [2023] N. Nayakanti, R. Al-Rfou, A. Zhou, K. Goel, K. S. Refaat, and B. Sapp.  Wayformer: Motion Forecasting via Simple & Efficient Attention Networks.  In _ICRA_ , 2023.
  * Wagner et al. [2024] R. Wagner, Ö. Ş. Taş, M. Klemp, C. Fernandez, and C. Stiller.  RedMotion: Motion Prediction via Redundancy Reduction.  _Transactions on Machine Learning Research_ , 2024.
  * Wang et al. [2023] X. Wang, T. Su, F. Da, and X. Yang.  ProphNet: Efficient Agent-Centric Motion Forecasting With Anchor-Informed Proposals.  In _CVPR_ , 2023.
  * Wayve Technologies Ltd. [2023] Wayve Technologies Ltd.  LINGO-1: Exploring Natural Language for Autonomous Driving, 2023.  URL https://wayve.ai/thinking/lingo-natural-language-autonomous-driving/.  Blog Post.
  * Kuo et al. [2022] Y.-L. Kuo, X. Huang, A. Barbu, S. G. McGill, B. Katz, J. J. Leonard, and G. Rosman.  Trajectory prediction with linguistic representations.  In _ICRA_ , 2022.
  * Sima et al. [2023] C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, P. Luo, A. Geiger, and H. Li.  DriveLM: Driving with Graph Visual Question Answering.  _arXiv:2312.14150_ , 2023.
  * Radford et al. [2021] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Alayrac et al. [2022] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al.  Flamingo: a Visual Language Model for Few-Shot Learning.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022\.
  * Liu et al. [2024] H. Liu, C. Li, Q. Wu, and Y. J. Lee.  Visual Instruction Tuning.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2024\.
  * Ramesh et al. [2021] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever.  Zero-shot text-to-image generation.  In _ICML_ , 2021.
  * Tevet et al. [2022] G. Tevet, B. Gordon, A. Hertz, A. H. Bermano, and D. Cohen-Or.  MotionCLIP: Exposing Human Motion Generation to CLIP Space.  In _ECCV_ , 2022.
  * Blattmann et al. [2023] A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis.  Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models.  In _CVPR_ , 2023.
  * Tan et al. [2023] S. Tan, B. Ivanovic, X. Weng, M. Pavone, and P. Kraehenbuehl.  Language Conditioned Traffic Generation.  In _CoRL_ , 2023.
  * Xu et al. [2023] Z. Xu, Y. Zhang, E. Xie, Z. Zhao, Y. Guo, K. K. Wong, Z. Li, and H. Zhao.  DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modell.  _arXiv:2310.01412_ , 2023.
  * Fu et al. [2024] D. Fu, X. Li, L. Wen, M. Dou, P. Cai, B. Shi, and Y. Qiao.  Drive Like a Human: Rethinking Autonomous Driving with Large Language Models.  In _WACV_ , 2024.
  * Zheng et al. [2024] X. Zheng, L. Wu, Z. Yan, Y. Tang, H. Zhao, C. Zhong, B. Chen, and J. Gong.  Large Language Models Powered Context-aware Motion Prediction.  _arXiv:2403.11057_ , 2024.
  * Wang et al. [2023] T.-H. Wang, A. Maalouf, W. Xiao, Y. Ban, A. Amini, G. Rosman, S. Karaman, and D. Rus.  Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models.  _arXiv:2310.17642_ , 2023.
  * Mao et al. [2023] J. Mao, J. Ye, Y. Qian, M. Pavone, and Y. Wang.  A Language Agent for Autonomous Driving.  _arXiv:2311.10813_ , 2023.
  * Wen et al. [2024] L. Wen, D. Fu, X. Li, X. Cai, T. MA, P. Cai, M. Dou, B. Shi, L. He, and Y. Qiao.  DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models.  In _ICLR_ , 2024.
  * Shao et al. [2024] H. Shao, Y. Hu, L. Wang, S. L. Waslander, Y. Liu, and H. Li.  LMDrive: Closed-Loop End-to-End Driving with Large Language Models.  In _CVPR_ , 2024.
  * Ouyang et al. [2022] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022\.
  * Chung et al. [2024] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, A. Castro-Ros, M. Pellat, K. Robinson, D. Valter, S. Narang, G. Mishra, A. Yu, V. Zhao, Y. Huang, A. Dai, H. Yu, S. Petrov, E. H. Chi, J. Dean, J. Devlin, A. Roberts, D. Zhou, Q. V. Le, and J. Wei.  Scaling Instruction-Finetuned Language Models.  _Journal of Machine Learning Research_ , 2024.
  * Raad et al. [2024] M. A. Raad, A. Ahuja, C. Barros, F. Besse, A. Bolt, A. Bolton, B. Brownfield, G. Buttimore, M. Cant, S. Chakera, et al.  Scaling Instructable Agents Across Many Simulated Worlds.  _arXiv:2404.10179_ , 2024.
  * Huang et al. [2024] Y. Huang, J. Sansom, Z. Ma, F. Gervits, and J. Chai.  DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiences.  In _First Vision and Language for Autonomous Driving and Robotics Workshop_ , 2024.
  * Seff et al. [2023] A. Seff, B. Cera, D. Chen, M. Ng, A. Zhou, N. Nayakanti, K. S. Refaat, R. Al-Rfou, and B. Sapp.  MotionLM: Multi-Agent Motion Forecasting as Language Modeling.  In _ICCV_ , 2023.
  * Hua et al. [2021] T. Hua, W. Wang, Z. Xue, S. Ren, Y. Wang, and H. Zhao.  On feature decorrelation in self-supervised learning.  In _ICCV_ , 2021.
  * Barbero et al. [2024] F. Barbero, A. Banino, S. Kapturowski, D. Kumaran, J. G. Araújo, A. Vitvitskyi, R. Pascanu, and P. Veličković.  Transformers need glasses! Information over-squashing in language tasks.  _arXiv:2406.04267_ , 2024.
  * Ben-Shaul et al. [2023] I. Ben-Shaul, R. Shwartz-Ziv, T. Galanti, S. Dekel, and Y. LeCun.  Reverse Engineering Self-Supervised Learning.  In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, _NeurIPS_ , 2023.
  * Alain and Bengio [2017] G. Alain and Y. Bengio.  Understanding intermediate layers using linear classifier probes.  In _ICLR_ , 2017.
  * Zou et al. [2023] A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A.-K. Dombrowski, et al.  Representation Engineering: A Top-Down Approach to AI Transparency.  _arXiv:2310.01405_ , 2023.
  * Subramani et al. [2022] N. Subramani, N. Suresh, and M. E. Peters.  Extracting Latent Steering Vectors from Pretrained Language Models.  In _Findings of the Association for Computational Linguistics_ , pages 566–581, 2022.
  * Turner et al. [2023] A. Turner, L. Thiergart, D. Udell, G. Leech, U. Mini, and M. MacDiarmid.  Activation Addition: Steering Language Models Without Optimization.  _arXiv:2308.10248_ , 2023.
  * Jaegle et al. [2021] A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira.  Perceiver: General perception with iterative attention.  In _ICML_ , 2021.
  * Beltagy et al. [2020] I. Beltagy, M. E. Peters, and A. Cohan.  Longformer: The Long-Document Transformer.  _arXiv:2004.05150_ , 2020.
  * Zhang et al. [2023] Z. Zhang, A. Liniger, C. Sakaridis, F. Yu, and L. Van Gool.  Real-Time Motion Prediction via Heterogeneous Polyline Transformer with Relative Pose Encoding.  In _NeurIPS_ , 2023.
  * Wilson et al. [2023] B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hartnett, J. K. Pontes, et al.  Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting.  _arXiv:2301.00493_ , 2023.
  * Ettinger et al. [2021] S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y. Chai, B. Sapp, C. R. Qi, Y. Zhou, et al.  Large Scale Interactive Motion Forecasting for Autonomous Driving: The Waymo Open Motion Dataset.  In _ICCV_ , 2021.
  * Chen and He [2021] X. Chen and K. He.  Exploring Simple Siamese Representation Learning.  In _CVPR_ , 2021.
  * Loshchilov and Hutter [2019] I. Loshchilov and F. Hutter.  Decoupled Weight Decay Regularization.  In _ICLR_ , 2019.
  * Ngiam et al. [2022] J. Ngiam, B. Caine, V. Vasudevan, Z. Zhang, H. L. Chiang, J. Ling, R. Roelofs, A. Bewley, C. Liu, A. Venugopal, D. Weiss, B. Sapp, Z. Chen, and J. Shlens.  Scene Transformer: A unified architecture for predicting multiple agent trajectories.  In _ICLR_ , 2022.
  * Zhou et al. [2023] Z. Zhou, J. Wang, Y.-H. Li, and Y.-K. Huang.  Query-Centric Trajectory Prediction.  In _CVPR_ , 2023.
  * Cui et al. [2023] A. Cui, S. Casas, K. Wong, S. Suo, and R. Urtasun.  Go Relative for Viewpoint-Invariant Motion Forecasting.  In _IEEE International Conference on Robotics and Automation (ICRA)_ , 2023.

## Appendix

(a) Default motion forecast

(b) Left control $\tau=10$

(c) Right control $\tau=100$

Figure 9: Controlling a left turning vehicle. In subfigure (b) and (c), we
apply our left-direction control vector and right-direction control vector.
The left control leads to accelerated future motion, which is consistent with
the common driving style of slowing down in front of a curve and accelerating
again when exiting the curve. A strong right control makes the focal agent
stationary. We hypothesize that it cancels out the actually driven left turn,
resulting in a virtually stationary past.

Title origin. The title of our work “words in motion“ is inspired by our
quantization method using natural language and a common notion in the computer
architecture literature. In computer architecture, a word is a natural unit of
data for a processing unit (e.g., CPU or GPU). In our work, words are classes
of motion features, which are embedded in the hidden states of motion
sequences processed by motion forecasting models.