# InCrowdFormer: On-Ground Pedestrian World Model From Egocentric Views

Mai Nishimura1,2
1OMRON SINIC X
Tokyo, Japan
<EMAIL_ADDRESS>Shohei Nobuhara2 Ko Nishino2
2Kyoto University
Kyoto, Japan
<EMAIL_ADDRESS>
###### Abstract

We introduce an on-ground Pedestrian World Model, a computational model that
can predict how pedestrians move around an observer in the crowd on the ground
plane, but from just the egocentric-views of the observer. Our model,
InCrowdFormer, fully leverages the Transformer architecture by modeling
pedestrian interaction and egocentric to top-down view transformation with
attention, and autoregressively predicts on-ground positions of a variable
number of people with an encoder-decoder architecture. We encode the
uncertainties arising from unknown pedestrian heights with latent codes to
predict the posterior distributions of pedestrian positions. We validate the
effectiveness of InCrowdFormer on a novel prediction benchmark of real
movements. The results show that InCrowdFormer accurately predicts the future
coordination of pedestrians. To the best of our knowledge, InCrowdFormer is
the first-of-its-kind pedestrian world model which we believe will benefit a
wide range of egocentric-view applications including crowd navigation,
tracking, and synthesis.

##  1 Introduction

Everyday, we successfully maneuver in a highly dynamic world with just our
egocentric, limited view of the surroundings. When we walk down the street, we
constantly map the positions of surrounding pedestrians on the ground plane.
We not only keep track of their current positions, but also predict their next
positions. When we play football, for instance, we are able to tell where
other players will end up, which is exactly why we can make that killer pass.

This predictive mental model of our dynamic surroundings is an illustrative
example of “World Models” [10], transition models of the environment. In this
paper, we are particularly interested in deriving a world model of pedestrians
that can continuously localize all visible surrounding people and predict
their movements with respect to the known actions of oneself in the next few
time steps. Unlike conventional trajectory forecasting methods [33, 1, 9]
which require accurate past on-ground pedestrian positions captured by a
static surveillance camera, our aim is to predict the on-ground pedestrian
positions in the future solely from the in-crowd observer’s ego-centric view.
The observer is either a person or mobile agent whose movement can be measured
(_i.e_., actions are known).

A world model of pedestrians that can be useful for such downstream tasks,
however, requires significant departures from past models [10, 19, 6]. First,
it must model the pedestrians on the ground plane but from inputs in
egocentric perspectives. This requires a nontrivial view transformation, often
referred to as Bird’s-Eye-View (BEV) transform. The world model, however,
needs to predict the next on-ground movements from the observed current
movements in the 2D egocentric view, unlike BEV transform that focuses on an
image to image (_i.e_., appearance) transform of only the current frame. We
refer to this inherently predictive purely geometric transform as predictive
ego2top transform.

Second, the model needs to be fundamentally object-centric. It must model each
pedestrian as an independent object that interacts with other objects
including the ego-viewer, too. The interaction between pedestrians lies at the
heart of the coordination of the pedestrians as a whole, and the transitions
of individual pedestrians are largely governed by these object interactions
especially in denser crowds. Third, the model needs to model and predict the
pedestrian movements and their interactions conditioned on the observer. The
observer itself is part of the crowd, and its/his/her actions influence and
are affected by the surrounding people. Finally, the viewer would be moving
and looking around while walking in the very crowd that needs to be modeled.
As a result, surrounding people will come in and out of sight. The crowd
itself will also consist of different numbers of people from time to time. The
model thus needs to naturally handle a varying number of people.

We derive a pedestrian world model that satisfies these requirements, namely
predictive ego2top transform, object-centric interaction encoding, observer
action conditioned modeling, and variable number of constituents. We refer to
our model as _InCrowdFormer_. The key idea is to model pedestrians as
individual tokens and fully leverage attention for modeling their interactions
and predictive ego2top transform in a Transformer architecture which also
naturally models varying numbers of pedestrians. Unlike past generic world
models [8, 6] which separate spatial encoding and temporal prediction, our
InCrowdFormer simultaneously encodes the social, temporal, and geometric
relationships of the viewer and surrounding pedestrians.

Two key challenges underlie learning an accurate Pedestrian World Model. The
model needs to decouple the ego-motion and pedestrian trajectories from
degenerated 2D movements in the image. In addition, the unknown absolute scale
of each object, _i.e_., pedestrian heights, introduces uncertainty in the
predictive ego2top transform. For this, we introduce a latent code for each
pedestrian and construct a generative model that outputs the on-ground future
location distribution conditioned on the observed movements in an ego-centric
view and the observer’s action. We efficiently model this conditional
probability with a Transformer consisting of self-attention that encodes the
social and temporal relationship between the observer’s action and each
pedestrian, and cross-attention that models the geometric relationship between
the ego-centric and on-ground views.

In summary, our contributions are threefold. (1) We introduce, to our
knowledge, the first egocentric Pedestrian World Model that models pedestrian
and in-crowd observer transitions on the ground plane from egocentric
observations, (2) derive it as a novel Transformer that leverages attention
for pedestrian interaction modeling and view transform for a variable number
of people, and (3) demonstrate its accuracy on real-world crowd motions.
Extensive experimental evaluation show that our unified Transformer World
Model can accurately predict the future coordination of pedestrians given the
observer’s action while taking into account uncertainty arising from imperfect
cues of depths. We also demonstrate the application of our pretrained model to
real video sequences. We believe our _InCrowdFormer_ will serve as a sound
foundation of pedestrian movement modeling for a wide range of applications.
We will release our code and data upon acceptance.

##  2 Related Work

Table 1: We introduce the first Pedestrian World Model (PWM) that models pedestrian movements solely from their in-crowd egocentric observations with a unified Transformer architecture that embodies predictive ego2top transform and action-conditioned prediction.  Method | state | ego2top | action | V | M
---|---|---|---|---|---
RSSM [10] | image | – | ✓ | CNN | GRU
TSSM [6, 8] | image | – | ✓ | CNN | Transformer
C-SWM [19] | object | – | – | CNN | GNN
G-SWM [21] | object | – | – | CNN | RNN
Slotformer [31] | object | – | – | CNN | Transformer
PWM (Ours) | object | ✓ | ✓ | Transformer

As shown in Table 1, to the best of our knowledge, we derive the first object-
centric world model that encodes pedestrian movements on the ground plane from
egocentric views together with the in-crowd observer’s actions and its mutual
dependency in a unified Transformer model.

#### World Models

A world model is an abstract representation of our environment and its
transitions [16]. Ha and Schmidhuber recently introduced the idea of building
a world model with a perception model (V) and a transition model (M), so that
a simple agent controller (C) [10] can be learned in the world model with
generated rollouts of simulated experiences [12, 11, 13, 8]. Most such world
models encode the whole image into a latent code as the environment
representation. Structured World Models [19, 31] and Object-centric World
Models [21] learn object-oriented representations about the world (_i.e_.,
individual objects constitute the environment representation). Even in these
models, however, the observer never interacts with the environment and object
transitions are simply observed from static viewpoints. In sharp contrast, we
are interested in modeling a human-populated environment with an object-
centric world model that also contains the dynamic observer itself.

#### Crowd Navigation

To plan a feasible path while avoiding collisions with nearby pedestrians,
typical approaches predict the future location of pedestrians with analytical
models [14, 27] or learning-based crowd models such as social pooling [5, 1]
and graph neural networks [4, 7, 15]. These methods, however, usually assume
direct top-down observations of on-ground pedestrian positions, which is
hardly plausible in the real world. Few works tackle ego-centric view
navigation, _i.e_., path planning directly from an observer’s ego-centric view
in the crowd. Dugas _et al_. [8] constructed synthetically generated human-
populated environments with a game-engine for vision-based navigation. The
domain gap between real and synthetic environments is, however, not negligible
both in appearances and pedestrian movements. In contrast, our egocentric
pedestrian world model is independent of pedestrian appearance and our dataset
can be easily augmented with arbitrary combinations of observer actions and
real crowd trajectories.

#### Ego2Top Transformation

Transforming first-person-view (FPV) images into top-down maps (BEV) images
has become important for autonomous driving [25, 35, 32]. Most works directly
learn frame-by-frame mapping in a data-driven fashion without taking into
account ego-motion [2, 34], _i.e_., they are only relative to the observer.
Nishimura et al. proposed view birdification which is the task of
reconstructing on-ground positions and trajectories of pedestrians and the
observer, _i.e_., simultaneous decoupling of ego-motion from observed 2D
motions in the FPV and anchoring of all motions in absolute coordinates [23].
Our work differs from BEV transform and view birdification in three critical
points. First, our goal is _predictive_ ego2top, not just a mapping between
two views but predicting constituent positions in this view transform. Second,
we aim at constructing a Pedestrian World Model not from the appearance, but
from the movements, which results in a compact and efficient representation
that can easily generalize. Third, unlike previous approaches which only
perform deterministic mappings [25], we derive a probabilistic formulation
that can handle uncertainty underlying object distances in the FPV.

##  3 Pedestrian World Model

Our goal is to derive an on-ground _Pedestrian World Model_ (PWM), an object-
oriented abstraction of a crowd on the ground, from egocentric views captured
in the crowd. PWM consists of one observer as an actor and pedestrians visible
to that observer as objects. We derive an action-conditioned transition model
of the environment consisting of these objects that can directly learn from
in-environment 2D egocentric perception.

Consider a mobile robot equipped with a vision sensor immersed in a crowd
consisting of people walking towards their own destinations while interacting
with each other. A key characteristic of the PWM we aim to derive is that,
unlike past object-oriented world models [21], the observer is the actor and
is also part of the environment who interacts with other objects
(pedestrians). In our running example, the mobile robot with a vision sensor
is the observer and its ego-motion can be obtained by an IMU or other sensors
including vision-based methods (_e.g_., SLAM [3] and View Birdification [23]).

The observer location at timestep $\tau$ is given by the relative rotation
$R(\theta_{\tau})\in SO(2)$ and translation $\bm{t}_{\tau}\in\mathbb{R}^{2}$
on the ground plane from the previous timestep $\tau-1$. The rotation
$R(\theta_{\tau})$ and $\bm{t}_{\tau}$ directly constitute the observer action
$\bm{a}_{\tau}=\left[\theta_{\tau}|\bm{t}_{\tau}\right]^{\top}\in\mathbb{R}^{3}$.

At every timestep $\tau\in\\{1,\dots,T\\}$, the robot captures an image
$I_{\tau}$ through which it observes the in-image states
$\bm{X}_{\tau}=\\{\bm{x}^{1},\dots,\bm{x}^{N_{\tau}}\\}$ of $N_{\tau}$ people
visible from the robot. Each state $\bm{x}^{n}=\left[u^{n},v^{n},\delta
u^{n},\delta v^{n}\right]^{\top}\;(n=1,\dots,N_{\tau})$ consists of the 2D
center position $\left[u^{n},v^{n}\right]^{\top}$ and its velocity
$\left[\delta u^{n},\delta v^{n}\right]^{\top}$ in $I_{\tau}$ calculated from
the previous and subsequent frames. These states can be computed from the
images with an off-the-shelf multi-object tracker [30], and the robot can keep
track of $\\{1,\dots,N_{\tau}\\}$ pedestrians appearing across frames within a
time window centered at $\tau$.

Given a set of in-image pedestrian states $\bm{X}_{\tau}$ at current time step
$\tau$ and the observer action $\bm{a}_{\tau}$, our objective is to predict
future on-ground states of pedestrians
$\bm{Y}_{\tau+1}=\\{\bm{y}^{1},\dots,\bm{y}^{N_{\tau}}\\}$, where
$\bm{y}^{n}=\left[x^{n},y^{n},\delta x^{n},\delta y^{n}\right]^{\top}$. The
on-ground pedestrian states $\bm{y}$ are described in the observer’s camera
coordinates, _i.e_., we predict 2D on-ground locations and their velocities
relative to the observer’s view which is then converted to absolute
coordinates with the known observer state. Our aim is to construct a
transition model
$\mathcal{T}(\bm{X}_{\tau}|\bm{a}_{\tau})\mapsto\bm{Y}_{\tau+1}$ that predicts
future on-ground pedestrian states
$\bm{Y}_{\tau+1}\in\mathbb{R}^{N_{\tau}\times 4}$ conditioned on the
observer’s action $\bm{a}_{\tau}$. The transition model can be formulated as

$\bm{H}_{\tau}=\mathcal{V}(\bm{X}_{\tau})\,,\quad\bm{Y}_{\tau+1}=\mathcal{G}(\bm{H}_{\tau},\bm{Y}_{\tau},\bm{a}_{\tau})\,,$
(1)

where $\mathcal{V}(\cdot)$ is the _Vision Module_ that encodes the ego-centric
view observation $\bm{X}_{\tau}\in\mathbb{R}^{N_{\tau}\times 4}$ into a
$d$-dimensional embedding $\bm{H}_{\tau}\in\mathbb{R}^{N_{\tau}\times d}$
which represents a set of pedestrian state embeddings in an ego-centric view,
and $\mathcal{G}(\cdot)$ is the _Geometric Memory Module_ which makes
predictions of the future on-ground pedestrian states $\bm{Y}_{\tau+1}$ based
on the past on-ground state estimates $\bm{Y}_{\tau}$ and the current
observation $\bm{X}_{\tau}$ with an implicit transform between the ego-centric
view and the top-down ground view.

##  4 InCrowdFormer

Figure 1: InCrowdFormer consists of two modules referred to as the Vision
Module and the Geometric Memory Module. The Vision Module encodes in-image
interactions with self-attention to produce the current state embeddings
$\bm{H}_{\tau}$ in an ego-centric view. The Geometric Memory Module predicts
on-ground future states of pedestrians with cross-attention between the ego-
centric view and on-ground past trajectories.

We introduce InCrowdFormer, a Transformer-based World Model, that realizes a
PWM with object-centric interaction, ego2top transform, action conditioning,
for a variable number of pedestrians. As Fig. 1 depicts, InCrowdFormer has two
modules, the Vision Module $\mathcal{V}$ and the Geometric Memory Module
$\mathcal{G}$.

###  4.1 Vision Module

The attention mechanism naturally provides a means to learn a mapping between
two sets with variable numbers of constituents. We fully leverage this to
learn the key ingredients, namely pedestrian interaction and ego2top transform
both naturally conditioned on the observer actions, of an on-ground Pedestrian
World Model from egocentric views.

The standard attention mechanism consists of a set of query vectors $\bm{Q}$,
key vectors $\bm{K}$, and value vectors $\bm{V}$. These vectors are generated
from the input tokens $\bm{f}_{s}$ and $\bm{f}_{t}$ as
$\bm{Q}=W_{Q}\bm{f}_{t}\,,\quad\bm{K}=W_{K}\bm{f}_{s}\,,\quad\bm{V}=W_{V}\bm{f}_{s}$,
where $W_{Q}$, $W_{K}$, and $W_{V}$ are linear embedding matrices. In _self-
attention_ $\bm{f}_{s}=\bm{f}_{t}$ and for _cross-attention_ we have
$\bm{f}_{s}\neq\bm{f}_{t}$. Unless otherwise noted, we use multi-head
attention [28] for both self-attention and cross-attention layers.

To model the interactions between pedestrians in the current frame, we use
self-attention to learn the object-wise interactions observed in the ego-
centric view during the state encoding process. As Fig. 1 left depicts, the
Vision Module $\mathcal{V}$ first tokenizes input first-person-view (FPV)
state vectors into $d_{s}$-dimensional tokens with multi-layer perceptron
(MLP) layers, $\bm{X}_{\tau}\mapsto\bm{f}_{s}\in\mathbb{R}^{N_{\tau}\times
d_{s}}$. It then computes self-attention over queries $\bm{Q}$ of tokens
$\bm{f}_{s}$, where we encode their interactions in the image space into
intermediate embeddings $\bm{H}_{\tau}$.

###  4.2 Geometric Memory Module

Learning a Pedestrian World Model is fundamentally different from standard
trajectory forecasting problems [33] in that the on-ground movements of a
crowd that needs to be predicted is deeply intertwined with the observer’s
ego-motion (_i.e_., action). That is, how the pedestrians move relative to the
observer is largely affected by the observer’s action. For this, we model a
mapping
$\mathcal{G}(\bm{a},\bm{H}_{\tau},\bm{Y}_{\tau})\mapsto\bm{Y}_{\tau+1}$ by
$\mathcal{G}=\mathcal{F}_{c}(\mathcal{F}_{s}(\bm{a}_{\tau},\bm{Y}_{\tau}),\bm{H}_{\tau})$,
where we use self-attention $\mathcal{F}_{s}$ to capture social interactions
and cross-attention $\mathcal{F}_{c}$ to model the ego2top transform. This
Geometric Memory Module first tokenizes the on-ground action and past
pedestrian states as
$\\{\bm{a}_{\tau},\bm{Y}_{\tau}\\}\mapsto\bm{f}_{t}\in\mathbb{R}^{(N_{\tau}+1)\times
d_{s}}$ with MLPs and computes self-attention over queries $\bm{Q}$ from
tokens $\bm{f}_{t}$ to encode the interactions between the action and on-
ground pedestrian trajectories. The cross-attention block takes queries
$\bm{Q}\in\mathbb{R}^{(N_{\tau}+1)\times d_{s}}$ from the output of the self-
attention layer and key, values $\bm{K},\bm{V}\in\mathbb{R}^{N_{\tau}\times
d_{s}}$ from the output of the vision module (Figure 1 left).

The geometric memory module is autoregressive, which means the module predicts
future on-ground states one step at a time and uses the current prediction as
input to make future predictions on a subsequent timestep. As the FPV view
state embeddings $\bm{H}_{\tau}$ are updated by the vision module
$\mathcal{V}$ at every timestep, their on-ground future predictions
$\bm{Y}_{\tau+1}$ are generated by the geometric memory module $\mathcal{G}$
from the past predictions $\bm{Y}_{\tau}$, the current observer’s action
$\bm{a}_{\tau}$, and the FPV state embeddings $\bm{H}_{\tau}$.

We use two linear projections;
$\varphi_{p}:\mathbb{R}^{4}\mapsto\mathbb{R}^{d_{s}}$ for encoding in-image
and on-ground pedestrian states, $\bm{X}$ and $\bm{Y}$, and
$\varphi_{a}:\mathbb{R}^{3}\mapsto\mathbb{R}^{d_{s}}$ for the observer’s
action $\bm{a}$. We hard-concatenate positional information into states as
$\bm{x}=\left[u,v\right]\oplus\left[\delta u,\delta v\right]$. To encourage
our model to generalize over diverse combinations of pedestrian tokens and
observer actions, we transform on-ground pedestrian tokens relative to the
observer’s action token at every timestep.

InCrowdFormer can handle a varying number of pedestrians, _i.e_., tokens, in
two ways. First, the attention matrix can be constructed with arbitrary sizes
of input keys $\bm{K}$ and queries $\bm{Q}$. Second, an attention mask
$M\in\mathbb{R}^{(N_{\tau}+1)\times N_{\tau}}$, where $M_{ij}=0$ can be
applied, if the pedestrian id $i$ is missing (_e.g_., occluded) at frame
$\tau$, otherwise $1$. The masked attention $\hat{A}$ becomes
$\hat{A}_{\tau}=M_{\tau}\odot A_{\tau}$, where $\odot$ denotes the Hadamard
product.

##  5 Probabilistic InCrowdFormer

We encode uncertainties arising from unknown pedestrian heights by making
InCrowdFormer reason probabilistically on the pedestrian positions and their
transitions. Figure 2 depicts an overview of the training and testing process
of our Probabilistic InCrowdFormer.

Figure 2: Overview of Probabilistic InCrowdFormer. The output MLP layer of the
Vision Module models the prior distribution. Each of the two geometric memory
modules models the approximated posterior distribution and likelihood
distribution, respectively. The latent codes are sampled from the approximated
posterior module during training, and from the prior module at inference time.

Given a triplet of the current observed pedestrian states in the ego-centric
view $\bm{X}_{\tau}$, the current observer’s action $\bm{a}_{\tau}$, and
future pedestrian states $\bm{Y}_{\tau+1}$, we construct a generative model
based on Conditional VAE (CVAE). The CVAE learns the target on-ground
pedestrian trajectory distributions in the future
$p(\bm{Y}_{\tau+1}|\bm{X}_{\tau},\bm{a}_{\tau})$ conditioned on the current
ego-centric view observation $\bm{X}_{\tau}$ and the observer’s action
$\bm{a}_{\tau}$ through a stochastic latent variable $\bm{Z}$. Our model
consists of a conditional prior network $p_{\psi}(\bm{Z}|\bm{X}_{\tau})$ to
model the latent variable $\bm{Z}$ from the observation $\bm{X}$, an
approximated posterior network
$q_{\phi}(\bm{Z}|\bm{Y}_{\tau+1},\bm{X}_{\tau},\bm{a}_{\tau})$, and a
likelihood network
$p_{\eta}(\bm{Y}_{\tau+1}|\bm{Z},\bm{X}_{\tau},\bm{a}_{\tau})$. The objective
of the CVAE model is to minimize the negative evidence lower bound (ELBO) in
our loss function $\mathcal{L}$

$\displaystyle\begin{split}\mathcal{L}_{\text{ELBO}}=&\mathbb{E}_{q_{\phi}(\bm{Z}|\bm{Y},\bm{X})}[\log\underbrace{p_{\eta}(\bm{Y}_{\tau+1}|\bm{Z},\bm{X}_{\tau},\bm{a}_{\tau})}_{\text{likelihood}}]\\\
&-\mathrm{KL}[\underbrace{q_{\phi}(\bm{Z}|\bm{Y}_{\tau+1},\bm{X}_{\tau},\bm{a}_{\tau})}_{\text{approximated
posterior}}\|\underbrace{p_{\psi}(\bm{Z}|\bm{X}_{\tau})}_{\text{prior}}]\,,\end{split}$
(2)

where the first term maximizes the expectation of the log-likelihood of the
target future state in the predicted distribution, and the second term
minimizes the Kullback-Leibler (KL) divergence between the approximated
posterior distribution and the prior distribution. Please see Supplementary
Material A. for a more detailed derivation. In what follows, we describe how
we implement each distribution in our Probabilistic InCrowdFormer.

#### Prior Module

The _Vision Module_ can be seen as as a prior network that produces latent
codes $\bm{Z}=\\{\bm{z}^{1},\dots,\bm{z}^{N_{\tau}}\\}$, each of which follows
a Gaussian distribution
$p(\bm{z}^{n}|\bm{X}_{\tau})\sim\mathcal{N}(\mu^{n},(\sigma^{n})^{2})$. We
process the output of the Vision Module
$\bm{H}_{\tau}=\\{\bm{h}^{1},\dots,\bm{h}^{N_{\tau}}\\}$ with a single MLP
layer to map them into parameters of each Gaussian distribution
($\mu^{n}_{p},\sigma^{n}_{p})$.

#### Approximated Posterior Module

To approximate the posterior distribution
$q_{\phi}(\bm{z}^{n}|\bm{Y}_{\tau+1},\bm{X}_{\tau},\bm{a}_{\tau})$, we use a
Transformer decoder with the same self-attention and cross-attention layers as
the _Geometric Memory Module_. The cross-attention layers in the Transformer
decoder can efficiently model conditional relationships by taking the
embeddings of FPV states $\bm{H}_{\tau}$ as keys and values, and the action
and the ground-truth future state on the ground
$\\{\bm{a}_{\tau},\bm{Y}_{\tau+1}\\}$ as queries. We also assume the
approximated posterior distribution follows a Gaussian distribution
$q_{\phi}(\bm{z}^{n}|\bm{Y}_{\tau+1},\bm{X}_{\tau},\bm{a}_{\tau})\sim\mathcal{N}(\mu^{n},(\sigma^{n})^{2})$.
MLP layers process the output of the Transformer Decoder to obtain the
Gaussian parameters $(\mu^{n}_{q},\sigma^{n}_{q})$ as in the Prior Module.
Note that this approximated posterior module is used only during training.

#### Likelihood Module

We can view the _Geometric Memory Module_ as a likelihood module to model
$p_{\eta}(\bm{Y}_{\tau+1}|\bm{Z},\bm{X}_{\tau},\bm{a}_{\tau})$. The latent
codes $\bm{Z}$ for pedestrians are computed from the approximated posterior
module during training, and from the vision module for inference. As we obtain
predictions of future pedestrian states autoregressively from the geometric
memory module, we feed $\bm{Y}$ concatenated with corresponding latent codes
$\bm{Z}$ alongside the observer’s action $\bm{a}_{\tau}$. We concatenate
$\bm{y}^{n}\oplus\bm{z}^{n}$ for each pedestrian and tokenize these as input
queries. The cross-attention layers in the geometric memory module effectively
models the conditional relationship between $\bm{Z}$ and $\bm{a}_{\tau}$ and
outputs a likelihood distribution. We consider two variants of the proposed
method based on the definition of the outputs, InCrowdFormer-D and
InCrowdFormer-G.

InCrowdFormer-D directly estimates the future pedestrian states and, for the
first term of Eq. 4, uses the mean squared error (MSE),
$\mathcal{L}_{\text{D}}=\|\hat{\bm{Y}}-\bm{Y}\|^{2}$, where $\hat{\bm{Y}}$ is
the ground-truth future state of pedestrians on the ground.

InCrowdFormer-G takes into account uncertainties arising from ego2top
transformation of unknown object scales as aleatoric uncertainty [17] computed
as a probability distribution over the InCrowdFormer outputs. We model the
uncertainty with a 2D Gaussian distribution on the ground
$p(\bm{Y}_{\tau+1}|\bm{Z},\bm{X}_{\tau},\bm{a}_{\tau})\sim\mathcal{N}(\bm{\mu}^{n},\Sigma^{n})$,
and make InCrowdFormer output these Gaussian parameters for each pedestrian by
defining the first term of Eq. 4 as a negative log-likelihood loss of a 2D
Gaussian function
$\mathcal{L}_{\text{G}}=\frac{\|\hat{\bm{Y}_{x}}-\bm{Y}_{x}\|^{2}}{(2\sigma_{x})^{2}}+\frac{\|\hat{\bm{Y}_{y}}-\bm{Y}_{y}\|^{2}}{(2\sigma_{y})^{2}}$.
Such a parametric representation of uncertainty of the future pedestrian state
would be useful for downstream applications.

##  6 Experiments

We evaluate the effectiveness of Probabilistic InCrowdFormer (hereafter simply
InCrowdFormer) trained on real-world crowd trajectories with augmented
observer’s actions in terms of its $T$-step prediction accuracy and also
demonstrate its application to real video sequences. We provide more
implementation details in the supplementary material.

Table 2: Quantitative Results of InCrowdFormer applied to the ego-centric view crowd dataset. (IV) and (CV) indicate the intra-scene, and the cross-scene validation split, respectively. ADET and FDET represent average and final displacement errors [m] in $T$-step prediction. InCrowdFormer variants clearly outperform baseline methods in terms of $5$-step and $10$-step prediction accuracy. Prediction results applied to the cross-scene validation split dataset demonstrate that our method can be generalized to unknown crowd scenarios. Dataset (IV) | Hotel (sparse) | ETH (mid) | Students (dense)
---|---|---|---
Metrics |  ADE5 |  FDE5 |  ADE10 |  FDE10 |  ADE5 |  FDE5 |  ADE10 |  FDE10 |  ADE5 |  FDE5 |  ADE10 |  FDE10
MLP-TSSM [8, 6] | 4.009 | 4.763 | 6.014 | 6.432 | 4.102 | 4.835 | 5.986 | 6.287 | 5.939 | 6.126 | 6.723 | 6.921
MLP-RSSM [10] | 4.251 | 4.861 | 6.027 | 6.853 | 4.562 | 4.925 | 6.102 | 6.489 | 6.102 | 6.615 | 6.532 | 6.911
C-SWM [19] | 1.475 | 1.592 | 1.972 | 2.018 | 1.535 | 1.682 | 1.920 | 2.012 | 1.428 | 1.492 | 2.156 | 2.215
SlotFormer [31] | 1.512 | 1.618 | 1.815 | 1.901 | 1.631 | 1.721 | 1.810 | 1.912 | 1.419 | 1.521 | 2.016 | 2.161
InCrowdFormer-D | 0.358 | 0.468 | 0.626 | 0.829 | 0.326 | 0.385 | 0.537 | 0.683 | 0.251 | 0.273 | 0.432 | 0.483
InCrowdFormer-G | 0.327 | 0.472 | 0.648 | 0.818 | 0.372 | 0.393 | 0.526 | 0.652 | 0.312 | 0.372 | 0.461 | 0.492
Dataset (CV) | Hotel (sparse) | ETH (mid) | Students (dense)
MLP-TSSM [8, 6] | 4.228 | 4.934 | 6.100 | 6.353 | 4.147 | 4.941 | 6.175 | 6.318 | 6.064 | 6.208 | 6.737 | 7.007
MLP-RSSM [10] | 4.315 | 4.856 | 6.223 | 6.843 | 4.549 | 5.020 | 6.105 | 6.615 | 5.849 | 6.649 | 6.544 | 7.002
C-SWM [19] | 1.515 | 1.613 | 2.215 | 2.251 | 1.721 | 1.810 | 2.018 | 2.231 | 1.615 | 1.712 | 2.381 | 2.519
SlotFormer [31] | 1.618 | 1.712 | 1.920 | 2.060 | 1.809 | 1.910 | 1.915 | 2.016 | 1.591 | 1.612 | 2.130 | 2.210
InCrowdFormer-D | 0.687 | 0.638 | 0.717 | 1.037 | 0.469 | 0.421 | 0.580 | 0.601 | 0.323 | 0.383 | 0.512 | 0.568
InCrowdFormer-G | 0.536 | 0.272 | 0.714 | 0.881 | 0.372 | 0.428 | 0.646 | 0.722 | 0.437 | 0.523 | 0.578 | 0.720

#### Ego-centric View Crowd Dataset

We construct a crowd dataset consisting of on-ground real pedestrian
trajectories paired with their state features in an ego-centric view. We first
extract trajectories referred to as Hotel, ETH, and Students from the ETH [24]
and the UCY [20] datasets. These three sets of trajectories correspond to
sparse, moderate, and dense crowds. To compute the egocentric views of those
pedestrians, we first sample pedestrian heights from a Gaussian distribution
$\mathcal{N}(\mu,\sigma)$ with $\mu=1.70$ m and $\sigma=0.07$ m according to
the statistics of European adults [29]. The virtual head points with the
height $h\sim\mathcal{N}(\mu,\sigma)$ are then projected to the 2D positions
$\left[u,v\right]$ by perspective projection with a known intrinsic matrix
$A$. For each pedestrian, we compute the state feature $\left[u,v,\delta
u,\delta v\right]$ in the ego-centric view and pair it with its corresponding
on-ground position. Since our proposed Pedestrian World Model is an
abstraction of the on-ground crowd movements, we do not need photo-realistic
renderings in the datasets. This allows us to augment datasets with diverse
combinations of ego-motion and crowd pedestrian trajectories easily and
efficiently, which is otherwise challenging, if not impossible, to collect in
the real world.

#### Observer Action Generation

To model the transition of the world in the observer’s view when navigating in
a crowd while avoiding potential collisions, we generate plausible observer’s
trajectories with an Optimal Reciprocal Collision Avoidance (ORCA) planner
[27] in the ego-centric view crowd dataset. We randomly sample starting
positions on a circle with a fixed radius $r=8.0$ m and the observer walks to
its destination set at the opposite side of the circle by the planner. We
mount two virtual, perspective cameras in front and rear on the observer, each
of which captures states of the pedestrians in the crowd.

#### Baselines

Our method is the first to realize an object-oriented, on-ground world model
conditioned on the observer’s action trajectory solely from ego-centric in-
crowd observations. Note that there is no existing method that achieves
exactly the same task. We compare with original, object-oriented baselines
composed of the cascaded object encoder and the memory module which are
components of the state-of-the-art world models [19, 31, 10, 6]. In
particular, we use a standard MLP object encoder/decoder and memory modules
referred to as Recurrent State Space Model (RSSM) [10] and Transformer State
Space Model (TSSM) [6]. We refer to these baselines as MLP-RSSM and MLP-TSSM.
We also compare our method with variants of recent object-oriented world
models. We extend C-SWM [19] and SlotFormer [31] to consider the observer’s
action in their memory module and combine it with the standard MLP object
encoder/decoder. In their original implementations, these world models encode
transitions in ego-centric views, unlike our model which encodes on-ground
transitions conditioned on the ego-views. Furhter, we consider two variants of
InCrowdFormer, InCrowdFormer-D (deterministic) and InCrowdFormer-G (Gaussian)
as introduced in Sec. 5.

Figure 3: Qualitative results of InCrowdFormer-G applied to the Hotel (top
row), ETH (middle), and Students (bottom) datasets. Left most: First frames in
the ego-centric views. Rest: On-ground future prediction results in subsequent
frames. Negative heights of bounding boxes depict observations from the rear
camera. The probabilities are rendered with red (high) to blue (low) heatmaps.
Stars depict the ground-truth positions. Our model successfully predicts on-
ground pedestrian states for crowds of diverse densities. Even in a dense
crowd, our method can successfully handle the varying number of pedestrians
and predict accurate future locations of nearby pedestrians. Figure 4:
Inference results of InCrowdFormer-G applied to real video sequences from the
JRDB dataset. Top row shows input ego-centric view video and pedestrian
bounding boxes. Bottom row shows on-ground $\tau+1$ future pedestrian
trajectories predicted by our model. Our method can easily be adapted to real
video sequences as it only requires the positions and scales of the bounding
boxes of pedestrians.

#### Metric

Given a sequence of observer’s actions and corresponding ego-centric view
observations, we predict the on-ground state of pedestrians in the future by
autoregressively feeding an action $\bm{a}_{\tau}$ and ego-centric view states
$\bm{X}_{\tau}$ into our InCrowdFormer model. For a set of action sequences
$\\{\bm{a}_{1},\dots,\bm{a}_{T}\\}$ and corresponding ego-centric view states
$\\{\bm{X}_{1},\dots,\bm{X}_{T}\\}$ during $T$-steps in the testing set, we
evaluate the prediction accuracy with Average Displacement Error (ADE) and
Final Displacement Error (FDE). We define ADET
$=\frac{1}{TN^{T}}\sum_{n=1}^{N^{T}}\sum_{\tau=1}^{T}\|\hat{\bm{y}}^{n}_{\tau}-\bm{y}^{n}_{\tau}\|$
and FDET
$=\frac{1}{N^{T}}\sum_{n=1}^{N^{T}}\|\hat{\bm{y}_{T}^{n}}-\bm{y}_{T}^{n}\|$,
respectively. For InCrowdFormer-D and InCrowdFormer-G, we compute the best-
of-10 results, _i.e_., the minimum ADE and FDE from 10 randomly-sampled latent
codes as in [9].

###  6.1 Quantitative evaluation of prediction accuracy

We prepare two validation splits for the ego-centric view crowd dataset. One
is the intra-scene validation split, and the other is the cross-scene
validation split, _i.e_., leave-one-out cross-validation. Table 2 shows the
prediction accuracy evaluated in ADET and FDET with $T\in\\{5,10\\}$. The RSSM
baseline makes a prediction without taking object interactions into account,
which results in the lowest accuracy in all the metrics. Both the
InCrowdFormer-D and InCrowdFormer-C outperform all the baselines in terms of
short-term ($T=5$) and long-term ($T=10$) prediction accuracy. Finally, our
method achieves high accuracy on the dataset with the cross-scene validation
split. This demonstrates that our method can be generalized to unknown scenes
by training the model with a diverse density of crowds. Most important, our
method still achieves high ($<1.0$ [m]) accuracy for long-term ($T=10$)
prediction sequences as well as short-term ($T=5$) sequences by attending to
observed ego-centric view features at every timestep.

#### Qualitative Results

Figure 3 visualizes the prediction results of InCrowdFormer-G from time $\tau$
to $\tau+5$ frames on the Hotel, ETH, and Students datasets. InCrowdFormer-G
outputs the future location of a pedestrian with a 2D Gaussian distribution
which captures uncertainty arising from imperfect cues of depths and
pedestrian interactions. Our method also predicts future locations of nearby
pedestrians in a dense crowd by masking the attention matrix for occluded
pedestrians. These outputs are beneficial to downstream applications such as
robot crowd navigation, where we should path-plan to avoid potential
collisions with nearby pedestrians from an ego-centric view and limited depth
information.

###  6.2 Ablation Study

#### Effect of ego-centric view for prediction

To evaluate the effectiveness of the cross-attention between the ego-centric
and the on-ground view inputs, we prepare a variant of our method that takes
only on-ground past features as inputs and outputs on-ground future
trajectories by self-attention layers. The first and the third row of Table 3
clearly demonstrate that our ego-centric view-conditioned future prediction
outperforms on-ground future trajectory prediction that depends only on the
on-ground past trajectories.

#### Effect of cross-attention

We also prepare a variant of our method consisting of only self-attention
layers. For this variant, we replace the geometric memory module with one that
consists of just two self-attention layers. The modified geometric memory
module takes the output of the vision module as inputs and calculates the on-
ground future states with self-attention. In Table 3, the variants of our
model without the cross-attention (second row) underperform in terms of
prediction accuracy, which clearly demonstrates that the cross-attention
between the vision module and the geometric memory module is essential to our
model.

#### Effect of uncertainty modeling

We compare two variants of the proposed method based on the choice of the
outputs. In Table 2, overall, InCrowdFormer-D achieves slightly higher
accuracy than InCrowdFormer-G, which can be attributed to the fact that
InCrowdFormer-D directly regresses future positions while InCrowdFormer-G
predicts uncertainty distributions.

Table 3: Ablation study of the cross-attention mechanism in our proposed method. These results clearly demonstrate that the cross-attention between the vision module and the geometric memory module is essential in our model. Dataset (IV) | Hotel (sparse) | ETH (mid) | Students (dense)
---|---|---|---
ego-view | cross-attn |  ADE5 |  ADE10 |  ADE5 |  ADE10 |  ADE5 |  ADE10
– | – | 0.482 | 0.648 | 0.495 | 0.482 | 0.391 | 0.523
✓ | – | 0.526 | 0.702 | 0.546 | 0.728 | 0.451 | 0.826
✓ | ✓ | 0.358 | 0.626 | 0.326 | 0.537 | 0.251 | 0.432

###  6.3 Inference on Real Data

Figure 4 shows the inference results of InCrowdFormer-G applied to real video
sequences from the JRDB dataset [22]. We first train our model with the same
camera parameter of JRDB. We train on the ETH dataset, which is similar in
crowd density to the JRDB sequences. We then apply our pre-trained model to
the target sequences. Our method can easily be adapted to real video sequences
as it requires only the position and scale of pedestrian bounding boxes and
abstracts away their appearance.

##  7 Conclusion

In this paper, we introduced InCrowdFormer, a Transformer-based Pedestrian
World Model that predicts on-ground pedestrian trajectories from an ego-
centric view observation. Our extensive evaluation demonstrates the
effectiveness of our approach in diverse density of crowds, and also show
promising results in zero-shot adaptation to real video sequences. We believe
our InCrowdFormer will serve as a sound foundation for crowd and pedestrian
movement modeling and enable a wide range of downstream applications including
but not limited to navigation. We will release all the code and data to
catalyze such use.

#### Limitation

In this study, we assumed perfect odometry observation for the observer’s
action, _i.e_., known observer ego-motion, and did not consider observation
noise in the ego-centric views. We believe that this can be modeled similarly
as uncertainty in depth arising from pedestrian’s height variances, which we
plan to explore in future work.

##  Appendix A Conditional VAE (CVAE)

We provide further details of our Probabilistic InCrowdFormer described in
Section 5. of the main manuscript. To model the distribution of the future on-
ground states of pedestrians $p(\bm{Y}_{\tau+1}|\bm{X}_{\tau},\bm{a}_{\tau})$
conditioned on the ego-centric view observation $\bm{X}_{\tau}$ and the
observer’s action $\bm{a}_{\tau}$, we formulate it as a Conditional
Variational Autoencoder (CVAE) [26]. CVAE is a conditional generative model
that outputs the target $\bm{Y}$ conditioned on the latent variable $\bm{Z}$
and the input observation $\bm{X}$. We introduce $d_{z}$-dimensional latent
codes for each pedestrian
$\bm{Z}=\\{\bm{z}_{1},\dots,\bm{z}_{N_{\tau}}\\}\in\mathbb{R}^{N_{\tau}\times
d_{z}}$ and re-formulate the future distribution of the pedestrian states
$\bm{Y}_{\tau+1}$ as

$\displaystyle
p(\bm{Y}_{\tau+1}|\bm{X}_{\tau},\bm{a}_{\tau})=\int\underbrace{p(\bm{Y}_{\tau+1}|\bm{Z},\bm{X}_{\tau},\bm{a}_{\tau})}_{\text{likelihood}}\underbrace{p(\bm{Z}|\bm{X}_{\tau})}_{\text{prior}}d\bm{Z}\,,$
(3)

where $p(\bm{Y}_{\tau+1}|\bm{Z},\bm{X}_{\tau},\bm{a}_{\tau})$ is the
conditional likelihood and
$p(\bm{Z}|\bm{X}_{\tau})=\prod_{n}p(\bm{z}^{n}|\bm{X}_{\tau})$ is the
conditional Gaussian prior factorized over pedestrians. The observer’s action
is deterministic in our Pedestrian World Model, so that latent codes are only
necessary for pedestrians and not the observer.

Due to the intractable integral computation in Eq. 3, we instead minimize the
negative evidence lower bound (ELBO) in our loss function $\mathcal{L}$

$\displaystyle\begin{split}\mathcal{L}_{\text{ELBO}}(\bm{X}_{\tau},&\bm{Y}_{\tau+1},\bm{a}_{\tau};\eta,\phi,\psi)=\\\
&\mathbb{E}_{q_{\phi}(\bm{Z}|\bm{Y},\bm{X})}[\log\underbrace{p_{\eta}(\bm{Y}_{\tau+1}|\bm{Z},\bm{X}_{\tau},\bm{a}_{\tau})}_{\text{likelihood}}]\\\
&-\mathrm{KL}[\underbrace{q_{\phi}(\bm{Z}|\bm{Y}_{\tau+1},\bm{X}_{\tau},\bm{a}_{\tau})}_{\text{approximated
posterior}}\|\underbrace{p_{\psi}(\bm{Z}|\bm{X}_{\tau})}_{\text{prior}}]\,,\end{split}$
(4)

where the first term maximizes the expectation of the log-likelihood of the
target future state in the predicted distribution, and the second term
minimizes the Kullback-Leibler (KL) divergence between the approximated
posterior distribution and the prior distribution.

Consequently, our CVAE-based probabilistic InCrowdFormer consists of the three
modules, a conditional prior network $p_{\psi}(\bm{Z}|\bm{X}_{\tau})$, an
approximated posterior network
$p_{\phi}(\bm{Z}|\bm{Y}_{\tau+1},\bm{X}_{\tau},\bm{a}_{\tau})$, and a
likelihood network
$p_{\eta}(\bm{Y}_{\tau+1}|\bm{Z},\bm{X}_{\tau},\bm{a}_{\tau}).$ Given the
current observed pedestrian states in the ego-centric view $\bm{X}_{\tau}$,
the observer’s action $\bm{a}_{\tau}$, and future pedestrian state relative to
the observer $\bm{Y}_{\tau+1}$, our goal is to learn a generative model
$p(\bm{Y}_{\tau+1}|\bm{X}_{\tau},\bm{a}_{\tau})$ by learning network
parameters $\phi,\psi$, and $\eta$ by minimizing Eq. 4.

##  Appendix B In-Depth Analysis of Qualitative Results

Figure 5: Prediction result of InCrowdFormer-G conditioned on a monotonous
straightforward ego-motion (Left) and a curving ego-motion (Right). The
variance of the Gaussian distribution corresponds to the uncertainty of the
estimated location.

Figure 5 compares prediction results with respect to the observer’s ego-
motion. In the left case, both the camera and pedestrians are going straight
and our model can easily localize the future locations of pedestrians
certainly. In the right case, the predicted Gaussian distributions have
heavier tails as the model needs to account for complex interactions between
the observer and the surrounding pedestrians. We expect that these
probabilistic representations are indeed beneficial for downstream
applications such as navigating in a crowd, where the observer seeks a path
while avoiding potential collisions with nearby pedestrians.

##  Appendix C Additional Experimental Details

We use an MLP with two layers and $16$ hidden dimension to encode the input
state $\bm{x}$ into $d_{s}=32$ dimensional input tokens. The latent dimensions
of $\bm{z}$ is set to $32$. We use two self-attention layers in the vision
module and also two self-attention and cross-attention layers in the geometric
memory module. We set the number of heads for multi-head attention (MHA) to
six in each attention layer. For RSSM and TSSM, we set the input embedding
size to $32$, which is the same size as $d_{s}$ of our method. We also use
another MLP with the same structure to encode and decode future state
$\bm{y}$. All of these models are implemented in PyTorch and trained with the
Adam optimizer [18] for $800$ epochs with minibatch size $32$ and learning
rate $0.001$. All experiments are run on a PC with Intel(R) Xeon(R) Gold 6752
CPU and an NVIDIA V100 GPU.

##  Appendix D Statistics of Crowd Dataset

Table 4 shows the crowd density of each dataset. Based on the average number
of people appearing in one frame, we refer to Hotel, ETH, and Students as
sparse, moderate, and dense crowd scenarios, respectively. In each scenario,
we generated $500$ trajectories of the moving observer with random starts and
goals with ORCA [27] for the training split and $200$ for the testing split.

Table 4: Crowd densities in each dataset. TrajLen denotes the average duration of each pedestrian’s trajectory. $N$ People in Crowd shows the average, minimum, and maximum numbers of people appearing in a frame. | TrajLen |  $N$ People in Crowd
---|---|---
Dataset | Avg | Avg | Min | Max
Hotel (sparse) | 15.0 | 6.31 | 3 | 15
ETH (moderate) | 14.4 | 9.29 | 3 | 26
Students (dense) | 45.8 | 44.2 | 13 | 75

##  Appendix E Quantitative Evaluation on JRDB sequence

JackRabbot Dataset and Benchmark (JRDB) [22] includes RGB images with 2D-3D
bounding box annotations for pedestrians captured by the JackRabbot moving
platform. We select two sequences where the moving platform is smoothly
navigating in a crowd. We refer to these two sequences, _clark-center-
intersection-2019-02-08_0_ and _clark-center-2019-02-08_0_ as clark-center I
and clark-center II. They correspond to Figure 5 left and Figure 5 right in
the main text. As described in the main text, we applied our InCrowdFormer-D
trained with the ETH dataset. We used the original camera parameter of JRDB
for these two sequences and evaluated in ADET and FDET. Since JRDB does not
provide the ground-truth of the ego-motion of the mobile platform, we fed the
IMU values as the observer’s action, which was parsed from their provided
rosbag data. Table 5 shows quantitative results of InCrowdFormer-G applied to
these two JRDB sequences. Even without re-training on the JRDB data, our
method shows good prediction accuracy on the real video sequences by using
only pedestrian bounding boxes extracted from the ego-centric images.

Table 5: Quantitative results of InCrowdFormer-G applied to JRDB sequences. Our model trained with ETH dataset demonstrates good prediction accuracy. Metric |  ADE5 |  FDE5 |  ADE10 |  FDE10
---|---|---|---|---
clark-center I | 0.325 | 0.386 | 0.421 | 0.459
clark-center II | 0.227 | 0.244 | 0.315 | 0.322
ETH (trained) | 0.372 | 0.393 | 0.526 | 0.652

## References

  * [1] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese.  Social lstm: Human trajectory prediction in crowded spaces.  In Proc. CVPR, pages 961–971, 2016.
  * [2] Lorenzo Bertoni, Sven Kreiss, and Alexandre Alahi.  Monoloco: Monocular 3d pedestrian localization and uncertainty estimation.  In Proc. ICCV, pages 6861–6871, 2019.
  * [3] Berta Bescos, José M Fácil, Javier Civera, and José Neira.  Dynaslam: Tracking, mapping, and inpainting in dynamic scenes.  IEEE Robotics and Automation Letters, 3(4):4076–4083, 2018.
  * [4] Changan Chen, Sha Hu, Payam Nikdel, Greg Mori, and Manolis Savva.  Relational graph learning for crowd navigation.  In Proc. IROS, 2020.
  * [5] Changan Chen, Yuejiang Liu, Sven Kreiss, and Alexandre Alahi.  Crowd-robot interaction: Crowd-aware robot navigation with attention-based deep reinforcement learning.  In Proc. ICRA, pages 6015–6022. IEEE, 2019.
  * [6] Chang Chen, Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn.  Transdreamer: Reinforcement learning with transformer world models.  arXiv preprint arXiv:2202.09481, 2022.
  * [7] Yuying Chen, Congcong Liu, Bertram E Shi, and Ming Liu.  Robot navigation in crowds by graph convolutional networks with attention learned from human gaze.  IEEE Robotics and Automation Letters, 5(2):2754–2761, 2020.
  * [8] Daniel Dugas, Olov Andersson, Roland Siegwart, and Jen Jen Chung.  NavDreams: Towards camera-only RL navigation among humans.  In Proc. IROS, 2022.
  * [9] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi.  Social gan: Socially acceptable trajectories with generative adversarial networks.  In Proc. CVPR, pages 2255–2264, 2018.
  * [10] David Ha and Jürgen Schmidhuber.  Recurrent world models facilitate policy evolution.  In Proc. NeurIPS, pages 2451–2463. Curran Associates, Inc., 2018\.
  * [11] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi.  Dream to control: Learning behaviors by latent imagination.  In Proc. ICLR, 2019.
  * [12] Danijar Hafner, Timothy P Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson.  Learning latent dynamics for planning from pixels.  In Proc. ICML, 2019.
  * [13] Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba.  Mastering atari with discrete world models.  In Proc. ICLR, 2021.
  * [14] Dirk Helbing and Peter Molnar.  Social force model for pedestrian dynamics.  Physical review E, 51(5):4282, 1995.
  * [15] Boris Ivanovic and Marco Pavone.  The trajectron: Probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs.  In Proc. ICCV, pages 2375–2384, 2019.
  * [16] Philip Nicholas Johnson-Laird.  Mental models: Towards a cognitive science of language, inference, and consciousness.  Number 6. Harvard University Press, 1983.
  * [17] Alex Kendall and Yarin Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  Proc. NeurIPS, 30, 2017.
  * [18] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [19] Thomas Kipf, Elise van der Pol, and Max Welling.  Contrastive learning of structured world models.  In Proc. ICLR, 2019.
  * [20] Alon Lerner, Yiorgos Chrysanthou, and Dani Lischinski.  Crowds by example.  Computer graphics forum, 26(3):655–664, 2007.
  * [21] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Bofeng Fu, Jindong Jiang, and Sungjin Ahn.  Improving generative imagination in object-centric world models.  In Proc. ICML, 2020.
  * [22] Roberto Martin-Martin, Mihir Patel, Hamid Rezatofighi, Abhijeet Shenoi, JunYoung Gwak, Eric Frankel, Amir Sadeghian, and Silvio Savarese.  Jrdb: A dataset and benchmark of egocentric robot visual perception of humans in built environments.  IEEE transactions on pattern analysis and machine intelligence, 2021\.
  * [23] Mai Nishimura, Shohei Nobuhara, and Ko Nishino.  View birdification in the crowd: Ground-plane localization from perceived movements.  In Proc. BMVC, 2021.
  * [24] Stefano Pellegrini, Andreas Ess, Konrad Schindler, and Luc Van Gool.  You’ll never walk alone: Modeling social behavior for multi-target tracking.  In Proc. ICCV, pages 261–268, 2009.
  * [25] Avishkar Saha, Oscar Mendez, Chris Russell, and Richard Bowden.  Translating images into maps.  In Proc. ICRA, pages 9200–9206. IEEE, 2022.
  * [26] Kihyuk Sohn, Honglak Lee, and Xinchen Yan.  Learning structured output representation using deep conditional generative models.  28, 2015.
  * [27] Jur Van Den Berg, Stephen J Guy, Ming Lin, and Dinesh Manocha.  Reciprocal n-body collision avoidance.  In Robotics research, pages 3–19. Springer, 2011.
  * [28] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Proc. NeurIPS, pages 5998–6008, 2017.
  * [29] Peter M. Visscher.  Sizing up human height variation.  Nature Genetics, 40:489–490, 2008.
  * [30] Zhongdao Wang, Liang Zheng, Yixuan Liu, and Shengjin Wang.  Towards real-time multi-object tracking.  In Proc. ECCV, 2020.
  * [31] Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg.  Slotformer: Unsupervised visual dynamics simulation with object-centric models.  Proc. ICLR, 2023.
  * [32] Weixiang Yang, Qi Li, Wenxi Liu, Yuanlong Yu, Yuexin Ma, Shengfeng He, and Jia Pan.  Projecting your view attentively: Monocular road scene layout estimation via cross-view transformation.  In Proc. CVPR, pages 15536–15545, 2021.
  * [33] Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris Kitani.  Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting.  In Proc. ICCV, 2021.
  * [34] Jianfeng Zhang, Dongdong Yu, Jun Hao Liew, Xuecheng Nie, and Jiashi Feng.  Body meshes as points.  Proc. CVPR, 2021.
  * [35] Brady Zhou and Philipp Krähenbühl.  Cross-view transformers for real-time map-view semantic segmentation.  In Proc. CVPR, pages 13760–13769, 2022.