[datatype=bibtex] [fieldsource=journal, match=Robotics: Science and Systems,
replace=RSS] [fieldsource=booktitle, match=Robotics: Science and Systems,
replace=RSS] [fieldsource=journal, match=International Conference on Robotics
and Automation, replace=ICRA] [fieldsource=booktitle, match=International
Conference on Robotics and Automation, replace=ICRA] [fieldsource=journal,
match=International Conference on Learning Representations, replace=ICLR]
[fieldsource=booktitle, match=International Conference on Learning
Representations, replace=ICLR] [fieldsource=booktitle, match=European
Conference on Computer Vision, replace=ECCV] [fieldsource=journal,
match=European Conference on Computer Vision, replace=ECCV]
[fieldsource=booktitle, match=International Conference on Computer Vision,
replace=ICCV] [fieldsource=journal, match=International Conference on Computer
Vision, replace=ICCV] [fieldsource=booktitle, match=IEEE Transactions on
Pattern Analysis and Machine Intelligence, replace=PAMI] [fieldsource=journal,
match=IEEE Transactions on Pattern Analysis and Machine Intelligence,
replace=PAMI] [fieldsource=booktitle, match=International Conference on
Intelligent Robots and Systems, replace=IROS] [fieldsource=booktitle,
match=Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, replace=CVPR] [fieldsource=booktitle, match=Conference on
Computer Vision and Pattern Recognition, replace=CVPR] [fieldsource=journal,
match=International, replace=Int.] [fieldsource=booktitle,
match=International, replace=Int.] [fieldsource=journal, match=Conference,
replace=Conf.] [fieldsource=booktitle, match=Conference, replace=Conf.]
[fieldsource=journal, match=Transactions, replace=Trans.]
[fieldsource=booktitle, match=Transactions, replace=Trans.]
[fieldsource=booktitle, match=Robotics and Automation Letters, replace=RA-L]
[fieldsource=journal, match=Robotics and Automation Letters, replace=RA-L]
[fieldsource=booktitle, match=Advances in Neural Information Processing
Systems, replace=NeurIPS] [fieldsource=journal, match=Advances in Neural
Information Processing Systems, replace=NeurIPS] [fieldsource=booktitle,
match=Neural Information Processing Systems, replace=NeurIPS]
[fieldsource=journal, match=Neural Information Processing Systems,
replace=NeurIPS]

# Robots That Can See:
Leveraging Human Pose for Trajectory Prediction

Tim Salzmann1,2, Hao-Tien Lewis Chiang1, Markus Ryll2, Dorsa Sadigh1,3,
Carolina Parada1, and Alex Bewley1 Manuscript received: May 2nd, 2023;
Revised: July 21st, 2023; Accepted: August 21st, 2023.This paper was
recommended for publication by Gentiane Venture upon evaluation of the
Associate Editor and Reviewers’ comments.1Tim Salzmann, Hao-Tien Lewis Chiang,
Dorsa Sadigh, Carolina Parada, and Alex Bewley are with Google DeepMind {tsal,
lewispro, dorsas, carolinap<EMAIL_ADDRESS>Salzmann and Markus Ryll
are with Technical University Munich {tim.salzmann<EMAIL_ADDRESS>Sadigh is with Stanford University<EMAIL_ADDRESS>Object
Identifier (DOI): see top of this page.

###### Abstract

Anticipating the motion of all humans in dynamic environments such as homes
and offices is critical to enable safe and effective robot navigation. Such
spaces remain challenging as humans do not follow strict rules of motion and
there are often multiple occluded entry points such as corners and doors that
create opportunities for sudden encounters. In this work, we present a
Transformer based architecture to predict human future trajectories in human-
centric environments from input features including human positions, head
orientations, and 3D skeletal keypoints from onboard in-the-wild sensory
information. The resulting model captures the inherent uncertainty for future
human trajectory prediction and achieves state-of-the-art performance on
common prediction benchmarks and a human tracking dataset captured from a
mobile robot adapted for the prediction task. Furthermore, we identify new
agents with limited historical data as a major contributor to error and
demonstrate the complementary nature of 3D skeletal poses in reducing
prediction error in such challenging scenarios.

Project page: https://human-scene-transformer.github.io/

###### Index Terms:

Autonomous Vehicle Navigation; Deep Learning Methods; Human-Aware Motion
Planning

##  I Introduction

The presence of robots sharing the environment with humans has led to a need
for effective methods to understand the human’s intention in order to avoid
collisions and ensure smooth interactions between humans and robots. A series
of steps are required for a robot to successfully navigate around humans in
dynamic scenes, namely perception, prediction and planning. Perception is
responsible for detecting the presence of humans and extracting other features
of the environment around the robot. Prediction aims to model how humans will
move into the future, and finally planning future actions towards a goal while
avoiding collisions. In this work we focus on the prediction step where the
inputs include perceived visual features and the outputs are predicted
trajectory distributions for motion planning.

While the trajectory prediction topic has been extensively studied in the
context of autonomous driving [1, 2, 3, 4], predicting human trajectories in
other environments where service robots could have a profound impact, such as
offices, homes, hospitals, and elderly care facilities, have received less
attention. In contrast to street scenes, the main dynamic agent in service
robotics environments are humans carrying out a wider range of tasks that
require diverse motions in unstructured environments, compared to the setting
of driving which has significant structure in place such as staying within
lanes or following the traffic rules. Additionally, the spatial environment is
generally smaller with a higher degree of perceptual obstructions (e.g. blind
corners or internal walls) resulting in a closer proximity upon first
observation due to occluded entry-points. Our goal is to enable more natural,
safe, smooth, and predictable navigation by anticipating where humans will be
moving in the near future using the robot’s onboard sensors.

Figure 1:  A service robot navigating a busy office space. To do so it
anticipates human motion using human-position and visual features such as head
orientation or 3D skeletal keypoints.

Several existing methods try to generalize vehicle trajectory prediction ideas
to human trajectory prediction by representing humans as 2D bounding boxes [5,
6, 7, 8, 9]. While bounding boxes can be sufficient for predicting vehicle
trajectories in outdoor environments [3, 10, 4, 1], the reduction of human
agents to bounding boxes neglects the plethora of perceptual information,
present in a human-centric scene – scenes with one or multiple humans sharing
limited space with the robot — such as a robot navigating in a hallway of a
busy office as in Fig. 1. In such settings, humans tend to take advantage of
information beyond each other’s position: There is substantial information
about people’s intent when _they turn their head_ or _look at where to walk
next_ — humans predict and anticipate each other’s intentions through vision-
based features: pose, gaze, and gestures. These visual cues are also apparent
for a human viewing a static image absent of temporal cues. Similarly, we will
demonstrate that a robot navigating in close proximity to humans as in Fig. 1
enables a more detailed model of the human beyond simple bounding boxes and
leads to more accurate prediction of human trajectories. Specifically, we
posit the research question: _“Can information from human visual features lead
to improved prediction accuracy?”_

We present the Human Scene Transformer (HST) which leverages different feature
streams: Historic positions of each human, vision-based features such as
skeletal keypoints (see Figure 1, joints of the human skeleton) or head
orientation when available. We specifically focus on demonstrating the
usefulness of noisy in-the-wild human skeletal information from a 3D human
pose estimator.

As such our contribution is threefold:

  * •

To the best of our knowledge, we are the first to adapt the trajectory
prediction task to the domain of human-centric navigation and demonstrate that
3D vision-based features improve prediction performance in a service robot
context notwithstanding imperfect in-the-wild data (without a dedicated motion
capture system). Specifically in the regime of limited historical data, which
is particularly under-explored, but a common scenario in indoor robot
navigation our key idea is to use 3D vision features as complimentary
predictive cues.

  * •

We present a prediction architecture, which flexibly processes and includes
detailed vision-based human features such as skeletal keypoints and head
orientation. To target crowded human-centric environments, we define HST
within a system of components from fields of Computer Vision, Machine
Learning, and Autonomous Driving to make use of real-world sensor data instead
of relying on ground truth labels. We demonstrate HST’s capability to
consistently model interactions which is critical in human-centric
environments.

  * •

We highlight a gap in prior work by showing the limitations of existing
datasets for human trajectory prediction in indoor navigation. We propose an
adaptation of existing datasets that can enable this new way of future
trajectory prediction. Using this adaptation, we demonstrate the feasibility
of our approach in human-centric environments. Simultaneously, we display
state-of-the art performance on a common outdoor pedestrian prediction
dataset.

##  II Related Work

Predicting the future trajectory of humans is a challenging task where prior
work has considered various motion models, scene context, and social
interaction. We will revisit three research fields which influence our
targeted domain of service robots and the use of vision-based human skeletal
(human pose) features. We will first outline current research in _Trajectory
Prediction_ , where the center position of an agent (which can be a human but
also a human driven vehicle) is forecasted over time. Subsequently, we will
introduce work in the field of _Human Pose Prediction_ , which uses a skeletal
representation of the human and predicts this pose over time. Finally, we
outline approaches in _Pose Estimation_ , where the estimated human pose is
directly used in trajectory prediction in application domains outside of
human-centric navigation.

Trajectory Prediction. From the early work of Pellegrini _et al_. [5] for
short-term future locations of humans for next frame data association to the
recent longer term multi-second prediction methods that are often used in
autonomous driving [4, 3, 2, 1, 11], trajectory prediction research has played
an important role in improving down-stream robotic tasks. Prior approaches
consider scene context [4], motion dynamics [4, 3], and the interaction
between agents [3]. Salzmann _et al_. [4] combine historic agent positions
with scene and dynamics constraints to make informed predictions. In an
autonomous driving context, extracting additional visual information about the
human actor (driver, cyclist, pedestrian) is challenging due to occlusion
(driver only partially visible in the car) or distance. Therefore, the
information representing each agent is reduced to a position estimate per
observation timestep [4, 3, 2, 1]. Unlike the large prediction range required
for self-driving [12], we focus on service-robot environments where people are
generally close enough to the robot to obtain a richer visual representation
of the human. As such, our work can benefit from recent approaches that fuse
LiDAR information adapting the human pose estimation problem to a robotic
sensor suit [13]. While prior works in trajectory prediction rely on
Generative Adversarial Networks (GANs) [14, 8] or Conditional Variational
Autoencoders (CVAEs) [7, 15, 4, 10, 16], this work follows the recent trend
towards Transformer architectures as they naturally lend themselves to the
set-to-set prediction problems such as multi-agent trajectory prediction and
are invariant to a varying number of agents. Specifically, we leverage the
fundamental idea of a Transformer based prediction framework [3, 1, 2]
inspired by Ngiam _et al_. [3]. Their Transformer architecture is used for
vehicle trajectory prediction in autonomous driving applications and captures
joint interactions between vehicles.

Pose Prediction. Another related area is human pose forecasting in 3D [17, 18,
19, 20, 21]. Corona _et al_. use scene context in the form of an influence
graph to refine a future trajectory of 3D human poses for a single subject in
a motion capture environment [17]. For multi-person pose prediction, [22]
extends DeTR [23] to predict multi-person 2D poses from a single image.
Towards social robot navigation, Narayanan _et al_. use a sequence of human
poses, also known as gait, to classify a person’s emotional state for setting
appropriate proximal distance constraints [24]. However, these approaches
commonly consider a single human motion relying on ground truth pose
information from a motion capture system, while we target multi-human in-the-
wild scenarios which are not limited to spaces with a motion capture system.

Pose Estimation in Trajectory Prediction. There have been prior efforts to
combine pose estimation with trajectory prediction, i.e., informing forecasted
trajectories by incorporating historic pose information. However, these works
are either limited to prediction in 2D image space [25, 26, 11] or operating
on motion capture datasets which do not exhibit diverse positional movement of
the human [17, 27]. Yagi _et al_. [25] showed that augmenting a convolutional
auto-encoder style model with scale and pose encoders reduces prediction error
compared to position only; however, their approach is applied to first-person
video using 2D pose detection and limiting the prediction to the 2D image
space. Similarly, Chen _et al_. [26] use a convolutional and recurrent
architecture to segment an image into heterogeneous traffic objects and body
parts, before using a Transformer decoder to attend to feature maps and
extract objects leading to improvements in 2D image space prediction. When
considering predicting multiple poses, Adeli _et al_. [28] use a form of graph
attention to capture dependencies between interacting agents but this work is,
again, limited to 2D first-person videos. Other works have explored attention
mechanisms between multiple human features in the image-view [11]. However,
for robotic navigation it is desired to obtain predictions for agents across
multiple sensors and ideally in a 3D or bird’s-eye metric space. In this work,
we follow these requirements by solely relying on onboard sensor information
of a robotic platform and predict in the metric global frame rather than in
image space.

##  III Human Scene Transformer

Our proposed method Human Scene Transformer (HST) follows the concept of
masked sequence to sequence prediction using an architecture with Transformer
blocks (see Figure 3 \- top right). This approach has shown promising vehicle
prediction results in the autonomous driving domain [3]. HST introduces
multiple important ideas extending the general Transformer architecture which
makes it suitable for trajectory prediction in human-centric environments.
These include the utilization of vision-based human features (Section III-A),
a feature attention mechanism to merge multiple potentially incomplete
features (Section III-B \- Input Embedding), an improved attention mechanism
facilitating a more complete information flow (Section III-B \- Full Self-
Attention), and a self-alignment layer which elegantly solves the problem of
discriminating between multiple masked agent timesteps while keeping
permutation equivariance (Section III-B \- Agent Self-Alignment). Notably,
implementing an attention based architecture end-to-end, the model is agnostic
to the number of humans per frame. This means the model can dynamically handle
a varying number of humans in different timesteps during inference. The
maximum number of jointly (single forward pass) predicted humans is only
limited by available memory.

###  III-A Model Inputs: Incorporating Vision-based Features

The robot’s observations for the last $H+1$ timesteps are processed as agent
features and scene context (Figure 3 blue box). The scene context can be an
occupancy grid or a raw point cloud at the current timestep, containing
information common to all nearby agents (e.g. static obstacles). Agent
features include the centroid position and vision-based features: skeletal
keypoints, and head orientation for each agent.

To extract these vision-based features from the raw data, image patches for
all agents are first obtained by projecting their detected 3D bounding boxes
into the 360 degree image using ex- and intrinsic camera calibrations (see
Figure 2-a). To extract skeletal key points from these patches, one could
choose from a plethora of off-the-shelf skeletal keypoints extractor for
images [29, 30, 31, 32, 33]. However, these extractors commonly output
keypoints in a 2D image coordinate frame. To produce 3D keypoints, we follow
the work of Grishchenko _et al_. [34] to estimate 3D keypoints from images
using a pre-trained model: As existing datasets commonly only include 2D
keypoint annotations, the 3D label required for supervised pre-training is
generated by fitting a parametric human shape model to available 2D keypoints
solving the following optimization problem:

$\small\text{argmin}_{\mathbf{k}}\left(\lVert
r(\mathbf{k})-\hat{\mathbf{k}}_{2}\rVert_{2}+\lambda p(\mathbf{k})\right),$
(1)

where $\mathbf{k}$ are the 3D skeletal keypoints, $\hat{\mathbf{k}}_{2}$ is
the 2D keypoints label, $r:\mathbb{R}^{33\times
3}\rightarrow\mathbb{R}^{33\times 2}$ is the re-projection function projecting
3D points into the 2D image space using the camera calibrations. Many
representations (from $<20$ [35] to $>500$ [36] keypoints) are present in the
literature; we settled for a 33 keypoints skeleton representation [34, 37] as
it presents itself as a minimal representation capturing _both_ head
information and limb articulation. The learned prior distribution over human
pose configuration $p(\mathbf{k})$ penalizes infeasible poses which can arise
in optimization for the underdetermined 3D-2D-projection problem, as multiple
3D poses can result in equivalent 2D projections. A infeasible pose would be a
configuration of human joints which is physically infeasible for a human to
achieve (e.g. head rotated full 180∘). The prior distribution is learned by
fitting a variational autoencoder to a dataset of feasible 3D human poses. The
3D keypoints in the camera space are transformed to the global coordinate
frame using extrinsic camera calibration. Given the skeletal keypoints
$\mathbf{k}$ from optimizing Equation 1, we can easily extract the head
orientation as the vector from in-between ears to in-between eyes. Note that
vision-based agent features may not be available for all agents at all
timesteps. This can be due to the agent being too far away to reliably extract
keypoints (e.g., the picture being too small). We will show in Section III-B
\- Input Embedding how we can leverage inherent properties of our HST
Transformer architecture to deal with these situations.

Figure 2: Process of estimating three dimensional vision-based features of the
human. (Left) Image of the human cropped based on the bounding box detection.
Lighting conditions can be sub-optimal in an human-centric environment.
(Middle) Inferred three dimensional pose from the trained pose estimator.
(Right) Head orientation post-processed from the pose keypoints.

Figure 3: Overview of the HST architecture. From the robot’s sensors we
extract the scene context, the historic position tracks of each agent, and
vision based skeletal keypoints/head orientation when feasible. All features
are encoded individually before the agent features are combined via cross-
attention (XA) using a learned query tensor. The resulting agent-timestep-
tokens is passed to our Agent Self-Alignment layer which enables the use of
subsequent full self-attention (FSA) layers. Embedded scene context is
attended to via cross-attention (XA). After multimodality is induced and
further FSA layers the model outputs the parameters of a Normal distribution
for each agent at each prediction timestep. We can represent the full output
structure as a Gaussian Mixture Model (formula in bottom right) over all
possible futures where the mixture coefficients $w$ come from the
Multimodality Induction. Both cross-attention (XA) and full self-attention
layers use the Transformer layer (top right) with different input
configurations for Query (Q), Key (K), and Value (V).

###  III-B Model Architecture

Figure 3 outlines the HST architecture. We will explain the individual
components in this section by first introducing the Transformer layer as core
concept and subsequently following the data flow depicted in Figure 3.

Transformer Layer. The primary building block of the model’s architecture is
the Transformer layer (shown in Figure 3 top right), which itself is comprised
of a Multi-Head Attention layer [38] and multiple dense and normalization
layers. The Transformer layer receives three tensors as input: Query (Q), Key
(K), and Value (V). However, a single tensor may be used for multiple of the
inputs. Consequently, we define a self-attention (SA) operation as a
Transformer Layer where inputs Q, K, and V are the same tensor: The tensor
attends to itself and conveys its information along one or more dimensions.
Similarly, we define cross-attention (XA) as a Transformer Layer where the Q
input is distinct from the K/V inputs. Intuitively the query attends to
additional information from a different tensor as means of merging multiple
streams of information. For a comprehensive explanation on the Attention
mechanism and its inputs we refer the reader to Vaswani _et al_. [38].

Input Embedding. The input agent features (blue) are tensors of shape
$[N,T,d]$, where $d=2$ for the x-y centroid position, $d=99$ for the x-y-z
position of 33 skeletal keypoints, and $d=1$ for the head orientation. These
tensors contain information of all $N$ nearby agents for all $H+1$ historic
and current input timesteps. If an agent’s feature is not observed at specific
timesteps, we mask those timesteps with $0$. As depicted in Figure 4, we also
mask all future $F$ timesteps for all agents by setting their feature value to
$0$, thus making only historical and current information available to the
model. This masking approach is a well known technique in missing-data
problems such as future prediction using Transformer based architectures [38,
3, 1]. Masking exploits the inductive bias inherent in the prediction problem,
which allows for the filling of the missing information using available
context in the vicinity of the gaps. As such, our approach allows for missing
keypoints in frames due to bad lighting or other influences as the Transformer
effectively “fills” in for the missing information.

The agent features are encoded independently and are combined by a learned
attention query. This masked attention mechanism offers scalability to systems
that have a large number of features with limited availability. The combined
agent features result in a latent tensor for each agent and timestep of shape
$[N,T,h]$ where $T=H+1+F$ and $h$ is the size of the token dimension. For
simplicity we will refer to such tensors of latent representations throughout
the network as _agent-timestep-tokens_.

Figure 4: Structure and masking of embedded tensors. Future agent-timestep-
tokens are masked and subsequently filled by the Transformer structure,
iteratively with updated latent representations and finally with position
distribution information on the output level.

Full Self-Attention Via Agent Self-Alignment. In contrast to previous methods
[3] which alternately attends to agents and time dimensions separately via
factorized attention, we propose a _full self-attention_ (FSA) operation where
each agent-timestep-token attends to all agent-timestep-tokens along _both_
agent and time dimensions. This provides a more direct path of information
flow. For example in social interactions, a change in action such as
adjustment in walking direction does not have an immediate influence on other
humans in proximity but rather influences their future. Following this
illustration, an agent at a given timestep in our Transformer architecture
should be able to attend not just to other agents at the current timestep
(factorized attention) but to _all agents_ at _all timesteps_ (full self-
attention).

Naïvely using full-self-attention can lead to sub-optimal outcomes. Since all
future agent-timestep-tokens are masked out, two agents with the same masked
future agent-timestep-tokens will also have the same input (Query)
representation to the Transformer layer (Figure 3 top right). This prevents
the model from associating future timesteps of an agent with the agent’s
history, since all future agents’ timesteps “look” the same (masked). The
problem could be addressed by enforcing an innate order on perceived agents,
where all agents are enumerated. This, however, would eliminate the
permutation invariant set-to-set prediction capabilities [39]; one of the core
strengths of Transformers: An agent’s future would be predicted differently
based on its enumeration embedding with the same historic features.

Instead, we solve this problem and achieve full self-attention via a simple
approach that we refer to as _agent self-alignment_. The agent-timestep-tokens
resulting from the feature combination are cross-attended with a learned query
tensor only in the time dimension. This query, a weight matrix jointly
optimized with all other network weights during training, learns to propagate
available historic information for each agent to future timesteps, enabling
the model to align future masked timesteps of an agent with historic ones
during full self-attention without an explicit enumeration embedding. After
this process, which is visualized in the dark green box in Figure 3, the
previously masked future agent-timestep-tokens hold information from the
respective agent’s history; differentiating them from another. As such, the
agent self-alignment mechanism preserves agents’ permutation equivariance and
enables full self-attention without restricting information flow along
matching timesteps [3] or utilizing special attention matrices which
explicitly separates agents [1]. The output agent-timestep-tokens of the agent
self-alignment then passes through $K$ transformer layers with full self-
attention across agent and time dimensions before cross-attending to the
encoded scene features.

Multimodality Induction. Our architecture can predict multiple consistent
futures (modes) for a scene. To do so, the Multimodality Induction module
repeats the agent-timestep-tokens by the number of future modes ($M$),
resulting in a tensor of shape $[N,T,M,h]$. To discriminate between modes it
is combined with a learned _mode-identifier_ tensor of shape $[1,1,M,h]$. Each
future’s logit probability $w_{m};\;m\in{1,\dots,M}$ is also inferred here by
having the _mode-identifier_ attend to the repeated input; resulting in shape
$[1,1,M,h]$ which is subsequently reduced by a MLP to output $w_{m}$ as
$[1,1,M,1]$.

Prediction Head. The agent-timestep-tokens updated with the learned mode-
identifier go through $L$ Transformer layers, again with full self-attention,
before predicting per mode parameters $\mu,\sigma$ using a dense layer as
_prediction head_.

###  III-C Producing Multimodal Trajectory Distributions

Combining $\mu$ and $\sigma$ with the mode likelihoods $w_{m}$ coming from the
Multimodality Induction, the Human Scene Transformer models the distribution
of the $i$-th agent’s centroid position at each timestep $t$ with a 2D
Gaussian Mixture Model (GMM):

$\small
P^{i}_{\theta}(\mathbf{x}_{t}|O(t),...,O(t-H))=\sum_{m=1}^{M}w_{m}\mathcal{N}(\mathbf{x};\mathbf{\sigma}_{m,i,t},\mathbf{\mu}_{m,i,t}),$
(2)

where $m$ is the $m$-th future mode. Here, we represent the position of an
agent at a specific timestep by a GMM with mixture weights $w$ equal to the
probability distribution of future modes.

We adopt a joint future loss function, that is, the cumulative negative log-
likelihood of the Gaussian mode ($m^{*}$) with the smallest mean negative log-
likelihood:

$\small\mathcal{L}_{\text{minNLL}}=\sum_{i,t}-\text{log}(\mathcal{N}(\mathbf{x}^{*}_{i,t};\mathbf{\sigma}_{m^{*},i,t},\mathbf{\mu}_{m^{*},i,t})),$
(3)

where

$\small
m^{*}=\text{argmin}_{m}(\sum_{i,t}-\text{log}(\mathcal{N}(\mathbf{x}^{*}_{i,t};\mathbf{\sigma}_{m,i,t},\mathbf{\mu}_{m,i,t}))),$
(4)

and $\mathbf{x}^{*}_{i,t}$ is the ground truth agent position. The resulting
prediction represents $M$ possible future realizations of all agents at once
in a consistent manner, where the mode mixture weights $w$ are shared by all
agents in the scene. The most likely future mode during inference is given as

$\small m^{+}=\text{argmax}_{m}(w_{m}).$ (5)

##  IV Experiments

We structure our experiments to support our contributions: First, we
qualitatively and quantitatively demonstrate how our architecture provides
accurate predictions for the human-centric service robot domain. We especially
demonstrate how HST can leverage and model interactions between humans
consistently over multiple possible futures. Further, we show that our
approach is cross-domain compatible with unconstrained outdoor pedestrian
prediction, where data from a surveillance camera is used to predict
pedestrians in different outdoor squares. Finally, we demonstrate how HST can
leverage vision-based features in human-centric environments to improve
prediction accuracy, specifically in short history situations where prediction
errors are high.

Datasets. To effectively investigate the performance of HST in human-centric
environments and the possible benefits that a detailed 3D skeletal
representation of the human body can have, a dataset should (I) include a
diverse range of indoor and outdoor environments, (II) capture humans’
movement from a robot viewpoint (III) in a natural unscripted environment,
(IV) provide labels for the position and skeletal keypoints ($\mathbf{k}$) of
all agents at all timesteps, and (V) be sufficiently large to prevent over-
fitting.

TABLE I: A summary of different prediction datasets indicating our desiderata addressed by each dataset. Parentheses indicate that a desiderata is only partially fulfilled. | I Diverse | II Robot | III Unscripted |  IV $\mathbf{k}$ | V Large
---|---|---|---|---|---
ETH [5] & UCY [6] | ✗ | ✗ | ✓ | (✗) | ✗
AD [40, 41, 42, 43] | ✗ | ✓ | ✓ | (✗) | ✓
Motion Capture [44, 45] | ✗ | ✗ | ✗ | (✓) | (✗)
3DPW [46] | ✗ | ✗ | ✗ | ✓ | ✗
Adapted JRDB | ✓ | ✓ | ✓ | (✓) | ✗

The evaluation of different datasets for human trajectory prediction in Table
I takes into account the satisfaction of these requirements. Many of existing
datasets are collected from a single top-down camera in a limited number of
environments, such as the ETH [5] and UCY [6] pedestrian datasets. Others are
specific to the autonomous driving domain [40, 41, 42, 43]. While none of
these datasets provide labels for skeleton keypoints, other datasets such as
H3.6.M [44], AMASS [45], and 3DPW [46] which are collected using a motion
capture system or wearable IMU devices [46], do offer such labels. However,
these datasets are limited to artificial environments and often feature
stationary or scripted motions. Finally, while all these datasets provide
labels for position, these labels are often hand labeled ground truth not
representing noisy input data a robot would experience in the real-world
during inference.

One dataset which is recorded in diverse human-centric environments using
sensors (2 x 16 Channel LIDAR, 5 x Stereo RGB Cameras) on a mobile robotic
platform is the JackRabbot Dataset and Benchmark (JRDB) [47]. However, JRDB
was created as a detection and tracking dataset rather than a prediction
dataset. To make the data suitable for a prediction task, we first extract the
robot motion from the raw sensor data to account for the robot’s motion.
Tracks are generated for both train and test split using the JRMOT [48]
detector and tracker. The ground truth labeled bounding-boxes on the train set
were disregarded as they were exposed to filtering during the labeling process
to the point where the smoothness eases the prediction task. We were able to
increase the number of human tracks for training by associating the JRMOT
detections to ground truth track labels via Hungarian matching, while on the
test split we solely use JRMOT predictions.

Due to factors such as distance, lighting and occlusion the pre-trained 3D
pose estimator model (Section III) is not guaranteed to produce keypoints for
all agents at all timesteps. We observed human keypoints information in $\sim
50\%$ of all timesteps for all agents in a distance of up to 7 meters from the
robot. The available data is split using $50\%$ of each scene as training
data, $20\%$ as validation data and $30\%$ as test data. We sub-sample the
dataset from $15$\mathrm{H}\mathrm{z}$$ to $3$\mathrm{H}\mathrm{z}$$ keeping
all of the intermediate samples and therefore increasing the number of
datapoints by a factor of five.

In addition, we also compare our model to the ETH [5] and UCY [6] datasets.
These are standard benchmarks for pedestrian trajectory prediction and enable
a fair comparison of our architecture against other methods.

Metrics. In consistency with prior work [4, 1, 8, 7], prediction quality is
evaluated using the following metrics:

  1. 1.

Minimum Average Displacement Error (minADE): Minimum Mean $l_{2}$ distance
between the ground truth and all $M$ future mode trajectories.

  2. 2.

Minimum Final Displacement Error (minFDE): Minimum $l_{2}$ distance between
the final positions of the ground truth and all $M$ future mode trajectories.

  3. 3.

Maximum Likelihood Average Displacement Error (MLADE): Mean $l_{2}$ distance
between the ground truth and the most likely mode trajectory.

  4. 4.

Negative Log Likelihood (NLL) of the ground truth under the full parametric
output distribution.

Lower is better for all metrics.

Baselines. We re-implement the autonomous driving Scene Transformer
architecture [3], where we match the number of Transformer layers to our
architecture. We further compare to trajectory prediction architectures
Trajectron++ [4], Agentformer [1], SoPhie [8], and PECNet [7].

Evaluation Protocol. For the JRDB prediction dataset we report minADE, MLADE,
and NLL for 128k scene snippets from the test split including partially
occluded agents. We use up to $2\text{\,}\mathrm{s}$ of history as input and
predict the next $4\text{\,}\mathrm{s}$ future of the scene. Note that if the
number of agents exceeds $N_{\text{max}}=16$, we randomly select one agent and
only consider and predict it and the $N_{\text{max}}-1$ closest agents. On the
ETH and UCY dataset we follow the standard procedure to train in a leave-one-
out fashion and evaluate minADE and minFDE on $20$ trajectories over a
prediction horizon of 12 timesteps ($4.8$\mathrm{s}$$) using 8 historic
timesteps as input. We match the evaluation protocol of AgentFormer [1] by
setting the number of modes to $M=20$.

(a)

(b)

(c)

Figure 6: Consistently modeled interactions in different predicted futures for
a single scene in the x-y-plane. Two humans approaching each other head on.
(a) History (solid) and ground truth future (dashed) of both humans. (b) Two
of the $M$ predicted futures (dots - increasing transparency with time) of the
scene by HST. Within each mode the influence and reaction of both agents is
consistent and reasonable. The humans’ futures are predicted without
collisions giving each other space to navigate within the specific predicted
future mode of the scene. (c) Two predicted futures of a crowded scene.

###  IV-A Trajectory Prediction in Human-centric Environments

In Table II and Figure 6 we show quantitative and qualitative results of HST’s
predictions in the human-centric environment. We show that in crowded human-
centric environments the influence of interaction between humans has large
benefits on the prediction accuracy of each individual. To show this, we
compare against a version of our model which is trained to predict a single
human at a time ignoring interactions with other agents. Subsequently, adding
our full self-attention via self-alignment mechanism additionally increases
the model’s ability to capture interactions across time, leading to
improvements across all metrics. The capability to account for interactions
between humans is qualitatively demonstrated in Figure 6 where we show
multiple predicted futures for a scene of two interacting humans. The two
humans approach each other head on. The possible interactions to avoid
collisions are modeled _consistently_ within each future.

TABLE II: Comparison against Scene Transformer on JRDB prediction dataset. HST outperforms the original Scene Transformer on all metrics. Ablation shows that the interaction attention to other agents improves performance by comparing to a model predicting a single human at time. We also show the positive impact of Full Self-Attention. Model Configuration | minADE | MLADE | NLL
---|---|---|---
Scene Transformer [3] | $0.53$ | $0.86$ | $0.25$
| Full Self-Attention | Interaction Attention |  |  |
HST | ✗ | ✗ | $0.57$ | $0.93$ | $0.89$
HST | ✗ | ✓ | $0.50$ | $0.84$ | $-0.02$
HST | ✓ | ✓ | $\mathbf{0.48}$ | $\mathbf{0.80}$ | $\mathbf{-0.13}$

###  IV-B Vision-based Features

In this section, we consider the adversarial setting, where the robot
encounters a human unexpectedly, i.e., the robot observes a new human with
little historical observations. Prediction architectures solely relying on
historic position information struggle in scenarios where no or only a limited
amount of history of the human position is available to the model.
Specifically, at the first instance of human detection, the experimental error
is 200% higher compared to full historic information over
$2\text{\,}\mathrm{s}$. Given the specifics of our targeted human-centric
environment, where we are mostly interested in humans close to the robot, we
are likely able to extract vision-based features for the human in addition to
the position. Specifically, we revisit our research question: _“Can
information from human visual features lead to improved prediction accuracy?”_

(a) First detection of person entering the scene.

(b) Prediction with keypoints.

(c) Prediction without keypoints.

Figure 7: A visualization of the predicted trajectory distributions for a new
human agent entering the scene through the door on the right as viewed in (a).
For _both_ (b) and (c) the HST model does not have any historic information
here and only has access to the current frame. The plot of future trajectory
distributions in (b) and (c) show the effect of using and not using skeletal
keypoints (respectively) as input in that single frame. Without pose keypoints
the HST model predicts the agent to be most-likely stationary while, with
keypoints as input, it can reason that the human is moving and correctly
anticipates the direction. Blue dot is the detected human at the initial
frame, orange dots are our most likely mode predictions with corresponding
distribution shown with blue shading, and green dots are the ground truth
human future (actually executed trajectory by the human).

Before answering this question quantitatively we show a clarifying visual
example in Figure 7 where a human just entered the scene through a door and is
first detected. When solely relying on historic position information the most
likely prediction by the model is a stationary agent. However, when we employ
the pre-trained skeleton keypoints estimator to provide pose keypoints as
additional input to our model the model correctly realizes if the human is in
a walking motion and how the human is oriented, accurately predicting the most
likely future trajectory.

Quantitatively, during evaluation, when keypoints are available on the first
detection we observe a substantial prediction improvement of up to 11% (Figure
8). When additional timesteps with position information are available the
improvement using keypoints vs not using keypoints averages between 5% and
10%. The relative improvement generally increases with the number of timesteps
with keypoints in the history and decreases with the number of historic
position information.

Figure 8: Impact of vision-based features conditioned on different number of
consecutive non occluded input timesteps.

Finally, we want to provide an outlook of how much vision-based features can
improve prediction performance if available for all agents at all timesteps.
We therefore enforce feature parity between position and skeletal keypoints
features by disregarding position information without associated keypoints
(Table III). We find that a relative improvement of around 10% is achievable
using our in-the-wild vision-based features. The baseline is naturally worse
than in Table II as we partially disregard historic position information.

TABLE III: Vision-based features relatively improve prediction across all metrics on a prediction horizon of $4\text{\,}\mathrm{s}$. To create feature parity between position and keypoints features we ignore all position history without detected skeletal keypoints during training and evaluation. | minADE | NLL |  minADE @ $2\text{\,}\mathrm{s}$ |  minADE @ $4\text{\,}\mathrm{s}$
---|---|---|---|---
Baseline | $0.56$ | $0\%$ | $1.04$ | $0\%$ | $0.46$ | $0\%$ | $0.87$ | $0\%$
Head Orient. | $0.53$ | $-5\%$ | $0.89$ | $-14\%$ | $0.42$ | $-9\%$ | $0.79$ | $-9\%$
Keypoints | $\mathbf{0.51}$ | $-9\%$ | $\mathbf{0.85}$ | $-18\%$ | $\mathbf{0.40}$ | $-13\%$ | $\mathbf{0.77}$ | $-11\%$

###  IV-C Pedestrian Dataset

In addition to showing HST’s capabilities in a robotics specific environment
we will further validate our architecture against a range of state-of-the-art
prediction methods. For this we use ETH/UCY which is prolific in the
trajectory prediction community while also being the dataset which we think is
the closest to the human-centric environments that we would like to explore:
On the ETH and UCY dataset, we either improve current state-of-the-art methods
or we are on par with them on 4 out of the 5 scenes.

TABLE IV: Results on ETH and UCY dataset. Our method is the best on four of five subsets. Method |  minADE20 / minFDE20
---|---
ETH |  Hotel |  Univ |  Zara1 |  Zara2 |  Average
SoPhie [8] |  $0.70\;/\;1.43$ |  $0.76\;/\;1.67$ |  $0.54\;/\;1.24$ |  $0.30\;/\;0.63$ |  $0.38\;/\;0.78$ |  $0.54\;/\;1.15$
PECNet [7] |  $0.54\;/\;0.87$ |  $0.18\;/\;0.24$ |  $0.35\;/\;0.60$ |  $0.22\;/\;0.39$ |  $0.17\;/\;0.30$ |  $0.29\;/\;0.48$
Trajectron++ [4] |  $0.43\;/\;0.86$ |  $0.12\;/\;0.19$ |  $\mathbf{0.22}\;/\;0.44$ |  $\mathbf{0.17}\;/\;0.32$ |  $\mathbf{0.11}\;/\;0.25$ |  $\mathbf{0.21}\;/\;0.41$
AgentFormer [1]1 |  $0.45\;/\;0.75$ |  $0.14\;/\;0.22$ |  $0.25\;/\;0.45$ |  $0.18\;/\;0.30$ |  $0.14\;/\;\mathbf{0.24}$ |  $0.23\;/\;0.39$
Scene Transformer [3] |  $0.50\;/\;0.76$ |  $0.14\;/\;0.20$ |  $0.29\;/\;\mathbf{0.42}$ |  $0.22\;/\;0.36$ |  $0.16\;/\;0.27$ |  $0.31\;/\;0.40$
HST |  $\mathbf{0.41}\;/\;\mathbf{0.73}$ |  $\mathbf{0.10}\;/\;\mathbf{0.14}$ |  $0.24\;/\;0.44$ |  $\mathbf{0.17}\;/\;\mathbf{0.30}$ |  $0.14\;/\;\mathbf{0.24}$ |  $\mathbf{0.21}\;/\;\mathbf{0.37}$

##  V Discussion

Simply representing a human as its spatial position, as commonly done in
autonomous driving environments, does deliver a baseline prediction
performance in human-centric service robot environments. However, it suffers
in challenging settings where the history of a human is limited. Specifically
in these situations we demonstrate how the Human Scene Transformer can
leverage vision-based features to improve prediction accuracy (Figure 7).
Beyond scenarios such as when robot and humans encounter each other in blind
corners, general improvement trends using in-the-wild skeletal pose detections
were also observed with more observations as shown in Figure 8. Another
intuitive assumption which we can support quantitatively in Table III is that
the full skeletal keypoints (full human pose) hold additional information over
just the head orientation (where is the human looking). This is expected as
the head orientation can give away a possible direction for the trajectory,
while the full keypoints can be more informative about the speed the human is
approaching, e.g., running or slowly walking.

Figure 6 and Table II demonstrate HST’s capability to model consistent
interactions between agents and use these influences to improve the overall
prediction substantially. This is especially useful for the crowded spaces a
service robot navigates and opens opportunities not just modeling human-to-
human interactions but robot-human interactions.

Limitations. Exploring this new domain of human-centric environments we
recognize that numerous limitations exist: We showed that an on par
relationship between positions and keypoint features leads to relative
improvements and expect the performance to be similarly correlated to the
quality of the detected 3D keypoints. We therefore think that an 3D skeletal
keypoints estimator, specifically designed for robotic applications,
increasing both the number of successful detections and the quality of each
detected skeleton would improve performance for the prediction task as well as
other robot tasks in close human contact (e.g. handover). Such an estimator
could make use of the full sensor suite of a robotic platform, fusing camera
and LiDAR information. We are happy to see first works [13] in this direction
and hope that our findings encourage the research community to pursue this
path within the domain of human-centric environments.

In Section III we presented a way to include scene context via a point cloud
but were unable to see any predictive benefit from this information. We
attribute this to three factors: the small size of the adapted JRDB dataset,
the limited number of locations (29) in which the data was recorded, and the
raw point lack semantic labels, e.g. a door is indistinguishable from a wall.
These factors highlight the potential for further work towards better
representations of point clouds.

11footnotetext: We report publicly available updated numbers for AgentFormer
(arxiv/github) which differ from the original publication.

Conclusion. This work introduced the task of human trajectory prediction into
the domain of human-centric service robots. We demonstrated that the proximity
of robot and humans in such environments can be leveraged to improve
prediction performance by explicitly incorporating vision-based human
features. We showed HST achieves strong prediction improvements using in-the-
wild 3D pose representations for the critical situation of agents being first
detected in close proximity to the robot. A quantitative ablation using paired
position and vision-based features highlighted the influence of different
visual features with skeletal keypoints providing the highest gains across all
metrics. Our Transformer based architecture is flexible in accommodating
different feature inputs while also achieving state-of-the-art results on a
common pedestrian prediction dataset without visual features outside the
domain of human-centric service robot environments. We think that the
unstructured and uncompressed nature of environment point-clouds fits nicely
with the permutation invariance property of Transformer architectures and are
therefore excited to further explore this direction in the future. We hope
that our findings will inspire further research in the human-centric domain
and in developing improved methods for generating accurate 3D vision-based
human representation for service robotics applications.

## References

  * [1] Ye Yuan, Xinshuo Weng, Yanglan Ou and Kris M Kitani “Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting” In _ICCV_ , 2021, pp. 9813–9823
  * [2] Nigamaa Nayakanti et al. “Wayformer: Motion forecasting via simple & efficient attention networks” In _arXiv:2207.05844_ , 2022
  * [3] Jiquan Ngiam et al. “Scene Transformer: A unified architecture for predicting future trajectories of multiple agents” In _ICLR_ OpenReview.net, 2022 URL: https://openreview.net/forum?id=Wm3EA5OlHsG
  * [4] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty and Marco Pavone “Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data” In _ECCV_ , 2020, pp. 683–700 Springer
  * [5] Stefano Pellegrini, Andreas Ess, Konrad Schindler and Luc Van Gool “You’ll never walk alone: Modeling social behavior for multi-target tracking” In _ICCV_ , 2009, pp. 261–268 IEEE
  * [6] Alon Lerner, Yiorgos Chrysanthou and Dani Lischinski “Crowds by example” In _Computer graphics forum_ 26, 2007, pp. 655–664 Wiley Online Library
  * [7] Karttikeya Mangalam et al. “It is not the journey but the destination: Endpoint conditioned trajectory prediction” In _ECCV_ , 2020, pp. 759–776 Springer
  * [8] Amir Sadeghian et al. “Sophie: An attentive gan for predicting paths compliant to social and physical constraints” In _CVPR_ , 2019, pp. 1349–1358
  * [9] Karttikeya Mangalam, Yang An, Harshayu Girase and Jitendra Malik “From goals, waypoints & paths to long term human trajectory forecasting” In _ICCV_ , 2021, pp. 15233–15242
  * [10] Boris Ivanovic and Marco Pavone “The trajectron: Probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs” In _ICCV_ , 2019, pp. 2375–2384
  * [11] Phillip Czech, Markus Braun, Ulrich Kreßel and Bin Yang “On-Board Pedestrian Trajectory Prediction Using Behavioral Features” In _arXiv:2210.11999_ , 2022
  * [12] Pei Sun et al. “Rsn: Range sparse net for efficient, accurate lidar 3d object detection” In _CVPR_ , 2021, pp. 5725–5734
  * [13] Andrei Zanfir et al. “HUM3DIL: Semi-supervised Multi-modal 3D Human Pose Estimation for Autonomous Driving” In _arXiv:2212.07729_ , 2022
  * [14] Agrim Gupta et al. “Social gan: Socially acceptable trajectories with generative adversarial networks” In _CVPR_ , 2018, pp. 2255–2264
  * [15] Boris Ivanovic, Karen Leung, Edward Schmerling and Marco Pavone “Multimodal deep generative models for trajectory prediction: A conditional variational autoencoder approach” In _RA-L_ 6.2 IEEE, 2020, pp. 295–302
  * [16] Boris Ivanovic, Edward Schmerling, Karen Leung and Marco Pavone “Generative modeling of multimodal multi-human behavior” In _IROS_ , 2018, pp. 3088–3095 IEEE
  * [17] Enric Corona, Albert Pumarola, Guillem Alenya and Francesc Moreno-Noguer “Context-aware human motion prediction” In _CVPR_ , 2020, pp. 6992–7001
  * [18] Ye Yuan and Kris Kitani “Dlow: Diversifying latent flows for diverse human motion prediction” In _ECCV_ , 2020, pp. 346–364 Springer
  * [19] Yan Zhang, Michael J Black and Siyu Tang “We are more than our joints: Predicting how 3d bodies move” In _CVPR_ , 2021, pp. 3372–3382
  * [20] Wei Mao, Miaomiao Liu and Mathieu Salzmann “History repeats itself: Human motion prediction via motion attention” In _ECCV_ , 2020, pp. 474–489 Springer
  * [21] Tim Salzmann, Marco Pavone and Markus Ryll “Motron: Multimodal Probabilistic Human Motion Forecasting” In _CVPR_ , 2022, pp. 6457–6466
  * [22] Edward Vendrow, Satyajit Kumar, Ehsan Adeli and Hamid Rezatofighi “SoMoFormer: Multi-Person Pose Forecasting with Transformers” In _arXiv:2208.14023_ , 2022
  * [23] Nicolas Carion et al. “End-to-end object detection with transformers” In _ECCV_ , 2020, pp. 213–229 Springer
  * [24] Venkatraman Narayanan et al. “Proxemo: Gait-based emotion learning and multi-view proxemic fusion for socially-aware robot navigation” In _IROS_ , 2020, pp. 8200–8207 IEEE
  * [25] Takuma Yagi, Karttikeya Mangalam, Ryo Yonetani and Yoichi Sato “Future person localization in first-person videos” In _CVPR_ , 2018, pp. 7593–7602
  * [26] Kai Chen, Xiao Song and Xiaoxiang Ren “Pedestrian Trajectory Prediction in Heterogeneous Traffic Using Pose Keypoints-Based Convolutional Encoder-Decoder Network” In _TCSVT_ 31.5 IEEE, 2020, pp. 1764–1775
  * [27] Mohammad Mahdavian, Payam Nikdel, Mahdi TaherAhmadi and Mo Chen “STPOTR: Simultaneous Human Trajectory and Pose Prediction Using a Non-Autoregressive Transformer for Robot Following Ahead” In _arXiv:2209.07600_ , 2022
  * [28] Vida Adeli et al. “Tripod: Human trajectory and pose dynamics forecasting in the wild” In _ICCV_ , 2021, pp. 13390–13400
  * [29] Z. Cao et al. “OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields” In _PAMI_ , 2019
  * [30] Hao-Shu Fang et al. “Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time” In _PAMI_ IEEE, 2022
  * [31] Debapriya Maji, Soyeb Nagori, Manu Mathew and Deepak Poddar “YOLO-Pose: Enhancing YOLO for Multi Person Pose Estimation Using Object Keypoint Similarity Loss” In _CVPR_ , 2022, pp. 2637–2646
  * [32] tensorflow.org “MoveNet: Ultra fast and accurate pose detection model.” In _TensorFlow_ Google, 2022 URL: https://www.tensorflow.org/hub/tutorials/movenet
  * [33] tensorflow.org “Real-time human pose estimation in the browser with tensorflow.js” In _The TensorFlow Blog_ Google, 2018 URL: https://blog.tensorflow.org/2018/05/real-time-human-pose-estimation-in.html
  * [34] Ivan Grishchenko et al. “BlazePose GHUM Holistic: Real-time 3D Human Landmarks and Pose Estimation” In _Computer Vision for AR/VR_ , 2022
  * [35] Tsung-Yi Lin et al. “Microsoft coco: Common objects in context” In _Computer Vision–ECCV 2014: 13th European Conf., Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , 2014, pp. 740–755 Springer
  * [36] Ivan Grishchenko and Valentin Bazarevsky “MediaPipe Holistic — Simultaneous Face, Hand and Pose Prediction, on Device” In _Google AI Blog_ Google, 2018 URL: hhttps://ai.googleblog.com/2020/12/mediapipe-holistic-simultaneous-face.html
  * [37] Hongyi Xu et al. “Ghum & ghuml: Generative 3d human shape and articulated pose models” In _CVPR_ , 2020, pp. 6184–6193
  * [38] Ashish Vaswani et al. “Attention is all you need” In _NeurIPS_ 30, 2017
  * [39] Juho Lee et al. “Set transformer: A framework for attention-based permutation-invariant neural networks” In _Int. conference on machine learning_ , 2019, pp. 3744–3753 PMLR
  * [40] Scott Ettinger et al. “Large Scale Interactive Motion Forecasting for Autonomous Driving: The Waymo Open Motion Dataset” In _ICCV_ , 2021, pp. 9710–9719
  * [41] Holger Caesar et al. “nuscenes: A multimodal dataset for autonomous driving” In _CVPR_ , 2020, pp. 11621–11631
  * [42] Benjamin Wilson et al. “Argoverse 2: Next Generation Datasets for Self-driving Perception and Forecasting” In _NeurIPS Datasets and Benchmarks_ , 2021
  * [43] John Houston et al. “One thousand and one hours: Self-driving motion prediction dataset” In _Conf. on Robot Learning_ , 2021, pp. 409–418 PMLR
  * [44] Catalin Ionescu, Dragos Papava, Vlad Olaru and Cristian Sminchisescu “Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments” In _PAMI_ 36.7 IEEE Computer Society, 2014, pp. 1325–1339
  * [45] Naureen Mahmood et al. “AMASS: Archive of Motion Capture as Surface Shapes” In _ICCV_ , 2019, pp. 5442–5451
  * [46] Timo Marcard et al. “Recovering Accurate 3D Human Pose in The Wild Using IMUs and a Moving Camera” In _ECCV_ , 2018
  * [47] Roberto Martin-Martin et al. “Jrdb: A dataset and benchmark of egocentric robot visual perception of humans in built environments” In _PAMI_ IEEE, 2021
  * [48] Abhijeet Shenoi et al. “Jrmot: A real-time 3d multi-object tracker and a new large-scale dataset” In _IROS_ , 2020, pp. 10335–10342 IEEE


Performance Evaluation of Selective Fixed-filter Active Noise Control based on
Different Convolutional Neural Networks

Zhengding<EMAIL_ADDRESS>Dongyuan<EMAIL_ADDRESS>Woon-Seng<EMAIL_ADDRESS>
School of Electrical and Electronic Engineering,

Nanyang Technological University, Singapore.

Libin<EMAIL_ADDRESS>
Huawei Technologies Co., Ltd.

Qirui<EMAIL_ADDRESS>
Huawei International Pte. Ltd.

ABSTRACT

Due to its rapid response time and a high degree of robustness, the selective
fixed-filter active noise control (SFANC) method appears to be a viable
candidate for widespread use in a variety of practical active noise control
(ANC) systems. In comparison to conventional fixed-filter ANC methods, SFANC
can select the pre-trained control filters for different types of noise. Deep
learning technologies, thus, can be used in SFANC methods to enable a more
flexible selection of the most appropriate control filters for attenuating
various noises. Furthermore, with the assistance of a deep neural network, the
selecting strategy can be learned automatically from noise data rather than
through trial and error, which significantly simplifies and improves the
practicability of ANC design. Therefore, this paper investigates the
performance of SFANC based on different one-dimensional and two-dimensional
convolutional neural networks. Additionally, we conducted comparative analyses
of several network training strategies and discovered that fine-tuning could
improve selection performance.

## . NTRODUCTION

Acoustic noise problems are becoming more prevalent as the quantity of
industrial equipment increases [1]. The attenuation of low-frequency noises is
quite difficult and expensive for passive noise control techniques such as
enclosures, barriers, silencers, etc. Different from passive techniques,
active noise control (ANC) involves the electro-acoustic generation of a sound
field to cancel an unwanted existing sound field [2]. Moreover, ANC can offer
a possible lower-cost alternative for the control of low-frequency noises.
Thus, it attracts much interest from the industry. When dealing with different
types of noises, traditional ANC systems typically use adaptive algorithms to
adjust control filter coefficients to minimize the error signal [3]. Among
adaptive algorithms, the filtered-X least mean square (FxLMS) and filtered-X
normalized least-mean-square (FxNLMS) algorithm are commonly used since they
can compensate for the delay involved by the secondary path to increase the
system robustness [4].

Figure 1: Block diagram of the CNN-based SFANC algorithm.

However, due to the least mean square (LMS) based algorithms’ inherent slow
convergence and poor tracking ability [5], FxLMS and FxNLMS are less capable
of dealing with rapidly varying or non-stationary noises. Their slow responses
to noises may impact customers’ perceptions of the noise reduction effect [6].
Fixed-filter ANC methods [7] can be adopted to tackle slow convergence, where
the control filter coefficients are pre-trained rather than adaptive updated.
However, the pre-trained control filter is only suitable for a specific noise
type, resulting in the degradation of noise reduction performance for other
types of noises. To rapidly select different pre-trained control filters given
different noise types, a selective fixed-filter active noise control (SFANC)
method based on the frequency band matching was proposed in [8].

Though the SFANC method [8] selects the most suitable pre-trained control
filters in response to different noise types, several critical parameters of
the method can only be determined through trials and errors. Considering the
limitations, deep learning techniques, particularly convolutional neural
networks (CNNs) [9, 10, 11, 12], appear to be powerful in classifying noises
in SFANC methods. Automatic learning of the SFANC algorithm’s critical
parameters based on deep learning would broaden its applications in real-world
scenarios.

With the learning ability of CNN models, the SFANC algorithm can automatically
learn its parameters from noise datasets and select the best control filter
given different noise types without resorting to extra-human efforts [13].
Additionally, a CNN model implemented on a co-processor can decouple the
computational load from the real-time noise controller. Therefore, in this
paper, we compared the performance of several one-dimensional (1D) CNNs and
two-dimensional (2D) CNNs in the SFANC method. Also, different network
training strategies are tried to choose the best one for training the
networks. Experiments show that the SFANC method based on CNN not only
achieves faster responses than FxLMS and FxNLMS but also exhibits good
robustness. Thus, it is expected to be used for attenuating dynamic noises
such as traffic noises and urban noises, etc.

## . NN-based SFANC Algorithm

The overall architecture of the CNN-based SFANC algorithm is depicted in
Figure 1. Throughout the control process, the real-time controller conducts
filtering to generate anti-noise while simultaneously sending the primary
noise to a co-processor (e.g., a mobile phone). Given the primary noise, the
co-processor employs a pre-trained CNN to produce the index for the most
appropriate control filter and delivers it to the real-time controller. The
controller then adjusts the control filter coefficients based on the received
filter index. Notably, if the network is a 1D CNN, its input is the raw
waveform [14]. However, if the network is a 2D CNN, its input is the Log Mel-
spectrogram [15].

Figure 2: Markov model of the ANC progress.

### 2.1. Concise Explanation of SFANC

An ANC progress can be abstracted as a first-order Markov chain [16] as shown
in Figure 2, where $\mathbf{w}_{o}(n)$ represents the optimal control filter
to attenuate the disturbance $d(n)$. To achieve the best noise reduction
performance, the best control filter $\mathbf{w}_{o}(n)$ can be selected from
a pre-trained filter set $\\{\mathbf{w}_{i}\\}^{C}_{i=1}$. Hence, the SFANC
method can be represented as follows:

$\mathbf{w}_{o}=\underset{\mathbf{w}\in\\{\mathbf{w}_{i}\\}^{C}_{i=1}}{\mathrm{argmin}}~{}\mathbb{E}\bigg{\\{}\left[d(n)-\mathbf{x}^{\mathrm{T}}(n)\mathbf{w}(n)\ast
s(n)\right]^{2}\bigg{\\}},$ (1)

where $\text{argmin}(\cdot)$ operator returns the input value for minimum
output; $\ast$ , $\mathbf{x}(n)$, and $\mathbf{s}(n)$ represent the linear
convolution, the reference signal, and the impulse response of the secondary
path, respectively. The reference signal is assumed to be the same as the
primary noise.

In practice, $d(n)$ is typically seen as the linear combination of
$\mathbf{x}(n)$. Thus, Equation 1 equals to

$\mathbf{w}_{o}=\underset{\mathbf{w}\in\\{\mathbf{w}_{i}\\}^{C}_{i=1}}{\mathrm{argmax}}P\left[\mathbf{w}|d(n)\right]=\underset{\mathbf{w}\in\\{\mathbf{w}_{i}\\}^{C}_{i=1}}{\mathrm{argmax}}P\left[\mathbf{w}|\mathbf{x}(n)\right],$
(2)

which means that the selected control filter is the one with maximum posterior
probability in the presence of reference signal $\mathbf{x}(n)$. Moreover,
according to Bayes’ theorem [17], the posterior probability can be replaced
with a conditional probability as

$\mathbf{w}_{o}=\underset{\mathbf{w}\in\\{\mathbf{w}_{i}\\}^{C}_{i=1}}{\mathrm{argmax}}P\left[\mathbf{x}(n)|\mathbf{w}\right],$
(3)

which predicts the most suitable control filter straight from the primary
noise $\mathbf{x}(n)$.

A classifier model $\hat{P}\left[\mathbf{x}(n)|\mathbf{w},\Theta\right]$ can
be developed to approximate ${P}\left[\mathbf{x}(n)|\mathbf{w}\right]$ from
the pre-recorded sampling set
$\\{\mathbf{x}^{j}(n),\mathbf{w}^{j}\\}^{N}_{j}$. The $\Theta$ denotes the
parameters of the classifier and can be obtained through maximum likelihood
estimation (MLE) [18] as

$\Theta=\underset{\Theta}{\mathrm{argmax}}\frac{1}{N}\sum^{N}_{j=1}\log{\hat{P}\left[\mathbf{x}^{j}(n)|\mathbf{w}^{j},\Theta\right]}.$
(4)

Therefore, we can utilize deep learning approaches to lean the classifier
model from the training set $\\{\mathbf{x}^{j}(n),\mathbf{w}^{j}\\}^{N}_{j}$.

### 2.2. CNN-based SFANC algorithm

Motivated by the work [13], this paper compares some 1D CNNs and 2D CNNs used
for classifying noises in the time domain and frequency domain, respectively.
The min-max operation firstly normalizes the input of the network:

$\hat{x}(n)=\frac{x(n)}{\max[\mathbf{x}(n)]-\min[\mathbf{x}(n)]},$ (5)

where $\max[\cdot]$ and $\min[\cdot]$ mean obtaining the maximum and minimum
value of $\mathbf{x}(n)$. It aims to rescale the input range into $(-1,1)$ and
retain the signal’s negative part that contains phase information. Phase
information is quite critical for ANC applications.

A lightweight 1D CNN illustrated in Figure 3 is proposed. Every residual block
in the network comprises two convolutional layers, subsequent batch
normalization, and ReLU non-linearity. Note that a shortcut connection is
adopted to add the input with the output in each residual block since residual
architecture is demonstrated easy to be optimized [19]. Additionally, the
network uses a broad receptive field (RF) in the first convolutional layer and
narrow RFs in the rest convolutional layers to fully exploit both global and
local information.

Figure 3: Architecture of the proposed 1D CNN. The configuration of
convolution layer is denoted as: (kernel size, channels, stride, padding).

Figure 4: The frequency bands of the noise tracks for pre-training control
filters.

### 2.3. Training of CNNs

The primary and secondary paths used in the training stage of the control
filters are band-pass filters with a frequency range of $20$Hz-$7,980$Hz.
Broadband noises with $15$ frequency ranges shown in Figure 4 are used to pre-
train $15$ control filters. The FxLMS algorithm is adopted to obtain the
optimal control filters for these broadband noises due to its low
computational complexity. Subsequently, the $15$ pre-trained control filters
are saved in the control filter database.

A noise dataset including synthetic and real noise tracks is used in this
work. Specifically, $80,000$ synthetic noise tracks and $80,000$ real noise
tracks are used for training, $2,000$ real noise tracks for validation, and
$2,000$ real noise tracks for testing. The synthetic noise tracks are randomly
generated with various frequency bands, amplitudes, and background noise
levels.

The SFANC system’s sample rate is $16,000$Hz, so each noise track of 1-second
duration consists of $16,000$ samples. Each noise track of 1 second duration
is taken as primary noise to generate disturbance. The class label of a noise
track corresponds to the index of the control filter that achieves the best
noise reduction performance on the disturbance.

## . xperiments

The Adam algorithm was employed to optimize the network during training. The
training epoch was set to be $30$. The glorot initialization [20] was used to
avoid bursting or vanishing gradients. Additionally, to prevent overfitting,
the weights of CNNs were subjected to $\ell_{2}$ regularization with a
coefficient of $0.0001$.

### 3.1. Comparison of Different Training Schemes

Four different training schemes are compared in training the proposed 1D CNN.
The comparison results are summarized in Table 1. According to Table 1,
training firstly on the synthetic noise tracks and then fine-tuning on the
real noise tracks achieves the highest testing accuracy. Noted that
simultaneously using synthetic dataset and real dataset for training has not
obtained a superior testing accuracy since the characteristics of synthetic
noises and real noises are quite different. As discussed above, in the SFANC
system, the CNN models can be firstly trained with the synthetic dataset and
then fine-tuned with the real noise dataset.

Table 1: The performance of different network training schemes.

Training Scheme | Testing Accuracy
---|---
Only using synthetic dataset | 46.4$\%$
Only using real dataset | 94.6$\%$
Fine-tuning method* | 95.3$\%$
Simultaneously using synthetic dataset and real dataset | 94.5$\%$

  * *

Training firstly by the synthetic dataset and then fine-tuning by the real
dataset.

### 3.2. Comparison of Different Networks

Based on above fine-tuning training scheme, we compared several different 1D
networks utilizing raw acoustic waveforms: the proposed 1D CNN, M3 [21], M5
[21], M11 [21], M18 [21], and M34-res [21]. Also, some light-weight 2D
networks including ShuffleNet v2 [22], MoblieNet v2 [23] and Attention Network
[24] are compared in the SFANC method. The performance of these networks on
the real testing dataset are summarised in Table 2.

Table 2: Comparisons of different networks used in the SFANC system. Network | Testing Accuracy | Network Parameters
---|---|---
1D Convolutional Neural Networks
Proposed 1D Network | 95.3$\%$ | 0.21M
M3 Network | 93.7$\%$ | 0.22M
M5 Network | 94.9$\%$ | 0.56M
M11 Network | 94.5$\%$ | 1.79M
M18 Network | 93.8$\%$ | 3.69M
M34-res Network | 94.4$\%$ | 3.99M
2D Convolutional Neural Networks
ShuffleNet v2 | 95.5$\%$ | 0.25M
MoblieNet v2 | 95.6$\%$ | 2.89M
Attention Network | 94.9$\%$ | 4.95M

As shown in Table 2, the proposed 1D network obtains the highest
classification accuracy of $95.3\%$ with the fewest network parameters among
the 1D networks. As for 2D networks, the ShuffleNet v2 achieves a similar
classification accuracy as MoblieNet v2 and requires far fewer parameters. By
considering both the testing accuracy and the number of parameters, the
ShuffleNet v2 performs best on the testing dataset among the 2D networks.
Compared to the proposed 1D network, the ShuffleNet v2 obtains a slight
improvement in classification accuracy but requires a little more network
parameters. Therefore, it is found that the proposed 1D network and ShuffleNet
v2 perform better in classifying noises in the SFANC system. The two light-
weight networks can be implemented on mobile platforms, but using acoustic
models directly from the raw waveform data is more convenient [25]. Hence, the
proposed 1D network is preferred.

### 3.3. Non-stationary Noise Cancellation

This section uses the SFANC algorithm based on the proposed 1D network, FxLMS
algorithm, and FxNLMS algorithm to attenuate a recorded aircraft noise. The
aircraft noise is non-stationary and has a frequency range of 50Hz-14,000Hz.
It does not belong to the training dataset. The step size of the FxLMS and
FxNLMS algorithm is set to $0.0001$, and the control filter length is $1,024$
taps. The noise reduction results using different ANC methods on the aircraft
noise are shown in Figure 5.

From the results in Figure 5, we can observe that the SFANC method responds to
the aircraft noise much faster than the FxLMS and FxNLMS algorithms. Also, the
SFANC method consistently outperforms the FxLMS and FxNLMS algorithm in the
noise reduction process. In particular, during $1$s-$2$s, the averaged noise
reduction level achieved by the SFANC algorithm is about 7dB and 8dB more than
that of FxLMS and FxNLMS, respectively. Therefore, the results on the aircraft
noise confirm that the SFANC method can rapidly select the most suitable pre-
trained control filter given the noise type. In contrast, adaptive algorithms
show slow responses to the aircraft noise due to adaptive updating.

Figure 5: (a)-(c): Error signals of different ANC algorithms, (d): Averaged
noise reduction level of every 1 second, on the aircraft noise.

## . ONCLUSIONS

Active noise control (ANC) technologies have been widely used to deal with
low-frequency noises. However, adaptive ANC algorithms are typically limited
by slow convergence speed. In this paper, CNNs are used to automatically
select the best pre-trained control filters given different noises. Also,
light-weight CNNs implemented on a co-processor can decouple the computational
load from the real-time noise controller. Numerical simulations show that the
CNN-based SFANC method improves response time while maintaining low
computational complexity and high robustness. Additionally, the effectiveness
of the proposed 1D network and the fine-tuning training strategy are confirmed
in the SFANC method. In future works, we will explore more efficient and
robust ANC algorithms based on deep learning.

## REFERENCES

  * [1] Colin N Hansen.  Understanding active noise cancellation.  CRC Press, 1999.
  * [2] Sen M Kuo and Dennis R Morgan.  Active noise control: a tutorial review.  Proceedings of the IEEE, 87(6):943–973, 1999.
  * [3] Rishabh Ranjan, Tatsuya Murao, Bhan Lam, and Woon-Seng Gan.  Selective active noise control system for open windows using sound classification.  In INTER-NOISE and NOISE-CON Congress and Conference Proceedings, volume 253, pages 1921–1931. Institute of Noise Control Engineering, 2016.
  * [4] Feiran Yang, Jianfeng Guo, and Jun Yang.  Stochastic analysis of the filtered-x lms algorithm for active noise control.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2252–2266, 2020.
  * [5] Rishabh Ranjan and Woon-Seng Gan.  Natural listening over headphones in augmented reality using adaptive filtering techniques.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(11):1988–2002, 2015.
  * [6] Dongyuan Shi, Woon-Seng Gan, Bhan Lam, Shulin Wen, and Xiaoyi Shen.  Active noise control based on the momentum multichannel normalized filtered-x least mean square algorithm.  In INTER-NOISE and NOISE-CON Congress and Conference Proceedings, volume 261, pages 709–719. Institute of Noise Control Engineering, 2020.
  * [7] Chuang Shi, Rong Xie, Nan Jiang, Huiyong Li, and Yoshinobu Kajikawa.  Selective virtual sensing technique for multi-channel feedforward active noise control systems.  In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8489–8493. IEEE, 2019.
  * [8] Dongyuan Shi, Woon-Seng Gan, Bhan Lam, and Shulin Wen.  Feedforward selective fixed-filter active noise control: Algorithm and implementation.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:1479–1492, 2020.
  * [9] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  nature, 521(7553):436–444, 2015.
  * [10] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015.
  * [11] Zhengding Luo, Qinghua Gu, Gege Qi, Song Liu, Yuesheng Zhu, and Zhiqiang Bai.  A robust single-sensor face and iris biometric identification system based on multimodal feature extraction network.  In 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI), pages 1237–1244, 2019.
  * [12] Zhengding Luo, Qinghua Gu, Guoxiong Su, Yuesheng Zhu, and Zhiqiang Bai.  An adaptive face-iris multimodal identification system based on quality assessment network.  In MultiMedia Modeling, pages 87–98. Springer International Publishing, 2021.
  * [13] Dongyuan Shi, Bhan Lam, Kenneth Ooi, Xiaoyi Shen, and Woon-Seng Gan.  Selective fixed-filter active noise control based on convolutional neural network.  Signal Processing, 190:108317, 2022.
  * [14] Zhengding Luo, Dongyuan Shi, and Woon-Seng Gan.  A hybrid sfanc-fxnlms algorithm for active noise control based on deep learning.  IEEE Signal Processing Letters, 29:1102–1106, 2022.
  * [15] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4510–4520, 2018.
  * [16] Paulo AC Lopes and Moisés S Piedade.  A kalman filter approach to active noise control.  In 2000 10th European Signal Processing Conference, pages 1–4. IEEE, 2000.
  * [17] Steven M Kay.  Fundamentals of statistical signal processing: estimation theory.  Prentice-Hall, Inc., 1993.
  * [18] Sjoerd van Ophem and Arthur P Berkhoff.  Multi-channel kalman filters for active noise control.  The Journal of the Acoustical Society of America, 133(4):2105–2115, 2013.
  * [19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [20] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010.
  * [21] Wei Dai, Chia Dai, Shuhui Qu, Juncheng Li, and Samarjit Das.  Very deep convolutional neural networks for raw waveforms.  In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 421–425. IEEE, 2017.
  * [22] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun.  Shufflenet v2: Practical guidelines for efficient cnn architecture design.  In Proceedings of the European conference on computer vision (ECCV), pages 116–131, 2018.
  * [23] Sainath Adapa.  Urban sound tagging using convolutional neural networks.  In Proceedings of the Detection and Classification of Acoustic Scenes and Events 2019 Workshop (DCASE2019), pages 5–9, 2019.
  * [24] Zhengding Luo, Junting Li, and Yuesheng Zhu.  A deep feature fusion network based on multiple attention mechanisms for joint iris-periocular biometric recognition.  IEEE Signal Processing Letters, 28:1060–1064, 2021.
  * [25] Erfan Loweimi, Peter Bell, and Steve Renals.  On the robustness and training dynamics of raw waveform models.  In INTERSPEECH, pages 1001–1005, 2020.