Technical report

Maciej Wielgosz0000-0002-4401-2957 Centre de Visió per Computador (CVC),
Barcelona, Spain

Antonio M. López0000-0002-6979-5783 Centre de Visió per Computador (CVC),
Barcelona, Spain

Muhammad Naveed Riaz Centre de Visió per Computador (CVC), Barcelona, Spain

# CARLA-BSP: a simulated dataset with pedestrians

Maciej Wielgosz

###### keywords:

CARLA; simulated data; pedestrians; dataset

We present a sample dataset featuring pedestrians generated using the ARCANE
framework, a new framework for generating datasets in CARLA (0.9.13). We
provide use cases for pedestrian detection, autoencoding, pose estimation, and
pose lifting. We also showcase baseline results. For more information, visit
https://project-arcane.eu/.

##  1 Introduction

Realistic simulation frameworks can offer many advantages compared to real-
world data. One of the most apparent is the instant availability of the ground
truth, as opposed to the time-consuming manual labeling of the collected data.
It is also relatively easy to generate a vast amount of samples, while the
real-world data is often limited and, by necessity, unbalanced.

Autonomous vehicle (AV) testing and training is one example of an application
where simulated data can provide an advantage. This boon is especially true
when learn-from-data methods, such as deep learning (DL), are employed.
Usually, the most dangerous situations occur rarely, and collecting enough
examples to ensure the system handles them correctly can be a challenge –
because how often will you be able to record a kid playing with the dog beside
the road?

Generation of such corner, adversarial cases is where simulation methods can
help the most. However, even the most ,,typical” outputs from the simulation
can be of use – imagine having an unlimited amount of data you can use to
preliminary train your models or validate your flow.

CARLA (Car Learning to Act) is an open simulator for urban driving
[Dosovitskiy et al.(2017)Dosovitskiy, Ros, Codevilla, López, and Koltun].
Among its other features, it offers a Python API that allows two-way
interaction with the simulated world. One of the applications it can be used
for is the creation of increasingly complex scenarios that can be recorded and
utilized in various models’ training and validation. Moreover, various
modalities are available: RGB camera, DVS camera, LIDAR, and others. The
ground truth annotations can be obtained, among others, using semantic and
instance segmentation cameras or semantic LIDAR.

Of particular interest to us were scenarios involving pedestrians. Therefore,
we decided to create a basic dataset containing simple scenes to validate the
usefulness of this approach in practice and invite early community feedback.
The dataset contains RGB camera data in the form of MP4 videos and semantic
segmentation data encoded as APNG (Animated PNG) along with 2D and 3D pose
keypoints. The videos involve a single pedestrian, with no other traffic data,
only background objects (buildings, vegetation, parked cars). Four main
pedestrian models are used: adult female and male, and children of both
genders. Additionally, each of the basic models has look variants.

In the next sections, we describe the dataset in more detail and showcase its
potential usage in various applications, briefly discussing state-of-the-art
for those and similar tasks. The code is separated into two main repositories,
the first one used to generate the dataset and the second one containing flows
and models for common tasks. There are also helper repositories for
orchestrating the whole setup and abstracting a common base. They are publicly
available via https://github.com/wielgosz-info/carla-
pedestrians?utm_source=link&utm_medium=latexutm_campaign=CARLA-
BSPhttps://github.com/wielgosz-info/carla-pedestrians. The dataset itself can
be downloaded from https://project-arcane.eu/datasets/binary-single-
pedestrian?utm_source=link&utm_medium=latexutm_campaign=CARLA-
BSPhttps://project-arcane.eu/datasets/binary-single-pedestrian.

##  2 High-level concepts

Familiarity with notions presented in this section can help better understand
the rest of this work. Some useful CARLA-related terminology and concepts,
overall project idea, and code architecture are briefly described, along with
some specific terms we use throughout the text.

###  2.1 CARLA

Figure 1: Coordinate system used by CARLA.

One crucial concept that must be introduced is the coordinate system. CARLA
uses a Z-up left-handed system, as presented in Fig. 1. One of the basic
structures that are used when working with the API is Transform, which
consists of a Location (an X, Y, Z point in space) expressed in meters
[$\mathrm{m}\text{/}$] and a Rotation (pitch, yaw, roll; note that the order
does not correspond to axes order!) that is expressed in degrees. All of the
following coordinates are expressed using this system unless otherwise stated.

In our work, we use a subset of the full CARLA skeleton that consists of 26
joints - 25 joints that correspond to what can be thought of as a ,,physical”
human skeleton and one meta point, the root point, that is used as a reference
for other points. We disregard the joints representing human hands as the
resolution of images we expect to work with at any point would be insufficient
to distinguish them. When describing a skeleton, a concept of T-pose needs to
be mentioned. T-pose is a basic position the model is developed with. It is
also the default position a model will appear in when added to the simulation
for the first time (spawned). As can be guessed, the name comes from a
position where a human appears like a ,,T” letter. The identifiers of the
joints and a close-up of the skeleton as it appears in the example frame can
be seen in Fig. 2.

Figure 2: (Left) CARLA skeleton in T-pose with joints’ names and indices
annotated. (Right) A zoomed-in look at the overlayed skeleton as it appears in
one of the dataset frames.

Actors in CARLA can be understood as entities we can interact with. It may be
a pedestrian, camera, vehicle, or other object. Actors are spawned from
blueprints – templates that hold object attributes, some configurable, and
have associated information on how the object should behave and look. The
available blueprints can be read from the library. We are particularly
interested in pedestrian blueprints, which start with walker.pedestrian.*, and
sensor blueprints sensor.*.*.

Each walker.pedestrian.* blueprint has some read-only attributes, like age and
gender, and some modifiable ones, like speed. As mentioned, pedestrians use
four basic skeletons as a base, one for an adult female, an adult male, a
child female, and a child male. Each blueprint differs visually in clothing
choice, hairstyle, and skin color. Included are some professional outfits,
e.g. a policeman, but as they may differ between the countries, they should
not be used to differentiate between civilians and uniformed pedestrians.

The sensors in CARLA that can be created and used to receive data about the
world can be divided into several groups. Of most interest in our case are
various cameras, with sensors belonging to the LIDAR class planned for
addition soon. Sensor blueprints offer a significantly wider selection of
modifiable attributes than pedestrian.walker.* ones, allowing a high degree of
customization.

All sensor.camera.* blueprints share some common characteristics. Among the
most useful is the possibility to modify the size of the image returned by the
camera and its field of vision (FOV). Currently, we capture the data from the
RGB camera and semantic segmentation camera. Dynamic Vision Sensor (DVS), also
known as Events camera, is in the testing phase.

###  2.2 Overall architecture

We believe that the easiest way to utilize CARLA, especially on a remote
server, is by using the Docker image provided by a CARLA team. Currently, we
use the image that provides version 0.9.13. Contrary to what seems a common
approach, we do not add our code to that base image, as that has the potential
to generate many conflicts, considering our use cases. Instead, we take
advantage of the possibilities offered by Docker Compose to connect several
containers together.

Describing the philosophy and usage of containers and related tools is beyond
the scope of this article, and to understand this work, it is enough to grasp
its similarities to (probably more common) virtual machines. To create a
container, an image is needed. The image acts as a template, similar to how a
class definition is a template to create instances in object programming.
Several containers can run on one physical machine simultaneously and be
connected via a private network. They also can share volumes used to store
data. One of the crucial advantages of Docker containers is an option to
specify Dockerfile, in which the contents of the image can be specified using
plain text. It is also possible to extend some base images with one’s specific
needs. We invite the reader to explore https://docs.docker.com/get-
started/overview/the Docker documentation website for more information about
containers and Docker.

In our main code repository, carla-pedestrians, we specify how the containers
we use should work together. The specific image contents are defined either in
subdirectories or in Git submodules, using docker-compose.yml and Dockerfile
files. A diagram of the high-level container structure can be found in Fig. 3.
Not all containers need to be running simultaneously, but in general
pedestrians-scenarios requires the CARLA server to be running; pedestrians-
video-2-carla only needs it if outputs rendering in CARLA is requested.
CarlaViz [Xu(2019)] allows an in-browser preview of what is happening on the
server, but it requires the hero vehicle to be present; it is not useful
during dataset generation and was included only for completeness.

Figure 3: Partial structure of the main repository and its relation to Docker
images. All mentioned repositories are hosted on GitHub.

###  2.3 Dataset generation

The pedestrians-scenarios repository and corresponding container allow to,
among others, record and save data from CARLA to create a dataset. The code is
organized into several logical modules, but two are important for dataset
generation. karma module contains various lower-level definitions and logic
that potentially can be reused in other applications. datasets module provides
the actual generation flow, with customization done mostly at the level of
BatchGenerators. Some of the simpler, non-behavioral modifications are also
possible with the configuration files, examples of which can be found in
pedestrians-scenarios/configs.

The following list contains some concepts we use concerning dataset
generation.

  * •

World – whenever we write about the world, we mean a newly loaded carla.World
instance with a specific carla.Map.

  * •

Scenario – a template used to create scenes. It describes the number of actors
and their possible behaviors. It may be as simple as specifying that there
should be a single pedestrian that can either walk along the street or cross
it, and there is a $50\text{\,}\mathrm{\char 37\relax}\text{/}$ chance for
each. Generally, an arbitrary number of pedestrians can participate in a
single scenario, although we have only implemented single-pedestrian
scenarios.

  * •

Scene or clip – an randomized instance of the specific scenario that takes
place inside the world. Pedestrians (and sensors) exist in the world for the
whole duration of the scene.

  * •

Batch – all scenes simulated synchronously. The generator can simultaneously
capture several clips in one world to save computation time. However, since
the starting points are randomized, actors belonging to the parallel scenes
may get captured on the sensors; therefore, this option should be used
carefully. By default, the batch size is set to one, so each clip will be
recorded in a newly loaded world. The batch size can be modified via the
config file.

  * •

Pedestrian profile – a structure describing possible types of pedestrians. For
now, it includes fields like age, gender, walking speed (expressed as mean and
standard deviation), and crossing speed (likewise). We prepared four basic
profiles, one for each available age-gender combination. It is possible to
modify/add new profiles or change the proportions with which they are used by
writing a custom config file. One of the examples, pedestrians-
scenarios/configs/children.yaml shows how to create a dataset that will
contain only children, with a higher probability of females.

  * •

Camera – this is a slightly misleading term for what could be more aptly named
,,point of view”. Several sensors can be associated with each camera (actual
RGB, DVS or semantic cameras, LIDAR, etc.). They are, however, synchronized in
their physical orientation in space and, where possible, attributes like
returned image size. This results in synchronized data, creating an illusion
of a single device operating in multiple ,,modes” at the same time. The
camera.idx used later in the dataset description refers to such a logical
device. We plan to add the ability to specify the required modes via a config
file in the future. It is already possible to add several points of view,
using camera_position_distributions field.

One of the available generators is BinarySinglePedestrian, which was used to
generate the dataset described in this work. It uses default pedestrian
profiles, each having a $25\text{\,}\mathrm{\char 37\relax}\text{/}$ chance to
be used in a scenario. It also uses only crossing speed, and the pedestrian’s
target speed does not change during the scene. The actual speed can be
different when forced by the obstacle. A single default camera is used to
capture the scene, and its position is sampled from the following
distributions: x-axis mean=-7.0, std=2.0 (about seven meters in front of the
scene); y-axis mean=0.0, std=0.25 (looking mostly in the perpendicular
direction), z-axis mean=1.0, std=0.25 (about one meter above the ground).

##  3 Binary Single Pedestrian dataset

pedestrian.age | pedestrian.gender | no. of scenes | no. of frames |  frame.pedestrian.is_crossing
---|---|---|---|---
2*adult | female | $88$ | $49\,951$ |  $17\,820$
| male | $109$ | $60\,775$ |  $22\,924$
2*child | female | $100$ | $55\,755$ |  $23\,038$
| male | $99$ | $58\,846$ |  $28\,082$
|  | $396$ | $225\,327$ |  $91\,864$
|  |  |  |
Table 1: Basic dataset stats including the number of annotated frames.

The dataset contains almost 400 scenes. Each clip was recorded for $900$
frames at $30\text{\,}\mathrm{fps}\text{/}$, yielding
$30\text{\,}\mathrm{s}\text{/}$ of video and associated semantic segmentation
data. We refer to it as binary, as it contains only two classes for use in
classification tasks: the pedestrian is either crossing or non-crossing. The
detailed stats can be found in Tab. 1.

As mentioned previously, each video contains only a single pedestrian. It may
contain only a fragment of a silhouette, and there is no guarantee that the
pedestrian is visible in all frames (often, there are only background frames
at the beginning and end of the video). Each frame where the pedestrian is in
the viewport has a corresponding entry in the annotations file (data.csv).
$63.22\text{\,}\mathrm{\char 37\relax}\text{/}$ of frames are annotated.

The annotations are provided in CSV format. It was decided that the ease of
working with and previewing such data trumps its bigger size. When running
actual tasks, this data is split into clips of the desired length and saved in
efficient HDF5 format. The annotations consist of 24 columns, including 20
data columns and four used for indexing purposes. An example of a single data
row can be found in Tab. 3. The index columns are:

  1. 1.

id – unique ID of the recorded scene/clip,

  2. 2.

camera.idx – index of the camera/point of view used to capture the data,

  3. 3.

frame.idx – frame index in the corresponding video and other files that this
row applies to; in general it is not continuous as only frames in which the
pedestrian is visible are annotated, while accompanying source files (videos,
APNGs with semantic labels) contain all frames as recorded,

  4. 4.

pedestrian.idx – the pedestrian the data in the row corresponds to; it is
unique only in the scene. It is always zero, as all scenarios have only one
pedestrian. In the future, we plan on introducing scenarios that involve
several actors at the same time.

Data columns can be split into two main groups. First of them are initial
attributes that are constant for the whole clip:

  1. [resume]

  2. 1.

world.map – this is the name of the CARLA map that was used to record the
scene,

  3. 2.

camera.transform – world coordinates of the camera with index camera.idx.
Currently, cameras are assumed to be static; in the future, we plan to add
moving cameras (like on vehicles),

  4. 3.

camera.width – pixel width of the image captured by the camera camera.idx,

  5. 4.

camera.height – like above, but for height,

  6. 5.

camera.recording – the relative path to the *.mp4 file that contains the RGB
recording of the camera.idx camera,

  7. 6.

camera.semantic_segmentation – the relative path to the *.apng file that
contains the semantic segmentation labels of the camera.idx camera,

  8. 7.

pedestrian.model – name of the blueprint (,,skin”) that was used to create the
pedestrian with index pedestrian.idx,

  9. 8.

pedestrian.age – age category as specified in blueprint; currently can be
either adult or child,

  10. 9.

pedestrian.gender – gender category as specified in blueprint; currently can
be either female or male,

  11. 10.

pedestrian.spawn_point – world coordinates where the pedestrian with index
pedestrian.idx first appeared.

The second group contains values that can change in each frame, and most are
specific to a particular pedestrian:

  1. [resume]

  2. 1.

world.frame – which simulation frame is this since the world was first
created; there can be a scenario-dependent number of ,,startup frames” needed
before the actual scene recording can start,

  3. 2.

frame.pedestrian.is_crossing – is the pedestrian considered to be crossing the
street in this frame; this is an automatic annotation that checks whether the
pedestrian is on a driving lane,

  4. 3.

frame.pedestrian.transform – pedestrian’s bounding box center coordinates in
the world,

  5. 4.

frame.pedestrian.velocity – pedestrian’s velocity in
$\mathrm{m}\text{/}\mathrm{s}$,

  6. 5.

frame.pedestrian.pose.in_frame – is at least one pedestrian’s joint in the
camera viewport; it does not mean it is actually visible, as the obstacles may
obscure the view, only that its coordinates fall into the correct range,

  7. 6.

frame.pedestrian.pose.in_segmentation – is at least one pixel in the
segmentation mask marked as a pedestrian, i.e. at least a fragment of the
pedestrian is in the viewport and any possible cover is less than
$100\text{\,}\mathrm{\char 37\relax}\text{/}$,

  8. 7.

frame.pedestrian.pose.world – position of each joint expressed in world
coordinates,

  9. 8.

frame.pedestrian.pose.component – position of each joint relative to the pivot
of the actor,

  10. 9.

frame.pedestrian.pose.relative – position of each joint relative to its parent
in a kinematic tree,

  11. 10.

frame.pedestrian.pose.camera – position of each joint projected to 2D with
camera camera.idx.

name | value |
---|---|---
id | 094b9fe1-babe-48d5-bd17-3ba3185690c5-0 |
camera.idx | 0
frame.idx | 0
pedestrian.idx | 0
world.map | /Game/Carla/Maps/Town04 |
camera.transform | [x, y, z ($\mathrm{m}\text{/}$), pitch, yaw, roll ($\mathrm{deg}\text{/}$)] | [ |  $-5.622\,942\,924\,499\,512$, |  $316.143\,859\,863\,281\,25$, |  $1.118\,919\,253\,349\,304\,2$, |  $-9.404\,664\,993\,286\,133$, |  $-65.501\,998\,901\,367\,19$, | $0.000\,000\,432\,702\,847\,774\,635\,3$ | ]
camera.width | $\mathrm{px}\text{/}$ | 1600
camera.height | $\mathrm{px}\text{/}$ | 600
camera.recording | clips/094b9fe1-babe-48d5-bd17-3ba3185690c5-0-0.mp4
camera.semantic_segmentation | clips/094b9fe1-babe-48d5-bd17-3ba3185690c5-0-1.apng
pedestrian.model | walker.pedestrian.0001
pedestrian.age | adult
pedestrian.gender | female
pedestrian.spawn_point | [x, y, z ($\mathrm{m}\text{/}$), pitch, yaw, roll ($\mathrm{deg}\text{/}$)] | [ |  $0.331\,657\,528\,877\,258\,3$, |  $308.923\,614\,501\,953\,1$, |  $0.430\,041\,074\,752\,807\,6$, |  $0.0$, |  $0.0$, | $0.0$ | ]
world.frame | 9 |
frame.pedestrian.is_crossing | False
frame.pedestrian.transform | [x, y, z ($\mathrm{m}\text{/}$), pitch, yaw, roll ($\mathrm{deg}\text{/}$)] | [ |  $0.330\,562\,859\,773\,635\,86$, |  $308.925\,964\,355\,468\,75$, |  $0.951\,499\,760\,150\,909\,4$, |  $0.0$, |  $114.875\,373\,840\,332\,03$, | $0.0$ | ]
frame.pedestrian.velocity | [x, y, z ($\mathrm{m}\text{/}\mathrm{s}$)] | [ |  $-0.032\,839\,961\,349\,964\,14$, |  $0.070\,898\,428\,559\,303\,28$, | $0.0$ | ]
frame.pedestrian.pose.in_frame | True
frame.pedestrian.pose.in_segmentation | True
frame.pedestrian.pose.world |  | [[ |  $0.328\,373\,283\,147\,811\,9$, |  $308.930\,694\,580\,078\,1$, |  $0.031\,499\,784\,439\,802\,17$, |  $0.0$, |  $24.875\,143\,051\,147\,46$, | $89.996\,185\,302\,734\,38$ | ],
| [[x, y, z ($\mathrm{m}\text{/}$), pitch, yaw, roll ($\mathrm{deg}\text{/}$)], …] | [ |  $0.326\,179\,981\,231\,689\,45$, |  $308.926\,635\,742\,187\,5$, |  $1.083\,359\,241\,485\,595\,7$, |  $-0.383\,303\,374\,052\,047\,73$, |  $22.558\,658\,599\,853\,516$, | $91.079\,765\,319\,824\,22$ | ],
|  | [ |  $0.330\,417\,543\,649\,673\,46$, |  $308.918\,304\,443\,359\,4$, |  $1.190\,052\,747\,726\,440\,4$, |  $0.055\,105\,961\,859\,226\,23$, |  $21.507\,951\,736\,450\,195$, | $91.767\,204\,284\,667\,97$ | ],
|  | [ |  $0.340\,895\,831\,584\,930\,4$, |  $308.891\,265\,869\,140\,6$, |  $1.353\,142\,023\,086\,547\,9$, |  $0.475\,299\,179\,553\,985\,6$, |  $19.955\,116\,271\,972\,656$, | $91.169\,647\,216\,796\,88$ | ],
|  | [ |  $0.375\,388\,205\,051\,422\,1$, |  $308.913\,421\,630\,859\,4$, |  $1.501\,969\,218\,254\,089\,4$, |  $-3.699\,148\,178\,100\,586$, |  $8.623\,555\,183\,410\,645$, | $95.497\,093\,200\,683\,6$ | ],
|  | [ |  $0.483\,258\,813\,619\,613\,65$, |  $308.929\,748\,535\,156\,25$, |  $1.494\,915\,485\,382\,08$, |  $-77.192\,932\,128\,906\,25$, |  $22.789\,136\,886\,596\,68$, | $-80.145\,248\,413\,085\,94$ | ],
|  | [ |  $0.542\,537\,152\,767\,181\,4$, |  $308.955\,627\,441\,406\,25$, |  $1.233\,181\,238\,174\,438\,5$, |  $-73.684\,356\,689\,453\,12$, |  $89.918\,136\,596\,679\,69$, | $-139.349\,349\,975\,585\,94$ | ],
|  | [ |  $0.542\,668\,223\,381\,042\,5$, |  $309.025\,573\,730\,468\,75$, |  $0.994\,058\,728\,218\,078\,6$, |  $-2.686\,436\,176\,300\,049$, |  $-53.122\,089\,385\,986\,33$, | $167.739\,318\,847\,656\,25$ | ],
|  | [ |  $0.341\,924\,458\,742\,141\,7$, |  $308.883\,697\,509\,765\,6$, |  $1.548\,534\,989\,356\,994\,6$, |  $0.827\,900\,826\,930\,999\,8$, |  $-157.035\,873\,413\,085\,94$, | $77.154\,792\,785\,644\,53$ | ],
|  | [ |  $0.331\,692\,636\,013\,031$, |  $308.911\,132\,812\,5$, |  $1.637\,737\,035\,751\,342\,8$, |  $2.255\,444\,526\,672\,363\,3$, |  $-154.068\,634\,033\,203\,12$, | $74.491\,333\,007\,812\,5$ | ],
|  | [ |  $0.324\,864\,298\,105\,239\,87$, |  $309.006\,958\,007\,812\,5$, |  $1.710\,412\,859\,916\,687$, |  $-2.260\,519\,266\,128\,54$, |  $25.957\,834\,243\,774\,414$, | $96.485\,504\,150\,390\,62$ | ],
|  | [ |  $0.265\,771\,359\,205\,245\,97$, |  $308.978\,240\,966\,796\,9$, |  $1.713\,000\,774\,383\,545$, |  $-2.260\,519\,266\,128\,54$, |  $25.957\,834\,243\,774\,414$, | $96.485\,504\,150\,390\,62$ | ],
|  | [ |  $0.297\,919\,243\,574\,142\,46$, |  $308.885\,284\,423\,828\,1$, |  $1.501\,290\,082\,931\,518\,6$, |  $-1.565\,479\,278\,564\,453\,1$, |  $-152.721\,893\,310\,546\,88$, | $-93.721\,328\,735\,351\,56$ | ],
|  | [ |  $0.200\,782\,507\,658\,004\,76$, |  $308.835\,174\,560\,546\,9$, |  $1.498\,303\,174\,972\,534\,2$, |  $76.904\,869\,079\,589\,84$, |  $16.132\,751\,464\,843\,75$, | $105.752\,090\,454\,101\,56$ | ],
|  | [ |  $0.137\,127\,429\,246\,902\,47$, |  $308.818\,206\,787\,109\,4$, |  $1.236\,869\,573\,593\,139\,6$, |  $72.412\,292\,480\,468\,75$, |  $-65.769\,081\,115\,722\,66$, | $30.698\,078\,155\,517\,578$ | ],
|  | [ |  $0.106\,404\,662\,132\,263\,18$, |  $308.888\,153\,076\,171\,9$, |  $0.999\,718\,606\,472\,015\,4$, |  $2.251\,667\,261\,123\,657\,2$, |  $80.102\,447\,509\,765\,62$, | $-25.211\,469\,650\,268\,555$ | ],
|  | [ |  $0.258\,733\,093\,738\,555\,9$, |  $308.881\,286\,621\,093\,75$, |  $0.996\,551\,096\,439\,361\,6$, |  $5.230\,073\,451\,995\,85$, |  $40.185\,192\,108\,154\,3$, | $-97.509\,773\,254\,394\,53$ | ],
|  | [ |  $0.244\,851\,887\,226\,104\,74$, |  $308.932\,128\,906\,25$, |  $0.536\,806\,643\,009\,185\,8$, |  $2.753\,897\,905\,349\,731\,4$, |  $40.128\,723\,144\,531\,25$, | $98.494\,132\,995\,605\,47$ | ],
|  | [ |  $0.277\,621\,179\,819\,107\,06$, |  $308.883\,087\,158\,203\,1$, |  $0.105\,696\,104\,466\,915\,13$, |  $0.483\,946\,204\,185\,485\,84$, |  $43.346\,218\,109\,130\,86$, | $-159.994\,369\,506\,835\,94$ | ],
|  | [ |  $0.202\,863\,350\,510\,597\,23$, |  $308.962\,554\,931\,640\,6$, |  $0.070\,813\,797\,414\,302\,83$, |  $-4.248\,117\,923\,736\,572$, |  $134.853\,775\,024\,414\,06$, | $90.536\,819\,458\,007\,81$ | ],
|  | [ |  $0.171\,125\,233\,173\,370\,36$, |  $308.994\,598\,388\,671\,9$, |  $0.055\,633\,764\,714\,002\,61$, |  $-0.762\,419\,819\,831\,848\,1$, |  $-135.157\,180\,786\,132\,8$, | $-166.283\,020\,019\,531\,25$ | ],
|  | [ |  $0.404\,810\,756\,444\,931\,03$, |  $308.941\,986\,083\,984\,4$, |  $0.995\,492\,815\,971\,374\,5$, |  $-4.093\,700\,885\,772\,705$, |  $11.055\,594\,444\,274\,902$, | $85.592\,895\,507\,812\,5$ | ],
|  | [ |  $0.387\,456\,476\,688\,385$, |  $308.961\,853\,027\,343\,75$, |  $0.533\,490\,478\,992\,462\,2$, |  $-0.711\,138\,784\,885\,406\,5$, |  $10.852\,561\,950\,683\,594$, | $-77.365\,280\,151\,367\,19$ | ],
|  | [ |  $0.425\,920\,963\,287\,353\,5$, |  $308.878\,051\,757\,812\,5$, |  $0.108\,245\,082\,199\,573\,52$, |  $1.468\,797\,922\,134\,399\,4$, |  $6.067\,918\,300\,628\,662$, | $-156.922\,576\,904\,296\,88$ | ],
|  | [ |  $0.415\,749\,579\,668\,045\,04$, |  $308.984\,649\,658\,203\,1$, |  $0.067\,582\,003\,772\,258\,76$, |  $-5.178\,396\,224\,975\,586$, |  $94.506\,080\,627\,441\,4$, | $91.121\,658\,325\,195\,31$ | ],
|  | [ |  $0.412\,461\,608\,648\,300\,17$, |  $309.029\,388\,427\,734\,4$, |  $0.051\,673\,810\,929\,059\,98$, |  $-0.887\,753\,784\,656\,524\,7$, |  $-175.352\,142\,333\,984\,38$, | $12.791\,315\,078\,735\,352$ | ]]

Table 2: Example data from a single row in data.csv. Coordinates were rounded
to four decimal places. Table spans several pages.

frame.pedestrian.pose.component |  | [[ |  $0.0$, |  $0.0$, |  $-0.919\,999\,957\,084\,655\,8$, |  $0.0$, |  $-90.000\,205\,993\,652\,34$, | $89.996\,185\,302\,734\,38$ | ],
---|---|---|---|---|---|---|---|---|---
| [[x, y, z ($\mathrm{m}\text{/}$), pitch, yaw, roll ($\mathrm{deg}\text{/}$)], …] | [ |  $-0.002\,762\,997\,988\,611\,459\,7$, |  $0.003\,698\,689\,863\,085\,746\,8$, |  $0.131\,859\,436\,631\,202\,7$, |  $-0.383\,303\,374\,052\,047\,73$, |  $-92.316\,711\,425\,781\,25$, | $91.079\,765\,319\,824\,22$ | ],
|  | [ |  $-0.012\,111\,622\,840\,166\,092$, |  $0.003\,362\,396\,964\,803\,338$, |  $0.238\,553\,002\,476\,692\,2$, |  $0.055\,105\,961\,859\,226\,23$, |  $-93.367\,431\,640\,625$, | $91.767\,204\,284\,667\,97$ | ],
|  | [ |  $-0.041\,042\,692\,959\,308\,624$, |  $0.005\,226\,826\,760\,917\,902$, |  $0.401\,642\,292\,737\,960\,8$, |  $0.475\,299\,179\,553\,985\,6$, |  $-94.920\,272\,827\,148\,44$, | $91.169\,647\,216\,796\,88$ | ],
|  | [ |  $-0.035\,458\,173\,602\,819\,44$, |  $-0.035\,382\,166\,504\,859\,924$, |  $0.550\,469\,517\,707\,824\,7$, |  $-3.699\,148\,178\,100\,586$, |  $-106.251\,861\,572\,265\,62$, | $95.497\,093\,200\,683\,6$ | ],
|  | [ |  $-0.066\,002\,540\,290\,355\,68$, |  $-0.140\,121\,623\,873\,710\,63$, |  $0.543\,415\,665\,626\,525\,9$, |  $-77.192\,932\,128\,906\,25$, |  $-92.086\,196\,899\,414\,06$, | $-80.145\,256\,042\,480\,47$ | ],
|  | [ |  $-0.067\,477\,472\,126\,483\,92$, |  $-0.204\,778\,060\,317\,039\,5$, |  $0.281\,681\,507\,825\,851\,44$, |  $-73.684\,379\,577\,636\,72$, |  $-24.957\,246\,780\,395\,508$, | $-139.349\,349\,975\,585\,94$ | ],
|  | [ |  $-0.004\,062\,328\,021\,973\,371\,5$, |  $-0.234\,325\,781\,464\,576\,72$, |  $0.042\,558\,975\,517\,749\,786$, |  $-2.686\,436\,176\,300\,049$, |  $-167.997\,436\,523\,437\,5$, | $167.739\,318\,847\,656\,25$ | ],
|  | [ |  $-0.048\,350\,445\,926\,189\,42$, |  $0.007\,481\,341\,250\,240\,803$, |  $0.597\,035\,229\,206\,085\,2$, |  $0.827\,907\,681\,465\,148\,9$, |  $88.088\,752\,746\,582\,03$, | $77.154\,792\,785\,644\,53$ | ],
|  | [ |  $-0.019\,133\,226\,945\,996\,284$, |  $0.005\,212\,557\,036\,429\,644$, |  $0.686\,237\,335\,205\,078\,1$, |  $2.255\,451\,202\,392\,578$, |  $91.055\,984\,497\,070\,31$, | $74.491\,333\,007\,812\,5$ | ],
|  | [ |  $0.070\,669\,651\,031\,494\,14$, |  $-0.028\,899\,049\,386\,382\,103$, |  $0.758\,913\,099\,765\,777\,6$, |  $-2.260\,519\,266\,128\,54$, |  $-88.917\,518\,615\,722\,66$, | $96.485\,504\,150\,390\,62$ | ],
|  | [ |  $0.069\,461\,882\,114\,410\,4$, |  $0.036\,796\,890\,199\,184\,42$, |  $0.761\,501\,014\,232\,635\,5$, |  $-2.260\,519\,266\,128\,54$, |  $-88.917\,518\,615\,722\,66$, | $96.485\,504\,150\,390\,62$ | ],
|  | [ |  $-0.028\,386\,835\,008\,859\,634$, |  $0.046\,730\,272\,471\,904\,755$, |  $0.549\,790\,322\,780\,609\,1$, |  $-1.565\,472\,483\,634\,948\,7$, |  $92.402\,725\,219\,726\,56$, | $-93.721\,328\,735\,351\,56$ | ],
|  | [ |  $-0.032\,976\,474\,612\,951\,28$, |  $0.155\,928\,522\,348\,403\,93$, |  $0.546\,803\,414\,821\,624\,8$, |  $76.904\,884\,338\,378\,9$, |  $-98.742\,729\,187\,011\,72$, | $105.752\,120\,971\,679\,69$ | ],
|  | [ |  $-0.021\,598\,204\,970\,359\,802$, |  $0.220\,817\,491\,412\,162\,78$, |  $0.285\,369\,873\,046\,875$, |  $72.412\,292\,480\,468\,75$, |  $179.355\,560\,302\,734\,38$, | $30.698\,091\,506\,958\,008$ | ],
|  | [ |  $0.054\,777\,774\,959\,802\,63$, |  $0.219\,269\,335\,269\,927\,98$, |  $0.048\,218\,838\,870\,525\,36$, |  $2.251\,667\,261\,123\,657\,2$, |  $-34.772\,960\,662\,841\,8$, | $-25.211\,469\,650\,268\,555$ | ],
|  | [ |  $-0.015\,518\,064\,610\,660\,076$, |  $0.083\,956\,941\,962\,242\,13$, |  $0.045\,051\,343\,739\,032\,745$, |  $5.230\,073\,451\,995\,85$, |  $-74.690\,162\,658\,691\,4$, | $-97.509\,773\,254\,394\,53$ | ],
|  | [ |  $0.036\,444\,187\,164\,306\,64$, |  $0.075\,164\,720\,416\,069\,03$, |  $-0.414\,693\,087\,339\,401\,25$, |  $2.753\,897\,905\,349\,731\,4$, |  $-74.746\,650\,695\,800\,78$, | $98.494\,132\,995\,605\,47$ | ],
|  | [ |  $-0.021\,850\,800\,141\,692\,16$, |  $0.066\,073\,566\,675\,186\,16$, |  $-0.845\,803\,678\,035\,736\,1$, |  $0.483\,946\,204\,185\,485\,84$, |  $-71.529\,159\,545\,898\,44$, | $-159.994\,369\,506\,835\,94$ | ],
|  | [ |  $0.081\,695\,348\,024\,368\,29$, |  $0.100\,465\,849\,041\,938\,78$, |  $-0.880\,685\,925\,483\,703\,6$, |  $-4.248\,117\,923\,736\,572$, |  $19.978\,401\,184\,082\,03$, | $90.536\,811\,828\,613\,28$ | ],
|  | [ |  $0.124\,123\,133\,718\,967\,44$, |  $0.115\,777\,410\,566\,806\,8$, |  $-0.895\,865\,976\,810\,455\,3$, |  $-0.762\,419\,819\,831\,848\,1$, |  $109.967\,422\,485\,351\,56$, | $-166.283\,020\,019\,531\,25$ | ],
|  | [ |  $-0.021\,911\,410\,614\,848\,137$, |  $-0.074\,094\,653\,129\,577\,64$, |  $0.043\,993\,070\,721\,626\,28$, |  $-4.093\,700\,885\,772\,705$, |  $-103.819\,831\,848\,144\,53$, | $85.592\,903\,137\,207\,03$ | ],
|  | [ |  $0.003\,426\,780\,691\,370\,368$, |  $-0.066\,714\,048\,385\,620\,12$, |  $-0.418\,009\,281\,158\,447\,27$, |  $-0.711\,145\,579\,814\,910\,9$, |  $-104.022\,865\,295\,410\,16$, | $-77.365\,272\,521\,972\,66$ | ],
|  | [ |  $-0.088\,786\,400\,854\,587\,55$, |  $-0.066\,356\,234\,252\,452\,85$, |  $-0.843\,254\,685\,401\,916\,5$, |  $1.468\,804\,836\,273\,193\,4$, |  $-108.807\,487\,487\,792\,97$, | $-156.922\,576\,904\,296\,88$ | ],
|  | [ |  $0.012\,203\,731\,574\,118\,137$, |  $-0.101\,970\,039\,308\,071\,14$, |  $-0.883\,917\,748\,928\,070\,1$, |  $-5.178\,396\,224\,975\,586$, |  $-20.369\,295\,120\,239\,258$, | $91.121\,658\,325\,195\,31$ | ],
|  | [ |  $0.054\,163\,854\,569\,196\,7$, |  $-0.117\,801\,174\,521\,446\,23$, |  $-0.899\,825\,930\,595\,398$, |  $-0.887\,753\,784\,656\,524\,7$, |  $69.772\,460\,937\,5$, | $12.791\,315\,078\,735\,352$ | ]]
frame.pedestrian.pose.relative |  | [[ |  $0.0$, |  $0.0$, |  $0.0$, |  $0.0$, |  $0.000\,001\,207\,458\,126\,373\,239\,8$, | $89.996\,177\,673\,339\,84$ | ],
| [[x, y, z ($\mathrm{m}\text{/}$), pitch, yaw, roll ($\mathrm{deg}\text{/}$)], …] | [ |  $-0.003\,698\,692\,191\,392\,183\,3$, |  $-1.051\,859\,378\,814\,697\,3$, |  $-0.002\,693\,099\,901\,080\,131\,5$, |  $-2.316\,465\,377\,807\,617$, |  $0.383\,463\,501\,930\,236\,8$, | $1.068\,060\,159\,683\,227\,5$ | ],
|  | [ |  $0.000\,000\,013\,589\,858\,838\,258\,806$, |  $-0.106\,501\,236\,557\,960\,51$, |  $-0.011\,336\,335\,912\,346\,84$, |  $-1.058\,788\,537\,979\,126$, |  $-0.418\,543\,308\,973\,312\,4$, | $0.688\,302\,814\,960\,479\,7$ | ],
|  | [ |  $-0.000\,004\,966\,258\,984\,495\,653$, |  $-0.162\,117\,749\,452\,590\,94$, |  $-0.034\,006\,297\,588\,348\,39$, |  $-1.565\,014\,839\,172\,363\,3$, |  $-0.372\,270\,852\,327\,346\,8$, | $-0.585\,288\,107\,395\,172\,1$ | ],
|  | [ |  $0.041\,213\,583\,201\,169\,97$, |  $-0.148\,644\,134\,402\,275\,09$, |  $0.006\,013\,965\,699\,821\,711$, |  $-11.218\,653\,678\,894\,043$, |  $4.480\,054\,378\,509\,521\,5$, | $3.567\,191\,123\,962\,402\,3$ | ],
|  | [ |  $0.109\,330\,080\,449\,581\,15$, |  $0.000\,001\,226\,663\,584\,930\,065\,6$, |  $-0.000\,011\,384\,188\,837\,837\,428$, |  $8.388\,673\,782\,348\,633$, |  $73.716\,339\,111\,328\,12$, | $-157.859\,848\,022\,460\,94$ | ],
|  | [ |  $0.269\,557\,446\,241\,378\,8$, |  $0.005\,116\,452\,928\,632\,498$, |  $-0.000\,004\,158\,020\,146\,860\,508$, |  $-15.855\,490\,684\,509\,277$, |  $-3.600\,116\,014\,480\,591$, | $6.738\,451\,480\,865\,478\,5$ | ],
|  | [ |  $0.249\,146\,774\,411\,201\,48$, |  $0.000\,002\,248\,287\,955\,808\,337$, |  $0.000\,045\,146\,935\,008\,233\,41$, |  $79.244\,285\,583\,496\,1$, |  $-163.840\,209\,960\,937\,5$, | $14.058\,793\,067\,932\,129$ | ],
|  | [ |  $0.000\,001\,516\,342\,081\,231\,414\,3$, |  $-0.195\,206\,329\,226\,493\,84$, |  $-0.011\,461\,276\,561\,021\,805$, |  $-3.034\,700\,632\,095\,337$, |  $-178.757\,385\,253\,906\,25$, | $168.282\,257\,080\,078\,12$ | ],
|  | [ |  $-0.000\,004\,050\,731\,604\,365\,865$, |  $-0.093\,487\,620\,353\,698\,73$, |  $-0.008\,710\,688\,911\,378\,384$, |  $3.208\,706\,617\,355\,346\,7$, |  $-0.735\,065\,519\,809\,722\,9$, | $-2.763\,886\,928\,558\,349\,6$ | ],
|  | [ |  $-0.032\,872\,978\,597\,879\,41$, |  $-0.095\,171\,183\,347\,702\,03$, |  $-0.066\,119\,141\,876\,697\,54$, |  $-0.026\,842\,642\,575\,502\,396$, |  $-179.997\,802\,734\,375$, | $170.977\,905\,273\,437\,5$ | ],
|  | [ |  $0.032\,884\,988\,933\,801\,65$, |  $-0.095\,172\,010\,362\,148\,28$, |  $-0.066\,122\,189\,164\,161\,68$, |  $-0.026\,842\,642\,575\,502\,396$, |  $-179.997\,802\,734\,375$, | $170.977\,905\,273\,437\,5$ | ],
|  | [ |  $-0.041\,205\,566\,376\,447\,68$, |  $-0.148\,648\,738\,861\,083\,98$, |  $0.006\,016\,416\,475\,176\,811$, |  $-7.296\,198\,844\,909\,668$, |  $178.746\,963\,500\,976\,56$, | $-2.341\,199\,159\,622\,192\,4$ | ],
|  | [ |  $0.109\,335\,452\,318\,191\,53$, |  $-0.000\,000\,424\,385\,063\,979\,571$, |  $-0.000\,007\,634\,163\,011\,971\,395$, |  $-6.113\,681\,316\,375\,732$, |  $104.492\,042\,541\,503\,9$, | $25.683\,034\,896\,850\,586$ | ],
|  | [ |  $-0.269\,557\,863\,473\,892\,2$, |  $-0.005\,109\,663\,121\,402\,264$, |  $-0.000\,004\,138\,946\,223\,974\,926$, |  $-19.590\,484\,619\,140\,625$, |  $-5.283\,465\,862\,274\,17$, | $5.751\,477\,241\,516\,113$ | ],
|  | [ |  $-0.249\,147\,638\,678\,550\,72$, |  $0.000\,000\,724\,792\,585\,060\,640\,7$, |  $0.001\,348\,362\,420\,685\,589\,3$, |  $77.007\,507\,324\,218\,75$, |  $160.926\,513\,671\,875$, | $-34.757\,709\,503\,173\,83$ | ],
|  | [ |  $-0.079\,094\,514\,250\,755\,31$, |  $0.087\,624\,944\,746\,494\,3$, |  $-0.014\,340\,247\,958\,898\,544$, |  $17.437\,143\,325\,805\,664$, |  $-6.207\,947\,254\,180\,908$, | $169.705\,383\,300\,781\,25$ | ],
|  | [ |  $-0.019\,800\,415\,262\,579\,918$, |  $-0.462\,156\,444\,787\,979\,1$, |  $0.012\,713\,927\,775\,621\,414$, |  $0.379\,444\,301\,128\,387\,45$, |  $-2.447\,601\,795\,196\,533$, | $-164.000\,274\,658\,203\,12$ | ],
|  | [ |  $-0.027\,271\,423\,488\,855\,362$, |  $0.434\,237\,927\,198\,410\,03$, |  $0.005\,568\,313\,412\,368\,298$, |  $3.517\,178\,297\,042\,846\,7$, |  $1.768\,803\,715\,705\,871\,6$, | $101.474\,884\,033\,203\,12$ | ],
|  | [ |  $-0.000\,109\,534\,259\,536\,303\,58$, |  $-0.114\,458\,255\,469\,799\,04$, |  $-0.004\,549\,493\,547\,528\,982$, |  $-15.763\,010\,025\,024\,414$, |  $-91.599\,380\,493\,164\,06$, | $-89.399\,665\,832\,519\,53$ | ],
|  | [ |  $0.046\,106\,513\,589\,620\,59$, |  $0.011\,797\,547\,340\,393\,066$, |  $0.000\,004\,539\,489\,509\,625\,127\,5$, |  $89.764\,251\,708\,984\,38$, |  $73.156\,143\,188\,476\,56$, | $-178.876\,266\,479\,492\,2$ | ],
|  | [ |  $0.079\,089\,857\,637\,882\,23$, |  $0.087\,624\,967\,098\,236\,08$, |  $-0.014\,339\,422\,807\,097\,435$, |  $-11.399\,851\,799\,011\,23$, |  $4.011\,956\,691\,741\,943$, | $-6.338\,530\,063\,629\,15$ | ],
|  | [ |  $0.019\,795\,626\,401\,901\,245$, |  $0.462\,157\,189\,846\,038\,8$, |  $-0.012\,709\,273\,956\,716\,06$, |  $0.057\,353\,094\,220\,161\,44$, |  $-3.388\,160\,705\,566\,406\,2$, | $-162.968\,383\,789\,062\,5$ | ],
|  | [ |  $0.027\,273\,092\,418\,909\,073$, |  $-0.434\,237\,629\,175\,186\,16$, |  $-0.005\,565\,528\,757\,870\,197$, |  $5.144\,894\,123\,077\,393$, |  $1.083\,651\,542\,663\,574\,2$, | $-79.476\,913\,452\,148\,44$ | ],
|  | [ |  $0.000\,111\,463\,661\,596\,644\,67$, |  $-0.114\,455\,841\,481\,685\,64$, |  $-0.004\,548\,492\,375\,761\,270\,5$, |  $-17.853\,416\,442\,871\,094$, |  $-88.505\,691\,528\,320\,31$, | $-90.940\,818\,786\,621\,1$ | ],
|  | [ |  $0.046\,099\,409\,461\,021\,42$, |  $0.011\,797\,904\,968\,261\,719$, |  $-0.000\,005\,195\,736\,775\,931\,436\,6$, |  $89.765\,922\,546\,386\,72$, |  $-105.196\,624\,755\,859\,38$, | $-177.225\,021\,362\,304\,7$ | ]]

Table 3: Example data from a single row in data.csv (continued).

frame.pedestrian.pose.camera | [[x, y ($\mathrm{px}\text{/}$)], [x, y ($\mathrm{px}\text{/}$)], …] | [[ |  $1013.429\,809\,570\,312\,5$, | $264.522\,888\,183\,593\,75$ | ], |  | [ |  $1017.147\,277\,832\,031\,2$, | $170.727\,081\,298\,828\,12$ | ],
---|---|---|---|---|---|---|---|---|---|---
|  | [ |  $1017.383\,666\,992\,187\,5$, | $161.020\,462\,036\,132\,8$ | ], |  | [ |  $1017.186\,523\,437\,5$, | $146.183\,151\,245\,117\,2$ | ],
|  | [ |  $1021.576\,904\,296\,875$, | $132.523\,986\,816\,406\,25$ | ], |  | [ |  $1030.233\,764\,648\,437\,5$, | $133.285\,888\,671\,875$ | ],
|  | [ |  $1034.884\,643\,554\,687\,5$, | $157.149\,230\,957\,031\,25$ | ], |  | [ |  $1038.115\,112\,304\,687\,5$, | $178.828\,125$ | ],
|  | [ |  $1017.592\,224\,121\,093\,8$, | $128.296\,600\,341\,796\,88$ | ], |  | [ |  $1018.846\,618\,652\,343\,8$, | $119.925\,735\,473\,632\,81$ | ],
|  | [ |  $1024.433\,959\,960\,937\,5$, | $112.637\,840\,270\,996\,1$ | ], |  | [ |  $1018.394\,958\,496\,093\,8$, | $112.405\,288\,696\,289\,06$ | ],
|  | [ |  $1014.329\,345\,703\,125$, | $132.561\,126\,708\,984\,38$ | ], |  | [ |  $1004.357\,666\,015\,625$, | $132.856\,384\,277\,343\,75$ | ],
|  | [ |  $997.795\,837\,402\,343\,8$, | $156.763\,687\,133\,789\,06$ | ], |  | [ |  $998.705\,932\,617\,187\,5$, | $178.382\,644\,653\,320\,3$ | ],
|  | [ |  $1009.314\,453\,125$, | $178.585\,784\,912\,109\,38$ | ], |  | [ |  $1009.512\,695\,312\,5$, | $220.118\,484\,497\,070\,3$ | ],
|  | [ |  $1007.401\,672\,363\,281\,2$, | $257.804\,534\,912\,109\,4$ | ], |  | [ |  $1006.501\,464\,843\,75$, | $261.912\,292\,480\,468\,75$ | ],
|  | [ |  $1006.014\,343\,261\,718\,8$, | $263.699\,493\,408\,203\,1$ | ], |  | [ |  $1023.316\,467\,285\,156\,2$, | $178.674\,911\,499\,023\,44$ | ],
|  | [ |  $1021.390\,930\,175\,781\,2$, | $220.229\,003\,906\,25$ | ], |  | [ |  $1017.535\,949\,707\,031\,2$, | $256.940\,521\,240\,234\,4$ | ],
|  | [ |  $1022.854\,370\,117\,187\,5$, | $261.495\,605\,468\,75$ | ], |  | [ |  $1025.192\,993\,164\,062\,5$, | $263.329\,132\,080\,078\,1$ | ]]
|  |  |  |  |  |  |  |  |  |

Table 4: Example data from a single row in data.csv (continued).

The example of captured ground truth data can be seen in Fig. 4, 5 and 6.

Figure 4: Example frame captured from RGB camera in Town10HD. Visible in the
frame is a female child (blueprint walker.pedestrian.0011).

Figure 5: The same frame as in Fig. 4, but captured by the semantic
segmentation camera and rendered using the Cityscapes palette.

Figure 6: (Above) The same frame as in Fig. 4, but with additional info
(including skeleton overlay) rendered using dataset_advanced_preview.ipynb
notebook from pedestrian_scenarios. (Below) Different frame from the same
video.

##  4 Example use cases

Our aim for creating and sharing this dataset is to encourage feedback. We
would also like to facilitate the comparison with other datasets in various
possible use cases. Therefore, the codebase was created to showcase our data’s
versatility in several applications. We connected some baseline models and
other commonly used sets to illustrate better how our data difficulty level
compares. In graphs and tables, we denote our dataset as CARLA-BSP.

The code, contained in pedestrians-video-2-carla repository, is written in
PyTorch [Paszke et al.(2019)Paszke, Gross, Massa, Lerer, Bradbury, Chanan,
Killeen, Lin, Gimelshein, Antiga, Desmaison, Kopf, Yang, DeVito, Raison,
Tejani, Chilamkurthy, Steiner, Fang, Bai, and Chintala], using PyTorch
Lightning [fal(2019)] as a framework. We also utilize PyTorch3D [Ravi et
al.(2020)Ravi, Reizenstein, Novotný, Gordon, Lo, Johnson, and Gkioxari] and
CameraTransform [Gerum et al.(2019)Gerum, Richter, Winterl, Mark, Fabry, Le
Bohec, and Zitterbart] libraries to provide crucial functionality.

Quality measures used to evaluate the models are described in Appendix A.

###  4.1 Crossing vs. non-crossing classification

One of the common tasks in analyzing the scene in the automotive domain is
deciding if the detected person will cross the street. Depending on the
dataset, there are usually two or three classes distinguished. The binary case
seems relatively straightforward – the pedestrian will cross or not. However,
even then, everyone needs to agree on what is meant by ,,crossing” – walking
from one side of the street to the other (even if separate from the future
path of the car)? Or is just entering the driving lane enough? Or maybe
crossing the planned car trajectory is required? What about the situations in
parking lots? What about cyclists? Should they be included, and under what
conditions?

When more classes are separated, the usual first criterion seems to be whether
the person comes into any interaction with the AV. If not, that pedestrian is
labeled as some kind of ,,irrelevant” class, and only pedestrians with any
interaction are separated into the crossing and non-crossing. Both of those
approaches to labeling are prone to create class imbalances, as shown in the
case of Joint Attention in Autonomous Driving (JAAD) [Rasouli et
al.(2017a)Rasouli, Kotseruba, and Tsotsos, Rasouli et al.(2017b)Rasouli,
Kotseruba, and Tsotsos] and Pedestrian Intention Estimation (PIE) [Rasouli et
al.(2019)Rasouli, Kotseruba, Kunic, and Tsotsos] datasets. Both datasets can
also be used with our codebase, with the ability to specify arbitrary clip
lengths and offsets or using the predefined benchmark data modules.

In our dataset, we mark the pedestrian crossing when he enters the driving
lane. All other situations are marked as non-crossing. As the scene is
recorded from the fixed point, there is no intersection of trajectories to
consider. Again, we provide two ways to utilize the data – the customizable
data module (CarlaRecorded) and the benchmark one (CarlaBenchmark), comparable
to benchmarks provided for JAAD and PIE [Kotseruba et al.(2021)Kotseruba,
Rasouli, and Tsotsos].

The state-of-the-art in pedestrian crossing and non-crossing classification
has seen considerable progress, driven by advances in deep learning and
diverse datasets. Several approaches have been developed, including temporal
models like the SRA-LSTM [Peng et al.(2021)Peng, Zhang, Shi, Xu, and Zheng],
which incorporates social relationship attention mechanisms to improve
trajectory prediction. CNN-based methods extract local features from image
patches for real-time classification. Spatial attention mechanisms, combine
attention with trajectory prediction to enhance understanding of pedestrian
behavior. There are also methods based on pose estimation and classification
MM-Pose [Contributors(2020)] which may also fuse multiple data modalities to
improve intention estimation. Graph-based methods, such as Social-STGCNN
[Mohamed et al.(2020)Mohamed, Qian, Elhoseiny, and Claudel], model
interactions between pedestrians and their environment for better insights
into crossing behavior. Despite progress, real-world challenges like noisy
data, occlusions, and complex interactions persist, emphasizing the need for
robust, reliable, and interpretable models in diverse scenarios.

We tested the classification flow using the 2D pose keypoints as the input. As
an example model, besides the basic LSTM architecture, we adapted the
Pedestrian Graph model [Cadena et al.(2019)Cadena, Yang, Qian, and Wang]. We
have used the benchmark setup for the experiments, with the clips’ length
equal to $16$ frames (a bit over $0.5\text{\,}\mathrm{s}\text{/}$). The
overlap between clips was ten frames. We also enabled class balancing during
the training, resulting in $344$ clips in each class. In the first test, the
data was perfect, i.e., no artificial noise or missing joints were introduced.
To our surprise, in that ideal setup, a simple LSTM model outperformed the
PedestrianGraph one (see Fig. 7).

Figure 7: Classification results in terms of F1-score when using perfect
CARLA-BSP data.

When we introduced artificial missing joints and gaussian noise, the
performance of the models became closer to that achieved for JAAD and PIE
datasets. This performance drop is partially due to the ground truth pose data
being incomplete and noisy. According to our tests, in the validation set of
the benchmark setup, JAAD is missing $85.74\text{\,}\mathrm{\char
37\relax}\text{/}$ of joints when using data provided in [Kotseruba et
al.(2021)Kotseruba, Rasouli, and Tsotsos] and $71.87\text{\,}\mathrm{\char
37\relax}\text{/}$ when using data we extracted using OpenPose [Cao et
al.(2021)Cao, Hidalgo Martinez, Simon, Wei, and Sheikh]. For PIE,
$93.28\text{\,}\mathrm{\char 37\relax}\text{/}$ keypoints is missing when
using data data from [Kotseruba et al.(2021)Kotseruba, Rasouli, and Tsotsos]
and $84.16\text{\,}\mathrm{\char 37\relax}\text{/}$ in data extracted using
OpenPose.

In the short experiments we have run ($400$ epochs), the performance of the
PedestrianGraph model on CARLA-BSP with requested missing joint probability
(MJP) $71.87\text{\,}\mathrm{\char 37\relax}\text{/}$ and gaussian noise with
STD=5.0 (added to ground truth pixel values), there was an F1-score drop of
$\approx 0.125$, which placed the results mid-way between the performance for
JAAD dataset and perfect CARLA data. Interestingly, a much smaller number of
missing joints ($\approx 0.3$) resulted in a similar performance. However,
when it was increased to $93.28\text{\,}\mathrm{\char 37\relax}\text{/}$, the
PedestrianGraph model failed completely (see Fig. 8).

Figure 8: Classification results for short experiments with PedestrianGraph
model.

LSTM seems to be more resilient to introducing artificial missing joints –
apart from the initial F1-score drop, the MJP value seems to make a minimal
difference. However, it performs slightly worse for JAAD and PIE data (see
Fig. 9).

Figure 9: Classification results for short experiments with LSTM model.

###  4.2 Pose autoencoder

Autoencoding is a powerful unsupervised learning technique that has emerged as
a popular first step in data processing pipelines, owing to its ability to
improve data quality and facilitate more effective downstream processing. In
the context of the JAAD dataset, autoencoding can be employed to enhance the
quality of skeletons extracted using OpenPose, a renowned human pose
estimation framework. By optimizing the representation of these skeletons, the
aim is to bolster the performance of classification tasks on the JAAD dataset,
which consists of annotated video sequences that focus on pedestrian behavior
and interactions in urban settings. By leveraging state-of-the-art
autoencoding techniques, it is possible to learn a compact and noise-resistant
representation of the input data, which can subsequently be used to refine the
extracted skeletal features [Carissimi et al.(2018)Carissimi, Rota, Beyan, and
Murino]. In turn, this refined representation is expected to yield improved
classification results, thus demonstrating the value of autoencoding as a
preliminary step in processing complex datasets such as JAAD.

Aside from the CARLA-BSP, as the input data we also used CMU [Carnegie Mellon
University()] and HumanEva [Sigal et al.(2010)Sigal, Balan, and Black] input
data, available as a part of AMASS [Mahmood et al.(2019)Mahmood, Ghorbani, F.
Troje, Pons-Moll, and Black], as well as JAAD and PIE. We used a simple
transformer-based model (with six layers and two heads) that managed to learn
clean CARLA-BSP and AMASS data almost perfectly.

Figure 10<EMAIL_ADDRESS>metrics during training for simple transformer-based
autoencoder and noisy CARLA-BSP and AMASS data contrasted with<EMAIL_ADDRESS>and
<EMAIL_ADDRESS>runs.  Figure 11<EMAIL_ADDRESS>metrics during training for simple
transformer-based autoencoder and noisy CARLA-BSP and AMASS data contrasted
with<EMAIL_ADDRESS>and<EMAIL_ADDRESS>runs.

The first conclusion we reached is a confirmation, that the amount of
available data plays a crucial role in the learning process. It can be
observed that while for the perfect data (CARLA-BSP and AMASS), the
convergence time and end quality of the model are very similar, the
introduction of the same amount of gaussian noise (STD=5.0) and artificial
missing joints (MJP=$29.44\text{\,}\mathrm{\char 37\relax}\text{/}$)
drastically changes the results (Fig. 10 and 11). The run using AMASS dataset,
containing $\approx 10$ times more training data, approached the ideal
results. In contrast, the one for CARLA-BSP resulted in a performance drop of
$\approx 10$ percentage points (p.p.) in terms of<EMAIL_ADDRESS>(percentage of
correct keypoints with a tolerance of $5\text{\,}\mathrm{\char
37\relax}\text{/}$ of bounding box; values are between 0.0 and 1.0) and almost
$18$ p.p. drop in terms of<EMAIL_ADDRESS>(percentage of correct keypoints with a
tolerance of $10\text{\,}\mathrm{\char 37\relax}\text{/}$ of hips-neck
distance; values are between 0.0 and 1.0).

This effect can also be noticed when comparing the runs on a strong_points=1.0
subset of PIE and JAAD datasets, where ’strong points’ denote the desired
percentage of non-zero, hopefully semi-correctly detected, keypoints in each
frame (denoted as<EMAIL_ADDRESS>and<EMAIL_ADDRESS>respectively). Run for<EMAIL_ADDRESS>eventually approaches the perfect metrics, while the one for<EMAIL_ADDRESS>reaches
plateau at<EMAIL_ADDRESS>$\approx 0.79$ and<EMAIL_ADDRESS>$\approx 0.56$. The subset of
<EMAIL_ADDRESS>data is $\approx 3$ times bigger than the one for<EMAIL_ADDRESS>
The second conclusion is that imperfections introduced during the pose
estimation process seem easier for the model to correct than the artificial
noise. This effect can be observed when comparing the<EMAIL_ADDRESS>run (which
eventually approaches perfect results) with the CARLA-BSP mentioned before
(Fig. 10), which resulted in a severe performance drop even though CARLA-BSP
has over $14$ times more training data than the<EMAIL_ADDRESS>subset!

The third conclusion is that training our simple transformer-based autoencoder
only on very incomplete<EMAIL_ADDRESS>or<EMAIL_ADDRESS>data (at least
$10\text{\,}\mathrm{\char 37\relax}\text{/}$ of data visible in each frame)
yields no usable results (see Fig. 12 and Fig. 13 for example outputs). The
maximal values of<EMAIL_ADDRESS>are $\approx 0.5$, and<EMAIL_ADDRESS>$\approx 0.3$. In
both cases, the model learns some approximation of the pose. However, it
significantly shifts some keypoints. Sometimes, the returned pose seems much
closer to some averaged pose and does not follow the actual person’s
movements, especially in the<EMAIL_ADDRESS>case.

Figure 12: Example<EMAIL_ADDRESS>rendering results. From left: fragment of a source
video with original pose data (red) and model output (green), normalized input
pose data as dots, model output as dots.

Figure 13: Example<EMAIL_ADDRESS>rendering results. From left: fragment of a source
video with original pose data (red) and model output (green), normalized input
pose data as dots, model output as dots.

In the next step, we trained the model on mixed datasets, meaning that the
model was shown data from either JAAD, CARLA-BSP or AMASS in a single epoch.
To make it possible, we mapped the various source pose data to common format
(CARLA skeleton one), which had a side effect of increasing the number of
missing joints and noise. This effect occurred because we only copied the
values from the existing close joints, without any additional processing.
Then, we tested all models on the same strong_points=1.0 subset of JAAD
<EMAIL_ADDRESS>and strong_points=1.0 subset of PIE (PIE@1.0). The results can be
seen in Tab. 5 and example rendered frames in the Fig. 4.2. It is important to
note however, that since<EMAIL_ADDRESS>and<EMAIL_ADDRESS>dataset variants were generated
independently, models trained using<EMAIL_ADDRESS>have seen about
$\nicefrac{{1}}{{3}}$ of<EMAIL_ADDRESS>test data during training, which likely
inflates the results.

| Train | Test<EMAIL_ADDRESS>| Test<EMAIL_ADDRESS>
---|---|---|---
(r)2-5(lr)6-7(l)8-9 |  Dataset | Proportions | Set size | Skeleton |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
1 |<EMAIL_ADDRESS>| $100\text{\,}\mathrm{\char 37\relax}\text{/}$ | $220$ | BODY_25 | $0.7635$ | $0.5581$ | $0.7865$ | $0.5558$
2 |<EMAIL_ADDRESS>| $100\text{\,}\mathrm{\char 37\relax}\text{/}$ | $8822$ | BODY_25 |  $0.4825$* |  $0.2642$* | $0.4965$ | $0.2605$
3 |<EMAIL_ADDRESS>/ CARLA-BSP / AMASS; MJP=$29.44\text{\,}\mathrm{\char 37\relax}\text{/}$ \+ gaussian noise (STD=5.0) for all datasets |  $10\text{\,}\mathrm{\char 37\relax}\text{/}$/$40\text{\,}\mathrm{\char 37\relax}\text{/}$/$50\text{\,}\mathrm{\char 37\relax}\text{/}$ | $2200$ | CARLA | $0.4898$ | $0.3214$ | $0.4246$ | $0.2414$
4 |<EMAIL_ADDRESS>/ CARLA-BSP / AMASS; no noise |  $10\text{\,}\mathrm{\char 37\relax}\text{/}$/$40\text{\,}\mathrm{\char 37\relax}\text{/}$/$50\text{\,}\mathrm{\char 37\relax}\text{/}$ | $21\,556$ | CARLA |  $0.4973$* |  $0.3214$* | $0.5352$ | $0.3019$
5 |<EMAIL_ADDRESS>/ CARLA-BSP / AMASS; MJP=$29.44\text{\,}\mathrm{\char 37\relax}\text{/}$ \+ gaussian noise (STD=5.0) for CARLA-BSP and AMASS datasets |  $10\text{\,}\mathrm{\char 37\relax}\text{/}$/$40\text{\,}\mathrm{\char 37\relax}\text{/}$/$50\text{\,}\mathrm{\char 37\relax}\text{/}$ | $21\,556$ | CARLA |  $0.3859$* |  $0.2240$* | $0.5048$ | $0.3152$
6 |  CARLA-BSP; MJP=$29.44\text{\,}\mathrm{\char 37\relax}\text{/}$ \+ gaussian noise (STD=5.0) | $100\text{\,}\mathrm{\char 37\relax}\text{/}$ | $8623$ | CARLA | $0.2146$ | $0.1130$ | $0.2162$ | $0.1063$
7 |  CARLA-BSP / AMASS; MJP=$29.44\text{\,}\mathrm{\char 37\relax}\text{/}$ \+ gaussian noise (STD=5.0) for all datasets |  $50\text{\,}\mathrm{\char 37\relax}\text{/}$ / $50\text{\,}\mathrm{\char 37\relax}\text{/}$ | $17\,246$ | CARLA | $0.1806$ | $0.1141$ | $0.1749$ | $0.1094$
|  |  |  |  |  |  |  |

* ca. $\nicefrac{{1}}{{3}}$ of the test data was seen during training.

Table 5: Setup and results for the tests of models trained using various
datasets.

1

2

3

4

5

6

7

Figure 14: Example rendering results for<EMAIL_ADDRESS>test set. Order (from the
top) follows rows in Tab. 5. Follow QR codes to see videos for each
pedestrian.

&utm_medium=latexutm_campaign=CARLA-BSP

[height=60pt]https://youtu.be/k9zPpkoWJhk?utm_source=QRCode&utm_medium=latexutm_campaign=CARLA-
BSP

[height=60pt]https://youtu.be/6GJB-
uLEpoo?utm_source=QRCode&utm_medium=latexutm_campaign=CARLA-BSP

It can be seen that training on<EMAIL_ADDRESS>yields the best results when testing
on<EMAIL_ADDRESS>and<EMAIL_ADDRESS>This may be due to the compatibility of the source
points data and the fact that there were no missing points in training.
However, when trained with missing data<EMAIL_ADDRESS>the results were
significantly worse. Interestingly, the visual examination of the results (see
Fig. 4.2 and linked videos) suggests that while autoencoder trained only on
CARLA-BSP / AMASS dataset combination (without seeing any JAAD data) scored
near the bottom in terms of<EMAIL_ADDRESS>and<EMAIL_ADDRESS>the generated pose is
often an approximation of the correct one. It seems to be deformed mainly due
to differences in points position when converted between BODY_25 and CARLA
skeletons – the difference between hips and neck positions makes the
denormalized pose taller than the actual one. On the other hand, using an
autoencoder trained only on CARLA-BSP seems to result in poses that are mostly
turned right from the observer’s point of view. This suggests an existence of
a bias in the training data, which should be eliminated in future CARLA-based
dataset versions.

###  4.3 Pose estimation

Pose estimation is another problem that can benefit from simulated data.
Historically, monocular images and videos were used for 2D pose estimation,
and 3D required specialized datasets obtained via motion capture technology or
at least stereoscopic images. Nowadays, many models offer at least basic 3D
pose estimation from monocular images and videos. Example pose estimation
solutions include OpenPose [Cao et al.(2021)Cao, Hidalgo Martinez, Simon, Wei,
and Sheikh] mentioned before, UniPose [Artacho and Savakis(2020)], AlphaPose
[Li et al.(2021)Li, Xu, Chen, Bian, Yang, and Lu], SMPLify-X [Pavlakos et
al.(2019)Pavlakos, Choutas, Ghorbani, Bolkart, Osman, Tzionas, and Black] and
others.

The advancements in 3D pose estimation from monocular images and videos have
been facilitated by the development of deep learning techniques, specifically
convolutional neural networks (CNNs) and their variants. These models have
been successful in capturing complex spatial dependencies and learning rich
feature representations from 2D images, which have proven beneficial for the
task of pose estimation. Additionally, the availability of large-scale
annotated datasets, such as MPII [Andriluka et al.(2014)Andriluka, Pishchulin,
Gehler, and Schiele], and Human3.6M [Ionescu et al.(2014)Ionescu, Papava,
Olaru, and Sminchisescu], has provided researchers with the necessary data to
train and fine-tune these models, resulting in improved performance on both 2D
and 3D pose estimation tasks.

Despite the progress made, pose estimation still faces several challenges,
such as occlusions, varying illumination, and diverse body shapes and
clothing. Simulated data can help address these challenges by providing a
controlled environment in which researchers can generate large amounts of
annotated data, manipulate object properties, and create a variety of
scenarios that are difficult or expensive to obtain with real-world data. For
example, synthetic human models can be generated with different body shapes,
clothing, and poses, while the virtual environment can be altered to simulate
different lighting conditions and occlusion patterns. This can lead to more
robust and generalizable models, as they are exposed to a broader range of
scenarios during training. Furthermore, simulated data can help overcome the
limitations of motion capture technology and stereoscopic images, which are
often costly and time-consuming to acquire. By leveraging simulated data,
researchers can accelerate the development of pose estimation solutions and
contribute to the advancement of the field.

Due to the limitations of computing power, we could not achieve any meaningful
results in the 2D pose estimation task. The models we ran reached a plateau of
about $0.5$<EMAIL_ADDRESS>and $0.35$<EMAIL_ADDRESS>The metrics are calculated based on
normalized skeletons aligned by hips, so they do not verify if the model even
managed to find a pedestrian in the frame. In practice, those results mean
that model learned the average skeleton position, as seen in Fig. 15.

Figure 15: Visualization of pose estimation results. It can be seen that the
model learned the average skeleton.

###  4.4 Pose lifting

Pose lifting can be considered a subset of the pose estimation tasks. In some
solutions, when trying to obtain a 3D pose, the 2D pose is estimated first,
and then, based on the 3D pose, it is lifted to 3D. Some of the example
solutions include 3DPoseBaseline [Martinez et al.(2017)Martinez, Hossain,
Romero, and Little] and PoseFormer [Zheng et al.(2021)Zheng, Zhu, Mendieta,
Yang, Chen, and Ding]. We trained both of those models, as well as LSTM, using
the CARLA-BSP dataset. All models were trained using the standard 2D (x,y)
$\rightarrow$ 3D (x,y,z) mapping that we refer to as the ’absolute_loc‘ model
output type. For the LSTM, we also tested the ’relative_rot’ mode – given the
2D (x,y) points, we expected the model to learn the rotations of each joint
relative to its parent in the kinematic tree.

We capped the training at $600$ epochs. The models reached close-to-final
performance after ca. $100$ epochs, although LSTM-based ones were still
showing small improvements until the end of training (see Fig. 16). As the
input, all models get 2D points normalized with hips-neck distance. The final
results, calculated on the test set, can be seen in Tab. 6 and the example
outputs in Fig. 3. We also experimented with loss functions; however, out of
all that we tested, the classic MSE calculated using normalized 3D input
points and model output was the most effective.

Figure 16: Change in MPJPE calculated on validation subset during the training.  Model |  Output type |  MPJPE [$\mathrm{mm}\text{/}$] |  N-MPJPE [$\mathrm{mm}\text{/}$] |  PA-MPJPE [$\mathrm{mm}\text{/}$] |  MPJVE [$\mathrm{mm}\text{/}$]
---|---|---|---|---|---
PoseFormer |  absolute_loc |  $85.54$ |  $82.855$ |  $23.563$ |  $26.218$
LSTM |  absolute_loc |  $109.359$ |  $107.939$ |  $64.029$ |  $29.567$
3DPoseBaseline |  absolute_loc |  $119.067$ |  $118.388$ |  $76.965$ |  $29.043$
LSTM |  relative_rot |  $131.905$ |  $131.871$ |  $99.412$ |  $29.596$
|  |  |  |  |
Table 6: Quality measures in pose lifting task calculated using a test set of
CARLA-BSP.

&utm_medium=latexutm_campaign=CARLA-BSP
PoseFormer
urlcolor=black[height=60pt]https://youtu.be/PM7B2ekQ5mo?utm_source=QRCode
&utm_medium=latexutm_campaign=CARLA-BSP
LSTM
urlcolor=black[height=60pt]https://youtu.be/POus_q63w-Q?utm_source=QRCode
&utm_medium=latexutm_campaign=CARLA-BSP
3DPoseBaseline
urlcolor=black[height=60pt]https://youtu.be/u8g8E30Yt3E?utm_source=QRCode
&utm_medium=latexutm_campaign=CARLA-BSP
LSTM (relative_rot)

Figure 17: Example outputs from pose lifting models. In visualization videos,
all model outputs are vertically flipped. Follow QR codes to see more
examples.

###  4.5 Other possible applications

Apart from the described cases, the synthetic datasets generated from CARLA
can also be used in other tasks. Some of them are obvious and well-researched,
like pedestrian (or people in general) detection and tracking. Others may
include scene semantic segmentation, motion retargeting, or style transfer. We
hope that increasing the complexity and realism of the generated pedestrians
and the world around them can open doors to even more exciting possibilities,
for example:

  * •

Advanced pedestrian behavior modeling: As AI and deep learning techniques
evolve, CARLA-based datasets can help create sophisticated pedestrian behavior
models, predicting their actions in real-time, resulting in safer driving
decisions by autonomous vehicles.

  * •

Adversarial training: The setup can be leveraged to create adversarial
scenarios that challenge the robustness of autonomous vehicle systems. By
exposing AI models to a variety of difficult situations, developers can ensure
that self-driving cars respond appropriately to extreme or uncommon events.

  * •

Multi-modal sensor fusion: By incorporating various sensor modalities like RGB
cameras, DVS cameras, LIDAR, and others, the setup enables the development of
algorithms that fuse data from multiple sources for more accurate perception
and decision-making.

  * •

Ethics and AI decision-making: The setup can be employed to develop and test
AI decision-making algorithms that consider ethical dilemmas and incorporate
human values, striking a balance between safety, efficiency, and fairness in
real-world scenarios.

##  5 Conclusions and future work

The generation of synthetic datasets for pedestrian crossing and non-crossing
detection problems offers a multitude of advantages, particularly in the
context of multimodal sensing and the increasing complexity of real-world
scenes. One of the primary benefits of synthetic datasets is their ability to
capture a diverse range of scenarios, environmental conditions, and pedestrian
behaviors that may not be readily available in real-world datasets. This is
crucial for training robust machine learning models capable of handling the
variability and intricacies of pedestrian crossing situations.

A key aspect of our future work will involve integrating multiple sensor
modalities, such as LiDAR, cameras, and radar, to form a holistic
representation of the environment. By fusing the complementary strengths of
these sensing modalities, we aim to build more robust and accurate pedestrian
detection models that can seamlessly adapt to the dynamic nature of urban
environments. The multimodal approach will also aid in addressing challenges
related to occlusions, varying lighting conditions, and other complexities
commonly encountered in real-world pedestrian crossing scenarios.

## Funding

The project on which these results are based has received funding from the
European Union’s Horizon 2020 research and innovation programme under Marie
Skłodowska-Curie grant agreement No. 801342 (Tecniospring INDUSTRY) and the
Government of Catalonia’s Agency for Business Competitiveness (ACCIÓ).

This work only expresses the opinion of the author and neither the European
Union nor ACCIÓ are liable for the use made of the information provided.

##  Appendix A Quality measures

###  A.1 Classification

In binary classification tasks usually we are interested only in the model
performance for the positive class, which in our case means that a pedestrian
is going to cross. Due to class imbalance inherent to most of the used
datasets, the standard accuracy measure is meaningless. Therefore, we only
report the results using positive-class F1-score, which is a compound metric
that combines precision and recall (also known as sensitivity).

$\mathrm{recall}=\frac{\mathit{tp}}{\mathit{tp}+\mathit{fn}},$ (1)
$\mathrm{precision}=\frac{\mathit{tp}}{\mathit{tp}+\mathit{fp}},$ (2)

where:

  * •

$\mathit{tp}$ – true positive – item correctly classified as belonging to
positive class,

  * •

$\mathit{fp}$ – false positive – item incorrectly classified as belonging to
positive class,

  * •

$\mathit{fn}$ – false negative – item incorrectly classified as belonging to
negative class.

The $\beta$ parameter controls the recall importance in relevance to the
precision when calculating an $\mathrm{F}_{\beta}$-measure:

$\mathrm{F}_{\beta}=(1+\beta^{2})\cdot\frac{\mathrm{recall}\cdot\mathrm{precision}}{\mathrm{recall}+\beta^{2}\cdot\mathrm{precision}}.$
(3)

During the experiments, we used $\beta=1$, which puts equal emphasis on both
components. This metric is known as F1-score.

###  A.2 Pose

####  A.2.1 PCK (and PCKhn variant)

The Percentage of Correct Keypoints is defined with some level of tolerance,
usually with respect to the bounding box. In our experiments, we have used a
value of $5\text{\,}\mathrm{\char 37\relax}\text{/}$ of the bounding box,
denoted as<EMAIL_ADDRESS>We also used a bit more restrictive<EMAIL_ADDRESS>variant,
which uses $10\text{\,}\mathrm{\char 37\relax}\text{/}$ of hips-neck distance
as a tolerance value.

$\mathrm{PCK}(\alpha,d)=\frac{1}{N}\sum_{i=1}^{N}1\text{~{}if~{}}\left\lVert\frac{\mathrm{pred}_{i}-\mathrm{gt}_{i}}{S}\right\rVert\leq\alpha\text{~{}else~{}}0$
(4)

  * •

$\alpha$: tolerance level.

  * •

$N$: The number of keypoints in the human body pose. Usually, only visible
keypoints are included in the calculation, which means that the set can differ
from frame to frame.

  * •

$i$: Index of a specific keypoint, ranging from 1 to N.

  * •

$\mathrm{pred}_{i}$: The predicted coordinates of keypoint $i$ from the pose
estimation algorithm. Can be 2D or 3D.

  * •

$\mathrm{gt}_{i}$: The ground truth coordinates of keypoint $i$ from the
annotated dataset.

  * •

$S$: The normalization factor – usually the bounding box size or a specific
body part distance (e.g., hips-neck distance).

####  A.2.2 MPJPE

Mean Per-Joint Position Error is used to quantify the error between the
predicted joint positions and the ground truth joint positions. It is
calculated as the mean Euclidean distance between the corresponding joint
positions of the predicted pose and the ground truth pose, usually measured in
millimeters.

$\text{MPJPE}=\frac{1}{N}\sum_{i=1}^{N}\left\lVert\mathrm{pred}_{i}-\mathrm{gt}_{i}\right\rVert$
(5)

####  A.2.3 N-MPJPE

The Normalized Mean Per-Joint Position Error metric is calculated similarly to
MPJPE, but with an additional normalization step to account for variations in
scale between different poses. This is especially useful when comparing errors
across datasets or subjects with different body sizes.

$\text{N-MPJPE}=\frac{1}{N}\sum_{i=1}^{N}\left\lVert
S\cdot\mathrm{pred}_{i}-\mathrm{gt}_{i}\right\rVert$ (6)

We’ve used the metric version from [Pavllo et al.(2019)Pavllo, Feichtenhofer,
Grangier, and Auli], which itself adapts from [Rhodin et al.(2018)Rhodin,
Salzmann, and Fua].

####  A.2.4 PA-MPJPE

The Procrustes Aligned Mean Per-Joint Position Error metric involves aligning
the predicted and ground truth poses using a similarity transformation
(translation, rotation, and scaling) before calculating the MPJPE. This
alignment removes any global differences between the poses, allowing the
evaluation to focus on local joint estimation errors.

We’ve used the metric version from [Pavllo et al.(2019)Pavllo, Feichtenhofer,
Grangier, and Auli].

####  A.2.5 MPJVE

MPJVE (Mean Per-Joint Velocity Error) is a metric used to evaluate the
performance of human pose estimation algorithms in predicting the velocities
of body joints in 3D space. This metric is particularly relevant for
applications such as motion capture, where the estimation of joint velocities
is essential for understanding human motion.

We’ve used the metric version from [Pavllo et al.(2019)Pavllo, Feichtenhofer,
Grangier, and Auli].

## References

  * [Dosovitskiy et al.(2017)Dosovitskiy, Ros, Codevilla, López, and Koltun] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio López, and Vladlen Koltun.  CARLA: An open urban driving simulator.  In _Proceedings of the 1st Annual Conference on Robot Learning_ , pages 1–16, 2017.
  * [Xu(2019)] Minjun Xu.  CarlaViz, 2019.  URL https://github.com/mjxu96/carlaviz.  Accessed on 2021-08-24.
  * [Paszke et al.(2019)Paszke, Gross, Massa, Lerer, Bradbury, Chanan, Killeen, Lin, Gimelshein, Antiga, Desmaison, Kopf, Yang, DeVito, Raison, Tejani, Chilamkurthy, Steiner, Fang, Bai, and Chintala] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  PyTorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc., 2019.
  * [fal(2019)] PyTorch Lightning, Mar 2019.  URL https://www.pytorchlightning.ai.  Accessed on 2022-09-14.
  * [Ravi et al.(2020)Ravi, Reizenstein, Novotný, Gordon, Lo, Johnson, and Gkioxari] Nikhila Ravi, Jeremy Reizenstein, David Novotný, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari.  Accelerating 3D deep learning with PyTorch3D.  Technical report, Facebook AI Research, 2020.  Accessed on 2021-09-18.
  * [Gerum et al.(2019)Gerum, Richter, Winterl, Mark, Fabry, Le Bohec, and Zitterbart] Richard C. Gerum, Sebastian Richter, Alexander Winterl, Christoph Mark, Ben Fabry, Céline Le Bohec, and Daniel P. Zitterbart.  CameraTransform: A Python package for perspective corrections and image mapping.  _SoftwareX_ , 10:100333, 2019.  ISSN 2352-7110.  10.1016/j.softx.2019.100333.
  * [Rasouli et al.(2017a)Rasouli, Kotseruba, and Tsotsos] Amir Rasouli, Iuliia Kotseruba, and John K Tsotsos.  Are they going to cross? A benchmark dataset and baseline for pedestrian crosswalk behavior.  In _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , pages 206–213, 2017a.  10.1109/ICCVW.2017.33.
  * [Rasouli et al.(2017b)Rasouli, Kotseruba, and Tsotsos] Amir Rasouli, Iuliia Kotseruba, and John K Tsotsos.  Agreeing to cross: How drivers and pedestrians communicate.  In _IEEE Intelligent Vehicles Symposium (IV)_ , pages 264–269, 2017b.  10.1109/IVS.2017.7995730.
  * [Rasouli et al.(2019)Rasouli, Kotseruba, Kunic, and Tsotsos] Amir Rasouli, Iuliia Kotseruba, Toni Kunic, and John K. Tsotsos.  PIE: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction.  In _International Conference on Computer Vision (ICCV)_ , 2019.  10.1109/ICCV.2019.00636.
  * [Kotseruba et al.(2021)Kotseruba, Rasouli, and Tsotsos] Iuliia Kotseruba, Amir Rasouli, and John K Tsotsos.  Benchmark for Evaluating Pedestrian Action Prediction.  In _Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 1258–1268, 2021.  10.1109/WACV48630.2021.00130.
  * [Peng et al.(2021)Peng, Zhang, Shi, Xu, and Zheng] Yusheng Peng, Gaofeng Zhang, Jun Shi, Benzhu Xu, and Liping Zheng.  Sra-lstm: Social relationship attention lstm for human trajectory prediction, 2021.
  * [Contributors(2020)] MMPose Contributors.  Openmmlab pose estimation toolbox and benchmark.  https://github.com/open-mmlab/mmpose, 2020.
  * [Mohamed et al.(2020)Mohamed, Qian, Elhoseiny, and Claudel] Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel.  Social-stgcnn: A social spatio-temporal graph convolutional neural network for human trajectory prediction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14424–14432, 2020.
  * [Cadena et al.(2019)Cadena, Yang, Qian, and Wang] Pablo Rodrigo Gantier Cadena, Ming Yang, Yeqiang Qian, and Chunxiang Wang.  Pedestrian graph: Pedestrian crossing prediction based on 2d pose estimation and graph convolutional networks.  In _2019 IEEE Intelligent Transportation Systems Conference (ITSC)_ , pages 2000–2005, 2019.  10.1109/ITSC.2019.8917118.
  * [Cao et al.(2021)Cao, Hidalgo Martinez, Simon, Wei, and Sheikh] Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh.  OpenPose: Realtime multi-person 2D pose estimation using part affinity fields.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 43:172–186, Jan 2021.  10.1109/TPAMI.2019.2929257.
  * [Carissimi et al.(2018)Carissimi, Rota, Beyan, and Murino] Nicoló Carissimi, Paolo Rota, Cigdem Beyan, and Vittorio Murino.  Filling the gaps: Predicting missing joints of human poses using denoising autoencoders.  In _ECCV Workshops_ , 2018.
  * [Carnegie Mellon University()] Carnegie Mellon University.  CMU MoCap Dataset.  URL http://mocap.cs.cmu.edu.
  * [Sigal et al.(2010)Sigal, Balan, and Black] L. Sigal, A. Balan, and M. J. Black.  HumanEva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion.  _International Journal of Computer Vision_ , 87(4):4–27, March 2010.  10.1007/s11263-009-0273-6.
  * [Mahmood et al.(2019)Mahmood, Ghorbani, F. Troje, Pons-Moll, and Black] Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black.  AMASS: Archive of Motion Capture as Surface Shapes.  In _The IEEE International Conference on Computer Vision (ICCV)_ , pages 5442–5451, Oct 2019.  URL https://amass.is.tue.mpg.de.
  * [Artacho and Savakis(2020)] Bruno Artacho and Andreas Savakis.  Unipose: Unified human pose estimation in single images and videos.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 7035–7044, June 2020.
  * [Li et al.(2021)Li, Xu, Chen, Bian, Yang, and Lu] Jiefeng Li, Chao Xu, Zhicun Chen, Siyuan Bian, Lixin Yang, and Cewu Lu.  Hybrik: A hybrid analytical-neural inverse kinematics solution for 3d human pose and shape estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3383–3393, 2021.
  * [Pavlakos et al.(2019)Pavlakos, Choutas, Ghorbani, Bolkart, Osman, Tzionas, and Black] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black.  Expressive body capture: 3D hands, face, and body from a single image.  In _Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 10975–10985, 2019.
  * [Andriluka et al.(2014)Andriluka, Pishchulin, Gehler, and Schiele] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele.  2d human pose estimation: New benchmark and state of the art analysis.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2014.
  * [Ionescu et al.(2014)Ionescu, Papava, Olaru, and Sminchisescu] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu.  Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 36(7):1325–1339, jul 2014.
  * [Martinez et al.(2017)Martinez, Hossain, Romero, and Little] Julieta Martinez, Rayat Hossain, Javier Romero, and James J. Little.  A simple yet effective baseline for 3d human pose estimation.  In _ICCV_ , 2017.
  * [Zheng et al.(2021)Zheng, Zhu, Mendieta, Yang, Chen, and Ding] Ce Zheng, Sijie Zhu, Matias Mendieta, Taojiannan Yang, Chen Chen, and Zhengming Ding.  3d human pose estimation with spatial and temporal transformers.  In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 11636–11645, 2021.  10.1109/ICCV48922.2021.01145.
  * [Pavllo et al.(2019)Pavllo, Feichtenhofer, Grangier, and Auli] Dario Pavllo, Christoph Feichtenhofer, David Grangier, and Michael Auli.  3d human pose estimation in video with temporal convolutions and semi-supervised training.  In _Proceedings of IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [Rhodin et al.(2018)Rhodin, Salzmann, and Fua] Helge Rhodin, Mathieu Salzmann, and Pascal Fua.  Unsupervised geometry-aware representation learning for 3d human pose estimation.  In _Proceedings of European Conference on Computer Vision (ECCV)_ , 2018.