# JRDB-Pose: A Large-scale Dataset for Multi-Person Pose Estimation and
Tracking

Edward Vendrow1 , Duy Tho Le211footnotemark: 1 , Jianfei Cai2, Hamid
Rezatofighi2
1Stanford University, 2Monash University
<EMAIL_ADDRESS>{tho.le<EMAIL_ADDRESS>equal
contribution

###### Abstract

Autonomous robotic systems operating in human environments must understand
their surroundings to make accurate and safe decisions. In crowded human
scenes with close-up human-robot interaction and robot navigation, a deep
understanding of surrounding people requires reasoning about human motion and
body dynamics over time with human body pose estimation and tracking. However,
existing datasets captured from robot platforms either do not provide pose
annotations or do not reflect the scene distribution of social robots. In this
paper, we introduce JRDB-Pose, a large-scale dataset and benchmark for multi-
person pose estimation and tracking. JRDB-Pose extends the existing JRDB which
includes videos captured from a social navigation robot in a university campus
environment, containing challenging scenes with crowded indoor and outdoor
locations and a diverse range of scales and occlusion types. JRDB-Pose
provides human pose annotations with per-keypoint occlusion labels and track
IDs consistent across the scene and with existing annotations in JRDB. We
conduct a thorough experimental study of state-of-the-art multi-person pose
estimation and tracking methods on JRDB-Pose, showing that our dataset imposes
new challenges for the existing methods. JRDB-Pose is available at
https://jrdb.erc.monash.edu/.

##  1 Introduction

Dataset | # Poses | # Boxes | Tracks | Crowd | ppF | Occlusion | Action |  Indoor +
Outdoor |  Robot
Navigation |  Multi-
Modal |  Multi-
Task
---|---|---|---|---|---|---|---|---|---|---|---
MPII [3] | 40K |  |  |  | 1-17 |  |  |  ✓ |  |  |
Penn Action [54] | 160k | 160k |  |  | 1 |  | ✓ |  ✓ |  |  |
COCO [26] | 250k | 500k |  |  | 1-20 | ✓ |  |  ✓ |  |  |  ✓
KITTI [18] |  | 80k | ✓ | ✓ |  | ✓ |  |  |  |  ✓ |  ✓
H3D [35] |  | 460k | ✓ | ✓ |  |  |  |  |  |  ✓ |
MOT20 [12] |  | 1.65M | ✓ | ✓ |  | ✓ |  |  ✓ |  |  |
THÖR [41] |  | 2.5M | ✓ |  |  |  |  |  |  ✓ |  ✓ |
PoseTrack21 [13] | 177k | 429k | ✓ | ✓ | 1-13 | ✓ |  |  ✓ |  |  |
Waymo [46] | 173K | 9.9M | ✓ | ✓ | unk | ✓ |  |  |  |  ✓ |  ✓
JRDB-Pose | 636k | 2.8M | ✓ | ✓ | 1-36 | ✓ | ✓ |  ✓ |  ✓ |  ✓ |  ✓
JTA† [16] | 10M |  |  | ✓ | 0-60 | ✓ |  |  ✓ |  |  |
MotSynth† [16] | 40M |  | ✓ | ✓ | 0-125 | ✓ |  |  ✓ |  |  |
Table 1: Comparison of existing public datasets related to 2D pose estimation
and tracking. For each dataset we report the numbers of poses, boxes, as well
as the availability of tracking information, crowd data, people per frame
(ppF), occlusion labels, action labels, scene type, and if the data comes from
robot navigation in human environments. We mark if a dataset has data
modalities besides RGB frames, and if it contains annotations for multi-task
types. Note that JRDB-Pose is a multi-modal dataset captured from a social
navigation robot, addressing different research challenges than many existing
works. †Synthetic dataset. unk: Unknown.

Visual scene understanding of human environments is a difficult and crucial
task for autonomous driving, human-robot interaction, safe robotic navigation,
and human action recognition. Although rough predictions of human location are
sufficient for some applications, a deep understanding of crowded human scenes
and close-up human-robot interaction requires reasoning about human motion and
body dynamics with human body pose estimation and tracking. Developing an AI
model to predict human body pose is made more difficult by the varied and
highly imbalanced range of human motion found in daily living environments,
including a variety of scales, occlusions, and overlapping humans,
representing a long-tailed distribution of human poses which is difficult for
existing methods.

Figure 1: JRDB-Pose provides high frequency annotations of tracks and body
joints in long scenes of crowded indoor and outdoor locations featuring
dynamic motion and occlusion.

Human pose estimation and tracking is an active research area with many new
large-scale datasets [13, 46, 26, 16] contributing to significant recent
progress; however, these datasets do not primarily target robotic perception
tasks in social navigation environments, and thus rarely reflect specific
challenges found in human-robot interaction and robot navigation in crowded
human environments, _e.g_. shopping malls, university campus, _etc_.

JRDB [28] previously introduced a large-scale dataset and a benchmark for
research in perception tasks related to robotics in human environments. The
dataset was captured using a social manipulator robot with a multi-modal
sensor suite including a stereo RGB 360° cylindrical video stream, 3D point
clouds from two LiDAR sensors, audio and GPS positions. JRDB [28] additionally
introduced annotations for 2D bounding boxes and 3D oriented cuboids.
Recently, JRDB-Act [15] further introduced new annotations on the JRDB videos
for individual actions, human social group formation, and social activity of
each social group.

JRDB was collected from a robotic navigation platform in crowded human
environments, diversely capturing both indoor and outdoor scenes. Additionally
since the robot’s camera is located at person-level, and moves around, the
data is not just collected from a far-off view but captures close-up scenes.

For robotic systems to safely navigate dynamic human environments and perform
collision risk prediction, they must be able to accurately track and forecast
motion of people in their surroundings. Human motion is often fast and
requires high frame rate data for accurate prediction and tracking, making
high-frequency annotated human pose data crucial for the development and
evaluation of robotic perception systems in human environments. Complex social
interactions add difficulty and similarly benefit from high-frequency data. In
crowded scenes with high levels of occlusions or overlap with other humans,
tracking may be also difficult.

We introduce JRDB-Pose, a large-scale dataset captured from a mobile robot
platform containing human pose and head box annotations. JRDB-Pose provides
600k pose annotations and 600k head box annotations, each with an associated
tracking ID. JRDB-Pose includes a wide distribution of pose scales and
occlusion levels, each with per-keypoint occlusion labels and consistent
tracking IDs across periods of occlusion. The combination of JRDB-Pose with
JRDB and JRDB-Act forms a valuable multi-modal dataset providing a
comprehensive suite of annotations suited for robotic interaction and
navigation tasks.
Our contributions are:

  * •

We introduce JRDB-Pose, a large-scale pose estimation and tracking dataset
providing pose annotations and head boxes with tracking IDs and per-keypoint
occlusion labels.

  * •

In addition to adopting the popular metrics, we introduce new metrics, OSPA-
Pose and OSPA(2)-Pose for pose estimation and tracking, respectively.

  * •

We conduct a comprehensive evaluation of state-of-the-art methods on JRDB-Pose
and discuss the strengths and weaknesses of existing methods.

##  2 Related Datasets

We summarize the commonly used public datasets for pose estimation [47, 54,
26, 3, 26], pose tracking [13, 46] and multi-object tracking [18, 12, 35, 41]
in Tab. 1.

Figure 2:  Various statistic for JRDB-Pose, which provides visibility labels
and track ids consistent across long periods of occlusion. From left to right:
1) The distribution of track lengths, with the long tail truncated. 2) A log-
scaled distribution showing all track lengths. JRDB-Pose track lengths are
varied and as high as 1700 frames long. 3) Number of pose annotations in each
panoramic frame. 4) Number of keypoints in each pose annotated as visible.

(a) Train set

(b) Test set

Figure 3: Distribution of keypoint visibility annotations in the JRDB-Pose
train/validation and test splits. While most joints are visible, JRDB-Pose
contains a large number of occluded and invisible joints.

Single and Multi-person Pose Estimation Datasets: The task of 2D pose
estimation involves predicting the pixel locations of human skeleton keypoints
on an image. The MPII Human Pose Dataset [3] is a popular multi-person pose
estimation dataset and benchmark including videos from everyday human
activities, and the larger Penn Action dataset [54] provides both pose and
action annotations in sports settings with a single pose per frame. MS COCO
Keypoints Challenge [26] proposed a large-scale dataset including a diverse
set of scenes and occlusion labels. All of these datasets label individual
frames, limiting them to single-frame pose estimation. Human3.6M [20]
annotates videos and single-person 3D poses from a controlled indoor scene,
while 3DPW [47] provides estimated 3D human meshes in-the-wild with up to 2
people per frames. In comparison, JRDB-Pose is captured from a robotic
platform in-the-wild with crowded and manually-annotated indoor and outdoor
scenes, addressesing a different set of challenges. JRDB-Pose also includes
diverse data modalities including cylindrical video, LiDAR point clouds, and
rosbags.

Multi-Person and Multi-Object Tracking Datasets: Multi-Person Pose Tracking
(MPPT) and Multi-Object Tracking (MOT) are crucial tasks in robotic perception
and navigation, where the challenge is to track, across a video, the body
keypoints of individuals or the locations of objects, respectively. MOT has
attracted significant attention from the community with large-scale challenges
[25, 30, 12], and MPPT is increasingly becoming recognized for its
significance in human activity understanding and human-object interaction.
Despite high performance on easy scenes, current MPPT methods struggle in
crowded environments with occlusions and scale fluctuations.

Posetrack 2018 [2] introduced a benchmark for pose estimation and tracking
using annotated video sequences featuring a variety of in-the-wild scenarios
such as sports and dancing. Since PoseTrack was not densely labeled in crowded
scenes, it required large “ignore regions” contains regions of images and
videos lacking annotations. More recently, Posetrack21 [13] used the videos
from PoseTrack 2018 to provide dense annotations of crowded scenes. Both
datasets target diverse pose tracking scenarios, containing videos such as
surveillance-like footage that are different from typical robotic navigation
and human-robot interaction scenarios. Additionally, each video sequence is
limited to five seconds, while sequences in JRDB-Pose are up to 2 minutes
long. JRDB-Pose is a challenging dataset and benchmark for real-world tracking
tasks, containing videos captured in crowded environments with humans fading
into and emerging from the crowd.

Figure 4: A portion of the panoramic frame with annotated pose instances. We
show some of our tracking annotations by visualizing pose histories for two
people. The gaps in the pose history correspond to periods of occlusion
denoted by the numbers marking the length of the occlusion in frames. JRDB-
Pose provides track IDs which are consistent even across long periods of
occlusion.

Autonomous Driving Datasets: Recently, autonomous vehicle datasets have been
released featuring large-scale and detailed annotations of the surrounding
environment, posing difficult MOT and MPPT challenges. H3D [35] annotates
human detections and tracks. More recently, the Waymo Open Dataset [46] filled
the gap by providing 172.6K human pose annotations from their autonomous
vehicle navigating outdoor road and highway environments. While the data is
valuable for driving applications, the dataset contains exclusively outdoor
road and highway environments making it unsuitable for non-driving robotic
tasks, _e.g_. navigating a crowded shopping mall.

Robotic Navigation: Robotics operating in a social environment must learn to
navigate in a socially-compliant manner by detecting and reacting to other
agents in the environment. Existing datasets for robotic navigation [28, 41,
23] provide relatively few annotations and are often created in limited
environment. The THÖR dataset provides human trajectories on a ground plane,
and is filmed in a controlled indoor environment. SCAND [23] captures a
variety of crowded indoor and outdoor environments without annotations for
human detection or tracking. In human environments with complex interactions,
it is crucial to understand human motion more deeply than a bounding box. To
this end, JRDB-Pose is one of the first datasets providing large-scale pose
annotations in a robotic navigation environment.

Synthetic Datasets: Several recent works have proposed large synthetic
datasets for human pose estimation and tracking by generating data using a
video game engine. The Joint Track Auto (JTA) [17] dataset generates 10M
annotated human body poses, which MotSynth [16] extends to 40M human poses
with associated track IDs. While synthetic datasets have been shown to be
helpful to the existing frameworks’ performance boost when combined with the
real-world datasets [16], they may not reflect underlying biases and
distributions in real-world data, preventing them from being the only training
data resources for evaluating approaches in real-world applications.
State-of-the-art Frameworks: Multi-person pose estimation involves predicting
body keypoint locations for all people in an image and identifying which
keypoints belong to which individuals. There are two common types of
approaches to this problem: top-down methods first detect all people and then
execute pose estimation for each individual [49, 24, 34, 53], while bottom-up
methods first identify keypoints directly and then group them into individual
people [7], including disentangled representations [19], associative embedding
[32], HGG [22], YOLO-Pose [27], and CID [48].

Multi-person pose tracking approaches [34, 53, 7, 44] often identify keypoints
or poses using a pose estimation method, and then predict tracks using the
estimated keypoints. Top-down techniques like openSVAI [34] and PGG [21]
decompose the task into three discrete stages: human detection, pose
estimation, and then pose tracking. Recently, UniTrack [50] proposed a unified
framework for multiple object and keypoint tracking utilizing shared general-
purpose appearance models.

The multi-object tracking task can be categorized into tracking by detection
(TBD) [55, 51, 8, 14, 42] and joint detection and tracking (JDT) [5, 56, 45,
29, 52, 11, 57]. SORT-based[6] approaches [51, 8, 14] have gained popularity
with methods including DeepSORT [51] and StrongSORT[14] In addition, OC-Sort
[8] improved these methods by proposing an enhanced filter and recovery
strategy suitable for tracking non-linearly moving and frequently occluded
objects. Recently, ByteTrack [55] achieved efficient and precise tracking
results using a general association technique that utilizes nearly every
detection box. Popular JDT methods include Tracktor++[5] which uses integrated
detection-tracking modules and CenterTrack[56] which employs two-frames
tracking algorithm for real-time and accurate tracking. Overall, since
significant research in multi-person pose and multi-object tracking focuses on
refining track consistency, occlusion recovery, and missing and false
detection handling, JRDB-Pose is a useful and challenging benchmark for
existing and future works.

##  3 The JRDB-Pose Dataset

We create JRDB-Pose using all videos from the JRDB dataset [28], including 64
minutes of sensory data consisting of 54 sequences captured from indoor and
outdoor locations in a university campus environment. JRDB-Pose includes
57,687 annotated panoramic (3760 x 480) frames, containing total of 636k
annotated pose instances with 11 million labeled keypoints with occlusion
labels for each keypoint. We annotate 17 keypoints for each body pose
including head, eyes, neck, shoulders, center-shoulder, elbows, wrists, hips,
center-hips, knees and ankles. Each pose includes a tracking ID which is
consistent for each person across the entire scene, including across long
periods of full occlusion (see Fig. 4), and is also consistent with person IDs
for existing 2D and 3D bounding box annotations from JRDB. Fig. 2 shows the
distribution of track lengths present in JRDB-Pose. In total, JRDB-Pose
provides 5022 pose tracks with an average length of 124 frames and some tracks
as long as 1700 frames.

Figure 5: Distribution of bounding box scales in the train and validation scenes. JRDB-Pose contains a wide distribution of pose scale that is different for each scene. Annotations | Quantity | Track IDs
---|---|---
2D Body Bounding Box [28] | 2,400,000 | ✓
3D Body Bounding Box [28] | 1,800,000 | ✓
Atomic Action [15] | 2,800,000 | N/A
Social Group [15] | 600,000 | N/A
2D Head Bounding Box | 600,000 | ✓
2D Body Pose | 600,000 | ✓
Table 2: A summary of annotations in by JRDB [28], JRDB-Act [15], and JRDB-
Pose, which together provide a unique multi-modal dataset suitable for multi-
person detection, pose estimation tracking, and activity recognition. Bolded
annotations are introduced in this paper.  Figure 6:  Bounding box
distribution of the train/validation and test splits. JRDB-Pose contains a
wide distribution of pose scales.

We started our annotation process using the JRDB bounding boxes and tracking
IDs for each person in the scene. Annotators used an in-house annotation tool
to manually label human poses, and all the annotations were carefully quality-
assessed multiple times by several annotators to ensure high-quality and
consistent annotations. We annotated all persons from the 5 camera views of
the 360º cylindrical video stream that had either a large enough bounding box
(area above 6000 pixels) or clear keypoint locations. Note that this means not
all visible people are labeled, especially those far away from the robot.
JRDB-Pose labels poses for a wide range of scales, as shown in Fig. 6. We
visualize the pose scales for each training and validation scene in Fig. 5,
showing that the distribution of scales also varies significantly by scene as
the types human motions and activities vary. Details on testing scenes are not
shown because test data is held out.

For researchers interested in panoramic views, we merge the annotations from
the 5 camera views into a single 360º panoramic image following the merging
protocol used in JRDB [28]. If a joint is labeled in two views, we use the
label from the joint with a higher visibility score; if the joints have
identical visibility scores, we use the pose with higher overall visibility.
Poses were annotated at 7.5 Hz and finally upsampled to 15 Hz, providing
accurate high-frequency annotations without significant jitter.

JRDB [28] and JRDB-Act [15] previously introduced annotations including 2D and
3D bounding boxes with tracking IDs, atomic action labels, and social groups.
Together with JRDB and JRDB-Act annotations, JRDB-Pose is a multi-task
learning dataset for human detection/tracking, pose estimation/tracking,
individual action, social group, and social activity detection. Tab. 2
summarizes all annotations now available on videos from JRDB.

Occlusion: Occlusion is a key problem for human pose estimation and tracking
because it causes incomplete information for predicting keypoints, leading to
major errors in pose estimation tasks [33]. Significant research in the field
of pose estimation has focused on occlusion-aware methods for improving pose
estimation [1, 10, 9], but may be limited by lack of quantity and diversity in
the existing data [9]. JRDB-Pose advances this research by providing large-
scale data including occluded scenarios, as well as detailed occlusion labels
for every keypoint which we hope will be useful for quantifying and improving
performance under occlusion. In particular, we assign each keypoint a label in
$\\{0,1,2\\}$ defining its occlusion type, where for every pose we annotate
the posistion and occlusion of each keypoint:

ID |  Meaning |  Description
---|---|---
0 |  Invisible |  The joint is out of frame or especially difficult to annotate, with a location inferred from context by annotators.
1 |  Occluded |  The joint is obscured by an object or another body part, but its location is apparent from the image context.
2 |  Visible |  The joint is fully visible and in view of the camera.

Figure 3 shows the occlusion label distribution in the train and test splits.
JRDB-Pose contains a large number of labeled occluded and invisible joints,
making it suitable for pose estimation under occlusion. Occlusion varies by
scene: Fig. 7 shows that the training scenes of JRDB-Pose contain varied
distributions of keypoint visibility, from densely populated indoor to dynamic
outdoor scenes in which the robot navigates and interacts with oncoming
pedestrians.

###  3.1 Benchmark and Metrics

JRDB-Pose Splits: We follow the splits of JRDB [28] to create training,
validation, and testing splits from the 54 captured sequences, with each split
containing an equal proportion of indoor and outdoor scenes as well as scenes
captured using a stationary or moving robot. All frames from a scene appear
strictly in one split.

Figure 7: Distribution of keypoint visibility annotations for each JRDB-Pose
train and validation scene. Best viewed in color.

####  3.1.1 Pose estimation

OKS. We define the ground-truth and a corresponding predicted human body
keypoints as $x_{i}\in\mathbb{R}^{2\times J}$ and $y_{j}\in\mathbb{R}^{2\times
J}$, respectively, where $J$ represents the number of the keypoints. We
measure a similarity score between $x_{i}$ and $y_{j}$ via Object Keypoint
Similarity (OKS) [26] as:

$OKS(x_{i},y_{j})=\exp\left(-\frac{d_{E}^{2}(x_{i},y_{j})}{2s^{2}k^{2}}\right)$
(1)

where $d_{E}(x_{i},y_{j})$ is the mean Euclidean distance between two set of
keypoints normalized by the product of ground-truth box area $s$ and a sigma
constants $k$. While this metric is commonly used for single-person pose
estimation, in multi-person settings the assignment between ground-truth and
predicted poses is not known, so it is unclear which pose pairs to use without
a matching mechanism.
Average Precision. Average Precision (AP) and mean AP (mAP) [36] are among the
most common multi-person pose estimation metrics. Similarity between predicted
and ground-truth poses are calculated via OKS[26], and poses are matched via a
non-optimal greedy assignment strategy that matches poses with higher
confidence scores first. True and false positive predictions are determined by
a threshold on the OKS score. Since the resulting AP score corresponds to the
performance at a specific OKS threshold rather than a continuous value based
on overall keypoint distances, the COCO benchmark [26] averages AP across
multiple OKS thresholds to reward methods with higher localization accuracy.
Nevertheless, this strategy does not fully capture keypoint distance nor is
the matching strategy optimal. We evaluate AP using an OKS threshold of $0.5$.

OSPA-Pose. Optimal Sub-Pattern Matching (OSPA) [43] is a multi-object
performance evaluation metric which includes the concept of miss-distance.
Recently, OSPA has been further adapted to detection and tracking tasks [38]
while preserving its metric properties. We propose OSPA-Pose
($\mathcal{O}_{pose}$), a modification of the OSPA metric for multi-person
pose estimation.

Let $X=\\{x_{1},x_{2},\dots x_{m}\\}$ and $Y=\\{y_{1},y_{2},\dots y_{n}\\}$ be
two sets of body keypoints for all ground-truth and predicted body poses,
respectively. Considering $d_{K}(x_{i},y_{i})=1-OKS(x_{i},y_{i})\in[0,1]$ as a
normalized distance metric between human poses, OSPA-Pose
($\mathcal{O}_{pose}$) error is calculated by

$\mathcal{O}_{pose}(X,Y)\\!=\\!\frac{1}{n}\bigg{(}\\!\\!\min_{\pi\in\Pi_{n}}\\!\\!\sum_{i=1}^{m}(d_{K}(x_{i},y_{\pi_{i}}))\\!+\\!(n-m)\\!\\!\bigg{)},$
(2)

where $n\geq m\geq 0$, $\Pi_{n}$ is the set of all permutations of
$\\{1,2,\dots,n\\}$, and $\mathcal{O}_{pose}(X,Y)$ = $\mathcal{O}_{pose}(Y,X)$
if $m<n$. We further define $\mathcal{O}_{pose}(X,Y)=1$ if either $X$ or $Y$
is empty, and $\mathcal{O}_{pose}(\emptyset,\emptyset)=0$.

While both AP and $\mathcal{O}_{pose}$ use OKS to calculate pose similarity,
$\mathcal{O}_{pose}$ measures an optimal distance from the set of continuous
keypoint distances consisting of the localization error (first term) and
cardinality mismatch (second term), eliminating the need for thresholding.

####  3.1.2 Pose tracking metrics

Commonly used metrics: We evaluate pose tracking performance with three
commonly used tracking metrics, MOTA [2], IDF1 [40], and IDSW [31], with
modifications to make them suitable for the pose tracking task. Rather than
using IoU or GIoU[39] to calculate similarity scores which are more
appropriate for bounding boxes, we apply OKS (as defined in Eq. 1) to obtain a
similarity score between keypoint sets, with any keypoint pair of OKS above a
threshold of 0.5 considered a positive prediction.
OSPA(2)-Pose Inspired by [4, 38] extending the OSPA metric for evaluating two
sets of trajectories, we propose a new metric for evaluating two sets of human
body pose tracks, namely OSPA(2)-Pose ($\mathcal{O}_{pose}^{2}$).

Let $\mathbf{X}=\\{X^{\mathcal{D}_{1}}_{1},X^{\mathcal{D}_{2}}_{2},\dots
X^{\mathcal{D}_{m}}_{m}\\}$ and
$\mathbf{Y}=\\{Y^{\mathcal{D}_{1}}_{1},Y^{\mathcal{D}_{2}}_{2},\dots
Y^{\mathcal{D}_{n}}_{n}\\}$ to be two sets of body keypoint trajectories for
ground-truth and predicted body poses, respectively. Note $\mathcal{D}_{i}$
represents the time indices which track $i$ exists (having a state-value).
Then, we calculate the time average distance of every pair of tracks
$X^{\mathcal{D}_{i}}_{i}$ and $Y^{\mathcal{D}_{j}}_{j}$:

$\underline{\widetilde{d}}(X^{\mathcal{D}_{i}}_{i},Y^{\mathcal{D}_{j}}_{j})=\sum_{t\in\mathcal{D}_{i}\cup\mathcal{D}_{j}}\frac{d_{O}\left(\\{X_{i}^{t}\\},\\{Y_{j}^{t}\\}\right)}{|\mathcal{D}_{i}\cup\mathcal{D}_{j}|},$
(3)

where $t\in\mathcal{D}_{i}\cup\mathcal{D}_{j}$ is the time-step when either or
both track presents. Note that $\\{X_{i}^{t}\\}$ and $\\{Y_{j}^{t}\\}$ are
singleton sets, _i.e_. $\\{X_{i}^{t}\\}=\emptyset$ or
$\\{X_{i}^{t}\\}=x_{i}^{t}\in\mathbb{R}^{2\times J}$ and
$\\{Y_{j}^{t}\\}=\emptyset$ or
$\\{Y_{j}^{t}\\}=y_{j}^{t}\in\mathbb{R}^{2\times J}$. Therefore, inspired by
the OSPA set distance, $d_{O}\left(\\{X_{i}^{t}\\},\\{Y_{j}^{t}\\}\right)$ can
be simplified into the following distance function, :

$\displaystyle d_{O}(\\{X_{i}^{t}\\},$ $\displaystyle\\{Y_{j}^{t}\\})=$
$\displaystyle\left\\{\begin{array}[]{lll}d_{K}(x^{t}_{i},y^{t}_{j})&\text{if
}|\\{X_{i}^{t}\\}|\wedge|\\{Y_{j}^{t}\\}|=1,\\\ 1&\text{if
}|\\{X_{i}^{t}\\}|\oplus|\\{Y_{j}^{t}\\}|=1,\\\
0&\text{Otherwise,}\end{array}\right.$ (7)

where $d_{K}(x_{i},y_{i})=1-OKS(x_{i},y_{i})\in[0,1]$.

Finally, we obtain the distance, $\mathcal{O}_{pose}^{2}$, between two sets of
pose tracks, _i.e_. $\mathbf{X}$ and $\mathbf{Y}$ by applying another OSPA
distance over Eq. 3, _i.e_.

$\displaystyle\mathcal{O}^{2}_{pose}$ $\displaystyle(\mathbf{X},\mathbf{Y})=$
$\displaystyle\frac{1}{n}$
$\displaystyle\left(\min_{\pi\in\Pi_{n}}\sum_{i=1}^{m}\underline{\widetilde{d}}(X^{\mathcal{D}_{i}}_{i},Y^{\mathcal{D}_{\pi_{i}}}_{\pi_{i}})+(n-m)\right),$
(8)

where $n\geq m\geq 0$, $\Pi_{n}$ is the set of all permutations of
$\\{1,2,\dots,n\\}$, and $\mathcal{O}^{2}_{pose}(X,Y)$ =
$\mathcal{O}^{2}_{pose}(Y,X)$ if $m<n$. We further define
$\mathcal{O}^{2}_{pose}(X,Y)=1$ if either $X$ or $Y$ is empty, and
$\mathcal{O}^{2}_{pose}(\emptyset,\emptyset)=0$.

Note that the first term $\underline{\widetilde{d}}$ reflects ID switches and
localization errors, whereas the cardinality error $(n-m)$ contains false and
missed track errors. In Tab. 5, we also present the $\mathcal{O}_{pose}^{2}$
per occlusion level, where each occlusion level is considered individually. We
do not apply localisation error to joints that do not belong to the occlusion
of interest by setting their Euclidean distance to zero, indicating that
$d_{E}(x_{i},y_{j})$ in Eq. 1 is now the average Euclidean distance between
two sets of keypoints of a certain occlusion level.

##  4 Multi-Person Pose Estimation and Tracking Baselines

In this section we evaluate the performance of various state-of-the-art
methods for pose estimation and tracking, verifying that JRDB-Pose is a
challenging dataset for existing frameworks seeking an opportunity for a
dedicated developments in this domain. All methods are evaluated on our
individual images and annotations.

###  4.1 Multi-Person Pose Estimation

We evaluate several popular or recent state-of-the-art methods for multi-
person pose estimation models [49, 19, 48, 27]. We evaluate one top-down
method using HRNet backbone [49]. The top-down method uses a Faster R-CNN [37]
detector to predict all humans in the scene, from which the high-confidence
(score $>$ 0.95) predictions are kept. We use a heatmap size of 192x256, and
the HRNet backbone with a high-resolution convolution width of 48. We further
evaluate three recent bottom-up models, which regress joint locations directly
without the need for human detections: DEKR [19], CID [48], an YoloPose [27].
All methods are trained from their respective initializations without COCO
pre-training. To help address training difficulties associated with wide
panoramic images, we train on individual camera images, and then combine them
together to form stitched view predictions. Duplicate poses in the stitched
annotation set are eliminated using NMS on the predicted boxes.

Method | $\mathcal{O}_{pose}$ | Loc | Card | AP | AR |
---|---|---|---|---|---|---
HRNet[49] | 0.480 | 0.210 | 0.270 | 24.6 | 47.5 |
DEKR[19] | 0.410 | 0.113 | 0.2968 | 31.7 | 46.9 |
CID[48] | 0.377 | 0.174 | 0.204 | 38.6 | 48.0 |
YOLO-Pose[27] | 0.368 | 0.172 | 0.196 | 47.9 | 72.5 |

Table 3: Multi-person pose estimation baselines evaluated on JRDB-Pose
stitched annotations. Loc and Card are the $\mathcal{O}_{pose}$ localization
and cardinality error, respectively.

Method | Total Contribution | Per-Keypoint Avg
---|---|---
V$\downarrow$ |  O$\downarrow$ |  I$\downarrow$ |  V$\downarrow$ |  O$\downarrow$ |  I$\downarrow$
HRNet[49] | 0.131 | 0.054 | 0.025 | 0.051 | 0.075 | 0.102
DEKR[19] | 0.070 | 0.031 | 0.012 | 0.030 | 0.052 | 0.089
CID[48] | 0.101 | 0.047 | 0.026 | 0.038 | 0.063 | 0.108
YOLO-Pose[27] | 0.098 | 0.048 | 0.026 | 0.033 | 0.055 | 0.091

Table 4: We break down $\mathcal{O}_{pose}$ localization error from Tab. 3 for
visible (V), occluded (O), and invisible (I) joints. The left and right
columns represent the total and average per-keypoint contribution (scaled by
1M) to the $\mathcal{O}_{pose}$ localization error, respectively.

Pose Estimation Method (Training) |  Tracking Method | MOTA $\uparrow$ | IDF1$\uparrow$ | IDSW$\downarrow$ | $\mathcal{O}_{pose}^{2}$$\downarrow$ | Components |  $\mathcal{O}_{pose}^{2}$$\downarrow$ by Visibility
---|---|---|---|---|---|---|---
Card$\downarrow$ |  Loc$\downarrow$ |  V$\downarrow$ |  O$\downarrow$ |  I$\downarrow$
Yolo-Pose[27] (COCO only) |  ByteTrack[55] | 61.32 | 55.80 | 4236 | 0.715 | 0.519 | 0.196 | 0.698 | 0.703 | 0.731
UniTrack[50] | 60.80 | 55.01 | 2854 | 0.727 | 0.473 | 0.253 | 0.713 | 0.716 | 0.733
OCTrack[8] | 66.09 | 59.93 | 2588 | 0.630 | 0.380 | 0.249 | 0.610 | 0.618 | 0.646
Yolo-Pose[27] (JRDB-Pose only) |  ByteTrack[55] | 61.17 | 52.11 | 3203 | 0.693 | 0.429 | 0.264 | 0.678 | 0.692 | 0.708
UniTrack[50] | 69.84 | 56.06 | 2565 | 0.725 | 0.454 | 0.271 | 0.710 | 0.722 | 0.734
OCTrack[8] | 69.22 | 60.62 | 1977 | 0.577 | 0.295 | 0.283 | 0.556 | 0.577 | 0.595
Yolo-Pose[27] (COCO$\rightarrow$ JRDB-Pose) |  ByteTrack[55] | 67.16 | 55.38 | 3325 | 0.690 | 0.456 | 0.234 | 0.674 | 0.688 | 0.708
UniTrack[50] | 72.82 | 57.69 | 2413 | 0.708 | 0.435 | 0.272 | 0.693 | 0.707 | 0.719
OCTrack[8] | 71.74 | 61.15 | 2260 | 0.594 | 0.331 | 0.263 | 0.573 | 0.592 | 0.613

Table 5: Multi-person pose tracking baselines evaluated on JRDB-Pose
individual camera images.

Tab. 3 summarises the pose estimation results for each method. To highlight
the large number of occluded and invisible labels in our dataset, we include
the total contribution of joints of each visibility type to the overall OSPA-
Pose as well as average contribution for joints of each visibility. We also
include cardinality and localization errors of OSPA-Pose. We find that Yolo-
Pose is the best performing baseline, outperforming the other methods in
$\mathcal{O}_{pose}$ and AP. DEKR achieves the lowest $\mathcal{O}_{pose}$
localization error but a higher cardinality error as a result of a high number
of missed detections. In Tab. 4 we see the contributions to the localization
error based on keypoint visibility. DEKR achieves the lower metrics due to its
low localization error, besides which Yolo-Pose again performs best. Although
visible joints contribute the most to localization error due to their higher
overall frequency, on a per-keypoint average the predictions on occluded and
invisible joints showed significantly higher errors for all models, confirming
that occlusion poses a difficult challenge for existing methods. JRDB-Pose
contains a significant number of occluded joints which we hope will be useful
for researchers to improve the robustness of pose estimation methods to
occlusions. We also find that all methods achieve lower AP as compared to
their results on common large-scale benchmarks [26], showing that JRDB-Pose
presents a reasonable and difficult challenge for multi-person pose
estimation.

###  4.2 Multi-Person Pose Tracking

We evaluate three recent state-of-the-art methods, such as ByteTrack [55],
Unitrack [50], and OC-SORT [8], in our tracking benchmark. All these methods
are from the tracking-by-detection category and predict tracks based on
predictions from a pose estimation method. We use the estimates from Yolo-
Pose, as our highest-performing baseline, to initialize the predictions for
all pose tracking methods.

In Tab. 5 we provide our pose tracking results for all trackers and training
strategies. OC-Sort achieves the best results by a wide margin. Since OC-SORT
targets improving tracking robustness for objects in non-linear motion with
the improved Kalman-filter and recovery strategy, we believe it is better
suited for JRDB-Pose which includes occluded periods during which people are
often not moving linearly with respect to the robot’s perspective (_e.g_.
sequences where both the robot and nearby people are turning or moving
simultaneously), thus better recovering from occlusions. This method’s lower
$\mathcal{O}_{pose}^{2}$cardinality error further confirms that it can better
find accurate tracks. Compared to their performance in other large-scale
tracking benchmarks [13, 12], these methods achieve relatively lower overall
performance in the same reported metrics, reflecting the unique challenges in
our dataset, which we hope will motivate further work within the research
community.

###  4.3 Study on Pre-training Methods

Using our highest performing pose estimation method, we further study how pre-
training strategies affect our final model performance. We try inference-only
with a COCO-pretrained model, finetuning a COCO-pretrained model, and training
from scratch. Tab. 6 and Tab. 5 show pose estimation and tracking performance
of our highest performing model across the different training strategies. For
pose estimation we find that finetuning from COCO generally performs better
than training from scratch in both pose estimation and tracking tasks with an
improvement of $1.7\%$ in $\mathcal{O}_{pose}^{2}$, while running inference
from a COCO model without finetune shows much weaker results especially in
$\mathcal{O}_{pose}$ . The relatively small improvement also suggests that
JRDB-Pose contains a varied distribution of scenes and poses suitable for
model training.

We also include results for pose tracking across the same three training
strategies of Yolo-Pose used to initialize the tracking. In Tab. 5 we find
that the JRDB-Pose model outperforms the COCO model finetuned on JRDB-Pose in
MOTA and IDF1 but is itself outperformed in IDSW and $\mathcal{O}_{pose}^{2}$.
The COCO model without finetuning performs worse than the other models. Since
the COCO model is not trained on the crowded human scenes and overall image
distribution present in JRDB-Pose, it struggles to find keypoints and thus has
a high miss-rate as indicated by a much higher
$\mathcal{O}_{pose}^{2}$cardinality component compared to the other training
schemes. JRDB-Pose contains a varied distribution of scenes in human
environments different from COCO.

Training strategy | Stitched | Per Camera
---|---|---
AP $\uparrow$ |  $\mathcal{O}_{pose}$ $\downarrow$ |  AP $\uparrow$ |  $\mathcal{O}_{pose}$ $\downarrow$
COCO [26] | 48.0 | 0.460 | 60.2 | 0.320
JRDB-Pose | 48.2 | 0.357 | 61.9 | 0.280
COCO $\rightarrow$ JRDB-Pose | 49.6 | 0.385 | 63.9 | 0.273

Table 6: We study how training protocols affect performance of YoloPose.
Finetuning from COCO achieves best results.

##  5 Conclusion

In this paper we have introduced JRDB-Pose, a large-scale dataset of human
poses and track IDs suitable for multi-person pose estimation and tracking
from videos. JRDB-Pose features high-frequency annotations for crowded indoor
and outdoor scenes with heavy occlusion, diverse actions, and longer videos
than existing large-scale pose estimation and tracking datasets. Finally, we
have introduced OSPA-Pose and OSPA${}^{(}2)$-Pose, new metrics for multi-
person pose estimation and tracking. We believe that JRDB-Pose will help
address limitations in human action understanding for human-robot interaction
and navigation in human environments and advance research by proving large-
scale and high-quality annotations.

## References

  * [1] Felix Achilles, Alexandru-Eugen Ichim, Huseyin Coskun, Federico Tombari, Soheyl Noachtar, and Nassir Navab.  Patient mocap: Human pose estimation under blanket occlusion for hospital monitoring applications.  In MICCAI, pages 491–499. Springer, 2016.
  * [2] Mykhaylo Andriluka, Umar Iqbal, Eldar Insafutdinov, Leonid Pishchulin, Anton Milan, Juergen Gall, and Bernt Schiele.  Posetrack: A benchmark for human pose estimation and tracking.  In CVPR, pages 5167–5176, 2018.
  * [3] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele.  2d human pose estimation: New benchmark and state of the art analysis.  In CVPR, June 2014.
  * [4] Michael Beard, Ba Tuong Vo, and Ba-Ngu Vo.  A solution for large-scale multi-object tracking.  IEEE Transactions on Signal Processing, 68:2754–2769, 2020.
  * [5] Philipp Bergmann, Tim Meinhardt, and Laura Leal-Taixe.  Tracking without bells and whistles.  In ICCV, pages 941–951, 2019.
  * [6] Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft.  Simple online and realtime tracking.  In 2016 ICIP, pages 3464–3468. IEEE, 2016.
  * [7] Guillem Brasó, Nikita Kister, and Laura Leal-Taixé.  The center of attention: Center-keypoint grouping via attention for multi-person pose estimation.  In ICCV, pages 11853–11863, 2021.
  * [8] Jinkun Cao, Xinshuo Weng, Rawal Khirodkar, Jiangmiao Pang, and Kris Kitani.  Observation-centric sort: Rethinking sort for robust multi-object tracking.  arXiv:2203.14360, 2022.
  * [9] Yu Cheng, Bo Yang, Bo Wang, and Robby T Tan.  3d human pose estimation using spatio-temporal networks with explicit occlusion training.  In AAAI, volume 34, pages 10631–10638, 2020.
  * [10] Yu Cheng, Bo Yang, Bo Wang, Wending Yan, and Robby T Tan.  Occlusion-aware networks for 3d human pose estimation in video.  In ICCV, pages 723–732, 2019.
  * [11] Peng Chu, Jiang Wang, Quanzeng You, Haibin Ling, and Zicheng Liu.  Transmot: Spatial-temporal graph transformer for multiple object tracking.  arXiv:2104.00194, 2021.
  * [12] Patrick Dendorfer, Hamid Rezatofighi, Anton Milan, Javen Shi, Daniel Cremers, Ian Reid, Stefan Roth, Konrad Schindler, and Laura Leal-Taixé.  Mot20: A benchmark for multi object tracking in crowded scenes.  arXiv:2003.09003, 2020.
  * [13] Andreas Doering, Di Chen, Shanshan Zhang, Bernt Schiele, and Juergen Gall.  Posetrack21: A dataset for person search, multi-object tracking and multi-person pose tracking.  In CVPR, pages 20963–20972, 2022.
  * [14] Yunhao Du, Yang Song, Bo Yang, and Yanyun Zhao.  Strongsort: Make deepsort great again.  arXiv:2202.13514, 2022.
  * [15] Mahsa Ehsanpour, Fatemeh Saleh, Silvio Savarese, Ian Reid, and Hamid Rezatofighi.  Jrdb-act: A large-scale dataset for spatio-temporal action, social group and activity detection.  In CVPR, pages 20983–20992, 2022.
  * [16] Matteo Fabbri, Guillem Brasó, Gianluca Maugeri, Orcun Cetintas, Riccardo Gasparini, Aljoša Ošep, Simone Calderara, Laura Leal-Taixé, and Rita Cucchiara.  Motsynth: How can synthetic data help pedestrian detection and tracking?  In ICCV, pages 10849–10859, 2021.
  * [17] Matteo Fabbri, Fabio Lanzi, Simone Calderara, Andrea Palazzi, Roberto Vezzani, and Rita Cucchiara.  Learning to detect and track visible and occluded body joints in a virtual world.  In ECCV (ECCV), 2018.
  * [18] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In CVPR, pages 3354–3361. IEEE, 2012.
  * [19] Zigang Geng, Ke Sun, Bin Xiao, Zhaoxiang Zhang, and Jingdong Wang.  Bottom-up human pose estimation via disentangled keypoint regression.  In CVPR, pages 14676–14686, 2021.
  * [20] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu.  Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.  TPAMI, 36(7):1325–1339, jul 2014.
  * [21] Sheng Jin, Wentao Liu, Wanli Ouyang, and Chen Qian.  Multi-person articulated tracking with spatial and temporal embeddings.  In CVPR, pages 5664–5673, 2019.
  * [22] Sheng Jin, Wentao Liu, Enze Xie, Wenhai Wang, Chen Qian, Wanli Ouyang, and Ping Luo.  Differentiable hierarchical graph grouping for multi-person pose estimation.  In ECCV, pages 718–734. Springer, 2020.
  * [23] Haresh Karnan, Anirudh Nair, Xuesu Xiao, Garrett Warnell, Sören Pirk, Alexander Toshev, Justin Hart, Joydeep Biswas, and Peter Stone.  Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation.  arXiv:2203.15041, 2022.
  * [24] Rawal Khirodkar, Visesh Chari, Amit Agrawal, and Ambrish Tyagi.  Multi-instance pose networks: Rethinking top-down pose estimation.  In ICCV, pages 3122–3131, 2021.
  * [25] Laura Leal-Taixé, Anton Milan, Ian Reid, Stefan Roth, and Konrad Schindler.  Motchallenge 2015: Towards a benchmark for multi-target tracking.  arXiv:1504.01942, 2015.
  * [26] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, pages 740–755. Springer, 2014.
  * [27] Debapriya Maji, Soyeb Nagori, Manu Mathew, and Deepak Poddar.  Yolo-pose: Enhancing yolo for multi person pose estimation using object keypoint similarity loss.  In CVPR, pages 2637–2646, 2022.
  * [28] Roberto Martin-Martin, Mihir Patel, Hamid Rezatofighi, Abhijeet Shenoi, JunYoung Gwak, Eric Frankel, Amir Sadeghian, and Silvio Savarese.  Jrdb: A dataset and benchmark of egocentric robot visual perception of humans in built environments.  TPAMI, 2021.
  * [29] Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer.  Trackformer: Multi-object tracking with transformers.  In CVPR, pages 8844–8854, 2022.
  * [30] Anton Milan, Laura Leal-Taixé, Ian Reid, Stefan Roth, and Konrad Schindler.  Mot16: A benchmark for multi-object tracking.  arXiv:1603.00831, 2016.
  * [31] Anton Milan, Laura Leal-Taixé, Ian Reid, Stefan Roth, and Konrad Schindler.  Mot16: A benchmark for multi-object tracking.  arXiv:1603.00831, 2016.
  * [32] Alejandro Newell, Zhiao Huang, and Jia Deng.  Associative embedding: End-to-end learning for joint detection and grouping.  Advances in neural information processing systems, 30, 2017.
  * [33] Alejandro Newell, Kaiyu Yang, and Jia Deng.  Stacked hourglass networks for human pose estimation.  In ECCV, pages 483–499. Springer, 2016.
  * [34] Guanghan Ning, Ping Liu, Xiaochuan Fan, and Chi Zhang.  A top-down approach to articulated human pose estimation and tracking.  In ECCV Workshops, pages 0–0, 2018.
  * [35] Abhishek Patil, Srikanth Malla, Haiming Gang, and Yi-Ting Chen.  The h3d dataset for full-surround 3d multi-object detection and tracking in crowded urban scenes.  In ICRA, pages 9552–9557. IEEE, 2019.
  * [36] Leonid Pishchulin, Eldar Insafutdinov, Siyu Tang, Bjoern Andres, Mykhaylo Andriluka, Peter V Gehler, and Bernt Schiele.  Deepcut: Joint subset partition and labeling for multi person pose estimation.  In CVPR, pages 4929–4937, 2016.
  * [37] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [38] Hamid Rezatofighi, Tran Thien Dat Nguyen, Ba-Ngu Vo, Ba-Tuong Vo, Silvio Savarese, and Ian Reid.  How trustworthy are performance evaluations for basic vision tasks?  arXiv:2008.03533, 2020.
  * [39] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese.  Generalized intersection over union: A metric and a loss for bounding box regression.  In CVPR, pages 658–666, 2019.
  * [40] Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi.  Performance measures and a data set for multi-target, multi-camera tracking.  In ECCV, pages 17–35. Springer, 2016.
  * [41] Andrey Rudenko, Tomasz P Kucner, Chittaranjan S Swaminathan, Ravi T Chadalavada, Kai O Arras, and Achim J Lilienthal.  Thör: Human-robot navigation data collection and accurate motion trajectories dataset.  RA-Letters, 5(2):676–682, 2020.
  * [42] Fatemeh Saleh, Sadegh Aliakbarian, Hamid Rezatofighi, Mathieu Salzmann, and Stephen Gould.  Probabilistic tracklet scoring and inpainting for multiple object tracking.  In CVPR, pages 14329–14339, 2021.
  * [43] Dominic Schuhmacher, Ba-Tuong Vo, and Ba-Ngu Vo.  A consistent metric for performance evaluation of multi-object filters.  IEEE transactions on signal processing, 56(8):3447–3457, 2008.
  * [44] Michael Snower, Asim Kadav, Farley Lai, and Hans Peter Graf.  15 keypoints is all you need.  In CVPR, pages 6738–6748, 2020.
  * [45] Peize Sun, Jinkun Cao, Yi Jiang, Rufeng Zhang, Enze Xie, Zehuan Yuan, Changhu Wang, and Ping Luo.  Transtrack: Multiple object tracking with transformer.  arXiv:2012.15460, 2020.
  * [46] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al.  Scalability in perception for autonomous driving: Waymo open dataset.  In CVPR, pages 2446–2454, 2020.
  * [47] Timo Von Marcard, Roberto Henschel, Michael J Black, Bodo Rosenhahn, and Gerard Pons-Moll.  Recovering accurate 3d human pose in the wild using imus and a moving camera.  In ECCV, pages 601–617, 2018.
  * [48] Dongkai Wang and Shiliang Zhang.  Contextual instance decoupling for robust multi-person pose estimation.  In CVPR, pages 11060–11068, 2022.
  * [49] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al.  Deep high-resolution representation learning for visual recognition.  TPAMI, 43(10):3349–3364, 2020.
  * [50] Zhongdao Wang, Hengshuang Zhao, Ya-Li Li, Shengjin Wang, Philip Torr, and Luca Bertinetto.  Do different tracking tasks require different appearance models?  Advances in Neural Information Processing Systems, 34:726–738, 2021\.
  * [51] Nicolai Wojke, Alex Bewley, and Dietrich Paulus.  Simple online and realtime tracking with a deep association metric.  In 2017 ICIP, pages 3645–3649. IEEE, 2017.
  * [52] Yihong Xu, Yutong Ban, Guillaume Delorme, Chuang Gan, Daniela Rus, and Xavier Alameda-Pineda.  Transcenter: Transformers with dense queries for multiple-object tracking.  arXiv:2103.15145, 2021.
  * [53] Dongdong Yu, Kai Su, Jia Sun, and Changhu Wang.  Multi-person pose estimation for pose tracking with enhanced cascaded pyramid network.  In ECCV Workshops, pages 0–0, 2018.
  * [54] Weiyu Zhang, Menglong Zhu, and Konstantinos G Derpanis.  From actemes to action: A strongly-supervised representation for detailed action understanding.  In ICCV, pages 2248–2255, 2013.
  * [55] Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang.  Bytetrack: Multi-object tracking by associating every detection box.  In ECCV, pages 1–21. Springer, 2022.
  * [56] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl.  Tracking objects as points.  In ECCV, pages 474–490. Springer, 2020.
  * [57] Tianyu Zhu, Markus Hiller, Mahsa Ehsanpour, Rongkai Ma, Tom Drummond, Ian Reid, and Hamid Rezatofighi.  Looking beyond two frames: End-to-end multi-object tracking using spatial and temporal transformers.  TPAMI, 2022.

Appendix

##  Appendix A Annotation Visualization

JRDB-Pose provides high-quality pose annotations at a high (15fps) frequency
over all JRDB scenes. In Figure Fig. 9 and Figure Fig. 8 we visualize of our
pose annotations for each of the training scenes. The visualizations show the
indoor and outdoor scenes, varying light conditions, and a diverse poses
representing a wide distribution of actions. To highlight the high frequency
and quality of annotations we also provide a video in the supplementary
material titled JRDB_Pose_Supp_Video.mp4.

##  Appendix B Additional Evaluations

In addition to the evaluation we provide in the main paper, we also include
metrics for additional modalities and camera types. In Tab. 7 we report the
tracking results for our baseline models and pre-training methods using the
panoramic stitched camera images. Similar to the results on individual camera
images, OCTrack outperforms the other baselines. We also similarly observe
that fine-tuning from COCO yields mixed results with improvements in MOTA and
IDF1 but drops in the other metrics as compared to training on JRDB-Pose only.

##  Appendix C Qualitative Analysis

We show predictions made by Yolo-Pose, our best performing pose estimation
method, on frames from each of the testing sequences in Fig. 10 and Fig. 11.
The model used for visualization was initialized from scratch. The predictions
are accurate for most poses, demonstrating that JRDB-Pose is sufficiently
large enough for the model to learn to predict a wide distribution of poses,
even without pre-training.

Figure 8: Visualization of JRDB-Pose annotations from each of the 27 training
sequences in JRDB-Pose. The images feature indoor and outdoor areas on a
university campus with varying lighting conditions, motion, pedestrian
density, and activities. Locations include roads, strip malls, sidewalks,
restaurants, plazas, parks, halls, classrooms, laboratories, and office
buildings. These scenes show a wide range of scenarios and human poses that a
social robot would encounter during operations.  Figure 9: Visualization of
JRDB-Pose annotations on the 27 training sequences (cont.)  Figure 10:
Visualization of Yolo-Pose pose estimation predictions on 27 testing sequences
in JRDB-Pose. The weights of the model are initialized from scratch,
demonstrating that JRDB-Pose is sufficiently large for the model to learn
accurate pose representations even without finetuning.

Pose Estimation Method (Training) |  Tracking Method | MOTA $\uparrow$ | IDF1$\uparrow$ | IDSW$\downarrow$ | $\mathcal{O}_{pose}^{2}$$\downarrow$ | Components |  $\mathcal{O}_{pose}^{2}$$\downarrow$ by Visibility
---|---|---|---|---|---|---|---
Card$\downarrow$ |  Loc$\downarrow$ |  V$\downarrow$ |  O$\downarrow$ |  I$\downarrow$
Yolo-Pose[27] (COCO only) |  ByteTrack[55] | 51.00 | 43.63 | 5160 | 0.914 | 0.815 | 0.099 | 0.910 | 0.912 | 0.911
UniTrack[50] | 45.74 | 41.35 | 4475 | 0.940 | 0.877 | 0.064 | 0.938 | 0.939 | 0.938
OCTrack[8] | 55.51 | 45.33 | 3906 | 0.895 | 0.766 | 0.129 | 0.892 | 0.894 | 0.892
Yolo-Pose[27] (JRDB-Pose only) |  ByteTrack[55] | 54.33 | 39.84 | 4730 | 0.920 | 0.796 | 0.124 | 0.917 | 0.920 | 0.917
UniTrack[50] | 55.90 | 44.32 | 4206 | 0.928 | 0.849 | 0.079 | 0.924 | 0.928 | 0.925
OCTrack[8] | 61.28 | 48.08 | 3296 | 0.861 | 0.692 | 0.169 | 0.856 | 0.862 | 0.855
Yolo-Pose[27] (COCO$\rightarrow$ JRDB-Pose) |  ByteTrack[55] | 57.69 | 43.68 | 4333 | 0.910 | 0.791 | 0.120 | 0.909 | 0.911 | 0.907
UniTrack[50] | 59.37 | 46.82 | 3779 | 0.921 | 0.841 | 0.080 | 0.919 | 0.921 | 0.918
OCTrack[8] | 63.02 | 49.04 | 3394 | 0.870 | 0.715 | 0.155 | 0.867 | 0.871 | 0.865

Table 7: Multi-person pose tracking baselines evaluated on JRDB-Pose stitched
camera images. Figure 11: Visualization of Yolo-Pose predictions on 27 testing
sequences in JRDB-Pose dataset (cont.)


11institutetext: JD Digits, Beijing, China22institutetext: Beijing University
of Posts and Telecommunications
22email: eagan-ywj<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
# RAF-AU Database: In-the-Wild Facial Expressions with Subjective Emotion
Judgement and Objective AU Annotations

Wen-Jing Yan These authors contributed equally to this work.11 Shan Li 2 ⋆2 ⋆
Chengtao Que 11 Jiquan Pei 11 Weihong Deng 22

###### Abstract

Much of the work on automatic facial expression recognition relies on
databases containing a certain number of emotion classes and their exaggerated
facial configurations (generally six prototypical facial expressions), based
on Ekman’s Basic Emotion Theory. However, recent studies have revealed that
facial expressions in our human life can be blended with multiple basic
emotions. And the emotion labels for these in-the-wild facial expressions
cannot easily be annotated solely on pre-defined AU patterns. How to analyze
the action units for such complex expressions is still an open question. To
address this issue, we develop a RAF-AU database that employs a sign-based
(i.e., AUs) and judgement-based (i.e., perceived emotion) approach to
annotating blended facial expressions in the wild. We first reviewed the
annotation methods in existing databases and identified crowdsourcing as a
promising strategy for labeling in-the-wild facial expressions. Then, RAF-AU
was finely annotated by experienced coders, on which we also conducted a
preliminary investigation of which key AUs contribute most to a perceived
emotion, and the relationship between AUs and facial expressions. Finally, we
provided a baseline for AU recognition in RAF-AU using popular features and
multi-label learning methods.

##  1 Introduction

Of all nonverbal behaviors—body movement, posture, gaze, proxemics and
voice—the face is probably the most commanding and complicated, and perhaps
the most confusing. The face, and especially facial movement (or expression),
is commanding because it is always visible and therefore always providing
information such as emotion and intent [1]. Due to their utility for
understanding a human being’s mental state, the recognition of automatic
facial expressions is becoming a popular field in computer vision.

Automatic facial expression recognition relies heavily on training datasets.
When training a system to recognize facial expressions, the investigator must
assume that the training and test data have been accurately labeled. This
assumption may or may not be accurate. Traditionally, researchers have
categorized facial expressions as expressing happiness, sadness, surprise,
fear, anger and disgust. Each of these prototypical facial expressions can be
described via a pattern of action units (AUs), based on the Facial Action
Coding System (FACS) developed by Ekman and colleagues [2] and supported by
Basic Emotion Theory (BET) [3].

Early in the process, samples of facial expressions are mostly viewed from the
front, collected from actors required to pose preset AUs that readily match
prototypical facial expressions [4, 5]. Thus, recognition of the emotion can
be judged against the “correct” expression adopted by the poser. Recently,
datasets of spontaneous facial expressions have multiplied [6, 7, 8, 9, 10].
This is remarkable progress because the goal of automatic facial expression
recognition is to apply to in the real world rather than lab situations (i.e.,
the ideal world). Facial expressions in real life are “in the wild”, meaning
that they are not necessarily frontal and direct-gazed, nor demonstrate only a
very limited number of AU patterns. There may be various combinations of AUs
with different gestures, head poses, gazes, and environments.

Due to the characteristics of in-the-wild facial expressions, applying an
emotion label to a certain face can be difficult. AU combinations do not
usually fall precisely into the six (or more) prototypical facial expressions
[11]. EMFACS [4, 12], usually understood as a guide for labeling emotions
expressed through AU combinations, has become inappropriate in this content.
Though previous research has summarized the relationship between AUs and
emotion [13, 14], researchers have only provided general or key AUs for
certain emotions. Moreover, the coherence problem has long being a topic of
debate in the field of facial expression [15, 16, 17]. Even for the six
universally used and prototypical facial expressions, psychologists have not
reached a consensus on the relationship between AU patterns and emotions, let
alone facial expressions in the wild. In other words, we cannot simply
categorize in-the-wild facial expressions by AU pattern, at least not for the
moment.

Generally, psychologists have proposed two approaches to studying nonverbal
behavior (including facial expressions), either judgement-based or sign-based
[1, 13]. In judgement-based approaches, observers make inferences about the
meaning of facial actions and assign corresponding labels. When classifying
facial expressions into a predefined number of emotions or mental activity
categories, the agreement of a group of annotators is taken as ground truth,
usually by computing the average of the responses of either experts or non-
experts. The rationale that judgement-based approaches can provide “correct”
labels can be explained from an evolutionary perspective: the facial
expressions broadcast from a sender (i.e., encoding the signal) should be
universally understood by a perceiver (i.e., decoding the signal), or they
would be useless and therefore removed. Thus, crowdsourced annotation, a
practice that employs many perceivers to heuristically label a target, might
be a useful way of labeling in-the-wild facial expressions. As for sign-based
approaches, facial motion and deformation are coded into visual classes.
Facial actions are then abstracted and described by their location and
intensity, such as in FACS. Ideally, a complete description framework would
contain all possible perceptible changes that might occur on a face. Most
automatic facial expression analysis approaches have attempted to directly
categorize facial expressions into basic emotion classes [18] by AUs, a sign-
based approach.

Figure 1: Example samples annotated with typical AUs in the RAF-AU dataset.

The judgement-based approach, with subjective estimation, provides the
perceivers’ judgement because facial expressions are born to be perceived and
understood by conspecifics. From this perspective, conspecifics can provide
the correct interpretation of a given facial expression. The sign-based
approach, with objective description, can tell the computer where and how the
facial movements will occur. Ideally, a qualified mechanism for annotating
facial expressions should include both subjective and objective elements.

With such an understanding of facial expressions and their annotation, the
present work provides an updated version of the existing RAF-ML database [9],
called RAF-AU database111http://whdeng.cn/RAF/model3.html, which consists of
in-the-wild facial expressions with both subjective (i.e., judgement-based)
and objective (i.e., sign-based) annotations. The RAF-ML database contains
facial images in different occlusions, illuminations and resolutions collected
from the social network and provides multiple labels via crowdsourcing
annotations for each facial image. To extend this database with objective
elements, we conducted AU coding according to strict criteria by two FACS
experts. Specially, two experienced coders were requested to independently
code each the facial image involving 26 kinds of AUs and the inter-coder
correlation was 0.6376. To our best knowledge, the RAF-AU database is the
first facial expression dataset that includes both subjective emotion
judgement and objective AU annotations for multi-label expression analysis in-
the-wild. Figure 1 shows example images annotated with typical AUs in the RAF-
AU dataset. Using both crowdsourced and AU annotations, we make the first
attempt to investigate the relationship between AUs and perceived emotions in
the wild. We explored which AUs contributed the most to each facial
expression. We then conducted AU detection experiments using popular features
and multi-label learning methods. A baseline was provided for AU recognition
in the RAF-AU.

##  2 Related Work

Several surveys have offered overviews of facial expression datasets [19, 20].
The present research does not repeat their work, but rather inspects how
facial movements were annotated with emotion labels. Many of these databases
classified facial expressions into types, based on rules such as those in
EMFACS (i.e., mapping AUs to facial expressions). Usually, participants were
required to pose “standard” AUs to express so-called prototypical facial
expressions, and those that qualified were included in the database.

In the Cohn-Kanade Expression Database [4], emotion labels refer to the
expression requested rather than what may actually have been performed. AUs
for each facial expression (i.e., the apex frame in a sequence) are converted
into emotion-specific expressions (e.g., happiness or anger). BU-3DFE [21] and
MMI [22] used a similar technique to assign emotion labels to specific facial
movements. Researchers have also used this approach to annotate emotions
expressed in spontaneous facial expressions. EmotioNet [6] is a large-scale
database with one million facial expression images collected from the
Internet. Most samples were annotated by an automatic AU detection algorithm,
and the remaining 10% were manually annotated using AUs. EmotioNet contains
six basic expressions and one neutral expression. The creators defined unique
AU patterns that mapped AUs to specific emotions. Therefore, the emotion
labels were inferred from the AUs. For example, if the face showed AU4$+$AU15,
the emotion label attributed was “sadness”. Similar mapping rules were used to
label compound facial expressions. EmotioNet defined 17 compound expressions,
depending on the AU combination. For example, AU4$+$AU20$+$AU25 was defined as
fearfully angry. Similarly in [14], subjects were required to practice their
expressions before each acquisition. They were required to express more than
one emotion based on prototypical facial expressions; these were then
evaluated by experts for validity. This annotation approach was also based on
prototypical facial expression protocol.

The dimension approach was used to annotate facial expressions from valence
(unpleasant to pleasant) and arousal (low to high). For example, AffectNet [7]
contains more than one million images from the Internet that were obtained by
querying different search engines using emotion-related tags. Over 450,000
images include manually annotated labels according to eight basic expressions.
Another valence-arousal dataset is Aff-Wild [23, 24] which contains 298 videos
with spontaneous facial behaviors collected in arbitrary recording conditions.
Most recently, the Aff-Wild dataset was extended to the Aff-wild2 [25, 26]
with more videos and other attribute annotations such as basic expressions and
action units. It is by far the largest database providing facial expressions
with all three types of behavior states, i.e., valence and arousal (VA), basic
expression and facial action unit.

The situation approach derives emotion labels from the situation (with an
anticipated emotion) where facial expressions elicited are taken as a cue. The
BP4D-Spontaneous database [27] used various tasks in a lab environment to
elicit spontaneous facial expressions. For example, smelling an unpleasant
odor should be disgusting and thus the facial expression during the task
should be one of disgust. The Aff-Wild dataset employed similar elicitation
method to record videos containing naturalistic emotional states in arbitrary
recording conditions.

The self-report approach is another way of annotating facial expression type.
The Belfast Induced Natural Emotion Database [28] is unique. Recordings are
accompanied by self-reports of emotion and intensity, serving as continuous
trace-style ratings of valence and intensity. Self-reporting may be the most
reliable approach to labeling emotion in facial expressions. Facial movements
in this database were also annotated with valence and intensity (or arousal)
according to the dimension of the emotion. In Belfast, they used many more
words to describe each emotion than the popular six prototypical facial
expressions. However, self-reported information is not easily collected and
may not reflect the inner mind [29] and thus not commonly used.

Crowdsourcing is a judgment-based approach that labels facial expressions
based on human perception. The Real-world Affective Face Database (RAF-DB) [8,
30] is a real-world database containing 29,672 highly diverse facial images
downloaded from the Internet. Images were randomly and equally assigned to
each labeler, ensuring that there was no direct correlation among the images
labeled by a single annotator. For the manually crowdsourced annotations,
seven basic (including neutral) and 11 compound emotion labels were applied to
the samples. Each image was ensured to be labeled by approximately 40
independent labelers. Later, RAF-ML [9] was developed. This database contains
facial expressions obtained from RAF-DB that offers multiple expressions, and
extends the sample collection, 4,908 in total. Another dataset, the Expression
in-the-Wild Database (ExpW) [10], contains 91,793 faces downloaded using
Google image searches. Each face image was manually annotated and categorized
by human beings into one of seven basic expression categories, but the study
didn’t explain whether this coding was based on heuristic judgement or mapping
rules.

In summary, posed facial expressions are uncommon in real life, so many
databases collect spontaneous samples. These spontaneous and in-the-wild
facial expressions are very difficult to annotate because they do not usually
display commonly accepted and pre-defined AU combinations. Mapping rules
comprise the most common approach to labeling facial expression type. There
are also dimension, situation, self-report, and crowdsourcing approaches. With
images of facial expressions downloaded from the Internet, the subject’s
emotional state and mindset are unknown, and researchers can’t simply infer
them by the AU combinations displayed. Therefore, a crowdsourcing approach
that leverages human perception to apply annotations is the most suitable of
those available. In the present work, we provide a dataset of both judgement-
based (i.e., subjective) and sign-based (i.e., objective) annotations. Only a
combination of subjective and objective annotations will properly disclose the
emotional implications of facial expressions. This is what is provided by RAF-
AU.

##  3 RAF-AU

###  3.1 AU annotations

Facial images from the Internet vary in quality; this is true not only in
terms of image resolution, but also spontaneity, since many are posed and thus
not emotion-motivated. There are certain individual differences in facial
appearance and habitual movement. Some AUs habitually appear, making emotional
perception confusing. For example, a smile with AU9 (i.e., a wrinkled nose) is
more likely to appear in women, while people with gag teeth are more likely to
display AU10 (a raised upper lip). AU coding of facial expressions requires a
baseline (or neutral) image, or it is difficult to judge whether certain
expressions were due to the face’s original appearance or facial movement. For
example, eyebrow AU4 can be a permanent feature or transient movement. If we
are forced to imagine a neutral face for each face image, then we have to
judge whether the expression is caused by a certain AU, perpetual feature, or
the influence of another AU.

In the present research, those with more than two whole faces (305 images in
total) were eliminated from the RAF-ML database. There were some images that
contained more than two faces. When we checked the voting, we found that
participants were not always be voting for the same face. For example, one
image contained a smiling face and a sad face, but about 30% of the
participants voted happiness while 25% voted sadness. After removing these
images, 4,601 were left for further coding.

Two experienced coders independently FACS-coded these face images and
arbitrated any disagreement. They usually disagreed on whether a trace of an
AU was actually a baseline or emotion-motivated. They also carefully checked
and discussed if AUs emerged due to other AUs. The inter-coder correlation
(i.e., reliability, see Eq. (1)) was 0.6376. This relatively low reliability
was due to the complexity of and lack of clarity in these in-the-wild and
blended facial expressions.

$R=\frac{2*\\#AU(C_{1}C_{2})}{\\#All\\_AU},$ (1)

where $\\#AU(C_{1}C_{2})$ is the number of AUs upon which Coders 1 and 2
agreed and $\\#All\\_AU$ is the total number of AUs in a facial expression
scored by two coders.

Figure 2: Examples of “null” images.

There were 253 images annotated as null. Some of these facial images didn’t
show any AUs. Though there were no AUs for these faces, they were still
selected for this database because according to perceivers, they expressed
emotions. This is not uncommon because perceived emotion is influenced by
gaze, head pose, gesture, and facial appearance. Figure 2 shows some examples
labeled as null. In the figure, (a) is null because the faintly discernible
AU17 (i.e., Chin raiser) is not sufficiently obvious; without a baseline, we
weren’t confident we were avoiding a mistake. In (b), the face of this elderly
person has many perpetual traces, and therefore was hard to annotate without a
baseline. The AU4 marker seemed locally obvious. However, this is an image of
an elderly person that did not show a clear vertical grain between the
eyebrows. Therefore, we didn’t annotate this image with AU4. The forehead in
(c) is covered and the mouth may be stretched by an outer force. These AUs are
either invisible or involuntary.

###  3.2 Profile of RAF-AU

This database contains 4,601 images obtained from RAF-ML. We have provided AU
annotations for all faces. There were 26 AUs used in the annotations, without
action descriptors. We list all used AUs and their frequency in Table 1. It
should be noted that the AU distribution was largely imbalanced. For example,
only a few AU39 (i.e., compressed nostrils) were found in this dataset. Some
representative examples with frequent AUs in the RAF-AU are shown in Figure 1.

AU | Name | Num | AU | Name | Num
---|---|---|---|---|---
1 | Inner Brow Raiser | 1028+(49) | 21 | Neck Tightener | 3
2 | Outer Brow Raiser | 701+(95) | 22 | Lip Funneler | 196+(148)
4 | Brow Lowerer | 1808+(9) | 23 | Lip Tightener | 80+(51)
5 | Upper Lid Raiser | 975+(10) | 24 | Lip Pressor | 165+(56)
6 | Cheek Raiser | 404+(46) | 25 | Lips part | 2830
7 | Lid Tightener | 347+(14) | 26 | Jaw Drop | 1089
9 | Nose Wrinkler | 749+(25) | 27 | Mouth Stretch | 810
10 | Upper Lip Raiser | 1274+(116) | 28 | Lip Suck | 13+(31)
12 | Lip Corner Puller | 1187+(81) | 29 | Jaw Thrust | 27
14 | Dimpler | 105+(22) | 30 | Jaw Sideways | 9
15 | Lip Corner Depressor | 290+(2) | 32 | Lip Bite | 6
16 | Lower Lip Depressor | 720 | 33 | Cheek Blow | 1
17 | Chin Raiser | 541 | 34 | Cheek Puff | 9+(1)
18 | Lip Puckerer | 118+(11) | 35 | Cheeck Suck | 10+(1)
19 | Tongue Visible | 34 | 39 | Nostril Compressor | 5
20 | Lip stretcher | 199+(33) | 43 | Eyes Closed | 148+(9)
Table 1: Frequency of all used AUs (without action descriptors) coded by manual FACS coders on the RAF-AU dataset. The numbers in parentheses refer to the sample size of one-sided AUs. Surprise | Fear | Disgust
---|---|---
AU25 | 0.5926 | AU25 | 0.5111 | AU10 | 0.5964
AU5 | 0.4665 | AU12 | 0.4033 | AU4 | 0.5330
AU26 | 0.3820 | AU27 | 0.3729 | AU17 | 0.2973
Happiness | Sadness | Anger
AU12 | 0.7040 | AU4 | 0.6723 | AU25 | 0.4659
AU25 | 0.5143 | AU25 | 0.3462 | AU9 | 0.4337
AU27 | 0.2491 | AU1 | 0.2979 | AU10 | 0.4236
Table 2: AUs that contribute most (variance) for these facial expressions in RAF-AU. | Sur | Fea | Dis | Hap | Sad | Ang | Neu
---|---|---|---|---|---|---|---
Sur | 1.0000 | 0.6802 | 0.1733 | 0.4908 | 0.3284 | 0.3688 | -0.3784
Fea | 0.6802 | 1.0000 | 0.3645 | 0.8168 | 0.5754 | 0.7665 | -0.8262
Dis | 0.1733 | 0.3645 | 1.0000 | 0.2497 | 0.6997 | 0.6266 | -0.4654
Hap | 0.4908 | 0.8168 | 0.2497 | 1.0000 | 0.5048 | 0.6207 | -0.7539
Sad | 0.3284 | 0.5754 | 0.6997 | 0.5048 | 1.0000 | 0.6342 | -0.4866
Ang | 0.3688 | 0.7665 | 0.6266 | 0.6207 | 0.6342 | 1.0000 | -0.8170
Neu | -0.3784 | -0.8262 | -0.4654 | -0.7539 | -0.4866 | -0.8170 | 1.0000
Table 3: The relationship of the facial expressions based on the AU combination for RAF-AU. Sur=Surprise, Fea=Fear, Dis=Disgust, Hap=Happiness, Sad=Sadness, Ang=Anger, Neu=Neutral. | Sur | Fea | Dis | Hap | Sad | Ang | Neu
---|---|---|---|---|---|---|---
Sur | 1.0000 | 0.2706 | -0.5585 | 0.1544 | -0.4168 | -0.3756 | -0.2598
Fea | 0.2706 | 1.0000 | -0.4242 | -0.2193 | -0.1224 | -0.1628 | -0.3568
Dis | -0.5585 | -0.4242 | 1.0000 | -0.2673 | 0.0686 | 0.0739 | 0.1447
Hap | 0.1544 | -0.2193 | -0.2673 | 1.0000 | -0.3023 | -0.2986 | -0.0387
Sad | -0.4168 | -0.1224 | 0.0686 | -0.3023 | 1.0000 | -0.2060 | 0.0817
Ang | -0.3756 | -0.1628 | 0.0739 | -0.2986 | -0.2060 | 1.0000 | -0.1858
Neu | -0.2598 | -0.3568 | 0.1447 | -0.0387 | 0.0817 | -0.1858 | 1.0000
Table 4: The correlation coefficients between the facial expressions for RAF-
AU. Sur=Surprise, Fea=Fear, Dis=Disgust, Hap=Happiness, Sad=Sadness,
Ang=Anger, Neu=Neutral.

Among these AU annotations, some were one-side AUs. Facial action units are
not always the same for left and right halves of faces; they may also vary in
intensity. One-sided AUs were labeled as L or R, depending on which half of
the face acted. Among the 4,601 faces, we found 219 right-only and 304 left-
only AUs. For the unweighted condition (the basic unit was facial expression,
and it did not matter if it contained one or more one-sided AUs), we found 163
right-only and 232 left-only AUs. This finding suggests that the left side of
the face is more expressive than the right. Psychologists may be interested in
these types of characteristics and thus should further investigate this topic.
In addition to the left- and right-only AUs, some AUs only contained top or
bottom elements. For example, AU23 indicates tightened lips, and sometimes
this is only expressed by half (top or bottom). For these instances, we
annotated T for top only and B for bottom only. There were 131 T and 179 B
faces in RAF-AU.

By annotating for both AU and perceived emotion, we were able to explore which
AUs contributed the most to blended emotions. Through a linear transformation,
the AUs and perceived emotions were made equivalent in the same space. That
is, the AUs of each image were linearly transformed to obtain the expression
corresponding to the image. Through matrix transformation, we will be able to
get the linear relationship between AUs and perceived emotions. Accordingly,
the AUs of each image can be linearly transformed to obtain the expression
possibilities corresponding to the image. Table 2 shows the three AUs
contributing the most to each facial expression. We explored the inner
relationship of the expression itself by using the expression matrix and
calculating the relationship of the facial expression based on the AU
combination (see Table 3). In addition, we provide the correlation
coefficients for the facial expressions in RAF-AU (see Table 4).

##  4 Baseline Evaluation on RAF-AU

###  4.1 Pre-processing and dataset split

We first filtered out all 253 images annotated with “null” as they may contain
irrelevant information which would largely distract the AU detection. Then, we
chose 13 kinds of AUs that occur more than 8% base rate in RAF-AU dataset for
experiment and analysis. Specifically, each image was annotated +1 or -1 if an
AU is present or absent, and 0 for one-sided AUs. During pre-processing,
manually annotated five facial landmarks provided in RAF-ML [9] were employed
to register all images to a reference face using an affine transformation,
resulting 100*100 cropped images. Then gray-scale samples were transformed for
the following feature extraction. For dataset split, we divided RAF-AU into
training part and test part, where the size of the training part is four times
larger than the test part.

###  4.2 Feature extraction and classification

For the comparison purpose, we implemented two handcrafted features and three
deep learning features. For hand-crafted feature extraction, we have tried
histogram of orientated gradients (HOG) [31] and Local binary patterns (LBP)
[32]. For HOG, we used this shape-based segmentation dividing the image into
10*10 pixel blocks of four 5*5 pixel cells with no overlapping. By setting 10
bins for each histogram, we got a 4000-dimensional feature vector per aligned
image. For LBP, we selected the 59-bin $LBP_{8,2}^{u2}$ operator, and divided
the 100*100 pixel images into 100 regions of 10*10 grid size, which was
empirically found to achieve relatively good performance for expression
classification.

For deep learning feature extraction, we first employed the already trained
baseDCNN and DBM-CNN provided in [9], then 2000-dimensional deep features
learned from raw data can be extracted from the penultimate fully connected
layer of these two DCNNs for both training set and test set in RAF-AU. We also
tried a multi-label CNN for AU detection. Let us assume that there are $k$ AU
categories and $n$ training samples. Given a set of ground truth label
$\boldsymbol{y}\in\\{-1,+1,0\\}^{k}$ and the corresponding prediction results
$\boldsymbol{p}\in\mathbb{R}^{k}$ for all $k$ AU labels, the goal is to
minimize the following multi-label cross entropy loss:

$L=\frac{-1}{n}\sum_{i=1}^{n}\sum_{j=1}^{k}\left\\{[y_{i}^{j}\textgreater
0]\log p_{i}^{j}+[y_{i}^{j}\textless 0]\log(1-p_{i}^{j})\right\\},$ (2)

where $[\cdot]$ is an indicator function that returns 1 if the Boolean
expression is true, and 0 otherwise. And $y_{i}^{j}$ is the ground truth for
the $i$-th sample of $j$-th AU, $p_{i}^{j}$ is the predicted probability for
the $i$-th sample of $j$-th AU. We then also extracted the output of the
penultimate fully connected layer as the final feature representation,
resulting in 2000-dimensional vectors. Table 5 displays detailed network
architectures of this AU-CNN trained on the RAF-AU dataset.

Layer Type | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
---|---|---|---|---|---|---|---|---|---
Conv | ReLu | MPool | Conv | ReLu | MPool | Conv | ReLu | Conv
Kernel | 3 | - | 2 | 3 | - | 2 | 3 | - | 3
output | 64 | - | - | 96 | - | - | 128 | - | 128
Stride | 1 | 1 | 2 | 1 | 1 | 2 | 1 | 1 | 1
Pad | 1 | 0 | 0 | 1 | 0 | 0 | 1 | 0 | 1
Layer Type | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18
ReLu | MPool | Conv | ReLu | Conv | ReLu | FC | ReLu | FC
Kernel | - | 2 | 3 | - | 3 | - |  | - |
output | - | - | 256 | - | 256 | - | 2000 | - | 13
Stride | 1 | 2 | 1 | 1 | 1 | 1 |  | 1 |
Pad | 0 | 0 | 1 | 0 | 1 | 0 |  | 0 |
Table 5: The network configuration parameters in the AU-CNN.

During AU detection, support vector machine with linear kernel implemented in
LIBSVM [33] was utilized for the one-versus-all binary classification. Given a
training set $\\{(x_{i},y_{i}),i=1,\dots,n\\}$, where $x_{i}\in\mathbb{R}^{d}$
and $y_{i}\in\\{+1,-1\\}$ (samples with one-sided AU has been omitted during
this AU detection), then the testing sample can be detected by optimizing:

$\min_{w}\frac{1}{2}||w||^{2}+C\sum_{i=1}^{n}max(1-y_{i}w^{T}x_{i},0).$ (3)

The penalty parameter $C$ of SVM was fixed to 1 for all different features.

AUs |  Number |  HOG |  LBP |  BaseDCNN |  DBM-CNN |  AU-CNN
---|---|---|---|---|---|---
1 |  1028 |  76.83 |  78.72 |  72.31 |  71.74 |  82.03
2 |  701 |  84.67 |  87.76 |  83.54 |  80.67 |  90.49
4 |  1808 |  72.95 |  74.27 |  75.67 |  78.73 |  86.28
5 |  975 |  78.19 |  76.43 |  83.25 |  87.84 |  88.86
6 |  404 |  84.61 |  80.68 |  81.98 |  84.29 |  87.41
9 |  749 |  84.09 |  88.49 |  86.79 |  86.76 |  90.03
10 |  1274 |  78.98 |  76.63 |  80.85 |  81.56 |  87.89
12 |  1187 |  80.51 |  82.27 |  86.23 |  86.17 |  88.48
16 |  720 |  78.47 |  77.84 |  77.71 |  79.67 |  86.11
17 |  541 |  81.81 |  79.55 |  80.07 |  78.29 |  88.58
25 |  2830 |  85.13 |  86.24 |  89.87 |  91.18 |  95.33
26 |  1089 |  66.27 |  67.72 |  75.12 |  78.82 |  86.31
27 |  810 |  91.48 |  93.11 |  93.42 |  93.65 |  95.77
AVG |  - |  80.31 |  80.75 |  82.06 |  83.03 |  88.73
Table 6: Performances of the AUC-ROC for AU detection on RAF-AU dataset using
different features.

###  4.3 Evaluation metrics

In terms of evaluating the performance on AU detection, two different metrics
were employed: the area underneath the receiver-operator characteristic (ROC)
curve (AUC-ROC) and the F1 score. The rank metric ROC curve visualizes the
trade-off between sensitivity and specificity by plotting both values as a
function of a varying classification threshold. And the threshold metric F1
score is defined as $F1=\frac{2RP}{R+P}$, where $R$ and $P$ denote recall (the
number of correctly recognized samples divided by the actual number of all
samples with the target AU) and precision (the number of correctly recognized
samples divided by the total number of samples detected with the target AU),
respectively. We then computed the average over all 13 AUs (AVG) to measure
the overall performance.

###  4.4 AU detection results

In Table 6 and Table 7, we show the AUC-ROC and F1 score results for each of
the 13 AUs in RAF-AU using four different features, respectively. We also list
the statistic regarding the AU occurrence, i.e., the number of positive
samples for each AU. It can be seen that the AU distribution in RAF-AU is more
imbalanced than those lab-controlled datasets. With regard to individual AU
detection, our baselines can yield relatively good performance on frequently
occurring AUs (e.g., AUs 25, 27, 5, 4 and 12). However, we also observe a
significant drop in the performance on other less common AUs. When comparing
different features, the deep learning feature AU-CNN can achieve comparable
and better AU detection rates in terms of both average AUC and F1 score.
Nevertheless, when compared to the accuracy achieved on other lab-controlled
datasets, there is still room for improvement on this challenging realistic
dataset which contains various naturalistic illuminations, occlusions, hear
poses and obvious imbalanced distribution.

AUs |  Number |  HOG |  LBP |  BaseDCNN |  DBM-CNN |  AU-CNN
---|---|---|---|---|---|---
1 |  1028 |  33.70 |  50.00 |  29.60 |  35.42 |  60.47
2 |  701 |  45.00 |  50.23 |  38.50 |  32.49 |  65.59
4 |  1808 |  58.86 |  57.89 |  62.35 |  65.47 |  73.44
5 |  975 |  45.43 |  37.29 |  51.53 |  59.52 |  69.69
6 |  404 |  26.09 |  26.80 |  17.78 |  18.18 |  58.21
9 |  749 |  52.94 |  59.29 |  52.24 |  51.48 |  67.44
10 |  1274 |  49.30 |  49.65 |  49.52 |  51.99 |  68.41
12 |  1187 |  54.72 |  60.29 |  64.76 |  62.09 |  69.62
16 |  720 |  21.65 |  27.03 |  25.37 |  31.63 |  59.38
17 |  541 |  7.27 |  20.74 |  18.60 |  16.26 |  25.64
25 |  2830 |  85.07 |  85.16 |  87.50 |  88.87 |  92.14
26 |  1089 |  20.00 |  17.90 |  42.11 |  48.84 |  64.65
27 |  810 |  68.69 |  72.11 |  70.99 |  71.23 |  82.67
AVG |  - |  43.75 |  47.26 |  46.99 |  48.73 |  65.95
Table 7: Performances of the F1 score for AU detection on RAF-AU dataset using
different features.

##  5 Conclusions

In real life, facial expressions occur in the wild, and thus their emotional
meaning cannot be absolutely defined. The crowdsourcing approach provides a
subjective label based on human perception. Thus, we update the previous RAF-
ML database by providing AU coding and removing certain confusing samples. The
present database, RAF-AU, provides both AU- and judgement-based annotations
from objective and subjective approaches, respectively. Thus, this database
provides a fuller picture of given facial expressions. Based on these
annotations, we were able to investigate the relationship between objective
description and subjective understanding. We also provide a set of baselines
for RAF-AU depending on different features. And the deep learning feature
obtained using a multi-label AU detection CNN achieve the best detection rate.
Further research should be conducted to study this relationship, with more
samples and across various cultures.

##  6 Acknowledgement

This work was partially supported by the National Natural Science Foundation
of China under Grants No. 61871052. We thank Yong Ke, Huaxiu Li for their
assistance in AU coding.

## References

  * [1] Harrigan, J., Rosenthal, R., Scherer, K.R., Scherer, K.:  New handbook of methods in nonverbal behavior research.  Oxford University Press (2008)
  * [2] Ekman, P., Friesen, W.V., Hager, J.C.:  Facial action coding system: The manual on cd rom.  A Human Face, Salt Lake City (2002) 77–254
  * [3] Russell, J.A., Rosenberg, E.L., Lewis, M.D.:  Introduction to a special section on basic emotion theory.  Emotion Review 3 (2011) 363
  * [4] Kanade, T., Cohn, J.F., Tian, Y.:  Comprehensive database for facial expression analysis.  In: Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580), IEEE (2000) 46–53
  * [5] Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.:  The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression.  In: Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference on, IEEE (2010) 94–101
  * [6] Fabian Benitez-Quiroz, C., Srinivasan, R., Martinez, A.M.:  Emotionet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2016) 5562–5570
  * [7] Mollahosseini, A., Hasani, B., Mahoor, M.H.:  Affectnet: A database for facial expression, valence, and arousal computing in the wild.  IEEE Transactions on Affective Computing PP (2017) 1–1
  * [8] Li, S., Deng, W.:  Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition.  IEEE Transactions on Image Processing 28 (2018) 356–370
  * [9] Li, S., Deng, W.:  Blended emotion in-the-wild: Multi-label facial expression recognition using crowdsourced annotations and deep locality feature learning.  International Journal of Computer Vision 127 (2019) 884–906
  * [10] Zhang, Z., Luo, P., Loy, C.C., Tang, X.:  From facial expression recognition to interpersonal relation prediction.  International Journal of Computer Vision 126 (2018) 550–569
  * [11] Yan, W.J., Wu, Q., Liang, J., Chen, Y.H., Fu, X.:  How fast are the leaked facial expressions: The duration of micro-expressions.  Journal of Nonverbal Behavior 37 (2013) 217–230
  * [12] Friesen, W.V., Ekman, P., et al.:  Emfacs-7: Emotional facial action coding system.  Unpublished manuscript, University of California at San Francisco 2 (1983) 1
  * [13] Ekman, P., Friesen, W.V.:  Nonverbal leakage and clues to deception.  Psychiatry 32 (1969) 88–106
  * [14] Du, S., Tao, Y., Martinez, A.M.:  Compound facial expressions of emotion.  Proceedings of the National Academy of Sciences 111 (2014) E1454–E1462
  * [15] Ruiz, A., van de Weijer, J., Binefa, X.:  From emotions to action units with hidden and semi-hidden-task learning.  In: 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, IEEE Computer Society (2015) 3703–3711
  * [16] Durán, J., Reisenzein, R., Fernández-Dols, J.M.:  Coherence between emotions and facial expressions.  The science of facial expression (2017) 107–129
  * [17] Kollias, D., Sharmanska, V., Zafeiriou, S.:  Face behavior à la carte: Expressions, affect and action units in a single network.  CoRR abs/1910.11111 (2019)
  * [18] Fasel, B., Luettin, J.:  Automatic facial expression analysis: a survey.  Pattern recognition 36 (2003) 259–275
  * [19] Ko, B.:  A brief review of facial emotion recognition based on visual information.  sensors 18 (2018) 401
  * [20] Li, S., Deng, W.:  Deep facial expression recognition: A survey.  IEEE Transactions on Affective Computing (2020) 1–1
  * [21] Yin, L., Wei, X., Sun, Y., Wang, J., Rosato, M.J.:  A 3d facial expression database for facial behavior research.  In: 7th international conference on automatic face and gesture recognition (FGR06), IEEE (2006) 211–216
  * [22] Pantic, M., Valstar, M., Rademaker, R., Maat, L.:  Web-based database for facial expression analysis.  In: 2005 IEEE international conference on multimedia and Expo, IEEE (2005) 5–pp
  * [23] Kollias, D., Tzirakis, P., Nicolaou, M.A., Papaioannou, A., Zhao, G., Schuller, B.W., Kotsia, I., Zafeiriou, S.:  Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures, and beyond.  Int. J. Comput. Vis. 127 (2019) 907–929
  * [24] Zafeiriou, S., Kollias, D., Nicolaou, M.A., Papaioannou, A., Zhao, G., Kotsia, I.:  Aff-wild: Valence and arousal ’in-the-wild’ challenge.  In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2017, Honolulu, HI, USA, July 21-26, 2017, IEEE Computer Society (2017) 1980–1987
  * [25] Kollias, D., Zafeiriou, S.:  Expression, affect, action unit recognition: Aff-wild2, multi-task learning and arcface.  In: 30th British Machine Vision Conference 2019, BMVC 2019, Cardiff, UK, September 9-12, 2019, BMVA Press (2019) 297
  * [26] Kollias, D., Schulc, A., Hajiyev, E., Zafeiriou, S.:  Analysing affective behavior in the first ABAW 2020 competition.  CoRR abs/2001.11409 (2020)
  * [27] Zhang, X., Yin, L., Cohn, J.F., Canavan, S., Reale, M., Horowitz, A., Liu, P., Girard, J.M.:  Bp4d-spontaneous: a high-resolution spontaneous 3d dynamic facial expression database.  Image and Vision Computing 32 (2014) 692–706
  * [28] Sneddon, I., McRorie, M., McKeown, G., Hanratty, J.:  The belfast induced natural emotion database.  IEEE Transactions on Affective Computing 3 (2011) 32–41
  * [29] Barrett, L.F.:  Feelings or words? understanding the content in self-report ratings of experienced emotion.  Journal of personality and social psychology 87 (2004) 266
  * [30] Li, S., Deng, W., Du, J.:  Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild.  In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2017) 2852–2861
  * [31] Dalal, N., Triggs, B.:  Histograms of oriented gradients for human detection.  In: CVPR, on. Volume 1., IEEE (2005) 886–893
  * [32] Ojala, T., Pietikäinen, M., Mäenpää, T.:  Multiresolution gray-scale and rotation invariant texture classification with local binary patterns.  Pattern Analysis and Machine Intelligence, IEEE Transactions on 24 (2002) 971–987
  * [33] Chang, C.C., Lin, C.J.:  LIBSVM: A library for support vector machines.  ACM Transactions on Intelligent Systems and Technology 2 (2011) 27:1–27:27 Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.