# Playing for 3D Human Recovery

Zhongang Cai⋆ Mingyuan Zhang⋆ Jiawei Ren⋆ Chen Wei Daxuan Ren
Zhengyu Lin Haiyu Zhao Lei Yang Chen Change Loy Ziwei Liu${}^{\href
<EMAIL_ADDRESS>$\star$ indicates equal contributions. Zhongang
Cai, Mingyuan Zhang, Jiawei Ren, Daxuan Ren, Chen Change Loy and Ziwei Liu are
with the S-Lab, Nanyang Technological University, Singapore, 639798. Zhongang
Cai, Haiyu Zhao, Chen Wei, Zhengyu Lin, and Lei Yang are with Shanghai AI
Laboratory. The corresponding author is Ziwei Liu<EMAIL_ADDRESS>
###### Abstract

Image- and video-based 3D human recovery (_i.e_., pose and shape estimation)
have achieved substantial progress. However, due to the prohibitive cost of
motion capture, existing datasets are often limited in scale and diversity. In
this work, we obtain massive human sequences by playing the video game with
automatically annotated 3D ground truths. Specifically, we contribute GTA-
Human, a large-scale 3D human dataset generated with the GTA-V game engine,
featuring a highly diverse set of subjects, actions, and scenarios. More
importantly, we study the use of game-playing data and obtain five major
insights. First, game-playing data is surprisingly effective. A simple frame-
based baseline trained on GTA-Human outperforms more sophisticated methods by
a large margin. For video-based methods, GTA-Human is even on par with the in-
domain training set. Second, we discover that synthetic data provides critical
complements to the real data that is typically collected indoor. Our
investigation into domain gap provides explanations for our data mixture
strategies that are simple yet useful. Third, the scale of the dataset
matters. The performance boost is closely related to the additional data
available. A systematic study reveals the model sensitivity to data density
from multiple key aspects. Fourth, the effectiveness of GTA-Human is also
attributed to the rich collection of strong supervision labels (SMPL
parameters), which are otherwise expensive to acquire in real datasets. Fifth,
the benefits of synthetic data extend to larger models such as deeper
convolutional neural networks (CNNs) and Transformers, for which a significant
impact is also observed. We hope our work could pave the way for scaling up 3D
human recovery to the real world. Homepage:
https://caizhongang.github.io/projects/GTA-Human/.

###### Index Terms:

Human Pose and Shape Estimation, 3D Human Recovery, Parametric Humans,
Synthetic Data, Dataset.

##  1 Introduction

Image- and video-based 3D human recovery, _i.e_., simultaneous estimation of
human pose and shape via parametric models such as SMPL [1], have transformed
the landscape of holistic human understanding. This technology is critical for
entertainment, gaming, augmented and virtual reality industries. However,
despite that the exciting surge of deep learning is arguably driven by
enormous labeled data [2, 3], the same is difficult to achieve in this field.
The insufficiency of data (especially in the wild) is attributed to the
prohibitive cost of 3D ground truth (particularly parametric model annotation)
[4]. Existing datasets are either small in scale [5, 6, 7], collected in
constrained indoor environment [8, 9, 10], or not providing the 3D parametric
model annotation at all [11, 12, 13, 14].

Inspired by the success of training deep learning models with video game-
generated data for various computer vision tasks such as instance segmentation
[15], 2D keypoint estimation [16], motion prediction [17], mesh reconstruction
[18], detection and tracking [19], we present GTA-Human in the hope to address
the aforementioned limitations of existing datasets. GTA-Human is built by
coordinating a group of computational workers (Figure 2) that simultaneously
play the popular video game Grand Theft Auto V (GTA-V), to put together a
large-scale dataset (Table I) with 1.4 million SMPL parametric labels
automatically annotated in 20 thousand video sequences. Besides the scale,
GTA-Human explores the rich resources of the in-game database to diversify the
data distribution that is challenging to achieve in real life (Figure 3, 4 and
5): more than 600 subjects of different gender, age, ethnicity, body shape and
clothing; 20,000 action clips comprising a wide variety of daily human
activities; six major categories of locations with drastically different
backgrounds from city streets to the wild; camera angles are manipulated in
each sequence to reflect a realistic distribution; subject-environment
interaction that gives rises to occlusion of various extents; time of the day
that affects lighting conditions, and weather system that mimics the real
climate changes.

Figure 1: GTA-Human dataset is built from GTA-V, an open-world action game that features a reasonably realistic functioning metropolis and virtual characters living in it. Our customized toolchain enables large-scale collection and annotation of highly diverse human data that we hope aid in-depth studies on 3D human recovery. We show here a few examples with SMPL annotations overlaid on the virtual humans. TABLE I: 3D human dataset comparisons. We compare GTA-Human with existing real datasets with SMPL annotations and synthetic datasets with highly realistic setups. GTA-Human has competitive scale and diversity. Datasets are divided into three types: real, synthetic and mixed. GTA-Human samples character action sequences from a large in-game database that allows a unique action to be assigned to each video sequence. Note that EFT [20] re-annotates 2D human pose estimation datasets where the number of subjects are difficult to trace. *: 3DPW and Panoptic Studio only have general descriptions of scene activities Dataset | Year | Type | In-the-Wild | Video | #SMPL | #Sequence | #Subject | #Action
---|---|---|---|---|---|---|---|---
HumanEva [5] | 2009 | Real | - | ✓ | NA | 7 | 4 | 6
Human3.6M [8] | 2013 | Real | - | ✓ | 312K | 839 | 11 | 15
MPI-INF-3DHP [21] | 2017 | Mixed | ✓ | ✓ | 96K | 16 | 8 | 8
3DPW [6] | 2018 | Real | ✓ | ✓ | 32K | 60 | 18 | *
Panoptic Studio [9] | 2019 | Real | - | ✓ | 736K | 480 |  $\sim$100 | *
EFT [20] | 2020 | Real | ✓ | - | 129K | NA | Many | NA
SMPLy [7] | 2020 | Real | ✓ | ✓ | 24K | 567 | 742 | NA
AGORA [22] | 2021 | Synthetic | ✓ | - | 173K | NA |  $>$350 | NA
GTA-Human | 2022 | Synthetic | ✓ | ✓ | 1.4M | 20K |  $>$600 | 20K

Equipped with GTA-Human, we conduct an extensive investigation in the use of
synthetic data for 3D human recovery. 1) Better 3D human recovery with data
mixture. Despite the seemingly unavoidable domain gaps, we show that practical
settings that mix synthetic data with real data, such as blended training and
pretraining followed by finetuning, are surprisingly effective. First, HMR
[23], one of the first deep learning-based methods for SMPL estimation with
relatively simplistic architecture, when trained with data mixture, is able to
outperform more recent methods with sophisticated designs or additional
information such as SPIN [24] and VIBE [25]. Moreover, PARE [26], a state-of-
the-art method also benefit considerably from GTA-Human. Second, our
experiments on the video-based method VIBE [25] further demonstrate the
effectiveness of data mixture: an equal amount of synthetic GTA-Human data is
as good as a real-captured indoor dataset as the frame feature extractor is
already pretrained on real datasets; the full set of GTA-Human is even on par
with in-domain training data.

2) Closing the domain gap with synthetic data. We then study the reasons
behind the effectiveness of game-playing data. An investigation into the
domain gaps provides insights into the complementary nature of synthetic and
real data: despite the reality gap, the synthetic data embodies the diversity
that most of the real data lack, as the latter is typically collected indoors.
Moreover, we experiment with mainstream domain adaptation methods to further
close the domain gaps and obtain improvements.

3) Dataset scale matters. We demonstrate that adding game-playing data
progressively improves the model performance. Considering the difficulty of
collecting real data with ground truth 3D annotations, synthetic data may thus
be an attractive alternative. Moreover, a multi-factor analysis reveals that
supervised learning leads to severe sensitivity to data density. Amongst
factors such as camera angles, pose distributions, and occlusions, a
consistent drop in performance is observed where data is scarce. Hence, our
observation suggests synthetic datasets may play a vital role to supplement
corner case scenarios in the future.

4) Strong supervision (SMPL) is key. Compared to large-scale pose estimation
benchmarks that only provide 3D keypoints, we demonstrate that strong
supervision in the form of SMPL parameters may be quintessential for training
a strong model. We discuss the potential reasons behind this observation,
which reaffirms the value of GTA-Human as a scalable training source with SMPL
annotations.

5) Big data benefits big models. Despite recent development in deeper
convolutional networks [27, 28] and vision transformers [29, 30] in computer
vision research, the mainstream backbone size remains unchanged for 3D human
recovery [23, 24, 25]. We extend our study to deeper CNNs and Transformers,
and show that training with GTA-Human not only gives rise to improvements but
allows smaller backbones to outperform larger counterparts.

##  2 Related Work

###  2.1 3D Human Recovery

Human Parametric Models. Human pose and shape estimation is typically
performed with 3D human parametric models, such as SMPL [1], SMPL-X [31] and
STAR [32], which take in parameters that represent pose and shape of the human
subject, and output 3D human mesh via linear blend skinning. We base our
discussion on SMPL version 1.0 in this work that consists of pose parameters
$\theta\in\mathbb{R}^{72}$ and shape parameters $\beta\in\mathbb{R}^{10}$, for
its popularity.

Registration-based Methods. As the output of the human parametric model is
manipulated by body parameters, SMPLify [33] and the following SMPLify-X [31]
are the pioneering works to optimize these parameters to minimize the distance
between ground truth 2D keypoints and reprojected human mesh joints. SMPLify
is also extended to videos with temporal constraints employed [34]. Although
optimization-based methods are able to achieve impressive results, they are
slow and typically take more than 60 seconds per frame. Hence, recent work
[35] has been proposed to accelerate optimization.

Regression-based Methods. Direct regression of body parameters using a trained
deep learning model has gained more popularity due to fast inference. The
recent works are categorized into image-based [36, 37, 38, 39, 40, 41, 26],
and video-based [42, 43, 44, 45, 46, 47] methods. HMR [23] is a pioneering
end-to-end deep learning-based work, which takes ResNet-50 [27] as its
backbone and directly regress the parameters of $\theta$ and $\beta$. VIBE
[25] is a milestone video-based work that leverages temporal information for
realistic pose sequences. Recently, a transformer encoder is introduced for
vertex-joint reweighting [48], but the method still uses a CNN backbone for
feature extraction.

Mixed Methods. There is a line of work that combines optimization-based and
regression-based techniques. SPIN [24], adds a SMPLify step to produce pseudo
parametric labels to guide the learning of the network. SPIN address the lack
of SMPL annotation but the optimization step results in slow training. Others
propose to refine the per-frame regression results by bundle adjustment of the
video sequence as a whole [49], designing a new swing-twist representation to
replace the original axis-angle representation of SMPL [40], and finetuning a
trained network to obtain refined prediction [20], and employing a network to
predict a parameter update rule in iterations of optimization [50].

###  2.2 Datasets

Datasets with 2D Keypoint Annotations. Many datasets contain in-the-wild
images, albeit the lack of SMPL annotations, they provide 2D keypoint labels.
Datasets such as LSP [11], LSP-Extended [12], COCO [13] and MPII [14] contain
images crawled from the Internet, and are annotated with 2D keypoints
manually. Such a strategy allows a large number of in-the-wild images to be
included in the dataset. To obtain 3D annotations that are crucial to human
pose and shape estimation, a common method is to fit an SMPL model on 2D
keypoints. SSP-3D [51] and 3DOH50K [52] leverages pre-trained model to perform
keypoint estimation as the first step, whereas UP-3D [53] and EFT [20]
performs fitting on ground truth keypoints. However, these datasets typically
suffer from the inherent depth ambiguity of images and the pseudo-SMPL may not
have the accurate scale.

Real Datasets. Motion capture facilities are built to achieve high-accuracy 3D
annotations. HumanEva [5] and Human3.6M [8] employ optical motion capture
systems, but intrusive markers are needed to be placed on the subjects. Total
Capture [54] MuPoTS-3D [55], Panoptic Studio [9], and HUMBI [10] make use of
multiple camera views and require no intrusive marker. However, the background
is constant and thus lacks diversity. 3DPW [6] combines inertial measurement
units (IMUs) and a moving camera to build an in-the-wild dataset with 3D
annotations. 3DPW has become an important benchmark for 3D human recovery.
Nevertheless, the IMU drift is still an obstacle and the dataset only contains
a relatively small number of videos. SMPLy [7] constructs point clouds from
multi-view capture of static people and fits SMPL on them. However, the scale
of the dataset is limited by the difficulty of collecting videos that meet the
special setup requirement. HuMMan [56] is the most recent large-scale multi-
modal 4D human dataset.

Synthetic or Mixed Datasets. SURREAL [57], Hoffmann _et al_. [58] render
textured SMPL body models in real-image backgrounds. However, this strategy
does not account for the geometry of the clothes, where the mismatch may
result in unrealistic subjects. 3DPeople [59] uses clothed human models while
MPI-INF-3DHP [21] takes segmented subjects from images and paste them onto new
backgrounds in the training set. However, the subject-background interaction
is still unnatural. AGORA [22] is a recent synthetic dataset featuring high-
quality annotations by rendering real human scans in a virtual world. However,
the dataset is image-based and does not support the training of video-based
methods. Richter _et al_. [60, 15], Krähenbühl _et al_. [61], JTA [16], GTA-IM
[17], SAIL-VOS 3D [18], MOTSynth [19] have demonstrated the potential of
obtaining nearly free and perfectly accurate annotations from video games for
various computer vision tasks. However, these datasets do not provide SMPL
annotation needed for our investigation. We take inspirations from these works
in building GTA-Human.

##  3 GTA-Human Dataset

Figure 2: Data collection toolchain. Our toolchain is highly scalable as the
cloud services are used to coordinate a large number of computation workers.
Left: the overview of the pipeline. Top right: an elaborate illustration of
Local GUI Worker. Bottom right: an elaborate illustration of Cluster Worker.

A scale comparison between GTA-Human and existing dataset is shown in Table I.
GTA-Human features 1.4 million individual SMPL annotations, which is highly
competitive compared to other real datasets and synthetic datasets with
realist setups. Moreover, GTA-Human consists of in-the-wild scenes that are
expensive and difficult to collect in real life. Notably, GTA-Human provides
video sequences instead of static frames and supports video-based human
recovery.

###  3.1 Toolchain

Inspired by existing works that use GTA-generated data for various vision
tasks [60, 15, 61, 16, 17, 18, 19], our toolchain extracts ground truth 2D and
3D keypoints, semantic and depth maps from the game engine, followed by
fitting SMPL models on the keypoints with temporal constraints. To achieve
scalability and efficiency, we design and deploy an automatic system (Fig. 2)
that leverages cloud-based services for parallel deployment and coordination
of our tools on a large number of computer instances and GPU cluster nodes.
GTA-Human consists of sequences of single-person scenes. More examples in GTA-
Human are found in Fig. 6.

Cloud-based NoSQL Database. The unit of data in GTA-Human is a single video
sequence. Hence, we employ a Database that is hosted on the cloud, to track
the progress of data generation and processing of each sequence. The status of
a sequence is updated at each stage in the toolchain, which we elaborate on
the details below.

Scenario File Generator. This tool reads from the Database to retrieve
sequence IDs that are either not generated before or failed in previous
processing attempts, and produce random scene attributes such as subject ID,
action ID, location in the 3D virtual world, camera position and orientation,
lighting, and weather settings.

Cloud-based Message FIFO Queue. The Message FIFO Queue parse the scenario
files from the Scenario File Generator as text strings, which can be fetched
in the first-in-first-out (FIFO) manner by multiple Local GUI Workers. Note
that the queue allows for multiple workers to retrieve their next jobs
simultaneously.

Local GUI Workers. We purchase multiple copies of GTA-V and install them on
regular gaming desktops. We refer to these desktops as Local GUI Workers. Each
worker runs three tools: Scenario Controller, Data Collector, and Data
Analyser that we elaborate on below.

Scenario Controller. Taking scenario files as the input, Scenario Controller
is essentially a plugin that interacts with the game engine via the designated
Application Programming Interface (API). It is thus able to control the
subject generation and placement, action assignment to the subject, camera
placement, in-game time, and weather.

Data Collector. This tool obtains data and some annotations from the API
provided by GTA-V. First, it extracts 3D keypoints from each subject via the
API provided by GTA-V. In addition to the original 98 keypoints available, we
further obtain head top [16] and nose from interpolation of existing
keypoints. We project 3D keypoints to the image plane with known intrinsic and
extrinsic parameters of the camera to obtain 2D keypoints. Second, we project
light rays at each joint to determine if the joint is occluded or self-
occluded by checking the entity that the light ray hits first [16]. Third, our
tool intercepts the rendering pipeline, powered by DirectX, for depth maps and
semantic masks. The pixel-wise depth is directly read from depth buffers.
Shader injection enables the segmentation of individual patches, and we
manually assign the semantic class to various shaders based on their variable
names. We refer interested readers to [61] for more details. Fourth, the
collector also records videos.

Data Analyser. To filter out low-quality data in the early stage, Data
Analyser imposes several constraints on 3D keypoints obtained. We compute
joint movement speed simply as the position different in consecutive frames to
filter out less expressive actions (slow-moving or stationary actions).
Severely occluded, or out-of-view subjects are also flagged at this stage. If
sequences pass the analysis, their data are transferred from the local storage
to a centralized storage space on our GPU cluster (Cluster Storage) for
further processing. The failed ones, however, are deleted. The Database is
notified of the result to get the status updated.

Cluster Workers and SMPL Annotator. On each Cluster Worker (a GPU in the
cluster), we run an instance of SMPL Annotator that takes keypoint annotation
from the Cluster Storage. We upgrade SMPLify [33] in two ways to obtain
accurate SMPL annotation. 1) we find out that compared to 2D keypoints that
have inherent depth ambiguity, exacerbated by weak perspective projection
[62], 3D keypoints are unambiguous. Minor modifications are needed to replace
the 2D keypoint loss of the original SMPLify with 3D keypoint loss. 2) Taking
advantage of the fact that GTA-Human consists of video sequences instead of
unrelated images, temporal consistency in the form of rotation smoothing and
unified shape parameters are enforced. The SMPL parameters include $\theta$
and $\beta$, and an additional translation vector, are optimized at an average
of one second per frame. We visualize more examples in GTA-Human that are
produced with our SMPL annotation tool in Fig. 6.

###  3.2 Data Diversity

Figure 3: Data diversity in GTA-Human. (a) GTA-Human contains subjects of
varied genders, ages, ethnic groups, clothing and body shapes. (b) locations
with diverse backgrounds. The example locations are pinpointed on the 3D game
world map. We discover in Section 4.2 that the outdoor scenes are critical to
the usefulness of GTA-Human. (c) Different weather conditions. (d) In-game
time are set to capture diverse lighting conditions. We capture the same scene
at one game hour interval. Note the shadow direction is affected by the sun
position.

Due to the difficulty and cost of data collection and SMPL annotation for the
3D human recovery task, most existing datasets are built at restrictive
locations such as indoor studios or laboratory environments. Furthermore, only
a small number of subjects are usually employed to perform a limited set of
actions. In contrast, GTA-Human is designed to maximize the variety in the
following aspects. We demonstrate the diversity in subjects, locations,
weather, and time (light conditions) in Fig. 3, actions in Fig. 4, and camera
angles in Fig. 5.

Subjects. GTA-Human collects over 600 subjects of different genders, ages,
ethnicities, clothing, and body shapes for a wide coverage of human
appearances. In addition, unlike motion capture systems in real life that rely
on intrusive markers to be placed on the subjects, accurate skeletal keypoints
are obtained directly from the game’s API.

Figure 4: Actions. GTA-Human contains 20 thousand actions that are expressive
and diverse. (a) The distribution of poses in GTA-Human and real datasets are
visualized after PCA dimension reduction. (b) We show five pose sequences,
represented by curves. Representative frames of sequence 1-5 are indicated by
the diamond-shaped nodes. Datasets are downsampled proportionally.

Actions. Existing datasets either design a small number of actions [5, 8, 21],
or lack a clearly defined action set [6, 9]. In contrast, we gain access to a
large database of motion clips (actions) that can be used to manipulate the
virtual characters, whose typical length is 30-80 frames at 30 FPS. These
actions provide a fairly holistic representation of city-dwellers’ daily
activities, and are reasonably realistic because they are originally produced
via motion capture of real human actors or actresses. We select 20,000 most
dynamic and expressive actions. In Fig. 4, the distribution of GTA-Human poses
does not only have the widest spread, but also covers existing poses in the
real datasets to a large extent. Note that these actions allows for the study
on video-based methods in Section 4.2.

Locations. The conventional optical [5, 8] or multi-view motion capture
systems [9, 10] require indoor environments, resulting in the scarcity of in-
the-wild backgrounds. Thanks to the open-world design of GTA, we have seamless
access to various locations with diverse backgrounds, from city streets to the
wilderness. Our investigation in Section 4.2 highlights these diverse
locations are complementary to real datasets that are typically collected
indoor.

Figure 5: Camera angles. (a) Visualization of camera angles sampled from
various datasets, normalized to a unit sphere. (b) Elevation angle (up-down,
with positive value indicating a camera placed higher than the waist and
looking down) distributions. The vertical axis represents normalized data
density. The colors of the points in (a) and line plots in (b) represent
different datasets, shown in the legend in (b).

Camera Angle. Recent studies [63, 64] have shown the critical impact of camera
angles on model performance, yet its effect in 3D human recovery is not fully
explored due to data scarcity: it is common to have datasets with fixed camera
positions [5, 8, 9, 21]. In GTA-Human, we choose to sample random camera
positions from the distribution of the real datasets [8, 21, 6] to balance
both diversity and realness. Our data collection tool enables the control of
camera placement position and orientation, thus allowing the study on camera
angles that is otherwise difficult in real life. We visualize the camera
angles in Fig. 5.

Figure 6: More examples of GTA-Human. We highlight that GTA-Human is a large-
scale, highly diverse (in terms of factors such as subjects, actions,
locations, and camera angles) dataset. Each frame of the video clips is
annotated with SMPL parameters.

Interaction. Compared to existing works that crop and paste subjects onto
random backgrounds [57, 21], the subjects in GTA-Human are rendered together
with the scenes to achieve a more realistic subject-environment interaction
empowered by the physics engine. Interesting examples include the subject
falling off the edge of a high platform, and the subject stepping into a muddy
pond causing water splashing. Moreover, taking advantage of the occlusion
culling mechanism [16], we are able to annotate the body joints as “visible”
to the camera, “occluded” by other objects, or “self-occluded” by the
subject’s own body parts.

Lighting and Weather. Instead of adjusting image exposure to mimic different
lighting, we directly control the in-game time to sample data around the
clock. Consequently, GTA-Human contains drastically different lighting
conditions and shadow projections. We also introduce random weather conditions
such as rain and snow to the scenes that would be otherwise difficult to
capture in real life.

##  4 Experiments

In this section, we study how to use game-playing data for 3D human recovery
for real-life applications.

###  4.1 Experiment Details

Datasets. We follow the original training convention of our baseline methods
[23, 24, 26]. we define the “Real” datasets used in the experiments to include
Human3.6M [8] (with SMPL annotations via MoSh [65]), MPI-INF-3DHP [21], LSP
[11], LSP-Extended [12], MPII [14] and COCO [13]. “Real” datasets consist of
approximately 300K frames. ”Blended” datasets are formed by simply mixing GTA-
Human data with the “Real” data. Amongst the standard benchmarks, 3DPW [6] has
60 sequences (51k frames) of unconstrained scenes. In contrast, MPI-INF-3DHP
[21] has only two sequences of real outdoor scenes (728 frames) and Human3.6M
[8] is fully indoor. Hence, we follow the convention [24, 25, 26] to evaluate
models mainly on 3DPW test set to gauge their in-the-wild performances.
Nevertheless, we also provide experiment results on Human3.6M and MPI-
INF-3DHP.

Metrics. The standard metrics are Mean Per Joint Position Error (MPJPE), and
Procrustes-aligned [66] Mean Per Joint Position Error (PA-MPJPE), _i.e_.,
MPJPE evaluated after rigid alignment of the predicted and the ground truth
joint keypoints, both in millimeters ($mm$). We highlight that PA-MPJPE is the
primary metric [24, 20], on which we conduct most of our discussions.

Training Details. We follow the original paper in implementing baselines [23,
24, 25, 26] on the PyTorch-based framework MMHuman3D [67]. HMR+ is a stronger
variant of the original HMR, for which we remove all adversarial modules from
the original HMR [23] for fast training and add pseudo SMPL initialization
(“static fits”) for keypoint-only datasets following SPIN [24] without further
in-the-loop optimization. For the Blended Training (BT), since GTA-Human has a
much larger scale than existing datasets, we run all our experiments on 32
V100 GPUs, with the batch size of 2048 (four times as SPIN [24]). The learning
rate is also scaled linearly by four times to 0.0002. The rest of the
hyperparameters are the same as SPIN [24]. For the Finetuning (FT)
experiments, we use the learning rate of 0.00001 with the batch size of 512,
on 8 V100 GPUs for two epochs.

Domain Adaptation Training Details. We use the same training settings as
Blended Training, except that an additional domain adaptation loss is added in
training. CycleGAN [68], we first train a CycleGAN between real data and our
synthetic GTA-Human data. Then we use a trained sim2real generator from the
CycleGAN to transform the input GTA-Human image into a real-style image during
training. For JAN [69], we use the default Gaussian kernel with a bandwidth
0.92, and set its loss weight to 0.001. For Chen _et al_. [70], we use the
default trade-off coefficient 0.1, and set its loss weight to 1e-4. For Ganin
_et al_. [71], we use a 3-layer MLP to classify the domain of given features
extracted from the backbone. The loss weight of the adversarial part is
progressively increased to 0.1 for more stable training.

###  4.2 Better 3D Human Recovery with Data Mixture

Despite that GTA-Human features reasonably realistic data, there inevitably
exists domain gaps. Surprisingly, intuitive methods of data mixture are
effective despite the domain gaps for both image- and video-based 3D human
recovery.

TABLE II: GTA-Human’s impact on model performance. The values are reported on 3DPW test set in mm. We employ two strategies: blended training (BT) that directly mixes GTA-Human data with real data to train an HMR model; finetuing (FT) that finetunes pretrained models with mixed data. Significant performance improvements are achieved with both settings. Including GTA-Human in the training boosts the HMR [23] baseline to outperform much more sophisticated methods such as SPIN [24] that leverages in-the-loop optimization (Registration) and VIBE [25] that utilizes temporal information (Video); State-of-the-art method PARE [26] also benefit from data mixture. We also conduct further experiments on video-based human recovery with VIBE in Table III. Mixture: data mixture strategies. Real: real datasets.  Method | Mixture | Registration | Video | Pretrain | Train | Finetune | MPJPE $\downarrow$ | PA-MPJPE $\downarrow$
---|---|---|---|---|---|---|---|---
HMR | - | - | - | ImageNet | Real | - | 112.3 | 67.5
HMR+ | - | - | - | ImageNet | Real | - | 98.5 | 61.7
SPIN | - | ✓ | - | ImageNet | Real | - | 96.9 | 59.2
VIBE | - | - | ✓ | ImageNet | Real | - | 93.5 | 56.5
PARE | - | - | - | ImageNet | Real | - | 82.0 | 50.9
HMR | BT | - | - | ImageNet | Mixed | - | 98.7 (-13.6) | 60.5 (-7.0)
HMR | FT | - | - | HMR | - | Mixed | 91.4 (-20.9) | 55.7 (-11.8)
HMR+ | BT | - | - | ImageNet | Mixed | - | 88.7 (-9.8) | 56.0 (-5.7)
HMR+ | FT | - | - | HMR+ | - | Mixed | 91.3 (-7.2) | 55.5 (-6.2)
SPIN | FT | - | - | SPIN | - | Mixed | 83.2 (-13.7) | 52.0 (-7.2)
PARE | FT | - | - | PARE | - | Mixed | 77.5 (-4.5) | 46.8 (-4.1)

Image-based 3D Human Recovery. We evaluate the use of synthetic data under two
data mixture settings: blended training (BT) and finetuning (FT). Results are
collated in Table II. In blended training (BT), synthetic GTA-Human data is
directly mixed with a standard basket of real datasets [24] (Human3.6M [8],
MPI-INF-3DHP [21], LSP [11], LSP-Extended [12], MPII [14] and COCO [13]).
Compared with the HMR and HMR+ baselines, blended training achieves 7.0 mm and
5.7 mm improvements in PA-MPJPE, surpassing methods such as SPIN [24] that
requires online registration or VIBE [25] that leverages temporal information.
As for finetuning (FT), we finetune a pretrained model with mixed data. Since
finetuing is much faster than blended training, this allows us to perform data
mixture on more base methods such as SPIN [24] and PARE [26]. Finetuning leads
to considerable improvements in PA-MPJPE compared to the original HMR (11.8
mm), HMR+ (6.2 mm), SPIN (7.2 mm) and PARE (4.1 mm) baselines.

Video-based 3D Human Recovery. In Table III, we validate that data mixture is
also effective for video-based methods. We conduct the study with the popular
VIBE [25] as the base model. VIBE uses a pretrained SPIN model as the feature
extractor for each frame, and we train the temporal modules with datasets
indicated in Table III. We obtain the following observations. First, when
training alone, GTA-Human outperforms MPI-INF-3DHP with an equal number of
training data. Second, the full set of GTA-Human is comparable with the in-
domain training source (3DPW train set), even slightly better in PA-MPJPE.
Third, GTA-Human is complementary to real datasets as blended training leads
to highly competitive results in all metrics.

TABLE III: Video-based 3D human recovery. The values are reported on 3DPW [6] test set with VIBE as the base model. MI3: MPI-INF-3DHP. GTA: GTA-Human. PA: PA-MPJPE. Accel: acceleration error ($mm/s^{2}$). *: downsampled GTA-Human data to match the size of MPI-INF-3DHP (96K SMPL poses). MI3 | 3DPW | GTA-Human | MPJPE $\downarrow$ | PA $\downarrow$ | Accel $\downarrow$
---|---|---|---|---|---
✓ | - | - | 95.0 | 56.5 | 27.1
- | - | ✓* | 93.7 | 55.0 | 26.3
- | ✓ | - | 87.9 | 54.7 | 23.2
- | - | ✓ | 91.3 | 54.1 | 24.7
- | ✓ | ✓ | 85.2 | 52.4 | 24.2
✓ | ✓ | ✓ | 86.0 | 51.9 | 23.3

Comparison with Other Data-driven Methods. We highlight that GTA-Human is a
large-scale, diverse dataset for 3D human recovery. In Table IV, we compare
GTA-Human with several other recent works that provide additional data for
human pose and shape estimation. We show that GTA-Human is a practical
training source that improves the performance of various base methods.
Notably, GTA-Human slightly surpasses AGORA, which is built with expensive
industry-level human scans of high-quality geometry and texture. This result
suggests that scaling with game-playing data at a lower cost achieves a
similar effect.

TABLE IV: Comparison with other data-driven methods. GTA-Human data effectively improves the base method performance. The numbers are reported on 3DPW test set, without using 3DPW in the training.  Dataset | Method | MPJPE $\downarrow$ | PA-MPJPE $\downarrow$
---|---|---|---
Arnab _et al_. [49] | HMR | - | 72.2
EFT [20] | SPIN | - | 54.2
AGORA [22] | SPIN | 85.7 | 55.3
GTA-Human (BT) | HMR | 98.7 | 60.5
GTA-Human (FT) | HMR | 91.4 | 55.5
GTA-Human (FT) | SPIN | 83.1 | 52.0

###  4.3 Closing the Domain Gap with Synthetic Data

After obtaining good results under both image- and video-based settings on
3DPW, an in-the-wild dataset and the standard test benchmark, we extend our
study to answer why is game playing data effective at all? To this end, we
also evaluate models on other (mostly) indoor benchmarks such as Human3.6M
(Protocol 2) and MPI-INF-3DHP in Table V. Interestingly, we notice that the
performance gains on these two benchmarks are not as significant as those on
3DPW. Moreover, existing methods [23, 24] commonly include in-the-wild COCO
data in the training set, in addition to popular training datasets are
typically collected indoors. We aim to explain the above-mentioned
observations and practices through both qualitative and quantitative
evaluations.

In Fig. 7, we visualize the feature distribution of various datasets. We
discover that there are indeed some domain gaps between real indoor data and
real outdoor data. Hence, models trained on real indoor data may not perform
well in the wild. We observe that in Fig. 7(a), indoor data has a significant
domain shift away from in-the-wild data. This result implies that models
trained on indoor datasets may not transfer well to in-the-wild scenes. In
Fig.7(b), blended training achieves better results as 3DPW test data are well-
covered by mixing real data or GTA-Human data. Specifically, even though the
domain gap between GTA-Human and real datasets persists, the distribution of
3DPW data is split into two main clusters, covered by GTA-Human and real
datasets separately. Hence, this observation may explain the effectiveness of
GTA-Human: albeit synthetic, a large amount of in-the-wild data provides
meaningful knowledge that is complementary to the real datasets.

Moreover, we further validate the synergy between real and synthetic data
through domain adaptation in Table VI. We select and implement several
mainstream domain adaption methods [72], and evaluate them on an HMR model
under BT with an equal amount of real data and GTA-Human ($1\times$). We
discover that learned data augmentation such as CycleGAN [68] may not be
effective, whereas domain generalization techniques (JAN [69] and Ganin _et
al_. [71]) and domain adaptive regression such as Chen _et al_. [70] further
improves the performance. In Fig.7(c), domain adaptation (Ganin _et al_. [71])
pulls the distributions of both real and GTA-Human data together, and they
jointly establish a better-learned distribution to match that of the in-the-
wild 3DPW data.

TABLE V: More benchmarks. We evaluate image-based methods trained with data mixture strategies on Human3.6M and MPI-INF-3DHP. We observe that the performance boosts are smaller than that on 3DPW. This may be attributed to the indoor-outdoor domain gaps that we discuss in Section 4.3. |  | Human3.6M | MPI-INF-3DHP
---|---|---|---
Method | Mixture | MPJPE $\downarrow$ | PA $\downarrow$ | MPJPE $\downarrow$ | PA $\downarrow$
HMR | - | 77.9 | 55.8 | 107.2 | 74.1
SPIN | - | - | 41.1 | 105.2 | 67.5
HMR | BT | 74.5 | 51.3 | 103.4 | 71.3
HMR | FT | 73.2 | 52.5 | 102.9 | 71.0
SPIN | FT | 60.9 | 40.8 | 96.4 | 67.0
TABLE VI: Domain adaptation with equal amount real and synthetic data. PA: PA-MPJPE. Method | Real | GTA-Human | PA-MPJPE $\downarrow$
---|---|---|---
HMR | ✓ | - | 76.7
HMR ($1\times$) | - | ✓ | 65.7
HMR (BT, $1\times$) | ✓ | ✓ | 58.6
CycleGAN [68] | ✓ | ✓ | 61.6
Chen _et al_. [70] | ✓ | ✓ | 57.9
JAN [69] | ✓ | ✓ | 56.5
Ganin _et al_. [71] | ✓ | ✓ | 55.5
Figure 7: Domain gap analysis. We visualize features extracted after the
trained backbones via UMAP [73] dimension reduction (the two axes are the
principal axes). (a) Training with real data only. (b) Blended training. (c)
Blended training with domain adaptation (Ganin _et al_. [71]). Figure 8:
Amount of GTA-Human Data. The horizontal axis indicate the amount of GTA-Human
data used as multiples of the amount of real data. HMR+ is used as the base
method.

###  4.4 Dataset Scale Matters

We study the data scale in two aspects. 1) Different amounts of GTA-Human data
are progressively added in the training to observe the trend in the model
performance. 2) The influence of a lack of data from the perspectives of
critical factors such as camera angle, pose, and occlusion.

Amount of GTA-Human Data. In Fig. 8, we delve deeper into the impact of data
quantity on 3DPW test set. HMR is used as the base model with BT setting. The
amount of GTA-Human data used is expressed as multiples of the total quantity
of real datasets ($\sim$300K [24]). For example, $2\times$ means the amount of
GTA-Human is twice as much as the real data in the BT. A consistent downward
trend in the errors ($\sim$6 mm decrease) with increasing GTA-Human data used
in the training is observed. Since real data is expensive to acquire,
synthetic data may play an important role in scaling up 3D human recovery in
real life.

Synthetic Data as a Scalable Supplement. We collate more experiments with
different real-synthetic data ratios in Table VII, using HMR+ as the base
method and BT as the data mixture strategy. We observe that 1) Adding more
data, synthetic and real alike, generally improve the performance. 2) Mixing
75% real data with 25% synthetic data performs well (200K to 400K data). 3)
When the data amount increases, high ratio of real data cannot be sustained
beyond 300K data due to insufficient real data. However, additional synthetic
data still improves model performance. These experiments reaffirm that
synthetic data complements real data, and more importantly, synthetic data
serves as an easily scalable training source to supplement typically limited
real data.

Impact of Data Scarcity. In Fig. 9, we systematically study the HMR+ model
trained with BT and evaluate its performance on GTA-Human, subjected to
different data density for factors such as camera angle, pose, and occlusion.
We discretize all examples evaluated to obtain and plot the data density with
bins, and compute the mean error for each bin to form the curves. A consistent
observation across factors is that the model performance deteriorates
drastically when data density declines, indicating high model sensitivity to
data scarcity. Hence, strategically collected synthetic data may effectively
supplement the real counterpart, which is often difficult to obtain.

TABLE VII: Synthetic Data as a Supplement. Different total data amount with different real data ratio are shown. Values are PA-MPJPE (mm) on 3DPW test set. Synthetic data are sampled from 4$\times$ set during training. N/A: this ratio cannot be sustained beyond 300K data due to insufficient real data. HMR+ (BT) is used as the base method. Real Ratio | 100K | 200K | 300K | 400K | 500K
---|---|---|---|---|---
0% | 70.6 | 64.5 | 65.7 | 65.0 | 64.9
25% | 62.4 | 60.9 | 58.0 | 57.6 | 57.3
50% | 61.7 | 58.9 | 57.9 | 56.3 | 55.6
75% | 62.4 | 58.4 | 56.8 | 55.7 | N/A
100% | 65.8 | 62.7 | 61.7 | N/A | N/A
Figure 9: Impact of data scarcity. We show that model performance is sensitive to data scarcity, and this observation is consistent on factors such as camera angles, poses, and occlusion. For c) and d), we follow [74] to encode pose as a set of 3D coordinates of the 24 key joints, and plot the distance from the mean pose and T-pose respectively. The data density of e) and f) are in log scale. TABLE VIII: Strong supervision is key. The first row is the HMR+ baseline without any GTA-Human data added. Keypoints | SMPL | MPJPE $\downarrow$ | PA-MPJPE $\downarrow$
---|---|---|---
- | - | 98.5 | 61.7
✓ | - | 93.4 | 60.9
- | ✓ | 92.0 | 56.3
✓ | ✓ | 88.7 | 56.0

###  4.5 Strong Supervision is Key

Due to the prohibitive cost of collecting a large amount of SMPL annotations
with a real setup, it is appealing to generate synthetic data that is
automatically labelled. In this section, we investigate the importance of
strong supervision and discuss the reasons. We compare weak supervision
signals (_i.e_., 2D and 3D keypoints) to strong counterparts (_i.e_., SMPL
parameters), and find out that the latter is critical to training a high-
performing model. We experiment under BT setting on 3DPW test set, Table VIII
shows that strong supervision of SMPL parameters $\theta$ and $\beta$, are
much more effective than weak supervision of body keypoints. Our findings are
in line with SPIN [24]. SPIN tests fitting 3D SMPL on 2D keypoints to produce
pseudo SMPL annotations during training and finds this strategy effective.
However, this conclusion still leaves the root cause of the effectiveness of
3D SMPL unanswered, as recent work suggests that 2D supervision is inherently
ambiguous [62]. In this work, we extend the prior study on the supervision
types by adding in 3D keypoints as a better part of the weak supervision and
find out that SMPL annotation is still far more effective.

As for reasons that make strong supervision (SMPL parameters) more effective
than weak supervision (keypoints), we argue that keypoints only provide
partial guidance to body shape estimation $\beta$ (bone length only), but
$\beta$ is required in joint regression from the parametric model. Moreover,
ground truth SMPL parameters is directly used in the loss computation with the
predicted SMPL parameters (Equation 1), which initiates gradient flow that
reaches the learnable SMPL parameters in the shortest possible route. On the
contrary, the 3D keypoints $\hat{X}_{3D}$ are obtained with joint regression
$\mathcal{J}$ of canonical keypoints with estimated body shape $\hat{\beta}$,
and the global rigid transformation $M$ derived from the SMPL kinematic tree
(Equation 2). The 2D keypoints $\hat{X}_{2D}$ further require extra estimation
of translation $\mathbf{\hat{t}}$ for the transformation $\mathcal{T}$ of the
3D keypoints, and 3D to 2D projection $\mathcal{K}$ with assumed focal length
$\mathbf{f}$ as well as camera center $\mathbf{c}$. The elongated route and
uncertainties introduced in the process to compute the loss for 2D keypoints
(Equation 3) hinder the effective learning.

$\displaystyle\mathcal{L}_{SMPL}$
$\displaystyle=||\theta-\hat{\theta}||+||\beta-\hat{\beta}||$ (1)
$\displaystyle\mathcal{L}_{3D}$ $\displaystyle=||\hat{X}_{3D}-X_{3D}||$ (2)
$\displaystyle\mathcal{L}_{2D}$ $\displaystyle=||\hat{X}_{2D}-X_{2D}||$ (3)

where

$\displaystyle\hat{X}_{3D}$
$\displaystyle=\mathcal{M}(J(\hat{\beta}),\hat{\theta})$ (4)
$\displaystyle\hat{X}_{2D}$
$\displaystyle=\mathcal{K}(\mathcal{T}(\hat{X}_{3D},\mathbf{\hat{t}}),\mathbf{f},\mathbf{c})$
(5)

###  4.6 Big Data Benefits Big Models

ResNet-50 remains a common backbone choice, since HMR is firstly introduced
for deep learning-based 3D human recovery. In this section, we extend our
study of the impact of big data on more backbone options, including deeper
CNNs such as ResNet-101 and 152 [27], as well as DeiT [30], as a
representative of Vision Transformers. In Table IX, we evaluate various
backbones for the HMR baseline. We highlight that including GTA-Human always
improves model performance by a considerable margin, regardless of the model
size or architecture. Note that using Transformers as the feature extractor
for human pose and shape estimation is under-explored in recent literature;
there may be some room for further improvement upon our attempts presented
here. Nevertheless, the same trend holds for the two transformer variants.
Interestingly, additional GTA-Human unleashes the full power of a small model
(_e.g_., ResNet-50), enabling it to outperform a larger model (_e.g_.,
ResNet-152) trained with real data only. This suggests data still remains a
critical bottleneck for accurate human pose and shape estimation.

TABLE IX: Big data benefits big models. Real: training with only the real datasets. +GTA: blended training setting is used with GTA-Human. Values in green indicate the error reduction in PA-MPJPE (mm) with blended training. Backbone | #Param | Real $\downarrow$ | +GTA-Human $\downarrow$
---|---|---|---
ResNet-50 | 26M | 61.7 | 56.0 (-5.7)
ResNet-101 | 45M | 60.1 | 54.5 (-5.6)
ResNet-152 | 60M | 58.4 | 54.3 (-4.1)
DeiT-Small | 22M | 66.5 | 60.7 (-5.8)
DeiT-Base | 86M | 61.2 | 56.2 (-5.0)

##  5 Conclusion

In this work, we evaluate the effectiveness of synthetic game-playing data in
enhancing human pose and shape estimation especially in the wild. To this end,
we present GTA-Human, a large-scale, diverse dataset for 3D human recovery.
Our experiments on GTA-Human provide five takeaways: 1) Training with diverse
synthetic data (especially with outdoor scenes) achieves a significant
performance boost. 2) The effectiveness is attributed to the complementary
relation between real and synthetic data. 3) The more data, the better because
model performance is highly sensitive to data density. 4) Strong supervision
such as SMPL parameters are essential to training a high-performance model. 5)
Deeper and more powerful backbones also benefit from a large amount of data.
As for future works, we plan to investigate beyond 1.4M data samples with more
computation budgets to explore the boundary of training with synthetic data.
Moreover, it would be interesting to study the sim2real problem for 3D
parametric human recovery more in-depth with GTA-Human, or even extend the
game-playing data to other human-related topics such as model-free
reconstruction that are out of the scope of this work.

## Acknowledgments

This work is supported by NTU NAP, MOE AcRF Tier 2 (T2EP20221-0033), and under
the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-
ICP) Funding Initiative, as well as cash and in-kind contribution from the
industry partner(s).

As we are aware that GTA is not a perfect depiction of real life, we address
some ethical concerns and explain our strategies to alleviate potential
negative impact.

### Privacy

The subjects present in GTA-Human are virtual humans extracted from the in-
game database, that usually do not have clear real-life references.
Protagonists may have some sort of real-life references, but the appearances
are altered to suit the corresponding characters in the context of the game
storyline.

### Violence and sexualized actions

We manually screen around 1k actions from the 20k classes and find that the
vast majority of the actions are used to depict the ordinary lives of the
city-dwellers (e.g. walking, drinking coffee, doing push-ups, and so on). This
is further supported by 1) the distribution of GTA-Human is center-aligned
with real datasets. 2) methods trained on GTA-Human can perform convincingly
better on standard real datasets. Both indicate that the domain shift in
actions may not be noticeably affected by the small portion of offensive
actions. Moreover, no weapon is depicted in the dataset.

### Stereotypes and Biases

The original storylines of GTA-V may insert strong stereotypes in the
depiction of characters depending on their attributes (_e.g_., gender and
race). We thus adopt the following strategies to minimize biases.

First, all factors including the characters and actions, are decoupled and
randomized in GTA-Human. Specifically, the examples in GTA-Human are not
linked to the original storylines; all the characters and the actions are
pulled out of the in-game database, randomly assigned at random locations all
over the map. Hence, it is very unlikely any character-specific actions could
be reproduced.

Second, we have conducted a manual analysis on clothing (which may serve as an
indication of social status) vs gender and race. As we find it difficult to
determine if a specific attire has certain social implications without context
(for example, skin-showing attire not be associated with sex workers as it is
also common to find people in bikinis at the beach), we thus categorize all
clothing into formal, semi-formal and casual. We observe that while there is
approximately the same number of men and women in formal attire (11% vs 9%);
more men in casual attires (_e.g_. tank tops, topless) than women (25% vs
11%); all races have approximately the same distribution of formal, semi-
formal and casual ($\sim 2\mathrel{\mathop{\mathchar
58\relax}}5\mathrel{\mathop{\mathchar 58\relax}}3$). Hence, we find the
character appearances mostly (albeit not perfectly) balanced across genders
and races.

Third, as much as we hope to perform a complete and thorough data screening
and cleaning, we highlight it is not very practical to manually inspect all
examples due to the sheer scale of the dataset. Hence, we anonymize the
characters, actions, and locations such that they exist in the dataset to
enrich the distribution, but cannot be retrieved for malicious uses.

### Copyright

The publisher of GTA-V allows for the use of game-generated materials provided
that it is non-commercial, and no spoilers are distributed***Policy on mods.
http://tinyurl.com/yc8kq7vn.†††Policy on copyrighted material.
http://tinyurl.com/pjfoqo5.. Hence, we follow prior works that generate data
on the GTA game engine [60, 15, 61, 16, 17, 18] to make GTA-Human publically
available.

## References

  * [1] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black, “SMPL: A skinned multi-person linear model,” _ACM transactions on graphics (TOG)_ , vol. 34, no. 6, pp. 1–16, 2015.
  * [2] J. Krause, B. Sapp, A. Howard, H. Zhou, A. Toshev, T. Duerig, J. Philbin, and L. Fei-Fei, “The unreasonable effectiveness of noisy data for fine-grained recognition,” in _ECCV_. Springer, 2016, pp. 301–320.
  * [3] C. Sun, A. Shrivastava, S. Singh, and A. Gupta, “Revisiting unreasonable effectiveness of data in deep learning era,” in _ICCV_ , 2017, pp. 843–852.
  * [4] Y. Rong, Z. Liu, C. Li, K. Cao, and C. C. Loy, “Delving deep into hybrid annotations for 3d human recovery in the wild,” in _ICCV_ , 2019, pp. 5340–5348.
  * [5] L. Sigal, A. O. Balan, and M. J. Black, “Humaneva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion,” _IJCV_ , vol. 87, pp. 4–27, 2009.
  * [6] T. von Marcard, R. Henschel, M. J. Black, B. Rosenhahn, and G. Pons-Moll, “Recovering accurate 3d human pose in the wild using imus and a moving camera,” in _ECCV_ , 2018, pp. 601–617.
  * [7] V. Leroy, P. Weinzaepfel, R. Br’egier, H. Combaluzier, and G. Rogez, “Smply benchmarking 3d human pose estimation in the wild,” _3DV_ , pp. 301–310, 2020.
  * [8] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu, “Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 36, no. 7, pp. 1325–1339, 2013.
  * [9] H. Joo, T. Simon, X. Li, H. Liu, L. Tan, L. Gui, S. Banerjee, T. Godisart, B. C. Nabbe, I. Matthews, T. Kanade, S. Nobuhara, and Y. Sheikh, “Panoptic studio: A massively multiview system for social interaction capture,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 41, pp. 190–204, 2019.
  * [10] Z. Yu, J. S. Yoon, I. Lee, P. Venkatesh, J. Park, J. Yu, and H. Park, “Humbi: A large multiview dataset of human body expressions,” _CVPR_ , pp. 2987–2997, 2020.
  * [11] S. Johnson and M. Everingham, “Clustered pose and nonlinear appearance models for human pose estimation,” in _BMVC_. British Machine Vision Association, 2010, pp. 1–11.
  * [12] ——, “Learning effective human pose estimation from inaccurate annotation,” _CVPR_ , pp. 1465–1472, 2011.
  * [13] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _ECCV_. Springer, 2014, pp. 740–755.
  * [14] M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele, “2d human pose estimation: New benchmark and state of the art analysis,” in _CVPR_ , 2014, pp. 3686–3693.
  * [15] S. R. Richter, Z. Hayder, and V. Koltun, “Playing for benchmarks,” in _ICCV_ , 2017, pp. 2213–2222.
  * [16] M. Fabbri, F. Lanzi, S. Calderara, A. Palazzi, R. Vezzani, and R. Cucchiara, “Learning to detect and track visible and occluded body joints in a virtual world,” in _ECCV_ , 2018, pp. 430–446.
  * [17] Z. Cao, H. Gao, K. Mangalam, Q.-Z. Cai, M. Vo, and J. Malik, “Long-term human motion prediction with scene context,” in _ECCV_. Springer, 2020, pp. 387–404.
  * [18] Y.-T. Hu, J. Wang, R. A. Yeh, and A. G. Schwing, “Sail-vos 3d: A synthetic dataset and baselines for object detection and 3d mesh reconstruction from video data,” in _CVPR_ , 2021, pp. 1418–1428.
  * [19] M. Fabbri, G. Brasó, G. Maugeri, O. Cetintas, R. Gasparini, A. Ošep, S. Calderara, L. Leal-Taixé, and R. Cucchiara, “Motsynth: How can synthetic data help pedestrian detection and tracking?” in _ICCV_ , 2021, pp. 10 849–10 859.
  * [20] H. Joo, N. Neverova, and A. Vedaldi, “Exemplar fine-tuning for 3d human pose fitting towards in-the-wild 3d human pose estimation,” _ArXiv_ , vol. abs/2004.03686, 2020.
  * [21] D. Mehta, H. Rhodin, D. Casas, P. Fua, O. Sotnychenko, W. Xu, and C. Theobalt, “Monocular 3d human pose estimation in the wild using improved cnn supervision,” in _3DV_. IEEE, 2017, pp. 506–516.
  * [22] P. Patel, C.-H. P. Huang, J. Tesch, D. T. Hoffmann, S. Tripathi, and M. J. Black, “AGORA: Avatars in geography optimized for regression analysis,” in _CVPR_ , Jun. 2021.
  * [23] A. Kanazawa, M. J. Black, D. W. Jacobs, and J. Malik, “End-to-end recovery of human shape and pose,” in _CVPR_ , 2018, pp. 7122–7131.
  * [24] N. Kolotouros, G. Pavlakos, M. J. Black, and K. Daniilidis, “Learning to reconstruct 3d human pose and shape via model-fitting in the loop,” in _ICCV_ , 2019, pp. 2252–2261.
  * [25] M. Kocabas, N. Athanasiou, and M. J. Black, “Vibe: Video inference for human body pose and shape estimation,” in _CVPR_ , 2020, pp. 5253–5263.
  * [26] M. Kocabas, C.-H. P. Huang, O. Hilliges, and M. J. Black, “Pare: Part attention regressor for 3d human body estimation,” _arXiv preprint arXiv:2104.08527_ , 2021.
  * [27] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016, pp. 770–778.
  * [28] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in _CVPR_ , 2017, pp. 1492–1500.
  * [29] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in _ICLR_. OpenReview.net, 2021.
  * [30] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” in _ICML_ , ser. Proceedings of Machine Learning Research, vol. 139. PMLR, 2021, pp. 10 347–10 357.
  * [31] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. Osman, D. Tzionas, and M. J. Black, “Expressive body capture: 3d hands, face, and body from a single image,” in _CVPR_ , 2019, pp. 10 975–10 985.
  * [32] A. A. A. Osman, T. Bolkart, and M. J. Black, “STAR: sparse trained articulated human body regressor,” in _ECCV (6)_ , ser. Lecture Notes in Computer Science, vol. 12351. Springer, 2020, pp. 598–613.
  * [33] F. Bogo, A. Kanazawa, C. Lassner, P. Gehler, J. Romero, and M. J. Black, “Keep it smpl: Automatic estimation of 3d human pose and shape from a single image,” in _ECCV_. Springer, 2016, pp. 561–578.
  * [34] Y. Huang, F. Bogo, C. Lassner, A. Kanazawa, P. V. Gehler, J. Romero, I. Akhter, and M. J. Black, “Towards accurate marker-less human shape and pose estimation over time,” in _3DV_. IEEE, 2017, pp. 421–430.
  * [35] T. Fan, K. V. Alwala, D. Xiang, W. Xu, T. Murphey, and M. Mukadam, “Revitalizing optimization for 3d human pose and shape estimation: A sparse constrained formulation,” _ICCV_ , 2021.
  * [36] G. Pavlakos, L. Zhu, X. Zhou, and K. Daniilidis, “Learning to estimate 3d human pose and shape from a single color image,” in _CVPR_ , 2018, pp. 459–468.
  * [37] M. Omran, C. Lassner, G. Pons-Moll, P. Gehler, and B. Schiele, “Neural body fitting: Unifying deep learning and model based human pose and shape estimation,” in _3DV_. IEEE, 2018, pp. 484–494.
  * [38] R. A. Guler and I. Kokkinos, “Holopose: Holistic 3d human reconstruction in-the-wild,” in _CVPR_ , 2019, pp. 10 884–10 894.
  * [39] G. Georgakis, R. Li, S. Karanam, T. Chen, J. Košecká, and Z. Wu, “Hierarchical kinematic human mesh recovery,” in _ECCV_. Springer, 2020, pp. 768–784.
  * [40] J. Li, C. Xu, Z. Chen, S. Bian, L. Yang, and C. Lu, “Hybrik: A hybrid analytical-neural inverse kinematics solution for 3d human pose and shape estimation,” in _CVPR_. Computer Vision Foundation / IEEE, 2021, pp. 3383–3393.
  * [41] Y. Sun, Q. Bao, W. Liu, Y. Fu, B. Michael J., and T. Mei, “Monocular, one-stage, regression of multiple 3d people,” in _ICCV_ , October 2021.
  * [42] A. Kanazawa, J. Y. Zhang, P. Felsen, and J. Malik, “Learning 3d human dynamics from video,” in _CVPR_ , 2019, pp. 5614–5623.
  * [43] Y. Sun, Y. Ye, W. Liu, W. Gao, Y. Fu, and T. Mei, “Human mesh recovery from monocular images via a skeleton-disentangled representation,” in _ICCV_ , 2019, pp. 5349–5358.
  * [44] D. Mehta, O. Sotnychenko, F. Mueller, W. Xu, M. Elgharib, P. Fua, H.-P. Seidel, H. Rhodin, G. Pons-Moll, and C. Theobalt, “Xnect: Real-time multi-person 3d motion capture with a single rgb camera,” _ACM Transactions on Graphics (TOG)_ , vol. 39, no. 4, pp. 82–1, 2020.
  * [45] G. Moon and K. M. Lee, “I2l-meshnet: Image-to-lixel prediction network for accurate 3d human pose and mesh estimation from a single RGB image,” in _ECCV (7)_ , ser. Lecture Notes in Computer Science, vol. 12352\. Springer, 2020, pp. 752–768.
  * [46] H. Choi, G. Moon, and K. M. Lee, “Beyond static features for temporally consistent 3d human pose and shape from a video,” in _CVPR_ , 2021.
  * [47] Z. Luo, S. A. Golestaneh, and K. M. Kitani, “3d human motion estimation via motion compression and refinement,” in _Proceedings of the Asian Conference on Computer Vision_ , 2020.
  * [48] K. Lin, L. Wang, and Z. Liu, “End-to-end human pose and mesh reconstruction with transformers,” in _CVPR_. Computer Vision Foundation / IEEE, 2021, pp. 1954–1963.
  * [49] A. Arnab, C. Doersch, and A. Zisserman, “Exploiting temporal context for 3d human pose estimation in the wild,” in _CVPR_ , 2019, pp. 3395–3404.
  * [50] J. Song, X. Chen, and O. Hilliges, “Human body model fitting by learned gradient descent,” in _ECCV (20)_ , ser. Lecture Notes in Computer Science, vol. 12365. Springer, 2020, pp. 744–760.
  * [51] A. Sengupta, R. Cipolla, and I. Budvytis, “Synthetic training for accurate 3d human pose and shape estimation in the wild,” in _BMVC_. BMVA Press, 2020.
  * [52] T. Zhang, B. Huang, and Y. Wang, “Object-occluded human shape and pose estimation from a single color image,” _CVPR_ , pp. 7374–7383, 2020.
  * [53] C. Lassner, J. Romero, M. Kiefel, F. Bogo, M. J. Black, and P. V. Gehler, “Unite the people: Closing the loop between 3d and 2d human representations,” in _CVPR_ , 2017, pp. 6050–6059.
  * [54] M. Trumble, A. Gilbert, C. Malleson, A. Hilton, and J. Collomosse, “Total capture: 3d human pose estimation fusing video and inertial sensors,” in _BMVC_ , 2017.
  * [55] D. Mehta, O. Sotnychenko, F. Mueller, W. Xu, S. Sridhar, G. Pons-Moll, and C. Theobalt, “Single-shot multi-person 3d pose estimation from monocular rgb,” _3DV_ , pp. 120–130, 2018.
  * [56] Z. Cai, D. Ren, A. Zeng, Z. Lin, T. Yu, W. Wang, X. Fan, Y. Gao, Y. Yu, L. Pan, F. Hong, M. Zhang, C. C. Loy, L. Yang, and Z. Liu, “Humman: Multi-modal 4d human dataset for versatile sensing and modeling,” October 2022.
  * [57] G. Varol, J. Romero, X. Martin, N. Mahmood, M. J. Black, I. Laptev, and C. Schmid, “Learning from synthetic humans,” _CVPR_ , pp. 4627–4635, 2017\.
  * [58] D. T. Hoffmann, D. Tzionas, M. Black, and S. Tang, “Learning to train with synthetic humans,” in _GCPR_ , 2019.
  * [59] A. Pumarola, J. Sanchez, G. Choi, A. Sanfeliu, and F. Moreno-Noguer, “3dpeople: Modeling the geometry of dressed humans,” _ICCV_ , pp. 2242–2251, 2019.
  * [60] S. R. Richter, V. Vineet, S. Roth, and V. Koltun, “Playing for data: Ground truth from computer games,” in _ECCV_. Springer, 2016, pp. 102–118.
  * [61] P. Krähenbühl, “Free supervision from video games,” in _CVPR_ , 2018, pp. 2955–2964.
  * [62] I. Kissos, L. Fritz, M. Goldman, O. Meir, E. Oks, and M. Kliger, “Beyond weak perspective for monocular 3d human pose estimation,” in _ECCV_. Springer, 2020, pp. 541–554.
  * [63] S. Madan, T. Henry, J. Dozier, H. Ho, N. Bhandari, T. Sasaki, F. Durand, H. Pfister, and X. Boix, “When and how do cnns generalize to out-of-distribution category-viewpoint combinations?” _arXiv preprint arXiv:2007.08032_ , 2020.
  * [64] Z. Cai, J. Zhang, D. Ren, C. Yu, H. Zhao, S. Yi, C. K. Yeo, and C. C. Loy, “Messytable: Instance association in multiple camera views,” in _ECCV_. Springer, 2020, pp. 1–16.
  * [65] M. Loper, N. Mahmood, and M. J. Black, “Mosh: Motion and shape capture from sparse markers,” _ACM Transactions on Graphics (TOG)_ , vol. 33, no. 6, pp. 1–13, 2014.
  * [66] J. C. Gower, “Generalized procrustes analysis,” _Psychometrika_ , vol. 40, no. 1, pp. 33–51, 1975.
  * [67] M. Contributors, “Openmmlab 3d human parametric model toolbox and benchmark,” https://github.com/open-mmlab/mmhuman3d, 2021.
  * [68] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _ICCV_ , 2017, pp. 2223–2232.
  * [69] M. Long, H. Zhu, J. Wang, and M. I. Jordan, “Deep transfer learning with joint adaptation networks,” in _ICML_. PMLR, 2017, pp. 2208–2217.
  * [70] X. Chen, S. Wang, J. Wang, and M. Long, “Representation subspace distance for domain adaptation regression,” in _ICML_. PMLR, 2021, pp. 1749–1759.
  * [71] Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” in _ICML_. PMLR, 2015, pp. 1180–1189.
  * [72] O. Wiles, S. Gowal, F. Stimberg, S. Alvise-Rebuffi, I. Ktena, T. Cemgil _et al._ , “A fine-grained analysis on distribution shift,” _arXiv preprint arXiv:2110.11328_ , 2021.
  * [73] L. McInnes, J. Healy, and J. Melville, “Umap: Uniform manifold approximation and projection for dimension reduction,” _arXiv preprint arXiv:1802.03426_ , 2018.
  * [74] Y. Rong, Z. Liu, and C. C. Loy, “Chasing the tail in monocular 3d human reconstruction with prototype memory,” _arXiv preprint arXiv:2012.14739_ , 2020.

|  Zhongang Cai is currently a Year 2 Ph.D. Student at Nanyang Technological
University (NTU) supervised by Prof Liu Ziwei and Prof Chen Change Loy, and a
Senior Algorithm Researcher at SenseTime International Pte. Ltd. He attained
his bachelor’s degree at Nanyang Technological University and was awarded the
Lee Kuan Yew Gold Medal as a top student. His research interest includes point
clouds and virtual humans. To date, he has published 10+ papers on top venues
such as ICLR, CVPR, ICCV and ECCV. He also serves as a reviewer for top
machine learning and computer vision conferences, such as NeurIPS, ICLR and
ICCV.
---|---
|  Mingyuan Zhang received a B.S. degree in computer science and engineering
from Beihang University, China. He is currently pursuing a Ph.D. degree at
MMLab@NTU, advised by Prof. Ziwei Liu. His research interests in computer
vision include motion synthesis, 3D pose estimation, and scene understanding.
He has published papers on top-tier conferences, including CVPR, ECCV, ICCV,
ICLR, and AAAI.
---|---
|  Jiawei Ren Jiawei Ren is a PhD student in Computer Science at MMLab@NTU,
advised by Prof. Ziwei Liu. His research focuses on robust learning for 3D
perception and generation. Previously, he was a full-time algorithm researcher
at X-Lab@SenseTime Research where he was advised by Prof. Hongsheng Li. He
obtained B.Eng. in Electrical and Electronic Engineering from Nanyang
Technological University where he was advised by Prof. Lap-Pui Chau. He has
published several papers on top-tier conferences including CVPR, ECCV,
NeurIPS, ICML and AAAI. He is also the recipient of AISG fellowship.
---|---
|  Chen Wei graduated from Nanyang Technological University. He was an
algorithm research intern at S-Lab @ NTU in 2021 and continued part-time
internship at SenseTime International Pte. Ltd in 2022. His work focuses on
the generation of synthetic human data from game engines.
---|---
|  Daxuan Ren is a PhD student in Computer Science at NTU advised by Prof.
Jianmin Zheng (NTU) and Prof. Jianfei Cai (Monash University). Prior to his
Ph.D, he was a software engineer in Autodesk working on large-scale 3D
Reconstruction Engine (Autodesk ReCap Photo) including SFM, MVS and Surface
Reconstruction. His research interest includes 3D Computer Vision and
Graphics. He has multiple papers published on top-tier conferences including
ICCV, ECCV.
---|---
|  Zhengyu Lin is an Algorithm Enginner at SenseTime International Pte. Ltd.
He attained his bachelor’s degree at Imperial College London. His research
interest includes 3D computer vision and virtual data. He is currently working
with human-based RGBD data.
---|---
|  Haiyu Zhao is currently a senior researcher at SenseTime International
Pte.Ltd. Previously, he was a research assistant at the Chinese University of
Hong Kong Supervised by Prof. Xiaogang Wang and Prof. Hongsheng Li. Before
that, Haiyu received his bachelor degree at Beihang University. He has
published over 10 papers (with more than 1,200 citations) on top-tier
conferences in relevant fields, including CVPR, ICCV, ECCV, NeurIPS and ICLR.
---|---
|  Lei Yang is currently a Research Director at SenseTime Group Inc., under
the supervision of Prof. Xiaogang Wang. Prior to that, Lei received his Ph.D.
degree from the Chinese University of Hong Kong in 2020, advised by Prof.
Dahua Lin and Prof. Xiaoou Tang. Before that, Lei obtained his B.E. degree
from Tsinghua University in 2015. He has published over 10 papers on top
conferences in relevant fields, including CVPR, ICCV, ECCV, AAAI, SIGGRAPH and
RSS.
---|---
|  Chen Change Loy (Senior Member, IEEE) received the PhD degree in computer
science from the Queen Mary University of London, in 2010. He is an associate
professor with the School of Computer Science and Engineering, Nanyang
Technological University. Prior to joining NTU, he served as a research
assistant professor with the Department of Information Engineering, The
Chinese University of Hong Kong, from 2013 to 2018. His research interests
include computer vision and deep learning with a focus on image/video
restoration and enhancement, generative tasks, and representation learning. He
serves as an Associate Editor of the International Journal of Computer Vision
(IJCV) and IEEE Transactions on Pattern Analysis and Machine Intelligence
(TPAMI). He also serves/served as an Area Chair of ICCV 2021, CVPR (2021,
2019), ECCV (2022, 2018), AAAI (2021-2023), and BMVC (2018-2021).
---|---
|  Ziwei Liu is currently an Assistant Professor at Nanyang Technological
University, Singapore. Previously, he was a senior research fellow at the
Chinese University of Hong Kong and a postdoctoral researcher at University of
California, Berkeley. Ziwei received his PhD from the Chinese University of
Hong Kong. His research revolves around computer vision, machine learning and
computer graphics. He has published extensively on top-tier conferences and
journals in relevant fields, including CVPR, ICCV, ECCV, NeurIPS, ICLR, ICML,
TPAMI, TOG and Nature - Machine Intelligence. He is the recipient of Microsoft
Young Fellowship, Hong Kong PhD Fellowship, ICCV Young Researcher Award and
HKSTP Best Paper Award. He also serves as an Area Chair of ICCV, NeurIPS and
ICLR.
---|---