# FrankMocap: A Monocular 3D Whole-Body Pose Estimation System
via Regression and Integration

Yu Rong1,3 Takaaki Shiratori2 Hanbyul Joo3
1The Chinese University of Hong Kong 2Facebook Reality Labs 3Facebook AI
Research
Work done during Yu Rong’s internship at Facebook AI Research

###### Abstract

Most existing monocular 3D pose estimation approaches only focus on a single
body part, neglecting the fact that the essential nuance of human motion is
conveyed through a concert of subtle movements of face, hands, and body. In
this paper, we present FrankMocap, a fast and accurate whole-body 3D pose
estimation system that can produce 3D face, hands, and body simultaneously
from in-the-wild monocular images. The idea of FrankMocap is its modular
design: We first run 3D pose regression methods for face, hands, and body
independently, followed by composing the regression outputs via an integration
module. The separate regression modules allow us to take full advantage of
their state-of-the-art performances without compromising the original accuracy
and reliability in practice. We develop three different integration modules
that trade off between latency and accuracy. All of them are capable of
providing simple yet effective solutions to unify the separate outputs into
seamless whole-body pose estimation results. We quantitatively and
qualitatively demonstrate that our modularized system outperforms both the
optimization-based and end-to-end methods of estimating whole-body pose. Code
and models and demo videos are available at
https://github.com/facebookresearch/frankmocap.

##  1 Introduction

Estimating 3D human pose from single RGB images is one of the core
technologies to build a computational model to understand human behavioral
cues. It can faciliate numerous applications including assistive technology
[23, 41, 53], sign language understanding [10], AR/VR [3, 68], and social
signal understanding [51]. Importantly, the essential nuance of human
behaviors is conveyed through a concert of subtlest movements of face, hands,
and body. Thus, it is necessary to estimate whole-body motions to capture the
authentic signals.

Estimating whole-body 3D poses including a face and hands, however, remains
challenging. One of the major difficulties comes from the fact that the scale
of faces and hands is much smaller than that of torsos and limbs. Furthermore,
hands in motion are susceptible to artifacts caused by abrupt viewpoint
changes, self-occlusions, and motion blur. These factors largely raise the
difficulties of creating large-scale datasets with whole-body 3D poses, even
in a controlled environment [32, 40, 56], not to mention capture such data in
the wild. The lack of whole-body 3D data is a major obstacle preventing the
construction of a unified system to estimate whole-body 3D poses
simultaneously. Therefore, most existing methods focus on one of the major
body parts individually: single image-based estimation of 3D body pose
(_i.e_., torso and limbs) [5, 33, 70, 37, 38, 69, 60], 3D hand pose [78, 9,
28, 8, 20, 75] or 3D face [65, 14, 19, 61, 12].

In this paper, we present FrankMocap, a modular system for estimating whole-
body 3D poses in a unified output in SMPL-X form [56], as visualized in Fig.
LABEL:fig:teaser. Our system is built upon the insight that training a single
model to jointly estimate whole parts is intrinsically limited by the lack of
accurate and diverse whole-body motion data. Instead, we design a modularized
system to first run 3D pose regression methods for face, hands, and body
independently. The three regression outputs are then composed via an
integration module. The separate regression modules allow us to take full
advantage of their state-of-the-art performances without compromising the
original accuracy and reliability in practice. Our integration module provides
a simple yet effective solution to unify them into seamless whole-body pose
estimation outputs. Unlike previous whole-body pose estimation methods that
require computationally heavy optimization [32, 70, 56], our method also
allows us to run at an interactive frame rate. We quantitatively and
qualitatively demonstrate that our modular framework outperforms existing
optimization-based methods [70, 56] and end-to-end models trained to produce
whole-body outputs jointly [18].

In conclusion, we make three major contributions. First, towards a practical
system for whole-body pose estimation, we present the idea of regression-and-
integration to take full advantage of existing single-part pose datasets and
circumvent the hurdle of lacking whole-body pose datasets. Especially, our
hand module is comparable with SOTA hand-only methods on in-the-wild
scenarios. Second, we present three effective strategies to integrate outputs
from separate single-part pose estimation modules, namely the fastest “copy-
paste” approach, a full-optimization approach, and an intermediate approach
via a simple integration network. Third, we quantitatively and qualitatively
demonstrate that our modular system outperforms the alternative approaches
based on optimization-based methods or end-to-end methods training all parts
jointly.

Figure 1: Overview of FrankMocap, the proposed whole-body 3D pose estimation
system. Given a single RGB image input, we first apply our part modules
separately to estimate 3D hands, body, and face. Our integration module is
then adopted to combine the outputs from each module into a unified whole-body
output.

##  2 Related Work

3D Parametric Human Body Models. 3D parametric human models are widely used as
a strong prior to estimate 3D pose and shape of humans. A main idea is to
model the deformation of 3D human (including face, hands and body) via low
dimension parameters [1, 57, 56, 58, 32]. SCAPE [1] is a pioneering work that
accounts for shape variations and pose deformations of human body. Loper  _et
al_. introduce SMPL [46] that learns local pose-dependent blendshape on top of
linear blend skinning for holistic mesh deformation as well as shape
variations. Similarly, there exists a hand deformation model called MANO [58]
and several face models [11, 4, 43]. Recent approaches also introduce whole-
body models such as Adam [32], SMPL-X [56], and GHUM & GHUML [71] that are
capable of expressing face, body, and hands in a unified space.

Single Image 3D Body, Hand and Face Estimation. Recent monocular 3D body
motion capture approaches adopt parametric model such as the SMPL [46] or Adam
model [32, 70] for 3D body representation111Note that body indicates torso and
limbs excluding finger joints and whole-body indicates all of face, hands and
body.. A pioneering work [5] uses an optimization framework to fit the 3D body
model to 2D observations. More recent methods [33, 34, 37, 66, 52, 59, 24, 72,
49, 30, 21, 44, 42] leverage a deep learning framework to regress parameters
of the body models from RGB images. Non-parametric methods have been also
introduced by directly regressing the model vertices [38, 17] or corresponding
UV maps [73]. There are also approaches that use a hybrid framework by using a
deep learning framework to produce intermediate representations such as 2D
heatmaps and then fitting the skeletal models on these outputs to reconstruct
joint angles [48, 70, 47]. Due to the lack of in-the-wild training data with
3D annotations, most of these models are trained with mixed datasets,
including indoor datasets such as Human3.6M [27] and in-the-wild datasets such
as COCO [45]. While most methods are based on single images as input, recent
approaches [34, 36] predict 3D motions from a video.

Monocular 3D hand pose estimation approaches share a similar pipeline as the
body counterparts, based on the parametric 3D hand models, such as MANO [58].
Deep neural networks are leveraged to either predict model parameters [8, 75,
2, 77, 16, 15] or directly regress hand mesh vertices [20, 39]. Similarly,
single image 3D face prediction methods [65, 14, 19, 61] also share similar
ideas by leveraging deep neural networks to regress the face landmarks or
parameters of face models, such as 3DMM [4] or FLAME [43]. We refer the
readers to recent surveys on 3D face reconstruction [80, 50] for more details.

Joint 3D Estimation of Body, Hands and Face. There are a few methods [70, 56,
18, 76] aiming to capture whole-body 3D motions. SMPLify-X optimizes the
parameter of the SMPL-X model [56] to fit it to 2D keypoints with additional
constraints, including body pose priors and collision penalizer. Monocular
Total Capture (MTC) [70] is based on the Adam model [32], and adopts deep
neural networks to get 2.5D predictions first. Then the parameters of Adam are
obtained through optimization. Both of these methods rely on optimization
procedures with relatively slow computational time (from 10 seconds to a few
minutes). Zhou  _et al_. [76] use SMPL-H [58] to represent body and hands and
3DMM face model [64] to represent 3D faces. They first predict 3D body and
hand poses and then separately predict the parameters of 3DMM model, including
shape, expression, albedo, and illumination. The predicted full-body motion is
not represented in a unified format and this paper does not present
generalization ability to in-the-wild scenarios. ExPose [18], avoids using the
optimization procedure by presenting a neural network to simultaneously
predict the body, hand, and face parameters of SMPL-X. In particular, ExPose
curates a pseudo-ground truth dataset by fitting SMPL-X model on in-the-wild
images, followed by manual quality checking by annotators, and use it to train
their model to jointly produces output.

##  3 Method

Given a single image as input, FrankMocap firstly estimates the 3D poses of
face, hands (both left and right), and body (the torso and limb parts), in the
form of SMPL-X model [56]. Each part’s output is produced by a separate 3D
pose regressor trained on public datasets in each sub-field. Their outputs are
combined through our integration module, to produce a seamless and unified
whole-body pose estimation output. Fig. 1 illustrates an overview of the
proposed framework.

###  3.1 Whole-Body Parameterization

SMPL-X Model. We formulate the SMPL-X model as:

$\boldsymbol{V}_{w}=W(\boldsymbol{\phi}_{w},\boldsymbol{\theta}_{w},\boldsymbol{\beta}_{w},\boldsymbol{\psi}_{f})\\\
$ (1)

where $W$ is parameterized by global orientation of the whole-body
$\boldsymbol{\phi}_{w}\in\mathbb{R}^{3}$, facial expressions parameters
$\boldsymbol{\psi}_{f}\in\mathbb{R}^{10}$, whole-body pose parameters
$\boldsymbol{\theta}_{w}\in\mathbb{R}^{(21+15+15+1)\times 3}$ accounting for
pose-dependent deformation, and shape parameters
$\boldsymbol{\beta}_{w}\in\mathbb{R}^{10}$ accounting for cross-identity shape
variations of the face, hands, and body. We divide $\boldsymbol{\theta}_{w}$
for each of the body, hands, and faces, namely body pose parameters
$\boldsymbol{\theta}_{w}^{b}\in\mathbb{R}^{21\times 3}$, left hand pose
parameters $\boldsymbol{\theta}_{w}^{lh}\in\mathbb{R}^{15\times 3}$, right
hand pose parameters $\boldsymbol{\theta}_{w}^{rh}\in\mathbb{R}^{15\times 3}$,
and face pose parameters222While the original SMPL-X model has jaw, left and
right eye poses for the face part, we do not estimate eye poses and only
consider jaw pose. $\boldsymbol{\theta}_{w}^{f}\in\mathbb{R}^{1\times 3}$. In
this way,
$\boldsymbol{\theta}_{w}=\\{\boldsymbol{\theta}_{w}^{b},\boldsymbol{\theta}_{w}^{lh},\boldsymbol{\theta}_{w}^{rh},\boldsymbol{\theta}_{w}^{f}\\}$.
All pose parameters are defined in the axis-angle representation, which stores
the relative rotation to the parent joints defined in the kinematics map. As
output, the SMPL-X model produces a mesh structure with 10,745 vertices,
$\boldsymbol{V}_{w}\in\mathbb{R}^{10475\times 3}$. The 3D joint locations of
the whole-body can be obtained by applying a joint regression function $R$
from the posed vertices as
$\boldsymbol{J}^{3D}_{w}=R_{w}(\boldsymbol{V}_{w})$, where
$\boldsymbol{J}^{3D}_{w}\in\mathbb{R}^{(22+20+20+3)\times 3}$.

Stand-Alone Hand Model. Our hand model is defined by taking the hand parts of
SMPL-X:

$\boldsymbol{V}_{h}=H(\boldsymbol{\phi}_{h},\boldsymbol{\theta}_{h},\boldsymbol{\beta}_{h}),\\\
$ (2)

where $\boldsymbol{\theta}_{h}\in\mathbb{R}^{3\times 15}$ is hand pose
parameters and $\boldsymbol{\beta}_{h}$ is the shape parameters for the hand
model. Since our hand model is taken from SMPL-X, $\boldsymbol{\beta}_{h}$
shares the same shape space as $\boldsymbol{\beta}_{w}$. For brevity, we use
$\boldsymbol{\theta}_{h}$, instead of $\boldsymbol{\theta}_{rh}$ or
$\boldsymbol{\theta}_{lh}$, to denote the hand pose parameters of either part
in our description. $\boldsymbol{\phi}_{h}\in\mathbb{R}^{3}$ represents the
global orientation of the hand meshes. Our hand model $H$ produces the hand
mesh structure with 778 vertices, $\boldsymbol{V}_{h}\in\mathbb{R}^{778\times
3}$, where the hand vertices are selected from the hand area of the original
SMPL-X vertices. Given hand vertices $\boldsymbol{V}_{h}$, 3D hand joints can
be regressed as $\boldsymbol{J}^{3D}_{h}=R_{h}(\boldsymbol{V}_{h})$, where
$\boldsymbol{J}^{3D}_{h}\in\mathbb{R}^{21\times 3}$ contains a wrist, 15
finger joints (3 joints per finger), and 5 finger tips. Hand visualization and
skeleton hierarchy are listed in Appendix A1.1. The major advantage of our
hand representation is that the components of this 3D hand model, including
pose parameters, vertices, and 3D joints, are directly compatible with the
whole-body parameterization. This enables us to efficiently integrate outputs
from the body module and the hand module.

Figure 2: Overview of hand module. Our hand module takes a cropped hand image
$\mathbf{I}_{H}$ as input, and produces the parameters of hand model,
$[\boldsymbol{\phi}_{h},\boldsymbol{\theta}_{h},\boldsymbol{\beta}_{h},\boldsymbol{c}_{h}]$.
The estimated parameters are then used to calculate the mesh of SMPL-X hand
part.

###  3.2 3D Hand Estimation Module

We present a monocular 3D hand pose estimation module, denoted by $M_{H}$, to
the parameters of the hand model $H$. In particular, our hand module is
inspired by the recently proposed monocular body pose estimation approaches
[33, 34, 37]. We follow similar model architecture, parameterization, and
training strategies as the pioneering body methods. In particular, we
introduce blur augmentation to make the hand module more robust to blurry
hands that are frequently observed in full body motion capture scenarios.

Network Architecture. Our hand module $M_{H}$ is built upon an end-to-end deep
neural network architecture that can predict hand model parameters defined in
Eq. (2) from given input images. This process is defined as:

$[\boldsymbol{\phi}_{h},\boldsymbol{\theta}_{h},\boldsymbol{\beta}_{h},\boldsymbol{c}_{h}]=M_{H}(\mathbf{I}_{H}),\\\
$ (3)

where $\mathbf{I}_{H}$ is an input RGB image cropped around the hand region.
$\boldsymbol{c}_{h}=(\boldsymbol{t}_{h},s_{h})$ is a set of weak-perspective
camera parameters to project a posed 3D hand model to the input image.
$\boldsymbol{t}_{h}\in\mathbb{R}^{2}$ is for 2D translation on the image
plane, and $s_{h}\in\mathbb{R}$ is a scale factor. Given camera parameters,
the $i$-th 3D hand joint, $\boldsymbol{J}^{3D}_{h,i}$ can be projected as:

$\boldsymbol{J}^{2D}_{h,i}=s_{h}\Pi(\boldsymbol{J}^{3D}_{h,i})+\boldsymbol{t}_{h},$
(4)

where $\Pi$ is an orthographic projection. Following the practice of HMR [33,
37], our hand module $M_{H}$ adopts an encoder-decoder structure, where the
encoder (ResNet-50 [26]) outputs features from an input image, and the decoder
regresses the hand model parameters from the features. See Fig. 2 for the
overview of our hand module. In particular, our hand module is trained with
the assumption that the inputs are right hands. The images and annotations for
the left hand are used after vertical flipping. During inference, the left-
hand images are first flipped and processed as if they were a right hand. Then
the outputs are flipped back to the original left-hand space. Note that the
shape parameter $\boldsymbol{\beta}_{h}$ is originally defined for whole-body
model $\boldsymbol{\beta}_{w}$, but we only consider the deformation for the
hand vertices defined in Eq. (2), ignoring the body part.

Figure 3: Motion Blur Augmentation. We show example images of motion blur
augmentation. From left to right: original images, augmented images after
applying different motion blur kernels.

Training Method. We consider three different types of annotations: (1) 3D pose
annotations (in axis-angle representation), (2) 3D keypoint (joint)
annotations, and (3) 2D keypoint annotations. The losses for each of the
annotation, namely $L_{\boldsymbol{\theta}}$, $L_{3D}$ and $L_{2D}$, are
defined as follows:
$L_{\boldsymbol{\theta}}=\lVert\boldsymbol{\theta}_{h}-{\boldsymbol{\hat{\theta}}}_{h}\rVert^{2}_{2}$,
$L_{3D}=\lVert\boldsymbol{J}^{3D}_{h}-\boldsymbol{\bar{J}}^{3D}_{h}\rVert^{2}_{2}$,
and
$L_{2D}=\lVert\boldsymbol{J}^{2D}_{h}-\boldsymbol{\bar{J}}^{2D}_{h}\rVert_{1}$,
where ${\boldsymbol{\hat{\theta}}}_{h}$, $\boldsymbol{\bar{J}}^{3D}_{h}$ and
$\boldsymbol{\bar{J}}^{2D}_{h}$ are the ground-truth axis-angle pose
parameters, 3D keypoints, and 2D keypoints, respectively. In particular, the
2D keypoint loss are leveraged to estimate the camera parameters and
facilitate the hand module generalize to in-the-wild images with only 2D
keypoints annotation. We do not use the shape parameters provided by the 3D
hand datasets such as FreiHAND [79], since these are defined for the MANO
model [58] and not compatible with our hand model from SMPL-X. Instead, an
additional shape parameter regularization loss
$L_{reg}=\lVert\boldsymbol{\beta}_{h}\rVert^{2}_{2}$ is applied. The overall
loss $L$ used to train our hand module is defined as follows:

$\displaystyle
L=\lambda_{1}L_{\boldsymbol{\theta}}+\lambda_{2}L_{3D}+\lambda_{3}L_{2D}+\lambda_{4}L_{reg}.$
(5)

In experiments, the balanced weights are set as $\lambda_{1}=10$,
$\lambda_{2}=100$, $\lambda_{3}=10$ and $\lambda_{4}=0.1$. Other hand module
training details, including data preprocessing, dataset information, and
implementation details, are included in the Appendix A1.1.

Motion Blur Augmentation. Performing data augmentations during training is a
common practice to enable a model with better generalizability. We firstly
apply common data augmentation strategies, including random scale, random
translation, color jittering, and random rotation. Importantly, we recognize
that in-the-wild videos are often accompanied by severe motion blur. To
achieve robustness to motion blur, we additionally apply motion blur
augmentation to the images. We use the methods [6, 7] to generate blur kernels
and then use 2D filtering to add blurriness to images. Examples of applying
motion blur augmentation are shown in Fig. 3. Our experiments clearly show
that our motion blur augmentation can effectively insist our hand module
generalize to in-the-wild scenarios.

###  3.3 3D Body Estimation Module

Our body module $M_{B}$ produces the torso and limb parameters defined in Eq.
(1) from a single image:

$[\boldsymbol{\phi}_{b},\boldsymbol{\theta}_{b},\boldsymbol{\beta}_{b},\boldsymbol{c}_{b}]=M_{B}(\boldsymbol{I}_{b}),\\\
$ (6)

where $\boldsymbol{I}_{b}$ is an input image cropped around a target single
person’s whole-body. Similar to Eq. (1),
$\boldsymbol{\phi}_{b}\in\mathbb{R}^{3}$ is the global body orientation,
$\boldsymbol{\theta}_{b}\in\mathbb{R}^{21\times 3}$ is the body pose
parameters (without any finger joints), and
$\boldsymbol{\beta}_{b}\in\mathbb{R}^{10}$ is the shape parameter.
$\boldsymbol{\beta}_{b}$ shares the same parameterization space as
$\boldsymbol{\beta}_{w}$, as defined in Eq. (1). Similar to Eq. (3), we use
weak perspective camera parameters
$\boldsymbol{c}_{b}=(\boldsymbol{t}_{b},s_{b})$.

We leverage the publicly available body pose estimation model (SPIN [37])
trained by the EFT dataset [30]. Since the original SPIN model is built with
SMPL, we fine-tune the network by replacing SMPL with SMPL-X. For the fine-
tuning process, we use the existing indoor 3D pose dataset Human3.6M [27] and
outdoor pseudo-GT dataset by EFT [30] that provides SMPL annotations. Since
the annotation (SMPL) and our target model (SMPL-X) are not exactly
compatible, we only use the pose parameters and 2D keypoint from the
annotations during fine-tuning process, ignoring the shape parameters 333Even
if the 3D joint locations of SMPL and SMPL-X are not exactly identical, we
found that this distinction is not very significant in practice..

###  3.4 3D Face Estimation Module

We use the off-the-shelf public 3D face estimation model RingNet [61] to
estimate facial expressions $\boldsymbol{\psi}_{f}$ and face poses
$\boldsymbol{\theta}_{f}$ as follows:

$[\boldsymbol{\theta}_{f},\boldsymbol{\psi}_{f}]=M_{F}(\boldsymbol{I}_{f}).\\\
$ (7)

Since RingNet is based on FLAME [43], its predictions are compatible with the
face part of SMPL-X. As an adjustment, we only keep the first $10$ expression
parameters from the original $50$ dimensional expression parameters
$\hat{\psi_{f}}$ predicted by RingNet, to make it compatible with the
expression space defined in SMPL-X. It is worth noting that the FLAME and
SMPL-X share the same PCA-based expression space, while SMPL-X only uses the
$10$ foremost principal components.

###  3.5 Whole-Body Integration Module

Our integration module combines the outputs from the face, hands, and body
modules into a unified representation of the SMPL-X [56]. While the output of
the face module can be easily applied to the face part of the SMPL-X model by
copying the expression parameters and jaw poses, additional procedures are
required to integrate the outputs of the hand and body modules because
different wrist poses are estimated from these two modules. For this
objective, we present three strategies: (1) by simple copy-paste composition
for fastest processing, (2) by an optimization framework, (3) by an
integration network to approximate the optimization via a simple neural
network.

By Copy-Paste. Our hand and body modules’ outputs can be efficiently combined,
since they are both defined in SMPL-X parameterization. As the simplest
strategy, we transfer the corresponding joint angle parameters from the
outputs of each module into the whole-body model. The wrist pose parameters
require additional processing because we obtain two different outputs from the
body module and the hand module (the wrist pose of hand module is represented
by the global hand orientation $\boldsymbol{\phi}_{h}$). Let us denote the
pose parameters for the wrist joint as $\boldsymbol{\theta}^{\text{wrist}}$,
then
$\boldsymbol{\theta}_{b}=\hat{\boldsymbol{\theta}}_{b}\cup\\{\boldsymbol{\theta}_{b}^{\text{rwrist}},\boldsymbol{\theta}_{b}^{\text{lwrist}}\\}$,
where $\hat{\boldsymbol{\theta}}_{b}$ represents all body pose parameters
except wrists. We use the similar notations for the whole-body pose
parameters,
$\boldsymbol{\theta}_{w}^{b}=\hat{\boldsymbol{\theta}}_{w}\cup\\{\boldsymbol{\theta}_{w}^{\text{rwrist}},\boldsymbol{\theta}_{w}^{\text{lwrist}}\\}$.
Then, whole-body integration by copy-paste can be performed as:

$\displaystyle\left(\boldsymbol{\phi}_{w},\boldsymbol{\beta}_{w},\boldsymbol{c}_{w},\hat{\boldsymbol{\theta}}_{w}^{b},\boldsymbol{\theta}_{w}^{lh},\boldsymbol{\theta}_{w}^{rh}\right)$
$\displaystyle=\left(\boldsymbol{\phi}_{b},\boldsymbol{\beta}_{b},\boldsymbol{c}_{b},\hat{\boldsymbol{\theta}}_{b},\boldsymbol{\theta}_{lh},\boldsymbol{\theta}_{rh}\right),$
(8)
$\displaystyle\left(\boldsymbol{\theta}_{w}^{\text{lwrist}},\boldsymbol{\theta}_{w}^{\text{rwrist}}\right)$
$\displaystyle=\left(\Gamma_{l}\left(\boldsymbol{\theta}_{b},\boldsymbol{\phi}_{lh}\right),\Gamma_{r}\left(\boldsymbol{\theta}_{b},\boldsymbol{\phi}_{rh}\right)\right),$

where $\Gamma_{\\{l,r\\}}$ are the functions to convert the global wrist
orientation $\boldsymbol{\phi}_{h}$ obtained from the hand module to the local
wrist pose parameters w.r.t. its parent joint in the SMPL-X skeleton
hierarchy. This can be implemented by comparing $\boldsymbol{\phi}_{h}$ with
the global orientation of the current wrist pose from
$\boldsymbol{\theta}_{b}$ that can be computed by following the forward
kinematics of the body skeleton hierarchy. This strategy requires almost no
extra computation, making our separate modules contribute a common whole-body
model simultaneously. We found this simple integration produces convincing
results, especially for the scenarios with computational bottlenecks as in our
live demo.

By Optimization. As an alternative integration method for better accuracy, we
build an optimization framework to fit the whole-body model parameters given
the outputs from body and hand modules. This strategy is particularly helpful
to reduce the artifact around the wrist parts over the copy-paste strategy. It
can also take advantage from the 2D keypoint estimation output [13] for better
2D localization quality. Our optimization framework finds the whole-body model
parameters that minimize the following objective cost function:

$\begin{gathered}\mathcal{F}([\boldsymbol{\phi}_{w},\boldsymbol{\theta}_{w},\boldsymbol{\beta}_{w},\boldsymbol{c}_{w}])=\mathcal{F}^{2d}+\mathcal{F}^{mesh}+\mathcal{F}^{pri},\end{gathered}$
(9)

where $\mathcal{F}^{2d}$ is the 2D re-projection cost term between the 2D
keypoint estimation [13] and the projection of 3D joints (hand, body and
faces). $\mathcal{F}^{mesh}$ is the 3D distance loss between the hand mesh of
whole-body model and the mesh vertices of the hand module’s output localized
on the body’s wrist joints, as shown in Fig. 4. The prior term
$\mathcal{F}^{pri}$ is needed to keep the 3D pose and shape parameters in
plausible space. We use the EFT introduced in [30] for the similar goal, by
applying neural network finetuning to replace explicit 3D pose prior terms
used by previous works [56]. Therefore, the prior term only accounts for the
shape regularization $\mathcal{F}^{pri}=\|\boldsymbol{\beta}_{w}\|^{2}_{2}$.

We use a multi-stage approach in which optimization is firstly performed
without $\mathcal{F}^{mesh}$. Then we optimize all terms in the second stage.
During the second stage, we add an additional constraint $\mathcal{F}^{3D}$ to
maintain the 3D joints within the final locations of the first stage. Our
experiments show that this term can produce reliable results in practice. See
Fig. 4 for the example of our optimization.

Figure 4: Optimizing the whole-body model (SMPL-X) with 3D hand prediction and
2D keypoint estimation. (a) Input images and estimated 2D keypoints by
OpenPose [13]; (b) 3D body pose estimation from body module; (c) The output of
3D hand module aligned to the wrist joints of SMPL-X; (d) Integration output
by copy-paste ; (e) Integration output by optimization framework.

By Wrist Integration Network. The result of the hand module shows precise 2D
localization quality as shown in Fig. 5. But we lose its precision by applying
copy-paste integration strategy since the wrist joint location is determined
by the body module, as shown in Fig. 4 (second last column). Although the
optimization method can achieve better precision and resolve this problem, it
suffers from relatively slow computation speed due to the requirement of
iterative gradient computations. As another alternative strategy to achieve
both precision and fast-runtime performance, we introduce an integration
network to adjust the arm pose of the whole-body model toward the given 2D
location without gradient computation. Specifically, we develop a small neural
network to achieve this:

$\begin{gathered}\tilde{\boldsymbol{\theta}}_{w}^{arm}=\mathcal{H}(\boldsymbol{\theta}_{w}^{arm},\textbf{d}),\end{gathered}$
(10)

where $\textbf{d}\in\mathbb{R}^{2}$ is a 2D directional vector that the arm
needs to follow in the image space. It is obtained from the wrist locations
calculated from the body module and those from the hand module, followed by a
normalization with the length of the arms. $\boldsymbol{\theta}_{w}^{arm}$ is
the pose parameters of the elbow and shoulder joints after the copy-paste
integration. Conceptually, $\mathcal{H}$ predicts an approximation of
optimization gradient to adjust the arm parameters, similar to the recent work
of Song  _et al_. [63]. We use a MLP with six layers to implement
$\mathcal{H}$. To train $\mathcal{H}$, we generate a synthetic dataset by
capturing our own a range of motion videos to cover diverse arm pose
variations and by simulating arbitrary d directions. More details of wrist
integration are described in Appendix A2.

##  4 Experiments

Table 1: Comparison of our hand module with the state-of-the-art 3D hand
methods using 2D/3D AUC of PCK.

Method $\rightarrow$ | Bouk [8] | Baek [2] | Ge [20] | Zhang [75] | Kulon [39] | CMR [15] | Ours
---|---|---|---|---|---|---|---
Dataset $\downarrow$
STB | 0.994 | - | 0.995 | 0.995 | - | - | 0.992
RHD | - | 0.926 | 0.92 | 0.901 | 0.95 | 0.944 | 0.934
MPII+NZSL | 0.501 | - | 0.15 | - | 0.701 | - | 0.655

Figure 5: Qualitative comparison with state-of-the-art 3D hand methods. Images
are selected from COCO [45]. Table 2: Ablation study on hand module data
augmentation. Models are evaluated on MPII+NZSL [62] with 2D AUC.

Translation | Rescale | Color | Rotation | Blur | 2D AUC
---|---|---|---|---|---
✗ | ✗ | ✗ | ✗ | ✗ | 0.608
✓ | ✓ |  |  |  | 0.610
✓ | ✓ | ✓ |  |  | 0.618
✓ | ✓ | ✓ | ✓ |  | 0.622
✓ | ✓ | ✓ | ✓ | ✓ | 0.655

Figure 6: Ablation study on data augmentations.

We first demonstrate that our hand module achieves high performance competing
to the previous state-of-the-art 3D hand methods, along with ablation studies
to examine the designs of our method. Then, we demonstrate that our whole-body
pose estimation method outperforms previous approaches on a public benchmark.
See Appendix A1 for the details of implementation and datasets.

###  4.1 Hand Module Evaluation

Comparison with State-of-the-art Methods. We compare our hand module with the
previous state-of-the-art hand approaches on three public hand benchmarks,
namely STB [74], RHD [78] and MPII+NZSL [62]. The results are listed in Tab. 1
and Fig. 5. For each validation dataset, we calculate the percentage of
correct keypoints (PCK) under different thresholds and calculate the
corresponding Area Under Curve (AUC) for PCK. For STB [74] and RHD [78], we
use 3D AUC and the threshold ranges from $20$ mm to $50$ mm. For MPII+NZSL
[62], we use 2D AUC and the threshold ranges from $0$ px to $30$ px. For fair
comparison, all the methods take a single RGB image as input. As shown in the
results, our hand module is comparable with the previous state-of-the-art
hand-only methods based on MANO [58], though our hand model, taken from the
whole-body model, has restricted shape variations 444Note that the code of
Kulton  _et al_. [39] is not publicly available, and, thus, we are unable to
make more comparisons in challenging in-the-wild scenes, our target
applications. CMR [15] requires camera intrinsic as known, which prevents
applying them to in-the-wild scenarios..

Ablation Study. We examine the data augmentation strategies used in training
hand module. The results are listed in Tab. 2 and Fig. 6. The results in Tab.
2 demonstrate that applying data augmentation leads to better results. The
qualitative results in Fig. 6 further show that adopting data augmentation
helps our model generalize better on challenging scenarios, including blur,
pose variations, and occlusion. In Fig. 6, “No Augment” refers to the model
trained without any data augmentation, and “No Blur” refers to the model
trained with all data augmentation strategies except the motion blur
augmentation. “Full” refers to the model trained with all data augmentation
strategies. We also do an ablation study on mixture usage of datasets,
demonstrating that using diverse datasets improves the performances. See
Appendix A1.3 for details on this part.

###  4.2 Body Module Evaluation

Table 3: Comparison of our body module with the state-of-the-art body-only
methods on 3DPW [67].

Method $\rightarrow$ | HMR [33] | CMR [38] | SPIN [37] |  EFT [30] | ExPose [18] | Ours
---|---|---|---|---|---|---
Metric $\downarrow$ |
MPJPE | 130 | 127.2 | 96.9 |  92.3 | 93.4 | 94.3
PA-MPJPE | 81.3 | 70.2 | 59.2 |  54.2 | 60.7 | 60.0

We compare our body module with the previous SOTA body-only methods on the
test set of 3DPW [67]. Following the previous approaches, we use Mean Per
Joint Position Errors (MPJPE) and PA-MPJPE (after Procrustes analysis [22]).
The results are listed in Tab. 3. Our body module is comparable to the
previous state-of-the-arts, including ExPose [18]. Our body model is a fine-
tuned version of SPIN [37] and EFT [30]. The slightly degraded performance
might be due to the reason that we only use fewer datasets, COCO and Human3.6M
datasets. We only compare with HMR [33]-based methods. We believe that the
experimental results shown in Tab. 3 can demonsrate the feasibility of
extending previous SMPL-based methods to SMPL-X based body module. Thanks to
our modularized design, we can also easily leverage the most recent 3D body
poses methods [44, 42] to build our body module.

###  4.3 Integration Module Evaluation

Table 4: Quantitative comparison with previous SMPL-X based whole-body model
on EHF dataset [56].

Methods $\rightarrow$ | SMPLify-X [56] | ExPose [18] | FM-CP | FM-WI | FM-OPT
---|---|---|---|---|---
Part & Metrics $\downarrow$
All | V2V | 146.2 | 81.9 | 69.9 | 69.3 | 63.5
PA-V2V | 68.0 | 57.4 | 58.2 | 57.1 | 54.7
Body | V2V | 231.3 | 116.4 | 83.7 | 83.9 | 93.1
PA-V2V | 75.4 | 55.1 | 52.7 | 53.2 | 53.3
L Hand | V2V | 34.6 | 39.3 | 27.1 | 27.0 | 28.6
PA-V2V | 11.4 | 13.1 | 10.9 | 10.9 | 10.8
R Hand | V2V | 34.3 | 39.3 | 25.4 | 25.5 | 25.6
PA-V2V | 13.1 | 12.9 | 11.2 | 11.2 | 11.2
Face | V2V | 23.6 | 19.5 | 30.7 | 30.7 | 20.4
PA-V2V | 8.4 | 5.6 | 5.7 | 5.7 | 5.4
Runtime (sec/frame) | 60 | 0.15 | 0.15 | 0.15 | 2

Figure 7: Qualitative comparison of FrankMocap model with different
integration modules. Figure 8: Qualitative comparison over the previous whole-
body pose estimation methods. We compare our method adopting the wrist
integration module (FM-WI) with MTC [70], SMPLify-X [56] and ExPose [18].

We quantitatively compare our methods with previous SMPL-X based whole-body
methods (SMPLify-X [56] and ExPose [18]) on EHF dataset [56]. The evaluation
metrics are vertex-to-vertex distance (V2V) and Procrustes Analysis vertex-to-
vertex distance (PA-V2V). The results are listed in Tab. 4. We run the
officially released codes of SMPLify-X and ExPose to obtain the evaluation
results. FM-CP, FM-WI, and FM-OPT indicate FrankMocap using the copy-paste,
wrist integration network, and optimization-based as the integration module,
respectively. The results demonstrate that all our methods significantly
outperform the previous methods in terms of whole-body estimation, body
estimation, and hand estimation. Our face estimation is comparable with
previous methods, where we only use an off-the-shelf 3D face model [61].
Comparing across different versions of FrankMocap reveal that the model with
optimization achieves the best performance in general. Applying our wrist
integration network can also lead to better performance than copy-paste
version. We also provide qualitative comparisons between different integration
modules in Fig. 7. It is revealed that using the wrist integration network can
bring better hand location while adopting optimization can further improve
hand localization while updating body shape and other poses.

We also qualitatively compare our method with previous whole-body motion
capture approaches, MTC [70], SMPLify-X [56], and ExPose [18], with our output
with wrist integration network. Results shown in Fig. 8 and the demo video
demonstrate that our method outperforms previous approaches, with fast and
accurate performance in challenging in-the-wild scenes. Qualitative comparison
with Zhou  _et al_. [76] is included in the Appendix A5.

##  5 Conclusion

We present FrankMocap, a monocular 3D whole-body motion capture system built
upon the motivation to take full advantages of state-of-the-art part
estimation modules, followed by effective integration algorithms. We develop
three integration modules, namely copy-paste, wrist integration network, and
an optimization-based method. The copy-paste and wrist integration modules are
suitable for time-sensitive applications, while the optimization-based module
is suitable for offline applications requiring better precision. Our system
surpasses previous state-of-the-art 3D whole-body estimation methods on both
public benchmarks and in-the-wild scenarios. Notably, our approach can
directly leverage future algorithms by replacing each module, which is a main
advantage of our modular design.

Acknowledgements. We acknowledge Yuting Ye from FRL, Donglai Xiang and Yu Ding
from CMU, Yu Xiong, Anyi Rao, and Jingbo Wang from CUHK, and Jie Chen from
Microsoft.

## Appendix

##  Appendix A1 Hand Module

In this section, we provide more details of our stand-alone hand module
described in Sec. 3.2. We first describe the hand mesh and hierarchy of the
hand skeleton. Then we discuss how to preprocess data from different datasets.
Then we discuss the implementation details of our hand module. In the end, we
discuss the hand datasets used in our paper and provide experiment results to
demonstrate the effectiveness of adopting different datasets.

###  A1.1 Hand Module Details

Hand Mesh. The visualization of cropped hand mesh and hierarchy of the hand
skeleton is depicted in Fig. 9.

Data preprocessing. Following the practice of 3D body pose estimation methods
[33, 34, 59, 37], we include as many publicly available datasets as possible
towards in-the-wild generalization ability. To handle the discrepancy of
annotations across different datasets, we perform several pre-processing steps
to make them consistent and compatible with our hand model. The pre-processing
includes: (1) Rescaling all 3D keypoint annotations to be compatible with our
hand model, by using the middle finger’s knuckle length as a reference. (2)
Reordering the 3D keypoints to be the same as our hand model’s skeleton
hierarchy.

Hand Module Implementation Details. Input images of the hand module are
center-cropped surrounding the hands, where the bounding boxes for cropping
are given by 2D hand keypoints. We use ground-truth 2D keypoints during
training and predicted keypoints from OpenPose [13] during testing. The
cropped images are further padded and resized to $224\times 224$. During
training, we apply data augmentations to each of the training images via
random scaling, translation, rotation, color jittering, and synthetic motion
blur. The hand module architecture is based on ResNet-50 [26] with two
additional fully connected layers to map the output features of ResNet to
vectors with $61$ dimension, which is composed of camera parameters
$\boldsymbol{c}_{h}$ (3 dimensions), hand global rotation
$\boldsymbol{\phi}_{h}$ (3 dimensions), hand pose parameters
$\boldsymbol{\theta}_{h}$ (45 dimensions) and shape parameters
$\boldsymbol{\beta}_{h}$ (10 dimensions). The hand module is implemented with
PyTorch [55]. The Adam optimizer [35] with learning rate $1\mathrm{e}{-4}$ is
used to train the model. The hand module is trained until converge, which
takes about $100$ epochs.

Figure 9: We take the hand part of SMPL-X as a stand-alone hand model for hand pose estimation. The example mesh is shown in (a) and the skeleton hierarchy is shown in (b). Table 5: In training our hand module, we use five datasets including: FreiHAND [79], HO-3D [25], MTC [70], STB [74] and RHD [78]. We use MPII+NZSL [62] for validation only. Information $\rightarrow$ | # of Samples | Pose Angles | 3D Joints | 2D Joints
---|---|---|---|---
Dataset $\downarrow$
FreiHAND [79] | 60K | ✓ | ✓ | ✓
HO-3D [25] | 60K | ✓ | ✓ | ✓
MTC [70] | 200K |  | ✓ | ✓
STB [74] | 20K |  | ✓ | ✓
RHD [78] | 50K |  | ✓ | ✓
MPII+NZSL [62] | 10K |  |  | ✓
Figure 10: Ablation study on training datasets. We show qualitative ablation
study on using different datasets in training our hand model. “Subset-01”
means using the combination of datasets FreiHADN [79] and HO-3D [25].
“Subset-02” means using the combination of datasets: STB [74], RHD [78] and
MTC [70]. “Full set” means using all the above datasets. The images are
selected from COCO dataset [45].

###  A1.2 Hand Datasets

To increase our hand module’s generalization ability, we adopt as many public
datasets as possible. The summary of different datasets is listed in Tab. 5.
The details of these datasets are listed below.

FreiHAND. FreiHAND [79] is a dataset with ground truth 3D hand joints and MANO
parameters for real human hand images. The 3D annotations are obtained by a
multi-camera system and a semi-automated approach. The obtained data is
further augmented with synthetic backgrounds. In our experiments, we randomly
select 80% of samples from the original training set as training data and use
the remaining 20% of samples for validation.

HO-3D. HO-3D dataset [25] is a dataset aiming to study the interaction between
hands and objects. The dataset has 3D joints and MANO pose parameters for
hands. It also has 3D bounding boxes for objects the hands interact with. In
this paper, we only use 3D annotations of hands. The training set is composed
of different sequences, each of which records one type of hand-object
interaction. Following the similar practice in processing FreiHAND, we
randomly choose 80% of sequences from the original training set as training
data and use the remaining 20% of sequences for validation.

MTC. Monocular Total Capture [70] is a dataset captured by Panoptic Studio
[29, 31] in a multi-view setup with 30 HD cameras. It has 3D hand joints
annotations for both body and hands. The sequences are mainly the range of
motion data of multiple subjects. To polish the dataset, we filter out
erroneous samples where hands are not visible or too small.

STB. Stereo Hand Pose Tracking Benchmark [74] is composed of 15,000 training
samples and 3,000 testing samples. During our experiments, we only use the RGB
images and the paired 3D joint annotations. We use the training set of STB to
train our model and compare our models with other state-of-the-art methods on
the validation set. To unify the definition of joints, following the practice
of [9, 20], we move the root joint from the palm center to the wrist.

RHD. Rendered Hand Dataset [78] is a synthetic dataset that has 2D and 3D hand
joint annotations. It is composed of 41,258 training samples and 2,728 testing
samples. We train our models on the training set and compare our models with
other state-of-the-art methods on the testing set.

MPII+NZSL. MPII+NZSL dataset [62] is composed of in-the-wild images with
manually annotated 2D hand joints. It includes challenging images with
occlusion, blur, and low resolution. To show our models’ generalization
ability, we do not incorporate MPII+NZSL into training set. We only use it for
validation.

###  A1.3 Ablation Study on Hand Datasets.

FreiHAND [79] |  HO-3D [25] |  MTC [70] |  STB [74] |  RHD [78] |  MPII+NZSL [62]
---|---|---|---|---|---
✓ |  |  |  |  | 0.482
| ✓ |  |  |  | 0.367
✓ | ✓ |  |  |  | 0.526
|  | ✓ | ✓ | ✓ | 0.556
✓ | ✓ | ✓ |  |  | 0.595
✓ | ✓ | ✓ | ✓ |  | 0.598
✓ | ✓ | ✓ |  | ✓ | 0.645
✓ | ✓ | ✓ | ✓ | ✓ | 0.655

Table 6: Ablation study on dataset. We show the results of our hand module
trained with different datasets. These models are evaluated on MPII+NZSL [62]
using 2D AUC as metric. For data augmentation, we use all the available
datasets.

We examine the influence of using different datasets in this subsection. The
results are listed in Tab. 6 and Fig. 10. As expected, the results in Tab. 6
show that using more datasets will lead to better performance. We also show
examples of qualitative comparison in Fig. 10. Similar to the conclusion from
the quantitative study, the qualitative results show that incorporating more
datasets can increase the models’ generalization ability and generate more
precise results for in-the-wild images. In the figure, “Subset-01” means using
the combination of datasets FreiHAND [79] and HO-3D [25]. “Subset-02” means
using the combination of datasets: STB [74], RHD [78] and MTC [70]. “Full set”
means using all the datasets.

Figure 11: Overview of the proposed wrist integration network.

##  Appendix A2 Wrist Integration Network.

###  A2.1 Framework Details.

The framework of wrist integration network is depicted in Fig. 11. Given hand
prediction from hand module and whole-body prediction from copy-paste
integration module, the role of wrist integration network is to adjust the arm
poses to move hands to the locations determined by the hand module. The input
of the wrist integration network is composed of two parts. The first part is
the 2D directional vector $d$ that the arm needs to follow in the image space.
It is obtained from the wrist locations calculated from the body module and
those from the hand module, followed by a normalization with the length of
arm. The second part $\boldsymbol{\theta}_{w}^{arm}$ is the pose parameters of
the elbow and shoulder joints the copy-paste integration. The original
shoulder poses $\boldsymbol{\theta}_{w}^{s}$ and elbow poses
$\boldsymbol{\theta}_{w}^{e}$ are all local rotations relative to their parent
defined in the kinematics of body skeleton. To ease the training of the wrist
integration network, we transform the shoulder poses to the global orientation
$\boldsymbol{\phi}_{w}^{s}$, so that the wrist integration network does not
need to consider other body parts’ poses. The elbow poses are kept as the
local rotation. In this way, the input arm poses
$\boldsymbol{\theta}_{w}^{arm}$ is composed of global orientation of shoulder
$\boldsymbol{\phi}_{w}^{s}$ and local rotations of elbow
$\boldsymbol{\theta}_{w}^{e}$.

$\begin{gathered}\boldsymbol{\theta}_{w}^{arm}=[\boldsymbol{\phi}_{w}^{s},\boldsymbol{\theta}_{w}^{e}]=[\Upsilon_{s}(\boldsymbol{\theta}_{w}^{s},\boldsymbol{\theta}_{w}),\boldsymbol{\theta}_{w}^{e}],\end{gathered}$
(11)

where $\Upsilon_{s}$ is the function to convert the local shoulder rotation
$\boldsymbol{\theta}_{w}^{s}$ to global shoulder orientation
$\boldsymbol{\phi}_{w}^{s}$. $\Upsilon_{s}$ can be implemented by forwarding
the kinematics of the SMPL-X body skeleton from the root to the shoulder.

Taking the arm poses $\boldsymbol{\theta}_{w}^{arm}$ and 2D directional vector
as input, the wrist integration network predicts the updated arm poses
$\ddot{\boldsymbol{\theta}}_{w}^{arm}$. The whole process is formulated as Eq.
(12).

$\begin{gathered}\ddot{\boldsymbol{\theta}}_{w}^{arm}=\mathcal{H}(\boldsymbol{\theta}_{w}^{arm},\textbf{d})\end{gathered}$
(12)

It is worth nothing that $\ddot{\boldsymbol{\theta}}_{w}^{arm}$ obtained in
this step is not the final updated arm poses
$\tilde{\boldsymbol{\theta}}_{w}^{arm}$ defined in Eq. (10).
$\ddot{\boldsymbol{\theta}}_{w}^{arm}=[\tilde{\boldsymbol{\phi}}_{w}^{s},\tilde{\boldsymbol{\theta}}_{w}^{e}]$
is composed of updated global shoulder orientation
$\tilde{\boldsymbol{\phi}}_{w}^{s}$ and updated local elbow rotation
$\tilde{\boldsymbol{\theta}}_{w}^{e}$. To make the obtained shoulder poses
compatible with the whole-body poses $\boldsymbol{\theta}_{w}$ defined in Eq.
(8), we convert the global shoulder orientation
$\tilde{\boldsymbol{\phi}}_{w}^{s}$ back to the local shoulder rotation
$\tilde{\boldsymbol{\theta}}_{w}^{s}$. This process is similar to converting
global hand orientation $\boldsymbol{\phi}_{h}$ to local hand rotation
$\boldsymbol{\theta}_{h}$ defined in Eq. (8). Since the updated local elbow
rotation $\tilde{\boldsymbol{\theta}}_{w}^{e}$ is already the local rotation,
there is no need to further process it. The final updated arm poses
$\tilde{\boldsymbol{\theta}}_{w}^{arm}$ with pose shoulder and elbow arms in
the local rotation, as defined in Eq. 10, can be formulated as:

$\begin{gathered}\tilde{\boldsymbol{\theta}}_{w}^{arm}=[\tilde{\boldsymbol{\theta}}_{w}^{s},\tilde{\boldsymbol{\theta}}_{w}^{e}]=[\Gamma_{s}(\boldsymbol{\phi}_{s},\boldsymbol{\theta}_{w}),\tilde{\boldsymbol{\theta}}_{e}]\end{gathered}$
(13)

where $\Gamma_{s}$ is the function to convert the global shoulder orientation
$\tilde{\boldsymbol{\phi}}^{s}_{w}$ to local shoulder rotation
$\tilde{\boldsymbol{\theta}}_{w}^{s}$. Its definition is similar to
$\Gamma_{l}$ and $\Gamma_{r}$ defined in Eq. (8).

###  A2.2 Implementation Details.

The wrist integration network is a MLP with 6 layers. The input dimension is
$8$, which is composed of $\boldsymbol{\theta}_{w}^{arm}\in\mathbb{R}^{2\times
3}$ and $d\in\mathbb{R}^{2}$. The output dimension is $6$ which is
$\tilde{\boldsymbol{\theta}}_{w}^{arm}\in\mathbb{R}^{2\times 3}$. The
dimension of the intermediate layers are $128$, $256$, $512$, $256$, $128$,
separately. For wrist integration network, we assume the input arm poses and
output arm poses are only right arm poses. During training, we only use right
arm poses and right hand directional movement. During inference, the input
initial left arm poses and 2D directional vector are vertically flipped as
there were right arms and hands. The output arm poses are then flipped back to
the left arm pose space.

Figure 12: Samples of synthesized data for training wrist integration network.
The right arrows are the random translation $\dot{d}$ to be added.The green
bodies are rendered from the original optimization output. The blue arms are
the obtained arm poses $\tilde{\boldsymbol{\theta}}_{w}^{arm}$ after applying
random translation.

###  A2.3 Training Data.

To train $\mathcal{H}$, we generate a synthetic dataset by capturing our own a
range of motion of right arms. We first capture videos with varying arm poses.
Then we use the optimization method introduced in Section 3.5 to obtain the
pseudo ground-truth 3D pose estimations for those video frames. Suppose for
one data sample, we obtain $\boldsymbol{\phi}_{w}$, $\boldsymbol{\theta}_{w}$,
$\boldsymbol{\beta}_{w}$, $\boldsymbol{c}_{w}$, which are global rotation,
whole-body pose, whole-body shapes and camera parameters. To increase the
model’s generalization ability to varying body shapes, we add random noise to
the obtain body shape $\boldsymbol{\beta}_{w}$. Then we use the updated body
shapes and other obtained parameters to get projected right 2D hand joints
$\boldsymbol{J}^{2D}_{rh}$ by projecting the predicted 3D right hand joints
$\boldsymbol{J}^{3D}_{rh}$ using Eq. 4. After that, we add random translation
$\dot{\textbf{d}}=[\dot{d}_{x},\dot{d}_{y}]$ to the original predicted 2D hand
joints to obtain permutated 2D hand joints
$\boldsymbol{\dot{J}}^{2D}_{rh}=\boldsymbol{J}^{2D}_{rh}+\dot{\textbf{d}}$. We
run the optimization method again to obtain the new poses using the permutated
2D hand joints $\boldsymbol{\dot{J}}^{2D}_{rh}$ in calculating
$\mathcal{F}^{2d}$ defined in Eq. (9). In this process, we fix other
parameters and only update the right shoulder and right elbow poses, obtaining
$\tilde{\boldsymbol{\theta}}_{w}^{arm}$. In the end, we obtain training data
with hand directional vector $\dot{\textbf{d}}$, original poses
$\boldsymbol{\theta}_{w}^{arm}$ and updated arm poses
$\tilde{\boldsymbol{\theta}}_{w}^{arm}$. They form the training data to train
wrist integration network defined in Fig. 11.

We visualize several generated data samples in Fig. 12. The right arrows are
the random translation $\dot{\textbf{d}}$ to be added. The green bodies are
rendered from the original optimization prediction. The blue arms are the
obtained arm poses $\tilde{\boldsymbol{\theta}}_{w}^{arm}$ after applying
randomly translating.

##  Appendix A3 Implementation Details

Bounding Bboxes. We use a lightweight Pytorch OpenPose implementation [54],
which only predicts 2D body keypoints, to obtain body bounding bboxes. Face
and hand bounding bboxes can also be obtained from OpenPose [13]. For live
demo requiring higher speed, the bounding boxes are obtained by projecting the
face and hand part of the estimated 3D body to image space.

Figure 13: Typical failure cases from FrankMocap. It is shown that typical
failure cases are caused by severe occlusion and blurry. Figure 14:
Quanlitative comparison with Zhou  _et al_. [76]. Hand regions are marked with
red circles. It is revealed that our method can generate more precise 3D hand
poses than Zhou  _et al_. [76] do.

##  Appendix A4 Failure Cases

We show several typical failure cases in Fig. 13. It is revealed that failure
cases are typically caused by severe occlusion and blurry.

##  Appendix A5 More Qualitative Results

Comparison with Zhou  _et al_. Firstly, we show qualitative comparisons with
Zhou  _et al_. [76] in Fig. 14. It is worth noting that Zhou  _et al_. [76]
have not released their official code yet. Therefore, we compare our methods
with them using images shown in their paper. The results demonstrate that our
methods can generate more accurate hand and body poses than Zhou  _et al_.
[76] does.

Demo. We show more qualitative results of FrankMocap in Fig.. 15. The
integration module is the Wrist Integration Network. More qualitative results
are shown in the demo videos.

Figure 15: Qualitative results for FrankMocap with wrist integration network
adopted.

## References

  * [1] Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis.  Scape: shape completion and animation of people.  ACM Transactions on Graphics, 2005.
  * [2] Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim.  Pushing the envelope for rgb-based dense 3d hand pose estimation via neural rendering.  In IEEE Conference on Computer Vision and Pattern Recognition, 2019\.
  * [3] Timur Bagautdinov, Chenglei Wu, Tomas Simon, Fabian Prada, Takaaki Shiratori, Shih-En Wei, Weipeng Xu, Yaser Sheikh, and Jason Saragih.  Driving-signal aware full-body avatars.  ACM Transactions on Graphics, 2021.
  * [4] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3d faces.  In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, 1999.
  * [5] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black.  Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image.  In European Conference on Computer Vision, 2016.
  * [6] Giacomo Boracchi and Alessandro Foi.  Uniform motion blur in poissonian noise: Blur/noise tradeoff.  IEEE Transactions on Image Processing, 2010.
  * [7] Giacomo Boracchi and Alessandro Foi.  Modeling the performance of image restoration from motion blur.  IEEE Transactions on Image Processing, 2012.
  * [8] Adnane Boukhayma, Rodrigo de Bem, and Philip HS Torr.  3d hand shape and pose from images in the wild.  In IEEE Conference on Computer Vision and Pattern Recognition, 2019\.
  * [9] Yujun Cai, Liuhao Ge, Jianfei Cai, and Junsong Yuan.  Weakly-supervised 3d hand pose estimation from monocular rgb images.  In European Conference on Computer Vision, 2018.
  * [10] Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden.  Multi-channel transformers for multi-articulatory sign language translation.  In European Conference on Computer Vision Workshop, 2020.
  * [11] Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou.  Facewarehouse: A 3d facial expression database for visual computing.  In IEEE Transactions on Visualization and Computer Graphics, 2014\.
  * [12] Kaidi Cao, Yu Rong, Cheng Li, Xiaoou Tang, and Chen Change Loy.  Pose-robust face recognition via deep residual equivariant mapping.  In IEEE Conference on Computer Vision and Pattern Recognition, 2018\.
  * [13] Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Openpose: realtime multi-person 2d pose estimation using part affinity fields.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019\.
  * [14] Feng-Ju Chang, Anh Tuan Tran, Tal Hassner, Iacopo Masi, Ram Nevatia, and Gerard Medioni.  Expnet: Landmark-free, deep, 3d facial expressions.  In IEEE International Conference on Automatic Face & Gesture Recognition, 2018.
  * [15] Xingyu Chen, Yufeng Liu, Chongyang Ma, Jianlong Chang, Huayan Wang, Tian Chen, Xiaoyan Guo, Pengfei Wan, and Wen Zheng.  Camera-space hand mesh recovery via semantic aggregation and adaptive 2d-1d registration.  In IEEE Conference on Computer Vision and Pattern Recognition, 2021\.
  * [16] Yujin Chen, Zhigang Tu, Di Kang, Linchao Bao, Ying Zhang, Xuefei Zhe, Ruizhi Chen, and Junsong Yuan.  Model-based 3d hand reconstruction via self-supervised learning.  In IEEE Conference on Computer Vision and Pattern Recognition, 2021\.
  * [17] Hongsuk Choi, Gyeongsik Moon, and Kyoung Mu Lee.  Pose2mesh: Graph convolutional network for 3d human pose and mesh recovery from a 2d human pose.  In European Conference on Computer Vision, 2020.
  * [18] Vasileios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black.  Monocular expressive body regression through body-driven attention.  In European Conference on Computer Vision, 2020.
  * [19] Yao Feng, Fan Wu, Xiaohu Shao, Yanfeng Wang, and Xi Zhou.  Joint 3d face reconstruction and dense alignment with position map regression network.  In European Conference on Computer Vision, 2018.
  * [20] Liuhao Ge, Zhou Ren, Yuncheng Li, Zehao Xue, Yingying Wang, Jianfei Cai, and Junsong Yuan.  3d hand shape and pose estimation from a single rgb image.  In IEEE Conference on Computer Vision and Pattern Recognition, 2019\.
  * [21] Georgios Georgakis, Ren Li, Srikrishna Karanam, Terrence Chen, Jana Kosecka, and Ziyan Wu.  Hierarchical kinematic human mesh recovery.  In European Conference on Computer Vision, 2020.
  * [22] John C Gower.  Generalized procrustes analysis.  Psychometrika, 40(1):33–51, 1975.
  * [23] Yiwen Gu, Shreya Pandit, Elham Saraee, Timothy Nordahl, Terry Ellis, and Margrit Betke.  Home-based physical therapy with an interactive computer vision system.  In IEEE International Conference on Computer Vision Workshop, 2019\.
  * [24] Riza Alp Guler and Iasonas Kokkinos.  Holopose: Holistic 3D human reconstruction in-the-wild.  In IEEE Conference on Computer Vision and Pattern Recognition, 2019\.
  * [25] Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit.  Honnotate: A method for 3d annotation of hand and object poses.  In IEEE Conference on Computer Vision and Pattern Recognition, 2020\.
  * [26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In IEEE Conference on Computer Vision and Pattern Recognition, 2016\.
  * [27] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu.  Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014\.
  * [28] Umar Iqbal, Pavlo Molchanov, Thomas Breuel Juergen Gall, and Jan Kautz.  Hand pose estimation via latent 2.5d heatmap regression.  In European Conference on Computer Vision, 2018.
  * [29] Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh.  Panoptic studio: A massively multiview system for social motion capture.  In IEEE Conference on Computer Vision and Pattern Recognition, 2015\.
  * [30] Hanbyul Joo, Natalia Neverova, and Andrea Vedaldi.  Exemplar fine-tuning for 3d human pose fitting towards in-the-wild 3d human pose estimation.  In arXiv preprint arXiv:2004.03686, 2020.
  * [31] Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, Sean Banerjee, Timothy Godisart, Bart Nabbe, Iain Matthews, et al.  Panoptic studio: A massively multiview system for social interaction capture.  In IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
  * [32] Hanbyul Joo, Tomas Simon, and Yaser Sheikh.  Total capture: A 3d deformation model for tracking faces, hands, and bodies.  In IEEE Conference on Computer Vision and Pattern Recognition, 2018\.
  * [33] Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik.  End-to-end recovery of human shape and pose.  In IEEE Conference on Computer Vision and Pattern Recognition, 2018\.
  * [34] Angjoo Kanazawa, Jason Y Zhang, Panna Felsen, and Jitendra Malik.  Learning 3d human dynamics from video.  In IEEE Conference on Computer Vision and Pattern Recognition, 2019\.
  * [35] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In International Conference on Learning Representation, 2015.
  * [36] Muhammed Kocabas, Nikos Athanasiou, and Michael J Black.  Vibe: Video inference for human body pose and shape estimation.  In IEEE Conference on Computer Vision and Pattern Recognition, 2020\.
  * [37] Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis.  Learning to reconstruct 3D human pose and shape via model-fitting in the loop.  In IEEE International Conference on Computer Vision, 2019.
  * [38] Nikos Kolotouros, Georgios Pavlakos, and Kostas Daniilidis.  Convolutional mesh regression for single-image human shape reconstruction.  In IEEE Conference on Computer Vision and Pattern Recognition, 2019\.
  * [39] Dominik Kulon, Riza Alp Guler, Iasonas Kokkinos, Michael M Bronstein, and Stefanos Zafeiriou.  Weakly-supervised mesh-convolutional hand reconstruction in the wild.  In IEEE Conference on Computer Vision and Pattern Recognition, 2020\.
  * [40] Gilwoo Lee, Zhiwei Deng, Shugao Ma, Takaaki Shiratori, Siddhartha S Srinivasa, and Yaser Sheikh.  Talking with hands 16.2 m: A large-scale dataset of synchronized body-finger motion and audio for conversational motion analysis and synthesis.  In IEEE International Conference on Computer Vision, 2019.
  * [41] Marco Leo, Antonino Furnari, Gerard G Medioni, Mohan Trivedi, and Giovanni M Farinella.  Deep learning for assistive computer vision.  In European Conference on Computer Vision Workshop, 2018.
  * [42] Jiefeng Li, Chao Xu, Zhicun Chen, Siyuan Bian, Lixin Yang, and Cewu Lu.  Hybrik: A hybrid analytical-neural inverse kinematics solution for 3d human pose and shape estimation.  In IEEE Conference on Computer Vision and Pattern Recognition, 2021\.
  * [43] Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero.  Learning a model of facial shape and expression from 4d scans.  ACM Transactions on Graphics, 2017.
  * [44] Kevin Lin, Lijuan Wang, and Zicheng Liu.  End-to-end human pose and mesh reconstruction with transformers.  In IEEE Conference on Computer Vision and Pattern Recognition, 2021\.
  * [45] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft COCO: Common objects in context.  In European Conference on Computer Vision, 2014.
  * [46] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black.  SMPL: A skinned multi-person linear model.  ACM Transactions on Graphics, 2015.
  * [47] Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Mohamed Elgharib, Pascal Fua, Hans-Peter Seidel, Helge Rhodin, Gerard Pons-Moll, and Christian Theobalt.  Xnect: Real-time multi-person 3d human pose estimation with a single rgb camera.  ACM Transactions on Graphics, 2019.
  * [48] Dushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafiei, Hans-Peter Seidel, Weipeng Xu, Dan Casas, and Christian Theobalt.  Vnect: Real-time 3d human pose estimation with a single rgb camera.  ACM Transactions on Graphics, 2017.
  * [49] Gyeongsik Moon and Kyoung Mu Lee.  I2l-meshnet: Image-to-lixel prediction network for accurate 3d human pose and mesh estimation from a single rgb image.  In European Conference on Computer Vision, 2020.
  * [50] Araceli Morales, Gemma Piella, and Federico M. Sukno.  Survey on 3d face reconstruction from uncalibrated images, 2020.
  * [51] Evonne Ng, Shiry Ginosar, Trevor Darrell, and Hanbyul Joo.  Body2hands: Learning to infer 3d hands from conversational gesture body dynamics.  In IEEE Conference on Computer Vision and Pattern Recognition, 2021\.
  * [52] Mohamed Omran, Christoph Lassner, Gerard Pons-Moll, Peter Gehler, and Bernt Schiele.  Neural body fitting: Unifying deep learning and model based human pose and shape estimation.  In International Conference on 3D Vision, 2018.
  * [53] Juan Diego Ortega, Neslihan Kose, Paola Cañas, Min-An Chao, Alexander Unnervik, Marcos Nieto, Oihana Otaegui, and Luis Salgado.  Dmd: A large-scale multi-modal driver monitoring dataset for attention and alertness analysis.  In European Conference on Computer Vision Workshop, 2020.
  * [54] Daniil Osokin.  Real-time 2d multi-person pose estimation on cpu: Lightweight openpose.  In arXiv preprint arXiv:1811.12004, 2018.
  * [55] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  In Advances in Neural Information Processing Systems, 2019.
  * [56] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black.  Expressive body capture: 3d hands, face, and body from a single image.  In IEEE Conference on Computer Vision and Pattern Recognition, 2019\.
  * [57] Gerard Pons-Moll, Javier Romero, Naureen Mahmood, and Michael J Black.  Dyna: A model of dynamic human shape in motion.  In ACM Transactions on Graphics, 2015.
  * [58] Javier Romero, Dimitrios Tzionas, and Michael J. Black.  Embodied hands: Modeling and capturing hands and bodies together.  ACM Transactions on Graphics, 2017.
  * [59] Yu Rong, Ziwei Liu, Cheng Li, Kaidi Cao, and Chen Change Loy.  Delving deep into hybrid annotations for 3D human recovery in the wild.  In IEEE International Conference on Computer Vision, 2019.
  * [60] Yu Rong, Ziwei Liu, and Chen Change Loy.  Chasing the tail in monocular 3d human reconstruction with prototype memory.  arXiv preprint arXiv:2012.14739, 2020.
  * [61] Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and Michael J Black.  Learning to regress 3d face shape and expression from an image without 3d supervision.  In IEEE Conference on Computer Vision and Pattern Recognition, 2019\.
  * [62] Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh.  Hand keypoint detection in single images using multiview bootstrapping.  In IEEE Conference on Computer Vision and Pattern Recognition, 2017\.
  * [63] Jie Song, Xu Chen, and Otmar Hilliges.  Human body model fitting by learned gradient descent.  In European Conference on Computer Vision, 2020.
  * [64] Ayush Tewari, Michael Zollhofer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Christian Theobalt.  Mofa: Model-based deep convolutional face autoencoder for unsupervised monocular reconstruction.  In IEEE International Conference on Computer Vision Workshop, 2017\.
  * [65] Anh Tuan Tran, Tal Hassner, Iacopo Masi, and Gérard Medioni.  Regressing robust and discriminative 3d morphable models with a very deep neural network.  In IEEE Conference on Computer Vision and Pattern Recognition, 2017\.
  * [66] Hsiao-Yu Tung, Hsiao-Wei Tung, Ersin Yumer, and Katerina Fragkiadaki.  Self-supervised learning of motion capture.  In Advances in Neural Information Processing Systems, 2017.
  * [67] Timo von Marcard, Roberto Henschel, Michael Black, Bodo Rosenhahn, and Gerard Pons-Moll.  Recovering accurate 3D human pose in the wild using IMUs and a moving camera.  In European Conference on Computer Vision, 2018.
  * [68] Jingbo Wang, Sijie Yan, Bo Dai, and Dahua Lin.  Scene-aware generative network for human motion synthesis.  In IEEE Conference on Computer Vision and Pattern Recognition, 2021\.
  * [69] Jingbo Wang, Sijie Yan, Yuanjun Xiong, and Dahua Lin.  Motion guided 3d pose estimation from videos.  In European Conference on Computer Vision, 2020.
  * [70] Donglai Xiang, Hanbyul Joo, and Yaser Sheikh.  Monocular total capture: Posing face, body, and hands in the wild.  In IEEE Conference on Computer Vision and Pattern Recognition, 2019\.
  * [71] Hongyi Xu, Eduard Gabriel Bazavan, Andrei Zanfir, William T Freeman, Rahul Sukthankar, and Cristian Sminchisescu.  Ghum & ghuml: Generative 3d human shape and articulated pose models.  In IEEE Conference on Computer Vision and Pattern Recognition, 2020\.
  * [72] Yuanlu Xu, Song-Chun Zhu, and Tony Tung.  Denserac: Joint 3D pose and shape estimation by dense render-and-compare.  In IEEE International Conference on Computer Vision, 2019.
  * [73] Wang Zeng, Wanli Ouyang, Ping Luo, Wentao Liu, and Xiaogang Wang.  3D human mesh regression with dense correspondence.  In IEEE Conference on Computer Vision and Pattern Recognition, 2020\.
  * [74] Jiawei Zhang, Jianbo Jiao, Mingliang Chen, Liangqiong Qu, Xiaobin Xu, and Qingxiong Yang.  3d hand pose tracking and estimation using stereo matching.  In IEEE International Conference on Image Processing, 2017.
  * [75] Xiong Zhang, Qiang Li, Hong Mo, Wenbo Zhang, and Wen Zheng.  End-to-end hand mesh recovery from a monocular rgb image.  In IEEE International Conference on Computer Vision, 2019.
  * [76] Yuxiao Zhou, Marc Habermann, Ikhsanul Habibie, Ayush Tewari, Christian Theobalt, and Feng Xu.  Monocular real-time full body capture with inter-part correlations.  In IEEE Conference on Computer Vision and Pattern Recognition, 2021\.
  * [77] Yuxiao Zhou, Marc Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, and Feng Xu.  Monocular real-time hand shape and motion capture using multi-modal data.  In IEEE Conference on Computer Vision and Pattern Recognition, 2020\.
  * [78] Christian Zimmermann and Thomas Brox.  Learning to estimate 3d hand pose from single rgb images.  In IEEE International Conference on Computer Vision, 2017.
  * [79] Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox.  Freihand: A dataset for markerless capture of hand pose and shape from single rgb images.  In IEEE International Conference on Computer Vision, 2019.
  * [80] Michael Zollhöfer, Justus Thies, Pablo Garrido, Derek Bradley, Thabo Beeler, Patrick Pérez, Marc Stamminger, Matthias Nießner, and Christian Theobalt.  State of the art on monocular 3d face reconstruction, tracking, and applications.  In Computer Graphics Forum, 2018.