# Learning Implicit Body Representations from Double Diffusion Based
Neural Radiance Fields

Guangming Yao1111Contact Author, Hongzhi Wu2, Yi Yuan1, Lincheng Li1, Kun
Zhou2, Xin Yu3, 1NetEase Fuxi AI Lab 2Zhejiang University 3University of
Technology Sydney
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In this paper, we present a novel double diffusion based neural radiance
field, dubbed DD-NeRF, to reconstruct human body geometry and render the human
body appearance in novel views from a sparse set of images. We first propose a
double diffusion mechanism to achieve expressive representations of input
images by fully exploiting human body priors and image appearance details at
two levels. At the coarse level, we first model the coarse human body poses
and shapes via an unclothed 3D deformable vertex model as guidance. At the
fine level, we present a multi-view sampling network to capture subtle
geometric deformations and image detailed appearances, such as clothing and
hair, from multiple input views. Considering the sparsity of the two level
features, we diffuse them into feature volumes in the canonical space to
construct neural radiance fields. Then, we present a signed distance function
(SDF) regression network to construct body surfaces from the diffused
features. Thanks to our double diffused representations, our method can even
synthesize novel views of unseen subjects. Experiments on various datasets
demonstrate that our approach outperforms the state-of-the-art in both
geometric reconstruction and novel view synthesis.

##  1 Introduction

Simultaneous reconstruction of body geometry and appearances from a sparse set
of views is highly challenging yet important in a wide variety of
applications, including special effects, game production and virtual reality.
Substantial research efforts have been made in the past, particularly with the
help of powerful deep learning techniques. When high-quality 3D body scans are
available for network training, deep networks can reconstruct human body
geometry and texture easily. However, such data are often expensive to acquire
and not publicly accessible.

When 3D supervision is not available, self-supervised neural radiance fields
(NeRF) have been proposed to render novel views of a specific subject.
However, this per-subject optimization process of NeRF is often time-
consuming. Recent works Yu et al. (2021); Chen et al. (2021); Wang et al.
(2021b) propose to first extract features of scene images and then utilize a
scene-shared NeRF to synthesize novel views from the scene features, thus
circumventing the inefficient per-subject optimization. However, these methods
would suffer artifacts when they are directly applied to human body
reconstruction or rendering due to the non-rigid deformations and large
variations of human bodies. As suggested in Peng et al. (2021), the
introduction of human body priors will significantly alleviate geometric
distortions. Unfortunately, the work Peng et al. (2021) is a per-subject
optimization method, thus restricting its applications in rendering various
subjects.

Figure 1:  Our method produces detailed body geometry and synthesizes high-
fidelity novel views, from a sparse set of person images.

To overcome the aforementioned shortcomings, we present a novel double
diffusion based neural radiance field, namely DD-NeRF. As illustrated in Fig.
1, DD-NeRF is developed to reconstruct human body geometry and render human
body appearances in novel views from a sparse set of human body images. To be
specific, we propose a novel double diffusion mechanism that can fully exploit
human body priors as well as image appearance details at two levels to achieve
representative features of input images.

At the coarse level, we first model the coarse human body poses and shapes via
an unclothed deformable human mesh as explicit guidance. We embed the deformed
vertices as a coarse-level feature, which indicates the coarse-scale human
body pose and shape. At the fine level, we develop a multi-view sampling
network to capture image appearance details and subtle geometric deformations,
such as clothing and hair, which are not modeled at the coarse level. Our
multi-view sampling network firstly samples 3D points on the deformable human
model as anchors and then projects them to each input view. Then, it
aggregates image features at the project positions of different views by
measuring their means and variances as the fine level features. In this
fashion, the acquired coarse level and fine level feature representations are
both sparse in the canonical space. Afterwards, we diffuse them into 3D
feature volumes via sparse convolutions to facilitate the decoding process.

Subsequently, we design an implicit field regression module to reconstruct
high-quality human bodies from the two level diffused features. Here, we
employ a signed distance function (SDF) to represent body geometry instead of
volume-density based radiance fields since SDF is accurate in modeling surface
geometry while volume density is more suitable to render texture Wang et al.
(2021a). Thus, we adopt multi-layer perceptrons (MLPs) to regress signed
distance from the tri-linearly interpolated coarse and fine level feature
volumes. In order to avoid the per-subject optimization as in Peng et al.
(2021), our radiance regression network is conditioned on image features and
pixels of input views in reasoning. Additionally, we employ a transformer to
fuse the multi-view image features and raw pixels into a joint representation
for radiance regression. With regressed signed distance and radiance, the
reconstructed meshes and novel views can be obtained from differential SDF
rendering Wang et al. (2021a). Extensive experiments demonstrate that once DD-
NeRF is trained, it can reconstruct human bodies of both seen and unseen
subjects and achieves superior performance compared to the state-of-the-art.

In summary, the contributions of this work are three-fold:

  * •

We present a novel double diffusion based neural radiance field, dubbed DD-
NeRF, which can reconstruct human body geometry from a sparse set of images in
a feed-forward fashion and is subject-agnostic.

  * •

We propose a double diffusion mechanism that fully exploits human body priors
and captures image appearance details in both coarse and fine levels, enabling
accurate and robust human body surface reconstruction.

  * •

We develop a signed distance function (SDF) regression network to reconstruct
human body surface in a differentiable manner, allowing the body surface to be
optimized with only 2d supervision.

##  2 Related Work

###  2.1 Body Shape Reconstruction

Recently, regressing a “freeform” 3D body shape with implicit representations
has achieved promising progress. Pixel-aligned implicit function (PIFu)
regresses a signed distance function for any given 3D location Saito et al.
(2019, 2020). It can infer both 3D body surface and texture from a single
image. Several works take multi-view images as input to pursue better
reconstruction performance. For instance, volumetric occupancy fields have
been proposed to learn dynamic clothed bodies from sparse viewpoints Gilbert
et al. (2018). The work Shao et al. (2021) combines a surface field and a
radiance field for body representations, and it is learned under the
supervision of ground-truth meshes. Note that all these methods require
ground-truth 3D scanned models as supervision. In contrast, our method does
not rely on 3D scanned models and only leverages a sparse set of multi-view
images to reconstruct high-quality 3D body geometry in a self-supervised
manner.

Figure 2:  An overview of _DD-NeRF_. Given multi-view input images and a
corresponding SMPL model, the double diffusion module constructs two feature
volumes to represent a coarse body prior and local features that depict local
geometric deformations. Next, an implicit field regression module predicts a
signed distance function (SDF) and scene radiance for each ray. All the
modules are learned jointly by minimizing the differences between the target
and rendered pixels.

###  2.2 Neural Representations

Neural implicit functions have emerged as an effective representation to learn
3D scenes from 2D images. NeRF Mildenhall et al. (2020) learns densities and
colors of a scene with volumetric rendering, and achieves impressive results
on novel view synthesis. Following NeRF, some methods Peng et al. (2021); Kwon
et al. (2021) capture a human body geometry from videos and reconstruct its
surface by performing marching cube Lorensen and Cline (1987) on learned
volumetric densities. Furthermore, several works Niemeyer et al. (2020); Yariv
et al. (2020); Kellnhofer et al. (2021); Wang et al. (2021a) represent a scene
with an SDF and thus extract surfaces by level set. The work Peng et al.
(2021) employs a skinned multi-person linear model (SMPL) Loper et al. (2015)
as a human body prior to preserve geometric details of human bodies, but it
needs to optimize the network according to each subject. Recent works Chen et
al. (2021); Wang et al. (2021b) propose conditional NeRF to bypass the tedious
per-subject optimization. They are trained on multiple subjects and can
perform novel view synthesis for unseen subjects which are not available
during training. Our method also learns across different subjects in order to
reconstruct unseen subjects in a feed-forward fashion. Moreover, we introduce
a human body model as geometric guidance, thus significantly facilitating
high-fidelity body geometry reconstruction and improving optimization
efficiency.

##  3 Proposed Method

Given a sparse set of multi-view posed body images, our method computes an
implicit radiance field that represents the geometry and appearance of the
subject. We denote the input images as $I=\\{I_{1},I_{2},...,I_{N_{c}}\\}$,
where $N_{c}$ is the number of pre-calibrated cameras. The calibration results
are stored as $\Phi=\\{\Phi_{1},\Phi_{2},...,\Phi_{N_{c}}\\}$. In general, our
network can be viewed as a conditional radiance field function as follows:

$\displaystyle s,c=F(x,d;I,\Phi),$ (1)

where $x$ represents a 3D location, $d$ is a viewing direction, $s$ denotes a
signed distance value at $x$ and $c$ is an RGB color as its appearance. The
output radiance field can be used to synthesize a novel view via
differentiable ray marching, or extract 3D surfaces with marching cube.

An overview of our model is illustrated in Fig. 2. There are three components:
an image encoder module, a double diffusion module, and an implicit field
regression module. The image encoder module is composed of a stacked hourglass
neural network Newell et al. (2016) for image feature extraction. Next, the
double diffusion module builds two volumes, a coarse body feature volume and a
fine detail feature volume. While the former is designed to encode the coarse
body prior, the latter is developed to depict the detailed geometric
deformations for high-fidelity body representation. The implicit field
regression module consists of two sub-modules that regress signed distance
values and colors, respectively. Specifically, an SDF regression sub-module
generates signed distance from diffused features of both volumes, and a
radiance regression sub-module takes as input surface normal, multi-view raw
pixels, image features and the view direction, and produces an RGB color as
output. Once we obtain the regressed signed distance and colors, a novel view
image of the subject can be synthesized by differential SDF rendering.

###  3.1 Double Diffusion

The motivation of double diffuse is to attain expressive representations by
exploiting both coarse human body priors and detailed multi-view image
features. We build a coarse body volume and a fine-level detail feature volume
in parallel. We use the resolution of $256^{3}$ for the coarse body volume,
and $512^{3}$ for the detail feature volume in order to capture high-frequency
geometric details. Inspired by Peng et al. (2021), we adopt an SMPL model as
the body prior and the coarse body volume encodes the body shapes and poses.
Given the multi-view images, the corresponding SMPL parameters are estimated
by Zheng and Shao (2021). We obtain the vertices
$V=\\{v_{1},v_{2},...,v_{6,890},v_{i}\in R^{3}\\}$ from the template model.
The vertices are fed into the embedding network to compute a structured latent
code $Z\in\mathbb{R}^{6,890\times 16}$, which serves as our coarse body prior
for implicit filed reconstruction.

The detail feature volume is designed to capture fine-grained geometric
characteristics of a subject by integrating multi-view image features.
Inspired by Yu et al. (2018), we sample the on-body and far-body anchors to
effectively capture the body-related features. The on-body anchors are
relatively dense vertices computed with subdivisions on the SMPL model. The
far-body anchors are randomly sampled on the exterior surfaces of spheres
centered at each vertex of the SMPL model. We empirically set the radius as
0.05m to account for objects that are relatively far away from the SMPL
surface (e.g., loose clothing, long hair). We uniformly sample 12,446 far-body
anchors, and utilize Trimesh Dawson-Haggerty et al. (2019) to obtain 27,554
on-body anchors from 6,890 SMPL vertices with subdivision. Next, we project
the anchors to different views and sample pixel-aligned multi-view features,
expressed as:

$\displaystyle\hat{a}_{i}^{j}$ $\displaystyle=K_{j}\cdot[R_{j}^{T}\cdot
R\cdot(a_{i}-T)+T_{j}],$ (2) $\displaystyle f_{i}^{j}$
$\displaystyle=E_{j}\\{\hat{a}_{i}^{j}\\},$

where $a_{i}$ is the 3D position of the $i$-th anchor, $R$ and $T$ are the
estimated rotation and translation of the SMPL model. $K_{j}$ and
$[R_{j},T_{j}]$ are the intrinsic and extrinsic matrices of $j$-th camera
respectively. $E_{j}\\{\cdot\\}$ denotes the image feature extracted from the
$j$-th view, $f_{i}^{j}\in\mathbb{R}^{c}$ denotes the sampled features of the
$i$-th anchor from the $j$-th view. Given the $N_{c}$ input views, we attain
features $f_{i}=\\{f_{1},f_{2},...,f_{N_{c}}\\}$ for each anchor and then
concatenate their mean and variance to achieve a feature vector
$\hat{f}_{i}\in\mathbb{R}^{2c}$ as the final representation of the anchor.
Then, those feature vectors are utilized to capture subtle geometric
deformations and image detailed appearances.

Since the sparse coarse-and-fine features are not suitable for dense surface
reconstruction, we employ SparseConvNet Graham and Engelcke (2018) to diffuse
them into the canonical space. Our double-diffusion module incorporates the
coarse body prior with elaborated geometric information while preserving the
local appearance and geometric details depicted in input images.

###  3.2 Implicit Field Regression

This step consists of two sub-modules: SDF regression $F_{S}$ and radiance
regression $F_{R}$. To obtain a high-quality output surface, we use signed
distance instead of volumetric density to represent the body geometry. For a
3D point $x$ that a ray traverses in the 3D bounding box $\Omega^{3}$, we
trilinearly interpolate features from the coarse and fine level feature
volumes to obtain its corresponding coarse feature $f_{c}$ and detail feature
$f_{d}$. Then, we adopt an MLPs to regress signed distance $s$ based on $x$,
$f_{c}$ and $f_{d}$ as follows:

$\displaystyle s,f_{s}=F_{S}(x,f_{c},f_{d}),$ (3)

where $f_{s}$ is a surface feature corresponding to $s$. Similar to NeRF
Mildenhall et al. (2020), $x$ is mapped to a higher dimensional space with
positional encoding. After computing $s$ for the feature volumes, the body
surface $\mathcal{S}$ can be expressed by the zero level-set of SDF, i.e.,
$\mathcal{S}=\\{x\in\Omega^{3}|s=0\\}$.

Once the geometry is obtained, the next step is to regress the corresponding
appearance. Given the geometry-related feature $f_{s}$, we take the view
direction, the geometry and the image feature into consideration. The radiance
regression is expressed by:

$\displaystyle
c=F_{R}(f_{s},g,x,d,\\{f_{xi}\\},\\{p_{xi}\\}|i=1,2,...,N_{c}),$ (4)

where $d$ is the view-direction, and $g=\nabla F_{S}(x,f_{c},f_{d})$
represents the normal of surface $\mathcal{S}$ at position $x$. We project $x$
to the $i$-th view, and then sample pixel-aligned image feature $f_{xi}$ from
$E_{i}$ and the raw RGB pixel $p_{xi}$. Notably, $p_{xi}$ and $d$ are also
mapped to a higher dimensional space with positional encoding for learning
high-frequency variations.

We adopt a transformer Vaswani and Shazeer (2017) to effectively fuse features
$\\{f_{xi}\\},\\{p_{xi}\\}|i=1,2,...,N_{c}\\}$ for appearance reasoning. The
transformer consists of an encoder and a decoder. The encoder $\mathcal{E}$
encodes the multi-view features with stacked multi-head attention layers to
obtain a fused feature $\hat{f}_{x}$, computed as:

$\displaystyle\hat{f}_{x}=\mathcal{E}(\\{f_{xi}\\},\\{p_{xi}\\}|i=1,2,...,N_{c}).$
(5)

The MLP-based decoder $\mathcal{D}$ regresses the final radiance:

$\displaystyle c=\mathcal{D}(f_{s},g,x,d,\hat{f}_{x}).$ (6)

###  3.3 Differential SDF Rendering

To optimize our representation, we adopt the SDF-based differential renderer
Wang et al. (2021a). Then, we can use our input images as ground-truth to
supervise novel view synthesis and reconstruction. For a pixel in the target
image, We accumulate the radiance along the ray emiited from the pixel to to
predict the pixel color by:

$\displaystyle\hat{C}=\sum_{i=1}^{n}T_{i}\alpha_{i}c_{i},$ (7)

where $T_{i}=\prod_{j=1}^{i-1}(1-\alpha_{j})$ is the discrete accumulated
transmittance, and $\alpha_{i}$ is a discrete opacity value, defined as:

$\displaystyle\alpha_{i}$
$\displaystyle=\max(\frac{\Phi_{s}(f(s_{i})-\Phi_{s}(f(s_{i+1})))}{\Phi_{s}(f(s_{i}))},0),$
(8) $\displaystyle\Phi_{s}$ $\displaystyle=(1+e^{-kx})^{-1},$

where $k$ is a learnable scalar that increases as the training iteration
progresses.

Table 1: Quantitative comparisons of novel view synthesis. “Ft” indicates that
fine-tuning is applied. Bold and underlined numbers correspond to the best and
the second-best values for each metric.

Model | Supervision | PNSR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PNSR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PNSR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---|---|---|---|---|---|---|---|---
|  | THuman2.0 | Twindom | ZJU-Mocap
MVSNeRF | images | 18.99 | 0.72 | 0.35 | 16.43 | 0.66 | 0.30 | 19.50 | 0.76 | 0.41
MVSNeRF(Ft) | images | 21.09 | 0.80 | 0.32 | 19.90 | 0.72 | 0.27 | 22.10 | 0.88 | 0.32
PIFu | 3D meshes | 18.39 | 0.58 | 0.33 | 16.43 | 0.70 | 0.30 | 19.12 | 0.75 | 0.43
NeuralBody | images | 21.90 | 0.85 | 0.20 | 23.52 | 0.76 | 0.26 | 25.65 | 0.92 | 0.27
Ours | images | 21.06 | 0.75 | 0.21 | 23.40 | 0.79 | 0.26 | 23.95 | 0.79 | 0.25
Ours(Ft) | images | 23.08 | 0.85 | 0.17 | 27.67 | 0.81 | 0.23 | 25.32 | 0.89 | 0.23

Table 2: Quantitative comparisons of surface reconstruction.

Model | Chamfer$\downarrow$ | P2S$\downarrow$ | Chamfer$\downarrow$ | P2S$\downarrow$
---|---|---|---|---
| THuman2.0 | Twindom
MVSNeRF | 1.597 | 1.146 | 1.528 | 1.126
MVSNeRF(Ft) | 0.930 | 0.913 | 0.925 | 0.741
PIFu | 1.510 | 1.524 | 1.170 | 1.630
NeuralBody | 0.915 | 0.931 | 0.815 | 0.725
Ours | 0.810 | 0.739 | 0.803 | 0.714
Ours(Ft) | 0.740 | 0.689 | 0.696 | 0.647

###  3.4 Loss Function

To learn the implicit body representations, we penalize the differences
between the rendered pixel colors and their counterparts in the input image.
We randomly select $n$ input views, and randomly sample $m$ pixels from each
view to train our network. The rendering loss $L_{r}$ measures the pixel-wise
L1 distance between the rendered colors and the ground-truth:

$\displaystyle L_{r}=\frac{1}{m}\sum_{i=1}^{m}|C_{i}-\hat{C}_{i}|,$ (9)

where $\hat{C}_{i}$ is the predicted pixel color and $C_{i}$ is the ground-
truth. Besides, the Eikonal loss $L_{e}$ serves as an implicit geometric
regularization Gropp and Yariv (2020), and enforces our SDF regression sub-
module to model signed distance:

$\displaystyle L_{e}=\frac{1}{nm}\sum_{i,j}(|\nabla(s_{ij})|-1)^{2}.$ (10)

Finally, the overall loss is defined as:

$\displaystyle L=\lambda_{r}L_{r}+\lambda_{e}L_{e},$ (11)

where $\lambda_{r}$ and $\lambda_{e}$ denote the corresponding weights.

##  4 Experiments

Figure 3: Comparisons of geometry reconstruction results by different methods.
Please zoom in for more details.

Figure 4: Comparisons of novel view synthesis. Our method produces the most
faithful results. Please zoom in for more details. Figure 5: Comparisons of
synthesized novel views and reconstructed surfaces of real-world images.

###  4.1 Datasets

We perform experiments on the synthesized datasets Twindom 222web.twindom.com,
THuman2.0 Zheng and Shao (2021) and the real-world dataset ZJU-Mocap Peng et
al. (2021). We use 1,200 body meshes from Twindom for training, 300 meshes for
evaluation. For THuman, 400 meshes are used for training, and 100 meshes for
evaluation. All the data in ZJU-Mocap are used for evaluation. To render a
scanned body mesh, we place it in the center of a unit sphere, and orient the
camera towards the sphere center with a distance of 2.4m. We move the camera
around the sphere, sample a yaw angle from $-30^{\circ}$ to $60^{\circ}$ with
an interval of $10^{\circ}$, and sample a roll angle from $0^{\circ}$ to
$360^{\circ}$ with an interval of $24^{\circ}$. For each body model, we render
135 images of resolution $1024^{2}$ for training.

###  4.2 Implementation Details

The SMPL parameters are estimated by EasyMocap Dong and Fang (2021) from input
images. Note that sampling strategies can heavily influence the final results
as reported by prior works Wang et al. (2021a). Therefore, we adopt the
hierarchical sampling method as in Wang et al. (2021a). We use the Adam
optimizer Kingma and Ba (2014), and set the learning rate to $1\times
10^{-4}$. The loss weights are set to 10, 1 for $\lambda_{r}$ and
$\lambda_{e}$, respectively.

###  4.3 Qualitative Comparisons

We compare our approach with state-of-the-art methods to validate the
superiority of our method. MVSNeRF Chen et al. (2021), PIFu Saito et al.
(2019) and our method are all generalizable. In other words, models are
trained on several subjects and tested on randomly sampled subjects that are
unseen during training. NeuralBody Peng et al. (2021) is a subject-specific
optimization based method. Following the training protocols of NeuralBody, we
train it with 500 epochs for each testing subject. It is worth mentioning that
we use the public-released weights of PIFu for comparisons. PIFu is trained
under the supervision of scanned 3D meshes and takes a single image as input.
For the remaining methods, we use four views (i.e., the left, front, right and
back views) as the input in testing.

We conduct comparison experiments on both the reconstructed geometry (Fig. 3)
and rendered appearances (Fig. 4). Please refer to the supplementary video for
animated results with varying views. The results show that PIFu suffers over-
smoothing artifacts and distortions in its reconstructed geometry and texture.
Benefiting from our double diffusion mechanism, our method effectively learns
both the coarse body prior and image details and thus produces the most
authentic geometry and appearances. MVSNeRF and NeuralBody generate plausible
novel views. However, some bumpy artifacts appear in their generated shape
results as seen in Fig. 3. Since we employ SDF to represent the geometry, our
results do not exhibit such artifacts. We also compare our method with
NeuralBody on real-world images. Note that we only need to spend 30 minutes to
finetune our method on real-world images, while NeuralBody requires 4 hours to
learn from scratch for each subject on an Nvidia RTX3090 GPU. As illustrated
in Fig. 5, our method produces better surface details and comparable novel
views. For more visual results, please refer to the supplementary material.

###  4.4 Quantitative Comparisons

We adopt a variety of metrics to quantitatively evaluate our results from the
perspectives of the image and geometry quality. Specifically, we employ peak
signal-to-noise ratio (PSNR), structural similarity index (SSIM) and learned
perceptual image patch similarity (LPIPS) Zhang et al. (2018) to evaluate the
similarity between the ground-truth and the synthesized image. The metrics of
Chamfer distance (Chamfer) and average point-to-surface Euclidean distance
(P2S) are applied to geometric quality assessment. For the image quality
evaluation, we randomly select 20 target views from the 135 rendered images as
the ground-truth. In the geometry quality assessment, marching cube is
performed for surface extraction. To further inspect the capability of
generalization, we fine-tune MVSNeRF and our method for 100 epochs.

The quantitative comparisons in terms of image and geometry quality are
reported in Tab. 1 and 2, respectively. Our method outperforms the other
competing methods on most metrics. In particular, our method achieves the
highest scores at Charmfer and P2S, indicating the effectiveness of our
proposed implicit body representations. In other words, our method attains the
most accurate geometric models. Though DD-NeRF works in a feed-forward way
during testing, we can fine-tune it, similar to the subject-specific
optimization based method NeuralBody. After fine-tuning, our method achieves
the highest scores in novel view synthesis. Moreover, all the metrics are also
improved. This indicates that DD-NeRF can not only be extended to subject-
specific novel view synthesis but also capture more expressive features.

###  4.5 Ablation Study

Figure 6: Qualitative ablation study on synthesized novel views. Our full
model leads to the best result compared to other variants. Table 3: Ablation
study on the Twindom dataset.

Model | Chamfer$\downarrow$ | P2S$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$
---|---|---|---|---
w/o$f_{c}$ | 0.843 | 0.91 | 17.03 | 0.70
w/o DD | 0.926 | 1.08 | 14.54 | 0.49
w/o transformer | 0.835 | 0.74 | 19.95 | 0.73
full model | 0.803 | 0.71 | 23.40 | 0.79

We conduct ablation studies on Twindom dataset to evaluate the contributions
of each component in DD-NeRF. We first remove the coarse-level feature $f_{c}$
(w/o$f_{c}$) to evaluate the effect of body prior. Note that the fine-level
feature $f_{d}$ cannot be removed since it provides all the clothes and hair
information. Then, we remove the double diffusion mechanism (w/o DD). The
ablated model is trained from scratch in a subject-specific way. To inspect
the effect of the transformer, we remove the transformer from the full model
(w/o transformer), and thus multi-view features are stacked as the input of
our radiance decoder.

The results are reported in Tab. 3. Our full model achieves the best scores
under all the metrics. Without the coarse-level feature, the performance of
w/o$f_{c}$ drops significantly, revealing the importance of body prior.
Without the double diffusion mechanism, w/o DD is hard to produce realistic
novel views when the input is a sparse set of views. This manifests that our
diffusion mechanism can effectively learn the body representation for the
implicit field regression. Besides, most metrics are negatively changed with
the absence of the transformer. This indicates that the transformer can
effectively integrate multi-view information and produce more satisfactory
results. We observed that when the SDF regression submodule is removed and the
diffused features are fed to the radiance field decoder, the network cannot
converge. The visual comparisons of the above variations are shown in Fig. 6,
and more results are provided in the supplementary material.

##  5 Conclusion

We propose a novel double diffusion based neural radiance field to represent
the human body implicitly from a sparse set of multi-view images. The proposed
double diffusion mechanism fully exploits human body priors and image
appearance details at both coarse and fine levels, thus leading to expressive
representations of subject geometry and appearances. Benefiting from the
diffused features, our proposed implicit field regression module can
reconstruct high-quality human bodies. Thanks to our self-supervised training
fashion, our method does not require any 3D supervision in training and is
able to reconstruct human bodies of unseen subjects. Extensive experiments
show that our method considerably outperforms concurrent works.

## References

  * Chen et al. [2021] Anpei Chen, Zexiang Xu, and Hao Su.  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.  arXiv preprint arXiv:2103.15595, 2021.
  * Dawson-Haggerty et al. [2019] Dawson-Haggerty et al.  trimesh, 2019.
  * Dong and Fang [2021] Junting Dong and Qi Fang.  Fast and robust multi-person 3d pose estimation and tracking from multiple views.  In T-PAMI, 2021.
  * Gilbert et al. [2018] Andrew Gilbert, Marco Volino, John Collomosse, and Adrian Hilton.  Volumetric performance capture from minimal camera viewpoints.  In ECCV, pages 566–581, 2018.
  * Graham and Engelcke [2018] Benjamin Graham and Engelcke.  3d semantic segmentation with submanifold sparse convolutional networks.  In CVPR, pages 9224–9232, 2018.
  * Gropp and Yariv [2020] Amos Gropp and Lior Yariv.  Implicit geometric regularization for learning shapes.  arXiv preprint arXiv:2002.10099, 2020.
  * Kellnhofer et al. [2021] Petr Kellnhofer, Lars C Jebe, and Gordon Wetzstein.  Neural lumigraph rendering.  In CVPR, pages 4287–4297, 2021.
  * Kingma and Ba [2014] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization, 2014.
  * Kwon et al. [2021] Youngjoong Kwon, Dahun Kim, Duygu Ceylan, and Henry Fuchs.  Neural human performer: Learning generalizable radiance fields for human performance rendering, 2021.
  * Loper et al. [2015] Matthew Loper, Naureen Mahmood, and Javier Romero.  Smpl: A skinned multi-person linear model.  ACM transactions on graphics (TOG), 34(6):1–16, 2015.
  * Lorensen and Cline [1987] William E Lorensen and Harvey E Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  ACM siggraph, 21(4):163–169, 1987.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P Srinivasan, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV, pages 405–421. Springer, 2020.
  * Newell et al. [2016] Alejandro Newell, Kaiyu Yang, and Jia Deng.  Stacked hourglass networks for human pose estimation.  In ECCV, pages 483–499. Springer, 2016.
  * Niemeyer et al. [2020] Michael Niemeyer, Lars Mescheder, and Andreas Geiger.  Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision.  In CVPR, pages 3504–3515, 2020.
  * Peng et al. [2021] Sida Peng, Yuanqing Zhang, Hujun Bao, and Xiaowei Zhou.  Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans.  In CVPR, pages 9054–9063, 2021.
  * Saito et al. [2019] Shunsuke Saito, Zeng Huang, and Hao Li.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization.  In CVPR, pages 2304–2314, 2019.
  * Saito et al. [2020] Shunsuke Saito, Tomas Simon, and Hanbyul Joo.  Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization.  In CVPR, pages 84–93, 2020.
  * Shao et al. [2021] Ruizhi Shao, Hongwen Zhang, Tao Yu, and Yebin Liu.  Doublefield: Bridging the neural surface and radiance fields for high-fidelity human rendering.  arXiv preprint arXiv:2106.03798, 2021.
  * Vaswani and Shazeer [2017] Ashish Vaswani and Noam Shazeer.  Attention is all you need.  In NeurIPS, pages 5998–6008, 2017.
  * Wang et al. [2021a] Peng Wang, Lingjie Liu, Yuan Liu, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  arXiv preprint arXiv:2106.10689, 2021.
  * Wang et al. [2021b] Qianqian Wang, Zhicheng Wang, and Kyle Genova.  Ibrnet: Learning multi-view image-based rendering.  In CVPR, 2021.
  * Yariv et al. [2020] Lior Yariv, Yoni Kasten, and Dror Moran.  Multiview neural surface reconstruction by disentangling geometry and appearance.  arXiv preprint arXiv:2003.09852, 2020.
  * Yu et al. [2018] Tao Yu, Zerong Zheng, and Yebin Liu.  Doublefusion: Real-time capture of human performances with inner body shapes from a single depth sensor.  In CVPR, pages 7287–7296, 2018.
  * Yu et al. [2021] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In CVPR, pages 4578–4587, 2021.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, and Alexei Efros.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.
  * Zheng and Shao [2021] Yang Zheng and Ruizhi Shao.  Deepmulticap: Performance capture of multiple characters using sparse multiview cameras, 2021.


# Spatial-Temporal Parallel Transformer for Arm-Hand Dynamic Estimation

Shuying Liu111These authors contributed equally to this work., Wenbin
Wu111These authors contributed equally to this work., Jiaxian Wu, Yue Lin
NetEase Games AI Lab, Guangzhou, China
{liushuying, wuwenbin02, wujiaxian<EMAIL_ADDRESS>
###### Abstract

We propose an approach to estimate arm and hand dynamics from monocular video
by utilizing the relationship between arm and hand. Although monocular full
human motion capture technologies have made great progress in recent years,
recovering accurate and plausible arm twists and hand gestures from in-the-
wild videos still remains a challenge. To solve this problem, our solution is
proposed based on the fact that arm poses and hand gestures are highly
correlated in most real situations. To fully exploit arm-hand correlation as
well as inter-frame information, we carefully design a Spatial-Temporal
Parallel Arm-Hand Motion Transformer (PAHMT) to predict the arm and hand
dynamics simultaneously. We also introduce new losses to encourage the
estimations to be smooth and accurate. Besides, we collect a motion capture
dataset including 200K frames of hand gestures and use this data to train our
model. By integrating a 2D hand pose estimation model and a 3D human pose
estimation model, the proposed method can produce plausible arm and hand
dynamics from monocular video. Extensive evaluations demonstrate that the
proposed method has advantages over previous state-of-the-art approaches and
shows robustness under various challenging scenarios.

Figure 1: We propose Spatial-Temporal Parallel Transformer to estimate arm and
hand dynamics from monocular video by utilizing the arm-hand correlation as
well as the temporal information. Row 1: Input video frames. Row 2: Estimated
arm and hand dynamics of our method. The proposed method shows robustness
under various challenging scenarios.

##  1 Introduction

Human arm-hand dynamics is an important part of full human motion capture, and
can also be used for the control of human-machine interface. However, although
some methods are proposed to capture full human motion including hand
gestures[30, 33, 40, 45, 47], most of them fail to take into account the
correlation between arm and hand, treating body motion capture and gesture
estimation as two separate tasks, resulting in inaccurate predictions in
challenging scenarios. Zhou et al.[47] propose to learn human motion by
considering body-hand correlation, but such correlation is only utilized in
predicting 2D hand key-points and 3D body key-points, the final motion of body
and hands are learned separately by different models. Ng et al.[29] introduce
to learn the body-hand correlation to estimate conversational hand gestures.
However, the method is restricted to the domain of conversational gesture
prediction and cannot produce arm motions.

In this paper, we focus on the task of capturing accurate and plausible arm-
hand dynamics from monocular video. Specifically, we introduce a spatial-
temporal Parallel Arm-Hand Motion Transformer (PAHMT) to take full advantage
of arm-hand correlations and inter-frame information. The estimation of 2D
hand key-points and 3D arm key-points are first achieved by a light weight
hand key-point detector and a 3D human key-point estimator respectively, which
are used as input in our model. The PAHMT mainly consists of a spatial
transformer and a temporal transformer. The spatial transformer is responsible
for extracting the spatial feature, namely global correlations between arm and
hand as well as local correlations between different joints, while the
temporal transformer is designed to utilize inter-frame information. Besides,
we introduce two losses to encourage the predictions to be smooth and
accurate. To train the proposed model, we collect a dataset of 200K frames of
human motion (including hand gestures). Most of the collected sequences are
dancing or sport motions that cover a lot of arm motions and hand gestures. We
demonstrate that the proposed model can produce plausible estimation of arm-
hand dynamics even in difficult scenarios such as occlusion or motion blur.

Our contributions can be summarized as follows:

  * •

We propose to capture arm and hand dynamics simultaneously by leveraging the
arm-hand correlations. By exploiting such correlations, the proposed model can
make reasonable estimations of arm twists and hand gestures;

  * •

We design a spatial-temporal parallel transformer model to make full use of
arm-hand correlation as well as inter-frame information, which enhances the
robustness of the prediction; In addition, we introduce two losses to
encourage smooth and accurate predictions;

  * •

Extensive evaluations demonstrate that the proposed method outperforms
existing state-of-the-art approaches and shows robustness under various
challenging scenarios.

##  2 Related Work

Great progress has been made on designing models to capture human pose or
dynamic motions from visual observations[11, 31, 34, 24, 44, 46, 7, 6, 41].
Previous work always divide the task of full human motion capture into three
independent sub-tasks, i.e., facial expression capture, body motion capture
and hand gesture capture. Since our work focuses on capturing the dynamics of
hands and arms (part of the body), we only present the related work as
follows.

###  2.1 Hand Pose Estimation

2D Hand Pose Estimation. Some preceding work put much effort on building
powerful datasets. Simon et al.[35] propose a hand key-point dataset with
geometrically consistent annotations, i.e., annotations are provided even for
occluded hand parts. With this dataset, a hand key-point detector is trained
that can rival the performance of state-of-the-art hand key-point detectors
using RGB-D inputs. To obtain more data for training, some researchers propose
to first synthesize 3D hand data and then transfer them to real image style
using GAN[28, 37]. Some work has been explored in network architecture design.
Wang et al.[39] introduce a novel architecture called SRHandNet for real-time
2D hand pose estimation from monocular RGB images, which can run at 40FPS
while achieving competing results. Zimmermann et al.[49] propose an
architecture of four different network streams to make a full estimation of
hand joints.

3D Hand Pose Estimation. The majority of state-of-the-art approaches predict
3D hand poses by recovering the hand meshes or regressing the coordinate of
hands from single RGB image. Zhou et al.[48] use the estimated kinematic chain
to estimate MANO[32] coefficients and develop an inverse kinematics network to
refine the predicted hand poses. Chen et al.[5] propose a camera-space mesh
recovery framework to recover 3D hand mesh in camera-centered space.
Kolotouros et al.[20] directly regress the 3D coordinates of mesh vertices by
training a graph convolutional network. On the other hand, some researchers
have studied how to achieve hand pose estimation based on monocular video.
Chen et al.[4] exploit video temporal consistency to address the uncertainty
caused by the lack of 3D joint annotations on training data. Apart from taking
temporal sequences as input, Ng et al.[29] propose a novel learned deep prior
of body motions for 3D hand shape synthesis and estimation in the domain of
conversational gestures.

###  2.2 Body Pose Estimation

Predicting 3D Joint Locations. Most previous attempts on predicting 3D human
joint locations take 2D human key-points as input and train models to learn
the 2D to 3D lifting[11, 31, 24, 44, 41]. Some of these work take 2D human
pose of a single frame as input[11, 41], while more of them consider the
inter-frame information and use a sequence of 2D key-points as input[31, 24].
There are also some approaches that train models to directly estimate the 3D
joint locations from visual input[7, 6]. However, a common problem with these
methods is that 3D skeletal rotations cannot be accurately recovered from 3D
joint locations when the goal is to capture human skeletal motion, i.e., the
twist component cannot be determined by joint locations.

Predicting 3D Joint Rotations. Most preceding work leverages parametric human
body models for pose representation, e.g., SMPL[25], ADAM[15]. A basic
pipeline of these work[16] is to train a model to predict the parameters of
SMPL model, such that the projected 2D joints on the input image can match the
2D joints detected by 2D pose estimators[3, 23, 42, 10]. Meanwhile, a
discriminator is introduced to guide the model to produce realistic 3D poses.
[18, 17] extend the baseline solution[16] for smoother results with video
inputs. Nikos et al.[19] propose to add an optimization process[2] after the
CNN model. Nevertheless, without high quality $<image,motion>$ pair, these
methods cannot directly utilize the motion capture data, i.e., motion capture
data is always used to train the GAN part. Therefore, these methods show
larger joint position error than methods that directly predict joint
locations.

###  2.3 Full Motion Capture

There are also some work focusing on capturing full human motion[15, 40, 33,
47, 30]. Rong et al. [33] propose to capture full human motion by combining
several existing technologies. Zhou et al.[47] utilize the body-hand
correlation to train a network to predict 2D hand key-points and 3D body key-
points, but key-points of hand and body are separately fed to different models
for human motion. And SMPLify-X[30] is an optimization process whose objective
is to find a set of parameters of SMPL-X model that best fit the 2D key-
points.

##  3 Methodology

Figure 2: Overall pipeline of our method. The whole pipeline consists two
modules, the key-point estimation module and the rotation estimation module.
The key-point estimation module consists of a 2D hand key-point detector and a
3D body key-point estimator. The rotation estimation module consists of a
carefully designed transformer-based network, which takes a sequence of 2D
hand key-points and 3D arm joints as input, and gives a output prediction of
arm-hand dynamics. Additionally, a discriminator is used to guide the network
for plausible results.

The goal of our work is to capture the dynamics of arms and hands from
monocular video. It is important to note that some researchers use human joint
locations as human poses, while others use skeletal rotation as human poses.
In this work, we primarily focus on training models to estimate rotations of
arms and hands. It is difficult to learn the image-to-rotation mapping
directly. Meanwhile, learning such mapping requires motion capture data with
corresponding video frames, which is hard to acquire. Therefore, we decouple
this task into two sub-tasks, namely the key-point estimation task and the
skeletal rotation estimation task. The overall pipeline of this work is
illustrated in Fig 2. As for the key-point estimation module, two models are
trained to predict 2D hand key-points and 3D body key-points respectively.
Then, the rotation estimation module takes the predicted 2D hand key-points
and 3D arm key-points (obtained from 3D body key-points) as inputs and trains
a model to learn the 3D rotations of arms and hands. In the following we will
briefly introduce the key-point estimation module while delving into the
details of the rotation estimation module.

###  3.1 Problem Definition

The goal of this system $\mathcal{F}$ is to capture the rotations of arms and
hands $\mathbf{Y}$ from visual observations $\mathbf{I}$.

$\mathbf{Y}\bm{=}\mathcal{F}(\mathbf{I})$ (1)

The system is decoupled into two sub-modules, i.e., key-point estimation
module $\mathcal{K}$ and rotation estimation module $\mathcal{R}$,

$\mathbf{K}\bm{=}\mathcal{K}(\mathbf{I})$ (2)

$\mathbf{Y}\bm{=}\mathcal{R}(\mathbf{K})$ (3)

where
$\bm{\mathbf{Y}}=\\{\bm{y}_{0},\bm{y}_{1},...,\bm{y}_{t},...,\bm{y}_{T}\\}$,
$\bm{\mathbf{K}=}\\{\bm{k}_{0},\bm{k}_{1},...,\bm{k}_{t},...,\bm{k}_{T}\\}$,
$\bm{\mathbf{I}=}\\{\bm{i}_{0},\bm{i}_{1},...,\bm{i}_{t},...,\bm{i}_{T}\\}$.
$\bm{y}_{t}\in\bm{\mathbb{R}}^{48\times 3}$ denotes the 3D rotation of arms
and hands at time $t$, which is represented by 3D axis-angle.
$\bm{k}_{t}\in\bm{\mathbb{R}}^{4\times 3+42\times 2}$ denotes the 3D local
location of arm joints and 2D pixel coordinates of hands at time $t$.
$\bm{i}_{t}$ denotes input image frame at time $t$.

Difference with Full Body Inverse Kinematics[43, 27]: Full-body inverse
kinematics (FBIK) is an optimization process which optimizes skeletal motions
to fit 3D joints. Thus FBIK requires accurate 3D hand key-points and body key-
points. Inaccurate 3D key-points might lead to implausible results. However,
it’s much more difficult to obtain accurate 3D hand key-points than to obtain
accurate 2D hand key-points from RGB image. Moreover, frame-by-frame IK
process might cause large jitter in the results. In contrast, our solution
does not require 3D hand key-points to produce smooth and plausible
estimations from video input.

###  3.2 Key-point Estimation Module

2D Hand Key-point Estimation. To obtain accurate 2D hand key-points
efficiently, we first crop the images of the hand regions according to
detected 2D wrist joints, and then feed them to our network. The backbone of
our model is based on the architecture of Mobilenetv3[13]. The model is
trained to fit the ground truth probability map using L2 loss. Detailed
configuration of this module can be found in the supplementary materials.

3D Body Key-point Estimation. We use the VPose3D model[31] trained on AMASS
dataset[26] to obtain 3D body key-points, and use arm key-points as input of
the rotation estimation module.

###  3.3 Rotation Estimation Module

To well utilize temporal information for better performance on capturing arm
and hand dynamics, our model takes a sequence of key-points as input. Since
transformer has been extensively proved to be efficient in various sequence-
to-sequence tasks[38, 22, 36], we design our network based on the transformer
architecture.

Arm-Hand Motion Transformer. We employ a temporal transformer called Arm-Hand
Motion Transformer (AHMT), as a baseline. The input pose sequence is treated
the same way as tokens (words) in Natural Language Processing applications[38,
22, 36]. In order to project each token to a high dimensional space, a pose
embedding module $\bm{\mathbf{E}}$ which consists of two convolution layers,
is designed. And positional embedding $\bm{\mathbf{E_{t}}_{\text{pos }}}$ is
used to retain positional information of the sequence. Given an input
sequence, the input tokens of our transformer can be represented as follows:

$\displaystyle\mathbf{z_{t}}_{0}$
$\displaystyle=\left[\mathbf{x}_{t}^{1}\mathbf{E};\mathbf{x}_{t}^{2}\mathbf{E};\cdots;\mathbf{x}_{t}^{f}\mathbf{E}\right]+\mathbf{E_{t}}_{\text{pos
}}$ (4)

where $X_{t}=[x_{t}^{1};x_{t}^{2};...;x_{t}^{f}]$ is the input sequence;
$\mathbf{E}\in\mathbb{R}^{\left(J\cdot 3\right)\times D_{t}}$ is the pose
embedding module; $\mathbf{E_{t}}_{\text{pos }}\in\mathbb{R}^{f\times D_{t}}$
is the positional embedding. $J$ is the number of total joints including arms
and hands. $f$ is set to 32 during training; $D_{t}=512$ is a constant latent
vector size of our transformer.

Then a standard transformer encoder, consisting of alternating layers of
Multi-head Self-Attention (MSA) and Multi-layer Perceptron (MLP) blocks
(Eq.5,6), is employed to extract temporal information from these high
dimensional features. We apply layer normalization (LN) before every block and
residual connections after every block. The processing of our transformer
encoder can be written as:

$\displaystyle\mathbf{z_{t}}^{\prime}{}_{\ell}$
$\displaystyle=\operatorname{MSA}\left(\operatorname{LN}\left(\mathbf{z_{t}}_{\ell-1}\right)\right)+\mathbf{z_{t}}_{\ell-1},\ell=1\ldots
L_{t}$ (5) $\displaystyle\mathbf{z_{t}}_{\ell}$
$\displaystyle=\operatorname{MLP}\left(\operatorname{LN}\left(\mathbf{z_{t}}^{\prime}{}_{\ell}\right)\right)+\mathbf{z_{t}}_{\ell}^{\prime},\ell=1\ldots
L_{t}$ (6)

where $L_{t}$ represents the number of transformer encoder layer, and
${\mathbf{z_{t}}_{L_{t}}}$ is the last output of transformer encoder.

Finally, the arm-hand rotations are regressed by a regression head consisting
of three convolution layers. We can get the output $\mathbf{y}$ as follows:

$\displaystyle\mathbf{y}$
$\displaystyle=\operatorname{RegressionHead}\left(\mathbf{z_{t}}_{L_{t}}\right)$
(7)

Spatial-Temporal Parallel Arm-Hand Motion Transformer. The AHMT baseline
mainly focuses on extracting temporal features from input sequence. However,
we believe that the spatial information, which represents the kinematic
information between different joints and the correlations between arms and
hands, is as important as the temporal information. Therefore, we design a
Parallel Arm-Hand Motion Transformer (PAHMT) network to fuse the spatial and
temporal information. As illustrated in Fig 2, the PAHMT mainly consists of
two core components, the spatial transformer and the temporal transformer. The
objective of the spatial transformer is to extract global correlations between
arm poses and hand gestures, as well as the local correlations between
different joints. Given an input sequence, we consider each frame, i.e., 3D
direction vectors of arms and 2D coordinates of hands, as a token, and reshape
the sequence $X_{t}\in\mathbb{R}^{f\times(J\cdot 3)}$ into
$X_{s}\in\mathbb{R}^{f\times J\times 3}$ (Eq.8). We conduct zero paddings on
the 2D positions of hands, i.e., $(x,y)\to(x,y,0)$. Considering that both the
global spatial correlations (e.g. waving arm always bring to waving hand) and
the local spatial correlations (e.g. the movements of middle finger and ring
finger are highly correlated) should be preserved across different frames, an
extra learnable ”regression token” ($\mathbf{z_{s}}_{0}^{0}$) is added to the
sequence, whose state at the output of our spatial transformer encoder
($\mathbf{z_{s}}_{L_{s}}^{0}$) serves as the spatial representation, similar
to the ”classification token ” proposed in ViT[9]. Each token is then fed into
a convolution layer and added by a learnable position embedding. The resulting
sequence of vectors is fed into a transformer encoder. The architecture of the
temporal transformer is the same to AHMT (Eq.4,5,6,7). Formally, we have:

$\displaystyle\mathbf{X_{s}}$
$\displaystyle=\operatorname{Reshape}\left(\operatorname{X_{t}}\right)$ (8)
$\displaystyle\mathbf{z_{s}}_{0}$
$\displaystyle=\left[\mathbf{x}_{\text{regress}};\mathbf{x}_{s}^{1}\mathbf{E};\mathbf{x}_{s}^{2}\mathbf{E};\cdots;\mathbf{x}_{s}^{N}\mathbf{E}\right]+\mathbf{E_{s}}_{\text{pos
}}$ (9) $\displaystyle\mathbf{z_{s}}^{\prime}{}_{\ell}$
$\displaystyle=\operatorname{MSA}\left(\operatorname{LN}\left(\mathbf{z_{s}}_{\ell-1}\right)\right)+\mathbf{z_{s}}_{\ell-1},\ell=1\ldots
L_{s}$ (10) $\displaystyle\mathbf{z_{s}}_{\ell}$
$\displaystyle=\operatorname{MLP}\left(\operatorname{LN}\left(\mathbf{z_{s}}^{\prime}{}_{\ell}\right)\right)+\mathbf{z_{s}}_{\ell}^{\prime},\ell=1\ldots
L_{s}$ (11)

where $N=48$ is the same as $J$, which represents the number of total joints
including arms and hands. $\mathbf{E}\in\mathbb{R}^{3\times D_{s}}$ is the
pose embedding module of spatial transformer, $\mathbf{E_{s}}_{\text{pos
}}\in\mathbb{R}^{N\times D_{s}}$ is the learnable positional encoding,
$D_{s}=64$ is the constant latent vector size of our spatial transformer,
$L_{s}$ is the number of the spatial transformer encoder layer.

Since the relationships between joints should be preserved across frames, only
the output of the spatial transformer ($\mathbf{z_{s}}_{L_{s}}^{0}$) is taken
as the spatial representation. Finally, we fuse the spatial features with the
temporal features by element-wise adding and feed them to the regression head
for the final predictions (Eq.12).

$\displaystyle\mathbf{y}$
$\displaystyle=\operatorname{RegressionHead}\left(\mathbf{z_{t}}_{L_{t}}+\mathbf{z_{s}}_{L_{s}}^{0}\right)$
(12)

###  3.4 Training

In this section, we discuss the training details of the rotation estimation
module.

Pre-processing: The 2D hand key-points are normalized. Specifically, the hand
key-points are first subtracted by the coordinates of the wrist joints, then
normalized by the bounding box of the corresponding hand. As for the frame in
which hands cannot be detected, key-points of the previous frame are used. If
the detector fails to detect hands in the first frame, we set the key-points
of the first frame to be a zero vector. Considering that the predicted 3D
skeletal joints might have different bone-length, for the 3D joints of arms,
the direction vectors of upperarm and forearm are taken as input.

Objective function: As illustrated in Fig 2, the objective function of the
rotation estimation module consists of two components. One is the
reconstruction loss that guides the generator to fit the ground truth
rotations of arms and hands. The other is the GAN loss[12] introduced by the
discriminator $\mathcal{D}$. Therefore, the full objective function can be
formulated as:

$\underset{\mathcal{R}}{\mathrm{min}}~{}\underset{\mathcal{D}}{\mathrm{max}}\bm{\mathfrak{L}}_{recon}(\mathcal{R})+\lambda\bm{\mathfrak{L}}_{GAN}(\mathcal{R},\mathcal{D})$
(13)

where $\bm{\mathfrak{L}}_{GAN}$ is a basic form of GAN loss and $\lambda$ is
the weight for this loss, $\bm{\mathfrak{L}}_{GAN}$ can be formulated as,

$\bm{\mathfrak{L}}_{GAN}(\mathcal{R},\mathcal{D})\bm{=}\mathbb{E}[\mathrm{log}(\mathcal{D}(\mathbf{Y}))]+\mathbb{E}[\mathrm{log}(1-\mathcal{R}(\mathbf{K}))]$
(14)

and $\bm{\mathfrak{L}}_{recon}$ consists of three components. The basic one is
a L1 loss, i.e.,

$\bm{\mathfrak{L}}_{L1}(\mathcal{R})\bm{=}\left\|\mathcal{R}(\mathbf{K})-\mathbf{Y}\right\|_{1}$
(15)

One objective of this module is to produce smooth sequential results.
Therefore a smooth loss is adopted, which can be regarded as a regularization
term of the inter-frame difference, i.e.,

$\bm{\mathfrak{L}}_{smooth}(\mathcal{R})\bm{=}{\textstyle\sum_{t=1}^{T}}\left\|\mathcal{R}(\mathbf{\mathit{k_{t}}})-\mathcal{R}(\mathbf{\mathit{k_{t-1}}})\right\|_{1}$
(16)

It is noteworthy that rotations of different joints should have different
importance in representing the poses. The rotation error of parent joints
would propagate to all child joints, making the rotations of parent joints are
more important than those of child joints. Taking this into consideration, we
introduce the FK loss. We first calculate the joint locations ${k_{i}^{*}}$
through the Forward Kinematic function $\mathcal{FK}$[21] of a pre-defined
character skeleton, then compute the L2 loss on the ground truth joint
location ${k_{i}^{*}}$ and the predicted joint location ${k_{i}}$,

$\bm{\mathfrak{L}}_{FK}(\mathcal{R})\bm{=}\left\|\mathcal{FK}(\mathcal{R}(\mathbf{K}))-\mathbf{K}^{*}\right\|_{2}$
(17)

$\mathbf{K}^{*}$ indicates 3D key-points of arms and hands, which can be
obtained from motion capture data.

Finally, the full objective function $\bm{\mathfrak{L}}_{recon}$ for the
generator can be defined as follows,

$\bm{\mathfrak{L}}_{recon}\bm{=}\bm{\mathfrak{L}}_{L1}+\gamma\bm{\mathfrak{L}}_{smooth}+\beta\bm{\mathfrak{L}}_{FK}$
(18)

##  4 Experiments

We analyse the performance of the proposed method via extensive experiments.
We conduct ablation study to validate each individual component of our method
and make both quantitative and perceptual comparisons with state-of-the-art
approaches to demonstrate the advantage of our method.

###  4.1 Dataset

Our Motion Capture Dataset: Although there are some publicly available motion
capture datasets[26, 14], body motion data and hand motion data are always
collected separately. We still lack a dataset of full human motion. We
therefore collect a dataset of full human motion using motion capture devices.
Specifically, we collect 500 sequences of full human motion data, with a total
frame count of 200K. The dataset primarily consists of sequences of dancing or
sports, covering a large number of body motions and hand gesture styles. We
manually split the dataset into training set (90%) and test set (10%). We
conduct ablation study on the test set. See the supplementary materials for
samples of this dataset.

In-the-wild 3D Body and Hand Gesture Dataset (BH Dataset): To make comparison
with the state-of-the-art gesture estimator[29], we conduct experiments on the
dataset released by the author[29]. This dataset consists of hours of in-the-
wild videos of 8 speakers and covers a wide range of gesture types.

Rendered Full Body Motion Dataset: Although we already have full body motion
capture data, we still lack in-the-wild data with full body motion
annotations. Therefore, we propose a rendered dataset to simulate in-the-wild
scenarios. Specifically, 10K frames of full body motion capture data are
retargeting to 3 character models from MIXAMO[1]. We compare our method with
state-of-the-art full body motion capture methods[33, 8] on this dataset. The
samples of this dataset is shown in the supplementary materials.

###  4.2 Implementation Details

We first retarget all the skeletal motion sequences to the MIXAMO character
model[1]. Then we obtain the rotation (axis-angle) and the 3D position of each
joint in world coordinate system. To generate training data, we project 3D
locations of hands into the 2D camera plane to get the 2D pixel coordinates of
hands. We apply a sliding window of size 32 with a step of 5 frames for each
motion sequence, producing 30K training sequences. We train the network using
Adam optimizer and we set the batch size, weight decay and momentum to 128,
0.0005 and 0.95, respectively. The initial learning rate is set to 1e-3 and
dropped by 50% every 50 epochs. We train our model for 300 epochs. As for the
objective functions, we empirically set $\lambda$, $\beta$, $\gamma$ to 0.05,
1.0, 1.0, respectively.

###  4.3 Evaluation Metric

As the goal of our work is to capture accurate skeletal rotations of arms and
hands from visual input, besides the commonly used mean per joint position
error (MPJPE, unit in meter), we also report the mean per joint rotation error
(MPJRE), which measures the average absolute difference between predicted
joint rotations and ground truth joint rotations. It is noteworthy that
accurate hand rotations can be recovered from accurate 3D hand joints by
inverse kinematic algorithms[27, 43], while arm rotations cannot be accurately
recovered from 3D arm joints. Therefore, we use 1) the overall MPJPE; 2) MPJPE
of hands; 3) MPJRE of arms to evaluate the performance.

Table 1: Ablation study on different components of the proposed method. Evaluation is conducted on our motion capture dataset. We evaluate the overall MPJPE, MPJPE of hands and MPJRE of arms. Results indicate that each component shows positive impact on learning high quality arm and hand dynamics. h2h represents the models which take 2D hand key-points as input and predict hand gestures. ah2ah represents the models which take arm-hand key-points as input and predict arm-hand dynamics. Architecture | h2h | ah2ah | Smooth Loss | FK Loss | MPJPE(hands)$\downarrow$ | MPJPE(overall)$\downarrow$ | MPJRE(arms)$\downarrow$
---|---|---|---|---|---|---|---
CNN | ✓ | ✗ | ✗ | ✗ | 0.0300 | - | -
CNN | ✗ | ✓ | ✗ | ✗ | 0.0130 | 0.0517 | 0.0604
CNN | ✗ | ✓ | ✓ | ✗ | 0.0133 | 0.0519 | 0.0603
CNN | ✗ | ✓ | ✗ | ✓ | 0.0129 | 0.0413 | 0.0579
CNN | ✗ | ✓ | ✓ | ✓ | 0.0127 | 0.0408 | 0.0577
AHMT | ✗ | ✓ | ✓ | ✓ | 0.0098 | 0.0316 | 0.0445
PAHMT | ✗ | ✓ | ✓ | ✓ | 0.0087 | 0.0274 | 0.0375
Table 2: Comparison with Body2Hands[29] on dataset released by [29]. We report MPJPE of hands for each method. Method | MPJPE(hands)$\downarrow$
---|---
Body2Hands w/o image[29] | 0.0422
Body2Hands w/ image[29] | 0.0400
Body2Hands*[29] | 0.0346
PAHMT(Ours) | 0.0281
Table 3: Comparison with state-of-the-art full body motion capture methods[33, 8] on our rendered dataset. We report overall MPJPE and MPJRE of arms for each method. Method | MPJPE(overall)$\downarrow$ | MPJRE(arms)$\downarrow$
---|---|---
FrankMocap[33] | 0.1556 | 0.2779
ExPose[8] | 0.1530 | 0.2984
PAHMT(Ours) | 0.1375 | 0.1614

###  4.4 Methods for comparison

Firstly, we conduct ablation study to assess the contribution of each part of
the method in estimating motions of arms and hands. Secondly, we compare our
method with several state-of-the-art approaches[29, 33, 8]. Note that our
method simultaneously predicts the motion (rotations) of arms and hands, and
the lack of a publicly available dataset with rotation annotations of both
arms and hands prevents us from making a full comparison of both arms and
hands rotations on an existing benchmark. Therefore, we only report the
performance of hand pose estimation comparing our method with Body2Hands[29]
on the dataset released by the authors. Furthermore, we make comparisons on
our rendered dataset to evaluate the overall arm-hand dynamics with previous
approaches. We also show results of our methods and state-of-the-arts
approaches on some in-the-wild videos for perceptual evaluation. Specifically,
we compare following methods:

  * •

Body2Hands w/wo image[29]: State-of-the-art 3D hand shape synthesis and
estimation by a learned deep prior of body motion. Models are re-trained using
the codes released by the authors, since the pre-trained model is not
available.

  * •

Body2Hands*[29]: The original Body2Hands model takes the output of MTC[40] as
input and predicts the rotations of hands. However, in our experiments, MTC
output is not provided. Thus we retrain a Body2Hands model of the same
structure as the original version which takes 3D body joints and 2D hand key-
points as input. We take the retrained version of Body2Hands as Body2Hands*.

  * •

ExPose[8]: State-of-the-art method for full body pose including hand gesture
estimation, which directly regresses the body, face, and hand poses from RGB
image.

  * •

FrankMocap[33]: State-of-the-art method for full body pose including hand
gesture estimation, which leverages multiple leading solutions for pose
estimation of different human body parts.

  * •

Our CNN: The same model structure with Body2Hands[29], trained with the
proposed objective functions, takes 3D arm joints and 2D hand key-points as
input and predicts the rotations of both arms and hands.

  * •

Our AHMT: Same configuration as Our CNN, except that the network architecture
is replaced by a temporal transformer.

  * •

Our PAHMT: Same configuration as Our CNN, except that the network architecture
is replaced by our carefully designed Spatial-Temporal Parallel Arm-Hand
Motion Transformer(PAHMT).

Figure 3: Effectiveness of leveraging arm-hand correlations. (a): Input frame.
(b): Results of ah2ah. (c): Results of h2h+IK. To compare the rotations of
arms, we set the rotations of wrists to zeros in the bottom row of (b) and
(c). Figure 4: Visual comparison with state-of-the-art methods[8, 33]. Row 1:
In-the-wild video frames of various arm poses and hand gestures in challenging
scenarios. Row 2: Results of FrankMocap[33]. Row 3: Results of ExPose[8]. Row
4: Results of our method. Despite occlusion and motion blur, our method
produces accurate and plausible results.

###  4.5 Ablation Study

To evaluate the importance of each individual component of the proposed PAHMT
and the impact of different hyperparameters of the network architecture,
extensive experiments are conducted on our motion capture dataset.

Baseline Setting. We choose the network architecture of Our CNN as the
baseline architecture. The objective function for the baseline model does not
include $\bm{\mathfrak{L}}_{smooth}$ and $\bm{\mathfrak{L}}_{FK}$. For fair
comparison, we fix the random seed for generating training batches in all the
experiments. Adam optimizer is used and we fix the batch size, learning rate,
weight decay and momentum to 128, 1e-3, 0.0005 and 0.95 in all the
experiments.

Effectiveness of Leveraging Arm-hand Correlations. As illustrated in Table 1,
h2h (hand input to hand output) performs worse than ah2ah (arm-hand input to
arm-hand output), indicating that the task of hand gesture estimation benefits
from arm-hand correlation. Fig 3 intuitively shows the advantage of leveraging
arm-hand correlations to predict arm-hand dynamics simultaneously. To
demonstrate the superiority of ah2ah in predicting arm rotations, we compare
ah2ah with h2h+IK[43] (hand input to hand output, with IK to solve the arm
rotations). We can observe that ah2ah gives more reasonable arm rotation
predictions than h2h+IK.

Effectiveness of the PAHMT Architecture. Table 1 presents the results. By
making better use of inter-trame information, traditional temporal transformer
(AHMT) significantly outperforms CNN (23% error reduction on both MPJPE and
MPJRE). We further investigate the impact of introducing the spatial
transformer. By utilizing the global correlation between arms and hands as
well as the local correlation between different body joints, our PAHMT further
improves the results of AHMT (13% error reduction on MPJPE, 16% error
reduction on MPJRE).

Effectiveness of the Proposed Objective Functions. We evaluate the impact of
$\bm{\mathfrak{L}}_{smooth}$ and $\bm{\mathfrak{L}}_{FK}$. Table 1 shows the
results, $\bm{\mathfrak{L}}_{FK}$ shows advantage in reducing both the MPJPE
and MPJRE. It is noteworthy that $\bm{\mathfrak{L}}_{smooth}$ is originally
designed for temporal smoothness, not for the error reduction, therefore
$\bm{\mathfrak{L}}_{smooth}$ shows no superiority in reducing the error when
used independently. However, $\bm{\mathfrak{L}}_{smooth}$ also makes
contribution to the accuracy improvement when integrated with
$\bm{\mathfrak{L}}_{FK}$.

Architecture Parameter Analysis. We investigate different parameters of
network architecture to search for a optimal setting. Detailed results can be
found in the supplementary materials.

###  4.6 Comparison with State-of-the-Arts

Table 2 shows the results on BH dataset. For fair comparison, the input 3D
joints for our model is derived from the MTC[40] output. We can see that our
PAHMT significantly outperforms the Body2Hands methods[29]. Note that BH
dataset is annotated by MTC[40], which means the label is not the ground
truth. We show that our method can produce more plausible and smoother results
than the annotations in supplementary materials.

Table 3 shows the results on the rendered dataset, which indicates that our
method significantly outperforms state-of-the-art full body motion capture
methods[33, 8] on both the metric of MPJPE and MPJRE. It is worth noticing
that our approach is significantly superior to existing methods in predicting
arm rotations, which might indicate that our approach can make much better use
of arm-hand correlation than state-of-the-arts.

###  4.7 Visual Comparison

To demonstrate the advantage of the proposed method, we perform experiments on
in-the-wild videos for visual comparison with ExPose[8] and FrankMocap[33].
For comparison, we conduct evaluation on in-the-wild videos including cases of
clear and challenging unclear hands, as well as various arm poses.

Fig 4 shows the visual comparison. We can observe that both ExPose[8] and
FrankMocap[33] fail to predict plausible hand gestures from unclear hand
images. Meanwhile, these two state-of-the-arts cannot accurately estimate the
rotations of arms. In contrast, our method produces the most accurate and
plausible results even in challenging scenarios. More results can be found in
the supplementary video.

##  5 Discussion

We propose a novel approach to estimate arm and hand dynamics from monocular
video by utilizing the correlations between arm and hand motions. To make full
use of the inter-frame information as well as the arm-hand correlations, a
spatial-temporal parallel transformer model, PAHMT is proposed. To obtain
smooth and accurate prediction, we propose new objective functions. Extensive
experiments demonstrate the advantage of our method over previous state-of-
the-arts. As for the limitation, our solution requires a motion capture
dataset of both body motions and hand gestures to train the model. Future work
may include leveraging the correlation of more body parts for high quality
full body dynamic estimation and leveraging various dataset of annotations of
different body parts.

## References

  * [1] Mixamo. animate 3d characters for games, film, and more.  http://https://www.mixamo.com.  Accessed: 2021-09-30.
  * [2] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J. Black.  Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image.  In Computer Vision – ECCV 2016, Lecture Notes in Computer Science. Springer International Publishing, Oct. 2016.
  * [3] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Realtime multi-person 2d pose estimation using part affinity fields.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291–7299, 2017.
  * [4] Liangjian Chen, Shih-Yao Lin, Yusheng Xie, Yen-Yu Lin, and Xiaohui Xie.  Temporal-aware self-supervised learning for 3d hand pose and mesh estimation in videos.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1050–1059, 2021.
  * [5] Xingyu Chen, Yufeng Liu, Chongyang Ma, Jianlong Chang, Huayan Wang, Tian Chen, Xiaoyan Guo, Pengfei Wan, and Wen Zheng.  Camera-space hand mesh recovery via semantic aggregation and adaptive 2d-1d registration.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13274–13283, 2021.
  * [6] Yu Cheng, Bo Yang, Bo Wang, and Robby T Tan.  3d human pose estimation using spatio-temporal networks with explicit occlusion training.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10631–10638, 2020.
  * [7] Yu Cheng, Bo Yang, Bo Wang, Wending Yan, and Robby T Tan.  Occlusion-aware networks for 3d human pose estimation in video.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 723–732, 2019.
  * [8] Vasileios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas, and Michael J Black.  Monocular expressive body regression through body-driven attention.  In European Conference on Computer Vision, pages 20–40. Springer, 2020.
  * [9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [10] Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu.  RMPE: Regional multi-person pose estimation.  In ICCV, 2017.
  * [11] Kehong Gong, Jianfeng Zhang, and Jiashi Feng.  Poseaug: A differentiable pose augmentation framework for 3d human pose estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8575–8584, 2021.
  * [12] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [13] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al.  Searching for mobilenetv3.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1314–1324, 2019.
  * [14] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu.  Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.  IEEE transactions on pattern analysis and machine intelligence, 36(7):1325–1339, 2013.
  * [15] Hanbyul Joo, Tomas Simon, and Yaser Sheikh.  Total capture: A 3d deformation model for tracking faces, hands, and bodies.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8320–8329, 2018.
  * [16] Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik.  End-to-end recovery of human shape and pose.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7122–7131, 2018.
  * [17] Angjoo Kanazawa, Jason Y Zhang, Panna Felsen, and Jitendra Malik.  Learning 3d human dynamics from video.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5614–5623, 2019.
  * [18] Muhammed Kocabas, Nikos Athanasiou, and Michael J. Black.  Vibe: Video inference for human body pose and shape estimation.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [19] Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis.  Learning to reconstruct 3d human pose and shape via model-fitting in the loop.  In ICCV, 2019.
  * [20] Nikos Kolotouros, Georgios Pavlakos, and Kostas Daniilidis.  Convolutional mesh regression for single-image human shape reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4501–4510, 2019.
  * [21] Serdar Kucuk and Zafer Bingul.  Robot kinematics: Forward and inverse kinematics.  INTECH Open Access Publisher, 2006.
  * [22] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  arXiv preprint arXiv:1910.13461, 2019.
  * [23] Jiefeng Li, Can Wang, Hao Zhu, Yihuan Mao, Hao-Shu Fang, and Cewu Lu.  Crowdpose: Efficient crowded scenes pose estimation and a new benchmark.  arXiv preprint arXiv:1812.00324, 2018.
  * [24] Wenhao Li, Hong Liu, Runwei Ding, Mengyuan Liu, and Pichao Wang.  Lifting transformer for 3d human pose estimation in video.  arXiv preprint arXiv:2103.14304, 2021.
  * [25] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black.  Smpl: A skinned multi-person linear model.  ACM transactions on graphics (TOG), 34(6):1–16, 2015.
  * [26] Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black.  Amass: Archive of motion capture as surface shapes.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5442–5451, 2019.
  * [27] Michael Mistry, Jun Nakanishi, Gordon Cheng, and Stefan Schaal.  Inverse kinematics with floating base and constraints for full body humanoid robot control.  In Humanoids 2008-8th IEEE-RAS International Conference on Humanoid Robots, pages 22–27. IEEE, 2008.
  * [28] Franziska Mueller, Florian Bernard, Oleksandr Sotnychenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, and Christian Theobalt.  Ganerated hands for real-time 3d hand tracking from monocular rgb.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 49–59, 2018.
  * [29] Evonne Ng, Shiry Ginosar, Trevor Darrell, and Hanbyul Joo.  Body2hands: Learning to infer 3d hands from conversational gesture body dynamics.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11865–11874, 2021.
  * [30] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black.  Expressive body capture: 3d hands, face, and body from a single image.  In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [31] Dario Pavllo, Christoph Feichtenhofer, David Grangier, and Michael Auli.  3d human pose estimation in video with temporal convolutions and semi-supervised training.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [32] Javier Romero, Dimitrios Tzionas, and Michael J Black.  Embodied hands: Modeling and capturing hands and bodies together.  ACM Transactions on Graphics (ToG), 36(6):1–17, 2017.
  * [33] Yu Rong, Takaaki Shiratori, and Hanbyul Joo.  Frankmocap: Fast monocular 3d hand and body motion capture by regression and integration.  arXiv preprint arXiv:2008.08324, 2020.
  * [34] Wenkang Shan, Haopeng Lu, Shanshe Wang, Xinfeng Zhang, and Wen Gao.  Improving robustness and accuracy via relative information encoding in 3d human pose estimation.  arXiv preprint arXiv:2107.13994, 2021.
  * [35] Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh.  Hand keypoint detection in single images using multiview bootstrapping.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1145–1153, 2017.
  * [36] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu.  Mass: Masked sequence to sequence pre-training for language generation.  arXiv preprint arXiv:1905.02450, 2019.
  * [37] Adrian Spurr, Jie Song, Seonwook Park, and Otmar Hilliges.  Cross-modal deep variational hand pose estimation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 89–98, 2018.
  * [38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [39] Yangang Wang, Baowen Zhang, and Cong Peng.  Srhandnet: Real-time 2d hand pose estimation with simultaneous region localization.  IEEE transactions on image processing, 29:2977–2986, 2019.
  * [40] Donglai Xiang, Hanbyul Joo, and Yaser Sheikh.  Monocular total capture: Posing face, body, and hands in the wild.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10965–10974, 2019.
  * [41] Bin Xiao, Haiping Wu, and Yichen Wei.  Simple baselines for human pose estimation and tracking.  In Proceedings of the European conference on computer vision (ECCV), pages 466–481, 2018.
  * [42] Yuliang Xiu, Jiefeng Li, Haoyu Wang, Yinghong Fang, and Cewu Lu.  Pose Flow: Efficient online pose tracking.  In BMVC, 2018.
  * [43] Katsu Yamane and Yoshihiko Nakamura.  Natural motion animation through constraining and deconstraining at will.  IEEE Transactions on visualization and computer graphics, 9(3):352–360, 2003.
  * [44] Sijie Yan, Yuanjun Xiong, and Dahua Lin.  Spatial temporal graph convolutional networks for skeleton-based action recognition.  In Thirty-second AAAI conference on artificial intelligence, 2018\.
  * [45] Yuxiang Zhang, Zhe Li, Liang An, Mengcheng Li, Tao Yu, and Yebin Liu.  Lightweight multi-person total motion capture using sparse multi-view cameras.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5560–5569, 2021.
  * [46] Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, and Dimitris N Metaxas.  Semantic graph convolutional networks for 3d human pose regression.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3425–3435, 2019.
  * [47] Yuxiao Zhou, Marc Habermann, Ikhsanul Habibie, Ayush Tewari, Christian Theobalt, and Feng Xu.  Monocular real-time full body capture with inter-part correlations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4811–4822, 2021.
  * [48] Yuxiao Zhou, Marc Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, and Feng Xu.  Monocular real-time hand shape and motion capture using multi-modal data.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5346–5355, 2020.
  * [49] Christian Zimmermann and Thomas Brox.  Learning to estimate 3d hand pose from single rgb images.  In Proceedings of the IEEE international conference on computer vision, pages 4903–4911, 2017.