# Mutual Adaptive Reasoning for Monocular 3D Multi-Person Pose Estimation

Juze Zhang1,2,3, Jingya Wang1,4∗, Ye Shi1,4∗, Fei Gao1, Lan Xu 1,4, Jingyi Yu
1,4 1 School of Information Science and Technology, ShanghaiTech University,
Shanghai 201210, China
2 Shanghai Advanced Research Institute, Chinese Academy of Sciences, Shanghai
201203, China
3 University of Chinese Academy of Sciences, Beijing 100049, China
4 Shanghai Engineering Research Center of Intelligent Vision and Imaging
<EMAIL_ADDRESS>
(2022)

###### Abstract.

Inter-person occlusion and depth ambiguity make estimating the 3D poses of
monocular multiple persons as camera-centric coordinates a challenging
problem. Typical top-down frameworks suffer from high computational redundancy
with an additional detection stage. By contrast, the bottom-up methods enjoy
low computational costs as they are less affected by the number of humans.
However, most existing bottom-up methods treat camera-centric 3D human pose
estimation as two unrelated subtasks: 2.5D pose estimation and camera-centric
depth estimation. In this paper, we propose a unified model that leverages the
mutual benefits of both these subtasks. Within the framework, a robust
structured 2.5D pose estimation is designed to recognize inter-person
occlusion based on depth relationships. Additionally, we develop an end-to-end
geometry-aware depth reasoning method that exploits the mutual benefits of
both 2.5D pose and camera-centric root depths. This method first uses 2.5D
pose and geometry information to infer camera-centric root depths in a forward
pass, and then exploits the root depths to further improve representation
learning of 2.5D pose estimation in a backward pass. Further, we designed an
adaptive fusion scheme that leverages both visual perception and body geometry
to alleviate inherent depth ambiguity issues. Extensive experiments
demonstrate the superiority of our proposed model over a wide range of bottom-
up methods. Our accuracy is even competitive with top-down counterparts.
Notably, our model runs much faster than existing bottom-up and top-down
methods.

3D Human Pose Estimation, Depth Regression, Geometry-Aware Depth Reasoning,
Adaptive Fusion

*Corresponding author

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
30th ACM International Conference on Multimedia ; October 10–14, 2022; Lisboa,
Portugal.††booktitle: Proceedings of the 30th ACM International Conference on
Multimedia (MM ’22), Oct. 10–14, 2022, Lisboa, Portugal††price: 15.00††isbn:
978-1-4503-9203-7/22/10††doi: 10.1145/3503161.3548148††ccs: Computing
methodologies Activity recognition and understanding Figure 1. (a) Existing
methods treat camera-centric 3D-MPE as two unrelated subtasks: 2.5D pose
representation and absolute depth estimation. (b) Our method can bridge the
gap between 2.5D pose representation and depth estimation and achieve mutual
benefit from them.

##  1\. Introduction

In recent years, 3D pose estimation has attracted a great deal of interest
from researchers due to its widespread application in a range of fields,
including video analysis, camera surveillance, human-computer interaction,
virtual/augmented reality, etc. Great successes have been achieved in
monocular 3D human pose estimation, especially when dealing with a single
person in an image(Sun et al., 2018; Zeng et al., 2021; Kocabas et al., 2019;
Iqbal et al., 2020; Cheng et al., 2019; Wandt and Rosenhahn, 2019). However,
when it comes to real-world scenarios, inferring poses can be much harder,
especially when multiple persons are present in the scene. This is because
body joints are often occluded by other objects or people. For this reason,
monocular 3D multi-person pose estimation (3D-MPE) is still a challenging
problem, yet, robustness to such occlusions is critical to real-world
applications.

Camera-centric 3D-MPE tasks aim to recover each pose as coordinates in the
camera-centric coordinate system. This requires estimating the absolute depth
of each person in 3D. Generally, existing methods for 3D-MPE can be divided
into two categories: top-down and bottom-up. Typical top-down approaches use
the single-person method with a 2D person detector to handle multi-person
scenes (Moon et al., 2019). This involves a human detector and pose estimation
in two stages. Since each person in the image is treated individually, these
methods ignore out-of-patch contexts. They also suffer from high computational
redundancy. By contrast, bottom-up 3D-MPE (Zhen et al., 2020) does not need
human detectors and can perceive global image contexts in a single shot. Thus,
multi-person interactions in pose estimation tasks do not present a problem.

Most existing bottom-up methods treat camera-centric 3D-MPE as two unrelated
subtasks: 2.5D pose representation and depth estimation as shown in Figure 1a.
We argue that a robust 2.5D pose representation, especially in crowded multi-
person scenes, requires the global depth cues over the whole image to be
aggregated for disambiguation. Further, the geometric information associated
with 2.5D poses can lead to a closed-form solution for individual depths. This
observation motivated us to design adaptive reasoning between pose and depth
estimation that mutually benefit each other. To this end, we developed a novel
unified model that combines the two subtasks as shown in Figure 1b. To improve
the representation learning of 2.5D pose estimation, we developed an end-to-
end method using the information of geometry depths. To disambiguate the
depths, we designed a depth fusion scheme that leverages both direct visual
perception and geometry to deal with inherent depth ambiguity issues in
monocular human depth estimation. Thus, the model can bridge the gap between
2.5D pose representation and depth estimation. Moreover, the framework
includes a process for robust structured pose refinement to handle occlusive
or out-of-image joints. This means the model is capable of end-to-end learning
without needing an additional network in the post-processing stage to complete
any missing joints. The whole pipeline is illustrated in figure 2. To
summarize, our main contributions are:

  * •

A unified bottom-up model that leverages the mutual benefits of 2.5D pose and
depth estimation. Exploiting each strength yields greater robustness for
disambiguation.

  * •

An end-to-end geometry aware depth reasoning method that first uses 2.5D pose
and geometry information to infer camera-centric root depths in a forward
pass, and then exploits the root depths to further improve representation
learning of 2.5D pose estimation in a backward pass.

  * •

An adaptive depth fusion that leverages both direct visual perception and
geometry to deal with inherent depth ambiguity issues in monocular human depth
estimation.

##  2\. Related Work

Multi-Person 2D Pose Estimation. As mentioned, existing methods for multi-
person 2D pose estimation can mainly be divided into top-down and bottom-up
approaches. Typical top-down frameworks deal with human detection and pose
estimation in two stages (Chen et al., 2018; Fang et al., 2017). Despite their
strength at handling scale variation, top-down methods also suffer from high
computational redundancy when attempting to detect additional persons.
Moreover, their performance tends to degrade severely with heavy occlusion
since they have no awareness of out-of-patch contexts. Bottom-up approaches
(Cao et al., 2017, 2021; Newell et al., 2017; Hidalgo et al., 2019; Nie et
al., 2019; Papandreou et al., 2018; Brasó et al., 2021; Geng et al., 2021;
Zhou et al., 2019; Li et al., 2019; Jin et al., 2020) localize all key points
in the image first and then group them to an individual. Examples include
OpenPose (Cao et al., 2017), a representative work, which was the first to
present the part-affinity field approach that links key points likely to lie
in the same person. Zhou et al. (Zhou et al., 2019) were the first to propose
a single-stage offset-based method for 2D human pose estimation. However, the
above method only focuses on the 2D level and is careless about 3D information
from the image.

Top-Down Multi-Person 3D Pose Estimation. Early works focused on human-centric
tasks by using a top-down manner without estimating individual depth (Rogez et
al., 2017, 2019). Only a few works tackle the problem of camera-centric multi-
person 3D pose estimation from a monocular RGB image or video. Moon et al.
(Moon et al., 2019) was the first to suggest locating a person’s root absolute
depth by learning a correction factor of the area of 2D bounding box, while
Lin et al. (Lin and Lee, 2020) proposed a pose-aware human depth estimation
network to address the problem of root joint localization for multi-person 3D
pose estimation in the camera-space. Similarly, Veges et al. (Véges and
Lőrincz, 2019) proposed to use two separate networks, pose estimator and depth
estimator, to recover camera-centric multi-person 3D pose. To handle inherent
depth ambiguity, Wang et al. (Wang et al., 2020a) proposed a novel
hierarchical multi-person ordinal relation. Another type of work utilizes
temporal information to recover 3D poses from a given video(Cheng et al.,
2021a, b). By applying a top-down scheme, the above method either directly
regresses the absolute 3D depth from a cropped image, or it computes it based
on a prior of the body size, ignoring global image contexts. Moreover, they
suffer from high computational redundancy with an additional detection stage.
By contrast, our methods can utilize global image contexts and enjoy low
computational costs as they are less affected by the number of humans.

Bottom-Up Multi-Person 3D Pose Estimation. A few bottom-up methods have been
proposed for 3D-MPE(Mehta et al., 2018; Fabbri et al., 2020; Mehta et al.,
2020; Zhen et al., 2020). Mehta et al. (Mehta et al., 2018) first proposed a
bottom-up method with an occlusion-robust pose-map (ORPM) to represent the
occlusive joints. Such methods can alleviate the occlusion problem to some
degree, but they only focus on person-centric and do not infer multiple
persons as camera-centric coordinates. While Fabbri et al. (Fabbri et al.,
2020) used an encoder-decoder network to compress a heatmap and then
decompress it back to the original resolution in the inference time for fast
HD image processing. They show superior results when tackling crowded scenes
but careless about the occlusion problem. Another method called XNect (Mehta
et al., 2020), a framework that encodes a 3D location map at the spatial
location of each visible joint, which also focuses on a person-centric
problem. Zhen et al. (Zhen et al., 2020) proposed an end-to-end network with
the depth-aware part association and bone-length constraints that benefits the
2.5D pose estimation branch of the scheme. However, these models heavily
depend on post-processing techniques or refinement. Above bottom-up methods
treat camera-centric 3D human pose estimation as two unrelated subtasks: 2.5D
pose estimation and camera-centric depth estimation. By contrast, our model
focus on exploring the mutual benefits 2.5D pose estimation and camera-centric
depths in an end-to-end manner.

Figure 2. Overview of the proposed Mutual Adaptive Reasoning (MAR) framework.

##  3\. Method

Our task is to estimate the 3D poses for multiple people in a monocular RGB
image as camera-centric coordinates considering the challenge of inter-person
occlusion and depth ambiguity. In this paper, we propose a Mutual Adaptive
Reasoning (MAR) method for this task. MAR is a unified bottom-up model that
leverages the mutual benefits of 2.5D pose and depth estimation. Within the
model, a robust structured 2.5D pose estimation is first designed to recognize
inter-person occlusion based on depth relationships. Additionally, we develop
an end-to-end geometry aware depth reasoning method that exploits the mutual
benefits of both 2.5D pose and camera-centric root depths. This method first
uses 2.5D pose and geometry information to infer camera-centric root depths in
a forward pass, and then exploits the root depths to further improve
representation learning of 2.5D pose estimation in a backward pass. Further,
we designed a depth fusion scheme that leverages both visual perception and
body geometry to alleviate inherent depth ambiguity issues. An overview of the
proposed Mutual Adaptive Reasoning (MAR) method is depicted in Figure 2. More
details on the structured pose estimation module are provided in Section 3.1,
followed by the direct depth regression module in Section 3.2, the end-to-end
geometry depth aware reasoning module in Section 3.3 and the adaptive fusion
for depth ambiguity in Section3.4.

###  3.1. Robust 2.5D Structured Pose Estimation

For monocular 3D multi-person pose estimation (3D-MPE), the goal is to recover
the absolute camera-centric coordinates of the keypoint of multiple people
$\\{{\bf{P}}^{3D}_{k}\\}^{J}_{k=1}$, where
${\bf{P}}^{3D}_{k}=[X_{k},Y_{k},Z_{k}]^{T}$ and $J$ represent the number of
joints. A 3D pose can be decomposed into representations of 2.5D pose
$\\{{\bf{P}}^{2.5D}_{k}\\}^{J}_{k=1}=\\{\left[u_{k},v_{k},Z_{k}^{r}\right]^{T}\\}^{J}_{k=1}$
and the absolute depth of the root joint $Z_{\text{root}}$. Here, the
coordinates $u_{k}$ and $v_{k}$ are the image pixel coordinates of the $k$-th
keypoint, $Z_{k}^{r}$ is its relative depth to the root keypoint and
$Z_{\text{root}}$ is the absolute camera-centric depth of each person. Given a
2.5D pose $\\{{\bf{P}}^{2.5D}_{k}\\}^{J}_{k=1}$, several elements can be
derived from back-projection. These are: the intrinsic camera parameters
$K\in\mathbb{R}^{3\times 3}$; the absolute human depth $Z_{\text{root}}$, and
the final absolute 3D pose $\\{{\bf{P}}^{3D}_{k}\\}^{J}_{k=1}$. Let
$f:({\bf{P}}^{2.5D}_{k},\bf{Z}_{\text{root}})\rightarrow{\bf{P}}^{3D}_{k}$ be
a back-projection function. Thus,

(1)
$f:{\bf{P}}^{3D}_{k}=\left(\bf{Z}_{\text{root}}+Z_{k}^{r}\right)K^{-1}[u_{k},v_{k},1]^{T},$

where

$K=\left[\begin{array}[]{ccc}f_{x}&0&c_{x}\\\ 0&f_{y}&c_{y}\\\
0&0&1\end{array}\right],$

$f_{x},f_{y}$ is focal lengths divided by the per-pixel distance factors
(pixel) of x-axis and y-axis and $c_{x},c_{y}$ is principal point of x-axis
and y-axis.

In real application scenarios, the body joints may be occluded, making it hard
to infer occluded joints from heatmaps. Thus, we designed the offset scheme
for pose estimation, which handles issues of occlusion as well as completely
out-of-frame joints as shown in Fig.3. Appearance-based heatmap
representations have a more precise spatial positioning accuracy, but they
usually fail to infer occluded joints. By contrast, 2.5D joint offset scheme
estimates its position from the offset between root joints, which is more
robust to occlusive joints. Here, we propose a robust structured pose
estimation that simultaneously estimates heatmaps and 2.5D joint offsets.

A heatmap of keypoints encodes the center of a 2D human body in the image,
where the activation value at each position indicates the degree to which the
keypoint lies in position. The position of each joint is represented as a
gaussian distribution in the heatmap, and we use PAFs to link the identity-
agnostic joints from the keypoint heatmaps after non-maximum suppression
following (Ding et al., 2020). Then, we can get the 2D joints location
$(u_{k}^{hm},v^{hm}_{k})^{T}$ at image pixel coordinate from heatmap scheme.

Moreover, we introduce 2.5D joint offset scheme that aims to estimate $(J-1)$
human joint offsets with respect to a root joint. The offset maps are encoded
at the spatial location of each root joint predicted by the keypoint heatmap.
Formally, the 2.5D offset-based pose estimation is formulated as follows:

(2)
$(u_{k}^{disp},v_{k}^{disp},Z_{k}^{r})=(u_{\text{root}},v_{\text{root}},0)+(\Delta_{u_{k}},\Delta_{v_{k}},\Delta_{Z_{k}}),$

where $(u_{\text{root}},v_{\text{root}},0)$ represents the spatial location of
the root joint as image coordinates and
$(\Delta_{u_{k}},\Delta_{v_{k}},\Delta_{Z_{k}})$ represents the offset of the
$k$-th body joint position with respect to the root joint. Note that the
relative depth of the root joint with respect to itself must be zero. Based on
above observation, we use heatmap and part affinity fields to estimate the
major keypoints. Occlusive or out-of-image joints can be inferred using the
offset scheme. The final structured 2.5D pose ${\bf{P}}^{2.5D}_{k}$ is then
inferred as follows:

(3)
${\bf{P}}^{2.5D}_{k}=\begin{cases}(u_{k}^{hm},v^{hm}_{k},Z_{k}^{r})^{T}&\text{if}\
c_{k}\geq\text{threshold},\\\
(u_{k}^{disp},v_{k}^{disp},Z_{k}^{r})^{T}&\text{else},\end{cases}$

where $c_{k}$ is the local maximum value of heatmaps for joint $k$. It is
worth noting that Eq.3 was only used in inference stage. The 2.5D pose
estimation is trained in an end-to-end manner with the depth estimation. The
total training loss of the pose estimation includes heatmap loss, PAF loss and
keypoints offset loss.

Figure 3. Robust 2.5D Structured Pose Estimation

###  3.2. Depth Regression

Generally, the human depth estimated from a single camera view is inherently
ambiguous. This ambiguity will make it hard to learn an accurate regression
model. To handle this issue, we exploit the capability of probability
prediction to capture the tolerance of depth ambiguity by predicting a depth
distribution instead of an absolute depth, which has also been used in the
field of object detection (He et al., 2019). Here, we assume that the depth
distribution of each person is independent and follows a Laplace distribution
$L({z}^{\text{reg}}_{\text{root}},\ \lambda)$. The probability density
function of a Laplace random variable $Z\sim
L({Z}^{\text{reg}}_{\text{root}},\ \lambda)$ is:

(4)
$p_{\text{reg}}(Z)=\frac{1}{2\lambda}e^{-\frac{|Z-{Z}^{\text{reg}}_{\text{root}}|}{\lambda}}$

where $\mu$ and $\lambda$ are parameters of the Laplace distribution. The
ground-truth depth can also be formulated as a Laplace distribution. Since it
is deterministic, it can be further represented by a Dirac delta function:

(5) $p_{D}(Z)=\delta(Z-\hat{Z}_{\text{root}}),$

where $\hat{Z}_{\text{root}}$ represents the ground truth of depth. The
distance between the distribution of direct depth and the ground truth is
measured by the Kullback Leibler(KL) Divergence.

$\displaystyle{L}_{\text{reg}}$
$\displaystyle=D_{KL}(p_{D}(Z)\|p_{\text{reg}}(Z))$ (6)
$\displaystyle\propto\frac{|{Z}^{\text{reg}}_{\text{root}}-\hat{Z}_{\text{root}}|_{1}}{\sigma_{1}}+\log(\sigma_{1}),$

where $\sigma_{1}=\sqrt{2}\lambda$ is the standard deviation of the Laplace
distribution to indicate the depth uncertainty. If the model lacks confidence
in its prediction, it will output a larger $\sigma$ so that $L_{\text{reg}}$
can be reduced. The term $\log(\sigma)$ avoids trivial solutions and
encourages the model to be optimistic about accurate predictions. Detailed
derivation is provided in Supplementary.

###  3.3. Geometry-Aware Depth Reasoning

The inherent ambiguity makes it hard to learn an accurate regression model.
Given the difficulty to regress depth directly, we develop a geometry depth
reasoning method that exploits the mutual benefits of both 2.5D pose and
camera-centric root depths. This method first uses 2.5D pose and geometry
information to infer camera-centric root depths in a forward pass, and then
exploits the root depths to improve representation learning of 2.5D pose
estimation in a backward pass.

####  3.3.1. Geometry-based Forward Pass

Once the 2.5D pose ${\bf{P}}^{2.5D}$ has been determined by Section 3.1, the
2D image coordinates should be back-projected to the camera-centred coordinate
space using the estimated depth value to get the final coordinates of
${\bf{P}}^{3D}$. Given a 2.5D pose ${\bf{P}}^{2.5D}$, the intrinsic camera
parameters $K$, and a torso length $\Omega$, the geometry reasoning depth can
be inferred as follows:

(7)
${Z}^{\text{geo}}_{\text{root}}=\mathop{\arg\min}\limits_{{Z}_{\text{root}}}\|d(f({Z_{\text{root}},{\bf{P}}^{2.5D}_{\text{n}}),f({Z}_{\text{root}},{\bf{P}}^{2.5D}_{\text{m}}))-\Omega\|_{2}^{2}}$

where the function $d(\cdot,\cdot)$ measures the distance between the root
joint and neck joint in the camera-centric space. Eq.7 leads to a closed-form
solution as follows:

(8) ${Z}^{\text{geo}}_{\text{root}}=\frac{-b+\sqrt{b^{2}-4ac}}{2a},$

where

$a=(f_{x}^{-1}\Delta_{u_{m}})^{2}+(f_{y}^{-1}\Delta_{v_{m}})^{2},$
$b=2Z_{m}^{r}[f_{x}^{-2}\Delta_{u_{m}}(u_{m}-c_{x})+f_{y}^{-2}\Delta_{v_{m}}(v_{m}-c_{y})],$
$c=(Z_{m}^{r})^{2}[f_{x}^{-2}(u_{m}-c_{x})^{2}+f_{y}^{-2}(v_{m}-c_{y})^{2}+1]^{2}-\Omega^{2}.$

Detailed derivation is provided in Supplementary. To optimize the final
geometry aware depth distribution, we apply the uncertainty regression loss
same as 3.2:

(9)
${L}_{\text{geo}}=\frac{|{Z}^{\text{geo}}_{\text{root}}-\hat{Z}_{\text{root}}|}{\sigma_{2}}+\log(\sigma_{2}).$

####  3.3.2. Geometry-based Backward Pass

To further improve the representation learning of 2.5D pose estimation, we
develop an end-to-end differentiable method using the geometry depth
information. The geometry depth, which is deduced by using 2.5D pose, can be
used to further update the feature extractor in a backward process. Since the
geometry-based root depth ${\bf{Z}}^{\text{geo}}_{\text{root}}$ can expressed
in a closed-form solution with respect to the 2.5D pose ${\bf{P}}^{2.5D}_{k}$
as shown in Eq.8, we can directly compute the backward derivative as follows:

$\displaystyle\frac{\partial{L}_{\text{geo}}}{\partial{\bf{P}}^{2.5D}_{k}}=\frac{\partial{L}_{\text{geo}}}{\partial{Z}^{\text{geo}}_{\text{root}}}\frac{\partial{Z}^{\text{geo}}_{\text{root}}}{\partial{\bf{P}}^{2.5D}_{k}}$
$\displaystyle=({Z^{\text{geo}}_{\text{root}}-\hat{Z}_{\text{root}})\cdot(\frac{\partial{Z}^{\text{geo}}_{\text{root}}}{\partial
a}\frac{\partial a}{\partial{\bf{P}}^{2.5D}_{k}}}$ (10)
$\displaystyle+\frac{\partial{Z}^{\text{geo}}_{\text{root}}}{\partial
b}\frac{\partial
b}{\partial{\bf{P}}^{2.5D}_{k}}+\frac{\partial{Z}^{\text{geo}}_{\text{root}}}{\partial
c}\frac{\partial c}{\partial{\bf{P}}^{2.5D}_{k}}).$

###  3.4. Adaptive Fusion for Depth Disambiguation

Existing approaches either use a neural network to regress the 3D depth of a
person from the cropped image based on the body size, or they directly regress
the depth based on visual perception of the object scale. However, both
methods focus on the local and global cues independently; they do not consider
any synergies between them. Moreover, direct depth regression will result in
satisfactory depth estimation, but that result may be inaccurate in some
occluded or crowded scenes. A key insight into solving the depth ambiguity
issue is, therefore, to design a mutual adaptive reasoning depth directly from
the visual perception that also includes a geometry-based reasoning depth.
Doing so should also bridge the gap between 2.5D pose representation and depth
estimation. Therefore, the loss function to train the whole network is as
follows:

(11) ${L}={L}_{\text{geo}}+\omega{L}_{\text{reg}},$

where $\omega$ is a hyperparameter to trade-off between the geometry loss and
regression loss.

At this point, we have two depths estimated from direct depth regression and
geometry reasoning. The uncertainty of the direct regressed depth and the
geometry aware depth is used to fuse the two depths, expressed as follows:

(12)
${Z}^{\text{fusion}}_{\text{root}}=(\frac{{Z}_{\text{root}}^{\text{reg}}}{\sigma_{1}}+\frac{{Z}^{\text{geo}}_{\text{root}}}{\sigma_{2}})/(\frac{1}{\sigma_{1}}+\frac{1}{\sigma_{2}}),$

where $\sigma_{1}$ is the direct regress uncertainty and $\sigma_{2}$ is the
geometry reasoning uncertainty.

##  4\. Experiments

|  | All people | Matched
---|---|---|---
Scheme | Method | $\text{PCK}_{\text{rel}}$ | $\text{PCK}_{\text{abs}}$ | $\text{PCK}_{\text{root}}$ | PCOD | $\text{PCK}_{\text{rel}}$ | $\text{PCK}_{\text{abs}}$
Top-down | Rogez et al. (Rogez et al., 2017) |  53.8 |  - |  - |  - |  62.4 |  -
Rogez et al. (Rogez et al., 2019) |  70.6 |  - |  - |  - |  74.0 |  -
Dabral et al. (Dabral et al., 2019) |  71.3 |  - |  - |  - |  74.2 |  -
Moon et al. (Moon et al., 2019) |  81.8 |  31.5 |  - |  92.6 |  82.5 |  31.8
Lin et al. (Lin and Lee, 2020) |  - |  - |  - |  - |  83.7 |  35.2
Wang et al. (Wang et al., 2020a) |  - |  - |  - |  - |  82.0 |  43.8
Guo et al. (Guo et al., 2021) |  82.5 |  - |  - |  - |  83.9 |  35.2
Bottom-up | Mehta et al. (Mehta et al., 2018) |  65.0 |  - |  - |  - |  69.8 |  -
Mehta et al. (Mehta et al., 2020) |  70.4 |  - |  - |  - |  75.8 |  -
Benzine et al. (Benzine et al., 2020) |  72.0 |  - |  - |  - |  - |  -
Benzine et al. (Benzine et al., 2021) |  67.5 |  19.8 |  - |  - |  72.7 |  20.9
Zhen et al. (Zhen et al., 2020) |  73.5 |  35.4 |  45.5 |  97.0 |  80.5 |  38.7
Ours |  76.2 |  36.3 |  48.9 |  97.1 |  81.5 |  39.5
Table 1. Results on the MuPoTS-3D dataset. All numbers are average values over
20 activities. “-” means the results that are not available. Best in bold.

###  4.1. Datasets and Evaluation Metrics

MuCo-3DHP and MuPoTS Datasets. MuCo-3DHP and MuPoTS-3D, two datasets proposed
by (Mehta et al., 2018), were used to assess the framework’s ability to
estimate multi-person 3D poses. MuCo-3DHP, the training set we used is a
large-scale synthesized dataset, which was generated from single person 3D
pose estimation dataset MPI-INF-3DHP (Mehta et al., 2017) by randomly
compositing the persons. We use the same set of MuCo-3DHP synthesized images
from (Moon et al., 2019) for a fair comparison. 400K frames of MuCo-3DHP are
used for training, among which half are background augmented. We used
MuPoTS-3D datasets as our test set which is a real-world outdoor scenes
dataset captured by a marker-less motion capture system. Besides, an
additional 2D human keypoint dataset COCO is used to train together with the
MuCo-3DHP dataset following Mehta et al. (Moon et al., 2019). Accordingly, we
set the loss value of depth becomes zero when the COCO dataset was imported.

Human3.6M Dataset. The Human3.6M dataset (Ionescu et al., 2013) is currently
the largest publicly available dataset for human 3D pose estimation. Two
experimental protocols are widely used for training and testing. _Protocol_ 1
uses S1, S5, S6, S7, S8, S9 in training and S11 in testing, while _Protocol_ 2
uses S1, S5, S6, S7, S8 in training and S9, S11 in testing. Same as the
configuration of MuCo-3DHP dataset, additional 2D human keypoint dataset COCO
is used to train together with the Human3.6M dataset. Accordingly, we set the
loss value of depth becomes zero when COCO dataset was imported. Following
previous work (Moon et al., 2019), we use _Protocol_ 2 and sample every 5th
and 64th frames in videos for training and testing respectively.

COCO Dataset. The COCO dataset (Lin et al., 2014) contains over 250, 000
person instances labeled and 200,000 images with 17 keypoints annotations.
COCO dataset is divided into three sets named train, val and test-dev,
containing 57k, 5k and 20k images respectively. In this paper, we used
train2017 as an additional 2D human keypoint dataset. Accordingly, we set the
loss value of depth becomes zero followed previous work (Moon et al., 2019)
when COCO dataset was imported.

Evaluation Metrics. Although our task is 3D-MPE at camera-centric coordinates,
we also perform person-centric 3D-MPE evaluation metrics. We use Percentage of
Correct 3D Keypoints(PCK) to evaluate the performance of 3D-MPE on MuPoTS-3D,
which calculate the percentage of correct joints if it lies within 15cm from
the ground truth joint location. Following (Moon et al., 2019), We report the
relative 3D$\text{PCK}_{\text{rel}}$ that with root alignment to evaluate the
person-centric 3D-MPE, and absolute 3D$\text{PCK}_{\text{abs}}$ that without
root alignment to evaluate the camera-centric 3D-MPE. To compare the human
depth location ability, we evaluate $\text{PCK}_{\text{root}}$ that only
measures the accuracy of root joints and percentage of correct ordinal depth
(PCOD) that measures the accuracy of ordinal depth following (Zhen et al.,
2020).

###  4.2. Implementation Details

We use HRNet-w32 (Sun et al., 2019; Wang et al., 2020b) pre-trained on the
ImageNet dataset as our backbone network and Adam as our optimizer with a
$5\times 10^{-4}$ learning rate and a $10^{-6}$ weight decay. All input images
were padded to the same size of 512 × 832. Every prediction head attached to
the backbone consists of one $3\times 3\times 256$ conv layer, BatchNorm, ReLU
and another $1\times 1\times c_{0}$ conv layer, where $c_{0}$ is the output
size. Following (Chen et al., 2018), we implemented multi-scale supervision
from the four scale output of HRNet-w32 at the heatmap and part affinity field
head. The model was trained for 50k iterations with a batch size of 32 on four
RTX 3090 GPUs; $50\%$ of data in each mini-batch was from COCO dataset (Lin et
al., 2014). The data augmentation process during the training included
rotation, horizontal flips, and color jittering.

###  4.3. Quantitative Evaluation on MuPoTS-3D

To evaluate the performance of 3D-MPE in complex scenarios, we perform
experiments on MuPoTS-3D and compare it to current state-of-the-art methods,
as shown in Table 1. After root alignment with the ground-truth poses, our
model achieves higher performance and outperforms the SOTA(Zhen et al., 2020)
on $\text{PCK}_{\text{rel}}$ by $1.3\%$. The results show that our person-
centric 3D-MPE outperforms the SOTA on $\text{PCK}_{\text{rel}}$ by $2.7\%$ so
that our structured 2.5D Pose can significantly improve the performance of the
2.5D pose. Additionally, our method is superior to the most top-down methods
and comparable with the SOTA top-down methods on person-centric 3D-MPE metrics
for matched ground truths. Since the existing top-down methods require an
additional SOTA human detector, such as Mask R-CNN model (He et al., 2017),
the metrics on $\text{PCK}_{\text{rel}}$ show an advantage for recognizing
instances, but at the expense of computational complexity. We also perform
camera-centric evaluation using $\text{PCK}_{\text{abs}}$, where we outperform
the SOTA by 0.9%. To evaluate our root joint localization performance, we
additionally calculate $\text{PCK}_{\text{root}}$ to evaluate the 3D
localization of people. Notably, we achieve better $\text{PCK}_{\text{root}}$
than SOTA by a large margin, which shows our method can alleviate inherent
depth ambiguity issues. Furthermore, our method can achieve 97.1% on PCOD. It
clearly indicates our method can correctly identify the ordinal depth and we
analyze this mainly with the help of geometric information. The evaluation on
MuPoTS-3D shows that our method outperforms the state-of-the-art methods in
both 3D pose and 2.5D pose performance.

###  4.4. Quantitative Evaluation on Human3.6M

The Human3.6M dataset has been widely used to evaluate 3D single person pose
estimation. As our task is focused on camera-centric 3D-MPE to handle inter-
person occlusion and depth ambiguity, we do not expect our method to perform
significantly better than the SOTA methods. The MRPE of root joint
localization results on Human3.6M dataset are shown in Table 2. The top 3 rows
are baseline results that take from the figures reported in (Lin and Lee,
2020). The baseline methods (Mehta et al., 2017; Rogez et al., 2017) employed
an optimization process to minimize the error of back-projection between 2D
and 3D pose. “w/o limb joints” refers to optimization using only head and body
trunk joints. “with RANSAC” refers to randomly sampling the set of joints used
for optimization with RANSAC. As shown in Table 1, our method is slightly
superior to the SOTA top-down method (Lin and Lee, 2020). The main reason is
that Human3.6M is an indoor single-person dataset, which is quite clean with
almost no crowded scenarios. Notably, (Moon et al., 2019) and (Lin and Lee,
2020) are used Mask-Rcnn (He et al., 2017) with the help of large detection
datasets, which resulting our method got slightly lower accuracy along x- and
y-axis. Despite this, our method shows better depth location ability at
camera-centric coordinates.

Figure 4. Quality results comparisons with SOTA top-down and bottom-up
methods.TD stand for top-down method (Moon et al., 2019), BU stand for bottom-
up method (Zhen et al., 2020). The first row shows the images from four
challenging frames from MuPoTS-3D which included extreme occlusion cases and
extreme hard poses; the second row shows the results from Posenet(Moon et al.,
2019); the third row shows the result from SMAP (Zhen et al., 2020); the last
row shows the results of our method. The occlusive person is highlighted with
red bounding box. Wrong estimations are highlighted with red solid circles.
Missing estimations are highlighted with red dotted circles. Correct
estimations are highlighted with red arrows.

###  4.5. Qualitative Comparisons on MuPoTS-3D and COCO

Fig. 4 shows the comparison among a typical top-down method Postnet(Moon et
al., 2019), a SOTA bottom-up method SMAP (Zhen et al., 2020) and our method.
We selected four challenging frames from MuPoTS-3D which included extreme
occlusion cases and extreme hard poses. As shown in the second row of Fig. 4,
we observe that although the top-down method achieves high accuracy with PCK
metrics, they suffer from high redundancy for the human detection bounding box
and cannot handle well in occlusion case. Specifically, when a person is
partially occluded, high redundancy detection bounding box will greatly affect
the performance. Besides, SOTA bottom-up method suffers from person scale
variation and also cannot handle partial occlusion cases. As shown in the
third row of Fig. 4, we observe that they cannot detect the partially occluded
person behind as well as cannot accurately regress extreme hard poses. By
contrast, our method can produce reasonable relative poses and reasonable
depth in the above cases as shown in the fourth row of Fig. 4. We also provide
results of the quality results on in-the-wild images from the COCO dataset of
val2017 set as shown in Fig. 5. The results indicate that our method is
applicable in in-the-wild cases.

###  4.6. Running Time Comparisons with SOTA

Table 3 reports the Running Time Comparisons of our approach and the existing
state-of-the-art approaches. In this experiment, the running time is measured
on a single RTX 2070 Super GPU, Intel i7-10700 CPU, and 32 GB RAM with batch
size 1. Since the multi-task structure of (Zhen et al., 2020) and our method,
the run times of the bottom show great superiority over any top-down method.
As shown in Tab. 3, our method achieves superior performance, significantly
faster than the competing methods. Additionally, our method processing time is
roughly constant regardless of the number of people and is more efficient in
terms of model size (#Params) and computation complexity (GFLOPs) as shown in
Fig 3.

Method |  $\text{MRPE}_{x}$ |  $\text{MRPE}_{y}$ |  $\text{MRPE}_{z}$
---|---|---|---
Baseline |  27.5 |  28.3 |  261.9
Baseline w/o limb joints |  24.5 |  24.9 |  220.2
Baseline with RANSAC |  24.3 |  24.3 |  207.1
Moon et al. (Moon et al., 2019)$\ast$ |  23.3 |  23.0 |  108.1
Lin et al. (Lin and Lee, 2020)$\ast$ |  15.6 |  13.6 |  69.9
Ours |  18.8 |  22.9 |  68.8
Table 2. MRPE results on the single person Human3.6M dataset. Best in bold.
$\ast$ means using Top-down manner with the help of large detection datasets.

###  4.7. Ablation Studies

Ablation studies are performed to validate the effectiveness of each branch.
To show how the performance of each branch affects the accuracy of the 2.5D
pose and absolute depth, we compare the performance of
$\text{PCK}_{\text{rel}},\text{PCK}_{\text{abs}}$ and
$\text{PCK}_{\text{root}}$ on each branch separation.

Effect of structured pose estimation. As appearance-based heatmap
representations have a more precise spatial positioning accuracy, from Table
4, we see obvious improvement on $\text{PCK}_{\text{rel}}$ by 2.9% when only
using heatmap representations. Although 2.5D joint offset scheme estimates
show compromising ability at positioning accuracy, they show more robustness
in inferring occluded joints. The result also shows that the offset scheme can
gain an improvement by 1.8% $\text{PCK}_{\text{rel}}$. It is able to infer
invisible joints and encode the information in a more robust way. The above
results show that the performance of our model will degrade severely without
our structured Pose Estimation scheme.

Figure 5. The quality results on in-the-wild images from COCO dataset. Method |  Input size |  Times(ms)-3 people |  Times(ms)-20 people |  FLOPs(G) |  Params(M)
---|---|---|---|---|---
_Top-down methods_
Moon et al. (Moon et al., 2019) |  512 $\times$ 512 |  169.4 |  663.1 |  331.19 |  112.51
Lin et al. (Lin and Lee, 2020) |  512 $\times$ 512 |  132.9 |  452.2 |  306.73 |  105.91
_Bottom-up methods_
Zhen et al. (Zhen et al., 2020) |  832 $\times$ 512 |  95.6 |  96.8 |  176.98 |  91.48
Ours |  832 $\times$ 512 |  43.2 |  43.5 |  112.87 |  32.04
Table 3. Runtime comparisons on a 2070S GPU. Best in bold.

Effect of geometry-based backward. By building a loop from 2.5D pose to depth,
we have deduced a geometry-aware depth from 2.5D pose, which can help the
network to the better penalty the 2.5D representation learning. Notably, the
geometry depth information is used to further improve the representation
learning of 2.5D pose estimation in an end-to-end differentiable manner. Table
4 shows that geometry-based backward pass is able to improve both relative and
absolute pose estimation by 4.1% $\text{PCK}_{\text{root}}$ and 3.6%
$\text{PCK}_{\text{root}}$ improvement.

Effect of adaptive fusion. In Section 3.3, we developed a depth fusion scheme
that leverages both direct visual perception and geometry to alleviate
inherent depth ambiguity issues. $\text{PCK}_{\text{root}}$ is the most direct
metric to reflect the ability of our model for absolute depth localization. As
shown in Table 4, our scheme shows significant improvement on
$\text{PCK}_{\text{root}}$ by $5.8\%$. Since the training and test images of
the MuPoTS-3D dataset are respectively from synthesized and outdoor, the
improvement is more significant in this dataset, which further validates the
robustness for generalization of our method.

##  5\. Conclusion

In this paper, we developed a unified bottom-up model that leverages the
mutual benefits of both 2.5D pose and depth estimation to handle the Monocular
3D-MPE problem. Different from existing top-down or bottom-up methods that
treat camera-centric 3D-MPE as two unrelated subtasks: 2.5D pose
representation and absolute depth estimation, our method can bridge the gap
between 2.5D pose representation and depth estimation and thus benefit from
each other. First, we designed a robust structured 2.5D pose estimation is
designed to recognize inter-person occlusion based on depth relationships.
Additionally, we developed an end-to-end differentiable geometry depth
reasoning method that exploits the mutual benefits of both 2.5D pose and
camera-centric root depths. This method first uses 2.5D pose and geometry
information to infer camera-centric root depths in a forward pass, and then
exploits the root depths to further improve representation learning of 2.5D
pose estimation in a backward pass. Further, we designed a depth fusion scheme
that leverages both visual perception and body geometry to alleviate inherent
depth ambiguity issues. Extensive experiments demonstrate the superiority of
our proposed model over a wide range of bottom-up methods. Our accuracy is
even competitive with top-down counterparts. Notably, our model runs much
faster than existing bottom-up and top-down methods.

Method |  PCK_rel |  PCK_abs |  PCK_root
---|---|---|---
Full model |  81.5 |  39.5 |  48.9
w/o offset |  79.7 |  38.2 |  48.7
w/o heatmap |  78.6 |  38.0 |  48.9
w/o GB backward |  77.4 |  35.9 |  44.7
w/o adaptive fusion |  81.4 |  35.1 |  43.1
Table 4. Ablation study of the structure design on the MuPoTS-3D dataset. GB
backward stand for geometry-based backward. Best in bold.

##  6\. Acknowledgements

This work was supported by the Shanghai Sailing Program (21YF1429400,
22YF1428800), Shanghai Local college capacity building program (22010502800),
NSFC programs (61976138, 61977047), the National Key Research and Development
Program (2018YFB2100500), STCSM (2015F0203-000-06), and SHMEC
(2019-01-07-00-01-E00003).

## References

  * (1)
  * Benzine et al. (2020) Abdallah Benzine, Florian Chabot, Bertrand Luvison, Quoc Cuong Pham, and Catherine Achard. 2020\.  Pandanet: Anchor-based single-shot multi-person 3d pose estimation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 6856–6865.
  * Benzine et al. (2021) Abdallah Benzine, Bertrand Luvison, Quoc Cuong Pham, and Catherine Achard. 2021.  Single-shot 3D multi-person pose estimation in complex images.  _Pattern Recognition_ 112 (2021), 107534.
  * Brasó et al. (2021) Guillem Brasó, Nikita Kister, and Laura Leal-Taixé. 2021\.  The Center of Attention: Center-Keypoint Grouping via Attention for Multi-Person Pose Estimation. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 11853–11863.
  * Cao et al. (2021) Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2021.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields.  _IEEE Transactions on Pattern Analysis & Machine Intelligence_ 43, 01 (2021), 172–186.
  * Cao et al. (2017) Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017\.  Realtime multi-person 2d pose estimation using part affinity fields. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 7291–7299.
  * Chen et al. (2018) Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun. 2018\.  Cascaded pyramid network for multi-person pose estimation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 7103–7112.
  * Cheng et al. (2021a) Yu Cheng, Bo Wang, Bo Yang, and Robby T Tan. 2021a.  Graph and temporal convolutional networks for 3d multi-person pose estimation in monocular videos. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 4. 12.
  * Cheng et al. (2021b) Yu Cheng, Bo Wang, Bo Yang, and Robby T Tan. 2021b.  Monocular 3D Multi-Person Pose Estimation by Integrating Top-Down and Bottom-Up Networks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7649–7659.
  * Cheng et al. (2019) Yu Cheng, Bo Yang, Bo Wang, Wending Yan, and Robby T Tan. 2019.  Occlusion-aware networks for 3d human pose estimation in video. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 723–732.
  * Dabral et al. (2019) Rishabh Dabral, Nitesh B Gundavarapu, Rahul Mitra, Abhishek Sharma, Ganesh Ramakrishnan, and Arjun Jain. 2019.  Multi-person 3d human pose estimation from monocular images. In _2019 International Conference on 3D Vision (3DV)_. IEEE, 405–414.
  * Ding et al. (2020) Mingyu Ding, Yuqi Huo, Hongwei Yi, Zhe Wang, Jianping Shi, Zhiwu Lu, and Ping Luo. 2020.  Learning depth-guided convolutions for monocular 3d object detection. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_. 1000–1001.
  * Fabbri et al. (2020) Matteo Fabbri, Fabio Lanzi, Simone Calderara, Stefano Alletto, and Rita Cucchiara. 2020\.  Compressed volumetric heatmaps for multi-person 3d pose estimation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7204–7213.
  * Fang et al. (2017) Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. 2017\.  Rmpe: Regional multi-person pose estimation. In _Proceedings of the IEEE international conference on computer vision_. 2334–2343.
  * Geng et al. (2021) Zigang Geng, Ke Sun, Bin Xiao, Zhaoxiang Zhang, and Jingdong Wang. 2021.  Bottom-up human pose estimation via disentangled keypoint regression. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 14676–14686.
  * Guo et al. (2021) Wen Guo, Enric Corona, Francesc Moreno-Noguer, and Xavier Alameda-Pineda. 2021.  PI-Net: Pose Interacting Network for Multi-Person Monocular 3D Pose Estimation. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 2796–2806.
  * He et al. (2017) Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017.  Mask r-cnn. In _Proceedings of the IEEE international conference on computer vision_. 2961–2969.
  * He et al. (2019) Yihui He, Chenchen Zhu, Jianren Wang, Marios Savvides, and Xiangyu Zhang. 2019.  Bounding box regression with uncertainty for accurate object detection. In _Proceedings of the ieee/cvf conference on computer vision and pattern recognition_. 2888–2897.
  * Hidalgo et al. (2019) Gines Hidalgo, Yaadhav Raaj, Haroon Idrees, Donglai Xiang, Hanbyul Joo, Tomas Simon, and Yaser Sheikh. 2019.  Single-network whole-body pose estimation. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 6982–6991.
  * Ionescu et al. (2013) Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. 2013.  Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.  _IEEE transactions on pattern analysis and machine intelligence_ 36, 7 (2013), 1325–1339.
  * Iqbal et al. (2020) Umar Iqbal, Pavlo Molchanov, and Jan Kautz. 2020\.  Weakly-supervised 3d human pose learning via multi-view images in the wild. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5243–5252.
  * Jin et al. (2020) Sheng Jin, Wentao Liu, Enze Xie, Wenhai Wang, Chen Qian, Wanli Ouyang, and Ping Luo. 2020.  Differentiable hierarchical graph grouping for multi-person pose estimation. In _European Conference on Computer Vision_. Springer, 718–734.
  * Kocabas et al. (2019) Muhammed Kocabas, Salih Karagoz, and Emre Akbas. 2019\.  Self-supervised learning of 3d human pose using multi-view geometry. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 1077–1086.
  * Li et al. (2019) Wenbo Li, Zhicheng Wang, Binyi Yin, Qixiang Peng, Yuming Du, Tianzi Xiao, Gang Yu, Hongtao Lu, Yichen Wei, and Jian Sun. 2019\.  Rethinking on multi-stage networks for human pose estimation.  _arXiv preprint arXiv:1901.00148_ (2019).
  * Lin and Lee (2020) Jiahao Lin and Gim Hee Lee. 2020.  Hdnet: Human depth estimation for multi-person camera-space localization. In _European Conference on Computer Vision_. Springer, 633–648.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context. In _European conference on computer vision_. Springer, 740–755.
  * Mehta et al. (2017) Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko, Weipeng Xu, and Christian Theobalt. 2017.  Monocular 3d human pose estimation in the wild using improved cnn supervision. In _2017 international conference on 3D vision (3DV)_. IEEE, 506–516.
  * Mehta et al. (2020) Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Mohamed Elgharib, Pascal Fua, Hans-Peter Seidel, Helge Rhodin, Gerard Pons-Moll, and Christian Theobalt. 2020.  XNect: Real-time multi-person 3D motion capture with a single RGB camera.  _ACM Transactions on Graphics (TOG)_ 39, 4 (2020), 82–1.
  * Mehta et al. (2018) Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Srinath Sridhar, Gerard Pons-Moll, and Christian Theobalt. 2018.  Single-shot multi-person 3d pose estimation from monocular rgb. In _2018 International Conference on 3D Vision (3DV)_. IEEE, 120–130.
  * Moon et al. (2019) Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee. 2019\.  Camera distance-aware top-down approach for 3d multi-person pose estimation from a single rgb image. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 10133–10142.
  * Newell et al. (2017) Alejandro Newell, Zhiao Huang, and Jia Deng. 2017.  Associative embedding: End-to-end learning for joint detection and grouping.  _Advances in neural information processing systems_ 30 (2017).
  * Nie et al. (2019) Xuecheng Nie, Jiashi Feng, Jianfeng Zhang, and Shuicheng Yan. 2019.  Single-stage multi-person pose machines. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 6951–6960.
  * Papandreou et al. (2018) George Papandreou, Tyler Zhu, Liang-Chieh Chen, Spyros Gidaris, Jonathan Tompson, and Kevin Murphy. 2018\.  Personlab: Person pose estimation and instance segmentation with a bottom-up, part-based, geometric embedding model. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 269–286.
  * Rogez et al. (2017) Gregory Rogez, Philippe Weinzaepfel, and Cordelia Schmid. 2017\.  Lcr-net: Localization-classification-regression for human pose. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 3433–3441.
  * Rogez et al. (2019) Gregory Rogez, Philippe Weinzaepfel, and Cordelia Schmid. 2019\.  Lcr-net++: Multi-person 2d and 3d pose detection in natural images.  _IEEE transactions on pattern analysis and machine intelligence_ 42, 5 (2019), 1146–1161.
  * Sun et al. (2019) Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. 2019\.  Deep high-resolution representation learning for human pose estimation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5693–5703.
  * Sun et al. (2018) Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei. 2018.  Integral human pose regression. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 529–545.
  * Véges and Lőrincz (2019) Márton Véges and András Lőrincz. 2019.  Absolute human pose estimation with depth prediction network. In _2019 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 1–7.
  * Wandt and Rosenhahn (2019) Bastian Wandt and Bodo Rosenhahn. 2019.  Repnet: Weakly supervised training of an adversarial reprojection network for 3d human pose estimation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7782–7791.
  * Wang et al. (2020a) Can Wang, Jiefeng Li, Wentao Liu, Chen Qian, and Cewu Lu. 2020a.  Hmor: Hierarchical multi-person ordinal relations for monocular multi-person 3d pose estimation. In _European Conference on Computer Vision_. Springer, 242–259.
  * Wang et al. (2020b) Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. 2020b.  Deep high-resolution representation learning for visual recognition.  _IEEE transactions on pattern analysis and machine intelligence_ (2020).
  * Zeng et al. (2021) Ailing Zeng, Xiao Sun, Lei Yang, Nanxuan Zhao, Minhao Liu, and Qiang Xu. 2021\.  Learning skeletal graph neural networks for hard 3d pose estimation. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 11436–11445.
  * Zhen et al. (2020) Jianan Zhen, Qi Fang, Jiaming Sun, Wentao Liu, Wei Jiang, Hujun Bao, and Xiaowei Zhou. 2020.  Smap: Single-shot multi-person absolute 3d pose estimation. In _European Conference on Computer Vision_. Springer, 550–566.
  * Zhou et al. (2019) Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. 2019\.  Objects as points.  _arXiv preprint arXiv:1904.07850_ (2019).