Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Image
Gyeongsik Moon Ju Yong Chang Kyoung Mu Lee
cs.CV


[
On Weissler's conjecture on the Hamming cube I
    [
    December 30, 2023
==============================================





    < g r a p h i c s >


figure

Qualitative results of applying our 3D multi-person pose estimation framework to COCO dataset <cit.> which consists of in-the-wild images. Most of the previous 3D human pose estimation studies mainly focused on the root-relative 3D single-person pose estimation. In this study, we propose a general 3D multi-person pose estimation framework that takes into account all factors including human detection and 3D human root localization.




]

On Weissler's conjecture on the Hamming cube I
    [
    December 30, 2023
==============================================

empty



Although significant improvement has been achieved recently in 3D human pose estimation, most of the previous methods only treat a single-person case. 
In this work, we firstly propose a fully learning-based, camera distance-aware top-down approach for 3D multi-person pose estimation from a single RGB image. 
The pipeline of the proposed system consists of human detection, absolute 3D human root localization, and root-relative 3D single-person pose estimation modules.
Our system achieves comparable results with the state-of-the-art 3D single-person pose estimation models without any groundtruth information and significantly outperforms previous 3D multi-person pose estimation methods on publicly available datasets. The code is available in [<https://github.com/mks0601/3DMPPE_ROOTNET_RELEASE>],[<https://github.com/mks0601/3DMPPE_POSENET_RELEASE>].






§ INTRODUCTION


The goal of 3D human pose estimation is to localize semantic keypoints of single or multiple human bodies in 3D space. 
It is an essential technique for human behavior understanding and human-computer interaction. 
Recently, many methods <cit.> utilize deep convolutional neural networks (CNNs) and have achieved noticeable performance improvement on large-scale publicly available datasets <cit.>.

Most of the previous 3D human pose estimation methods <cit.> are designed for single-person case. 
They crop the human area in an input image with a groundtruth bounding box or the bounding box that is predicted from a human detection model <cit.>. 
The cropped patch of a human body is fed into the 3D pose estimation module, which then estimates the 3D location of each keypoint. As their models take a single cropped image, estimating the absolute camera-centered coordinate of each keypoint is difficult. 
To handle this issue, many methods <cit.> estimate the relative 3D pose to a reference point in the body, e.g., the center joint (i.e., pelvis) of a human, called root. 
The final 3D pose is obtained by adding the 3D coordinates of the root to the estimated root-relative 3D pose. 
Prior information on the bone length <cit.> or the groundtruth <cit.> has been commonly used for the localization of the root.


Recently, many top-down approaches <cit.> for the 2D multi-person pose estimation have shown noticeable performance improvement. 
These approaches first detect humans by using a human detection module, and then estimate the 2D pose of each human by a 2D single-person pose estimation module. 
Although they are straightforward when used in 2D cases, extending them to 3D cases is nontrivial. 
Note that for the estimation of 3D multi-person poses, we need to know the absolute distance to each human from the camera as well as the 2D bounding boxes. 
However, existing human detectors provide 2D bounding boxes only. 

In this study, we propose a general framework for 3D multi-person pose estimation. 
To the best of our knowledge, this study is the first to propose a fully learning-based camera distance-aware top-down approach of which components are compatible with most of the previous human detection and 3D human pose estimation methods. 
The pipeline of the proposed system consists of three modules. First, a human detection network (DetectNet) detects the bounding boxes of humans in an input image. 
Second, the proposed 3D human root localization network (RootNet) estimates the camera-centered coordinates of the detected humans' roots. Third, a root-relative 3D single-person pose estimation network (PoseNet) estimates the root-relative 3D pose for each detected human. Figures <ref> and <ref> show the qualitative results and overall pipeline of our framework, respectively.

We show that our approach outperforms previous 3D multi-person pose estimation methods <cit.> on several publicly available 3D single- and multi-person pose estimation datasets <cit.> by a large margin. Also, even without any groundtruth information (i.e., the bounding boxes and the 3D location of the roots), our method achieves comparable performance with the state-of-the-art 3D single-person pose estimation methods that use the groundtruth in the inference time. 
Note that our framework is new but follows previous conventions of object detection and 3D human pose estimation networks. 
Thus, previous detection and pose estimation methods can be easily plugged into our framework, which makes the proposed framework quite flexible and generalizable. 

Our contributions can be summarized as follows.



  * We propose a new general framework for 3D multi-person pose estimation from a single RGB image. 
The framework is the first fully learning-based, camera distance-aware top-down approach, of which components are compatible with most of the previous human detection and 3D human pose estimation models.


  * Our framework outputs the absolute camera-centered coordinates of multiple humans' keypoints. 
For this, we propose a 3D human root localization network (RootNet). 
This model makes it easy to extend the 3D single-person pose estimation techniques to the absolute 3D pose estimation of multiple persons.


  * We show that our method significantly outperforms previous 3D multi-person pose estimation methods on several publicly available datasets. 
Also, it achieves comparable performance with the state-of-the-art 3D single-person pose estimation methods without any groundtruth information.




§ RELATED WORKS


2D multi-person pose estimation.
There are two main approaches in the multi-person pose estimation. 
The first one, top-down approach, deploys a human detector that estimates the bounding boxes of humans. 
Each detected human area is cropped and fed into the pose estimation network. 
The second one, bottom-up approach, localizes all human body keypoints in an input image first, and then groups them into each person using some clustering techniques. 

 <cit.> are based on the top-down approach. Papandreou  <cit.> predicted 2D offset vectors and 2D heatmaps for each joint. 
They fused the estimated vectors and heatmaps to generate highly localized heatmaps. Chen  <cit.> proposed a cascaded pyramid network whose cascaded structure refines an initially estimated pose by focusing on hard keypoints. Xiao  <cit.> used a simple pose estimation network that consists of a deep backbone network and several upsampling layers.

 <cit.> are based on the bottom-up approach. Cao  <cit.> proposed the part affinity fields (PAFs) that model the association between human body keypoints. 
They grouped the localized keypoints of all persons in the input image by using the estimated PAFs. Newell  <cit.> introduced a pixel-wise tag value to assign localized keypoints to a certain human. Kocabas  <cit.> proposed a pose residual network for assigning detected keypoints to each person.




3D single-person pose estimation.
Current 3D single-person pose estimation methods can be categorized into single- and two-stage approaches. 
The single-stage approach directly localizes the 3D body keypoints from the input image. 
The two-stage methods utilize the high accuracy of 2D human pose estimation. They initially localize body keypoints in a 2D space and lift them to a 3D space.

 <cit.> are based on the single-stage approach. 
Li  <cit.> proposed a multi-task framework that jointly trains both the pose regression and body part detectors. Tekin  <cit.> modeled high-dimensional joint dependencies by adopting an auto-encoder structure. 
Pavlakos  <cit.> extended the U-net shaped network to estimate a 3D heatmap for each joint. They used a coarse-to-fine approach to boost performance. Sun  <cit.> introduced compositional loss to consider the joint connection structure. 
Sun  <cit.> used soft-argmax operation to obtain the 3D coordinates of body joints in a differentiable manner.

 <cit.> are based on the two-stage approach. Park  <cit.> estimated the initial 2D pose and utilized it to regress the 3D pose. Martinez  <cit.> proposed a simple network that directly regresses the 3D coordinates of body joints from 2D coordinates. Zhou  <cit.> proposed a geometric loss to facilitate weakly supervised learning of the depth regression module with images in the wild. 
Yang  <cit.> utilized adversarial loss to handle the 3D human pose estimation in the wild.


3D multi-person pose estimation.
Few studies have been conducted on 3D multi-person pose estimation from a single RGB image. Rogez  <cit.> proposed a top-down approach called LCR-Net, which consists of localization, classification, and regression parts.
The localization part detects a human from an input image, and the classification part classifies the detected human into several anchor-poses.
The anchor-pose is defined as a pair of 2D and root-relative 3D pose. It is generated by clustering poses in the training set.
Then, the regression part refines the anchor-poses. Mehta  <cit.> proposed a bottom-up approach system.
They introduced an occlusion-robust pose-map formulation which supports pose inference for more than one person through PAFs <cit.>.


3D human root localization in 3D multi-person pose estimation.
Rogez  <cit.> estimated both the 2D pose in the image coordinate space and the 3D pose in the camera-centered coordinate space simultaneously. 
They obtained the 3D location of the human root by minimizing the distance between the estimated 2D pose and projected 3D pose, similar to what Mehta  <cit.> did. 
However, this strategy cannot be generalized to other 3D human pose estimation methods because it requires both the 2D and 3D estimations. For example, many works <cit.> estimate the 2D image coordinates and root-relative depth values of keypoints. 
As their methods do not output root-relative camera-centered coordinates of keypoints, such a distance minimization strategy cannot be used. 
Moreover, contextual information cannot be exploited because the image feature is not considered. For example, it cannot distinguish between a child close to the camera and an adult far from the camera because their scales in the 2D image is similar.






§ OVERVIEW OF THE PROPOSED MODEL







The goal of our system is to recover the absolute camera-centered coordinates of multiple persons' keypoints {𝐏^abs_j}_j=1^J, where J denotes the number of joints. To address this problem, we construct our system based on the top-down approach that consists of DetectNet, RootNet, and PoseNet. 
The DetectNet detects a human bounding box of each person in the input image. 
The RootNet takes the cropped human image from the DetectNet and localizes the root of the human 𝐑=(x_R,y_R,Z_R), in which x_R and y_R are pixel coordinates, and Z_R is an absolute depth value. 
The same cropped human image is fed to the PoseNet, which estimates the root-relative 3D pose 𝐏^rel_j=(x_j,y_j,Z^rel_j), in which x_j and y_j are pixel coordinates in the cropped image space and Z^rel_j is root-relative depth value. 
We convert Z^rel_j into Z^abs_j by adding Z_R and transform x_j and y_j to the original input image space. 
Then, the final absolute 3D pose {𝐏^abs_j}_j=1^J is obtained by simple back-projection.






§ DETECTNET


We use Mask R-CNN <cit.> as the framework of DetectNet. 
Mask R-CNN <cit.> consists of three parts. 
The first one, backbone, extracts useful local and global features from the input image by using deep residual network (ResNet) <cit.> and feature pyramid network <cit.>. 
Based on the extracted features, the second part, region proposal network, proposes human bounding box candidates. 
The RoIAlign layer extracts the features of each proposal and passes them to the third part, which is the classification head network. 
The head network determines whether the given proposal is a human or not and estimates the bounding box refinement offsets. 
It achieves the state-of-the-art performance on publicly available object detection datasets <cit.>. 
Due to its high performance and publicly available code <cit.>, we use Mask R-CNN <cit.> as a DetectNet in our pipeline.






§ ROOTNET




 §.§ Model design

The RootNet estimates the camera-centered coordinates of the human root 𝐑=(x_R,y_R,Z_R) from a cropped human image. 
To obtain them, RootNet separately estimates the 2D image coordinates (x_R,y_R) and the depth value (i.e., the distance from the camera Z_R) of the human root. 
The estimated 2D image coordinates are back-projected to the camera-centered coordinate space using the estimated depth value, which becomes the final output. 



Considering that an image provides sufficient information on where the human root is located in the image space, the 2D estimation part can learn to localize it easily. 
By contrast, estimating the depth only from a cropped human image is difficult because the input does not provide information on the relative position of the camera and human. 
To resolve this issue, we introduce a new distance measure, k, which is defined as follows:

    k=√(α_xα_yA_real/A_img),

where α_x, α_y, A_real, and A_img are focal lengths divided by the per-pixel distance factors (pixel) of x- and y-axes, the area of the human in real space (mm^2), and image space (pixel^2), respectively. 
k approximates the absolute depth from the camera to the object using the ratio of the actual area and the imaged area of it, given camera parameters. 
Eq <ref> can be easily derived by considering a pinhole camera projection model. 
The distance d (mm) between the camera and object can be calculated as follows:

    d=α_xl_x,real/l_x,img = α_yl_y,real/l_y,img,

where l_x,real, l_x,img, l_y,real, l_y,img are the lengths of an object in real space (mm) and in image space (pixel), on the x and y-axes, respectively. 
By multiplying the two representations of d in Eq <ref> and taking the square root of it, we can have the 2D extended version of depth measure k in Eq <ref>. 
Assuming that A_real is constant and using α_x and α_y from datasets, the distance between the camera and an object can be measured from the area of the bounding box. 
As we only consider humans, we assume that A_real is 2000mm × 2000mm. 
The area of the human bounding box is used as A_img after extending it to fixed aspect ratio (i.e., height:width = 1:1). 
Figure <ref> shows that such an approximation provides a meaningful correlation between k and the real depth values of the human root in 3D human pose estimation datasets <cit.>.



Although k can represent how far the human is from the camera, it can be wrong in several cases because it assumes that A_img is an area of A_real (i.e., 2000mm × 2000mm) in the image space when the distance between the human and the camera is k. 
However, as A_img is obtained by extending the 2D bounding box, it can have a different value according to its appearance, although the distance to the camera is the same. 
For example, as shown in Figure <ref>(a), two humans have different A_img although they are at the same distance to the camera. 
On the other hand, in some cases, A_img can be the same, even with different distances from the camera. 
For example, in Figure <ref>(b), a child and an adult have similar A_img however, the child is closer to the camera than the adult. 

To handle this issue, we design the RootNet to utilize the image feature to correct A_img, eventually k. 
The image feature can give a clue to the RootNet about how much the A_img has to be changed. 
For example, in Figure <ref>(a), the left image can tell the RootNet to increase the area because the human is in a crouching posture. 
Also, in Figure <ref>(b), the right image can tell the RootNet to increase the area because the input image contains a child. 
Specifically, the RootNet outputs the correction factor γ from the image feature. 
The estimated γ is multiplied by the given A_img, which becomes A_img^γ. From A_img^γ, k is calculated and it becomes the final depth value.





 §.§ Camera normalization

Our RootNet outputs correction factor γ only from an input image. Therefore, data from any camera intrinsic parameters (i.e., α_x and α_y) can be used during training and testing. We call this property camera normalization, which makes our RootNet very flexible. For example, in the training stage, data from different α_x and α_y can be used together. Also, in the testing stage, RootNet can be used when α_x and α_y are not available, likely for in-the-wild images. In this case, α_x and α_y can be set to any values α_x^' and α_y^', respectively. Then, estimated Z_R represents distance between an object and camera whose α_x and α_y are α_x^' and α_y^', respectively. 




 §.§ Network architecture

 The network architecture of the RootNet, which comprises three components, is visualized in Figure <ref>. 
 First, a backbone network extracts the useful global feature of the input human image using ResNet <cit.>. 
 Second, the 2D image coordinate estimation part takes a feature map from the backbone part and upsamples it using three consecutive deconvolutional layers with batch normalization layers <cit.> and ReLU activation function. 
 Then, a 1-by-1 convolution is applied to produce a 2D heatmap of the root. Soft-argmax <cit.> extracts 2D image coordinates x_R,y_R from the 2D heatmap. 
 The third component is the depth estimation part. It also takes a feature map from the backbone part and applies global average pooling. 
 Then, the pooled feature map goes through a 1-by-1 convolution, which outputs a single scalar value γ. 
 The final absolute depth value Z_R is obtained by multiplying k with 1/√(γ). 
 In practice, we implemented the RootNet to output γ^'=1/√(γ) directly and multiply it with the k to obtain the absolute depth value Z_R (i.e., Z_R=γ^'k).



 §.§ Loss function

We train the RootNet by minimizing the L1 distance between the estimated and groundtruth coordinates. 
The loss function L_root is defined as follows:

    L_root = 𝐑 - 𝐑^*_1,

where * indicates the groundtruth.




§ POSENET







 §.§ Model design

The PoseNet estimates the root-relative 3D pose 𝐏^rel_j=(x_j,y_j,Z^rel_j) from a cropped human image. 
Many works have been presented for this topic <cit.>. 
Among them, we use the model of Sun  <cit.>, which is the current state-of-the-art method. 
This model consists of two parts. The first part is the backbone, which extracts a useful global feature from the cropped human image using ResNet <cit.>. 
Second, the pose estimation part takes a feature map from the backbone part and upsamples it using three consecutive deconvolutional layers with batch normalization layers <cit.> and ReLU activation function. 
A 1-by-1 convolution is applied to the upsampled feature map to produce the 3D heatmaps for each joint. The soft-argmax operation is used to extract the 2D image coordinates (x_j,y_j), and the root-relative depth values Z^rel_j.



 §.§ Loss function

We train the PoseNet by minimizing the L1 distance between the estimated and groundtruth coordinates. The loss function L_pose is defined as follows:

    L_pose = 1/J∑_j=1^J𝐏_j^rel - 𝐏_j^rel*_1,

where * indicates groundtruth.





§ IMPLEMENTATION DETAILS


Publicly released Mask R-CNN model <cit.> pre-trained on the COCO dataset <cit.> is used for the DetectNet without fine-tuning on the human pose estimation datasets <cit.>. 
For the RootNet and PoseNet, PyTorch <cit.> is used for implementation. Their backbone part is initialized with the publicly released ResNet-50 <cit.> pre-trained on the ImageNet dataset <cit.>, and the weights of the remaining part are initialized by Gaussian distribution with σ=0.001. 
The weights are updated by the Adam optimizer <cit.> with a mini-batch size of 128. 
The initial learning rate is set to 1× 10^-3 and reduced by a factor of 10 at the 17th epoch. 
We use 256×256 as the size of the input image of the RootNet and PoseNet. 
We perform data augmentation including rotation (±30), horizontal flip, color jittering, and synthetic occlusion <cit.> in training. 
Horizontal flip augmentation is performed in testing for the PoseNet following Sun  <cit.>. 
We train the RootNet and PoseNet for 20 epochs with four NVIDIA 1080 Ti GPUs, which took two days, respectively.




§ EXPERIMENT




 §.§ Dataset and evaluation metric

Human3.6M dataset. 
Human3.6M dataset <cit.> is the largest 3D single-person pose benchmark. 
It consists of 3.6 millions of video frames. 11 subjects performing 15 activities are captured from 4 camera viewpoints. 
The groundtruth 3D poses are obtained using a motion capture system. Two evaluation metrics are widely used. 
The first one is mean per joint position error (MPJPE) <cit.>, which is calculated after aligning the human root of the estimated and groundtruth 3D poses. 
The second one is MPJPE after further alignment (i.e., Procrustes analysis (PA) <cit.>). 
This metric is called PA MPJPE. To evaluate the localization of the absolute 3D human root, we introduce the mean of the Euclidean distance between the estimated coordinates of the root 𝐑 and the groundtruth 𝐑^*, i.e., the mean of the root position error (MRPE), as a new metric:

    MRPE=1/N∑_i=1^N||𝐑^(i)-𝐑^(i)*||_2,

where superscript i is the sample index, and N denotes the total number of test samples.


MuCo-3DHP and MuPoTS-3D datasets.
These are the 3D multi-person pose estimation datasets proposed by Mehta  <cit.>.
The training set, MuCo-3DHP, is generated by compositing the existing MPI-INF-3DHP 3D single-person pose estimation dataset <cit.>. 
The test set, MuPoTS-3D dataset, was captured at outdoors and it includes 20 real-world scenes with groundtruth 3D poses for up to three subjects. 
The groundtruth is obtained with a multi-view marker-less motion capture system. For evaluation, a 3D percentage of correct keypoints (3DPCK_rel) and area under 3DPCK curve from various thresholds (AUC_rel) is used after root alignment with groundtruth. 
It treats a joint's prediction as correct if it lies within a 15cm from the groundtruth joint location. 
We additionally define 3DPCK_abs which is the 3DPCK without root alignment to evaluate the absolute camera-centered coordinates. 
To evaluate the localization of the absolute 3D human root, we use the average precision of 3D human root location (AP_25^root) which considers a prediction is correct when the Euclidean distance between the estimated and the groundtruth coordinates is smaller than 25cm.




 §.§ Experimental protocol

Human3.6M dataset. 
Two experimental protocols are widely used. Protocol 1 uses six subjects (S1, S5, S6, S7, S8, S9) in training and S11 in testing. 
PA MPJPE is used as an evaluation metric. Protocol 2 uses five subjects (S1, S5, S6, S7, S8) in training and two subjects (S9, S11) in testing. 
MPJPE is used as an evaluation metric. 
We use every 5th and 64th frames in videos for training and testing, respectively following  <cit.>. 
When training, besides the Human3.6M dataset, we used additional MPII 2D human pose estimation dataset <cit.> following  <cit.>. 
Each mini-batch consists of half Human3.6M and half MPII data. For MPII data, the loss value of the z-axis becomes zero for both of the RootNet and PoseNet following Sun  <cit.>.
















MuCo-3DHP and MuPoTS-3D datasets.
Following the previous protocol, we composite 400K frames of which half are background augmented. 
For augmentation, we use images from the COCO dataset <cit.> except for images with humans. 
We use an additional COCO 2D human keypoint detection dataset <cit.> when training our models on the MuCo-3DHP dataset following Mehta  <cit.>. 
Each mini-batch consists of half MuCo-3DHP and half COCO data. For COCO data, loss value of z-axis becomes zero for both of the RootNet and PoseNet following Sun  <cit.>.



 §.§ Ablation study

In this study, we show how each component of our proposed framework affects the 3D multi-person pose estimation accuracy. 
To evaluate the performance of the DetectNet, we use the average precision of bounding box (AP^box) following metrics of the COCO object detection benchmark <cit.>. 

Disjointed pipeline.
To demonstrate the effectiveness of the disjointed pipeline (i.e., separated DetectNet, RootNet, and PoseNet), we compare MRPE, MPJPE, and running time of joint and disjointed learning of the RootNet and PoseNet in Table <ref>. 
The running time includes DetectNet and is measured using a single TitanX Maxwell GPU. 
For the joint learning, we combine the RootNet and PoseNet into a single model which shares backbone part (i.e., ResNet <cit.>). 
The image feature from the backbone is fed to each branch of RootNet and PoseNet in a parallel way. Compared with the joint learning, our disjointed learning gives lower error under a similar running time. 
We believe that this is because each task of RootNet and PoseNet is not highly correlated so that jointly training all tasks can make training harder, resulting in lower accuracy.


Effect of the DetectNet.
To show how the performance of the human detection affects the accuracy of the final 3D human root localization and 3D multi-person pose estimation, we compare AP_25^root, AUC_rel, and 3DPCK_abs using the DetectNet in various backbones (i.e., ResNet-50 <cit.>, ResNeXt-101-32 <cit.>) and groundtruth box in the second, third, and fourth row of Table <ref>, respectively. 
The table shows that based on the same RootNet (i.e., Ours), better human detection model improves both of the 3D human root localization and 3D multi-person pose estimation performance. However, the groundtruth box does not improve overall accuracy considerably compared with other DetectNet models. 
Therefore, we have sufficient reasons to believe that the given boxes cover most of the person instances with such a high detection AP. 
We can also conclude that the bounding box estimation accuracy does not have a large impact on the 3D multi-person pose estimation accuracy. 


Effect of the RootNet.
To show how the performance of the 3D human root localization affects the accuracy of the 3D multi-person pose estimation, we compare AUC_rel and 3DPCK_abs using various RootNet settings in Table <ref>. 
The first and second rows show that based on the same DetectNet (i.e., R-50), our RootNet exhibits significantly higher AP_25^root and 3DPCK_abs compared with the setting in which k is directly utilized as a depth value. We use the x and y of the RootNet when the k is used as a depth value. 
This result demonstrates that the RootNet successfully corrects the k value. The fourth and last rows show that the groundtruth human root provides similar AUC_rel, but significantly higher 3DPCK_abs compared with our RootNet. 
This finding shows that better human root localization is required to achieve more accurate absolute 3D multi-person pose estimation results.

Effect of the PoseNet.
All settings in Table <ref> provides similar AUC_rel. Especially, the first and last rows of the table show that using groundtruth box and human root does not provide significantly higher AUC_rel. 
As the results in the table are based on the same PoseNet, we can conclude that AUC_rel, which is an evaluation of the root-relative 3D human pose estimation highly depends on the accuracy of the PoseNet. 











 §.§ Comparison with state-of-the-art methods

Human3.6M dataset.
We compare our proposed system with the state-of-the-art 3D human pose estimation methods on the Human3.6M dataset <cit.> in Tables <ref> and  <ref>. 
As most of the previous methods use the groundtruth information (i.e., bounding boxes or 3D root locations) in inference time, we report the performance of the PoseNet using the groundtruth 3D root location. 
Note that our full model does not require any groundtruth information in inference time. The tables show that our method achieves comparable performance despite not using any groundtruth information in inference time. 
Moreover, it significantly outperforms previous 3D multi-person pose estimation methods <cit.>. 

MuCo-3DHP and MuPoTS-3D datasets.
We compare our proposed system with the state-of-the-art 3D multi-person pose estimation methods on the MuPoTS-3D dataset <cit.> in Tables <ref> and  <ref>. 
The proposed system significantly outperforms them in most of the test sequences and joints. 





§ DISCUSSION


Although our proposed method outperforms previous 3D multi-person pose estimation methods by a large margin, room for improvement is substantial. As shown in Table <ref>, using the groundtruth 3D root location brings significant 3DPCK_abs improvement. Recent advances in depth map estimation from a single RGB image <cit.> can give a clue for improving the 3D human root localization model. 

Our framework can also be used in applications other than 3D multi-person pose estimation. For example, recent methods for 3D human mesh model reconstruction <cit.> reconstruct full 3D mesh model from a single person. Joo  <cit.> utilized 2D multi-view input for 3D multi-person mesh model reconstruction. In our framework, if the PoseNet is replaced with existing human mesh reconstruction model <cit.>, 3D multi-person mesh model reconstruction can be performed from a single RGB image. This shows our framework can be applied to many 3D instance-aware vision tasks which take a single RGB image as an input.









§ CONCLUSION

We propose a novel and general framework for 3D multi-person pose estimation from a single RGB image. Our framework consists of human detection, 3D human root localization, and root-relative 3D single-person pose estimation models. Since any existing human detection and 3D single-person pose estimation models can be plugged into our framework, it is very flexible and easy to use. The proposed system outperforms previous 3D multi-person pose estimation methods by a large margin and achieves comparable performance with 3D single-person pose estimation methods without any groundtruth information while they use it in inference time. To the best of our knowledge, this work is the first to propose a fully learning-based camera distance-aware top-down approach whose components are compatible with most of the previous human detection and 3D human pose estimation models. We hope that this study provides a new basis for 3D multi-person pose estimation, which has only barely been explored.





§ ACKNOWLEDGMENTS

This work was partially supported by the Visual Turing Test project (IITP-2017-0-01780) from the Ministry of Science and ICT of Korea.







Supplementary Material of Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Image





In this supplementary material, we present more experimental results that could not be included in the main manuscript due to the lack of space.



§ DERIVATION OF EQUATION 1


We provide a derivation of Equation 1 of the main manuscript with reference to Figure <ref> ,which shows a pinhole camera model. The green and blue arrows represent the human root joint centered x and y-axes, respectively. The yellow lines show rays, and c is the hole. d, f, and l_sensor are distance between camera and the human root joint (mm), focal length (mm), and the length of human on the image sensor (mm), respectively. 

According to the definition of tan,


    tanθ_x = 0.5l_x,real/d = 0.5l_x,sensor/f,


Let p_x be per pixel distance factor in x-axis. Then,

    d = f l_x,real/l_x,sensor = f p_x l_x,real/l_x,sensor p_x = α_x l_x,real/l_x,img,


Above equations are also valid in y-axis. Therefore, 

    d = f l_y,real/l_y,sensor = f p_y l_y,real/l_y,sensor p_y = α_y l_y,real/l_y,img,


Finally,

    d = √(α_x α_y l_x,real/l_x,imgl_y,real/l_y,img) = √(α_x α_y A_real/A_img).




§ COMPARISON OF 3D HUMAN ROOT LOCALIZATION WITH PREVIOUS APPROACHES






We compare previous absolute 3D human root localization methods <cit.> with the proposed RootNet on the Human3.6M dataset <cit.> based on protocol 2.










Previous approaches <cit.> simultaneously estimate 2D image coordinates and 3D camera-centered root-relative coordinates of keypoints. Then, absolute camera-centered coordinates of the human root are obtained by minimizing the distance between 2D predictions and projected 3D predictions. For optimization, linear least-squares formulation is used. To measure the errors of their method, we implemented and used ResNet-152-based model of Sun  <cit.> as a 2D pose estimator and model of Martinez  <cit.> as a 3D pose estimator, which are state-of-the-art methods. In addition, to minimize the effect of outliers in 3D-to-2D fitting, we excluded limb joints when fitting. Also, we performed RANSAC with a various number of joints to get optimal joint set for fitting instead of using heuristically selected joint set.

Table <ref> shows our RootNet significantly outperforms previous approaches. Furthermore, the RootNet can be designed independently of the PoseNet, giving design flexibility to both models. In contrast, the previous 3D root localization methods <cit.> require both of 2D and 3D predictions for the root localization, which results in lack of generalizability.



§ RUNNING TIME OF THE PROPOSED FRAMEWORK

In Table <ref>, we report seconds per frame for each component of our framework. The running time is measured using a single TitanX Maxwell GPU. As the table shows, most of the running time is consumed by DetectNet. It is hard to directly compare running time with previous works <cit.> because they did not report it. However, we guess that there would be no big difference because models of  <cit.> and  <cit.> are similar with  <cit.> and  <cit.> whose speed is 0.2 and 0.11 seconds per frame, respectively.



§ ABSOLUTE 3D MULTI-PERSON POSE ESTIMATION ERRORS

For the continual study of the 3D multi-person pose estimation, we report 3DPCK_abs in Table <ref> and  <ref>. As previous works <cit.> did not report 3DPCK_abs, we only report our result. 




§ QUALITATIVE RESULTS

Figures <ref> and  <ref> show qualitative results of our 3D multi-person pose estimation framework on the MuPoTS-3D <cit.> and COCO <cit.> datasets, respectively. Note that COCO dataset consists of in-the-wild images which are hardly included in the 3D human pose estimation training sets <cit.>.








ieee_fullname




