These authors contributed equally to this work.

These authors contributed equally to this work.

[1]Yifan Peng

1]Department of Population Health Sciences, Weill Cornell Medicine, New York,
USA

2]Cockrell School of Engineering, The University of Texas at Austin, Austin,
USA

3]School of Information, The University of Texas at Austin, Austin, USA

4]Department of Preventive Medicine, Northwestern University, Chicago, USA

# An empirical study of using radiology reports and images to improve ICU
mortality prediction

Mingquan Lin  Song Wang  Ying Ding  Lihui Zhao  Fei Wang
<EMAIL_ADDRESS>[  [  [  [

###### Abstract

Background: The predictive Intensive Care Unit (ICU) scoring system plays an
important role in ICU management because it predicts important outcomes,
especially mortality. Many scoring systems have been developed and used in the
ICU. These scoring systems are primarily based on the structured clinical data
in the electronic health record (EHR), which may suffer the loss of important
clinical information in the narratives and images.

Methods: In this work, we build a deep learning based survival prediction
model with multi-modality data to predict ICU mortality. Four sets of features
are investigated: (1) physiological measurements of Simplified Acute
Physiology Score (SAPS) II, (2) common thorax diseases pre-defined by
radiologists, (3) BERT-based text representations, and (4) chest X-ray image
features. We use the Medical Information Mart for Intensive Care IV (MIMIC-IV)
dataset to evaluate the proposed model.

Results: Our model achieves the average C-index of 0.7829 (95% confidence
interval, 0.7620-0.8038), which substantially exceeds that of the baseline
with SAPS-II features (0.7470 (0.7263-0.7676)). Ablation studies further
demonstrate the contributions of pre-defined labels (2.00%), text features
(2.44%), and image features (2.82%).

Conclusions: Our model achieves a higher average C-index than the traditional
machine learning methods under the same feature fusion setting, suggesting
that the deep learning methods can outperform the traditional machine learning
methods in ICU mortality prediction. These results highlight the potential of
deep learning models with multimodal information to enhance ICU mortality
prediction. We make our work publicly available at
https://github.com/bionlplab/mimic-icu-mortality.

###### keywords:

Mortality prediction, Deep learning, Multimodal fusion

##  1 Introduction

Predictive ICU scoring systems measure the disease severity to predict
outcomes, typically mortality, of patients in the intensive care unit (ICU)
[1]. Such measurements help standardize research and compare the quality of
patient care across ICUs. For example, the Acute Physiology and Chronic Health
Evaluation [2], Simplified Acute Physiology Score (SAPS) II [3], and Mortality
Probability Model [4] were created explicitly for the use in the ICU,
validated in the critically ill, and relied primarily on the physiologic data
to predict mortality. These scoring systems have been primarily based on
structured clinical data including the risk factors used by the scoring system
such as demongraphsics, vital signs and lab tests, which are frequently
documented in the electronic health record (EHR).

The recent development of machine learning offers great potential to improve
ICU mortality prediction [5, 6, 7, 8]. However, these studies used only
structured, coded approaches for data entry, thus may result in the loss of
significant clinical information typically contained in the narratives and
images [9, 10]. To overcome this issue, many studies focus on mining
unstructured clinical notes for patient mortality prediction [11, 12, 13].
However, most of these works were not compared with the current scoring
system, making it challenging to compare these models fairly.

Moreover, the practice of modern medicine usually relies on multimodal
information. Consequently, many feature fusion strategies were proposed to
enhance the performance of prediction algorithms, such as early fusion, late
fusion, and joint fusion [14]. Early fusion combines multimodal features into
a single vector by concatenating or averaging [15, 16, 17]. Late fusion
combines the predictions of multiple models to make the final decision [18,
19, 20]. Joint fusion combines the features from the intermediate layer of the
neural network with the features of other modalities. The loss during the
training process will propagate back to the feature extraction neural network,
thereby creating a better feature representation through training iterations
[14, 21, 22, 23]. Despite these encouraging findings, we note that most
competitive approaches studied the classification tasks. Thus, the integration
of text and images in the survival analysis framework remains an important
yet, to date, insufficiently studied problem.

We, therefore, sought to overcome these limitations by (1) incorporating the
potentials of natural language processing (NLP) and medical image analysis to
identify the hidden features of critical illness among ICU patients in the
radiology reports and chest X-rays that may not be found in the structured EHR
fields [24]; and (2) investigating deep learning models that may provide the
superior discrimination of ICU mortality predictions compared to traditional
machine learning models [25]. Specifically, we first build the clinical
prediction models to predict ICU mortality using the SAPS-II risk factors such
as demographics, vital signs, and lab tests. These measurements were obtained
in the first 24 hours of ICU admission. We then enrich the model with
multimodal features extracted from radiology reports and chest X-rays. The
radiology imaging and reading were studied in the first 24 hours. We
hypothesize that including free texts and images provides better predictions
of ICU mortality than including clinical measurements alone. Experiments on
the MIMIC-IV dataset [26] show that our multimodal models are substantially
more accurate than the unimodal ones.

Our framework has several important strengths. First, we present a method to
fuse multimodal data from EHR for ICU mortality prediction. Second, we
demonstrate that our survival analysis model outperforms the existing clinical
standards (SAPS-II). Third, we make our work publicly available for
reproduction by others.

##  2 Methods

###  2.1 Task

We first formulate the survival analysis task, which predicts a patient’s
survival probability in ICU as a function of their features. We have $n$
patients $(x_{i},y_{i},\delta_{i})$. Each patient record consists of $d$
potential covariants $x_{i}\in R^{d}$, and the time $T_{i}$ when the death
occurred or the time $C_{i}$ of censoring. Since death and censoring are
mutually exclusive, we use the indicator $\delta_{i}\in\\{0,1\\}$ and the
observed survival time $y_{i}$, defined as below.

$y_{i}=min(T_{i},C_{i})=\begin{cases}T_{i}&\text{if $\delta_{i}$ = 1}\\\
C_{i}&\text{if $\delta_{i}$ = 0}\end{cases}$ (1)

The goal is to estimate the survival probability $S_{i}(t)=Pr_{i}(T>t)$ of a
patient who was not dead beyond time $t$.

In this study, we use one of the most popular survival analysis models, the
Cox model [27], where the survival function is assumed to be

$S_{i}(t|x_{i})=S_{0}(t)^{e^{\psi(x_{i})}}.$ (2)

In this model, $S_{0}(t)$ is the baseline survival function that describes the
risk for individuals with $x_{i}=\mathbf{0}$, and $\psi(x_{i})=x_{i}\beta$ is
the relative risk based on the covariants. Note that $S_{0}(t)$ is shared by
all patients at time $t$. It is NOT associated with any individual covariants.
The effect of the covariate values $x_{i}$ on the survival function is to
raise it to a power given by the relative risk.

In the Cox model, $\psi(x_{i})$ has the form of a linear function, but we can
also extend it to a non-linear risk function of a neural network, called the
DeepSurv-based model. The DeepSurv-based model has three steps: features
extraction, multimodal feature fusion, and survival analysis. The main
difference between our model and the DeepSurv model in [28] is that our deep
network performs multimodal feature fusion. When there is only a single
modality as input, our model is equivalent to the DeepSurv model. The detail
of the neural network via feature fusion is as described in the next section.

###  2.2 Neural network via feature fusion

The practices of physicians rely heavily on the synthesis of data from
multiple sources. This includes, but is not limited to, structured laboratory
data, unstructured text data, and imaging pixel data. Therefore, automated
predictive models that can successfully utilize multimodal data may lead to
better performance.

In this paper, we expand $\psi(x_{i})$ by introducing a deep neural network
with the fusion features from multiple sources: SAPS-II risk factors
$x_{saps}$, text features $x_{text}$, and imaging features $x_{img}$ (Figure
1). The extracted text features $x_{text}$ and image features $x_{img}$ are
respectively passed to two separate Multilayer Perceptron (MLP) modules where
the output dimensions are equal. We then use the two hidden features by
element-wise averaging. Finally, we concatenate it to $x_{saps}$.

$x_{i}=Avg(\text{DNN}_{img}(x_{img}),\text{DNN}_{text}(x_{text}))\oplus
x_{saps}$ (3)

In terms of fusion strategy, our approach is similar to “early fusion” which
refers to the process of combining features from multiple input modalities
into one single feature vector before feeding into the survival model [14].
The difference is that our loss is propagated back to the DNNs during
training, thus creating better feature selections for each training iteration.
In addition, our approach is not “joint fusion” because the parameters of the
features are not updated during the training iteration.

Figure 1: Multimodal feature fusion network.

###  2.3 Feature Extraction

####  2.3.1 SAPS-II score and risk factors

SAPS-II was designed to measure the severity of disease for patients aged 18
or more admitted to ICU [3]. Twenty-four hours after admission to the ICU, the
measurements have been completed and resulted in an integer point score
between 0 and 163. The score is calculated from 15 routine physiological
measurements including information about previous health status, and some
information obtained at admission. These measurements are: Age, Heart rate,
Blood pressure, Temperature, PaO2/FiO2, Blood urea nitrogen, Urine output,
Sodium, Potassium, Bicarbonate, Bilirubin, White blood count, Glasgow Coma
Scale, Chronic disease, and Admission type.

####  2.3.2 Text Features

In this work, we investigate three sets of text features.

Common thorax diseases from radiology reports. The first set of features
consists of 13 pre-defined diseases commonly found in radiology reports
(Atelectasis, Cardiomegaly, Consolidation, Edema, Enlarged Cardiomediastinum,
Fracture, Lung lesion, Lung opacity, Pleural effusion, Pleural other,
Pneumonia, Pneumothorax, Support devices) and Normal [27, 29, 30]. These
labels were extracted from radiology reports using NegBio [31] and can be
obtained from the MIMIC-CXR website111https://physionet.org/content/mimic-cxr-
jpg/2.0.0/.

Transformer-based features. The second set of features are text embeddings
extracted by the BERT model - taking advantage of the pre-training on large-
scale biomedical and clinical text corpora. Utilizing clinical texts in
survival analysis is difficult because they are largely unstructured. The
above lung diseases may fail to capture the textual information
comprehensively since their labels are still limited in scope. In this work,
we use the BERT-based hidden layer representations as text features. For an
input report that contains $m$ tokens, the BERT model will produce a
$d$-dimension embedding vector for each token, resulting in an $m\times d$
representation vector of the report in the latent space. We apply average
pooling over the token embeddings from the last layer of the BERT model to
obtain an aggregate latent report representation.

GCN-based features. We build a graph convolutional neural network (GCN) to
model the inner correlations among radiology concepts. The graph was manually
defined by domain experts (Figure 1 in Zhang et al. [32]. Disease findings are
defined as nodes in the graph, and correlated findings are connected to
influence each other during graph propagation. We take the $m\times d$ hidden
representation vectors from the last layer of the BERT model. To initialize
GCN node features, we apply a 1-dimension convolution over the text features
with the kernel size $k$ and the number of output channels equal to the number
of graph nodes. In this way, the graph nodes are initialized by aggregating
the hidden features of all the tokens in the report.

The GCN updates its node representations by message passing. We first
calculate $\hat{A}=D^{-1/2}\tilde{A}D^{-1/2}$ in a pre-processing step.
$\tilde{A}$ = $A$ \+ $I_{N}$ is the adjacency matrix with added self-
connections, where $A$ is the graph adjacency matrix, $I_{N}$ is the
$N$-dimension identity matrix, $D=\text{diag}\sum_{j}A_{ij}$ is the diagonal
node degree matrix. Then based on [33], the graph convolution can be expressed
as:

$\displaystyle H^{l}$ $\displaystyle=ReLu(\hat{A}H^{0}W^{0}+b^{0})$
$\displaystyle Z$ $\displaystyle=softmax(\hat{A}H^{1}W^{1}+b^{1})$

where $H^{l}$ is the states in the $l$-th layer, with $H^{0}$ initialized
using the aggregate report text hidden features, and $W^{l}$ is a trainable
layer-specific weights matrix.

####  2.3.3 Image Features

For image feature extraction, we use the ChexNet, a DenseNet-121 model
pretrained on CheXpert dataset [29, 34, 35]. For each input image, we extract
the image features of dimension $d_{img}$ from the global average pooling
layer of DenseNet-121.

##  3 Experiments

###  3.1 Study population and patient selection

We use the MIMIC-IV dataset (Medical Information Mart for Intensive Care IV)
to evaluate the proposed model [26]. MIMIC-IV is a de-identified clinical
database composed of 382,278 patients admitted in the ICUs at Beth Israel
Deaconess Medical Center. Of those, we excluded patients who had no CXR
studies before the measurements have been completed and resulted in the SAPS-
II score. Therefore, there are in total 9,928 patients included in this study.
Out of these patients, 2,213 patients (22%) were deceased in the ICU. Table 1
lists the information of the ICU admission group studied in this work. Details
of the SAPS-II can be found in Table LABEL:saps_ICU.

Table 1:  Information of the ICU admission group. EC: Enlarged Cardiomediastinum. | ICU Discharge | ICU morality
---|---|---
Patient, n | 7,715 | 2,213
Age, mean (SD), y | 61.78 (18.20) | 69.63 (14.84)
Gender, male/female, % | 45/55 | 46/54
Race, % |  |
American Indian | 0.27 | 0.18
Asian | 3.25 | 0.39
Black/African American | 11.43 | 11.16
Hispanic/Latino | 3.88 | 2.67
White | 65.37 | 63.76
Other | 5.22 | 4.34
Unable to obtain | 0.73 | 1.04
Unknown | 9.85 | 12.97
Common thorax diseases, % |  |
Atelectasis | 19.90 | 21.69
Cardiomegaly | 17.64 | 18.08
Consolidation | 4.19 | 8.00
Edema | 13.45 | 19.52
EC | 3.54 | 4.07
Fracture | 2.68 | 2.21
Lung Lesion | 2.64 | 4.97
Lung Opacity | 26.36 | 36.69
Pleural Effusion | 17.04 | 28.29
Pleural Other | 0.62 | 0.72
Pneumonia | 0.70 | 9.31
Pneumothorax | 2.44 | 2.94
Support Devices | 34.50 | 43.83

###  3.2 Evaluation metrics

To assess the accuracy of our models, we use the C-index, defined as:

$\displaystyle{L}_{s}=\frac{\sum_{i,j}I(T_{i}\geq T_{j})\cdot I(R_{i}\leq
R_{j})}{\sum_{i,j}I(T_{i}\geq T_{j})\cdot d_{j}},$

where $I(c)=\begin{cases}1&\text{if c is true}\\\
0&\text{otherwise}\end{cases}$, $d_{j}=\begin{cases}1&\text{if $T_{j}$
exist}\\\ 0&\text{otherwise}\end{cases}$ , $j\in\\{1,2,\cdots,N\\}$, and
$j\textgreater i$. $N$ is the number of sample.

Intuitively, the C-index measures the extent to which the model can assign
logical risk scores. An individual with a shorter time-to-event $T$ should
have a higher risk score $R$ than those with a longer time-to-event. C-index
assigns a random model 0.5 and a perfect model 1.

###  3.3 Implementation and Experimental Settings

We perform a grid search to find the optimal hyperparameters based on the
metrics and use them for all configurations. The MLP layer for SAPS-II risk
factors takes an input of 15 dimensions, and fully connects to 15 output
dimensions. The MLP layer for labels features fully connects the 14-dimension
inputs to the 14-dimension outputs. The MLP layer for report text features
fully connects the 768-dimension inputs to the 32-dimension outputs, and the
MLP layer for chest X-ray image features fully connects the 1024-dimension
inputs to the 32-dimension outputs.

We use 200 bootstrap samples to obtain a distribution of the C-index and
report the 95% confidence intervals (CI). For each bootstrap experiment, we
sample $n$ patients with replacement from the whole set of $n$ patients. We
then split the sampled set into training (70%), validation (10%), and test
(20%) sets. We iterate the training process for 250 epochs with batch size 72
and early stop if the validation loss does not decrease. The dropout rate is
0.5. The learning rate is 0.001 with an Adam optimizer [36].

We obtained the SAPS-II scores using the scripts in the MIMIC-IV
repository222https://github.com/MIT-LCP/mimic-iv. The text embeddings are
extracted using BlueBERT [37], which was pre-trained on the PubMed abstracts
and MIMIC-III notes. We use pycox333https://github.com/havakv/pycox, scikit-
survival [38], and PyTorch to implement the framework. Intel Core i9-9960X 16
cores processor and NVIDIA Quadro RTX 5000 GPU are used in this work.

##  4 Results

We first compare the baseline ICU scoring model and our models with four
different feature settings. The SAPS-II score is an integer point score
between 0 and 163 directly obtained from the MIMIC-IV website. The SAPS-II
risk factors model is trained using the 15 routine physiological measurements.
The SAPS-II risk factors + GCN features model is enriched with the GCN-based
features. The SAPS-II risk factors + Image features model is enriched with
chest X-ray image features. The multimodal features model is trained using
SAPS-II risk factors combined with text features and chest X-ray image
features using early average fusion.

Table 2 shows that the ICU scoring model achieves an average C-index of 0.7470
(95% confidence interval, 0.7263-0.7676). The mean C-index of our model with
SAPS-II risk factors achieves 0.7545 (0.7240-0.7849), which brings 0.75%
improvements to the ICU scoring baseline model. When combining the SAPS-II
risk factors with GCN-based text features and image features, the models
obtain the average C-index of 0.7720 (0.7517-0.7923) and 0.7752
(0.7518-0.7985), respectively, yielding increases of 2.50% and 2.82%. Using
the multimodal features, the performance of the model can further be boosted.
We obtain the average C-index of 0.7829 (0.7620-0.8038), resulting in an
improvement of 3.60% over the ICU scoring model. We also train the multimodal
features model with SAPS-II risk factors combined with GCN features and chest
X-ray image features using early average fusion. The average C-index is 0.7805
(0.7570-0.8040), which is slightly lower than the proposed multimodal features
model.

Table 2:  C-index comparisons of the models using different sets of features. Model | C-index | (95% CI)
---|---|---
SAPS-II scores (ICU scoring baseline) | 0.7477 | (0.7238-0.7716)
SAPS-II risk factors | 0.7555 | (0.7220-0.7890)
SAPS-II risk factors + GCN features | 0.7745 | (0.7486-0.8004)
SAPS-II risk factors + Image features | 0.7757 | (0.7522-0.7992)
Multimodal features | 0.7847 | (0.7625-0.8068)

Figure 2 shows more details on bootstrapping. The violin shape reflects the
distribution of the C-index: the thicker, the higher frequency. We find that
the average C-index associated with the multimodal features model is
statistically higher than the other four settings.

Figure 3 shows the C-index results of our SAPS-II risk factors model and
multimodal features model, marked in red and blue respectively. Both are
trained on the entire dataset and tested on the patients with normal or
abnormal chest X-rays. It is clear that our multimodal features model
outperforms the SAPS-II risk factors model and the normal subjects can be more
accurately predicted by our model. Figure 4 further breaks chest X-ray
abnormalities into 13 pre-defined thorax diseases.

Figure 2: C-index comparisons of the models using different sets of features.
$*$: $p-$value $\leq$ 0.05; $**$: $p-$value $\leq$ 0.01. Figure 3: The C-index
results of the models trained on the entire dataset and tested on the normal
patients or patients with chest X-ray abnormalities. Figure 4: The C-index
results of the models trained on the entire dataset and tested on the patients
with different chest X-ray abnormalities.

##  5 Discussion

###  5.1 Comparison of different types of text features.

First, we compare the results of our model using different types of text
features. SAPS-II risk factors + labels, SAPS-II risk factors + transformer
features, and SAPS-II risk factors + GCN features. They are trained using 15
routine physiological measurements combined with 14 thorax disease labels,
transformer-based features, and GCN-based features, respectively. Table 3
lists the results of our model using these three feature settings. The mean
C-indexes for these three settings are 0.7669 (0.7456-0.7882), 0.7714
(0.7488-0.7941), and 0.7720 (0.7517-0.7923), respectively. Models with
transformer features or GCN features outperform the model that uses only
labels. But there is no significant difference between the transformer and GCN
features.

Table 3:  The C-index results of the models using different types of text features. Model | C-index | (95% CI)
---|---|---
SAPS-II risk factors + labels | 0.7669 | (0.7456-0.7882)
SAPS-II risk factors + transformer features | 0.7714 | (0.7488-0.7941)
SAPS-II risk factors + GCN features | 0.7720 | (0.7517-0.7923)

###  5.2 Contribution of thorax diseases in survival analysis

Next, we analyze the multivariate association of chest X-ray abnormalities to
ICU mortality based on Cox Proportion Hazards (CoxPH model) (Table 4). The
p-values of these four findings: enlarged cardiomediastinum, fracture,
pneumonia, pneumothorax are greater than 0.05, indicating that there is no
statistically significant difference. In other words, these findings do not
contribute to mortality prediction.

Table 4:  Multivariate associations of chest X-ray abnormalities to ICU-mortality. *: $p-$value $\leq$ 0.05; **: $p-$value $\leq$ 0.01; ***: $p-$value $<$ 0.001. | Hazard ratio | 95% CI |  $p$-value
---|---|---|---
Atelectasis | 0.84 | 0.75– | 0.94 | **
Cardiomegaly | 0.85 | 0.76– | 0.96 | **
Consolidation | 1.33 | 1.14– | 1.55 | ***
Edema | 1.23 | 1.10– | 1.38 | ***
Enlarged Cardiomediastinum | 0.91 | 0.75– | 1.12 | 0.37
Fracture | 0.96 | 0.72– | 1.28 | 0.77
Lung Lesion | 1.37 | 1.13– | 1.67 | **
Lung Opacity | 1.29 | 1.17– | 1.42 | ***
Pleural Effusion | 1.13 | 1.02– | 1.26 | *
Pleural Other | 0.64 | 0.41– | 1.00 | *
Pneumonia | 1.07 | 0.93– | 1.23 | 0.34
Pneumothorax | 1.10 | 0.86– | 1.41 | 0.45
Support Devices | 1.27 | 1.16– | 1.39 | ***

###  5.3 Comparison of linear and deep survival models

We then compare the performances of the linear machine learning model and deep
learning model: CoxPH [40] and DeepSurv-based model.

Table 5 shows the results for both models with two feature settings. The
average C-indexes of the CoxPH model with SAPS-II risk factors and SAPS-II
risk factors + labels are 0.7510 (0.7300-0.7720) and 0.7617 (0.7414-0.7819),
respectively, in comparison with 0.7545 (0.7240-0.7849) and 0.7669
(0.7456-0.7882) obtained by our DeepSurv-based model. The results demonstrate
that deep learning models outperform CoxPH on high-dimensional features. The
p-value for CoxPH and DeepSurv-based model using SAPS-II is 0.01 and p-value
is 1.08e-6 when using SAPS-II + labels.

Table 5:  The C-index results of the conventional machine learning models and the deep learning models trained and tested on the entire dataset. Model | C-index | (95% CI)
---|---|---
SAPS-II risk factors |  |
CoxPH | 0.7510 | (0.7300-0.7720)
DeepSurv-based | 0.7545 | (0.7240-0.7849)
SAPS-II risk factors + labels |  |
CoxPH | 0.7617 | (0.7414-0.7819)
DeepSurv-based | 0.7669 | (0.7456-0.7882)

###  5.4 Error Analysis

Error analysis (i.e., examining the reasons behind inaccurate predictions)
revealed that the multimodal accounted for fewer errors. Table 7 demonstrates
one example case of ICU mortality. According to physiological measurements,
SAPS-II graded patient #1 the score of 38 and patient #2 36. However, patient
#1 was decreased at hour 198, but patient #2 was deceased at hour 75. Hence,
the SAPS-II incorrectly assigned the score. However, our multimodal approach
correctly assigned a higher survival probability to patient #1 (0.9903) than
to patient #2 (0.9562). In one bootstrap sample, we observed a total of 40,529
such errors (patient #1 has normal chest x-ray, and SAPS-II gives wrong
predictions but our multimodal method gives correct predicitons) with 1,802
distinct patients, out of which 527 patients have normal chest X-rays and
1,275 patients have abnormal chest x-rays. Figure 5 shows the distribution of
thorax diseases among 1,275 patients. It shows that Lung Opacity (38.98%)
contributes most to the ICU mortality prediction.

Figure 5: Distribution of thorax diseases among patients where our multimodal
model made more accurate predictions than SAPS-II.

##  6 Conclusion

In this paper, we propose a deep learning method that combines text and image
features that may not be found in the structured EHR field before, to improve
the ICU mortality prediction. Experiments on the MIMIC-IV dataset show that
our multimodal features model (using SAPS-II risk factors and early fusion of
text and X-ray image features) obtains a superior average C-index of 0.7829
(0.7620-0.8038) than several baselines. This demonstrates that the additional
information provided by the multimodal features can improve the ICU mortality
prediction performance. We also investigate whether deep learning methods are
more powerful than traditional machine learning methods in predicting ICU
mortality. Through experiments, our model achieves a better average C-index
than the CoxPH model under the same feature fusion setting, proving the
superior performance of deep learning methods in ICU mortality prediction.

There are several limitations to this work. First, we use a fusion strategy
similar to “early fusion” to fuse the text and image features extracted by
BlueBERT and ChexNet, respectively, but the parameters of them are not updated
during the training iterations. In the future, we plan to use joint fusion to
propagate the loss back to the feature extraction modules during training,
which may improve the representation learning performance. Second, a knowledge
graph is a popular tool for representing background knowledge, which can
improve several aspects of the model. We will explore other domain knowledge
and try different ways of incorporating the knowledge graph into ICU mortality
prediction. Third, the longitudinal EHR data contain the information regarding
the disease progressions that may help ICU mortality prediction, but are not
utilized in this work. In the future, we can employ the longitudinal EHR to
assist in predicting ICU mortality. Fourth, there is a risk of selction bias
in this study. For instance, we only included in our analysis patients with
imaging studies after ICU admission. Imaging studies are usually performed
when a patient is sicker, for example, to confirm central line placement. This
selection could lead to a sample that is not representative of the ICU
population. However, selection bias is a common problem in machine learning
[39], statistics [40], and epidemiology [41]; as a result, a number of
techniques have been developed to correct it. In the future, we will
investigate these techniques. Fifth, machine learning models are vulnerable to
adversarial attacks [42]. For example, images can be attacked by adding a
small perturbation to the original images. Texts can be attacked by adding a
small number of words. These attacks are imperceptible to humans but mislead a
model into producing incorrect outputs. Like selection bias, the adversarial
attack is a common problem in the medical domain, where accurate diagnostic
results are of paramount importance [43]. Previous studies suggest that if a
model could eliminate noises in their learned feature representations, they
would be more robust against adversarial perturbations [44]. We will study
these techniques to improve the robustness of the model in the future.

While our work only scratches the surface of multimodal fusion for survival
analysis, we hope it will shed light on the future directions for ICU
mortality prediction.

Supplementary information

Abbreviations

  * •

ICU: Intensive Care Unit

  * •

EHR: Electronic Health Record

  * •

SAPS: Simplified Acute Physiology Score

  * •

MIMIC: Medical Information Mart for Intensive Care

  * •

NLP: Natural Language Processing

  * •

GP: Gaussian Process

  * •

CNN: Convolutional Neural Network

  * •

MLP: Multilayer Perceptron

  * •

GCN: Graph Convolutional Network

  * •

CI: Confidence intervals

## Declarations

Ethics approval and consent to participate

The dataset supporting the conclusions of this article is available in the
Medical Information Mart for Intensive Care version IV (MIMIC-IV), which is a
public de-identified database thus informed consent and approval of the
Institutional Review Board was waived. Our access to the database was approved
after completion of the Collaborative Institutional Training Initiative (CITI
program) web-based training.

Consent for publication

N/A

Availability of data and materials

We made our codes publicly available at https://github.com/bionlplab/mimic-
icu-mortality. The dataset we are using in this work is Medical Information
Mart for Intensive Care IV (MIMIC-IV), which is also publicly available at
https://physionet.org/content/mimiciv/0.4/.

Competing interests

The authors declare that they have no competing interests.

Funding

This work was funded by the National Library of Medicine under award number
4R00LM013001 and Amazon Machine Learning Grant.

Author’s contributions

ML and SW implemented the methods, conducted the experiments and wrote the
paper. YP advised on all aspects of the work involved in this project and
assisted in the paper writing. YD, LZ, and FW advised on the overall direction
of the project and edited the paper. All authors read and approved the final
manuscript.

##  Appendix A Support materials

Table 6: SAPS-II physiological measurements of the ICU admission group. | Score | ICU Discharge | ICU Morality
---|---|---|---
|  | % | %
Age, year
$<$40 | 0 | 12.51 | 3.21
40-59 | 7 | 25.91 | 17.67
60-69 | 12 | 21.63 | 20.70
70-74 | 15 | 9.16 | 11.84
75-79 | 16 | 9.19 | 11.21
$\geq$80 | 18 | 21.59 | 35.38
Heart rate
$<$40 | 11 | 21.04 | 31.22
40-69 | 2 | 42.28 | 28.11
70-119 | 0 | 34.71 | 32.90
120-159 | 4 | 0.93 | 2.94
$\geq$160 | 7 | 1.04 | 4.84
Systolic BP, mmHg
$<$70 | 13 | 5.51 | 22.01
70-99 | 5 | 62.49 | 59.51
100-199 | 0 | 30.82 | 17.44
$\geq$200 | 2 | 1.18 | 1.04
Temperature $\leq 39$°C
No | 0 | 95.58 | 93.76
Yes | 3 | 4.42 | 6.24
PaO2/FiO2, mm Hg
$<$100 | 11 | 3.47 | 13.29
100-199 | 9 | 7.57 | 13.92
$\geq$200 | 6 | 13.66 | 17.40
No ventilation | 0 | 75.29 | 55.40
Blood urea nitrogen, mg/dL
$<$28 | 0 | 70.28 | 47.40
28-93 | 6 | 26.80 | 46.18
$\geq$84 | 10 | 2.92 | 6.42
Urine output, mL/day
$<$500 | 11 | 7.66 | 29.60
500-999 | 4 | 16.84 | 19.84
$\geq$1000 | 0 | 75.50 | 50.56
Sodium, mEq/L
$<$125 | 5 | 2.20 | 2.58
125-144 | 0 | 89.31 | 81.16
$\geq$145 | 1 | 8.49 | 16.27
Potassium
3.0-4.9 | 0 | 82.31 | 69.95
$<$3.0 or $\geq$5.0 | 3 | 17.69 | 30.05
Bicarbonate, mEq/L
$<$15 | 6 | 4.72 | 16.49
15-19 | 3 | 18.09 | 26.25
$\geq$20 | 0 | 77.19 | 67.25
Bilirubin, mg/dL
$<$4.0 | 0 | 96.64 | 90.24
4.0-5.9 | 4 | 1.24 | 2.76
$\geq$6.0 | 9 | 2.11 | 7.00
White blood count, x103/mm3
$<$1.0 | 12 | 0.45 | 2.08
1.0-19.9 | 0 | 89.38 | 77.09
$\geq$20.0 | 3 | 10.16 | 20.83
Glasgow Coma Scale
14-15 | 0 | 78.70 | 68.59
11-13 | 5 | 11.67 | 10.94
9-10 | 7 | 3.69 | 9.47
6-8 | 13 | 3.85 | 7.14
$<$6 | 26 | 2.09 | 7.86
Chronic disease
None | 0 | 90.56 | 76.59
Metastatic cancer | 9 | 6.13 | 15.68
Hematologic malignancy | 10 | 2.40 | 7.05
AIDS | 17 | 0.91 | 0.68
Type of admission
Scheduled surgical | 0 | 0.75 | 0.09
Medical | 6 | 83.50 | 86.58
Unscheduled surgical | 8 | 15.75 | 13.33
Table 7:  An example case of ICU mortality prediction where our multimodal model made more accurate prediction than SAPS-II. According to physiological measurements, SAPS-II graded patient 1 the score of 38 and patient 2 36. However, patient 1 was decreased at hour 198, but patient 2 was deceased at hour 75. Hence, the SAPS-II incorrectly assigned the score. However, our multimodal approach correctly assigned a higher survival probability to patient 1 (0.9903) to patient 2 (0.9562). |  Patient 1 |  Patient 2
---|---|---
Age, y |  85 |  62
Gender |  Male |  Male
Race |  African American |  White
Time-to-event, hr |  198 |  75
Mortality score |  38 |  36
Findings |  No Findings |  Atelectasis, Consolidation, Lung Lesion, Pleural Effusion
Report |  FINDINGS: Portable AP chest radiograph demonstrates no focal consolidation, pleural effusion, pulmonary vascular engorgement, or pneumothorax. Multiple vascular stents are noted in the right upper extremity, superior mediastinum, and mid left upper extremity. The aorta is tortuous. The cardiomediastinal silhouette is otherwise normal. IMPRESSION: No acute cardiopulmonary process. |  FINDINGS: There continues to be near-complete opacification of the right lung compatible with components of pleural effusion and post-obstructive consolidation secondary to a known right chest mass. Additionally, there is increasing left pleural effusion with basilar atelectasis. Assessment of the heart size is limited due to these opacities. There is no pneumothorax. IMPRESSION: Stable appearance of right-sided pleural effusion and post-obstructive consolidation in the setting of a known right chest mass; increasing left pleural effusion with basal atelectasis.
Chest X-ray |  |

## References

  *   * Lipshutz et al. [2016] Lipshutz, A.K., Feiner, J.R., Grimes, B., Gropper, M.A.: Predicting mortality in the intensive care unit: a comparison of the university health consortium expected probability of mortality and the mortality prediction model iii. Journal of intensive care 4(1), 1–8 (2016)
  * Zimmerman et al. [2006] Zimmerman, J.E., Kramer, A.A., McNair, D.S., Malila, F.M.: Acute physiology and chronic health evaluation (apache) iv: hospital mortality assessment for today’s critically ill patients. Critical care medicine 34(5), 1297–1310 (2006)
  * Le Gall et al. [1993] Le Gall, J.-R., Lemeshow, S., Saulnier, F.: A new simplified acute physiology score (saps ii) based on a european/north american multicenter study. JAMA 270(24), 2957–2963 (1993)
  * Teres et al. [1987] Teres, D., Lemeshow, S., Avrunin, J.S., Pastides, H.: Validation of the mortality prediction model for icu patients. Critical care medicine 15(3), 208–213 (1987)
  * El-Rashidy et al. [2020] El-Rashidy, N., El-Sappagh, S., Abuhmed, T., Abdelrazek, S., El-Bakry, H.M.: Intensive care unit mortality prediction: An improved patient-specific stacking ensemble model. IEEE Access 8, 133541–133564 (2020)
  * Ghassemi et al. [2015] Ghassemi, M., Pimentel, M., Naumann, T., Brennan, T., Clifton, D., Szolovits, P., Feng, M.: A multivariate timeseries modeling approach to severity of illness assessment and forecasting in icu with sparse, heterogeneous clinical data. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 29 (2015)
  * Zhao et al. [2020] Zhao, Z., Chen, A., Hou, W., Graham, J.M., Li, H., Richman, P.S., Thode, H.C., Singer, A.J., Duong, T.Q.: Prediction model and risk scores of icu admission and mortality in covid-19. PloS one 15(7), 0236618 (2020)
  * Johnson and Mark [2017] Johnson, A.E., Mark, R.G.: Real-time mortality prediction in the intensive care unit. In: AMIA Annual Symposium Proceedings, vol. 2017, p. 994 (2017). American Medical Informatics Association
  * Murdoch and Detsky [2013] Murdoch, T.B., Detsky, A.S.: The inevitable application of big data to health care. Jama 309(13), 1351–1352 (2013)
  * Ching et al. [2018] Ching, T., Himmelstein, D.S., Beaulieu-Jones, B.K., Kalinin, A.A., Do, B.T., Way, G.P., Ferrero, E., Agapow, P.-M., Zietz, M., Hoffman, M.M., et al.: Opportunities and obstacles for deep learning in biology and medicine. Journal of The Royal Society Interface 15(141), 20170387 (2018)
  * Mechcatie and Rosenberg [2018] Mechcatie, E., Rosenberg, K.: Nursing notes are predictive of outcomes in icu patients. AJN The American Journal of Nursing 118(10), 70 (2018)
  * Yang et al. [2021] Yang, H., Kuang, L., Xia, F.: Multimodal temporal-clinical note network for mortality prediction. Journal of Biomedical Semantics 12(1), 1–14 (2021)
  * Grnarova et al. [2016] Grnarova, P., Schmidt, F., Hyland, S.L., Eickhoff, C.: Neural document embeddings for intensive care patient mortality prediction. arXiv preprint arXiv:1612.00467 (2016)
  * Huang et al. [2020] Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., Lungren, M.P.: Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. npj Digital Medicine 3(1), 136 (2020) https://doi.org/10.1038/s41746-020-00341-z
  * Liu et al. [2017] Liu, N., Wang, K., Jin, X., Gao, B., Dellandréa, E., Chen, L.: Visual affective classification by combining visual and text features. PloS one 12(8), 0183018 (2017)
  * Liu et al. [2018a] Liu, J., Chen, Y., Lan, L., Lin, B., Chen, W., Wang, M., Li, R., Yang, Y., Zhao, B., Hu, Z., et al.: Prediction of rupture risk in anterior communicating artery aneurysms with a feed-forward artificial neural network. European radiology 28(8), 3268–3275 (2018)
  * Liu et al. [2018b] Liu, M., Lan, J., Chen, X., et al.: Bone age assessment model based on multi-dimensional feature fusion using deep learning. Academic Journal of Second Military Medical University 39, 909–916 (2018)
  * Bakkali et al. [2020] Bakkali, S., Ming, Z., Coustaty, M., Rusinol, M.: Visual and textual deep feature fusion for document image classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 562–563 (2020)
  * Reda et al. [2018] Reda, I., Khalil, A., Elmogy, M., Abou El-Fetouh, A., Shalaby, A., Abou El-Ghar, M., Elmaghraby, A., Ghazal, M., El-Baz, A.: Deep learning role in early diagnosis of prostate cancer. Technology in cancer research & treatment 17, 1533034618775530 (2018)
  * Qiu et al. [2018] Qiu, S., Chang, G.H., Panagia, M., Gopal, D.M., Au, R., Kolachalama, V.B.: Fusion of deep learning models of mri scans, mini–mental state examination, and logical memory test enhances diagnosis of mild cognitive impairment. Alzheimer’s & Dementia: Diagnosis, Assessment & Disease Monitoring 10, 737–749 (2018)
  * Yala et al. [2019] Yala, A., Lehman, C., Schuster, T., Portnoi, T., Barzilay, R.: A deep learning mammography-based model for improved breast cancer risk prediction. Radiology 292(1), 60–66 (2019)
  * Kawahara et al. [2018] Kawahara, J., Daneshvar, S., Argenziano, G., Hamarneh, G.: Seven-point checklist and skin lesion classification using multitask multimodal neural nets. IEEE journal of biomedical and health informatics 23(2), 538–546 (2018)
  * Yoo et al. [2019] Yoo, Y., Tang, L.Y., Li, D.K., Metz, L., Kolind, S., Traboulsee, A.L., Tam, R.C.: Deep learning of brain lesion patterns and user-defined clinical and mri features for predicting conversion to multiple sclerosis from clinically isolated syndrome. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization 7(3), 250–259 (2019)
  * Ford et al. [2016] Ford, E., Carroll, J.A., Smith, H.E., Scott, D., Cassell, J.A.: Extracting information from the text of electronic medical records to improve case detection: a systematic review. Journal of the American Medical Informatics Association 23(5), 1007–1015 (2016)
  * Weissman et al. [2018] Weissman, G.E., Hubbard, R.A., Ungar, L.H., Harhay, M.O., Greene, C.S., Himes, B.E., Halpern, S.D.: Inclusion of unstructured clinical text improves early prediction of death or prolonged icu stay. Critical care medicine 46(7), 1125 (2018)
  * [26] Johnson, A., Bulgarelli, L., Pollard, T., Horng, S., Celi, L.A., Mark, R.: MIMIC-IV. PhysioNet. https://doi.org/10.13026/A3WN-HQ05
  * Wang et al. [2017] Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: Chestx-ray8: hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3462–3471. IEEE, ??? (2017). https://doi.org/10.1109/CVPR.2017.369
  * Katzman et al. [2018] Katzman, J.L., Shaham, U., Cloninger, A., Bates, J., Jiang, T., Kluger, Y.: Deepsurv: personalized treatment recommender system using a cox proportional hazards deep neural network. BMC medical research methodology 18(1), 1–12 (2018)
  * Irvin et al. [2019] Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al.: Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 590–597 (2019)
  * Johnson et al. [2019] Johnson, A.E.W., Pollard, T.J., Greenbaum, N.R., Lungren, M.P., Deng, C.-y., Peng, Y., Lu, Z., Mark, R.G., Berkowitz, S.J., Horng, S.: MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. arXiv preprint (2019) arXiv:1901.07042v5
  * Peng et al. [2018] Peng, Y., Wang, X., Lu, L., Bagheri, M., Summers, R., Lu, Z.: NegBio: a high-performance tool for negation and uncertainty detection in radiology reports. In: AMIA Joint Summits on Translational Science Proceedings. AMIA Joint Summits on Translational Science, vol. 2017, pp. 188–196 (2018)
  * Zhang et al. [2020] Zhang, Y., Wang, X., Xu, Z., Yu, Q., Yuille, A., Xu, D.: When radiology report generation meets knowledge graph. ArXiv abs/2002.08277 (2020)
  * Kipf and Welling [2017] Kipf, T.N., Welling, M.: Semi-Supervised Classification with Graph Convolutional Networks. In: Proceedings of the 5th International Conference on Learning Representations. ICLR ’17 (2017). https://openreview.net/forum?id=SJU4ayYgl
  * Huang et al. [2017] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)
  * Rajpurkar et al. [2017] Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C., Shpanskaya, K., et al.: Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225 (2017)
  * Kingma and Ba [2014] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * Peng et al. [2019] Peng, Y., Yan, S., Lu, Z.: Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets. In: Proceedings of the 2019 Workshop on Biomedical Natural Language Processing (BioNLP 2019), pp. 58–65 (2019)
  * Pölsterl [2020] Pölsterl, S.: scikit-survival: A library for time-to-event analysis built on top of scikit-learn. The Journal of Machine Learning Research 21(1), 8747–8752 (2020)
  * Cortes et al. [2008] Cortes, C., Mohri, M., Riley, M., Rostamizadeh, A.: Sample selection bias correction theory. In: Algorithmic Learning Theory: 19th International Conference, ALT 2008, Budapest, Hungary, October 13-16, 2008. Proceedings 19, pp. 38–53 (2008). Springer
  * Whittemore [1978] Whittemore, A.S.: Collapsibility of multidimensional contingency tables. Journal of the Royal Statistical Society: Series B (Methodological) 40(3), 328–340 (1978)
  * Robins [2001] Robins, J.M.: Data, design, and background knowledge in etiologic inference. Epidemiology, 313–320 (2001)
  * Goodfellow et al. [2014] Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014)
  * Paschali et al. [2018] Paschali, M., Conjeti, S., Navarro, F., Navab, N.: Generalizability vs. robustness: investigating medical imaging networks using adversarial examples. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part I, pp. 493–501 (2018). Springer
  * Finlayson et al. [2019] Finlayson, S.G., Bowers, J.D., Ito, J., Zittrain, J.L., Beam, A.L., Kohane, I.S.: Adversarial attacks on medical machine learning. Science 363(6433), 1287–1289 (2019)


# Physically Plausible 3D Human-Scene Reconstruction from Monocular RGB Image
using an Adversarial Learning Approach

Sandika Biswas1,3, Kejie Li2, Biplab Banerjee3, Subhasis Chaudhuri3, Hamid
Rezatofighi1 Manuscript received: March, 11, 2023; Revised: June, 5, 2023;
Accepted:July, 5, 2023.1,3Sandika Biswas is with the Indian Institute of
Technology, Bombay, and Monash University. Email:
<EMAIL_ADDRESS>Li is with the University of Oxford. Email:
<EMAIL_ADDRESS>Banerjee and Subhasis Chaudhuri are with the
Indian Institute of Technology, Bombay. Email<EMAIL_ADDRESS><EMAIL_ADDRESS>Rezatofighi is with the Faculty of Information Technology,
Monash University. Email<EMAIL_ADDRESS>Object
Identifier (DOI): see top of this page.

###### Abstract

Holistic 3D human-scene reconstruction is a crucial and emerging research area
in robot perception. A key challenge in holistic 3D human-scene reconstruction
is to generate a physically plausible 3D scene from a single monocular RGB
image. The existing research mainly proposes optimization-based approaches for
reconstructing the scene from a sequence of RGB frames with explicitly defined
physical laws and constraints between different scene elements (humans and
objects). However, it is hard to explicitly define and model every physical
law in every scenario. This paper proposes using an implicit feature
representation of the scene elements to distinguish a physically plausible
alignment of humans and objects from an implausible one. We propose using a
graph-based holistic representation with an encoded physical representation of
the scene to analyze the human-object and object-object interactions within
the scene. Using this graphical representation, we adversarially train our
model to learn the feasible alignments of the scene elements from the training
data itself without explicitly defining the laws and constraints between them.
Unlike the existing inference-time optimization-based approaches, we use this
adversarially trained model to produce a per-frame 3D reconstruction of the
scene that abides by the physical laws and constraints. Our learning-based
method achieves comparable 3D reconstruction quality to existing optimization-
based holistic human-scene reconstruction methods and does not need inference
time optimization. This makes it better suited when compared to existing
methods, for potential use in robotic applications, such as robot navigation,
_etc_.

###### Index Terms:

Holistic Scene Reconstruction, Scene Graph Discriminator, Human and Object
Reconstruction.

##  I Introduction

Human-scene reconstruction in 3D is crucial for successful robot navigation
and interactions in the 3D world. Robots need to perceive and understand the
surrounding environment to navigate and interact effectively. From the 3D
reconstruction of the scene, robots are able to gain a more comprehensive
understanding of their surroundings, including the shapes, positions, and
spatial relationships of objects. To make informed decisions and plan its
actions robot should have an accurate knowledge of the scene structure, hence
physically plausible reconstruction is necessary for robot navigation.
Significant progress has been made in the independent reconstruction of the
humans [1, 2, 3], [4, 5, 6], and the objects [7, 8, 9], [10, 11] present in a
scene. But only a few recent works [12, 13] focus on holistic 3D human-scene
reconstruction.

The human and object elements of a scene constrain each other’s positions,
orientations, and alignments to avoid a collision or to ensure contact while
interacting with each other. Hence, for the holistic scene reconstruction
system, it is necessary to learn these constraints based on the human-object
(_e.g_., a person sitting on a chair) and object-object (_e.g_., a chair in
front of a table) relations or interactions for physically plausible
reconstruction. The scene reconstruction system should also ensure the
physical laws in the produced reconstruction _e.g_., humans/objects can not
float over the floor or humans cannot walk through the wall _etc_.

Figure 1: Overview of our approach. From a monocular RGB image, we learn to
generate physically plausible 3D human-scene reconstruction. We propose using
adversarial training for analyzing the physical plausibility of the human and
object alignments in the generated scene using implicit features from the 3D
representations of the scene elements. Specifically, we form a graph using
features extracted from the 3D bounding boxes around each scene element
(Section III-C). We use a graph convolution network (PNAConv) to differentiate
a plausible scene representation from an implausible one using the generated
graph and to penalize the scene reconstruction system for physically
implausible scene prediction.

The current optimization-based 3D human-scene reconstruction methods [12, 13]
explicitly define some hand-crafted rules (contact and non-collision losses)
to impose these physical laws and constraints between different scene
elements. Explicitly encoding every physical law between different elements in
the scene is difficult, as the applicability of these laws depends on the
specific interactions between the elements, which may vary from frame to
frame. For example, a person is moving a chair from one place to another and a
person is sitting on a chair. [12, 13] employ contact and non-collision losses
to ensure plausible reconstructions in such scenarios. Contact loss minimizes
the distance between the human body mesh and the scene mesh, whereas, non-
collision loss penalizes the penetration of human body mesh into the scene
mesh or one object into another. However, for these two scenarios, such losses
should be applied to different parts of the human body and the chair. Also,
not only the contact or collision constraints but also the proper placement of
the human on the chair is constrained by their relative orientations. In this
paper, we aim to learn these constraints implicitly from the training data for
building a human-scene reconstruction system that can produce a physically
plausible reconstruction without explicitly defining any physical laws and
constraints.

During training, we propose representing the holistic scene as a graph and
employing adversarial training with a graph discriminator to improve the
realism of the reconstructions. Given a single RGB image, we utilize off-the-
shelf networks [14, 15] to predict 2D bounding boxes around human and object
elements. Generator networks are then used to predict 3D reconstructions of
these elements and form a graphical representation of the scene using the 3D
bounding boxes around these 3D reconstructions. Nodes in the graph correspond
to the human and object elements, while edges are defined by distance and
angle features between pairs of elements (Fig. 1). This helps comprehend the
human-scene interactions and object-object relations in the scene for
analyzing their relative positioning, orientations, and sizes from the
training data. We propose using a graph neural network (GNN)-based
discriminator [16] to differentiate between a graph formed from the predicted
or reconstructed scene elements with implausible alignments in 3D violating
the physical laws/constraints from the real graph formed from the actual 3D
scenes present in the training data with physically plausible object
alignments. If the predicted scene reconstruction does not match with the real
distribution seen in the training data, it penalizes the human and object
reconstruction generator to produce a physically plausible scene. Using a
graph discriminator helps our method to learn the relations between different
elements of the scene from the data implicitly, using different features that
define the alignments of the elements in the scene.

Moreover, [12, 13] optimize the initial predictions from the off-the-shelf
networks over multiple frames which is unsuitable for robotic applications in
an online framework. Our method does not require the whole sequence
information a-priori, instead reconstructs the scene from a single monocular
RGB image, resulting in much faster reconstruction compared to the existing
methods. In summary, our contributions are:

  * •

We propose representing a scene in the form of a graph using distance and
angle features between each pair of elements in the scene that gives an
implicit knowledge of the alignments of the scene elements.

  * •

We propose an adversarial learning approach using a graph neural network-based
discriminator for building a physical plausibility-aware 3D human-scene
reconstruction system. Using a GNN-based discriminator helps the generator to
produce more accurate localizations and reconstructions of the scene elements.

  * •

Our single-image reconstruction method achieves comparable performance with
the existing optimization-based methods that use multiple frames’ information
for reconstruction. Our learning-based model for holistic human-scene
reconstruction is computationally more efficient compared to the existing
methods.

##  II Related Work

Scene-guided human pose and motion estimation: Recently, few methods have
leveraged the scene information for accurate human reconstruction and motion
predictions [17, 18, 19, 20, 21]. These methods optimize the human pose based
on its contacts and collisions with the given ground-truth scene. Shimada _et
al_. [19] use ground reaction forces to predict a temporally stable human
pose. Hassan _et al_. [20] propose character animation with challenging scene
interactions using an adversarial discriminator that assess the realism of the
human motion in the context of the scene. Luo _et al_. [21] propose human 3D
pose estimation by utilizing motion imitation in prescanned 3D scenes. Few
approaches have utilized the ground-truth scene information for accurate
placement of the human [22] or generating human [23] in the given scene. While
previous works focus on enhancing human reconstruction and localization with
ground-truth scene knowledge, our objective is to reconstruct both humans and
the scene without any prior knowledge of the scene.

Coarse scene reconstruction: Few techniques [24, 25] focus on the coarse
reconstruction of the scene. iMapper [24] estimates object layout by
leveraging human-object interactions inferred from 3D human skeleton data,
while Chen _et al_. [25] utilize an HOI (human-object interactions) graph to
learn spatial relations between the scene elements and enforce physical
constraints. This method jointly infers 3D human skeleton, object bounding
boxes, room layout, and camera pose from the image and scene graph. These
methods only generate coarse reconstruction and do not produce the detailed
shape of the scene objects and humans. Some recent approaches [26, 27, 28]
partially reconstruct the scene, restricting their scope to only those objects
which are in contact with the human. Zhang _et al_. [26] use prior knowledge
of the object size distribution to infer the intrinsic scale of the objects.
Dabral _et al_. [28] propose a gravity-aware system for reconstructing 3D
human and free-flight objects by joint optimization of the human and object
trajectories. Xie _et al_. [27] reconstruct the scene via an implicit
representation (unsigned distance fields) of the humans and objects. These
methods do not provide a holistic reconstruction of the scene and focus only
on a single object that is in contact with the human. In contrast, our aim is
to generate a detailed mesh representation of the entire scene (or multiple
objects).

Holistic scene reconstruction: Weng _et al_. [12] first propose a holistic
scene reconstruction. In this work, the authors jointly optimize the human
mesh and object reconstruction network parameters based on explicitly defined
physical constraints on surface contacts and interpenetration without any
precise knowledge about the HSI (Human-Scene Interactions). Yi _et al_. [13]
incorporate explicit human scene interactions using POSA (Pose with
prOximitieS and contActs) [22]. POSA predicts the probable contact regions of
a human body for a given human pose and scene. It also predicts the probable
scene elements with which the human surface may come into contact. Yi _et al_.
use this information for explicitly incorporating human-scene interactions as
constraints during optimization over a sequence of frames for final scene
reconstruction. Moreover, both these methods [12, 13] use optimization at the
inference time, which is time-consuming. [12] performs 4 stages of
optimization, 1) initial human prediction, and optimization of the scene
prediction network using 2) within-scene losses 3) body-scene losses, and 4)
final optimization of human mesh based on the optimized scene. Moreover, they
optimize the network parameters over the whole sequence for the final
reconstruction. Whereas, [13] does not optimize the network parameters but the
initial predictions over a sequence. These methods rely on multiple frames for
a reasonable reconstruction of the scene. However, our goal is to build a
learning-based model that learns the physical laws and constraints implicitly
from the training data and produces per-frame plausible reconstruction without
inference-time optimization. This makes our model more time efficient and
applicable for robotic applications in an online framework.

##  III Method

(a)

(b)

Figure 2: (a) Human reconstruction module: At inference time, given an input
image, using information about bounding box around human (size, $b$ and
relative distance of bounding box center _w.r.t._ image center,
$(c_{x},c_{y})$), and relation feature of human with respect to the other
objects in the scene ($\mathcal{F}_{h-obj}$), the human reconstruction network
predicts the SMPL pose and shape parameters along with location information of
the human, $T^{full}$ within the scene. (b) Object reconstruction module: At
inference time, given an input image, using object binary masks, and relation
feature with respect to the other objects in the scene ($\mathcal{F}_{obj-
obj}$), the object detection network predicts the object orientation, size,
and centroid. Object mesh generation network weights are initialized from
Total3D [9] proposed mesh generation network and have not been updated in the
training process.

Figure 3: Scene reconstruction results on 2 scenes (N3OpenArea and MPH16) of
PROX qualitative dataset. For every example, the second and third rows show
reconstruction results from the top and side views. For a fair comparison with
our single image-based method, results of HolisticMesh [12] are generated by
optimizing their network on a single image. For MOVER [13], results are
rendered from reconstructed results shared on MOVER’s website.

Given an indoor image, we use pre-trained networks to get 2D bounding boxes
around objects and humans and generate 3D reconstructions. The predicted 3D
reconstructions are used to form a graphical representation of the scene and
analyzed using a graph discriminator compared to the graph created from the
ground truth. This section explains our 1) human mesh reconstruction (Fig.
2(a)), 2) object reconstruction (Fig. 2(b)), and 3) the graph discriminator
modules.

###  III-A Human mesh reconstruction:

We use a learning-based model [29] for human body reconstruction. Given an
image firstly we use OpenPose [15] network for human bounding box detection.
The human reconstruction network takes the cropped human image and bounding
box information _i.e_., location of the bounding box center ($c_{x},c_{y}$)
relative to the actual image center ($W/2,H/2$), and the size of the squared
bounding box ($b$) in actual image scale (before cropping and resizing) as
input to predict the SMPL (Skinned Multi-Person Linear model) [30] body
parameters ($\theta\in\mathbb{R}^{72}$, orientation of each body joint
_w.r.t._ the root _i.e_., pelvis joint and $\beta\in\mathbb{R}^{10}$,
representing body shape) and the root joint translations,
$T^{full}=[t_{X}^{full},t_{Y}^{full},t_{Z}^{full}]$, relative to the original
camera.

Object-aware human reconstruction: A human’s pose and location are conditioned
by the other objects’ positions in the scene; hence it is necessary to
incorporate the positional relation of the human with the other scene
elements. Nie _et al_. [9] propose using the relative positions of surrounding
objects in the scene for each object bounding box prediction. Following a
similar approach, we use RelationNet [31] to extract the relation features of
the human, $\mathcal{F}_{h-obj}\in\mathbb{R}^{2048}$, with respect to the
other elements in the scene. The inputs to RelationNet [31] are i) the
ResNet-34 feature for each of the scene elements, and ii) geometric features
($\in\mathbf{R}^{n\times 64}$) between each pair of scene elements ($n$)
defined by the relative distance between the 2D bounding boxes. We add this
relation feature with the image feature
$\mathcal{F}_{img}\in\mathbb{R}^{2048}$ of the cropped human instance.

The combined bounding box feature and image-object feature
$\mathcal{F}_{bbox}\oplus(\mathcal{F}_{img}+\mathcal{F}_{h-obj})$ is passed
through an MLP regressor proposed in [29] to predict SMPL pose ($\theta$),
shape ($\beta$) parameters and the perspective camera translation
$t^{crop}=[t^{crop}_{X},t^{crop}_{Y},t^{crop}_{Z}]$ with respect to the
cropped image. This perspective camera translation is then modified to get the
body root translation with respect to the original camera, $T^{full}$ using
the focal length $f$ and the human bounding box information $(c_{x},c_{y},b)$
[29]. $T^{full}$ is added to the SMPL body vertices to get the human body
reconstruction in the world coordinates. From the SMPL body vertices, 3D body
joints are predicted using the joint regressor [32]. The losses used for
training the human body generator are as follows,

$\begin{split}\mathcal{L}_{human}&=\lambda_{pose}\mathcal{L}_{pose}+\lambda_{shape}\mathcal{L}_{shape}+\lambda_{2D}\mathcal{L}_{2D}\\\
+\lambda_{3D}&\mathcal{L}_{3D}+\lambda_{verts}\mathcal{L}_{verts}+\lambda_{h-obj_{gan}}\mathcal{L}_{h-obj_{gan}}\end{split}$
(1)

Where, $\mathcal{L}_{pose}=||\theta-\theta^{\prime}||_{2}$,
$\mathcal{L}_{shape}=|\beta-\beta^{\prime}|$,
$\mathcal{L}_{3D}=||J_{3D}-J_{3D}^{\prime}||_{2}$,
$\mathcal{L}_{2D}=||J_{2D}-J_{2D}^{\prime}||_{2}$, and
$\mathcal{L}_{verts}=||\mathcal{B}_{verts}-\mathcal{B}_{verts}^{\prime}||_{2}$
are losses applied on predicted SMPL pose, shape parameter, 3D body joints,
reprojected 3D joints, and body mesh vertices
$\\{\theta^{\prime},\beta^{\prime},J_{3D}^{\prime},J_{2D}^{\prime},\mathcal{B}_{verts}\\}$.
The adversarial loss is defined as follows,

$\begin{split}\mathcal{L}_{h-obj_{gan}}=\mathbb{E}_{\\{\theta^{\prime},\beta^{\prime},\mathcal{B}_{v}^{\prime}\\}}&[log(D(\mathcal{G}_{h}^{\prime})]\\\
&+\mathbb{E}_{\\{\theta,\beta,\mathcal{B}_{v}\\}}[log(1-D(\mathcal{G})]\end{split}$
(2)

where $\mathcal{G}_{h}^{\prime}$ is the graph formed from predicted human mesh
and ground-truth object meshes and $\mathcal{G}$ is formed from ground-truth
human mesh and object meshes. Details about graph formation are described in
Section III-C.

###  III-B Object reconstruction:

Similar to [12] and [13], we use the object reconstruction module proposed by
Nie _et al_. [9]. It consists of three sub-networks, i.e., Layout Estimation
Network (LEN), Object Detection Network (ODN) for object bounding box
prediction, and Mesh Generation Network (MGN). From a given image, first
object labels and 2D bounding boxes are extracted using an off-the-shelf
network PointRend [14]. Then ResNet-34 [33] is used to get the image feature
from the object patches and RelationNet [31] is used to extract the object-
object relation feature $\mathcal{F}_{obj-obj}$. Using this feature the Object
Detection Network predicts each object’s size ($\mathbb{B}_{s}^{\prime}$),
orientation ($\mathbb{B}_{\theta}^{\prime}$), and 2D offset of the bounding
box center. We use ground truth camera parameters to unproject the predicted
2D object center for computing the 3D object centroid
($\mathbb{B}_{c}^{\prime}$). The pre-trained Mesh Generation Network (MGN)
from [9] is used to get the object meshes, which are then resized, rotated,
and translated using the predicted object orientation
($\mathbb{B}_{\theta}^{\prime}$), centroid ($\mathbb{B}_{c}^{\prime}$) and
size ($\mathbb{B}_{s}^{\prime}$) information. We use the same losses as
proposed in [9] for training.

As we perform per-frame reconstruction human occlusion highly affects the
object reconstruction. Hence, to further enhance the object detection
performance for occlusion cases, we propose using occlusion masks’ features
along with the ResNet image features (Fig. 2(b)). For training purposes, we
use ground-truth occlusion masks. Ground-truth occlusion masks are extracted
by using $M_{obj}-M_{obj}\cap M_{human}$, where $M_{obj}$ are unoccluded
object masks that are extracted manually from a keyframe (without occlusion by
the human) and $M_{human}$ is the rendered body mask. The losses used for
training the human body generator are as follows,

$\begin{split}\mathcal{L}_{obj}&=\lambda_{size}\mathcal{L}_{size}+\lambda_{ori}\mathcal{L}_{ori}\\\
&+\lambda_{centroid}\mathcal{L}_{centroid}+\lambda_{obj-
obj_{gan}}\mathcal{L}_{obj-obj_{gan}}\end{split}$ (3)

Where, $\mathcal{L}_{size}=||\mathbb{B}_{s}-\mathbb{B}_{s}^{\prime}||_{2}$,
$\mathcal{L}_{ori}=||\mathbb{B}_{\theta}-\mathbb{B}_{\theta}^{\prime}||_{2}$,
and $\mathcal{L}_{centroid}=||\mathbb{B}_{c}-\mathbb{B}_{c}^{\prime}||_{2}$
are losses applied on predicted size, orientation and centroid of the object
bounding boxes
$\\{\mathbb{B}_{s}^{\prime},\mathbb{B}_{\theta}^{\prime},\mathbb{B}_{c}^{\prime}\\}$.
The adversarial loss is defined as,

$\begin{split}\mathcal{L}_{obj-
obj_{gan}}=\mathbb{E}_{\\{\mathbb{B}_{s}^{\prime},\mathbb{B}_{\theta}^{\prime},\mathbb{B}_{c}^{\prime}\\}}&[log(D(\mathcal{G}_{obj}^{\prime})]+\\\
&\mathbb{E}_{\\{\mathbb{B}_{s},\mathbb{B}_{\theta},\mathbb{B}_{c}\\}}[log(1-D(\mathcal{G})]\end{split}$
(4)

where $\mathcal{G}_{obj}^{\prime}$ and $\mathcal{G}$ are the graphs formed
from predicted and ground-truth object meshes.

###  III-C Graph Discriminator:

We propose using the GNN [16] for analyzing the physical laws and constraints
using implicit feature representation of the reconstructed 3D scene elements.
Specifically, we use a graph discriminator to distinguish a physically
plausible real scene alignment from a reconstructed physically implausible
scene to further strengthen the performance of the object and human
reconstruction modules.

Graph formation: We form a graph, $\mathcal{G}(\mathcal{V},\mathcal{E})$,
$\mathcal{V}\in\mathbb{R}^{P},P=N+M$ from the scene, where, nodes
$\mathcal{V}$, are represented by the objects $\\{O_{n}\\}_{n=1,2,...N}$ and
human body segments $\\{H_{m}\\}_{m=1,2,...M}$, that come in contact (_e.g_.,
feet while walking or standing on the floor, hands while touching any objects
_etc_.) with the scene or scene objects. These body segments,
$\\{H_{m}\\}_{m=1,2,...M}$ are defined by the vertices of the corresponding
regions of the body mesh as proposed by Hassan _et al_. in [17]. $N$ is the
number of objects in the scene and $M$ is the number of body segments under
consideration. We consider the corner points of the 3D bounding box around the
objects or the human body segments as the node features
$\mathcal{F}_{\mathcal{V}_{l}}\in\mathbb{R}^{8\times 3}$. Implementation
details for 3D bounding box formation around human body segments are described
in Section IV-B. The edge features
$\mathcal{F}_{\mathcal{V}_{l}->\mathcal{V}_{k}}=\mathcal{F}_{\mathcal{V}_{l},\mathcal{V}_{k}}^{dist}\oplus\mathcal{F}_{\mathcal{V}_{l},\mathcal{V}_{k}}^{surf-
norm}$ are defined by the Euclidean distance
$\mathcal{F}_{\mathcal{V}_{l},\mathcal{V}_{k}}^{dist}$ between every pair of
corners (Eqn. 5) and surface normals
$\mathcal{F}_{\mathcal{V}_{l},\mathcal{V}_{k}}^{surf-norm}$ of every pair of
faces (Eqn. 6) of the bounding boxes, between each pair of nodes or scene
elements $\mathcal{V}_{l}$ and $\mathcal{V}_{k}$.
$\mathcal{F}_{\mathcal{V}_{l},\mathcal{V}_{k}}^{dist}$ helps to analyze the
relative distance, positioning, and sizes of the interacting elements whereas,
$\mathcal{F}_{\mathcal{V}_{l},\mathcal{V}_{k}}^{surf-norm}$ helps to analyze
the relative orientations.

$\mathcal{F}_{\mathcal{V}^{i}_{l},\mathcal{V}^{j}_{k}}^{dist}=\lVert\mathbb{B}^{l}_{i}-\mathbb{B}^{k}_{j}\rVert,\;$
(5)

where, $\mathbb{B}^{l}_{i},\mathbb{B}^{k}_{j}\in\mathbb{R}^{8\times 1}$
represents the $i^{th}$ and $j^{th}$ corner points of the 3D bounding boxes of
$l^{th}$\- and $k^{th}$-object.

$\mathcal{F}_{\mathcal{V}^{i}_{l},\mathcal{V}^{j}_{k}}^{surf-
norm}=\mathcal{B}^{l}_{n_{i}}\times\mathcal{B}^{k}_{n_{j}},\;$ (6)

where, $\mathbb{B}^{l}_{n_{i}},\mathcal{B}^{k}_{n_{j}}\in\mathcal{R}^{6\times
3}$ represents the surface normals on $i^{th}$ and $j^{th}$ faces of
$l^{th}$\- and $k^{th}$-object bounding boxes. More details about the network
architecture and training are given in Section IV-B.

Scene representation learning: We have used Principal Neighbourhood
Aggregation (PNA) graph convolution [16] network for scene representation
learning. Unlike the traditional graph convolution networks, PNA use multiple
aggregators for effective feature representation learning, as a single message
aggregator fails to capture meaningful representations from different kinds of
messages [16]. Also, it uses a scaler to either amplify or attenuate the
aggregated message at a node, based on the number of messages coming to the
node. We have used four aggregators i.e mean, max, min and std and three-
scalers i.e. identity, amplification ($\alpha=1$), attenuation ($\alpha=-1$)
for effective feature aggregation from neighborhood nodes. Node embeddings
learned using PNAConv layers are passed through a graph pooling layer to get
the scene representation. Finally, an MLP is used with an output layer to
classify the graph between real and predicted. Hence, the output of this graph
discriminator consists of binary labels: 0 for predicted and 1 for ground
truth or real.

##  IV Experiments and Results

###  IV-A Dataset

We use the PROX (Proximal Relationships with Object eXclusion) dataset [17]
for our experiments. It consists of a) qualitative and b) quantitative
datasets. PROX qualitative dataset contains 12 different scenes including,
bedrooms, libraries, offices, and living rooms. It contains around 100K RGB-D
frames with humans interacting with the scenes. It contains pseudo-ground
truth for the body poses generated from the SMPLify-X [34] method using depth
and RGB frames. The PROX quantitative contains a single scene where human
activities or interactions with the scene have been captured using MoCap. The
ground truth of human body mesh is generated using MoSh++ [35]. The scene is
scanned and reconstructed using Structure Sensor and Skanect. This dataset
contains a total of 18 sequences with 180 RGB-D frames. We have used this
dataset for cross-dataset evaluation. We use PROX qualitative and perform data
augmentation (scaling and translation) for the training of both human and
object reconstruction modules. We use a similar train and test split as
defined by Hassan _et al_. in POSA [22]. Amongst 12 scenes, 7 scenes have been
used for training and 3 scenes (N3OpenArea, MPH1Library, and MPH16) have been
used for testing. We also provide qualitative results for human and scene
reconstruction on PROX quantitative dataset.

###  IV-B Implementation Details

We use the CLIFF (Carrying Location Information in Full Frames) model proposed
by Li _et al_. [29] as the backbone of our human mesh reconstruction module
and train this module on the PROX qualitative dataset. We additionally use
object alignment information in 2D as input for more accurate localization of
the human in the scene (Fig. 2(a)). For object reconstruction, we use the
network proposed by [9] as our backbone network and initialized the weights of
our object reconstruction network using the model proposed in [9]. We
additionally use object occlusion masks as input (Fig. 2(b)). For creating a
graph from the predicted scene elements, we calculate 3D bounding boxes around
human body segments that come in contact with the scene, _e.g_., feet, hands
_etc_..
3D bounding box creation around human mesh: We perform PCA [36] on the
projection of vertices of each body segment on each plane for calculating the
orientation around the perpendicular axis and size of the bounding boxes. For
example, to calculate the orientation of a body segment around the $x$-axis,
the body segment vertices are projected on the $yz$ plane ($x=0$). PCA is
performed on the projected 2d points (on $yz$ plane) to calculate the
principal orientation ($\theta_{x}$) of the point cluster. Using the
calculated orientation, the points are aligned with the axis (applying
$-\theta_{x}$) to calculate the size of the body segment. The orientations
around the $y$\- and $z$-axis are calculated in a similar manner. Finally, the
3D bounding box for each body segment vertices is formed using these size,
orientation, and centroid information for graph formation.

Our proposed graph discriminator consists of 4 PNAConv layers [16] followed by
a graph pooling layer, global$\\_$add$\\_$pool, and 3 fully connected layers
with ReLU at the intermediate and Sigmoid activation at the last layer. The
global$\\_$add$\\_$pool layer calculates a graph level output _i.e_., a
representation of the input graph, by adding the node features across the node
dimension of the input graph. We have used global$\\_$add$\\_$pool layer from
pytorch_geometric library [37].

We train human and object reconstruction modules independently. We use Adam
optimizer [38] with the initial learning rate of $1e-4$ and $1e-3$ for the
object and human reconstruction network training. We have used PyTorch and
PyTorch geometric library for implementation and GTX 1080i GPU for training
and inference. In the human reconstruction training loss (Eqn. 1)
$\lambda_{verts}=1.0$, $\lambda_{pose}=1.0$, $\lambda_{shape}=0.1$,
$\lambda_{2D}=5.0$ and $\lambda_{3D}=5.0$. The total loss is scaled by 60. In
the object reconstruction loss (Eqn. 3) are $\lambda_{ori}=1.0$,
$\lambda_{size}=1.0$, $\lambda_{centroid}=1.0$, $\lambda_{gen}=0.1$ and
$\lambda_{disc}=0.1$. The total loss is scaled by 100.

Method | Input | Object Reconstruction Quality | Physical Metrics
---|---|---|---
|  | IoU3D $\uparrow$ | IoU2D $\uparrow*$ | Non-collision $\uparrow$ | Contact $\uparrow$
MOVER [13] | Image Sequence | 0.3665 | 0.6179 | 0.9992 | 0.1601
HolisticMesh [12] | Image Sequence | 0.2607 | 0.4361 | 0.8878 | 0.7725
Total3D [9] | Single Image | 0.1917 | 0.3038 | - | -
Baseline $\dagger$ | Single Image | 0.3156 | 0.4037 | - | -
Ours | Single Image | 0.3470 | 0.5515 | 0.9011 | 0.7897

TABLE I: Quantitative comparison of our object reconstruction method with SoTA
on PROX qualitative dataset. Blue denotes the best-performing method that uses
single image input and performs per-frame prediction. Red denotes the overall
best-performing method. $*$ IoU2D is calculated as bird’s eye view 2D bounding
box IoU. $\dagger$ Baseline is Total3D [9] trained on PROX qualitative
dataset.

| Object reconstruction
---|---
Method | IoU3D $\uparrow$ | IoU2D $\uparrow$
HolisticMesh [12] | 0.2048 | 0.5167
Baseline + object mask | 0.2514 | 0.5208
Baseline + gan + object mask (Ours) | 0.2647 | 0.6097

TABLE II: Quantitative comparison of our object reconstruction method with
SoTA on PROX qualitative dataset using predicted 2D bounding box and occlusion
masks. The model trained with graph discriminator is defined as ’gan’.

Dataset | Method | Localization | Pose Estimation
---|---|---|---
|  | MPJPE | V2V | p-MPJPE | p-V2V
PROX quantitative | MOVER [13] | 174.37 | 178.31 | 73.60 | 67.89
HolisticMesh [12] | 190.78 | 192.21 | 72.72 | 61.01
| Ours $\dagger$ | 266.31 | 268.08 | 72.31 | 84.14
PROX quantitative (Testset) | HolisticMesh [12] | 187.24 | 191.00 | 65.13 | 64.07
Ours $\ddagger$ | 181.02 | 187.89 | 55.82 | 67.62
PROX qualitative | Ours $\dagger$ | 182.57 | 187.79 | 59.51 | 70.14

TABLE III: Quantitative comparison of our human reconstruction with SoTA. Ours
$\dagger$: Human reconstruction model trained on PROX qualitative dataset;
Ours $\ddagger$: Human reconstruction model fine-tuned on PROX quantitative
dataset using 5-fold cross-validation method.

| Object reconstruction
---|---
Method | IoU3D $\uparrow$ | IoU2D $\uparrow$
Baseline | 0.3156 | 0.4037
Baseline + gan | 0.3279 | 0.4206
Baseline + object mask | 0.3059 | 0.5208
Baseline + gan + object mask (Ours) | 0.3470 | 0.5515

TABLE IV: Ablation for object reconstruction on PROX qualitative.

###  IV-C Quantitative Results:

Object reconstruction We have evaluated and compared our object reconstruction
results against the state-of-the-art methods HolisticMesh [12] and MOVER [13]
in Table I (with GT 2D bounding box and occlusion masks) and II (with
predicted 2D bounding box and occlusion masks by PointRend [14]). It should be
noted that MOVER optimizes the object and human reconstruction based on the
HSI information collected from a sequence of frames taken as input. And
HolisticMesh performs optimization of the object reconstruction network
parameters over the sequence of frames. We have used the same metrics proposed
in [12, 13] for the performance evaluation. As we use PROX qualitative dataset
for training, the performance metrics are evaluated on 3 test scenes of the
dataset for all three methods. Similar to [12, 13, 9] we have used 3D and 2D
IoU for quantifying the object localization accuracy. We train the Total3D
model [9] on PROX qualitative dataset and consider it as our Baseline model.
Our method _i.e_., Baseline with graph discriminator outperforms the results
of Baseline and HolisticMesh in 3D IoU using both predicted (Table II) and
ground-truth (Table I) bounding box and occlusion masks. Following MOVER [13],
we have computed the Non-collision and Contact scores to assess the physical
plausibility of the reconstructed scenes with respect to the reconstructed
human (Table I). Total3D [9] performs only object reconstruction, hence does
not have non-collision and contact scores.

Figure 4: Comparison of execution time vs. performance of our scene
reconstruction method with respect to state-of-the-art methods.

Human reconstruction Following the state-of-the-art methods we use Mean Per
Joint Error (MPJPE) and vertex-to-vertex (V2V) distance for assessing the
quality of human reconstruction (Table III) [32]. MPJPE is calculated on the
3D body skeleton joints without making the root position zero. Our human
reconstruction module is trained on human body meshes from 7 scenes of the
PROX qualitative dataset. Our method gives a reasonable performance on cross-
dataset (PROX quantitative) compared to the state-of-the-art methods. We also
fine-tune the human reconstruction module using 12 sequences of PROX-
quantitative data, which performs better than that of HolisticMesh [12]. We
use p-MPJPE and p-V2V for estimating the quality of human pose reconstruction
similar to the state-of-the-art methods. Our method achieves the best p-MPJPE
compared to the state-of-the-art methods, hence the human reconstruction
module can capture the body pose correctly.

###  IV-D Qualitative Results:

We demonstrate our scene reconstruction results on two unseen scenes from the
PROX qualitative dataset (Fig. 3). For each frame, the reprojection of
reconstructed object meshes on the image plane is shown in the first row, and
3D reconstructions from different viewing angles are in the second and third
rows. Our approach, with the discriminator network (second column, Fig. 3),
produces physically plausible scene reconstructions with accurate relative
positioning of humans and objects compared to the backbone model (without
discriminator network, third column, Fig. 3). However, in the case of scene
MPH16, the model struggles to capture the correct size of the bed due to
limited bed instances in the training dataset. Our method performs well on
objects like sofas and chairs, outperforming HolisticMesh and yielding
comparable results to MOVER.

###  IV-E Performance vs. Inference Time:

In Fig. 4 we present an analysis of performance vs. inference time for all the
methods. Our method (0.75 sec/frame) and Total3D (0.24 sec/frame) [9] perform
per-frame prediction, whereas MOVER [13] and HolisticMesh [12] perform
optimization over a sequence of images. Weng _et al_. [12] take around 4-5
mins/frame and optimize over the whole sequence which takes days to produce
the final reconstruction. Whereas, Yi _et al_. [13] take around 30 mins to
optimize over the whole sequence and take the same time for a scene regardless
of the number of frames in the input video. For a fair comparison, the
execution time for all the methods is reported over a sequence of frames (avg.
1200 frames). All methods are evaluated under the same system configuration.
Although the inference time of Total3D [9] is low, it should be noted that
Total3D performs only object reconstruction. Our method achieves comparable
performance with MOVER at a much lower per-frame execution time.

| PROX quantitative | PROX qualitative
---|---|---
Method | MPJPE | V2V | MPJPE | V2V
Ours w/o gan | 181.97 | 198.80 | 181.74 | 188.17
Ours w/ gan | 181.02 | 187.89 | 182.57 | 187.79

TABLE V: Ablation for human reconstruction module.

###  IV-F Ablation Study:

Object reconstruction In Table IV, we have compared the effect of using the
graph discriminator (represented as ’$+$ gan’ in Table) for the scene
reconstruction using 3D and 2D IoU values on the PROX qualitative test set.
Using the graph discriminator helps in achieving better reconstruction
compared to the baseline. Also, we show the effectiveness of using binary
object masks as input for object reconstruction. Object masks help in
increasing the performance, specifically for the occluded objects, and give
the best IoU3D.

Human reconstruction In Table V we have shown the effectiveness of our scene-
aware human body reconstruction on the test set of both quantitative and
qualitative datasets. Using a discriminator to analyze the human placement
with respect to the object alignments in the scene helps in achieving better
MPJPE and V2V values on PROX quantitative.

##  V Conclusions

We have proposed a fully learning-based scene reconstruction method, which
relies on implicit feature representation of a scene for differentiating a
physically plausible scene reconstruction from implausible human and object
alignments without explicitly defining physical laws and constraints. The
reconstruction generator learns from training data to produce a plausible 3D
scene and performs per-frame prediction without any test time optimization.
Our method achieves comparable performance as the state-of-the-art methods but
with a faster reconstruction speed. However, the execution time needs to be
further improved for utilizing it in an actual robotics platform. Due to
limited variability in the training data, our method suffers from a lack of
generalization in terms of dynamic views, camera setup, resolution _etc_.. It
fails to perform well for out-of-training distribution data. In the future,
with the availability of a more generalized dataset, robust scene
reconstruction would be possible.

## References

  * [1] C. Zhang, Z. Cui, Y. Zhang, B. Zeng, M. Pollefeys, and S. Liu, “Holistic 3d scene understanding from a single image with implicit representation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 8833–8842.
  * [2] V. Choutas, G. Pavlakos, T. Bolkart, D. Tzionas, and M. J. Black, “Monocular expressive body regression through body-driven attention,” in _European Conference on Computer Vision_. Springer, 2020, pp. 20–40.
  * [3] M. Kocabas, N. Athanasiou, and M. J. Black, “Vibe: Video inference for human body pose and shape estimation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 5253–5263.
  * [4] A. Kanazawa, M. J. Black, D. W. Jacobs, and J. Malik, “End-to-end recovery of human shape and pose,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7122–7131.
  * [5] Y. Rong, T. Shiratori, and H. Joo, “Frankmocap: A monocular 3d whole-body pose estimation system via regression and integration,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 1749–1759.
  * [6] M. Kocabas, C.-H. P. Huang, O. Hilliges, and M. J. Black, “Pare: Part attention regressor for 3d human body estimation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 11 127–11 137.
  * [7] M. Dahnert, J. Hou, M. Nießner, and A. Dai, “Panoptic 3d scene reconstruction from a single rgb image,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 8282–8293, 2021.
  * [8] G. Gkioxari, J. Malik, and J. Johnson, “Mesh r-cnn,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 9785–9795.
  * [9] Y. Nie, X. Han, S. Guo, Y. Zheng, J. Chang, and J. J. Zhang, “Total3dunderstanding: Joint layout, object pose and mesh reconstruction for indoor scenes from a single image,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [10] S. Song, F. Yu, A. Zeng, A. X. Chang, M. Savva, and T. Funkhouser, “Semantic scene completion from a single depth image,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1746–1754.
  * [11] Y. Nie, A. Dai, X. Han, and M. Nießner, “Learning 3d scene priors with 2d supervision,” _arXiv preprint arXiv:2211.14157_ , 2022.
  * [12] Z. Weng and S. Yeung, “Holistic 3d human and scene mesh estimation from single view images,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 334–343.
  * [13] H. Yi, C.-H. P. Huang, D. Tzionas, M. Kocabas, M. Hassan, S. Tang, J. Thies, and M. J. Black, “Human-aware object placement for visual environment reconstruction,” _arXiv preprint arXiv:2203.03609_ , 2022.
  * [14] A. Kirillov, Y. Wu, K. He, and R. Girshick, “Pointrend: Image segmentation as rendering,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 9799–9808.
  * [15] Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh, “Openpose: Realtime multi-person 2d pose estimation using part affinity fields,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2019.
  * [16] G. Corso, L. Cavalleri, D. Beaini, P. Liò, and P. Veličković, “Principal neighbourhood aggregation for graph nets,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 13 260–13 271, 2020.
  * [17] M. Hassan, V. Choutas, D. Tzionas, and M. J. Black, “Resolving 3d human pose ambiguities with 3d scene constraints,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 2282–2292.
  * [18] D. Rempe, T. Birdal, A. Hertzmann, J. Yang, S. Sridhar, and L. J. Guibas, “Humor: 3d human motion model for robust pose estimation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 11 488–11 499.
  * [19] S. Shimada, V. Golyanik, W. Xu, and C. Theobalt, “Physcap: Physically plausible monocular 3d motion capture in real time,” _ACM Transactions on Graphics (ToG)_ , vol. 39, no. 6, pp. 1–16, 2020.
  * [20] M. Hassan, Y. Guo, T. Wang, M. Black, S. Fidler, and X. B. Peng, “Synthesizing physical character-scene interactions,” _arXiv preprint arXiv:2302.00883_ , 2023.
  * [21] Z. Luo, S. Iwase, Y. Yuan, and K. Kitani, “Embodied scene-aware human pose estimation,” _arXiv preprint arXiv:2206.09106_ , 2022.
  * [22] M. Hassan, P. Ghosh, J. Tesch, D. Tzionas, and M. J. Black, “Populating 3d scenes by learning human-scene interaction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 14 708–14 718.
  * [23] Y. Zhang, M. Hassan, H. Neumann, M. J. Black, and S. Tang, “Generating 3d people in scenes without people,” in _Computer Vision and Pattern Recognition (CVPR)_ , June 2020. [Online]. Available: https://arxiv.org/abs/1912.02923
  * [24] A. Monszpart, P. Guerrero, D. Ceylan, E. Yumer, and N. J. Mitra, “imapper: interaction-guided scene mapping from monocular videos,” _ACM Transactions On Graphics (TOG)_ , vol. 38, no. 4, pp. 1–15, 2019.
  * [25] Y. Chen, S. Huang, T. Yuan, S. Qi, Y. Zhu, and S.-C. Zhu, “Holistic++ scene understanding: Single-view 3d holistic scene parsing and human pose estimation with human-object interaction and physical commonsense,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 8648–8657.
  * [26] J. Y. Zhang, S. Pepose, H. Joo, D. Ramanan, J. Malik, and A. Kanazawa, “Perceiving 3d human-object spatial arrangements from a single image in the wild,” in _European Conference on Computer Vision_. Springer, 2020, pp. 34–51.
  * [27] X. Xie, B. L. Bhatnagar, and G. Pons-Moll, “Chore: Contact, human and object reconstruction from a single rgb image,” _CVPR_ , 2022.
  * [28] R. Dabral, S. Shimada, A. Jain, C. Theobalt, and V. Golyanik, “Gravity-aware monocular 3d human-object reconstruction,” _arXiv preprint arXiv:2108.08844_ , 2021.
  * [29] Z. Li, J. Liu, Z. Zhang, S. Xu, and Y. Yan, “Cliff: Carrying location information in full frames into human pose and shape estimation,” _arXiv preprint arXiv:2208.00571_ , 2022.
  * [30] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black, “Smpl: A skinned multi-person linear model,” _ACM transactions on graphics (TOG)_ , vol. 34, no. 6, pp. 1–16, 2015.
  * [31] H. Hu, J. Gu, Z. Zhang, J. Dai, and Y. Wei, “Relation networks for object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 3588–3597.
  * [32] N. Kolotouros, G. Pavlakos, M. J. Black, and K. Daniilidis, “Learning to reconstruct 3d human pose and shape via model-fitting in the loop,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 2252–2261.
  * [33] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [34] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. Osman, D. Tzionas, and M. J. Black, “Expressive body capture: 3d hands, face, and body from a single image,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 10 975–10 985.
  * [35] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black, “Amass: Archive of motion capture as surface shapes,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 5442–5451.
  * [36] K. P. F.R.S., “Liii. on lines and planes of closest fit to systems of points in space,” _The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science_ , vol. 2, no. 11, pp. 559–572, 1901.
  * [37] M. Fey and J. E. Lenssen, “Fast graph representation learning with PyTorch Geometric,” in _ICLR Workshop on Representation Learning on Graphs and Manifolds_ , 2019.
  * [38] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.