predicted scene hierarchy to be consistent with the groundtruth. Specifically, the score of a correct merging operation is required to be larger than that of any incorrect merging operation $\widehat{a}$ with a constant margin $\triangle$, i.e., $q(a)\geq q(\widehat{a}) + \triangle$. Thus, we define the loss for scene hierarchy construction as, $$\label{eq_structure_loss}
\begin{split}
\mathcal{J}_{struc}(\mathbf{W}; \mathbf{I}, T^S) & =  \frac{1}{P_S} \sum_{p=1}^{P_S} [ \:\: \max_{\widehat{a}_p\notin\mathcal{A}(\mathbf{I},T^S)}
q(\widehat{a}_p)\\
& - q(a_p) + \triangle \:]
\end{split}$$ Intuitively, this loss intends to maximize the score of correct merging operation while minimizing the scores of incorrect merging operations. To improve efficiency, only the highest score of the incorrect merging operation is considered during training.

**Relation Categorization.** Denote by $\{kl\}$ the combination of two child nodes $k$ and $l$. Let $y_{kl}$ be the softmax classification result by the Categorizer sub-network in Eqn. , and $\widehat{y}_{kl}$ be the groundtruth relation from $T^R$. The loss on relation categorization is then defined as the cross entropy between $y_{kl}$ and $\widehat{y}_{kl}$, $$\label{eq_relation_loss}
\begin{split}
& \mathcal{J}_{rel}(\mathbf{W}; \mathbf{I}, T^R) = \frac{1}{|N_R|}  \sum_{\{kl\}} \mathcal{L}_r (\widehat{y}_{kl},y_{kl} )
\end{split}$$ where $y_{kl}$ is the predicted relation probability in Eqn. (\[eq\_node\_score\]). $|N_R|$ denotes the number of relations in $T^R$.

  \
Training samples ($\mathbf{I_1},T_1$),($\mathbf{I_2},T_2$),...,($\mathbf{I_Z},T_Z$).   \
The parameters of our CNN-RsNN model 