hierarchy construction and the other for relation prediction, $$\label{eq_rnn_loss}
    \mathcal{J}_{R}(\mathbf{W}; \mathbf{I}, T) = \mathcal{J}_{struc}(\mathbf{W}_1; \mathbf{I}, T^S) + \mathcal{J}_{rel}(\mathbf{W}_2; \mathbf{I}, T^R)$$ where $T^S$ and $T^R$ indicate the groundtruth of hierarchical scene structure and inter-object relations, respectively. $\mathbf{W}_1 = \{ \mathbf{W}_C, \mathbf{W}_{tran}, \mathbf{W}_{com}, \mathbf{W}_{int}, \mathbf{W}_{score} \}$ and $\mathbf{W}_2 = \{ \mathbf{W}_C, \mathbf{W}_{tran}, \mathbf{W}_{com}, \mathbf{W}_{int}, \mathbf{W}_{cat} \}$. The above two items are jointly used to optimize the parameters of CNN and RsNN. The difference is that $\mathbf{W}_{score}$ in Eqn. (\[eq\_node\_score\]) and $\mathbf{W}_{cat}$ in Eqn. (\[eq\_rel\_category\]) are optimized by the structure loss and relation loss, respectively.

**Scene Hierarchy Construction.** Scene hierarchy construction aims to learn a transformation $\mathbf{I} \rightarrow \mathcal{P}_I$. The predicted scene hierarchy $\mathcal{P}_I$ is said to be valid if the merging order between regions is consistent with that in the groundtruth hierarchical scene structure. Given the groundtruth hierarchical scene structure $T^S$, we extract a sequence of “correct” merging operations as $\mathcal{A}(\mathbf{I}, T^S) = \{a_1,...,a_{P_S}\}$, where $P_S$ is the total number of merging operation. Given an operation $a$ on the input image $\mathbf{I}$, we use $q(a)$ to denote the merging score produced by the Scorer sub-network. Based on the merging score $q(a)$ calculated in Eqn. (\[eq\_node\_score\]), we define the loss to encourage the 