from $\Psi$. By this way, the nodes are recursively combined to generate the hierarchical scene structure until all the object categories in an image are combined into a root node.

Fig. 3 illustrates the process of RsNN in merging two nodes $\mathbf{v}_k$ and $\mathbf{v}_l$. In general, the RsNN model is composed of five subnetworks: (i) transition mapper, (ii) combiner, (iii) interpreter, (iv) categorizer, and (v) scorer. The *transition mapper* is a one-layer fully-connected neural network to generate $\mathbf{x}_k$ and $\mathbf{x}_l$ from $\mathbf{v}_k$ and $\mathbf{v}_l$. Based on $\mathbf{x}_k$ and $\mathbf{x}_l$, the *combiner* is used to obtain the feature representation $\mathbf{x}_{kl}$. Then, both $\mathbf{x}_{kl}$ and their contextual information $\mathbf{b}_{kl}$ are considered in the *interpreter* to produce the enhanced feature representation $\mathbf{g}_{kl}$. Finally, the *categorizer* and *scorer* are used to predict the relation label and confidence score for merging $\mathbf{v}_k$ and $\mathbf{v}_l$. In the following, we further present more detailed explanation on each subnetwork.

**Network Annotations.** Following [@DBLP:Recursive_Socher] and [@DBLP:RecursiveContext2], object feature $\mathbf{v}_k$ produced by CNN is first mapped onto a transition space by the ***Transition Mapper***, which is a one-layer fully-connected neural network. $$\label{eq_sem}
\mathbf{x}_k = F_{tran}(\mathbf{v}_k; \mathbf{W}_{tran})$$ where $\mathbf{x}_k$ is the mapped feature, $F_{tran}$ is the network transformation and $\mathbf{W}_{tran}$ indicates 