novel deep CNN-RsNN architecture. The CNN model is introduced to perform semantic segmentation by assigning an entity label (*i.e.* object category) to each pixel, and the RsNN model is introduced to discover hierarchical structure and interaction relations among entities.

Fig. \[fig:inference\] illustrates the proposed CNN-RsNN architecture for structured scene parsing. First, the input image $\mathbf{I}$ is directly fed into revised VGG-16 network [@vggnet] to produce different levels of feature maps. According to these feature maps, multi-scale prediction streams are combined to produce final score maps $\mathcal{S} = \{\mathbf{s}^0, ..., \mathbf{s}^k, ..., \mathbf{s}^K\}$ for object categories. Based on the softmax normalization of score maps, the $j$-th pixel is assigned with an object label $c_j$. We further group the pixels with the same label into an object, and obtain the feature representations of objects. By feeding these feature representations of objects to the RsNN, a greedy aggregation procedure is implemented for constructing the parsing tree $\mathcal{P}_I$. In each recursive iteration, two input objects (denoted by the child nodes) are merged into a higher-level object (denoted by the parent node), and generated root note represents the whole scene. Different from the RsNN architecture in [@DBLP:Recursive_Socher][@DBLP:RecursiveContext2], our model also predicts the relation between two 