an intermediate label map by inferring the classification probability of each pixel. Multi-scale information of the image is adopted to improve the accuracy. (ii) With the label map, the model groups the pixels into semantic objects and predicts the scene hierarchy and inter-object relations through the RsNN. (iii) With the fixed scene labeling and structure, it updates the parameters of the CNN and RsNN by back propagation.

![An example of structured scene parsing generated by our framework. An input scene image is automatically parsed into a structured configuration that comprises hierarchical semantic objects (black labels) and the interaction relations (red labels) of objects.[]{data-label="fig:application"}](fig_application.pdf){width="3.2in"}

The main contributions of our work are summarized as follows. i) We present a novel CNN-RsNN framework for generating meaningful and hierarchical scene representations, which helps gain a deeper understanding of the objects in the scene compared with traditional scene labeling. The integration of CNN and RsNN models can be extended to other high-level computer vision tasks. ii) We present a EM-type training method by leveraging descriptive sentences that associate with the training images. This method is not only cost-effective but also beneficial to the introduction of rich contexts and semantics. iii) The advantages of our method 