Training a scene parsing model usually relies on very expensive manual annotations, e.g., labeling pixel-wise semantic maps, hierarchical representations and inter-object relations.

To address these above issues, we develop a novel deep neural network architecture for hierarchical scene parsing. Fig. \[fig:application\] shows a parsing result generated by our framework, where a semantic object hierarchy with object interaction relations is automatically parsed from an input scene image. Our model is inspired by the effectiveness of two widely successful deep learning techniques: convolutional neural networks (CNN) [@DBLP:AlexNet][@DBLP:FCnetwork] and recursive neural network (RsNN) [@DBLP:Recursive_Socher]. The former category of models is widely applied for generating powerful feature representations in various vision tasks such as image classification and object detection. Meanwhile, the RsNN models (such as  [@DBLP:Recursive_Socher][@DBLP:RecursiveContext][@DBLP:RecursiveContext2]) have been demonstrated as an effective class of models for predicting hierarchical and compositional structures in image and natural language understanding [@DBLP:RNN-NLP]. One important property of RsNN is the ability to recursively learn the representations in a semantically and structurally coherent way. In our deep CNN-RsNN architecture, the CNN and RsNN models are collaboratively integrated for accomplishing the scene parsing from complementary aspects. We utilize the CNN to extract features from the input scene image and generate the 