method include not only the objects but also the interaction relations of objects. Second, we introduce a novel objective function to discover the scene structure. Third, we incorporate convolutional feature learning into our deep model for joint optimization.

![image](fig_inference_new.pdf){width="2\columnwidth"}

Most of the existing scene labeling / parsing models are studied in the context of supervised learning, and they rely on expensive annotations. To overcome this issue, one can develop alternative methods that train the models from weakly annotated training data, e.g., image-level tags and contexts [@DBLP:Weakly2][@DBLP:Weakly-MultiInstance][@DBLP:WeaklySegmentation][@DBLP:CCNN]. Among these methods, the one that inspires us is [@DBLP:WeaklySegmentation], which adopted an EM learning algorithm for training the model with image-level semantic labels. This algorithm alternated between predicting the latent pixel labels subject to the weak annotation constraints and optimizing the neural network parameters. Different from this method, our model applies the sentence description to label the salient semantic object in the image. By employing such knowledge transfer, the model can deal with object labeling and relation prediction simultaneously according to human perception.

CNN-RsNN Architecture {#CNN-RNN Architecture}
=====================

This work aims to jointly solve three tasks: semantic labeling, scene structure generation, and the inter-object relation prediction. To achieve these goals, we propose a 