the input image and the labeling results of two entities, *i.e*, motorcycle and person. The center of each entity is also given, *i.e.* $\alpha_1$ and $\alpha_2$. Based on the centers and labeling results, the bottom row illustrates three spatial relations, *i.e.*, distance $\gamma$, relative angle $\theta$, and area ratio $\beta_1 / \beta_2$, to characterize the contextual information between the two entities.[]{data-label="fig:context"}](context.pdf){width="0.7\linewidth"}

Model Training {#sec:training}
==============

Fully supervised training of our CNN-RsNN model requires expensive manual annotations on pixel-level semantic maps, inter-object relations, and hierarchical structure configuration. To reduce the burden on annotations, we present a weakly-supervised learning method to train our CNN-RsNN by leveraging a much cheaper form of annotations, *i.e.*, image-level sentence description. To achieve this goal, the descriptive sentence is first converted to the semantic tree to provide weak annotation information. Then we formulate the overall loss function for structured scene parsing based on the parsing results and the semantic trees. Finally, an Expectation-Maximization (EM) algorithm is developed to train CNN-RsNN by alternatively updating structure configuration and network parameters. In the E-step, guided by the sentence description, we update scene configurations (*i.e.*, intermediate label map $\widehat{\mathbf{C}}$, scene hierarchy and inter-object relations) together with the intermediate CNN and RsNN 