losses. In the M-step, the model parameters are updated via back-propagation by minimizing the intermediate CNN and RsNN losses.

Sentence Preprocessing {#sub:preprocessing}
----------------------

For guiding semantic labeling and scene configuration, we convert each sentence into a semantic tree by using some common techniques in natural language processing. As shown in the bottom of Fig. \[fig:language\], a semantic tree $T$ only includes both entity labels (*i.e.* nouns) and their interaction relations (i.e., verb/ prepositional phrases). Therefore, in sentence preprocessing, we first generate the constituency tree from the descriptive sentence, and then remove the irrelevant leaf nodes and recognize the entities and relations to construct the semantic tree.

The conversion process generally involves four steps. In the first step, we adopt the Stanford Parser [@DBLP:conf/acl/SocherBMN13] to generate the constituency tree (*i.e.* the tree in the top of Fig. \[fig:language\]) from the descriptive sentence. Constituency trees are two-way trees with each word in a sentence as a leaf node and can serve as suitable alternative of structured image tree annotation. However, such constituency trees inevitably contain irrelevant words (e.g., adjectives and adverbs) that do not denote semantic entities or interaction relations. Thus, in the second step, we filter the leaf nodes by their 