representations of semantic objects. Then, the RsNN is sequentially stacked based on the CNN feature representations, generating the structured configuration of the scene.

On the other hand, to avoid affording the elaborative annotations, we propose to train our CNN-RsNN model by leveraging the image-level descriptive sentences. Our model training approach is partially motivated but different from the recently proposed methods for image-sentence embedding and mapping [@DBLP:VisualAlign-feifei][@DBLP:TellMeShowYou], since we propose to transfer knowledge from sentence descriptions to discover the scene configurations.

In the initial stage, we decompose each sentence into a semantic tree consisting of nouns and verb phrases with a standard parser [@DBLP:conf/acl/SocherBMN13], WordNet [@wordnet] and a post-processing method. Then, we develop an Expectation-Maximization-type learning method for model training based on these semantic trees and their associated scene images. Specifically, during the weakly-supervised training, the semantic tree facilitators discover the latent scene configuration in the two following aspects: 1) the objects (*i.e.*, nouns) determine the object category labels existing in the scene, and 2) the relations (*i.e.*, verb phrases) among the entities help produce the scene hierarchy and object interactions. Thus, the learning algorithm iterates in three steps. (i) Based on the object labels extracted from the sentence, it estimates 