part-of-speech, preserving only nouns as object candidates, and verbs and prepositions as relation candidates (*i.e.* the tree in the middle of Fig. \[fig:language\]). In the third step, nouns are converted to object categories. Note that sometimes different nouns (e.g. “cat” and “kitten”) represent the same category. The lexical relation in WordNet [@wordnet] is employed to unify the synonyms belonging to the same defined category. The entities that are not in any defined object categories (e.g. “grass” in “a sheep stands on the grass”) are also removed from the trees. In the fourth step, relations are also recognized and refined. Let $\mathcal{R}$ denote a set of defined relations. We provide the list of relations we defined for different datasets in Table \[tbl:relation\]. Note that $\mathcal{R}$ also includes an extra relation category, *i.e.* “others”, to denote all the other relations that are not explicitly defined. Let $\mathcal{T}$ be the set of triplets with the form of $(entity1, verb/prep, entity2)$. We construct a mapping $\mathcal{T} \rightarrow \mathcal{R}$ to recognize the relations and construct the semantic tree (i.e., the tree in the bottom of Fig. \[fig:language\]).

![image](fig_learning.pdf){width="1.0\linewidth"}

![An illustration of the tree conversion process. The top is the constituency tree generated by language parser, 