of our EM algorithm is described as follows:

**(i) Estimate the intermediate semantic map $\widehat{\mathbf{C}}$.** As illustrated in Fig. \[fig:learning\] (*i.e.* blue part), the input image $\mathbf{I}$ first goes through the convolutional neural network to generate predicted semantic map. Then the intermediate semantic map $\widehat{\mathbf{C}}$ is estimated based on the predicted map the annotated entities $T^E$, $$\label{eq_adaptive}
\begin{split}
\widehat{\mathbf{C}} = \arg\max_{\mathbf{C}} \log P(\mathbf{C}|\mathbf{I}; \mathbf{W}_C^{'}) + \log P(T^E|\mathbf{C}).
\end{split}$$ The classification probability $P(\mathbf{C}|\mathbf{I}; \mathbf{W}_C^{'})$ of each pixel can be computed using Eqn. . Inspired by the effectiveness of cardinality potentials [@DBLP:conf/uai/TarlowSZAF12][@DBLP:conf/icml/LiZ14], we define $\log P(T^E|\mathbf{C})$ as entity-dependent bias $\psi^{k}$ for the class label $k$, and set $\psi^{k}$ adaptively in a manner similar to [@DBLP:WeaklySegmentation].

For multi-scale prediction streams, the score in the $e$-th stream is calculated by $G_j^e(k) = \log \sigma_e(s_j^{k,e}) + \psi^{k,e}$. The fused score is $G_j^f(k) = \log \sigma_f(s_j^{k})+ \psi^{k,f}$. Then the intermediate label of pixel $j$ can be estimated by, $$\label{eq_multifusion}
\begin{split}
\widehat{c}_j=\arg\max_k \: \left[ \: \sum_{e=1}^E G_j^e(k) + G_j^f(k) \: \right]
\end{split}$$ Algorithm \[alg:algorithm\] summarizes our semantic map estimation method.

**(ii) Predict the object hierarchy and inter-object relations.** Given the semantic labeling result, we group the pixels into semantic objects and obtain the object feature representations according 