$\mathbf{W}$

  \
Initialize the CNN model with the pre-trained networks on ImageNet\
Initialize the RsNN model with Gaussian distribution

-   Estimate the intermediate semantic maps $\{\widehat{\mathbf{C}}_i\}_{i=1}^Z$ according to Algorithm \[alg:algorithm\]

-   Predict the scene hierarchy and inter-object relations for each image $\mathbf{I}_i$

-   Replace the groundtruth semantic maps $\{\mathbf{C}_i\}_{i=1}^Z$ in Eqn. (\[eq\_overallloss\]) with intermediate semantic maps $\{\widehat{\mathbf{C}}_i\}_{i=1}^Z$.

-   Update the parameters $\mathbf{W}$ according to Eqn. (\[eq\_overallloss\])

The optimization algorithm converges

  \
Annotated entities $T^E$ in the semantic tree, normalized prediction score $\sigma_e(s_j^{k,e})$ and final prediction score $\sigma_f(s_j^{k})$, where $j\in\{1,..,M\}$, $k\in\{0,..,K\}$, $e\in\{1,...,E\}$.   \
Intermediate label map $\widehat{\mathbf{C}}=\{\widehat{c}_j\}_{j=1}^M$

  \
(1) To simplify, let $f$ be the $E+1$ scale.\
(2) Set $\psi^{k,e} = 0$ and $G_j^e(k) = \log \sigma_e(s_j^{k,e})$ for all $e\in \{1,..,E+1\}$ and $k \in \{0,...,K\}$;\
(3) Let $\rho_{bg},\rho_{fg}$ indicate the number of pixels being assigned to background and foreground. Set $\rho_k = \rho_{bg}$ if $k=0$, $\rho_k = \rho_{fg}$ if $k\in\{1,...,K\}$.

-   Compute the maximum score at each position $j$,\
    $[G_j^e]_{max} = \max_{k\in T^E} G_j^e(k)$

-   **repeat**

    **if** the $k$-th semantic category appears in annotated entities $T^E$,\

    -   Set $\delta_j^{k,e} = [G_j^e]_{max} - G_j^e(k)$.

    -   Rank $\{\delta_j^{k,e}\}_{j=1}^M$ according to the ascending sorting and obtain the ranking list.

    -   Select $\delta_i^{k,e}$ in the 