$\rho_k$-th position of the ranking list, and let $\psi^{k,e} = \delta_i^{k,e}$

    **else** Set $\psi^{k,e} = -\infty$ to suppress the labels not present in $T^E$.\
     \
    Update $G_j^e(k)$ with $G_j^e(k) = \log \sigma_e(s_j^{k,e})+\psi^{k,e}$.\
     \
    **until** Handling all of $K+1$ semantic categories.

Updating all of the prediction score in $E+1$ scales.\

Calculate the intermediate label of each pixel using Eqn. (\[eq\_adaptive\])

EM Method for Weakly Supervised Learning {#sub:em_learning}
----------------------------------------

In our weakly supervised learning setting, the only supervision information is the descriptive sentence for each training image. By converting the descriptive sentence to the semantic tree $T$, we can obtain the entities $T^E$ (i.e., nouns), the relations $T^R$ (i.e., verbs or prepositional phrases) and the composite structure $T^S$ between entities, but cannot directly get the semantic map $\mathbf{C}$. Therefore, we treat the semantic labeling map $\mathbf{C}$ as latent variable and adopt a hard EM approximation for model training. In the E-step, we estimate the intermediate semantic map $\widehat{\mathbf{C}}$ based on the previous model parameters and the annotated entities $T^E$, and replace the $\mathbf{C}_i$ in Eqn. (\[eq\_overallloss\]) with its estimate $\widehat{\mathbf{C}}_i$. In the M-step, mini-batch SGD is deployed to update the CNN and RsNN parameters by minimizing the overall loss function. The detail 