the middle is the constituency tree after POS tag filtering, and the bottom is the converted semantic tree.[]{data-label="fig:language"}](fig_language.pdf){width="3.5in"}

Loss Functions
--------------

Before introducing the weakly supervised training algorithm, we first define the loss function in the fully supervised setting. For each image $\mathbf{I}_i$, we assume that both the groundtruth semantic map $\mathbf{C}_i$ and the groundtruth semantic tree $T_i$ are known. Then, the loss function is defined as the sum of three terms: semantic label loss $\mathcal{J}_{C}$, scene structure loss $\mathcal{J}_{R}$, and regularizer $R(\mathbf{W})$ on model parameters. With a training set containing $N$ images $\{(\mathbf{I}_1,\mathbf{C}_1, T_1),...,(\mathbf{I}_N, \mathbf{C}_N, T_N)\}$, the overall loss function can be defined as, $$\label{eq_overallloss}
\mathcal{J}(\mathbf{W})  =  \frac{1}{N} \sum_{i=1}^N ( \mathcal{J}_{C}(\mathbf{W}_C;\mathbf{I}_i,\mathbf{C}_i) + \mathcal{J}_{R}(\mathbf{W}; \mathbf{I}_i, T_i) ) + \lambda R(\mathbf{W})$$ where $\mathbf{I}_i$ is the $i$-th image. $T_i$ is the groundtruth semantic tree including both hierarchical scene structure and inter-object relation. $\mathbf{W} = \{\mathbf{W}_C, \mathbf{W}_R\}$ denotes all model parameters. $\mathbf{W}_C$ and $\mathbf{W}_R$ are the model parameters of the CNN and RsNN, respectively. Note that $\mathbf{W}_R$ includes the parameters of the five subnetworks defined in Sec.3.2, *i.e*. $\mathbf{W}_R = \{ \mathbf{W}_{tran}, \mathbf{W}_{com}, \mathbf{W}_{int}, \mathbf{W}_{cat}, \mathbf{W}_{score} \}$. The regularization term is defined as $R(\mathbf{W}) = \frac{\lambda}{2}||\mathbf{W}||^2$ and $\lambda$ is the regularization parameter.

### 