Semantic Label Loss {#sub:cnn_loss}

The goal of semantic labeling is to assign the category labels to each pixel. Let $\mathbf{C}^f$ be the final predicted semantic map, $\mathbf{C}^e$ the $e$-th semantic map of the multi-scale prediction streams. The semantic label loss for an image $\mathbf{I}$ is defined as, $$\label{eq_cnn_loss}
\begin{split}
&\mathcal{J}_{C}(\mathbf{W}_C; \mathbf{I}, \mathbf{C}) = \frac{ \sum_{e=1}^E\mathcal{L}_e(\mathbf{C}, \mathbf{C}^e)}{E} + \mathcal{L}_f(\mathbf{C}, \mathbf{C}^f )
\end{split}$$ where $\mathcal{L}_f$ indicates the loss generated by the final predicted semantic map $\mathbf{C}^f$. Each element in $\mathbf{C}^f$ is calculated by Eqn. (\[eq\_softmax\]), and we have $\mathbf{C}^{t,f}(j) = \sigma_f(s_j^{t})$. $\mathbf{C}$ is the groundtruth label map. By considering the multi-scale prediction streams, we also define the loss $\mathcal{L}_e, \{e = 1, 2, ..., E\}$ for multiple feature streams (*i.e.* the red line in Fig. \[CNN-RNN Architecture\]). Same as the $\mathbf{C}^f$, each element in $\mathbf{C}^e$ is defined by $\mathbf{C}^{t,e}(j) = \sigma_e(s_j^{t,e})$. The cross entropy is adopted in $\mathcal{L}_f$ and $\mathcal{L}_e$ as the error measure.

### Scene Structure Loss {#sub:rnn_loss}

The purpose of constructing scene structure is to generate the meaningful configurations of the scene and predict the interaction relations of the objects in the scene. To achieve this goal, the scene structure loss can be divided into two parts: one for scene 