objects when they are combined into a higher-level object. Please refer to Fig. \[fig:inference\] for more information about the proposed architecture. In the following, we discuss the CNN and RsNN models in details.

CNN Model {#sub:cnn_model}
---------

The CNN model is designed to accomplish two tasks: semantic labeling and generating feature representations for objects. For semantic labeling, we adopt the fully convolutional network with parameters $\mathbf{W}_C$ to yield $K+1$ score maps $\{\mathbf{s}^0, ..., \mathbf{s}^k, ..., \mathbf{s}^K\}$, corresponding to one extra background category and $K$ object categories. Following the holistically-nested architecture in [@DBLP:HNED-tuzhuowen] we adopt $E=3$ multi-scale prediction streams, and each stream is associated with $K+1$ score maps with the specific scale. Let $s_j^{t,e}$ indicate the score value at pixel $j$ in the $t$-th map of $e$-th scale. We normalize $s_j^{t,e}$ in the $e$-th stream using softmax to obtain the corresponding classification score: $$\label{eq_softmax}
\sigma_e(s_j^{t,e}) = \frac{\exp(s_j^{t,e})} {\sum_{k=0}^K \exp(s_j^{k,e})}$$ Then the final classification score $\sigma_f(s_j^{t})$ is further calculated by $\sigma_f(s_j^{t}) = \sum_{e=1}^E \alpha_e \:\sigma_e(s_j^{t,e})$, where $\alpha_e>0 $ is the fusion weight for the $e$-th stream, and $\sum_{e=1}^{E}  \alpha_e = 1$. The learning of this fusion weight is equivalent to training $1\times1$ convolutional filters on the concatenated score maps from all multi-scale 