streams. $\sigma_f(s_j^t)$ denotes the probability of $j$-th pixel belonging to $t$-th object category with $\sum_{t=1}^K \sigma_f(s_j^t)=1$. The set $\{c_j\}_{j=1}^M$ denotes the predicted labels of pixels in the image $\mathbf{I}$, where $c_j\in \{0,...,K\}$ and $M$ is the number of pixels of image $\mathbf{I}$. With $\sigma_f(s_j^t)$, the label of the $j$-th pixel can be predicted by:

$$\label{eq_prediction}
c_j = \arg\max_t \: \sigma_f(s_j^t)$$

To generate feature representation for each entity category, we group the pixels with the same label into one semantic category.

Considering that the pixel numbers vary with the semantic entity categories, the pooling operation is generally required to obtain fixed-length representation for any object category. Conventional sum-pooling treats feature representation from different pixels equally, while max-pooling only considers the most representative one and ignores the contribution of the other. For the tradeoff between sum-pooling and max-pooling, we use *Log-Sum-Exp* (LSE), a convex approximation of the *max* function, as the pooling operator to fuse the features of pixels, $$\label{eq_feature}
\mathbf{v}_k = \frac{1}{\pi}\log \left[\sum_{c_j = k} \exp (\pi \bar{\mathbf{v}}_j) \right]$$ where $\mathbf{v}_k$ denotes the feature representation of the $k$-th entity category, $\bar{\mathbf{v}}_j$ denotes the feature representation of the $j$-th pixel by concatenating all feature maps at the layer before softmax at position 