model, HST used a quantized grammar, rather than the neural networks which can adopt the transfer learning to obtain better initialization for higher training efficiency.

With the resurgence of neural network models, the performances of scene understanding have been improved substantially. The representative works, the fully convolutional network (FCN) [@DBLP:FCnetwork] and its extensions [@DBLP:CNN-CRF], have demonstrated effectiveness in pixel-wise scene labeling. A recurrent neural network model was proposed in [@DBLP:CRF-RNN], which improved the segmentation performance by incorporating the mean-field approximate inference, and similar idea was also explored in [@DBLP:CNN-MRF]. For the problem of structured scene parsing, recursive neural network (RsNN) was studied in [@DBLP:Recursive_Socher][@DBLP:RecursiveContext2]. For example, Socher et al. [@DBLP:Recursive_Socher] proposed to predict hierarchical scene structures with a max-margin RsNN model. Inspired by this work, Sharma et al. proposed the deep recursive context propagation network (RCPN) in [@DBLP:RecursiveContext] and [@DBLP:RecursiveContext2]. This deep feed-forward neural network utilizes the contextual information from the entire image to update the feature representation of each superpixel to achieve better classification performance. The differences between these existing RsNN-based parsing models and our model are three folds. First, they mainly focused on parsing the semantic entities (e.g., buildings, bikes, trees), while the scene configurations generated by our 