as 10 following [@simonyan2014two].

The feature encoding network of S-CNN and T-CNN follows the base architecture of the first five convolutional blocks in Two Stream CNN [@simonyan2014two], while omitting the final max pooling layer. We choose to use the output feature map of the last convolution layer from the 5-th convolutional group, i.e., *conv5\_3*. Further analysis of different choices of deep feature maps from other layers is described in Section \[Experiment:AA\].

Saliency Prediction Module
--------------------------

Biologically, human tends to gaze at an image region with high saliency, i.e., a region containing unique and distinctive visual features [@sugano2013appearance]. In the saliency prediction module of our gaze prediction model, we learn to generate a visual saliency map which reflects image regions that are likely to attract human gaze. We fuse the latent representations $F^{S}_t$ and $F^{T}_t$ as an input to a saliency prediction decoder (denoted as $S$) to obtain the initial gaze prediction map $G^s_t$ (Eq. \[equa:sp\]). We use the “3dconv + pooling" method of [@feichtenhofer2016convolutional] to fuse the two input feature streams. Since our task is different from [@feichtenhofer2016convolutional], we modify the kernel sizes of the fusion part, which can be seen in detail in Section \[details\]. The decoder outputs a 