details in this section. Our implementation is based on the PyTorch [@paszke2017automatic] library. The feature encoding module follows the base architecture of the first five convolutional blocks (*conv1 $\sim$ conv5*) of VGG16 [@vgg] network. We remove the last max-pooling layer in the 5-th convolutional block. We initialize these convolutional layers using pre-trained weights on ImageNet [@deng2009imagenet]. Following [@simonyan2014two], since the input channels of T-CNN is changed to 20, we average the weights of the first convolution layer of T-CNN part. The saliency prediction module is a set of 5 convolution layer groups following the inverse order of VGG16 while changing all max pooling layers into upsampling layers. We change the last layer to output 1 channel and add sigmoid activation on top. Since the input of the saliency prediction module contains latent representations from both S-CNN and T-CNN, we use a 3d convolution layer (with a kernel size of $1 \times 3 \times 3$) and a 3d pooling layer (with a kernel size of $2 \times 1 \times 1$) to fuse the inputs. Thus, the input and output sizes are all 224 $\times$ 224. The fixation state predictor is a set of fully connected (FC) layers, whose output sizes are 