likely to attract human attention and thus gaze fixation [@borji2013state]. Traditional saliency models are based on the feature integration theory [@treisman1980feature] telling that an image region with high saliency contains distinct visual features such as color, intensity and contrast compared to other regions. After Itti *et al.*’s primary work [@itti1998model] on a computational saliency model, various bottom-up computational models of visual saliency have been proposed such as a graph-based model [@harel2007graph] and a spectral clustering-based model [@hou2012image]. Recent saliency models [@lin2014saliency][@huang2015salicon][@pan2016shallow] leveraged a deep Convolutional Neural Network (CNN) to improve their performance. More recently, high-level context has been considered in deep learning-based saliency models. In [@simonyan2013deep][@cao2015look], class labels were used to compute the partial derivatives of CNN response with respect to input image regions to obtain a class-specific saliency map. In [@zhao2015saliency], a salient object is detected by combining global context of the whole image and local context of each image superpixel. In [@ramanishka2017top], region-to-word mapping in a neural saliency model was learned by using image captions as high-level input.

However, none of the previous methods explored the patterns in the transition of human attention inherent in a complex task. In this work, we propose to learn the task-dependent attention 