system, such as implying a person where to look next in performing a complex task. We conduct a new experiment on the GTEA-sub dataset (as introduced in Section \[gtea-sub\]) to test the attention transition module of our model. Since here we focus on the module’s ability of attention transition, we omit the fixation state predictor in the module and assume the output of the fixation state predictor as $f_{t}=0$ in the test frame. The module takes $w_{t}$ calculated from the region of current fixation as input and outputs an attention map on the same frame which represents the predicted region of the next fixation. We extract a 2D position from the maximum value of the predicted heatmap and calculate its rate of falling within the annotated bounding box as the transition accuracy.

We conduct experiments based on different latent representations extracted from the convolutional layer: *conv5\_1*, *conv5\_2*, and *conv5\_3* of S-CNN. The accuracy based on the above three convolutional layers are 71.7%, 83.0%, and 86.8% respectively, while the accuracy based on random position is 10.7%. We also tried using random channel weight as the output of channel weight predictor to compute attention map based on the latent representation of *conv5\_3*, 