4096,1024,2 sequentially. The LSTM is a 3-layer LSTM whose input and output sizes are both 512. The late fusion module consists of 4 convolution layers followed by sigmoid activation. The first three layers have a kernel size of 3 $\times$ 3, 1 zero padding, and output channels 32,32,8 respectively, and the last convolution layer has a kernel size of 1 with a single output channel. We empirically set both the height $H_c$ and width $W_c$ for cropping the latent representations to be 3.

The whole model is trained using Adam optimizer [@kingma2014adam] with its default settings. We fix the learning rate as 1e-7 and first train the saliency prediction module for 5 epochs for the module to converge. We then fix the saliency prediction module and train the LSTM-based weight predictor and the fixation state predictor in the attention transition module. Learning rates for other modules in our framework are all fixed as 1e-4. After training the attention transition module, we fix the saliency prediction and the attention transition module to train the late fusion module in the end.

Experiments
===========

We first evaluate our gaze prediction model on two public egocentric activity datasets (**GTEA Gaze** and **GTEA Gaze Plus**). 