# On estimating gaze by self-attention
augmented convolutions

Gabriel Lefundes<EMAIL_ADDRESS>Luciano Oliveira<EMAIL_ADDRESS>Intelligent Vision Research Lab,
Federal University of Bahia, Bahia, Brazil

###### Abstract

Estimation of 3D gaze is highly relevant to multiple fields, including but not
limited to interactive systems, specialized human-computer interfaces, and
behavioral research. Although recently deep learning methods have boosted the
accuracy of appearance-based gaze estimation, there is still room for
improvement in the network architectures for this particular task. Therefore
we propose here a novel network architecture grounded on self-attention
augmented convolutions to improve the quality of the learned features during
the training of a shallower residual network. The rationale is that self-
attention mechanism can help outperform deeper architectures by learning
dependencies between distant regions in full-face images. This mechanism can
also create better and more spatially-aware feature representations derived
from the face and eye images before gaze regression. We dubbed our framework
ARes-gaze, which explores our Attention-augmented ResNet (ARes-14) as twin
convolutional backbones. In our experiments, results showed a decrease of the
average angular error by 2.38% when compared to state-of-the-art methods on
the MPIIFaceGaze data set, while achieving a second-place on the EyeDiap data
set. It is noteworthy that our proposed framework was the only one to reach
high accuracy simultaneously on both data sets.

###### keywords:

Appearance-based gaze estimation , self-attention , augmented 2D convolution.

††journal: arxiv.org

##  1 Introduction

Gaze estimation is an active area of research within computer vision, and its
relevance spans a large array of fields. For instance, Gaze can be a valuable
source of information in behavioral and health research [1, 2, 3], augmented
and virtual reality [4, 5, 6], mobile applications [7, 5], human-computer
interaction [8], and even natural language processing (NLP) pipelines [9].

Methods of gaze estimation can be categorized as model-based or appearance-
based [10]. The former relies on explicitly modeling the subject’s eye and
using some type of feedback (usually with the need for specialized hardware
such as active infrared LEDs [11] or wearable devices [12]) to infer the gaze
direction geometrically. This approach can reach accurate results in
controlled environments, although suffering from hardware cost and
installation overhead. Also, model-based gaze estimation is usually limited by
external factors such as lighting conditions and lower tolerance for subject
pose and distance. In contrast, appearance-based approaches attempt to
directly predict the gaze vector from RGB images of the subject by mapping a
regression function that can be ultimately learned from data.

While challenges like lighting conditions and unconstrained subject pose
remain, the use of deep learning and in-the-wild large-scale data sets [13,
14, 15] have greatly improved the accuracy of appearance-based methods, which
in general only need simple monocular cameras as input sensors. Recent
publications in the field have focused on exploring different neural network
architectures and training conditions to raise the performance of the current
state-of-the-art. Notably, many works have remarked that by using full-face
images along with the usually extracted eye-patches as the input, can improve
the prediction accuracy significantly [16, 17, 18]. This is so since full-face
images carry relevant information about the subject’s pose.

Here, we explore the recent trend of attention mechanisms in deep learning
[19] as a way to produce higher quality features by improving the spatial
awareness of the network. The rationale is to better leverage the relationship
between coarse pose information from face images and fine information from
eye-patches.

###  1.1 Related works

Appearance-based gaze estimation: Early works in appearance-based gaze
estimation used well-established machine-learning algorithms like adaptive
linear regression [20], support vector regression [21], and random forests
[22] to learn the mapping function from eye images to gaze vectors. Recently,
convolutional neural networks (CNNs) have shown great success in gaze
estimation, with its first published iteration [13] reporting significant
gains over the previous state-of-the-art works. Subsequent publications have
then built upon the notion of using CNNs by proposing different input models
for the convolutional networks like images of the entire face and a binary
grid to encode head size and position [17]. Another explored avenue was to use
different strategies to combine features extracted from the eyes and the face.
Geometry constraints are used in [23] to connect head pose and eye movement in
an informed manner bound by physical limits of pose and movement. In [24], an
attention gating strategy is proposed as a way to refine gaze predictions made
on full-face images adaptively by using residuals from isolated-eye images in
a coarse-to-fine manner.

Other works have proposed taking into account domain knowledge and
peculiarities of the gaze estimation task while designing the architecture of
the CNN itself. For example, in [25], the asymmetrical nature of left and
right eyes is posited to have relevance on the result of gaze estimation, and
accuracy gains are reported when encoding and leveraging that asymmetry in a
deep neural network. Another example of domain-specific modeling is found in
[18] where dilated convolutions are used as a replacement for max-pooling
layers to better capture small differences in eye images. Since finer eye
movement is highly relevant to gaze estimation, it can easily be lost in the
downsampling stages of neural networks. In [26], recurrent CNNs are used and
shown to improve prediction accuracy significantly on continuous inference.
This is done so because it is plausible to consider gaze an inherently
temporal phenomenon, which is grounded by the notion that where people are
looking at, in a particular moment in time, directly depends on where they
were looking at, in a previous moment. In [16], a spatial-weights mechanism is
proposed to learn spatial importance maps and predict gaze directions using
only face images as input. This map serves as a guide to the following layers
of the CNN, learning to locate important features on the normalized input
image (eyes, nose) while pointing to where the network focus should be. The
rationale behind this approach is similar to ours, except that we estimate
what would be the spatial importance in the form of multiple attention layers
maps, doing so in an implicit way. This serves not only as a way to learn
possible locations for important facial regions, but the use of self-attention
also allows the maps to correlate these often distant features in a more
abstract way not feasible for regular convolutional layers.

Attention mechanisms: Recently, there has been a great success in using
attention in sequence modeling with deep learning. Recurrent neural networks,
long short-term memory [27], and gated recurrent units [28] are known methods
of handling sequential data such as video, text, and speech. For a long time,
these were held as state-of-the-art methods but not without flaws. In
particular, while these methods can successfully represent dependencies when
they are close across input/output sequences, their performance suffers when
representing distant relationships. Self-attention is an alternative way to
tackle that issue, being capable of modeling relationships among elements in
different positions of an input sequence while creating a rich representation.
This characteristic of self-attention methods brings a clear advantage over
traditional sequence handling, and the Transformer [19] was the first method
to show that it is possible to discard convolutions completely. Transformer
networks rely only on self-attention to model representations of input/output
sequences, representing the very first choice in more recent NLP applications
(usually sequential by nature). Similarly, in image-related tasks, the use of
context-aware mechanisms like attention has been shown to generally improve
the accuracy of deep neural networks. There is a particular interest in
techniques that can be used with minimal effort to improve existing
architectures. The Squeeze-and-Excitation (SE) [29] blocks are drop-in
components that model contextual dependencies in channel-wise relationships in
feed-forward networks. The Bottleneck Attention Module (BAM) [30] and the
Convolutional Block Attention Module (CBAM) [31] are these types of components
that propose to do the similar job, additionally integrating both spatial- and
channel-wise relationships.

In [32], the principle of multi-headed self-attention from the Transformer
network is adapted for 2D inputs, presenting a hybrid layer with attention and
convolution operations performed in parallel. These are shown to be compatible
with current established deep network architectures, being able to completely
replace regular convolutional layers. Unlike BAM and CBAM, which refine
existing convolutional feature maps with attention, self-attention augmented
convolutions create new attention maps to be fused with their convolutional
counterparts. This allows for the network to create more spatially-aware
representations, potentially presenting accuracy gains.

###  1.2 Contributions

In this paper, we introduce a ResNet-inspired [33] network, dubbed Attention-
augmented ResNet (ARes-14), conceived upon a self-attention-based mechanism as
proposed by [32]. ARes-14 was intuitively driven to improve appearance-based
gaze estimation, which needs spatial awareness but does not require very deep
architectures to be effective (indeed, it only uses 14 layers, as the name
suggests).

Method |  | 3D gaze
---
output
| Full-face
---
as input
| Eye as
---
input
| Multimodal
---
inputs
| Spatial
---
awareness
| Attention
---
augmented
MPIIGaze [13] | ✓ | – | ✓ | – | – | –
iTracker [17] | – | ✓ | ✓ | ✓ | – | –
Spatial Weights [16] | ✓ | ✓ | – | – | ✓ | –
RT-Gene [14] | ✓ | ✓ | ✓ | – | – | –
Recurrent CNN [26] | ✓ | ✓ | ✓ | ✓ | – | –
Dilated Net [18] | ✓ | ✓ | ✓ | – | – | –
FAR-Net [25] | ✓ | ✓ | ✓ | – | – | –
Ours | ✓ | ✓ | ✓ | – | ✓ | ✓

Table 1: Summary of the state-of-the-art on appearance-based gaze estimation.

To effectively provide gaze estimation from a monocular camera, we also
propose a framework called ARes-gaze, which is comprised of two ARes-14
networks that act as twin feature extractors, taking as inputs full-face
images and isolated eye-patches. We showed that, as reported in [32] for
classification tasks, some of the weights of early attention maps can learn to
highlight geometric structures from the full-face images, leading us to
hypothesize that self-attention augmented convolutions can fulfill a similar
role to the spatial importance maps conceptualized in [16]. This ability can
help the network better focus on facial regions relevant to gaze estimation.

ARes-gaze achieved state-of-the-art results on two challenging data sets. When
compared with similar methods of appearance-based gaze estimation, we found a
decrease in the average angular error by 2.38% on the MPIIFaceGaze data set,
achieving the second-best result on the EyeDiap data set. Table 1 summarizes
the characteristics of our framework in comparison with other state-of-the-art
works.

##  2 Gaze estimation with self-attention augmented convolutions

Before going into details of our proposed framework, we review important
concepts of appearance-based gaze estimation and the motivation behind the use
of attention-augmented convolutional layers.

###  2.1 Gaze vector

3D appearance-based gaze estimation can be comprehended as to find a function
capable of mapping an input image, I, to a gaze vector, ĝ. Given that the gaze
direction is usually also dependent on head pose, (h), we include this latter
into the formulation, thus generically obtaining:

$\textbf{\^{g}}=f(\textbf{I},\textbf{h})\,,$ (1)

where ĝ is a 2D unit vector with the origin being in the middle point between
the subject’s eyes. The components that form ĝ are the pitch
($\mathrm{\hat{\textbf{g}}}_{\theta}$) and yaw
($\mathrm{\hat{\textbf{g}}}_{\phi}$) angles. Here, the mapping function is the
proposed trained neural network, and h is implicitly inferred from full-face
images. We can then rewrite the generic appearance based formula as
$\textbf{\^{g}}=f(\mathbf{I}^{eyes},\mathbf{I}^{face})$.

###  2.2 Attention-augmented convolutional layer

First proposed as an alternative base layer for classification [32],
attention-augmented convolutions (AAConv) extend the multi-head attention
concept from the Transformer network [19] by applying self-attention to 2D
arrays. In regular convolution layers, inter-pixel correlation is usually
spatially constrained by the convolutional kernel. This limits the degree to
which is possible to relate distant sections from an image that could have
relevant relationships.

Figure 1: Attention augmented convolutional (AAConv) layer as proposed by
[32]. Nh attention maps are computed for every (w, h) location of the input.
The self-attention step consists of performing weighted averaging in these
maps, then combining the results by concatenating, reshaping, and mixing
($1\times 1$ convolution). The resulting feature map is then concatenated with
a feature map obtained from a regular convolution performed directly on the
input, producing the final output. $W_{i}$, $H_{i}$ and $W_{o}$, $H_{o}$ are
the width and height of the input and output maps, respectively, with
$W_{o}<W_{i}$ and $H_{o}<H_{i}$.

Similar to what is done in Transformer networks with 1D sequences, attention-
augmented convolutions use self-attention to handle pixel matrices, as
depicted in Fig. 1. Each pass through an AAConv layer can be split into two
main parts: The first one through a regular convolutional layer, while the
second through a multi-headed attention layer. The outputs ($W_{o},H_{o}$) of
each individual attention-head are concatenated and projected onto the
original spatial dimensions of height and width of the input ($W_{i},H_{i}$).
Additionally, relative positional embeddings [34] are expanded to two
dimensions in order to encode spatially-relevant information while maintaining
translation equivalence [32]. In the end, the results from both passes of the
convolutional and the multi-headed attention layers are concatenated, forming
spatially-aware convolutional feature maps from the input image.

Expanding the neural network principle of long-distant spatial relationships,
it is possible to achieve a clear positive effect when applied to straight-
forward classification tasks across a range of different architectures [32].
Particularly, we explore if and how to use these concepts as a building-block
of CNNs to improve accuracy on a regression task of gaze estimation.

###  2.3 ARes-14: A self-attention augmented convolutional backbone

Figure 2: ARes-14: Self-attention augmented ResNet with 14 layers. All
convolutions in residual blocks are augmented with self-attention, while the
input stem remains with conventional convolutions.

In appearance-based gaze estimation performed by CNNs, shallow networks can be
sufficient as long as the task is performed in relatively constrained
conditions [14] (across a limited range of head pose and with short distances
between subject and camera). These conditions are intrinsic to some available
data sets, although they are not a reasonable expectation for in-the-wild
applications. These constraints can be simulated in more challenging data by
applying preprocessing and normalization procedures (see Section 3.2 for more
details). The use of these strategies allows us to train with more structured
data. Also, procedures like those should still perform well in more complex
environments by normalizing the input data before sending it through the
prediction network during inference time. The use of shallower networks is of
particular importance given the significant computational overhead of training
with self-attention in convolutional networks (see [32] for a more detailed
discussion).

ResNet is a widespread and well understood general-purpose CNN, turning it
onto an ideal candidate for a baseline comparison against self-attention
augmentation. We started with the shallow version, ResNet-18, and replaced
every convolutional layer for an equivalent self-attention augmented
convolution with compatible dimensions. The number of parameters was further
reduced by removing the last-layer block, essentially shrinking the
architecture to 14 layers. Each convolution and AAConv is followed by a batch
normalization and activation (ReLU) operation. The ratio between attention
channels and output filters (k), as well as, the ratio between the key depth
and output filters (v) were both fixed to 0.25 for every self-attention
augmented convolution. Unless otherwise specified, the number of attention
heads, $Nh$, is fixed to 8. We called this novel network architecture as
ARes-14, which is used as the backbone in our proposed framework for gaze
estimation. Figure 2 depicts ARes-14 architecture.

###  2.4 ARes-gaze: A framework for gaze estimation

Figure 3: ARes-gaze framework. Face- and eye-patches are extracted and
separately normalized from the source image, subsequently being sent through
twin ARes-14 backbones. The resulting features from each backbone are then
concatenated and passed through a prediction stage consisting of two fully-
connected layers.

To perform gaze estimation, we propose a fairly conventional framework: A two-
stemmed network where each branch is an instance of ARes-14, and the extracted
features are joined by a shared prediction layer, as shown in Fig. 3. We used
a feature vector of 256 elements obtained from each convolutional backbone
after the global average pooling layer, resulting in 512 features to be sent
through the prediction layers (see Fig. 2).

Many works have used multi-input frameworks in appearance-based gaze
estimation [25, 14, 18, 26, 23] since the gaze direction of a subject relies
heavily on more than one factor (eyes, head pose, and location, distance,
etc). Here our inputs are RGB-face images, normalized for pose and distance,
and grayscale eye images, histogram-normalized.

To extract information from the isolated eye-patches, while some published
methods with similar topologies use two networks (one for each eye) [18, 14]
or a single network with shared weights (making separate passes for each
input) [25], we employed a single-pass, single-network strategy for the eye
branch by stacking the left- and right-eye regions, creating a $1:1$ ratio
square input. We study the practical implications of the use of this method in
comparison with the other mentioned works in Section 4.2.1. The extracted-
feature vectors from the face and the eyes are then joined by concatenation
and passed through a prediction block to output the two values of our gaze
vector prediction.

##  3 Materials and methods

Characteristics | Data sets
---|---
MPIIFaceGaze [13, 16] | EyeDiap [35]
Size | 45,000 images | 94 videos
Image type | RGB | RGB-D
Subjects | 15 | 16
Subject distance | 40 - 60cm | 80 - 120cm
Normalized | ✓ | –
Head-pose annotation | ✓ | ✓
Extreme-head pose | – | ✓
Extreme-lighting variation | ✓ | –
Eye-position annotation | ✓ | ✓
Table 2: Comparison of the relevant characteristics of both data sets used in
the experiments.

###  3.1 Training data

We performed evaluations on the most used publicly-available data sets. Figure
4 shows samples of training data of both data sets. Table 2 summarizes the
relevant characteristics of both data sets, described below:

Figure 4: Data samples from the EyeDiap [35] (top row) and MPIIFaceGaze [13]
(bottom row) data sets. The samples on the top row were normalized by
following the procedure described in Section 3.2. The bottom samples were
taken from the already-normalized MPIIFaceGaze data set. [16]

The MPIIFaceGaze data set. The MPIIGaze data set [13] was the first to provide
unconstrained data for gaze estimation in-the-wild. 15 subjects (9 males, 6
females, and 5 subjects with glasses) were recorded in various sessions during
day-to-day use of their laptops, where targets were occasionally displayed at
random positions in the screen. The recorded data contains a large number of
different conditions of recording locale (inside and outside), illumination,
head pose and position, and overall recording quality. Since the original
MPIIGaze data set provides cropped-eye regions already, we used its modified
version MPIIFaceGaze [16], which provides 3,000 full-face, already normalized
images for each subject. Figures 5(c) and 5(f) show the gaze data distribution
for the MPIIFaceGaze data set.

The EyeDiap data set is a collection of 94 videos with 16 different subjects
in 3 different modalities: Discrete screen target \- where a target was
displayed in regular intervals on random locations on a screen, continuous
screen target – in which the target moved along random trajectories in the
screen, and 3D floating target – where a small ball was moved along the 3D
space between the participant and the camera with the assistance of a thin
thread. In our experiments, we used only the modalities where the target was
projected onto the screen (continuous and discrete), since, in the floating
target-sessions, the small ball would sometimes occlude the subject’s face.
Two subjects only have video recordings on floating target sessions, so we are
left with a total of 14 subjects and 56 videos. For each one of the 14
subjects, there are two different versions of each session: One where the
subject’s head is fixed and the target is pursued with eye movements only, and
another where the subject also moves the head.

Figures 5(a), 5(b), 5(d), and 5(e) show the gaze and head-pose data
distribution for the EyeDiap data set grouped by whether the subject was
instructed to move its head (static or mobile). We sampled every $5^{th}$
frame from the recordings to collect the processed data for training. We
cleaned the resulting frames by excluding those with missing annotations for
the head pose, screen target location, or eye position. Annotations for
invalid frames because the subjects have their eyes closed (blinking) or are
distracted (looking away from the target) are available for some of the data
set’s sessions, but not all. For those sessions without such annotations, we
manually parsed extracted frames, removing those ones considered invalid. In
the end, approximately 44.000 examples remained to carry out with the leave-
one-person-out cross-validation.

(a) EyeDiap g (static)

(b) EyeDiap g (mobile)

(c) MPIIFaceGaze g

(d) EyeDiap h (static)

(e) Eyediap h (mobile)

(f) MPIIFaceGaze h

Figure 5: Gaze ground-truth (g) and normalized head pose (h) distribution on
the MPIIFaceGaze and EyeDiaps data sets. The latter is split into static or
mobile according the subject’s head movement. Angle values are displayed in
radians.

###  3.2 Data set normalization

Similar to [36], we applied an affine transformation to rotate the image as to
cancel out the roll-axis angle of the head, and to scale it to the desired
size (standardizing the distance from the face to the virtual camera). The
effect of that transformation is that relevant facial features are always in
the same regions on the input, making the network job easier to recognize
patterns in important regions. This procedure is only applied on the EyeDiap
[35] data set since the MPIIFaceGaze data set [16] is already normalized.
Figure 6 illustrates these procedures.

Figure 6: Normalization procedure on a sample frame from the EyeDiap data set.
The line drawn between the subject’s eyes is used to determine the $\alpha$
and $S$ parameters needed to build the transformation matrix $M$.

The affine transformation is represented as the matrix $M$, which is defined
as $M=[R\ T]$, where $R$ is the rotation component that normalizes the roll-
axis rotation, and $T$ is the translation component that ensures the relevant
patch from the original frame is in the normalized image. To build $R$, we
need the parameters $\alpha$ (rotation) and $S$ (scale). We estimate the angle
of the line between the subject’s eyes, and use that value as the parameter
$\alpha$, which can be obtained with the help of a facial landmark detector,
although here we use the annotated position provided by the Eyediap data set.
The scale parameter, $S$, controls the distance to the subject in the image.
For a squared input, we defined that the distance, $d$, between the left- and
right-eye centers should be 40% of the image width. Given that the face should
be vertically centered, this gives us left- and right-eye centers to be
$(0.7,0.35)$ and $(0.3,0.35)$ on a normalized 0 to 1 scale relative to the
input dimensions.

$S$ can then be given as

$S=\frac{Z*d}{D},$ (2)

where $Z$ is the final image size in pixels (considering square images), and
$D$ is the original distance between the subject’s eye center.

With the angle and scale parameters in hands, we can then build $R$ as follows

$R=\begin{bmatrix}a&b\\\ -b&a\end{bmatrix},$ (3)

where a = S * $\cos(\alpha)$, b = S * $\sin(\alpha)$.

$T$ is defined as

$T=\begin{bmatrix}(1-a)*GO.x-b*GO.y\\\
b*GO.x+(1-a)*GO.y\end{bmatrix}+\begin{bmatrix}tX-GO.x\\\ tY-
GO.y\end{bmatrix},$ (4)

where tX = Z * 0.5 , tY = Z * 0.35, and $GO$ is the coordinates of the gaze
origin vector (here, defined as the middle point between the left- and right-
eye centers). The $tX$ and $tY$ are the correction factors that enable re-
positioning the gaze vector in the normalized image.

For the face patch, we used RGB images with an input size of $112\times 112$
pixels (thus, for our case $Z=112$). The eye patches are cropped from the
normalized face, converted to grayscale, and the histogram normalized before
being resized to the input shape of $30\times 60$. These steps are carried out
for both data sets.

###  3.3 Implementation Details

The code for the models was written with the PyTorch [37] framework111Link to
git repository will be available after paper acceptance. All the models were
trained for 120 epochs with a batch size of 48 on an HPC cluster equipped with
8 NVidia V100 GPUs. The high computational overhead of training attention-
based methods is prohibitive with regards to the batch size, and this needs to
be taken into account during the hyperparameter tuning. We used a stochastic
gradient descendant (SGD) [38] solver with a momentum equal to 0.9, and a
weight decay of 0.0003 was empirically found to be optimal in preventing over-
fitting. The learning rate is linearly warmed-up for 5% of the epochs until
reaching the value of 0.128, then gradually decreased by cosine annealing
[39]. The loss used for training the whole model was the smooth $L1$ cost
function defined as

$\text{smooth}_{L1}(x,y)=\begin{cases}0.5(x-y)^{2},&\text{if }|x-y|<1\\\
|x-y|-0.5,&\text{otherwise}\,,\end{cases}$ (5)

where $x$ and $y$ are the ground-truth and predicted vectors, respectively.

##  4 Experimental results

To characterize our proposed framework properly, a group of experiments was
carried out and divided into two main parts: First, a set of ablation studies
were performed to assess the impact of the self-attention augmentation modules
on aspects of the ARes-gaze architecture. The main goal of this part is to
better understand the optimal conditions to apply AAConvs in our framework. In
this part, the studies were namely: The input models of the eye images, the
performance of ARes-14 versus ResNet14, and the number of attention heads for
the self-attention augmented layers. In the second part, some of the external
factors that directly impact the performance of gaze estimation were analyzed
to explore how our proposed framework can deal with them. Two experiments were
performed in this part: Head-pose variation and illumination conditions of the
input image.

###  4.1 Evaluation methodology

To improve reproducibility and reduce the effects of subject’s dependence on
our evaluations, a leave-one-out cross-validation strategy was used across the
subjects from each data set. Considering the characteristics of the data sets
used in the experiments (see Table 2), N models were trained, where N is the
number of available subjects in a data set. For each model, a different
subject is held out and used for testing. The final result is the average of
the evaluations of all models. On the EyeDiap data set, for example, the final
scores are the average performance of the 14 trained models on the held-out
subject, each time. Similarly, on the MPIIFaceGaze data set, 15 models were
trained and their performance scores were averaged into the final results.

###  4.2 Ablation studies

The first evaluation here is grounded on different input models for the eye
branch. The goal was to compare our proposed single-branch, single-pass
vertical stacking scheme (see Section 2.4) with other strategies adopted by
similar methods. The second study was to untangle the effect of self-attention
augmentation on different inputs (face and eyes) and the network schemes. The
aim is to understand how and where self-attention is effective on the task of
gaze estimation, and how it ultimately impacts the performance of the proposed
framework. By evaluating isolated portions of the proposed framework with and
without self-attention augmentation, these experiments are useful in
generating insights on how ARes-14 can be best applied, guiding future
researches. Finally, we evaluate the effect of choosing different numbers of
attention heads for ARes-14. The multi-headed attention mechanism can present
significant computational overhead, so an investigation on the trade-off
between number of attention heads and evaluation error drives this choice.

####  4.2.1 Evaluating different models of the eye images

For the eye-patch branch of our network, the input consists of images of both
left and right eyes from the subject. Other published works with similar
network topologies either need to perform two forward passes [25] or use a
dedicated network branch for each eye [18, 14]. We propose the vertical
stacking of eye images to obtain a 1:1 input image that can be processed in a
single pass.

We evaluate ARes-gaze against the other mentioned models, considering the
following parameters: the number of network parameters, approximated floating-
point operations (FLOPs), and average angular error. Three models were
considered for the eye branch:

  * 1.

Stacked-eyes input (SE);

  * 2.

Double-pass with shared weights (DP);

  * 3.

Separate branches (three-branch pipeline) (TB).

Model type | Average angular error | # Params (M) | FLOPs (M)
---|---|---|---
MPIIFaceGaze | EyeDiap
SE | 5.40° | 7.27° | 2.810 | 414
DP | 5.54° | 7.42° | 2.842 | 422
TB | 5.45° | 7.36° | 5.619 | 422
Table 3: Results on different input models of the eye images. The evaluated
parameters considered are: Average angular error on the EyeDiap and
MPIIFaceGaze data sets, number of trainable parameters (Millions) and
approximate floating operations (FLOPs, also in Millions) for the three
evaluated input models.

As summarized in Table 3, although there is arguably only a small difference
in the average angular error, the stacked-input model performed better than
the other ones on both data sets. Also the stacked-input model presents
roughly the same number of trainable parameters of the shared-weights variety
and a significantly lower number when compared to the twin-branch network.
These results further validate the adoption of the stacked-eye for all
subsequent evaluations.

####  4.2.2 ARes-14 evaluation

With the aim of gauging the effect of self-attention augmentation in multiple
stages of ARes-gaze, we evaluated and compared multiple models based on the
ARes-14 architecture. First, to see how attention affects different types of
input, we trained single-branch networks with and without self-attention
augmentation in isolated versions with only eye images as inputs, or only face
images as inputs. Second, we applied the ARes-gaze and compare models
switching between ResNet-14 and ARes-14 backbones for each input branch. The
goal is to explore the contrast between fully convolutional features and self-
attention augmented features for gaze estimation.

| Input | Dataset
---|---|---
Network type | Eyes | Face | MPIIFaceGaze | EyeDiap
Regular | $\blacksquare$ |  | 5.40° | 7.27°
Attention | $\blacksquare$ |  | 5.33° | 6.02°
Regular |  | $\blacksquare$ | 4.71° | 7.42°
Attention |  | $\blacksquare$ | 4.46° | 6.10
Regular | $\blacksquare$ | $\blacksquare$ | 4.46° | 6.09°
Regular | $\blacksquare$ |  | 4.42° | 5.81°
Attention |  | $\blacksquare$
Regular |  | $\blacksquare$ |  |
Attention | $\blacksquare$ |  | 4.52° | 5.84°
Attention | $\blacksquare$ | $\blacksquare$ | 4.17° | 5.58°
Table 4: Results of attention-augmented versus regular convolutional layers on
the backbones of ARes-gaze. Best results are highlighted.

The results for each model are laid out in Table 4. The network types are
comprised of single regular, single attention or both regular/attention
branches. For the single-branch networks (with either only face or only eyes
as inputs), we observe a drop of more than 17% on the average angular error on
the EyeDiap data set when using self-attention augmented convolutions. When
compared with its regular convolutional form, ARes-gaze reduces the average
error by 6.5% on the MPIIFaceGaze data set and by 8.4% on EyeDiap.

####  4.2.3 Determining the number of attention heads

In all evaluations reported in [32], on the use of AAConvs in classification
tasks, the accuracy gains are on architectures using a fixed number of
attention-heads, specifically $Nh=8$. In this section, we evaluate ARes-gaze
considering other values of $Nh$.

Table 5 shows the average angular errors found on the MPIIFaceGaze and EyeDiap
data sets. Notably, for the MPIIFaceGaze data set, when using less than 4
attention-heads, the ARes-gaze architecture performs worse than the purely
convolutional baseline, with the evaluation error proportionally decreasing
with the increase of attention-heads. On the EyeDiap data set, the results
follow the same tendency with $Nh=2$ and $Nh=4$, which are only marginally
better than the baseline network. In both data sets, there is a sudden and
significant improvement in the results when $Nh=8$. It is worth noting that
given the high computational overhead of AAConv layers, with the available
hardware (as described in Section 3.3) it was not viable to perform
experiments with $Nh>8$.

Method | MPIIFaceGaze | EyeDiap
---|---|---
Baseline | 4.46° | 6.09°
ARes-gaze (Nh=2) | 4.93° | 5.98
ARes-gaze (Nh=4) | 4.36° | 5.99
ARes-gaze (Nh=8) |  4.17° |  5.58°
Table 5: Results of average angular errors on different numbers of attention-
heads per attention layer. Best results are highlighted.

###  4.3 Comparison of ARes-gaze with other appearance-based methods

We selected six appearance-based methods that take as input either full-face
images or a combination of full-face images and other inputs. All these
methods output a single-gaze vector with origin in the center of the face or
in the middle-point of the eye. The selected methods were: the iTracker in its
original form [17] and with AlexNet backbone [16], the CNN with spatial-
weights mechanism [16], RT-Gene (a version of 4 ensembles with the best
reported results) [14], the CNN with dilated convolutions proposed in [18],
and the eye-asymmetry based FAR-Net [25]. These are approaches we consider
similar to ours, which were compared over the average 3D-angular error on the
chosen data sets. Except for RT-Gene and iTracker (AlexNet), which do not
report evaluations on the EyeDiap data set, all compared methods use the same
or a similar protocol to extract data from the videos, as described in Section
3.1.

The results are reported by considering two versions of our architecture: The
full ARes-gaze and ARes-gaze without self-attention augmentation.

Method | MPIIFaceGaze | EyeDiap
---|---|---
iTracker [17] | 6.2° | 8.3°
iTracker (AlexNet) [17, 16] | 5.6° | –
Spatial Weights CNN [16] | 4.8° | 6.0°
RT-Gene (4 Ensemble) [14] | 4.3° | –°
Dilated CNN [18] | 4.5° | 5.4°
FAR-Net [25] | 4.3° | 5.7°
Baseline | 4.5° | 6.1°
ARes-gaze (Nh = 8) | 4.2° | 5.6°
Table 6: Results of average angular error compared with other appearance-based
methods. Best results are highlighted.

When compared to the other methods, our ARes-gaze framework with twin ARes-14
backbones reached state-of-the-art results on the MPIIFaceGaze data set, and
the second-best place on the EyeDiap data set, being only 0.2 degrees behind
the Dilated CNN [18] (see Table 6). It is worth noting that no other method
was able to have superior results on both data sets.

###  4.4 Evaluating external factors in gaze estimation

In in-the-wild gaze estimation applications, the subject’s head pose and
external illumination conditions are to be considered unconstrained. As such,
these are highly relevant factors to be considered when proposing new
approaches for gaze estimation. As discussed in Section 3.2, we applied color-
level normalization, enforcing roll-angle normalization during training and
inference to reduce the complexity of our model when the data set does not
offer already normalized images. Regardless, edge cases of extreme conditions
are still challenging, and there are yet the pitch and yaw angles to be
concerned. To see how self-attention augmentation in convolutions affects
robustness to these factors, we evaluated both our completely attention-
augmented model and the traditional convolutional baseline in isolated
scenarios.

####  4.4.1 Head pose

(a)

(b)

(c)

Figure 7: Samples with extreme head pose angles from the EyeDiap dataset.

To evaluate how each model performs to the subject’s head pose, we used the
EyeDiap data set due to its larger variety (refer to Fig. 5) of head pose and
edge cases of extreme pose conditions (see image samples in Fig. 7). We used
the data set annotation for head-pose angles to correlate every sample
prediction error to the subject’s head pose. We then performed 2D binning to
obtain the average angular error on intervals of 0.20 radians for pitch and
yaw.

(a) Baseline convolutional model

(b) attention-augmented model

Figure 8: Distribution of mean angular error of baseline and attention-
augmented models across head poses in the EyeDiap dataset.

Figure 8 shows the results for both self-attention augmented and regular
convolutional based architectures. The plots make clear that the overall gains
are obtained across most of the pitch and yaw head-pose spectrum. This notion
is further reinforced by Fig. 9 where the average angular error difference
between the two plots is plotted. The overall decrease in average error
appears mostly uniform outside of the most extreme cases. For those, we
observe that the larger gains obtained by the ARes-gaze model were in regions
of extreme pitch angle (negative and positive), and the heavier losses were in
regions of high yaw angles. It should also be noted that the overall magnitude
of the gains in average accuracy is greater than that of eventual losses
suffered by the model (color bar in Fig 9).

Figure 9: Distribution of angle-error difference between attention-augmented
and baseline models on the head-pose evaluation. Blue boxes (negative numbers)
mean an improvement over the baseline model, or a drop in the average angular
error. Similarly, red boxes mean regions where there was an increase in the
average angular error.

####  4.4.2 Illumination conditions

Figure 10: Distribution of average light intensity per sample on the
MPIIFaceGaze and EyeDiap datasets.

(a) Sample with average light level greater than 220.

(b) Sample with average light level lower than 20.

Figure 11: Samples of edge cases w.r.t lighting conditions from the
MPIIFaceGaze data set. These examples were randomly sampled from images with
average light level greater than 220 and lower than 20.

In Fig. 10, the illumination distribution is plotted on both the MPIIFaceGaze
and EyeDiap data sets. The light level values are obtained per sample by
averaging the pixel intensity to obtain a value between 0 (completely dark)
and 255 (completely bright). The plot shows that the images from MPIIFaceGaze
data set is more varied to lighting conditions (illustrated by the sample
edge-case from both extreme light and darkness shown in Fig. 11), while the
average intensity of image pixels approximate to a normal distribution. These
properties should be beneficial to our evaluation, so we chose to analyze the
effect of self-attention augmentation versus lighting on the MPIIFaceGaze data
set.

Figure 12: Evaluation of model accuracy versus lighting conditions of input
images. The MPIIFaceGaze data was split into 10 bins with regard to light
levels, with the X-axis showing the average level of each bin. The Y-axis is
the average angular error in degrees. A regression line drawn across each
model bars, with its slope value ($m$) being shown in the plot legend.

Figure 12 shows an overlapping evaluation of both baseline and ARes-gaze
models by light-level intervals. We split the 0-255 light-level range into 10
bins, and the results are averaged across all 15 subjects in relation with
these bins. Clearly there is an inverse relationship between light level and
angular error that behaves somewhat linearly. Additionally, the last bin,
representing extreme high-light levels (overly lit images) shows a small spike
in the averaged angular error. This situation reinforces the intuitive notion
that appearance-based gaze estimation models have worse accuracy with both
poorly lit and overexposed input images.

To quantify the sensibility of each model to lighting conditions, we fit a
regression line across the angle error of each bin, and calculated its slope
($m$). The closer to zero the slope is, the lower is the model sensibility to
light. This experiment showed that ARes-gaze had a slightly smaller slope
inclination, although the difference was not enough to justify conclusions
about its robustness to lighting conditions in comparison with the purely
convolutional baseline.

###  4.5 Result analysis

####  4.5.1 On the use of self-attention augmented convolutions for gaze
estimation

First, we evaluated the difference between using eye images versus using the
entire face as inputs. In these experiments, the eye images were stacked
before being forwarded through the network, as reported in Section 4.2.1.
Intuitively, the difference between using full-face images and isolated-eye
regions as inputs is the scope of the information that the network is able to
extract. With full-face images, CNN has the chance to learn not only from the
eyes themselves but also extract head-pose information from regions such as
the nose and mouth. This comes with the drawback of the subject’s eyes having
a lower resolution, thus limiting the amount of information present in their
regions. In contrast, using isolated eye-patches should allow the network to
extract more detailed information about the pupils’ positions, turning the
network to be more sensitive to smaller changes in the eye movement. In this
case, the drawback is the absence of elements that can inform the network
about the subject’s head pose, which has relevance to the final prediction.

Figure 13: Comparison of average angular error for single-branch gaze
estimation networks on the MPIIFaceGaze and EyeDiap data sets. Blue bars
represent evaluations with ResNet-14 as the backbone, while red bars represent
those with ARes-14.

Figure 13 shows a visual summary of the results of the single-branch networks
from Table 4. There is a clear and consistent decrease of the average angular
error in all instances when using the networks with self-attention augmented
convolutions (ARes-14). As to which kind of input benefits the most from
attention, on the EyeDiap data set, an error decrease of 17.19% with eyes as
input versus 17.78% with faces can be observed. On the MPIIFaceGaze data set,
the decreases were of 1.28% and 5.31%, respectively. The larger magnitude of
gains on the EyeDiap data set can be inferred from the fact that it is a more
challenging data set with regards to head pose, which is an issue we guess
self-attention augmentation benefits when applied to gaze estimation.

The slightly larger gains on the face-only network may be explained by the
fact that while in eye images the region of interest is essentially the pupil,
the regions of interest for gaze estimation in a facial picture (eyes, nose,
mouth) are further apart from each other, which is exactly where self-
augmentation can be the most useful. Although the error decrease in single-
branch networks seems promising, results in Table 4 shows that the advantage
of using multiple-branches networks is very clear.

Figure 14: Comparison of average angular error for four different versions of
the proposed gaze estimation framework. From left to right, respectively:
regular convolutional baseline (blue), a version with ARes-14 on the face
branch and ResNet-14 on the eye branch (purple), a version with ResNet-14 on
the face branch and ARes-14 on the eye branch (green), and the fully
attention-augmented ARes-gaze (red).

Figure 14 shows a comparison of the results obtained from multiple inputs
across different iterations of our proposed gaze estimation architecture
(replacing ResNet backbones by ARes-14 in each branch). It is worth noting
that between the networks using ARes-14 as the backbone for only one of the
branches, the one with self-attention augmentation on the face branch wins by
a slight margin. On the MPIIFaceGaze data set, the one with attention only on
the eye branch even had a small but noticeable drop in performance when
compared with the regular CNN baseline. When analyzing the results from the
single-branch network evaluation, it is possible to note that the face branch
benefits slightly more from self-attention augmented convolutions due to
having more distant elements that can be correlated by self-attention. This is
reinforced by our results on the evaluation of mixed attention and regular
convolution networks.

All in all, our findings indicate that self-attention augmented convolutions
can be used as drop-in replacements to convolutional layers in gaze estimation
networks to reduce the angular error in evaluation. Yet, while self-attention
augmented convolutions work well with both face and eye-input images, our
experiments showed that networks working with the full-face image as input
were more prone to improvement when augmented by self-attention. The ARes-gaze
framework which uses ARes-14 networks for both face and eye-inputs had the
best results on our ablation studies, in some scenarios outperforming and at
worst being comparable to state-of-the-art similar appearance-based gaze
estimation methods on the MPIIFaceGaze and EyeDiap data sets.

####  4.5.2 On the number of attention heads per convolutional layer

Figure 15: Visualization of weights for intermediate feature maps from
attention heads on the first attention-augmented convolution when performing
inference.

(a)

(b)

(c)

(d)

(e)

Figure 16: Stages of attention maps: (a) is the input image, (b) is the
projected prediction (red) and ground truth (blue) vectors, (c) is the last
attention map on a random pixel in the image for the first convolutional
layer, (d) is the attention map after thresholding and smoothing, and (e) is
an overlay of the smooth map and the input image, evidencing relevant
features.

We obtained the counter-intuitive results that for a number of attention-heads
less than eight, $Nh<8$, the ARes-gaze framework sometimes actually performed
worse than the regular convolutional baseline. As detailed in Section 2.2, the
output of a self-attention augmented convolution is the concatenation of a
regular convolution and a self-attention layer, which is made up of feature
maps obtained from the attention heads. We hypothesize that for lower numbers
of attention-heads, the attention portion of the output may not be able to
sufficiently represent relevant features, and may even harm the convolutional
feature maps upon joining with them. This hypothesis is further reinforced by
a visual analysis of early feature maps extracted from a trained network, as
depicted in Fig. 15. Figure 15 shows the weights computed from attention maps
(one per attention head), extracted from the first augmented layer of an
ARes-14 face branch during prediction. These weights are subsequently used to
compute the final attention feature maps that will be joined with the
convolution results for the output layer. Each row represents a different
trained network with the number of attention-heads $Nh=2,4$ and $8$, as seen
in Table 5. It is worth observing that, for some inputs, the self-attention
augmented layer is capable of highlighting semantically relevant regions of
the image. We verified however that when this happens, it is only on the map
of the eighth attention head. This leads us to conclude that the attention
layer might need a certain depth of attention-heads in order to specialize in
very particular tasks. In turn, when this specialization is not reached
properly, the overall output of the self-attention augmented convolution
(concatenation of a regular convolution and an attention layer) can have lower
quality than a regular convolutional layer with a larger feature space. This
would explain the counter-intuitive results reached when using self-attention
augmented layers with values lower than 8 for $Nh$, being less accurate than a
regular convolutional CNN.

To more easily interpret the highlighted regions by the weights of the eighth
attention map, Fig. 16 illustrates an example of this phenomenon on a sample
image from the MPIIFaceGaze data set. Figure 16(b) shows the projected
prediction and ground-truth gaze direction arrows over the input image (Fig.
16(a)). Figures 16(c), 16(d), and 16(e) show respectively the last attention
map’s weights for a random pixel on the input image, a refined version of
these weights after operations of thresholding and smoothing, and an overlay
with the input image. As expected, the eyes and eyebrows are the most relevant
regions for the gaze estimation task. Notably, the nose, mouth, and background
are also highlighted, which we speculate to be the source for head-pose
related information implicitly used for the prediction.

##  5 Conclusion

In this paper, we addressed the question ”can self-attention augmented
convolutions be used to reduce angular error in appearance-based gaze
estimation?”, and we found that when compared to an equivalent regular
convolutional network, the use of our 2D self-attention-based architecture can
indeed produce more accurate results. We used ARes-14 twin branches as self-
attention augmented CNNs in our experiments, and we believe that further
research is merited on the design of optimal architectures for each branch of
a multi-input attention-augmented framework such as the proposed ARes-gaze. We
showed that the input face images had more to gain from using AAConvs than the
input eye images, so incorporating domain knowledge of both attention
mechanisms and gaze estimation to refine each branch for its particular input
(face and eyes) may produce even better results than those reported. The
spacial awareness afforded to the framework by the self-attention augmented
convolutions can also be a promising way to develop joint head pose and gaze
direction estimation networks, with the possibility of including other types
of input images such as facial landmarks and explore their behavior in
attention-augmented convolutional networks.

## Acknowledgments

The authors acknowledge the National Laboratory for Scientific Computing
(LNCC/MCTI, Brazil) for providing HPC resources of the SDumont supercomputer,
which have contributed to the research results reported within this paper.
URL: http://sdumont.lncc.br.

## Funding

This work was supported by the Fundação de Amparo à Pesquisa do Estado da
Bahia (FAPESB) [grant number 892/2019]; and the Conselho Nacional de
Desenvolvimento Científico e Tecnológico [grant number 307550/2018-4].

## References

  * [1] M. A. Hollands, A. E. Patla, J. N. Vickers, “look where you’re going!”: gaze behaviour associated with maintaining and changing the direction of locomotion, Experimental brain research 143 (2) (2002) 221–230.
  * [2] G. Warlop, P. Vansteenkiste, M. Lenoir, J. Van Causenbroeck, F. J. Deconinck, Gaze behaviour during walking in young adults with developmental coordination disorder, Human Movement Science 71 (2020) 102616.
  * [3] T. Nakano, K. Tanaka, Y. Endo, Y. Yamane, T. Yamamoto, Y. Nakano, H. Ohta, N. Kato, S. Kitazawa, Atypical gaze patterns in children and adults with autism spectrum disorders dissociated from developmental changes in gaze behaviour, Proceedings of the Royal Society B: Biological Sciences 277 (1696) (2010) 2935–2943.
  * [4] S. Nilsson, T. Gustafsson, P. Carleberg, Hands free interaction with virtual information in a real environment: Eye gaze as an interaction tool in an augmented reality system., PsychNology Journal 7 (2).
  * [5] M. Lankes, B. Stiglbauer, Gazear: Mobile gaze-based interaction in the context of augmented reality games, in: International Conference on Augmented Reality, Virtual Reality and Computer Graphics, Springer, 2016, pp. 397–406.
  * [6] J.-Y. Lee, H.-M. Park, S.-H. Lee, T.-E. Kim, J.-S. Choi, Design and implementation of an augmented reality system using gaze interaction, in: 2011 International Conference on Information Science and Applications, IEEE, 2011, pp. 1–8.
  * [7] M. Barz, F. Daiber, D. Sonntag, A. Bulling, Error-aware gaze-based interfaces for robust mobile gaze interaction, in: Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, 2018, pp. 1–10.
  * [8] D. Rozado, F. B. Rodriguez, P. Varona, Low cost remote gaze gesture recognition in real time, Applied Soft Computing 12 (8) (2012) 2072–2084.
  * [9] D. Hakkani-Tür, M. Slaney, A. Celikyilmaz, L. Heck, Eye gaze for spoken language understanding in multi-modal conversational interactions, in: Proceedings of the 16th International Conference on Multimodal Interaction, 2014, pp. 263–266.
  * [10] D. W. Hansen, Q. Ji, In the eye of the beholder: A survey of models for eyes and gaze, IEEE transactions on pattern analysis and machine intelligence 32 (3) (2009) 478–500.
  * [11] T. Ohno, N. Mukawa, A free-head, simple calibration, gaze tracking system that enables gaze-based interaction, in: Proceedings of the 2004 symposium on Eye tracking research & applications, 2004, pp. 115–122.
  * [12] Z. Ye, Y. Li, A. Fathi, Y. Han, A. Rozga, G. D. Abowd, J. M. Rehg, Detecting eye contact using wearable eye-tracking glasses, in: Proceedings of the 2012 ACM conference on ubiquitous computing, 2012, pp. 699–704.
  * [13] X. Zhang, Y. Sugano, M. Fritz, A. Bulling, Appearance-based gaze estimation in the wild, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 4511–4520.
  * [14] T. Fischer, H. Jin Chang, Y. Demiris, Rt-gene: Real-time eye gaze estimation in natural environments, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 334–352.
  * [15] D. Lian, L. Hu, W. Luo, Y. Xu, L. Duan, J. Yu, S. Gao, Multiview multitask gaze estimation with deep convolutional neural networks, IEEE transactions on neural networks and learning systems 30 (10) (2018) 3010–3023.
  * [16] X. Zhang, Y. Sugano, M. Fritz, A. Bulling, It’s written all over your face: Full-face appearance-based gaze estimation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017, pp. 51–60.
  * [17] K. Krafka, A. Khosla, P. Kellnhofer, H. Kannan, S. Bhandarkar, W. Matusik, A. Torralba, Eye tracking for everyone, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2176–2184.
  * [18] Z. Chen, B. E. Shi, Geddnet: A network for gaze estimation with dilation and decomposition, arXiv preprint arXiv:2001.09284.
  * [19] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, in: Advances in neural information processing systems, 2017, pp. 5998–6008.
  * [20] F. Lu, Y. Sugano, T. Okabe, Y. Sato, Adaptive linear regression for appearance-based gaze estimation, IEEE transactions on pattern analysis and machine intelligence 36 (10) (2014) 2033–2046.
  * [21] T. Schneider, B. Schauerte, R. Stiefelhagen, Manifold alignment for person independent appearance-based gaze estimation, in: 2014 22nd international conference on pattern recognition, IEEE, 2014, pp. 1167–1172.
  * [22] Y. Sugano, Y. Matsushita, Y. Sato, Learning-by-synthesis for appearance-based 3d gaze estimation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1821–1828.
  * [23] W. Zhu, H. Deng, Monocular free-head 3d gaze tracking with deep learning and geometry constraints, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 3143–3152.
  * [24] Y. Cheng, S. Huang, F. Wang, C. Qian, F. Lu, A coarse-to-fine adaptive network for appearance-based gaze estimation., in: AAAI, 2020, pp. 10623–10630.
  * [25] Y. Cheng, X. Zhang, F. Lu, Y. Sato, Gaze estimation by exploring two-eye asymmetry, IEEE Transactions on Image Processing 29 (2020) 5259–5272.
  * [26] C. Palmero, J. Selva, M. A. Bagheri, S. Escalera, Recurrent cnn for 3d gaze estimation using appearance and shape cues, arXiv preprint arXiv:1805.03064.
  * [27] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural computation 9 (8) (1997) 1735–1780.
  * [28] J. Chung, C. Gulcehre, K. Cho, Y. Bengio, Empirical evaluation of gated recurrent neural networks on sequence modeling, arXiv preprint arXiv:1412.3555.
  * [29] J. Hu, L. Shen, G. Sun, Squeeze-and-excitation networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132–7141.
  * [30] J. Park, S. Woo, J.-Y. Lee, I. S. Kweon, Bam: Bottleneck attention module, arXiv preprint arXiv:1807.06514.
  * [31] S. Woo, J. Park, J.-Y. Lee, I. So Kweon, Cbam: Convolutional block attention module, in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 3–19.
  * [32] I. Bello, B. Zoph, A. Vaswani, J. Shlens, Q. V. Le, Attention augmented convolutional networks, in: Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 3286–3295.
  * [33] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [34] P. Shaw, J. Uszkoreit, A. Vaswani, Self-attention with relative position representations, arXiv preprint arXiv:1803.02155.
  * [35] K. A. Funes Mora, F. Monay, J.-M. Odobez, Eyediap: A database for the development and evaluation of gaze estimation algorithms from rgb and rgb-d cameras, in: Proceedings of the Symposium on Eye Tracking Research and Applications, 2014, pp. 255–258.
  * [36] Y. Sugano, Y. Matsushita, Y. Sato, Learning-by-synthesis for appearance-based 3d gaze estimation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1821–1828.
  * [37] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., Pytorch: An imperative style, high-performance deep learning library, in: Advances in neural information processing systems, 2019, pp. 8026–8037.
  * [38] L. Bottou, Large-scale machine learning with stochastic gradient descent, in: Proceedings of COMPSTAT’2010, Springer, 2010, pp. 177–186.
  * [39] I. Loshchilov, F. Hutter, Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint arXiv:1608.03983.