#  Knowing When to Quit: Selective Cascaded Regression with Patch Attention
for Real-Time Face Alignment

Gil Shapira  Samsung Semiconductor Israel R&D Center (SIRC)  Faculty of
Engineering, Bar-Ilan University  Noga Levy  Samsung Semiconductor Israel R&D
Center (SIRC)  Ishay Goldin  Samsung Semiconductor Israel R&D Center (SIRC)
Roy J. Jevnisek  Samsung Semiconductor Israel R&D Center (SIRC)

###### Abstract

Facial landmarks (FLM) estimation is a critical component in many face-related
applications. In this work, we aim to optimize for both accuracy and speed and
explore the trade-off between them. Our key observation is that not all faces
are created equal. Frontal faces with neutral expressions converge faster than
faces with extreme poses or expressions. To differentiate among samples, we
train our model to predict the regression error after each iteration. If the
current iteration is accurate enough, we stop iterating, saving redundant
iterations while keeping the accuracy in check. We also observe that as
neighboring patches overlap, we can infer all facial landmarks (FLMs) with
only a small number of patches without a major accuracy sacrifice.
Architecturally, we offer a multi-scale, patch-based, lightweight feature
extractor with a fine-grained local patch attention module, which computes a
patch weighting according to the information in the patch itself and enhances
the expressive power of the patch features. We analyze the patch attention
data to infer where the model is attending when regressing facial landmarks
and compare it to face attention in humans. Our model runs in real-time on a
mobile device GPU, with 95 Mega Multiply-Add (MMA) operations, outperforming
all state-of-the-art methods under 1000 MMA, with a normalized mean error of
8.16 on the 300W challenging dataset. The code is available at
_https://github.com/ligaripash/MuSiCa_

Figure 1: Selectively choosing the number of iterations according to the
estimated error. Each column presents the predictions of one image after
iterations 1, 2 and 3. The estimated error is printed above each sample. In
this example, the success threshold is set to 2.4. Estimated errors below this
value are marked in green and the following iterations are skipped (grayed
out), saving redundant computation. The final output of each sample is framed
in blue.

##  1 Introduction

Face alignment, or facial landmarks regression, is a vital step in many face
related applications, such as face recognition, detection of facial
expressions, face beautification, avatar rendering, and more. Many of these
applications need to run on mobile devices in real-time, but optimizing for
speed has been largely overlooked in previous years. Most of the published
facial alignment papers [1, 2, 3, 4], focus on accuracy optimization at the
expense of enormous computation demands. The most accurate models achieve
their accuracy at a price tag of more than 10 Giga Mutiply-Add operations
(GMA), which is a far cry from real-time implementations on mobile devices. On
the other end of the spectrum, regression-trees algorithms [5, 6] are super
fast but lack the accuracy of CNN based algorithms.

In this work, we aim to close this gap, putting our attention on both accuracy
and speed, and exploring the trade-off between them. We base our solution on
the cascaded regression paradigm as it lends itself to computation
optimization. The input to the model is cropped patches around the initial
estimate of the location of the landmarks, instead of the full image. Each
iteration in the process improves upon the estimate of the previous. All
previous models in this paradigm had a fixed number of iterations. Our
observation is that the input face complexity should determine the number of
iterations: simple frontal faces can be regressed accurately with only one
iteration. Further accuracy increase in the next iteration would be marginal
and may not justify the additional compute. On the other hand, faces with
extreme poses, expressions, lighting conditions or occlusions may need all
iterations to gain the required accuracy. This idea is illustrated in Figure
1. To implement it, we train our model to predict the regression error in each
iteration. We then need to determine a threshold for skipping the next
iterations. Every threshold value determines a specific operating point in the
compute vs. accuracy domain. In this paper we explore this domain and show how
to select a working point judiciously.

Architecturally, we take inspiration from MDM [7] for its lightweight feature
extractor with only two convolutions and effective information sharing between
iterations. We improve their architecture in several ways. Firstly, we use a
coarse-to-fine approach, cropping patches from a coarse face image on the
first iteration, and progressively higher resolutions on later iterations. As
patches size is fixed, each patch covers a large portion of the face in the
first iteration, which assists the model to infer the correct location and
pose of the face, and smaller portion in later iterations allowing the
algorithm for better local precision. Secondly, prior cascaded regression
models all had one to one relationship between facial landmarks (FLMs) and
patches. Observing that neighboring patches overlap, we propose models
regressing 68 FLMs with 34 or 19 predefined patches. These models reduce the
computation load considerably, with only a small accuracy decrease. These 34
or 19 patches have proved to be sufficient also in localizing many more facial
keypoints (100-300 in some applications) without any added computational cost.
Thirdly, we improved the expressive power of the model by utilizing a fine-
grained weighting for each component in the patch feature vector according to
the patch feature itself, emphasizing salient information and suppressing
irrelevant patch data.

Our main contributions include:

  * •

We expand the current cascaded regression paradigm of a fixed number of
iterations by learning the expected regression error and thresholding it to
get the appropriate number of iterations per sample.

  * •

We relax the one to one relation between patches and landmarks in cascaded
regression to reduce the computation load, with only a minor accuracy impact.

  * •

We devise an iterative coarse-to-fine architecture with local patch attention
to increase the model’s expressiveness while keeping computation demands in
check.

  * •

We offer a new data augmentation approach to reduce overfitting to the ‘mean
face’ by identifying faces that are ‘far’ from the mean face and oversampling
them.

Utilizing these ideas, our multi-scale Selective Cascaded Regression with
patch attention (MuSiCa) model achieves the best accuracy for models under 1
[GMA] operations on the 300W and WFLW datasets. We implement our model on
mobile-phone GPU and it runs in real-time.

##  2 Related Work

There are two dominating approaches to face alignment in modern literature:
direct regression and heatmap regression.

Direct coordinates regression methods regress the coordinates of the landmarks
directly from the image or image patches. Most of these methods use the
cascaded regression paradigm [8, 9, 6, 7, 5, 10] which provides a good
tradeoff between accuracy and speed. The algorithm starts from an initial FLMs
configuration, usually the ’mean face’. On each iteration of the cascade,
patches are cropped around the FLMs current estimated location. The model
tries to infer the displacement from the initial position to the correct FLMs
location, according to features extracted from the image patches. Each
regressor in the cascade is trained to infer the correction step from the
locations predicted by the previous regressor. ERT [5] implements this concept
by using an ensemble of regression trees with gradient boosting. SDM [10]
extract SIFT features from local patches to estimate the FLMs coordinates. MDM
[7] extends SDM by replacing SIFT with lightweight CNNs and sharing
information between iterations in the cascade.

Heatmap regression methods: instead of regressing the landmarks coordinates,
these methods regress one heatmap per landmark. The heatmap designates the
probability of the landmark location in each pixel. These heatmaps are created
by deep-stacked hourglass [3, 11, 2] or U-net architectures [12], and achieve
very high accuracy at the expanse of enormous computation resources. Most of
them don’t achieve real-time performance on strong GPUs, let alone mobile
devices.

Real-time deep methods have been mostly overlooked in the past but gained some
traction recently. [13] purpose a two-stage network. The first network is
lightweight, and only normalizes the input face image to a canonical pose, and
the second more massive network regresses the landmarks on the normalized
face. Another two-stage architecture is presented by [14]. They optimize for
speed and achieve 1100 fps at the expanse of degraded accuracy.

Learning to attend to the salient information is one of the essential concepts
in deep learning, with applications in natural language processing [15, 16],
speech recognition [17, 18] and computer vision [19, 20, 21]. In the face
alignment domain, attention mechanism is used in single-stage models to attend
to the location of the landmarks selectively. [22] offers a single-stage
regression model with a multi-scale attention map learned by direct
supervision to help the model attend to the location of the landmarks.
Similarly, [12] offers a cascade of U-nets with the same kind of supervised
multi-scale attention maps. We inspect the attention maps learned by our model
to better understand the facial features useful for this task, and offer an
insightful comparison to human face attention.

Error Estimation papers are scarce in facial alignment literature inspite of
its importance to downstream applications. [23] offer two types of landmarks
confidence measures: local and global. The local measure is inferred from the
local feature of each landmark, and the global is computed from a 3D rendered
face model. The motivation for the confidence measure is to assist the face
recognition task. [24] assess landmarks localization uncertainty per landmark
and also detect their visibility. In our work, the average normalized error is
directly predicted for the whole face, as we are interested in fail/pass value
for the complete regression for early stopping purposes.

##  3 MuSiCa: Multi-scale Selective Cascaded Regression with Patch Attention

###  3.1 Solution Details

Given a cropped face image $\mathbb{I}\in\mathbb{R}^{w\times h}$, face
alignment is the task of localizing $N$ predefined landmarks
$\mathbf{S}\in\mathbb{R}^{N\times 2}$. To save cycles, we take the multi-scale
cascaded regression approach. In iteration $i$, the model receives the current
estimate for the locations of the landmarks, $\mathbf{S}^{(i)}$, and a face
image $I^{(i)}$. The image resolution doubles with every iteration. In the
first iteration, $\mathbf{S}^{(0)}$ is set to an initial guess (the ’mean
face’) and $I^{(0)}$ to the lowest resolution image. In each iteration the
model infers the displacement $\Delta\mathbf{x}^{(i)}$ from $\mathbf{S}^{(i)}$
to the ground truth as expressed in Eq. 1

$\mathbf{S}^{(i+1)}=2\mathbf{S}^{(i)}+\Delta\mathbf{x}^{(i+1)}$ (1)

The multiplication by 2 is due to the image up-scaling in the next iteration.
Our solution is illustrated in Figure 3. Following MDM [7], our model contains
a recurrent component that transfers information between iterations, assisting
fast convergence. Unlike MDM, each iteration regresses images in different
scales; hence we use separate parameterization for each iteration. In each
iteration, we feed our network with small (14x14) patches cropped around the
landmarks. As our patches are fixed in size, they cover a large portion of the
small face at the first iteration and smaller parts as the resolution
increases, and localization improves, zooming in on the targets. Observing
that some landmarks are densely packed (for instance, landmarks around the
eyes) and patches overlap, we relax the standard paradigm in cascaded
regression of cropping patches around each landmark. To reduce computation, we
crop patches around $\mathbf{P}\subseteq\mathbf{S}$ landmarks skipping tightly
packed patches. The patch-less landmarks are regressed using the information
in patches of neighboring landmarks. Determining the size of $\mathbf{P}$ is a
useful handle moving the model on the accuracy-computation tradeoff curve. In
Figure 3, we mark landmarks in $\mathbf{S}\setminus\mathbf{P}$ in yellow (only
the pupils in this example) and landmarks in $\mathbf{P}$ in red. The cropped
patches are aggregated and passed to a lightweight patch feature extractor
$f_{c}^{(i)}$, where $i$ is the iteration index, see figure 4. The feature
extractor is composed of two regular convolutions and max-pooling in between.
To get the final multi-scale patch descriptor, we crop the center of the
second convolution, and concatenate it with the second max-pool output.

Local patch attention: At the next stage, we compute a fine-grained patch
attention function $f_{lpa}^{(i)}$ , inferring a single weight to each
component of the patch feature vector, with a fully connected layer (FC)
appended with a sigmoid activation to get weights between 0 and 1. The weight
vector is later element-wise multiplied with the patch feature vector to
produce the weighted feature vector.

Recurrence: To share information between iterations, We compute a hidden state
vector $\textbf{h}^{(i)}$ using the function $f_{r}^{(i)}$. The input to
$f_{r}^{(i)}$ is the concatenation of all patch vectors with the hidden state
of the previous iteration $\textbf{h}^{(i-1)}$. $f_{r}^{(i)}$ is implemented
with a FC layer followed by a tanh activation (Eq 2, 3).

$f^{(i)}:=f_{r}^{(i)}\circ f_{lpa}^{(i)}\circ f_{c}^{(i)}$ (2)
$\mathbf{h}^{(i)}=f^{(i)}(\cdot,\mathbf{S}^{(i-1)},\mathbf{\theta}^{(i)},\mathbf{h}^{(i-1)})$
(3)

In Eq. 3, $\theta^{(i)}$ denotes the weights of iteration network $f^{(i)}$.
To infer the final landmarks displacements, we use the function $f_{l}^{(i)}$.
$f_{l}^{(i)}$ is implemented as another FC layer taking $\mathbf{h}^{(i)}$ as
input (Eq. 4).

$\mathbf{\Delta
x}^{(i)}=f_{l}^{(i)}(\mathbf{h}^{(i)};\mathbf{\theta}_{l}^{(i)})$ (4)

###  3.2 Error Estimation (Knowing When to Quit)

In addition to the landmarks displacement, we compute the normalized estimated
error $E(i)$ for each iteration (Eq. 5),

$\displaystyle
E^{(i)}=f_{e}^{(i)}(\mathbf{h}^{(i)};\mathbf{\theta}_{e}^{(i)})$ (5)

and compare the estimated error against a regression success threshold
$T_{s}$. If $E(i)<T_{s}$, then the computation is terminated with the current
iteration; Otherwise, it continues. If the estimated error in the final
iteration is above the failure threshold $T_{f}$ then we declare the output is
invalid. Employing this early stopping scheme allows us to assign computation
resources per sample as needed. Simple frontal faces can usually be regressed
in one iteration while faces with extreme pose or expression may need all
iterations (Figure 1).

Choosing $\mathbf{T_{s}}$ allows us to control the accuracy-computation
tradeoff without retraining the model. To judiciously choose a threshold, we
would like to know the computation complexity and the expected error of the
model induced by each $T_{s}$ value. For each sample, we select the number of
iterations required to produce an estimated error below $Ts$; If this
criterion is not met, we choose the last iteration.

After setting the required number of iterations per sample for a given $Ts$,
we compute the average number of iterations and the normalized mean error for
this policy (blue curve in Figure 2). As the baseline to assess this policy’s
effectiveness, we count how many samples are assigned to each iteration and
randomly assign iterations to samples with the same count. We repeat this
random assignment 50 times to get a stable result (red curve in Figure 2).
These randomized policies have the same computation demand as the original
policy, so the accuracy improvement is attributed to our policy’s
effectiveness. Relative to a point with an average of $2$ iterations on the
baseline (green point), we improve the accuracy by $\sim 10\%$ if we keep the
average iterations constant (yellow point), or reduce the average number of
iterations by approximately half while maintaining the same accuracy (red
point). All samples are taken from the 300W challenging set.

The green line describes the theoretical optimal iteration selection policy,
achieved by using the true posterior error, instead of the estimated error.

Figure 2: The effectiveness of our early stopping policy. By selecting the
number of iterations per sample according to our error estimator (blue graph),
we can control the accuracy/time complexity tradeoff and significantly improve
over random iteration selection (red graph). Selecting the number of
iterations per sample according to the actual error (green graph) marks the
best possible assignment Figure 3: An illustrative example of MuSiCa. We start
with a coarse version of the input face and crop M small patches (14x14) at
the location of the ‘mean face’. To save computation, the number of patches
can be smaller than the number of landmarks. Landmarks without corresponding
patches are marked in yellow. A lightweight CNN,
$f_{c}^{(i)}(\cdot,\theta_{c}^{(i)})$ extracts features from the patches. The
local patch attention module $f_{lpa}^{(i)}(\cdot,\theta_{lpa}^{(i)})$
computes attention weights for each patch feature according to the patch
feature itself. We aggregate the patch features to a single descriptor and
pass it to the recurrent module $f_{r}^{(i)}(\cdot,\theta_{r}^{(i)})$ that
computes a hidden representation $\mathbf{h}(i)$ from the patch features of
the current iteration and the hidden state of the previous iteration. The
displacement $\mathbf{\Delta x_{i}}$ is computed from the hidden vector of
each iteration. At the next iteration, we crop patches from a finer resolution
according to the previously computed displacements. As patches are cropped
from different scales, the weights are not shared between iterations. In
addition to the estimated displacement, we compute the estimated error
$E^{(i)}$ after each iteration. We use the estimated error to stop the
computation if the accuracy is good enough $(E^{(i)}<T_{s})$. If the estimated
error after the last iteration is above the failure threshold $T_{f}$ then we
declare a regression failure. Figure 4: An illustrative example of a single
MuSiCa iteration. Small (14x14) patches are cropped around the current
estimated landmarks location. We extract features from each patch using a
lightweight CNN with two convolutions. Each patch feature is used to compute a
fine grained, local patch attention vector using a fully-connected layer with
sigmoid activation. The weight vector is later element-wise multiplied by the
patch vector. The weighted features are all concatenated together and with the
hidden state $h^{(i-1)}$ from previous iteration. The concatenated vector is
used to compute the new hidden state $h^{(i)}$. The estimated $N$ landmarks
and the error estimation $E^{(i)}$ are computed from the hidden state using a
fully-connected layer.

###  3.3 Training and Loss function

We train our model end-to-end. Following [25] we use the $L1$ Loss with
rectification of small errors to mitigate the impact of ground truth labeling
errors. To compute the estimated error we add a loss term which is the
absolute value of the difference between the estimated error and the actual
error.

Data Augmentation is crucial for training a robust facial alignment system
that generalizes well. This is especially true when training sets contain only
a few thousand samples, like the 300W training set, and the face bounding
boxes are not consistent in size and location, which is a common situation.
[26] conducted a thorough study, evaluating how different augmentation
strategies contribute to face alignment. One of their significant findings is
that geometric transformations (rotation, shear) are more important than
texture transformations. Building on their work, we apply the following
augmentations: non-uniform scaling, shearing, flip, in-plane rotation,
bounding box perturbation, color jetting, and gray-scale transformation as
described in [26]

###  3.4 Data Balancing

Many face alignment datasets suffer from unbalanced data where frontal faces
with regular expressions dominate the sample population. This imbalance causes
overfitting to standard pose and hurts the performance of the algorithm. To
mitigate this issue, [27] suggests a pose-based data balancing (PDB) strategy:
They align each ground-truth shape in the training set to the mean shape by
Procrustes Analysis with the mean shape as the reference shape. Next, they
apply a PCA to the aligned shapes and project each shape to the first
principal axis, associated with pose variation. Later, they create a histogram
of the projected values. Small bins in this histogram are related to extreme
poses that are under-represented in the dataset. They flatten this
distribution by duplicating samples with rare poses. Our generalized data
balancing method (GDB) extends this idea by trying to identify not only faces
with extreme poses but also extreme expressions and other, more subtle
idiosyncrasies.

We also perform a PCA to the aligned shapes, but instead of using only the
first principal component, we take the first K components. Each shape is
represented as a point in K-space. By measuring the Mahalanobis distance
between a shape and the average shape, we quantify how well these shape
features are represented in the training set. Shapes can have large
Mahalanobis distance not only due to the pose, but also due to extreme
expressions, age, or any other facial features that are under represented in
the training set, as depicted in Figure 5. The top row in this figure contains
a random sample from 20 faces with the maximal Mahalanobis distance out of
300W training set. These samples have faces with extreme pose, extreme
expressions, baby face etc. which are lacking in the dataset. The bottom row
includes examples with extreme first principal axis values related to pose
variability only. We over-sample faces in proportion to their Mahalanobis
distance to enhance under represented samples.

Figure 5: Face variability induced by Mahalanobis distance vs. first principal
axis. The top row depicts a random sample from the top 20 faces with the
largest Mahalanobis distance out of 300W training set. They exhibit extreme
values in pose, expression, age, etc. On the bottom row are six faces with
extreme first principal axis values. This axis is related to pose only, and
all other dimensions are ignored.

###  3.5 What are you looking at?

We were curious to know what part of the human face is the most indicative for
the task of face alignment; in other words, which patches draw most of the
model’s attention. To answer this question, we extracted the patch attention
vectors for all samples in the 300W-Fullset, averaged all 256 components of
each vector, and then averaged over all samples in the set. The results are
depicted in Figure 6. Most of the model’s attention is concentrated around the
upper center of the nose. This appeals to our intuition, as the nose is the
most ’out-of-plane’ organ in a face. As such, it exhibits the highest visual
variability under out-of-plane rotations and is most indicative of a face
pose.

We were also interested to know how the human visual system attends to the
human face. [28] answers this question by measuring the EEG signal, which
appears 170 [ms] (N170) after the face stimulus, and is linked to early-stage
face processing in the brain. [29] and [30] support these findings by
measuring the gaze time on each face location. Interestingly, As depicted in
Figure 6, humans and our face alignment model, are all focusing their
attention at the upper center of the nose when processing a face image.

Figure 6: The resemblance of patch attention in our model to human attention
heatmaps. From upper-right clockwise: [30], [29], [28], and our patch
attention averaged over 300W common test set. The patch attention points are
rendered at the FLMs initial position (the ’mean face’). In all cases the
attention maxima is located on the upper part of the nose. Figure 7:
Normalized mean error vs. Multiply-Adds count on 300W full test set. MuSiCa
models create a segmented line in this space, that is induced by different
early skipping thresholds.

##  4 Experiments

###  4.1 300W

300W [31] is currently the most widely used dataset for face alignment. It is
annotated semi-automatically with 68 landmarks, and also contains a bounding
box for each face. The training set is composed of 3148 images, and the test
set is composed of 689 faces, which are divided into the common set (554 faces
mostly frontal or semi-frontal) and a challenging test set of 135 faces with
extreme pose/expression/illumination. In our experiments, we put emphasis not
only on the accuracy but also on the computation resources and the tradeoff
between them. We normalize the errors by the inter-pupil distance as this is
the common practice. The 300W-test results are summarized in Table 2. The
papers are divided into two groups: the green group with models less than
1[GMA] and the red group with models above 1[GMA]. Our MuSiCa68 model (MuSiCa
with 68 patches) with 95 [MMA] dominate the green scoreboard, and outperforms
much heavier models like MDM [7] with 500[MMA], and DeFA [32] with 1.4 [GMA].
Figure 7 contains a graph of Normalized Mean Error vs. Multiply-Add operations
for several models, measured on 300W full test set. Each one of the MuSiCa X
models, where X is the number of patches, induces a segmented line in this
space. Each point on this line is related to a different value of $T_{s}$
threshold value for early exit. This gives the freedom to choose the
appropriate model according to the specific requirements of compute and
accuracy without retraining the model. Table 1 contains the computation
complexity of different MuSiCa models.

#FLMs | #Patches | MA ops | Memory [MB]
---|---|---|---
68 | 68 | 95 | 9.10
68 | 34 | 48 | 4.60
68 | 19 | 26 | 2.64
98 | 98 | 138 | 13.02
98 | 49 | 69 | 5.94
Table 1: MuSiCa variants Multiply Adds (Millions) with fixed 3 iterations

| 300-W
---|---
Method | Common | Challenging | Fullset | MA
LAB [2] | 3.42 | 6.98 | 4.12 |
DCFE [33] | 3.83 | 7.54 | 4.55 |
DAN [34] | 4.42 | 7.57 | 5.03 |
BL+AFS [35] | 3.13 | 7.34 | 3.82 |
DVLN [36] | 3.94 | 7.62 | 4.66 |
SIR [37] | 4.29 | 8.14 | 5.04 |
TR-DRN [38] | 4.36 | 7.56 | 4.99 |
DeFA [32] | 5.37 | 9.38 | 6.10 |
SDM [10] | 5.60 | 17.00 | 7.58 |
CFAN [39] | 5.50 | 16.78 | 7.69 |
MDM [7] | 4.83 | 10.14 | 5.88 |
FASTER [14] | - | - | 6.25 |
FNM_LAM [13] | 5.09 | 8.32 | 5.72 |
ERT [5] | - | - | 6.40 |
3DDFA [40] | 6.15 | 10.59 | 7.01 |
TCDCN [41] | 4.80 | 8.60 | 5.54 |
CFSS [42] | 4.73 | 9.98 | 5.76 |
MuSiCa19 (ours) | 5.22 | 9.50 | 6.05 |
MuSiCa34 (ours) | 4.95 | 8.76 | 5.69 |
MuSiCa68 (ours) | 4.63 | 8.16 | 5.32 |

Table 2: Results on 300W testset. Errors are normalized by the inter-pupil
distance. The blue models are below 1 [GMA] operations, and the red models are
above this threshold. Our method with 68 patches is the most accurate in the
light models category even though it has a complexity of only 90 [MMA], a far
cry from the 1 [GMA] threshold.

###  4.2 WLFW

[2] created the WFLW dataset based on WIDER Face [43] dataset. Each sample
contains 98 points annotations. There are 7500 training images and 2500
testing images which are separated to several subsets according to face
attributes: large pose, expression, illumination, make-up, occlusion and blur.
Table 3 contains our results on this dataset. As before, we split the methods
by computation complexity. Models above 1[GMA] operations are marked in red,
and methods below this threshold are marked in green. Our MuSiCa98 model is
the most accurate in the lightweight category. We follow the common practice
for this dataset and normalized the errors by the inter-ocular distance.

Method | Testset |  | Pose
---
Subset
| Expression
---
Subset
| Illumination
---
Subset
| Make-Up
---
Subset
| Occlusion
---
Subset
| Blur
---
Subset
MA
DVLN [36] | 6.08 | 11.54 | 6.78 | 5.73 | 5.98 | 7.33 | 6.88 |
LAB [2] | 5.27 | 10.24 | 5.51 | 5.23 | 5.15 | 6.79 | 6.32 |
Wing [27] | 5.11 | 8.75 | 5.36 | 4.93 | 5.41 | 6.37 | 5.81 |
HRNet [44] | 4.60 | 7.86 | 4.78 | 4.57 | 4.26 | 5.42 | 5.36 |
AWing [45] | 4.36 | 7.38 | 4.58 | 4.32 | 4.27 | 5.19 | 4.96 |
ESR [46] | 11.13 | 25.88 | 11.47 | 10.49 | 11.05 | 13.75 | 12.20 |
SDM [10] | 10.29 | 24.10 | 11.45 | 9.32 | 9.38 | 13.03 | 11.28 |
CFSS [46] | 9.07 | 21.36 | 10.09 | 8.30 | 8.74 | 11.76 | 9.96 |
MuSiCa44 (ours) | 8.38 | 17.89 | 8.82 | 8.12 | 9.11 | 10.6 | 9.67 |
MuSiCa98 (ours) | 7.90 | 15.8 | 8.52 | 7.49 | 8.56 | 10.04 | 8.92 |
Table 3: Results on WFLW. Models above 1[GMA] operations are marked in red,
and models below this threshold are marked in blue. Both of our 98 and 44
patches models are more accurate than other models in the lightweight
category.

###  4.3 Implementation on a mobile phone

We implemented a version of our model with 165 landmarks, 23 patches and, 2
iterations and without local patch attention on a Mali-G76 MP5 GPU of the
Samsung A71 mobile phone. The measured run-time of our algorithm on this
device is 14.77 [ms]. Samples of the output of this algorithm can be found in
the supplementary material.

##  5 Ablation Studies

To assess the contribution of a specific aspect of our method, we create a
model with the specific feature ablated and measure its accuracy on the 300W
challenging test set. Ablating the local patch attention mechanism reduces the
results by 8.72% (see Table 4). Using PDB [27] for data balancing instead of
our improved GDB reduced the accuracy by 3.3%. We also experimented with other
architectures for our feature extractor. Specifically, we replace the regular
convolutions layer with inverted bottlenecks [47], but this model degraded the
accuracy by 12.74%. We also experimented with a global patch attention
mechanism to complement our local patch attention. Global patch attention
computes a weighting to each patch feature vector according to the data in
other patches. We implemented a global attention module with both FC layer and
a GCN [48], but both of them were below the baseline (degraded accuracy of
10.29% and 12.50%, respectfully)

Method |  | 300W
---
Challenging
Set
| Change
---
relative to
baseline (%)
| No local
---
patch attention
8.72 | 6.86
|  Trained with PDB [27]
---
data balancing
instead of
our GDB method
8.43 | 3.31
| Inverted bottlenecks
---
feature extractor
9.20 | 12.74
| Global attention
---
with FC
9.17 | 10.29
| Global attention
---
with GCN
8.60 | 12.37
MuSiCa68 (baseline) | 8.16 | 0
Table 4: Ablation experiments on the 300W challenging set. Both local patch
attention and GDB contribute to increased accuracy. Our experiments with
inverted bottlenecks [47], and global patch attention did not improve the
accuracy.

##  6 Conclusion

In this paper, we study the tradeoff between accuracy and computation in face
alignment by offering a cascaded regression method that selectively chooses
when to quit the computation according to the estimated error, and by varying
the number of patches used for landmarks regression. Using different
thresholds for early skipping, we show how to choose a working point that
meets specific accuracy and computation demands. To increase our model’s
expressiveness, we offer a local patch attention mechanism to highlight the
important information and suppress the redundant patch data. We study what
face part draws the model attention, compare it to human face attention, and
draw parallels between them. We also offer an improved data balancing strategy
and prove its effectiveness. Our MuSiCa68 model offers an excellent tradeoff
between accuracy and compute and achieves the best results for models under
1[GMA] on 300W and WLFW dataset. We implemented a version of our model on a
mobile device GPU, and it runs in real-time.

##  7 Acknowledgement

We’d like to thank Geunhee Yang, Donghoon Kim, Yosi Keller, Tal Hassner, Ran
Vitek and the paper reviewers for their constructive feedback and discussions.

## References

  * [1] Shaun Zheng, Xiuxiu Bai, Lele Ye, and Zhan Fang.  Hafanet: An efficient coarse-to-fine facial landmark detection network.  IEEE Access, 8:123037–123043, 2020.
  * [2] Wayne Wu, Chen Qian, Shuo Yang, Quan Wang, Yici Cai, and Qiang Zhou.  Look at boundary: A boundary-aware face alignment algorithm.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2129–2138, 2018.
  * [3] Lisha Chen, Hui Su, and Qiang Ji.  Face alignment with kernel density deep neural network.  In Proceedings of the IEEE International Conference on Computer Vision, pages 6992–7002, 2019.
  * [4] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al.  Deep high-resolution representation learning for visual recognition.  IEEE transactions on pattern analysis and machine intelligence, 2020\.
  * [5] Vahid Kazemi and Josephine Sullivan.  One millisecond face alignment with an ensemble of regression trees.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1867–1874, 2014.
  * [6] Donghoon Lee, Hyunsin Park, and Chang D Yoo.  Face alignment using cascade gaussian process regression trees.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4204–4212, 2015.
  * [7] George Trigeorgis, Patrick Snape, Mihalis A Nicolaou, Epameinondas Antonakos, and Stefanos Zafeiriou.  Mnemonic descent method: A recurrent process applied for end-to-end face alignment.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4177–4187, 2016.
  * [8] Yue Wu, Tal Hassner, KangGeon Kim, Gerard Medioni, and Prem Natarajan.  Facial landmark detection with tweaked convolutional neural networks.  IEEE transactions on pattern analysis and machine intelligence, 40(12):3067–3074, 2017.
  * [9] Shahar Mahpod, Rig Das, Emanuele Maiorana, Yosi Keller, and Patrizio Campisi.  Facial landmarks localization using cascaded neural networks.  Computer Vision and Image Understanding, 205:103171, 2021.
  * [10] Xuehan Xiong and Fernando De la Torre.  Supervised descent method and its applications to face alignment.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 532–539, 2013.
  * [11] Zhiwei Liu, Xiangyu Zhu, Guosheng Hu, Haiyun Guo, Ming Tang, Zhen Lei, Neil M Robertson, and Jinqiao Wang.  Semantic alignment: Finding semantically consistent ground-truth for facial landmark detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3467–3476, 2019.
  * [12] Arnaud Dapogny, Kevin Bailly, and Matthieu Cord.  Decafa: deep convolutional cascade for face alignment in the wild.  In Proceedings of the IEEE International Conference on Computer Vision, pages 6893–6901, 2019.
  * [13] Zhiwei Liu, Xiangyu Zhu, Ming Tang, Zhen Lei, and Jinqiao Wang.  Efficient face alignment with fast normalization and contour fitting loss.  ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 15(3s):1–16, 2019.
  * [14] Pengfei Duan, Xin Ning, Yuan Shi, Shaolin Zhang, and Weijun Li.  Faster real-time face alignment method on cpu.  In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pages 398–408. Springer, 2019.
  * [15] Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao.  Attention-based lstm for aspect-level sentiment classification.  In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 606–615, 2016.
  * [16] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [17] Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio.  Attention-based models for speech recognition.  In Advances in neural information processing systems, pages 577–585, 2015.
  * [18] Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio.  End-to-end attention-based large vocabulary speech recognition.  In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4945–4949. IEEE, 2016.
  * [19] Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo.  Image captioning with semantic attention.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4651–4659, 2016.
  * [20] Volodymyr Mnih, Nicolas Heess, Alex Graves, et al.  Recurrent models of visual attention.  In Advances in neural information processing systems, pages 2204–2212, 2014.
  * [21] Niki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens.  Stand-alone self-attention in vision models.  In Advances in Neural Information Processing Systems, pages 68–80, 2019.
  * [22] Lei Yue, Xin Miao, Pengbo Wang, Baochang Zhang, Xiantong Zhen, and Xianbin Cao.  Attentional alignment networks.  In BMVC, volume 2, page 7, 2018.
  * [23] KangGeon Kim, Feng-Ju Chang, Jongmoo Choi, Louis-Philippe Morency, Ramakant Nevatia, and Gérard Medioni.  Local-global landmark confidences for face recognition.  In 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), pages 666–672. IEEE, 2017.
  * [24] Abhinav Kumar, Tim K Marks, Wenxuan Mou, Ye Wang, Michael Jones, Anoop Cherian, Toshiaki Koike-Akino, Xiaoming Liu, and Chen Feng.  Luvli face alignment: Estimating landmarks’ location, uncertainty, and visibility likelihood.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8236–8246, 2020.
  * [25] Zhen-Hua Feng, Josef Kittler, Muhammad Awais, and Xiao-Jun Wu.  Rectified wing loss for efficient and robust facial landmark localisation with convolutional neural networks.  International Journal of Computer Vision, pages 1–20, 2019.
  * [26] Zhen-Hua Feng, Josef Kittler, and Xiao-Jun Wu.  Mining hard augmented samples for robust facial landmark localization with cnns.  IEEE Signal Processing Letters, 26(3):450–454, 2019.
  * [27] Zhen-Hua Feng, Josef Kittler, Muhammad Awais, Patrik Huber, and Xiao-Jun Wu.  Wing loss for robust facial landmark localisation with convolutional neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2235–2245, 2018.
  * [28] Younes Zerouali, Jean-Marc Lina, and Boutheina Jemel.  Optimal eye-gaze fixation position for face-related neural responses.  PloS one, 8(6):e60128, 2013.
  * [29] Markus Bindemann, Christoph Scheepers, and A Mike Burton.  Viewpoint and center of gravity affect eye movements to human faces.  Journal of vision, 9(2):7–7, 2009.
  * [30] Line Sæther, Werner Van Belle, Bruno Laeng, Tim Brennen, and Morten Øvervoll.  Anchoring gaze when categorizing faces’ sex: evidence from eye-tracking data.  Vision research, 49(23):2870–2880, 2009.
  * [31] Christos Sagonas, Georgios Tzimiropoulos, Stefanos Zafeiriou, and Maja Pantic.  300 faces in-the-wild challenge: The first facial landmark localization challenge.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 397–403, 2013.
  * [32] Yaojie Liu, Amin Jourabloo, William Ren, and Xiaoming Liu.  Dense face alignment.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 1619–1628, 2017.
  * [33] Roberto Valle, Jose M Buenaposada, Antonio Valdes, and Luis Baumela.  A deeply-initialized coarse-to-fine ensemble of regression trees for face alignment.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 585–601, 2018.
  * [34] Marek Kowalski, Jacek Naruniec, and Tomasz Trzcinski.  Deep alignment network: A convolutional neural network for robust face alignment.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 88–97, 2017.
  * [35] Teng Wang, Xinjie Tong, and Wenzhe Cai.  Attention-based face alignment: A solution to speed/accuracy trade-off.  Neurocomputing, 2020.
  * [36] Wenyan Wu and Shuo Yang.  Leveraging intra and inter-dataset variations for robust face alignment.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 150–159, 2017.
  * [37] Tao Hu, Honggang Qi, Jizheng Xu, and Qingming Huang.  Facial landmarks detection by self-iterative regression based landmarks-attention network.  arXiv preprint arXiv:1803.06598, 2018.
  * [38] Jiangjing Lv, Xiaohu Shao, Junliang Xing, Cheng Cheng, and Xi Zhou.  A deep regression architecture with two-stage re-initialization for high performance facial landmark detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3317–3326, 2017.
  * [39] Jie Zhang, Shiguang Shan, Meina Kan, and Xilin Chen.  Coarse-to-fine auto-encoder networks (cfan) for real-time face alignment.  In European conference on computer vision, pages 1–16. Springer, 2014.
  * [40] Xiangyu Zhu, Zhen Lei, Xiaoming Liu, Hailin Shi, and Stan Z Li.  Face alignment across large poses: A 3d solution.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 146–155, 2016.
  * [41] Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang.  Facial landmark detection by deep multi-task learning.  In European conference on computer vision, pages 94–108. Springer, 2014.
  * [42] Shizhan Zhu, Cheng Li, Chen Change Loy, and Xiaoou Tang.  Face alignment by coarse-to-fine shape searching.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4998–5006, 2015.
  * [43] Shuo Yang, Ping Luo, Chen-Change Loy, and Xiaoou Tang.  Wider face: A face detection benchmark.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5525–5533, 2016.
  * [44] Ke Sun, Yang Zhao, Borui Jiang, Tianheng Cheng, Bin Xiao, Dong Liu, Yadong Mu, Xinggang Wang, Wenyu Liu, and Jingdong Wang.  High-resolution representations for labeling pixels and regions.  arXiv preprint arXiv:1904.04514, 2019.
  * [45] Xinyao Wang, Liefeng Bo, and Li Fuxin.  Adaptive wing loss for robust face alignment via heatmap regression.  In Proceedings of the IEEE International Conference on Computer Vision, pages 6971–6981, 2019.
  * [46] Xudong Cao, Yichen Wei, Fang Wen, and Jian Sun.  Face alignment by explicit shape regression.  International Journal of Computer Vision, 107(2):177–190, 2014\.
  * [47] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [48] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  arXiv preprint arXiv:1609.02907, 2016.