# AI methods for approximate compiling of unitaries

David Kremer, Victor Villar, Sanjay Vishwakarma, Ismael Faro and Juan Cruz-
Benito  IBM Quantum. IBM T.J. Watson Research Center,
Yorktown Heights, NY, USA

###### Abstract

This paper explores artificial intelligence (AI) methods for the approximate
compiling of unitaries, focusing on the use of fixed two-qubit gates and
arbitrary single-qubit rotations typical in superconducting hardware. Our
approach involves three main stages: identifying an initial template that
approximates the target unitary, predicting initial parameters for this
template, and refining these parameters to maximize the fidelity of the
circuit. We propose AI-driven approaches for the first two stages, with a deep
learning model that suggests initial templates and an autoencoder-like model
that suggests parameter values, which are refined through gradient descent to
achieve the desired fidelity. We demonstrate the method on 2 and 3-qubit
unitaries, showcasing promising improvements over exhaustive search and random
parameter initialization. The results highlight the potential of AI to enhance
the transpiling process, supporting more efficient quantum computations on
current and future quantum hardware.

###### Index Terms:

Artificial Intelligence, Quantum Computing, Unitary synthesis, Approximate
compiling

##  I Introduction

Transpiling is a critical stage in the current quantum computing workflows,
and consists of adapting a given circuit to a set of target instructions
supported on specific quantum devices. Modern quantum circuit transpilers
include several passes dedicated to different tasks [1, 2, 3], and aim to
adapt the given quantum circuit to the hardware with as little gate and depth
overhead as possible to reduce the effects of noise on the quantum hardware.

Unitary Synthesis is a fundamental task within most transpilers. It consists
of, given a unitary matrix, finding a sequence of gates within the hardware’s
instruction set that implement the given matrix.

In this paper we focus on exploring how AI methods can improve Unitary
Sythesis, a central part within typical transpiling workflows. We focus on the
case where the instruction set is composed of fixed two-qubit gates (such as
CZ or CX) and single qubit rotations (such as RZ) that can implement arbitrary
rotations, as is common in current superconducting hardware [4, 5].

Within this setting, exising algorithms broadly fall into two categories:

  * •

Exact methods, such as the two qubit KAK-based decomposition described in [6]
or the more general Quantum Shannon Decomposition [7, 8] that exactly
implement the given unitary. These methods provide an upper bound on the
number of two-qubit gates for general unitaries, but except for the two qubit
case, the circuits they produce are far from optimal for unitaries that can be
found in typical applications.

  * •

Approximate methods that, based on a given circuit template (a circuit with
fixed gates and adjustable single qubit rotations), adjust the circuit
parameters to maximize the fidelity of the circuit with respect to the given
unitary [9, 10].

Although the approximate methods scale beyond 2 qubits, they have some
drawbacks that make their widespread use impractical for transpiling:

  * •

Template selection is not straightforward, and most methods resort to search
methods that require iterating over several templates to find one that can
successfully implement the target unitary (up to some high fidelity
threshold).

  * •

Optimization of the parameters is time consuming, and it often reaches local
minima, resulting in sub-optimal fidelities even when the given template is
capable of reaching perfect fidelity for the target unitary.

There have been recent attempts to improve this process. In [9], they use
tensor networks to reduce the amount of computation during the optimization,
allowing them demonstrate approximate compiling on shallow circuits on 27
qubits, but only for state preparation and not for the full unitary
transformation.

In [11], they propose a tree-based search method where they train a deep
learning model to predict which template to use from a set of 3-qubit
templates, and start a tree-based template search from there, but they don’t
address the problem of parameter optimization.

Here we explore a variety of Machine Learning based methods to potentially
speed up the process and allow synthesis of larger unitaries on typical
transpiling pipelines, and propose simple AI-based algorithms for template
selection and circuit parameter optimization.

The paper is organized as follows: Section II describes the general approach
and the Machine Learning methods used and the data generation procedures used
for the template selection and the parameter optimization. Section III shows
the results obtained by the method for 2 and 3 qubit unitary synthesis.
Finally, in Section IV we discuss the results and future research directions
of interest.

##  II Materials and methods

For addressing the unitary synthesis problem, we develop a pipeline for
unitary synthesis composed of three stages:

  1. 1.

Template selection. A deep learning model that suggests a template given a
target unitary.

  2. 2.

Parameter prediction. A deep learning model that suggests an initial value for
the template’s parameters as a starting point for the optimization.

  3. 3.

Parameter optimization. Further Optimization of the parameters via gradient
descent, up to the desired fidelity.

For synthesizing a given unitary, the unitary is first assessed by the
template selection model, which suggests a first template, and then the
unitary is passed to the second model that provides suggested values for the
template. These values are then used as starting point for an optimization via
gradient descent. If the desired fidelity is not reached, the procedure starts
again by selecting the second template suggested by the model, and continues
until the objective fidelity is reached, or defaults to other methods if the
suggestions are exhausted.

When used in the context of circuit optimization, where circuit blocks are
collected from a larger circuit and re-synthesized, the two qubit depth and
count of the block presents an upper bound on the template size. This can be
used to filter the suggestions to the ones that incur lower cost. If the model
is confident that the cost of a block cannot be improved, the block can be
left as is without attempting optimization.

For the cases when the template selection model guesses correctly, and where
the second model suggests a useful starting point, this may result in
significant speedups with respect to exhaustive template search, and might
result in better fidelities by avoiding local minima.

In the next subsections we focus on the first and second stages and describe
the network architectures used and the data generation processes.

###  II-A Template prediction

For the first stage we use supervised deep learning techniques to select the
right template to be used in the second step.

####  II-A1 Network architecture and training

We train a multi layer neural network to predict a template, which poses a
supervised classification problem. The input to the neural network is a vector
with the raw values of the entries of the unitary matrix, split in their real
and imaginary parts (this gives a total number of elements of $2\cdot 2^{N}$,
being N the number of qubits). For our experiments, we have seen that simply
using standard fully connected layers for the network architecture is enough
for achieving high accuracy.

The output of the network is a vector with the probabilities assigned to each
of the possible templates. Since we only allow one ”correct” template for a
given unitary, we train the network to minimize the cross entropy loss between
the probabilities provided by the network and the template for which the
unitary was generated. As it is often done in multiclass classification, the
last layer of the neural network normalizes the output with a softmax so that
the probabilities sum up to one.

####  II-A2 Data generation

During the training, we generate batches of unitary-template pairs on the fly.
This ensures an ”unlimited” supply of data and acts as a form of
regularization preventing overfitting to a particular fixed set of unitaries.

We first define a set of templates for which we want to predict. Note that
this doesn’t need to include the complete set of possible templates for a
given number of qubits; in particular we are typically interested in shallower
unitaries that appear frequently on larger circuits, and we can always include
an extra term in the model’s output to indicate if a given unitary is not
implementable in any of the templates of the set. In our experiments, we build
the templates by combining single-qubit unitary gates and two-qubit CZ gates.
A few examples of templates for 3 qubits can be found in Figure 1.

(a) Template 9

(b) Template 12

Figure 1: Examples of two templates for 3 qubits.

For generating each row of the batch, we first randomly pick a template from
the template set, which will be the label that will be predicted by the model.
We then generate a random instance of the chosen template by assigning random
values for the parameters of the single-qubit unitaries, and we obtain the
unitary matrix corresponding to the circuit.

(a) Encoder network

(b) Decoder

Figure 2: Encoder and decoder networks for template parameter suggestion

###  II-B Parameter suggestion

For the second stage we define a network architecture and training procedure
to generate good suggested values for the template parameters for a given
template.

A naive approach for predicting good parameters for a given unitary would be
to simply train a neural network model on unitary-parameter pairs. For
generating a unitary-parameter pair one would first uniformly sample a group
of parameters from the allowed parameter range (typically $[-\pi,\pi]$ for
rotation angles), and then obtain the corresponding unitary by setting these
parameters in the template. Repeating this for multiple parameter values would
yield a dataset for training. However, we have found that with this method the
models fail to produce useful results, or to learn anything at all. Because a
given unitary can be implemented by multiple (very different) sets of
parameters for the given template, by using this procedure we may generate
multiple different target parameters for unitaries that are potentially very
close. This results in a dataset of unitary-parameters pairs that cannot be
described by a mapping that assigns a single value for the parameters given a
unitary matrix, and prevents the network from learning any useful
relationship.

Although one could overcome this by generating the dataset in a consistent
manner, for example by first sampling a unitary matrix and then using gradient
descent from a fixed starting point, this approach still has some fundamental
limitations. One of the main drawbacks is that by training on unitary-
parameter pairs we are teaching the network to, in some way, interpolate
between the parameters in the dataset; however, the actual objective is not to
provide parameters that mimick the dataset, but to provide parameters that
maximize the fidelity of the circuit.

####  II-B1 Network architecture and training

For overcoming this limitation, we define a deep learning model based on the
autoencoder architecture [12, 13, 14], consisting on an encoder network and a
decoder. Here, the encoder is a deep learning network (as in standard
decoders) that takes the unitary matrix as input and outputs a value for each
of the parameters on the template. The decoder, unlike typical autoencoders,
is fixed, and corresponds to the simulation of the actual template of the
quantum circuit from the parameters output by the encoder.

For training, we first take an input unitary and pass it through the encoder
to generate the predicted parameters. We then pass this parameters through the
decoder to reconstruct the unitary. With both unitaries, we calculate the
reconstruction fidelity of the output unitary with respect to the input
unitary. Since the simulation of the circuit is also differentiable, we
backpropagate the gradients of the fidelity with respect to the encoder
network’s weights and use them to train the network. This allows us to train
the network directly to maximize the reconstruction fidelity.

For the encoder network architecture, we use again fully connected layers, but
this time we choose complex-valued weights for the trainable parameters. This
allows us to directly input the complex values of the unitary matrix and
output complex values for the entries of the single-qubit unitary matrices of
the template that represent the template’s parameters.

An illustration of the encoder and decoder architecture can be found in Figure
2.

####  II-B2 Data generation

With the autoencoder architecture, our dataset now only consists of ”x” points
that correspond to sampled unitaries, since we don’t need to generate any
parameters to train on. For the training, as in previous sections, we generate
the unitaries on the fly. We generate the unitaries by sampling from the given
template, to ensure they can be implemented on that template.

##  III Results

(a) Probability assigned to each template as a function of $n/pi$.

(b) Template predicted as a function of $n/pi$.

Figure 3: Probability assigned to each template and predicted template as a
function of $n/pi$, where $n$ is the angle used in gate CZ for the circuit
described in the text.

In this section we show the results obtained in training models for 2 and 3
qubits for both stages. For the models we trained, we selected the instruction
set of the IBM Quantum Heron devices [15]: we use CZ for the two-qubit gates,
and implement the single-qubit unitaries as a sequence of 3 CZ rotations
separated by SX gates.

###  III-A Template prediction

For the 2 qubits model, we use 4 different templates, containing from 0 to 3
CZ gates respectively. As pointed out in [6], this set of templates is
complete in the sense that it allows to implement any two qubit unitary with
optimal number of two qubit gates.

For 3 qubits, we train two models: one for 15 different templates, covering
all possible combinations using up to 3 CZ gates, and another for 63 possible
templates, covering all possible combinations up to 5 CZ gates. This set of
templates is not enough to cover all possible unitary circuits, but it is
useful for optimizing small blocks that typically appear on circuits.

In order to assess if model capacity was a limiting factor, we train multiple
models of different sizes. The results are summarized on Table I.

TABLE I: Template prediction training parameters. | Train configuration | Validation result
---|---|---
#Qubits | #Network Parameters ($10^{6}$) | Size (MB) | Accuracy
2 | 0.2 | 0.8 | 0.95
7.4 | 29.5 | 0.96
12.6 | 50.6 | 0.97
3 (3 CZ) | 3.3 | 13.2 | 0.74
5.4 | 21.5 | 0.75
21.3 | 85.1 | 0.77
3 (5 CZ) | 9.6 | 38.5 | 0.37

For 2 qubits we reach a validation accuracy of $\simeq 96\%$, with a
difference of only $2\%$ between the worst and the best models. Even if we see
a trend on decreasing accuracy with decreasing model sizes, the difference is
small and may be improved with different training hyperparameters or longer
training time. Overall, an interesting thing to note is that we can reach high
accuracy with very small models, with the smallest one we tested being only
$0.8$ MB.

To test the robustness of the template prediction for 2 qubits, we run a
simple experiment where we predict the template for a circuit consisting of a
two consecutive CNOT gates separated by an RZ gate on the first qubit. We vary
the angle of the RZ gate from $-\pi$ to $\pi$, allowing us to test how the
model transitions between templates: the two-CZ template for most of the
range, the zero-CZ template when the angle is a multiple of $\pi$ (allowing
the two CNOTs to cancel out) and a less evident one-CZ template when the angle
is $\pm\pi/2$.

In Figure 3(a) we can see the probability the model assigns to each template
(with 0, 1, 2 or 3 CZ gates). The model predicts with high probability the
right template in most of the range, except around $n/\pi\simeq 0.45$, where
it incorrectly predicts the template with 3 CZ gates instead of the one with 2
CZ gates by a close margin. In figure 3(b) we validate the predictions
obtained against the solution provided by the $Qiskit$ transpiler (using the
KAK-based method described in [6]).

For 3 qubits with up to 3 CZ, the best model reaches an accuracy of $\simeq
77\%$ when selecting the highest probability template. If we also consider the
second highest probability, the correct template is present in this set
approximately $\simeq 98\%$ of the time. The model 3 qubits and up to 5 CZ
reaches $\simeq 37\%$ after 24h of training on a single GPU (NVIDIA V100
16GB), and $\simeq 75\%$ if considering the templates with top 5 probability,
which can likely improve with longer training. If we use this model as a
template recommender, and ”visit” the templates in the order suggested by the
model, on average we would take $\simeq 3.5$ visits until we reach the correct
template. This is in contrast with the $63/2$ visits on average that we would
have to make if we did an exhaustive template search.

To understand the kind of errors the model makes, we show the confusion matrix
for the 3-CZ model in Figure 4.

Figure 4: Confusion matrix for the 3 qubits template prediction model, for
1000 unitaries sampled uniformly from each template.

In the table we can see that the model guesses correctly for most of the
templates, but fails for specific templates. In particular, the model usually
has problems telling apart templates with the same CZ gates but where they
appear in different order, such as templates 9 and 12 shown in Figure 1. This
is likely due to these templates generating unitaries that have a significant
overlap, and might improve with more careful generation of the dataset.

Figure 5: Template used for the 3 qubit, 10 layer parameter suggestion model.
Each single qubit unitary gate is decomposed as three RZ gates separated by SX
gates, following the instruction set of ibm_torino [16].

###  III-B Parameter suggestion

For parameter suggestion, we train models for the 4 different two-qubit
templates, and for 6 and 10 layer 3-qubit templates. The 10 layer 3-qubit
template is shown in Figure 5.

For the two qubit templates, the models achieve a very high fidelity of 0.95
on average. For the three qubit templates, the models achieve around 0.5
fidelity after around 24h of training. The results are summarized in Table II.

#Qubits | #Template layers | #Template parameters | Avg. Fidelity
---|---|---|---
2 | 0 | 6 | 0.98
1 | 12 | 0.96
2 | 16 | 0.96
3 | 20 | 0.95
3 | 6 | 45 | 0.52
10 | 69 | 0.51
TABLE II: Parameter suggestion model results

In Figure 6 we show the fidelities corresponding to the parameters suggested
by the model for the 10 layer 3-qubit template, for 100 random unitaries,
compared against the fidelities obtained by selecting random parameters.
Although the results may still improve with further training, we see that the
suggestions are already useful when paired with the optimization procedure to
refine the results.

Figure 6: Fidelity achieved at the starting point of the optimization by the
parameter suggestion model for the 3-qubit, 10-layer template, on a set of 100
3 qubit unitaries sampled from the template. For comparison, we show the
fidelity achieved on the starting point by selecting the parameters at random.

Figure 7 shows the result of compiling the Toffoli gate for the same template
by using different starting points. For the parameters selected by the model,
we see that the fidelity starts high and quickly converges to 1 at around 1000
iterations. For comparison, out of the 5 random starting points shown, two got
stuck in a local minimum and the other three converged to fidelity 1, but took
20-30% more iterations.

Figure 7: Unitary reconstruction error as a function of the gradient descent
optimization step for the Toffoli gate, using the parameters suggested by the
model vs using 5 different random starting points.

##  IV Discussion

In this paper we have described an AI-based procedure for approximate
synthesis of unitaries. We address the problem in three stages: template
selection, parameter suggestion, and parameter refinement, and propose AI-
based methods for the first two stages.

We demonstrate the idea by training two and three qubit models for the two
stages. The results look in general promising: for the two qubit models, we
reach almost perfect accuracy for template selection, and very high fidelity
for the parameter suggestion. The accuracy and fidelity results for 3 qubits
are not as high, but the models we show have only been partially trained for
24h (on a single GPU), and as described in Section III they already provide
advantage over exhaustive template search and random parameter initialization.

Interestingly, we have not found the model capacity to be a limited factor in
the learning, at least in the range we have tried, and we have been able to
train relatively small models (less than 1MB) that still produce high
accuracy. This hints at the possibility that further improvements can be made
by a more careful generation of the datasets and by tuning the training
hyperparameters.

## Acknowledgments

We would like to thank Ali Javadi for the useful discussions on the topic of
approximate compiling of unitaries.

## References

  * [1] Qiskit contributors, “Qiskit: An open-source framework for quantum computing,” 2023.
  * [2] A. Javadi-Abhari, M. Treinish, K. Krsulich, C. J. Wood, J. Lishman, J. Gacon, S. Martiel, P. D. Nation, L. S. Bishop, A. W. Cross _et al._ , “Quantum computing with qiskit,” _arXiv preprint arXiv:2405.08810_ , 2024.
  * [3] D. Kremer, et al., “Practical and efficient quantum circuit synthesis and transpiling with reinforcement learning,” _arXiv:2405.13196_ , 2024.
  * [4] J. M. Chow, J. M. Gambetta, A. D. Corcoles, S. T. Merkel, J. A. Smolin, C. Rigetti, S. Poletto, G. A. Keefe, M. B. Rothwell, J. R. Rozen _et al._ , “Universal quantum gate set approaching fault-tolerant thresholds with superconducting qubits,” _Physical review letters_ , vol. 109, no. 6, p. 060501, 2012.
  * [5] A. Niskanen, K. Harrabi, F. Yoshihara, Y. Nakamura, S. Lloyd, and J. S. Tsai, “Quantum coherent tunable coupling of superconducting qubits,” _Science_ , vol. 316, no. 5825, pp. 723–726, 2007.
  * [6] A. W. Cross, L. S. Bishop, S. Sheldon, P. D. Nation, and J. M. Gambetta, “Validating quantum computers using randomized model circuits,” _Physical Review A_ , vol. 100, no. 3, Sep. 2019. [Online]. Available: http://dx.doi.org/10.1103/PhysRevA.100.032328
  * [7] B. Drury and P. Love, “Constructive quantum shannon decomposition from cartan involutions,” _Journal of Physics A: Mathematical and Theoretical_ , vol. 41, no. 39, p. 395305, Sep. 2008. [Online]. Available: http://dx.doi.org/10.1088/1751-8113/41/39/395305
  * [8] V. Shende, S. Bullock, and I. Markov, “Synthesis of quantum-logic circuits,” _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , vol. 25, no. 6, p. 1000–1010, Jun. 2006. [Online]. Available: http://dx.doi.org/10.1109/TCAD.2005.855930
  * [9] N. F. Robertson, A. Akhriev, J. Vala, and S. Zhuk, “Approximate quantum compiling for quantum simulation: A tensor network based approach,” 2023.
  * [10] L. Madden and A. Simonetto, “Best approximate quantum compiling problems,” _ACM Transactions on Quantum Computing_ , vol. 3, no. 2, pp. 1–29, 2022.
  * [11] M. Weiden, E. Younis, J. Kalloor, J. Kubiatowicz, and C. Iancu, “Improving quantum circuit synthesis with machine learning,” 2023.
  * [12] G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” _science_ , vol. 313, no. 5786, pp. 504–507, 2006\.
  * [13] Y. Bengio _et al._ , “Learning deep architectures for ai,” _Foundations and trends® in Machine Learning_ , vol. 2, no. 1, pp. 1–127, 2009.
  * [14] D. Bank, N. Koenigstein, and R. Giryes, “Autoencoders,” _Machine learning for data science handbook: data mining and knowledge discovery handbook_ , pp. 353–374, 2023.
  * [15] J. Gambetta, “Quantum-centric supercomputing: The next wave of computing,” _IBM Research Blog_ , 2022.
  * [16] IBM Quantum, “ibm_torino system,” https://quantum.ibm.com/services/resources?tab=systems&system=ibm_torino, [Accessed 22-04-2024].


# DAD-3DHeads: A Large-scale Dense, Accurate and Diverse
Dataset for 3D Head Alignment from a Single Image

Tetiana Martyniuk1,2 * Orest Kupyn1,2 * Yana Kurlyak1,2 Igor Krashenyi1,2
Jiři Matas3 Viktoriia Sharmanska4, 5
1 Ukrainian Catholic University 2 Piñata Farms, Los Angeles, USA
3 Visual Recognition Group, Center for Machine Perception, FEE, CTU in Prague
4 University of Sussex 5 Imperial College London

###### Abstract

We present DAD-3DHeads, a dense and diverse large-scale dataset, and a robust
model for 3D Dense Head Alignment in-the-wild. It contains annotations of over
3.5K landmarks that accurately represent 3D head shape compared to the ground-
truth scans. The data-driven model, DAD-3DNet, trained on our dataset, learns
shape, expression, and pose parameters, and performs 3D reconstruction of a
FLAME mesh. The model also incorporates a landmark prediction branch to take
advantage of rich supervision and co-training of multiple related tasks.
Experimentally, DAD-3DNet outperforms or is comparable to the state-of-the-art
models in (i) 3D Head Pose Estimation on AFLW2000-3D and BIWI, (ii) 3D Face
Shape Reconstruction on NoW and Feng, and (iii) 3D Dense Head Alignment and 3D
Landmarks Estimation on DAD-3DHeads dataset. Finally, diversity of DAD-3DHeads
in camera angles, facial expressions, and occlusions enables a benchmark to
study in-the-wild generalization and robustness to distribution shifts. The
dataset webpage is https://p.farm/research/dad-3dheads.

Figure 1: DAD-3DHeads, a Dense, Accurate, and Diverse 3D Head dataset, labeled
with over 3.5K verified accurate landmarks. A model trained on DAD-3DHeads
achieves superior performance on diverse 3D head tasks. It is robust to domain
shifts common in the wild, including head pose changes, occlusions, facial
expressions, age groups, illumination conditions, and image quality. Best
viewed in color.

**footnotetext: These authors contributed equally to this work.

##  1 Introduction

Tremendous progress in 3D face analysis has been made since the first 3D
morphable model (3DMM) [4] from an image had been proposed [23]. The use cases
for precise 3D face models are abundant: accurate face recognition and face
detection [16], realistic 3D avatars and animation for VR and games [39], face
re-enactment and synthesis for dubbing [63], virtual mirrors and try-on,
statistical shape models for medical tasks such as segmentation and analysis
of variations in anatomical structures [81].

These applications require not only accurate 3D face geometry but also (1)
handling the diversity, e.g., ethnic, age, gender subgroups, and (2)
generalizing to in-the-wild deployment conditions, i.e., beyond controlled
capture and beyond the data they are trained on. The largest face models up-
to-date [48, 8] have focused on the (1) aspect by collecting diverse 3D face
and head scans, and building 3DMMs models for different age, gender and
ethnicity. In-the-wild generalization has been identified as a pressing
challenge of the next generation 3D face models [23]. This (2) aspect of in-
the-wild generalization is the focus of our study.

The progress that we have witnessed with deep learning has impacted closely
related facial analysis tasks such as Landmark Localisation [61, 49, 54, 14,
20], Facial Alignment in 2D and 3D [78, 17, 1, 33, 12, 10, 51, 11, 34, 68],
and Face Detection [49, 78, 17, 1, 27, 20]. This has been driven by the
community effort towards collecting and annotating large image datasets
captured in unconstrained conditions, building enhanced models that can take
advantage of such large datasets, and most importantly _openness_ , i.e.,
making the models and datasets publicly available for research use.

Nevertheless, 3D face or head alignment from a single image in the wild
remains an open challenge. The difficulty comes from (1) lack of 2D-3D ground-
truth data and, as a result, (2) ambiguity of the task and reliance on 3D
shape priors. Many methods have been developed to fill the gap of missing
2D-3D annotations (1), primarily using 2D landmarks datasets for fitting, or
exploring extra knowledge such as identity invariance [56], or co-training
with related face detection [20], [16] tasks to drive the recovery of 3D face
geometry. Up until now, evaluation of the efficiency of these approaches has
been problematic due to the lack of ground-truth data. Regarding (2), the
state-of-the-art 3D face reconstruction methodologies such as non-linear 3DMMs
and deep learning models [5, 8, 6, 41, 48] are based on learning a statistical
3D facial model and fitting it to the image as a shape (or shape and texture)
prior. This direction has a long history tracing back to the seminal work of
Blanz and Vetter [4]. It relies on a large and diverse dataset of 3D/4D scans
to build the statistical 3D face model that can be decomposed into facial
shape (identity and expression), and the camera parameters. This comes at the
cost of laborious data collection with expensive 3D acquisition devices, and
the fact that 3D acquisition devices cannot operate in arbitrary conditions.
Hence, the current 3D facial databases have limited data sample size and have
been captured not-quite-in-the-wild [56].

In this work, we show that _without expensive devices, like scanners, that are
difficult to deploy in the wild, we can collect accurate annotations of 3D
landmarks directly from images, which is labor-efficient and effective to push
the state-of-the-art results for 3D head recovery from images_.

Our contributions are as follows:

  * •

A new Dense, Accurate and Diverse dataset for 3D Dense Head Alignment in-the-
wild, DAD-3DHeads. It has over 3.5K verified accurate landmarks, the densest
annotations for 3D dense head alignment in-the-wild currently available.
DAD-3DHeads contains a variety of extreme poses, facial expressions,
challenging illuminations, and severe occlusions cases. Accuracy and
consistency of the annotations are compared to the ground truth 4D scans and
head pose labels.

  * •

A novel way to address the problems of shape reconstruction and pose
estimation _simultaneously_ during training via optimizing two loss
components: (i) Shape+Expression Loss and (ii) Reprojection Loss. (i) is based
on the normalized 3D vertices that enables disentangling the shape and
expression information from the pose; (ii) is based on the full head dense 2D
landmarks and assesses the pose accuracy. That makes the rich annotations
fully utilized, which could not have been done previously due to the lack of
GT annotations. Extensive ablation studies show the importance of both loss
components.

  * •

DAD-3DNet model that maps an input image to 3D mesh representation consistent
with the FLAME topology. The model is trained end-to-end by regressing the
3DMM parameters and recovering the 3D head geometry with differential FLAME
decoder. The proposed approach learns the head shape, pose, and expression
simultaneously. DAD-3DNet outperforms state-of-the-art on a range of tasks,
suggesting that dense supervision as provided in our dataset, enables a
holistic framework for 3D Head Analysis from images.

  * •

A novel benchmark with the evaluation protocol for quantitative assessment of
3D dense head fitting, i.e. 3D Head Estimation from dense annotations. Our
evaluation protocol introduces two novel metrics: Reprojection NME computing
the NME of the reprojected 3D vertices onto the image plane, and Zn Accuracy
evaluating the ordinal distance of the $Z$-coordinate and accuracy of the 3D
fitting.

##  2 Related Work

This section provides an overview of the available 3D face datasets, followed
by a survey of the methods targeting 3D head-related tasks.

3D Face Datasets. Existing 3D face datasets differ based on registration of a
3D face model. Model fitting datasets [32, 7, 5] fit the 3DMM to the images,
which makes it suitable for large-scale datasets. The main limitation of such
approach is shape detalization. To get a precise 3D facial shape, multi-view
camera systems are applied [80, 19] or depth camera [57, 74, 79, 75, 55, 18],
however, these sensors suffer from limited spatial resolution. The FaceScape
dataset [71] contains textured 3D faces recorded using a dense camera array
under controlled lighting, which retrieves the 3D facial model preserving low-
level details such as small wrinkles and pores. The 3DFAW-Video dataset [36]
lacks subjects diversity, and is not really ”in-the-wild”; 300W-LP [53, 84] is
synthetic and focuses only on faces. In contrast to our dataset, none of the
datasets is diverse, accurate, dense, and in-the-wild at the same time.

3D Head Pose Estimation. Classical methods for head pose estimation are based
on traditional techniques such as cascade detectors [64] or template matching
[9]. Cascade detectors localize the head for each pose [37], while a template
matching approach compares query image with a set of pre-labeled templates and
finds a corresponding pose [44, 58]. Geometric methods use facial landmarks
retrieved from the input image and estimate the head pose empirically [13,
31]. Regression and classification methods include wide-ranging methods that
fit a mathematical model to predict the head pose from labeled training data
or discretized set of poses[59, 88, 43, 52, 70, 3]. _Multi-task_ approach
combines a head pose estimation learning with other facial analysis tasks,
such as Face Detection [50, 49, 86], Face Recognition [50], Landmark
Localization [50, 49, 86], Alignment [17, 78, 50]. Our approach is related to
the latter one, where the study of 3D Head Reconstruction is coupled with
learning parameters of a 3D head model and Landmark Localization.

3D Face Alignment. Early 3D Morphable Face Models (3DMM) [4, 47] were derived
from a small amount of registered 3D scans, e.g., Basel Face Model (BFM) [47]
has 200 human faces. More recent models such as FLAME [41] are learnt from a
significantly larger amount of scans, i.e., FLAME uses 3,800 3D scans of human
heads. Nevertheless, the diversity of the scans is limited.

RingNet[56] is trained to estimate the 3D face shape from a single image
without direct 3D supervision to overcome this limitation. In contrast, we
train our model to perform 3D head reconstruction from an image directly by
the 2D-3D supervision as provided in our dataset. Similarly motivated is 3DDFA
[85], a Cascaded CNN model which directly predicts a dense 3DMM from the
facial image. This approach has been further extended and optimized in [30]
with meta-joint optimization to facilitate parameters regression. Another
approach called DECA [26] is trained to regress a parameterized face model. A
recent FAN model [12] has been constricted by stacking four Hourglass models
[45, 72] in which all bottleneck blocks were replaced with the hierarchical,
multi-scale and parallel binary residual blocks [11]. Instead of using
Landmark Localization, in [16] the authors propose to align human faces
directly from an image using 6 degrees of freedom (6DoF 3D) – rotations and
translations along $X$, $Y$, $Z$ axis. [33] introduces a model with a
lightweight attention mechanism for Face Alignment. In contrast, we collect a
large-scale, diverse dataset with annotations directly in 3D and correspond
with FLAME topology. This enables efficient training of the DAD-3DNet for a
range of 3D head tasks.

##  3 DAD-3DHeads Dataset

To create a large-scale dataset of in-the-wild images, we repurpose a modern
3D modeling tool and introduce a novel annotation scheme that addresses the
problems exhibited by existing labeling tools, such as ”guessing” the
positions of the correct landmarks for invisible parts of the head, thus
enabling accurate annotations for any head images. In this section we verify
that obtained annotations are accurate compared to the GT 3D scans, and of
high quality, i.e., reducing annotator’s errors by half.

###  3.1 Data acquisition

Figure 2: A labeling tool example. The annotator fits the 3D Head model to the
image by anchoring pinpoints. The corresponding 3D textured render is
available to ensure the visual plausibility of the head shape.

We fit a 3D Morphable Model of a human head to a given photo with a simple
interface. The annotators do not explicitly control or label either the 3DMM
parameters or the blendshapes. The fitting is conditioned upon the visible
part of the head and the prior FLAME model [41]. The annotators ”pin” the
points on the 3D mesh surface (see Fig. 2, left) to the specific pixels of the
image. The mesh then undergoes the optimization of the 3DMM parameters, so
that the ”pin” reprojection error is minimized. During the labeling process,
labelers can see the texture rendered onto the 3D mesh with respect to their
fitting to verify that the results are visually plausible (Fig. 2, right). We
use the 2D reprojection of the 3D mesh onto the image to ensure that the
boundaries of the facial features and the skull are correct, and the relative
depth information to confirm that the image provides realistic texture mapping
onto the human head model. The details of the annotation procedure along with
the visuals - images of the intermediate steps and the full video example -
are provided in Sec. A.3. In total we receive 5,023 dense landmarks consistent
with the FLAME topology, namely, FLAME mesh vertices.

###  3.2 Dataset Statistics

DAD-3DHeads dataset consists of 44,898 images collected from various sources
(37,840 in the training set, 4,312 in the validation set, and 2,746 in the
test set). For each image, we provide 5,023 vertices of the FLAME mesh, 3,669
of which are accurately labeled (we demonstrate it in Sec. 3.3), neck and
eyeballs excluded. We refer to this subset of 3,669 landmarks as ”head” (see
Fig. 12). We also provide the model-view and frustum projection matrices that
map the 3D mesh from model space onto the image for different training
scenarios and evaluation purposes. In addition, we release rich attribute
information, showing the variability and unbiasedness of the data. DAD-3DHeads
attributes include head poses, presence of emotions, occlusions (see Fig. 3),
as well as gender, age group, image quality, and illumination labels. The
detailed dataset card can be found in Sec. A.1.

Figure 3: Dataset Properties: DAD-3DHeads is well balanced over a wide range
of poses, face expressions, and occlusions. The attribute labels are a
valuable signal for subgroup analysis and for generalization to in-the-wild
deployment conditions.

###  3.3 Annotation accuracy

To check the accuracy of our annotations, we calculate the accuracy of the
head shape reconstruction and head pose estimation compared to ground-truth 3D
scans.

Figure 4: DAD-3DHeads accuracy on selected samples from the NoW dataset. (a)
GT scan; (b) input image; (c) the result of our annotation; (d) alignment of
the mesh (wireframe) and the GT scan (with color-coded errors overlayed). The
scale of the errors relates to the real-world size of the scans. Note that the
resulting meshes accurately capture the coarse shape of the frontal part of
the head, the regions of higher error heavily overlap with finer facial
structures. We provide more examples, in high resolution, in Fig. 16
visualising this phenomenon. Best viewed zoomed in and in color.

3D Head Shape Reconstruction. To validate that DAD-3DHeads annotations fit the
head shape correctly, we compare the 3D meshes to the ground-truth scans
provided in NoW [56] and Stirling [60] datasets, following the correspondent
evaluation protocols (see Sec. 5.3).

As both benchmarks provide scans only of the frontal part of the face, the
reconstruction of the whole skull can not be validated by those methods - that
is where we resort to visual verification by our labelers as shown in Fig. 2
(right).

We explicitly validate the accuracy on neutral images only since the 3D scans
do not capture emotions, see the quantitative results in Tab. 1(a), Tab. 1(b).
For visual comparison, see Fig. 4. Note that the representation is coarse
(same as FLAME topology [41]), and we do not aim to model wrinkles and other
tiny details that scanners can capture.

(a) NoW[56] Dataset, ”multiview_neutral” subset.

Model Median(mm) Mean(mm) Std(mm) 3DDFA-V2 [29, 30] 1.360 1.762 1.621 RingNet
[56] 1.316 1.659 1.392 DAD-3DHeads 1.109 1.386 1.166

(b) Stirling[60] Database, ”Neutral expression, four views” subset.

Model 3DRMSE(mm) Median(mm) Mean(mm) Std(mm) RingNet [56] 2.793 1.633 2.112
1.828 3DDFA-V2 [29, 30] 2.550 1.508 1.927 1.670 DAD-3DHeads 2.488 1.447 1.873
1.638

Table 1: DAD-3DHeads accuracy of 3D Face Shape Reconstruction on NoW and
Stirling DBs; SOTA methods as reference.

3D Head Pose Estimation. To validate the goodness-of-fit of the head pose, we
compare the rotation matrices from our annotations to the ground-truth
matrices from the BIWI dataset [25]. They are captured by Kinect v2 sensors,
the measurement error of which is 20mm [46].

To compare the matrices $R_{1}$ and $R_{2}$, we calculate the difference
rotation $R_{1}R_{2}^{T}$, and measure (i) Frobenius norm of the matrix
$I-R_{1}R_{2}^{T}$, as in [32], and (ii) the angle in axis-angle
representation of $R_{1}R_{2}^{T}$, see Tab. 2.

###  3.4 Annotation quality

To verify the quality of our annotations, we have selected a subset of $N=30$
images from different categories in the dataset. Each image was manually
labeled with 68 facial landmarks, in the traditional configuration of [28], by
$m=10$ different annotators. The same pictures were labeled following our
annotation scheme (Sec. 3.1) in the 3D labeling tool. The 68 reprojected
landmarks were computed from the 3D annotations to be comparable with the
manual 2D-point labels. We compute the quality score $\mathcal{F}_{Q}$ for
each approach (see Table 3), averaging across the images, as a normalized mean
error between each pair of labels:

$\mathcal{F}_{Q}=\frac{1}{N}\sum_{n=1}^{N}\frac{1}{d_{n}}\cdot\frac{2}{m(m-1)}\sum_{i=1}^{m}\sum_{j>i}\Big{|}\Big{|}\overrightarrow{x_{n}^{i}}-\overrightarrow{x_{n}^{j}}\Big{|}\Big{|}_{2},$
(1)

where $d_{n}$ is the head bounding box size, as used in [76, 34],
$\overrightarrow{x}$ is an array of 68 labeled landmarks. As our data is
mainly non-frontal, we do not use eye landmark distances as a normalization
factor.

Method | $||I-R_{1}R_{2}^{T}||_{F}$ | Angle error (degrees)
---|---|---
Img2Pose [1] | 0.228 | 9.336
DAD-3DHeads | 0.149 | 6.037
Table 2: DAD-3DHeads accuracy of 3D Head Pose estimation on BIWI[25]; SOTA method as reference. The measure of $R_{1}R_{2}^{T}$ deviation from identity matrix lies in the (0, $2\sqrt{2}$) range [32]. Figure 5: Annotation Consistency. Images labeled with our 3D annotation scheme (left) and with 68 2D points (right). Different colors correspond to different labelers. The annotators are consistent due to the conditioning by a 3D head model prior, which ensures high quality of the DAD-3DHeads dataset even under extremely diverse conditions. Labeling of invisible landmarks is highly inconsistent with the traditional approach, while using the 3D mesh fitting ensures high consistency even on occluded parts.  Method |  $\mathcal{F}_{Q}$ (avg NME) | Best sample NME
---|---|---
2D 68 keypoints | 3.210 | 2.326
DAD-3DHeads 68 landm. |  1.737 ($\downarrow 45.8\%$) | 1.302
Table 3: Quality score. Annotation in 3D reduces the global average NME by
45.8%, see e.g. Fig. 5.

Limitations. Such labelling scheme provides only partial control over depth.
To mitigate this issue, we (i) provide the annotators with the ability to see
the rendered texture onto the mesh in 3D, so they can inspect visually whether
the lack of depth information corrupted the skull shape, and if the image
provides realistic texture; (ii) propose $Z_{n}$ metric (see Sec. 5.1) that
assesses the depth quality.

##  4 Method

Figure 6: DAD-3DNet architecture design and model training benefit from the
rich annotations in a multi-branch setup. The Gaussian heatmap estimator
predicts coarse locations of the head landmarks. The fusion block combines the
coarse heatmap, BiFPN feature map, and CNN encoder output to regress a set of
3D head model parameters and finer locations of head landmarks.

Our goal is to estimate a compact 3D Head representation from a single image.
Given an image, we assume the head is detected, loosely cropped, and
approximately centered. We introduce a novel architecture, DAD-3DNet, that
predicts a vector of 3DMM parameters disentangled into shape, expression and
pose, and a dense set of 2D landmarks. The landmarks serve as additional
supervision and regularization and extend the range of applications that could
benefit from the DAD-3DNet model. The DAD-3DNet architecture is illustrated in
Fig. 6.

###  4.1 DAD-3DNet Architecture

Our architecture consists of (i) a CNN Encoder to extract features from the
image, (ii) a Landmark Heatmap Estimator based on the BiFPN [62] to predict
coarse locations of 2D landmarks, (iii) a Fusion Module that fuses the heatmap
predictions with the encoder features, and (iv) a Regression Module that
predicts finer facial landmarks locations and 3DMM parameters. We also use (v)
a differential FLAME Layer that maps the 3DMM vector to the 3D mesh vertices.

A pre-trained CNN Encoder extracts features from the first four stages of a
backbone network. The Landmark Heatmap Estimator takes second to fourth stage
feature maps as an input and predicts coarse Gaussian heatmaps using BiFPN,
allowing easy and fast multi-scale feature fusion. The Gaussian heatmaps with
$1/4$ of the original spatial resolution are then interpolated to the size of
the fourth stage feature maps. The Fusion Layer incorporates the interpolated
Gaussian heatmaps, the original feature map, and the BiFPN feature maps to
encode a multi-scale representation with an Inception Module. A linear layer
follows encoder representation to extract 2D landmarks locations.

###  4.2 Objective Function

We introduce a multi-component loss function for the end-to-end training of
DAD-3DNet to provide supervision for different branches of the network. The
loss function consists of four different parts: Shape+Expression Loss
measuring goodness-of-fit of the 3D Head Shape ($L_{3D}$), Reprojection Loss
($L_{proj}$) that incorporates pose information, Landmark Regression ($L_{1}$)
and Gaussian Heatmap Loss ($L_{AWing}$ [65]) to provide the supervision for
the 2D Facial Landmarks prediction branch. The detailed ablation studies (Sec.
5.4) show the importance of each component.

Shape+Expression Loss: Following the notations used in [56] , we denote the
3DMM coefficients as follows: shape coefficients
$\overrightarrow{\beta}\in\mathbb{R}^{|\overrightarrow{\beta}|}$, expression
coefficients $\overrightarrow{\psi}\in\mathbb{R}^{|\overrightarrow{\psi}|}$.
The global rotation pose is modeled by
$\overrightarrow{\theta_{r}}\in\mathbb{R}^{6}$ for continuity of
representation [82], and is separated from the jaw rotation pose vector
$\overrightarrow{\theta_{j}}\in\mathbb{R}^{3}$. In our approach we assume that
neck $\overrightarrow{\theta_{n}}\in\mathbb{R}^{3}$ and eyeballs
$\overrightarrow{\theta_{e}}\in\mathbb{R}^{6}$ rotation coefficients are equal
to zero. The global rotation predictions are set to zero to evaluate the
discrepancy between our predictions and the ground truth in 3D. The 3D
vertices are computed from the 3DMM parameters using a differentiable FLAME
layer. As FLAME model [41] contains both the head and the neck, but our task
is narrowed down to the head mesh estimation, we subsample the vertices vector
$\overrightarrow{v}=\overrightarrow{v}\big{(}\overrightarrow{\beta},\overrightarrow{\psi},\overrightarrow{\theta_{j}}\big{)}$
over the set of ”head” vertex indices $I$: $\overrightarrow{v}|_{I}$.

The ground truth and the predicted mesh can differ in scale and location, so
we normalize $\varphi$ both to fit into the unit cube after subsampling.

The final loss term measures discrepancy between normalized subsampled
vertices:

$L_{3D}\Big{(}\overrightarrow{\beta},\overrightarrow{\psi},\overrightarrow{\theta_{j}}\Big{)}=\Big{|}\varphi\Big{(}\overrightarrow{v_{pred}|_{I}}\Big{)}-\varphi\Big{(}\overrightarrow{v_{GT}|_{I}}\Big{)}\Big{|}_{2}.$
(2)

Reprojection Loss is computed by projecting the 3D vertices of the posed mesh
onto the image. The posed mesh is a ”zero-pose” mesh described above, to which
we apply the similarity transform (rotation $R(\overrightarrow{\theta_{r}})$,
uniform scaling $s$, and translation $\overrightarrow{t}$). The reprojection
then is a simple orthographic projection onto the image plane. Here, as well,
only the ”head” vertices are included in the loss computation. The $L_{1}$
criterion is used as a discrepancy measure between the reprojected subsampled
vertices.

The overall loss is a combination of the four terms:
$\text{\quad\quad\quad}L=\lambda_{1}L_{3D}+\lambda_{2}L_{1}+\lambda_{3}L_{proj}+\lambda_{4}L_{AWing}.$
We use 50.0, 1.0, 0.05, 1.0 as $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$,
$\lambda_{4}$ respectively.

###  4.3 Implementation details

We implemented all of our models using PyTorch. The backbone network is
initialized using the pre-trained weights on ImageNet. The differentiable
FLAME layer is kept fixed during the training. The number of learnable head
shape and expression parameters are set to 300 and 100, respectively. All the
models are trained using 1 RTX A6000 GPU, with a batch size of 256. We use an
ADAM optimizer with a learning rate = $1*10^{-4}$ and a plateau learning rate
reducer with a reduce factor = 0.5 every six epochs when the validation loss
stops decreasing. The training takes one day to converge. To preserve the
scale ratio and shape of the head, images are padded to the square size and
then resized to 256x256. We trained all models without any image
augmentations.

##  5 Experimental Evaluation

Table 4: Comparison with state-of-the-art 3D Dense Head Alignment models on
DAD-3DHeads Benchmark: We compute the metrics on full test dataset as well as
on challenging atypical poses (Pose), compound expressions (Expr.) and heavy
occlusions (Occl.) subsets. DAD-3DNet shows superior performance on all
subsets. Note: $Z_{n}$ is computed only for methods that use FLAME mesh
topology.

.  Pose NME$\downarrow$ Z5 Accuracy$\uparrow$ Chamfer Distance$\downarrow$
Pose Error$\downarrow$ Overall Pose Expr. Occl. Overall Pose Expr. Occl.
Overall Pose Expr. Occl. Overall Pose Expr. Occl. 3DDFA-V2 [29, 30] 3.580
7.630 3.168 3.195 - - - - 6.17 8.878 6.410 6.400 0.527 0.790 0.455 0.542
RingNet [56] 8.757 26.732 5.010 12.660 0.880 0.743 0.913 0.860 5.166 5.704
5.792 5.993 0.438 1.076 0.294 0.551 DAD-3DNet 2.302 6.049 1.748 2.036 0.954
0.916 0.958 0.943 3.178 4.094 3.375 3.774 0.138 0.343 0.112 0.203

We propose DAD-3DHeads Benchmark for evaluating (i) the task of 3D Dense Head
Alignment from an image, (ii) in-the-wild model generalization (when trained
on our data) to a range of 3D Head Learning tasks, and (iii) robustness to
extreme poses. To address (i), we provide a comprehensive analysis of
DAD-3DNet and several existing methods on our benchmark, and report the
findings in Tab. 4. To test generalization (ii), we analyze the performance of
DAD-3DNet on the established benchmarks for 3D Face Shape Reconstruction and
3D Head Pose Estimation, detailed in Sec. 5.2, Sec. 5.3. To test robustness
(iii), we evaluate DAD-3DNet under train/test distribution shift in camera
poses and report our findings in Sec. B.2 (see Tab. 9).

###  5.1 Metrics

Given a ground-truth mesh $M$ on a particular frame, and post-processed model
output - predicted 3D vertices $V$, we calculate how well $V$ fit $M$. The
goodness-of-fit measures the pose fitting, and both face and head shape
matching. We propose two new metrics for the evaluation protocol: Reprojection
NME and $Z_{n}$ accuracy, in addition to Chamfer Distance and Pose Error
reported previously for the 3D Head Learning tasks.

Reprojection NME: we compute the normalized mean error of the reprojected 3D
vertices onto the image plane, taking $X$ and $Y$ coordinates into account.
Similar to Eq. 1, we use head bounding box size for normalization. The metric
is computed on 68 landmarks [28].

Zn Accuracy: as our annotation scheme is conditioned only upon model prior and
the reprojection onto the image, we do not guarantee the absolute depth values
to be as accurate as sensor data. We address this issue by measuring the
_relative depth_ as an ordinal value of the $Z$-coordinate. For each of $K$
vertices $v_{i}$ of the GT mesh, we choose $n$ closest vertices
$\\{v_{i}^{1},...,v_{i}^{n}\\}$ , and calculate which of them are closer to
(or further from) the camera. Then, we compare if for every predicted vertex
$w_{i}$ this configuration is the same:
$\text{\quad}Z_{n}=\frac{1}{K}\frac{1}{n}\sum_{i=1}^{K}\sum_{j=1}^{n}\Big{(}(v_{i}\succeq_{z}v_{i}^{j})==(w_{i}\succeq_{z}w_{i}^{j})\Big{)}.$
We do so on the ”head” subset of the vertices only.

Chamfer Distance: as the $Z_{n}$ metric is valid only for predictions that
follow FLAME mesh topology, we add Chamfer distance to measure the accuracy of
fit. To ensure generalization to any number of predicted vertices, we measure
a one-sided Chamfer distance from our ground-truth mesh to the predicted one.
We align them by seven key points correspondences [56], and compute the
distances from the ”face” subset of the vertices only (see Fig. 12 in
Appendix), following the traditional approach [56, 42].

Pose Error: measuring the accuracy of pose prediction, we want to overcome the
issues observed in AFLW2000-3D[38] Dataset. Creators of AFLW2000-3D measure
the head pose resorting to Euler angles. Such representation is highly
dependent on the order in which the rotations are applied. Whenever the second
rotation reaches over $\frac{\pi}{2}$ in any direction, i.e., extreme head
poses, other rotation axes become linearly dependent, yielding an infinite
number of representations for the same transformation [15]. One can observe
inconsistencies caused by this in the AFLW2000-3D[38] benchmark in Fig. 7.

Figure 7: AFLW2000-3D label inconsistencies. Some labels of side or extreme
atypical poses are inconsistent as the Euler angle representation used is
ambiguous due to gimbal lock.

To avoid that, we measure accuracy of pose prediction based on rotation
matrices [32] (see Sec. 3.3):
$\text{\quad\quad\quad\quad}Error_{pose}=||I-R_{1}R_{2}^{T}||_{F}$

###  5.2 3D Head Pose Estimation

We evaluate DAD-3DNet on AFLW2000-3D and BIWI datasets for the task of 3D Head
Pose Estimation.

BIWI Dataset [25] is gathered in a laboratory setting by recording RGB-D video
of different subjects across various head poses using a Kinect v2 device. It
contains frames with the rotations ±75∘ for yaw, ±60∘ for pitch, and ±50∘ for
roll. A 3D model was fit to each individual’s point cloud, and the head
rotations were tracked to produce the pose annotations.

AFLW2000-3D Dataset [84] consists of the first 2,000 subjects of the in-the-
wild AFLW dataset, which has been re-annotated with image-level 68 3D
landmarks and consequently, contain fine-grained pose annotations.

Results: We report the results in Tab. 5(a), Tab. 5(b). The proposed model
outperforms all other 3DMM estimation methods by a large margin, and shows
comparable performance to other state-of-the-art methods for head pose
estimation.

Table 5: 3D head pose estimation results.

Model MAE ↓ Pitch MAE ↓ Roll MAE ↓ Yaw MAE ↓ 3DDFA [29, 85] 19.07 12.25 8.78
36.18 Fan (12 points)[16] 7.88 7.48 7.63 8.53 Dlib (68 points)[35] 12.25 13.80
6.19 16.76 HopeNet[21] 4.90 6.61 3.27 4.81 Img2Pose[1] 3.79 3.55 3.24 4.57
3DDFA-V2[29, 30] 8.81 12.08 7.54 6.80 RingNet[56] 7.34 5.37 7.82 8.82
WHENet[83] 3.81 4.39 3.06 3.99 DAD-3DNet 3.98 5.24 2.92 3.79

(a) BIWI[25]

Model MAE ↓ Pitch MAE ↓ Roll MAE ↓ Yaw MAE ↓ 3DDFA[29, 85] 7.39 8.53 7.39 5.40
Fan (12 points)[16] 9.12 12.28 8.71 6.36 Dlib (68 points)[35] 13.29 12.60 9.00
18.27 HopeNet[21] 6.16 6.56 5.44 6.47 RetinaNet[20] 6.22 9.64 3.92 5.10
Img2Pose[1] 3.91 5.03 3.28 3.43 SynergyNet[67] 3.35 4.09 2.55 3.42
3DDFA-V2[29, 30] 7.56 8.48 9.89 4.30 RingNet[56] 8.27 4.39 13.51 6.92
DAD-3DNet 3.66 4.76 3.15 3.08

(b) AFLW2000-3D[84]

###  5.3 3D Face Shape Reconstruction

For the task of 3D Face Shape reconstruction, we compare the performance of
DAD-3DNet with two state-of-the-art publicly available methods: 3DDFA-V2 [29,
30], and RingNet [56] on two 3D Face Shape reconstruction benchmarks: NoW [56]
and Feng et al. [42].

NoW Face Challenge: NoW benchmark is designed for the task of 3D face
reconstruction from single monocular images. The dataset contains 2054 2D
images of 100 subjects. Following the evaluation protocol, we predict the
meshes that are then rigidly aligned with corresponding ground truth scans
based on seven landmark points. The scan-to-mesh distance is computed between
them. The calculated mean, median, and standard deviation errors are reported
in Table 6(a).

Table 6: 3D face shape reconstruction results.

Model Median(mm) ↓ Mean(mm) ↓ Std(mm) ↓ 3DDFA-V2[29, 30] 1.234 1.566 1.391
RingNet[56] 1.207 1.535 1.306 DAD-3DNet 1.236 1.541 1.285

(a) NoW[56]

Model 3DRMSE↓ Median(mm)↓ Mean(mm)↓ Std(mm)↓ HQ LQ HQ LQ HQ LQ 3DDFA-V2[29,
30] 2.998 1.500 1.779 1.942 2.350 1.704 2.149 RingNet[56] 2.809 1.698 1.634
2.161 2.113 1.832 1.831 DAD-3DNet 2.749 1.558 1.624 1.940 2.082 1.581 1.795

(b) Feng et al.[42]

Feng et al. Benchmark: [42] provides a subset of Stirling/ESRC 3D face
database as the test dataset for their challenge. The test dataset consists of
2,000 2D various expression facial images, including 656 high-quality (HQ)
images taken in controlled scenarios and 1,344 low-quality (LQ) images
extracted from video frames [60]. Following [42] protocol that rehearses [56]
we perform the aforementioned steps and compute the scan-to-mesh distance
between the predicted meshes and the ground truth scans. These distances are
used to compute the 3DRMSE. We also compute mean, median and standard
deviation errors for HQ and LQ images separately for in-depth analysis. The
results of evaluation are provided in the Table 6(b).

Results: DAD-3DNet shows superior performance to other methods for coarse 3D
dense head alignment without using explicit Shape and Expression
disentanglement loss.

###  5.4 Ablation study

In this section, we verify the efficiency of the separate loss components and
demonstrate the impact of the training data. We report the results of an
ablation study in Table 7.

| Component | NME$\downarrow$ |  Z5 Acc.$\uparrow$ | Pose$\downarrow$
---|---|---|---|---
1 | baseline | 2.576 | 0.880 | 0.267
2 | \+ full face reprojection loss | 2.395 | 0.873 | 0.263
3 | \+ full head reprojection loss | 2.500 | 0.943 | 0.172
4 | \+ shape+expression loss | 2.471 | 0.951 | 0.139
5 | \+ landmark prediction head | 2.302 | 0.954 | 0.138
Table 7:  DAD-3DNet ablation study on DAD-3DHeads: The loss terms have
significant impact on the fitting accuracy, and the multi-head architecture
improves the model generalization.

Reprojection Loss: Supervision based on reprojected landmarks is a core part
of the training algorithms. Compared to the models that use supervision based
on 68 keypoints, we have only added the reprojection loss based on all
available face and head points. Incorporating information about other facial
landmarks improves the accuracy of reprojected 68 landmarks but does not
impact the other metrics and does not improve the 3D fitting; adding points of
the whole head improved all the metrics by a large margin. Additional full
head supervision improves the model stability by enforcing to learn the entire
head shape.

Shape+Expression Loss: Rich supervision of the normalized 3D vertices
locations enables the model to encode more nuanced information about the 3D
head pose. As shown in Table 7 this component improves all of the metrics and
reduces the 3D head pose error significantly.

Landmarks Head: Multi-task training improves the model stability and enforces
the model to prefer more general representations. With the landmarks
regression and coarse heatmap estimation modules, the model achieves a
significant boost in performance on all metrics yet again.

##  6 Conclusions

We introduce DAD-3DHeads, a dense, accurate, and diverse 3D Head dataset in
the wild. We demonstrate the efficiency and accuracy of the data and novel
loss components by training a data-driven DAD-3DNet model. DAD-3DNet achieves
superior performance on diverse 3D head tasks and successfully generalizes to
in-the-wild conditions.

Acknowledgements. We thank the Armed Forces of Ukraine for providing security
to complete this work.

## References

  * [1] Vitor Albiero, Xingyu Chen, Xi Yin, Guan Pang, and Tal Hassner.  img2pose: Face alignment and detection via 6dof, face pose estimation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7617–7627, 2021.
  * [2] Peter N. Belhumeur, David W. Jacobs, David J. Kriegman, and Neeraj Kumar.  Localizing parts of faces using a consensus of exemplars.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(12):2930–2940, 2013.
  * [3] Ben Benfold and Ian Reid.  Colour invariant head pose classification in low resolution video.  In British Machine Vision Conference (BMVC), pages 1–10. Citeseer, 2008.
  * [4] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3d faces.  In Conference on Computer graphics and interactive techniques, pages 187–194, 1999.
  * [5] James Booth, Epameinondas Antonakos, Stylianos Ploumpis, George Trigeorgis, Yannis Panagakis, and Stefanos Zafeiriou.  3d face morphable models ”in-the-wild”.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 48–57, 2017.
  * [6] James Booth, Anastasios Roussos, Allan Ponniah, David Dunaway, and Stefanos Zafeiriou.  Large scale 3d morphable models.  International Journal of Computer Vision, 126(2):233–254, 2018\.
  * [7] James Booth, Anastasios Roussos, Evangelos Ververas, Epameinondas Antonakos, Stylianos Ploumpis, Yannis Panagakis, and Stefanos Zafeiriou.  3d reconstruction of “in-the-wild” faces in images and videos.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(11):2638–2652, 2018.
  * [8] James Booth, Anastasios Roussos, Stefanos Zafeiriou, Allan Ponniah, and David Dunaway.  A 3d morphable model learnt from 10,000 faces.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 5543–5552, 2016.
  * [9] Roberto Brunelli.  Template matching techniques in computer vision: theory and practice.  John Wiley & Sons, 2009.
  * [10] Adrian Bulat and Georgios Tzimiropoulos.  Two-stage convolutional part heatmap regression for the 1st 3d face alignment in the wild (3dfaw) challenge.  In European Conference on Computer Vision (ECCV), pages 616–624. Springer, 2016.
  * [11] Adrian Bulat and Georgios Tzimiropoulos.  Binarized convolutional landmark localizers for human pose estimation and face alignment with limited resources.  In International Conference on Computer Vision (ICCV), Oct 2017\.
  * [12] Adrian Bulat and Georgios Tzimiropoulos.  How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks).  In International Conference on Computer Vision (ICCV), pages 1021–1030, 2017.
  * [13] Patrick Burger, Martin Rothbucher, and Klaus Diepold.  Self-initializing head pose estimation with a 2d monocular usb camera.  Technical report, Lehrstuhl für Datenverarbeitung, 2014.
  * [14] Xavier P Burgos-Artizzu, Pietro Perona, and Piotr Dollár.  Robust face landmark estimation under occlusion.  In International Conference on Computer Vision (ICCV), pages 1513–1520, 2013.
  * [15] Zhiwen Cao, Zongcheng Chu, Dongfang Liu, and Yingjie Chen.  A vector-based representation to enhance head pose estimation.  In IEEE Winter Conference on Applications of Computer Vision, pages 1188–1197, 2021.
  * [16] Feng-Ju Chang, Anh Tuan Tran, Tal Hassner, Iacopo Masi, Ram Nevatia, and Gerard Medioni.  Faceposenet: Making a case for landmark-free face alignment.  In International Conference on Computer Vision (ICCV) Workshops, Oct 2017.
  * [17] Dong Chen, Shaoqing Ren, Yichen Wei, Xudong Cao, and Jian Sun.  Joint cascade face detection and alignment.  In European Conference on Computer Vision (ECCV), pages 109–122. Springer, 2014.
  * [18] Shiyang Cheng, Irene Kotsia, Maja Pantic, and Stefanos Zafeiriou.  4dfab: A large scale 4d facial expression database for biometric applications.  arXiv preprint arXiv:1712.01443, 2017.
  * [19] Darren Cosker, Eva Krumhuber, and Adrian Hilton.  A facs valid 3d dynamic action unit database with applications to 3d dynamic morphable facial modeling.  In 2011 International Conference on Computer Vision, pages 2296–2303, 2011.
  * [20] Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou.  Retinaface: Single-shot multi-level face localisation in the wild.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [21] Bardia Doosti, Shujon Naha, Majid Mirbagheri, and David J. Crandall.  Hope-net: A graph-based model for hand-object pose estimation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [22] Shichuan Du, Yong Tao, and Aleix M Martinez.  Compound facial expressions of emotion.  Proceedings of the National Academy of Sciences, 111(15):E1454–E1462, 2014.
  * [23] Bernhard Egger, William AP Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, et al.  3d morphable face models—past, present, and future.  ACM Transactions on Graphics (TOG), 39(5):1–38, 2020.
  * [24] Eran Eidinger, Roee Enbar, and Tal Hassner.  Age and gender estimation of unfiltered faces.  IEEE Transactions on Information Forensics and Security, 9(12):2170–2179, 2014.
  * [25] Gabriele Fanelli, Thibaut Weise, Juergen Gall, and Luc Van Gool.  Real time head pose estimation from consumer depth cameras.  In Joint pattern recognition symposium, pages 101–110. Springer, 2011.
  * [26] Yao Feng, Haiwen Feng, Michael J Black, and Timo Bolkart.  Learning an animatable detailed 3d face model from in-the-wild images.  ACM Transactions on Graphics (TOG), 40(4):1–13, 2021.
  * [27] Golnaz Ghiasi and Charless C Fowlkes.  Occlusion coherence: Detecting and localizing occluded faces.  arXiv preprint arXiv:1506.08347, 2015.
  * [28] Ralph Gross, Iain Matthews, Jeffrey Cohn, Takeo Kanade, and Simon Baker.  Multi-pie.  Image and vision computing, 28(5):807–813, 2010.
  * [29] Jianzhu Guo, Xiangyu Zhu, and Zhen Lei.  3ddfa.  https://github.com/cleardusk/3DDFA, 2018.
  * [30] Jianzhu Guo, Xiangyu Zhu, Yang Yang, Fan Yang, Zhen Lei, and Stan Z Li.  Towards fast, accurate and stable 3d dense face alignment.  In European Conference on Computer Vision (ECCV), 2020.
  * [31] Rainer Herpers, Markus Michaelis, K-H Lichtenauer, and Gerald Sommer.  Edge and keypoint detection in facial regions.  In International Conference on Automatic Face and Gesture Recognition, pages 212–217. IEEE, 1996.
  * [32] Du Q Huynh.  Metrics for 3d rotations: Comparison and analysis.  Journal of Mathematical Imaging and Vision, 35(2):155–164, 2009\.
  * [33] Lei Jiang, Xiao-Jun Wu, and Josef Kittler.  Dual attention mobdensenet(damdnet) for robust 3d face alignment.  In International Conference on Computer Vision (ICCV) Workshops, Oct 2019.
  * [34] Amin Jourabloo and Xiaoming Liu.  Pose-invariant 3d face alignment.  In International Conference on Computer Vision (ICCV), pages 3694–3702, 2015.
  * [35] Davis E. King.  Dlib-ml: A machine learning toolkit.  Journal of Machine Learning Research, 10:1755–1758, 2009.
  * [36] Rohith Krishnan Pillai, Laszlo Attila Jeni, Huiyuan Yang, Zheng Zhang, Lijun Yin, and Jeffrey F. Cohn.  The 2nd 3d face alignment in the wild challenge (3dfaw-video): Dense reconstruction from video.  In International Conference on Computer Vision (ICCV) Workshops, Oct 2019.
  * [37] Jeffrey Ng Sing Kwong and Shaogang Gong.  Composite support vector machines for detection of faces across views and pose estimation.  Image Vis. Comput., 20:359–368, 2002.
  * [38] Martin Köstinger, Paul Wohlhart, Peter M. Roth, and Horst Bischof.  Annotated Facial Landmarks in the Wild: A Large-scale, Real-world Database for Facial Landmark Localization.  In IEEE International Workshop on Benchmarking Facial Image Analysis Technologies, 2011.
  * [39] Avisek Lahiri, Vivek Kwatra, Christian Frueh, John Lewis, and Chris Bregler.  Lipsync3d: Data-efficient learning of personalized 3d talking faces from video using pose and lighting normalization.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2755–2764, June 2021.
  * [40] Vuong Le, Jonathan Brandt, Zhe Lin, and Lubomir Bourdev.  Interactive facial feature localization.  10 2012.
  * [41] Tianye Li, Timo Bolkart, Michael J. Black, Hao Li, and Javier Romero.  Learning a model of facial shape and expression from 4d scans.  ACM Trans. Graph., 36(6), 2017.
  * [42] Feng Liu, Luan Tran, and Xiaoming Liu.  3d face modeling from diverse raw scan data.  In International Conference on Computer Vision (ICCV), Seoul, South Korea, October 2019.
  * [43] Sankha Mukherjee and Neil Robertson.  Deep head pose: Gaze-direction estimation in multimodal video.  IEEE Transactions on Multimedia, 17(11):2094–2107, 2015.
  * [44] Erik Murphy-Chutorian and Mohan Manubhai Trivedi.  Head pose estimation in computer vision: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 31(4):607–626, 2009.
  * [45] Alejandro Newell, Kaiyu Yang, and Jia Deng.  Stacked hourglass networks for human pose estimation.  In European Conference on Computer Vision (ECCV), pages 483–499. Springer, 2016.
  * [46] Diana Pagliari and Livio Pinto.  Calibration of kinect for xbox one and comparison between the two generations of microsoft sensors.  Sensors, 15(11):27569–27589, 2015.
  * [47] Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter.  A 3d face model for pose and illumination invariant face recognition.  In IEEE International Conference on Advanced Video and Signal Based Surveillance, pages 296–301, 2009.
  * [48] Stylianos Ploumpis, Haoyang Wang, Nick Pears, William A. P. Smith, and Stefanos Zafeiriou.  Combining 3d morphable models: A large scale face-and-head model.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [49] Rajeev Ranjan, Vishal M Patel, and Rama Chellappa.  Hyperface: A deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 41(1):121–135, 2017.
  * [50] Rajeev Ranjan, Swami Sankaranarayanan, Carlos D Castillo, and Rama Chellappa.  An all-in-one convolutional neural network for face analysis.  In IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), pages 17–24. IEEE, 2017.
  * [51] Zeyu Ruan, Changqing Zou, Longhai Wu, Gangshan Wu, and Limin Wang.  Sadrnet: Self-aligned dual face regression networks for robust 3d dense face alignment and reconstruction.  IEEE Transactions on Image Processing, 2021.
  * [52] Nataniel Ruiz, Eunji Chong, and James M Rehg.  Fine-grained head pose estimation without keypoints.  In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 2074–2083, 2018.
  * [53] Christos Sagonas, Epameinondas Antonakos, Georgios Tzimiropoulos, Stefanos Zafeiriou, and Maja Pantic.  300 faces in-the-wild challenge: Database and results.  Image and Vision Computing, 47:3–18, 2016.
  * [54] Christos Sagonas, Georgios Tzimiropoulos, Stefanos Zafeiriou, and Maja Pantic.  300 faces in-the-wild challenge: The first facial landmark localization challenge.  In International Conference on Computer Vision (ICCV) Workshops, pages 397–403. IEEE, 2013.
  * [55] Wojciech Sankowski, Piotr Stefan Nowak, and Paweł Krotewicz.  Multimodal biometric database dmcsv1 of 3d face and hand scans.  In International Conference Mixed Design of Integrated Circuits & Systems (MIXDES), pages 93–97. IEEE, 2015.
  * [56] Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and Michael J Black.  Learning to regress 3d face shape and expression from an image without 3d supervision.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7763–7772, 2019.
  * [57] Arman Savran, Neşe Alyüz, Hamdi Dibeklioğlu, Oya Çeliktutan, Berk Gökberk, Bülent Sankur, and Lale Akarun.  Bosphorus database for 3d face analysis.  In European Workshop on Biometrics and Identity Management, pages 47–56. Springer, 2008.
  * [58] Jamie Sherrah, Shaogang Gong, and Eng-Jon Ong.  Understanding pose discrimination in similarity space.  In British Machine Vision Conference (BMVC), pages 1–10. Citeseer, 1999.
  * [59] Sujith Srinivasan and Kim L Boyer.  Head pose estimation using view based eigenspaces.  In Object recognition supported by user interaction for service robots, volume 4, pages 302–305. IEEE, 2002.
  * [60] Stirling ESRC 3D Face database.  http://pics.stir.ac.uk/ESRC/.
  * [61] Yi Sun, Xiaogang Wang, and Xiaoou Tang.  Deep convolutional network cascade for facial point detection.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2013.
  * [62] Mingxing Tan, Ruoming Pang, and Quoc V Le.  Efficientdet: Scalable and efficient object detection.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 10781–10790, 2020.
  * [63] Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner.  Neural voice puppetry: Audio-driven facial reenactment.  In European Conference on Computer Vision (ECCV), pages 716–731. Springer, 2020.
  * [64] Paul Viola and Michael Jones.  Rapid object detection using a boosted cascade of simple features.  In IEEE Computer Vision and Pattern Recognition (CVPR). IEEE, 2001\.
  * [65] Xinyao Wang, Liefeng Bo, and Li Fuxin.  Adaptive wing loss for robust face alignment via heatmap regression.  In International Conference on Computer Vision (ICCV), pages 6971–6981, 2019.
  * [66] Erroll Wood, Tadas Baltrušaitis, Charlie Hewitt, Sebastian Dziadzio, Thomas J. Cashman, and Jamie Shotton.  Fake it till you make it: Face analysis in the wild using synthetic data alone.  In International Conference on Computer Vision (ICCV), pages 3681–3691, October 2021.
  * [67] Cho-Ying Wu, Qiangeng Xu, and Ulrich Neumann.  Synergy between 3dmm and 3d landmarks for accurate 3d facial geometry.  arXiv preprint arXiv:2110.09772, 2021.
  * [68] Wayne Wu, Chen Qian, Shuo Yang, Quan Wang, Yici Cai, and Qiang Zhou.  Look at boundary: A boundary-aware face alignment algorithm.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2129–2138, 2018.
  * [69] Wayne Wu, Chen Qian, Shuo Yang, Quan Wang, Yici Cai, and Qiang Zhou.  Look at boundary: A boundary-aware face alignment algorithm.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [70] Yan Yan, Elisa Ricci, Ramanathan Subramanian, Oswald Lanz, and Nicu Sebe.  No matter where you are: Flexible graph-guided multi-task learning for multi-view head pose classification under target motion.  In International Conference on Computer Vision (ICCV), pages 1177–1184, 2013.
  * [71] Haotian Yang, Hao Zhu, Yanru Wang, Mingkai Huang, Qiu Shen, Ruigang Yang, and Xun Cao.  Facescape: a large-scale high quality 3d face dataset and detailed riggable 3d face prediction.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 601–610, 2020.
  * [72] Jing Yang, Qingshan Liu, and Kaihua Zhang.  Stacked hourglass network for robust facial landmark localisation.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 2025–2033, 2017.
  * [73] Shuo Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang.  Wider face: A face detection benchmark.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [74] Baocai Yin, Yanfeng Sun, Chengzhang Wang, and Yun Ge.  Bjut-3d large scale 3d face database and information processing.  Journal of Computer Research and Development, 46(6):1009, 2009.
  * [75] Lijun Yin, Xiaozhou Wei, Yi Sun, Jun Wang, and Matthew J Rosato.  A 3d facial expression database for facial behavior research.  In International Conference on Automatic Face and Gesture Recognition (FGR06), pages 211–216. IEEE, 2006.
  * [76] Xiang Yu, Junzhou Huang, Shaoting Zhang, Wang Yan, and Dimitris N Metaxas.  Pose-free facial landmark fitting via optimized part mixtures and cascaded deformable shape model.  In International Conference on Computer Vision (ICCV), pages 1944–1951, 2013.
  * [77] Hongwen Zhang, Qi Li, and Zhenan Sun.  Joint voxel and coordinate regression for accurate 3d facial landmark localization.  In 2018 24th International Conference on Pattern Recognition (ICPR), pages 2202–2208. IEEE, 2018.
  * [78] Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao.  Joint face detection and alignment using multitask cascaded convolutional networks.  IEEE Signal Processing Letters, 23(10):1499–1503, 2016.
  * [79] Xing Zhang, Lijun Yin, Jeffrey F Cohn, Shaun Canavan, Michael Reale, Andy Horowitz, and Peng Liu.  A high-resolution spontaneous 3d dynamic facial expression database.  In IEEE International Conference and workshops on Automatic Face and Gesture Recognition (FG), pages 1–6. IEEE, 2013.
  * [80] Xing Zhang, Lijun Yin, Jeffrey F Cohn, Shaun Canavan, Michael Reale, Andy Horowitz, Peng Liu, and Jeffrey M Girard.  Bp4d-spontaneous: a high-resolution spontaneous 3d dynamic facial expression database.  Image and Vision Computing, 32(10):692–706, 2014.
  * [81] Guoyan Zheng, Shuo Li, and Gabor Szekely.  Statistical shape and deformation analysis: methods, implementation and applications.  Academic Press, 2017.
  * [82] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li.  On the continuity of rotation representations in neural networks.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 5745–5753, 2019.
  * [83] Yijun Zhou and James Gregson.  WHENet: Real-time fine-grained estimation for wide range head pose.  arXiv preprint arXiv:2005.10353, 2020.
  * [84] Xiangyu Zhu, Zhen Lei, Xiaoming Liu, Hailin Shi, and Stan Z Li.  Face alignment across large poses: A 3d solution.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 146–155, 2016.
  * [85] Xiangyu Zhu, Xiaoming Liu, Zhen Lei, and Stan Z Li.  Face alignment in full pose range: A 3d total solution.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(1):78–92, 2017.
  * [86] Xiangxin Zhu and Deva Ramanan.  Face detection, pose estimation, and landmark localization in the wild.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2879–2886. IEEE, 2012.
  * [87] Xiangxin Zhu and Deva Ramanan.  Face detection, pose estimation, and landmark localization in the wild.  In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 2879–2886, 2012.
  * [88] Youding Zhu and Kikuo Fujimura.  Head pose estimation for driver monitoring.  In IEEE Intelligent Vehicles Symposium, 2004, pages 501–506. IEEE, 2004.

##  Appendix A DAD-3DHeads Dataset

The images in DAD-3DHeads dataset are anonymised without additional metadata.
The results of labeling in the form of 3D head model do not contain any
private or sensitive information. The data gathered is not being used for
identification purposes or in connection with any other personal data.

###  A.1 Dataset card

As stated in Sec. 3.2, along with the dataset of images and annotations, we
provide additional information per image such as gender, age, illumination
conditions, image quality, pose, presense of expression, and occlusions (see
Fig. 14 and Fig. 3).

We use multiple sources to construct our dataset, among which WIDER FACE
dataset [73], the Adience dataset [24], Compound Facial Expressions of
Emotions Dataset[22], WFLW[69], AFW[87], Helen[40], LFPW[2].

###  A.2 DAD-3DHeads annotations accuracy

We provide more visualizations of the results of DAD-3DHeads annotations
compared to GT scans on neutral faces from NoW dataset[56] in Fig. 16. This is
in addition to (and in the context of) Fig. 4.

###  A.3 Annotation process

We provide here more details along with visuals to prevent possible
misunderstandings around the labeling process. The full video is available via
the project webpage.

As was already mentioned in the Sec. 3.1, the annotators do not explicitly
control or label either the 3DMM parameters or the blendshapes. They also do
not have to disambiguate between identity and expression. The annotators only
have to ”pin” a 3D head model - a mesh - to the head image. They do so
iteratively having the initial generic mesh optimized after every ”pin” being
placed (see Fig. 8). The nonlinear optimization is performed, indeed, over the
shape and expression parameters, along with the pose.

Figure 8: The mesh is being deformed due to the under-the-hood optimization
after the ”pin” is placed on the ear. Figure 9: The rendered texture here has
”holes” due to the occlusions, but can also be ”torn” if the ”pins” are placed
poorly.

After each step, the annotators can inspect if the fitted mesh aligns well
with the image in several ways: they might view (i) the reprojected set of
selected landmarks that correspond to the recognizible features on any face or
head (see Fig. 13 as an example), such as contours of the eyes (eyelash line),
lips, etc.; (ii) the image rendered onto the mesh as a texture given the
fitting - it helps to inspect if there appeared texture ”holes” due to poor
labeling or occlusions (see Fig. 9); (iii) the mesh itself is visible in
$360^{o}$ to inspect if the skull shape is not deformed (see Fig. 9, right).
These measures help to partially overcome limitations introduced by the
absence of camera parameters for the input images, thus possible ambiguities
caused by the effects of perspective projection.

Another important issue is that our annotations consist only of the 3D
vertices and transformation matrices, we do not interpret the resulting mesh
w.r.t. identity, expression, or any other ambiguous and ill-defined over a
single-image input feature. The concerns arise as there might be cases, e.g.,
faces with extreme expressions, where it is impossible to perfectly detect the
shape of the head with neutral expression based on a single image. Indeed, the
3D scanners would provide an accurate 3D head model that the manual annotation
cannot guarantee. However, they operate under controlled capture, i.e., _3D
scans have been in-the-lab instead of in-the-wild_. We provide the community
with the complementary data. It has a _known_ trade-off in accuracy, but the
performance is sufficient for many applications that operate in-the-wild.

##  Appendix B Experimental results on DAD-3DHeads benchmark

###  B.1 3D Landmark localization

We evaluate the state-of-the-art methods such as JVCR[77], FaceSynthetics[66],
3DDFA-v2[30], and proposed DAD-3DNet on DAD-3DHeads benchmark for the task of
3D Landmark Localization. We report normalized mean error (NME) for the
predicted landmarks reprojected onto the image plane (see Tab. 8). We analyse
the NME metric on full test dataset as well as across challenging subgroups
(atypical poses, compound expressions, heavy occlusions). _DAD-3DNet shows
superior performance in all cases._

When computing NME of the competitor methods, we only use the images where the
landmarks have been localized, otherwise the NME is ill-defined. See examples
where the landmarks are not localized in Fig. 17, along with other challenging
cases.

Performance on the ”Expr.” subset is better than the overall average (Table 8)
for all of the methods. We attribute this to the fact that for heavily
occluded faces and large extreme poses, where the landmarks are not clearly
visible (and therefore the emotions), the images are labeled as ”neutral” by
default.

Model | NME$\downarrow$
---|---
Overall | Pose | Expr. | Occl.
3DDFA-V2 [30] | 3.580 | 7.630 | 3.168 | 3.195
FaceSynthetics [66] | 4.363 | 15.781 | 3.159 | 4.934
JVCR [77] | 4.455 | 12.514 | 3.843 | 4.949
DAD-3DNet | 2.302 | 6.049 | 1.748 | 2.036

Table 8: 3D Landmark Localization on DAD-3DHeads benchmark. We compute the
normalized mean error (NME, the lower the better) on full test dataset as well
as on challenging atypical poses (Pose), compound expressions (Expr.) and
heavy occlusions (Occl.) subsets. DAD-3DNet performs superior in all cases.

.

###  B.2 DAD-3DNet performance across multiple subgroups

We present an extension of Tab. 4 from the main paper, performing in-depth
analysis of the results across multiple subgroups. We define subgroups w.r.t.
_pose_ (front, side, atypical), _age_ (child, young, middle age, senior),
_image quality_ (high, low), _occlusions_ (true, false), _expressions_
(neutral, non-neutral), _lightning_ (standard, non-standard). The results of
the DAD-3DNet model are reported in Tab. 9. This analysis shows robustness of
the proposed approach across various conditions (distribution shifts) in-the-
wild.

Pose | NME |  Z5 Accuracy | Chamfer Dist. | Pose Error
---|---|---|---|---
front | 1.496 | 0.965 | 3.146 | 0.089
side | 2.257 | 0.952 | 3.180 | 0.143
atypical | 6.190 | 0.916 | 4.027 | 0.343

Age | NME |  Z5 Accuracy | Chamfer Dist. | Pose Error
---|---|---|---|---
child | 1.662 | 0.960 | 3.546 | 0.103
young | 2.421 | 0.953 | 3.178 | 0.150
middle | 2.438 | 0.955 | 3.393 | 0.133
senior | 1.756 | 0.958 | 2.989 | 0.113

Quality | NME |  Z5 Accuracy | Chamfer Dist. | Pose Error
---|---|---|---|---
high | 2.065 | 0.957 | 3.194 | 0.129
low | 4.755 | 0.928 | 3.643 | 0.259

Occlusion | NME |  Z5 Accuracy | Chamfer Dist. | Pose Error
---|---|---|---|---
True | 4.242 | 0.9436 | 3.784 | 0.1986
False | 2.134 | 0.955 | 3.182 | 0.1359

Expression | NME |  Z5 Accuracy | Chamfer Dist. | Pose Error
---|---|---|---|---
Non-neutral | 1.156 | 0.959 | 3.412 | 0.116
Neutral | 1.644 | 0.950 | 3.088 | 0.164

Lighting | NME |  Z5 Accuracy | Chamfer Dist. | Pose Error
---|---|---|---|---
Standard | 2.350 | 0.954 | 3.182 | 0.142
Non-standard | 2.235 | 0.954 | 3.569 | 0.142

Table 9: In-depth analysis of the benchmark results reported in Table 4 of the
main paper across multiple subgroups such as _camera pose_ , _age_ , _image
quality_ , _occlusions_ , _expressions_ , _lighting_. This analysis shows
robustness of the proposed approach across various conditions (distribution
shifts) in-the-wild.

###  B.3 3D Head Pose Estimation

The comparison of DAD-3DNet with the state-of-the-art 3D Head Pose Estimation
method img2pose [1] is provided in Table 10. We calculate two metrics on the
rotation matrices $R_{1}$ (GT) and $R_{2}$ (prediction), similar to Table 2:
(i) Frobenius norm of the matrix $I-R_{1}R_{2}^{T}$, and (ii) the angle in
axis-angle representation of $R_{1}R_{2}^{T}$.

Method | $||I-R_{1}R_{2}^{T}||_{F}$ | Angle error (degrees)
---|---|---
Img2Pose [1] | 0.226 | 9.122
DAD-3DNet | 0.138 | 5.360
Table 10: 3D Head Pose estimation on DAD-3DHeads benchmark. DAD-3DNet
outperforms state-of-the-art img2pose[1]. The measure of $R_{1}R_{2}^{T}$
deviation from identity matrix lies in the (0, $2\sqrt{2}$) range [32].

###  B.4 Failure cases

DAD-3DNet still sometimes fails on some cases of severe occlusions, extremely
low quality, or very atypical poses (like faces upside-down) (see Fig. 10).
Our current model was trained with no augmentations, this creates a suitable
venue for future explorations.

Figure 10: Failure cases of DAD-3DNet on DAD-3DHeads benchmark, 3D Landmark
Localization.

##  Appendix C Miscellaneous

Zn Metrics. We measure performance of DAD-3DNet via $Z_{n}$ for different
values of $n$, see the results in Fig. 11. The accuracy does not change
dramatically with $n$, so for DAD-3DHeads benchmark we use $n=5$ as a trade-
off between computational complexity and robustness.

Figure 11: $Z_{N}$ accuracy of DAD-3DNet for different values of $n$. $x$-axis
is in log-scale.

”Head” and ”face” mesh vertices. In the paper, we refer to two subsets of the
FLAME mesh vertices: ”head” and ”face”. The former ones are used in the loss
terms calculation (see Shape+Expression loss and Reprojection loss in Sec.
4.2) and in the $Z_{n}$ accuracy measurements for DAD-3DHeads benchmark (see
Sec. 5.1), and the latter ones - in the Chamfer distance measurements for
DAD-3DHeads benchmark (see Sec. 5.1). We provide visual examples of what these
subsets represent on meshes with various head shape and face expressions in
Fig. 12.

Figure 12: ”Face” and ”head” subsets of FLAME mesh vertices. Upper row:
”face”, capture frontal part of the head without ears. Lower row: ”head”,
capture the head without neck. The eyeballs are excluded in both.

Various number of landmarks. As DAD-3DHeads dataset is dense, it allows for
training different models, localizing many more than the usual 68
landmarks[28]. This flexibility saves the human annotator efforts, because the
data should not be relabeled every time a different setup is needed.
Moreoever, DAD-3DNet training pipeline allows for inference on any subset of
head vertices, given its 3DMM prediction branch, as they can be subsampled
after the entire mesh is predicted (see examples of different landmark subsets
in Fig. 13).

Figure 13: DAD-3DHeads dataset allows for flexibly choosing the desired
landmark subset for predicting as many dense landmarks as needed. Left to
right: 68 landmarks [28], 191 landmarks, 445 landmarks. Figure 14: Attribute
labels (gender, age, illumination, and image quality) and the distribution
across ethnic groups in DAD-3DHeads.

Figure 15: DAD-3DHeads dataset, more visual examples. The source images cover
large variation in poses [2, 4, 9, 11, 14, 16, 18-20], expressions [3, 6, 7,
8, 15], occlusions [1-3, 8, 19], non-standard illumination conditions [6, 13,
14], low image quality [2, 4, 13].

Figure 16: DAD-3DHeads accuracy on selected samples from the NoW dataset.
First row: input image; second row: GT scan; third row: the result of our
annotation; fourth row: alignment of the mesh (wireframe) and the GT scan
(with color-coded errors overlayed).

Figure 17: Qualitative comparison of DAD-3DNet and state-of-the-art methods on
challenging cases from DAD-3DHeads benchmark. Left to right: 3DDFA-v2[30],
FaceSynthetics[66], JVCR [77], DAD-3DNet (ours), ground truth.

Figure 18: Qualitative comparison of DAD-3DNet against state-of-the-art
methods on challenging cases from DAD-3DHeads benchmark (cont.) Left to right:
3DDFA-v2[30], FaceSynthetics[66], JVCR[77], DAD-3DNet (ours), ground truth.

Figure 19: Qualitative comparison of DAD-3DNet against state-of-the-art
methods on challenging cases from DAD-3DHeads benchmark (cont.) Left to right:
3DDFA-v2[30], FaceSynthetics[66], JVCR[77], DAD-3DNet (ours), ground truth.