# Deep Learning-Based Quasi-Conformal Surface Registration for Partial 3D
Faces Applied to Facial Recognition

Yuchen Guo
Department of Mathematics
The Chinese University of Hong Kong
&Hanqun Cao
Department of Mathematics
The Chinese University of Hong Kong
&Lok Ming Lui
Department of Mathematics
The Chinese University of Hong Kong

###### Abstract

3D face registration is an important process in which a 3D face model is
aligned and mapped to a template face. However, the task of 3D face
registration becomes particularly challenging when dealing with partial face
data, where only limited facial information is available. To address this
challenge, this paper presents a novel deep learning-based approach that
combines quasi-conformal geometry with deep neural networks for partial face
registration. The proposed framework begins with a Landmark Detection Network
that utilizes curvature information to detect the presence of facial features
and estimate their corresponding coordinates. These facial landmark features
serve as essential guidance for the registration process. To establish a dense
correspondence between the partial face and the template surface, a
registration network based on quasiconformal theories is employed. The
registration network establishes a bijective quasiconformal surface mapping
aligning corresponding partial faces based on detected landmarks and curvature
values. It consists of the Coefficients Prediction Network, which outputs the
optimal Beltrami coefficient representing the surface mapping. The Beltrami
coefficient quantifies the local geometric distortion of the mapping. By
controlling the magnitude of the Beltrami coefficient through a suitable
activation function, the bijectivity and geometric distortion of the mapping
can be controlled. The Beltrami coefficient is then fed into the Beltrami
solver network to reconstruct the corresponding mapping. The surface
registration enables the acquisition of corresponding regions and the
establishment of point-wise correspondence between different partial faces,
facilitating precise shape comparison through the evaluation of point-wise
geometric differences at these corresponding regions. Experimental results
demonstrate the effectiveness of the proposed method.

_Keywords_ Facial landmark detection $\cdot$ surface registration $\cdot$
quasi-conformal geometry $\cdot$ convolutional neural networks

##  1 Introduction

The shape analysis of the human face has garnered significant interest among
researchers, emerging as a captivating and widely explored field of study[29,
4, 9]. It holds significant importance across diverse domains such as computer
vision, computer graphics, and medical imaging, where it finds extensive
applications. In computer vision, facial shape analysis plays a pivotal role
in tasks such as facial recognition and facial expression analysis[41, 31, 22,
45, 36]. Similarly, in the realm of medical imaging, the analysis of facial
shape proves invaluable in medical diagnosis and treatment planning[7].

2D facial analysis has been extensively studied and is the subject of a
surplus of literature. Nevertheless, recent advancements in 3D acquisition
devices have revolutionized the field by facilitating the availability of rich
3D data of human faces. This surpasses the limitations of 2D images by
providing a wealth of geometric information. Notably, numerous mathematical
models for 3D facial analysis have been developed, yielding promising
results[20, 24, 32]. A critical step in conducting shape analysis of 3D faces
involves establishing accurate and dense pointwise correspondence among
different human faces. This dense correspondence permits systematic and
quantitative comparison, enabling comprehensive shape analysis[35, 48]. While
3D surface registration of complete faces has been extensively studied, real-
world scenarios often present challenges such as occlusions, partial view
angles, or self-occlusions caused by hands or accessories. When faced with
limited data availability, achieving correspondence between 3D partial faces
becomes a more intricate task, commonly referred to as the partial face
registration problem. The challenge lies in accurately determining the
corresponding regions between two partially observed faces and establishing
point-wise correspondence between these regions[50, 8, 18]. This particular
class of surface registration problem is comparatively less studied. Without
precise registration between partial faces, shape analysis and comparison
among them cannot be performed accurately.

In this paper, we introduce a novel deep learning-based method that addresses
the challenge of partial face registration. Our approach combines
quasiconformal theories and deep neural networks to achieve accurate landmark
detection and surface registration, even when dealing with partial facial
data. This method can be further applied to facial recognition tasks. Our
framework begins with a Landmark Detection Network (LD-Net), which precisely
detects landmarks on the partial face surfaces using the curvature values
derived from each vertex. This innovative approach allows us to accurately
locate facial landmarks, even in scenarios with limited facial information or
occlusions. These detected facial landmarks play a crucial role in guiding the
subsequent registration process. To establish a dense correspondence between
the partial face and the template surface, we employ a registration network
based on quasiconformal theories. This network consists of the Coefficients
Prediction Network (CP-Net), which outputs the optimal Beltrami coefficient
representing the surface mapping. The Beltrami coefficient quantifies the
local geometric distortion of the mapping. By controlling the magnitude of the
Beltrami coefficient through a suitable activation function, we can
effectively control the bijectivity and geometric distortion of the mapping.
The optimal Beltrami coefficient is then fed into the Beltrami solver network
to reconstruct the corresponding mapping. The surface registration achieved by
our method enables the acquisition of corresponding regions and the
establishment of point-wise correspondence between different partial faces.
This facilitates precise shape comparison by evaluating point-wise geometric
differences at these corresponding regions. To thoroughly evaluate the
effectiveness of our proposed method, we conducted extensive experiments on
partial facial surfaces. The results obtained from these experiments provide
strong evidence for the robustness and reliability of our approach in
achieving accurate landmark detection and surface registration for partial
faces. The results of facial classification using the partial face
registration further demonstrate the effectiveness of our proposed framework
for analyzing partial faces.

In summary, the contribution of our work is as follows:

  1. (I)

Automatic extraction of feature landmarks on partial faces: We propose a deep
neural network that automatically extracts feature landmarks on partial faces
by leveraging surface curvature information. The network accurately identifies
their presences and precise locations, thereby facilitating subsequent
registration.

  2. (II)

Quasi-conformal deep neural network for 3D partial face registration: We
propose a novel approach that combines quasi-conformal (QC) geometry with
convolutional neural networks (CNNs) to address the challenge of registering
partial face surfaces. By integrating these techniques, we can improve both
the accuracy and efficiency of the registration process, even in the presence
of occlusions or limited facial information.

  3. (III)

Facial recognition of partial faces: By achieving precise surface registration
of partial faces, we can identify the overlapping regions on each partial face
and establish point-wise correspondences between them. This facilitates shape
comparison and enables the recognition of partial faces through the evaluation
of geometric differences at key feature landmarks.

##  2 Related Works

###  2.1 Quasi-Conformal Mapping

Surface parameterization and mapping problems are widely investigated, and
algorithms have been proposed in recent decades [15, 38]. In particular, as
conformal maps preserve the local geometry, researchers developed algorithms
that could be applied to various fields [11, 26, 33, 47]. However, the
computation of conformal mapping is expensive and quasi-conformal theory has
been found useful with trade-offs between conformality and other prescribed
constraints [19, 25]. The Beltrami coefficient is introduced to measure the
distortion of the mapping. In [16], Gardiner et al. proved the 1-1
correspondence between Beltrami coefficients and surface diffeomorphisms.

The utilization of quasi-conformal geometry in registration tasks allows for
the alignment of different surfaces by minimizing distortion, which enables
accurate and consistent alignment, even when dealing with complex deformations
and variations in shape. In segmentation, quasi-conformal geometry has been
applied to delineate boundaries and segment regions of interest by leveraging
the topological properties of the underlying complex domains. Additionally,
shape-matching techniques based on quasi-conformal geometry enable efficient
and effective matching of shapes by preserving their local geometric
properties.

By incorporating quasi-conformal geometry into our approach, we leverage its
robustness and versatility to achieve accurate and reliable registration of
partial face surfaces. This enables us to align and integrate partial faces
with minimal distortion, facilitating subsequent analysis and recognition
tasks.

###  2.2 Facial Landmark Detection

Facial landmark detection plays a crucial role in facial analysis as it
provides key points representing anatomical structures, facial features, and
regions of interest for various high-level tasks such as face recognition,
registration, and editing [45]. Traditional methods for landmark detection
rely on machine learning algorithms, including Support Vector Machines (SVM)
[43] and Random Forests (RF) [28], using manually crafted key points. However,
these approaches require expert annotations and have limited generalization
capabilities across different datasets.

Recent advancements in landmark detection leverage the powerful feature
extraction capabilities of Convolutional Neural Networks (CNN) to perform
unsupervised landmark detection [23, 5]. To further improve detection accuracy
and generalization abilities, researchers have proposed improved network
architectures [53, 51], learning patterns [49], and well-designed loss
functions [14, 44]. However, these existing approaches still face challenges
in identifying facial landmarks with significant variations or on partial
faces.

In this work, we present our landmark detection network specifically designed
for partial human faces with diverse variations. Our network addresses the
limitations of existing methods by effectively detecting landmarks on partial
faces, thereby providing crucial information for subsequent registration
processes.

###  2.3 Facial Registration

Traditional face registration methods primarily focus on aligning distorted
facial landmarks with their corresponding landmarks using CNN-based networks
and L2 loss [50]. These methods often incorporate advanced techniques such as
inception blocks [39] and siamese networks [37, 40] to extract multi-scale
features and learn discriminative embeddings. To handle variations in facial
landmarks, techniques like wing loss and triplet loss have been proposed. 3D
surface registration methods have also been developed to address non-linear
distortions. These methods[46, 42] utilize 3D models to capture complex facial
deformations.

However, acquiring a complete face for registration purposes can be
challenging, leading to a growing interest in methods for partial surface
registration. In recent years, several approaches have been proposed to tackle
this problem. In [6], Cao et al. introduced a method that utilizes deep
functional maps for unsupervised training to achieve shape correspondence even
for partial shapes. Bracha et al. proposed a novel idea in [3], where they
establish a direct correspondence between partial and full face shapes through
feature matching. These recent advancements in the field of partial face
registration highlight the growing interest in finding effective solutions for
this challenging problem. However, existing research often lacks focus on the
analysis of face surfaces. Consequently, the challenges associated with
feature extraction and region correspondence for partial faces remain a
significant problem that requires further attention and investigation.

To address this challenge, we propose a quasi-conformal-based registration
model for partial faces’ correspondence. Our model combines the benefits of
quasi-conformal methods while handling partial components and non-rigid
deformations, filling the existing gaps in face registration research.

##  3 Mathematical Background

###  3.1 Quasi-conformal theory

In this section, we introduce the theories related to quasi-conformal maps.
For more details, please refer to [2, 16].

Compared with the conformal map, the quasi-conformal map is more generalized
and can be understood to be a map with bounded conformality distortion. An
orientation-preserving homeomorphism
$f:\Omega\subset\mathbb{C}\rightarrow\Omega^{\prime}\subset\mathbb{C}$ is said
to be a quasi-conformal map if it satisfies the Beltrami equation:

$\displaystyle\frac{\partial f}{\partial\bar{z}}=\mu_{f}(z)\frac{\partial
f}{\partial z},$ (1)

where $\mu_{f}(z)$ is a complex-valued Lebesgue-measurable function satisfying
$\|\mu_{f}(z)\|_{\infty}<1$ is the Beltrami coefficient of $f$ and it
represents the conformality distortion of $f$.

Figure 1: The illustration of how Beltrami coefficient $\mu$ determine the
conformal distortion.

If we are given a Beltrami coefficient $\mu$ with $\|\mu\|_{\infty}<1$, we can
solve its corresponding quasi-conformal mapping satisfying the Beltrami
equation[16].

######  Theorem 1.

Suppose $\mu:\mathbb{D}\rightarrow\mathbb{C}$ is Lebesgue measurable with
$\|\mu\|_{\infty}<1$. There is a quasi-conformal homeomorphism $\phi$ from
$\mathbb{D}$ to itself, which is in the Sobolev space $W^{1,2}(\mathbb{D})$
and satisfies the Beltrami equation (1) in the distribution sense.
Furthermore, by fixing 0 and 1, $\phi$ is uniquely determined.

Conversely, if an orientation preserving homeomorphism $\phi$ is preseted, we
can get its Beltrami coefficient by:

$\displaystyle\mu_{\phi}=\frac{\partial\phi}{\partial\bar{z}}/\frac{\partial\phi}{\partial
z}.$ (2)

Thus, the Jacobian Matrix $J$ of $\phi$ can be expressed according to
$\mu_{\phi}$ as follows:

$J(\phi)=\left|\frac{\partial\phi}{\partial
z}\right|^{2}\left(1-\left|\mu_{\phi}\right|^{2}\right).$ (3)

Given that $\phi$ is an orientation preserving homeomorphism, $J(\phi)>0$ and
$\left|\mu_{\phi}\right|<1$ everywhere. Hence,
$\left\|\mu_{\phi}\right\|_{\infty}<1$ should hold. Theorem 1 indicates that
under suitable normalization, every $\mu$ with $\left\|\mu\right\|_{\infty}<1$
is associated with a unique homeomorphism. Therefore, a homeomorphism from
$\mathbb{C}$ or $\mathbb{D}$ onto itself can be uniquely determined by its
associated Beltrami coefficient.

In addition, we can also composite two quasi-conformal maps and the Beltrami
coefficient can be computed. Suppose we have $f,g$:
$\mathbb{C}\rightarrow\mathbb{C}$, two quasi-conformal maps, associated with
Beltrami coefficients $\mu_{f}$ and $\mu_{g}$ respectively. Thus, the Beltrami
coefficient of $g\circ f$ is

$\mu_{g\circ f}=\frac{\mu_{f}+(\mu_{g}\circ
f)\tau}{1+\bar{\mu}_{f}(\mu_{g}\circ f)\tau},\tau=\frac{\bar{f}_{z}}{f_{z}}$
(4)

###  3.2 Conformal Parameterization

In this section, we are going to introduce the variational formulation of the
conformal parameterization from [11].

Suppose we have a piecewise linear mesh patch $M$ and we want to construct a
piecewise linear mapping $\psi$ between $M$ and a planar triangulation
$U\in\mathcal{R}$ conformally. We denote $x_{i}$ to be 3D positions of the
i-th vertex on $M$ and $u_{i}$ is its corresponding position on $U$.

In [11], they introduced the Dirichlet energy and Chi energy on
triangulations.

$\displaystyle
E_{A}=\sum_{oriented\,edges(i,j)}\cot\alpha_{ij}|u_{i}-u_{j}|^{2},$ (5)
$\displaystyle E_{\chi}=\sum_{j\in
N(i)}\frac{\cot\gamma_{ij}+\cot\delta_{ij}}{|x_{i}-x_{j}|^{2}}(u_{i}-u_{j})^{2}$
(6)

in which $\alpha_{ij}$ is the opposite left angle of edge $(i,j)$ on the same
triangular, $\gamma_{ij}$ and $\delta_{ij}$ are the angles on the both side of
edge $(i,j)$ near vertex $x_{j}$. In this way, the distortion measure $E$ can
be defined to be

$\displaystyle E=\lambda E_{A}+\zeta E_{\chi}$ (7)

in which $\lambda$ and $\zeta$ are parameters. By minimizing $E$, we can get a
smooth parameterization.

###  3.3 Surface Curvature

Curvature is an important quantity in differential geometry for assessing how
a surface deviates from a plane. In the human face surface, the curvature
values contain the personal features that could be used for recognition.

We define $N:S\rightarrow\mathbb{S}^{2}\subseteq\mathbb{R}^{3}$ to be the
_normal map_ giving unit vector at each point $p$. Suppose $C$ is a regular
curve on $S$, $p$ is a point on $S$ and $k$ is the curvature of $C$ at $p$. We
set $\cos\theta=\langle n,N\rangle$, where $N$ is the normal vector to $S$ at
$p$ and $n$ is normal to $C$. $k_{n}=k\cos\theta$ is called the _normal
curvature_ of $C$ at $p$. The _principal curvatures_ at $p$ are the maximum
and minimum of the normal curvature, denoted as $k_{1}$ and $k_{2}$
respectively. The _mean curvature_ at $p$ is defined to be
$H=\frac{1}{2}(k_{1}+k_{2})$.

Figure 2: Demonstration of the surface matching problem. $S_{1}$ and $S_{2}$
are two partial faces with similar parts. $\Omega_{1}$ and $\Omega_{2}$ are
the intercepted parts and labeled in magenta. The goal is to find a bijective
mapping $f$ such that the regional correspondence can be constructed for
$\Omega_{1}$ and $\Omega_{2}$.

##  4 Deep Surface Registration of Partial Faces

The problem of partial surface correspondence and registration is highly
significant, particularly in scenarios with limited data availability, as not
all situations are ideal. To tackle this problem, we initiate our analysis by
introducing the subsequent matching problem of partial surfaces.

######  Problem.

Given two surfaces $S_{1}$ and $S_{2}$, where $S_{1}$ and $S_{2}$ has similar
regions. Suppose there exist regions $\Omega_{1}$ and $\Omega_{2}$ such that
$\Omega_{1}\subseteq S_{1}$ and $\Omega_{2}\subseteq S_{2}$ respectively with
similar features, the problem is to find an optimal bijective mapping
$f:\Omega_{1}\rightarrow\Omega_{2}$ such that $f(\Omega_{1})=\Omega_{2}$.

To visualize this problem, we demonstrate it using human faces as examples in
Fig. 2. In this paper, we are introducing a deep learning based method for
finding the optimal bijective mapping $f$ for partial faces’ matching problem,
which is then applied to facial recognition.

###  4.1 Framework Overview

In this section, we present a detailed overview of our framework for partial
face registration, and the strategy is shown in Fig. 3.

Our method aims to take a mean face as the template and find the optimal
mapping that registers the partial face to the template. In Fig. 3, $S_{1}$
and $S_{2}$ are two partial faces that share similar regions. For each partial
face, our model could generate a quasi-conformal mapping $f_{1}$, which is
bijective, and map the partial face $S_{1}$ to the corresponding region on
template face $T$. By applying the same method to the partial face $S_{2}$, we
obtain a bijective quasi-conformal mapping $f_{2}$. Consequently, we can
compute an inverse mapping of $f_{2}$ and compose it with $f_{1}$. This
enables us to register the partial face $S_{1}$ to $S_{2}$ by mapping
$f_{1}\cdot f_{2}^{-1}$.

Figure 3: Demonstration of the strategy for solving partial surface
correspondence problem. $S_{1}$ and $S_{2}$ are two partial faces in cyan and
magenta, respectively. $T$ is the template face in gray. $f_{1}$ and $f_{2}$
are bijective mappings.

###  4.2 Method Overview

Figure 4: Demonstration of the framework for registration. The first row
demonstrates how we generate a quasi-conformal mapping based on the input face
mesh. Then, we demonstrate how the mapping registers an input face mesh to the
template face.

In this section, we introduce the details of finding the optimal mapping $f$
for mapping a partial face to the corresponding region on the template face
$T$. The overview of it is shown in Fig. 4. Initially, we parameterize the
partial face to the 2D domain by conformal and quasi-conformal mapping with
the least angle distortion. Subsequently, a network is constructed to identify
the landmarks of the parameterized face and ascertain the existence of each
point. The final step involves feeding the output from the preceding network,
along with the parameterized partial face, into the registration network. This
process facilitates the acquisition of appropriate Beltrami Coefficients,
which in turn generate a corresponding quasi-conformal mapping to register the
input face to the template face, as shown in Fig. 4. Details for each
component will be illustrated in the following parts.

####  4.2.1 Parameterization Method

The partial face is a 3D surface mesh, and to adopt the learned method, we
need to parameterize it into the 2D domain. We parameterize the 3D mesh
conformally and then use the quasi-conformal theory to get a visualizable face
mesh while minimizing the angle distortion and keeping the majority of
information of the surface mesh as shown in Fig. 5.

Figure 5: The flow on the left demonstrates the parameterization by discrete
natural conformal parameterization(DNCP) and modified by least-squares quasi-
conformal map (LSQC). The chart on the right shows the angle distortion which
is small.

Initially, we adopt the Discrete Natural Conformal Parameterization(DNCP) in
[11] and thus get a free boundary map from the surface. The mapping generated
by DNCP is conformal, which preserves every angle of the mesh. However, as
shown in Fig. 5, the parameterized face is hard to visualize as a face and
also hard to determine the location of eyes or mouth; thus, we are going to
use quasi-conformal theory to modify it by using an approach called least-
squares quasi-conformal map (LSQC)[34, 52] with proper boundary conditions.
Suppose $f:\Omega\subset\mathbb{C}\rightarrow\Omega^{\prime}\subset\mathbb{C}$
is a quasi-conformal map and $f=u+iv$, $\mu_{f}=\rho+i\tau$ where $u,v,\rho$
and $\tau$ are real-valued functions. We can transfer the Beltrami equation to

$\displaystyle\begin{pmatrix}u_{x}\\\ u_{y}\end{pmatrix}=\begin{pmatrix}0&1\\\
-1&0\end{pmatrix}A\begin{pmatrix}v_{x}\\\ v_{y}\end{pmatrix},$ (8)

in which

$\displaystyle
A=\frac{1}{1-|\mu|^{2}}\begin{pmatrix}(\rho-1)^{2}+\tau^{2}&-2\tau\\\ \
-2\tau&(1+\rho)^{2}+\tau^{2}\end{pmatrix}.$ (9)

By the relation $u_{xy}=u_{yx}$, we can get the equation

$\displaystyle\nabla\cdot(A\nabla v(z))=0,\,\nabla\cdot(A\nabla u(z))=0.$ (10)

These are the Euler-Lagrange equations of the following two Dirichelet-type
energies respectively:

$\displaystyle E_{A}(u)=\frac{1}{2}\int_{\Omega}\|A^{1/2}\nabla
u\|^{2},\,E_{A}(v)=\frac{1}{2}\int_{\Omega}\|A^{1/2}\nabla u\|^{2}.$ (11)

Here we introduce the energy $E_{LSQC}$ from [34], and we set the boundary of
$\Omega$ to be mapped to a given boundary and get a quasi-conformal map from
it by minimizing:

$\displaystyle\left\\{\begin{aligned}
&E_{LSQC}(u,v)=\frac{1}{2}\int_{\Omega}\|P\nabla u+JP\nabla v\|^{2}dxdy\\\
&f(\partial\Omega)=\partial\Omega^{\prime}\end{aligned}\right.,$ (12)

in which

$\displaystyle P=\frac{1}{\sqrt{1-|\mu|^{2}}}\begin{pmatrix}1-\rho&-\tau\\\
-\tau&1+\rho\end{pmatrix}\,\,\text{and}\,\,J=\begin{pmatrix}0&-1\\\
1&0\end{pmatrix}$ (13)

It is observed that $P^{T}P=A$ and the Beltrami equation holds if and only if
$E_{LSQC}(u,v)=0$. There is a relationship between $E_{A}(u)$, $E_{A}(v)$,
$E_{LSQC}(u,v)$ and $\mathcal{A}(u,v)$. $\mathcal{A}(u,v)$ is the area of
$\Omega^{\prime}=f(\Omega)$:

$\displaystyle
E_{A}(u)+E_{A}(v)-E_{LSQC}(u,v)=\mathcal{A}(u,v)=\int_{\Omega}(u_{x}v_{y}-v_{x}u_{y})dxdy$
(14)

In our setting, the face surface is discretized, and $f$ is linear on each
triangular face. Thus, we have $\mu$ constant on each face. On an oriented
triangular face $T=[w_{0},w_{1},w_{2}]$, the gradient of a function
$f=(f_{0},f_{1},f_{2})$ is given by:

$\displaystyle\nabla f=\frac{1}{2Area(T)}\begin{pmatrix}0&-1\\\
1&0\end{pmatrix}\sum_{i=0,1,2}f_{i}(w_{2+i}-w_{1+i})$ (15)

where the index should modulo 3. In this way, $E_{A}(u)$ and $E_{A}(v)$ can be
discretized by summing over all faces which could be expressed with the
generalized Laplacian matrix $\mathcal{L}_{\mu}$:

$\displaystyle
E_{A}(u)=u^{T}\mathcal{L}_{\mu}u,\,E_{A}(v)=v^{T}\mathcal{L}_{\mu}v.$ (16)

We can also discretize the area matrix from Equ.14 by some skew-symmetric
matrix $U$:

$\displaystyle\mathcal{A}(u,v)=\begin{pmatrix}u^{T}&v^{T}\end{pmatrix}\begin{pmatrix}0&U\\\
-U&0\end{pmatrix}\begin{pmatrix}u\\\ v\end{pmatrix}$ (17)

We can derive a symmetric matrix $N$:

$\displaystyle N=\begin{pmatrix}\mathcal{L}_{\mu}&0\\\
0&\mathcal{L}_{\mu}\end{pmatrix}-\begin{pmatrix}0&U\\\ -U&0\end{pmatrix}$ (18)

In this way, the energy $E_{LSQC}$ can be discretized to be:

$\displaystyle
E_{LSQC}(u,v)=\begin{pmatrix}u^{T}&v^{T}\end{pmatrix}N\begin{pmatrix}u\\\
v\end{pmatrix}$ (19)

To solve the corresponding mapping, we set $\mu=0$ with landmark constraints.
The aim of this step is to have a 2D face mesh with a proper shape for
visualization. Suppose the input 3D face mesh is $M_{0}$ and the boundary
points are $\partial M_{0}=\\{m_{0},m_{1},...,m_{k}\\}$, in which
$m_{i}=(x_{i},y_{i},z_{i})$. We project the boundary points of the face mesh
to the 2D domain as $\hat{m}_{i}=(x_{i},y_{i})$. Since we parameterize the
mesh $M_{0}$ by DNCP first, we denote the parameterized face mesh to be
$M_{1}$ and the boundary points are $\partial
M_{1}=\\{n_{0},n_{1},...,n_{k}\\}$. In this way, we are solving the following
problem:

$\displaystyle\left\\{\begin{aligned}
&E_{LSQC}(u,v)=\begin{pmatrix}u^{T}&v^{T}\end{pmatrix}N\begin{pmatrix}u\\\
v\end{pmatrix}\\\ &f(n_{i})=\hat{m}_{i},\,\,\,\,\,n_{i}\in\partial
M_{1}\end{aligned}\right..$ (20)

Minimizing the energy $E_{LSQC}$ could generate a mapping satisfying the input
value $\mu$ if there are no boundary constraints. However, due to the added
boundary conditions, very few angle distortions are introduced. In this way,
the Beltrami coefficients $\mu$ would close to zero, and the mapping could be
quasi-conformal, as shown in Fig. 5.

####  4.2.2 Landmark Detection

In this section, we outline our proposed Landmark Detection Network (LD-net)
architecture, designed specifically for detecting significant points on a
parameterized partial face. The detected landmarks will be used to assist the
registration process.

We previously introduced the process for flattening the partial face using
quasi-conformal theory. We also computed the curvature for each vertex,
storing this data for future use. To leverage the capabilities of
convolutional neural networks, we transformed the parameterized data into the
image domain, with pixel values determined by their corresponding curvature
values. The rest of the pixels are remaining zero. This transformation allows
us to easily use the data as input for our landmark detection network.

Besides recording the vertex value on the image domain, we also save the
location and the existence of the landmark points as labels for the supervised
training. In the inference part, the input data is a partial face on the image
domain, and the objective of this network extends beyond merely determining
point locations; it also involves confirming the presence of each point. The
structure of our proposed network is illustrated in Fig. 6.

Figure 6: Demonstration for the LD-net. The input parameterized partial faces
pass through convolution layers, and we take features from different scales
and concatenate them into a column. Then we take the column data into two
different multilayer perceptrons then generate the output masks and landmark
locations.

We first pass the smoothed image with curvature value on each vertex into
convolution layers as shown in Fig. 6. Inspired by the structure by [17], we
take the mean value of each channel at different stages and concatenate it to
be a column for further process. The advantage is that the features at
different scales are all taken into the multi-layer perceptron(MLP). Next, we
take the concatenated value to two MLPs separately, and MLPs could learn from
the features to generate the location and the existence of the landmarks. In
the inference part, we take the detected prominent points with probability
over 90% as existing.

Loss Function: For the training of LD-net, we adopt a supervised mode. The
loss function of this network is as follows:

$\mathcal{L}_{\text{LD}}=\frac{\alpha}{N}\sum_{i}^{N}-[\hat{p}_{i}\cdot\log(p_{i})+(1-\hat{p}_{i})\cdot\log(1-p_{i})]+\frac{\beta}{N}\sum_{i}^{N}\|(\hat{x}_{i}-x_{i})\odot
p_{i}\|,$ (21)

in which $N$ is the number of prominent points predefined on the face,
$\hat{p}_{i}$ and $\hat{x}_{i}$ are the output mask probability and landmark
locations for the i-th point from the network, $p_{i}$ and $x_{i}$ are the
label mask and label location for the i-th point. In our setting, the
predicted $\hat{p}_{i}\in[0,1]$ and the labeled $p_{i}=0$ or $1$.

The first term in Equ. 21 is designed for the existence prediction. The
network generated the probability of the existence of each point, and we
adopted the cross-entropy loss between the output and the label in the
training part. The second term constrains the distance between the output
location and the labeled locations.

####  4.2.3 Registration

As introduced in the Section Mathematical Background, a quasi-conformal
mapping is bijective when the Beltrami coefficient $\mu$ is less than 1, and
we can easily get its inverse mapping. Based on this theory, we will introduce
the procedure for registration between different partial face meshes in this
part.

Figure 7: Demonstration for the registration process. One path takes the
detected landmarks’ locations and masks as input, and the second path’s input
is the partial face. Template face is NOT the input of the Coefficient
Prediction Network(CP-net). Here is for visualization and comparison. The CP-
net generates the Beltrami Coefficient $\mu$, and the Beltrami Solver Network
generates the quasi-conformal mapping $f$ from $\mu$. Then we can register the
input partial face by $f$ to the template face.

Initially, as shown in Fig. 7, we define the mean face as the template. Then,
we design a network that is able to take the parameterized face and detected
landmarks as input, and by using the Coefficients Prediction Network, we
generate the Beltrami coefficients for producing the quasi-conformal mapping
to register the input face to the template. To generate the quasi-conformal
mapping in a fast and accurate way, we use the Beltrami Solver Network
introduced by Chen et al.[8]. We demonstrate its structure in Fig. 9. In this
way, different face meshes could be registered to the template mesh, and thus,
we can easily compute its inverse mapping, which allows us to compare
differences between face meshes.

Figure 8: Demonstration for the Coefficient Prediction Network. The input is
separated into two paths. The first one is taking landmarks and masks as input
to the Landmark Encoder, which consists of the fully connected layers, and
then reshaping the output to 2D. The second path takes the input partial face
into the Image Encoder which consists of the convolutional neural networks.
Then, the reshaped features are concatenated and passed through the Decoder to
get the Beltrami coefficients $\mu$. The Decoder consists of the transposed
convolutions, and we use an activation function to keep each output value
lower than 1 to generate a bijective quasi-conformal mapping.

Next, we introduce the details of the Coefficient Prediction Network as shown
in Fig. 8. We separate the input into two paths for landmarks and
parameterized faces. For the Landmark Encoder, we take the detected landmarks
and masks as input and use fully connected layers to process and then reshape
them into 2D channels. The input partial face on the second path is in the
image domain, and we pass it through the convolutional layers and concatenate
it with the reshaped output from the Landmark Encoder. To preserve the
bijectivity and keep the least distortion in the generated mapping, we use an
activation function at the end of the Decoder to suppress the norm of Beltrami
Coefficients to be less than 1 which represents no folding, and the lower, the
better.

The following activation function $\Phi$ is used:

$\Phi(m)(T)=\frac{e^{|m(T)|}-e^{-|m(T)|}}{e^{|m(T)|}+e^{-|m(T)|}}e^{iarg(m(T))},$

in which $m$ is the output from the last layer and $m(T)$ is a complex number
on triangular face $T$. In this way, We have $\|\Phi(m)\|_{\infty}<1$.

Figure 9: Demonstration of the structure of BSnet from [8].

Loss Function: To solve the quasi-conformal mapping from the generated
Beltrami Coefficent $\mu$, we use the BS-Net from [8] which is pre-trained. In
the training process, we take $\mu$, $\nabla\mu$, landmark loss, and curvature
loss into consideration, which is specified in Equ. 22. The usage of $\mu$ and
$\nabla\mu$ in the first two terms of Equ. 22 is to minimize the distortion
and have a smooth map for registration. The third term is the landmark loss
and the aim is to register the existing prominent points to their
corresponding target. The last term is the curvature loss, which ensures the
partial face can be registered even with no 1-to-1 correspondence. Thus, in
the computation of the last term, we only compute the intercepted parts of the
registered face and the template face.

$\mathcal{L}_{Reg}=\frac{\kappa}{N}\sum^{N}_{i=1}\|\mu_{i}\|^{2}_{2}+\frac{\tau}{N}\sum^{N}_{i=1}\|\nabla\mu_{i}\|^{2}_{2}+\frac{\varsigma}{M}\sum^{M}_{i=1}\|(x_{i}-\hat{x}_{i})\odot
p_{i}\|^{2}_{2}+\sigma(I_{2}\cdot P_{I_{2}\cap(I_{1}\cdot f^{\mu})}-I_{1}\cdot
f^{\mu})^{2}$ (22)

in which $x_{i}$ is the target points and $\hat{x}_{i}$ are the registrated
points. $M$ and $N$ are the number of prominent points and the number of
triangular faces separately. $p_{i}$ is the input mask for the prominent point
and $p_{i}=0$ or $1$. If the output probability from the LD-net is larger than
90%, $p_{i}=1$. Otherwise $p_{i}=0$. Since the $\mu_{i}$ and $\nabla\mu_{i}$
are defined on each triangular face. $I_{1}$ represents the starting face’s
curvature, and $I_{2}$ represents the channel of the target face’s curvature.
$P_{I_{2}\cap(I_{1}\cdot f^{\mu})}$ is the mask for overlapping points of the
target face mesh $I_{2}$ with the registered face $I_{1}\cdot f^{\mu}$.
$f^{\mu}$ is the mapping generated from $\mu$ by BSnet.

##  5 Recognition

With our approach, we are able to determine the existence and location of
features in each input partial face, facilitating their registration to the
template. By aligning the registered partial faces, we can perform recognition
through the overlapping of features. We demonstrate it in Fig. 10.

Figure 10: Demonstration for the registration process of two faces. The first
column consists of two input 3D faces and the second column are their
corresponding parameterized faces. The third column is the registered result
after two faces pass through our model separately. The fourth column is the 3D
visualization of the registered faces. We use the intercepted landmarks’
normal vector to compute the distance and generate the recognition confidence.

To perform the 3D face recognition, we adopt the algorithm proposed by Marras
et al.[30] which introduced how to recognize a face by using the azimuth angle
of surface normals. In this part, we use the facial features for recognition.
The normal field is defined as a set of local surface normals
$n(\boldsymbol{x})=(n_{x}(\boldsymbol{x}),n_{y}(\boldsymbol{x}),n_{z}(\boldsymbol{x}))$
and $\|n(\boldsymbol{x})\|=1$, where $\|\cdot\|$ is the $\mathcal{L}_{2}$
norm. The azimuth angle is defined as
$\phi(\boldsymbol{x})=\arctan\frac{n_{y}(\boldsymbol{x})}{n_{x}(\boldsymbol{x})}$.
Then we can define the cosine-based dissimilarity measure between two vectors
of azimuth angles $\boldsymbol{\phi}_{i}$ and $\boldsymbol{\phi}_{j}$ as:

$d^{2}(\boldsymbol{\phi}_{i},\boldsymbol{\phi}_{j})\triangleq\sum_{\boldsymbol{x}}\\{1-cos[\phi_{i}(\boldsymbol{x})-\phi_{j}(\boldsymbol{x})]\\}$

in which $\boldsymbol{x}$ are selected landmarks for recognition. Details for
the experiment will be shown in the following part.

##  6 Experiments

###  6.1 Landmark Detection

Dataset. To train our model, we first generate faces by an CelebA dataset [27]
and 3D Morphable Model (3DMM)[10]. To start, we generate the reconstructed
faces from the image as shown in 11.

Figure 11: Demonstration for generating 3D meshes from image [10]. The image
on the left is the original one and the image on the right is the mesh
generated from the input image. Figure 12: Demonstration for the training loss
of the LD-net(Left) and Reg-net(Right). The value on the vertical axis is the
error and the value on the horizontal axis is the training step.

We generate 20000 face meshes and randomly cut the faces in different ways.
The first dataset is generated by cutting off part of the faces, which means
the output is still genus-1. The second dataset is generated by cutting off
holes in random sizes and locations on the face. Then, the generated
parameterized surfaces are interpolated on the image domain with its mean
curvature value on each vertex.

Benchmarks. We evaluate our model on the above-mentioned generated datasets.
The interpolated images are the size of 256 by 256. The mesh surface template
consists of 35709 vertexes and 70865 facets. To increase the robustness of our
model, every parameterized face will be rotated and translated randomly. The
training data and testing data are in the ratio of 5:1.

In training, we train the location-predicting path for the initial 50 epochs
by using only landmark constraints. Then we add the mask loss gradually from
epoch 50. Here you can see the loss in Fig. 12.

Figure 13: Demonstration for landmarks detection. The black points are the
detected facial features.

| Method | Landmark Loss | Std. of Landmark Loss | Mask Accuracy | Std. of Mask
---|---|---|---|---|---
Partial | LD-net | 0.935 | 0.057 | 98.2% | 0.46%
SBR [13] | 87.537 | 0.025 | $\backslash$ | $\backslash$
AWing [44] | 13.354 | 0.151 | $\backslash$ | $\backslash$
PFLD [17] | 1.167 | 0.051 | $\backslash$ | $\backslash$
SRT [12] | 1.088 | 0.042 | $\backslash$ | $\backslash$
Holes | LD-net | 1.066 | 0.042 | 99.0% | 0.41%
SBR [13] | 87.529 | 0.015 | $\backslash$ | $\backslash$
AWing [44] | 14.066 | 0.097 | $\backslash$ | $\backslash$
PFLD [17] | 1.868 | 0.027 | $\backslash$ | $\backslash$
SRT [12] | 0.947 | 0.037 | $\backslash$ | $\backslash$

Table 1: Comparison with the existing method for facial landmark detection on
the dataset of partial faces and the dataset of faces with holes. We also list
the standard deviation while testing. The best result is shown in red, second
best in blue.

In table 1, we compare our method with other facial landmark detection
networks. All the networks are trained on partial faces and face with holes,
and all the compared networks have no mask output. The testing set consists of
5000 images, and the images are not contained in the training set.

As shown in table 1, our model outperforms all other methods. SRT and PFLD can
also achieve comparable loss in landmark loss but have no mask output for the
prominent points’ detection. SBR and AWing failed in this sort of dataset.

###  6.2 Registration

For the registration process, we use the same dataset as we used for
detection. In the training of the registration network, we set a face with no
missing part as a template and register all the input faces to the template by
a generated quasi-conformal mapping. Since the quasi-conformal mapping is
bijective and we can easily get the inverse mapping based on the output quasi-
conformal mapping.

In Fig. 12, we demonstrate the training loss of the registration network. We
set the constraint of value $\mu$ at a low level while starting training and
adding up its parameter gradually. It is easy to see that value of $\mu$ will
drop close to zero at the compensation of a little bit higher rigid error.

In Fig. 14, we demonstrate our model is able to register a partial face to the
template full face and have all landmarks aligned.

Figure 14: On the first column, the input partial face is in pink and the
template face is in cyan. The overlapping part is in pink. The second column
is the registered partial face with the template face. It can be viewed that
the facial features can be aligned with the template. The third column is the
mapping generated by our model.

Since every partial face can be registered to the template and the generated
mapping is quasi-conformal, we can get its inverse mapping easily. Thus, for
every input partial face, after being registered by our model, we can easily
analyze the overlapping features for recognition. In Fig. 10, we demonstrate
two different partial faces which are registered together.

In Fig. 15, we demonstrate that if two faces with very few overlapping, we can
also register both faces and easy to analyze the overlapping features for
recognition. We will test our model for recognition in the next part.

Figure 15: Demonstration of two faces with few overlapping.

###  6.3 Recognition

Dataset. We utilize the Labeled Faces in the Wild (LFW) dataset, which
comprises 13,233 facial images from 5749 identities. The dataset includes a
diverse collection of real-world facial images captured under various
conditions, and we select the ones that are easier for 3DMM to reconstruct the
3D face meshes. This enables comprehensive evaluations of our approach. This
labeled large dataset allows us to assess the performance and robustness of
our method effectively.

The data we use for recognition is Labeled Faces in the Wild(LFW)[21] and we
mainly compare with the non-rigid ICP[1] and the result is shown in 2. To be
precise, our method is trained on the previous dataset(use the dataset name)
so the LFW dataset is unseen for our model. In Table 2, we can achieve the
recognition accuracy of 91.73%. It is acceptable that it is not close to 100%
since the overlap landmarks could reduce to just a few points. Besides testing
the recognition accuracy, we also perform statistical analyses of the result.
Suppose $x_{0}$ is a vector that contains the distances of the same person and
$x_{1}$ contains the distances between different persons. Our hypothesis is
that $H_{0}$: the mean of $x_{0}$ equals the mean of $x_{1}$, $H_{1}$: the
mean of $x_{0}$ less than the mean of $x_{1}$. In Table 2, the p-value of our
output is $3.59\times 10^{-13}$ and it can be fully convinced that our method
could recognize different people reliably.

Method | Recognition Accuracy | T-test | Time
---|---|---|---
Our Method | 91.73% | $3.59\times 10^{-13}$ | 0.102
Non-rigid ICP($\epsilon$=1) | 51.24% | 0.8244 | 116.719
Non-rigid ICP($\epsilon$=0.1) | 51.65% | 0.0997 | 315.766
Non-rigid ICP($\epsilon$=0.01) | 52.07% | 0.6888 | $1.56\times 10^{3}$

Table 2: Comparison with non-rigid ICP with different parameters for
recognition. Our network’s performance is far better than non-rigid ICP with
any parameter. The t-test shows that the distance of the same person is
significantly lower than the distance of the different person after being
processed by our model.

##  7 Conclusion

In conclusion, this work emphasizes the importance of 3D facial analysis and
introduced effective techniques for analyzing and comparing partial faces. By
using quasi-conformal theory, we achieved accurate parameterization of 3D
partial face meshes with minimal distortion. Our study introduced an automated
method for detecting landmarks and prominent points on partial faces,
eliminating the need for manual annotation. This allowed for precise alignment
of partial faces, enabling comprehensive analysis. By generating Beltrami
coefficients and applying quasi-conformal mapping, we ensured accurate
comparisons between partial faces while preserving bijectivity. The derived
distribution of optimal Beltrami coefficients can be used for metric learning
in partial face registration problems. Extensive experiments have demonstrated
the effectiveness of our approach to landmark detection and partial face
registration. Overall, this research contributes to advancing 3D facial
analysis by providing efficient techniques for parameterization, landmark
detection, and registration of partial faces. These findings have potential
implications in various fields, such as biometrics, computer vision, and
human-computer interaction. Further research can build upon these methods to
improve the accuracy and efficiency of facial analysis.

## Acknowledgement

This work is partially supported by HKRGC GRF (Project ID: 14306721).

## References

  * [1] Brian Amberg, Sami Romdhani, and Thomas Vetter.  Optimal step nonrigid icp algorithms for surface registration.  In 2007 IEEE conference on computer vision and pattern recognition, pages 1–8. IEEE, 2007.
  * [2] Kari Astala, Tadeusz Iwaniec, and Gaven Martin.  Elliptic partial differential equations and quasiconformal mappings in the plane (pms-48).  In Elliptic Partial Differential Equations and Quasiconformal Mappings in the Plane (PMS-48). Princeton University Press, 2008.
  * [3] Amit Bracha, Thomas Dagès, and Ron Kimmel.  On partial shape correspondence and functional maps.  arXiv preprint arXiv:2310.14692, 2023.
  * [4] Alan Brunton, Augusto Salazar, Timo Bolkart, and Stefanie Wuhrer.  Review of statistical shape spaces for 3d data with comparative analysis for human faces.  Computer Vision and Image Understanding, 128:1–17, 2014.
  * [5] Xavier P Burgos-Artizzu, Pietro Perona, and Piotr Dollár.  Robust face landmark estimation under occlusion.  In Proceedings of the IEEE international conference on computer vision, pages 1513–1520, 2013.
  * [6] Dongliang Cao, Paul Roetzer, and Florian Bernard.  Unsupervised learning of robust spectral shape matching.  arXiv preprint arXiv:2304.14419, 2023.
  * [7] Hei-Long Chan, Hoi-Man Yuen, Chun-Ting Au, Kate Ching-Ching Chan, Albert Martin Li, and Lok-Ming Lui.  Quasi-conformal geometry based local deformation analysis of lateral cephalogram for childhood osa classification.  arXiv preprint arXiv:2006.11408, 2020.
  * [8] Qiguang Chen, Zhiwen Li, and Lok Ming Lui.  A deep learning framework for diffeomorphic mapping problems via quasi-conformal geometry applied to imaging.  arXiv preprint arXiv:2110.10580, 2021.
  * [9] Fernando De la Torre and Jeffrey F Cohn.  Facial expression analysis.  Visual analysis of humans: Looking at people, pages 377–409, 2011.
  * [10] Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong.  Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 0–0, 2019.
  * [11] Mathieu Desbrun, Mark Meyer, and Pierre Alliez.  Intrinsic parameterizations of surface meshes.  Comput. Graph. Forum, 21(3):209–218, 2002.
  * [12] Xuanyi Dong, Yi Yang, Shih-En Wei, Xinshuo Weng, Yaser Sheikh, and Shoou-I Yu.  Supervision by registration and triangulation for landmark detection.  2020\.  doi:10.1109/TPAMI.2020.2983935.
  * [13] Xuanyi Dong, Shoou-I Yu, Xinshuo Weng, Shih-En Wei, Yi Yang, and Yaser Sheikh.  Supervision-by-Registration: An unsupervised approach to improve the precision of facial landmark detectors.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 360–368, 2018.
  * [14] Zhen-Hua Feng, Josef Kittler, Muhammad Awais, Patrik Huber, and Xiao-Jun Wu.  Wing loss for robust facial landmark localisation with convolutional neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2235–2245, 2018.
  * [15] Michael S Floater and Kai Hormann.  Surface parameterization: a tutorial and survey.  Advances in Multiresolution for Geometric Modelling, pages 157–186, 2005.
  * [16] Frederick P Gardiner and Nikola Lakic.  Quasiconformal teichmuller theory.  Number 76. American Mathematical Soc., 2000.
  * [17] Xiaojie Guo, Siyuan Li, Jinke Yu, Jiawan Zhang, Jiayi Ma, Lin Ma, Wei Liu, and Haibin Ling.  Pfld: A practical facial landmark detector.  arXiv preprint arXiv:1902.10859, 2019.
  * [18] Yuchen Guo, Qiguang Chen, Gary Choi, and Lok Ming Lui.  Automatic landmark detection and registration of brain cortical surfaces via quasi-conformal geometry and convolutional neural networks.  arXiv preprint arXiv:2208.07010, 2022.
  * [19] Juha Heinonen and Pekka Koskela.  Quasiconformal maps in metric spaces with controlled geometry.  Acta Mathematica, 181(1):1–61, 1998.
  * [20] Xiaobin Hu, Wenqi Ren, John LaMaster, Xiaochun Cao, Xiaoming Li, Zechao Li, Bjoern Menze, and Wei Liu.  Face super-resolution guided by 3d facial priors.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16, pages 763–780. Springer, 2020.
  * [21] Gary B. Huang, Manu Ramesh, Tamara Berg, and Erik Learned-Miller.  Labeled faces in the wild: A database for studying face recognition in unconstrained environments.  Technical Report 07-49, University of Massachusetts, Amherst, October 2007.
  * [22] Kostiantyn Khabarlak and Larysa Koriashkina.  Fast facial landmark detection and applications: A survey.  Journal of Computer Science and Technology, 22(1):e02–e02, 2022.
  * [23] Marek Kowalski, Jacek Naruniec, and Tomasz Trzcinski.  Deep alignment network: A convolutional neural network for robust face alignment.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 88–97, 2017.
  * [24] Alexandros Lattas, Stylianos Moschoglou, Baris Gecer, Stylianos Ploumpis, Vasileios Triantafyllou, Abhijeet Ghosh, and Stefanos Zafeiriou.  Avatarme: Realistically renderable 3d facial reconstruction” in-the-wild”.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 760–769, 2020.
  * [25] Olli Lehto and Kaarlo Ilmari Virtanen.  Quasiconformal mappings in the plane, volume 126.  Citeseer, 1973.
  * [26] Bruno Lévy, Sylvain Petitjean, Nicolas Ray, and Jérome Maillot.  Least squares conformal maps for automatic texture atlas generation.  ACM Trans. Graph., 21(3):362–371, 2002.
  * [27] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In Proceedings of International Conference on Computer Vision (ICCV), 2015.
  * [28] Changwei Luo, Zengfu Wang, Shaobiao Wang, Juyong Zhang, and Jun Yu.  Locating facial landmarks using probabilistic random forest.  IEEE Signal Processing Letters, 22(12):2324–2328, 2015.
  * [29] Riccardo Marin, Arianna Rampini, Umberto Castellani, Emanuele Rodolà, Maks Ovsjanikov, and Simone Melzi.  Spectral shape recovery and analysis via data-driven connections.  International journal of computer vision, 129:2745–2760, 2021.
  * [30] Ioannis Marras, Stefanos Zafeiriou, and Georgios Tzimiropoulos.  Robust learning from normals for 3d face recognition.  In Computer Vision–ECCV 2012. Workshops and Demonstrations: Florence, Italy, October 7-13, 2012, Proceedings, Part II 12, pages 230–239. Springer, 2012.
  * [31] Qiang Meng, Shichao Zhao, Zhida Huang, and Feng Zhou.  Magface: A universal representation for face recognition and quality assessment.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14225–14234, 2021.
  * [32] José Augusto Cadena Moreano and NBLS Palomino.  Global facial recognition using gabor wavelet, support vector machines and 3d face models.  Journal of Advances in Information Technology, 11(3), 2020.
  * [33] Patrick Mullen, Yiying Tong, Pierre Alliez, and Mathieu Desbrun.  Spectral conformal parameterization.  Comput. Graph. Forum, 27(5):1487–1494, 2008.
  * [34] Di Qiu, Ka-Chun Lam, and Lok-Ming Lui.  Computing quasi-conformal folds.  SIAM Journal on Imaging Sciences, 12(3):1392–1424, 2019.
  * [35] Di Qiu and Lok Ming Lui.  Inconsistent surface registration via optimization of mapping distortions.  Journal of Scientific Computing, 83:1–31, 2020.
  * [36] Evangelos Sariyanidi, Hatice Gunes, and Andrea Cavallaro.  Automatic analysis of facial affect: A survey of registration, representation, and recognition.  IEEE transactions on pattern analysis and machine intelligence, 37(6):1113–1133, 2014.
  * [37] Florian Schroff, Dmitry Kalenichenko, and James Philbin.  Facenet: A unified embedding for face recognition and clustering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823, 2015.
  * [38] Alla Sheffer, Emil Praun, and Kenneth Rose.  Mesh parameterization methods and their applications.  Found. Trends Comput. Graph. Vis., 2(2):105–171, 2007.
  * [39] Yi Sun, Ding Liang, Xiaogang Wang, and Xiaoou Tang.  Deepid3: Face recognition with very deep neural networks.  arXiv preprint arXiv:1502.00873, 2015.
  * [40] Yi Sun, Xiaogang Wang, and Xiaoou Tang.  Deeply learned face representations are sparse, selective, and robust.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2892–2900, 2015.
  * [41] Murat Taskiran, Nihan Kahraman, and Cigdem Eroglu Erdem.  Face recognition: Past, present and future (a review).  Digital Signal Processing, 106:102809, 2020.
  * [42] Luan Tran and Xiaoming Liu.  Nonlinear 3d face morphable model.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7346–7355, 2018.
  * [43] Michal Uřičář, Vojtěch Franc, and Václav Hlaváč.  Detector of facial landmarks learned by the structured output svm.  VIsAPP, 12:547–556, 2012.
  * [44] Xinyao Wang, Liefeng Bo, and Li Fuxin.  Adaptive wing loss for robust face alignment via heatmap regression.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6971–6981, 2019.
  * [45] Yue Wu and Qiang Ji.  Facial landmark detection: A literature survey.  International Journal of Computer Vision, 127(2):115–143, 2019.
  * [46] Tarun Yenamandra, Ayush Tewari, Florian Bernard, Hans-Peter Seidel, Mohamed Elgharib, Daniel Cremers, and Christian Theobalt.  i3dmm: Deep implicit 3d morphable model of human heads.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12803–12813, 2021.
  * [47] Mei-Heng Yueh, Wen-Wei Lin, Chin-Tien Wu, and Shing-Tung Yau.  An efficient energy minimization for conformal parameterizations.  J. Sci. Comput., 73(1):203–227, 2017.
  * [48] Wei Zeng, Lok Ming Lui, and Xianfeng Gu.  Surface registration by optimization in constrained diffeomorphism space.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4169–4176, 2014.
  * [49] Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang.  Facial landmark detection by deep multi-task learning.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 94–108. Springer, 2014.
  * [50] Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang.  Learning deep representation for face alignment with auxiliary attributes.  IEEE transactions on pattern analysis and machine intelligence, 38(5):918–930, 2015.
  * [51] Yang Zhao, Yifan Liu, Chunhua Shen, Yongsheng Gao, and Shengwu Xiong.  Mobilefan: Transferring deep hidden representation for face alignment.  Pattern Recognition, 100:107114, 2020.
  * [52] Zhipeng Zhu, Gary PT Choi, and Lok Ming Lui.  Parallelizable global quasi-conformal parameterization of multiply connected surfaces via partial welding.  SIAM Journal on Imaging Sciences, 15(4):1765–1807, 2022.
  * [53] Xu Zou, Sheng Zhong, Luxin Yan, Xiangyun Zhao, Jiahuan Zhou, and Ying Wu.  Learning robust facial landmark detection via hierarchical structured ensemble.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 141–150, 2019.


# Reply to Chen et al.: Parametric methods for cluster inference perform worse
for two-sided t-tests

Anders Eklund<EMAIL_ADDRESS>Hans Knutsson  Thomas E. Nichols  Division
of Medical Informatics, Department of Biomedical Engineering, Linköping
University, Linköping, Sweden  Division of Statistics & Machine Learning,
Department of Computer and Information Science, Linköping University,
Linköping, Sweden  Center for Medical Image Science and Visualization (CMIV),
Linköping University, Linköping, Sweden  Big Data Institute, University of
Oxford, Oxford, United Kingdom  Wellcome Trust Centre for Integrative
Neuroimaging (WIN-FMRIB), University of Oxford, Oxford, United Kingdom
Department of Statistics, University of Warwick, Coventry, United Kingdom

###### Abstract

One-sided t-tests are commonly used in the neuroimaging field, but two-sided
tests should be the default unless a researcher has a strong reason for using
a one-sided test. Here we extend our previous work on cluster false positive
rates, which used one-sided tests, to two-sided tests. Briefly, we found that
parametric methods perform worse for two-sided t-tests, and that non-
parametric methods perform equally well for one-sided and two-sided tests.

###### keywords:

fMRI, false positives, cluster inference, permutation, one-sided, two-sided

##  1 Introduction

Chen et al. [2018] discuss an important topic which is often neglected in the
neuroimaging field, the use of one-sided or two-sided tests and the lack of
multiple comparison correction for two one-sided tests. As mentioned in their
paper, in our work on massive empirical evaluation of task fMRI inference
methods with resting state fMRI [Eklund et al., 2016] we used one-sided tests
(familywise error rate $\alpha_{\mathrm{FWE}}=0.05$). We made this choice for
two reasons. The first reason was simply that for analyses of randomly created
groups of healthy controls, it should make no difference if one uses a one-
sided or a two-sided test. The second reason was more practical. FSL and SPM
both run one-sided tests by default, and we wished to reflect the typical (if
ill-advised) practices of the community. Furthermore, to perform a two-sided
permutation test [Winkler et al., 2014], it would be necessary to run two
permutation tests per group analysis (which would double the processing time),
since normally only the maximum test value over the brain (or the largest
cluster) is saved for every permutation (to form the maximum null
distribution).

##  2 Methods

To investigate if performing a two-sided test (as implemented by two tests at
$\alpha_{\mathrm{FWE}}=0.025$) lead to different false positive rates compared
to a single one-sided test (at $\alpha_{\mathrm{FWE}}$ = 0.05), we performed
new group analyses for a subset of all the parameter settings used in our
previous work [Eklund et al., 2016, 2018]. Specifically, we only performed
two-sample t-tests for the Beijing data [Biswal et al., 2010], using 40
subjects (i.e. 20 subjects per group) and a cluster defining threshold of p =
0.001. All group analyses were performed for 4 mm, 6 mm, 8 mm and 10 mm FWHM
of smoothing. See our recent work [Eklund et al., 2018] for a description of
the six designs (B1, B2, E1, E2, E3, E4) applied to every subject in the first
level analysis.

For FSL, group analyses were only performed using FSL OLS, and not using
FLAME1 (which is the default option); FLAME1 leads to conservative results if
resting state fMRI data is used, while null task fMRI analyses (control-
control) with FLAME1 gives FWE rates comparable to FSL OLS [Eklund et al.,
2016]. For AFNI, we used the new ACF (autocorrelation function) option in
3dClustSim [Cox et al., 2017], which uses a long-tail spatial ACF instead of a
Gaussian one. It should be noted that AFNI provides another function for
cluster thresholding, ETAC (equitable thresholding and clustering) [Cox,
2018], which may perform better than the long-tail ACF function used here, but
we used the ACF approach to be able to compare the two-sided results to our
recent work [Eklund et al., 2018]. Contrary to Chen et al. [2018], we did not
change the cluster defining threshold to p = 0.0005 when performing two one-
sided tests (for SPM, FSL or AFNI), as this represents yet another change in
the inference configuration that we rather leave fixed to facilitate the
comparison of these results to previous one-sided findings.

##  3 Results

Figure 1 shows estimated familywise error rates for one-sided and two-sided
tests, where both should exhibit a nominal 5% familywise false positive rate.
The non-parametric permutation test produces similar results in both cases,
while the parametric methods perform worse for two-sided tests.

Figure 1: _A comparison of empirical familywise error rates for one-sided
(left) and two-sided (right) tests, for a cluster defining threshold of p =
0.001. Designs B1 and B2 represent two block based activity paradigms, while
E1, E2, E3 and E4 represent event related paradigms. Design E4 is randomized
over subjects, while all other designs are the same for all subjects. The
parametric methods perform worse for two one-sided tests at
$\alpha_{\mathrm{FWE}}$ = 0.025, compared to a single one-sided test at
$\alpha_{\mathrm{FWE}}$ = 0.05, while the permutation test produces nominal
results in both cases. _

##  4 Discussion

We have extended our original work on cluster false positive rates [Eklund et
al., 2016, 2018] to two-sided tests, showing that parametric methods perform
worse for two-sided tests. RFT p-values depend on a number of approximations:

  1. 1.

Joint normality over the image,

  2. 2.

Sufficient smoothness for lattice images to behave like continuous processes,

  3. 3.

Homogeneous smoothness (stationarity), so that the null distribution of
cluster size does not vary over space,

  4. 4.

Spatial dependence mostly local, i.e. the spatial autocorrelation function is
proportional to a Gaussian density, and

  5. 5.

Sufficiently high cluster-forming threshold so that the approximate
distribution for cluster size is accurate.

On this last assumption, the control of FWE depends on the accuracy of the
cluster size distribution in its tail. For example, it is of little
consequence if the true cluster size FWE p-value is 0.6 and RFT estimates it
as 0.5; in contrast, two-sided inference demands accuracy in the RFT
approximation down to FWE 0.025, and then any inaccuracies are doubled as both
positive and negative excursions are considered. In our findings, it appears
that modest inaccuracies in the null cluster size distribution corresponding
to FWE 0.05 (see Figure 1 (a), and general tendency to over estimate FWE) grow
into larger inaccuracies when the more stringent FWE level 0.025 is used (the
inference used twice for each result contributing to Figure 1 (b)).

In contrast, the non-parametric permutation test for a two-sample t-test is
only based on the assumption of exchangeability between subjects, and
therefore performs equally well for two one-sided tests at
$\alpha_{\mathrm{FWE}}$ = 0.025.

## Acknowledgements

The authors have no conflict of interest to declare. This study was supported
by Swedish research council grants 2013-5229 and 2017-04889. Funding was also
provided by the Center for Industrial Information Technology (CENIIT) at
Linköping University, and the Knut and Alice Wallenberg foundation project
”Seeing organ function”. Thomas E. Nichols was supported by the Wellcome Trust
(100309/Z/12/Z) and the NIH (R01 EB015611). The Nvidia Corporation, who
donated the Nvidia Quadro P6000 graphics card used to run all permutation
tests, is also acknowledged. This study would not be possible without the
recent data-sharing initiatives in the neuroimaging field. We therefore thank
the Neuroimaging Informatics Tools and Resources Clearinghouse and all of the
researchers who have contributed with resting-state data to the 1,000
Functional Connectomes Project.

## References

  * Biswal et al. [2010] Biswal, B., Mennes, M., …, X. Z., & Milham, M. (2010).  Toward discovery science of human brain function.  PNAS, 107, 4734–4739.
  * Chen et al. [2018] Chen, G., Cox, R. W., Glen, D. R., Rajendra, J. K., Reynolds, R. C., & Taylor, P. A. (2018).  A tail of two sides: Artificially doubled false positive rates in neuroimaging due to the sidedness choice with t-tests.  Human Brain Mapping, .
  * Cox et al. [2017] Cox, R., Chen, G., Glen, D., Reynolds, R., & Taylor, P. (2017).  FMRI Clustering in AFNI: False-Positive Rates Redux.  Brain Connectivity, 7, 152–171.
  * Cox [2018] Cox, R. W. (2018).  Equitable thresholding and clustering.  bioRxiv, 10.1101/295931, .
  * Eklund et al. [2018] Eklund, A., Knutsson, H., & Nichols, T. (2018).  Cluster failure revisited: impact of first level design and physiological noise on cluster false positive rates.  Human Brain Mapping, .
  * Eklund et al. [2016] Eklund, A., Nichols, T., & Knutsson, H. (2016).  Cluster failure: why fMRI inferences for spatial extent have inflated false positive rates.  PNAS, 113, 7900–7905.
  * Winkler et al. [2014] Winkler, A., Ridgway, G., Webster, M., Smith, S., & Nichols, T. (2014).  Permutation inference for the general linear model.  NeuroImage, 92, 381–397.