# Physics-Guided Spoof Trace Disentanglement for Generic Face Anti-Spoofing

Yaojie Liu, and Xiaoming Liu Y. Liu and X. Liu are with the Department of
Computer Science and Engineering, Michigan State University, East Lansing, MI
48824, USA.
Email<EMAIL_ADDRESS>
###### Abstract

Prior studies show that the key to face anti-spoofing lies in the subtle image
pattern, termed “spoof trace”, e.g., color distortion, $3$D mask edge, Moiré
pattern, and many others. Designing a generic face anti-spoofing model to
estimate those spoof traces can improve not only the generalization of the
spoof detection, but also the interpretability of the model’s decision. Yet,
this is a challenging task due to the diversity of spoof types and the lack of
ground truth in spoof traces. In this work, we design a novel adversarial
learning framework to disentangle spoof faces into the spoof traces and the
live counterparts. Guided by physical properties, the spoof generation is
represented as a combination of additive process and inpainting process.
Additive process describes spoofing as spoof material introducing extra
patterns (e.g., moire pattern), where the live counterpart can be recovered by
removing those patterns. Inpainting process describes spoofing as spoof
material fully covering certain regions, where the live counterpart of those
regions has to be “guessed”. We use 3 additive components and 1 inpainting
component to represent traces at different frequency bands. The disentangled
spoof traces can be utilized to synthesize realistic new spoof faces after
proper geometric correction, and the synthesized spoof can be used for
training and improve the generalization of spoof detection. Our approach
demonstrates superior spoof detection performance on $3$ testing scenarios:
known attacks, unknown attacks, and open-set attacks. Meanwhile, it provides a
visually-convincing estimation of the spoof traces. Source code and pre-
trained models will be publicly available upon publication.

###### Index Terms:

Face Anti-Spoofing, Low-level Vision, Weak Supervision, Synthesis, Spoof
Traces, Deep Learning.

##  1 Introduction

In recent years, the vulnerability of face biometric systems has been widely
recognized and increasingly brought attention to the computer vision
community. The attacks to the face biometric systems attempt to deceive the
systems to make wrong identity recognition: either recognize the attackers as
a target person (i.e., impersonation), or cover up the original identity
(i.e., obfuscation).

Figure 1: The proposed approach can detect spoof faces, disentangle the spoof
traces, and reconstruct the live counterparts. It can be applied to diverse
spoof types and estimate distinct traces (e.g., Moiré pattern in replay
attack, artificial eyebrow and wax in makeup attack, color distortion in print
attack, and specular highlights in $3$D mask attack). Zoom in for details.

There are various types of digital and physical attacks, including face
morphing [1, 2, 3], face adversarial attacks [4, 5, 6], face manipulation
attacks (e.g., deepfake, face swap) [7, 8], and face spoofing [9, 10, 11].
Among the above-mentioned attacks, face spoofing is the only physical attack
to deceive the systems, where attackers present faces from spoof mediums, such
as photograph, screen, mask and makeup, instead of a live human. These spoof
mediums can be easily manufactured by ordinary people, and hence they pose
huge threats to face biometric applications such as mobile face unlock,
building access control, and transportation security. Therefore, face
biometric systems need to be secured with face anti-spoofing (FAS) techniques
to distinguish the source of the face before performing the face recognition
task.

Figure 2: The comparison of different deep-learning based face anti-spoofing.
(a) direct FAS only provides a binary decision of spoofness; (b) auxiliary FAS
can provide simple interpretation of spoofness. $\mathbf{M}$ denotes the
auxiliary task, such as depth map estimation; (c) generative FAS can provide
more intuitive interpretation of spoofness, but only for a limited number of
spoof attacks; (d) the proposed method can provide spoof trace estimation for
generic face spoof attacks.

As most face recognition systems are based on a monocular RGB camera,
monocular RGB based face anti-spoofing has been studied for over a decade, and
one of the most common approaches is based on texture analysis [12, 13, 14].
Researchers noticed that presenting faces from spoof mediums introduces
special texture differences, such as color distortions, unnatural specular
highlights, Moiré patterns, etc. Those texture differences are inherent within
spoof mediums and thus hard to remove or camouflage. Conventional approaches
build a feature extractor plus classifier pipeline, such as LBP+SVM and
HOG+SVM [15, 16], and show good performance on several small databases with
constraint environments. In recent years, many works leverage deep learning
techniques and show great progress in face anti-spoofing performance [17, 18,
19, 20, 21]. Deep learning based methods can be generally grouped into $3$
categories: direct FAS, auxiliary FAS, and generative FAS, as illustrated in
Fig. 2. Early works [22, 23] build vanilla CNN with binary output to directly
predict the spoofness of an input face (Fig.2a). Methods [18, 21] propose to
learn an intermediate representation, e.g., depth, rPPG, reflection, instead
of binary classes, which can lead to better generalization and performance
(Fig.2b). [24, 25, 26] additionally attempt to generate the visual patterns
existing in the spoof samples (Fig.2c), providing a more intuitive
interpretation of the sample’s spoofness.

Despite the success, there are still at least three unsolved problems in the
topic of deep learning-based face anti-spoofing. First, most prior works are
designed to tackle limited spoof types, either print/replay or $3$D mask
solely, while a real-world anti-spoofing system may encounter a wide variety
of spoof types including print, replay, various $3$D masks, facial makeup, and
even unseen attack types. Therefore, to better reflect real-world performance,
we need a benchmark to evaluate face anti-spoofing under known attacks,
unknown attacks, and their combination (termed open-set setting). Second, many
approaches formulate face anti-spoofing as a classification/regression
problem, with a single score as the output. Although auxiliary FAS and
generative FAS attempt to offer some extent of interpretation by fixation,
saliency, or noise analysis, there is little understanding on what the exact
differences are between live and spoof, and what patterns the classifier’s
decision is based upon. A better interpretation can be estimating the exact
patterns differentiating a spoof face and its live counterpart, termed spoof
trace. Thirdly, compared with other face analysis tasks such as recognition or
alignment, the data for face anti-spoofing has several limitations. Most FAS
databases are captured in the constraint indoor environment, which has limited
intra-subject variation and environment variation. For some special spoof
types such as makeup and customized silicone mask, they require highly skilled
experts to apply or create, with high cost, which results in very limited
samples (i.e., long-tail data). Thus, how to learn from data with limited
variations or samples is a challenge for FAS.

In this work, we aim to design a face anti-spoofing model that is applicable
to a wide variety of spoof types, termed generic face anti-spoofing. We equip
this model with the ability to explicitly disentangle the spoof traces from
the input faces. Some examples of spoof trace disentanglement are shown in
Fig. 1. This is a challenging objective due to the diversity of spoof traces
and the lack of ground truth during model learning. However, we believe that
fulfilling this objective can bring several benefits:

  1. 1.

Binary classification for face anti-spoofing would harvest any cue that helps
classification, which might include spoof-irrelevant cues such as lighting,
and thus hinder generalization. In contrast, spoof trace disentanglement
explicitly tackles the most fundamental cue in spoofing, upon which the
classification can be more grounded and witness better generalization.

  2. 2.

With the trend of pursuing explainable AI [27, 28], it is desirable for the
face anti-spoofing model to generate the spoof patterns that support its
binary decision, since spoof trace serves as a good visual explanation of the
model’s decision. Certain properties (e.g., severity, methodology) of spoof
attacks might potentially be revealed from the traces.

  3. 3.

Disentangled spoof traces can enable the synthesis of realistic spoof samples,
which addresses the issue of limited training data for the minority spoof
types, such as special $3$D masks and makeup.

As shown in Fig. 2d, we propose a Physics-guided Spoof Trace Disentanglement
(PhySTD) to explore the spoof traces for generic face anti-spoofing. To model
all types of spoofs, we formulate the spoof trace disentanglement as a
combination of additive process and inpainting process. Additive process
describes spoofing as spoof material introducing extra patterns (e.g., moire
pattern), where the live counterpart can be recovered by removing those
patterns. Inpainting process describes spoofing as spoof material fully
covering certain regions of the original face, where the live counterpart of
those regions has to be “guessed” [29, 30]. We further decompose the spoof
traces into frequency-dependent components, so that traces with different
frequency properties can be equally handled. For the network architecture, we
extend a backbone network for auxiliary FAS with a decoder to perform the
disentanglement. With no ground truth of spoof traces, we adopt an overall
GAN-based training strategy. The generator takes an input face, estimates its
spoofness, and disentangles the spoof trace. After obtaining the spoof trace,
we can reconstruct the live counterpart from the spoof and synthesize new
spoof from the live. The synthesized samples are then sent to multiple
discriminators with real samples for adversarial training. The synthesized
spoof samples are further utilized to train the generator in a fully
supervised fashion, thanks to disentangled spoof traces as ground truth for
the synthesized samples. To correct possible geometric discrepancy during
spoof synthesis, we propose a novel $3$D warping layer to deform spoof traces
toward the target live face.

A preliminary version of this work was published in the Proceedings European
Conference on Computer Vision (ECCV) 2020 [31]. We extend the work from three
aspects. $1$) Guided by the physics of how a spoof is generated, we introduce
a spoof generation function (SGF) to model the spoof trace disentanglement as
a combination of additive and inpainting processes. SGF has a better and more
natural modeling of generic spoof attacks, such as paper glass. $2$) Previous
trace components $\\{\mathbf{S},\mathbf{B},\mathbf{C},\mathbf{T}\\}$ are not
supervised hierarchically so that there exists semantic ambiguity. In this
work, we introduce several hierarchical designs in the GAN framework to remedy
such ambiguity. $3$) We propose an open-set testing scenario to further
evaluate the real-world performance for face anti-spoofing models. Both known
and unknown attacks are included in the open-set testing. We perform a side-
by-side comparison between the proposed approach and the state-of-the-art
(SOTA) face anti-spoofing solutions on multiple datasets and protocols.

In summary, the main contributions of this work are as follows:

  * $\bullet$

We for the first time study spoof trace for generic face anti-spoofing, where
a wide variety of spoof types are tackled with one unified framework;

  * $\bullet$

We propose a novel physics-guided model to disentangle spoof traces, and
utilize the spoof traces to synthesize new data samples for enhanced training;

  * $\bullet$

We propose novel protocols for a generic open-set face anti-spoofing;

  * $\bullet$

We achieve SOTA anti-spoofing performance and provide convincing visualization
for a wide variety of spoof types.

##  2 Related Work

Face Anti-Spoofing Face anti-spoofing has been studied for more than a decade
and its development can be roughly divided into three stages. In the early
years, researchers leverage spontaneous human movement, such as eye blinking
and head motion, to detect simple print photograph or static replay attacks
[32, 33]. However, when facing counter attacks, such as print face with eye
region cut, and replaying a face video, those methods would fail. In the
second stage, researchers pay more attention to texture differences between
live and spoof, which are inherent to spoof mediums. Researchers mainly
extract hand-crafted features from the faces, e.g., LBP [12, 15, 34, 35], HoG
[16, 36], SIFT [14] and SURF [13], and train a classifier to split the live
vs. spoof, e.g., SVM and LDA.

Recently, face anti-spoofing solutions equipped with deep learning techniques
have demonstrated significant improvements over the conventional methods.
Methods in [37, 38, 39, 22] train a deep neural network to learn a binary
classification between live and spoof. In [17, 18, 19, 20, 21], additional
supervisions, such as face depth map and rPPG signal, are utilized to help the
network to learn more generalizable features. As the latest approaches
achieving saturated performance on several benchmarks, researchers start to
explore more challenging cases, such as few-shot/zero-shot face anti-spoofing
[19, 40, 41] and domain adaptation in face anti-spoofing [20, 42].

In this work, we aim to solve an interesting yet very challenging problem:
disentangling and visualizing the spoof traces from an input face. A related
work [24] also adopts GAN seeking to estimate the spoof traces. However, they
formulate the traces as low-intensity noises, which is limited to print and
replay attacks only and cannot provide convincing visual results. In contrast,
we explore spoof traces for a much wider range of spoof attacks, visualize
them with novel disentanglement, and also evaluate the proposed method on the
challenging cases, e.g., zero-shot face anti-spoofing.

Disentanglement Learning Disentanglement learning is often adopted to better
represent complex data and features. DR-GAN [43] disentangles a face into
identity and pose vectors for pose-invariant face recognition and view
synthesis. Similarly in gait recognition, [44] disentangles the
representations of appearance, canonical, and pose features from an input gait
video. $3$D reconstruction works [45, 46] also disentangle the representation
of a $3$D face into identity, expressions, poses, albedo, and illuminations.
For image synthesis, [47] disentangles an image into appearance and shape with
U-Net and Variational Auto Encoder (VAE).

Different from [45, 43, 44], we intend to disentangle features that have
different scales and contain geometric information. We leverage the multiple
outputs to represent features at different scales, and adopt multiple-scale
discriminators to properly learn them. Moreover, we propose a novel warping
layer to tackle the geometric discrepancy during the disentanglement and
reconstruction.

Image Trace Modeling Image traces are certain signals existing in the image
that can reveal information about the capturing camera, imaging setting,
environment, and so on. Those signals often have much lower energy compared to
the image content, which needs proper modeling to explore them. [48, 49, 50]
observe the difference of image noises, and use them to recognize the capture
cameras. From the frequency domain, [25] shows the image noises from different
cameras obey different noise distributions. Such techniques are applied to the
field of image forensics, and later [51, 52] propose methods to remove such
traces for image anti-forensics.

Recently, image trace modeling is widely used in image forgery detection and
image adversarial attack detection [53, 54]. In this work, we attempt to
explore the traces of spoof face presentation. Due to different spoof mediums,
spoof traces show large variations in content, intensity, and frequency
distribution. We propose to disentangle the traces as additive traces and
inpainting trace. And for additive traces, we further decompose them based on
different frequency bands.

Figure 3: Overview of the proposed Physics-guided Spoof Trace Disentanglement
(PhySTD).

##  3 Physics-based Spoof Trace Disentanglement

###  3.1 Problem Formulation

Let the domain of live faces be denoted as
$\mathcal{L}\\!\subset\\!\mathbb{R}^{N\\!\times\\!N\\!\times\\!3}$ and spoof
faces as $\mathcal{S}\\!\subset\\!\mathbb{R}^{N\\!\times\\!N\\!\times\\!3}$,
where $N$ is the image size. We intend to obtain not only the correct
prediction (live vs. spoof) of the input face, but also a convincing
estimation of the spoof trace and live face reconstruction. To represent the
spoof trace, our preliminary version assumes an additive relation between live
and spoof, and uses $4$ trace components
$\\{\mathbf{S},\mathbf{B},\mathbf{C},\mathbf{T}\\}$ at different frequency
bands as:

$\mathbf{I}_{\textit{spoof}}=(1+\lfloor\mathbf{S}\rfloor_{n_{1}})\mathbf{I}_{\textit{live}}+\lfloor\mathbf{B}\rfloor_{n_{1}}+\lfloor\mathbf{C}\rfloor_{n_{2}}+\mathbf{T},$
(1)

where $\mathbf{S},\mathbf{B}$ represent low-frequency traces, $\mathbf{C}$
represents mid-frequency ones, and $\mathbf{T}$ represents high-frequency
ones. $\lfloor\cdot\rfloor$ is the low bandpass filtering operation, and in
practice, we achieve this by downsampling the original image and upsampling it
back. In the previous setting, $n_{1}\\!=\\!1$ and $n_{2}\\!=\\!64$. Compared
to the simple representation with only a single component [24], this multi-
scale representation of $\\{\mathbf{S},\mathbf{B},\mathbf{C},\mathbf{T}\\}$
can largely improve disentanglement quality and suppress undesired artifacts
due to its coarse-to-fine process. The model is designed to provide a valid
estimation of spoof traces $\\{\mathbf{S},\mathbf{B},\mathbf{C},\mathbf{T}\\}$
without respective ground truth. Our preliminary version [31] aims to find a
minimum intensity change that transfers an input face to the live domain:

$\operatorname*{arg\,min}_{\hat{\mathbf{I}}}\|\mathbf{I}-\hat{\mathbf{I}}\|_{F}\;s.t.\;\mathbf{I}\in(\mathcal{S}\cup\mathcal{L})\;\text{and}\;\hat{\mathbf{I}}\in\mathcal{L},$
(2)

where $\mathbf{I}$ is the source face, $\hat{\mathbf{I}}$ is the target face
to be optimized, and $\mathbf{I}-\hat{\mathbf{I}}$ is defined as the spoof
trace. When the source face is live $\mathbf{I}_{\text{live}}$,
$\mathbf{I}-\hat{\mathbf{I}}$ should be $0$ as $\mathbf{I}$ is already in
$\mathcal{L}$. When the source face is spoof $\mathbf{I}_{\text{spoof}}$,
$\mathbf{I}-\hat{\mathbf{I}}$ should be regularized to prevent unnecessary
changes such as identity shift.

Despite the effectiveness of this representation, there are still two
drawbacks: First, the spoof trace disentanglement is mainly formulated as an
additive processing. The optimization of Eqn. 2 limits the trace intensity,
and the reconstruction for spoof regions with large appearance divergence
might be sub-optimal, such as spoof glasses or mask. For those spoof regions,
the physical relationship between the live and the spoof is better described
as replacement rather than addition; Second, while our preliminary version
representing the traces with hierarchical components, these components are
learned with losses on their summation. Without careful supervision, the
learned components can be ambiguous in their semantic meanings, e.g., the
high-frequency component may include low-frequency information.

To address the first drawback, we introduce a spoof generation function (SGF)
as an additive process followed by an inpainting process:

$\mathbf{I}_{\textit{spoof}}=(1-\mathbf{P})(\mathbf{I}_{\textit{live}}+\mathbf{T}_{A})+\mathbf{P}\cdot\mathbf{T}_{P},$
(3)

where $\mathbf{T}_{A}\\!\in\mathbb{R}^{N\\!\times\\!N\\!\times\\!3}$ indicates
the traces from additive process, $\mathbf{T}_{P}$ indicates the traces from
inpainting process, and
$\mathbf{P}\in\mathbb{R}^{N\\!\times\\!N\\!\times\\!1}$ denotes the inpainting
region. Given a spoof face, one may reconstruct the live counterpart by
inversing Eqn. 3:

$\hat{\mathbf{I}}_{\textit{live}}=(1-\mathbf{P})(\mathbf{I}_{\textit{spoof}}-\mathbf{T}_{\text{A}})+\mathbf{P}\cdot(\hat{\mathbf{I}}_{\textit{live}}+\mathbf{I}_{\textit{spoof}}-\mathbf{T}_{P}),$
(4)

As the inpainting physically replaces content, the spoof trace
$\mathbf{T}_{P}$ in the inpainting region $\mathbf{P}$ is identical to the
spoof image $\mathbf{I}_{\textit{spoof}}$ in the same region, and thus both
cancel out in the second term of Eqn. 4. We further rename the
$\hat{\mathbf{I}}_{\textit{live}}$ in the second term as $\mathbf{I}_{P}$ to
indicate the inpainting content within the inpainting region that should be
estimated from the model. Therefore, the reconstruction of the live image
becomes:

$\hat{\mathbf{I}}_{\textit{live}}=(1-\mathbf{P})(\mathbf{I}_{\textit{spoof}}-\mathbf{T}_{\text{A}})+\mathbf{P}\cdot\mathbf{I}_{P},$
(5)

where
$\mathbf{T}_{A}\\!=\\!\lfloor\mathbf{B}\rfloor_{n_{1}}\\!+\\!\lfloor\mathbf{C}\rfloor_{n_{2}}\\!+\\!\mathbf{T}$
denotes the additive trace represented by three hierarchical components.
$n_{1}$ and $n_{2}$ are set to be $32$ and $128$ respectively. With a larger
$n_{1}$, the effect of component $\mathbf{S}$ in the preliminary version can
be incorporated into $\mathbf{B}$, and hence we remove $\mathbf{S}$ for
simplicity. Besides the additive traces, the model is further required to
estimate the inpainting region $\mathbf{P}$ and inpainting live content
$\mathbf{I}_{P}$. $\mathbf{I}_{P}$ is estimated based on the rest of the live
facial region without intensity constraint. We use a function $G(\cdot)$ to
represent the reconstruction process of Eqn. 5. Accordingly, the optimization
of Eqn. 2 is re-formulated by replacing $\hat{\mathbf{I}}$ with Eqn. 5 as:

$\begin{gathered}\operatorname*{arg\,min}_{\mathbf{T}_{A},\mathbf{P},\mathbf{I}_{P}}\|\mathbf{I}-(1-\mathbf{P})(\mathbf{I}-\mathbf{T}_{A})-\mathbf{P}\cdot\mathbf{I}_{P}\|_{F}\;\\\
\rightarrow\operatorname*{arg\,min}_{\mathbf{T}_{A},\mathbf{P},\mathbf{I}_{P}}\|(1-\mathbf{P})\mathbf{T}_{A}\|_{F}+\|\mathbf{P}\cdot(\mathbf{I}-\mathbf{I}_{P})\|_{F}.\;\\\
\end{gathered}$ (6)

As we do not wish to impose any intensity constraint on $\mathbf{I}_{P}$, the
final objective is formulated as:

$\begin{gathered}\operatorname*{arg\,min}_{\mathbf{T}_{A},\mathbf{P}}\|(1\\!-\\!\mathbf{P})\mathbf{T}_{A}\|_{F}+\lambda\|\mathbf{P}\|_{F}\;s.t.\;\mathbf{I}\\!\in\\!\mathcal{S}\cup\mathcal{L},\hat{\mathbf{I}}\in\mathcal{L},\end{gathered}$
(7)

where $\lambda$ is a weight to balance two terms. In addition, based on Eqn.
3, we can define another function $G^{-}(\cdot)$ to synthesize new spoof
faces, by transferring the spoof traces from $\mathbf{I}^{i}$ to
$\mathbf{I}^{j}$:

$\hat{\mathbf{I}}_{\textit{spoof}}^{i\rightarrow
j}=G^{-}(\mathbf{I}^{j}|\mathbf{I}^{i})=(1-\mathbf{P}^{i})(\mathbf{I}^{j}+\mathbf{T}_{\text{A}}^{i})+\mathbf{P}^{i}\cdot\mathbf{I}^{i}.$
(8)

Note that $\mathbf{T}_{P}$ in Eqn. 3 has been replaced with $\mathbf{I}^{i}$
since the spoof image $\mathbf{I}^{i}$ contains the spoof trace for the
inpainting region.

Figure 4: The proposed PhySTD network architecture. Except the last layer,
each conv and transposed conv is concatenated with a batch normalizstion layer
and a leaky ReLU layer. ‘k3c64s2’ indicates the kernel size of $3\times 3$,
the convolution channel of $64$ and the stride of $2$.

Estimating $\\{\mathbf{T}_{\text{A}},\mathbf{P},\mathbf{I}_{P}\\}$ from an
input face $\mathbf{I}$ is termed as spoof trace disentanglement. Given that
no ground truth of traces is available, this disentanglement can be achieved
via generative adversarial based training. As shown in Fig. 3, the proposed
Physics-guided Spoof Trace Disentanglement (PhySTD) consists of a generator
and discriminator. Given an input image, the generator is designed to predict
the spoofness (represented by the pseudo depth map) as well as estimate the
additive traces $\\{\mathbf{B},\mathbf{C},\mathbf{T}\\}$ and the inpainting
components $\\{\mathbf{P},\mathbf{I}_{P}\\}$. With the traces, we can apply
function $G(\cdot)$ to reconstruct the live counterpart and function
$G^{-}(\cdot)$ to synthesize new spoof faces. We adopt a set of discriminators
at multiple image resolutions to distinguish the real faces
$\\{\mathbf{I}_{\textit{live}},\mathbf{I}_{\textit{spoof}}\\}$ with the
synthetic faces
$\\{\hat{\mathbf{I}}_{\textit{live}},\hat{\mathbf{I}}_{\textit{spoof}}\\}$. To
remedy the semantic ambiguity during $\\{\mathbf{B},\mathbf{C},\mathbf{T}\\}$
learning, three trace component combinations, $\\{\mathbf{B}\\}$,
$\\{\mathbf{B},\mathbf{C}\\}$, and $\\{\mathbf{B},\mathbf{C},\mathbf{T}\\}$,
will contribute to the synthesis of live reconstruction at one particular
resolution, which is then supervised by a respective discriminator (details in
Sec.3.3). To learn a proper inpainting region $\mathbf{P}$, we leverage both
the prior knowledge and the information from the additive traces.

In the rest of this section, we present the details of the generator, the
discriminators, the details of face reconstruction and synthesis, and the
losses and training steps used in PhySTD.

###  3.2 Disentanglement Generator

As shown in Fig. 4, the disentanglement generator consists of a backbone
encoder, a spoof trace decoder and a depth estimation network. The backbone
encoder aims to extract multi-scale features, the depth estimation network
leverages the features to estimate the facial depth map, and a spoof trace
decoder to estimate the additive trace components
$\\{\mathbf{B},\mathbf{C},\mathbf{T}\\}$ and the inpainting components
$\\{\mathbf{P},\mathbf{I}_{P}\\}$. The depth map and the spoof traces will be
used to compute the final spoofness score.

Figure 5: The visualization of image decomposition for different input faces:
(a) live face (b) $3$D mask attack (c) replay attack (d) print attack.

Backbone encoder Backbone encoder extracts features from the input images for
both depth map estimation and spoof trace disentanglement. As shown in our
preliminary work [31], the spoof traces consists of components from different
frequency bands: low-frequency traces includes color distortion, mid-frequency
traces includes makeup strikes, and high-frequency traces includes Moiré
patterns and mask edges. However, a vanilla CNN model might overlook high-
frequency traces since the energy of high-frequency traces is often much
weaker than that of low-frequency traces. In order to encourage the network to
equally regard traces with different physical properties, we explicitly
decompose the image into three elements
$\\{\mathbf{I}_{\mathbf{B}},\mathbf{I}_{\mathbf{C}},\mathbf{I}_{\mathbf{T}}\\}$
as:

$\begin{split}\mathbf{I}_{\mathbf{B}}=&\lfloor\mathbf{I}\rfloor_{n_{1}},\\\
\mathbf{I}_{\mathbf{C}}=&\lfloor\mathbf{I}\rfloor_{n_{2}}-\lfloor\mathbf{I}\rfloor_{n_{1}},\\\
\mathbf{I}_{\mathbf{T}}=&\mathbf{I}-\lfloor\mathbf{I}\rfloor_{n2},\\\
\end{split}$ (9)

where $n_{1}\\!=\\!32$, $n_{2}\\!=\\!128$ and the image size $N\\!=\\!256$. In
addition, we amplify the value in
$\mathbf{I}_{\mathbf{C}},\mathbf{I}_{\mathbf{T}}$ by two constants $15$ and
$25$, and then feed the concatenation of three elements to the backbone
network. Fig. 5 provides the visualization of image decomposition. We observe
that the traces that are less distinct in the original images become more
highlighted in the $\mathbf{I}_{\mathbf{T}}$ component: $3$D mask and replay
attack bring unique patterns different with the live face pattern, while print
attack is lacking of necessary high frequency details. Semantically,
$\mathbf{I}_{\mathbf{B}},\mathbf{I}_{\mathbf{C}},\mathbf{I}_{\mathbf{T}}$
share the same frequency domains with $\mathbf{B},\mathbf{C},\mathbf{T}$
respectively, and thus the decomposition potentially eases the learning of
$\mathbf{B},\mathbf{C},\mathbf{T}$.

After that, the encoder progressively downsamples the decomposed image
components $3$ times to obtain features
$\mathbf{F}_{1}\\!\in\\!\mathbb{R}^{128\\!\times\\!128\\!\times\\!64}$,
$\mathbf{F}_{2}\\!\in\\!\mathbb{R}^{64\\!\times\\!64\\!\times\\!96}$,
$\mathbf{F}_{3}\\!\in\\!\mathbb{R}^{32\\!\times\\!32\\!\times\\!128}$ via conv
layers.

Spoof trace decoder The decoder upsamples the feature $\mathbf{F}_{3}$ with
transpose conv layers back to the input face size $256$. The last layer
outputs both additive traces $\\{\mathbf{B},\mathbf{C},\mathbf{T}\\}$ and
inpainting components $\\{\mathbf{P},\mathbf{I}_{P}\\}$. Similar to U-Net
[55], we apply the short-cut connection between the backbone encoder and
decoder to bypass the multiple scale details for a high-quality trace
estimation.

Depth estimation network We still recognize the importance of the
discriminative supervision used in auxiliary FAS, and thus introduce a depth
estimation network to perform the pseudo-depth estimation for face anti-
spoofing, as proposed in [18]. The depth estimation network takes the
concatenated features of $\mathbf{F}_{1}$, $\mathbf{F}_{2}$, $\mathbf{F}_{3}$
from the backbone encoder and $\mathbf{U}_{3}$ from the decoder as input. The
features are put through a spatial attention mechanism from [56] and resize to
the same size of $K=32$. It outputs a face depth map
$\mathbf{M}\\!\in\\!\mathbb{R}^{32\\!\times\\!32}$, where the depth values are
normalized within $[0,1]$. Regarding the number of parameters, both spoof
trace decoder and depth estimation network are light weighed, while the
backbone network is much heavier. With more network layers being shared to
tackle both depth estimation and spoof trace disentanglement, the knowledge
learnt from spoof trace disentanglement can be better shared with depth
estimation task, which can lead to a better anti-spoofing performance.

Final scoring In the testing phase, we use the norm of the depth map and the
intensity of spoof traces for real vs. spoof classification:

$\text{score}=\frac{1}{2K^{2}}\|\mathbf{M}\|_{1}\\!+\\!\frac{\alpha_{0}}{2N^{2}}(\|\mathbf{B}\|_{1}\\!+\\!\|\mathbf{C}\|_{1}\\!+\\!\|\mathbf{T}\|_{1}\\!+\\!\|\mathbf{P}\|_{1}),$
(10)

where $\alpha_{0}$ is the weight for the spoof trace.

###  3.3 Reconstruction and Synthesis

There are multiple options to use the disentangled spoof traces: $1$) live
reconstruction, $2$) spoof synthesis, and $3$) “harder” sample synthesis,
which will be described below respectively.

Live reconstruction: Based on Eqn. 5, we propose a hierarchical reconstruction
of the live face counterpart from the input images. To reconstruct faces at a
certain resolution, each additive trace is included only if its frequency
domain is lower than the target resolution. We apply
$\\{\textit{hi},\textit{mid},\textit{low}\\}$ three resolution settings as:

$\begin{split}\hat{\mathbf{I}}_{\textit{hi}}=&(1-\mathbf{P})(\mathbf{I}-\lfloor\mathbf{B}\rfloor_{n_{1}}-\lfloor\mathbf{C}\rfloor_{n_{2}}-\mathbf{T})+\mathbf{P}\cdot\mathbf{I}_{P},\\\
\hat{\mathbf{I}}_{\textit{mid}}=&(1-\mathbf{P})(\lfloor\mathbf{I}\rfloor_{n_{2}}-\lfloor\mathbf{B}\rfloor_{n_{1}}-\lfloor\mathbf{C}\rfloor_{n_{2}})+\mathbf{P}\cdot\mathbf{I}_{P},\\\
\hat{\mathbf{I}}_{\textit{low}}=&(1-\mathbf{P})(\lfloor\mathbf{I}\rfloor_{n_{1}}-\lfloor\mathbf{B}\rfloor_{n_{1}})+\mathbf{P}\cdot\mathbf{I}_{P}.\\\
\end{split}$ (11)

Spoof synthesis: Based on Eqn. 8, we can obtain a new spoof face via applying
the spoof traces disentangled from a spoof face $\mathbf{I}_{i}$ to a live
face $\mathbf{I}_{j}$. However, spoof traces may contain face-dependent
content associated with the original spoof subject. Directly applying them to
a new face with different shapes or poses may result in mis-alignment and
strong visual implausibility. Therefore, the spoof trace should go through a
geometry correction before performing this synthesis. We propose an online
$3$D warping layer and will introduce it in the following subsection.

“Harder” sample synthesis: The disentangled spoof traces can not only
reconstruct live and synthesize new spoof, but also synthesize “harder” spoof
samples by removing or amplifying part of the spoof traces. We can tune one or
some of the trace elements $\\{\mathbf{B},\mathbf{C},\mathbf{T},\mathbf{P}\\}$
to make the spoof sample to become “less spoofed”, which is thus closer to a
live face since the spoof traces are weakened. Such spoof data can be regarded
as harder samples and may benefit the generalization of the disentanglement
generator. For instance, while removing the low frequency element $\mathbf{B}$
from a replay spoof trace, the generator may be forced to rely on other
elements such as high-level texture patterns. To synthesize the “harder”
sample $\hat{\mathbf{I}}_{\textit{hard}}$, we follow Eqn. 8 with two minor
changes: 1) generate $3$ random weights between $[0,1]$ and multiple each with
one component of $\\{\mathbf{B},\mathbf{C},\mathbf{T}\\}$; 2) randomly remove
the inpainting process (i.e., set $\mathbf{P}=0$) with a probability of $0.5$.
Compared with other methods, such as brightness and contrast change [57],
reflection and blurriness effect [21], or $3$D distortion [58], our approach
can introduce more realistic and effective data samples, as shown in Sec. 4.

####  3.3.1 Online $3$D Warping Layer

We propose an online $3$D warping layer to correct the shape discrepancy. To
obtain the warping, previous methods in [59, 18] use offline face swapping and
pre-computed dense offset respectively, where both methods are non-
differentiable as well as memory intensive. In contrast, our warping layer is
designed to be both differentiable and computationally efficient, which is
necessary for online synthesis during the training.

Figure 6: The online $3$D warping layer. (a) Given the corresponding dense
offset, we warp the spoof trace and add them to the target live face to create
a new spoof. E.g. pixel $(x,y)$ with offset $(3,5)$ is warped to
pixel$(x+3,y+5)$ in the new image. (b) To obtain a dense offsets from the
spare offsets of the selected face shape vertices, Delaunay triangulation
interpolation is adopted.

First, the live reconstruction of a spoof face $\mathbf{I}^{i}$ can be
expressed as:

$G^{i}=G(\mathbf{I}^{i})[\mathbf{p}_{0}],$ (12)

where $\mathbf{p}^{0}=\\{(0,0),(0,1),...,(255,255)\\}\in\mathbb{R}^{256\times
256\times 2}$ enumerates pixel locations in $\mathbf{I}^{i}$. To align the
spoof traces while synthesizing a new spoof face, a dense offset
$\Delta\mathbf{p}^{i\rightarrow j}\in\mathbb{R}^{256\times 256\times 2}$ is
required to indicate the deformation between face $\mathbf{I}^{i}$ and face
$\mathbf{I}^{j}$. A discrete deformation can be acquired from the distances of
the corresponding facial landmarks between two faces. During the data
preparation, we use [60] to fit a $3$DMM model and extract the $2$D locations
of $Q$ facial vertices for each face:

$\mathbf{s}=\\{(x_{0},y_{0}),(x_{1},y_{1}),...,(x_{N},y_{N})\\}\in\mathbb{R}^{Q\times
2}.$ (13)

A sparse offset on the corresponding vertices can then be computed two faces
as $\Delta\mathbf{s}^{i\rightarrow j}=\mathbf{s}^{j}-\mathbf{s}^{i}$. To
convert the sparse offset $\Delta\mathbf{s}^{i\rightarrow j}$ to the dense
offset $\Delta\mathbf{p}^{i\rightarrow j}$, we apply a triangulation
interpolation:

$\Delta\mathbf{p}^{i\rightarrow
j}=\text{Tri}(\mathbf{p}^{0},\mathbf{s}^{i},\Delta\mathbf{s}^{i\rightarrow
j}),$ (14)

where $\text{Tri}(\cdot)$ is the interpolation, $\mathbf{s}^{i}$ denotes the
vertex locations, $\Delta\mathbf{s}^{i\rightarrow j}$ are the vertex values,
and we adopt Delaunay triangulation. The warping operation can be denoted as:

$G^{-i\rightarrow
j}=G^{-}(\mathbf{I}^{j}|\mathbf{I}^{i})[\mathbf{p}^{0}+\Delta\mathbf{p}^{i\rightarrow
j}],$ (15)

where the offset $\Delta\mathbf{p}^{i\rightarrow j}$ applies to all subject
$i$ related elements $\\{\mathbf{T}_{A}^{i},\mathbf{I}^{i},\mathbf{P}^{i}\\}$.
Since the offset $\Delta\mathbf{p}^{i\rightarrow j}$ is typically composed of
fractional numbers, we implement the bilinear interpolation to sample the
fractional pixel locations. We select $Q=140$ vertices to cover the face
region so that they can represent non-rigid deformation, due to pose and
expression. As the pixel values in the warped face are a linear combination of
pixel values of the triangulation vertices, this entire process is
differentiable. This process is illustrated in Fig. 6.

Input: live faces $\mathbf{I}_{\textit{live}}$ and facial landmarks
$\mathbf{s}_{\textit{live}}$, spoof faces $\mathbf{I}_{\textit{spoof}}$ and
facial landmarks $\mathbf{s}_{\textit{spoof}}$, ground truth depth map
$\mathbf{M}_{0}$, preliminary mask $\mathbf{P}_{0}$;

Output: reconstructed live $\hat{\mathbf{I}}_{\textit{live}}$, synthesized
spoof $\hat{\mathbf{I}}_{\textit{spoof}}$, spoof traces
$\\{\mathbf{T}_{A}^{\textit{l}}$, $\mathbf{P}^{\textit{l}}$,
$\mathbf{I}_{P}^{\textit{l}},\mathbf{T}_{A}^{\textit{s}}$,
$\mathbf{P}^{\textit{s}}$, $\mathbf{I}_{P}^{\textit{s}}\\}$, depth maps
$\\{\mathbf{M}^{\textit{l}}$, $\mathbf{M}^{\textit{s}}\\}$ ;

while  _iteration $<$ max_iteration_ do

// training step $1$

1: compute $\mathbf{T}_{A}^{\textit{l}}$, $\mathbf{P}^{\textit{l}}$,
$\mathbf{I}_{P}^{\textit{l}}$ $\leftarrow$ $G(\mathbf{I}_{\textit{live}})$ and
compute $\mathbf{T}_{A}^{\textit{s}}$, $\mathbf{P}^{\textit{s}}$,
$\mathbf{I}_{P}^{\textit{s}}$ $\leftarrow$ $G(\mathbf{I}_{\textit{spoof}})$;

2: estimate the depth map $\mathbf{M}^{\textit{l}}$,
$\mathbf{M}^{\textit{s}}$;

3: compute losses $L_{\textit{depth}}$, $L_{P}$, $L_{R}$;

// training step $2$

4: compute $\hat{\mathbf{I}}_{\textit{low}}$,
$\hat{\mathbf{I}}_{\textit{mid}}$, $\hat{\mathbf{I}}_{\textit{hi}}$ from
$\mathbf{T}_{A}^{s}$, $\mathbf{P}^{s}$, $\mathbf{I}_{P}^{s}$ and
$\mathbf{I}_{\textit{spoof}}$ (Eqn. 4);

5: compute warping offset $\Delta\mathbf{p}^{s\rightarrow l}$ from
$\mathbf{s}_{\textit{live}}$, $\mathbf{s}_{\textit{spoof}}$ (Eqn. 14);

6: compute $\hat{\mathbf{I}}_{\textit{spoof}}$ from warped
$\mathbf{T}_{A}^{s\rightarrow l}$, $\mathbf{P}^{s\rightarrow l}$ and
$\mathbf{I}_{\textit{live}}$ (Eqn. 15);

7: send $\mathbf{I}_{\textit{live}}$, $\mathbf{I}_{\textit{spoof}}$,
$\hat{\mathbf{I}}_{\textit{low}}$, $\hat{\mathbf{I}}_{\textit{mid}}$,
$\hat{\mathbf{I}}_{\textit{hi}}$, $\hat{\mathbf{I}}_{\textit{spoof}}$ to
discriminators;

8: compute the adversarial loss for generator $L_{\textit{G}}$ and for
discriminators $L_{\textit{D}}$;

// training step $3$

9: create harder samples $\mathbf{I}_{\textit{hard}}$ from
$\mathbf{T}_{A}^{s\rightarrow l}$, $\mathbf{P}^{s\rightarrow l}$ and
$\mathbf{I}_{\textit{live}}$ with random perturbation on traces;

10: compute $\mathbf{T}_{A}^{h}$, $\mathbf{P}^{h}$, $\mathbf{I}_{P}^{h}$
$\leftarrow$ $G(\mathbf{I}_{\textit{hard}})$;

11: compute depth map $\mathbf{M}^{h}$ for $\mathbf{I}_{\textit{hard}}$;

12: compute losses $L_{S}$, $L_{H}$;

// back propagation

13: back-propagate the losses from step $3,8,12$ to corresponding parts and
update the network;

end while

Algorithm 1 PhySTD Training Iteration.

###  3.4 Multi-scale Discriminators

Motivated by [61], we adopt multiple discriminators at different resolutions
(e.g., $32$, $96$, and $256$) in our GAN architecture. We follow the design of
PatchGAN [62], which essentially is a fully convolutional network. Fully
convolutional networks are shown to be effective to not only synthesize high-
quality images [62, 61], but also tackle face anti-spoofing problems [18]. For
each discriminator, we adopt the same structure but do not share the weights.

As shown in Fig. 4, we use in total $4$ discriminators in our work: $D_{1}$,
working in the lowest resolution of $32$, focuses on low frequency elements
since the higher-frequency traces are erased by downsampling. $D_{2}$, working
at the resolution of $96$, focuses on the middle level content pattern.
$D_{3}$ and $D_{4}$, working on the highest resolution of $256$, focus on the
fine texture details. Our preliminary version resizes real and synthetic
samples $\\{\mathbf{I},\hat{\mathbf{I}}\\}$ to different resolutions and
assign to each discriminator. To remove semantic ambiguity and provide
correspondence to the trace components, we instead assign the hierarchical
reconstruction from Eqn. 11 to the discriminators: we send low frequency pairs
$\\{\mathbf{I}_{\textit{live}},\hat{\mathbf{I}}_{\textit{low}}\\}$ to $D_{1}$,
middle frequency pairs
$\\{\mathbf{I}_{\textit{live}},\hat{\mathbf{I}}_{\textit{mid}}\\}$ to $D_{2}$,
high frequency pairs
$\\{\mathbf{I}_{\textit{live}},\hat{\mathbf{I}}_{\textit{hi}}\\}$ to $D_{3}$,
and real/synthetic spoof
$\\{\mathbf{I}_{\textit{spoof}},\hat{\mathbf{I}}_{\textit{spoof}}\\}$ to
$D_{4}$. Each discriminator outputs a $1$-channel map in the range of $[0,1]$,
where $0$ denotes fake and $1$ denotes real.

###  3.5 Loss Functions and Training Steps

We utilize multiple loss functions to supervise the learning of depth maps and
spoof traces. Each training iteration consists of three training steps. We
first introduce the loss function, followed by how they are used in the
training steps.

Depth map loss: We follow the auxiliary FAS [18] to estimate an auxiliary
depth map $\mathbf{M}$, where the depth ground truth $\mathbf{M}_{0}$ for a
live face contains face-like shape and the depth for spoof should be zero. We
apply the $\mathcal{L}$-$1$ norm on this loss as:

$L_{\textit{depth}}=\frac{1}{K^{2}}\mathbb{E}_{i\sim\mathcal{L}\cup\mathcal{S}}{\lVert}\mathbf{M}^{i}-\mathbf{M}^{i}_{0}{\rVert}_{F},$
(16)

where $K\\!=\\!32$ is the size of $\mathbf{M}$. We apply the dense face
alignment [60] to estimate the $3$D shape and render the depth ground truth
$\mathbf{M}_{0}$.

Figure 7: Preliminary mask $\mathbf{P}_{0}$ for the negative term in
inpainting mask loss. White pixels denote $1$ and black pixels denote $0$.
White indicates the area should not be inpainted. $\mathbf{P}_{0}$ for: (a)
print, replay; (b) 3D mask and makeup; (c) partial attacks that cover the eye
portion; (d) partial attacks that cover the mouth portion.

Adversarial loss for $G$: We employ the LSGANs [63] on reconstructed live
faces and synthesized spoof faces. It encourages the reconstructed live to
look similar to real live from domain $\mathcal{L}$, and the synthesized spoof
faces to look similar to faces from domain $\mathcal{S}$:

$\displaystyle
L_{G}=\mathbb{E}_{i\sim\mathcal{L},j\sim\mathcal{S}}\Big{[}{\lVert}D_{1}(\hat{\mathbf{I}}_{\textit{low}}^{j})\\!-\\!\mathbf{1}{\rVert}_{F}^{2}+{\lVert}D_{2}(\hat{\mathbf{I}}_{\textit{mid}}^{j})\\!-\\!\mathbf{1}{\rVert}_{F}^{2}+$
(17)
$\displaystyle{\lVert}D_{3}(\hat{\mathbf{I}}_{\textit{hi}}^{j})\\!-\\!\mathbf{1}{\rVert}_{F}^{2}+{\lVert}D_{4}(\hat{\mathbf{I}}_{\textit{spoof}}^{j\rightarrow
i})\\!-\mathbf{1}{\rVert}_{F}^{2}\Big{]}.$

Adversarial loss for $D$: The adversarial loss for discriminators encourages
$D(\cdot)$ to distinguish between real live vs. reconstructed live, and real
spoof vs. synthesized spoof:

$\displaystyle
L_{D}=\mathbb{E}_{i\sim\mathcal{L},j\sim\mathcal{S}}\Big{[}{\lVert}D_{1}(\mathbf{I}^{i})\\!-\\!\mathbf{1}{\rVert}_{F}^{2}+{\lVert}D_{2}(\mathbf{I}^{i})\\!-\\!\mathbf{1}{\rVert}_{F}^{2}+$
(18)
$\displaystyle{\lVert}D_{3}(\mathbf{I}^{i})\\!-\\!\mathbf{1}{\rVert}_{F}^{2}+{\lVert}D_{4}(\mathbf{I}^{j})\\!-\mathbf{1}{\rVert}_{F}^{2}+{\lVert}D_{1}(\hat{\mathbf{I}}_{\textit{low}}^{j}){\rVert}_{F}^{2}+$
$\displaystyle{\lVert}D_{2}(\hat{\mathbf{I}}_{\textit{mid}}^{j}){\rVert}_{F}^{2}+{\lVert}D_{3}(\hat{\mathbf{I}}_{\textit{hi}}^{j}){\rVert}_{F}^{2}+{\lVert}D_{4}(\hat{\mathbf{I}}_{\textit{spoof}}^{j\rightarrow
i}){\rVert}_{F}^{2}\Big{]}.$

Inpainting mask loss: The ground truth inpainting region for all spoof attacks
is barely possible to obtain, hence a fully supervised training [64] for
inpainting mask is out of the question. However, we may still leverage the
prior knowledge of spoof attacks to facilitate the estimation of inpainting
masks. The inpainting mask loss consists of a positive term and a negative
term. First, the positive term encourages certain region to be inpainted. As
the goal of inpainting process is to allow certain region to change without
intensity constraint, the region with larger magnitude of additive traces
would have a higher probability to be inpainted. Hence, the positive term
adopts a $\mathcal{L}$-$2$ norm between the inpainting region $\mathbf{P}$ and
the region where the additive trace is larger than a threshold $\beta$.

Second, the negative term discourages certain region to be inpainted. While
the ground truth inpainting mask is unknown, it’s straightforward to mark a
large portion of region that should not be inpainted. For instance, the
inpainting region for funny eye glasses should not appear in the lower part of
a face. Hence, we provide a preliminary mask $\mathbf{P}_{0}$ to indicate the
not-to-be-inpainted region, and adopt a normalized $\mathcal{L}$-$2$ norm on
the masked inpainting region $\mathbf{P}\cdot\mathbf{P}_{0}$ as the negative
term. The preliminary mask $\mathbf{P}_{0}$ is illustrated in Fig. 7. Overall,
the inpainting mask loss is formed as:

$L_{P}=\mathbb{E}_{\mathbf{i}\sim\mathcal{S}}\Big{[}{\lVert}\mathbf{P}^{i}-(\mathbf{T}_{A}^{i}>\beta){\rVert}_{F}^{2}+\frac{{\lVert}\mathbf{P}^{i}\cdot\mathbf{P}_{0}^{i}{\rVert}_{F}^{2}}{{\lVert}\mathbf{P}_{0}^{i}{\rVert}_{F}^{2}}\Big{]}.$
(19)

Trace regularization: Based on Eqn. 6 with $\lambda=1$, we regularize the
intensity of additive traces $\\{\mathbf{B},\mathbf{C},\mathbf{T}\\}$ and
inpainting region $\mathbf{P}$. The regularizer loss is denoted as:

$L_{R}=\mathbb{E}_{\mathbf{i}\sim\mathcal{L}\cup\mathcal{S}}\Big{[}{\lVert}\mathbf{B}{\rVert}_{F}^{2}+{\lVert}\mathbf{C}{\rVert}_{F}^{2}+{\lVert}\mathbf{T}{\rVert}_{F}^{2}+{\lVert}\mathbf{P}{\rVert}_{F}^{2}\Big{]}.$
(20)

Synthesized spoof loss: Synthesized spoof data come with ground truth spoof
traces. As a result, we are able to define a supervised pixel loss for the
generator to disentangle the exact spoof traces that were added:

$L_{S}=\mathbb{E}_{\mathbf{i}\sim\mathcal{L},\mathbf{j}\sim\mathcal{S}}\Big{[}{\lVert}G(\lceil
G^{-j\rightarrow i}\rceil)-\lceil G^{j\rightarrow
i}\rceil{\rVert}_{F}^{1}\Big{]},$ (21)

where $G^{j\rightarrow i}$ is the overall effect of
$\\{\mathbf{P}^{j},\mathbf{I}_{P}^{j},\mathbf{B}^{j},\mathbf{C}^{j},\mathbf{T}^{j}\\}$
after warping to subject $i$, and $\lceil\cdot\rceil$ is the `stop_gradient`
operation. Without stopping the gradient, $G^{j\rightarrow i}$ may collapse to
$0$.

Depth map loss for “harder” samples: We send the “harder” synthesized spoof
data to depth estimation network to improve the data diversity, and hope to
increase the FAS model’s generalization:

$L_{\textit{H}}=\frac{1}{K^{2}}\mathbb{E}_{i\sim\hat{\mathcal{S}}}\Big{[}{\lVert}\mathbf{M}^{i}-\mathbf{M}_{0}^{i}{\rVert}_{F}\Big{]},$
(22)

where $\hat{\mathcal{S}}$ denotes the domain of synthesized spoof faces.

Training steps and total loss: Each training iteration has $3$ training steps.
In the training step $1$, live faces $\mathbf{I}_{\textit{live}}$ and spoof
faces $\mathbf{I}_{\textit{spoof}}$ are fed into generator $G(\cdot)$ to
disentangle the spoof traces. The spoof traces are used to reconstruct the
live counterpart $\hat{\mathbf{I}}_{\textit{live}}$ and synthesize new spoof
$\hat{\mathbf{I}}_{\textit{spoof}}$. The generator is updated with respect to
the depth map loss $L_{\textit{depth}}$, adversarial loss $L_{G}$, inpainting
mask loss $L_{P}$, and regularizer loss $L_{R}$:

$L=\alpha_{1}L_{\textit{depth}}+\alpha_{2}L_{G}+\alpha_{3}L_{P}+\alpha_{4}L_{R}.$
(23)

In the training step $2$, the discriminators are supervised with the
adversarial loss $L_{D}$ to compete with the generator. In the training step
$3$, $\mathbf{I}_{\textit{live}}$ and $\hat{\mathbf{I}}_{\textit{hard}}$ are
fed into the generator with the ground truth label and trace to minimize the
synthesized spoof loss $L_{S}$ and depth map loss $L_{H}$:

$L=\alpha_{5}L_{S}+\alpha_{6}L_{H},$ (24)

where $\alpha_{1}$-$\alpha_{6}$ are the weights to balance the multitask
training. To note that, we send the original live faces
$\mathbf{I}_{\textit{live}}$ with $\hat{\mathbf{I}}_{\textit{hard}}$ for a
balanced mini-batch, which is important when computing the moving average in
the batch normalization layer. We execute all $3$ steps in each minibatch
iteration, but reduce the learning rate for discriminator step by half. The
whole training process is depicted in Alg. 1.

##  4 Experiments

In this section, we first introduce the experimental setup, and then present
the results in the known, unknown, and open-set spoof scenarios, with
comparisons to respective baselines. Next, we quantitatively evaluate the
spoof traces by performing a spoof medium classification, and conduct an
ablation study on each design in the proposed method. Finally, we provide
visualization results on the spoof trace disentanglement, new spoof synthesis
and t-SNE visualization.

TABLE I: The evaluation on four protocols in OULU-NPU. Bold indicates the best
score in each protocol.

Protocol | Method | APCER (%) | BPCER (%) | ACER (%)
---|---|---|---|---
1 |  STASN[21] | $1.2$ | $2.5$ | $1.9$
Auxiliary [18] | $1.6$ | $1.6$ | $1.6$
DeSpoof [24] | $1.2$ | $1.7$ | $1.5$
DRL [65] | $1.7$ | $0.8$ | $1.3$
STDN [31] | $0.8$ | $1.3$ | $1.1$
CDCN [56] | $0.4$ | $1.7$ | $1.0$
HMP [66] | $0.0$ | $1.6$ | $0.8$
CDCN++ [56] | $0.4$ | 0.0 | 0.2
| Ours | 0.0 | $0.8$ | $0.4$
2 |  DeSpoof [24] | $4.2$ | $4.4$ | $4.3$
Auxiliary [18] | $2.7$ | $2.7$ | $2.7$
DRL [65] | $1.1$ | $3.6$ | $2.4$
STASN[21] | $4.2$ | 0.3 | $2.2$
STDN [31] | $2.3$ | $1.6$ | $1.9$
HMP [66] | $2.6$ | $0.8$ | $1.7$
CDCN [56] | $0.4$ | $1.7$ | $1.5$
CDCN++ [56] | $1.8$ | $0.8$ | $1.3$
| Ours | 1.2 | $1.3$ | 1.3
3 |  DeSpoof [24] | $4.0\pm 1.8$ | $3.8\pm 1.2$ | $3.6\pm 1.6$
Auxiliary [18] | $2.7\pm 1.3$ | $3.1\pm 1.7$ | $2.9\pm 1.5$
STDN [31] | $1.6\pm 1.6$ | $4.0\pm 5.4$ | $2.8\pm 3.3$
STASN[21] | $4.7\pm 3.9$ | $\textbf{0.9}\pm\textbf{1.2}$ | $2.8\pm 1.6$
HMP [66] | $2.8\pm 2.4$ | $2.3\pm 2.8$ | $2.5\pm 1.1$
CDCN [56] | $2.4\pm 1.3$ | $2.2\pm 2.0$ | $2.3\pm 1.4$
DRL [65] | $2.8\pm 2.2$ | $1.7\pm 2.6$ | $2.2\pm 2.2$
CDCN++ [56] | $1.7\pm 1.5$ | $2.0\pm 1.2$ | $\textbf{1.8}\pm\textbf{0.7}$
| Ours | $\textbf{1.7}\pm\textbf{1.4}$ | $2.2\pm 3.5$ | $1.9\pm 2.3$
4 |  Auxiliary [18] | $9.3\pm 5.6$ | $10.4\pm 6.0$ | $9.5\pm 6.0$
STASN[21] | $6.7\pm 10.6$ | $8.3\pm 8.4$ | $7.5\pm 4.7$
CDCN [56] | $4.6\pm 4.6$ | $9.2\pm 8.0$ | $6.9\pm 2.9$
DeSpoof [24] | $5.1\pm 6.3$ | $6.1\pm 5.1$ | $5.6\pm 5.7$
HMP [66] | $2.9\pm 4.0$ | $7.5\pm 6.9$ | $5.2\pm 3.7$
CDCN++ [56] | $4.2\pm 3.4$ | $5.8\pm 4.9$ | $5.0\pm 2.9$
DRL [65] | $5.4\pm 2.9$ | $\textbf{3.3}\pm\textbf{6.0}$ | $4.8\pm 6.4$
STDN [31] | $2.3\pm 3.6$ | $5.2\pm 5.4$ | $3.8\pm 4.2$
| Ours | $\textbf{2.3}\pm\textbf{3.6}$ | $4.2\pm 5.4$ | $\textbf{3.6}\pm\textbf{4.2}$

###  4.1 Experimental Setup

Databases We conduct experiments on three major databases: Oulu-NPU [67], SiW
[18], and SiW-M [19]. Oulu-NPU and SiW include print/replay attacks, while
SiW-M includes $13$ spoof types. We follow all the existing testing protocols
and compare with SOTA methods. Similar to most prior works, we only use the
face region for training and testing.

Evaluation metrics Two common metrics are used in this work for comparison:
EER and APCER/BPCER/ACER. EER describes the theoretical performance and
predetermines the threshold for making decisions. APCER/BPCER/ACER[68]
describe the practical performance given a predetermined threshold. For both
evaluation metrics, lower value means better performance. The threshold for
APCER/BPCER/ACER is computed from either training set or validation set. In
addition, we also report the True Detection Rate (TDR) at a given False
Detection Rate (FDR). This metric describes the spoof detection rate at a
strict tolerance to live errors, which is widely used to evaluate real-world
systems [69]. In this work, we report TDR at FDR$=0.5\%$. For TDR, the higher
the better.

Parameter setting PhySTD is implemented in Tensorflow with an initial learning
rate of $5e$-$5$. We train in total $150,000$ iterations with a batch size of
$8$, and decrease the learning rate by a ratio of $10$ every $45,000$
iterations. We initialize the weights with $[0,0.02]$ normal distribution.
$\\{\alpha_{1},\alpha_{2},\alpha_{3},\alpha_{4},\alpha_{5},\alpha_{6}\\}$ are
set to be $\\{100,5,1,1e$-$4,10,1\\}$, and $\beta=0.1$. $\alpha_{0}$ is
empirically determined from the training or validation set. We use the open-
source face alignment [70] and $3$DMM fitting [60] to crop the face and
provide $140$ landmarks.

TABLE II: The evaluation on three protocols in SiW Dataset. We compare with
the top $7$ performances.

Protocol | Method | APCER (%) | BPCER (%) | ACER (%)
---|---|---|---|---
1 |  Auxiliary[18] | $3.6$ | $3.6$ | $3.6$
STASN[21] | $-$ | $-$ | $1.0$
Meta-FAS-DR[41] | $0.5$ | $0.5$ | $0.5$
HMP [66] | $0.6$ | $0.2$ | $0.5$
DRL [65] | $0.1$ | $0.5$ | $0.3$
CDCN [56] | $0.1$ | $0.2$ | $0.1$
CDCN++ [56] | $0.1$ | $0.2$ | $0.1$
Ours | 0.0 | 0.0 | 0.0
2 |  Auxiliary[18] | $0.6\pm 0.7$ | $0.6\pm 0.7$ | $0.6\pm 0.7$
Meta-FAS-DR[41] | $0.3\pm 0.3$ | $0.3\pm 0.3$ | $0.3\pm 0.3$
STASN[21] | $-$ | $-$ | $0.3\pm 0.1$
HMP [66] | $0.1\pm 0.2$ | $0.2\pm 0.0$ | $0.1\pm 0.1$
DRL [65] | $0.1\pm 0.2$ | $0.1\pm 0.1$ | $0.1\pm 0.0$
CDCN [56] | $0.0\pm 0.0$ | $0.1\pm 0.1$ | $0.1\pm 0.0$
CDCN++ [56] | $0.0\pm 0.0$ | $0.1\pm 0.1$ | $0.0\pm 0.1$
Ours | $\textbf{0.0}\pm\textbf{0.0}$ | $\textbf{0.0}\pm\textbf{0.0}$ | $\textbf{0.0}\pm\textbf{0.0}$
3 |  STASN[21] | $-$ | $-$ | $12.1\pm 1.5$
Auxiliary[18] | $8.3\pm 3.8$ | $8.3\pm 3.8$ | $8.3\pm 3.8$
Meta-FAS-DR[41] | $8.0\pm 5.0$ | $7.4\pm 5.7$ | $7.7\pm 5.3$
DRL [65] | $9.4\pm 6.1$ | $1.8\pm 2.6$ | $5.6\pm 4.4$
HMP [66] | $2.6\pm 0.9$ | $2.3\pm 0.5$ | $2.5\pm 0.7$
CDCN [56] | $2.4\pm 1.3$ | $2.2\pm 2.0$ | $2.3\pm 1.4$
CDCN++ [56] | $\textbf{1.7}\pm\textbf{1.5}$ | $\textbf{2.0}\pm\textbf{1.2}$ | $\textbf{1.8}\pm\textbf{0.7}$
Ours | $13.1\pm 9.4$ | $1.6\pm 0.6$ | $7.4\pm 4.3$

###  4.2 Anti-Spoofing for Known Spoof Types

Oulu-NPU Oulu-NPU[67] is a commonly used face anti-spoofing benchmark due to
its high-quality data and challenging testing protocols. Tab. I shows our
anti-spoofing performance on Oulu-NPU, compared with SOTA algorithms. Our
method achieves the best overall performance on this database. Compared with
our preliminary version [31], we demonstrate improvements in all $4$
protocols, with significant improvement on protocol $1$ and protocol $3$,
i.e., reducing the ACER by $63.6\%$ and $32.1\%$ respectively. Compared with
the SOTA, our approach achieves similar best performances on the first three
protocols and outperforms the SOTA on the fourth protocol, which is the most
challenging one. To note that, in protocol $3$ and protocol $4$, the
performances of testing camera $6$ are much lower than those of cameras
$1$-$5$: the ACER for camera $6$ are $6.4\%$ and $10.2\%$, while the average
ACER for the other cameras are $1.0\%$ and $2.0\%$ respectively. Compared with
other cameras, we notice that camera $6$ has stronger sensor noises and our
model recognizes them as unknown spoof traces, which leads to an increased
false negative rate (i.e., BPCER). How to separate sensor noises from spoof
traces can be an important future research topic.

TABLE III: The evaluation and ablation study on SiW-M Protocol I: known spoof
detection.

Metrics(%) | Method | Replay | Print | 3D Mask |  | Makeup |  | Partial Attacks | Overall
---|---|---|---|---|---|---|---|---|---
Half | Silic. | Trans. | Paper | Mann. |  | Ob. | Im. | Cos. |  | Funny. | Papergls. | Paper
ACER | Auxiliary[18] | $5.1$ | $5.0$ | $5.0$ | $10.2$ | $5.0$ | $9.8$ | $6.3$ |  | $19.6$ | $5.0$ | $26.5$ |  | $5.5$ | $5.2$ | $5.0$ | $6.3$
SDTN[31] | $3.2$ | $3.1$ | $3.0$ | $9.0$ | $3.0$ | $3.4$ | $4.7$ |  | 3.0 | $3.0$ | $24.5$ |  | $4.1$ | $3.7$ | $3.0$ | $4.1$
Step$1$ | $6.1$ | $5.4$ | $5.4$ | $5.4$ | $5.4$ | $5.4$ | $5.4$ |  | $22.7$ | $5.4$ | $26.8$ |  | $5.4$ | $5.5$ | $5.4$ | $10.9$
Step$1$+Step$2$ w/ single trace | $8.7$ | $7.8$ | $7.8$ | $7.8$ | $7.8$ | $7.8$ | $7.8$ |  | $25.0$ | $7.9$ | $28.8$ |  | $7.8$ | $7.8$ | $7.8$ | $13.8$
Step$1$+Step$2$ | $4.1$ | $3.9$ | $3.9$ | $3.9$ | $4.0$ | $3.9$ | $4.0$ |  | $13.5$ | $4.0$ | $25.1$ |  | $3.9$ | $3.9$ | $3.9$ | $4.6$
Step$1$+Step$2$+Step$3$ (Ours) | 3.2 | 1.4 | 1.0 | 2.3 | 1.3 | 2.9 | 2.5 |  | $12.4$ | 1.2 | 18.5 |  | 1.7 | 0.4 | 1.6 | 2.8
EER | Auxiliary[18] | $4.7$ | $0.0$ | $1.6$ | $10.5$ | $4.6$ | $10.0$ | $6.4$ |  | $12.7$ | $0.0$ | $19.6$ |  | $9.3$ | $7.5$ | $0.0$ | $6.7$
SDTN[31] | $2.1$ | $2.2$ | $0.0$ | $7.2$ | $0.1$ | $3.9$ | $4.8$ |  | 0.0 | $0.0$ | $19.6$ |  | $5.3$ | $5.4$ | 0.0 | $4.8$
Step$1$ | $3.8$ | $2.7$ | $1.5$ | $2.7$ | $1.9$ | 1.8 | $2.4$ |  | $15.1$ | $0.7$ | $28.7$ |  | $4.1$ | $4.9$ | $1.0$ | $4.3$
Step$1$+Step$2$ w/ single trace | $6.7$ | $5.3$ | $0.8$ | 1.5 | $1.4$ | $3.3$ | $3.2$ |  | $21.5$ | $1.0$ | $27.1$ |  | $6.5$ | $6.1$ | $1.5$ | $5.8$
Step$1$+Step$2$ | $2.4$ | $3.1$ | $0.4$ | $2.6$ | $1.2$ | $3.0$ | $2.4$ |  | $9.5$ | $0.4$ | $23.5$ |  | $1.1$ | $0.5$ | $0.6$ | $2.8$
Step$1$+Step$2$+Step$3$ (Ours) | 2.5 | 1.0 | 0.0 | $2.1$ | 1.0 | $1.9$ | 2.2 |  | $8.2$ | 0.0 | 18.5 |  | 0.8 | 0.0 | $0.4$ | 2.5
| SDTN[31] | 90.1 | $76.1$ | $80.7$ | $71.5$ | $62.3$ | $74.4$ | $85.0$ |  | 100 | $100$ | $33.8$ |  | $49.6$ | $30.6$ | $97.7$ | $70.4$
TPR@ | Step$1$ | $43.8$ | $43.3$ | $47.2$ | $44.5$ | $62.9$ | $54.8$ | $55.4$ |  | $16.7$ | $90.6$ | $31.5$ |  | $60.3$ | $56.7$ | $77.1$ | $59.3$
FNR=$.5\%$ | Step$1$+Step$2$ w/ single trace | $58.9$ | $76.8$ | $97.6$ | 94.2 | $94.9$ | $66.3$ | $78.3$ |  | $13.3$ | $94.1$ | $49.1$ |  | $62.4$ | $58.5$ | $92.1$ | $74.8$
| Step$1$+Step$2$ | $84.7$ | $74.7$ | $100$ | $70.1$ | 96.6 | $77.5$ | $89.6$ |  | $36.9$ | $100$ | $40.1$ |  | $96.3$ | $99.4$ | $99.4$ | $89.7$
| Step$1$+Step$2$+Step$3$ (Ours) | $85.7$ | 85.4 | 100 | $76.6$ | $96.3$ | 80.2 | 93.8 |  | $41.1$ | 100 | 55.8 |  | 98.1 | 100 | 99.8 | 91.2

TABLE IV: The evaluation on SiW-M Protocol II: unknown spoof detection.

Metrics | Method | Replay | Print | 3D Mask |  | Makeup |  | Partial Attacks | Average
---|---|---|---|---|---|---|---|---|---
(%) | Half | Silic. | Trans. | Paper | Mann. |  | Ob. | Im. | Cos. |  | Fun. | Papergls. | Paper
APCER |  Auxiliary[18] | $23.7$ | $7.3$ | $27.7$ | $18.2$ | $97.8$ | $8.3$ | $16.2$ |  | $100.0$ | $18.0$ | $16.3$ |  | $91.8$ | $72.2$ | $0.4$ | $38.3\pm 37.4$
LBP+SVM [67] | $19.1$ | $15.4$ | $40.8$ | $20.3$ | $70.3$ | $0.0$ | $4.6$ |  | $96.9$ | $35.3$ | $\mathbf{11.3}$ |  | $53.3$ | $58.5$ | $0.6$ | $32.8\pm 29.8$
DTL[19] | 1.0 | $0.0$ | $0.7$ | $24.5$ | $58.6$ | $0.5$ | $3.8$ |  | $73.2$ | $13.2$ | $12.4$ |  | $17.0$ | $17.0$ | $0.2$ | $17.1\pm 23.3$
CDCN [56] | $8.2$ | $6.9$ | $8.3$ | $7.4$ | $20.5$ | $5.9$ | $5.0$ |  | $43.5$ | $1.6$ | $14.0$ |  | $24.5$ | $18.3$ | $1.2$ | $12.7\pm 11.7$
SDTN[31] | $1.6$ | $\mathbf{0.0}$ | $\mathbf{0.5}$ | $\mathbf{7.2}$ | $9.7$ | $0.5$ | $\mathbf{0.0}$ |  | $96.1$ | $0.0$ | $21.8$ |  | $\mathbf{14.4}$ | $\mathbf{6.5}$ | $0.0$ | $12.2\pm 26.1$
CDCN++[56] | $9.2$ | $6.0$ | $4.2$ | $7.4$ | $18.2$ | $\mathbf{0.0}$ | $5.0$ |  | $39.1$ | $0.0$ | $14.0$ |  | $23.3$ | $14.3$ | $0.0$ | $10.8\pm\ 11.2$
HMP[66] | $12.4$ | $5.2$ | $8.3$ | $9.7$ | $13.6$ | $\mathbf{0.0}$ | $2.5$ |  | $\mathbf{30.4}$ | $0.0$ | $12.0$ |  | $22.6$ | $15.9$ | $1.2$ | $\mathbf{10.3}\pm\mathbf{9.1}$
Ours | $10.0$ | $4.9$ | $5.3$ | $16.7$ | $\mathbf{3.5}$ | $2.0$ | $2.8$ |  | $92.8$ | $\mathbf{0.0}$ | $37.5$ |  | $33.7$ | $23.2$ | $0.2$ | $17.9\pm 25.8$
BPCER |  LBP+SVM [67] | $22.1$ | $21.5$ | $21.9$ | $21.4$ | $20.7$ | $23.1$ | $22.9$ |  | $21.7$ | $12.5$ | $22.2$ |  | $18.4$ | $20.0$ | $22.9$ | $21.0\pm 2.9$
DTL[19] | $18.6$ | $11.9$ | $29.3$ | $12.8$ | $13.4$ | $8.5$ | $23.0$ |  | $11.5$ | $9.6$ | $16.0$ |  | $21.5$ | $22.6$ | $16.8$ | $16.6\pm 6.2$
SDTN[31] | $14.0$ | $14.6$ | $13.6$ | $18.6$ | $18.1$ | $8.1$ | $13.4$ |  | $10.3$ | $9.2$ | $17.2$ |  | $27.0$ | $35.5$ | $11.2$ | $16.2\pm 7.6$
CDCN[56] | $9.3$ | $8.5$ | $13.9$ | $10.9$ | $21.0$ | $\mathbf{3.1}$ | $7.0$ |  | $45.0$ | $2.3$ | $16.2$ |  | $26.4$ | $20.9$ | $5.4$ | $14.6\pm 11.7$
CDCN++[56] | $12.4$ | $8.5$ | $14.0$ | $13.2$ | $19.4$ | $7.0$ | $6.0$ |  | $45.0$ | $1.6$ | $14.0$ |  | $24.8$ | $20.9$ | $3.9$ | $14.6\pm 11.4$
HMP[66] | $13.2$ | $6.2$ | $13.1$ | $10.8$ | $16.3$ | $3.9$ | $\mathbf{2.3}$ |  | $34.1$ | $\mathbf{1.6}$ | $13.9$ |  | $23.2$ | $17.1$ | $2.3$ | $12.2\pm 9.4$
Auxiliary[18] | $10.1$ | $6.5$ | $10.9$ | $11.6$ | $\mathbf{6.2}$ | $7.8$ | $9.3$ |  | $11.6$ | $9.3$ | $7.1$ |  | $\mathbf{6.2}$ | $8.8$ | $10.3$ | $8.9\pm 2.0$
Ours | $\mathbf{3.8}$ | $\mathbf{6.3}$ | $\mathbf{4.4}$ | $\mathbf{5.5}$ | $11.3$ | $3.5$ | $6.0$ |  | $\mathbf{6.6}$ | $1.8$ | $\mathbf{2.7}$ |  | $6.5$ | $\mathbf{8.0}$ | $\mathbf{1.1}$ | $\mathbf{5.7}\pm\mathbf{2.8}$
ACER |  LBP+SVM [67] | $20.6$ | $18.4$ | $31.3$ | $21.4$ | $45.5$ | $11.6$ | $13.8$ |  | $59.3$ | $23.9$ | $16.7$ |  | $35.9$ | $39.2$ | $11.7$ | $26.9\pm 14.5$
Auxiliary[18] | $16.8$ | $6.9$ | $19.3$ | $14.9$ | $52.1$ | $8.0$ | $12.8$ |  | $55.8$ | $13.7$ | $\mathbf{11.7}$ |  | $49.0$ | $40.5$ | $5.3$ | $23.6\pm 18.5$
DTL[19] | $9.8$ | $6.0$ | $15.0$ | $18.7$ | $36.0$ | $4.5$ | $13.4$ |  | $48.1$ | $11.4$ | $14.2$ |  | $19.3$ | $19.8$ | $8.5$ | $16.8\pm 11.1$
CDCN[56] | $8.7$ | $7.7$ | $11.1$ | $\mathbf{9.1}$ | $20.7$ | $4.5$ | $5.9$ |  | $44.2$ | $2.0$ | $15.1$ |  | $25.4$ | $19.6$ | $3.3$ | $13.6\pm 11.7$
SDTN[31] | $7.8$ | $7.3$ | $7.1$ | $12.9$ | $13.9$ | $4.3$ | $6.7$ |  | $53.2$ | $4.6$ | $19.5$ |  | $20.7$ | $21.0$ | $5.6$ | $14.2\pm 13.2$
CDCN++ [56] | $10.8$ | $7.3$ | $9.1$ | $10.3$ | $18.8$ | $3.5$ | $5.6$ |  | $42.1$ | $0.8$ | $14.0$ |  | $24.0$ | $17.6$ | $1.9$ | $12.7\pm 11.2$
HMP[66] | $12.8$ | $5.7$ | $10.7$ | $10.3$ | $14.9$ | $\mathbf{1.9}$ | $\mathbf{2.4}$ |  | $\mathbf{32.3}$ | $\mathbf{0.8}$ | $\mathbf{12.9}$ |  | $22.9$ | $16.5$ | $1.7$ | $\mathbf{11.2}\pm\mathbf{9.2}$
Ours | $\mathbf{6.9}$ | $\mathbf{5.6}$ | $\mathbf{4.8}$ | $11.1$ | $\mathbf{7.4}$ | $2.7$ | $4.4$ |  | $49.7$ | $0.9$ | $20.1$ |  | $\mathbf{20.1}$ | $\mathbf{15.6}$ | $\mathbf{0.6}$ | $11.5\pm 13.2$
EER |  LBP+SVM [67] | $20.8$ | $18.6$ | $36.3$ | $21.4$ | $37.2$ | $7.5$ | $14.1$ |  | $51.2$ | $19.8$ | $16.1$ |  | $34.4$ | $33.0$ | $7.9$ | $24.5\pm 12.9$
Auxiliary[18] | $14.0$ | $4.3$ | $11.6$ | $12.4$ | $24.6$ | $7.8$ | $10.0$ |  | $72.3$ | $10.1$ | $\mathbf{9.4}$ |  | $21.4$ | $18.6$ | $4.0$ | $17.0\pm 17.7$
DTL[19] | $10.0$ | $\mathbf{2.1}$ | $14.4$ | $18.6$ | $26.5$ | $5.7$ | $9.6$ |  | $50.2$ | $10.1$ | $13.2$ |  | $19.8$ | $20.5$ | $8.8$ | $16.1\pm 12.2$
CDCN[56] | $8.2$ | $7.8$ | $8.3$ | $\mathbf{7.4}$ | $20.5$ | $5.9$ | $5.0$ |  | $47.8$ | $1.6$ | $14.0$ |  | $24.5$ | $18.3$ | $1.1$ | $13.1\pm 12.6$
SDTN[31] | $7.6$ | $3.8$ | $8.4$ | $13.8$ | $14.5$ | $5.3$ | $4.4$ |  | $35.4$ | $0.0$ | $19.3$ |  | $21.0$ | $20.8$ | $1.6$ | $12.0\pm 10.0$
CDCN++[56] | $9.2$ | $5.6$ | $\mathbf{4.2}$ | $11.1$ | $19.3$ | $5.9$ | $5.0$ |  | $43.5$ | $0.0$ | $14.0$ |  | $23.3$ | $14.3$ | $\mathbf{0.0}$ | $11.9\pm 11.8$
HMP[66] | $13.4$ | $5.2$ | $8.3$ | $9.7$ | $13.6$ | $5.8$ | $\mathbf{2.5}$ |  | $\mathbf{33.8}$ | $0.0$ | $14.0$ |  | $23.3$ | $16.6$ | $1.2$ | $11.3\pm 9.5$
Ours | $\mathbf{5.2}$ | $4.4$ | $4.4$ | $10.1$ | $\mathbf{8.6}$ | $\mathbf{2.6}$ | $4.3$ |  | $47.2$ | $\mathbf{0.0}$ | $19.6$ |  | $\mathbf{18.6}$ | $\mathbf{12.4}$ | $0.7$ | $\mathbf{10.6}\pm\mathbf{12.6}$
TPR@ |  SDTN[31] | $45.0$ | $40.5$ | $45.7$ | $36.7$ | $11.7$ | $40.9$ | $74.0$ |  | $0.0$ | $67.5$ | $16.0$ |  | $13.4$ | $9.4$ | $62.8$ | $35.7\pm 23.9$
FNR=$.5\%$ | Ours | $\mathbf{55.1}$ | $\mathbf{46.4}$ | $\mathbf{57.3}$ | $\mathbf{65.1}$ | $\mathbf{33.0}$ | $\mathbf{91.7}$ | $\mathbf{76.7}$ |  | $\mathbf{0.0}$ | $\mathbf{100.0}$ | $\mathbf{46.4}$ |  | $\mathbf{31.8}$ | $\mathbf{15.4}$ | $\mathbf{97.7}$ | $\mathbf{53.7}\pm\mathbf{31.8}$

SiW SiW[18] is another recent high-quality database. It includes fewer capture
cameras but more spoof mediums and environment variations, such as pose,
illumination, and expression. The comparison on three protocols is shown in
Tab. II. We outperform the previous works on the first two protocols and rank
in the middle on protocol $3$. Protocol $3$ aims to test the performance of
unknown spoof detection, where the model is trained on one spoof attack (print
or replay) and tested on the other. As we can see from Fig.8-9, the traces of
print and replay are significantly different, where the replay traces are more
on the high-frequency part (i.e., trace component T) and the print traces are
more on the low-frequency part (i.e., trace component S). These pattern
divergence leads to the adaption gap of our method while training on one
attack and testing on the other.

SiW-M SiW-M[19] contains a large diversity of spoof types, including print,
replay, $3$D mask, makeup, and partial attacks. This allows us to have a
comprehensive evaluation of the proposed approach with different spoof
attacks. To use SiW-M for known spoof detection, we randomly split the data of
all types into train/test set with a ratio of $60\%$ vs. $40\%$, and the
results are shown in Tab. III. Compared to the preliminary version [31], our
method outperforms on most spoof types as well as the overall EER performance
by $47.9\%$ relatively, which demonstrates the superiority of our anti-
spoofing on known spoof attacks.

For experiments on SiW-M (protocol I, II, and III), we additionally report the
TPR at FNR equal to $0.5\%$. While EER and ACER provide the theoretical
evaluation, the users in real-world applications care more about the true
spoof detection rate under a given live detection error rate, and hence TPR
can better reflect how well the model can detect one or a few spoof attacks in
practices. As shown in Tab. III, we improve the overall TDR of our preliminary
version [31] by $29.5\%$.

TABLE V: The evaluation on SiW-M Protocol III: openset spoof detection.

Metrics | Method | Replay | Print | 3D Mask |  | Makeup |  | Partial Attacks | Overall
---|---|---|---|---|---|---|---|---|---
(%) | Half | Silic. | Trans. | Paper | Mann. |  | Ob. | Im. | Cos. |  | Funny. | Papergls. | Paper
ACER | Auxiliary[18] | $6.7$ | $5.6$ | $8.5$ | $7.5$ | $11.6$ | $6.7$ | $6.4$ |  | $8.9$ | $5.7$ | $6.1$ |  | $14.3$ | $15.9$ | $5.4$ | $8.4\pm 3.4$
Ours | 4.7 | 3.5 | 3.4 | 3.3 | 6.4 | 2.6 | 3.8 |  | 7.0 | 2.3 | 3.2 |  | 10.7 | 7.3 | 3.2 | $\textbf{4.7}\pm\textbf{2.4}$
EER | Auxiliary[18] | $6.4$ | $5.6$ | $7.7$ | $6.5$ | $10.3$ | $6.1$ | $6.1$ |  | $8.4$ | $5.1$ | $6.3$ |  | $15.3$ | $13.1$ | $5.7$ | $7.9\pm 3.2$
Ours | 4.1 | 2.8 | 3.4 | 3.1 | 5.6 | 3.6 | 3.0 |  | 6.7 | 2.2 | 3.4 |  | 10.2 | 8.6 | 2.2 | $\textbf{4.5}\pm\textbf{2.5}$
TPR@ | Auxiliary [18] | $60.4$ | $65.5$ | $64.4$ | $70.4$ | $47.5$ | $67.0$ | $71.6$ |  | $64.3$ | $75.1$ | $69.8$ |  | $45.8$ | $47.8$ | $62.9$ | $62.5\pm 9.7$
FNR=$.5\%$ | Ours | 87.4 | 78.7 | 81.0 | 84.5 | 69.0 | 86.3 | 84.7 |  | 85.0 | 91.0 | 89.3 |  | 66.6 | 64.4 | 91.1 | $\textbf{81.6}\pm\textbf{9.2}$

Figure 8: Examples of each spoof trace components. (a) the input sample faces.
(b) B. (c) C. (d) T. (e) P. (f) the final live counterpart reconstruction and
zoom-in details. (g) results from [31]. (h) results from Step$1$+Step$2$ with
a single trace representation.

###  4.3 Anti-Spoofing for Unknown and Open-set Spoofs

Another important aspect is to test the anti-spoofing performance on unknown
spoof. To use SiW-M for unknown spoof detection, The work [19] defines the
leave-one-out testing protocols, termed as SiW-M Protocol II. In this
protocol, each model (i.e., one column in Tab. IV) is trained with $12$ types
of spoof attacks (as known attacks) plus the $80\%$ of the live faces, and
tested on the remaining $1$ attack (as unknown attack) plus the $20\%$ of live
faces. As shown in Tab. IV, our PhySTD achieves significant improvement over
our preliminary version, with relatively $11.7\%$ on the overall EER, $19.0\%$
on the overall ACER, $50.4\%$ on the overall TPR. Specifically, we reduce the
EERs of half mask, paper glasses, transparent mask, replay attack, and partial
paper relatively by $47.6\%$, $40.4\%$, $37.7\%$, $31.6\%$, $56.3\%$,
respectively. Overall, compared with the top $7$ performances, we outperform
the SOTA performance of EER/TPR and achieve comparable ACER. Among all, the
detection of silicone mask, paper-crafted mask, mannequin head, impersonation
makeup, and partial paper attacks are relatively good, with the detection
accuracy (i.e<EMAIL_ADDRESS>above $65\%$. Obfuscation makeup is the most
challenging one with TPR of $0\%$, where we predict all the spoof samples as
live. This is due to the fact that the makeup looks very similar to the live
faces, while being dissimilar to any other spoof types. However, once we
obtain a few samples, our model can quickly recognize the spoof traces on the
eyebrow and cheek, synthesize new spoof samples, and successfully detect the
attack (TPR=$41.1\%$ in Tab. III).

Figure 9: Examples of spoof trace disentanglement on SiW (a-h) and SiW-M
(i-x). (a)-(d) items are print attacks and (e)-(h) items are replay attacks.
(i)-(x) items are live, print, replay, half mask, silicone mask, paper mask,
transparent mask, obfuscation makeup, impersonation makeup, cosmetic makeup,
paper glasses, partial paper, funny eye glasses, and mannequin head. The first
column is the input face, the second column is the overall spoof trace
($\textbf{I}-\hat{\textbf{I}}$), the third column is the reconstructed live.

Moreover, in the real-world scenario, the testing samples can be either a
known spoof attack or an unknown one. Thus, we propose SiW-M Protocol III to
evaluate this open-set testing situation. In Protocol III, we first follow the
train/test split from protocol I, and then further remove one spoof type as
the unknown attack. During the testing, we test on the entire unknown spoof
samples as well the test split set of the know spoof samples. The results are
reported in Tab. V. Compared to the SOTA face anti-spoofing method [18], our
approach substantially outperforms it in all three metrics.

TABLE VI: Confusion matrices of spoof mediums classification based on spoof
traces. The results are compared with the previous method [24]. Green
represents improvement over [24]. Red represents performance drop.

Label Predict | Live |  Print$1$ |  Print$2$ |  Replay$1$ |  Replay$2$
---|---|---|---|---|---
Live |  $56(\color[rgb]{1,0,0}{-4}$$)$ |  $1(\color[rgb]{1,0,0}{+1}$$)$ |  $1(\color[rgb]{1,0,0}{+1}$$)$ |  $1(\color[rgb]{1,0,0}{+1}$$)$ |  $1(\color[rgb]{1,0,0}{+1}$$)$
Print$1$ | $0$ | $43({\color[rgb]{0,0.88,0}+2})$ | $11({\color[rgb]{1,0,0}+9})$ | $3({\color[rgb]{0,0.88,0}-8})$ | $3({\color[rgb]{0,0.88,0}-3})$
Print$2$ | $0$ | $9({\color[rgb]{0,0.88,0}-25})$ | $48({\color[rgb]{0,0.88,0}+37})$ | $1({\color[rgb]{0,0.88,0}-8})$ | $2({\color[rgb]{0,0.88,0}-4})$
Replay$1$ | $1({\color[rgb]{0,0.88,0}-9})$ | $2({\color[rgb]{0,0.88,0}-1})$ | $3({\color[rgb]{1,0,0}+3})$ | $51({\color[rgb]{0,0.88,0}+38})$ | $3({\color[rgb]{0,0.88,0}-28})$
Replay$2$ | $1({\color[rgb]{0,0.88,0}-7})$ | $2({\color[rgb]{0,0.88,0}-5})$ | $2({\color[rgb]{1,0,0}+2})$ | $3({\color[rgb]{0,0.88,0}-3})$ | $52({\color[rgb]{0,0.88,0}+13})$

TABLE VII: Confusion matrices of $6$-class spoof traces classification on
SiW-M database.

Label Predict | Live | Print | Replay | Masks | Makeup | Partial
---|---|---|---|---|---|---
Live | $116$ | $6$ | $6$ | $3$ | $0$ | $0$
Print | $1$ | $40$ | $1$ | $3$ | $0$ | $1$
Replay | $3$ | $1$ | $32$ | $1$ | $0$ | $1$
Masks | $3$ | $1$ | $1$ | $90$ | $0$ | $3$
Makeup | $3$ | $0$ | $0$ | $0$ | $36$ | $0$
Partial | $2$ | $0$ | $0$ | $2$ | $0$ | $146$

Figure 10: Examples of the spoof data synthesis. The first row are the source
spoof faces, the first column are the target live faces, and the remaining are
the synthesized spoof faces from the live face with the corresponding spoof
traces.

###  4.4 Spoof Traces Classification

To quantitatively evaluate the spoof trace disentanglement, we perform a spoof
medium classification on the disentangled spoof traces and report the
classification accuracy. The spoof traces should contain spoof medium-specific
information, so that they can be used for clustering without seeing the face.
To make a fair comparison with [24], we remove the additional spoof type
information from the preliminary mask $\mathbf{P}_{0}$. That is, for this
specific experiment, we only use the additive traces
$\\{\mathbf{B},\mathbf{C},\mathbf{T}\\}$ to learn the trace classification.
After $\\{\mathbf{B},\mathbf{C},\mathbf{T}\\}$ finish training with only
binary labels, we fix PhySTD and apply a simple CNN (i.e., AlexNet) on the
estimated additive traces to do a supervised spoof medium classification. We
follow the same $5$-class testing protocol in [24] in Oulu-NPU Protocol $1$.
We report the classification accuracy as the ratio between correctly predicted
samples from all classes and all testing samples. Shown in Tab. VI. Our model
can achieve a $5$-class classification accuracy of $83.3\%$. If we treat two
print attacks as the same class and two replay as the same class, our model
can achieve a $3$-class classification accuracy of $92.0\%$. Compared with the
prior method [24], we show an improvement of $29\%$ on the $5$-class model. In
addition, we train the same CNN on the original images instead of the
estimated spoof traces for the same spoof medium classification task, and the
classification accuracy can only reach $80.6\%$. This further demonstrates
that the estimated traces do contain significant information to distinguish
different spoof mediums.

We also execute the spoof traces classification task on more spoof types in
SiW-M database. We leverage the train/test split on SiW-M Protocol $1$. We
first train the PhySTD till convergence, and use the estimated traces from the
training set to train the trace classification network. We explore the
$6$-class scenario, shown in Tab. VII. Our $6$-class model can achieve the
classification accuracy of $92.0\%$. Since the traces are more distinct among
different spoof types, this performance is even better than $5$-class
classification on print/replay scenario in Oulu-NPU Protocol $1$. This further
demonstrates that PhySTD can estimate spoof traces that contain significant
information of spoof mediums and can be applied to multiple spoof types.

###  4.5 Ablation Study

In this section, we show the importance of each design of our proposed
approach on the SiW-M Protocol I, in Tab.III. Our baseline is the auxiliary
FAS [18], without the temporal module. It consists of the backbone encoder and
depth estimation network. When including the image decomposition, the baseline
becomes the training step $1$ in Alg. 1, as the traces are not activated
without the training step $2$. To validate the effectiveness of GAN training,
we report the results from the baseline model with our GAN design, denoted as
Step$1$+Step$2$. We also provide the control experiment where the traces are
represented by a single component to demonstrate the effectiveness of the
proposed $5$-element trace representation. This model is denoted as
Step$1$+Step$2$ with single trace. In addition, we evaluate the effect of
training with more synthesized data via enabling the training step $3$ as
Step$1$+Step$2$+Step$3$, which is our final approach.

As shown in Tab. III, the baseline model (Auxiliary) can achieve a decent
performance of EER $6.7\%$. Adding image decomposition to the baseline (Step
1) can improve the EER from $6.7\%$ to $4.3\%$, but more live samples are
predicted with higher scores, causing a worse ACER. Adding simple GAN design
(Step$1$+Step$2$ with single trace) may lead to a similar EER performance of
$5.8\%$, but based on the TPR ($59.3\%\rightarrow 74.8\%$) its practical
performance may be improved. With the proper physics-guided trace
disentanglement, we can improve the EER to $2.8\%$ and TPR to $89.7\%$. And
our final design can achieve the performance of HTER $2.8\%$, EER $2.5\%$, and
TPR $91.2\%$. Compared with our preliminary version, the EER is improved by
$47.9\%$, HTER is improved by $31.7\%$ and TPR is improved by $29.5\%$.

Figure 11: The tSNE visualization of features from different scales and
layers. The first $3$ visualization are from the encoder feature
$F_{1}$,$F_{2}$,$F_{3}$, and the last $2$ visualization are from the features
that produce $\\{\textbf{B},\textbf{C},\textbf{T}\\}$ and
$\\{\textbf{P},\textbf{I}_{P}\\}$.

Figure 12: The illustration of removing the disentangled spoof trace
components one by one. The estimated spoof trace elements of input spoof (the
first column) are progressively removed in the order of
$\mathbf{\mathbf{B},\mathbf{C},\mathbf{T},\mathbf{T}_{P}}$. The last column
shows the reconstructed live image after removing all three additive trace
components and the inpainting trace. (a) Replay attack; (b) Makeup attack; (c)
Mask attack; (d) Paper glasses attack.

###  4.6 Visualization

Spoof trace components In Fig.8, we provide illustration of each spoof trace
component. Strong color distortion (low-frequency trace) shows up in the print
attacks. Moiré patterns in the replay attack are well detected in the high-
frequency trace. The local specular highlights in transparent mask are well
presented in the low- and mid-frequency components, and the inpainting process
further fine-tunes the most highlighted area. For the two glasses attacks, the
color discrepancy is corrected in the low-frequency trace, and the sharp edges
are corrected in the mid- and high-frequency traces. Each component shows a
consistent semantic meaning on different spoof samples, and this successful
trace disentanglement can lead to better final visual results. As shown on the
right side of Fig. 8, we compare with our preliminary version [31] and the
ablated GAN design with a single trace representation. The result of single
trace representation shows strong artifacts on most of the live
reconstruction. The multi-scale from our preliminary version has already shown
a large visual quality improvement, but still have some spoof traces (e.g.,
glass edges) remained in the live reconstruction. In contrast, our approach
can further handle the missing traces and achieve better visualization.

Live reconstruction In Fig. 9, we show more examples from different spoof
types in SiW and SiW-M databases. The overall trace is the exact difference
between the input face and its live reconstruction. For the live faces, the
trace is zero, and for the spoof faces, our method removes spoof traces
without unnecessary changes, such as identity shift, and make them look like
live faces. For example, strong color distortion shows up in print/replay
attacks (Fig. 9a-h) and some $3$D mask attacks (Fig. 9l-o). For makeup attacks
(Fig. 9q-s), the fake eyebrows, lipstick, artificial wax, and cheek shade are
clearly detected. The folds and edges (Fig. 9t-w) are well detected and
removed in paper-crafted masks, paper glasses, and partial paper attacks.

Spoof synthesis Additionally, we show examples of new spoof synthesis using
the disentangled spoof traces, which is an important contribution of this
work. As shown in Fig. 10, the spoof traces can be precisely transferred to a
new face without changing the identity of the target face. Due to the
additional inpainting process, spoof attacks such as transparent mask and
partial attacks can be better attached to the new live face. Thanks to the
proposed $3$D warping layer, the geometric discrepancy between the source
spoof trace and the target face can be corrected during the synthesis.
Especially on the second source spoof, the right part of the traces is
successfully transferred to the new live face while the left side remains to
be still live. It demonstrates that our trace regularization can suppress
unnecessary artifacts generated by the network. Both the live reconstruction
results in Fig. 9 and the spoof synthesis results in Fig. 10 demonstrate that
our approach disentangles visually convincing spoof traces that help face
anti-spoofing.

Spoof trace removing process As shown in Fig. 12, we illustrate the effects of
trace components by progressively removing them one by one. For the replay
attack, the spoof sample comes with strong over-exposure as well as clear
Moiré pattern. Removing the low-frequency trace can effectively correct the
over-exposure and color distortion caused by the digital screen. And removing
the texture pattern in the high-frequency trace can peel off the high-
frequency grid effect and reconstruct the live counterpart.

For the makeup attack, since there is no strong color range bias, removing
estimated low-frequency trace would mainly remove the lip-stick color and fake
eyebrow, but in the meantime bring a few artifacts at the edges. Next, while
removing the content pattern, the shadow on the cheek and the fake eyebrows
are adequately lightened. Finally, removing the texture pattern would
significantly correct the spoof traces from artificial wax, eyeliner, and
shadow on the cheek. Similarly, in mask and partial attacks, the
reconstruction will be gradually refined as we removing components one by one.

t-SNE visualization We use t-SNE [71] to visualize the encoder features
$\mathbf{F}_{1}$,$\mathbf{F}_{2}$,$\mathbf{F}_{3}$, and the features that
produce $\\{\mathbf{B},\mathbf{C},\mathbf{T}\\}$ and
$\\{\mathbf{P},\mathbf{I}_{P}\\}$. The t-SNE is able to project the output of
features from different scales and layers to $2$D by preserving the KL
divergence distance. As shown in Fig. 11, among the three feature scales in
the encoder, $F_{3}$ is the most separable feature space, the next is $F_{1}$,
and the worst is $F_{2}$. The features for additive traces
$\\{\textbf{B},\textbf{C},\textbf{T}\\}$ are well-clustered as semantic sub-
groups of live, makeup, mask, and partial attacks. As we know the inpainting
masks for live samples are close to zero, the feature for inpainting traces
$\\{\textbf{P},\textbf{I}_{P}\\}$ shows the inpainting process mostly update
the partial attacks, and then some makeup attacks and mask attacks, i.e., the
green dots being further away from the black dots means they have greater
magnitude. This validates our prior knowledge of the inpainting process.

##  5 Conclusions

This work proposes a physics-guided spoof traces disentanglement network
(PhySTD) to tackle the challenging problem of disentangling spoof traces from
the input faces. With the spoof traces, we reconstruct the live faces as well
as synthesize new spoofs. To correct the geometric discrepancy in synthesis,
we propose a $3$D warping layer to deform the traces. The disentanglement not
only improves the SOTA of face anti-spoofing in known, unknown, and open-set
spoof settings, but also provides visual evidence to support the model’s
decision.

## Acknowledgments

This research is based upon work supported by the Office of the Director of
National Intelligence (ODNI), Intelligence Advanced Research Projects Activity
(IARPA), via IARPA R$\&$D Contract No. 2017-17020200004. The views and
conclusions contained herein are those of the authors and should not be
interpreted as necessarily representing the official policies or endorsements,
either expressed or implied, of the ODNI, IARPA, or the U.S. Government. The
U.S. Government is authorized to reproduce and distribute reprints for
Governmental purposes notwithstanding any copyright annotation thereon.

## References

  * [1] K. Dale, K. Sunkavalli, M. K. Johnson, D. Vlasic, W. Matusik, and H. Pfister, “Video face replacement,” in _TOG_. ACM, 2011.
  * [2] E. Zakharov, A. Shysheya, E. Burkov, and V. Lempitsky, “Few-shot adversarial learning of realistic neural talking head models,” _arXiv preprint arXiv:1905.08233_ , 2019.
  * [3] M. Zollhöfer, J. Thies, P. Garrido, D. Bradley, T. Beeler, P. Pérez, M. Stamminger, M. Nießner, and C. Theobalt, “State of the art on monocular 3D face reconstruction, tracking, and applications,” in _Computer Graphics Forum_. Wiley Online Library, 2018.
  * [4] D. Deb, J. Zhang, and A. K. Jain, “Advfaces: Adversarial face synthesis,” _arXiv preprint arXiv:1908.05008_ , 2019.
  * [5] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” _arXiv preprint arXiv:1412.6572_ , 2014.
  * [6] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” _arXiv preprint arXiv:1312.6199_ , 2013.
  * [7] J. F. Boylan, “Will deep-fake technology destroy democracy?” in _The New York Times_ , 2018.
  * [8] J. Thies, M. Zollhofer, M. Stamminger, C. Theobalt, and M. Nießner, “Face2face: Real-time face capture and reenactment of RGB videos,” in _CVPR_. IEEE, 2016.
  * [9] J. Bigun, H. Fronthaler, and K. Kollreider, “Assuring liveness in biometric identity authentication by real-time face tracking,” in _International Conference on Computational Intelligence for Homeland Security and Personal Safety (CIHSPS)_. IEEE, 2004.
  * [10] R. W. Frischholz and A. Werner, “Avoiding replay-attacks in a face recognition system using head-pose estimation,” in _International SOI Conference. (Cat. No. 03CH37443)_. IEEE, 2003.
  * [11] S. A. Schuckers, “Spoofing and anti-spoofing measures,” _Information Security technical report_ , 2002.
  * [12] Z. Boulkenafet, J. Komulainen, and A. Hadid, “Face anti-spoofing based on color texture analysis,” in _ICIP_. IEEE, 2015, pp. 2636–2640.
  * [13] ——, “Face antispoofing using speeded-up robust features and fisher vector encoding,” _Signal Processing Letters_ , 2016.
  * [14] K. Patel, H. Han, and A. K. Jain, “Secure face unlock: Spoof detection on smartphones,” _TIFS_ , 2016.
  * [15] T. de Freitas Pereira, A. Anjos, J. M. De Martino, and S. Marcel, “LBP-TOP based countermeasure against face spoofing attacks,” in _ACCV_. Springer, 2012.
  * [16] J. Komulainen, A. Hadid, and M. Pietikäinen, “Context based face anti-spoofing,” in _BTAS_. IEEE, 2013, pp. 1–8.
  * [17] Y. Atoum, Y. Liu, A. Jourabloo, and X. Liu, “Face anti-spoofing using patch and depth-based CNNs,” in _IJCB_. IEEE, 2017.
  * [18] Y. Liu, A. Jourabloo, and X. Liu, “Learning deep models for face anti-spoofing: Binary or auxiliary supervision,” in _CVPR_. IEEE, 2018.
  * [19] Y. Liu, J. Stehouwer, A. Jourabloo, and X. Liu, “Deep tree learning for zero-shot face anti-spoofing,” in _CVPR_. IEEE, 2019.
  * [20] R. Shao, X. Lan, J. Li, and P. C. Yuen, “Multi-adversarial discriminative deep domain generalization for face presentation attack detection,” in _CVPR_. IEEE, 2019.
  * [21] X. Yang, W. Luo, L. Bao, Y. Gao, D. Gong, S. Zheng, Z. Li, and W. Liu, “Face anti-spoofing: Model matters, so does data,” in _CVPR_. IEEE, 2019.
  * [22] J. Yang, Z. Lei, and S. Z. Li, “Learn convolutional neural network for face anti-spoofing,” _arXiv preprint arXiv:1408.5601_ , 2014.
  * [23] Z. Xu, S. Li, and W. Deng, “Learning temporal features using lstm-cnn architecture for face anti-spoofing,” in _2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR)_. IEEE, 2015.
  * [24] A. Jourabloo, Y. Liu, and X. Liu, “Face de-spoofing: Anti-spoofing via noise modeling,” in _ECCV_. Springer, 2018\.
  * [25] J. Stehouwer, A. Jourabloo, Y. Liu, and X. Liu, “Noise modeling, synthesis and classification for generic object anti-spoofing,” in _CVPR_ , 2020.
  * [26] H. Feng, Z. Hong, H. Yue, Y. Chen, K. Wang, J. Han, J. Liu, and E. Ding, “Learning generalized spoof cues for face anti-spoofing,” _arXiv preprint arXiv:2005.03922_ , 2020.
  * [27] “Explainable Artificial Intelligence (XAI),” https://www.darpa.mil/program/explainable-artificial-intelligence.
  * [28] A. B. Arrieta, N. Díaz-Rodríguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. García, S. Gil-López, D. Molina, R. Benjamins _et al._ , “Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai,” _Information Fusion_ , 2020.
  * [29] Y. Liu and C. Shu, “A comparison of image inpainting techniques,” in _Sixth International Conference on Graphic and Image Processing (ICGIP 2014)_ , 2015.
  * [30] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, “Image inpainting,” in _Proceedings of the 27th annual conference on Computer graphics and interactive techniques_ , 2000.
  * [31] Y. Liu, J. Stehouwer, and X. Liu, “On disentangling spoof traces for generic face anti-spoofing,” in _ECCV_ , 2020.
  * [32] K. Kollreider, H. Fronthaler, M. I. Faraj, and J. Bigun, “Real-time face detection and motion analysis with application in “liveness” assessment,” _TIFS_ , 2007.
  * [33] G. Pan, L. Sun, Z. Wu, and S. Lao, “Eyeblink-based anti-spoofing in face recognition from a generic webcamera,” in _ICCV_. IEEE, 2007.
  * [34] T. de Freitas Pereira, A. Anjos, J. M. De Martino, and S. Marcel, “Can face anti-spoofing countermeasures work in a real world scenario?” in _ICB_. IEEE, 2013, pp. 1–8.
  * [35] J. Määttä, A. Hadid, and M. Pietikäinen, “Face spoofing detection from single images using micro-texture analysis,” in _IJCB_. IEEE, 2011.
  * [36] J. Yang, Z. Lei, S. Liao, and S. Z. Li, “Face liveness detection with component dependent descriptor,” in _ICB_. IEEE, 2013.
  * [37] L. Feng, L.-M. Po, Y. Li, X. Xu, F. Yuan, T. C.-H. Cheung, and K.-W. Cheung, “Integration of image quality and motion cues for face anti-spoofing: A neural network approach,” _Journal of Visual Communication and Image Representation_ , 2016.
  * [38] L. Li, X. Feng, Z. Boulkenafet, Z. Xia, M. Li, and A. Hadid, “An original face anti-spoofing approach using partial convolutional neural network,” in _Sixth International Conference on Image Processing Theory, Tools and Applications (IPTA)_. IEEE, 2016.
  * [39] K. Patel, H. Han, and A. K. Jain, “Cross-database face antispoofing with robust feature representation,” in _CCBR_. Springer, 2016.
  * [40] Y. Qin, C. Zhao, X. Zhu, Z. Wang, Z. Yu, T. Fu, F. Zhou, J. Shi, and Z. Lei, “Learning meta model for zero-and few-shot face anti-spoofing,” _arXiv preprint arXiv:1904.12490_ , 2019.
  * [41] C. Zhao, Y. Qin, Z. Wang, T. Fu, and H. Shi, “Meta anti-spoofing: Learning to learn in face anti-spoofing,” _arXiv preprint arXiv:1904.12490_ , 2019.
  * [42] R. Shao, X. Lan, and P. C. Yuen, “Regularized fine-grained meta face anti-spoofing,” _arXiv preprint arXiv:1911.10771_ , 2019.
  * [43] L. Tran, X. Yin, and X. Liu, “Disentangled representation learning GAN for pose-invariant face recognition,” in _CVPR_. IEEE, 2017.
  * [44] Z. Zhang, L. Tran, X. Yin, Y. Atoum, J. Wan, N. Wang, and X. Liu, “Gait recognition via disentangled representation learning,” in _CVPR_. IEEE, 2019.
  * [45] F. Liu, D. Zeng, Q. Zhao, and X. Liu, “Disentangling features in 3D face shapes for joint face reconstruction and recognition,” in _CVPR_. IEEE, 2018.
  * [46] L. Tran and X. Liu, “On learning 3d face morphable model from in-the-wild images,” _T-PAMI_ , Jan 2021.
  * [47] P. Esser, E. Sutter, and B. Ommer, “A variational U-Net for conditional appearance and shape generation,” in _CVPR_. IEEE, 2018.
  * [48] A. Abdelhamed, S. Lin, and M. S. Brown, “A high-quality denoising dataset for smartphone cameras,” in _CVPR_ , 2018.
  * [49] T. H. Thai, R. Cogranne, and F. Retraint, “Camera model identification based on the heteroscedastic noise model,” _TIP_ , 2013.
  * [50] T. H. Thai, F. Retraint, and R. Cogranne, “Camera model identification based on the generalized noise model in natural images,” _Digital Signal Processing_ , 2016.
  * [51] Y. Wang, L. Bondi, P. Bestagini, S. Tubaro, D. J. Edward Delp _et al._ , “A counter-forensic method for cnn-based camera model identification,” in _CVPRW_ , 2017.
  * [52] C. Chen, Z. Xiong, X. Liu, and F. Wu, “Camera trace erasing,” in _CVPR_ , 2020\.
  * [53] Y. Wu, W. AbdAlmageed, and P. Natarajan, “Mantra-net: Manipulation tracing network for detection and localization of image forgeries with anomalous features,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 9543–9552.
  * [54] H. Dang, F. Liu, J. Stehouwer, X. Liu, and A. K. Jain, “On the detection of digital face manipulation,” in _CVPR_ , 2020.
  * [55] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015.
  * [56] Z. Yu, C. Zhao, Z. Wang, Y. Qin, Z. Su, X. Li, F. Zhou, and G. Zhao, “Searching central difference convolutional networks for face anti-spoofing,” in _CVPR_. IEEE, 2020\.
  * [57] Y. Liu, J. Stehouwer, A. Jourabloo, and X. Liu, “Presentation attack detection for face in mobile phones,” _Selfie Biometrics_ , 2019.
  * [58] J. Guo, X. Zhu, J. Xiao, Z. Lei, G. Wan, and S. Z. Li, “Improving face anti-spoofing by 3D virtual synthesis,” _arXiv preprint arXiv:1901.00488_ , 2019.
  * [59] H. Chang, J. Lu, F. Yu, and A. Finkelstein, “Paired cycleGAN: Asymmetric style transfer for applying and removing makeup,” in _CVPR_. IEEE, 2018.
  * [60] Y. Liu, A. Jourabloo, W. Ren, and X. Liu, “Dense face alignment,” in _ICCV Workshops_. IEEE, 2017.
  * [61] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro, “High-resolution image synthesis and semantic manipulation with conditional GANs,” in _CVPR_. IEEE, 2018.
  * [62] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _CVPR_. IEEE, 2017.
  * [63] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley, “Least squares generative adversarial networks,” in _ICCV_. IEEE, 2017.
  * [64] L. Tran, X. Liu, J. Zhou, and R. Jin, “Missing modalities imputation via cascaded residual autoencoder,” in _CVPR_. IEEE, 2017.
  * [65] K.-Y. Zhang, T. Yao, J. Zhang, Y. Tai, S. Ding, J. Li, F. Huang, H. Song, and L. Ma, “Face anti-spoofing via disentangled representation learning,” in _ECCV_ , 2020.
  * [66] Z. Yu, X. Li, X. Niu, J. Shi, and G. Zhao, “Face anti-spoofing with human material perception,” in _ECCV_ , 2020.
  * [67] Z. Boulkenafet, J. Komulainen, L. Li, X. Feng, and A. Hadid, “OULU-NPU: A mobile face presentation attack database with real-world variations.” IEEE, 2017.
  * [68] ISO/IEC JTC 1/SC 37 Biometrics, “Information technology biometric presentation attack detection part 1: Framework. international organization for standardization,” https://www.iso.org/obp/ui/iso, 2016.
  * [69] “IARPA research program Odin),” https://www.iarpa.gov/index.php/research-programs/odin.
  * [70] A. Bulat and G. Tzimiropoulos, “How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3d facial landmarks),” in _ICCV_. IEEE, 2017.
  * [71] L. v. d. Maaten and G. Hinton, “Visualizing data using t-SNE,” _Journal of machine learning research_ , 2008.

|  Yaojie Liu is a PhD candidate at at Michigan State University in Computer
Science and Engineering Department. He received his B.S. in Communication
Engineering from University of Electronic Science and Technology of China, and
M.S. in Computer Science from the Ohio State University. His research areas of
interest are face representation and analysis, including face anti-spoofing,
2D/3D large pose face alignment, and 3D face reconstruction.
---|---
|  Xiaoming Liu is a Professor at the Department of Computer Science and
Engineering of Michigan State University. He received the Ph.D. degree in
Electrical and Computer Engineering from Carnegie Mellon University in 2004.
Before joining MSU in Fall 2012, he was a research scientist at General
Electric (GE) Global Research. His research interests include computer vision,
machine learning, and biometrics. As a co-author, he is a recipient of Best
Industry Related Paper Award runner-up at ICPR 2014, Best Student Paper Award
at WACV 2012 and 2014, Best Poster Award at BMVC 2015, and Michigan State
University College of Engineering Withrow Endowed Distinguished Scholar Award.
He has been Area Chairs for numerous conferences, including CVPR, ICCV, ECCV,
ICLR, NeurIPS, the Program Co-Chair of BTAS’18, WACV’18, and AVSS’21
conferences, and General Co-Chair of FG’23 conference. He is an Associate
Editor of Pattern Recognition Letters, Pattern Recognition, and IEEE
Transaction on Image Processing. He has authored more than 150 scientific
publications, and has filed 27 U.S. patents. He is a fellow of IAPR.
---|---