# FDLS: A Deep Learning Approach to Production Quality, Controllable, and
Retargetable Facial Performances

Wan-Duo Kurt Ma  1234-5678-9012-3456 Weta Digital + UnityWellingtonNew Zealand
<EMAIL_ADDRESS>,  Muhammad Ghifary  1234-5678-9012 Weta DigitalWellingtonNew
Zealand<EMAIL_ADDRESS>,  J.P. Lewis  Weta DigitalWellingtonNew Zealand
<EMAIL_ADDRESS>,  Byungkuk Choi  Weta Digital + UnitySeoulKorea78229
<EMAIL_ADDRESS>and  Haekwang Eom  Weta Digital + UnitySeoulKorea
<EMAIL_ADDRESS>
(2022)

###### Abstract.

Visual effects commonly requires both the creation of realistic synthetic
humans as well as retargeting actors’ performances to humanoid characters such
as aliens and monsters. Achieving the expressive performances demanded in
entertainment requires manipulating complex models with hundreds of
parameters. Full creative control requires the freedom to make edits at any
stage of the production, which prohibits the use of a fully automatic “black
box” solution with uninterpretable parameters. On the other hand, producing
realistic animation with these sophisticated models is difficult and
laborious.
This paper describes FDLS (Facial Deep Learning Solver
111https://www.youtube.com/watch?v=39W4eGjMFiA), which is Weta Digital’s
solution to these challenges. FDLS adopts a coarse-to-fine and _human-in-the-
loop_ strategy, allowing a solved performance to be verified and (if needed)
edited at several stages in the solving process. To train FDLS, we first
transform the raw motion-captured data into robust graph features. The feature
extraction algorithms were devised after carefully observing the artists’
interpretation of the 3d facial landmarks. Secondly, based on the observation
that the artists typically finalize the jaw pass animation before proceeding
to finer detail, we solve for the jaw motion first and predict fine
expressions with region-based networks conditioned on the jaw position.
Finally, artists can optionally invoke a non-linear finetuning process on top
of the FDLS solution to follow the motion-captured virtual markers as closely
as possible. FDLS supports editing if needed to improve the results of the
deep learning solution and it can handle small daily changes in the actor’s
face shape.
FDLS permits reliable and production-quality performance solving with minimal
training and little or no manual effort in many cases, while also allowing the
solve to be guided and edited in unusual and difficult cases. The system has
been under development for several years and has been used in major movies.

Facial animation, Deep learning, Motion capture, Optimization

††journalyear: 2022††copyright: acmcopyright††conference: The Digital
Production Symposium; August 7, 2022; Vancouver, BC, Canada††booktitle: The
Digital Production Symposium (DigiPro ’22), August 7, 2022, Vancouver, BC,
Canada††price: 15.00††doi: 10.1145/3543664.3543672††isbn:
978-1-4503-9418-5/22/08††ccs: Computing methodologies Animation

##  1\. Introduction

Figure 1. Facial Deep Learning Solve (FDLS) is an animator oriented tool
system for solving facial animation given motion captured sparse marker set
and limited training data.

Digital characters representing either humans or creatures are ubiquitous in
the film and video game industries. Creating believable performances with
these characters is one of the grand challenges of computer graphics.
Realistic digital characters are often animated using performances captured
from actors, as this allows the characters to inherit the “personality” of
selected actors. On the other hand, performance capture presents its own
difficulties arising from the complexity of the models, the limited amounts of
data available (to retain fidelity to a particular actor’s performance of a
particular character, it may not be possible to use data from other actors,
nor even from the same actor performing a different character), and the need
to potentially edit the resulting performances.

This paper describes FDLS, Weta Digital’s solution to this challenge. FDLS has
been in development since 2016 and has been used on movies such as _Gemini
Man_ 222Gemini Man VFX Breakdown - Junior — Weta Digital: “All of this was to
ensure we captured the youthfulness, not just the likeness, of a 23-year-old
Will Smith.” https://www.youtube.com/watch?v=V1lTuNLuIO8. Our system has the
following assumptions and requirements:

  * •

Production quality, with high and verifiable accuracy. For example we assume
that an actor’s face shape may change slightly across days, and account for
this. A stereo or mono head-mounted camera (HMC) with virtual (painted)
markers is used in order to allow simultaneous capture of the head and body
motion, since marker-free tracking methods are considered to not yet be
sufficiently accurate.

  * •

Perceptual evaluation is used (as is commonly the case in image quality
assessment as well (Zhang et al., 2018)), with guidance provided by the VFX
supervisor.

  * •

Editable and retargetable results. Editing is needed to retain full creative
control. A closed “black box” solver is not suitable because the supervisor
may _require_ that the performance be edited for various reasons. The actor’s
performance often needs to be re-targeted to other characters, such as
humanoid aliens and monsters. Occasionally a particular expression may not be
conveyed correctly on a character with very different proportions (e.g. a
dragon). FDLS excels at post editing on rig (aka  _puppet_) parameters such as
FACS (Facial Action Coding System) action units (Ekman and Friesen, 1978).

  * •

We target expert animators rather than casual or novice users, while greatly
reducing the effort needed to manipulate complex facial models,

  * •

The ability to handle a rig with nonlinear effects, including intermediate and
corrective blendshapes (Osipa, 2010) and additional deformers e.g. for
handling collisions between the lips.

  * •

As mentioned above, we assume limited training data relative to the massive
datasets commonly used to train deep neural networks. To address this issue we
introduce strong inductive biases by way of specifically designed vertex graph
features, that better capture the facial expression while remaining
insensitive to irrelevant factors.

  * •

We require performance capture that has been stabilized to remove rigid motion
of the skull (Beeler and Bradley, 2014; Lamarre et al., 2018). We use a stereo
or mono head-mounted camera, however the use of a head-mounted camera is not
itself sufficient for stabilization since the camera may slip or the actor’s
scalp may move with respect to the skull.

This paper describes the ”face solving” component of an overall character
pipeline. Other components such as modeling, rigging, adaptation of the rig to
specific actors (Ma et al., 2016; Seol et al., 2016), tracking, stabilization,
muscle simulation, and rendering are separate topics not covered here.

The primary contributions of this paper are:

  * •

A presentation of the motivation and design decisions for Weta Digital’s
Facial Deep Learning Solver (FDLS) system, a novel _multi-stage face solving_
approach for production quality performance-driven facial animation using deep
neural networks.

  * •

Graph features that allow accurate animation solving with limited training
data.

  * •

A nonlinear post-finetuning method conditioned on jaw animation to allow
better matching of some difficult face expressions.

  * •

A method to accommodate the changes in the marker layout or actor’s face shape
without requiring re-training of the network.

##  2\. Related work

The literature on facial performance capture dates from (Williams, 1990) and
is too large to fully review, for example (Zollhöfer et al., 2018) primarily
targets the subset of monocular and optimization-based approaches yet contains
more than 200 references. We will focus on methods that using deep learning as
well as those that give professional quality and editable results, while
providing a few pointers to other literature. Broader surveys of facial
performance capture and related areas include (Zollhöfer et al., 2018; Klehm
et al., 2015; Tagliasacchi et al., 2016).

Computer vision research often targets fully automatic approaches that produce
non-editable representations such as dense meshes or uninterpretable (e.g.
PCA) parameters. (Blanz and Vetter, 1999) introduced the morphable model
approach in which scanned facial geometry and texture are approximated with
PCA linear subspaces. This approach has been extended in a large body of
research over the past two decades (Egger et al., 2020). Performance capture
of dense facial meshes usually requires complex camera setups (Fyffe et al.,
2015; Bhat et al., 2013; Bradley et al., 2010) and multi-view stereo or
photogrammetric reconstruction (Fyffe et al., 2015; Furukawa and Ponce, 2009;
Zhang et al., 2004). Other research introduces algorithms for specific facial
regions such as the lips (Garrido et al., 2016) and eyelids (Bermano et al.,
2015).

Real-time tracking and re-targeting from monocular RGB video has seen
widespread use in face “filters” for video conferencing systems. While these
methods have impressive ease of use and robustness, they do not offer
_verifiable_ accuracy and may not capture fine facial detail, as well as being
prone to errors from significant lighting changes and occlusions. See
(Zollhöfer et al., 2018) for a recent survey of the literature in this area.
Higher quality markerless capture has also been demonstrated recently (Disney,
2022), albeit under constrained viewing and lighting conditions that would
prevent simultaneous capture of facial and body performances. Though there are
differences in opinion, some believe that such simultaneous capture is
essential to capture natural correlations between face and body motion.

The facial animation solvers used in visual effects production and games often
use physical or virtual (painted) markers as input, due to accuracy
requirements and reliability of marker-based systems (Bickel et al., 2007;
Huang et al., 2011). However, these approaches have less ability to capture
fine detail in complex regions of the face, in particular around the lips. One
of the accommodations in these marker-driven methods is to add additional
information extracted from the captured footage. For instance, (Bhat et al.,
2013) adds separately tracked contours of the lips and eyelid to the solver’s
input.

Keyframe animation is widely used as it allows artists to specify intuitively
understandable parameters at a subset of frames, and then interpolate these
parameters to the remaining frames using (for example) Catmull-Rom (Li and
Deng, 2008) or B-spline (Choe et al., 2001) curves. On the other hand,
keyframe animation is quite laborious, and various techniques have been
proposed to speed up the artist’s workflow. For example (Seol et al., 2011)
introduced a successive refinement scheme in which large motions (e.g. the
jaw) are solved before the fine details. Temporal animation editing has been
developed in several directions, includings gradient-domain space-time
editing(Seol et al., 2012) and systems that allow interactive tuning of the
solution using nonlinear regression with radial basis function networks (Seol
and Cozens, 2019; Seol and Lewis, 2014). Intuitive user guidance in the form
of 2D sketches has been used for generating plausible lip corrections (Dinev
et al., 2018), guiding blendshape models (Cetinaslan et al., 2015) and doing
space-time editing of general animation (Choi et al., 2016). (Berson et al.,
2020, 2019) use a learning based approach to perform temporal animation
editing.

Performance capture systems developed in production settings target high
quality and (in most cases) editable rig representations. (Smith et al., 2017)
demonstrates high-quality blendshape solving using an optimization approach.
Convolutional neural networks have been employed to regress directly from
video to dense meshes (Laine et al., 2017). (Moser et al., 2018) describes a
production-proven system for regressing 3D marker positions. In addition to
this research, there are several commercial systems that address aspects of
professional performance capture, including DI4D, Dynamixyz, Synthesia,
Medusa, and Imagemetrics (Various, 2022).

“Deepfake” systems have recently been applied to face replacement for stunt
doubles (Seymour, 2022). In these systems an autoencoder with separate
decoders for the actor and stunt double is used to learn a shared latent
space, allowing the double’s performance to be decoded using the actor’s
likeness. In this stunt double application the resolution limitations of
current neural rendering is not a limiting issue, since the view of the stunt
generally includes the full body and perhaps the surround (thus the face is a
smaller part of the whole image), and there may be motion blur as well. On the
other hand, neural rendering approaches currently struggle with producing
“hero” shots where the face occupies most of a 2K or 4K image, and they are
also not suited for scenarios where artist editing is sometimes required.
(Moser et al., 2021; Serra et al., 2022) avoid the resolution issue by using a
deepfake approach only to bridge the domain gap between CG and real input
images, allowing a regression from input images to PCA model coefficients to
be trained with synthetic data.

In summary, there is a large body of research on facial performance capture,
including both fully automatic neural approaches, and approaches that target
editable high quality performance capture using classic methods. However,
there is relatively little existing research on deep learning methods that
target artist-in-the-loop, editable, and high-quality performance capture
suitable for high-resolution (2K or 4K) “hero” facial shots (Seymour, 2019).

##  3\.  Hybrid Method for Facial Animation Capture

We focus on solving performance-driven facial animation problems with
blendshape models, i.e., inferring the blendshape weights $\mathbf{\hat{w}}$
such that the corresponding blendshape expression matches the captured actor’s
performance. Specifically, we denote a face model by
$g:\mathcal{W}\rightarrow\mathcal{X}$, where
$\mathcal{W}\subseteq\mathbb{R}^{D}$ is the blendshape weight space and
$\mathcal{X}\subseteq\mathbb{R}^{3n}$ is the face expression space, each
element of which is the $n$ vertices with the coordinates vectorized as
$[x_{1},y_{1},z_{1},\ldots,x_{n},y_{n},z_{n}]$. Correspondingly, each
blendshape is given by a vector $\mathbf{b}_{k}\in\mathbb{R}^{3n}$. In the
linear case a blendshape model in the ”delta” formulation is:

(1) $\displaystyle
g(\mathbf{w})=\mathbf{b}_{0}+\sum_{k=1}^{D}w_{k}(\mathbf{b}_{k}-\mathbf{b}_{0})=\mathbf{b}_{0}+\mathbf{B}\mathbf{w},$

where $w_{k}$ are the blendshape weights (typically $0\leq w_{k}\leq 1$),
$\mathbf{b}_{0}$ corresponds to the neutral shape, and
$\mathbf{B}\in\mathbb{R}^{3n\times D}$ contains the ”delta” blendshape targets
$\mathbf{b}_{k}-\mathbf{b}_{0}$.

Let us define a distance measure
$\mathcal{M}:\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R}$. Given a
captured actor expression at a particular frame $\mathbf{x}\in\mathcal{X}$ ,
we seek $\hat{\mathbf{w}}$ such that
$\mathcal{M}(g(\hat{\mathbf{w}}),\mathbf{x})$ is small. In our case,
$g(\mathbf{w})$ and $\mathbf{x}$ represent sparse markers attached on either
the face puppet or the real actor’s face. In general, we can frame the problem
either as a face matching problem or a regression problem.

##### Matching approach.

A prevalent approach to solving the blendshape weights is to frame it as a
_face matching_ optimization problem (Choe et al., 2001). Choosing Euclidean
distance as $\mathcal{M}(\cdot,\cdot)$ and setting $g(\mathbf{w})$ as in (1),
we can express the problem as the following optimization:

(2)
$\displaystyle\hat{\mathbf{w}}:=\arg\min_{\mathbf{w}}\left\|\mathbf{b}_{0}+\mathbf{B}\mathbf{w}-\mathbf{x}\right\|_{2}^{2}\quad\text{s.t.}\quad
0\preceq\mathbf{w}\preceq 1,$

which is a constrained quadratic programming (QP) problem. The solution can be
obtained by applying a standard QP solver run on each frame (Boyd and
Vandenberghe, 2004; Fascione et al., 2017).

##### Learning-based approach.

Instead of solving the face matching problem by optimization, one can also
apply a regression approach to inferring the blendshape weights
$\hat{\mathbf{w}}$ from the captured expression $\mathbf{x}$ through a
multivariate regression function
$f_{\theta}:\mathcal{X}\rightarrow\mathcal{W}$.

The regression function is often decomposed as $f=h\circ\phi$, where we call
$\phi:\mathcal{X}\rightarrow\mathcal{H}$ the _feature extractor_ and
$h:\mathcal{H}\rightarrow\mathcal{W}$ the _regressor_. A machine learning
technique can identify the function $f_{\hat{\theta}}$ given a set of labeled
examples
$\mathcal{D}=\\{(\mathbf{x}^{(i)}_{v},\mathbf{w}^{(i)}_{v})\\}_{i=1}^{N}$
(Vapnik, 1998). This trained regressor is expected to predict valid blendshape
weights given the _previously unseen_ captured expression $\mathbf{x}_{u}$,
i.e., $\hat{\mathbf{w}_{u}}=f_{\hat{\theta}}(\mathbf{x}_{u})$, such that the
3d model expression $g(\hat{\mathbf{w}_{u}})$ is correctly generated.

##### Nonlinear rig.

For simplicity, the preceding description presents the matching and learning-
based approaches in terms of a linear blendshape model. Our blendshape puppets
incorporate natural non-linear expressions, for example using in-between and
corrective shapes (Seo et al., 2011; Osipa, 2010). In addition, the overall
puppet consists of a large deformation chain with numerous tweaks, deformers
and skin clusters applied on top of the blendshape system.

A full description of the puppet is outside the scope of this paper. The
details of the puppet are not needed to understand the application of the
learning and matching approaches, however: the learning-based approach
captures the puppets’s behavior (_including_ nonlinearities) since the puppet
itself is used to produce training data (Section 4.1). For the matching
approach, rather than seek a global optimum, our optimization fine-tunes a
local linear subset of parameters that are selected by the artist (see _Hybrid
approach_ , next, and Section 4.5).

##### Hybrid approach.

In FDLS, we apply the learning-based approach as the _main animation solver_ ,
specifically using deep learning regression (He et al., 2016), followed by the
matching approach to fine-tune some facial parameters. This hybrid approach
has two main advantages. First, the main animation solving is reduced to the
_forward pass_ of the network $f_{\theta}$, which is time-efficient. Second,
the forward pass provides an initialization for the nonlinear matching
optimization that is in the neighborhood of the correct local minimum and is
close to the desired solution, resulting in temporally consistent and
efficient fine tuning to generate high-fidelity final animations. In the
actual implementation, FDLS comprises several components to enable the _human-
in-the-loop_ solving process. These are fully elaborated in the next section.

##  4\. The Complete FDLS Pipeline

Figure 2. The Facial Deep Learning Solver (FDLS) comprises three main stages:
data preparation, training and solving, where the letters denote: (A) salient
sample selection (Section 4.2); (B) feature engineering (Section 4.3); (C) jaw
kinematics conditional training (Section 4.4.1); (D) shape alignment (Section
4.4.2); (E) finetuning (Section 4.5); (F) anchor poses (Section 4.4.2).

As shown in Fig. 2, the end-to-end FDLS pipeline consists of two main stages:
i) the development stage, and ii) the operation stage. This partitioning forms
a _human-in-the-loop_ MLOps lifecycle in Weta Digital’s production that
ensures reproducibility, testability, and maintainability of our deep learning
system. In the development stage, data preparation steps are performed,
including synthetic data generation (Section 4.1), salient sample selection
(Section 4.2), feature engineering (Section 4.3), and training the deep
learning models (Section 4.4). The animation solving happens in the operation
stage, which makes use of the trained deep learning models as the main solvers
complemented by a few other techniques: shape alignment with anchor poses
(Section 4.4) and fine-tuning optimization (Section 4.5).

###  4.1. Synthetic Data Generation and Augmentation

The training data for FDLS are in the form of 3d virtual markers created from
the actor’s facial landmarks of his/her neutral expression using
photogrammetry – we call these ”neutral markers” throughout the paper for
brevity. We then attach the neutral markers to the corresponding actor’s
blendshape puppet and synthetically generate the training markers driven by
the puppet motion $\mathbf{x}^{(i)}_{v}=g(\mathbf{w}^{(i)}_{v})$ forming a set
of training tuples
$\mathcal{D}=\\{(\mathbf{x}^{(i)}_{v},\mathbf{w}^{(i)}_{v})\\}_{i=1}^{N}$.
Note that the training blendshape weights $\mathbf{w}^{(i)}$ are designed and
selected by animation experts, typically containing both FACS and some actor-
specific expressions. Since the training markers are driven by the puppet, the
training data thus implicitly includes nonlinear effects from the puppet
without requiring a differentiable chain of deformations.

Additionally, we also utilize a data augmentation or oversampling technique to
further increase the variation of the synthetic markers $\mathbf{x}_{v}$ and
therefore avoid overfitting of the trained deep learning models. This is
achieved by randomly injecting Gaussian noise in the original markers
$\mathbf{x}_{e}=\mathbf{x}_{v}+\mathcal{N}(0,\Sigma)$. We empirically
determine the covariance $\Sigma$ by analyzing the jitter of the tracked
markers, for instance, a marker at the nose bridge has less variance than
those on the chin region. This can produce a significant quality gain in the
solved animations.

###  4.2. Salient Training Sample Selection

As is mentioned in Section 4.1, the training data for FDLS can be flexibly
generated with various combinations of simple “one-hot” single action unit
FACS expressions (producing a minimum viable solution) and more complex and
realistic data from various sources such as dynamic scans of the actor,
previously animated or solved performances, etc.

While we generally expect that more realistic and complex training data should
result in better performance, counterintuitively we observed that this is not
always the case. We found that the root cause is data imbalance – for example,
some solved performances may consist mostly of a neutral expression. This
issue can be avoided if the additional training examples are carefully
selected by the animators, but manually selecting from among many training
expressions is laborious.

To simultaneously encourage data diversity and salience, we introduce a
salient sample selection technique to automatically remove redundant examples
from the training set. This eliminates the need for tedious manual sample
selection. We reduce the number of training examples from $N$ to $M<N$ by
sequentially applying the following sample selection rule

(3)
$s_{i}=\frac{1}{N}\left(\sum_{j=1}^{N}k(\mathbf{x}^{(i)},\mathbf{x}^{(j)})\right)<\sigma,$

where $s_{i}\in\\{0,1\\}$ is a binary variable indicating whether the shape
$i$ is selected, $\sigma$ is a tunable threshold, and
$k:\mathcal{X}\times\mathcal{X}\rightarrow[0,1]$ is the similarity measure. In
the extreme case, this salient sample selection picks only a single datapoint
from the dataset $\mathcal{D}$ if it contains mostly similar expressions
according to Eq. (3), e.g., neutral non-dialogue expressions. We choose an
exponential Radial Basis Function (RBF) (Broomhead and Lowe, 1988)
representing $k(\cdot,\cdot)$ which provides a good set of salient shapes due
to its effectiveness in capturing similarities in a non-linear feature space.

###  4.3. Feature Engineering

Solving animation using solely sparse marker coordinates
$\mathbf{x}=[\mathbf{m}_{1},\ldots,\mathbf{m}_{n}]$, where
$\mathbf{m}_{i}\in\mathbb{R}^{3}$ to represent faces has its own limitations
(e.g., (Seol et al., 2016)). Sparse markers are less informative than the full
face representation (e.g., facial depth from LiDAR), and it is not possible to
fully characterize certain facial expressions using only marker coordinates.

To partially address this and simultaneously help with the limited amount of
training data available in this domain, we developed several _graph features_
to increase the information richness of the input representations, thereby
inducing robust and highly accurate deep learning solves.

In the initial enthusiasm following the success of (Krizhevsky et al., 2012),
it was argued that features emerging from neural net training should
outperform manually designed features aka ”feature engineering”. While this is
arguably true especially in computer vision and natural language processing
domains, engineered features continue to be used in state-of-the-art deep
learning – the positional encoding in transformers and Fourier/positional
encoding that distinguishes NeRF (Mildenhall et al., 2020) are prominent
examples. In practice, the need for engineered features can be justified when
it is impractical to search across a sufficient variety of architectures to
find those that result in ideal features, or to impose an inductive bias in
cases where ”correct” features might not be discovered due to limited data –
as is often the case in visual effects!

Figure 3. FDLS graph features. From left to right: pairwise distance, pairwise
directions, delta from neural to the target expression.

FDLS benefits from carefully engineered features extracted from the marker
coordinates. Fig. 3 illustrates the variants of the feature extractor
$\phi:\mathcal{X}\rightarrow\mathcal{H}$ that we design as part of the
learning-based approach (recall Section 3). These are described next.

##### Pairwise distance.

The pairwise distance features
$\boldsymbol{\phi}^{\text{dist}}\in\mathbb{R}^{n^{2}}$ are a feature vector,
whose values represent the Euclidean distances over all possible combinations
of two marker coordinates in the input representation
$\\{\mathbf{m}_{i}\\}_{i=1}^{n}$. The $k$-th element of the pairwise distance
feature $\phi^{\text{dist}}_{k}\in\mathbb{R}$ is given by
$\phi^{\text{dist}}_{k}=\left\lVert\mathbf{m}_{i}-\mathbf{m}_{j}\right\rVert_{2}.$

##### Pairwise direction.

The pairwise direction features
$\boldsymbol{\phi}^{\text{dir}}\in\mathbb{R}^{3n^{2}}$ are a collection of the
directional vectors from one coordinate to another. This gives complementary
information to eliminate the inherent ambiguity in
$\boldsymbol{\phi}^{\text{dist}}$. For instance, the Euclidean distance of two
particular coordinates between different expressions, e.g. neutral and
smiling, could be indistinguishable due to symmetry.

Analogous to the pairwise distance computation, the $k$-th coordinate
$\boldsymbol{\phi}^{\text{dir}}_{k}\in\mathbb{R}^{3}$ is computed as
$\boldsymbol{\phi}^{\text{dir}}_{k}=(\phi^{\text{dist}}_{k})^{-1}(\mathbf{m}_{i}-\mathbf{m}_{j}).$
Note that the pairwise distance and direction features are both invariant to
small slippage of the head camera.

##### Delta pose.

Lastly, we define the delta pose features
$\boldsymbol{\phi}^{\text{delta}}\in\mathbb{R}^{3n}$, which are the coordinate
differences between a particular face expression and its neutral pose. These
features are invariant to the global displacement of the marker coordinates
and have become a common representation used in traditional blendshape systems
in the graphics community (Lewis et al., 2014). Given the specific expression
$\mathbf{x}$ and the corresponding neutral pose $\mathbf{x}_{0}$, the delta
pose feature vector $\boldsymbol{\phi}^{\text{delta}}\in\mathbb{R}^{3n}$ is
calculated as $\boldsymbol{\phi}^{\text{delta}}=\mathbf{x}-\mathbf{x}_{0}$.

The complete set of input features for FDLS is the concatenation of these
three features, i.e.,
$\boldsymbol{\phi}=[\boldsymbol{\phi}^{\text{dist}},\boldsymbol{\phi}^{\text{dir}},\boldsymbol{\phi}^{\text{delta}}]\in\mathcal{H}$.
Intuitively, these features are a graph-like representation of facial
expressions that contains not only the positional information but also
pairwise relationships over the marker coordinates. These features can be
directly applied in our region-based training by extracting the features only
in a specific face region, as will be shown later. Graph features have been
used in concurrent work such as (Qi et al., 2017). Our use is somewhat
distinguished in that we intentionally do not introduce convolution. The
translational equivariance of convolution is appropriate for recognition of
general point clouds (for example an edge may appear at any location) however
the features of a canonically positioned face are in relatively fixed
locations and we seek only to capture changes due to expression.

###  4.4. Region-based Training and Solving

The deep learning regression $f_{\theta}$ defined in Section 3 is the core of
FDLS. In its implementation, we define seven deep residual networks, each
responsible for a particular face region containing a subset of the sparse
markers. We denote the face regions by $R\in\\{\textit{upper-
face},\textit{lower-
face},\textit{jaw},\textit{lips},\textit{cheek},\textit{eye-
lids},\textit{eyeballs}\\}$. The marker subset is selected according to the
semantic meaning of the face region, but a marker on the nose bridge is always
included regardless of the region to insure sufficiently diverse and spatially
supported features. Each face region is also linked only to a subset of
blendshape weights or channels that are relevant to that region. In other
words, we train
$\\{f_{\theta_{r}}:\mathcal{X}_{r}\rightarrow\mathcal{W}_{r};\forall r\in
R\\}$.

Solving with the trained deep learning models is straightforward, i.e., we
simply run the forward pass
$\hat{\mathbf{w}}_{r}=f_{\hat{\theta}_{r}}(\mathbf{x}_{r})$ for each face
region on a per frame basis. The full solved weights are the concatenation of
$\\{\hat{\mathbf{w}}_{r}\\}_{r\in R}$, which we refer to as the _raw
animation_ – later this will be used as the basis for editing or fine-tuning.
As described next, a substantial improvement in the quality of the raw
animation is obtained by extending the forward pass with two additional
techniques.

Figure 4.  The network $f_{\hat{\theta}_{r}}(\mathbf{x}_{r})$ for a regular
face region (left) and a jaw-conditioned network
$f_{\hat{\theta}_{l}}(\mathbf{x}_{l};\hat{\mathbf{w}}_{\text{jaw}})$ (right).

####  4.4.1. Jaw Kinematics Conditional Training.

Jaw kinematics plays an important role in affecting the shape and expressions
of the face, especially the lower half (Yang et al., 2019). For instance, one
can have different smiling expressions under different jaw conditions (open,
close, sideway movement, etc). In other words, the effect of face muscle
activations controlling the lips and cheeks strongly depend on the jaw
position.

For this reason, in our pipeline the jaw motions are established _before_
solving other regions. FDLS supports the artist in this process by providing a
regression specifically for the jaw position. Specifically, we denote the jaw
activation weights by $\mathbf{w}_{\text{jaw}}\in\mathbb{R}^{z}$ where in our
production face models, normally $z=3$, namely, jawOpen, jawThrust, and
jawSideways. The jaw motions $\hat{\mathbf{w}}_{\text{jaw}}$ are obtained from
computing $f_{\hat{\theta}_{\text{jaw}}}(\mathbf{x}_{\text{jaw}})$ and are
used in the following regions $l\in\\{\emph{lower-
face},\emph{lips},\emph{cheek}\\}$. We solve the animation by evaluating
$f_{\hat{\theta}_{l}}(\mathbf{x}_{l};\hat{\mathbf{w}}_{\text{jaw}})$, which is
trained by the risk minimization:

(4)
$\displaystyle\hat{\theta}_{l}:=\arg\min_{\theta_{l}}\frac{1}{N}\sum_{i=1}^{N}\ell\left(f_{\theta_{l}}(\mathbf{x}^{(i)}_{l};\mathbf{w}^{(i)}_{\text{jaw}}),\mathbf{w}^{(i)}_{l}\right),$

where $\ell:\mathcal{W}x\mathcal{W}\rightarrow\mathbb{R}$ is the standard
regression loss function. Note the use of $\mathbf{w}_{\text{jaw}}$ (the
groundtruth jaw motion) in the training, versus
$\hat{\mathbf{w}}_{\text{jaw}}$.

As we can see the solved jaw weight vector $\hat{\mathbf{w}}_{\text{jaw}}$ is
part of the inputs to the deep learning model $f_{\theta_{l}}$. However, due
to its low dimensionality, we embed it as additional nodes in a hidden layer
after the series of deep residual blocks rather than as part of the input
layer, as shown in Fig. 4.

####  4.4.2. Shape Alignment with Anchor Poses.

In real productions lasting several days or more, the positions of markers on
a single actor’s face may shift over time. There are several reasons for this.
First, although the landmarks are painted using a perforated mask and their
relative positions do not change, the mask may be placed in slightly different
positions on different days. Second, the accuracy of the tracked and
3d-reconstructed markers can be inconsistent across different shots, depending
on their complexity. Lastly, the actor may slightly gain or lose weight. These
changes can introduce a small shape mismatch between the puppet-snapped
markers and the performance-captured markers.

Figure 5. Shape alignment using an anchor pose. The green and red circles
represent the puppet-snapped markers and the performance-captured markers,
respectively.

To mitigate this problem, we introduce a simple shape alignment method that
can be performed directly by an animator before (re)running the deep learning
solver $f_{\hat{\theta}}(\cdot)$. The basic idea is to first calculate the
offsets between the synthetic markers and the shot-captured markers producing
an _anchor pose_ , i.e., a hand-picked expression at a particular frame in a
shot, and then propagate the offsets to displace the other face markers within
the entire frame range. Specifically, suppose that we have a performance shot
with $T$ frames and an anchor pose at frame $a$ is chosen. The artist then
sets the anchor weights $\mathbf{w}_{a}$ such that $g(\mathbf{w}_{a})$
portrays the desirable anchor pose. The shape alignment is executed as
follows:

(5) $\begin{split}&\mathbf{d}_{a}=\mathbf{x}_{a}-g(\mathbf{w}_{a}),\\\
&\tilde{\mathbf{x}}_{f}=\mathbf{x}_{f}-\mathbf{q}_{fa}\odot\mathbf{d}_{a},\
f\in\\{1,...,T\\},\end{split}$

where $\mathbf{d}_{a}$ are the marker offsets and
$\mathbf{q}_{fa}\propto\exp{(-\|\mathbf{x}_{a}-\mathbf{x}_{f}\|^{2}_{2})}$
denotes the weighting factors of the alignment by calculating the pose
similarity. After this we can rerun the deep learning solver
$\tilde{w}=f_{\hat{\theta}}(\tilde{\mathbf{x}}_{f}),\forall
f=\\{1,\ldots,T\\}$ _without retraining_ to produce a more accurate animation.
Note that this shape alignment process can be done iteratively by choosing a
few other anchor poses as needed – for the first anchor pose, all elements of
$\mathbf{q}$ are set to one.

###  4.5. Fine-tuning Optimization

Editing can optionally be performed once the raw animation has been solved by
the deep learning model
$\hat{\mathbf{w}}_{u}=f_{\hat{\theta}}(\mathbf{x}_{u})$. In cases with unusual
expressions, we sometime see a small discrepancy between the solved expression
of the actor puppet $g(\hat{\mathbf{w}}_{u})$ and the actual captured
expression $\mathbf{x}_{u}$, mainly because the shape $\mathbf{x}_{u}$ has not
been well represented in the training dataset for constructing
$f_{\hat{\theta}}$. The artists can simply adjust the solved weights (most of
the time only a few channels are needed) to get more accurate expressions.

To make the aforementioned process more scalable, we provide a _fine-tuning
optimization_ as a post-processing step in FDLS. It essentially further
minimizes the error $\|g(\hat{\mathbf{w}}_{u})-\mathbf{x}_{u}\|_{2}$. We run
the minimization by solving the face matching problem in Eq. (2), in which the
deep learning solved weights are set as the starting point for a minimization
using the L-BFGS algorithm as the optimizer (Liu and Nocedal, 1989). We found
this provides better results than starting the minimization from zero or a
random point. Note that we also form a restricted blendshape basis (matrix
$\mathbf{B}$ in Eq.(2)) that only includes artist-selected degrees of freedom
to be finetuned while the remaining weights remain frozen.

With this implementation, we provide the artists with flexible options such as
choosing only a specific frame range, partially selecting the face markers as
objects of comparison, and selecting only a few blendshape weights to be
finetuned. This makes the fine-tuning optimization an interactive tool as part
of an overall editing flow to efficiently finalize high quality animations.

##  5\. Experiments

In this section we review the end-to-end FDLS pipeline and present the main
results as well as ablation tests. While the use case shown here is for a
specific actor and a single shot performance, the workflow is generic for any
actor and performance capture.

Figure 6. FDLS workspace and GUI. The system is implemented using the
commercial Maya API (Autodesk, Inc., 2019). Figure 7. A complex FDLS solve
example. (a) raw solve with shape alignment from a single neutral-ish frame
(top), finetune with the additional anchor pose outlined in red (middle), and
the final manual edits from artist on top of the FDLS solve (bottom). (b) and
(c) are the RMSE performance of ablation experiments at the frame marked with
star with/without an anchor pose and finetuning, and on the salient data
selection, respectively. Note that the RMSE generally includes an irreducible
offset due to slight mismatch between the actor and puppet virtual marker
positions.

Next we describe the implementation details of the FDLS training and solving
stages as depicted in Fig. 2.

###  5.1. Training

As described in Section 4.1, we synthetically manufacture the training data in
the form of a series of sparse markers by assigning animation weights to the
blendshape channels. In the FDLS training, two types of training animations
are utilized: “one-hot” FACS expressions and the range of motion (ROM) of an
actor.

The one-hot FACS animation has individual FACS expressions such as jaw drop
(AU26) and lip corner puller (AU12) at successive frames, thus representing
the full extent of the blendshape basis.

By using only the one-hot FACS training animation, we can already construct a
_minimum viable solver_ before more advanced training animation is available.

The ROM captures the motion of the specific actor. This is more realistic than
the one-hot animation, as some people have difficulty activating individual
muscles in isolation. One can think of the FACS-based animation as determining
the range and extremes of individual muscles, while the ROM provides
information about their distribution, for example, what muscle combinations
actually occur and do not occur for the particular actor. In addition, the ROM
allows an expert artist to specify exactly which blendshape weights contribute
to an expression, thus guiding the solver to disambiguate cases where nearly
(but not exactly) identical marker positions can be produced with different
weight configurations.

###  5.2. Solving

Fig. 6 shows the FDLS workspace and GUI. Note that we do not require the
artist to specify any parameters to perform solving, thus the tool is
convenient to use without significant artist training. Our users have reported
that FDLS system takes only about one week to train a new animator, which
boosts the production speed.

After the workspace has been setup (Fig. 6), FDLS allows the artist to pick a
neutral-ish frame as an anchor frame (Section 4.4.2), and correct the jaw
animation given the raw jaw solve, then run the jaw-conditioned networks
(Section 4.4.1). After raw solve, one or more additional anchor poses can be
added to refine the projected markers. Finally, the post-finetuning optimizes
the face conditioned on jaw channels to match the performance markers.

Fig. 7(a) shows a complex FDLS solve that illustrates most aspects of the
system. The top row depicts the raw solved expressions induced by the region-
based models $\\{f_{\theta_{r}}(\cdot)\\}_{r\in R}$ after shape alignment
using one neutral-ish frame. The middle row displays the solved expressions
with shape alignment using the additional anchor pose highlighted in red
followed by finetuning. This shows a substantial improvement in that the
result is close to the final desired expressions (bottom).

Fig. 7(b) compares the performance of post-finetuning versus the raw solve,
with or without an anchor pose at the marked frame. The minimization ensures
the RMSE loss value decreases (e.g., curves A and B), however the error
remains high across frames in this case. By adding one anchor pose at around
frame 55 (shown in (a) with the red box), the error is globally decreased with
no further effort.

Fig. 7(c) shows the results of salient frame selection by varying the
threshold $\sigma$ in (3). Specifically, it compares the test performance with
$\sigma=0.1$, $\sigma=0.3$, $\sigma=0.5$, and the full dataset, corresponding
to using $50\%$, $66\%$, $83\%$ and $100\%$ of the original dataset,
respectively. One can see that the network trained with data selected using
$\sigma=0.3$ (orange) has better performance than the others, including the
network trained with the full dataset (red).

Many performances include a certain amount of repeated information, for
instance, in some performances a large percentage of the frames show a mostly
neutral expression. The experiment in Fig. 7(c) provides evidence of an
important point, which is that the solve actually _improves_ when this
redundant data is removed. However, a limitation of this approach is that the
$\sigma$ must be empirically tuned for the particular dataset. If $\sigma$ is
set to an extreme (e.g., $\sigma=0.1$) or moderate value (e.g., $\sigma=0.5$)
it may give poor results.

##  6\. Conclusion and future work

This paper presents a human-in-the-loop deep learning based approach to facial
animation solving. The FDLS system is a pioneering application of deep
learning to facial solving in movie production. It was initiated in 2016 and
has seen ongoing development and improvement since then. It presents a simple
and light-weight training and solving pipeline to our artists. The approach is
suitable for driving sophisticated and non-linear blendshape rigs with
hundreds of parameters. FDLS produces production quality results while
requiring limited training data and artist effort. The design trade-offs and
several specific features of the system are guided by consideration of artist
practice and differ from other published work.

The graph features have proven successful at allowing accurate solves with
limited training data. The quality of the intermediate solution can be
verified and adjusted if needed following both the jaw estimation and prior to
the post fine-tuning, and since FDLS produces interpretable parameters, the
freedom to do further editing on the final solution is fully preserved. The
anchor pose mechanism allows the network to adapt to slight changes in both
the marker positions and the actor’s face shape without re-training.

In our experience animators can learn to use the system in approximately a
week and a large majority of shots require little or no human editing. While
we could cherry pick results to provide an “objective” characterization of the
production performance, in all honesty such a characterization is almost
meaningless. This is due to wide variety of different performance and
retargeting situations, the different ways that FDLS can be used, the quality
of training data, choice of anchor poses, and other factors, and particularly
because of the ultimate perceptual evaluation of the results. Instead, we
highlight that FDLS succeeds as a system that can be employed in _all_ these
scenarios – from fully automatic solves to extremely challenging shots that
require iterative solution and evolving guidance from a supervisor.

One of the disadvantages of using a sparse marker set is the ambiguity of the
lip positions in certain expressions. In the future we would like to enhance
the solver with additional information such the lip contours or LIDAR depth.

###### Acknowledgements.

We thank Joe Letteri, Marco Revelant, Luca Fascione, Dejan Momcilovic, Stephen
Cullingford, Stuart Adcock, Allison Orr, David Luke, Millie Maier, Andrew
Moffat, Kenneth Gimpelson, Nivedita Goswami, and Zhicheng Ye for supporting
this project. Moreover, we appreciate the anonymous reviewers for their
suggestions.

## References

  * (1)
  * Autodesk, Inc. (2019) Autodesk, Inc. 2019\.  _Maya_.  https://autodesk.com/maya
  * Beeler and Bradley (2014) Thabo Beeler and Derek Bradley. 2014.  Rigid Stabilization of Facial Expressions.  _ACM Trans. Graph._ 33, 4 (2014), 44:1–44:9.
  * Bermano et al. (2015) Amit Bermano, Thabo Beeler, Yeara Kozlov, Derek Bradley, Bernd Bickel, and Markus Gross. 2015\.  Detailed Spatio-Temporal Reconstruction of Eyelids.  _ACM Trans. Graph._ 34, 4, Article 44 (jul 2015).
  * Berson et al. (2019) Eloïse Berson, Catherine Soladié, Vincent Barrielle, and Nicolas Stoiber. 2019.  A Robust Interactive Facial Animation Editing System.  _Motion, Interaction and Games_ (Oct 2019).
  * Berson et al. (2020) Eloïse Berson, Catherine Soladié, and Nicolas Stoiber. 2020\.  Intuitive Facial Animation Editing Based On A Generative RNN Framework.  _Computer Graphics Forum_ 39, 8 (Nov 2020), 241–251.
  * Bhat et al. (2013) Kiran S. Bhat, Rony Goldenthal, Yuting Ye, Ronald Mallet, and Michael Koperwas. 2013.  High Fidelity Facial Animation Capture and Retargeting with Contours. In _Proceedings of the 12th ACM SIGGRAPH/Eurographics Symposium on Computer Animation_ (Anaheim, California) _(SCA ’13)_. Association for Computing Machinery, New York, NY, USA, 7–14.
  * Bickel et al. (2007) Bernd Bickel, Mario Botsch, Roland Angst, Wojciech Matusik, Miguel Otaduy, Hanspeter Pfister, and Markus Gross. 2007.  Multi-Scale Capture of Facial Geometry and Motion. In _ACM SIGGRAPH 2007 Papers_ (San Diego, California) _(SIGGRAPH ’07)_. Association for Computing Machinery, 33–es.
  * Blanz and Vetter (1999) Volker Blanz and Thomas Vetter. 1999.  A Morphable Model for the Synthesis of 3D Faces. In _Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques_ _(SIGGRAPH ’99)_. ACM Press/Addison-Wesley Publishing Co., USA, 187–194.
  * Boyd and Vandenberghe (2004) Stephen Boyd and Lieven Vandenberghe. 2004.  _Convex Optimization_.  Cambridge University Press.
  * Bradley et al. (2010) Derek Bradley, Wolfgang Heidrich, Tiberiu Popa, and Alla Sheffer. 2010.  High Resolution Passive Facial Performance Capture.  _ACM Trans. Graph._ 29, 4 (2010).
  * Broomhead and Lowe (1988) D.S. Broomhead and D. Lowe. 1988.  Multivariable Functional Interpolation and Adaptive Networks.  _Complex Systems_ 2 (1988), 321–355.
  * Cetinaslan et al. (2015) Ozan Cetinaslan, Verónica Orvalho, and J.P. Lewis. 2015.  Sketch-Based Controllers for Blendshape Facial Animation. In _Eurographics_. 25–28.
  * Choe et al. (2001) Byoungwon Choe, Hanook Lee, and Hyeong seok Ko. 2001\.  Performance-Driven Muscle-Based Facial Animation.  _The Journal of Visualization and Computer Animation_ 12 (2001), 67–79.
  * Choi et al. (2016) Byungkuk Choi, Roger Blanco i Ribera, J.P. Lewis, Yeongho Seol, Seokpyo Hong, Haegwang Eom, Sunjin Jung, and Jun-yong Noh. 2016.  SketchiMo: sketch-based motion editing for articulated characters.  _ACM Trans. Graph._ 35, 4 (2016), 146:1–146:12.
  * Dinev et al. (2018) Dimitar Dinev, Thabo Beeler, Derek Bradley, Moritz Bächer, Hongyi Xu, and Ladislav Kavan. 2018.  User‐Guided Lip Correction for Facial Performance Capture.  _Computer Graphics Forum_ 37 (2018).
  * Disney (2022) Disney. 2022.  _Anyma_.  https://studios.disneyresearch.com/anyma
  * Egger et al. (2020) Bernhard Egger, William A. P. Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhöfer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, Christian Theobalt, Volker Blanz, and Thomas Vetter. 2020.  3D Morphable Face Models - Past, Present, and Future.  _ACM Trans. Graph._ 39, 5 (2020), 157:1–157:38.
  * Ekman and Friesen (1978) Paul Ekman and Wallace V Friesen. 1978.  Facial action coding system.  _Environmental Psychology & Nonverbal Behavior_ (1978).
  * Fascione et al. (2017) Luca Fascione, J.P. Lewis, and Iain Matthews. 2017\.  FACETS Sci-Tech Academy Award.  http://oscars.org.
  * Furukawa and Ponce (2009) Yasutaka Furukawa and Jean Ponce. 2009.  Dense 3D motion capture for human faces.  _IEEE Conference on Computer Vision and Pattern Recognition_ (Jun 2009).
  * Fyffe et al. (2015) Graham Fyffe, Andrew Jones, Oleg Alexander, Ryosuke Ichikari, and Paul Debevec. 2015.  Driving High-Resolution Facial Scans with Video Performance Capture.  _ACM Trans. Graph._ 34, 1, Article 8 (2015).
  * Garrido et al. (2016) Pablo Garrido, Michael Zollhöfer, Chenglei Wu, Derek Bradley, Patrick Pérez, Thabo Beeler, and Christian Theobalt. 2016\.  Corrective 3D Reconstruction of Lips from Monocular Video.  _ACM Trans. Graph._ 35, 6, Article 219 (2016).
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep Residual Learning for Image Recognition. In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 770–778.
  * Huang et al. (2011) Haoda Huang, Jinxiang Chai, Xin Tong, and Hsiang-Tao Wu. 2011\.  Leveraging Motion Capture and 3D Scanning for High-Fidelity Facial Performance Acquisition. In _ACM SIGGRAPH 2011 Papers_ (Vancouver, British Columbia, Canada) _(SIGGRAPH ’11)_. Association for Computing Machinery, New York, NY, USA, Article 74.
  * Kingma and Ba (2015) Diederick P Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization. In _International Conference on Learning Representations (ICLR)_.
  * Klehm et al. (2015) Oliver Klehm, Fabrice Rousselle, Marios Papas, Derek Bradley, Christophe Hery, Bernd Bickel, Wojciech Jarosz, and Thabo Beeler. 2015\.  Recent Advances in Facial Appearance Capture.  _Comput. Graph. Forum_ 34, 2 (2015), 709–733.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012\.  ImageNet Classification with Deep Convolutional Neural Networks. In _Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1_ (Lake Tahoe, Nevada) _(NIPS’12)_. Curran Associates Inc., Red Hook, NY, USA, 1097–1105.
  * Laine et al. (2017) Samuli Laine, Tero Karras, Timo Aila, Antti Herva, Shunsuke Saito, Ronald Yu, Hao Li, and Jaakko Lehtinen. 2017\.  Production-level facial performance capture using deep convolutional neural networks. In _Proceedings of the ACM SIGGRAPH / Eurographics Symposium on Computer Animation, Los Angeles, CA, USA, July 28-30, 2017_ , Joseph Teran, Changxi Zheng, Stephen N. Spencer, Bernhard Thomaszewski, and KangKang Yin (Eds.). Eurographics Association / ACM, 10:1–10:10.
  * Lamarre et al. (2018) Mathieu Lamarre, J.P. Lewis, and Etienne Danvoye. 2018\.  Face Stabilization by Mode Pursuit for Avatar Construction in the Universe. In _2018 International Conference on Image and Vision Computing, IVCNZ 2018_. IEEE, 1–6.
  * Lewis et al. (2014) J. P. Lewis, Ken Anjyo, Taehyun Rhee, Mengjie Zhang, Fred Pighin, and Zhigang Deng. 2014\.  Practice and Theory of Blendshape Facial Models. In _Eurographics 2014 - State of the Art Reports_ , S. Lefebvre and M. Spagnuolo (Eds.).
  * Li and Deng (2008) Qing Li and Zhigang Deng. 2008.  Orthogonal-Blendshape-Based Editing System for Facial Motion Capture Data.  _IEEE Comput. Graph. Appl._ 28, 6 (nov 2008), 76–82.
  * Liu and Nocedal (1989) Dong C. Liu and Jorge Nocedal. 1989.  On the limited memory BFGS method for large scale optimization.  _Mathematical Programming_ 45, 1 (01 Aug 1989), 503–528.
  * Ma et al. (2016) Wan-Chun Ma, Mathieu Lamarre, Etienne Danvoye, Chongyang Ma, Manny Ko, Javier von der Pahlen, and Cyrus A. Wilson. 2016.  Semantically-aware blendshape rigs from facial performance measurements. In _SIGGRAPH ASIA 2016, Macao, December 5-8, 2016 - Technical Briefs_ , Johannes Kopf and Phillip Chi-Wing Fu (Eds.). ACM, 3\.
  * Mildenhall et al. (2020) Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020\.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In _ECCV_.
  * Moser et al. (2021) Lucio Moser, Chinyu Chien, Mark Williams, Jose Serra, Darren Hendler, and Doug Roble. 2021\.  Semi-Supervised Video-Driven Facial Animation Transfer for Production.  _ACM Trans. Graph._ 40, 6, Article 222 (dec 2021), 18 pages.
  * Moser et al. (2018) Lucio Moser, Mark Williams, Darren Hendler, and Doug Roble. 2018.  High-Quality, Cost-Effective Facial Motion Capture Pipeline with 3D Regression. In _ACM SIGGRAPH 2018 Talks_ (Vancouver, British Columbia, Canada) _(SIGGRAPH ’18)_. 2 pages.
  * Osipa (2010) Jason Osipa. 2010\.  _Stop Staring: Facial Modeling and Animation Done Right, 3rd Ed._ Sybex.
  * Qi et al. (2017) Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. 2017\.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_. 5099–5108.
  * Seo et al. (2011) Jaewoo Seo, Geoffrey Irving, J.P. Lewis, and Junyong Noh. 2011\.  Compression and direct manipulation of complex blendshape models.  _ACM Trans. Graph._ 30, 6, Article 164 (Dec. 2011), 164:1–164:10 pages.
  * Seol and Cozens (2019) Yeongho Seol and Michael Cozens. 2019.  Interactive editing of performance-based facial animation. In _SIGGRAPH Asia 2019 Technical Briefs, SA 2019, Brisbane, QLD, Australia, November 17-20, 2019_. ACM, 61–64.
  * Seol et al. (2012) Yeongho Seol, J.P. Lewis, Jaewoo Seo, Byungkuk Choi, Ken Anjyo, and Junyong Noh. 2012\.  Spacetime expression cloning for blendshapes.  _ACM Trans. Graph._ 31, 2, Article 14 (April 2012), 14:1–14:12 pages.
  * Seol and Lewis (2014) Yeongho Seol and J. P. Lewis. 2014.  Tuning Facial Animation in a Mocap Pipeline. In _ACM SIGGRAPH 2014 Talks_ (Vancouver, Canada) _(SIGGRAPH ’14)_. Association for Computing Machinery, New York, NY, USA, Article 13, 1 pages.
  * Seol et al. (2016) Yeongho Seol, Wan-Chun Ma, and J. P. Lewis. 2016.  Creating an Actor-Specific Facial Rig from Performance Capture. In _Proceedings of the 2016 Symposium on Digital Production_ (Anaheim, California) _(DigiPro ’16)_. Association for Computing Machinery, New York, NY, USA, 13–17.
  * Seol et al. (2011) Yeongho Seol, Jaewoo Seo, Paul Hyunjin Kim, John P. Lewis, and Junyong Noh. 2011.  Artist friendly facial animation retargeting.  _ACM Trans. Graph._ 30, 6 (2011), 162.
  * Serra et al. (2022) Jose Serra, Mark Williams, and Lucio Moser. 2022\.  Accelerating Facial Motion Capture with Video-Driven Animation Transfer. In _ACM SIGGRAPH 2022 Talks_ (Vancouver, BC, Canada) _(SIGGRAPH ’22)_. Association for Computing Machinery, New York, NY, USA, Article 19, 2 pages.
  * Seymour (2019) Mike Seymour. 2019\.  _Face it Will: Gemini Man_.  https://www.fxguide.com/fxfeatured/face-it-will-gemini-man
  * Seymour (2022) Mike Seymour. 2022\.  _Deep Dive on Wētā FX Face Fabrication System_.  https://www.fxguide.com/fxfeatured/deep-dive-on-weta-fx-face-fabrication-system
  * Smith et al. (2017) Alex Smith, Sven Pohle, Wan-Chun Ma, Chongyang Ma, Xian-Chun Wu, Yanbing Chen, Etienne Danvoye, Jorge Jimenez, Sanjit Patel, Mike Sanders, and Cyrus A. Wilson. 2017.  Emotion challenge: building a new photoreal facial performance pipeline for games. In _Proc. ACM SIGGRAPH Digital Production Symposium_ , Christopher Horvath, Cary B. Phillips, Andrew Pearce, Corban Gossett, and Stephen N. Spencer (Eds.). ACM, 8:1–8:2.
  * Tagliasacchi et al. (2016) Andrea Tagliasacchi, Sofien Bouaziz, Mark Pauly, and Hao Li. 2016.  Modern techniques and applications for real-time non-rigid registration. In _SIGGRAPH ASIA 2016, Macao, December 5-8, 2016 - Courses_. 11:1–11:25.
  * Vapnik (1998) Vladimir Vapnik. 1998\.  _Statistical learning theory_.  Wiley.
  * Various (2022) Various 2022.  Commercial performance capture systems:.  https://di4d.com, https://www.dynamixyz.com, https://www.synthesia.io, https://studios.disneyresearch.com/medusa, https://image-metrics.com.
  * Williams (1990) Lance Williams. 1990\.  Performance-Driven Facial Animation.  _SIGGRAPH Comput. Graph._ 24, 4 (sep 1990), 235–242.
  * Yang et al. (2019) Wenwu Yang, Nathan Marshak, Daniel Sýkora, Srikumar Ramalingam, and Ladislav Kavan. 2019\.  Building anatomically realistic jaw kinematics model from data.  _The Visual Computer_ 35, 6-8 (2019), 1105–1118.
  * Zhang et al. (2004) Li Zhang, Noah Snavely, Brian Curless, and Steven M. Seitz. 2004\.  Spacetime Faces: High Resolution Capture for Modeling and Animation.  _ACM Trans. Graph._ 23, 3 (aug 2004), 548–558.
  * Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In _CVPR_.
  * Zollhöfer et al. (2018) M. Zollhöfer, J. Thies, P. Garrido, D. Bradley, T. Beeler, P. Pérez, M. Stamminger, M. Nießner, and C. Theobalt. 2018.  State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications.  _Computer Graphics Forum_ 37, 2 (2018), 523–550.

FDLS: Supplementary Material

##  7\. Data Acquisition

The data acquisition step is crucial to generate the FDLS training tuples
$\mathcal{D}=\\{(\mathbf{x}^{(i)}_{v},\mathbf{w}^{(i)}_{v})\\}_{i=1}^{N}$. It
uses the following three components: neutral markers, the facial puppet, and
facial motion (aka blendshape weights).

The neutral markers are created in a motion capture session by triangulating
from images of the actor’s neutral expression captured through multiple
cameras. To make the static neutral markers animatable, we project each marker
on the facial puppet.

The facial motion is created by the artist and comprises FACS expressions and
range of motion (ROM). The FACS motion contains about 500 frames. This
includes linear interpolation of the one-hot controls, yielding nonlinear
motion of the corresponding synthetic markers. The range of motion (ROM)
generally has about 2000 frames of phoneme and dialogue clips. In addition,
the artist can add the animations of previously approved shots to enhance the
diversity of the training set.

We use Autodesk Maya (Autodesk, Inc., 2019) to generate the synthetic markers
for the training tuples. Artist-created animations are applied to the facial
controller attributes to drive the facial puppet. The performance of the
generating process is about 5 frames per second with an AMD Ryzen Threadripper
PRO 3995WX 64-Core CPU and Quadro RTX A5000 24GB graphics card.

##  8\. Architecture and Training Setup

The FDLS training is performed offline in our production environment. We use
the Adam optimizer (Kingma and Ba, 2015) to train the neural networks for each
facial region. The region-specific hyper-parameters are given in the following
table:

Region network hyperparameters
---
Group |  lr |  dp |  bs |  #ep |  l2 |  time
lower-face |  1e-4 |  0.01 |  64 |  450 |  1e-5 |  20m
upper-face |  5e-4 |  0.01 |  64 |  300 |  1e-5 |  15m
cheek |  1e-4 |  0.01 |  64 |  300 |  1e-5 |  15m
eyelids |  2e-4 |  0.0 |  64 |  300 |  1e-5 |  10m
eyeball |  1e-4 |  0.0 |  64 |  250 |  1e-5 |  5m
jaw |  1e-4 |  0.01 |  64 |  300 |  1e-5 |  15m
lips |  1e-4 |  0.01 |  64 |  450 |  1e-5 |  20m

The learning rate, dropout, batch size, epochs, L2-norm coefficient and
training time are denoted as lr, dp, bs, ep, l2, time, respectively. We
highlight that the 3d eye marker set shown in Fig. 8 is constructed by ray-
casting from the upper camera center through the 2d tracked eye markers in the
dewarped head camera footage. The intersection between the ray and the face
puppet determines the 3d eye marker. We find that the projected eye marker set
is relatively more stable than other regions, thus we remove dropout in
training the eyelids and eyeball.

The next table shows the per-region network architectures:

Region network architectures
---
Group |  feature |  rb dim |  #rb |  jaw cond?
lower-face |  dist-delta |  800 |  3 |  ✓
upper-face |  dist-dir |  300 |  3 |
cheek |  dist-delta |  500 |  2 |  ✓
eyelids |  dist-dir |  300 |  2 |
eyeball |  dist-dir |  150 |  2 |
jaw |  dir-delta |  800 |  2 |
lips |  dir |  600 |  3 |  ✓

All the neural networks are fully-connected ResNets (HeZRS15), each with an
input layer followed with a varied number of residual blocks (#rb) finally
followed by layers to match target dimensions. Furthermore, the jaw kinematic
conditioning values are concatenated to the first hidden layer following a
series of residual blocks.

##  9\. Facial Region Definitions

Controls per region. The following table shows the FACS controls (i.e.
blendshape weights or channels) for each face region. The face puppet in our
paper is constructed with symmetrically split channel configuration, for
instance, cheekRaiserR and cheekRaiserL represent the right and left-side
cheekRaiser respectively. In the table we list the generic (not split) name
for brevity.

FACS controls used per facial region
---
upper-face |  cheek |  lower-face |  lips
innerBrowRaiser |  cheekRaiser |  incisivus |  lipPuckerer
outerBrowRaiser |  noseWrinkler |  lipRaiser |  lipFunneler
procerus |  cheekPuff |  lipDepressor |  lipTightener
browLowerer |  noseDepressor |  chinRaiser |  innerOO
|  nasolabialFurrow |  lipPressor |  outerOO
|  lidTightener |  incisivus |
|  squint |  |

“OO” denotes the orbicularis oris muscle. These remaining regions were omitted
from the table due to limited space: jaw: jawOpen, jawSideway, jawThrust.
eyeball: eyeLeftRight, eyeUpDown. eyelids: eyeClose.

Markers per region. Fig. 8 shows the marker selection for each facial region.

Notice that although upper-face/cheek and lower-face/lips have the same marker
set, the features of each group are different and thus the neural network
training behaves differently. The “lips” group especially benefits from the
pairwise direction feature as the depth information is particularly important
in this region.

Figure 8. The per-region marker groups.


11institutetext: Institute WeST, University of Koblenz-Landau, Germany
11email<EMAIL_ADDRESS>
#  _ECOL_ : Early Detection of COVID Lies Using Content, Prior Knowledge and
Source Information

Ipek Baris  Zeyd Boukhers

###### Abstract

Social media platforms are vulnerable to fake news dissemination, which causes
negative consequences such as panic and wrong medication in the healthcare
domain. Therefore, it is important to automatically detect fake news in an
early stage before they get widely spread. This paper analyzes the impact of
incorporating content information, prior knowledge, and credibility of sources
into models for the early detection of fake news. We propose a framework
modeling those features by using BERT language model and external sources,
namely Simple English Wikipedia and source reliability tags. The conducted
experiments on CONSTRAINT datasets demonstrated the benefit of integrating
these features for the early detection of fake news in the healthcare domain.

###### Keywords:

Fake news detection Deep learning Prior knowledge

##  1 Introduction

Social media is replacing traditional media as a source of information due to
the ease of access, fast sharing, and the freedom to create content. However,
social media is also responsible for spreading the massive amount of fake news
[31]. Fake news propagation can manipulate significant events such as
political elections or severely damage the society during crisis [14]. For
example, a rumor that initially occurred in a UK tabloid paper claimed that
neat alcohol could cure COVID-19. As a consequence of the spread of this
rumor, hundreds of Iranians have lost their lives due to alcohol
poisoning111https://www.independent.co.uk/news/world/middle-east/iran-
coronavirus-methanol-drink-cure-deaths-fake-a9429956.html. Therefore, it is
crucial to detect potentially false claims early before they reach large
audiences and cause damage.

Since the U.S presidential elections in 2016, tremendous efforts have been
devoted by the research community to automate fake news detection. Most prior
studies rely on leveraging propagation information, user engagement and
content of news/social media posts [39, 26, 11]. However, the methods relying
on propagation information [37] and/or on user engagement [16, 3, 2, 27] are
not applicable for detecting fake news at an early stage since they are only
available when the news starts disseminating. The methods solely based on
content (e.g [2, 33]) could be misguided by claims that require additional
context for their interpretation. For instance, the post in Figure 1 may sound
very plausible for readers who know the relationship between the politicians
mentioned in the post. However, the source is a satiric website which
indicates that the post is fake.

Commonly, a lot of fake claims/news that are partially sharing similar content
occur in different sources in a relatively long time-span. For example, in the
healthcare domain, the most common fake claims are about unproven and
alternative cures (e.g using alcohol) against diseases [28, 34]. These types
of fake news have also been observed during COVID-19 [18]. The assumption is
that early published claims are fact-checked and can be employed to detect
later ones. Therefore, investigating previously published claims/news can
provide important information in determining the truthfulness of posts [12,
25]. Specifically, encoding previously published fake news in healthcare
domain as prior knowledge, content and source information could help detecting
posts that would potentially be missed by solely content-based methods.

Figure 1: System architecture of _ECOL_ when metadata is unavailable.

In this paper, we investigate and analyse those intrinsic and extrinsic
features to detect fake news in the healthcare domain. To this end, we propose
a Neural Network model that integrates (1) a contextual representation of the
news content, (2) a representation of the relationship to similar validated
fake news and (3) a source representation the embeds the reliability of the
source. The main contributions of this paper are as follows:

  * •

We introduced a classification framework that models prior knowledge, content
and source information by using BERT[7] and reliability tags to predict the
truthfulness of social media posts in the healthcare domain. We share our
source code and trained models on
Github222https://github.com/isspek/FakeNewsDetectionFramework.

  * •

To evaluate the effectiveness of proposed model, we conducted an extensive
experiment on the CONSTRAINT dataset. According to the obtained F1 score,
_ECOL_ is ranked 14 among 167 submissions at CONSTRAINT competition [21].

The rest of the paper is organized as follows. Section 2 briefly presents
related work. Section 3 describes the proposed framework. Section 4 presents
the dataset, baselines, ablation models and implementation details. Section 5
presents and discusses the results of the experiment. Finally, Section 6
concludes the paper and gives insights on future work.

##  2 Related Work

This section presents related work of textual content-based methods and
approaches using external information to assess truthfulness.

###  2.1 Textual Content based Fake News Detection on Social Media

Textual content-based methods for fake news detection on social media [39, 26,
11] vary from traditional machine learning models [3, 13, 38] to neural
networks [23, 2, 7]. For instance, the methods [3, 13] leverage features such
as the sentiment and metadata information (e.g. replies, likes of social media
posts) in addition to statistics derived from both post and metadata. Zhou et
al. [38] investigate the features derived by the theories in social and
forensic psychology. As examples of neural network models, CNN [33], RNN [4]
and most recently context-aware language models such as ELMo [23, 2] and BERT
[9, 7] have also been used. While CNN, RNN models ignore the context
information, BERT and ELMo can learn the different meanings of the words
depending on the context. Among context-aware language models, BERT has shown
state-of-art results in many NLP tasks [7]. Therefore, in our study, we encode
content information with BERT.

###  2.2 Extrinsic Features for Determining Truthfulness of Claims

As an extrinsic feature, encoding the top $N$ relevant evidence pages
retrieved by commercial search engines (e.g. Google) has been a widely
preferred approach [29, 1, 15] to determine the truthfulness of claims or
posts. However, Augenstein et al. [1] stated that this method has a drawback
of affecting veracity assessments when the results change over time. Firstly,
this drawback could prevent reproducible results. Secondly, it would not be
applicable to evaluate the posts which cannot be supported or denied with
evidences when initially occurred.

Another extrinsic feature is leveraging the information of previously analyzed
claims. Claim similarity between previously fact-checked claims in the
political domain has been studied as part of fact-checking system [12, 6, 36]
or as an information retrieval task [25]. Those studies aim to find claims or
posts reporting about the same event. However, we aim to learn the similarity
of the posts with previously detected fake news in the healthcare domain, not
necessarily reporting about the same event.

Lastly, the credibility of user-profiles [16, 37] and source websites [17, 8,
20, 10] are also strong extrinsic features for determining truthfulness at an
early stage [19]. To detect rumors and fake news on social media, Yuan et al.
[37] used user credibility as weak signals in their graph-based neural network
model. Li et al. [16] combined the credibility of users with post features and
post embeddings for rumor detection. These two studies require propagation and
metadata information to encode the aforementioned features. Other studies [17,
20, 10] focus on determining the credibility of sources that mostly report
political news. Using only the credibility information of political news
sources could be limited. Therefore, we leverage the content of the Simple
English Wikipedia and source credibility information by Gruppi et al. [10] in
our study.

In summary, _ECOL_ utilizes the content of the post, its similarity to prior
knowledge, and the credibility of its embedded source URLs, to detect their
truthfulness early.

##  3 _ECOL_ Approach

The overall architecture of _ECOL_ framework is illustrated in Figure 1.
Firstly, as a pre-processing step, the framework (1) lowers all cased words,
(2) fixes the Unicode errors, (3) translates the text to the closest ASCII
representation and (4) replaces exclusive content with tags. Specifically,
URLs are replaced with with <URL>, emails with <EMAIL>, numbers with <NUMBER>,
digits with <DIGIT>, phone numbers with <PHONE> and currency symbols with
<CUR>. Secondly, the framework encodes (1) content information from solely
posts (Section 3.1), (2) relation with top 10 relevant fake news in health
domain (Section 3.2) and (3) the sources, that are embedded within the post,
by concatenating reliability tags and their Simple Wiki descriptions (Section
3.3). Next, it concatenates the encoded features and feeds them into a fully
connected layer. Finally, a softmax layer classifies the features as fake or
real to express their truthfulness.

###  3.1 Content (C)

In this unit, _ECOL_ tries to capture the writing style of fake and true
posts. To well learn the content information, we encode the texts with BERT
[7] which is a context-aware language model based on the transformer network
and has been pre-trained on massive text corpus [30]. BERT learns specific NLP
tasks after fine-tuning its pre-trained models. In order to obtain the post
representations, we encode the input sequences of the post with the uncased
base version of pre-trained BERT by using the transformers library [35].
Uncased base BERT consists of 12 layers and 12 attention heads and outputs
768-dimensional vectors for each word in the post. The first token of the
input sequences is called [CLS] and indicates the classification label. The
final hidden state of the [CLS] is used as a content representation. We tune
the maximum size of texts to 128 and padded short texts.

###  3.2 Prior Knowledge (PK)

To leverage the relation of news event with previously published and proved
fake news, we encoded a post’s relation with a set of similar fake news
disseminated before COVID-19. Given a post as a query, an ad-hoc semantic
search engine retrieves the top 10 fake news from a repository indexed with
fake news in the health domain. To obtain a fake news vector (FN), we encoded
each retrieved news with the BERT and took their average. Afterwards, we
computed the _cosine_ similarity between FN and the post (P) as follows
$\mathbf{R}=cos(\mathbf{FN},\mathbf{P})$, where $\mathbf{R}$ is an one
dimensional relatedness vector.

3.2.1 Ad Hoc Search and Indexing

To obtain the similar fake news, we used Elasticsearch
333docker.elastic.co/elasticsearch/elasticsearch:7.6.1 to retrieve validated
fake news from FakeHealth [5] which is a dataset containing real and fake news
stories and releases published in 2009 and 2018 from the health fatchecking
organization HealthNewsReview444https://www.healthnewsreview.org/. The dataset
covers diseases such as cancer, alzheimer, etc.
We indexed the title and article of the news in text format and to add the
ability of semantic retrieval to the search engine, we encode title and the
article of news also with the pre-trained sentence-BERT [24]. The search
engine retrieves the top 10 documents whose fields match and have high cosine
similarity with the query. If the number of retrieved documents is smaller
than 10, the list of the documents is appended with an empty strings.

###  3.3 Source (S)

To encode information of the sources, for each source, we first unshortened
any shortened links, such as the URL in Figure 1. Then, we extract the name of
the source (e.g thespoof). We retrieve then the reliability and Simple Wiki
description of the source and vectorized source information by concatenating
the retrieved information.

3.3.1 Reliability

NELA 2019 is a dataset containing 260 news sources proposed by Gruppi et al
[10]. The dataset contains source reliability labels from various assessment
websites such as Media Bias/Fact Check
(MBFC)555https://mediabiasfactcheck.com/,
Politifact666https://www.politifact.com/, etc. Moreover, The authors
aggregated the labels from MBFC by assigning a label _unreliable_ to sources
with low factual reporting or listed as conspiracy/pseudoscience source.
Similarly, they assigned _reliable_ to sources with high factual reporting. We
determined the source reliability by combining the aggregated labels and
satire sources from MBFC. In the end, the source types that we used for
reliability are _reliable_ , _unreliable_ , and _satire_. We assign the na
(not available) tag to the sources that do not occur in NELA 2019. Afterwards,
we vectorize the reliability of each source with one hot encoder. For the
posts that do not have URLs or the number of URLs less than 5 URLs, we append
the source lists with zero vectors of a size equal to the length of the
reliability tags.

Simple Wiki Source Descriptions

The Simple English Wikipedia aims to provide access to an English encyclopedia
for non-native English speakers and children. The entity descriptions of
Simple English Wikipedia can be a clue for the trustworthiness of the sources.
We downloaded the Simple English Wikipedia, February 2019
dump777https://archive.org/details/simplewiki-20190201 to ensure that the
contents were written before COVID-19. Using the tool Wiki
Extractor888https://github.com/attardi/wikiextractor, we constructed a
dictionary mapping entities to their descriptions. For simplicity, we ignored
ambiguous entities that have more than one wiki pages and mapped the
description of each source in the posts if a key of the dictionary exactly
matched the source name. Afterwards, we encoded each source descriptions as
BERT representations.

##  4 Experiments

###  4.1 Dataset

The CONSTRAINT dataset [22] contains fake and real news about COVID-19 in the
form of social media posts. Fake news samples were collected from various
fact-checking websites and tools such as Politifact, IFCN chatbot. Real news
samples were collected from verified Twitter accounts and manually checked by
the organizers. The dataset is split into train, dev, and test splits. Table 1
presents statistical details of the datasets.

Table 1: Statistical details of the task’s dataset [22]. w: post with links, w/o: posts without links | Real | Fake |  | Total
---|---|---|---|---
| w | w/o | w | w/o
Train | 2321 | 1039 | 1002 | 2058 | 6420
Dev | 780 | 340 | 327 | 693 | 2140
Test | 779 | 341 | 319 | 701 | 2140

###  4.2 Baselines

We compared _ECOL_ framework against the baseline classifiers provided by the
organizers, which are Support Vector Machine (SVM), Logistic Regression (LR),
Gradient Boost, and Decision Trees (DT). The baseline classifiers are trained
on term frequency-inverse document frequency (tf-idf) features.

###  4.3 Models

We compare and analyse the following variations of _ECOL_ framework models:

  * •

C uses solely content information as feature.

  * •

PK uses solely prior knowledge as feature.

  * •

C_PK uses concatenation of content and prior knowledge as feature.

  * •

C_S uses concatenation of content and source as feature.

  * •

C_PK_S uses concatenation of content, source and prior knowledge.

###  4.4 Implementation

We implemented _ECOL_ models using PyTorch Lightning999https://pytorch-
lightning.readthedocs.io/en/0.7.1/introduction_guide.html. We trained the
models with 42, 0, 36 random seeds, three epochs, and one batch size on a
NVIDIA TITAN RTX 16 GB GPU.

##  5 Results and Discussion

We present the experimental results on development and test sets in Table 2.
We report Precision (P), Recall (R), F1 scores per class, and accuracy,
weighted P, R, and F1 scores as the models’ overall performance. C$\mu$,
PK$\mu$, C_S$\mu$, C_PK_S$\mu$ average the predictions by the models trained
with 42, 36, 0 as random seeds. The other models use a random seed of 42,
which gave the highest F1 scores in our experiments. We entered the CONSTRAINT
shared task with three entries: an average over the three C_PK_S models and
the two C_PK_S models with the highest F1 scores (random seeds are 42, 36).
The best performing model with random seed 42 ranked 14 among 167 submissions
[21].

Table 2: Precision, recall and F1-score of the baseline and proposed models, trained on the CONSTRAINT datasets. Highlighted scores indicate the highest values for each metric. Set | Model | Fake |  | Real |  | Overall |
---|---|---|---|---|---|---|---
P | R | F1 | P | R | F1 | Acc | P | R | F1
Dev | SVM | 92.07 | 94.41 | 93.22 | 94.79 | 92.59 | 93.68 | 93.46 | 93.48 | 93.46 | 93.46
LR | 91.07 | 94.02 | 92.52 | 94.39 | 91.61 | 92.98 | 92.76 | 92.79 | 92.76 | 92.75
GB | 83.41 | 90.20 | 86.67 | 90.36 | 83.66 | 86.88 | 86.78 | 87.03 | 86.78 | 86.77
DT | 85.53 | 83.43 | 84.47 | 85.24 | 87.14 | 86.18 | 85.37 | 85.42 | 85.37 | 85.38
C | 98.12 | 97.06 | 97.59 | 97.35 | 98.30 | 97.82 | 97.71 | 97.72 | 97.71 | 97.71
PK | 97.69 | 95.20 | 96.43 | 95.72 | 97.95 | 96.82 | 96.63 | 96.67 | 96.64 | 96.64
C_PK | 98.99 | 95.78 | 97.36 | 96.27 | 99.11 | 97.67 | 97.52 | 97.57 | 97.52 | 97.53
CS | 98.39 | 95.98 | 97.17 | 96.42 | 98.57 | 97.48 | 97.34 | 97.37 | 97.34 | 97.34
C_PK_S | 98.51 | 97.25 | 97.88 | 97.53 | 98.66 | 98.09 | 97.99 | 98.00 | 97.99 | 97.99
C$\mu$ | 99.28 | 95.20 | 97.20 | 95.78 | 99.38 | 97.55 | 97.38 | 97.47 | 97.38 | 97.39
PK$\mu$ | 99.17 | 93.53 | 96.27 | 94.40 | 99.29 | 96.78 | 96.54 | 96.70 | 96.54 | 96.55
C_PK$\mu$ | 99.49 | 94.71 | 97.04 | 95.38 | 99.55 | 97.42 | 97.24 | 97.35 | 97.24 | 97.24
C_S$\mu$ | 99.49 | 95.10 | 97.24 | 95.71 | 99.55 | 97.59 | 97.43 | 97.52 | 97.43 | 97.43
C_PK_S$\mu$ | 99.09 | 96.08 | 97.56 | 96.52 | 99.20 | 97.84 | 97.71 | 97.75 | 97.71 | 97.71
Test | SVM | 92.20 | 93.92 | 93.05 | 94.37 | 92.77 | 93.56 | 93.32 | 93.33 | 93.32 | 93.32
LR | 90.08 | 93.43 | 91.72 | 93.81 | 90.62 | 92.19 | 91.96 | 92.01 | 91.96 | 91.96
GB | 83.39 | 90.59 | 86.84 | 90.70 | 83.57 | 86.99 | 86.92 | 87.20 | 86.92 | 86.91
DT | 85.39 | 84.22 | 84.80 | 85.80 | 86.88 | 86.34 | 85.61 | 85.62 | 85.61 | 85.61
C | 98.21 | 96.96 | 97.58 | 97.26 | 98.39 | 97.83 | 97.71 | 97.72 | 97.71 | 97.71
PK | 97.78 | 95.20 | 96.47 | 95.73 | 98.04 | 96.87 | 96.68 | 96.72 | 96.68 | 96.68
C_PK | 99.29 | 95.59 | 97.40 | 96.11 | 99.38 | 97.72 | 97.57 | 97.64 | 97.57 | 97.57
CS | 98.69 | 96.08 | 97.37 | 96.51 | 98.84 | 97.66 | 97.52 | 97.56 | 97.52 | 97.53
C_PK_S | 99.10 | 96.96 | 98.02 | 97.29 | 99.20 | 98.23 | 98.13 | 98.15 | 98.13 | 98.13
C$\mu$ | 99.49 | 94.80 | 97.09 | 95.46 | 99.55 | 97.47 | 97.29 | 97.40 | 97.30 | 97.29
PK$\mu$ | 98.56 | 94.02 | 96.24 | 94.77 | 98.75 | 96.72 | 96.50 | 96.60 | 96.50 | 96.50
C_PK$\mu$ | 99.38 | 93.82 | 96.52 | 94.65 | 99.46 | 97.00 | 96.78 | 96.93 | 96.78 | 96.78
C_S$\mu$ | 99.79 | 94.90 | 97.29 | 95.56 | 99.82 | 97.64 | 97.48 | 97.59 | 97.48 | 97.48
C_PK_S$\mu$ | 99.59 | 95.29 | 97.39 | 95.88 | 99.64 | 97.72 | 97.57 | 97.66 | 97.57 | 97.57

(a) Fake news posts with and without links

(b) Real news with and without links

Figure 2: F1 Scores of the $\mu$ models when predicting posts with (blue
color) and without links (red color).

By applying a T-test at the 0.01 significance level, we observe that the
proposed models significantly outperformed the baselines. When we compare the
content-based models (C and C$\mu$) with the other proposed models, we first
see that the prior knowledge and content information (C_PK) complements the
source information (C_S). Moreover, the prior knowledge and content
information helps to identify false news, but source information helps in
identifying real news. Therefore, among the proposed models, C_PK_S and
C_PK_S$\mu$ achieve the highest F1 scores by balancing the predictions towards
real and fake news. However, the improvement is not significant. For instance,
C misclassified only 49 samples while the C_PK_S, classified 9 samples more
correctly.

PK and C_PK which incorporate the prior knowledge are the least successful
models among the proposed models. We observed that the indexing method for the
retrieval unit yields false-positive predictions. However, the models also
outperformed the official baselines which implies that prior knowledge could
be useful for fake news detection. For better healthcare retrieval, we plan to
improve the indexing schema with the semantic concepts that define the health
claim types such as treatment, alternative medicine.

We also analyzed how the presence of links in posts change the model
predictions and present F1 scores of the models by grouping them into posts
with and without links in Figure 2. The presence of links in fake news
drastically degrades the performance of the models (Figure 2a). For example,
while C_S$\mu$ scores the posts with the links as 98.77, its F1 score is
reduced to 93.22. However, encoding source information into the models
(C_PK_S$\mu$ and C_S$\mu$) improves identifying fake news posts with links.
When we analyse the links, we see that they are Twitter accounts, medical
websites and delete links that are not present in Simple English Wikipedia nor
reliability dictionary. However, we found some samples in the test set, which
have links to a fact-checking website (Politifact) but were annotated as fake,
potentially yielding false predictions.

As seen in Figure 2b, the content information is the only key feature for
identifying real news. Prior knowledge and source information could not
improve the prediction of real news posts with links. When we analyze real
news posts that were misclassified by the models, we see that although the
posts are written in reporting language, they also contain judgemental
language. For example, Coronavirus: Donald Trump ignores COVID-19 rules with
’reckless and selfish’ indoor rally [URL] might confuse the models by
combining the two language types.

##  6 Conclusion

In this paper, we presented a promising framework for the early detection of
fake news on social media. The framework encodes content, prior knowledge, and
credibility of sources from the URL links in the posts. We analyzed the impact
of each encoded information on the models to detect fake news in the
healthcare domain. We observed that using three perspectives could lead to
precisely distinguish between fake and real news. In future work, we will
improve the source linking by using structured data such as Wikidata [32] in
order to encode more source knowledge. For a better retrieval in the
healthcare domain, we plan also to index prior knowledge by categorizing it
into semantic concepts such as cure, treatment and symptoms.

## References

  * [1] Augenstein, I., Lioma, C., Wang, D., Lima, L.C., Hansen, C., Hansen, C., Simonsen, J.G.: Multifc: A real-world multi-domain dataset for evidence-based fact checking of claims. In: EMNLP/IJCNLP (1). pp. 4684–4696. Association for Computational Linguistics (2019)
  * [2] Baris, I., Schmelzeisen, L., Staab, S.: Clearumor at semeval-2019 task 7: Convolving elmo against rumors. In: SemEval@NAACL-HLT. pp. 1105–1109. Association for Computational Linguistics (2019)
  * [3] Castillo, C., Mendoza, M., Poblete, B.: Information credibility on twitter. In: WWW. pp. 675–684. ACM (2011)
  * [4] Chen, T., Li, X., Yin, H., Zhang, J.: Call attention to rumors: Deep attention based recurrent neural networks for early rumor detection. In: PAKDD (Workshops). Lecture Notes in Computer Science, vol. 11154, pp. 40–52. Springer (2018)
  * [5] Dai, E., Sun, Y., Wang, S.: Ginger cannot cure cancer: Battling fake health news with a comprehensive data repository. In: ICWSM. pp. 853–862. AAAI Press (2020)
  * [6] Denaux, R., Gómez-Pérez, J.M.: Linked credibility reviews for explainable misinformation detection. In: ISWC (1). Lecture Notes in Computer Science, vol. 12506, pp. 147–163. Springer (2020)
  * [7] Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT (1). pp. 4171–4186. Association for Computational Linguistics (2019)
  * [8] Esteves, D., Reddy, A.J., Chawla, P., Lehmann, J.: Belittling the source: Trustworthiness indicators to obfuscate fake news on the web. CoRR abs/1809.00494 (2018)
  * [9] Fajcik, M., Smrz, P., Burget, L.: BUT-FIT at semeval-2019 task 7: Determining the rumour stance with pre-trained deep bidirectional transformers. In: SemEval@NAACL-HLT. pp. 1097–1104. Association for Computational Linguistics (2019)
  * [10] Gruppi, M., Horne, B.D., Adali, S.: NELA-GT-2019: A large multi-labelled news dataset for the study of misinformation in news articles. CoRR abs/2003.08444 (2020)
  * [11] Guo, B., Ding, Y., Yao, L., Liang, Y., Yu, Z.: The future of false information detection on social media: New perspectives and trends. ACM Comput. Surv. 53(4), 68:1–68:36 (2020)
  * [12] Hassan, N., Zhang, G., Arslan, F., Caraballo, J., Jimenez, D., Gawsane, S., Hasan, S., Joseph, M., Kulkarni, A., Nayak, A.K., Sable, V., Li, C., Tremayne, M.: Claimbuster: The first-ever end-to-end fact-checking system. Proc. VLDB Endow. 10(12), 1945–1948 (2017)
  * [13] Kwon, S., Cha, M., Jung, K., Chen, W., Wang, Y.: Prominent features of rumor propagation in online social media. In: ICDM. pp. 1103–1108. IEEE Computer Society (2013)
  * [14] Lazer, D.M., Baum, M.A., Benkler, Y., Berinsky, A.J., Greenhill, K.M., Menczer, F., Metzger, M.J., Nyhan, B., Pennycook, G., Rothschild, D., et al.: The science of fake news. Science 359(6380), 1094–1096 (2018)
  * [15] Li, Q., Zhou, W.: Connecting the dots between fact verification and fake news detection. In: COLING. pp. 1820–1825. International Committee on Computational Linguistics (2020)
  * [16] Li, Q., Zhang, Q., Si, L.: Rumor detection by exploiting user credibility information, attention and multi-task learning. In: ACL (1). pp. 1173–1179. Association for Computational Linguistics (2019)
  * [17] Mensio, M., Alani, H.: News source credibility in the eyes of different assessors. In: TTO (2019)
  * [18] Naeem, S.B., Bhatti, R., Khan, A.: An exploration of how fake news is taking over social media and putting public health at risk. Health Information & Libraries Journal (2020)
  * [19] Nakov, P.: Can we spot the ”fake news” before it was even written? CoRR abs/2008.04374 (2020)
  * [20] Nørregaard, J., Horne, B.D., Adali, S.: NELA-GT-2018: A large multi-labelled news dataset for the study of misinformation in news articles. In: ICWSM. pp. 630–638. AAAI Press (2019)
  * [21] Patwa, P., Bhardwaj, M., Guptha, V., Kumari, G., Sharma, S., PYKL, S., Das, A., Ekbal, A., Akhtar, M.S., Chakraborty, T.: Overview of constraint 2021 shared tasks: Detecting english covid-19 fake news and hindi hostile posts. In: Proceedings of the First Workshop on Combating Online Hostile Posts in Regional Languages during Emergency Situation (CONSTRAINT). Springer (2021)
  * [22] Patwa, P., Sharma, S., Srinivas, P., Guptha, V., Kumari, G., Akhtar, M.S., Ekbal, A., Das, A., Chakraborty, T.: Fighting an infodemic: COVID-19 fake news dataset. CoRR abs/2011.03327 (2020)
  * [23] Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep Contextualized Word Representations. In: NAACL-HLT. pp. 2227–2237. ACL (2018)
  * [24] Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. In: EMNLP/IJCNLP (1). pp. 3980–3990. Association for Computational Linguistics (2019)
  * [25] Shaar, S., Babulkov, N., Martino, G.D.S., Nakov, P.: That is a known lie: Detecting previously fact-checked claims. In: ACL. pp. 3607–3618. Association for Computational Linguistics (2020)
  * [26] Shu, K., Sliva, A., Wang, S., Tang, J., Liu, H.: Fake news detection on social media: A data mining perspective. SIGKDD Explor. 19(1), 22–36 (2017)
  * [27] Shu, K., Zheng, G., Li, Y., Mukherjee, S., Awadallah, A.H., Ruston, S., Liu, H.: Leveraging multi-source weak social supervision for early detection of fake news. CoRR abs/2004.01732 (2020)
  * [28] The, L.O.: Oncology,” fake” news, and legal liability. The Lancet. Oncology 19(9), 1135 (2018)
  * [29] Thorne, J., Vlachos, A., Cocarascu, O., Christodoulopoulos, C., Mittal, A.: The fact extraction and verification (FEVER) shared task. CoRR abs/1811.10971 (2018)
  * [30] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30, 5998–6008 (2017)
  * [31] Vosoughi, S., Roy, D., Aral, S.: The spread of true and false news online. Science 359(6380), 1146–1151 (2018)
  * [32] Vrandečić, D., Krötzsch, M.: Wikidata: a free collaborative knowledgebase. Communications of the ACM 57(10), 78–85 (2014)
  * [33] Wang, W.Y.: ”liar, liar pants on fire”: A new benchmark dataset for fake news detection. In: ACL (2). pp. 422–426. Association for Computational Linguistics (2017)
  * [34] Waszak, P.M., Kasprzycka-Waszak, W., Kubanek, A.: The spread of medical fake news in social media–the pilot quantitative study. Health policy and technology 7(2), 115–118 (2018)
  * [35] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T.L., Gugger, S., Drame, M., Lhoest, Q., Rush, A.M.: Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pp. 38–45. Association for Computational Linguistics, Online (Oct 2020), https://www.aclweb.org/anthology/2020.emnlp-demos.6
  * [36] Woloszyn, V., Schaeffer, F., Boniatti, B., Cortes, E.G., Mohtaj, S., Möller, S.: Untrue.news: A new search engine for fake stories. CoRR abs/2002.06585 (2020)
  * [37] Yuan, C., Ma, Q., Zhou, W., Han, J., Hu, S.: Early detection of fake news by utilizing the credibility of news, publishers, and users based on weakly supervised learning. In: COLING. pp. 5444–5454. International Committee on Computational Linguistics (2020)
  * [38] Zhou, X., Jain, A., Phoha, V.V., Zafarani, R.: Fake news early detection: A theory-driven model. Digital Threats: Research and Practice 1(2), 1–25 (2020)
  * [39] Zhou, X., Zafarani, R.: A survey of fake news: Fundamental theories, detection methods, and opportunities. ACM Comput. Surv. 53(5) (Sep 2020). https://doi.org/10.1145/3395046, https://doi.org/10.1145/3395046