# On Generating Identifiable Virtual Faces

Zhuowen Yuan  Fudan University<EMAIL_ADDRESS>,  Zhengxin You  Fudan
University<EMAIL_ADDRESS>,  Sheng Li  Fudan University
<EMAIL_ADDRESS>,  Zhenxing Qian  Fudan University<EMAIL_ADDRESS>,
Xinpeng Zhang  Fudan University<EMAIL_ADDRESS>and  Alex C. Kot
Nanyang Technological University<EMAIL_ADDRESS>
(2022)

###### Abstract.

Face anonymization with generative models have become increasingly prevalent
since they sanitize private information by generating virtual face images,
ensuring both privacy and image utility. Such virtual face images are usually
not identiﬁable after the removal or protection of the original identity. In
this paper, we formalize and tackle the problem of generating identiﬁable
virtual face images. Our virtual face images are visually different from the
original ones for privacy protection. In addition, they are bound with new
virtual identities, which can be directly used for face recognition. We
propose an Identiﬁable Virtual Face Generator (IVFG) to generate the virtual
face images. The IVFG projects the latent vectors of the original face images
into virtual ones according to a user speciﬁc key, based on which the virtual
face images are generated. To make the virtual face images identiﬁable, we
propose a multi-task learning objective as well as a triplet styled training
strategy to learn the IVFG. We evaluate the performance of our virtual face
images using different face recognizers on diffident face image datasets, all
of which demonstrate the effectiveness of the IVFG for generate identiﬁable
virtual face images.

Face authentication, Face anonymization, Generative adversarial networks

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
30th ACM International Conference on Multimedia; October 10–14, 2022; Lisboa,
Portugal††booktitle: Proceedings of the 30th ACM International Conference on
Multimedia (MM ’22), October 10–14, 2022, Lisboa, Portugal††price: 15.00††doi:
10.1145/3503161.3548110††isbn: 978-1-4503-9203-7/22/10††ccs: Security and
privacy Biometrics

##  1\. Introduction

Since the advent of the digital era, millions of images are shared over the
Internet with the help of personal devices. Although they enable us to build
stronger connections with the society, there is growing concern that these
techniques may introduce privacy issues. Often, the images that are uploaded
without sanitization contain privacy-sensitive information of individuals,
such as identity, location, etc. Such private information may be leveraged by
adversaries for malicious purposes. The public available images may also be
collected and utilized to train machine learning models without consent
(Harvey and LaPlace, 2019).

To tackle these problems, obfuscation techniques such as pixelization and
blurring (Hill et al., 2016) are widely adopted. However, these traditional
methods fail to protect against modern attacks based on deep neural networks.
McPherson _et al._ (McPherson et al., 2016) show that a trained attack model
can re-identify the obfuscated face images with over 95% of accuracy. On the
other hand, the quality of these obfuscated images are severely distorted,
which could hardly be used for other computer vision tasks. Recently,
researchers propose to add adversarial noise to the face images (Shan et al.,
2020; Raval et al., 2017) to mislead the face recognizer. These models rely on
the prior knowledge of the face recognizer for high performance. People can
easily reveal the original identity of the obfuscated face image by human
perception (Li and Choi, 2021).

A growing trend of researches take advantage of generative adversarial
networks (GANs) (Goodfellow et al., 2014) for generating realistic face images
from original face images (Maximov et al., 2020; Mirjalili et al., 2020; Li
and Choi, 2021; Kuang et al., 2021; Gu et al., 2020). One branch of these
methods performs attribute editing while preserving the original identity. For
instance, Mirjalili _et al._ (Mirjalili et al., 2020) propose to remove
sensitive soft attributes so that the generated faces can be used for face
authentication, but the facial attributes can no longer be classified. Such a
scheme achieves a low level of face private protection as the original
identity is still revealed in the anonymized face image.

Figure 1. A typical application of our IVFG framework. Suppose that a user
does not want to send his original face images to the untrusted applications,
but still wishes to be identified. He can generate a virtual face image with
our framework locally, and then feed the faces of his unique VID to the
untrusted applications. Downstream applications are all performed in the
virtual space, thus protecting the privacy of PIDs.

Another category anonymizes the identity of the face image, where the original
identity is not revealed in the generated face images. Various face image
anonymization networks have been proposed, including the deepblur network (Li
and Choi, 2021), the conditional identity anonymization GAN (Maximov et al.,
2020), and the de-identification generative adversarial network (DeIdGAN)
(Kuang et al., 2021). These approaches are able to generate anonymized face
images with relatively good resuability in terms of visual quality and face
detection. However, the identity information in the original face images are
removed forever, which can not be used for authentication when needed.

Recently, attempts have been made to encrypt the original face images into
real look alike anonymized face images (Gu et al., 2020). Different anonymized
face images could be generated from the original face images using different
keys. In addition, the original face images can be recovered from the
anonymized face images with the correct key available. As such, face
authentication can be carried out in the original space after decryption.
However, the original identity is leaked to the third party who performs face
authentication.

With the widespread applications of face identification techniques, it is
urgent to develop countermeasures against illegal usage of face images. Unlike
other images, face images contain unique identity information, which can not
be replaced or reissued. Once stolen, people’s identities will be compromised
forever. Therefore, it is important that the original identity can not be
revealed from the anonymized face images. On the other hand, the collection of
face images is expensive and requires a lot of human effort. Thus, the
resuability should not be compromised after anonymization, while the most
important usability of a face image is the identifiability.

In this paper, we try to tackle the aforementioned issues by generating
identifiable virtual face images for privacy protection. A virtual face image
is generated from the original face image according to a key, which satisfies
the following requirements.

  * •

The virtual face images are visually different from the original face images
for anonymization.

  * •

Different virtual face images can be generated from the same original face
image with different keys.

  * •

The virtual face images can be directly used for face authentication in the
virtual space.

Concretely, we propose a novel Identifiable Virtual Face Generator (IVFG) to
fulfill such requirements. The IVFG maps an original face image into different
virtual face images using different keys. For face images from the same
original identity, the IVFG is able to generate virtual face images belonging
to a unique virtual identity (VID) using a user specific key. In the following
discussions, we also term the original identity as the physical identity
(PID). The virtual face images can be used in different applications that
require the face images as inputs for anonymization purpose. Take the online
face authentication as an example, we can submit the virtual faces and only
the VID is used for registration or authentication. Thus, the face
authentication can be conducted in the virtual space to protect the user’s
privacy, where the PID is never exposed online as shown in Fig. 1.

Our proposed IVFG framework is based on an Encoder-Decoder architecture. We
propose to project the features of the original face images to the latent
space of the Style-GAN (Karras et al., 2019) according to a user specific key.
This is to encode the VID information into the virtual face images which are
the outputs of the Style-GAN decoder. To make the virtual face images
identifiable, a multi-task learning objective is further proposed to achieve
the following properties, including the anonymization, diversity,
identifiability and separability. The performance of our virtual faces are
shown to be promising using different face recognizers on different face image
datasets.

The main contributions are summarized below.

  * •

We first explore the possibility to generate identifiable virtual face images
which protects the PID and can be identified directly in the virtual space
with off-the-shelf face recognizers.

  * •

We propose a novel framework IVFG to map PIDs to VIDs conditioned on a user-
specified key.

  * •

We propose a multi-task learning objective as well as the corresponding
training strategy to facilitate the training of IVFG.

##  2\. Related Work

In this section, we brieﬂy summarize related work on image obfuscation, GANs,
and face anonymization.

###  2.1. Image Obfuscation

Traditional methods such as pixelization, blurring (Hill et al., 2016) are
widely adopted for image obfuscation. These methods severely reduce the image
quality and utility. They also fail to protect private information in images
against deep models as already demonstrated by (McPherson et al., 2016).
Another category of methods leverage the power of adversarial machine learning
(Kurakin et al., 2017; Shan et al., 2020; Raval et al., 2017) to fool the
classiﬁer. Shan _et al._ (Shan et al., 2020) propose to add cloaks (i.e.,
adversarial noise) to the images. It achieves over 95 % protection rate across
different feature extractors. However, such obfuscated images are still
recognizable by human perception (Li and Choi, 2021). Raval _et al._ (Raval et
al., 2017) train an adversarial network by jointly optimizing the privacy and
utility objectives to perturb the images for privacy protection. This method
relies on the prior knowledge of the attacker’s network, which may not be
applicable in many real-world scenarios.

###  2.2. Generative Adversarial Networks

The idea of GAN is initially introduced by (Goodfellow et al., 2014), which
has widely been used for image generation. A typical GAN consists of a
discriminator and a generator, which are jointly trained in a minimax game.
The discriminator learns to distinguish between real and fake faces while the
generator learns to generate realistic samples to fool the discriminator.
Karras _et al._ (Karras et al., 2019) propose a style-based generator StyleGAN
and achieve good performance. Developments in GANs have also enabled
controllable generation given input labels (Mirza and Osindero, 2014). Abdal
_et al._ (Abdal et al., 2021) investigate the latent space of StyleGAN and
propose StyleFlow for attribute-conditioned semantic editing on projected real
images. Richardson _et al._ (Richardson et al., 2021) propose an encoder for
StyleGAN inversion, which demonstrates the possibility of face translation
using pre-trained models. Various optimization strategies have been proposed
(Arjovsky et al., 2017; Mao et al., 2017) to facilitate the training of GANs.

###  2.3. Face Anonymization

Face anonymization refers to publishing face images with the identity
information sanitized. The image obfuscation techniques mentioned earlier can
also be utilized for face anonymization. Recent advances take advantage of GAN
to generate sanitized face images (Yan et al., 2019; Mirjalili et al., 2020).
Li and Lin (Li and Lin, 2019) propose AnonymousNet that performs face
attribute editing based on a set of privacy metrics. Similarly, Mirjalili _et
al._ (Mirjalili et al., 2020) propose to generate anonymized face images by
removing sensitive soft attributes. These schemes only protect the privacy of
the face attributes, the original identities (i.e., the PIDs) are preserved in
the anonymized face images for authentication.

In order to protect the PIDs of the face images, Li and Choi (Li and Choi,
2021) propose DeepBlur to perform blurring in the latent space of a pre-
trained generative model, which is capable of synthesizing realistic face
images with the PIDs removed. Maximov _et al._ (Maximov et al., 2020) propose
a general face anonymization framework by mixing styles of different
identities using GAN. Kuang _et al._ (Kuang et al., 2021) propose an de-
identification generative adversarial network (DeIdGAN) for face
anonymization, which seamlessly replaces the original face image by a visually
different but realistic face image. The anonymized face images generated by
such schemes can only be used for face detection, which are not able to be
utilized for authentication anymore due to the removal of PIDs. To deal with
this issue, Gu _et al._ (Gu et al., 2020) propose a face image encryption
framework to generate visually realistic encrypted face images bases on
different passwords. The encrypted face images are generated based on a
password conditioned GAN, which well protect the PIDs and can be decrypted
into the original face images for authentication. Despite the advantage, this
method fails to protect the PIDs after the face image decryption. The
protection mainly relies on the secrecy of the password, and the anonymized
face images can not be used directly for face authentication.

In this paper, we propose to generate identiﬁable virtual face images for
anonymization. The virtual face images are generated based on both the
original face image and a user speciﬁc key. Unlike the existing face
anonymization schemes, our virtual faces can be authenticated directly in the
virtual space without leaking the PIDs.

##  3\. Problem Formulation

Let $\mathcal{X}$ and $\mathcal{Y}$ denote two sets of face images, where
images within each set belong to the same person (i.e., the same PID). The
anonymization process is conditioned on a user-specified key
$k\in\mathcal{K}$. We further denote $T(x,k)$ as the virtual face image
generated from the original face image $x$ and key $k$, and $I(x)$ as the
identity that can be extracted from $x$.

Given the original face images $x_{1},x_{2}\in\mathcal{X}$, $y\in\mathcal{Y}$
and two disparate keys $k_{1},k_{2}\in\mathcal{K}$, our goal is to generate
virtual face images that meet the following properties.

###### Property 1 (Anonymization).

The original face image and virtual face image should belong to different
identities, namely VID $\neq$ PID:

(1) $I(T(x_{1},k_{1}))\neq I(x_{1}).$

###### Property 2 (Diversity).

The key controls the VID of the anonymized face images, i.e., the virtual face
images of the same PID belong to different VIDs under different keys:

(2) $I(T(x_{1},k_{1}))\neq I(T(x_{1},k_{2})).$

###### Property 3 (Identifiability).

Given two original face images belonging to the same PID, the corresponding
virtual face images should belong to the same VID under the same key:

(3) $I(T(x_{1},k_{1}))=I(T(x_{2},k_{1})).$

###### Property 4 (Separability).

Given two original face images belonging to different PIDs, the corresponding
virtual face images should belong to different VIDs under the same or
different keys:

(4) $\begin{split}I(T(x_{1},k_{1}))&\neq I(T(y,k_{1}))\\\
I(T(x_{1},k_{1}))&\neq I(T(y,k_{2})).\end{split}$

Figure 2. (a) The proposed IVFG framework. (b) Overview of our training
strategy.

##  4\. Methodology

In this section, we introduce our IVFG framework, learning objective and
training strategy.

###  4.1. The Proposed Framework

The overall structure of our proposed IVFG framework is illustrated in Fig.
2(a). The IVFG framework is based on an encoder-decoder architecture, which is
composed of three modules: an encoder $E$, a latent projector $P$ and a face
generator $G$. The encoder $E$ takes an image tensor $x$ as input and outputs
the extracted facial representation, say $r$. We propose a latent projector
$P$ to project the combination of facial representation $r$ and key $k$ to the
latent space of the generator, which is composed of a concatenation operation
and a multilayer perceptron (MLP). We use the generator of a pre-trained GAN
as our $G$, which maps a vector in latent space $z\in\mathcal{Z}$ to realistic
images from the distribution of training data $x\sim p(x)$, namely:
$G:z\mapsto x$.

###  4.2. Learning Objective

We design a multi-task learning objective to satisfy the four properties
mentioned in Section 3 for training the IVFG. We leverage the following cosine
embedding loss $\mathcal{L}_{emb}$ to measure the similarity between two
facial features $f_{1}$ and $f_{2}$, where

(5) $\mathcal{L}_{emb}(f_{1},f_{2},l)=\\\
\begin{cases}1-\cos(f_{1},f_{2}),&l=1\\\ \max(m,\cos(f_{1},f_{2})),&l=-1\\\
\end{cases},$

where $cos$ denotes cosine similarity, $m$ is a hyperparameter, and $l$ is the
identity indicator. $l=1$ refers to the case that $f_{1}$ and $f_{2}$ to be
from the same identity, and we are pushing $f_{1}$ and $f_{2}$ to be close to
each other by optimizing $\mathcal{L}_{emb}$. On the contrary, we try to
separate $f_{1}$ and $f_{2}$ to make them belong to different identities if
$l$ is set to $-1$.

Privacy Loss. To satisfy Property 1, we propose privacy loss
$\mathcal{L}_{pri}$ to modify the identity of the original face image. In
other words, $\mathcal{L}_{pri}$ guides the anonymization process so that PID
$\neq$ VID, where

(6)
$\mathcal{L}_{pri}=\mathbb{E}_{\mathcal{X},\mathcal{K}}[\mathcal{L}_{emb}(R(T(x_{1},k_{1})),R(x_{1}),-1)],$

where $R(x)$ extracts the features from the face image $x$.

Conditional Loss. To satisfy Property 2, we propose conditional loss
$\mathcal{L}_{con}$ to make sure that the generated virtual identity can be
controlled by a user-specified key. Given two different keys $k_{1},k_{2}$ and
an original face image, the model should be able to generate two distinct VIDs
by using $\mathcal{L}_{con}$ below

(7)
$\mathcal{L}_{con}=\mathbb{E}_{\mathcal{X},\mathcal{K}}[\mathcal{L}_{emb}(R(T(x_{1},k_{1})),R(T(x_{1},k_{2})),-1)].$

Intra-group Classification Loss. To satisfy Property 3, we have to guarantee
that, given two original face images from the same PID:
$x_{1},x_{2}\in\mathcal{X}$, the corresponding virtual face images generated
using same key should belong to the same VID. Therefore, the intra-group
classification loss is formulated as

(8)
$\mathcal{L}_{intra}=\mathbb{E}_{\mathcal{X},\mathcal{K}}[\mathcal{L}_{emb}(R(T(x_{1},k_{1})),R(T(x_{2},k_{1})),1)].$

Inter-group Classification Loss. To satisfy Property 4, we propose the
following inter-group classification loss $\mathcal{L}_{inter}$ to maximize
the distance between VIDs that are generated from different PIDs

(9)
$\mathcal{L}_{inter}=\mathbb{E}_{\mathcal{X},\mathcal{Y},\mathcal{K}}[\mathcal{L}_{emb}(R(T(x_{1},k_{1})),R(T(y,k_{1})),-1)],$

where $x$ and $y$ are face images from different PIDs.

Regularization. With the above losses, the predicted latent vectors may
deviate from the latent space of $G$, leading to non-convergence. To deal with
this issue, we further propose a regularization term to bound the distribution
of the projected latent vectors such that they are within the latent space of
$G$. Specifically, we first calculate the mean latent vector $\bar{z}$ by
sampling _m_ latent vectors $z_{1},z_{2},...,z_{m}$ from the latent
distribution of $G$. Then, we minimize the L2 distance between the predicted
latent vector $z$ and $\bar{z}$ by

(10) $\mathcal{L}_{reg}=||z-\bar{z}||_{2}^{2}.$

Our full objective is a weighted average over the above mentioned loss
functions, which is given below

(11)
$\begin{split}\mathcal{L}&=\lambda_{pri}\mathcal{L}_{pri}+\lambda_{con}\mathcal{L}_{con}\\\
&+\lambda_{intra}\mathcal{L}_{intra}+\lambda_{inter}\mathcal{L}_{inter}+\lambda_{reg}\mathcal{L}_{reg}.\end{split}$

###  4.3. Training Strategy

During training, we leverage an auxiliary face recognizer $R$ for facial
feature extraction. We use pre-trained models for $E$, $G$, $R$ whose
parameters will be frozen in the training. We only update the parameters of
$P$ by

(12) $P^{*}=\arg\min_{P}{\mathcal{L}}.$

We split the dataset $\mathcal{S}$ into triplets of images. For each triplet
$s\in\mathcal{S}$, we have $s=\\{x_{1},x_{2},y\\}$ with $x_{1},x_{2}$
belonging to the same PID and $y$ for another PID. Given two different keys
$k_{1}$ and $k_{2}$, we can derive a quaternion $s^{\prime}$ of four virtual
images: $T(x_{1},k_{1})$, $T(x_{1},k_{2})$, $T(x_{2},k_{1})$, $T(y,k_{1})$.
Then, we calculate the objective function and perform back propagation. Such a
triplet styled training strategy is illustrated in Fig. 2(b) with the details
given in Algorithm 1.

Algorithm 1 IVFG Training

Input: $E$: pre-trained encoder; $G$: pre-trained generator; $R$: pre-trained
face recognizer; $P(f,k;W)$: latent projector; $W$: parameters of $P$,
$\mathcal{S}_{n}$: dataset containing $n$ triplets
Parameter:
$\lambda_{pri},\lambda_{con},\lambda_{intra},\lambda_{inter},\lambda_{reg}$:
weights for the training objective; $\alpha$: learning rate

1: Freeze $E$, $G$, $R$.

2: // Generate $\hat{z}$ by sampling from latent space.

3: $z_{1},...,z_{m}=sample(z),z\sim p(z)$

4: $\hat{z}=\frac{1}{m}\sum_{j=1}^{m}z_{j}$

5: for $i\leftarrow 1$ to $n$ do

6: $k_{1},k_{2}\leftarrow$ generate two distinct keys

7: $x_{1},x_{2},y\leftarrow\mathcal{S}_{i}$

8: $r_{x_{1}}\leftarrow E(x_{1})$ // Extract facial representations.

9: $r_{x_{2}}\leftarrow E(x_{2})$

10: $r_{y}\leftarrow E(y)$

11: $z_{x_{1},k_{1}}\leftarrow P(r_{x_{1}},k_{1};W_{i})$ // Project to latent
vectors.

12: $z_{x_{1},k_{2}}\leftarrow P(r_{x_{1}},k_{2};W_{i})$

13: $z_{x_{2},k_{1}}\leftarrow P(r_{x_{2}},k_{1};W_{i})$

14: $z_{y,k_{1}}\leftarrow P(r_{y},k_{1};W_{i})$

15: $T(x_{1},k_{1})\leftarrow G(z_{x_{1},k_{1}})$ // Generate virtual faces.

16: $T(x_{1},k_{2})\leftarrow G(z_{x_{1},k_{2}})$

17: $T(x_{2},k_{1})\leftarrow G(z_{x_{2},k_{1}})$

18: $T(y,k_{1})\leftarrow G(z_{y,k_{1}})$

19: Calculate full objective $\mathcal{L}$ from Eq. 11.

20: $\nabla W_{i}=\partial\mathcal{L}/\partial W_{i}$

21: $W_{i+1}=W_{i}-\alpha\nabla W_{i}$

22: end for

23: return $W_{i+1}$

##  5\. Experiments

###  5.1. Datasets

Our model is trained on LFW (Huang et al., 2007) and evaluated on both LFW
(Huang et al., 2007) and CelebA (Liu et al., 2015). The LFW dataset consists
of 13,233 face images from 5,749 identities, including 1,680 identities with
two or more images. We perform an 8:1:1 split based on the identities for
training, validation, and testing. Then, we split each set into triplets of
images as discussed in the previous section. We also randomly select 10% of
the face images in the CelebA dataset as another test set for cross dataset
evaluation. All face images are aligned and cropped to $128\times 128$ by
MTCNN (Zhang et al., 2016).

###  5.2. Implementation Details and Settings

We use FaceNet Inception-ResNet-v1 (Szegedy et al., 2017) for both $E$ and $R$
in training. Both of them are pre-trained on VGGFace2 (Cao et al., 2018). For
the input key, we use an $n$-digit binary string, which is represented by a
$n$-dimensional vector $k\in\\{0,1\\}^{n}$. We set $n=8$ as the default value.
$G$ is a StyleGAN (Karras et al., 2019) generator trained from scratch on LFW
with progressive growing (Karras et al., 2018). We stop after the 128-pixel
training step since we only use the model for generating $128\times 128$
images 111If a higher resolution is required, we can pre-train the StyleGAN
for more iterations with progressive growing until the generator learns how to
generate high resolution images.. For effective and stable evaluation, all
noises included in the original StyleGAN implementation are removed during
both training and evaluating. Our projector $P$ maps the concatenation of
facial representation and key into the intermediate latent space $\mathcal{W}$
of StyleGAN instead of $Z$, since $\mathcal{W}$ achieves better
disentanglement as pointed out by (Karras et al., 2019). $w\in\mathcal{W}$
directly controls the generator through adaptive instance normalization (Huang
and Belongie, 2017) at each convolution layer.

We set the number of hidden layers of the MLP in $P$ as 2, and
$\lambda_{pri}=0.1$, $\lambda_{con}=1$, $\lambda_{intra}=1$,
$\lambda_{inter}=1$, $\lambda_{reg}=20$, and learning rate as $3\times
10^{-4}$. We set the identity indicator $l$ in the cosine embedding loss as
0.4. We train the model for 20 epochs with the Adam optimizer (Kingma and Ba,
2017), where $\beta_{1}=0.9$ and $\beta_{2}=0.999$. The training process takes
only 1 hour on a single NVIDIA GTX 1080Ti GPU.

To demonstrate the generality of our framework, instead of using the FaceNet
(which is used for training) as the face recognizer, we use a different
architecture for recognizing the virtual face images in testing. Unless
otherwise stated, we use the SphereFace (Liu et al., 2017) which is pre-
trained on CASIA-WebFace (Yi et al., 2014) as the face recognizer for
evaluating the performance of our virtual face images.

Figure 3. Examples of our virtual face images generated from the LFW test set.
Images in first two rows are the original faces images, where the
corresponding virtual face images are given in the third and fourth rows.

###  5.3. Performance

To the best of our knowledge, we are the first to propose and formalize the
task of generating identifiable virtual face images. Here, we compare the
performance of our method with the most relevant work proposed by Gu _et al._
(Gu et al., 2020). Both our scheme and the work in (Gu et al., 2020) use a key
and an original face image for input, which are able to generate real-look
alike virtual/encrypted face images using different keys.

Table 1. Face recognition performance on different image sets.  Face image sets | EER | AUC
---|---|---
Original | 0.024 | 0.998
Gu _et al._ (Gu et al., 2020) (Set-A) | 0.105$\pm$0.009 | 0.957$\pm$0.004
Gu _et al._ (Gu et al., 2020) (Set-B) | 0.136$\pm$0.012 | 0.929$\pm$0.006
Ours (Set-A) | 0.018$\pm$0.005 | 0.999$\pm$0.001
Ours (Set-B) | 0.103$\pm$0.009 | 0.963$\pm$0.004
Table 2. Face recognition performance of our virtual face images generated from CelebA with ArcFace model as the face recognizer.  Face image sets | EER | AUC
---|---|---
Original | 0.048 | 0.975
Set-A-CelebA | 0.035$\pm$0.002 | 0.994$\pm$0.001
Set-B-CelebA | 0.196$\pm$0.004 | 0.887$\pm$0.003

####  5.3.1. Face Recognition

In this section, we evaluate the performance of our virtual faces in terms of
face recognition, including Equal Error Rate (EER) and Area Under Curve (AUC).
We generate the following two sets of virtual face images from the LFW test
set:

  * •

Set-A: We randomly assign a key to each identity to generate the virtual face
images.

  * •

Set-B: Each identity is assigned with the same key to generate the virtual
face images.

Set-A refers to the case that the user’s key is not known by the adversary,
while Set-B is generated for evaluating the key stolen case. We generate the
same sets of encrypted face images using the method proposed in (Gu et al.,
2020) for comparison. Table 1 gives the EER and AUC among different sets of
face images, where “original” refers to the set of the original face images.
It can be seen that, our virtual face images achieve the best recognition
performance when the key is well protected, which is even better than that of
the original face images. When the key is stolen, the EER drops around 8%
using our virtual face images.

Next, we evaluate our method on the CelebA test set using another face
recognizer which is the pre-trained Arcface model (Deng et al., 2019). We test
on two sets of virtual face images as mentioned before, including Set-A-CelebA
and Set-B-CelebA, for virtual face images generate using random keys and the
same key for different identities. The results are shown in Table2. Again,
when the keys are protected (i.e., Set-A-CelebA), our virtual face images
offers higher recognition accuracy compared with the original face images.
This indicates that our scheme works on arbitrary face images and can be
recognized using arbitrary face recognizers.

Our IVFG supports different key spaces according to the requirements of
different applications. For a different key space, the only thing we have to
do is to make the dimension of the input of the MLP (which is in the latent
projector $P$ of the IVFG) to be the same as the dimension of the combination
of the facial representation $r$ and the key $k$. We adjust and retrain our
IVFG on a 128-bit key space with $k\in\\{0,1\\}^{128}$, where we are still
able to generate identifiable virtual face images. The EER of the virtual face
recognition on the LFW test set is 0.016$\pm$0.005 and 0.112$\pm$0.013 for the
key protected and key stolen cases, respectively. These are similar to the
results of using the default 8-bit keys (see Table 1). Thus, our IVFG is
flexible and robust against different key spaces.

Figure 4. Examples of our virtual face images generated from the CelebA test
set. Images in first two rows are the original faces images, where the
corresponding virtual face images are given in the third and fourth rows.

####  5.3.2. Anonymization

We evaluate the anonymization performance of our virtual face images in terms
of protection rate, which is computed as the unsuccessful match rate between
the original and anonymized face images. Specifically, we match the original
face image and the corresponding anonymized face image. If the match score is
less than the EER threshold of the face recognizer, these two images are
considered to be from different identities and an unsuccessful match is
yielded. Table 3 reports the protection rate of our virtual face images and Gu
_et al._ (Gu et al., 2020)’s encrypted face images on the test set of LFW. It
can be seen that our virtual face images offer excellent anonymization with
protection rate of 0.988, which is significantly better than Gu _et al._ ’s
method.

Figure 5. Diverse virtual face images generated from the same face image with
different keys. In each row, the first image is the original face images, the
rest images are the corresponding virtual face images generated using
different keys. The original face images of the first two rows are randomly
selected from the LFW test set, while those of the last two rows are randomly
selected from the CelebA test set.

####  5.3.3. Diversity

We evaluate here how different virtual face images could be generated using
different keys. For each face image from the test set of LFW, we use two
different keys to generate two virtual face images. We match these two virtual
face images to see if the matching is successful (i.e., the matching score is
over the EER threshold). The diversity is measured as the unsuccessful match
rate among the virtual face images generated using different keys, the results
of which are given in Table 3. It can be seen that our method performs
significantly better than the work in (Gu et al., 2020) in terms of diversity.

Table 3. Performance on anonymization and diversity. Metrics | Gu _et al._ (Gu et al., 2020) | Ours
---|---|---
Protection Rate | 0.052$\pm$0.007 | 0.988$\pm$0.005
Diversity | 0.008$\pm$0.003 | 0.750$\pm$0.013
Table 4. Detection rate and FID scores between the original and anonymized faces. The scores are calculated on features from the second max pooling layer with a dimension of 192. Methods | Detection | FID
---|---|---
Pixelization | 0.984 | 44.451
Blurring | 0.991 | 62.786
Gu _et al._ (Gu et al., 2020) | 1.0$\pm$0.001 | 4.782$\pm$0.027
Ours | 1.0$\pm$0.0 | 6.170$\pm$0.050

####  5.3.4. Recoverability

Next, we evaluate the possibility for the attackers to recover the original
face image from the virtual face images when the key is stolen. In particular,
we use the virtual face images in Set A and the corresponding keys for input
and run the IVFG again to generate a set of recovered face images. We match
these recovered face images with the original face images, the successful
match rate of which is 0$\pm$0.001 at the EER threshold. This is to say, it is
difficult to recover the original face images when our virtual face image, the
key and the trained IVFG are compromised. While in Gu _et al._ (Gu et al.,
2020)’s work, the original face image can be directly decrypted from the
encrypted face image using the correct password.

Table 5. Ablation study of our model. From the second row to the sixth row, each row shows the performance of our virtual faces with one loss component removed. Methods | Protection Rate | Diversity | EER (same key) | EER (different keys) | FID
---|---|---|---|---|---
w/o $\mathcal{L}_{pri}$ | 0.978$\pm$0.006 | 0.718$\pm$0.010 | 0.105$\pm$0.008 | 0.021$\pm$0.009 | 6.066$\pm$0.056
w/o $\mathcal{L}_{con}$ | 0.987$\pm$0.003 | 0.0$\pm$0.0 | 0.108$\pm$0.005 | 0.096$\pm$0.004 | 7.641$\pm$0.008
w/o $\mathcal{L}_{intra}$ | 0.968$\pm$0.008 | 0.764$\pm$0.019 | 0.190$\pm$0.008 | 0.112$\pm$0.008 | 5.776$\pm$0.086
w/o $\mathcal{L}_{inter}$ | 0.982$\pm$0.004 | 0.803$\pm$0.015 | 0.180$\pm$0.012 | 0.0$\pm$0.0 | 6.339$\pm$0.037
w/o $\mathcal{L}_{reg}$ | 0.996$\pm$0.002 | 0.740$\pm$0.021 | 0.096$\pm$0.007 | 0.016$\pm$0.005 | 15.535$\pm$0.144
full objective | 0.988$\pm$0.005 | 0.750$\pm$0.013 | 0.103$\pm$0.009 | 0.018$\pm$0.005 | 6.170$\pm$0.050
Figure 6. t-SNE visualization of the feature space of different face image
sets, left: ours, right: Gu _et al._ (Gu et al., 2020)’s

####  5.3.5. Visual Quality

To evaluate the visual quality of the virtual face images, we report the
detection rate using the face detector MTCNN (Zhang et al., 2016) as well as
the Fréchet Inception Distance (FID) (Heusel et al., 2018) to quantify the
similarity of data distributions between the virtual face images and real
ones. The face detection thresholds using MTCNN are set as the default values.
Table 4 gives the results for our proposed method, Gu _et al._ ’s method and
traditional methods including pixelization (with pixel size of 18) and
blurring (Gaussian blur with kernel size of 37 and $\sigma$=6). Our method
achieves perfect detection rate of 1.0. In terms of FID, our method
significantly outperforms traditional anonymization methods and achieves
comparable results with the work in (Gu et al., 2020).

Fig. 3 and Fig. 4 give a few examples of our virtual face images as well as
the corresponding original ones from the LFW and CelebA test sets,
respectively. In these two figures, the four face images in each column belong
to the same identity, and each identity is bound with a user specific key for
virtual face generation. It can be seen that our virtual face images are
visually real-look alike, which are completely different from the original
ones. On the other hand, the virtual face images generated from the face
images of the same identity are similar. This is because they are generated
from the user specific key for the same identity, and the outputs of our
network are highly bound with the identity and key.

Fig. 5 further illustrates the diverse virtual face images that are generated
from the same face image with different keys. We can see that, by changing the
keys, we are able to generate virtual face images with diverse visual
appearances. Therefore, when one of the virtual face images is compromised, it
can be easily replaced with another using a different key.

###  5.4. Feature Space Visualization

In this section, we visualize the feature space of the original and virtual
face image for qualitative analysis. We randomly select 10 identities from the
LFW test set with at least 30 face images, from which we generate different
sets of virtual face images using different keys (i.e., Set A) or the same key
(i.e., Set B). We perform t-SNE on the extracted facial features of the
original and virtual faces (Van der Maaten and Hinton, 2008) in Set A and Set
B, the visualization results of which are shown in Fig. 6. We can see that, in
the case of different keys (i.e., Set A), the virtual facial features are more
sparsely distributed than the original ones, which can be well classified in
the virtual space. On the other hand, the features of our virtual face images
can also be well separated from the original ones, leading to good
anonymization ability. When the key is stolen (Set B), the virtual features
are still clustered to a certain extent. We also compare our virtual feature
space with that of Gu _et al._ ’s. It can be seen that, by using Gu _et al._
’s method, the features of the encrypted face images are not well clustered,
which are also difficult to be differentiated from the original ones.

###  5.5. Ablation Study

In this section, we present ablation studies to demonstrate the effectiveness
of each loss component in our multi-task learning objective on the test set of
LFW. We remove each loss component from the overall training objective
separately and report the results in Table 5.

It can be seen from Table 5 that the $\mathcal{L}_{pri}$ effectively pushes
the features apart, the protection rate drops if $\mathcal{L}_{pri}$ is
removed. Moreover, the average cosine similarity drops from 0.0052 to 0.0024
between virtual and original facial features when $\mathcal{L}_{pri}$ is
incorporated during training. Without $\mathcal{L}_{con}$, the model fails to
be conditioned on the user-specified key to generate different virtual face
images. The results of w/o $\mathcal{L}_{intra}$ and w/o $\mathcal{L}_{inter}$
demonstrate that both the intra-group and inter-group classification losses
are necessary for high face recognition accuracy. The results of w/o
$\mathcal{L}_{reg}$ shows that $\mathcal{L}_{reg}$ regularizes the
anonymization process to produce realistic face images.

##  6\. Conclusions

In this paper, we propose and formalize a novel task in face anonymization:
generating identifiable virtual face images. To this end, we propose a
framework termed as the IVFG for the generation of identifiable virtual face
images. The input of the IVFG is the original face image and a user specific
key, the output is a virtual face image bound with a unique virtual identity.
Our virtual face images are visually different from the original ones for
protecting the original identity, which could also be directly recognized in
the virtual space for authentication. We also propose a multi-task learning
objective and a triplet styled training strategy to effectively train the
IVFG. Experimental results demonstrate that our virtual face images achieve
good performance for face anonymization and identification of virtual
identities.

###### Acknowledgements.

This work was supported in part by the National Natural Science Foundation of
China under Grant 62072114, U20A20178,U20B2051 and U1936214, in part by
Alibaba Group through Alibaba Innovative Research (AIR) Program.

## References

  * (1)
  * Abdal et al. (2021) Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka. 2021\.  Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows.  _ACM Transactions on Graphics (TOG)_ 40, 3 (2021), 1–21.
  * Arjovsky et al. (2017) Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017\.  Wasserstein GAN.  arXiv:1701.07875 [stat.ML]
  * Cao et al. (2018) Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. 2018.  Vggface2: A dataset for recognising faces across pose and age. In _2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018)_. IEEE, 67–74.
  * Deng et al. (2019) Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. 2019\.  Arcface: Additive angular margin loss for deep face recognition. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 4690–4699.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014\.  Generative adversarial nets.  _Advances in neural information processing systems_ 27 (2014).
  * Gu et al. (2020) Xiuye Gu, Weixin Luo, Michael S Ryoo, and Yong Jae Lee. 2020\.  Password-conditioned anonymization and deanonymization with face identity transformers. In _European Conference on Computer Vision_. Springer, 727–743.
  * Harvey and LaPlace (2019) Adam Harvey and Jules LaPlace. 2019.  MegaPixels: origins, ethics, and privacy implications of publicly available face recognition image datasets.  _Megapixels_ (2019).
  * Heusel et al. (2018) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2018\.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium.  arXiv:1706.08500 [cs.LG]
  * Hill et al. (2016) Steven Hill, Zhimin Zhou, Lawrence K Saul, and Hovav Shacham. 2016\.  On the (In) effectiveness of Mosaicing and Blurring as Tools for Document Redaction.  _Proc. Priv. Enhancing Technol._ 2016, 4 (2016), 403–417.
  * Huang et al. (2007) Gary B. Huang, Manu Ramesh, Tamara Berg, and Erik Learned-Miller. 2007.  _Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments_.  Technical Report 07-49. University of Massachusetts, Amherst.
  * Huang and Belongie (2017) Xun Huang and Serge Belongie. 2017.  Arbitrary style transfer in real-time with adaptive instance normalization. In _Proceedings of the IEEE International Conference on Computer Vision_. 1501–1510.
  * Karras et al. (2018) Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018\.  Progressive Growing of GANs for Improved Quality, Stability, and Variation.  arXiv:1710.10196 [cs.NE]
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila. 2019.  A style-based generator architecture for generative adversarial networks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 4401–4410.
  * Kingma and Ba (2017) Diederik P. Kingma and Jimmy Ba. 2017.  Adam: A Method for Stochastic Optimization.  arXiv:1412.6980 [cs.LG]
  * Kuang et al. (2021) Zhenzhong Kuang, Huigui Liu, Jun Yu, Aikui Tian, Lei Wang, Jianping Fan, and Noboru Babaguchi. 2021.  Effective De-Identification Generative Adversarial Network for Face Anonymization. In _Proceedings of the 29th ACM International Conference on Multimedia_. 3182–3191.
  * Kurakin et al. (2017) Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2017\.  Adversarial examples in the physical world.  arXiv:1607.02533 [cs.CV]
  * Li and Choi (2021) Tao Li and Min Soo Choi. 2021.  DeepBlur: A Simple and Effective Method for Natural Image Obfuscation.  arXiv:2104.02655 [cs.CV]
  * Li and Lin (2019) Tao Li and Lei Lin. 2019\.  Anonymousnet: Natural face de-identification with measurable privacy. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_. 0–0.
  * Liu et al. (2017) Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. 2017\.  Sphereface: Deep hypersphere embedding for face recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 212–220.
  * Liu et al. (2015) Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2015\.  Deep learning face attributes in the wild. In _Proceedings of the IEEE international conference on computer vision_. 3730–3738.
  * Mao et al. (2017) Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. 2017\.  Least squares generative adversarial networks. In _Proceedings of the IEEE international conference on computer vision_. 2794–2802.
  * Maximov et al. (2020) Maxim Maximov, Ismail Elezi, and Laura Leal-Taixé. 2020\.  Ciagan: Conditional identity anonymization generative adversarial networks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5447–5456.
  * McPherson et al. (2016) Richard McPherson, Reza Shokri, and Vitaly Shmatikov. 2016\.  Defeating Image Obfuscation with Deep Learning.  arXiv:1609.00408 [cs.CR]
  * Mirjalili et al. (2020) Vahid Mirjalili, Sebastian Raschka, and Arun Ross. 2020\.  PrivacyNet: semi-adversarial networks for multi-attribute face privacy.  _IEEE Transactions on Image Processing_ 29 (2020), 9400–9412.
  * Mirza and Osindero (2014) Mehdi Mirza and Simon Osindero. 2014.  Conditional Generative Adversarial Nets.  arXiv:1411.1784 [cs.LG]
  * Raval et al. (2017) Nisarg Raval, Ashwin Machanavajjhala, and Landon P Cox. 2017\.  Protecting visual secrets using adversarial nets. In _2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_. IEEE, 1329–1332.
  * Richardson et al. (2021) Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. 2021.  Encoding in style: a stylegan encoder for image-to-image translation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2287–2296.
  * Shan et al. (2020) Shawn Shan, Emily Wenger, Jiayun Zhang, Huiying Li, Haitao Zheng, and Ben Y Zhao. 2020\.  Fawkes: Protecting privacy against unauthorized deep learning models. In _29th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 20)_. 1589–1604.
  * Szegedy et al. (2017) Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. 2017.  Inception-v4, inception-resnet and the impact of residual connections on learning. In _Thirty-first AAAI conference on artificial intelligence_.
  * Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ 9, 11 (2008).
  * Yan et al. (2019) Bin Yan, Mingtao Pei, and Zhengang Nie. 2019.  Attributes Preserving Face De-Identification.. In _ICCV Workshops_. 1217–1221.
  * Yi et al. (2014) Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z. Li. 2014\.  Learning Face Representation from Scratch.  arXiv:1411.7923 [cs.CV]
  * Zhang et al. (2016) Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. 2016\.  Joint face detection and alignment using multitask cascaded convolutional networks.  _IEEE Signal Processing Letters_ 23, 10 (2016), 1499–1503.