# A secure and private ensemble matcher using multi-vault obfuscated templates

Babak Poorebrahim Gilkalaye, Shubhabrata Mukherjee, Reza Derakhshani

###### Abstract

Given the irrevocability of biometric samples and mounting privacy concerns,
biometric template security and secure matching are among the essential
features of any well-designed modern biometric system. In this paper, we
propose an obfuscation method that hides the biometric template information
with just enough chaff. The main idea is to reduce the number of chaff points
to a practical level by creating $n$ sub-templates from the original template
and hiding each sub-template with $m$ chaff points. During verification, $s$
closest vectors to the biometric query are retrieved from each vault and then
combined to generate hash values that are compared with the stored hash value.
We demonstrate the effectiveness of synthetic facial images, generated by a
Generative Adversarial Network (GAN), as “random chaff points” within a
secure-vault authorization system. This approach safeguards user identities
during training and deployment. We tested our protocol using the AT&T, GT, and
LFW face datasets, with the ROC areas under the curve being 0.99, 0.99, and
0.90, respectively. These numbers were close to those of the unprotected
templates, showing that our method does not adversely affect accuracy.

##  I Introduction

Biometric user identification has become a popular and convenient way for
users to assert their identity across different devices and services, from
cellular phones and laptops to access control and payment systems and even
government services. However, many biometric systems are still vulnerable to
certain attack vectors, such as sensor hijacks, injection of forged biometric
templates, and network attacks on servers [1]. Such vulnerabilities may expose
a biometric system to correlation attacks [2], replay attacks, Man-in-the-
Middle attacks, Denial-of-Service attacks, or Brute-Force attacks [3, 4, 5].
Security incidents and privacy breaches through social media platforms,
healthcare data, and government services [6, 7, 8] exemplify such
vulnerabilities. Per the recent report from the HIPAA journal [9], there has
been a substantial increase in data breaches across US hospitals, including
data theft, unauthorized access, and hacking, highlighting the need for
securing biomedical and biometric reference data.The same report also shows a
large increase in the monetary penalties issued to hospitals due to such data
breaches.

To protect stored biometric references from such attacks, researchers have
offered various security schemes such as random projection, feature
disentanglement, fuzzy vaults (with helper data), and deep learning-based
concealable multi-biometrics to mitigate the earlier-mentioned privacy and
security issues [10, 11, 12, 13]. However, many of these approaches are either
computationally too expensive or not completely secure against certain attack
vectors. In this work, we propose an encrypted vault approach to securely
store biometric templates. The reference feature vector, derived from the
enrollment image and used for comparison with another incoming images, is
called an enrollment template. For example, in the context of a Convolutional
Neural Network (CNN) trained for face matching, this could be the flattened
input to an FC layer. If the target image corresponds to the same person or
object, for instance by the corresponding templates having a cosine similarity
below a set threshold, the claimant is accepted. In this case the claimant is
genuine, and the acceptance is a true positive of genuine accept. If the
claimant is an impostor (the incoming image is not from the same identity) and
they mange to pass the template comparison, the result is a false or impostor
accept. A genuine comparison is anticipated to yield a higher cosine
similarity, indicating that the feature vectors derived from the images
closely match (Fig. 1). Our approach uses mutually exclusive embeddings
generated by various deep learning models, combined and then securely
encrypted using a SHA-512 based encryption. This technique also stores a large
number of face embeddings generated from random identities (2000-4000 or
higher), all stored in a vault, making it practically impossible to identify
the original template. This attribute makes our approach robust against brute
force attacks. We note that many other encryption-based approaches have been
proposed previously to overcome such attacks [14, 15], but these approaches
often adversely increase the time complexity of the biometric pipeline. Our
approach does not inflict such a high computational burden, as we demonstrate
later for the face verification use case, while it maintains the security of
the process.

The rest of this paper is organized as follows: section II focuses on related
work. Section III describes our methodology and the main idea behind the
framework, and explains the improvement in True Positive Rate (TPR) and the
enhanced security of our approach. Section IV lays out the experimental setup.
Section V details our results and their implications. Finally, we conclude by
summarizing the unique attributes of our method, the difficulties faced, and
future work.

##  II Related Work

Many methods have been introduced over the years to preserve the privacy of
biometric data. For instance, [16] used a de-identification technique.
However, there was no guarantee of anonymity. Other methods tried encryption-
based schemes, such as [17]. However, such solutions are often vulnerable to
attack vectors such as the False Acceptance Rate (FAR) attack [18], brute
force, and linkage attacks. Our approach can be shown to be secure against
such attacks, as we demonstrate in Section III.B. Some authors used CNNs for
protecting face templates while attaining good matching performance [19]. In
another group of papers such as [20], researchers discuss blockchain-based
methods for storing biometric templates. However, often execution time is
higher for such schemes. Some studies have explored methods involving
adversarial learning to conceal sensitive information within a learned space
[21]. Specifically, there is a focus on disentangling identity from attribute
information to mitigate soft-biometric information in face templates [22, 11].
Recent studies have investigated combining two biometric modalities, like
fingerprint and iris scan, using various fusion techniques [23, 24]. While
enhancing security through redundancy, this approach introduces a single point
of failure. If one biometric trait is compromised or unavailable, the entire
authentication process could be vulnerable.

Figure 1: Cosine similarity for template comparison

###  II-A Fuzzy Extractor

Fig. 2 describes a fuzzy extractor scheme and its terminology. We will use the
same structure and terminology for our proposed obfuscation method. The Gen
function generates a secret key and a public helper $P$ from a biometric
template $t$. The Rep function takes a biometric query $q$ and the helper as
its input. If dist($q,t$) is smaller than a preset threshold, then it
generates the correct key. In other words, the helper $P$ helps to remove
noise from query $q$ in order to generate $t$. The helper $P$ is required to
be secure and reveal limited information about template $t$ to preserve the
privacy of the template $t$. To satisfy the information-theoretic security,
the entropy loss must be at most $\mathcal{L}$, as further detailed below
[25],[26],[27].

Figure 2: Fuzzy extractor scheme

##  III Methodology

In this section, we will first informally explain the main idea behind the
proposed method, and then provide the formalized algorithm. The idea is to
divide the biometric template into $m$ sub-templates and hide them with $n$
chaff points to make it computationally impractical for an adversary to find
the template. One solution would be to hide a template $t$ with $n$ chaff
points in a public vault, $V=[r_{1},...,t,...,r_{n}]$ and store $f(t)$, where
$f$ is a one-way function such as SHA-256. Now the question is how hard it
would be for an adversary to find the index of $t$ in $V$. To make it
computationally impractical for the adversary to find $t$, $n$ must be a very
big number $(\lambda$ (security parameter)), as the best strategy to find the
template $t$ is to compute $f(x)$ for all $x$ in the vault to find the
template $t$. However, this method is also computationally impractical for the
legitimate user. By way of example, assume we will have $f(t)$ and one vault
containing $1+2^{80}$ vectors as public information (helper). We assume the
inserted chaff points are indistinguishable from the template $t$ in the
collection. By indistinguishable chaff points, we mean that, for example, if
we were working with face templates stored in the vault, the chaff points
would be representing (albeit different) faces in a way that an attacker
wouldn’t be able to distinguish them from the actual template. For matching a
query $q$, we compute the distance of $q$ from all points inside the vault and
retrieve $m$ (depending on the threshold of the system) closest points to $q$:
${p_{1},p_{2},...,p_{m}}$, and compute $f(p_{1}),f(p_{2}),...,f(p_{m})$ and
check if there are any matches between these values and $f(t)$. This system is
secure under certain conditions, but impractical because during the
authentication the user needs to compute the desired distance of the query $q$
vs. $1+2^{80}$ vectors.

Thus we propose generating $n$ sub-templates $t_{1},t_{2},...,t_{n}$ from
template $t$, using protocol $\mathcal{G}$, such that
$\mathcal{G}(t):t_{1},t_{2},...,t_{n}$ and hide each of them with $m$ chaff
points. In the end, we store $f([t_{1},t_{2},...,t_{n}])$ (, where $f$ is a
one-way function), in $n$ vaults such that each vault has $m+1$ vectors. To
make it computationally impractical for the adversary to find the template by
brute-force, we need $m^{n}$ to be very large. Unlike the earlier impractical
example, we can choose $m$ and $n$ to garner computational security against
brute force attacks while having a practical solution. In the forthcoming
experimental demonstration we use $m=2000$, and $n=5$, so that
$m^{n}=2^{54.82}$ to get acceptable security against a brute force attack.
When matching query $q$, we generate $q_{1},q_{2},...q_{n}$ using protocol
$\mathcal{G}(q):q_{1},q_{2},...,q_{n}$ (protocol $\mathcal{G}$ will be
explained later). The distances between $q_{i}$ and all $m+1$ vectors in the
corresponding vault are then computed and the top $s$ vectors from each vault
are retrieved. The threshold of the system is $s$ and the distance can be
computed by any distance metric, such as cosine similarity. At this point, we
have retrieved $sn$ vectors. Next we construct all $s^{n}$ possible
combinations of vectors $v_{1},v_{2},...v_{s^{n}}$ and compute
$f(v_{1}),f(v_{2}),...f(v_{s^{n}})$ and check if there are any matches between
the stored value of $f([t_{1},t_{2},...,t_{n}])$ and the $f(v_{i})$. If $q$ is
a genuine vector, it should be able to retrieve $t_{1},t_{2},...,t_{n}$ with
the threshold $s$. If $q$ is an imposter, it will fail to retrieve
$t_{1},t_{2},...t_{n}$ from the vaults. It should be noted that $s$ cannot be
a large number as it would render the biometric matching computationally
impractical.

###  III-A True Positive Rate (TPR) Improvement

It can be seen that a query needs to satisfy all $n$ vaults in order to
succeed. Thus we relax this condition and build the system such that a query
can succeed if it satisfies $k$ out of $n$ vaults. To do so, after the
generation of sub-templates $t_{1},t_{2},...t_{n}$, we need to store all
$n\choose k$ possible combinations of $f(t_{i})$. To satisfy the asymptotic
security, we need to make sure $m^{k}$ is computationally impractical for a
computationally bounded adversary, where $m$ is the number of chaff points per
vault.

###  III-B The Algorithm

Our algorithm Alg consists of a Gen protocol, a Rep protocol, a key generation
protocol $\mathcal{K}$, a sub-template Generator $\mathcal{G}$, a one-way
function $f$, and a security parameter $\gamma$. We assume that the template
$t$ is sampled from the distribution $\mathcal{D}$.

Key Generation $\mathcal{K}(\gamma)$: This protocol chooses $n$ (number of
vaults) and $m$ (number of chaff points) so that $m^{n}>2^{\gamma}$. Then it
generates $mn$ random samples from distribution $\mathcal{D}$ for $mn$ chaff
points $K=c_{11},c_{12}...,c_{1m},c_{21},c_{22},c_{21},...c_{2m},...,c_{nm}$.
Two types of random numbers $R_{1n}$ and $R_{2n}$ are generated. Where
$R_{1n}$ are $n$ scalars and $R_{2n}$ are small vectors with the same
dimension as the template. These will be used to hide the template in each
vault further, as follows. Let $d$ be the cosine similarity of two vectors,
then:

$d(A,B)$ = $d(A,rB)$ for any arbitrary scalar $r$, and:

$d(A,B)$ $\approx$ $d(A,r_{1}B+R_{1})$ for any arbitrary scalar $r_{1}$ and
small vector $R_{1}$

Sub-template Generator $\mathcal{G}(t,n)$: This protocol takes template $t$
and the number of vaults $n$ as its inputs and generates $n$ independent sub
templates $T=(t_{1},t_{2},...,t_{n})$. In the experimental section, we propose
one solution for such a function.

Gen: $\textsc{Gen}(T,K)$ This protocol places each sub-template $t_{i}$ and
its corresponding chaff points $c_{i,j}$, for all $j={1,..,m}$ inside a vault
in no specific order. It then stores $P_{1}=f(t_{1},t_{2},...t_{n})$ and
outputs the helper $P$:

$v_{1}=\\{R_{11}t_{1}+R_{12},R_{11}c_{11}+R_{12},R_{11}c_{12}+R_{12},...,R_{11}c_{1m}+R_{12}\\}$

$v_{2}=\\{R_{21}t_{2}+R_{22},R_{21}c_{21}+R_{22},R_{21}c_{22}+R_{22},...,R_{21}c_{2m}+R_{22}\\}$

…

$v_{n}=\\{R_{n1}t_{n}+R_{n2},R_{n1}c_{n1}+R_{n2},R_{n1}c_{n2}+R_{n2},...,R_{n1}c_{nm}+R_{n2}\\}$

$P_{2}=(v_{1},v_{2},...,v_{n})$

$P=(P_{1},P_{2})$

Rep: $\textsc{Rep}(C,q,tr)$: With $q$ representing a new biometric query using
protocol $\mathcal{G}$, $Q=(q_{1},q_{2},,...q_{n})$ is generated and the
closest $tr$ vectors are retrieved from each vault. The threshold of the
system is denoted by $tr$. Note that $tr$ is different from the original well-
known face matching threshold. $tr$ may be adjusted to increase or decrease
the False Acceptance Rate (FAR). In other words, if $tr=1$ we only retrieve
the closest vector in each vault. This will drastically reduce the False
Positive Rate (FPR), but simultaneously it reduces the TPR.

The following are retrieved from the $n$ vaults:

$\\{w_{11},w_{12},...,w_{1tr}\\}$, $\\{w_{21},w_{22},...,w_{2tr}\\}$…

$\\{w_{n1},w_{n2},...,w_{ntr}\\}$

All $n\choose tr$ possible combinations of these vectors are created and
passed to the one-way function $f$. If there is any match between the
generated hash values and the stored hash value $P_{1}$, then the query $q$ is
authenticated and the protocol outputs $1$ (success), otherwise, it outputs
$0$ (fail).

Figure 3: Embedding generated by the secure hashing verification

##  IV Security

An ideal biometric security scheme needs to satisfy Revocability, Diversity,
Accuracy, and Security [28] requirements. Our proposed scheme meets all these
requirements, as detailed below.

Revocability: A biometric system needs to be revocable because if it is
compromised, or if the user wishes to no longer be identified, they should be
able to revoke their reference. Our scheme’s templates are revocable by
choosing a new set of chaff points and a new sub-template generator.

Diversity: A secure template must not allow cross-matching across databases,
thereby ensuring the user’s privacy. In our system, for a false positive to
occur, the imposter must meet the authentication criteria of each vault and
successfully retrieve all segments of the template, hence our system improves
the cross-matching as compared to systems using one model. It also could be
verified throughout the experimental section, the FPR is very small, because
an imposter needs to satisfy multiple vaults to be able to positively match.

Accuracy: Our method shows minimum accuracy degradation with a 0.99 area under
the curve (AUC) for both AT&T database of faces [29] and Georgia Tech (GT)
face database [30].

Security: To show that our system is secure, we need to show that the helper
data $P=(P_{1},P_{2})$ is not revealing too much information and there is
enough entropy in this system. Chaff points are generated from the same
distribution that the corresponding sub-template is generated. Hence we can
say the sub-templates inside each vault are indistinguishable from other chaff
points there. Getting this indistinguishability between the sub-template and
chaff points inside a vault is straightforward; the challenge is to achieve
indistinguishability between sub-templates and chaff points. In other words,
if an adversary knows that $t_{1}$ is a sub-template, other $t_{i}$ should
remain indistinguishable from chaff points, we achieve this by using
independent models, given two models $M_{1},M_{2}$ and template $t$ and a
random face $r$, let $===$ be coming from the same image.

$|Pr[M_{1}(t)===M_{2}(t)]-Pr[M_{1}(t)===M_{2}(r)|=\epsilon$.

To achieve this, we must make sure that the sub-templates are independent. If
we assume everything looks totally random inside vaults, then we can claim the
best strategy to find the template for any adversary is to check all possible
combinations of elements in vaults, which we have made impractical by choosing
proper $n$,$m$. As a result, our approach provides superior security against
brute force attacks, as well as faster face acceptance and rejection
capabilities.

It is clear that given a face template $t$, a random face $r$, and two
biometric models $M1$ and $M2$, $y_{1}=M1(t)$ and $y_{2}~{}=~{}M2(t)$ cannot
be completely independent. However, in our case we are using big neural
networks. With this assumption and if $M1$ and $M2$ have completely different
structures, it is not going to be easy to distinguish between $M1(t)$ and
$M1(r)$ given $M2(t)$. In other words, if we tell the adversary $M2(t)$
belongs to person $A$, and then if we ask him to tell us which one of $M1(t)$
and $M1(r)$ belongs to the same person, it will not be easy for the adversary
to solve this problem.

One approach is to reverse engineer the template to recreate the template and
see which one of reverse ($M1(t))$ and reverse ($M1(r))$ are closer to the
reverse ($M2(t)$). We have addressed this attack by adding randomness now the
adversary needs to distinguish between $R_{2}M1(t)+R_{2}^{\prime}$ ,
$R_{2}M1(r)+R_{2}^{\prime\prime}$ , given $R_{1}M2(t)+R_{1}^{\prime}$.

##  V Experimental setup

Per common practice in face recognition, we use cosine similarity for matching
deep face templates [31]. Cosine similarity between any two vectors is given
by:

$\cos({\bf A},{\bf B})={{\bf A}{\bf B}\over\|{\bf A}\|\|{\bf
B}\|}=\frac{\sum_{i=1}^{n}{{\bf A}_{i}{\bf B}_{i}}}{\sqrt{\sum_{i=1}^{n}{({\bf
A}_{i})^{2}}}\sqrt{\sum_{i=1}^{n}{({\bf B}_{i})^{2}}}}$ (1)

Where A and B are the two feature vectors being compared.

In our research, a face image is transformed into an embedding using five
different deep learning methods (Fig. 4). Each of these embeddings is produced
by a different convolutional neural network (CNN) model. In the current
context, each of the embeddings ($e1,e2,e3,e4,e5$) is a 512 feature vector and
numerically sufficiently different from each other. A total of five secure
vaults is constructed. Each vault contains 2001 embeddings, of which 2000
embeddings are generated using 2000 random images of human faces and only 1
embedding is generated from an intended face image. In our research, we used a
technique called StyleGAN2, developed by Karras et al [32] to generate
realistic images of human faces. These images were then used to create random
chaff points for the secure vault.111To achieve this, we conducted an
iterative process, generating 2000 embeddings from 2000 realistic face images
obtained from the website thispersondoesnotexist.com This approach improves
the privacy safeguards for the proposed method during the development and
evaluation of the Multi-Vault Obfuscated Templates algorithm. The same
structure, consisting of 2000 embeddings generated using random images and 1
embedding from an intended face image (i.e. the template), is replicated in
all five vaults. To ensure the security, the hash value of the five different
embeddings $H=hash(e1,e2,e3,e4,e5)$ is stored.

###  V-A Identification process

The process for a single vault is shown in Fig. 3. For a biometric
identification match:

  * •

Five distinct embeddings ($A^{\prime}$, $B^{\prime}$, $C^{\prime}$,
$D^{\prime}$, $E^{\prime}$) are generated using five separate classifiers
based on the image of the person seeking verification (referred to as the
query).

  * •

Each query is compared against all embeddings across the five vaults,
resulting in the calculation of a total of 2001*5 cosine similarity scores.

  * •

From each vault, the embedding with the highest cosine similarity score is
selected, leading to the choice of five winners, one from each vault.

  * •

A secure 64-bit hash, denoted as $m^{\prime}$, is constructed by combining the
five embeddings selected in the previous step.

  * •

Identification is matched only if this hash matches the previously generated
hash (m) of the template.

A higher number of random images can be used to increase security. A lower and
higher number of classifiers can be used depending upon the scalability
requirements.

###  V-B Pre trained models

A set of five different pre-trained models have been used here to construct
the end-to-end secure privacy framework. The first model is InceptionResnetV1;
this particular model [33] has been pretrained on the VGGFace2dataset. [34,
35]. As a second classifier, another variation of InceptionResnetV1 has been
used, but this version was pretrained with CASIA-Webface [36]. For both the
models one and two, MTCNN [37] has been used as a preprocessing for face
cropping. More precisely, MTCNN cropped a 250x250 dimension face to improve
the model accuracy. As a third and fourth classifier, two different versions
of insightFace [38] have been used. The third pre-trained classifier, named
“buffalo”, is an implementation of the ResNet50 model pre-trained on the
WebFace600K dataset. This dataset contains 600,000 unique identities and is
derived from the larger WebFace42M dataset [39] which contains 42 million
identities. The fourth pre-trained model antelopev2 is an implementation of
ResNet100 pretrained using Glint360K [40], which contains 17091657 images of
360232 individuals. The fifth model is an arcface-based implementation of
Resnet100 [41]; for this version some additional pre-processing has been used
including image resizing, detection of faces using a face detector, face
alignment, etc.

Figure 4: Generating key using five different classifiers

##  VI Result and analysis

Our results are mainly described from two perspectives, accuracy and time
complexity performance. Fig. 5 shows the comparison between the distribution
of cosine similarity scores for genuine and imposter face images. The leftmost
distribution is for the genuine cosine similarities, which mostly lie between
0.8 to 1. The middle distribution is for the imposter cosine similarity
computed from the target template and other (non-mated) class images from a
same dataset, with the score distribution mostly covering 0.2 to 0.4. The
rightmost distribution also represents imposter cosine similarities but is
computed by comparing the target templates and the random face images
generated by GAN (as mentioned in the experimental setup), with the scores
being mostly between -0.25 and 0.25. The constrained feature space of faces
within a single dataset promotes higher cosine similarity (0.2-0.4) due to
inherent structural similarities. Conversely, the inclusion of random external
faces introduces greater vector variance, leading to a broader cosine
similarity distribution (-0.25 to 0.25). These comparisons show how well the
cosine similarity can distinguish the embeddings generated from face images.
The corresponding ROC AUC of cosine similarity scores reinforces this fact.

Figure 5: Distribution of cosine similarity for Genuine, Other and Random
classes

###  VI-A Individual classifier performance

Figure 6: ROC performance of individual classifiers based on the AT&T face dataset TABLE I: Individual classifier AUC for different datasets Dataset used | Classifier used | AUC
---|---|---
AT&T | InceptionResnetV1-VGG | .9999
AT&T | InceptionResnetV1-casia | .9996
AT&T | InsightFace Bufalo | .9952
AT&T | InsightFace Antelopev2 | .9978
AT&T | Resnet100 with preprocessing | 1.000
GT | InceptionResnetV1-VGG | .9998
GT | InceptionResnetV1-casia | .9995
GT | InsightFace Bufalo | .9780
GT | InsightFace Antelopev2 | .9745
GT | Resnet100 with preprocessing | 1.000
LFW | InceptionResnetV1-VGG | .9904
LFW | InceptionResnetV1-casia | .9909
LFW | InsightFace Bufalo | .9070
LFW | InsightFace Antelopev2 | .9175
LFW | Resnet100 with preprocessing | .9701

All five classifiers chosen for the secure hash pipeline performed
exceptionally well on the AT&T dataset, achieving an AUC exceeding 0.99 (shown
in Table I). Similar success was observed on the GT dataset, where at least
three classifiers surpassed an AUC of 0.99, and the remaining two achieved an
AUC exceeding 0.97. The LFW dataset [42] yielded comparable results, with at
least three classifiers again exceeding an AUC of 0.99. The designation
‘genuine’ indicates a positive class, and ‘imposter’ is marked as a negative
class.

###  VI-B End-to-end benchmarking

TABLE II: End-to-end benchmarking performance Dataset | Chaff points | TPR | TNR | Classifier used
---|---|---|---|---
GT | 4000 | 81.20% | 100.00% | 5
GT | 2000 | 84.63% | 100.00% | 5
GT | 4000 | 91.46% | 100.00% | 4
AT&T | 4000 | 91.58% | 100.00% | 5
AT&T | 2000 | 93.27% | 100.00% | 5
AT&T | 4000 | 96.19% | 100.00% | 4

As shown in Table II, the True Negative Rate (TNR) remained mostly at 100% for
the proposed framework when tested with both AT&T and GT datasets. However,
when the number of chaff points was increased from 2000 to 4000, the true
positive rate slightly decreased. It was also observed that if the system had
to satisfy only 4 out of all 5 classifiers, the true positive rate could
improve even with more chaff points. Using a higher number of chaff points can
increase the computation complexity, which in turn may make the system robust
against attack. Introducing additional classifiers strengthens the system by
exponentially increasing the imposter’s comparison workload
($(n+1)^{C}>n^{C}$), thus making the system further resilient against brute
force attacks. Here, $C$ is the number of classifiers and $n$ is the number of
chaff points. A combination of 4000 chaff points and 4 classifiers yielded the
best TPR and TNR.

Figure 7: End-to-end performance based on the AT&T face dataset

Figure 8: End-to-end performance based the on GT face dataset

Figure 9: End-to-end performance based the on LFW face dataset

###  VI-C ROC Analysis

Figures 7, 8, and 9 depict the study ROC curves for various datasets, all
generated using the presented secure hash pipeline.Instead of evaluating
individual classifiers, these ROC curves assess the performance of the
complete registration pipeline that combines all five classifiers. Within this
pipeline, a sample is considered a true positive if a genuine person is
successfully registered, and a false positive otherwise. The ROC curves
(Figures 7, 8, 9) depict the trade-off between security and accuracy in the
registration process. These curves show how the rate of incorrectly accepting
imposters (FPR) increases as we loosen the criteria for accepting genuine
users (TPR). Imagine a vault containing 2001 images. The strictest system
would only accept a single image with the highest cosine similarity score,
while the most lenient would accept all 2001. Practically, accepting the top
five highest-scoring images might be a good balance. This means the person can
be authenticated if their true identity is among the top five closest matches
out of the 2001 comparisons. As the ROC curves show, there’s a point beyond
which accepting more images doesn’t significantly improve the chance of
correctly identifying a genuine person, while increasing the risk of
mistakenly accepting someone else.

TABLE III: End-to-end AUC performance on different datasets Dataset for benchmarking | AUC
---|---
AT&T face dataset | 0.9939
Georgia Tech Face dataset | 0.9942
LFW dataset | 0.9042

While benchmarking with various datasets, from Table III it can be seen that
both the AT&T and GT face datasets yield an AUC greater than 0.99, whereas the
LFW dataset yields an AUC greater than 0.9.

###  VI-D Time complexity analysis

TABLE IV: Time complexity performance Event | Time taken in seconds
---|---
Image registration | 1.15
Authentication | 0.31
Secure hash generation | 0.01
Complete end-to-end procedure | 1.47

Our system was developed and run on a Windows laptop, with an Intel i9 11th
generation 64 bit CPU, 32 GB RAM, and 1 TB of storage. The system was equipped
with an NVIDIA GeForce RTX 3080 GPU. As seen from Table IV, our method is
computationally extremely efficient with an average of 1.15 seconds for image
registration, 0.31 seconds for biometric matching, and 0.01 seconds of hash
generation using the aforementioned system. Overall, the complete end-to-end
procedure takes approximately 1.47 seconds on average.

##  VII Conclusion

In this paper, we proposed a cryptographic biometric template security
protocol similar to a fuzzy extractor. We showed that $2^{\gamma}$ security
against brute-force attacks could be achieved by choosing appropriate
parameters for the protocol. We tested our scheme against the AT&T, GT, and
LFW datasets and showed that our protocol can preserve the average accuracy of
the models used to generate the embeddings. We introduced a new approach to
compute the receiver operating characteristic (ROC) curve which is compatible
with our protocol. Our work leverages the power of generative models to create
synthetic facial images, demonstrating their potential in building robust and
secure end-to-end biometric system. Our approach offers distinct advantages as
it eliminates the risk of exposing real individuals’ identities during the
system’s training and operation.

## Acknowledgments

This work was supported in part by grant from Center for Big Learning, an NSF
IUCRC. Dr. Derakhshani is also a consultant for Jumio.

## References

  * [1] Z. Rui and Z. Yan, “A survey on biometric authentication: Toward secure and privacy-preserving identification,” _IEEE Access_ , vol. 7, pp. 5994–6009, 2019.
  * [2] X. Zhou, S. D. Wolthusen, C. Busch, and A. Kuijper, “Feature correlation attack on biometric privacy protection schemes,” in _2009 Fifth International Conference on Intelligent Information Hiding and Multimedia Signal Processing_ , 2009, pp. 1061–1065.
  * [3] A. K. Das, M. Wazid, N. Kumar, A. V. Vasilakos, and J. J. P. C. Rodrigues, “Biometrics-based privacy-preserving user authentication scheme for cloud-based industrial internet of things deployment,” _IEEE Internet of Things Journal_ , vol. 5, no. 6, pp. 4900–4913, 2018.
  * [4] M. Martinez-Diaz, J. Fierrez-Aguilar, F. Alonso-Fernandez, J. Ortega-Garcia, and J. Siguenza, “Hill-climbing and brute-force attacks on biometric systems: A case study in match-on-card fingerprint verification,” in _Proceedings 40th Annual 2006 International Carnahan Conference on Security Technology_ , 2006, pp. 151–159.
  * [5] D. F. Smith, A. Wiliem, and B. C. Lovell, “Face recognition on consumer devices: Reflections on replay attacks,” _IEEE Transactions on Information Forensics and Security_ , vol. 10, no. 4, pp. 736–745, 2015.
  * [6] I. ur Rehman, “Facebook-cambridge analytica data harvesting: What you need to know,” _Library Philosophy and Practice_ , pp. 1–11, 2019.
  * [7] O. F. Segun and F. B. Olawale, “Healthcare data breaches: Biometric technology to the rescue,” _Int. Res. J. Eng. Technol._ , vol. 4, no. 11, pp. 946–950, 2017.
  * [8] P. Singh, “Aadhaar and data privacy: biometric identification and anxieties of recognition in india,” _Information, Communication & Society_, vol. 24, no. 7, pp. 978–993, 2021. [Online]. Available: https://doi.org/10.1080/1369118X.2019.1668459
  * [9] T. H. Journal, “Healthcare data breach statistics,” 2024. [Online]. Available: https://www.hipaajournal.com/healthcare-data-breach-statistics/
  * [10] Y. Wang and K. N. Plataniotis, “An analysis of random projection for changeable and privacy-preserving biometric verification,” _IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)_ , vol. 40, no. 5, pp. 1280–1293, 2010.
  * [11] B. Bortolato, M. Ivanovska, P. Rot, J. Križaj, P. Terhörst, N. Damer, P. Peer, and V. Štruc, “Learning privacy-enhancing face representations through feature disentanglement,” in _2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)_ , 2020, pp. 495–502.
  * [12] U. Uludag and A. Jain, “Securing fingerprint template: Fuzzy vault with helper data,” in _2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’06)_ , 2006, pp. 163–163.
  * [13] E. Abdellatef, N. A. Ismail, S. E. S. Abd Elrahman, K. N. Ismail, M. Rihan, A. El-Samie, and E. Fathi, “Cancelable multi-biometric recognition system based on deep learning,” _The Visual Computer_ , vol. 36, no. 6, pp. 1097–1109, 2020.
  * [14] J. Bringer, H. Chabanne, and A. Patey, “Privacy-preserving biometric identification using secure multiparty computation: An overview and recent trends,” _IEEE Signal Processing Magazine_ , vol. 30, no. 2, pp. 42–52, 2013\.
  * [15] K. Simoens, P. Tuyls, and B. Preneel, “Privacy weaknesses in biometric sketches,” in _2009 30th IEEE Symposium on Security and Privacy_ , 2009, pp. 188–203.
  * [16] E. Newton, L. Sweeney, and B. Malin, “Preserving privacy by de-identifying face images,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 17, no. 2, pp. 232–243, 2005.
  * [17] I. Natgunanathan, A. Mehmood, Y. Xiang, G. Beliakov, and J. Yearwood, “Protection of privacy in biometric data,” _IEEE Access_ , vol. 4, pp. 880–892, 2016.
  * [18] K. Karabina and A. Robinson, “Revisiting the false acceptance rate attack on biometric visual cryptographic schemes,” in _Information Theoretic Security: 9th International Conference, ICITS 2016, Tacoma, WA, USA, August 9-12, 2016, Revised Selected Papers 9_. Springer, 2016, pp. 114–125.
  * [19] A. K. Jindal, S. Chalamala, and S. K. Jami, “Face template protection using deep convolutional neural network,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2018, pp. 575–5758.
  * [20] O. Delgado-Mohatar, J. Fierrez, R. Tolosana, and R. Vera-Rodriguez, “Biometric template storage with blockchain: A first look into cost and performance tradeoffs,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2019, pp. 2829–2837.
  * [21] A. Morales, J. Fierrez, R. Vera-Rodriguez, and R. Tolosana, “Sensitivenets: Learning agnostic representations with application to face images,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 43, no. 6, pp. 2158–2164, 2021.
  * [22] P. Terhörst, D. Fährmann, N. Damer, F. Kirchbuchner, and A. Kuijper, “On soft-biometric information stored in biometric face embeddings,” _IEEE Transactions on Biometrics, Behavior, and Identity Science_ , vol. 3, no. 4, pp. 519–534, 2021.
  * [23] S. P. Singh and S. Tiwari, “A dual multimodal biometric authentication system based on woa-ann and ssa-dbn techniques,” _Sci_ , vol. 5, no. 1, p. 10, 2023\.
  * [24] K. Sasireka and R. Rajesh, “Dual biometric authentication scheme for privacy protection,” in _2014 International Conference on Communication and Network Technologies_ , 2014, pp. 105–108.
  * [25] Y. Dodis, L. Reyzin, and A. Smith, “Fuzzy extractors: How to generate strong keys from biometrics and other noisy data,” in _International conference on the theory and applications of cryptographic techniques_. Springer, 2004, pp. 523–540.
  * [26] Y. Dodis, R. Ostrovsky, L. Reyzin, and A. Smith, “Fuzzy extractors: How to generate strong keys from biometrics and other noisy data,” _SIAM journal on computing_ , vol. 38, no. 1, pp. 97–139, 2008.
  * [27] E.-C. Chang and Q. Li, “Hiding secret points amidst chaff,” in _Annual International Conference on the Theory and Applications of Cryptographic Techniques_. Springer, 2006, pp. 59–72.
  * [28] D. Maltoni, D. Maio, A. K. Jain, and S. Prabhakar, _Handbook of fingerprint recognition_. Springer Science & Business Media, 2009.
  * [29] A. L. Cambridge, “The database of faces,” cam-orl.co.uk/facedatabase.html.
  * [30] A. V. Nefian, “Georgia tech face database,” 1999. [Online]. Available: http://www.anefian.com/research/face_reco.htm
  * [31] H. V. Nguyen and L. Bai, “Cosine similarity metric learning for face verification,” in _Asian conference on computer vision_. Springer, 2010, pp. 709–720.
  * [32] T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 4401–4410.
  * [33] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” in _Thirty-first AAAI conference on artificial intelligence_ , 2017.
  * [34] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman, “Vggface2: A dataset for recognising faces across pose and age,” in _2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018)_. IEEE, 2018, pp. 67–74.
  * [35] T. Esler, “facenet-pytorch,” https://github.com/timesler/facenet-pytorch, 2020.
  * [36] D. Yi, Z. Lei, S. Liao, and S. Z. Li, “Learning face representation from scratch,” _arXiv preprint arXiv:1411.7923_ , 2014.
  * [37] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, “Joint face detection and alignment using multitask cascaded convolutional networks,” _IEEE signal processing letters_ , vol. 23, no. 10, pp. 1499–1503, 2016.
  * [38] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 4690–4699.
  * [39] Z. Zhu, G. Huang, J. Deng, Y. Ye, J. Huang, X. Chen, J. Zhu, T. Yang, J. Lu, D. Du _et al._ , “Webface260m: A benchmark unveiling the power of million-scale deep face recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 10 492–10 502.
  * [40] X. An, X. Zhu, Y. Gao, Y. Xiao, Y. Zhao, Z. Feng, L. Wu, B. Qin, M. Zhang, D. Zhang _et al._ , “Partial fc: Training 10 million identities on a single machine,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 1445–1449.
  * [41] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [42] G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeled faces in the wild: A database for studying face recognition in unconstrained environments,” University of Massachusetts, Amherst, Tech. Rep. 07-49, October 2007.