# Signed Input Regularization

Saeid Asgari Taghanaki  Kumar Abhishek  Ghassan Hamarneh

###### Abstract

Over-parameterized deep models usually over-fit to a given training
distribution, which makes them sensitive to small changes and out-of-
distribution samples at inference time, leading to low generalization
performance. To this end, several model-based and randomized data-dependent
regularization methods are applied, such as data augmentation, which prevents
a model from memorizing the training distribution. Instead of the random
transformation of the input images, we propose SIGN, a new regularization
method, which modifies the input variables using a linear transformation by
estimating each variable’s contribution to the final prediction. Our proposed
technique maps the input data to a new manifold where the less important
variables are de-emphasized. To test the effectiveness of the proposed idea
and compare it with other competing methods, we design several test scenarios,
such as classification performance, uncertainty, out-of-distribution, and
robustness analyses. We compare the methods using three different datasets and
four models. We find that SIGN encourages more compact class representations,
which results in the model’s robustness to random corruptions and out-of-
distribution samples while also simultaneously achieving superior performance
on normal data compared to other competing methods. Our experiments also
demonstrate the successful transferability of the SIGN samples from one model
to another.

##  1 Introduction

The classical computer vision methods generally achieve lower performance
compared to the current deep neural networks [19]. This is mainly because
classical methods are designed to capture human-level information from images
at the expense of curtailing their level of freedom from the beginning of the
exploration of a solution to the problem. In contrast, deep models might not
be able to provide human-communicable latent information from signals, but
they achieve superior performance in various machine learning tasks. However,
extensive regularization of the deep networks reduces their freedom of
exploration.

Several different regularization techniques have been successfully applied to
over-parameterized models to reduce over-fitting and to generalize to unseen
data, including the $L_{2}$ norm penalty for model parameters, weight decay
[17], early stopping, and dropout [27]. Although initially not introduced as a
regularization method, batch normalization [8] performs regularization by
considering fluctuations within mini-batches, albeit with a generalization
performance of deep neural networks that often depends on the size of the
mini-batch [11]. Similarly, stochastic gradient descent (SGD) [29] can also be
interpreted as a regularized gradient descent impaired by noisy gradients.
However, SGD’s fluctuations do not lead to a smooth convergence. Other types
of regularization are data-dependent, such as traditional data augmentation
techniques (rotation, flipping, etc.) [13, 25], AutoAugment [5], and the mixup
method [33]. The regularization properties of mixup were studied by Guo et al.
[7]. Although a few works have attempted to improve the mixup method [24, 16],
besides the smoothed decision boundaries which can cause a model to be highly
under-confident, the random sample selection step of mixup may produce wrong
labels, as shown in Figure 1, thus failing to generalize well. Another issue
with the mixup method is that the linear blending results in implausible
images. As we detail later, our proposed method modifies the input to generate
new samples as well, but it does so while avoiding the drawbacks of the linear
blending.

Figure 1: The mixup method leads to wrong labels for the samples on the
boundaries due to random sample selection. Blue and red dots represent samples
from two different classes. The dashed line represents the interpolation
function ($\lambda(p_{1})+(1-\lambda)p_{2}$) for both images and labels, which
was proposed by the authors [33] for the mixing step. The samples inside the
red dotted circle should be assigned a blue label; however, due to random
sampling, they get a red label, and vice-versa for the blue dotted circle.
$\mathcal{J}$ represents the image space.

Adversarial training is another data-dependent regularization approach [6, 21]
in which an adversarially perturbed version of data is used as a type of
augmentation. However, an adversarially perturbed image contains perceptible
information about the correct class and imperceptible information about a
wrong or a random label. Ideally, a model should learn to cancel out the
deliberate implicit patterns specific to the wrong class, but with an
arbitrary level of perturbation (which is the case in adversarial training), a
model might get biased towards an implausible distribution. In other words,
from an adversarial attack perspective, it is ideal to destroy a well-
distributed manifold. Therefore, using adversarial training as data
augmentation for regularization can skew the feature space towards an
arbitrary class distribution.

In contrast to adversarial training that perturbs an input image to effectuate
a wrong prediction, Taghanaki et al. [31] proposed a gradient based method
that perturbs the input image by transforming it to a new space to effectuate
a more accurate prediction. However, since their method relies on the true
labels, which are not available at the inference time, they leverage a second
model to learn the mapping between the input and the corresponding transformed
images, and show an improvement in performance with a computational complexity
trade-off. As we describe later, our proposed method also generates new
perturbed images that lead to improved predictions, however, it does not
require ground truth labels for mapping.

Relation to causality. Understanding the effect of input variables on the
prediction function’s output is essentially a causality problem. Recently,
several approaches have successfully applied reverse gradients towards the
input to decode, and in particular, to visualize the causal effect of input
variables on the overall final prediction [1, 23, 26, 28]. However, they have
not studied the causal influence of each single input variable towards a
better prediction performance. In this paper, we attempt to answer the
question: “can the casual influence of each variable in the input manifold be
modified to limit the number of input variables by discarding the less
effective ones?”. If a model can successfully capture only the truly
correlated variables in the input space to the labels, then the model should
be robust to manifold shifts and out-of-distribution samples [2] without the
need to perform explicit regularization. Leveraging prior knowledge, such as
location information of target objects, might help regularize the model and
steer it to focus only on the relevant variables in the input, which has
motivated recent works on attention priors [32, 34]. However, such location
labels are not always provided and, similar to over-regularization, excessive
prior knowledge might restrict the model’s exploration of the data and miss
other more discriminative latent information.

To facilitate the model’s exploration with only relevant input variables, we
seek a transformation that projects the input to a new space where the
important variables are emphasised. Therefore, we adopt the push-forward
approach [9, 14] to approximate the changes in the output for each input
variable. In contrast to methods that limit the exploration of the model by
turning off the neurons and scaling the model parameters, such as dropout and
batch normalization, and inspired by the adversarial training concept, we
introduce a new method called Signed Input Regularization (SIGN).

The SIGN method re-weights the input (pixels in the image space,
$\mathcal{J}$) such that an input sample $p\in\mathbb{R}^{+}$ is transformed
to $p+\delta\in\mathbb{R}$. The variables $\in\mathbb{R}^{-}$ are discarded by
a model with ReLU non-linearity. Thus the model focuses on fewer input
variables.

In this paper, we make the following contributions:

  * •

We introduce a new Jacobian-based regularization method which facilitates the
model’s exploration by reducing the variables in the input.

  * •

We show how the push-forward and manifold mapping concepts can be used for the
linear estimation of input using a layer’s output in a model.

  * •

Our proposed method achieves significantly better results compared to
traditional data augmentation and the mixup regularization schemes. We also
discuss and demonstrate the potential shortcomings of the mixup method.

##  2 Method

Given a training sample $p$, the goal is to transform it to a new space. Let
$\Phi:\mathcal{N}\rightarrow\mathcal{M}$ be a map between manifolds
$\mathcal{N}$ and $\mathcal{M}$. Then, the derivative $d\Phi(p)$, the
derivative of $\Phi$ at point $p$ in the tangent space of $\mathcal{N}$ is the
best linear approximation of $\Phi$ near $p$ in the tangent space of
$\mathcal{M}$. Therefore, $d\Phi(p)$ pushes tangent vectors on $\mathcal{N}$
forward to tangent vectors on $\mathcal{M}$ [14], as shown in Figure 2.

Figure 2: Visualization of the push-forward map from the tangent space in
$\mathcal{N}$ to the tangent space in $\mathcal{M}$. Compared to the mixup
method (Figure 1), which transforms the sample in the same manifold using a
linear combination, our proposed method moves the samples to a new manifold.
$\mathcal{J}$ represents the image space.

###  2.1 The Push Forward Map

Let $\Phi:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$ be a smooth function
($C^{\infty}$) and let $p\in\mathbb{R}^{n}$. The goal is to define the push
forward map [9, 14] as

$\Phi_{*,p}:T_{p}\mathbb{R}^{n}\rightarrow T_{\Phi(p)}\mathbb{R}^{m}.$ (1)

Let $X_{p}\in T_{p}\mathbb{R}^{n}$ be a tangent vector (in tangent space $T$)
which by definition is completely determined by its action on $C^{\infty}(p)$.
In order that $\Phi_{*,p}\left(X_{p}\right)$ to be a tangent vector at
$q=\check{\Phi}(p)\in\mathbb{R}^{m}$, we need to define

$\Phi_{*,p}\left(X_{p}\right)(g)\quad\forall g\in C^{\infty}(q).$ (2)

Let $g\in C^{\infty}(q)$ be a function in the space of $\Phi$. The composition
$g\circ\Phi:\mathbb{R}^{n}\rightarrow\mathbb{R}$ is a smooth function on an
open subset of $\mathbb{R}^{n}$ which contains $p$, and thus
$X_{p}(g\circ\Phi)$ is well defined. Let
$\Phi:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$ be a smooth function and let
$q=\Phi(p)$. Given $X_{p}\in T_{p}\mathbb{R}^{n}\text{ ,
}\Phi_{*,p}\left(X_{p}\right):C^{\infty}(q)\rightarrow\mathbb{R}$ is defined
by

$\Phi_{*,p}\left(X_{p}\right)(g)=X_{p}(g\circ\Phi)\quad\forall g\in
C^{\infty}(q)$ (3)

Equation 3 can be written using the coordinate bases. Let
$\beta=\left\\{\left.\partial_{x^{i}}\right|_{p}\right\\}_{1\leq i\leq n}$ be
the coordinate basis for $T_{p}\mathbb{R}^{n}$ and let
$\gamma=\left\\{\left.\partial_{y^{a}}\right|_{q}\right\\}_{1\leq a\leq m}$ be
the coordinate basis for $T_{q}\mathbb{R}^{m}$. If $X_{p}\in
T_{p}\mathbb{R}^{n}$ is given by

$X_{p}=\left.\sum_{i=1}^{n}\xi^{i}\partial_{x^{i}}\right|_{p},\quad\xi^{i}\in\mathbb{R},$
(4)

then,

$\Phi_{*,p}X_{p}=\left.\sum_{a=1}^{m}\left(\left.\sum_{i=1}^{n}\xi^{i}\frac{\partial\Phi^{a}}{\partial
x^{i}}\right|_{p}\right)\frac{\partial}{\partial y^{a}}\right|_{q}.$ (5)

Therefore, the matrix representation of the linear transformation
$\Phi_{*,p}:T_{p}\mathbb{R}^{n}\rightarrow T_{q}\mathbb{R}^{m}$ in the basis
$\beta=\left\\{\left.\partial_{x^{i}}\right|_{p}\right\\}_{1\leq i\leq n}$ for
$T_{p}\mathbb{R}^{n}$ and the basis
$\gamma=\left\\{\left.\partial_{y^{a}}\right|_{q}\right\\}_{1\leq a\leq m}$
for $T_{\Phi(p)}\mathbb{R}^{m}$ is given by the Jacobian

$\left[\Phi_{*,p}\right]=\left.\frac{\partial\Phi^{a}}{\partial
x^{i}}\right|_{p}$ (6)

Proof. If we write the matrix representation of $\Phi_{*,p}$ using the matrix
$\left(J_{i}^{a}\right)\in M_{m\times n}(\mathbb{R})$, we get

$\Phi_{*,p}\left(\left.\partial_{x^{i}}\right|_{p}\right)=\left.\sum_{a=1}^{m}J_{i}^{a}\partial_{y^{a}}\right|_{q}.$
(7)

The matrix $J_{i}^{a}$ can then be calculated using Equation 5, and therefore

$\Phi_{*,p}\left(\left.\partial_{x^{i}}\right|_{p}\right)=\left.\left.\sum_{a=1}^{m}\frac{\partial\Phi^{a}}{\partial
x^{i}}\right|_{p}\partial_{y^{a}}\right|_{q}$ (8)

$\implies\left[\Phi_{*,p}\right]=\left.\frac{\partial\Phi^{a}}{\partial
x^{i}}\right|_{p}.$

Therefore, the map $\Phi_{*,p}$ can be thought of as a matrix of partial
derivatives:

$\Phi_{*,p}=\left(\begin{array}[]{cccc}{\frac{\partial\phi^{1}}{\partial
x^{1}}}&{\frac{\partial\phi^{1}}{\partial
x^{2}}}&{\cdots}&{\frac{\partial\phi^{1}}{\partial x^{m}}}\\\
{\frac{\partial\phi^{2}}{\partial x^{1}}}&{\frac{\partial\phi^{2}}{\partial
x^{2}}}&{\cdots}&{\frac{\partial\phi^{2}}{\partial x^{m}}}\\\
{\vdots}&{\vdots}&{\ddots}&{\cdots}\\\ {\frac{\partial\phi^{n}}{\partial
x^{1}}}&{\frac{\partial\phi^{n}}{\partial
x^{2}}}&{\cdots}&{\frac{\partial\phi^{n}}{\partial x^{m}}}\end{array}\right).$
(9)

###  2.2 Iterative SIGN Method

Given a trained model, the goal is to transform the input with a linear
estimation using the model’s $l^{th}$ layer under the push-forward definition,
which is realized by the Jacobian, as explained in Section 2.1. Not only the
Jacobian provides the linear estimation of the input, it also highlights the
variables in the input which contribute the most to the output of the layer.
The input variables with negative sign are discarded after they are
transferred to manifold $\mathcal{M}\in\mathbb{R}$ because the following ReLU
activation function does not let them pass on to the subsequent layers.

In order to emphasize the input variable discarding step, we use a similar
approach to the momentum method [20], which is a technique for accelerating
gradient descent algorithms by accumulating a velocity vector in the gradient
direction of the loss function across iterations. Our iterative process is
summarized in Algorithm 1.

Input : A well-trained model $\Phi$, a real sample $p$, and number of
iterations $K$

Output : Transformed input sample $p^{*}$

1 $p_{0}^{*}=p$;

2 for  _$k=0$ to $K-1$_ do

3 $\delta_{k+1}=\sum_{a}\left.\frac{\partial\Phi^{a}}{\partial
x^{i}}\right|_{p}$ ;

4 $p_{k+1}^{*}=p^{*}+\delta_{k+1}$

5 end for

6$p^{*}=p_{K}^{*}$;

return _$p^{*}$_

Algorithm 1 Iterative SIGN Table 1: Class-wise and overall classification
performances of different methods on CIFAR-10 using MobileNetV2.

Model | Method | airplane | automobile | bird | cat | deer | dog | frog | horse | ship | truck | Mean
---|---|---|---|---|---|---|---|---|---|---|---|---
MobileNetV2 [22] | Classical [25] | $0.834$ | $0.940$ | $0.850$ | $0.691$ | $0.858$ | $0.758$ | $0.914$ | $0.875$ | $0.909$ | $0.921$ | $0.8550$
mixup [33] | 0.870 | $0.902$ | 0.861 | 0.793 | $0.810$ | $0.716$ | $0.886$ | 0.884 | $0.935$ | $0.927$ | $0.8584$
SIGN (proposed) | $0.868$ | 0.931 | $0.806$ | $0.717$ | 0.864 | 0.763 | 0.936 | $0.875$ | 0.941 | 0.936 | 0.8637
BasicCNN | DeepAugment [18] | 0.885 | $0.930$ | $0.794$ | 0.733 | 0.911 | $0.746$ | $0.893$ | 0.915 | 0.924 | $0.917$ | $0.8648$
(Section 3) | DeepAugment + SIGN | $0.866$ | 0.939 | 0.819 | $0.728$ | $0.867$ | 0.816 | 0.920 | $0.898$ | $0.919$ | 0.919 | 0.8691

Table 2: Aleatoric uncertainty analysis of different methods on CIFAR-10
dataset.

Method | Metric | airplane | automobile | bird | cat | deer | dog | frog | horse | ship | truck | Mean
---|---|---|---|---|---|---|---|---|---|---|---|---
Classical [25] | Overall accuracy | $0.851$ | $0.930$ | $0.716$ | $0.721$ | $0.842$ | $0.631$ | 0.934 | $0.864$ | $0.816$ | 0.929 | $0.8234$
| $p\leq 0.5$
---
(# images)
| $0.4325$
---
$(17)$
| $0.4081$
---
$(6)$
| $0.4370$
---
$(31)$
| 0.4414
---
$(40)$
| $0.4346$
---
$(30)$
| $0.4449$
---
$(41)$
| $0.4402$
---
$(9)$
| $0.4435$
---
$(19)$
| $0.4244$
---
$(12)$
| $0.4102$
---
$(7)$
| $0.4370$
---
$(212)$
[uncertainty] | $[6.5198]$ | $[5.6784]$ | $[7.1620]$ | $[7.3614]$ | $[7.3126]$ | $[7.5829]$ | $[7.0295]$ | $[7.5014]$ | $[5.8672]$ | $[6.1368]$ | $[7.1265]$
mixup [33] | Overall accuracy | 0.866 | $0.900$ | $0.707$ | $0.625$ | $0.819$ | $0.689$ | $0.913$ | $0.912$ | $0.910$ | $0.919$ | $0.8260$
| $p\leq 0.5$
---
(# images)
| $0.3326$
---
$(30)$
| $0.4394$
---
$(15)$
| $0.4136$
---
$(46)$
| $0.4110$
---
$(93)$
| 0.4348
---
$(50)$
| $0.4175$
---
$(59)$
| $0.4296$
---
$(25)$
| $0.4053$
---
$(27)$
| $0.4314$
---
$(12)$
| 0.4376
---
$(6)$
| $0.4123$
---
$(363)$
[uncertainty] | $[3.4691]$ | $[4.5749]$ | $[4.2924]$ | $[4.6358]$ | $[4.3798]$ | $[4.6574]$ | $[4.3065]$ | $[4.5253]$ | $[4.6718]$ | $[4.4918]$ | $[4.4295]$
SIGN (proposed) | Overall accuracy | $0.864$ | 0.949 | 0.807 | 0.725 | 0.849 | 0.764 | $0.895$ | 0.918 | 0.926 | $0.894$ | 0.8591
| $p\leq 0.5$
---
(# images)
| 0.4292
---
$(\textbf{2})$
| 0.4571
---
$(\textbf{1})$
| 0.4468
---
$(\textbf{4})$
| $0.4199$
---
$(\textbf{6})$
| $0.4275$
---
$(\textbf{2})$
| 0.4576
---
$(\textbf{3})$
| 0.4535
---
$(\textbf{3})$
| 0.4574
---
$(\textbf{3})$
| 0.4984
---
$(\textbf{1})$
| $0.4322$
---
$(\textbf{1})$
| 0.4428
---
$(\textbf{26})$
[uncertainty] | $[6.4461]$ | $[8.4259]$ | $[6.1156]$ | $[5.9603]$ | $[5.7440]$ | $[6.7875]$ | $[6.4557]$ | $[6.8555]$ | $[8.9375]$ | $[8.4732]$ | $[6.5668]$

##  3 Experiments

We evaluate our proposed method on three datasets: CIFAR-10 [12], Tiny
ImageNet [3], and 2D RGB skin lesion classification dataset from the 2017 IEEE
International Skin Imaging Collaboration (ISIC) ISBI Challenge [4], and four
different deep neural network architectures: MobileNetV2 [22], Inception-
ResNet-v2 [30], NASNetMobile [35], and a simple small network (called
BasicCNN) with the following architecture to show the effectiveness of the
proposed method regardless of the complexity of the architecture.

[Conv2D_a $\rightarrow$ ReLu $\rightarrow$ Conv2D_a $\rightarrow$ ReLu
$\rightarrow$ MaxPool2D $\rightarrow$ Dropout] $\rightarrow$ [Conv2D_b
$\rightarrow$ ReLu $\rightarrow$ Conv2D_b $\rightarrow$ ReLu $\rightarrow$
MaxPool2D $\rightarrow$ Dropout] $\rightarrow$ [FC512 $\rightarrow$ ReLu
$\rightarrow$ Dropout],

where Conv2D_a and Conv2D_b layers have 32 and 64 filter channels each. All
the Conv2D layers use $3\times 3$ kernels, all the MaxPool2D layers use
$2\times 2$ kernels, and all the dropout layers have a drop probability of
$0.3$.

We compare our proposed SIGN method with classical data augmentation
strategies (horizontal and vertical flipping, shifting, and rotations) on
normal data, the mixup transformed samples, and DeepAugment [18], the Bayesian
version of the AutoAugment [5] methods. We train all the models with a batch
size of 128 except for the Inception-ResNet-v2 in the skin lesion
classification experiment, for which we set the batch size to 32. We train all
the models except BasicCNN for 100 epochs and select the epochs corresponding
to the highest accuracies on the validation sets. The BasicCNN model, which
was used to test DeepAugment was trained for 200 epochs.

We set $K$ (the number of iterations) for iterative SIGN (Algorithm 1) to 50
and 100, thereby augmenting the training set with two different versions. For
the source manifold, we use the output of the last layer (before the logits)
of the model to estimate the target manifold which is the input variables’
space.

In the following subsections, we start with a general classification
performance analysis in subsection 3.1. In subsection 3.2, we design a
MobileNetV2-based model to capture the aleatoric uncertainty [10] of different
regularization methods. Next, we set up an experiment to evaluate the models’
robustness to random perturbations and out-of-distribution samples in
subsection 3.3. Finally, we study the transferability of the SIGN-generated
samples to other models in subsection 3.4.

###  3.1 Classification Performance Analysis

In this subsection, we study the performance of the proposed approach and
compare it with other methods in terms of the classification accuracy. We
first test all the three regularization methods on the CIFAR-10 dataset using
the MobileNetV2 and the BasicCNN models. As reported in Table 1, the proposed
method achieves the highest overall classification accuracy for both the
models. Note that for the BasicCNN model, we apply the learned augmentation
policies from normal CIFAR-10 images to both normal and SIGN transformed
samples. We do this to study the transferability of the augmentation policies
on the SIGN samples.

In Figure 3, we visualize a few samples from CIFAR-10 before and after
applying the SIGN regularization method. As can be seen, the color contrasts
and the boundaries of the objects are enhanced after applying SIGN.

Figure 3: CIFAR-10 samples before (top) and after (bottom) applying SIGN.

Next, we apply the winner strategy (SIGN) from the previous experiments on a
separate dataset and a model, i.e., the ISIC dataset and the Inception-
ResNet-v2 model. The task here is to predict whether a skin lesion is a
melanoma or not. The dataset consists of 2000 dermoscopic images of skin
lesions, out of which only 374 belong to the positive class, indicating a
considerable class imbalance. Our proposed SIGN method improves the baseline
model’s area under the curve (AUC) by $\sim 5\%$ and $\sim 2\%$ on validation
and test sets ($0.7633$ versus $0.8131$ and $0.7611$ versus $0.7787$),
respectively. Note that for this experiment, we do not apply any data
augmentation, so as to ascertain that the improvement is from the application
of the SIGN method. It can be seen in Figure 4 that our SIGN method improves
the dermoscopic features of the skin lesions, which leads to an improvement in
the classification performance.

Figure 4: ISIC skin lesion samples before (top) and after (bottom) applying
SIGN.

Next, we visualize the t-SNE [15] feature space of the different approaches by
transforming and projecting their high dimensional features into a 2D space.
As can be seen in Figure 5, the proposed SIGN method results in more compact
class representations for both the training and the validation samples.
However, for the mixup method, the linear interpolation function causes label
smoothing that is manifested as fuzzy boundaries across the different classes.

(a) Classical [25]

(b) mixup [33]

(c) SIGN (proposed)

Figure 5: Feature space visualization for models trained under different
schemes. The first and second rows represent plots for training and validation
sets, respectively.

In order to certify that there is useful signal for correct classification in
the perturbation $\delta$ added by our iterative SIGN method to the samples,
we train and evaluate a MobileNetV2 model with only the $\delta$s, i.e., (the
linear estimation of the input) from the last layer of the model. We obtain a
mean classification accuracy of 46.2% for CIFAR-10 using only $\delta$s which
shows that $\delta$ captures the classes information as expected.

###  3.2 Uncertainty Analysis

To capture uncertainties in the models’ predictions, which arise because of
the inherent noise in the observations, we model the aleatoric uncertainty
[10]. Particularly, we define a stochastic loss function which models the
aleatoric uncertainty as

$\mathcal{L}_{x}=\sum_{i}\log\frac{1}{T}\sum_{t}\exp\left(\hat{x}_{i,t,c}-\log\sum_{c^{\prime}}\exp\hat{x}_{i,t,c^{\prime}}\right),$
(10)

where $c$ is the class label for sample input $i$. We design the model
architecture such that it outputs $\mathbf{f}_{i}^{\mathbf{W}}$ and
$\sigma_{i}^{\mathbf{W}}$ with parameters $\mathbf{W}$, where $\mathbf{f}_{i}$
is unary for input $i$ and $\hat{\mathbf{x}}_{i,1\cdots t}$ are the $t$ Monte
Carlo approximations (samples) of the unaries using the learned
$\sigma_{i}^{\mathbf{W}}$ which is defined as

$\hat{\mathbf{x}}_{i,t}=\mathbf{f}_{i}^{\mathbf{W}}+\sigma_{i}^{\mathbf{W}}\epsilon_{t},\quad\epsilon_{t}\sim\mathcal{N}(0,I)$
(11)

Table 2 summarizes the results of this experiment on CIFAR-10 dataset. As
expected, the mixup method makes the decision boundaries smooth (see Figure
5), which causes the model(s) trained with this data augmentation strategy to
be highly under-confident, i.e., failing to know when it fails. In particular,
the mixup method’s threshold for correctly predicted classes is slightly above
$0.1$, while the same value is $0.26$ and $0.33$ for the standard data
augmentation and SIGN methods, respectively. This means that for a model
trained with mixup, the lowest probability an image could have while still
being correctly predicted is a little over $0.1$, which is just above the
probability of randomly assigning a correct label for CIFAR-10 images.

Table 3: Out-of-distribution results on the Tiny ImageNet dataset using a MobileNetV2 model trained on the CIFAR-10 dataset. Method | Metric | automobile | bird | cat | dog | frog | Mean
---|---|---|---|---|---|---|---
| (total images) | (49) | (146) | (50) | (298) | (100) | (643)
Classical [25] | Accuracy | $0.8163$ | 0.5479 | $0.6800$ | $0.3289$ | 0.8300 | $0.5210$
| $p\leq 0.5$
---
(# images)
| $0.4423$
---
(1)
| $0.4225$
---
$(12)$
| 0.4438
---
$(2)$
| $0.4135$
---
$(17)$
| $0.4701$
---
$(3)$
| $0.4240$
---
$(35)$
[uncertainty] | $[5.9622]$ | $[6.0362]$ | $[8.0968]$ | $[7.0100]$ | $[7.4542]$ | $[6.7464]$
mixup [33] | Accuracy | $0.7551$ | $0.4726$ | $0.6800$ | $0.2919$ | $0.7900$ | $0.4759$
| $p\leq 0.5$
---
(# images)
| 0.4759
---
$(3)$
| 0.4314
---
$(9)$
| $0.4133$
---
$(6)$
| $0.4250$
---
$(20)$
| $0.3867$
---
$(3)$
| 0.4256
---
$(41)$
[uncertainty] | $[4.8710]$ | $[4.5847]$ | $[4.4742]$ | $[4.8388]$ | $[3.8331]$ | $[4.6584]$
SIGN (proposed) | Accuracy | 0.9184 | $0.3973$ | 0.7200 | 0.4497 | $0.8000$ | 0.5490
| $p\leq 0.5$
---
(# images)
| $0.3854$
---
(1)
| $0.3125$
---
(1)
| N/A
---
(0)
| 0.4611
---
(3)
| N/A
---
(0)
| $0.4162$
---
(5)
[uncertainty] | $[4.8829]$ | $[4.1209]$ | [N/A] | $[6.3664]$ | [N/A] | $[5.6206]$
Table 4: Transferability results of the proposed SIGN method. Evaluating a
NASNetMobile model trained using CIFAR-10 samples transformed using SIGN
method from a MobileNetV2 model.

Method | airplane | automobile | bird | cat | deer | dog | frog | horse | ship | truck | Mean
---|---|---|---|---|---|---|---|---|---|---|---
Classical [25] | $0.846$ | $0.873$ | $0.717$ | $0.620$ | $0.722$ | 0.741 | 0.962 | $0.804$ | $0.907$ | $0.893$ | $0.8085$
Transfer (with SIGN) | 0.858 | 0.927 | 0.813 | 0.672 | 0.829 | $0.740$ | $0.897$ | 0.898 | 0.934 | 0.910 | 0.8478

Table 5: Robustness of the methods to additive noise and pixel corruption. Model | Method | Normal accuracy | Pixels turned off | Gaussian noise
---|---|---|---|---
(50 random pixels) |  ($\mu=0,\sigma=10$)
MobileNetV2 [22] |  Classical [25] | $0.8550$ | $0.6672\pm$310-4$$ | $0.6552\pm$410-5$$
mixup [33] | $0.8584$ | $0.5970\pm$210-4$$ | $0.6590\pm$910-5$$
SIGN (proposed) | 0.8637 | $\textbf{0.6690}\pm\textbf{3e--4}$ | $\textbf{0.7122}\pm\textbf{3e--5}$
BasicCNN (Section 3) |  DeepAugment [18] | $0.8648$ | $0.8361\pm$110-5$$ | $0.7875\pm$110-5$$
DeepAugment + SIGN | 0.8691 | $\textbf{0.8479}\pm\textbf{4e--6}$ | $\textbf{0.7980}\pm\textbf{2e--5}$

In Table 2, we report the mean probability values of the correctly classified
test samples with a correct class probabilities less than $0.5$, i.e.,
relatively low confidence. As can be seen in the table, for all the classes
combined, our proposed method only has $26$ samples in total with
probabilities below the threshold value, while there are $363$ and $212$
samples for mixup and the standard data augmentation methods respectively. As
can be seen in the last row of the same table, the SIGN method obtains
reasonably high uncertainty values for the samples with a low confidence
level, which is desirable, since we want a model to know when it fails by
producing high uncertainty. However, the mixup method causes the model to
produce relatively smaller uncertainty values while it is still highly under-
confident, indicating that the model is unaware of its failures.

###  3.3 Out-of-distribution and Robustness Analysis

Next, we study the performance of the models trained with different data
augmentation techniques by evaluating on out-of-distribution and corrupted
samples, a few examples of which are shown in Figure 6. For evaluating on out-
of-distribution samples, we train a MobileNetV2 model on the CIFAR-10 dataset
and test on Tiny ImageNet. In particular, we extract images from multiple
classes in the TinyImageNet dataset so as to correspond to the classes present
in CIFAR-10 and resize them to the CIFAR-10 image resolution ($32\times 32$).
We use 49 images of ‘automobile’ class, 146 images of ‘bird’ class, 50 images
of ‘cat’ class, 298 images of ‘dog’ class, and 100 images of ‘frog’ class,
totaling to 643 images overall. As can be seen in the third row of Table 3,
our method achieves significantly better results ($\sim 7\%$ improvement over
mixup) compared to other techniques.

Next, to test the robustness of the methods to different input corruptions
using two different models and five different augmentation approaches, we
apply two types of corruptions to the test samples ( which are assumed to be
drawn from the same distribution as the train samples): (a) we add Gaussian
noise with mean $0$ and standard deviation of $10$ to the images, and (b) we
turn off 50 randomly selected pixels in each image. Table 5 summarized the
results for this experiment. The SIGN method shows higher resistance from
other methods. The DeepAugment method’s results are improved from $83.61\%$ to
$84.79\%$ when it uses the SIGN transformed samples.

This experiment shows that if a method is restricted to learn from a fewer
input variables, as SIGN does, it should be robust when a few variables are
missing, which is simulated in this experiment by zeroing out random pixels.

Figure 6: Samples of the corrupted CIFAR-10 images. Rows from top to bottom
show the normal, pixel-off, and Gaussian noise, respectively.

###  3.4 Transferability of SIGN Samples

Finally, we examine whether the transformed samples obtained with our SIGN
method are effective in other unknown models. To this end, we leverage the
samples mapped to a new space using a MobileNetV2 model to train a
NASNetMobile model. As reported in Table 4, the SIGN method’s samples are
transferable as they improve the NASNetMobile baseline model’s mean
classification accuracy on the CIFAR-10 dataset by $\sim 4\%$, i.e., from
$0.8085$ to $0.8478$, including considerable improvements in class-wise
accuracies for 8 out of 10 classes.

##  4 Conclusion

We proposed SIGN, a regularization method that can be used as a data
augmentation strategy. Our proposed iterative SIGN technique produces
different transformations of the input data, which are then used to train a
model. We showed how the push-forward concept in manifold transformation could
be applied to both obtain the linear estimation of the input using a layer in
the model and diminish the effect of the non-important input variables by
assigning them negative signs. We showed that the iterative SIGN method could
help for better generalization performance of deep models.

We also discussed the critical limitations and risks of using the mixup and
the adversarial training methods as regularization techniques. We evaluated
the proposed idea for several classification tasks and demonstrated the
superior classification accuracy obtained using iterative SIGN regularization.
Moreover, we note that the proposed method can also be extended to other
problems such as dense image labeling and object detection. Another possible
future direction is to study the feasibility of using the SIGN method to
cancel out the adversarial perturbations added by adversarial attack
strategies.

## References

  * [1] S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Müller, and W. Samek.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  PloS one, 10(7):e0130140, 2015.
  * [2] Y. Bengio, T. Deleu, N. Rahaman, R. Ke, S. Lachapelle, O. Bilaniuk, A. Goyal, and C. Pal.  A meta-transfer objective for learning to disentangle causal mechanisms.  arXiv preprint arXiv:1901.10912, 2019.
  * [3] P. Chrabaszcz, I. Loshchilov, and F. Hutter.  A downsampled variant of ImageNet as an alternative to the CIFAR datasets.  arXiv preprint arXiv:1707.08819, 2017.
  * [4] N. C. Codella, D. Gutman, M. E. Celebi, B. Helba, M. A. Marchetti, S. W. Dusza, A. Kalloo, K. Liopyris, N. Mishra, H. Kittler, et al.  Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC).  In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pages 168–172. IEEE, 2018.
  * [5] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le.  Autoaugment: Learning augmentation policies from data.  arXiv preprint arXiv:1805.09501, 2018.
  * [6] I. J. Goodfellow, J. Shlens, and C. Szegedy.  Explaining and harnessing adversarial examples.  arXiv preprint arXiv:1412.6572, 2014.
  * [7] H. Guo, Y. Mao, and R. Zhang.  Mixup as locally linear out-of-manifold regularization.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3714–3722, 2019.
  * [8] S. Ioffe and C. Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  arXiv preprint arXiv:1502.03167, 2015.
  * [9] J. Jost and J. Jost.  Riemannian geometry and geometric analysis, volume 42005.  Springer, 2008.
  * [10] A. Kendall and Y. Gal.  What uncertainties do we need in Bayesian deep learning for computer vision?  In Advances in Neural Information Processing Systems, pages 5574–5584, 2017.
  * [11] N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang.  On large-batch training for deep learning: Generalization gap and sharp minima.  arXiv preprint arXiv:1609.04836, 2016.
  * [12] A. Krizhevsky, G. Hinton, et al.  Learning multiple layers of features from tiny images.  Technical report, Citeseer, 2009.
  * [13] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, et al.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [14] J. M. Lee.  Smooth manifolds.  In Introduction to Smooth Manifolds, pages 1–31. Springer, 2013\.
  * [15] L. v. d. Maaten and G. Hinton.  Visualizing data using t-SNE.  Journal of Machine Learning Research, 9(Nov):2579–2605, 2008.
  * [16] Z. Mai, G. Hu, D. Chen, F. Shen, and H. T. Shen.  Metamixup: Learning adaptive interpolation policy of mixup with meta-learning.  arXiv preprint arXiv:1908.10059, 2019.
  * [17] S. J. Nowlan and G. E. Hinton.  Simplifying neural networks by soft weight-sharing.  Neural Computation, 4(4):473–493, 1992.
  * [18] B. Ozmen.  deepaugment.  https://github.com/barisozmen/deepaugment, 2019.
  * [19] N. O’Mahony, S. Campbell, A. Carvalho, S. Harapanahalli, G. V. Hernandez, L. Krpalkova, D. Riordan, and J. Walsh.  Deep learning vs. traditional computer vision.  In Science and Information Conference, pages 128–144. Springer, 2019.
  * [20] B. T. Polyak.  Some methods of speeding up the convergence of iteration methods.  USSR Computational Mathematics and Mathematical Physics, 4(5):1–17, 1964.
  * [21] K. Roth, Y. Kilcher, and T. Hofmann.  Adversarial training generalizes data-dependent spectral norm regularization.  arXiv preprint arXiv:1906.01527, 2019.
  * [22] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen.  MobileNetV2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4510–4520, 2018.
  * [23] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra.  Grad-CAM: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE International Conference on Computer Vision, pages 618–626, 2017.
  * [24] T. Shimada, S. Yamaguchi, K. Hayashi, and S. Kobayashi.  Data interpolating prediction: Alternative interpretation of mixup.  arXiv preprint arXiv:1906.08412, 2019.
  * [25] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [26] D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg.  SmoothGrad: removing noise by adding noise.  arXiv preprint arXiv:1706.03825, 2017.
  * [27] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  The Journal of Machine Learning Research, 15(1):1929–1958, 2014\.
  * [28] M. Sundararajan, A. Taly, and Q. Yan.  Axiomatic attribution for deep networks.  In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3319–3328. JMLR. org, 2017.
  * [29] I. Sutskever, J. Martens, G. Dahl, and G. Hinton.  On the importance of initialization and momentum in deep learning.  In International Conference on Machine Learning, pages 1139–1147, 2013.
  * [30] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi.  Inception-v4, Inception-ResNet and the impact of residual connections on learning.  In Thirty-First AAAI Conference on Artificial Intelligence, 2017\.
  * [31] S. A. Taghanaki, K. Abhishek, and G. Hamarneh.  Improved inference via deep input transfer.  In D. Shen, T. Liu, T. M. Peters, L. H. Staib, C. Essert, S. Zhou, P.-T. Yap, and A. Khan, editors, Medical Image Computing and Computer Assisted Intervention – MICCAI 2019, pages 819–827, Cham, 2019. Springer International Publishing.
  * [32] Y. Yan, J. Kawahara, and G. Hamarneh.  Melanoma recognition via visual attention.  In International Conference on Information Processing in Medical Imaging, pages 793–804. Springer, 2019.
  * [33] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz.  mixup: Beyond empirical risk minimization.  arXiv preprint arXiv:1710.09412, 2017.
  * [34] M. Zhao and G. Hamarneh.  Retinal image classification via vasculature-guided sequential attention.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [35] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le.  Learning transferable architectures for scalable image recognition.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 8697–8710, 2018.


#  Towards Zero-trust Security for the Metaverse

Ruizhi Cheng, Songqing Chen, and Bo Han  George Mason University
Email<EMAIL_ADDRESS>
###### Abstract

By focusing on immersive interaction among users, the burgeoning Metaverse can
be viewed as a natural extension of existing social media. Similar to
traditional online social networks, there are numerous security and privacy
issues in the Metaverse (_e.g.,_ attacks on user authentication and
impersonation). In this paper, we develop a holistic research agenda for zero-
trust user authentication in social virtual reality (VR), an early prototype
of the Metaverse. Our proposed research includes four concrete steps:
investigating biometrics-based authentication that is suitable for
continuously authenticating VR users, leveraging federated learning (FL) for
protecting user privacy in biometric data, improving the accuracy of
continuous VR authentication with multimodal data, and boosting the usability
of zero-trust security with adaptive VR authentication. Our preliminary study
demonstrates that conventional FL algorithms are not well suited for
biometrics-based authentication of VR users, leading to an accuracy of less
than 10%. We discuss the root cause of this problem, the associated open
challenges, and several future directions for realizing our research vision.

††publicationid: pubid: ©2023 IEEE

##  I Introduction

Metaverse, with the combination of the prefix “meta” (meaning transcending)
and the word “universe”, has been deemed as a hypothetical next-generation
Internet [5]. While there is no consensus on the definition, a narrow
depiction of the Metaverse is a universal virtual world for social
interaction, by connecting multiple 3D virtual environments via the Internet.
In 2021, Facebook – now Meta – planned to invest at least $10 billion in its
extended reality (XR) and Metaverse business for creating novel XR hardware,
software, and content.

Undoubtedly, there will be numerous security and privacy issues in the
Metaverse, for example, known attacks on user authentication and impersonation
[26]. Also, there would be new types of challenges, for instance, defending
against immersive attacks [3] that adjust the location and orientation of
users, which may cause collisions with real objects and motion sickness.
Hence, we should take security and privacy into consideration from day one
when designing the Metaverse instead of treating them as an afterthought.

In this paper, we focus on user authentication in the Metaverse and argue that
it should satisfy the following four key design principles: zero trust, non-
intrusive interaction, high reliability, and privacy preservation. Zero trust
[27] assumes that no entities (_e.g.,_ users, transactions, or network
traffic) should be always trusted unless verified, as trust itself is a
vulnerability. Thus, it requires pervasive and continuous deployment of
security mechanisms such as authentication and access control [9].

Figure 1: System architecture of $\sf\small{MetaGuard}$.

While one-shot authentication on head-mounted displays (HMDs) [22] and
continuous authentication for mobile devices [9] have been extensively
investigated, it is still unclear how to systematically leverage continuous
authentication in an immersive environment for effectively supporting zero-
trust security. Existing studies bear the following limitations. First, they
often utilize a single data source (_e.g.,_ head movement [2]), which may not
always be available for continuous authentication and usually fails to
guarantee a high accuracy (_i.e.,_ only around 90% [2]). Second, they all
benefit from collecting biometric information from users, which is highly
sensitive and may potentially lead to severe security and privacy issues.
Third, they rely on intrusive methods (_e.g.,_ throwing a ball [22]) to
authenticate users, making them not suitable for continuous authentication.
Lastly, they do not consider the practical challenges of continuous
authentication on HMDs (_e.g.,_ usability and resource utilization).

To address these issues, we explore the emerging security threats and privacy
concerns in social virtual reality (VR), an early prototype of the Metaverse
[6], and develop a holistic research agenda to protect it with zero-trust
authentication mechanisms. In particular, we propose a first-of-its-kind,
privacy-preserving continuous and adaptive authentication framework, dubbed
$\sf\small{MetaGuard}$ (Figure 1), by employing a federated-learning (FL)
based scheme that authenticates users with multimodal biometric data (§IV). We
build a proof-of-concept of $\sf\small{MetaGuard}$, focusing on preserving
user privacy with FL. Our preliminary study reveals that existing FL models
such as FedAvg [21] lead to extremely poor performance since each client holds
only positive-label data (§V). We discuss the root cause of the issues related
to this setup and suggest several potential directions for future improvements
(§VI).

##  II Background

Zero-trust Security. Zero trust [27] is a fine-grained security approach that
shifts defenses from static perimeters of a protected system to its users,
resources, and assets. It is rooted in the principle of “never trust, always
verify” [27] and eliminates vulnerable permissions and unnecessary access to
help service providers better manage and protect identities, applications, and
machines across their networks. Zero trust requires the system to ensure that
all requests are continuously verified before accessing any system asset
(_e.g.,_ continuous user authentication [9]).

Metaverse. Metaverse could be considered as a large-scale virtual world
connected via the Internet. In recent years, with the rapid development of
mobile immersive computing [12], providing immersive experiences for users via
VR headsets has become one of the most crucial goals of the Metaverse [5].
Therefore, emerging social VR is regarded as a key component of the Metaverse
[6]. In social VR, users are free to explore the virtual scene and socialize
with others, such as trading virtual content via the non-fungible token (NFT)
[6]. Thus, a continuous authentication method is required to protect the
account and asset security of social VR users.

Figure 2: Biometric-based authentication process in VR.

VR Authentication.  There are two main approaches for user authentication in
VR: knowledge-based and biometric-based. Since users wear HMDs in VR,
knowledge-based methods, such as PINs, become inconvenient and vulnerable to
attacks [22]. Therefore, biometric-based authentication has become a more
practical option for authenticating VR users [28]. Biometric-based
authentication in VR often involves training a classification model using
machine learning (ML) or deep learning (DL), as shown in Figure 2. However,
traditional ML/DL models require clients to upload raw data to a server, which
raises privacy concerns. To address this issue, utilizing FL [30] for VR user
authentication is a promising solution.

##  III Threat Models

In this section, we describe the threats in VR authentication, demonstrating
the necessity of continuous biometric-based authentication, which also serve
as threat models for $\sf\small{MetaGuard}$.

Shoulder Surfing Attack refers to a scenario in which an attacker can visually
observe a user’s interactions with a device during authentication. This attack
is particularly concerning in VR authentication, as users are fully immersed
in the virtual environment and may be unaware of their surroundings, making
them vulnerable to shoulder-surfing attackers [28]. Unlike knowledge-based
authentication methods such as PINs, biometric-based authentication methods
could be less susceptible to shoulder-surfing attacks, as they do not lead to
direct exposure of the authentication information.

Insider Attack refers to an individual who has been granted trust, such as a
friend or family member, exploits the absence of a device owner to gain
unauthorized access to the device or account. Such attacks pose a significant
threat to the security of VR authentication systems, as the insider has
already been granted a certain level of access, which they can use to commit
account takeover or make unauthorized transactions. To defend against this
type of attack, a continuous authentication mechanism that can detect and lock
the system as soon as the legitimate user is not wearing the device is
required [33].

Human Mimicry Attack is a form of security threat in which an attacker records
a legitimate authentication and subsequently replays it to gain unauthorized
access to a device or account. In the context of VR authentication, an
attacker could record a user’s interactions with the device, such as body
movements, and then mimic the recorded movements to impersonate the user and
gain access to the device or account. This type of attack is difficult to
detect and prevent, especially in the one-shot authentication [4], as the
authentication information used in the replay may appear to be coming from the
actual user. Continuous authentication, with its capability to continuously
monitor the user’s behavior and detect anomalies, can mitigate such attacks by
providing an additional layer of security to authentication.

Man-in-the-Middle (MitM) Attack occurs when an attacker intercepts and
manipulates the communication between the user and the authentication system.
When training the FL-based authentication model, the attackers can intercept
the model sent by the server, train the model using their own biometric data,
and then upload the updated model to the server. This tampering can result in
a model that fails to accurately recognize the legitimate user, which enables
the attacker to impersonate the user and gain unauthorized access to the
device or account. Thus, one-shot authentication faces a huge risk under this
attack. Even continuous authentication alone does not guarantee security since
the attackers can continuously use their data to train the model. Hence, it is
imperative to design an additional layer of protection in the continuous
authentication scheme to defend against this attack.

Data Poisoning Attack is a malicious tactic used by attackers to upload
counterfeit data to a ML model with the aim of compromising its performance.
In ML-based classification tasks, such as user authentication, this attack
involves the injection of incorrect data for certain labels (users), making
them unable to be successfully recognized and classified. In the context of
FL-based authentication systems, if the counterfeit data is irrelevant to
other labels, this attack has a limited impact on the authentication
performance of others. However, if the counterfeit data resembles that of
other users, such an attack can be considered a variant of the human mimicry
attack.

##  IV Research Agenda

In this section, we propose a holistic research agenda to empower zero-trust
authentication for the Metaverse with four concrete steps by presenting the
design of $\sf\small{MetaGuard}$.

###  IV-A Benchmarking Biometrics-based Authentication for Metaverse

Early research on authenticating mobile devices [9] and HMDs [2] suggested
that collecting physiological (_e.g.,_ iris, face, fingerprint, voice, and
brain wave) or behavioral (_e.g.,_ typing, touch, hand gesture, and gait)
biometrics from users to build the ML model would be an effective way to
authenticate them. While biometrics-based authentication has been extensively
explored, it is still unclear how to effectively adapt it for
$\sf\small{MetaGuard}$ to offer privacy-preserving zero-trust security
mechanisms.

To bridge this gap, we plan to systematically investigate different biometric
modalities in terms of their usability, reliability, vulnerability,
collectability, sensitivity, and adaptability. For example, continuous
authentication in $\sf\small{MetaGuard}$ should be non-intrusive, which
excludes fingerprint as a candidate, although it is more reliable and accurate
than behavioral biometrics such as gait and hand gestures. Also, behavioral
biometrics such as head and body motion may be more vulnerable to human
mimicry attack than others [22], whereas physiological biometrics such as face
and voice are more sensitive to the environment [9] and the collection of
brain waves is still a challenging task. Moreover, continuously collecting and
processing biometrics, such as gaze movement, may consume more system
resources than others. To this end, we plan to conduct a large-scale user
study by designing a VR environment, where participants will engage in various
common activities such as walking, reading, and playing games. While these
activities are taking place, we will collect various biometric data, such as
body/gaze movements and pupil size, which have been demonstrated to be
appropriate for user authentication in VR [28]. By conducting this study, we
aim to gain a deep understanding of the efficacy of different biometric
modalities for authentication, and to create an open, diverse, and
comprehensive dataset for the design and evaluation of $\sf\small{MetaGuard}$.

Figure 3: A schematic diagram of training FL-based user authentication model
in VR.

###  IV-B Leveraging Federated Learning to Protect User Privacy

Biometrics-based authentication poses severe privacy concerns to users if such
a scheme is blindly employed in the Metaverse, as the collected data often
carries sensitive and private information. If biometric data is misused,
leaked, or stolen, users could be endangered. To address this concern, we will
resort to the recent advances of FL [30]. As shown in Figure 3, when training
the FL model, the sensitive user data will not be uploaded to a central place
(_e.g.,_ a cloud or edge server). Only the updated gradient of local models
will be uploaded to the server for updating the global model, which will, in
turn, be used to improve local models. This would effectively mitigate the
potential risks and protect user privacy.

To train a user authentication model with FL, an open challenge is that each
client holds only positive-label data, which may lead to scalability issues of
the model. ML-based user authentication is essentially a binary decision
problem. During the training process, the loss function tends to increase the
similarity between class embeddings with the same label ($L_{pos}$) and
minimize the similarity between those with different labels ($L_{neg}$). While
in the FL-based authentication scenario, since each client has only its own
data (_i.e.,_ positive label), the local model cannot optimize $L_{neg}$.
Optimizing only $L_{pos}$ will lead to a trivial solution where all class
embeddings collapse into a single point in the feature space [15]. Hence, the
model cannot separate feature embeddings among different users, leading the
authentication accuracy to decrease as the number of users increases (§V). To
address this problem, we propose to utilize the time-series feature of
biometric data and variational recurrent neural network (VRNN) [7], as well as
contextual information to design the FL-based authentication model (§VI-B).

###  IV-C Improving Zero-trust Authentication Accuracy with Personalized
Multimodal Biometric Data

The Metaverse necessitates a reliable zero-trust security framework with
highly accurate authentication schemes. Given the increasing stake in the
virtual world, conducting traditional authentication methods (_e.g.,_ password
or two-factor authentication) only once at the beginning of the session is no
longer sufficient for the Metaverse. With more users joining the Metaverse,
the uniqueness of the authentication features for each user is reducing,
making it crucial to continuously collect and train the authentication model.
Moreover, as we discuss in §IV-A, some biometric information, such as voice,
has a dynamic nature that can change over time. Thus, one-shot authentication
may not be robust under different environments.

We point out that a naive continuous authentication approach, which monitors
when the headset is taken off from the user’s head and performs authentication
each time the headset is put on, is unreliable. This is because it requires
additional sensors to monitor device usage. Even though it has been integrated
into the commercial VR headset, such as Oculus Quest 2, it can be disabled by
software-level attacks111https://bit.ly/3YUUGjt [accessed on 14-February-2022]
or physically covered, for example, using tape222We have confirmed that using
tape to cover the Oculus Quest 2’s proximity sensor can make it unable to
recognize if we take off the headset.. Additionally, conducting authentication
only after the device is put on is vulnerable to the MitM attack, as it
presents a noticeable pattern for the attacker to recognize and intercept the
authentication model. Thus, we need to design a fine-grained continuous
authentication mechanism in the Metaverse.

While prior research has proposed continuous authentication [9], a major
caveat of these strategies is that, as we mentioned in §I, they often rely on
a single trait of users. However, depending solely on a single biometric
modality for authentication in VR may lead to sub-optimal performance [22] and
be vulnerable to human mimicry attacks [28]. Moreover, the free-interaction
nature of the Metaverse presents challenges to the effectiveness of single-
modality authentication. For example, a user may intermittently walk and sit,
leaving a scheme that solely relies on users’ gait powerless when they do not
walk. Similarly, an authentication scheme monitoring users’ typing would leave
a loophole when they do not behave as expected.

To this end, we will design a multimodal FL model where multiple modalities of
biometric data (_e.g.,_ pupil size, head motion, body motion, voice, _etc._)
are collected and fused together for authentication. Moreover, we plan to
propose a personalized FL model [19] for each user with a unique combination
of modalities that reflects their unique biometric features. This approach
will not only improve authentication accuracy but also effectively defend
against the MitM attack. This is because users’ multimodal biometric data will
present unique features. Even if MitM attackers also perform the human mimicry
attack, our proposed model is still capable of defending against it since
precisely mimicking multiple biometric information is challenging.
Furthermore, some biometric data, such as gaze movement, is hard to be
mimicked since the user’s eyes are covered by the VR headset. However,
integrating multiple biometric modalities, particularly when utilizing body
movement data, into authentication systems may lead to noise and a decrease in
authentication accuracy (§V). To address this challenge, we propose a density-
based within-client modality selection method that chooses the best modality
combination for each client (§VI-B).

###  IV-D Boosting Usability of Zero-trust Security with Adaptive
Authentication

The Metaverse requires a non-intrusive and easy-to-use authentication that
will not impact users’ experience while protecting their privacy. Our proposed
FL-based framework with multimodal biometric data as input is expected to
provide strong protection to end-users. However, HMDs are not well designed to
handle computation-intensive tasks such as FL training and continuous data
collection, processing, fusion, and inference, which could inadvertently
impact user experience, for example, quickly depleting the battery.

To this end, we will optimize $\sf\small{MetaGuard}$ with adaptive and dynamic
authentication. For example, a coarse-grain data collection and authentication
frequency can be used when users’ behavior does not change. Otherwise,
intensive authentications will be triggered. Also, we plan to investigate
contextual information that $\sf\small{MetaGuard}$ can utilize to effectively
reduce authentication costs. An inherent and practical challenge of adaptive
authentication is to balance the trade-off between reliability, usability, and
resource consumption. For example, in traditional FL algorithms such as FedAvg
[21], the selection weights of all clients are the same by default. In each
communication round, the server randomly selects clients to participate in the
training. However, clients may present heterogeneous resource and data issues,
affecting the training efficiency and model performance [8]. To address this
issue, we need to design an adaptive client-selection algorithm that chooses
the appropriate clients to jointly train the global model in each round. A
unique challenge of $\sf\small{MetaGuard}$ is that we cannot simply drop a
client for any reason, as everyone should be continuously authenticated in the
Metaverse.

##  V Preliminary Results

To better understand the technical challenges of $\sf\small{MetaGuard}$, we
build a proof-of-concept that partially implements our proposed research in
§IV-B and §IV-C. In this section, we present our preliminary results of FL-
based VR authentication that shed light on realizing the fully-fledged
$\sf\small{MetaGuard}$.

Dataset. We use the dataset released by Miller  _et al._ [22] for our initial
experiments. This dataset was collected with an application of throwing a ball
in VR. It contains the trajectories of 41 users throwing the ball using the
Oculus Quest VR headset and has 6 modalities, including the position and
orientation of the headset and both controllers. Each user threw the ball 10
times per day for two days. We use the first-day trajectories for enrollment
(training set) and the second-day trajectories for authentication (test set).
Note that leveraging ball-throwing for authentication is intrusive and may not
be suitable for continuous authentication. However, given that this is the
largest publicly available biometrics-based VR authentication dataset, in
terms of the number of users, we believe it can help us gain some initial
insights into designing privacy-preserving continuous authentication for the
Metaverse.

Implementation Details. We implement three well-known deep learning (DL)
models that have been demonstrated to achieve high accuracy in VR
authentication tasks [22]: siamese neural network, FCN, and ResNet. FCN and
ResNet are based on a benchmark of DL-based classification [16]. Both models
have a similar architecture, consisting of three convolutional layers and
ending with a dense layer using softmax as the activation function. The output
of these two models is a set of $N$ scores, where $N$ is the number of users,
representing the probability of each candidate being the authenticated user.
The candidate with the highest score is then identified as the authenticated
user. The siamese network has the same limb as the FCN but does not have a
dense layer. It receives two pieces of data as input (_e.g.,_ enrollment and
authentication motion trajectories), calculates their Euclidean distance, and
outputs the distance. The user ID of the enrollment trajectory with the
closest distance to the authentication trajectory is the identified ID. The
server trains the above three models with all users’ data (_i.e.,_
trajectories of headset and controllers), compromising their privacy.

Next, we set up and train the FL model that ensures each client utilizes only
its own data and does not share it with others, including the server. We
leverage a widely used FL algorithm – FedAvg [21] – to enable the server and
clients to jointly train the model. The process of training the FL model via
the FedAvg algorithm is as follows:

  1. (a)

In the $k^{th}$ communication round, the server sends the class embedding
$W_{k}$ and model parameters $\theta_{k}$ of the global model to all clients.

  2. (b)

The $t^{th}$ client updates ($W_{k}$, $\theta_{k}$) to ($W_{k}^{t}$,
$\theta_{k}^{t}$) based on its local data and the loss function.

  3. (c)

The server receives the updated ($W_{k}^{t}$, $\theta_{k}^{t}$) from all
clients and updates the ($W_{k+1}$, $\theta_{k+1}$) of the global model by
taking a weighted average of ($W_{k}^{t}$, $\theta_{k}^{t}$).

  4. (d)

The server transmits the ($W_{k+1}$, $\theta_{t+1}$) to all clients and keeps
repeating steps (b)– (d) until the global model is converged.

Based on the above algorithm, the authentication model would not leak user
privacy. We use the FCN as the network architecture of the global model for
FL, namely FedAvg + FCN. We use the Adam optimizer with a learning rate of
0.001 for training. For the three non-privacy-preserving models, each training
runs 2,000 epochs. For the FedAvg + FCN model, we train it with 100
communication rounds and let all clients participate in the training during
each round. For each training, we save the model with the best authentication
accuracy in the test set. We train each model five times and report the
average accuracy.

Model | Privacy Preservation | Accuracy
---|---|---
Siamese Network | ✗ | 90.2%
FCN | ✗ | 89.3%
ResNet | ✗ | 87.2%
FedAvg + FCN | ✓ | 6.34%
TABLE I: Accuracy of different authentication models.

Results. Table I shows the accuracy of different models when using all 41
users’ trajectories with all 6 modalities, which leads to the following two
observations. First, on this intrusive authentication dataset, the accuracy of
the best non-privacy-preserving model, the siamese network, is only $\sim$90%.
However, as we discussed in §IV-A, to perform zero-trust continuous
authentication, $\sf\small{MetaGuard}$ must utilize non-intrusive approaches,
whose accuracy may be even lower than the intrusive solutions. For example,
Pfeuffer  _et al._ [25] leveraged users’ gait movement to perform
authentication, and the accuracy is only $\sim$50%. One possible reason is
that it is more difficult to differentiate the features of user data in non-
intrusive methods, which introduces a key challenge: How to improve the
accuracy of continuous authentication based on non-intrusive methods to ensure
the effectiveness of $\sf\small{MetaGuard}$?

Figure 4: Impact of the number of clients on the FedAvg + FCN and FCN models.

Second, the FedAvg + FCN method can achieve an accuracy of only 6.34%, while
all non-privacy-preserving models have higher than 87% accuracy. The reason
may be that FedAvg + FCN cannot distinguish the features of most users,
resulting in a low authentication accuracy when the number of users is large.
To understand the impact of the number of users on the authentication
performance of FedAvg + FCN, we train the FCN and FedAvg + FCN models with
different numbers of users using all 6 modalities and show the result in
Figure 4. As we can see, the accuracy of the FCN model only slightly drops
when the number of users increases. However, FedAvg + FCN is significantly
affected by the number of users. When there are only two users, it can achieve
95% accuracy. However, when the number of users increases to 10, the accuracy
reduces to $\textless$30%; when the number of users further increases to 25,
the accuracy is under 10%. Given that the Metaverse will accommodate tens of
thousands of users from all over the world [5], the user authentication model
must be scalable and robust. Thus, it leads to another key challenge for
$\sf\small{MetaGuard}$: How to ensure the accuracy of $\sf\small{MetaGuard}$
remains stable when the number of users increases?

Next, in order to understand the impact of the number of modalities on
authentication performance, we train the FCN and FedAvg + FCN models with
different modality combinations. Among the 6 modalities, there are a total of
$\binom{6}{1}+\cdots+\binom{6}{6}=63$ modality combinations. The result is
shown in Figure 5. We observe that the accuracy of FCN is stable with
different numbesr of modalities. However, using fewer modalities may result in
higher accuracy for the FedAvg + FCN model. For example, when using all 6
modalities, the accuracy is merely 6.34%. However, when using only 3
modalities, the average accuracy is 19.32%. Moreover, the best modality
combination may be different for different users. During the training phase,
we record the modality combinations with the highest local accuracy for each
of the 41 users and observe that there are 30 unique modality combinations
among the recordings. These findings demonstrate the importance of
personalized modality selection for improving the performance of FL-based
authentication models in VR.

Figure 5: The average accuracy of FedAvg + FCN and FCN models using different
numbers of modalities for 41 users. The bands represent 95% confidence
intervals. For each number of modalities $x$, it has $\binom{6}{x}$ modality
combinations.

##  VI Discussion and Future Directions

###  VI-A Discussion

The preliminary results validate our identified challenges in §IV-B and
demonstrate that the FL-based authentication model may not scale due to the
fact that each user holds only positive label data. Next, we discuss the
solutions to this problem proposed in existing works and why they do not apply
to VR authentication. For example, recent work proposed to address this issue
by improving the model update process. FedAwS [31] is a method where the
server, in addition to averaging the uploaded gradients, performs a geometric
regularization to ensure users’ embeddings are separated by a pre-defined
margin. FedUV [15] uses an error-correction code to generate a unique secret
vector for each user to update the model. However, these models do not
consider the quality of local data (_i.e.,_ each client simply uses all of its
local data for training). This can be problematic, as low-quality local data,
such as noise, can negatively affect the accuracy of the trained model.

Other recent work on face recognition reported an accuracy comparable to the
state-of-the-art non-privacy-preserving deep learning models. For example,
FedFace [1] proposes to first deploy a feature extractor on the server that
has been pre-trained on a large-scale dataset. This feature extractor is then
used to initiate the class embedding for each client to ensure that it lies
inside the generated space of the feature extractor. FedFR [20] is motivated
by FedFace and introduces extra data created from the public datasets that
have low similarity to the data held by the client for computing its
$L_{neg}$. However, both solutions have the prerequisite of large-scale
datasets. This is feasible for face recognition because there are numerous
public large-scale datasets available, such as the
DigiFace1M333https://github.com/microsoft/DigiFace1M [accessed on
14-February-2022], which has 1.2M images obtained from 110K individuals. On
the other hand, biometric datasets, especially for VR headsets, are limited in
their scale. For example, as we introduced before, the dataset released by
Miller  _et al._ [22], which is the largest among the public biometrics-based
VR authentication datasets, has only 41 users.

The root cause for this disparity in size between the face recognition and
biometrics-based authentication datasets is that face images are relatively
easy to collect, requiring only devices with cameras, and many people
voluntarily share their photos on the Internet. Whereas collecting users’
biometric data on HMDs requires additional setup for sensors, and users may be
reluctant to share biometric data because it could contain sensitive
information. Therefore, it is probably unrealistic to expect the publicly
available biometric dataset for VR authentication to be as large as the face
recognition datasets. Nevertheless, the size of the dataset is directly
related to the accuracy of the trained models. Moreover, as discussed in IV-C,
capturing the user’s whole face poses challenges in VR since the headset
encloses half of it. Hence, it may not be feasible to apply the methods for
face recognition directly to VR authentication.

###  VI-B Future Directions

To improve the performance of the authentication model while preserving user
privacy, we first propose a density-based within-client modality selection
algorithm. Our approach is motivated by the observation that the optimal
modality combinations may differ among users (§V) and that modalities
beneficial for authentication are typically well-clustered [23]. To select the
optimal modality combination for each user, we plan to leverage the density-
based clustering algorithm, such as mean shift and OPTICS [29], for reducing
noise and calculating the density for each modality combination. First, after
applying the clustering algorithm on each modality combination, the clustered
group with only a single point will be considered as noise and removed.
Second, by calculating the distance between the centroids of each clustered
group and the center of all data, we will obtain the density of each modality
combination. In this manner, each client can select the modality combination
with the highest density for training the authentication model.

Next, we exploit the unique features of the biometric data collected on VR
headsets to improve the scalability of $\sf\small{MetaGuard}$. We notice an
important fact that since all data are captured from the continuous activities
of the user, they may present time-series features, such as lag features and
rolling window features [11], which are not available in other authentication
methods, for example, those that benefit from face recognition. To leverage
the time-series features of biometric data on VR headsets, we can perform
time-series analysis (TSA) [11] to extract data with salient features for
authentication. For example, by considering the excellent performance of the
recurrent neural network (RNN), such as long short-term memory (LSTM) [14], in
time-series classification tasks, we can train the RNN-based model to improve
the accuracy of non-intrusive authentication.

In particular, we plan to design an LSTM-based FL model for preserving privacy
in VR authentication. In contrast to FCN, training LSTM on the client is less
subjective to the negative impact of having only positive labels, mitigating
the scalability issue, as shown in Figure 4. The reason is that when updating
the gradient of the loss function, LSTM considers only the current time-series
data instead of comparing it with other data. However, since users still have
only their own time-series data, if we use the traditional softmax loss-based
training approach, this may lead to the model overconfidence problem [10].
Assuming that there are $N$ users, in the softmax loss-based training model,
the last layer of the model is a dense layer with $N$ units, and the output of
the model is the recognition probability of these users. With this training
model, the user with the highest probability is considered the recognized
user. However, since each user in our case does not have other users’ data,
the local model updated by a given user does not have any recognition
capability of others, making the output of the other $N-1$ probabilities
meaningless. In this secenairo, model’s false positive rate (FPR) will
increase as $N$, leading to potential scalability issues [10].

To address this problem, we can resort to the VRNN model [7] for privacy-
preserving authentication. VRNN integrates the variational autoencoder (VAE)
[18] model into the RNN model. In VRNN, RNN is the encoder, VAE samples the
output of RNN to reconstruct the time series similar to the input data, and
the loss function is the reconstruction error. VAE is based on the assumption
that it can generate a sample with high similarity to the input data if its
label has been trained before. Conversely, if the model cannot generate a
sufficiently similar sample, this input data is considered anomalous. Hence,
when training VRNN in the FL setting, each user-trained model focuses on
minimizing the error of reconstructing its time series. After training, the
converged VRNN model will be able to reconstruct the time series of all users
involved in the training. In the authentication stage, the model will make a
judgment based on the reconstruction error. It will consider the
authentication successful when the reconstruction error is lower than the pre-
defined threshold, instead of directly using the aforementioned probability.
As a result, this approach increases the confidence level and stability of the
authentication model.

To further improve the scalability of our system, we plan to utilize
contextual information in VR to conduct authentication. For example, there is
a large number of user-generated content (UGC) in the Metaverse. Thus, we can
design a fine-grained data processing method, which categorizes time-series
data based on recent user interactions with the UGC. During authentication,
the user will be verified using the historical information of their
interactions with the UGC. Given that users may exhibit distinct patterns in
their interactions with different UGC, this approach can help the model learn
more useful representations of the user’s behavior by considering mainly the
most relevant historical data (_i.e.,_ the interaction with the UGC),
enhancing the authentication performance.

We finally discuss three potential directions for further improving the
practicality of $\sf\small{MetaGuard}$. First, training the VRNN-based FL
model on VR headsets may consume substantial resources [13]. As we discussed
in §IV-D, clients with insufficient available resources may affect the
convergence speed of the global model, which motivates us to design a client-
selection algorithm to accelerate the convergence of the model and reduce the
communication and computation overhead, without sacrificing the authentication
accuracy of $\sf\small{MetaGuard}$. Second, continuous authentication
necessitates non-intrusive data collection. However, given that the data
collected in this way may exhibit random patterns, especially for behavioral-
based biometrics, training the authentication model using raw data may result
in poor authentication performance [25]. Applying signal processing
algorithms, such as fast Fourier transform (FFT), to extract features from the
raw data and use them to train the authentication model can potentially
improve authentication performance [24]. Third, although FL can prevent the
exposure of raw user data during the learning process, the model updates
communicated in $\sf\small{MetaGuard}$ may still reveal some user information
[17]. In the future, we will integrate other privacy-preserving techniques,
such as homomorphic encryption [32], in $\sf\small{MetaGuard}$, and
investigate the trade-off between privacy and utility.

##  VII Conclusion

In this position paper, we presented a holistic research agenda for securing
the Metaverse through a zero-trust continuous authentication framework. We
first analyzed the challenges of conducting privacy-preserving continuous
authentication in the context of social VR (_i.e.,_ a prototype of the
Metaverse). We then proposed $\sf\small{MetaGuard}$, a first-of-its-kind, FL-
based adaptive and continuous authentication framework for VR users by
leveraging multimodal biometric data. Our preliminary study through a proof-
of-concept of $\sf\small{MetaGuard}$ revealed that blindly applying FL to VR
authentication will lead to an accuracy of lower than 10%. We finally
discussed the root cause of these problems, which shed light on future
improvements. We hope our study can inspire more research to realize the grand
vision of the Metaverse by tackling its security and privacy challenges.

## References

  * [1] D. Aggarwal, J. Zhou, and A. K. Jain.  FedFace: Collaborative Learning of Face Recognition Model.  In Proceedings of IEEE International Joint Conference on Biometrics (IJCB), 2021.
  * [2] A. Bhalla, I. Sluganovic, K. Krawiecka, and I. Martinovic.  MoveAR: Continuous Biometric Authentication for Augmented Reality Headsets.  In Proceedings of ACM Cyber-Physical System Security Workshop (CPSS), 2021.
  * [3] P. Casey, I. Baggili, and A. Yarramreddy.  Immersive Virtual Reality Attacks and the Human Joystick.  IEEE Transactions on Dependable and Secure Computing, 18(2):550–562, 2019.
  * [4] Y. Chen, Z. Yang, R. Abbou, P. Lopes, B. Y. Zhao, and H. Zheng.  User authentication via electrical muscle stimulation.  In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI), 2021.
  * [5] R. Cheng, N. Wu, S. Chen, and B. Han.  Will Metaverse be NextG Internet? Vision, Hype, and Reality.  IEEE Network, 36(5):197 – 204, 2022.
  * [6] R. Cheng, N. Wu, M. Varvello, S. Chen, and B. Han.  Are We Ready for Metaverse? A Measurement Study of Social Virtual Reality Platforms.  In Proceedings of ACM IMC, 2022.
  * [7] J. Chung, K. Kastner, L. Dinh, K. Goel, A. C. Courville, and Y. Bengio.  A Recurrent Latent Variable Model for Sequential Data.  In Proceedings of Conference on Neural Information Processing Systems (NIPS), 2015.
  * [8] E. Diao, J. Ding, and V. Tarokh.  HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients.  In Proceedings of ICLR, 2021.
  * [9] H. Feng, K. Fawaz, and K. G. Shin.  Continuous Authentication for Voice Assistants.  In Proceedings of ACM MobiCom, 2017.
  * [10] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger.  On Calibration of Modern Neural Networks.  In Proceedings of International Conference on Machine Learning (ICML), 2017.
  * [11] J. D. Hamilton.  Time Series Analysis.  Princeton University Press, 2020.
  * [12] B. Han.  Mobile Immersive Computing: Research Challenges and the Road Ahead.  IEEE Communications Magazine, 57(10):112–118, 2019.
  * [13] A. Hard, K. Rao, R. Mathews, S. Ramaswamy, F. Beaufays, S. Augenstein, H. Eichner, C. Kiddon, and D. Ramage.  Federated Learning for Mobile Keyboard Prediction.  https://arxiv.org/pdf/1811.03604.pdf, 2018.  [accessed on 14-February-2022].
  * [14] S. Hochreiter and J. Schmidhuber.  Long Short-Term Memory.  Neural Computation, 9(8):1735–1780, 1997.
  * [15] H. Hosseini, H. Park, S. Yun, C. Louizos, J. Soriaga, and M. Welling.  Federated Learning of User Verification Models Without Sharing Embeddings.  In Proceedings of ICML, 2021.
  * [16] H. Ismail Fawaz, G. Forestier, J. Weber, L. Idoumghar, and P.-A. Muller.  Deep Learning for Time Series Classification: a Review.  Data mining and knowledge discovery, 33(4):917–963, 2019.
  * [17] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, et al.  Advances and Open Problems in Federated Learning.  Foundations and Trends® in Machine Learning, 14(1–2):1–210, 2021.
  * [18] D. P. Kingma and M. Welling.  An introduction to variational autoencoders.  Foundations and Trends® in Machine Learning, 12(4):307–392, 2019.
  * [19] T. Li, A. K. Sahu, A. Talwalkar, and V. Smith.  Federated Learning: Challenges, Methods, and Future Directions.  IEEE Signal Processing Magazine, 37(3):50–60, 2020.
  * [20] C.-T. Liu, C.-Y. Wang, S.-Y. Chien, and S.-H. Lai.  FedFR: Joint Optimization Federated Framework for Generic and Personalized Face Recognition.  In Proceedings of AAAI, 2022.
  * [21] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas.  Communication-Efficient Learning of Deep Networks from Decentralized Data.  In Proceedings of International Conference on Artificial Intelligence and Statistics (AISTATS), 2017.
  * [22] R. Miller, N. K. Banerjee, and S. Banerjee.  Using Siamese Neural Networks to Perform Cross-System Behavioral Authentication in Virtual Reality.  In Proceedings of IEEE VR, 2021.
  * [23] P. Oza and V. M. Patel.  Federated Learning-based Active Authentication on Mobile Devices.  In Proceedings of IEEE International Joint Conference on Biometrics (IJCB), 2021.
  * [24] V. M. Patel, R. Chellappa, D. Chandra, and B. Barbello.  Continuous User Authentication on Mobile Devices: Recent progress and remaining challenges.  IEEE Signal Processing Magazine, 33(4):49–61, 2016.
  * [25] K. Pfeuffer, M. J. Geiger, S. Prange, L. Mecke, D. Buschek, and F. Alt.  Behavioural Biometrics in VR: Identifying People from Body Motion and Relations in Virtual Reality.  In Proceedings of ACM CHI, 2019.
  * [26] R. D. Pietro and S. Cresci.  Metaverse: Security and Privacy Issues.  In Proceedings of IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA), 2021\.
  * [27] S. Rose, O. Borchert, S. Mitchell, and S. Connelly.  Zero Trust Architecture (NIST Special Publication 800-207).  https://doi.org/10.6028/NIST.SP.800-207, 2020.  [accessed on 14-February-2022].
  * [28] S. Stephenson, B. Pal, S. Fan, E. Fernandes, Y. Zhao, and R. Chatterjee.  Sok: Authentication in Augmented and Virtual Reality.  In Proceedings of IEEE Symposium on Security and Privacy (S&P), pages 267–284, 2022.
  * [29] R. Xu and D. Wunsch.  Survey of Clustering Algorithms.  IEEE Transactions on neural networks, 16(3):645–678, 2005.
  * [30] Q. Yang, Y. Liu, T. Chen, and Y. Tong.  Federated Machine Learning: Concept and Applications.  ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1–19, 2019.
  * [31] F. Yu, A. S. Rawat, A. Menon, and S. Kumar.  Federated Learning with Only Positive Labels.  In Proceedings of ICML, 2020.
  * [32] C. Zhang, S. Li, J. Xia, W. Wang, F. Yan, and Y. Liu.  BatchCrypt: Efficient Homomorphic Encryption for Cross-Silo Federated Learning.  In Proceedings of USENIX Annual Technical Conference (ATC), 2020\.
  * [33] Y. Zhang, W. Hu, W. Xu, C. T. Chou, and J. Hu.  Continuous Authentication Using Eye Movement Response of Implicit Visual Stimuli.  In Proceedings of ACM IMWUT, 2017.