# UserBoost: Generating User-specific Synthetic Data for Faster Enrolment into
Behavioural Biometric Systems

George Webber  0009-0003-2573-7690 King’s College LondonLondonUK
<EMAIL_ADDRESS>,  Jack Sturgess  0000-0001-5708-3052 University of
OxfordOxfordUK<EMAIL_ADDRESS>and  Ivan Martinovic
0000-0003-2340-3040 University of OxfordOxfordUK<EMAIL_ADDRESS>
###### Abstract.

Behavioural biometric authentication systems entail an enrolment period that
is burdensome for the user. In this work, we explore generating synthetic
gestures from a few real user gestures with generative deep learning, with the
application of training a simple (i.e. non-deep-learned) authentication model.
Specifically, we show that utilising synthetic data alongside real data can
reduce the number of real datapoints a user must provide to enrol into a
biometric system. To validate our methods, we use the publicly available
dataset of WatchAuth, a system proposed in 2022 for authenticating smartwatch
payments using the physical gesture of reaching towards a payment terminal. We
develop a regularised autoencoder model for generating synthetic user-specific
wrist motion data representing these physical gestures, and demonstrate the
diversity and fidelity of our synthetic gestures. We show that using synthetic
gestures in training can improve classification ability for a real-world
system. Through this technique we can reduce the number of gestures required
to enrol a user into a WatchAuth-like system by more than 40% without
negatively impacting its error rates.

authentication, synthetic data, mobile payment, smartwatch, tap gesture,
wearable, biometrics, activity recognition, autoencoder

††copyright: none

##  1\. Introduction

Many biometric authentication systems require a burdensome enrolment period
for the user, where they must provide many instances of their biometric
signal. This reduces the usability of the authentication system. Ideally, a
user would be able to enrol into the security system with fewer instances of
their biometric signal.

Generating synthetic data to supplement small, real-world datasets has
potential cost, privacy and speed advantages for many tasks when compared to
collecting large datasets from real-world users. While used widely in such
computer science domains as healthcare technology and image analysis,
synthetic data remains relatively underutilised in security.

In this work, we investigate generating synthetic gesture data for a
smartwatch. A gesture is a pattern of user movements and is measured over time
by the Inertial Measurement Unit (IMU) on the device. We focus on smartwatch
authentication for this work because smartwatch authentication models must be
compact to satisfy hardware requirements. As a result, state-of-the-art deep
learning models that could improve the authentication system are too memory-
intensive to install on the smartwatch. However, given a corpus of user data,
a deep learning model could teach a simpler, smartwatch-compatible
authentication model by generating synthetic gesture data. We investigate the
specific use case for this data in improving the training of a motion-based
authentication system for smartwatch contactless payments.

Making contactless payments using near-field communication (NFC) is
commonplace in today’s society. Wearable devices such as smartwatches are an
increasingly popular medium for utilising contactless payment technology — US
payments made by smartwatch are projected to increase from $\sim$US$40 billion
in 2020 to $\sim$US$500 billion by 2024 (Magnates, 2023). This rise in
adoption drives a need for biometric smartwatch authentication to prevent
fraudulent users making payments. Smartwatches typically lack sophisticated
fingerprint sensors or cameras, but usually include an accelerometer and
gyroscope for motion sensing. In 2022, Sturgess et al. introduced WatchAuth
(Sturgess et al., 2022), a machine learning system for authenticating a user’s
payment based on the physical act of extending the smartwatch to a payment
terminal, a motion we call a payment gesture (see Figure 1).

The attractiveness of smartwatch payments lies mostly in convenience, and so
WatchAuth was conceived to improve the security of a payment without
negatively affecting usability. However, similar to many biometric
authentication systems, it requires a burdensome enrolment phase to become a
useful security measure. In this phase, a user must provide many example
gestures (on the order of 100) to create a unique template, which the system
then compares to future gestures for authentication. Reducing the burden of
this enrolment phase would encourage users to use the additional security on
offer.

We consider the application of generative deep learning to produce synthetic
data with a view to reducing the enrolment burden.

This work makes the following contributions:

  * •

We formulate and present a novel use case for reducing the enrolment burden
for an authentication system through the application of synthetic data.

  * •

We propose an autoencoder-based model that attempts to model the distribution
of payment gestures, and can be used to generate synthetic gestures.

  * •

We show that on a real-world dataset with a limited number of users (n=16),
our system can reduce WatchAuth enrolment effort by more than 40%, without
negatively affecting its security or usability.

  * •

We make available our codebase for training generative models, as well as for
formatting the WatchAuth dataset for deep learning applications at
https://github.com/GeorgeWebber/WatchAuth-DL.

##  2\. Related Work

The general problem of data scarcity for classification tasks on IMU data is
well known, and many attempts to solve it in different contexts with synthetic
data have been made. Kim & Brown (Kim and O’Neill-Brown, 2019) apply pre-
defined transformations to real data to generate additional training data for
the task of American Sign Language recognition from IMU data. Panneer et al.
(Santhalingam et al., 2023) take a different approach, presenting a video-to-
IMU translation framework for converting a large number of publicly available
videos to IMU training data. Alzantot et al. (Alzantot et al., 2017) use deep
learning for generating synthetic timeseries sensor data capable of fooling a
discriminator network, aiming to replace personal data with synthetic data for
increased privacy. However, they do not show the ability to generate user-
specific samples.

In their review of timeseries data augmentation methods, Iglesias et al.
(Iglesias et al., 2023) identify VAEs and generative adversarial networks
(GANs) as the primary methods for deep generative data augmentation.
Particular works of relevance for this paper are Alawneh et al. (Alawneh et
al., 2021), who use VAEs for data augmentation in a human activity recognition
context, and Goubeaud et al. (Goubeaud et al., 2021), who show that data
generated from a VAE improves the performance of random forest classifiers in
contexts unrelated to security.

The related field of Human Activity Recognition (HAR) has explored the
potential of deep-learning generation of synthetic data, and shown that this
synthetic data can augment training of machine learning systems. Zhang et al.
(Zhang et al., 2022) details a number of successful studies using GANs to
augment training datasets for HAR. For example, de Souza et al. (de Souza et
al., 2023) show the feasibility of improving HAR performance from wearable
data. Their study shows that wearable timeseries data can be generated that is
representative, diverse, and accurate, and that training a machine learning
system with this data can improve the quality of HAR classification. Similar
conclusions are reached by other authors using adversarial methods, including
Alharbi et al. (Alharbi et al., 2020) and Konak et al. (Konak et al., 2023).

Zhang et al. also highlights the use case of transfer learning for HAR to
improve performance across different subjects, sensors and sampling rates. In
the most relevant work, Soleimani & Nazerfard (Soleimani and Nazerfard, 2021)
investigate the use of GANs to transfer user-specific characteristics onto
data from a second user, and show that this augmented data can make an
activity classifier more independent of the user. This work showcases a method
for generating user-dependent synthetic data, but does not consider its
usefulness in an authentication context.

##  3\. Background

###  3.1. Motivation

Sturgess et al. (Sturgess et al., 2022) show that a payment gesture, performed
when a user taps a smartwatch against an NFC-enabled payment terminal, is a
biometric that can authenticate a user. The machine learning system they
describe for this purpose is called WatchAuth, and can be run on a smartwatch
without modifying payment terminals or smartwatch hardware.

The system model of WatchAuth assumes that a user is wearing a smartwatch on
their wrist and uses it to make an NFC-enabled payment at a point-of-sale
terminal in a typical setting (for example, a shop or entry barriers to a
transport system). It is further assumed that the user performs a payment
gesture by moving and rotating their wrist towards the terminal until it is
near enough to exchange data via NFC ($<10$ cm), at which point the gesture
ends and payment is either approved or denied. WatchAuth only requires access
to the watch’s inertial sensors. As it is a machine learning system, to
achieve strong authentication performance it is trained on many positive and
negative samples and can’t be employed for user authentication until the user
has exhibited sufficiently many payment gestures (i.e. positive samples),
either naturally over a long time frame or as a burdensome initial enrolment
phase.

A potential solution to the issue of a burdensome enrolment phase would be to
generate user-specific synthetic data and augment the initial dataset with
these synthetic positive instances to train WatchAuth. Smartwatch-based models
are necessarily resource-constrained. This approach offers the benefit of
using an unconstrained source of computing power to generate synthetic
positive training examples, which are then used to train the smartwatch-
compatible WatchAuth model. The synthetic positive examples may themselves be
generated by a machine learning model. The approach offers the possibility of
exploiting commonalities between the target user and a large corpus of seen
users, which could include transferring user-specific characteristics of a
payment gesture to a new position of payment terminal or leveraging
characteristics common to age, height or sex.

###  3.2. WatchAuth Dataset

All experiments in this paper make use of the publicly available WatchAuth
dataset (Sturgess, 2023). This dataset contains data collected in a physical
study with 16 participants. Across three sessions, participants performed
gestures simulating smartwatch payments at seven differently-positioned
contactless payment terminals (see Figure 1), with a watch on their wrist for
data collection.

Figure 1.  Left: A smartwatch user making a payment gesture. Right: The
positioning of six contactless payment terminals used in WatchAuth
experiments; the seventh terminal (centre) was handheld (image courtesy of
(Sturgess et al., 2022)).

Data was sampled at 50 Hz from the watch’s inbuilt sensors, returning vectors
for acceleration (from an accelerometer), angular velocity (from a gyroscope),
linear acceleration (from a linear accelerometer) and device orientation
(derived from other sensors). The dataset includes timestamps showing when
contact between the watch and payment terminal was initiated (via NFC).

In addition to the authentication gesture data collected, the dataset also
contains ”non-gesture” data collected by users outside of the lab while
performing everyday tasks. This data forms a useful negative class for gesture
recognition. In total, 3,484 gesture and 30,771 non-gesture datapoints were
collected.

(a) MSE

(b) DTW

(c) Keogh’s lower bound

Figure 2. The dissimilarity of two WatchAuth gestures, measured with (a) Mean
Square Error (MSE) distance, (b) Dynamic Time Warping (DTW) distance, and (c)
Keogh’s lower bound. Matching between points is shown in grey.

WatchAuth was designed to be memory-efficient and resource-sparse in order to
work with the limited memory and battery capacity of a smartwatch. To classify
payment gestures, it first windows a gesture’s data to the 4 seconds before
NFC contact. It then extracts manually-defined features (e.g. mean, variance,
skew, peak count). These features are calculated along each of the gesture
data’s channels (e.g. acceleration x-axis, gyroscope z-axis, acceleration
Euclidean norm). A random forest classifier with 100 decision trees is then
trained to classify these features, using the target user’s data as the
positive class and other users’ data as the negative class.

###  3.3. Generative Modelling

Autoencoder models. One common family of generative deep learning models are
autoencoders. Under this semi-supervised framework, we have two neural
networks: an encoder $E$ and decoder $D$. Given input $x$, we first apply the
encoder $E$ to generate an embedding in a latent space (typically a Euclidean
space of low dimension). We then apply the decoder $D$ to $E(x)$ to
reconstruct a signal in the original domain. For a given loss function $\ell$,
the objective of the network is to minimise the reconstruction loss
$\ell(x,(D\circ E)(x))$.

An autoencoder’s latent space must be sufficiently regular to be useful for
generating synthetic data. In particular, we require it to be sufficiently
continuous so that close together points decode to semantically similar
representations (Rocca, 2021), and to be complete, i.e. without ”holes”.
Unregularised autoencoders only minimise reconstruction loss, which usually
results in a latent space that does not encode useful semantic information and
is full of holes where the decoder has never been trained (Tolstikhin et al.,
2018).

The most well-known version of a regularised autoencoder is a variational
autoencoder (VAE), proposed by Kingma & Welling (Kingma and Welling, 2014).
Instead of outputting a single vector in the latent space, the encoder is
modified to output a latent mean and variance representing the parameters of a
Gaussian distribution. The decoder then samples from this Gaussian and decodes
the sampled point.

To comply with our latent space requirements of continuity and completeness,
we want our latent variances to be sufficiently large and our latent means
sufficiently close such that the latent distributions of different points
overlap. We achieve this using the Kullback-Leibler (KL) divergence to
penalise the distance between each latent distribution and a standard
Gaussian. This gives rise to the following KL loss used as a regularisation
term in the VAE loss function111In practice, our encoder outputs the logarithm
of the latent variance, resulting in a change of variables in this formula.:

$\displaystyle\ell_{KL}(\mu,\sigma^{2})$
$\displaystyle=D_{KL}(\mathcal{N}(0,1)||\mathcal{N}(\mu,\sigma^{2}))$
$\displaystyle=-\frac{1}{2}(1+log(\sigma^{2})-\mu^{2}-\sigma^{2})$

The full loss function used in a VAE is the sum of the reconstruction and KL
losses.

Dynamic Time Warping (DTW). DTW is an algorithm for quantifying the
dissimilarity between two timeseries that may not be perfectly aligned. The
DTW distance is computed as the minimum Euclidean distance between the
timeseries across all temporal alignments. A given temporal alignment may be
described by a matching between two sequences (see Figure 2).

Given sequences $\mathbf{x}=(x_{i})_{i=1}^{m}$ and
$\mathbf{y}=(y_{i})_{i=1}^{n}$, a valid matching must obey the following
rules:

  1. (1)

The first and last points of the sequences must be matched to each other (but
may also match other points).

  2. (2)

Every index from both sequences must be matched with at least one index from
the other sequence.

  3. (3)

Indices from the first sequence must be matched in a monotonically increasing
manner to indices from the second sequence, and vice versa.

The cost of a matching is the sum of distances between matched points. An
optimal matching may be computed via dynamic programming in time and space
complexity equal to the product of the lengths of the sequences.

Keogh’s lower bound for DTW. Keogh’s lower bound is a linear time lower bound
on DTW that makes use of derived series, termed envelopes, around a
timeseries. For a sequence $\mathbf{x}=(x_{i})_{i=1}^{n}$, its upper and lower
envelopes $U_{x}$ and $L_{x}$ are the maximum and minimum values of
$\mathbf{x}$ within a moving window of half-width $w$:

$\displaystyle\qquad U_{i}^{\mathbf{x}}$
$\displaystyle=\max_{\max\\{1,i-w\\}\leq
j\leq\min\\{l,i+w\\}}\\{x_{j}\\}\qquad$ $\displaystyle\qquad
L_{i}^{\mathbf{x}}$ $\displaystyle=\min_{\max\\{1,i-w\\}\leq
j\leq\min\\{l,i+w\\}}\\{x_{j}\\}$

Keogh’s lower bound is the sum of distances at points where sequence
$\mathbf{y}=(y_{i})_{i=1}^{n}$ is not contained within $\mathbf{x}$’s
envelopes (see Figure 2):

$\qquad
d_{KLB}^{(w)}(\mathbf{x},\mathbf{y})=\sum_{i=1}^{n}\begin{cases}\ell(y_{i},U_{i}^{\mathbf{x}})\qquad\text{if
$y_{i}>U_{i}^{\mathbf{x}}$},\\\ \ell(y_{i},L_{i}^{\mathbf{x}})\qquad\text{if
$y_{i}<L_{i}^{\mathbf{x}}$},\\\
0\qquad\qquad\quad\text{otherwise.}\end{cases}$

##  4\. Approach

###  4.1. Challenges for Authentication

Related works demonstrate that methods are known for generating synthetic data
to augment the training of activity recognition classifiers. However,
authentication tasks require discriminating between similar actions by
different users, and so are a more involved set of tasks that have not yet
been addressed by the literature.

The cheapest and least complex approach to generating synthetic data is
applying class-invariant transformations to real-world data. Unfortunately, it
is not well understood which transformations preserve class labels for
authentication using gesture data. Common timeseries transformations such as
jittering, temporal scaling and intensity scaling have been shown not to
preserve class labels in the related context of user identification (Benegui
and Ionescu, 2020).

As a result, we considered two of the most popular deep generative frameworks
(Iglesias et al., 2023), namely adversarial models such as GANs and
autoencoder-based models such as VAEs. While GANs typically generate data of
higher fidelity than autoencoder-based models, they frequently suffer from
unstable training (Goodfellow et al., 2020) and mode collapse (Zhang, 2021).
These issues proved insurmountable in our preliminary tests, so attention was
restricted to autoencoder-based models.

###  4.2. Our Approach

We are in the setting where we have a large number of gestures from existing
users and a small number of gestures from a target user. Our research question
is as follows: How can we generate additional realistic gestures for the
target user?

To use an autoencoder to generate synthetic data, we must first train the
encoder and decoder together to embed gestures into a latent space and
reconstruct gestures from their embeddings. After training, points can be
sampled from the learnt latent space and passed through the decoder to
generate synthetic gestures.

To generate user-specific gestures successfully, an autoencoder model must
satisfy the following two requirements:

  1. (1)

High reconstruction quality — the decoder must reconstruct realistic gestures
from embeddings in the latent space.

  2. (2)

Meaningful latent space representations — the encoder must learn a latent
space representation where nearby embeddings decode to semantically similar
gestures (ideally gestures from the same user).

To achieve this, we first build an autoencoder-based generative model with
high reconstruction quality and meaningful latent space representations. We
then use the generative model to generate synthetic gestures for a particular
user and evaluate the usefulness of our method.

As our trained generative model will not have been exposed to the new user, we
rely on the assumption that given a large enough corpus of users we may find a
latent space that adequately captures common discriminative features between
users. We select points close to a user’s previously seen latent space
embeddings to decode, with the intention of retaining user-specific gesture
characteristics.

We call our final system for generating additional user-specific synthetic
data UserBoost.

###  4.3. Threat Model

Throughout this paper, we adopt the threat model of WatchAuth (Sturgess et
al., 2022). That is, we assume that an adversary has possession of a
legitimate user’s unlocked smartwatch, and that the adversary attempts to make
a contactless payment using it. This is a zero-effort attack, as the adversary
is attempting to fool the authentication system by submitting their own
biometric signal.

###  4.4. Evaluating Authentication Models

Given biometric data, a biometric authentication system outputs a probability
that the data belongs to a given user. If this probability is above a
threshold $T$, the system accepts the user; otherwise, it rejects them.
Security may be balanced against usability by modifying the threshold value
$T$.

In the case of WatchAuth, maximum usability is desirable. The relevant metric
in this case is the False Acceptance Rate (FAR) when $T$ is optimised to set
the false rejection rate to 0, i.e. ”the false acceptance rate when we permit
no false rejections”. Henceforth, we abbreviate this quantity to FAR@0.

For models trained on limited data, many test samples may be predicted the
same probability, particularly for random forest models whose implied
probabilities are output from a discrete set. This poses a problem when
calculating Equal Error Rate (EER), as the crossover point between FAR and
False Rejection Rate (FRR) can lie in a large interval (see Figure 3). For
reporting transparency, in several tables we present the EER as an interval,
with the lower value representing the minimum value of FAR where FAR is
greater than FRR, and the upper value representing the FRR value at this same
threshold. We also report the Area Under Receiver Operating Characteristic
curve (AUROC) value as a secondary metric for evaluating the average
authentication ability of a classifier.

Figure 3. A plot of FAR and FRR against decision threshold $T$ from a RF100
classifier (defined in Section 5.3). Marked points show thresholds. Observe
that the true EER may lie between $\sim 30\%$ and $\sim 45\%$.

###  4.5. Evaluating Generative Models

We may evaluate the quality of an autoencoder’s reconstructions via its
validation reconstruction loss. However, this metric is not guaranteed strong
correspondence with the usefulness of reconstructed gestures for training an
authentication system. Instead, the quality of reconstructions was evaluated
quantitatively by the train-synthetic test-real (TSTR) method (see (Yadav et
al., 2023), or ”GAN-train” in (Figueira and Vaz, 2022)). This is a strategy
for evaluating the similarity between synthetic data and true data by training
a classifier on the synthetic data and testing it on the real test dataset.
Strong classification performance demonstrates explicitly that the synthetic
data is of sufficiently high quality to be useful as training data.

##  5\. Methods

###  5.1. Data Preparation

The WatchAuth dataset is provided as per-user files of partially-processed
gesture data. Each row is associated with a gesture ID, timestamp and one of
the watch’s four sensors with sensor readings.

A standard input format for timeseries prediction models is a real vector of
dimension $N_{timesteps}\times N_{channels}$. We therefore pre-process each
row to be indexed by gesture ID and timestep rather than by gesture ID,
timestep and sensor. The data provided per gesture covers a timespan from 4
seconds before NFC contact to 2 seconds after. For real-time authentication,
time after the payment is accepted is irrelevant, so we slice all gestures to
only cover the 4 seconds before payment is made. Non-gesture data provided in
the WatchAuth dataset is also partitioned into 4 second windows.

Restricting to the sensors of interest, we represent each gesture by a
separate $200\times 6$ array (3 channels each for accelerometer and gyroscope
data). Each gesture array is then passed through a Butterworth low-pass filter
to remove noise. We also store descriptive information for each gesture,
including user ID and payment terminal position.

###  5.2. Experiment Details

The choice of experiment setup can strongly influence reported results. To
align with best practice, we preserved the temporal order of samples when
constructing train/test splits. The first two-thirds of all users’ gestures
was used for training and validation, with the final third used for testing.
For authentication classifiers, weighting the training data 4:1 in favour of
the positive class showed the strongest classification performance on
validation data.

For the timeseries data used by deep learning models, channel-wise means and
variances were initialised using the training data and used to normalise all
data prior to classification.

All models were implemented in TensorFlow (Martín Abadi et al., 2015), with
the exception of one model using the Soft-DTW loss function, which was
implemented in PyTorch (Paszke et al., 2019) (for compatibility with Maghoumi
et al.’s CUDA implementation of Soft-DTW (Maghoumi et al., 2021)).

All deep learning models were trained with the Adam optimiser (with learning
rate $1e-4$). Hyperparameters were tuned manually to minimise average
validation loss. During training, validation loss was used for model selection
(using early stopping with a patience of 150 epochs).

###  5.3. Authentication Models

To perform TSTR as described, we require some independent models for
performing authentication. We consider both a Random Forest model, similar to
WatchAuth, and an alternative machine learning architecture with additional
expressive power (similar to the encoder in our autoencoder model)222These
models were chosen as they exhibited the best authentication performance on
the unaltered dataset..

RF100. Sturgess et al. (Sturgess et al., 2022) showed that the accelerometer
and gyroscope are the most informative sensors for authentication.
Furthermore, as many smartwatches only have these two sensors, this study
restricts the data considered to just the acceleration and angular velocity
vectors. In the rest of the paper, we refer to the WatchAuth random forest
authentication system with 100 trees using just accelerometer and gyroscope
data as RF100. (We also investigated the equivalent system with 1000 trees,
but observed only minor gains that did not justify the additional resources
required for training and classification with forests of this size.)

Conv+GRU. We wish to construct a deep learning architecture that exploits both
the timeseries nature of our input data, as well as the relationship between
different channels.

Empirically, we obtained the best authentication performance by combining
convolutional and Gated Recurrent Unit (GRU) layers in sequence. Intuitively,
this architecture can first extract translation-invariant features using
convolutions, and then exploit intra-sequence dependencies with the GRU333GRUs
were found to greatly outperform the more complex LSTM (Long Short-Term
Memory) units, which could not be trained effectively.. This approach also
mitigates training difficulties when using GRUs for long sequences as the
convolutional layers reduce the spatial dimension of the data.

Our Conv+GRU model employs varying kernel sizes to extract features at
multiple scales (similar to Chrononet (Roy et al., 2019)). These features are
then concatenated and passed through 1x1 kernels to reduce the overall number
of parameters. Four layers of this architecture are employed to reduce the
spatial dimension of the data to thirteen timesteps. Data is then passed
through three stacked GRUs before a classification multi-layer perceptron with
two hidden layers, one with 25 units and one with 10 units, both using the
Rectified Linear Unit (ReLU) activation function. An output neuron using
sigmoid activation then is used to output a classification probability.

###  5.4. Autoencoder Architecture

We now seek to construct an encoder-decoder pair for modelling payment
gestures.

We take our encoder to be the Conv+GRU model described in Section 5.3.
However, we remove the output sigmoid neuron so that its output is a vector of
dimension equal to the latent space size (the latent embedding vector).

We construct our decoder as an inverted version of the encoder. Our decoder
consists of stacked GRU layers acting on the latent embedding vector, followed
by a sequence of upsampling and convolutional layers to reconstruct a full-
length output signal.

The size of an autoencoder’s latent space must be a compromise between the two
requirements of Section 4.2, with larger spaces favouring reconstruction
quality and smaller spaces favouring more meaningful latent space
representations. Following extensive experimentation, the latent space size
was fixed at 10 dimensions.

###  5.5. Autoencoder Reconstruction Loss

For synthetic data to be useful, it must be of sufficiently high quality to
improve a classifier’s performance on an authentication task. For an
autoencoder, the quality of gestures is determined by the fidelity of the
decoder’s outputs. Given that the decoder is trained to minimise
reconstruction loss, the choice of loss function strongly influences the
quality of reconstructed gestures.

In the absence of a literature consensus on the choice of loss function for
timeseries comparison, we investigate the effectiveness of several choices:

  * •

Mean Squared Error (MSE): A naive choice of loss is pointwise MSE. MSE only
captures the similarity between two signals in the intensity dimension.

  * •

Soft-DTW: DTW is an alternative quantity for measuring timeseries
dissimilarity that is invariant to translation and warping in the time
dimension. However, DTW is known to perform poorly as a loss function because
the min operation used to calculate it is not differentiable. In 2017, Cuturi
& Blondel introduced Soft-DTW (Cuturi and Blondel, 2017) as a smoothed version
of DTW that replaces min with a differentiable soft-min operation. The level
of smoothing of the soft-min operation is controlled by a hyperparameter
$\gamma$, which we set to a standard value of $0.1$. We calculate Soft-DTW
separately for each channel — otherwise, the time warping heavily penalises
reconstructions that slightly misalign peaks for two separate channels. We use
Maghoumi et al.’s open-source CUDA implementation for PyTorch (Maghoumi et
al., 2021), which follows the formulation of Soft-DTW given by Zhu et al. (Zhu
et al., 2018).

  * •

Keogh’s lower bound for DTW: Keogh’s lower bound for DTW is much more
efficient to compute than Soft-DTW. Furthermore, it is tolerant of small
translational shifts / warping within its Sakoe-Chiba bandwidth. However, it
provides zero reconstruction loss when one curve is entirely within the
envelope of another curve, which would limit an autoencoder’s reconstruction
resolution to the coarseness of the bandwidth. We overcome this issue through
a modification of Keogh’s lower bound, which we name KLB-mod and construct as
the weighted average of Keogh’s lower bound at exponentially varying
bandwidths:

$\qquad d_{\text{{KLB-mod}}}(x,y)=\sum_{w=1}^{5}(6-w)\cdot
d_{KLB}^{(2^{w})}(x,y)$

where $d_{KLB}^{(w)}$ is Keogh’s lower bound. This formulation results in a
theoretically smoother loss landscape than MSE or Keogh’s lower bound with
fixed bandwidth.

  * •

Feature-based approaches: WatchAuth compares gestures based on their feature
similarity rather than on explicit differences between the original curve
amplitudes. To reconstruct gestures that are highly discriminative for
authentication, we consider augmenting the above reconstruction losses with a
feature-based loss. We represent a curve as a vector of differentiable
features and apply a standard Euclidean kernel. The features used in WatchAuth
that are compatible with TensorFlow are used for this feature vector
representation, namely max, min, mean, standard deviation, variance, skew,
kurtosis, median and inter-quartile range.

We trained autoencoders with each of the reconstruction losses above, as well
as the combination of feature-based loss and each other loss. We then
generated synthetic data with the trained autoencoders, and evaluated this
synthetic data with a TSTR task classifying timeseries data into gesture and
non-gesture classes (details in Appendix A). KLB-mod + Feature, a loss
additively combining KLB-mod and our Feature loss achieved the best empirical
performance and so we use this loss function going forwards.

Figure 4. The first two principal components (PCs) of the training data’s
latent space embedding for an unregularised autoencoder trained using the KLB-
mod + Feature loss. Figure 5. Training data latent space mean embeddings for
VAEs trained with decreasing $\beta$ values, decreasing the influence of
regularisation. Different colours represent different users.

###  5.6. Regularising the Autoencoder

Figure 4 visualises the latent space of an autoencoder model trained using
KLB-mod + Feature. It is promising that there appears to be a degree of
natural clustering by user. However, the latent space lacks the necessary
continuity to be useful for generating synthetic gestures and so requires
regularisation.

We used the VAE framework for regularisation, which introduces a KL
regularisation term into the autoencoder loss function, weighted by scalar
hyperparameter $\beta$:

$\qquad\ell_{VAE}=\ell_{reconstruction}+\beta\cdot\ell_{regularisation}$

We exponentially varied the value of $\beta$ between 1 and $1e-6$ and trained
a VAE on the training dataset for each $\beta$.

Figure 5 visualises how the latent space becomes less regularised as $\beta$
decreases. We can see that, for $\beta\geq 1e-2$, all points are encoded with
approximate mean 0 and standard deviation 1. For $\beta\leq 1e-3$
regularisation still has an effect but is balanced with reconstruction ability
(see Appendix B, Figure 15). As a result we set $\beta=1e-4$.

Appendix B details an empirical comparison of Wasserstein autoencoders to VAEs
as a regularisation strategy.

###  5.7. Enforcing User Similarity in the Latent Space

To be able to sample points from the latent space and have confidence in their
user label, we must achieve much more consistent clustering by user in the
latent space than has been observed in the models so far. This motivates the
explicit enforcement of user-based clustering in the latent space.

To this end, we add a simple Multi-Layer Perceptron (MLP) classifier to our
VAE model, and train it to classify points in the latent space according to
their user. We strictly limit the expressive power of this classifier. As a
result, to improve the latent space user classification the encoder must learn
to organise the latent space by user. Under the hypothesis that the
distribution of gestures is determined both by user and non-user variables
(e.g. terminal position, independent random effects), we only apply this
classifier to the first 5 dimensions of the 10-dimensional latent space.

Figure 6. Latent space visualisation for VAE-based generative models trained
with varying values of $\alpha$, increasing the emphasis put on a structured
latent space in the modified VAE loss function.

The natural loss function for this classifier is categorical cross-entropy.
Empirically, we observed that this loss caused the model to increase its
confidence in points it was correctly classifying rather than focusing on
classifying all points correctly. An alternative choice of loss that we found
to induce stronger clustering was a rank-based loss function, where points in
the latent space are assigned scores per user, and the objective is to
maximise the Mean Reciprocal Rank (MRR) of the true user (to reward higher
predicted ranks to the true user). For differentiability and compatibility
with our deep learning architecture, we used approximate MRR (Qin et al.,
2010) and implemented in TensorFlow Ranking (Pasumarthi et al., 2019). This
latent space authentication loss leads to a new loss function for our model:

$\qquad\ell_{total}=\ell_{\text{{KLB-mod +
Feature}}}+\beta\cdot\ell_{KL}+\alpha\cdot\ell_{auth}$

We introduce a new hyperparameter $\alpha$ to control the relative effect of
the authentication loss. As with $\beta$ in Section 5.6, we experimented with
different values of $\alpha$ to balance reconstruction quality, regularisation
and classification by user. Figure 6 shows the latent space at varying values
of $\alpha$. Note that setting $\alpha$ too high results in excellent grouping
by user and very poor regularity.

We set $\alpha=1e-2$, guided by the effect on reconstruction loss (see
Appendix C, Figure 14). The validation approximate MRR is $\approx 0.1$
without and $\approx 0.3$ with the authentication MLP, which shows it induces
an improvement in clustering by user.

###  5.8. Model Design Summary

Informed by the preliminary experiments and discussion in this section, we
constructed the following autoencoder model to generate synthetic payment
gestures. We constructed a VAE-based model, with encoder and decoder as
specified in Section 5.4, with the following weighted sum loss:

  * •

Reconstruction loss ($\times 1$): the sum of a feature-based loss, to reward
reconstructions that preserve key features of curves such as $\max$ and
$\min$, and the weighted average of Keogh’s lower bound at exponentially
varying bandwidths, to smoothly reward faithful reconstructions.

  * •

Regularisation loss ($\times 1e-4$): KL-divergence (as normal for VAE), acting
on magnitude of latent neurons in the model.

  * •

Latent space authentication loss ($\times 1e-2$): a rank-based loss penalising
latent spaces that do not display sufficient organisation by user. A simple
MLP is trained in parallel with with main network to classify users from their
latent space embeddings (only using half of the latent space neurons), and the
loss is applied to these predictions.

##  6\. Results

###  6.1. Verifying Model Properties

Latent space structure. For synthetic gestures to be useful, our encoder must
assign similar embeddings to training gestures and unseen gestures for a given
user. Figure 7 shows the result of embedding one user’s gestures in the test
dataset for a model trained using gestures in the training dataset.

It may be observed that the gestures observe a degree of localisation to a
region of the embedding manifold. It is unrealistic to expect perfect
clustering given that authentication classifiers trained on this dataset do
not achieve perfect classification. With more users, the quality of gesture
clustering would be expected to improve.

Diversity and fidelity of synthetic gestures. We verify that the decoder is
generating diverse, high fidelity samples and is not just linearly
interpolating between existing gestures. This can be seen in Figure 8, which
we created by sampling two nearby points from the latent space, calculating
points on the line between them, and decoding these intermediate points.

Figure 7. t-distributed Stochastic Neighbour Embedding (t-SNE) visualisation
of a trained VAE-based model’s latent space, demonstrating that an example
user’s test gestures are embedded similarly to the user’s training gestures.
Figure 8. Visualisation of decoding points along a short line segment in the
latent space. For points $x$ and $y$, we visualise the decoded gesture from
latent space point $\mu\cdot x+(1-\mu)\cdot y$ for each $0\leq\mu\leq 1$. The
reconstructions with $\mu=0$ and $\mu=1$ are represented by blue and red
dotted lines respectively. Colours associated with other values of $\mu$ are
shown by the colour bar.

###  6.2. Generating User-specific Synthetic Gestures

We now seek to use our final VAE-based model (from Section 5.8) to generate
user-specific synthetic gestures, and evaluate whether these gestures are
useful for an authentication use case.

Given a VAE-based model trained on other users, we can apply the encoder to
embed the target user’s gestures into the model’s latent space. We can then
use these embeddings to guide the selection of further points in the latent
space, which we then decode to generate our user-specific synthetic data.

In this section, we firstly investigate the effectiveness of four simple
strategies for selecting points in the latent space. We then proceed with the
most promising sampling strategy to answer the central question of whether our
methodology can improve the classification performance of a simple
authentication model.

###  6.3. Latent Space Sampling Strategies

Neighbourhood sampling. We simply sample points from a Gaussian distribution
using the embedding means and variances for the points we already know for the
user.

Self-mixed sampling. We pick several of the existing embedding means for the
user, and generate a new point in the latent space by calculating a randomly
weighted convex combination of these points.

Adversarial sampling. We randomly pick two embeddings: one from our target
user and one from a different user. We then calculate a convex combination of
them weighted heavily towards the target user (85:15). The rationale behind
this strategy is to generate gestures that should be classed as the target
user but have some characteristics of other users, to generate a more diverse
training set.

Same-user sampling. We take existing embeddings for the target user. We assume
that as the authentication MLP is trained on the first 5 dimensions of the
latent space that these dimensions are user-specific and other dimensions are
not. We therefore set the embedding’s last 5 dimensions to randomly sampled
embeddings for other users’ points.

###  6.4. Experiments

To evaluate whether our methodology improves the classification performance of
a simple authentication model, we performed TSTR for authentication using the
classifier RF100 (defined in Section 5.3). We trained 16 models, leaving out
training data for one user each time. This mimics the realistic scenario of
pre-training a generative model before seeking to enrol a new user into an
authentication system with the aid of synthetic gestures.

For each trained model, we allowed access to two real user gestures per
terminal position (14 in total), which were used to generate 500 synthetic
gestures with each of our four sampling strategies. The original and synthetic
gestures together formed the positive class for training an authentication
classifier. For the results in Section 6.5, our negative class contained the
VAE-based model’s reconstruction of each of the other users’ gestures. We then
converted our training data (using both real and synthetic gestures) into
feature vectors and trained a random forest classifier for authentication.

In Section 6.6, we also included the original gestures from other users as
negative samples, and varied the number of real user gesture samples per
terminal to investigate how effective our approach is at reducing enrolment
burden.

As before, the full test dataset of gestures unseen during training was used
to evaluate all trained authentication classifiers.

Table 1. TSTR authentication results for a random forest classifier aided by
synthetic data generated by different strategies.
Sampling Strategy |  AUROC |  EER interval |  FAR@0
---|---|---|---
Original data only |  0.83 |  (0.14, 0.28) |  1.00
Neighbourhood |  0.83 |  (0.23,0.26) |  0.79
Self-mixed |  0.82 |  (0.23,0.27) |  0.78
Adversarial |  0.84 |  (0.21,0.25) |  0.74
Same-user |  0.82 |  (0.23,0.27) |  0.77

###  6.5. Results

Overall performance. TSTR results for authentication, using synthetic data
generated with different sampling strategies, are presented in Table 1.

The synthetic data provided a significant improvement to FAR@0, irrespective
of sampling strategy. Adversarial sampling yielded the strongest performance
on all metrics amongst the sampling strategies, with an average reduction in
FAR@0 from 100% to 74% and favourable AUROC compared to using no synthetic
data. The addition of the synthetic data also reduced the uncertainty around
the true EER value. To investigate the cause of this improvement it is
necessary to consider performance on a per-user basis.

Per-user performance. Figure 10 shows the FAR@0 for each user when synthetic
gestures were generated using adversarial sampling. It is clear that
performance varied widely between users. For example, User 7 experienced a
reduction in FAR@0 from $1.0$ to $\sim 0.2$, while Users 6, 10 and 12 had very
little or no reduction.

We visualise the embeddings of test gestures for Users 7 and 10 in Figure 10.
In the case of User 7, our VAE-based model trained without User 7’s gestures
learnt a user clustering that is highly relevant to User 7. For User 10, no
such effective organisation of the latent space took place. This example shows
the importance of the assumption that closely localised points are more likely
to share a user. It is a positive finding that when this assumption is met,
synthetic gesture data is of high quality and improves classification
performance.

###  6.6. Impact on Enrolment Burden

Figure 9. Average FAR@0 for random forest classifiers, trained with and
without synthetic gestures (generated using adversarial sampling). Results are
broken down by user.

Figure 10. Left: User 7. Right: User 10. t-SNE visualisation of latent space
embeddings of test data for User $x$ and training data for other users, for a
VAE-based model trained without User $x$ ($x=7$ or $x=10$).

(a) Upper bound on EER

(b) FAR@0

(c) AUROC

Figure 11. Authentication metrics for random forest classifiers given
increasing quantities of training data, with and without the addition of
synthetic data.

The primary aim of this work was to explore whether using synthetic gestures
can reduce the enrolment burden of WatchAuth. To this end, in Figure 11 we
visualise how increasing the number of real user gestures used in training
affects the quality of an authentication classifier. We observe that the
addition of synthetic gestures does not adversely affect the EER and AUROC of
our trained classifier. Furthermore, the FAR@0 is markedly improved.

It is clear from Figure 11 that the enrolment burden on a user can be reduced
with our approach. For example, with synthetic gestures, only 9 gestures per
terminal were required to reach a FAR@0 value of 70%, while without synthetic
gestures 16 gestures per terminal were needed, an improvement of 42 fewer
gestures (in total) and a reduction of over 40%.

##  7\. Discussion and Limitations

Variation in performance between users. In Section 6.5 we showed a large
disparity between the effectiveness of our methodology for enrolling different
users. This could be due to fundamental differences between the
distinctiveness and stability of user gestures. However, it is more likely to
be because some users are more similar to others in the dataset collected, or
a result of improper data collection for some users. Both when training simple
authentication models on real data and when incorporating synthetic data, we
observed large differences in the quality of authentication for different
users, with a small subset of users proving difficult to authenticate.

If a user provided a poor quality gestures, the synthetic data for that user
would also likely be poor quality; however, as more real gestures were
provided, the authentication model would be retrained so this does not detract
from the benefits we show.

Improvement of average measures. FAR@0 is a measure specific to our use case
of minimising enrolment burden, prioritising security over usability. The more
general measures of EER and AUROC capture a blend of the authentication
system’s security and usability. Our results did not show any improvement in
EER or AUROC when adding synthetic data to the training process for RF100.

The sampling methodologies used are inherently likely to improve usability at
a potential cost to security, because there are no guarantees over what points
in the model’s latent space are associated with which users. As a result, it
is a likely scenario that models are made more permissive by the inclusion of
training data that shares characteristics associated with other users.

However, with a larger corpus of users there remains the possibility of
improving security as well as usability through the exploitation of
commonalities between user groups.

Latent space assumptions. The success of our synthetic data generation
methodology rests strongly on the assumption that we can find a latent space
of gesture data embeddings in which the gestures of unseen users cluster
meaningfully. This work has not fully validated that assumption, although the
success of synthetic gesture generation for some users is promising
(especially when considering Figures 10 and 10 together).

Synthetic gesture generalisability. Qualitatively, we found that our
reconstructed gestures were sometimes of lower fidelity than their
corresponding real gestures. As the gestures reconstructed by our generative
model were only tested for our authentication enrolment use case it remains
unknown how useful they are in other contexts. Additional evaluation would
clarify the wider usefulness of our synthetic gestures, while integrating this
work with GANs or diffusion models provides an avenue to generating higher
fidelity synthetic data.

Other use cases. The methodology employed in this work could be applicable to
other hardware-constrained authentication tasks. It also has potential as a
student-teacher learning framework for reducing the memory requirements of
authentication models.

##  8\. Conclusion

We have shown the feasibility of generating synthetic IMU data from scratch
corresponding to a specific user’s physical gesture using a deep-learning
pipeline. When used to augment the training process of WatchAuth, a simple
machine-learned biometric authentication system, this synthetic data reduces
the enrolment burden for a user by up to $40\%$ without compromising security.
It also reduced the false acceptance rate when no false rejections are
permitted by an average of $26\%$ for a case of minimal training data. Once
trained, our synthetic data generation solution is efficient and generates
diverse synthetic data.

The usefulness of the synthetic data varied between users, with some users
seeing excellent results with synthetic data and others only low to moderate
improvements. It is hoped that larger studies would be better able to exploit
the potential of transferring learnt semantic information from a large corpus
of users’ data to the authentication template for a new user.

## Acknowledgements

This work was supported financially by Mastercard and the
Engineering and Physical Sciences Research Council [grant number
EP/P00881X/1]. The authors would like to thank these organisations for their
support.

## References

  * (1)
  * Alawneh et al. (2021) Luay Alawneh, Tamam Alsarhan, Mohammad Al-Zinati, Mahmoud Al-Ayyoub, Yaser Jararweh, and Hongtao Lu. 2021.  Enhancing Human Activity Recognition Using Deep Learning and Time Series Augmented Data.  _Journal of Ambient Intelligence and Humanized Computing_ 12, 12 (Dec. 2021), 10565–10580.  https://doi.org/10.1007/s12652-020-02865-4
  * Alharbi et al. (2020) Fayez Alharbi, Lahcen Ouarbya, and Jamie A Ward. 2020.  Synthetic Sensor Data for Human Activity Recognition. In _2020 International Joint Conference on Neural Networks (IJCNN)_. IEEE, Glasgow, UK, 1–9.  https://doi.org/10.1109/IJCNN48605.2020.9206624
  * Alzantot et al. (2017) Moustafa Alzantot, Supriyo Chakraborty, and Mani Srivastava. 2017.  SenseGen: A Deep Learning Architecture for Synthetic Sensor Data Generation.  _2017 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops)_ (March 2017), 188–193.  https://doi.org/10.1109/PERCOMW.2017.7917555
  * Benegui and Ionescu (2020) Cezara Benegui and Radu Tudor Ionescu. 2020.  To Augment or Not to Augment? Data Augmentation in User Identification Based on Motion Sensors. In _Neural Information Processing: 27th International Conference, ICONIP 2020, Bangkok, Thailand, November 18–22, 2020, Proceedings, Part V 27_. Springer, 822–831.
  * Cuturi and Blondel (2017) Marco Cuturi and Mathieu Blondel. 2017.  Soft-DTW: A Differentiable Loss Function for Time-Series. In _Proceedings of the 34th International Conference on Machine Learning_. PMLR, 894–903.
  * de Souza et al. (2023) Maynara Donato de Souza, Clesson Roberto Silva Junior, Jonysberg Quintino, André Luis Santos, Fabio Q B da Silva, and Cleber Zanchettin. 2023.  Exploring the Impact of Synthetic Data on Human Activity Recognition Tasks.  _Procedia Computer Science_ 222 (Jan. 2023), 656–665.  https://doi.org/10.1016/j.procs.2023.08.203
  * Figueira and Vaz (2022) Alvaro Figueira and Bruno Vaz. 2022.  Survey on Synthetic Data Generation, Evaluation Methods and GANs.  _Mathematics_ 10, 15 (Jan. 2022), 2733.  https://doi.org/10.3390/math10152733
  * Goodfellow et al. (2020) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020.  Generative Adversarial Networks.  _Commun. ACM_ 63, 11 (Oct. 2020), 139–144.  https://doi.org/10.1145/3422622
  * Goubeaud et al. (2021) Maxime Goubeaud, Philipp Joußen, Nicolla Gmyrek, Farzin Ghorban, Lucas Schelkes, and Anton Kummert. 2021.  Using Variational Autoencoder to Augment Sparse Time Series Datasets. In _2021 7th International Conference on Optimization and Applications (ICOA)_. 1–6.  https://doi.org/10.1109/ICOA51614.2021.9442619
  * Iglesias et al. (2023) Guillermo Iglesias, Edgar Talavera, Ángel González-Prieto, Alberto Mozo, and Sandra Gómez-Canaval. 2023.  Data Augmentation Techniques in Time Series Domain: A Survey and Taxonomy.  _Neural Computing and Applications_ 35, 14 (May 2023), 10123–10145.  https://doi.org/10.1007/s00521-023-08459-3
  * Kim and O’Neill-Brown (2019) Jungi Kim and Patricia O’Neill-Brown. 2019.  Improving American Sign Language Recognition with Synthetic Data. In _Proceedings of Machine Translation Summit XVII: Research Track_. European Association for Machine Translation, Dublin, Ireland, 151–161.
  * Kingma and Welling (2014) Diederik P. Kingma and Max Welling. 2014.  Auto-Encoding Variational Bayes. In _2nd International Conference on Learning Representations_. Banff, Canada, 14–16.
  * Konak et al. (2023) Orhan Konak, Lucas Liebe, Kirill Postnov, Franz Sauerwald, Hristijan Gjoreski, Mitja Lusˇtrek, and Bert Arnrich. 2023.  Overcoming Data Scarcity in Human Activity Recognition.  (2023).
  * Maghoumi et al. (2021) Mehran Maghoumi, Eugene Matthew Taranta, and Joseph LaViola. 2021.  DeepNAG: Deep Non-Adversarial Gesture Generation. In _26th International Conference on Intelligent User Interfaces_ _(IUI ’21)_. Association for Computing Machinery, New York, NY, USA, 213–223.  https://doi.org/10.1145/3397481.3450675
  * Magnates (2023) Finance Magnates. 2023.  Paying with Your Wrist: The Future of Wearable Payments.  https://www.financemagnates.com/fintech/payments/paying-with-your-wrist-the-future-of-wearable-payments/.
  * Martín Abadi et al. (2015) Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems.  https://www.tensorflow.org/.
  * Pasumarthi et al. (2019) Rama Kumar Pasumarthi, Sebastian Bruch, Xuanhui Wang, Cheng Li, Michael Bendersky, Marc Najork, Jan Pfeifer, Nadav Golbandi, Rohan Anil, and Stephan Wolf. 2019.  TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. Association for Computing Machinery, Anchorage, AK, 2970–2978.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _Advances in Neural Information Processing Systems 32_. Curran Associates, Inc., 8024–8035.
  * Qin et al. (2010) Tao Qin, Tie-Yan Liu, and Hang Li. 2010.  A General Approximation Framework for Direct Optimization of Information Retrieval Measures.  _Information Retrieval_ 13, 4 (Aug. 2010), 375–397.  https://doi.org/10.1007/s10791-009-9124-x
  * Rocca (2021) Joseph Rocca. 2021.  Understanding Variational Autoencoders (VAEs).  https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73.
  * Roy et al. (2019) Subhrajit Roy, Isabell Kiral-Kornek, and Stefan Harrer. 2019.  ChronoNet: A Deep Recurrent Neural Network for Abnormal EEG Identification. In _Artificial Intelligence in Medicine_ , David Riaño, Szymon Wilk, and Annette Ten Teije (Eds.), Vol. 11526. Springer International Publishing, Cham, 47–56.  https://doi.org/10.1007/978-3-030-21642-9_8
  * Santhalingam et al. (2023) Panneer Selvam Santhalingam, Parth Pathak, Huzefa Rangwala, and Jana Kosecka. 2023.  Synthetic Smartwatch IMU Data Generation from In-the-wild ASL Videos.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 7, 2 (June 2023), 74:1–74:34.  https://doi.org/10.1145/3596261
  * Soleimani and Nazerfard (2021) Elnaz Soleimani and Ehsan Nazerfard. 2021.  Cross-Subject Transfer Learning in Human Activity Recognition Systems Using Generative Adversarial Networks.  _Neurocomputing_ 426 (Feb. 2021), 26–34.  https://doi.org/10.1016/j.neucom.2020.10.056
  * Sturgess (2023) Jack Sturgess. 2023.  WatchAuth Dataset (Extended).  University of Oxford.
  * Sturgess et al. (2022) Jack Sturgess, Simon Eberz, Ivo Sluganovic, and Ivan Martinovic. 2022.  WatchAuth: User Authentication and Intent Recognition in Mobile Payments Using a Smartwatch. In _2022 IEEE 7th European Symposium on Security and Privacy (EuroS &P)_. IEEE Computer Society, Los Alamitos, CA, USA, 377–391.  https://doi.org/10.1109/EuroSP53844.2022.00031
  * Tolstikhin et al. (2018) Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. 2018.  Wasserstein Auto-Encoders. In _International Conference on Learning Representations_.
  * Yadav et al. (2023) Parul Yadav, Manish Gaur, Nishat Fatima, and Saqib Sarwar. 2023.  Qualitative and Quantitative Evaluation of Multivariate Time-Series Synthetic Data Generated Using MTS-TGAN: A Novel Approach.  _Applied Sciences_ 13, 7 (Jan. 2023), 4136.  https://doi.org/10.3390/app13074136
  * Zhang (2021) Kaifeng Zhang. 2021.  On Mode Collapse in Generative Adversarial Networks. In _Artificial Neural Networks and Machine Learning – ICANN 2021_ _(Lecture Notes in Computer Science)_ , Igor Farkaš, Paolo Masulli, Sebastian Otte, and Stefan Wermter (Eds.). Springer International Publishing, Cham, 563–574.  https://doi.org/10.1007/978-3-030-86340-1_45
  * Zhang et al. (2022) Shibo Zhang, Yaxuan Li, Shen Zhang, Farzad Shahabi, Stephen Xia, yu Deng, and Nabil Alshurafa. 2022.  Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances.  _Sensors_ 22 (Feb. 2022), 1476.  https://doi.org/10.3390/s22041476
  * Zhu et al. (2018) Huanzhou Zhu, Zhuoer Gu, Haiming Zhao, Keyang Chen, Chang-Tsun Li, and Ligang He. 2018.  Developing a Pattern Discovery Method in Time Series Data and Its GPU Acceleration.  _Big Data Mining and Analytics_ 1, 4 (Dec. 2018), 266–283.  https://doi.org/10.26599/BDMA.2018.9020021

## Appendix

##  Appendix A Autoencoder Reconstruction Loss Experiments

Preliminary experiments. We conducted experiments with loss functions defined
as the weighted sum of this feature loss and each of the MSE and KLB-mod loss
functions, which we call MSE + Feature and KLB-mod + Feature respectively. The
relative weight of the non-feature and feature losses was heuristically set to
1:0.1 and 1:0.01 respectively to reconstruct visually realistic gestures444We
also attempted to calculate features on a rolling basis for a timeseries and
compare two ”feature timeseries”. This approach resulted in noisy
reconstructions and took prohibitively long to run, so we did not investigate
it further..

The same naïve autoencoder (with architecture detailed in Section 5.4) was
trained on the WatchAuth gesture data using different loss functions. Each
model was trained until convergence with early stopping on its validation
error, using the same random 20% validation split for each choice of loss.

The quality of reconstructions was evaluated quantitatively by the TSTR method
(detailed in Section 4.5).

We began by using TSTR for classification of timeseries data into gesture and
non-gesture classes, to investigate whether synthetic gestures were reliably
distinguishable from non-gesture data. 240 synthetic gestures were generated
as the direct reconstructions of randomly selected gestures for the positive
class, with 240 non-gesture samples used as the negative class.

Preliminary results. Table 2 summarises the AUROC and EER values computed for
our TSTR gesture recognition task.

We firstly note that models using all loss functions except Soft-DTW achieved
strong results on both TSTR tasks. The Soft-DTW model generated samples that
achieved superior gesture recognition ability to the original curves on the
RF100 TSTR task but completely failed the Conv+GRU TSTR task.

Visualising reconstructed gestures gives an explanation. We see in Figure 12
that the reconstructed curve from the model using Soft-DTW is very noisy. This
could be a result of difficulty in training effectively — gradients from the
loss must be backpropagated through a dynamic programming algorithm, resulting
in a highly complex optimisation landscape. In Figure 13 we see a further
visualisation of a different channel, with all loss functions except Soft-DTW
achieving near-perfect reconstruction.

These reconstructed gestures are clearly extremely different to non-gesture
data; perhaps this extreme difference to non-gesture data has resulted in the
strong performance on the RF100 TSTR task despite the unrealistic generated
gestures.

Figure 12. Visualising reconstructed gestures along the gyroscope z-axis for
autoencoders trained with different loss functions. Table 2. TSTR results for
classification between gesture and non-gesture data for autoencoders trained
with different choices of reconstruction loss, using both Conv+GRU and RF100
as classifiers.
| Conv+GRU | RF100
---|---|---
Loss function |  AUROC |  EER |  AUROC |  EER
No reconstruction |  0.94 |  0.14 |  0.97 |  0.054
MSE |  0.88 |  0.19 |  0.97 |  0.054
Soft-DTW |  0.30 |  0.65 |  0.99 |  0.033
KLB-mod |  0.88 |  0.18 |  0.97 |  0.056
MSE + Feature |  0.90 |  0.18 |  0.97 |  0.055
KLB-mod + Feature |  0.92 |  0.16 |  0.97 |  0.056

##  Appendix B Autoencoder Regularisation Schema

###  B.1. Wasserstein Autoencoders

Wasserstein autoencoders (WAEs), proposed by Tolstikhin et al. (Tolstikhin et
al., 2018), are an alternative method for regularising an autoencoder. Instead
of modelling points as distributions in the latent space and regularising
these individual distributions, WAEs use a regularisation term that encourages
the distribution of all latent space embeddings to match a Gaussian. Given $n$
training points $\\{x_{1},...,x_{n}\\}$, and $n$ points
$\\{z_{1},...,z_{n}\\}$ sampled independently from a standard Gaussian, we
calculate the following regularisation loss:

$\displaystyle\ell_{WAE\\_regularisation}=\frac{1}{n(n-1)}$
$\displaystyle\sum_{i\neq j}(E(x_{i})-E(x_{j}))^{2}$
$\displaystyle+\frac{1}{n(n-1)}\sum_{i\neq j}(z_{i}-z_{j})^{2}$
$\displaystyle-\frac{2}{n^{2}}\sum_{i,j}(E(x_{i})-z_{j})^{2}$

Figure 13. Visualising reconstructed gestures along the accelerometer z-axis
for autoencoders trained with different loss functions.

Tolstikhin et al. claim that WAEs retain the favourable qualities of VAEs
including stable training and nice latent manifold structure while achieving
higher quality reconstructions (which they demonstrate empirically for image
data).

###  B.2. VAE vs WAE

We trialled two approaches for regularising an autoencoder, namely the
variational (VAE) and Wasserstein autoencoders (WAE).

Both methods introduce a regularisation term into the autoencoder loss
function:

$\qquad\ell_{VAE}=\ell_{reconstruction}+\beta\cdot\ell_{regularisation}$

The hyperparameter $\beta$ controls the level of regularisation. We
exponentially varied the value of $\beta$ between 1 and $1e-6$ and trained a
VAE and WAE on the training dataset for each $\beta$.

We repeated the preliminary TSTR experiments of Appendix A using Conv+GRU with
the VAE and WAE models to investigate whether adding regularisation affected
the ability to reconstruct realistic gestures. These results are summarised in
Table 3.

Figure 14. Validation reconstruction loss for VAE-based models trained with
varying values of $\alpha$, showing the effect of additional weighting on the
authentication loss in our modified VAE loss function. Figure 15. Validation
reconstruction losses for VAEs trained with varying $\beta$, showing the
effect of additional weighting on the KL (regularisation) loss in the VAE loss
function

We can see that the regularisation had a slight effect, but that realistic
gestures are still being reconstructed. WAEs performed worse than VAEs, so we
proceeded with the VAE regularisation schema.

Table 3. TSTR gesture recognition results using Conv+GRU, shown for different
regularised autoencoders.
Regularisation |  AUROC |  EER
---|---|---
Unregularised |  0.92 |  0.16
VAE |  0.90 |  0.17
WAE |  0.87 |  0.20

##  Appendix C Effect of authentication loss

Figure 14 shows reconstruction losses for different values of $\alpha$. Figure
15 is discussed in Section 5.6.