# Distributed Monitoring for Data Distribution Shifts in Edge-ML Fraud
Detection

Nader Karayanni1, Robert J. Shahla2, Chieh-Lien Hsiao1

###### Abstract

The digital era has seen a marked increase in financial fraud. edge ML emerged
as a promising solution for smartphone payment services fraud detection,
enabling the deployment of ML models directly on edge devices. This approach
enables a more personalized real-time fraud detection. However, a significant
gap in current research is the lack of a robust system for monitoring data
distribution shifts in these distributed edge ML applications. Our work
bridges this gap by introducing a novel open-source framework designed for
continuous monitoring of data distribution shifts on a network of edge
devices. Our system includes an innovative calculation of the Kolmogorov-
Smirnov (KS) test over a distributed network of edge devices, enabling
efficient and accurate monitoring of users behavior shifts. We comprehensively
evaluate the proposed framework employing both real-world and synthetic
financial transaction datasets and demonstrate the framework’s effectiveness.

## Introduction

In recent years, smartphone payment services such as Apple Pay, smart wallets,
and banking apps have revolutionized financial transactions, offering
unprecedented convenience to users. However, these advancements have
significantly increased fraud risk, by methods including identity theft,
phishing, and smartphone theft (Bosamia and Patel 2019; Wang, Hahn, and
Sutrave 2016).

To combat these risks, payment service providers (referred to as operators)
increasingly rely on Machine Learning (ML) to identify and prevent fraudulent
activities with high accuracy (Omar, Fred, and Swaib 2018). Recently, there
has been increased adoption of on-device ML, also known as edge ML, which is a
paradigm where lightweight ML models are deployed directly on edge devices
such as smartphones, sensors, and IoT devices. One advantage to this approach
is the lower latency in detecting fraud since the inference doesn’t require
traveling over the network. Another advantage is enabling the deployment of a
personalized fine-tuned ML model per user, significantly enhancing the
understanding of user-specific spending patterns and other relevant variables,
therefore opening new avenues in improving the detection of fraudulent
activities (Sharma, Sharma, and Lal 2019; Chen et al. 2019).

ML-based fraud detection systems often operate under a critical assumption:
the data used for training these models closely mirrors the data encountered
in real-world applications. However, this assumption doesn’t always hold. Data
distribution shift is a phenomenon where the statistical properties of the
data distribution change over time. This change can possibly lead to a
significant undermining of the ML model’s performance, as the encountered data
no longer reflects the distribution on which the ML model was originally
trained (Widmer and Kubat 1994), denoted as originial training data. In the
context of financial transactions, as an example, consumer behavior changes
lead to shifts in spending patterns that ML models may not be trained to
handle, invalidating the assumption that the ML model original training data
mirrors the real-world encountered data (Dal Pozzolo et al. 2015).

Identifying the distribution shift is typically done via a proactive
monitoring process, where the newly incoming data distribution is continuously
monitored and compared against the original training data distribution. This
monitoring empowers the operators to promptly identify instances of data
distribution shift and take action accordingly, preventing prolonged
performance degradation of the ML models.

In the context of edge ML, monitoring shifts in data distribution becomes more
challenging. The newly incoming data is distributed across a network of edge
devices, and new samples are continuously being generated at each device.
Identifying distribution shifts across the edge devices requires consideration
of the data generated by the distributed edge devices. A monitoring system
deployed in the operator’s cloud cluster is tasked with aggregating and
monitoring the data distribution from these devices. The naïve solution is for
the edge devices to send each new sample to the cloud. However, this approach
is impractical due to its overhead for both the clients and the operator. The
transmission of a vast number of samples would demand excessive bandwidth and
runtime at the edge devices. Furthermore, the operator would need substantial
processing power to handle the volume of incoming data.

The challenge mentioned above remains underexplored. This work addresses this
gap, We introduce a novel framework for continuous monitoring of data
distribution shifts in such environments. Our primary contribution lies in the
design and implementation of this framework, with a specific emphasis on its
application in edge ML fraud detection scenarios. We conduct thorough
evaluations of our framework highlighting its efficiency and effectiveness.

## Background and Related Work

### Edge ML for Fraud Detection

The detection of fraudulent transactions is a crucial task in the finance
industry. Fraud Detection Systems (FDS) heavily rely on ML as a backbone for
classifying fraud. Researchers and industry leaders alike employ ML-based
methods, with neural networks being the most adopted method (Omar, Fred, and
Swaib 2018). RB and KR (2021) have demonstrated the effectiveness of neural
networks in classifying transactions as fraudulent or legitimate. Similarly,
FDS industry leaders such as Featurespace (2023) and ComplyAdvantage (2023),
heavily rely on ML for fraud detection.

Edge ML refers to the deployment of ML models on edge devices, such as
smartphones, IoT devices, and other computing resources, rather than in the
cloud. This approach provides many benefits, such as enabling the use of a
personalized fine-tuned ML model per edge device, minimizing latency,
conserving bandwidth, and preserving privacy. However, the primary drawback of
edge ML lies in the limited resources of edge devices and the absence of
centralized performance monitoring for the algorithms (Wang et al. 2022).

Edge ML is attractive in the context of mobile payments fraud detection. Chen
et al. (2005) have shown an effective way to build a personalized ML model
based on a questionnaire to assess the consumer’s behavior. Similarly Chen et
al. (2019) provided a fraud detection algorithm that leverages individual
behaviors to improve accuracy. Beyond personalization, edge ML provides other
advantages for financial applications. For example, Yang et al. (2019) have
utilized the privacy provided by edge ML to develop a federated learning-based
FDS that enables banks to collaboratively learn and improve fraud detection
models while keeping their data localized and secure.

### Handling Data Distribution Shift

Data distribution shift refers to the phenomenon where the distribution of the
newly observed data differs from the original training data of the ML model.
This shift can lead to a decline in the model’s performance as the original
training data no longer reflects the current real-world data distribution
(Widmer and Kubat 1994). In the dynamic world of financial transactions, and
with the continuous changes in the users and market behaviors, the phenomenon
of data distribution shift poses a significant challenge (Dal Pozzolo et al.
2018).

A key step to detect data distribution shift is the continuous monitoring of
incoming data. This process entails comparing the distribution of the new data
against the distribution of the original training data. A divergence between
these two distributions signals the occurrence of data distribution shift.
Detecting such shift is vital as it helps the operator recognize potential
declines in model performance, and take timely and informed actions
accordingly.

For cloud-based ML solutions, monitoring the incoming data for identifying
data distribution shifts is an essential step (AWS 2023; GCP 2023). NannyML
(2023) is a framework dedicated to monitoring cloud-based ML models’ post-
deployment. Monitoring data distribution shift is one of the critical methods
NannyML relies on to help operators. Besides monitoring the incoming data,
assessing data shifts can be done by utilizing the ML model’s outputs. For
example, Hendrycks and Gimpel (2017) and Hao et al. (2023) utilize the
outputted probabilities of the neural network, where systematic low model
confidence is correlated with distribution shift.

Figure 1: KS-statistic visualization (Wikipedia 2023).

### Two Distributions Comparison

A widely adopted test for comparing two distributions is the Goodness of Fit
(GoF) test. A GoF test can be used to statistically determine how well the
distribution of new data aligns with the distribution of the original training
data. This involves quantifying the differences between the two distributions.
Among the various GoF tests, the Kolmogorov-Smirnov (KS) test stands out as a
popular GoF test in ML applications (Massey 1951; Dos Reis et al. 2016). This
non-parametric test is favored for its distribution-free nature. The KS test
operates by calculating the maximum distance between the Cumulative
Distribution Function (CDF) of two distributions (Figure 1). Formally, if we
denote by $F_{1}$ and $F_{2}$ the CDFs of two distributions, then the KS
statistic is given by:

$KS(F_{1},F_{2})=\sup_{x}|F_{1}(x)-F_{2}(x)|$

A KS statistic close to 0 suggests that the two compared distributions are
aligned. Conversely, as the KS statistic increases, so does the indicated
divergence between the distributions, where 1 is the maximum KS statistic.

Recent work has been expanding the original KS test to suit the current ML
solutions. Lall (2015) and dos Reis et al. (2016) provide algorithms based on
the KS test, which are designed for monitoring streaming data. In Nguyen
(2018), an extended version of the KS test tailored for scenarios involving
big data was introduced. While the paper briefly touched upon the potential
extension to a distributed setting, the details were not discussed.

In this work, we focus on utilizing the KS test as our GoF test for monitoring
data distribution shifts. this decision is due to the high adoption and proven
effectiveness of the KS test. As an essential part of our framework, we
propose a novel calculation of the KS test in a distributed setting and
evaluate its efficiency and accuracy.

###  $t$-digest Data Structure

The $t$-digest (Dunning and Ertl 2019) data structure is designed to
dynamically represent a distribution based on continuously generated samples.
It represents the distribution with a series of centroids, each summarizing a
subset of the data. The $t$-digest creates and updates the centroids
dynamically according to the new samples’ distribution, Providing finer
resolution in areas with a high density of data points and coarser resolution
in areas of lower density. This adaptability makes it robust in representing
different data distributions with minimal memory footprint.

The $t$-digest offers two knobs, the $delta$ knob controls compression,
affecting both the size and accuracy of the data structure. A lower $delta$
results in higher compression but lower accuracy. The $k$ knob limits the size
of centroids, balancing quantile precision with memory usage.

## Proposed Framework

In this section, we introduce our framework designed to assist in effectively
monitoring data distribution shifts in edge ML deployments. Our framework
provides continuous distributed edge data aggregation and calculation of the
KS test for the aggregated data. A significant challenge in the framework is
calculating the KS statistic across a distributed network of devices, without
transmitting the entire data to the cloud. The solution is designed to
accurately and efficiently calculate the KS statistic while minimizing storage
and network bandwidth usage on edge devices.

To overcome the above challenges, each edge device maintains a compact data
structure, which encapsulates the distribution of samples generated within a
specific time window. Its primary function is to represent the CDF of the data
observed by the edge device using sketches optimized for minimal storage
overhead, efficiency, accuracy, and the ability to be merged with little to no
loss in accuracy.

To meet these needs, the $t$-digest data structure is used. It is efficient
both in its memory footprint and computation overhead, accurate, and can be
easily merged across devices to yield a global view of the data distribution.
Thereby facilitating the distributed calculation of the CDF. The merge is
carried out by iteratively combining the centroids of each $t$-digest, and
adjusting their weights and positions to reflect the combined data
distribution. The merged $t$-digest accurately represents the combined
distribution represented by the two original $t$-digest objects.

### High-level Overview

Each edge device stores a $t$-digest object, which is updated with every new
sample generated by the device. When $d$ samples are generated, the device
sends the $t$-digest object to the operator. Once the objects from the client
devices are received, the operator aggregates them into a unified $t$-digest
object. This unified object reflects the data distribution generated by the
entire network of devices. Consequently, the KS statistic can be calculated
once a sufficient number of samples have been received by the operator. The
system architecture includes two primary components (Figure 2).

Figure 2: Framework overview

  * •

Client: A client is deployed on each edge device and continuously tracks newly
generated samples. The goal of a client is to encapsulate the seen samples’
distribution efficiently, and eventually, send this data to the backend.

  * •

Backend: The backend manages a queue to which the clients’ data is added. In
addition, it analyzes the collected data and implements the monitoring.

## Framework Implementation

The framework is developed in Python and is available as an open-source
project on GitHub111https://github.com/karayanni/Distributed-KS-Test. The
repository includes all necessary code, detailed setups, and instructions for
replicating our experiments.

### Client

The client component of the framework is the first layer of the distributed
data monitoring system. This Python-based client code is lightweight, and
designed to be deployed on edge devices. The implementation of $t$-digest in
Davidson-Pilon (2023) was used. At its core, the client code is responsible
for managing the $t$-digest object which represents the sampled CDF by the
client. For every sample generated by the device, the $t$-digest object data
is updated. Note that updating the $t$-digest can also be done in batches by
grouping new samples and updating the $t$-digest object once per batch.

A key configurable parameter is the number of samples inserted into the
$t$-digest object, before transmitting it to the operator, denoted as $d$.
Once the number of processed samples surpasses this threshold, the client
initiates transmitting the data structure, along with its associated metadata
to the operator. This metadata contains information such as the used ML model
identifier, user payment location, age, etc.

### Backend

The backend architecture is designed in a serverless fashion, leveraging AWS
services for scalability and resource efficiency. The backend consists of the
following components.

Queue

Simple Queue Service (SQS) from AWS (2023), used as the data queue. All data
received from the clients is inserted into this queue in a FIFO manner.

Redis

To compute the aggregated CDF from multiple clients, a $t$-digest object is
maintained in a Redis (2023) instance, chosen for its fast in-memory data
handling capabilities. Multiple CDFs can be monitored by maintaining different
$t$-digest objects accordingly.

Lambda

Upon receiving data from a client, SQS triggers a Lambda function dedicated to
processing this data. The purpose of this function is to:

  1. 1.

Identify the proper $t$-digest object(s) (if there are multiple), based on the
provided metadata.

  2. 2.

Retrieve the relevant $t$-digest object(s).

  3. 3.

Merge the newly received $t$-digest object with the relevant existing one(s).
optimistic locking is used for concurrency control.

Figure 3 presents an example of handling a queue data insertion, where 4
different distributions are monitored, for 4 population types. An object
corresponding to a male, 65 years old from New York (NY) is received. Only the
objects of groups 1, 2, and 4 are updated in this case.

Figure 3: Handling queue message.

After merging the newly received $t$-digest object, the Lambda function
assesses if the updated $t$-digest object(s) represents a sufficient number of
samples, surpassing a pre-defined threshold, denoted as aggregation size. If
so, for each such object, it performs the KS test, comparing the current batch
distribution with the original training dataset distribution. The operators
can then integrate alerting logic for monitoring a distribution shift based on
a KS score threshold. Following the KS test, the $t$-digest object is reset
and is ready to repeat the aggregation process.

## Evaluation

In this section, we rigorously evaluate our framework, addressing the
following key aspects:

  1. 1.

The accuracy of our distributed KS statistic calculation.

  2. 2.

The effect of our KS calculation on false positive and false negative alerting
of distribution shift.

  3. 3.

The runtime overhead of the server.

  4. 4.

The runtime overhead for each client.

### Datasets

Our framework is designed to assist operators in detecting shifts in their
users’ financial data distribution. To accommodate the expected use cases, we
conducted experiments on datasets containing financial transactions. We use a
real-world dataset containing 284,807 credit card transactions from 2013, as
described in (Pozzolo et al. 2015), denoted as real-world dataset. We also use
two synthetic data sets, collectively comprising over 8 million simulated
financial transactions (Shenoy 2020; Roy 2021), denoted as synthetic datasets.
These datasets provide detailed insights into transaction amounts, cardholder
information, and fraud indicators. Closely mirroring the data expected to be
encountered by the operators.

### Accuracy

In real-world scenarios, aggregating all the data from edge devices to compute
the KS statistic is often impractical. This is why it’s crucial for our
framework to provide a precise estimation of the KS statistic, reflecting the
collective data observed by the edge devices. In this part of the evaluation,
we thoroughly compare the accuracy of the proposed distributed method, denoted
as T-Digest-KS, against the scenario where complete data is accessible,
denoted as Optimal-KS. The tests cover the impact our distributed computation
has on false positive and false negative identifications of data shifts. In
all our experiments, we use the default $t$-digest configurations,
specifically setting $delta$ to 0.01 and $K$ to 25.

#### Real-World Dataset Evaluation

This experiment evaluates the T-Digest-KS using the real-world dataset. The
objective is to determine how accurate the T-Digest-KS calculation of the KS
statistic is, with different data shift degrees.

Due to the dataset’s small size, a random 100,000 transactions are used as the
original data. Another 100,000 transactions, selected randomly from the
remaining pool, served as our newly encountered data, representing newly
encountered transaction patterns. To simulate shift, we introduced random
increases in the spending patterns with varying degrees of increase to result
in different levels of shift. For each degree of shift, the process was
repeated 15 times.

Figure 4: Real-world dataset accuracy with different shifts.

Figure 4 depicts the experiment results. The Y-axis represents the absolute
difference between the KS statistic calculated by T-Digest-KS and Optimal-KS.
The X-axis represents the KS value range of the Optimal-KS across the repeated
runs. The results indicate a consistent low error in the T-Digest-KS
estimations. The median error remained below 0.004. Interestingly, in cases
with slight to significant shifts, T-Digest-KS demonstrated even higher
accuracy, with a median error rate below 0.002. The explanation for this
phenomenon can be attributed to the lossy-ness of the $t$-digest data
structure, and the definitions of the KS statistic, which identifies the
greatest divergence in the CDFs of two distributions.

In instances with no shift, the difference between the two CDFs is
predominantly zero, and the $t$-digest’s lossy-ness becomes the primary factor
in determining the discrepancy between the CDFs. However, in scenarios with
data shifts, the differences in CDFs are more pronounced. $t$-digest’s lossy-
ness is comparatively less impacting because it affects a smaller proportion
of critical comparisons (points where the difference in the CDFs is a
potential candidate to be the KS statistic).

#### Immediate Shift in Users Behavior

An immediate shift is a distribution shift due to a certain percentage of
users drastically altering their behavior. Such scenarios are particularly
relevant for fraud detection ML models. For instance, a model developed using
data prior to COVID-19 might not perform accurately during the pandemic where
immediate distribution shifts occur to consumer behavior (Loxton et al. 2020).
We assess the accuracy of our system using the synthetic datasets mentioned
above, which emulate distinct yet realistic spending scenarios. We harness the
large dataset size to calculate the KS statistic using 500,000 samples per
test. The test involves five varied percentages of users exhibiting immediate
shifts, repeating each test 15 times.

Figure 5: Accuracy with a percentage of shifted users.

As depicted in Figure 5, T-Digest-KS shows remarkable accuracy in this set of
tests. The enhanced performance can be attributed to three key differences:
the increased number of samples per test, differences in the distribution
characteristics, and the nature of the data distribution shift.

#### Takeaways

Building on the findings from the two experiments, T-Digest-KS exhibits a
notably low error in estimating the KS statistic. In standard practice, a KS
statistic exceeding the range of 0.03-0.05 is often indicative of a
significant change in data distribution, as established in (Massey 1951).

Our comprehensive evaluation underscores that when the KS statistic remains
below the 0.01 threshold, the median error in T-Digest-KS estimations is
consistently under 0.004 across both tests and the max error measured is below
0.006. This low error significantly minimizes the likelihood of false
positives in identifying distribution changes. Moreover, in instances where
the KS statistic surpasses the 0.01 threshold, T-Digest-KS demonstrates an
even lower error margin. This heightened accuracy further reduces the risk of
false negatives.

These results highlight the reliability and precision of T-Digest-KS in
detecting shifts in data distribution, ensuring a high degree of confidence in
its practical application, particularly in scenarios where accurate detection
of data shifts is critical.

### Runtime

This section assesses the runtime associated with adopting our framework of
maintaining a $t$-digest object on each edge device and merging them in the
cloud, denoted as T-Digest-Merge. As a baseline, a batch-streaming approach
was employed. In this method, clients transmit data to the server in batches,
without maintaining a $t$-digest object, and the samples are then inserted to
the $t$-digest object in the cloud. This method is denoted as T-Digest-Stream.

To ensure a fair comparison, the batch size in T-Digest-Stream was set to
1,000 samples, mirroring our system’s memory footprint on the client side.
This consideration is essential for maintaining a balance between performance
and resource consumption on client devices.

We examine the overhead for both the backend server and the client code
running on the edge devices. All experiments use random samples from the real-
world dataset. The results were averaged across 10 runs. The tests were
performed on a system with an 11th Gen Intel Core i7 processor, 3.00GHz, and
16GB of memory. To simulate cloud-based components, LocalStack (Mishra 2023)
was used, enabling us to mimic AWS services locally.

#### Backend Runtime

This evaluation focuses on the runtime overhead of our backend server,
including all processes from the receipt of client data in the queue to the
completion of the KS calculation. The client stores $d=20,000$ samples, and
only then sends them to the cloud. The reference dataset comprises randomly
selected 100,000 samples from the real-world transactions dataset.

Figure 6: Backend cycle runtime by server aggregation size

Figure 6 demonstrates the backend runtime as a function of the aggregation
size. The backend’s runtime scales linearly with the increase in aggregation
size. This linear trend is indicative of the system’s ability to scale
efficiently under varying loads. Additionally, a notable improvement is
observed when compared to T-Digest-Stream, which reaches a ratio of $1.5$ at
aggregation size of 5 million between the T-Digest-Stream and T-Digest-Merge.

#### Client Runtime

Given the deployment of the client on edge devices, which have limited
computational resources, our framework must operate with minimal resource
consumption. This experiment compares T-Digest-Merge runtime on the client
side against T-Digest-Stream, considering different client batch sizes. The
client runtime includes data structure management, updating with new samples,
and the processes of serialization of the data structure and transmission over
the network.

Figure 7: Client runtime as a function of $d$

Figure 7 depicts the client runtime as a function of $d$, the number of
samples stored at the client before sending them to the cloud. T-Digest-Merge
demonstrates superior client runtime performance compared to T-Digest-Stream.
Notably, even though our method involves managing the $t$-digest structure on
the client, it remains more efficient overall. This efficiency is attributed
to the reduced network overhead from sending less data, and to the high
efficiency of the $t$-digest operations.

## Discussion and Future Work

This research introduces a framework for monitoring data distribution shifts
in edge ML deployments, specifically aimed to enable edge ML fraud detection.
Our findings demonstrate the framework’s accuracy and efficiency in real-world
settings, highlighting its potential as a vital component for fraud detection
using edge ML. The rapid growth in IoT devices and the widespread adoption of
smartphones with increasing capabilities underscore the significance of edge
ML, particularly in the context of fraud detection.

While edge ML is promising, it also brings forth significant challenges that
require addressing. The proposed framework, though effective in certain
aspects, falls short of providing a holistic solution to the array of
challenges in this new paradigm. This shortfall is also seen in the fraud
detection use case. Specifically, in fraud detection, it’s not sufficient to
monitor data distribution shifts, as the data faced is highly imbalanced, and
legitimate transactions significantly outnumber fraudulent ones. Thus, shifts
in the overall data distribution might overlook critical changes in fraudulent
activity patterns. Such scenarios are undetectable in the proposed framework,
as the data distribution shift will be negligible.

To tackle these challenges effectively, future research must focus on
developing a holistic solution. One that reflects the diverse strategies used
in cloud monitoring systems. A robust solution for edge ML needs to
incorporate a variety of methods, creating a framework that offers operators a
comprehensive overview and addresses the various obstacles. Our work is
envisioned to be an integral component of these future solutions, contributing
to the development of edge ML operations.

An interesting avenue for future research is extending our framework to offer
privacy-preserving guarantees. Given the critical importance of privacy in
certain edge ML use cases, developing methods to accurately measure and
monitor collective data distributions while maintaining privacy is essential.
Another interesting improvement of our system is incorporating a sliding
window approach to continuously calculate the aggregated distribution. This
approach would reduce the latency of detecting a distribution shift.

## Conclusion

In this study, we developed a novel framework aimed at monitoring data
distribution shifts in edge ML applications, with a specific focus on fraud
detection. In particular, we proposed a novel implementation of the KS test
over a distributed network of edge devices. Our extensive evaluations
underscore the framework’s accuracy and efficiency, confirming its
effectiveness for practical, real-world deployment. This research marks a step
towards building a comprehensive edge ML monitoring solution.

## References

  * AWS (2023) AWS. 2023.  Detecting data drift using Amazon SageMaker.  [Online; accessed 24-November-2023].
  * AWS (2023) AWS. 2023.  SQS.  https://aws.amazon.com/sqs/.  Accessed: Nov 24, 2023.
  * Bosamia and Patel (2019) Bosamia, M.; and Patel, D. 2019.  Wallet payments recent potential threats and vulnerabilities with its possible security measures.  _Int. J. Comput. Sci. Eng_ , 7(1): 810–817.
  * Chen et al. (2019) Chen, L.; Zhang, Z.; Liu, Q.; Yang, L.; Meng, Y.; and Wang, P. 2019.  A Method for Online Transaction Fraud Detection Based on Individual Behavior.  In _Proceedings of the ACM Turing Celebration Conference - China_ , ACM TURC ’19. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450371582.
  * Chen et al. (2005) Chen, R.-C.; Luo, S.-T.; Liang, X.; and Lee, V. 2005.  Personalized Approach Based on SVM and ANN for Detecting Credit Card Fraud.  In _2005 International Conference on Neural Networks and Brain_ , volume 2, 810–815.
  * ComplyAdvantage (2023) ComplyAdvantage. 2023.  complyadvantage.  https://complyadvantage.com/.  Accessed: Nov 24, 2023.
  * Dal Pozzolo et al. (2015) Dal Pozzolo, A.; Boracchi, G.; Caelen, O.; Alippi, C.; and Bontempi, G. 2015.  Credit card fraud detection and concept-drift adaptation with delayed supervised information.  In _2015 International Joint Conference on Neural Networks (IJCNN)_ , 1–8.
  * Dal Pozzolo et al. (2018) Dal Pozzolo, A.; Boracchi, G.; Caelen, O.; Alippi, C.; and Bontempi, G. 2018.  Credit Card Fraud Detection: A Realistic Modeling and a Novel Learning Strategy.  _IEEE Transactions on Neural Networks and Learning Systems_ , 29(8): 3784–3797.
  * Davidson-Pilon (2023) Davidson-Pilon, C. 2023.  T-Digest data structure in Python.  https://github.com/CamDavidsonPilon/tdigest.  Accessed: 2023-11-11.
  * Dos Reis et al. (2016) Dos Reis, D. M.; Flach, P.; Matwin, S.; and Batista, G. 2016.  Fast unsupervised online drift detection using incremental kolmogorov-smirnov test.  In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , 1545–1554.
  * dos Reis et al. (2016) dos Reis, D. M.; Flach, P.; Matwin, S.; and Batista, G. 2016.  Fast Unsupervised Online Drift Detection Using Incremental Kolmogorov-Smirnov Test.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’16, 1545–1554. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450342322.
  * Dunning and Ertl (2019) Dunning, T.; and Ertl, O. 2019.  Computing extremely accurate quantiles using t-digests.  _arXiv preprint arXiv:1902.04023_.
  * Featurespace (2023) Featurespace. 2023.  featurespace.  https://www.featurespace.com/.  Accessed: Nov 24, 2023.
  * GCP (2023) GCP. 2023.  Monitor feature skew and drift.  [Online; accessed 24-November-2023].
  * Hao et al. (2023) Hao, W.; Wang, Z.; Hong, L.; Li, L.; Karayanni, N.; Mao, C.; Yang, J.; and Cidon, A. 2023.  Monitoring and Adapting ML Models on Mobile Devices.  _arXiv preprint arXiv:2305.07772_.
  * Hendrycks and Gimpel (2017) Hendrycks, D.; and Gimpel, K. 2017.  A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks.  In _International Conference on Learning Representations_.
  * Lall (2015) Lall, A. 2015.  Data streaming algorithms for the Kolmogorov-Smirnov test.  In _2015 IEEE International Conference on Big Data (Big Data)_ , 95–104.
  * Loxton et al. (2020) Loxton, M.; Truskett, R.; Scarf, B.; Sindone, L.; Baldry, G.; and Zhao, Y. 2020.  Consumer Behaviour during Crises: Preliminary Research on How Coronavirus Has Manifested Consumer Panic Buying, Herd Mentality, Changing Discretionary Spending and the Role of the Media in Influencing Behaviour.  _Journal of Risk and Financial Management_ , 13(8).
  * Massey (1951) Massey, F. J. 1951.  The Kolmogorov-Smirnov Test for Goodness of Fit.  _Journal of the American Statistical Association_ , 46(253): 68–78.
  * Mishra (2023) Mishra, H. 2023.  LocalStack’s Docker Extension.  https://blog.localstack.cloud/2023-01-13-introducing-localstack-extension-for-docker-desktop/.  Accessed: Nov 24, 2023.
  * NannyML (2023) NannyML(2023). 2023.  NannyML (release 0.9.1).  https://github.com/NannyML/nannyml.  Accessed: Nov 24, 2023.
  * Nguyen (2018) Nguyen, H. D. 2018.  A Two-Sample Kolmogorov-Smirnov-Like Test for Big Data.  In Boo, Y. L.; Stirling, D.; Chi, L.; Liu, L.; Ong, K.-L.; and Williams, G., eds., _Data Mining_ , 89–106. Singapore: Springer Singapore.  ISBN 978-981-13-0292-3.
  * Omar, Fred, and Swaib (2018) Omar, S. J.; Fred, K.; and Swaib, K. K. 2018.  A State-of-the-Art Review of Machine Learning Techniques for Fraud Detection Research.  In _Proceedings of the 2018 International Conference on Software Engineering in Africa_ , SEiA ’18, 11–19. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450357197.
  * Pozzolo et al. (2015) Pozzolo, A. D.; Caelen, O.; Johnson, R. A.; and Bontempi, G. 2015.  Calibrating Probability with Undersampling for Unbalanced Classification.  In _2015 IEEE Symposium Series on Computational Intelligence_ , 159–166.
  * RB and KR (2021) RB, A.; and KR, S. K. 2021.  Credit card fraud detection using artificial neural network.  _Global Transitions Proceedings_ , 2(1): 35–41.  1st International Conference on Advances in Information, Computing and Trends in Data Engineering (AICDE - 2020).
  * Redis (2023) Redis. 2023.  Redis.io.  [Online; accessed 24-November-2023].
  * Roy (2021) Roy, R. 2021.  Online Payments Fraud Detection Dataset.  https://www.kaggle.com/datasets/rupakroy/online-payments-fraud-detection-dataset/data.
  * Sharma, Sharma, and Lal (2019) Sharma, B.; Sharma, L.; and Lal, C. 2019.  Anomaly Detection Techniques using Deep Learning in IoT: A Survey.  In _2019 International Conference on Computational Intelligence and Knowledge Economy (ICCIKE)_ , 146–149.
  * Shenoy (2020) Shenoy, K. 2020.  Credit Card Transactions Fraud Detection Dataset.  https://www.kaggle.com/datasets/kartik2112/fraud-detection/.
  * Wang, Hahn, and Sutrave (2016) Wang, Y.; Hahn, C.; and Sutrave, K. 2016.  Mobile payment security, threats, and challenges.  In _2016 Second International Conference on Mobile and Secure Services (MobiSecServ)_ , 1–5.
  * Wang et al. (2022) Wang, Y.; Wang, J.; Zhang, W.; Zhan, Y.; Guo, S.; Zheng, Q.; and Wang, X. 2022.  A survey on deploying mobile deep learning applications: A systemic and technical perspective.  _Digital Communications and Networks_ , 8(1): 1–17.
  * Widmer and Kubat (1994) Widmer, G.; and Kubat, M. 1994.  Learning in the Presence of Concept Drift and Hidden Contexts.  _Machine Learning_ , 23.
  * Wikipedia (2023) Wikipedia. 2023.  Kolmogorov–Smirnov test — Wikipedia, The Free Encyclopedia.  [Online; accessed 24-November-2023].
  * Yang et al. (2019) Yang, W.; Zhang, Y.; Ye, K.; Li, L.; and Xu, C.-Z. 2019.  FFD: A Federated Learning Based Method for Credit Card Fraud Detection.  In Chen, K.; Seshadri, S.; and Zhang, L.-J., eds., _Big Data – BigData 2019_ , 18–32. Cham: Springer International Publishing.  ISBN 978-3-030-23551-2.