# Financial Fraud Detection using Quantum Graph Neural Networks

Nouhaila<EMAIL_ADDRESS>Abhishek Sawaika3,
Ashim Dhor4, Siddhant Dutta5, Sairupa Thota6, Husayn Gokal7, Nandan Patel8,
Muhammad Al-Zafar<EMAIL_ADDRESS>Ioannis Theodonis10
and Mohamed Bennai1 0000-0002-1014-3457 0000-0002-1147-7782 1 Quantum Physics
and Magnetism Team, LPMC, Faculty of Sciences Ben M’sick, Hassan II University
of Casablanca, Morocco  2 Quantum Formalism Fellow, Zaiku Group Ltd,
Liverpool, United Kingdom  3 Indian Institute of Technology, Bombay, India  4
Department of Data Science and Engineering, Indian Institute of Science
Education and Research Bhopal, India  5 Dwarkadas J. Sanghvi College of
Engineering  6 Guru Nanak Institutions Technical Campus, Hyderabad, India  7
ASTI Academy, Dubai, UAE  8 Vellore Institute of Technology, Chennai, India  9
Robotics, Autonomous Intelligence, and Learning Laboratory (RAIL), School of
Computer Science and Applied Mathematics, University of the Witwatersrand, 1
Jan Smuts Ave, Braamfontein, Johannesburg 2000, Gauteng, South Africa  10
Department of Physics, School of Applied Mathematical and Physical Sciences,
National Technical University of Athens, Greece.

###### Abstract

Financial fraud detection is essential for preventing significant financial
losses and maintaining the reputation of financial institutions. However,
conventional methods of detecting financial fraud have limited effectiveness,
necessitating the need for new approaches to improve detection rates. In this
paper, we propose a novel approach for detecting financial fraud using Quantum
Graph Neural Networks (QGNNs). QGNNs are a type of neural network that can
process graph-structured data and leverage the power of Quantum Computing (QC)
to perform computations more efficiently than classical neural networks. Our
approach uses Variational Quantum Circuits (VQC) to enhance the performance of
the QGNN. In order to evaluate the efficiency of our proposed method, we
compared the performance of QGNNs to Classical Graph Neural Networks using a
real-world financial fraud detection dataset. The results of our experiments
showed that QGNNs achieved an AUC of $0.85$, which outperformed classical
GNNs. Our research highlights the potential of QGNNs and suggests that QGNNs
are a promising new approach for improving financial fraud detection.

  *

Keywords: Quantum Machine Learning, Quantum Graph Neural Network, Variational
Quantum Circuit, Fraud Detection

##  1 Introduction

In today’s dynamic technological landscape, the rapid surge in digital
transactions and financial activities has ushered in unprecedented
conveniences. However, this digital transformation has also fostered a
parallel escalation in financial fraud, posing significant threats to global
industries and economies. As financial operations increasingly adopt digital
complexities, the susceptibility to exploitation from malicious actors
increases, leading to the pursuit of innovative fraud detection methods [1].

The importance of fraud detection extends beyond a single industry, resonating
across diverse sectors, including Banking, e-Commerce, Healthcare, FinTech,
and Insurance. The integrity of financial transactions is pivotal for optimal
functionality in these domains, making robust fraud detection systems
indispensable. The banking sector, exemplifying the forefront of this
challenge, confronts expanding attack surfaces due to rising online and mobile
banking adoption [2]. Conventional rule-based approaches, while partially
effective, struggle to match the ever-evolving tactics of fraudsters. This
necessitates advanced, adaptable techniques in order to keep pace with the
dynamic landscape of financial fraud [3].

The research community is constantly exploring new ways to detect fraud, as
the methods used by fraudsters become increasingly sophisticated. An emerging
avenue combines Quantum Machine Learning (QML), and Graph Neural Networks
(GNNs); a groundbreaking approach poised to revolutionise financial fraud
detection [4]. By harnessing the distinct attributes of Quantum Computing (QC)
–- superposition and entanglement –- and integrating them with the
effectiveness of GNNs, this hybrid paradigm endeavours to tackle the intricate
and ever-changing realm of financial fraud. In this paper, we investigate the
challenges of financial fraud detection and the potential of Quantum Graph
Neural Networks (QGNNs) as a transformative approach to address these
challenges. We conduct comprehensive analyses and experiments to demonstrate
the efficacy and viability of QGNNs for fraud detection. Our findings
contribute to the discourse on advanced fraud detection techniques and pave
the way for enhanced security in the financial ecosystem.

QML emerges at the intersection of QC and conventional Machine Learning (ML)
techniques, offering promising avenues for solving complex problems in various
domains. In the context of financial fraud detection, QML presents an
intriguing approach that harnesses the inherent power of quantum computations
to address the intricate challenges posed by fraudulent activities within the
financial sector. By leveraging QGNNs, the combination of quantum and
graphical techniques offers a novel framework for analysing and classifying
financial transactions.

In financial fraud detection, QML encodes transactional data into quantum
states. This can improve the efficiency and accuracy of fraud detection by
enabling efficient parallel processing and enhanced feature representation.
The unique property of quantum entanglement facilitates the capture of
intricate relationships among transactional features. This fosters a more
comprehensive understanding of fraudulent patterns.

However, this emerging paradigm also poses significant challenges, including
the interpretability of quantum computations and concerns over quantum system
security. Nonetheless, the promising outcomes showcased in this study
highlight the potential of QML to significantly impact fraud detection by
improving accuracy and scalability, opening the door to a new era of enhanced
financial security. The inherent ability of quantum systems to process and
model complex relationships among variables aligns well with the intricacies
of financial transactions.

In the realm of fraud detection, conventional methods often struggle to
navigate the intricate tapestry of relationships and intricate patterns that
underlie fraudulent activities. As the complexity and diversity of fraudulent
schemes continue to evolve, the demand for adaptable and nuanced detection
techniques becomes increasingly apparent. In response, graph-based methods
have emerged as a promising avenue, harnessing the intrinsic strength of graph
structures to capture and dissect the multifaceted interconnections that
define fraudulent behaviours.

Built upon the foundation of the mathematical study of graph theory, these
methods provide an intuitive and comprehensive framework to model, depict, and
analyse the labyrinthine-like relationships among entities. From individuals
and accounts to transactions and organisations, a broad spectrum of entities
can be seamlessly encapsulated within interconnected nodes and edges. This
unique arrangement empowers graph structures to illustrate the intricate
dynamics inherent in fraudulent schemes vividly. Diverging from conventional
approaches, which rely on isolated data points and predetermined rules, graph-
based methods can unveil concealed connections and dependencies, thereby
unveiling anomalous or suspicious behaviours that might go unnoticed [5].

The true effectiveness of graph structures in fraud detection lies in their
unparalleled ability to uncover latent patterns that elude customary detection
techniques. Fraudulent endeavours rarely occur in isolation; rather, they
unravel through intertwined actions involving multiple entities. These
connections, often obscured by noise and obfuscation, are revealed through
meticulous graph analysis. By representing and exploring these relationships,
graph-based methods empower fraud detection systems to transcend the confines
of isolated data points, capturing the holistic context in which fraudulent
activities manifest and thrive.

This quantum potential extends seamlessly to the detection of financial fraud,
where quantum-inspired techniques can transform the landscape. By integrating
the strengths of quantum mechanics, QC could potentially revolutionise
classical ML algorithms used for fraud detection. Techniques such as Quantum
Neural Networks (QNNs) can harness the property of quantum parallelism to
process and analyse financial data more efficiently. Exploiting superposition,
these networks can simultaneously examine multiple data points, accelerating
tasks like pattern recognition and data classification, crucial elements in
detecting fraudulent activities. Furthermore, quantum feature mapping enhances
the ability to identify intricate patterns in financial transactions that
might evade classical methods. This intersection of QML and financial fraud
detection promises a future where quantum-inspired techniques amplify the
accuracy and efficiency of identifying and combating fraudulent behaviours.

The integration of quantum properties into graph-based fraud detection
methodologies is motivated by the imperative to confront the escalating
sophistication of fraudulent activities and the inherent limitations of
classical computing paradigms. In today’s landscape of intricate financial
systems, fraudulent behaviours manifest intricate and interwoven patterns that
defy conventional computational approaches. These conventional methods
struggle to unravel the complexities inherent in fraud networks, marked by
exponential data growth and non-linear dynamics. As a result, a critical need
arises for innovative techniques that transcend classical computing
boundaries, leading us to explore the distinctive attributes of QC.

While graph-based fraud detection methods have shown their ability to capture
the intricate relationships within fraudulent activities, they face
limitations in handling the explosive data growth and complex dynamics of
fraud networks [6]. The conventional algorithms grounded in classical
computing struggle to navigate the intricate patterns that define these
networks, often characterised by non-linear structures and extensive solution
spaces. This disparity underscores a compelling necessity for pioneering
methodologies capable of transcending the confines of classical computation.
By delving into quantum properties, we seek to bridge this gap, illuminating
concealed fraud schemes and redefining the boundaries of fraud detection in
the context of modern financial systems [7].

The primary objective of this study is to conduct a comprehensive
investigation into the effectiveness of QGNNs for financial fraud detection.
Specifically, we aim to comprehensively assess and analyse the potential
advantages and enhancements that QGNNs offer in contrast to classical GNNs,
specifically concerning their ability to identify and mitigate fraudulent
activities within intricate networks of financial transactions. By
meticulously evaluating and comparing the performance, adaptability, and
overall effectiveness of QGNNs benchmarked against classical GNNs, we intend
to shed light on the transformative possibilities of leveraging QC for
refining fraud detection methodologies.

Through this research endeavour, we aspire to comprehensively understand QGNNs
potential to enhance the landscape of fraud detection strategies [8]. By
closely examining the strengths and constraints of QGNNs _vis-á-vis_ classical
GNNs, our study seeks to contribute meaningful insights to the QC and fraud
detection disciplines.

The structure of this paper is as follows:

  * •

Sec. 2, we present a comprehensive literature review on financial fraud
detection, graph neural networks, and QML. We highlight the fundamental
concepts of GNNs and emphasise the limitations of conventional methods.

  * •

In Sec. 3, we examine the theory of GNNs and different models, focusing on
GraphSAGE architecture.

  * •

In Sec. 4, we conduct a comprehensive study of our proposed quantum graph
neural network model for financial fraud detection.

  * •

In Sec. 5, we provide a comprehensive understanding of our selected dataset
characteristics, including attribute highlighting, class imbalance, and
preprocessing steps.

  * •

In Sec. 6, we evaluate the performance of QGNN and GNN on a real-world
financial fraud dataset and discuss the results of our experiments.

  * •

In Sec. 7, we summarise the key findings of our study and suggest directions
for future research.

##  2 Literature Review

Transitioning our inquiry from the domain of conventional fraud detection
techniques, we direct our attention towards the emerging promise of QGNNs.
This transition represents a notable departure from established methodologies
and carries the potential to reshape our understanding and implementation of
fraud detection strategies. In adopting this advanced technological paradigm,
we aim to uncover novel insights and methodologies that may prove instrumental
in addressing the complexities of contemporary fraud detection challenges.

In the classical computing area, there exists a plethora of noteworthy
research. We highlight some of the outstanding application-based research
works below.

Ma et al., $2020$ [9] provide an in-depth exploration of using Deep Learning
(DL) techniques to detect anomalies in graph data. It emphasises the
importance of anomaly detection in domains such as security, finance, and
medicine. The paper highlights how conventional methods struggle with graph
data complexities, leading to the rise of DL approaches. The authors
systematically categorise and analyse contemporary techniques for graph
anomaly detection based on their ability to identify different types of
anomalous graph objects. The survey underscores the challenges inherent in
this field and identifies future research directions.

In today’s rapidly evolving technological landscape, we are witnessing a
growing variety of ways in which fraudulent transactions take place. This
makes it crucial to understand how we detect fraud and to determine which
approach works best to get accurate results. Researchers are on a widespread
quest to explore different QML models that fit well for spotting fraud.

In [10], Ahmad et al., $2021$ explore the combination of QC and Deep Neural
Networks (DNNs) to address network security through anomaly detection.
Emphasising the transformative potential of QC, it develops a DNN-based
quantum autoencoder to process quantum-encoded data. Applying dimensionality
reduction techniques, the research enhances the model’s ability to detect
subtle anomalies within network data. The results reveal impressive accuracy
rates of $100\%$ during training and $99\%$ during testing, demonstrating the
framework’s efficacy in capturing patterns and its potential to revolutionize
anomaly detection in network security. This innovative research establishes a
symbiotic relationship between QC and DL, offering a novel avenue for
cybersecurity advancements.

In [11], Zheng et al., $2021$ introduce a Quantum Graph Convolutional Neural
Network (QGCNN) model to address the challenge of applying a QNN to non-
Euclidean spatial data, particularly graph-structured data. The QGCNN model
efficiently captures graph topology and node features using quantum parametric
circuits, making graph-level classification tasks effective. The study bridges
the gap in QML research by extending QNNs to handle irregular spatial
structures in graph data. The proposed QGCNN model showcases the potential of
QC in enhancing graph-based ML tasks, contributing to the advancement of QML
research.

QGNNs offer a transformative approach to fraud detection in the financial
sector, addressing the escalating complexity of fraudulent schemes. By
leveraging their capacity for intricate pattern recognition, enhanced data
representation, and adaptability to evolving strategies, these networks hold
the potential to significantly enhance the accuracy of fraud detection while
mitigating false positives.

The study by Kyriienko et al., $2022$ [12] focuses on utilising QML to enhance
the detection of internal fraud in the financial sector, particularly
fraudulent activities conducted by individuals within an organisation. The
study employs quantum kernels generated through quantum feature maps based on
Instantaneous Quantum Polynomial (IQP) circuits to transform classical data
from credit card transactions into a higher-dimensional space for analysis.
Applying the one-class support vector machine (OC-SVM) algorithm within this
quantum feature space enables to detection of abnormal transactions.
Simulations confirm the effectiveness of this quantum kernel-based approach in
detecting internal fraud, highlighting the potential of QC to improve fraud
detection accuracy in complex financial data.

Recent research efforts, such as [13], Innan et al., $2023$ investigate the
utility of QML models in enhancing fraud detection within the financial
sector. The authors address the pressing need for accurate fraud detection and
propose leveraging QML techniques to improve system efficiency. Through a
comparative analysis of four QML models –- Quantum Support Vector Classifier,
Variational Quantum Classifier, Estimator Quantum Neural Network, and Sampler
Quantum Neural Network –- the study assesses their performances in detecting
financial fraud. Employing a synthetic dataset derived from BankSim, the
authors employ data preprocessing, logical analysis, and visualisations to
identify key discriminatory features. The QSVC emerges as the most effective
model, yielding high F1 scores, exceeding $98\%$ for both fraud and non-fraud
classes. The study highlights the potential of QML in fraud detection,
providing insights into its capabilities and implications while acknowledging
areas for future advancement.

In [14], Xiang et al., $2023$ address the challenge of credit card fraud
detection by introducing a novel semi-supervised approach utilising attribute-
driven graph representation and a Gated Temporal Attention Network (GTAN). The
proposed method effectively enhances fraud detection performance, especially
with limited labelled data, according to the study’s key findings. The authors
constructed a temporal-transaction graph based on transaction records,
incorporating temporal transactions and interactions, and utilised GTAN for
message passing and learning transaction representations. Risk patterns are
further modelled through risk propagation amongst transactions. The research
showcases that GTAN outperforms state-of-the-art baselines in detecting fraud
across multiple datasets, demonstrating its ability to perform well with only
a small fraction of labelled data. The study contributes a comprehensive
framework that amalgamates attribute-driven embeddings, temporal attention
mechanisms, and risk propagation to advance fraud detection methodologies,
offering promising avenues for future research and innovation in the field.

In [15], Schetakis et al., $2023$ explore the application of fraud detection
for small and medium-sized enterprises. The study employs a hybrid approach,
comparing quantum classifiers with classical Artificial Neural Networks (ANN),
demonstrating more efficient training and potential advantages in credit
scoring. The study focuses on a classical-quantum hybrid model’s performance
using quantum classifiers and classical ANNs, revealing significant efficiency
improvements with promising outcomes in credit scoring for SMEs.

In [16], Beer et al., $2023$ delve into the realm of QML by investigating the
use of quantum neural networks for processing graph-structured quantum data.
Addressing challenges and opportunities in the Noisy Intermediate-Scale
Quantum (NISQ) devices era, the study introduces a novel concept of
associating quantum states with graph vertices and correlations with edges.
The paper proposes using modified loss functions and quantum training
algorithms to learn and characterise complex graph structures with QNNs. The
research contributes to the taxonomy of QML approaches, emphasising the
significance of correlations and graph structure in advancing quantum
information processing capabilities for various applications.

##  3 Classical Graph Neural Networks

In this section, we present the theoretical foundations of classical graph
neural networks, focusing on the type used in this study. Firstly, we briefly
describe the mechanics of ANNs and thereafter describe the GNNs.

Artificial Neural Networks are a class of ML algorithms whose structure is
derived from the workings of the Biological neurons in sentient beings.
Analogous to how the neurological systems of these beings transmit information
from one neuron to another, the ANN consists of different layers of neurons
that transmit information. Below, we clarify further.

  * •

Neurons: These are the basic units of computation in the ANN. Each neuron
consists of an input, hidden, and output layer.

    * –

The input layers ingest the initial data.

    * –

The hidden layer transforms the data received by the input layer and performs
certain computations.

    * –

The output layer then receives the transformed data from the hidden layer and
returns the predictive result.

  * •

Weights and Biases: Each connection between neurons has an associated weight,
$w_{ij}\in\left[0,1\right]$, between nodes $i$ and $j$ respectively, which
indicates the strength of connections between neurons and is learned and
updated during the training process. The actual interpretation of weights is
different for different types of ANNs.

In an ANN, a bias term is added to the weighted sum of the inputs to a neuron
before the activation function is applied. The bias term allows the neuron to
learn a non-linear decision boundary, which is necessary for ANNs to learn
complex functions.

Figure 1: The basic structure of an ANN shows the weights and bias along with
the respective layers.

In Fig. 1, the inputs are represented by $x_{1},x_{2},x_{3}$, and the weights
are represented by $w_{1},w_{2},w_{3}$ shown in a summation form in the second
layer.

In terms of the bias, $b$, the output layer is expressed as

$\displaystyle\hat{y}=\sum_{i}w_{i}x_{i}+b=w_{1}x_{1}+w_{2}x_{2}+w_{3}x_{3}+b.$
(1)

  * •

Activation Functions: The ANNs consist of an activation function that
transforms the given output non-linearly. Several examples of activation
functions include the sigmoid, the ReLU and its variations, the hyperbolic
tangent, the swish, etc., as shown in Fig. 2. The choice of each of these
functions is an artefact of the problem at hand that one is trying to solve,
and each has its advantages and disadvantages.

Figure 2: Plot showing the ReLU, Sigmoid, and Tanh activation functions along
with vertical and horizontal lines at $x=0$ and $y=0$, respectively.

  * •

Final Output: The final layer of a basic ANN is the output layer, which yields
the output of the computations from the hidden layers.

As we discussed, artificial neural networks are ML models that can be used to
learn from data. However, ANNs cannot capture the relationships between
entities in a sequence or a grid, as they cannot consider the complex
relationships between entities. To solve this problem, we use the where graph
neural networks.

GNNs are a type of ANN specifically designed to process data structured as a
graph. A graph is a group of nodes and edges representing relationships
between them. The nodes in a GNN can represent entities such as people,
objects, or places, and the edges can represent relationships such as
friendships, kinship, or spatial proximity.

GNNs can process graph-structured data by iteratively aggregating information
from neighbouring nodes. This allows GNNs to learn the complex relationships
between entities in a graph, making them a powerful tool for node
classification, link prediction, and graph classification tasks.

To express the edges and nodes of a graph as inputs to a neuron in an ANN, we
first construct an adjacency matrix $\mathbf{A}$. It is, essentially, a matrix
of dimensions $n\times n$ where $n$ is the number of nodes in a graph. The
entries in the matrix are $1$ if an edge exists between the nodes and $0$ if
there is no edge. Mathematically for each $A_{ij}\in\mathbf{A}$, and $1\leq
i,j\leq n$, we have that

$A_{ij}=\begin{cases}1,\text{if there exists an edge between
nodes}\;i\;\text{and}\;j,\\\ 0,\text{otherwise}.\end{cases}$ (2)

The adjacency matrix approach has several advantages: If the graph is
undirected, the adjacency matrix becomes symmetric and has several interesting
properties, including the diagonals representing self-loop vertices, the
degree of the graph can be inferred from the sum of any row or column, the
number of edges in the graphs (the number of connections required in the ANN)
is just half the count of the number of nonzero entries, and properties
relating the eigenvalues of $\mathbf{A}$.

In Tab. 1, we observe no self-loops in the graph structure; thus, the diagonal
entries of the adjacency matrix will be $0$, This is illustrated in Fig. 3,
which shows a directed graph with no self-loops.

Table 1: Adjacency matrix representing directed edges between nodes $A$, $B$, $C$, $D$, and $E$. | A | B | C | D | E
---|---|---|---|---|---
A | 0 | 1 | 0 | 0 | 0
B | 0 | 0 | 1 | 1 | 0
C | 0 | 0 | 0 | 0 | 1
D | 1 | 0 | 0 | 0 | 0
E | 0 | 0 | 0 | 1 | 0
Figure 3: A directed graph. The arrows indicate the direction of flow. The
graph consists of nodes labelled $A$, $B$, $C$, $D$, and $E$.

Below, we elaborate on the critical components of GNNs.

  * •

Graph Convolution This is the key concept underpinning GNN architectures.
Graph convolution predicts the features of the node in the next layer as a
function of the neighbours’ features. It transforms the node’s features
$x_{i}$ in a latent space $h_{i}$ that can be used for various reasons, as
shown in Fig. 4.

Figure 4: A graphical representation showing the dependency of the feature of
nodes on the features of its neighbouring nodes.

  * •

Node classification For a GNN, the objective is firstly to generate latent,
from which we would then be able to work on a wide variety of standard tasks.
Mathematically, we have that

$Z_{i}=f(h_{i}),$ (3)

where $h_{i}$ is the feature vector associated with node $i$, and it can also
be represented in the Fig. 5.

Figure 5: Graphical representation illustrating node classification. Nodes are
associated with expressions $Z_{i}=f(h_{i})$, where $i$ ranges from 1 to 7,
capturing relationships between features $Z_{i}$ and hidden features $h_{i}$.

  * •

Edge Classification: Similarly, we can use it to classify edges based on their
features. To accomplish this, we generally need both the adjacent node vectors
and the edge features, if they exist. Mathematically, we have that

$Z_{ij}=f(h_{i},h_{j},e_{ij}),$ (4)

where $h_{i},h_{j}$ are the feature vectors associated with nodes $i$ and $j$
respectively, and $e_{ij}$ is the edge connecting nodes $i$ and $j$ as shown
in Fig. 6.

Figure 6: Graphical representation showcasing edge classification

###  3.1 Types of Graph Neural Networks

There are a variety of graph neural networks. Below, we focus on some of the
important types of GNNs:

####  3.1.1 Feed-forward GNNs:

Also known as “Deep feedforward Networks” or “Multi-layer Perceptrons”. In
these model architectures, input data is propagated through a graph of neurons
to produce output by applying the transfer function at the edge weights
between each node as represented in Fig. 7. Feedforward graph neural networks
are simplified versions of GNNs that work straightforwardly. They process
graph data in just one pass and update node information using neighbouring
node data.

Figure 7: Basic structure of Feed-Forward GNNs. The network consists of an
input layer with $3$ nodes, a hidden layer with $4$ nodes, and an output
layer. Figure 8: Architecture of Graph Convolutional Neural Network. The
network includes a convolution layer followed by a ReLU activation function, a
subsequent convolution layer with another ReLU activation function, an
intermediate dense layer, and finally, a sigmoid activation function before
the output layer.

####  3.1.2 Graph Recurrent Neural Networks (GRNNs):

GRNNs relay information from previous to neighbouring nodes. At each step, the
GRNN uses a hidden state as a memory of what the network has encountered thus
far in the sequence, as represented in Fig. 9. The network is updated with
each new step based on the current input and the previous hidden state. When
working with sequences, the RNN “unrolls” itself through time, treating each
step as a separate layer of the network. This helps the GRNN handle sequences
of different lengths.

Figure 9: Structure of a Graph Recurrent Neural Network. The GRNN consists of
an input layer with $4$ nodes, a hidden layer with $5$ nodes, and an
additional layer with $3$ nodes. The architecture is completed with an output
layer.

In [17], it was observed that Recurrent neural networks allow for loops and
cycles, giving rise to dynamical systems and flexible behaviour in
computation.

In [18], it has been observed that graph neural Networks are an evolution of
Recurrent Neural Networks (RNNs) that address the limitations of RNNs by
combining gated RNN architectures with graph convolutions.

####  3.1.3 Graph Convolutional Neural Networks (GCNNs):

These architectures are one of the most popular and widely used GNNs in the
literature, and they can be represented as shown in Fig. 8. The main property
of GCNNs is their ability to maintain stability under stochastic
perturbations. This means that GCNNs are less likely to be affected by noise
or randomness in the data, which can be a major advantage in many
applications. In [19], a new RES and a GFT model were employed over a GCNNs to
test the robustness of the graph filter. GCNN is stable to stochastic
perturbations with a factor proportional to the link loss probability. In
particular, a wider and deeper GCNN decreases the stability while improving
the performance, indicating a trade-off between these two factors. In [20],
the study leverages the stability property of GNNs to seek stable
representations within a distribution.

####  3.1.4 GraphSAGE (Graph Sample and Aggregated) Networks:

GraphSAGE is a node classification algorithm that samples and aggregates
information from a node’s local neighbourhood in a graph.

It improves over GCNNs by allowing nodes to aggregate information from a
sampled set of neighbours, making GraphSAGE more scalable and generalisable to
different graph structures.

In [21], Lo and et al. proposed E-GraphSAGE, a GNN-based intrusion detection
system which outperforms state-of-the-art methods in evaluating network-based
intrusion detection systems.

These networks use a message-passing scheme to update node representations by
aggregating and transforming information from the neighbourhood. They have
been shown to be effective in capturing both the edge features and topological
information of a graph, making them suitable for analysing interconnected
patterns in network flows. The modified E-GraphSAGE and E-ResGAT algorithms
integrate residual learning into GNNs to address the challenge of class
imbalance and improve performance in minority classes.

The GraphSAGE model architecture includes two graph convolution layers and
uses the mean aggregation method to effectively capture node information from
neighbours in the graph, as shown in Fig. 10.

Figure 10: Architecture of the GraphSage. It comprises successive GraphSAGE
layers, each followed by a ReLU activation function. An intermediate dense
layer bridges the structure, culminating in a sigmoid activation function in
the output layer.

For homogeneous graphs, the feature update rule is the mean aggregator, where
the aggregation of features of the neighbours of the node v are given by

$h_{N(v)}^{i}=\frac{1}{||N(v)||}D_{p}\left[h_{u}^{i-1}\right],\forall u\in
N(v),$ (5)

where the forward passes through layer k can be added in the future nodes and
is given by

$h_{v}^{i}=\sigma\left(\text{concat}\left[W^{k}_{\text{self}}D_{p}\left[h_{v}^{i-1}\right],W^{i}_{\text{neigh}}h_{N(v)}^{i}\right]+b^{k}\right).$
(6)

In Eq.(5) $D_{p}\left[\ldots\right]$ is a random dropout with probability $p$
applied to its argument vector, and $b^{k}$ is an optional bias.

Eqs.(5) and (6) are the rules deciding how the features get updated in the
GNN. The GraphSAGE algorithm was implemented on the dataset to perform
predictions using StellarGraph. From the original graph, $30\%$ of edges were
removed to form a test set, $65\%$ acted as the train set, and $5\%$ acted as
the validation set where the internal parameters were optimised over $10$
epochs using the ADAM optimiser. The classifier was then applied to the test
set; Scores were generated for the classifier’s predictions and are provided
in the Sec. 6. In the following, we describe the steps of the GraphSAGE
algorithm.

Algorithm 1 GraphSAGE

0: data (training, validation)

for all data in training, validation do

Assign unique node identifier to sample

end for

Create a StellarGraph $G$ using data with source column as “node_id”

Initialise a GraphSAGENodeGenerator with graph $G$, batch size $5$, and number
of samples $\left[2,32\right]$

Extract node labels $y$ from data

Generate a flow using the GraphSAGENodeGenerator and $y$ as targets

if data is training then

Initialise a GraphSAGE model with layer sizes $\left[128,128\right]$ and
activations [“relu”, “relu”]

Define input tensor and output tensor using the GraphSAGE model

Add a Dense layer with $1$ unit and “sigmoid” activation to obtain predictions

Create a Keras model with input and output tensors

Compile the model with “adam” optimiser, “binary_crossentropy” loss, and
“accuracy” metric

for epoch in $1$ to $10$ do

Train the model using the flow and perform validation for $1$ epoch

end for

end if

if data is validation then

Predict on the flow using the trained model.

Calculate mode threshold and convert predictions to binary

Calculate precision-recall curve points and AUC-PR using true labels and
predicted scores

end if

The algorithm 1 first assigns unique node identifiers to each sample in the
dataset. A graph is then constructed using the StellarGraph library, where
nodes are identified by their unique identifiers. The GraphSAGENodeGenerator
is then used to generate a stream of node samples and their corresponding
labels. During the training phase, the GraphSAGE model is instantiated by
specifying the sizes of the layers and activation functions. The model
includes a dense output layer with a sigmoid activation function for
generating predictions. After defining the input and output tensors, a Keras
model is created and compiled using the Adam optimiser, binary cross-entropy
loss, and accuracy metric. The model is trained over a number of epochs using
the generated data stream.

For non-training data, the trained model is used to predict labels. The
predictions are then post-processed by calculating the mode threshold and
converting the predicted scores to binary form. Precision-recall curves are
then computed to evaluate the model’s performance, and the area under the
precision-recall curve (AUC-PR) is used to quantify its effectiveness.

##  4 Quantum Graph Neural Networks

Our research methodology draws inspiration from the pioneering work of Hu et
al., $2022$ [22] in quantum graph neural network development. Our approach
unfolds in the following manner:

  * •

Step $1$: Graph Construction and Representation

Our initial step involves constructing the graph, where each graph corresponds
to dataset transactions. This dataset encompasses $30$ encoded attributes,
predominantly processed via Principal Component Analysis (PCA) techniques,
except for two attributes – transaction amount and time. In this framework,
nodes represent encoded features. At the same time, edges symbolise their
fusion to get novel graph configurations, and each graph in our feature
grouping employs interconnected nodes, totalling $28$ nodes per graph.

  * •

Step $2$: Topological Data Analysis (TDA) for Graph Projection

Utilising topological data analysis, the graph is embedded into a one-
dimensional scatter plot. This technique discriminates between graphs based on
their distinct elements, facilitating differentiation between fraudulent and
non-fraudulent cases. Following scatter plot creation, node clustering reduces
nodes while encapsulating the initial $28$ features, minimising quantum
resource requirements. A final $28$-dimensional feature vector representing
the positional attributes is generated.

  * •

Step $3$: Encoding within Quantum Circuit and Linear Transformation

Upon acquiring the graphical representation of each transaction, the
subsequent step entails encoding it into a quantum state utilising the angle-
encoding technique, employing a range of qubits from $1$ to $28$. To initiate,
the encoding process is applied to all node-level features ($N\times 28$),
resulting in a dimensionally apt vector ($N\times q$) facilitated by a fully
connected linear layer. This vector is subsequently channelled into the
designated quantum circuit with $q$ qubits as defined

$\ket{x}=\bigotimes_{i=1}^{N}\cos(x_{i})\ket{0}+i\sin(x_{i})\ket{1}.$ (7)

  * •

Step $4$: Enhancing Node Features with Variational Quantum Circuits

Following the previous steps, a sequence of two-local ansatz constructs
orchestrates the implementation of a multi-layered Variational Quantum Circuit
(VQC) calibrated for the transactional dataset. Fig. 11 visually portrays a
circuit encompassing a $6$-qubit with the angle encoding alongside $2$
concealed VQC layers. Notably, the ansatz capitalises on Rx, Ry, and CNOT
operations while adopting Rx rotations for angle encoding, as explained in the
$3$rd step. Subsequently, circuit measurement culminates in using Pauli-Z
expectation. This strategic circuit design is a conscientious outcome of
considerations concerning circuit expressibility and intricate node-node
interactions embedded within graphical data.

Figure 11: Quantum circuit for angle encoding and VQC in the QGNN architecture
with RY, RX, and CNOT gates.

We enhance node features using the VQC, emulating the operational principles
of conventional classical graph neural networks with fully connected layers.
VQC entails the successive application of variational layers. A single layer
transformation $L$ is depicted as

$L:\ket{\psi(x)}\xrightarrow{}\ket{\psi(y)}=U(w)\ket{\psi(x)},$ (8)

where $U(w)$ represents the VQC with $N$ layers, defined as

$U(w)=T\bigotimes_{k=1}^{N}R_{y}(w_{k})\ket{\psi(x)}.$ (9)

  * •

Step $5$: Integration of Unitary Gates and Classification

In order to meet the requirements of a quantum graph neural network, it is
necessary to have unitary gates that act on two states simultaneously while
preserving the others. As we explained, rotational gates are essential to
integrate the graph into the quantum circuit. Once these gates are in place, a
linear layer can classify the graph as fraudulent or non-fraudulent. The
comprehensive procedure outlined in Fig. 12 sheds light on the architecture
and operational aspects of the QGNN, highlighting the distinct components and
their produced interplay in fraud detection. Additionally, Algorithm 2
provides a step-by-step algorithm description.

Figure 12: Architecture of the quantum graph neural network. The QGNN starts
with a graph as input, followed by a feature compression step. Angle encoding
prepares the data for the VQC processing. The output of the VQC is subjected
to average pooling, then fed through a linear layer, and finally to the output
layer Algorithm 2 Quantum Graph Neural Networks

Inputs: Graph data (training, validation)

Initialise QGNN

for each epoch in training do

for each graph batch in training data do

Extract graph features and edges

Apply angle encoding to the graph features

Initialise Quantum Circuit for VQC

Generate tunable parameters for VQC

Update Quantum Circuit using inputs and VQC parameters

Run the Quantum circuit (VQC) to obtain quantum output

Perform average pooling for graph-level features

Make predictions using a linear layer

Compute loss and update parameters using Adam optimiser

end for

Evaluate QGNN on validation data

end for

Calculate optimal threshold for binary classification

Convert predictions to binary using the threshold

Calculate Precision-Recall curve and AUC-PR

Return Classification metrics, loss values

##  5 The Dataset

We used the publicly available credit card fraud detection dataset [23] for
our analysis. This dataset contains credit card transactions of European
cardholders with encrypted feature labels obtained from principal component
analysis on actual user data. Each transaction is labelled as either fraud
$(1)$ or not fraud $(0)$ and contains $28$ encrypted features along with
amount and time values. The dataset is highly imbalanced, with only $492$
frauds out of $284\;807$ transactions. To address this imbalance, we used
random undersampling on the non-fraud transactions.

For a comprehensive understanding of the dataset characteristics, we utilised
visual representations. In Fig. 13, we present the correlation matrix
$(v_{1}-v_{28})$ of the dataset. The figure effectively illustrates the
relationships between the features $v_{1}$ to $v_{28}$ by displaying their
correlation values. This information offers insights into how these features
interact and influence one another.

Figure 13: Correlation matrix $(v_{1}-v_{28})$ of the dataset. The figure
illustrates the relationships between the features $v_{1}$ to $v_{28}$ by
displaying their correlation values.

Additionally, Fig. 14 offers histograms that vividly depict the distribution
of all features in the dataset. These histograms visually represent the data
distribution patterns, assisting us in identifying potential trends and
outliers. Furthermore, our approach involved translating each transaction as a
graph using topological data analysis [24]. This technique allowed us to
capture intricate relationships among the various feature vectors, enhancing
our ability to analyse the data effectively. In the following steps, we
explain how to apply and implement this technique.

Figure 14: Histograms illustrating the distribution of all features in the
dataset.

Given a transaction described by a vector

$\mathbf{V}=[\text{Time},V_{1},V_{2},\ldots,V_{28},\text{Amount}],$ (10)

We create the graph $G=\left(N,E\right)$, where
$N\in\\{R^{1},R^{2},\ldots,R^{28}\\}$ represents the feature vector, and $E$
represents the edge list, where, $V_{1},V_{2},\ldots,V_{28}$ denotes the
encrypted feature labels in the dataset.

Next, we focus on converting the transaction,
$\left\\{T_{i}\right\\}_{i=1}^{284\;807}$, into its graphical representations
$G_{i}$ via the following steps using TDA:

  1. 1.

Create a set $\\{S_{i}\\}$ of 3D data points, where
$S_{i}=\big{\\{}P_{j}\;|\;P_{j}=\left[\text{Time}_{i},V_{j},\text{Amount}_{i}\right],\>\forall{j}\in[1,28]\>\big{\\}}$.
This helps in capturing the correlation of $V_{j}$ with time and amount.

  2. 2.

Project all data points $P_{j}\in S_{i}$ onto a 1D plane. We denote these
projected points as $f_{j}$.

  3. 3.

Draw covers around all the projected data points $\\{f_{j}\\}_{j=1}^{28}$ and
cluster them using a suitable clustering algorithm, here, we have used DBSCAN.
Let these clusters form a set $\\{C_{i}\\}$, where
$C_{i}=\big{\\{}c_{k}\>|\>\exists_{j,k}\;:\;f_{j}\in c_{k},\>\forall
j\in[1,28]\;and\;1<k<|C_{i}|\big{\\}}$

  4. 4.

Draw edges between clusters with common data points to form the graph $G_{i}$.

Here, a node $N_{k}\in G_{i}$, represents a cluster
$c_{k}\in\left\\{C_{i}\right\\}$, s.t. $N_{k}=\left(0\ldots V_{j}\ldots
0\right)_{28},\;\forall f_{j}\in c_{k}$. Thus, the relation between different
features is captured and grouped using the clustering algorithm of high-
dimensional data via a more representative graphical structure. Each graph
represents a transaction labelled fraud or non-fraud for the classification
task.

##  6 Results and Discussion

We tested the performance of our two classical and quantum algorithms on the
same dataset described in Sec. 5. The results showed that the GraphSAGE model
achieved promising results on the fraud classification task. The model was
trained for $10$ epochs using the Adam optimiser with a batch size of $5$
using Keras, as shown in Tab. 2. The aggregation method was set to mean. These
hyperparameters were chosen based on preliminary experiments and empirical
observations to balance model complexity and generalisation performance.

Table 2: Hyperparameters and training details for the GraphSAGE model. Hyperparameter |  Value
---|---
Model Architecture |  GraphSAGE
Number of Epochs |  $10$
Optimiser |  Adam
Batch Size |  $5$
Aggregation Method |  Mean

The receiver operating characteristic (ROC) curve presented in Fig. 15 shows
the trade-off between the true positive rate (TPR) and the false positive rate
(FPR) for the GraphSAGE model. The Area Under Cover (AUC) score measures how
well a model can distinguish between positive and negative instances. A higher
AUC score indicates a better model. The GraphSAGE model achieves an AUC of
$0.77$, within the acceptable range for fraud classification tasks. The
performance metrics for the GraphSAGE model are also within the acceptable
range, with an accuracy of $92.3\%$, and F1 score of $0.83$. These scores are
reported in Tab. 3.

Figure 15: Receiver Operating Characteristic Curve for the GraphSAGE model with an AUC of $0.77$. Table 3: Performance test metrics of GraphSAGE Accuracy (%) | Precision (%) | Recall (%) | F1 score
---|---|---|---
92.3 | 95.2 | 76.3 | 0.83

The loss curves in Fig. 16 show that the training loss decreases steadily over
time while the validation loss decreases more slowly, which suggests that the
model is well-fitted on both the training and validation data.

Figure 16: Training and validation loss curves for the GraphSAGE model.

We performed experiments on the quantum graph neural network using qiskit [25]
under different scenarios, namely qubit count and hidden layers. The results
in Tab. 4 show that a dense encoding scheme of node features using fewer
qubits performed better than using more qubits and sparser representations,
and this is evident as the model with $6$ qubits, with an accuracy of $94.5\%$
and F1 score of $0.86$, performed best compared to that with 16 qubits, with
an accuracy of $92.0\%$ and F1-score of $0.81$.

Table 4: Performance test metrics of QGNN. The table presents results for various configurations, including the number of qubits and layers. Num Qubits | Layers | Accuracy (%) | Precision (%) | Recall (%) | F1 score
---|---|---|---|---|---
6 | 1 | 94.5 | 96.1 | 79.5 | 0.86
16 | 1 | 92.06 | 86.5 | 77.3 | 0.81
6 | 2 | 91.5 | 93.3 | 65.6 | 0.76
16 | 2 | 89.2 | 86.6 | 72 | 0.77

The AUC for the precision-recall curve from Fig. 17 is estimated to be roughly
$0.85$, which shows high tolerance in our method towards biased datasets with
high class imbalances, apart from having highly accurate predictions. A
smoother transition curve leads to a better F1 score, and the loss curves show
a decrease in the loss function over time for both training and validation,
which shows that our model is well-fitted, as presented in Fig. 18.

Figure 17: Receiver Operating Characteristic curve for QGNN model with $6$
qubits and $1$ hidden layer, with an AUC of $0.85$ Figure 18: Training and
validation loss curves for the QGNN model with $6$ qubits and $1$ hidden
layer.

These results are observed on a basic network architecture of quantum circuits
and outperform various classical approaches. This is due to the enhanced
representation of graphical structures through quantum circuits and efficient
feature relations using topological data analysis. We can see an improvement
of $\approx 3\%$ over classical GNN, such as GraphSAGE, across various
metrics. However, it was observed that graphical techniques, in general, help
enhance the precision and accuracy of these classification systems more than
other non-graphical approaches.

Careful analysis of quantum techniques suggests that better representation,
deeper search space, state entanglement, and qubit parallelism are key factors
contributing to the supremacy of quantum circuits for classification tasks. In
our experiments, we observed the same with a smaller system with 6 qubits and
$\approx$200 parameters, which makes our model QGNN outperform the classical
GNN. These results suggest that with more advanced features and complex
quantum systems, we can achieve a real quantum advantage in solving problems
such as fraud detection in highly imbalanced datasets.

##  7 Conclusion

Our research has demonstrated the potential of quantum graph neural networks
in detecting financial fraud. We have compared the performance of QGNNs with
classical graph neural networks and found that QGNNs outperformed GNNs in
terms of accuracy and efficiency. Our experiments on a real-world financial
dataset show that QGNNs can detect fraud with a remarkable accuracy of
$94.5\%$, which is significantly higher than GNNs’ accuracy of $92.3\%$.

Moreover, we have also studied the influence of the number of qubits and
layers on QGNNs’ performance. Our research indicates that QGNNs with $6$
qubits and one layer exhibit the best performance, with an accuracy of
$94.5\%$. This can be attributed to the fact that fewer qubits are less prone
to noise, and a single layer is sufficient to grasp the essential features of
the data.

Our research shows that QGNNs have the potential to revolutionise fraud
detection and improve the security of financial systems. It also highlights
the need for further research to address the challenges of implementing QC in
real-world applications.

## Reference

## References

  * [1] Ali A, Abd Razak S, Othman S H, Eisa T A E, Al-Dhaqm A, Nasser M, Elhassan T, Elshafie H and Saif A 2022 Applied Sciences 12 9637
  * [2] Sadgali I, Sael N and Benabbou F 2019 Procedia computer science 148 45–54
  * [3] Zojaji Z, Atani R E, Monadjemi A H et al. 2016 arXiv preprint arXiv:1611.06439
  * [4] Tüysüz C, Rieger C, Novotny K, Demirköz B, Dobos D, Potamianos K, Vallecorsa S, Vlimant J R and Forster R 2021 Quantum Machine Intelligence 3 1–20
  * [5] Pourhabibi T, Ong K L, Kam B H and Boo Y L 2020 Decision Support Systems 133 113303
  * [6] Yu S, Ren J, Li S, Naseriparsa M and Xia F 2022 Frontiers in Artificial Intelligence 5 922589
  * [7] Grossi M, Ibrahim N, Radescu V, Loredo R, Voigt K, Von Altrock C and Rudnik A 2022 IEEE Transactions on Quantum Engineering 3 1–12
  * [8] Ghosh S and Reilly D L 1994 Credit card fraud detection with a neural-network System Sciences, 1994. Proceedings of the Twenty-Seventh Hawaii International Conference on vol 3 (IEEE) pp 621–630
  * [9] Ma X, Wu J, Xue S, Yang J, Zhou C, Sheng Q Z, Xiong H and Akoglu L 2021 IEEE Transactions on Knowledge and Data Engineering
  * [10] Ahmad Z, Shahid Khan A, Nisar K, Haider I, Hassan R, Haque M R, Tarmizi S and Rodrigues J J 2021 Applied Sciences 11 7050
  * [11] Zheng J, Gao Q and Lü Y 2021 Quantum graph convolutional neural networks 2021 40th Chinese Control Conference (CCC) (IEEE) pp 6335–6340
  * [12] Kyriienko O and Magnusson E B 2022 arXiv preprint arXiv:2208.01203
  * [13] Innan N, Khan M A Z and Bennai M 2023 arXiv preprint arXiv:2308.05237
  * [14] Xiang S, Zhu M, Cheng D, Li E, Zhao R, Ouyang Y, Chen L and Zheng Y 2023 Semi-supervised credit card fraud detection via attribute-driven graph representation Proceedings of the AAAI Conference on Artificial Intelligence vol 37 pp 14557–14565
  * [15] Schetakis N, Aghamalyan D, Boguslavsky M, Rees A, Raktomalala M and Griffin P 2023 arXiv preprint arXiv:2308.03575
  * [16] Beer K, Khosla M, Köhler J, Osborne T J and Zhao T 2023 Physical Review A 108 012410
  * [17] Siegelmann H T 2005 Computer Science Today: Recent Trends and Developments 29–45
  * [18] Kiran Kadambari S and Prabhakar Chepuri S 2020 arXiv e-prints arXiv–2001
  * [19] Gao Z, Isufi E and Ribeiro A 2021 Signal Processing 188 108216
  * [20] Cerviño J, Ruiz L and Ribeiro A 2022 Training stable graph neural networks through constrained learning ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (IEEE) pp 4223–4227
  * [21] Lo W W, Layeghy S, Sarhan M, Gallagher M and Portmann M 2022 E-graphsage: A graph neural network based intrusion detection system for iot NOMS 2022-2022 IEEE/IFIP Network Operations and Management Symposium (IEEE) pp 1–9
  * [22] Hu Z, Li J, Pan Z, Yang S Z L, Ding C, Khan O, Geng T and Jiang W 2022 On the design of quantum graph convolutional neural network in the nisq-era and beyond IEEE 40th International Conference on Computer Design (ICCD) pp 3–5
  * [23] ULB M L G 2013 Credit card fraud detection URL https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud
  * [24] Chazal F and Michel B 2021 Frontiers in artificial intelligence 4 108
  * [25] Aleksandrowicz G and et al 2019 Qiskit: An Open-source Framework for Quantum Computing URL https://doi.org/10.5281/zenodo.2562111


# Deep learning-based stereo camera multi-video synchronization

1st Nicolas Boizard  ISIA Lab - University of Mons
Mons, Belgium
<EMAIL_ADDRESS>2nd Kevin El Haddad 12  0000-0003-1465-6273 1ISIA Lab
- University of Mons
2The Big Projects
Mons, Belgium
<EMAIL_ADDRESS>3rd Thierry Ravet  4th François Cresson  ISIA Lab
- University of Mons
Mons, Belgium
<EMAIL_ADDRESS>ISIA Lab - University of Mons
Mons, Belgium
<EMAIL_ADDRESS>5th Thierry Dutoit  ISIA Lab - University of
Mons
Mons, Belgium
<EMAIL_ADDRESS>
###### Abstract

Stereo vision is essential for many applications. Currently, the
synchronization of the streams coming from two cameras is done using mostly
hardware. A software-based synchronization method would reduce the cost,
weight and size of the entire system and allow for more flexibility when
building such systems. With this goal in mind, we present here a comparison of
different deep learning-based systems and prove that some are efficient and
generalizable enough for such a task. This study paves the way to a production
ready software-based video synchronization system.

©2023 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

##  I INTRODUCTION

Computer vision applications such as visual depth estimation, useful in the
areas of autonomous navigation and human-agent interaction, require stereo
vision. Currently, the most accurate stereo systems are based on specific
hardware solutions such as stereo cameras. The synchronization of video
streams within these cameras is therefore achieved using electronic systems
that increase the cost, the weight, limit the flexibility of the systems and
require more space. Replacing these electronic solutions with robust software
would make these systems more flexible, allowing them to be used with
different camera configurations and settings. Also, making these software
solutions open-source would help reduce the costs to build stereo vision
systems and thus make them more accessible to a wider community. In this
article, we present a study on the automatic synchronisation in time of two
video sequences from two different lenses/cameras filming a common area with a
deep learning method. Research for this specific application is surprisingly
neglected. Indeed previous work did tackle the problem of software-based
multi-camera synchronization [1, 2, 3, 4]. But, the novelty and advantage of
our work is that it relies solely on the image content (the pixels) and is not
dependant on hardware components or their features (internal clocks,
timestamps, network, etc.). It does not require prior knowledge to function
nor is it limited to a specific application. To the best of our knowledge, the
closest related work on our previously defined problem to date is the work
[5]. The objective of their research was to synchronize several travelling
paths despite changing weather conditions. Based on the triplet loss method,
their model was able to synchronize two video sequences with 38% of the
predicted frame pairs having a shift of less than four frames. Their objective
is quite different from our problematic. Indeed, our problem focuses on the
synchronization of videos acquired by two cameras with overlapping scenes
captured over a similar time periods, as opposed to the alignment of videos
recorded by a single camera following an identical trajectory but at different
time periods, which involves serious changes in the scenes.

In this paper, we chose to divide the problem of video synchronization into
two subproblems: given two video sequences of twenty frames captured at the
same frame rate by two different cameras with a distance between their optical
centres limited to a few centimeters and oriented towards the same direction,
a first module computes one by one the correspondence scores between each
frame of the two sequences; then, when all the correspondence scores are
computed, a second module takes as input all these scores to estimate the
average delay between the sequences. This final frame delay can then be used
to synchronise the two sequences. The need for a second submodule in our
system is due to the fact that the output of the first one does not give
consistent results in terms of frame distance when assigning similar images:
images A and B of sequence 1 could be assigned to images C and D of sequence 2
with a frame distance from A to C different from the one from B to D.

A commonly used method for the similarity estimation between a pair of images
is to use the SIFT algorithm [6]. This algorithm allows to detect, describe
and locate common features between two images. However, preliminary tests with
it gave poor results, so we decided to focus our experiments, in this work, on
deep learning-based systems which initially showed more promising results.
Siamese networks are another common choice for image pair comparison. The
authors in [7], use such architecture and show that the features proposed
using a Siamese network improve the matching performance compared to the
features obtained with a classical CNN networks trained for image
classification. Other works also based on Siamese networks [8, 9, 10, 11] as
well as triplet networks [12, 13] showing encouraging results, motivated our
choice towards considering them in our experiments. The main contribution of
this work is a comparative study of different architectures and approaches, a
product of which being what we believe to be a first deep learning-based
stereo video time synchronization system. We thus prove the feasibility of a
software based video synchronization. The source code and the dataset used
here are provided to the community
111https://github.com/numediart/multi_video_sync.

We therefore first present the dataset collected in Section II. We then
present the systems, experiments and results in Sections III, IV and V before
finally concluding in Section VI.

##  II DATASET

TABLE I: List of the different datasets (image + optical flow), with their properties. These are: the camera used, the environment (2 possible locations), whether the camera is in motion or not, and whether the scenes contain moving objects (human walking). The columns starting with the abbreviation No. indicate the Number of image and optical flow pairs that the dataset contains as well as the number of video sequences formed on these two types of data. To differentiate a dataset according to the type of data chosen, ”*” in the name will be replaced by ”img” for images and ”flw” for the optical flow in the rest of this paper. Name |  Camera |  Location |  Moving Camera |  Moving Objects |  No. of raw img |  No. of optical flow img |  No. of raw img seq |  No. of optical flow seq
---|---|---|---|---|---|---|---|---
*_zed_p |  Stereolab Zed 2 |  Loc1 |  Pushed |  No |  23892 |  23890 |  40000 |  40000
*_zed_p2 |  Stereolab Zed 2 |  Loc2 |  Pushed |  No |  16104 |  16102 |  40000 |  40000
*_zed_m |  Stereolab Zed 2 |  Loc1 |  Still |  Yes |  3960 |  3958 |  3000 |  3000
*_zed_pm |  Stereolab Zed 2 |  Loc1 |  Pushed |  Yes |  4200 |  4198 |  3000 |  3000
*_oak_p |  Oak D Lite |  Loc1 |  Pushed |  No |  7910 |  7908 |  3000 |  3000
*_oak_m |  Oak D Lite |  Loc1 |  Still |  Yes |  5714 |  5712 |  3000 |  3000
*_oak_pm |  Oak D Lite |  Loc1 |  Pushed |  Yes |  4784 |  4782 |  3000 |  3000

For our experiments, matching/non-matching pairs of images and image sequences
were needed. It is also important to have videos recorded in different
configurations/environments to study the robustness of our models. We
therefore recorded videos using stereo cameras at 30 fps in RGB mode using two
different cameras (Stereolab Zed 2222https://www.stereolabs.com/zed-2/ and OAK
D Lite333https://shop.luxonis.com/products/oak-d-lite-1) and in different
environments. Other parameters considered were: the camera being in motion or
fixed and the presence or not of moving objects in the scene (humans walking
in our case). Note that recording videos with stereo cameras allowed us to
desynchronize them manually by applying different delays. The results are
summarized in Table I.

The first module of our system mentioned in the introduction, the image
matching model, required unmatched image pairs along with the matching pairs
for the training. We therefore created matching and non-matching pairs of
images based on the recordings. Matching pairs were formed by the
corresponding images of the left and right lenses of the camera. The non-
matching pairs were formed by randomly pairing images from the left and right
lenses with the following conditions: they must come from the same set and the
frame coming from the right must be located in an interval $t\in[-10,-1]$ or
$t\in[1,10]$ with t the capture time of the left frame. We also ensure that
all images in the sets are 224x224px in size and in grayscale by post-
processing them.

Also, optical flow features were extracted using the Farneback algorithm [14]
from the videos recorded. Our raw image datasets being relatively small, we
rely on the optical flow to provide additional motion information to our
models in order to avoid overfitting in some cases. Indeed, as we show in the
following sections, these features turned out to be more efficient than the
raw images in several cases. This results in total of fourteen sets summarized
in Table I (seven of raw images and seven of optical flow data).

Concerning the second module mentioned in the introduction, the delay
estimation module, we desynchronized two corresponding sequences of twenty
images coming from the left and the right lenses of a stereo camera, by a
random delay varying from zero to twenty frames. A delay of zero frames,
corresponds to synchronized sequences while a delay of twenty frames
corresponds to completely desynchronized ones (there are therefore no frames
in common between the left and right videos). The delay between each pair of
video clips is randomly assigned to ensure balanced datasets. The total number
of sequences for each of the raw images and optical flow sequences can be seen
in Table I.

##  III SYSTEMS

As mentioned in the introduction, our problem is divided into two subproblems:
firstly, finding the matching frames between our videos and secondly,
calculating the delay using these matches. Our model therefore consists of two
submodels. The first submodule, called ”Matching Frames”, will compute a
matching score between two frames. Using the calculation of these matching
scores, we construct a matching matrix. Indeed, by calculating the score of
the first frame of the left video with all the frames of the right video, then
the score of the second frame of the left video with all the frames of the
right video and so on, we obtain a matching matrix on all the frames of the
videos. Finally, with this correspondence matrix, our second submodule called
”Delay Estimation” estimates the delay that may exist between the two
sequences.

For the ”Matching Frames” submodule, several network architectures were
considered. The first one is a Siamese network called ”CNNSiamese” and
described in the Fig 1. ”CNNSiamese” is trained with the binary crossentropy
loss and the Adam optimizer. As input, the network takes a frame from the left
lens and another from the right lens. On the output, a neuron activated by a
Sigmoid function gives a matching score between these two frames. The closer
the score is to 1, the higher the network considers that the pair of frames
match. The stride of the convolution layers is set to (1,1) and activated with
the ReLu function.

Figure 1: Diagram of ”CNNSiamese” model. The CNN blocks are summarised in the
Table II and the Distance block performs a subtraction.

Another version of ”CNNSiamese” with CNN blocks based on a pre-trained
EfficientNet [15] architecture was also tested in this work. Due to poor
results in the early stages of the experiments, this model was dropped from
the study.

TABLE II: Layers in the CNN blocks for ”CNNSiamese”, ”TripletEuc” and ”TripletSim” model. Layers | Units/Filters | Kernel Size
---|---|---
Conv2D | 32 | (3,3)
MaxPooling2D | X | (2,2)
Conv2D | 48 | (3,3)
MaxPooling2D | X | (2,2)
Conv2D | 48 | (3,3)
MaxPooling2D | X | (2,2)
Conv2D | 64 | (3,3)
MaxPooling2D | X | (2,2)
GlobalAveragePooling2D | X | X

The second architecture considered and depicted in Figure 2 is based on the
Triplet Loss method [16]. A triplet loss architecture aims at minimizing the
distance between two positively associated data and maximize the distance with
two negatively associated data (the anchor associated with the positive and
with negative input respectively as shown in Fig 2). The distance obtained of
either the positive/anchor or negative/anchor branch is then used here to fill
the matching matrix forwarded to the second submodule. Two versions of the
triplet loss network are designed. The first one, ”TripletEuc” for which the
distance is an Euclidean distance and whose training aims at minimizing the
loss:

$\sum_{i}^{N}\left[\left\|f\left(x_{i}^{a}\right)-f\left(x_{i}^{p}\right)\right\|_{2}^{2}-\left\|f\left(x_{i}^{a}\right)-f\left(x_{i}^{n}\right)\right\|_{2}^{2}+\alpha\right]$

while for the second one, ”TripletSim”, the distance is the cosine similarity
and trained to minimize the loss:

$\sum_{i}^{N}\left[\max(\cos(f\left(x_{i}^{a}\right),f\left(x_{i}^{p}\right))-\cos(f\left(x_{i}^{a}\right),f\left(x_{i}^{n}\right))+\alpha,0.0)\right]$

with $N$ the number of samples per batch, $f\left(x\right)$ the output of a
(CNN+Dense) branch, $x^{a}$ an anchor frame (frame of reference), $x^{p}$ a
positive one (frame matching with reference), $x^{n}$ a negative one (frame
non-matching with reference) and $\alpha$ the margin fixed at 0.5 for both
formulas.

Figure 2: Diagram of the triplet loss network. The layers used in the CNN
blocks are summarised in the Table II. The Distance block computes the
distance between the positive/anchor pair and between the anchor/negative
pair. The distance method is either the Euclidean distance or the Cosine
similarity.

For the second submodule ”Delay Estimation”, two methods were compared. The
first, labelled ”HeatMap”, consists in finding the diagonal of the similarity
matrix containing the highest amount of maximum scores in order to calculate
the distance between this diagonal and the main diagonal. This distance would
represent the delay in number of frames since the main diagonal corresponds to
a delay of zero. By nature this method is limited since it is more likely to
choosing a diagonal with a higher number of values. The second method uses a
dense neural network coupled with a categorical crossentropy loss. The network
takes as input the correspondence matrix flattened and outputs forty neurons
of a single dense layer activated with the Softmax function corresponding to a
positive or negative delay between zero and twenty frames. The layers of this
model named ”DenseDelay” are detailed in the Table III.

TABLE III: The layers making up ”DenseDelay”. Layers |  Units |  Activation
---|---|---
Input |  400 |  X
Dense |  64 |  ReLu
Dense |  32 |  ReLu
Dense |  40 |  Softmax

A CNN architecture taking the similarity matrix as input as well as a
bidirectional LSTM network have also been tested. However, due to their poor
performance, these two networks are not included in this study.

##  IV EXPERIMENTS

In order to test the robustness of our models in different environments and
with different cameras, several experiments were performed. They all aim at
evaluating both the efficiency and the robustness of several systems built by
combining different submodules of ”Matching Frames” with different ”Delay
Estimation” submodules. First we train the different possible models for the
”Matching Frames” submodule seen at in section III. All the trainings were
performed on 100 epochs with a batch of 1 and a learning rate of 0.01 for the
”CNNSiamese” networks and a batch of 32 with a learning rate of 0.001 for the
”TripletEuc” and ”TripletSim” networks. Each model were trained on 4 different
datasets, ”img_zed_p”, ”img_zed_p2”, ”flw_zed_p” and ”flw_zed_p2”. Training on
these datasets allows to analyze the impact of the type of the data used
(optical flow or raw images), evaluate the robustness on different cameras,
and within different environments (see Table I). The trained first submodule
is then combined, here with either the ”DenseDelay” and ”HeatMap” methods. The
obtained systems are then evaluated based on the final delay estimation
performances. As the ”HeatMap” method is a simple comparison algorithm, it did
not require any training. However, ”DenseDelay” is trained each time on the
data of the training set used to train the first submodule associated with it.
These trainings are performed over 50 epochs with a learning rate of 0.01. The
tests are then run with all possible combinations of first and second
submodules, on all datasets that were not used for training.

##  V RESULTS

Figure 3: F1 score & MAE of different combination of submodules on different
datasets (on the x-axis). The models were all trained with the ”img_zed_p”
dataset. Figure 4: F1 score & MAE for systems with different first submodules
and the same ”DenseDelay” as the second submodule tested on different test
sets (on the x-axis). The * in the names is to be replaced with the same type
of data as the training set (in parentheses in the legend): image or optical
flow.

All submodule combinations mentioned in Section IV were compared in our
experiments. But for the shake of clarity, the results not bringing any
valuable information were not reported here. Figures 3 and 4 show the F1 score
estimating the amount of times the systems provided the exact delay, and Mean
Absolute Error (MAE) estimating the number of frames these systems were
mistaken in, and this for different systems and on several datasets
representing different acquisition contexts and data content. The first and
main conclusion from these results is that deep learning-based systems do have
the potential to be used for video synchronization tasks. Indeed as can be
seen in Fig. 4 the TripletEuc+DenseDelay when trained and used with optical
flow data shows high F1 scores ranging between 60 and 90% with an average of
74.8%, low MAE across different datasets with a maximum average error of 4.14
frames with an average of 2.80 (ranging between 0.89 and 4.14).This shows both
its effectiveness, generalizability and robustness on data, material and
context. The second one is the use of optical flow data instead of raw images
improves significantly the robustness of the system as can be seen when
comparing both figures. To go further into detail, based on Fig. 3 we can see
that the DenseDelay method for the delay estimation module outperforms the
HeatMap one.

##  VI CONCLUSION AND FUTURE WORK

In this work, we showed that a stereo video synchronization solution using
deep learning is feasible. We compared several systems and showed that some
architectures perform and generalize well. Although further analyses are
necessary to have a better understanding about the performance and
generalization of the systems, the results presented here pave the way for the
implementation of a production-level system in a near future. In perspective,
we will perform a more thorough study including more parameter variations,
especially regarding the environments and recording setup. We will also
consider the use of the transformer models [17] that proved their efficiency
in computer vision applications in general.

## References

  * [1] S. Ansari, N. Wadhwa, R. Garg, and J. Chen, “Wireless software synchronization of multiple distributed cameras,” in 2019 IEEE International Conference on Computational Photography (ICCP), pp. 1–9, 2019.
  * [2] M. Faizullin, A. Kornilova, A. Akhmetyanov, and G. Ferrer, “Twist-n-sync: Software clock synchronization with microseconds accuracy using mems-gyroscopes,” Sensors, vol. 21, no. 1, p. 68, 2020.
  * [3] L. Zini, A. Cavallaro, and F. Odone, “Action-based multi-camera synchronization,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 3, no. 2, pp. 165–174, 2013.
  * [4] L. Yin, R. Han, W. Feng, and S. Wang, “Self-supervised human pose based multi-camera video synchronization,” in Proceedings of the 30th ACM International Conference on Multimedia, pp. 1739–1748, 2022.
  * [5] I. Freeman, P. Wieschollek, and H. P. A. Lensch, “Robust video synchronization using unsupervised deep learning,” CoRR, vol. abs/1610.05985, 2016.
  * [6] D. Lowe, “Distinctive image features from scale-invariant keypoints,” International Journal of Computer Vision, vol. 60, pp. 91–, 11 2004.
  * [7] I. Melekhov, J. Kannala, and E. Rahtu, “Siamese network features for image matching,” in 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 378–383, 2016.
  * [8] B. Tao, L. Huang, H. Zhao, G. Li, and X. Tong, “A time sequence images matching method based on the siamese network,” Sensors, vol. 21, no. 17, p. 5900, 2021.
  * [9] S. Chopra, R. Hadsell, and Y. LeCun, “Learning a similarity metric discriminatively, with application to face verification,” in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 1, pp. 539–546, IEEE, 2005.
  * [10] R. R. Varior, M. Haloi, and G. Wang, “Gated siamese convolutional neural network architecture for human re-identification,” in European conference on computer vision, pp. 791–808, Springer, 2016.
  * [11] J. Bromley, I. Guyon, Y. LeCun, E. Säckinger, and R. Shah, “Signature verification using a” siamese” time delay neural network,” Advances in neural information processing systems, vol. 6, 1993.
  * [12] D. Cheng, Y. Gong, S. Zhou, J. Wang, and N. Zheng, “Person re-identification by multi-channel parts-based cnn with improved triplet loss function,” in Proceedings of the iEEE conference on computer vision and pattern recognition, pp. 1335–1344, 2016.
  * [13] A. Gordo, J. Almazán, J. Revaud, and D. Larlus, “Deep image retrieval: Learning global representations for image search,” in Computer Vision – ECCV 2016 (B. Leibe, J. Matas, N. Sebe, and M. Welling, eds.), (Cham), pp. 241–257, Springer International Publishing, 2016.
  * [14] G. Farnebäck, “Two-frame motion estimation based on polynomial expansion,” in Image Analysis (J. Bigun and T. Gustavsson, eds.), (Berlin, Heidelberg), pp. 363–370, Springer Berlin Heidelberg, 2003.
  * [15] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in International conference on machine learning, pp. 6105–6114, PMLR, 2019.
  * [16] E. Hoffer and N. Ailon, “Deep metric learning using triplet network,” 2014.
  * [17] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” 2020.