# Medical image classification via quantum neural networks

Natansh Mathur  QC Ware, Palo Alto, USA and Paris, France  Indian Institute of
Technology Roorkee, India  Jonas Landman  QC Ware, Palo Alto, USA and Paris,
France  IRIF, CNRS - University of Paris, France  Yun Yvonna Li  F. Hoffmann
La Roche AG  Martin Strahm  F. Hoffmann La Roche AG  Skander Kazdaghli  QC
Ware, Palo Alto, USA and Paris, France  Anupam Prakash  QC Ware, Palo Alto,
USA and Paris, France  Iordanis Kerenidis  QC Ware, Palo Alto, USA and Paris,
France  IRIF, CNRS - University of Paris, France

(September 2021)

###### Abstract

Machine Learning provides powerful tools for a variety of applications,
including disease diagnosis through medical image classification. In recent
years, quantum machine learning techniques have been put forward as a way to
potentially enhance performance in machine learning applications, both through
quantum algorithms for linear algebra and quantum neural networks. In this
work, we study two different quantum neural network techniques for medical
image classification: first by employing quantum circuits in training of
classical neural networks, and second, by designing and training quantum
orthogonal neural networks. We benchmark our techniques on two different
imaging modalities, retinal color fundus images and chest X-rays. The results
show the promises of such techniques and the limitations of current quantum
hardware.

##  I Introduction

Medical image classification is an important application of machine learning
in the healthcare domain. As deep learning algorithms continue to achieve
accuracy levels on par with humans, numerous efforts are taken to automate
diagnostic, or even prognostic tasks Benjamens . Despite the success of
artificial neural networks to classify, segment, and perform other image
related tasks, complexity of such models and the cost of training continues to
increase. One of the latest visual transformer architecture trained on
imageNet boasts of more than 2 billion parameters, trained used more than 10
000 TPU core-days Zhai . This begs the question whether alternative
technologies can provide better solutions in terms of scalability.

Recently with the advent of the first small-scale noisy quantum computers, the
study of quantum machine learning, Quantum Neural Network (QNN) techniques in
particular, has received a lot of attention. While current quantum hardware is
far from being powerful enough to compete with classical machine learning
algorithms, the first proof of concept demonstrations of interesting quantum
machine learning algorithms have started to appear QNN2018 ; CNN2019 ;
Image2020 ; Semisupervised2020 ; Polyadic2020 ; Dressed2020 ; Supervised2018 ;
Hierarchical2018 ; NearestCentroid2021 ; Kiani20 . In Supervised2018 , the
authors use parametrized quantum circuits where parameters are trained through
stochastic gradient descent, and results on synthetic data are presented in
comparison with results using Support Vector Machines classification. In
Hierarchical2018  the authors outline different classification methods based
on different quantum parametrized circuits, where the classical data is
encoded as separable qubits. A 4-qubit hardware experiment for a binary
classification task between two of the IRIS dataset classes was performed on
an IBM machine with high accuracy. In Polyadic2020 , the authors performed
2-qubit experiments on IBM machines using the IRIS dataset. Other near term
implementations include small fully connected neural networks beer2020training
; abbas2021power , convolutional neural networks cong2019quantum , and
generative models xiaodi_neurips ; coyle2020born . They all rely on the
parameter shift rule mitarai2018quantum  method to compute the gradients of
the gate’s parameters, and give encouraging results on small scale
experiments. More details on noisy intermediate scale quantum algorithms can
also be found in bharti2021noisy .

While such first demonstrations are promising, we neither have solid
theoretical evidence that such quantum architectures can be easily trained to
provide high accuracy, nor can we perform large enough simulations to get
convincing practical evidence of their performance, since we do not have large
enough quantum hardware and classical simulations usually incur an exponential
overhead. For example, architectures that use quantum circuits of constant
depth and gates connecting only neighboring qubits are suitable for
implementation on near-term quantum computers, but cannot act as a fully
connected neural networks since each input qubit can only affect a constant
number of output qubits. Further, the time to train such quantum parametrized
circuits can be quite large, considering phenomena such as barren plateaus and
the fact that designing the architectures, choosing cost-functions and
initializing the parameters are far more complex and subtle than one may
naively think QCNNplateaus2020 ; VQCA2020 ; Train2020 . Further work is
certainly needed to understand the power and limitations of parametrized
quantum circuits for machine learning applications.

Another avenue for quantum supervised learning involves the use of similarity-
based techniques to increase speed and accuracy. In NearestCentroid2021 , a
quantum Nearest Centroid classifier was executed on an 11-qubit trapped-ion
machine of IonQ, and classification of all ten classes of the MNIST
handwritten-digit dataset was performed with an accuracy of $77.5\%$. These
quantum similarity-based machine learning methods are interpretable and their
performance can be theoretically studied, but may require considerably bigger
and better quantum hardware to compete with classical deep learning
classification methods.

Our work is a collaboration between experts in classical machine learning
methods for medical image classification and experts in quantum algorithms to
advance the state-of-the-art of quantum methods for medical image
classification. The hardware demonstration remains a simple proof of concept,
which nevertheless marks progress towards unblocking a number of theoretical
and practical bottlenecks for quantum machine learning.

We focus on quantum neural networks for medical image classification and
implement two different methods: the first uses quantum circuits to assist the
training and inference of classical neural networks by adapting the work in
QNN2020 ; QCNN2019  to be amenable to current quantum hardware; the second
method builds on the recent work on quantum orthogonal neural networks klm21 ,
where quantum parametrized circuits are trained to enforce orthogonality of
weight matrices. Orthogonality is an inherent property of quantum operations,
which are described by unitary matrices, and it has been shown that it can
improve performance of deep neural nets and help avoid vanishing or exploding
gradients nosarzewski2018deep ; wang2020orthogonal ; jia2019orthogonal .

The datasets used to demonstrate the utilities of the two different quantum
techniques come from the MedMNIST series medmnist . Two very different imaging
modalities, the RetinaMNIST dataset of retina fundus images retina  and the
PneumoniaMNIST dataset of chest X-rays pneumonia  are chosen. The same
techniques can also be easily adapted to be tested on all other datasets from
the MedMNIST.

The hardware experiments both for training the quantum neural networks and
forward inference have been performed on the IBM quantum hardware. More
specifically, 5-qubit experiments have been performed on three different
machines with 5, 7, and 16 qubits, and 9-qubit experiments have been performed
in a 16-qubit machine. The IBM quantum hardware is based on superconducting
qubits, one of the two most advanced technologies for quantum hardware right
now along with trapped ions. We also performed the same training and inference
experiments using classical methods as well as simulators of the quantum
neural networks for benchmarking the accuracy of the quantum methods.

Our results show quantum neural networks could be used as an alternative to
classical neural networks for medical image classification. Results of
simulation of the quantum neural networks provide similar accuracy to the
classical ones. From the hardware experiments, we see that for a number of
classification tasks the quantum neural networks can be indeed trained to the
same level of accuracy as their classical counterparts, while for more
difficult tasks, quantum accuracies drop due to the quantum hardware
limitations. The MedMNIST datasets provide good benchmarks for tracking the
advances in quantum hardware, to act as a proxy for checking how well
different hardware perform in classification tasks, and also for testing
different classification algorithms and neural network architectures in the
literature.

##  II Results

###  II.1 Datasets and pre-processing

In order to benchmark our quantum neural network techniques for medical images
we used datasets from MedMNIST, a collection of 10 pre-processed medical open
datasets medmnist . The collection has been standardized for classification
tasks on lightweight $28\times 28$ medical images of 10 different imaging
modalities.

In this work we looked specifically at two different datasets. The first is
the PneumoniaMNIST pneumonia , a dataset of pediatric chest X-ray images. The
task is binary-class classification between pneumonia-infected and healthy
chest X-rays. The second is the RetinaMNIST, which is based on DeepDRiD retina
, a dataset of retinal fundus images. The original task is ordinal regression
on a 5-level grading of diabetic retinopathy severity, which has been adapted
in this work to a binary classification task to distinguish between normal
(class 0) and different levels of retinopathy (classes 1 to 4).

In the PneumoniaMNIST the training set consists of 4708 images (class 0: 1214,
class 1: 3494) and the test set has 624 points (234 - 390). In the RetinaMNIST
the training set has 1080 points (486 - 594) and the test set 400 (174 - 226).
Note that in the RetinaMNIST we have considered the class 0 (normal) versus
classes $\\{1,2,3,4\\}$ together (different levels of retinopathy).

Though the image dimension of the MedMNIST is small compared to the original
images, namely 784 pixels, one cannot load such data on the currently
available quantum computers and thus a standard dimensionality reduction pre-
processing has been done with Principal Component Analysis to reduce the
images to 4 or 8 dimensions. Such a pre-processing indeed may reduce the
possible accuracy of both classical and quantum methods but our goal is to
benchmark such approaches on current quantum hardware and understand if and
when quantum machine learning methods may become competitive.

###  II.2 Quantum data-loaders

Once the classical medical image datasets have been pre-processed we need to
find ways of loading the data efficiently into the quantum computer. We use
exactly one qubit per feature and outline three different ways of performing
what is called a unary amplitude encoding of the classical data points in the
following section.

We use a two-qubit parametrized gate, called Reconfigurable Beam Splitter gate
(RBS) in NearestCentroid2021  and that has appeared also as partial SWAP or
fSIM gate, which is defined as

$RBS(\theta)=\left(\begin{array}[]{cccc}1&0&0&0\\\
0&\cos\theta&\sin\theta&0\\\ 0&-\sin\theta&\cos\theta&0\\\
0&0&0&1\end{array}\right)$ (1)

The RBS gate can be easily implemented on hardware through the following
decomposition in Fig. 1, where $H$ is the Hadamard gate, $R_{y}(\theta)$ is a
single qubit rotation with angle $\theta$ around the $y$-axis, and the two
two-qubit gates represent the CZ gate that flips the sign when both qubits are
in state 1.

Figure 1: A possible decomposition of the $RBS(\theta)$ gate.

The first step is a procedure that given access to a classical data point
$x=(x_{1},x_{2},\ldots,x_{d})\in\mathbb{R}^{d}$, pre-processes the classical
data efficiently, i.e. spending only $\widetilde{O}(d)$ total time (where we
hide logarithmic factors), in order to create a set of parameters
$\theta=(\theta_{1},\theta_{2},\ldots,\theta_{d-1})\in\mathbb{R}^{d-1}$, that
will be the parameters of the $(d-1)$ two-qubit gates we will use in our
quantum circuit. In the pre-processing, we also keep track of the norms of the
vectors.

In the end of the data loader circuits we have created the state

$\mathinner{|{x}\rangle}=\frac{1}{\left\lVert
x\right\rVert}\sum_{i=1}^{d}x_{i}\mathinner{|{e_{i}}\rangle}$ (2)

where the states $\mathinner{|{e_{i}}\rangle}$ are a unary representation of
the numbers $1$ to $d$, using $d$ qubits. Three different types of data loader
circuits appear in Figure 2.

The shallowest data loader is a parallel version which loads $d$-dimensional
data points using $d$ qubits, $d-1$ RBS gates and circuits of depth only $\log
d$ (see first circuit in Figure 2).

While this loader has the smallest depth, it also requires connectivity beyond
the available connectivity on the IBM hardware. For experiments on quantum
computers with a small number of qubits, as the ones available nowadays, the
difference in the depth between the different data loader circuits is not
significant. However as the number of qubits increases, being able to use the
parallel data loader can be very beneficial: for example, for a
1024-dimensional data point, the circuit depth can be reduced from 1024 to 10.

The two data loaders we use in this work have worse asymptotic depth but
respect the nearest neighbors connectivity of the IBM hardware. The first is a
simple diagonal unary loader that still uses $d$ qubits and $d-1$ RBS gates
between neighboring qubits, but it has circuit depth of $d-1$ (see second
circuit in Figure 2). The second one is another unary loader whose depth now
decreases to $d/2$, and that we refer to as semi-diagonal (see third circuit
in Figure 2).

We provide the details for finding these parameters in the Methods section.
Such unary amplitude encoding loaders have been used before in different
scenarios and the details of how to calculate the set of parameters $\theta$
from a classical data point $x$ appear in NearestCentroid2021 ; unary2019 .

Figure 2: The parallel, diagonal, and semidiagonal data loader circuit for an
8-dimensional data point. The X corresponds to the single-qubit Pauli X gate,
while the B-S symbols between two qubits correspond to the two-qubit RBS gate
(B: first qubit, S: second qubit).

###  II.3 Quantum-assisted Neural Networks

Our first method for exploring QNN is a hybrid approach where the quantum
computer is used in the framework of a classical neural network architecture
in order to train the neural network and perform forward inference.

In QNN2020 ; QCNN2019 , it was shown how quantum techniques from linear
algebra can be used in order to potentially offer speedups in the training of
classical neural networks, for both fully connected and convolution nets.
These quantum methods are still outside the capabilities of current quantum
hardware, so we design and implement in this work Noisy Intermediate Scale
Quantum computing (NISQ) friendly subroutines for performing matrix-matrix
multiplications through simpler quantum circuits to estimate inner products
between two vectors. These circuits have the same behaviour as the more
complex quantum linear algebraic procedures, where the quantities that need to
be computed can be estimated through sampling a quantum state that encodes
these quantities in its amplitudes. This allows us to study how this
estimation, instead of an exact computation as is the case in practice
classically, affects the performance of the quantum deep learning methods;
this also allows us to find ways to mitigate the errors of the quantum
hardware.

Important to note here is that simply sampling a quantum state is enough to
estimate the squares of the amplitudes of the desired state, which can be
enough if we know that the quantity we are interested in is positive. For
estimating the inner product of two vectors, which can be either positive or
negative, one needs to find efficient ways to estimate the sign of the
amplitudes as well.

We start by giving a simpler circuit that can be used for computing the square
of the inner product between two normalized vectors $x$ and $w$, as shown in
the left circuit in Figure 3, which is the same type of circuit used in
NearestCentroid2021 . The first part of the circuit is the data loader for
some vector $x$, which in our case will be the normalized data point, and the
second part is the adjoint data loader circuit for some vector $w$, which in
our case will be a row of the weight matrix. The parameters of the gates in
the first part of the circuit are fixed and correspond to the input data
points, while the parameters of the gates of the second part are updated as we
update the weight matrix through a gradient descent computation. Note that in
NearestCentroid2021  the parallel loader was used, since the experiments were
performed in a fully connected trapped ion quantum computer, while here we use
the semi-diagonal ones, due to the restricted connectivity of the
superconducting hardware.

Figure 3: The square inner product and the inner product estimation circuits
with semi-diagonal data loaders for 8-dimensional data points $x$ and $w$. The
RBS gates on the left part of the circuits have parameters that correspond to
the first vector $x$, while the RBS gates on the right part of the circuits
have parameters that correspond to the second vector $w$.

The final state of this circuit has the form

$w\cdot x\mathinner{|{e_{1}}\rangle}+\mathinner{|{G}\rangle}$ (3)

where $\mathinner{|{G}\rangle}$ is an unnormalised state orthogonal to
$e_{1}$, in other words whose first qubit is in state
$\mathinner{|{0}\rangle}$. This implies that when we measure the first qubit
of this quantum state, the probability we get outcome $1$ is exactly $(w\cdot
x)^{2}$, while the outcome is 0 with the remaining probability $1-(w\cdot
x)^{2}$. Thus, if we run the same quantum circuit $N_{shot}$ times and count
the number of times the measurement of the first qubit gave outcome 1, we
denote it by $N_{1}$, then we can estimate the square inner product between
the two vectors as $N_{1}/N_{shot}$. A simple Chernoff bound shows that if we
want the estimate to be within $\epsilon$ of the correct answer, then we need
to take $N_{shot}=O(1/\epsilon^{2})$.

We now need to extend the above circuit in order to be able to estimate the
inner product directly and not its square. While there are different possible
ways to do so, we decided on the way shown in the second circuit in Figure 3,
which adds one extra qubit in the quantum circuit and only a constant number
of gates, independent of the dimension of the data points, and it does not
need multiple measurements that form a complete operator basis on the Hilbert
space of the system.

In short, in order to compute the inner product of two vectors $x$ and $w$,
the extra qubit, denoted as $q0$, is initialized with an $X$ gate to
$\mathinner{|{1}\rangle}$ and used as a control qubit through an RBS gate
between the qubits $q0$ and $q1$ with $\theta=\pi/4$. This way when the qubit
$q0$ is 1 nothing happens, and when it is 0, the quantum circuit described
above consisting of a quantum data loader for $x$ and the adjoint quantum data
loader for $w$ is performed. By performing a final RBS gate with
$\theta=\pi/4$ we end up with a final state of the form

$\left(\frac{1}{2}-\frac{1}{2}w\cdot
x\right)\mathinner{|{e_{1}}\rangle}+\mathinner{|{G}\rangle}$ (4)

where the $\mathinner{|{G}\rangle}$ is an unnormalised state orthogonal to
$e_{1}$, in other words whose first qubit is in state
$\mathinner{|{0}\rangle}$. Thus, as described above, measuring the first qubit
enough times and counting the number of times the outcome is 1, we can
estimate the quantity $\left(\frac{1}{2}-\frac{1}{2}w\cdot x\right)^{2}$ from
which, using the fact that $\left(\frac{1}{2}-\frac{1}{2}w\cdot x\right)\geq
0$, we can provide an estimate of $w\cdot x$ directly. Note also that the
connectivity of the new circuit necessitates one qubit with three neighbors
while all other qubits can be on a line, which is available on the latest IBM
machines.

To see how the above quantum circuits can assist in the training and inference
of classical neural networks: any time there is need for multiplication
between data points and weights, instead of classically performing this
operation, we can employ the quantum circuit. The advantage these circuits
offer is two-fold. First, with the advent of faster quantum computers that
also have the possibility to apply gates in parallel (see for example
parallelGates ), one could take advantage of the fact that these quantum
circuits require only logarithmic depth (using the parallel loaders) in order
to provide a sample from the random variable we are estimating, and thus for
larger feature spaces one can expect a speedup in the estimation. A second
advantage can come from the fact that one can perform training in a different
optimization landscape, that of the parameters of the angles of the quantum
gates and not the elements of the weight matrices. This can provide different
and potentially better models. We will show how to perform such training for
the second quantum method in the section below.

###  II.4 Quantum Orthogonal Neural Networks

The second method we use for medical image classification is based on quantum
orthogonal neural networks, recently defined in klm21 . An orthogonal neural
network is a neural network where the trained weight matrices are orthogonal
matrices, a property that can improve accuracy and avoid vanishing gradients
jia2019orthogonal .

The input data is first loaded into the quantum circuit using a diagonal data
loader and then a circuit in the form of a pyramid made of RBS gates is
applied in order to implement an orthogonal layer of the neural network.

Figure 4: The 8 dimensional diagonal data loader circuit (in red) is
efficiently embedded before the quantum pyramid circuit that corresponds to an
$8\times 8$ orthogonal layer. The RBS gates are here denoted by a vertical
line between the neighboring qubits. The $\alpha$ parameters correspond to the
angles of the RBS gates of the diagonal data loader, while the $\theta$
parameters correspond to the angles of the RBS gates of the quantum pyramid
circuit klm21 .

The circuit in Figure 4 corresponds to an $8\times 8$ orthogonal layer with a
diagonal loader followed by a quantum pyramid circuit. In cases where the
number of inputs and outputs are not the same, the pyramid circuit can be
curtailed to provide a smaller circuit as shown in Figure 5.

Figure 5: A quantum pyramid circuit that corresponds to an $8\times 4$
orthogonal layer. The parameters $\theta$ correspond to the angles in the RBS
gates of the quantum pyramid circuit klm21 .

The important property to note is that the number of parameters of the quantum
pyramid circuit corresponding to a neural network layer of size $n\times d$ is
$(2n-1-d)\cdot d/2$, exactly the same as the number of degrees of freedom of
an orthogonal matrix of dimension $n\times d$. In fact, there is a one-to-one
mapping between any $n\times n$ orthogonal matrix $W$ and the parameters of
the quantum gates of the quantum pyramid circuit, so that the matrix $W$
appears as the submatrix of the unitary matrix specified by the quantum
pyramid circuit when we restrict to the unary basis vectors. In other words,
given an orthogonal weight matrix one can find the corresponding parameters of
the quantum gates and vice versa. The mapping holds for the case of $n\times
d$ matrices as well.

If we denote by $x$ the classical data point that is loaded by the diagonal
loader, and $W$ the orthogonal weight matrix corresponding to the pyramid
circuit with rows $W_{j}$, then the state that comes out of the circuit in
Figure 4 is

$\sum_{j}W_{j}\cdot x\mathinner{|{e_{j}}\rangle}$ (5)

where we see that the amplitudes correspond to the multiplication of the
matrix $W$ with the input $x$.

To be more precise, while the above mentioned pyramid circuit creates the
quantum state that corresponds to the multiplication of the orthogonal matrix
with the datapoint, in order to get the signs of the coordinates of the
resulting vector (and not simply the squared values) one needs to augment the
circuit by adding one extra qubit that works as a control qubit, as well as a
fixed loader for the uniform vector. The final circuit that we implement on
the hardware has the form shown in Figure 6(b).

(a)

(b)

Figure 6:  The final quantum circuit for (a) an 8x8 and (b) an 4x4 quantum
orthogonal layer, where the symbols @-X correspond to a CNOT gate (@: control
qubit, X: target qubit) klm21 .

The state at the end of this circuit is in fact

$\frac{1}{2}\mathinner{|{0}\rangle}\sum_{j=1}^{n}\left(W_{j}x+\frac{1}{\sqrt{n}}\right)\mathinner{|{e_{j}}\rangle}+\frac{1}{2}\mathinner{|{1}\rangle}\sum_{j=1}^{n}\left(W_{j}x-\frac{1}{\sqrt{n}}\right)\mathinner{|{e_{j}}\rangle}$
(6)

On this final state, we can see that the difference in the probabilities of
measuring the first qubit in state $1$ or $0$ and the remaining qubits in the
unary state $e_{j}$ is given by
$\Pr[0,e_{j}]-\Pr[1,e_{j}]=\frac{1}{4}\left(\frac{1}{\sqrt{n}}+W_{j}x\right)^{2}-\frac{1}{4}\left(\frac{1}{\sqrt{n}}-W_{j}x\right)^{2}=W_{j}x/\sqrt{n}$.
Therefore, for each $j$, we can deduce both the amplitude and the sign of
$W_{j}x$.

More details about these calculations appear in klm21 .

Let us provide some remarks about these orthogonal neural networks. First, the
quantum circuits have only linear depth and thus one can use a quantum
computer to perform inference on the datasets in only linear number of steps.
Of course, one can imagine combining such quantum orthogonal layers with other
types of quantum layers, in the form of more general parametrized quantum
circuits, thus creating more advanced quantum neural network architectures.
Second, in klm21  an efficient classical algorithm for training such
orthogonal neural networks is presented that takes the same asymptotic time as
training normal dense neural networks, while previously used training
algorithms were based on singular value decompositions that have a much worse
complexity. The new training of the orthogonal neural network is done with a
gradient descent in the space of the angles of the gates of the quantum
circuit and not on the elements of the weight matrix. We denote this training
algorithm by QPC, after the quantum pyramid circuit which inspired its
creation. Note here that the fact the optimization is performed in the angle
landscape, and not on the weight landscape, can produce very different and at
times better models, since these landscapes are not the same. This is indeed
corroborated by our experimental results as we will discuss below.

###  II.5 Hardware demonstration

####  II.5.1 Superconducting quantum computer

The hardware demonstrations were performed on three different superconducting
quantum computers provided by IBM, with the majority of the experiments
performed on the 16-qubit ibmq_guadalupe machine (see Figure 7).

Figure 7: The 16-qubit ibmq_guadalupe quantum computer.

The other two machines that were used were the 7-qubit ibmq_casablanca and the
5-qubit ibmq_bogota. The quantum hardware was accessed through the IBM Quantum
Experience for Business cloud service.

####  II.5.2 Quantum software

The quantum software development was performed using tools from the QC Ware
Forge platform, including the quasar language, the data loader and the inner
product estimation procedures, as well as the quantum pyramid circuits. The
final circuits were translated into qiskit circuits that were then sent to the
IBM hardware.

The datasets were downloaded from the MedMNIST repository medmnist  and pre-
processed by the sklearn implementation of PCA.

For benchmarking as accurately as possibly against classical fully-connected
neural networks, we used the code from nielsen  for training classical neural
networks, and for the quantum-assisted neural networks we adapted the code to
use a quantum procedure for the dot product computations. For the orthogonal
neural networks, we first designed a new training algorithm for quantum
orthogonal neural networks, based on klm21 , and we also developed code for
classical orthogonal neural networks based on singular value decomposition,
following jia2019orthogonal .

####  II.5.3 Optimizations for the hardware demonstration

(a)

(b)

Figure 8:  The optimized qiskit circuits for (a) a [4,2] quantum-assisted NN
layer and (b) a [4,2] quantum orthogonal NN layer

Before describing the experimental results, we outline here the optimizations
on the circuit design.

First, using a unary encoding for loading the data is very useful in order to
mitigate errors that arise from the hardware. In fact all results that
correspond to outcomes which are not unary strings can be discarded and this
dramatically increases the accuracy of the computation, as we see in the
experimental results below.

Further optimizations were performed with respect to the layout of the
hardware machines and the translation of the RBS gates into native hardware
gates. We provide in Fig. 13 the final qiskit circuits that were used for a
$[8,4]$ layer of the quantum-assisted neural network and of the quantum
orthogonal neural network, where a compilation of the original circuits have
resulted in a reduction of the number of gates, in particular by removing a
number of Hadamard gates that were appearing as pairs. The purple boxes
correspond to the Ry single qubit gates with a parameter noted within the box.
We also provide a $[8,2]$ layer of the quantum orthogonal neural network in
the Methods.

####  II.5.4 Experimental results

Figure 9: Results of experiments for the Quantum assisted Neural Network.
Figure 10: Results of experiments for the Orthogonal Neural Network. QPC
stands for Quantum Pyramidal Circuit and is the classical algorithm simulating
our quantum circuit. QHW is the quantum circuit on the real quantum hardware.
SVB stands for the classical Singular Value Bounded algorithm.

We tested our methods using a number of different datasets, classification
tasks, and architectures that we detail below, and we performed both
simulations and hardware experiments.

In Figures 9 and 10 and Table 2, we show our results, where we provide the AUC
(area under curve) and ACC (accuracy) for all different types of neural
network experiments, for both the training and test sets, for the Pneumonia
and Retina datasets.

TYPES | PNEUMONIA AUC | PNEUMONIA ACC | RETINA AUC | RETINA ACC
---|---|---|---|---
method | layers | training | inference | TRAIN | TEST | TRAIN | TEST | TRAIN | TEST | TRAIN | TEST
qNN | [4,4,2] | CLA | CLA | 0,96 | 0,91 | 0,90 | 0,84 | 0,76 | 0,75 | 0,71 | 0,70
qNN | [4,4,2] | SIM | SIM | 0,96 | 0,91 | 0,90 | 0,84 | 0,77 | 0,75 | 0,72 | 0,73
qNN | [4,4,2] | SIM | QHW | 0,96 | 0,90 | 0,90 | 0,83 | 0,77 | 0,75 | 0,71 | 0,67
qNN | [4,4,2] | QHW | QHW | 0,97 | 0,87 | 0,88 | 0,79 | 0,72 | 0,65 | 0,61 | 0,60
qNN | [8,4,2] | CLA | CLA | 0,98 | 0,90 | 0,93 | 0,85 | 0,86 | 0,83 | 0,80 | 0,78
qNN | [8,4,2] | SIM | SIM | 0,98 | 0,90 | 0,93 | 0,83 | 0,86 | 0,83 | 0,81 | 0,80
qNN | [8,4,2] | SIM | QHW | 0,98 | 0,91 | 0,92 | 0,87 | 0,85 | 0,82 | 0,80 | 0,80
qOrthNN | [4,2] | SVB | SVB | 0,95 | 0,90 | 0,85 | 0,80 | 0,74 | 0,75 | 0,71 | 0,71
qOrthNN | [4,2] | QPC-SIM | QPC-SIM | 0,95 | 0,90 | 0,85 | 0,80 | 0,74 | 0,74 | 0,71 | 0,71
qOrthNN | [4,2] | QPC-SIM | QPC-QHW | 0,95 | 0,89 | 0,82 | 0,76 | 0,75 | 0,73 | 0,71 | 0,68
qOrthNN | [8,2] | SVB | SVB | 0,96 | 0,88 | 0,86 | 0,81 | 0,83 | 0,83 | 0,78 | 0,80
qOrthNN | [8,2] | QPC-SIM | QPC-SIM | 0,96 | 0,88 | 0,86 | 0,80 | 0,83 | 0,84 | 0,78 | 0,79
qOrthNN | [8,2] | QPC-SIM | QPC-QHW | 0,96 | 0,89 | 0,84 | 0,83 | 0,81 | 0,77 | 0,75 | 0,74
Table 1: Results of our experiments on Pneumonia and Retina datasets, reported
for the train set and the test set, both using the AUC and ACC metrics.
Methods are quantum-assisted neural networks (qNN) or quantum orthogonal
neural network (qOrthNN) with different layer architectures. The training and
inference are done classically via standard feedforward/backpropagation (CLA),
or in a quantum-assisted way both on a quantum simulator (SIM) and on quantum
hardware (QHW) for the qNN; and with the Singular-Value Bounded algorithm
(SVB), or with a quantum pyramid circuit algorithm both on a quantum simulator
(QPC-SIM) or on quantum hardware (QPC-QHW) for the qOrthNN. The experiments
that do not involve quantum hardware have been repeated ten times, the mean
values appear on the table and the standard deviation is in most cases $\pm
0.01$ and up to $\pm 0.04$.

As general remarks from these results, one can see that for the PneumoniaMNIST
dataset, both the AUC and ACC are quite close for all different experiments,
showing that the quantum simulations and the quantum hardware experiments
reach the performance of the classical neural networks. We also note that the
quantum-assisted neural networks achieved somewhat higher accuracies than the
orthogonal neural networks. For the RetinaMNIST dataset, the experiments with
8-dimensional data achieve higher AUC and ACC than the ones with 4-dimensional
data. The quantum simulations achieve similar performance to the classical
one, while for the case where both the training and the inference was
performed on a quantum computer, a drop in the performance is recorded, due to
the noise present in the hardware and the higher difficulty of the dataset.

We provide now a more detailed description of the performed experiments.

For the quantum-assisted neural networks, we used two architectures of size
$[4,4,2]$ and $[8,4,2]$, meaning the input size is four or eight dimensional
respectively, there is one hidden layer of four nodes, we used the sigmoid
activation function, and a binary classification task at the end. We also used
two small orthogonal neural networks of size $[4,2]$ and $[8,2]$, meaning the
input size is four or eight dimensional respectively, there is no hidden
layer, we used the sigmoid activation, and a binary classification task at the
end. This is due to the fact that the circuit complexity for each layer is
still quite high for current quantum machines and better quality qubits would
be required in order to scale these architectures up. We performed two
classification tasks, one for the PneumoniaMNIST dataset, and one between
class 0 and classes $\\{1,2,3,4\\}$ of the RetinaMNIST dataset. For each task,
the training and the inference was performed in combinations of classically,
on a quantum simulator, and on quantum hardware.

The experiments involving classical methods or quantum simulators have been
repeated ten times each to extract mean values and error bars for the AUC and
ACC quantities. The mean values appear in Table 1 and the errors are bounded
by $\pm 0.01$ for most cases and up to $\pm 0.04$, which reflects the
randomness in initializing the weights and randomness in the quantum
estimation procedures. For the hardware experiments, we performed the
experiments once per different type of neural network, of layer architecture,
of training method, of inference method and on the training and test sets of
the RetinaMNIST and PneumoniaMNIST datasets. The hardware experiments took
between 45 minutes to several hours. The longest one, training the $[4,4,2]$
quantum-assisted neural network, took more than 10 hours, with the majority of
this time not on the actual quantum hardware but on handling a really large
number of jobs within the quantum cloud service. The training and inference
with classical methods or the quantum simulator took a few seconds to
complete.

In order to benchmark more precisely the performance of the quantum hardware
for the different types of quantum circuits we used, we provide in Methods an
analysis, where we plot the simulated versus the experimental value of the
output of the quantum circuits we used in the quantum-assisted neural networks
over the entire test sets. This allows us to see how the quantum hardware
behaves on real data. For the 5 qubit experiments the hardware results are
very close to the exepcted ones, while for the 9-qubit experiments we see that
with some non-trivial probability the hardware execution diverges from the
simulation.

We note also that the simulations show that the models trained through quantum
methods are robust and the differences in AUC and ACC between different runs
are small, and comparable to the classical models. On the other hand, the
behaviour of the quantum hardware can be quite unstable over time, where
repeating an experiment in different days or weeks can provide quite different
results, not due to the randomness in the training or inference (these
differences can be seen from the simulation results to be very small, namely
about $\pm 0.01$), but due to the levels of noise of the hardware that is
better or worse calibrated at different time periods or due to which specific
subset of qubits one uses within the same quantum computer, or which quantum
hardware machine one uses.

##### Simulation results

Looking at the quantum simulation results, the AUC and ACC of the quantum-
assisted neural networks match those of the corresponding classical ones. This
is to be expected, since the quantum circuits are assisting in the training
but are not changing the classical architectures. The main difference stems
from quantum procedures which estimate and not compute exactly quantities such
as inner products between data points and weights. This way, the quantum-
assisted neural-nets are closer to classical FNNs where noise has been
injected artificially during training and inference. Such noise injection can
actually be beneficial at times, especially in the presence of small training
sets, since they make the models more robust and potentially generalize
better. For the quantum orthogonal neural networks, the performance varied
more compared to the performance of same-size classical orthogonal neural
networks based on Singular Value Decomposition (SVB), and this difference can
be explained by the training methods which are completely different. The new
quantum-inspired way of performing classical training, using the parameters of
the gates of the quantum circuit instead of the elements of the weight
matrices, as described in klm21 , allows for a training in time $O(n^{2})$
instead of the previously known $O(n^{3})$ jia2019orthogonal . Moreover, the
models produced are quite different due to the gradient optimization on the
landscape of the circuit parameters and thus can be a powerful source of more
accurate models. We provide a specific example of a substantially different
training in Figure 11, where we see substantial difference in the ACC and
confusion matrices, where the quantum-inspired way of training achieves $75\%$
accuracy and the SVB-based one does not train and outputs practically always
1.

Figure 11: Results of training of a [32,16,2] orthogonal neural network on the
RetinaMNIST (classes 0 vs. {1,2,3,4}) with training set of size 1080 (486-594)
and test size 400 (174-226) with the SVB-based algorithm (left side: ACC (test
set) = 58.25%) and the simulated quantum pyramid circuit algorithm (right
side: ACC (test set) = 75.25

Thanks to tailor-made quantum simulators for the specific circuits we use, we
were able to perform larger-scale simulations which provide strong evidence of
the scalability of our methods. In Figure 12 we provide an example of
simulation results for quantum-assisted neural networks on the 784 dimensional
images that match the accuracy of the classical NNs.

Figure 12: Training of a [784,64,2] quantum-assisted neural network on the
PneumoniaMNIST with training set of size 4708 (1214 – 3494) and test size of
624 (234-390) with classical training (left side: AUC (test set) = 0.90, ACC
(test set) = 83.65%) and the simulated quantum-assisted training algorithm
(right side: AUC (test set) = 0.89, ACC (test set) = 80.77%).

Last, in Methods we provide simulations of the training of the quantum
orthogonal neural networks for different layer sizes to show in practice the
asymptotic running time of $O(n^{2})$ of the quantum pyramid circuit training
algorithm. We also provide more details about the number of steps that the
quantum-assisted neural networks take in comparison to the classical training.

##### Hardware results

Looking at the results of the hardware demonstration, it is clear that current
hardware is not ready to perform medical image classification in a way that is
competitive to classical neural networks, since the hardware does not have
sufficient number and quality of qubits. Possibly better algorithms and
heuristics are needed to train better and faster models. Nevertheless, the
experiments showed promising results with 5-qubit and to a lesser extent with
9-qubit circuits, providing small-scale confirmation of the methods, while the
larger simulations we performed provide more evidence about the scalability
and future performance.

One can see the overall results of the simulations and hardware experiments in
Table 1.

Overall, classically-trained architectures and quantum inference matches the
classical and quantum simulator performance, while when we performed both
training and inference on a quantum computer, the $[4,4,2]$ neural network
managed to train with a small drop in the accuracy, while the $[8,4,2]$ neural
network did not train, outputting always the same classification value for the
entire dataset.

For the orthogonal neural networks, the results of the hardware demonstration
match the quantum simulation ones (with a small drop only for the RetinaMNIST
and the 9-qubit experiments), showing that overall the hardware has sufficient
quality to perform the necessary quantum circuits. Note that in this case, the
training can always be performed on a classical computer with the optimal
training algorithm developed in klm21 .

We provide some further results, including AUC, ACC and confusion matrices for
a number of the experiments in the Methods.

##  III Methods

###  III.1 Additional qiskit circuits

We provide here the circuits for the $[8,4]$ layer of the quantum-assisted
neural network and a $[8,2]$ layer of the quantum orthogonal neural network.
We have seen above how these circuits scale for larger dimensions.

Figure 13: The optimized qiskit circuit for a [8,4] quantum-assisted NN layer
Figure 14: The optimized qiskit circuit for a [8,2] quantum orthogonal NN
layer

###  III.2 Additional simulation results

As was proved in klm21 , the scaling of the running time of the training of
the orthogonal neural networks based on the quantum pyramid circuit is linear
with respect to the number of parameters. This is corroborated by our results
here as shown in Figure 15. In particular, we trained $[n,n,2]$ quantum
orthogonal neural networks, for different values of $n\in[2,392]$ and saw that
the running time grows indeed as the number of parameters which is
$0.5n^{2}+1.5n-3$. This is asymptotically better than the previously known
training algorithms for orthogonal neural networks that run in time
$O(n^{3})$.

Note as well that the running time of the quantum-assisted neural networks can
be analysed theoretically as in NearestCentroid2021 , where the main
difference with classical fully-connected neural networks is on the
computation of the inner product between vectors. While classically an inner-
product computation between two $n$-dimensional vectors takes $n$ steps, a
quantum circuit for estimating the inner product that uses the parallel
loaders from Fig. 2 has depth of only $2\log(n)-1$. These shallow quantum
circuits need to be repeated a number of times (shots) in order to get a
accurate estimation of the inner product, and in theory to get an
$\epsilon$-accurate result the total number of steps of the quantum circuits
will be $O(\log n/\epsilon^{2})$. We find that repeating the quantum circuits
400 times (independent of the dimension $n$) suffices to get the desired
accuracies. In Fig. 16 we compare the scaling of the steps of the quantum
versus classical computation and see that for images of size $100\times 100$
the quantum steps start becoming fewer. A smaller number of steps does not
imply immediately a faster running time, since one needs to assume that in the
future a single quantum processor can apply gates on different qubits in
parallel (see e.g. parallelGates ) and also take into account the time to
apply one quantum or one classical step.

Figure 15: The actual scaling of the running time for growing dimension $n$ of
training a $[n,n,2]$ orthogonal neural network and the predicted scaling
0.006($0.5n^{2}+1.5n-3$). Figure 16: The scaling of quantum steps for the
inner product estimation between two n-dim vectors (blue: $400(2\log n-1)$)
versus for the classical inner product computation (red: $n$). The crossover
point is for $100\times 100$ pixel images.

###  III.3 Additional hardware results

In Fig. 17,18, and 19 we provide the AUC curves, ACC and confusion matrices
for some more of the experiments we performed, including both the
PneumoniaMNIST and RetinaMNIST datasets, both quantum-assisted and orthogonal
neural networks, and for training and inference on simulators or quantum
hardware.

Figure 17: Training of a [8,4,2] quantum-assisted neural network on the
PneumoniaMNIST with training set of size 2428 (1214 – 1214) and test size 336
(168-168) with training on a quantum simulator and inference on a quantum
hardware (left: AUC (test set) = 0.94, ACC = (test set) 86.01%), and on the
quantum simulator(right: AUC = (test set) 0.95, ACC (test set) = 88.39%).
Figure 18: Training of a [4,4,2] quantum-assisted neural network on the
RetinaMNIST with training set of size 100 (50-50) and test size 400 (174-226)
with both training and testing on a quantum simulator (left: AUC (test set) =
0.74, ACC (test set) = 66.5%), the training on a quantum simulator and the
inference on the quantum hardware (middle: AUC (test set) = 0.73, ACC (test
set) = 63.75%), and last with both the training and testing on the quantum
hardware (right: AUC (test set) = 0.65, ACC (test set) = 60%).  Figure 19:
Training of a [4,2] quantum orthogonal neural network on the RetinaMNIST with
training set of size 1080 (486-594) and test size 400 (174-226) with testing
on a quantum simulator (left: AUC (test set) = 0.73, ACC (test set) = 70%),
and testing on the quantum hardware (right: AUC (test set) = 0.74, ACC (test
set) = 70.75%).

We also present here an analysis for the quantum circuits we used in order to
perform the quantum-assisted neural networks, namely the quantum inner product
estimation circuits, which are useful for other applications beyond training
neural networks.

In particular, we present inference results from the 624 data points in the
test set of the PneumoniaMNIST and the 400 data points in the test set of the
RetinaMNIST, and also both with the [4,4,2] and [8,4,2] quantum-assisted
neural networks, and we check the estimated value of the first node in the
final layer of the neural network (which corresponds to the first class of the
binary classification task) on the simulator versus on the real hardware. The
weights we used were found by performing the training on the quantum
simulator. This allows us to perform the same quantum circuits a large number
of times on different inputs and get an estimate of how well the quantum
hardware can perform this particular application task.

In Figure 20, we see that for the $[4,4,2]$ architecture the results of the
hardware executions are quite close to the expected ones from the simulation
for both data sets. In Figure 21, we see that for the $[8,4,2]$ architecture,
while most of the hardware results agree with the simulations, there is a non-
trivial fraction of the points where an error occurs in the quantum circuit
that causes the results between simulation and hardware execution to diverge.
This does not necessarily translate to a drop in accuracy, this indeed does
not happen as we see in Table 1, since some points that were mis-classified by
the simulator can now be classified correctly on the quantum hardware due to
possible errors in the quantum circuits. It is quite clear that the larger
circuits we used are pushing the boundaries of what the current quantum
machines can perform reliably and one would need better and more robust
hardware to increase confidence on how quantum machine learning techniques can
help in image classification tasks.

Figure 20: Experimental versus simulated value of the first node of the final
layer of the [4,4,2] qNN. Pneumonia: slope=$0.94\pm 0.09$, intercept=$0.04\pm
0.06$; Retina: slope=$0.90\pm 0.20$, intercept=$0.04\pm 0.14$ Figure 21:
Experimental versus simulated value of the first node of the final layer of
the [8,4,2] qNN. Pneumonia: slope=$0.63\pm 0.09$, intercept=$0.19\pm 0.06$;
Retina: slope=$0.50\pm 0.15$, intercept=$0.26\pm 0.10$

##  IV Discussion

We provided here an initial study of quantum neural network techniques for
medical image classification. We initiated the use of the MedMNIST suite of
datasets as a clear, standardized and lightweight benchmark for classification
via quantum techniques and we showed via simulations and hardware
demonstrations how quantum-assisted neural networks and quantum orthogonal
neural networks can be powerful tools for medical image classification. Our
current methods have been designed in a way to retain a clear connection to
classical methods, in order to be able to study theoretically their
scalability, performance and training time. They can of course be combined
with other types of parametrized quantum circuits that explore larger Hilbert
spaces, though if and how such techniques can enhance the training of models
for classical data is still an open question. The simulations and hardware
experiments we performed were extensive with more than a billion shots in
total, showing the power and limitations of current hardware.

Regarding the potential advantages of quantum neural network techniques: as
far as speed is concerned, with the coming of faster quantum machines that
will also have the ability to perform parallel gates on qubits, we can expect
faster quantum training and inference via the log-depth quantum inner product
estimation circuits and more importantly with more advanced quantum procedures
for linear-algebra. However, highly specialized classical hardware, e.g. GPUs,
are extremely efficient in matrix-vector multiplications, so it is very likely
that more powerful quantum techniques will have to be combined with the
simpler techniques described here. We note also that the study of quantum
computing can also lead to novel, faster classical training methods, as is the
case for orthogonal neural networks klm21 .

With respect to the accuracy of the quantum models, we see the potential of
quantum neural networks to provide different and at times better models, by
having access to different optimization landscapes that could avoid effects
like barren plateaus due to quantum properties like, for example,
orthogonality. Note also, that there is a possibility for the quantum-assisted
neural networks to actually provide different models via gradient optimization
on the landscape of the circuit parameters and not directly on the elements of
the weight matrices. We performed such gradient computations for the more
complex orthogonal neural network case, and the techniques can be readily
transferred to this easier case. Whether such different quantum models are
advantageous will depend on the specifics of the use case and further
experimentation is certainly needed.

Last, the application of neural networks is ubiquitous in almost all data
domains, not only in image classification, and, thus, we expect the quantum
methods developed and tested in this work to have much broader impact both in
life sciences and beyond. Such methods will of course continue to be refined
as the hardware evolves.

##  V Data Availability

The data supporting the findings is available from the corresponding author
upon reasonable request.

##  VI Author Contribution

NM developed code for the quantum neural networks, compiled the quantum
circuits, and performed the corresponding experiments and data analysis. JL
developed code for the orthogonal neural networks and performed data analysis.
YL and MS formulated the use case and relevant experiments to demonstrate
usefulness of this work. SK has developed code for the different types of data
loaders. AP has contributed in the theoretical analysis of the quantum neural
networks. IK conceived the project and contributed in all aspects.

##  VII Acknowledgements

This work is a collaboration between QC Ware and Roche. We acknowledge the use
of IBM Quantum services for this work. The views expressed are those of the
authors, and do not reflect the official policy or position of IBM or the IBM
Quantum team. The following members of the Roche pRED Quantum Computing
Taskforce also contributed to this work: Marielle van de Pol, Agnes Meyder,
Detlef Wolf, Stanislaw Adaszewski.

##  VIII Competing Interests

The authors declare no competing interests.

## References

  * [1] Deepdr diabetic retinopathy image dataset (deepdrid), ”the 2nd diabetic retinopathy – grading and image quality estimation challenge,”.  https://isbi.deepdr.org/data.html, 2020.
  * [2] Amira Abbas, David Sutter, Christa Zoufal, Aurélien Lucchi, Alessio Figalli, and Stefan Woerner.  The power of quantum neural networks.  Nature Computational Science, 1(6):403–409, 2021.
  * [3] J Allcock, CY Hsieh, I Kerenidis, and S Zhang.  Quantum algorithms for feedforward neural networks.  ACM Transactions on Quantum Computing 1 (1), 1-24, 2020.
  * [4] Kerstin Beer, Dmytro Bondarenko, Terry Farrelly, Tobias J Osborne, Robert Salzmann, Daniel Scheiermann, and Ramona Wolf.  Training deep quantum neural networks.  Nature communications, 11(1):1–6, 2020.
  * [5] Stan Benjamens, Pranavsingh Dhunnoo, and Bertalan Mesko.  The state of artificial intelligence-based fda-approved medical devices and algorithms: an online database.  npj Digital Medicine volume 3, Article number: 118, 2020.
  * [6] Kishor Bharti, Alba Cervera-Lierta, Thi Ha Kyaw, Tobias Haug, Sumner Alperin-Lea, Abhinav Anand, Matthias Degroote, Hermanni Heimonen, Jakob S Kottmann, Tim Menke, et al.  Noisy intermediate-scale quantum (nisq) algorithms.  arXiv preprint arXiv:2101.08448, 2021.
  * [7] William Cappelletti, Rebecca Erbanni, and Joaquín Keller.  Polyadic quantum classifier.  arXiv:2007.14044, 2020.
  * [8] M Cerezo, A Sone, L Cincio, and P Coles.  Barren plateau issues for variational quantum-classical algorithms.  Bulletin of the American Physical Society 65, 2020.
  * [9] M. Cerezo, Akira Sone, Tyler Volkoff, Lukasz Cincio, and Patrick J. Coles.  Cost-function-dependent barren plateaus in shallow quantum neural networks.  arXiv:2001.00550, 2020.
  * [10] Shouvanik Chakrabarti, Huang Yiming, Tongyang Li, Soheil Feizi, and Xiaodi Wu.  Quantum wasserstein generative adversarial networks.  In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [11] Iris Cong, Soonwon Choi, and Mikhail D. Lukin.  Quantum convolutional neural networks.  Nature Physics 15, 2019.
  * [12] Iris Cong, Soonwon Choi, and Mikhail D Lukin.  Quantum convolutional neural networks.  Nature Physics, 15(12):1273–1278, 2019.
  * [13] Brian Coyle, Daniel Mills, Vincent Danos, and Elham Kashefi.  The born supremacy: quantum advantage and training of an ising born machine.  npj Quantum Information, 6(1):1–11, 2020.
  * [14] Edward Farhi and Hartmut Neven.  Classification with quantum neural networks on near term processors.  arXiv:1802.06002, 2018.
  * [15] Hector Ivan Garcıa-Hernandez, Raymundo Torres-Ruiz, and Guo-Hua Sun.  Image classification via quantum machine learning.  arXiv:2011.02831, 2020.
  * [16] Edward Grant, Marcello Benedetti, Shuxiang Cao, Andrew Hallam, Joshua Lockhart, Vid Stojevic, Andrew G. Green, and Simone Severini.  Hierarchical quantum classifiers.  npj Quantum Information 4, arXiv:1804.03680, 2018.
  * [17] Nikodem Grzesiak, Reinhold Blümel, Kenneth Wright, Kristin M. Beck, Neal C. Pisenti, Ming Li, Vandiver Chaplin, Jason M. Amini, Shantanu Debnath, Jwo-Sy Chen, and Yunseong Nam.  Efficient arbitrary simultaneously entangling gates on a trapped-ion quantum computer.  Nat Commun, 11, 2020.
  * [18] Vojtech Havlicek, Antonio D. Córcoles, Kristan Temme, Aram W. Harrow, Abhinav Kandala, Jerry M. Chow, and Jay M. Gambetta.  Supervised learning with quantum enhanced feature spaces.  Nature volume 567, arXiv:1804.11326, 2018.
  * [19] Kui Jia, Shuai Li, Yuxin Wen, Tongliang Liu, and Dacheng Tao.  Orthogonal deep neural networks.  IEEE transactions on pattern analysis and machine intelligence, 2019\.
  * [20] S Johri, S Debnath, A Mocherla, A Singh, A Prakash, J Kim, and I Kerenidis.  Nearest centroid classification on a trapped ion quantum computer.  npj Quantum Information (to appear), arXiv:2012.04145, 2021.
  * [21] I Kerenidis, J Landman, and A Prakash.  Quantum algorithms for deep convolutional neural networks.  EIGHTH INTERNATIONAL CONFERENCE ON LEARNING REPRESENTATIONS ICLR, 2019.
  * [22] Iordanis Kerenidis, Jonas Landman, and Natansh Mathur.  Classical and quantum algorithms for orthogonal neural networks.  arXiv:2106.07198, 2021.
  * [23] Daniel S. Kermany, Michael Goldbaum, and et al.  Identifying medical diagnoses and treatable diseases by image-based deep learning.  Cell, vol. 172, no. 5, pp. 1122 – 1131.e9,, 2018.
  * [24] Bobak Toussi Kiani, Agnes Villanyi, and Seth Lloyd.  Quantum medical imaging algorithms.  arXiv:2004.02036, 2020.
  * [25] Saurabh Kumar, Siddharth Dangwal, and Debanjan Bhowmik.  Supervised learning using a dressed quantum network with ”super compressed encoding”: Algorithm and quantum-hardware-based implementation.  arXiv:2007.10242, 2020.
  * [26] Kosuke Mitarai, Makoto Negoro, Masahiro Kitagawa, and Keisuke Fujii.  Quantum circuit learning.  Physical Review A, 98(3):032309, 2018.
  * [27] Kouhei Nakaji and Naoki Yamamoto.  Quantum semi-supervised generative adversarial network for enhanced data classification.  arXiv:2010.13727, 2020.
  * [28] Michael A. Nielsen.  Neural networks and deep learning.  Determination Press, 2015.
  * [29] Benjamin Nosarzewski.  Deep orthogonal neural networks.  2018\.
  * [30] Sergi Ramos-Calderer, Adrián Pérez-Salinas, Diego García-Martín, Carlos Bravo-Prieto, Jorge Cortada, Jordi Planagumà, and José I. Latorre.  Quantum unary approach to option pricing.  arXiv:1912.01618, 2019.
  * [31] K Sharma, M Cerezo, L Cincio, and PJ Coles.  Trainability of dissipative perceptron-based quantum neural networks.  arXiv preprint arXiv:2005.12458, 2020.
  * [32] Jiayun Wang, Yubei Chen, Rudrasis Chakraborty, and Stella X Yu.  Orthogonal convolutional neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11505–11515, 2020.
  * [33] Jiancheng Yang, Rui Shi, and Bingbing Ni.  Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis.  arXiv preprint arXiv:2010.14925, 2020.
  * [34] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer.  Scaling vision transformers.  arxiv:2106.04560, 2021.


#  Authentic Emotion Mapping: Benchmarking Facial Expressions in Real News

Qixuan Zhang1, Zhifeng Wang1, Yang Liu1, Zhenyue Qin1, Kaihao Zhang1, Sabrina
Caldwell1, Tom Gedeon2
Australian National University1, Curtin University2
Email: {qixuan.zhang, zhifeng.wang, kaihao.zhang,
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In this paper, we present a novel benchmark for Emotion Recognition using
facial landmarks extracted from realistic news videos. Traditional methods
relying on RGB images are resource-intensive, whereas our approach with Facial
Landmark Emotion Recognition (FLER) offers a simplified yet effective
alternative. By leveraging Graph Neural Networks (GNNs) to analyze the
geometric and spatial relationships of facial landmarks, our method enhances
the understanding and accuracy of emotion recognition. We discuss the
advancements and challenges in deep learning techniques for emotion
recognition, particularly focusing on Graph Neural Networks (GNNs) and
Transformers. Our experimental results demonstrate the viability and potential
of our dataset as a benchmark, setting a new direction for future research in
emotion recognition technologies. The codes and models are at:
https://github.com/wangzhifengharrison/benchmark_real_news

###### Index Terms:

emotion recognition, graph neural network, human-centered computing

##  I Introduction

Emotion recognition has become a part of interactive technology, finding its
place in numerous applications ranging from content delivery [1] to medical
diagnostics [2] and immersive virtual experiences [3]. Traditional approaches
to emotion recognition have heavily relied on RGB images, demanding
significant processing resources to analyze complex facial features in such
high-resolution data [4], [5].

Facial Landmark Emotion Recognition (FLER) presents an alternative that
simplifies this process through the use of basic geometric constructs on the
face [6].

###  I-A Introduction to Facial Landmark Emotion Recognition

Emotion recognition through facial expressions is a significant aspect of
artificial intelligence, aiming to interpret the emotional states by human
facial gestures. The use of facial landmarks, or distinct facial points, is
essential in this domain, enabling the detection of subtle emotional
variations. The subtle shifts in these landmarks are closely linked to
different emotional states and their precise recognition is critical for
enhancing the performance of emotion recognition systems. Compared to
traditional image-based emotion recognition, facial landmark-based emotion
recognition can protect privacy since it does not involve processing entire
facial images [6]. Moreover, in resource-constrained scenarios like edge
computing, landmark-based methods are favored for their efficiency and low
resource consumption. Consequently, Facial Landmark Emotion Recognition (FLER)
has emerged as a pivotal technology across various applications, prominently
in smart devices and real-time interactive systems [7].

###  I-B Graph Neural Networks in Emotion Recognition

The Graph Neural Networks (GNNs) [8], [9] has provided novel methods to handle
complex structured graph data. In the domain of facial landmark emotion
recognition, the GNNs are especially valuable because they can directly work
with graphs composed of facial landmarks, where the edges define the structure
of facial expressions. GNNs are adept at capturing the intricate interactions
among facial landmarks, which are essential for understanding the facial
dynamics associated with different emotional states. Although the application
of GNNs in facial emotion recognition is still in its nascent stages, their
theoretical advantages in handling graph-structured data suggest immense
potential. For example, [4] introduces a directed graph neural network (DGNN)
using graph convolution and landmark features for Facial Emotion Recognition
(FER), efficiently capturing geometric and temporal information from facial
landmarks and addressing the vanishing gradient issue through a stable
temporal block. Rao et al. [10] presents a novel multiscale graph
convolutional network based on facial landmark graphs for Facial Expression
Recognition (FER), addressing issues of data redundancy and bias. However, the
deployment of GNNs in this field also presents challenges such as how to
design effective graph structures to accurately reflect the dynamics of facial
expressions, and balancing the complexity of the models with the performance
constraints of edge computing devices [11].

###  I-C Gaps in Current Research and Our Contribution

Despite numerous studies in the field of facial landmark emotion recognition,
there is still a lack of a systematic benchmarking framework for evaluating
and comparing different neural network models. Such a benchmark is vital for
understanding how various models perform in real-world applications. Most
existing studies have focused on specific models or algorithms without
providing a comprehensive perspective on the performance of different methods
when dealing with facial landmark data. Furthermore, the absence of
benchmarking makes it difficult for researchers and developers to determine
which models are best suited for deployment on resource-limited edge devices.
To address these issues, this paper presents a benchmarking test, including a
range of neural network models and their performance evaluations across
different emotion classification tasks. We consider not only the accuracy of
the models but also their computational efficiency, thus providing significant
references for applications on edge computing devices. The motivation behind
this study is to advance model performance and facilitate practical
applications on edge devices. We believe that through these benchmarks, the
research community can gain a deeper understanding of the strengths and
limitations of different algorithms, leading to the design of solutions that
are more in tune with practical requirements.

The contribution of this paper lies in establishing a comprehensive and
systematic benchmarking for facial landmark emotion recognition. We ensure the
reliability and consistency of our test results through strict experimental
design, providing a solid benchmark for assessing the performance of various
neural network models in the task of facial emotion recognition. Furthermore,
we delve into how the number and configuration of landmarks impact model
performance, revealing the significance of landmark selection in optimizing
model efficiency. Additionally, we have created our dataset and benchmarking
code open source for the community to facilitate further research and
development. These contributions not only provide the academic community with
research resources but also lay a solid foundation for the industry to
implement efficient algorithms, thereby promoting the development of
technology geared towards practical applications.

##  II Related Work

In this section, we explore the landscape of emotion recognition, starting
from its fundamental tasks and the role of facial landmarks, to the latest
advancements in deep learning, particularly focusing on Convolutional Neural
Networks (CNNs) and Transformers. Each subsection delves into significant
developments, challenges, and potential future directions, laying out a
comprehensive overview of the field’s progression and the cutting-edge
techniques shaping the future of facial landmark-based emotion recognition.

###  II-A Introduction to Emotion Recognition and Facial Landmarks

The journey toward automating emotion recognition has covered various fields,
from early psychophysiological studies to the current artificial intelligence
techniques. This section will introduce significant past works, specifically
focusing on the use of facial landmarks as indicators for emotion recognition.
Traditional emotion recognition systems mainly utilized methods based on
geometric features, employing distances and angles between facial landmarks as
direct inputs to classifiers such as Support Vector Machines (SVMs) [12] and
simple neural networks[13].

###  II-B Advancements in Deep Learning for Emotion Recognition

With the revival of neural networks, especially Convolutional Neural Networks
(CNNs), the focus shifted to deeper neural networks for emotion recognition.
Yang et al. [14] introduces a 3D facial expression recognition algorithm using
CNNs and landmark features, robust to pose and lighting changes due to its
reliance on 3D geometric models. Akhand et al. [15] introduces a highly
accurate facial emotion recognition system employing a deep CNN model
optimized via transfer learning and a novel pipeline strategy, addressing
limitations of shallow networks and frontal-only images by fine-tuning with
emotion data for enhanced feature extraction. However, these techniques often
overlooked facial expressions’ temporal dynamics. Addressing this, Kollias et
al. [16] introduces a CNN-RNN approach for dimensional emotion recognition
using multiple features and multi-task frameworks on large emotion datasets,
significantly outperforming existing methods, but these still fell short in
capturing the long-range dependencies crucial for understanding emotional
expressions.

###  II-C Emotion Recognition with Transformers

Enter the era of Transformers, which have redefined the possibilities in
handling sequential data thanks to their self-attention mechanisms. Vaswani et
al. [17] revolutionized sequence modeling with the introduction of the
Transformer model, which has since been adapted beyond the boundaries of
language processing. In the realm of facial emotion recognition, recent
studies have begun to explore the application of Transformers. For instance,
Zhao et al. [18] introduces a geometry-guided FER framework using graph
convolutional networks and transformers, enhancing emotion recognition from
videos by constructing spatial-temporal graphs with facial landmarks and
incorporating attention mechanisms for more informative feature emphasis.
Zheng et al. [19] introduces POSTER, a two-stream Pyramid cross-fusion
Transformer network, aiming to address the key FER challenges: inter-class
similarity, intra-class discrepancy, and scale sensitivity by fusing facial
landmark and image features and employing a pyramid structure for scale
invariance. Hybrid models have also emerged, combining the strengths of CNNs
and Transformers. Karatay et al. [20] presents a deep neural network framework
combining Gaussian mixture models with CNN and Transformer for emotion
detection from videos and images using facial and body features extracted by
OpenPose, addressing various basic emotions.

###  II-D Addressing Diversity and Data Augmentation in Emotion Recognition

Moreover, the impact of data diversity and representativeness has also been a
focal point in the literature. Studies like that of Li et al. [21] highlighted
the challenges posed by variations in ethnicity, age, and lighting conditions,
prompting the development of more robust models. The employment of Generative
Adversarial Networks (GANs) for data augmentation, as explored by
Hajarolasvadi et al. [22], has been one avenue to enhance the diversity and
quantity of training data, thereby improving the generalizability of emotion
recognition systems.

###  II-E Conclusion and Future Directions in Facial Landmark-based Emotion
Recognition

The review of related work underscores a gradual but significant shift towards
sophisticated models that consider both the spatial and temporal aspects of
facial landmarks in emotion recognition. The exploration into Transformer-
based models marks a new frontier in this domain, promising enhanced accuracy
and deeper understanding of emotional states through the advanced modeling of
sequential landmark data. This paper builds upon these foundations, aiming to
further the efficacy and applicability of facial landmark-based emotion
recognition systems.

TABLE I: Emotion Distribution Summary. Emotion | Training | Validation | Test | Total
---|---|---|---|---
Angry | 914 | 100 | 200 | 1,214
Fear | 2,031 | 100 | 200 | 2,331
Happy | 3,091 | 100 | 200 | 3,391
Neutral | 3,536 | 100 | 200 | 3,836
Sad | 3,100 | 100 | 200 | 3,400
Total | 12,672 | 500 | 1,000 | 14,172

##  III Facial expressions in real news dataset

###  III-A Dataset overview

In Table I, striking feature of this distribution is the significant variance
in the number of samples across different emotions. For instance, Neutral,
Happy, and Fear emotions are notably more represented in the dataset, with
counts exceeding 3,500, 3,000, and 2,000 respectively in the Training subset.
The Total column sums the Training and Validation and Test samples, offering a
holistic view of the dataset’s composition, with a grand total of 14,172
instances. This comprehensive distribution is critical for understanding the
dataset’s balance and evaluating the potential biases in the training,
validating testing of emotion recognition models.

###  III-B Basic emotion categories

We capture 5 different daily basic emotion categories in real news. Fig.1
illustrates the distribution of 5 different emotion categories, each
represented by a unique color. The largest portion is ‘Neutral’ at 27.1%,
followed closely by ‘Sad’ at 24.0%. ‘Happy’ represents 23.9% of the chart,
while ‘Fear’ is at 16.4%. The smallest slice is ‘Angry’, making up 8.6% of the
chart. Each category’s percentage is provided.

Figure 1: The number of emotion categories. Figure 2: Landmarks faces

###  III-C Data acquisition and preparation

We initiated our dataset by downloading 318 news videos from internationally
recognized media sources such as ABC, Al Jazeera, NBC News, BBC, CBC, and CCTV
and involved extracting clear, well-defined facial images from these videos.
We established a quality control process that included evaluating image
sharpness, ensuring all facial features were visible and distinct, and
preventing the selection of repetitive images. Additionally, we employed
adaptive padding technology to adjust each image’s frame size based on the
facial dimensions, thereby enhancing the overall quality and consistency of
our image dataset.

To efficiently process and extract faces, we developed scripts to organize
data into a structured directory, calculate adaptive padding, and use the
Laplacian method for sharpness and mean squared error for image similarity. We
also used Dlib’s[23] frontal face detector to identify each face and extract
its key features, making sure all essential landmarks were fully captured and
within the image frame. This method ensured efficient and high-quality face
extraction.

Our comprehensive facial analysis tool is a product of these efforts, adeptly
identifying faces, pinpointing facial features, and recognizing emotional
expressions. We utilized libraries such as DeepFace [24] and Mediapipe [25] to
detect faces, analyze facial characteristics, and determine prevalent
emotional states. Each piece of analysis was precisely documented, offering a
rich dataset for future machine learning applications or detailed statistical
analysis. This tool marks a significant advancement in the field of facial
analysis.

We manually reviewed the images and corresponding landmark data, removing any
irrelevant or low-quality images. After ensuring the integrity and relevance
of the data, we shuffled and divided it into training and test sets,
maintaining a balanced representation of each emotion category. A ’type’
column was added to distinguish between the sets, laying the groundwork for a
robust and reliable facial landmark emotion recognition benchmark. This
structured, detailed approach to data acquisition and processing ensures a
high-quality, diverse dataset that will significantly contribute to
advancements in emotion recognition and facial analysis.

##  IV Methods

In the field of emotion recognition via facial landmarks, leveraging Graph
Neural Networks (GNNs) offers significant potential. This stems from the
structured arrangement of facial features. Facial landmarks, including mouth
corners, eyebrow edges, and nose tips, naturally form a graph-like network.
Their interconnectedness and spatial relationships are key to understanding
emotional expressions. GNNs excel in this context, adeptly identifying the
nuanced arrangements of these landmarks that represent various emotions.

###  IV-A Innovative Convolution Techniques in Graph Neural Networks

The application of convolution techniques in graph neural networks,
particularly through Graph Convolutional Networks (GCNs)[26] and Chebyshev
Spectral CNNs (ChebNets)[27], has been pivotal in decoding complex spatial
interactions among facial landmarks. GCNs contribute by learning localized
features, capturing the subtleties of emotional expressions. In contrast,
ChebNets employ spectral graph theory, focusing on landmark configuration
frequencies for feature extraction. Collectively, these methods provide a
thorough framework for analyzing facial expressions, integrating both spatial
and frequency domain insights to interpret static and dynamic expression
elements.

###  IV-B Attention and Aggregation Strategies for Enhanced Feature Analysis

Graph Attention Networks (GATs)[28] and Graph Sample and Aggregate (GraphSAGE)
[29] networks adopt distinct approaches to emphasize the importance of
individual facial landmarks. GATs utilize attention mechanisms to dynamically
highlight key features, thereby improving the model’s ability to focus on
facial regions most indicative of emotions. Conversely, GraphSAGE introduces a
versatile aggregation framework. This framework is adaptable to various facial
structures and expressions, ensuring stable performance across a wide range of
facial data.

###  IV-C Dynamic and Edge-Enhanced Learning for Expressive Feature
Interpretation

Dynamic Graph CNNs (DGCNNs)[30] and Edge-Conditioned Convolution (ECC)[31]
networks present innovative approaches for analyzing the evolving
characteristics of facial expressions. DGCNNs are specifically designed for
time-series data, effectively tracking the development of emotional indicators
over time, a critical aspect in video-based emotion recognition. ECC networks
complement this approach by factoring in the attributes of edges connecting
facial landmarks. This addition enhances the model’s insight into the impact
of landmark interrelations on the expression of emotions.

###  IV-D Spatial-Temporal and Topological Insights with Advanced GNN
Architectures

Spatial Graph Convolutional Networks[32] and Graph Isomorphism Networks
(GIN)[33] offer novel perspectives in understanding facial landmark
structures, utilizing both spatial-temporal and topological data. Spatial
networks are adept at directly learning from the distribution of landmarks,
effectively identifying the spatial patterns associated with various emotions.
Conversely, GIN focuses on the intrinsic topological structures of these
patterns. This approach enables the model to distinguish between closely
related yet distinct emotional states by analyzing the configuration and
characteristics of neighboring landmarks.

##  V Experiments

###  V-A Experiment settings

We executed experiments on our proposed dataset to evaluate 5 classes of
expressions, utilizing an Ubuntu 20.04 workstation equipped with an NVIDIA
GeForce GTX 3090Ti GPU. Network implementation was carried out using the
Pytorch framework.

###  V-B Accuracy

Accuracy is calculated by the formula:

$\text{Accuracy (\%)}=\left(\frac{P}{N}\right)\times 100$ (1)

where $P$ denotes the number of correct predictions, and $N$ is the total
observations for each emotion category.

TABLE II: Emotion recognition performance by different evaluated approaches on our proposed dataset. Emotion | MLP[34] | GINFormer[35] | GIN[33] | SAGE[29]
---|---|---|---|---
Ave Acc | 28.00% | 33.10% | 30.20% | 32.80%

###  V-C Experimental Results

Multilayer Perceptron (MLP)[34] is a fundamental type of neural network
architecture that has been extensively applied in the field of machine
learning. As a form of a feedforward artificial neural network, MLP consists
of an input layer, multiple hidden layers, and an output layer. In our
experiments, we use one hidden layer as baseline for comparison with other
methods. Each node, except for the input nodes, is a neuron with a RELU
activation function. MLP utilizes backpropagation for training the network. In
our proposed dataset, MLP achieved average score of 28.00% on five basic
emotion categories shown in Table II. Compared with GIN [33], MLP can’t
capture the intrinsic topological structures of facial landmarks. GIN enables
the model to distinguish between closely related yet distinct emotional states
by analyzing the configuration and characteristics of neighboring landmarks.
So, GIN achieved higher accuracy with average score of 30.20%. SAGE [29] is a
novel inductive framework for facial landmarks embedding that leverages facial
landmark features to learn a function capable of generalizing to unseen facial
landmarks, distinguishing it from matrix factorization-based methods and
allowing simultaneous learning of neighborhood topology and facial landmarks
distribution, applicable to both feature-rich and standard structural graphs.
Compared with GIN, SAGE can learn the distribution of facial landmarks and
give prediction of unseen facial landmarks which are important for unseen
emotion categories in the inference process. So, SAGE achieve higher average
accuracy with score of 32.80%. In order to achieve linear complexity,
GINFormer [35] proposes a Graph Transformer, adept at handling diverse
benchmarks. This paper provides a foundational understanding of positional and
structural encodings, categorizing them into local, global, and relative
types. They introduce a novel architecture that scales linearly with the graph
size, serving as a universal function approximator. The proposed framework
integrates three core elements—positional/structural encoding, local message-
passing, and global attention—to achieve state-of-the-art results in graph
representation learning.

TABLE III: Performance comparison of five emotion categories on the our proposed dataset. Emotion |  MLP [34] |  GINFormer [35] |  GIN [33] |  SAGE [29]
---|---|---|---|---
Happy | 33.50% | 46.50% | 40.00% | 49.50%
Sad | 40.50% | 31.50% | 27.00% | 27.00%
Fear | 31.50% | 45.50% | 29.50% | 19.00%
Neutral | 27.50% | 23.50% | 18.00% | 43.00%
Angry | 7.00% | 18.50% | 36.50% | 25.50%

The Table III presents a comparison of performance across different methods on
a proposed dataset. The methods compared include MLP, GINformer, GIN, and
SAGE, with their performances reported in percentages for five different
emotions: Happy, Sad, Fear, Neutral, and Angry.

SAGE outperforms the other methods in recognizing Happy emotions with a 49.50%
accuracy rate. For Sad emotions, MLP leads with 40.50%, while for Fear,
GINformer shows the highest performance at 45.50%. In recognising Neutral
emotions, SAGE is the best performer with 43.00% accuracy, and for Angry
emotions, GIN achieves the highest accuracy at 36.50%.

Figure 3: Five emotional landscapes on transformer attention.

The figure in 3 provides an illustrative comparative study of facial
expression analysis using both advanced deep learning algorithms and classical
psychological techniques. The first row displays a series of original images
portraying a range of emotions—anger, fear, neutrality, sadness, and
happiness—extracted from various news broadcasts. The second row showcases the
application of facial landmark detection by utilizing a system like
MediaPipe[25], which maps out key facial points such as the corners of the
mouth or eyes that are essential for analyzing expressions. The third row
introduces a layer of complexity with the inclusion of a GINFormer’s [35]
attention mechanism, a state-of-the-art deep learning model that employs
transformer architecture to decode the web of facial landmarks and their
connections, pinpointing the areas most relevant for identifying each emotion.
The fourth row adopts a psychological lens, employing the Facial Action Coding
System (FACS) to focus on the activation of Action Units (AUs)—specific facial
muscles associated with emotion display. The patterns overlaying the faces in
this row indicate the intensity and activity of AUs, offering a more granular
perspective on how emotions manifest physically.

The differences of the third and fourth rows illuminates the methodological
divergence between computational and psychological analysis: the GINFormer
model abstracts emotion recognition into patterns within the facial landmark
network, while psychological analysis, through FACS, delves into the intricate
details of muscle movements linked with emotional states. This comparative
visualization underscores the distinctive, yet potentially complementary,
natures of these approaches. Deep learning models like GINFormer excel in
identifying and leveraging facial feature patterns to deduce emotions, while
psychological methods provide an in-depth, muscle-by-muscle dissection of
facial expressions as per the FACS. Such findings highlight the synergy
between AI’s objective pattern recognition and the qualitative, detailed
muscle movement analysis at the heart of psychological research, suggesting a
multidisciplinary fusion that could enrich the field of emotion recognition.

##  VI Conclusion

In this study, we propose a novel benchmark using realistic news videos,
complemented with RGB images and facial landmark coordinates, to enhance
emotion recognition research. Our benchmark provides detailed emotion labels
and facial landmark data, proving to be a reliable and effective tool in our
evaluations. We have demonstrated the practicality and reliability of our
dataset, making it a valuable asset for understanding and analyzing emotions.
We believe this work will inspire further research in emotion recognition,
landmark analysis, and psychological studies. Our contributions aim to support
and advance these fields, offering a robust dataset that reflects real-world
complexity and variability in human emotions.

## References

  * [1] A. Saxena, A. Khanna, and D. Gupta, “Emotion recognition and detection methods: A comprehensive survey,” _Journal of Artificial Intelligence and Systems_ , vol. 2, no. 1, pp. 53–79, 2020.
  * [2] J. Chen, M. Short, and E. Kemps, “Interpretation bias in social anxiety: A systematic review and meta-analysis,” _Journal of Affective Disorders_ , vol. 276, pp. 1119–1130, 2020.
  * [3] J. Marín-Morales, C. Llinares, J. Guixeres, and M. Alcañiz, “Emotion recognition in immersive virtual reality: From statistics to affective computing,” _Sensors_ , vol. 20, no. 18, p. 5163, 2020.
  * [4] Q. T. Ngoc, S. Lee, and B. C. Song, “Facial landmark-based emotion recognition via directed graph neural network,” _Electronics_ , vol. 9, no. 5, p. 764, 2020.
  * [5] K. Zhang, Y. Huang, Y. Du, and L. Wang, “Facial expression recognition based on deep evolutional spatial-temporal networks,” _IEEE Transactions on Image Processing_ , vol. 26, no. 9, pp. 4193–4203, 2017.
  * [6] B. Chen, W. Guan, P. Li, N. Ikeda, K. Hirasawa, and H. Lu, “Residual multi-task learning for facial landmark localization and expression recognition,” _Pattern Recognition_ , vol. 115, p. 107893, 2021.
  * [7] A. Poulose, J. H. Kim, and D. S. Han, “Feature vector extraction technique for facial emotion recognition using facial landmarks,” in _2021 International Conference on Information and Communication Technology Convergence (ICTC)_. IEEE, 2021, pp. 1072–1076.
  * [8] X. Jin, Z. Lai, and Z. Jin, “Learning dynamic relationships for facial expression recognition based on graph convolutional network,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 7143–7155, 2021.
  * [9] L. Zhong, C. Bai, J. Li, T. Chen, S. Li, and Y. Liu, “A graph-structured representation with brnn for static-based facial expression recognition,” in _2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019)_. IEEE, 2019, pp. 1–5.
  * [10] T. Rao, J. Li, X. Wang, Y. Sun, and H. Chen, “Facial expression recognition with multiscale graph convolutional networks,” _IEEE MultiMedia_ , vol. 28, no. 2, pp. 11–19, 2021.
  * [11] R. Zhao, T. Liu, Z. Huang, D. P.-K. Lun, and K. K. Lam, “Geometry-aware facial expression recognition via attentive graph convolutional networks,” _IEEE Transactions on Affective Computing_ , 2021.
  * [12] X. Liu, X. Cheng, and K. Lee, “Ga-svm-based facial emotion recognition using facial geometric features,” _IEEE Sensors Journal_ , vol. 21, no. 10, pp. 11 532–11 542, 2020.
  * [13] B. Hasani and M. H. Mahoor, “Facial expression recognition using enhanced deep 3d convolutional neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2017, pp. 30–40.
  * [14] H. Yang and L. Yin, “Cnn based 3d facial expression recognition using masking and landmark features,” in _2017 seventh international conference on affective computing and intelligent interaction (ACII)_. IEEE, 2017, pp. 556–560.
  * [15] M. Akhand, S. Roy, N. Siddique, M. A. S. Kamal, and T. Shimamura, “Facial emotion recognition using transfer learning in the deep cnn,” _Electronics_ , vol. 10, no. 9, p. 1036, 2021.
  * [16] D. Kollias and S. Zafeiriou, “Exploiting multi-cnn features in cnn-rnn based dimensional emotion recognition on the omg in-the-wild dataset,” _IEEE Transactions on Affective Computing_ , vol. 12, no. 3, pp. 595–606, 2020.
  * [17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [18] R. Zhao, T. Liu, Z. Huang, D. P. Lun, and K.-M. Lam, “Spatial-temporal graphs plus transformers for geometry-guided facial expression recognition,” _IEEE Transactions on Affective Computing_ , 2022.
  * [19] C. Zheng, M. Mendieta, and C. Chen, “Poster: A pyramid cross-fusion transformer network for facial expression recognition,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 3146–3155.
  * [20] B. Karatay, D. Beştepe, K. Sailunaz, T. Özyer, and R. Alhajj, “Cnn-transformer based emotion classification from facial expressions and body gestures,” _Multimedia Tools and Applications_ , pp. 1–43, 2023.
  * [21] S. Li and W. Deng, “Deep facial expression recognition: A survey,” _IEEE transactions on affective computing_ , vol. 13, no. 3, pp. 1195–1215, 2020.
  * [22] N. Hajarolasvadi, M. A. Ramirez, W. Beccaro, and H. Demirel, “Generative adversarial networks in human emotion synthesis: A review,” _IEEE Access_ , vol. 8, pp. 218 499–218 529, 2020.
  * [23] D. E. King, “Dlib-ml: A machine learning toolkit,” _The Journal of Machine Learning Research_ , vol. 10, pp. 1755–1758, 2009.
  * [24] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “Deepface: Closing the gap to human-level performance in face verification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2014, pp. 1701–1708.
  * [25] C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, M. G. Yong, J. Lee _et al._ , “Mediapipe: A framework for building perception pipelines,” _arXiv preprint arXiv:1906.08172_ , 2019\.
  * [26] L. Q. Nguyen, Y. Li, H. Wang, L. M. Dang, H.-K. Song, H. Moon _et al._ , “Facial landmark detection with learnable connectivity graph convolutional network,” _IEEE Access_ , vol. 10, pp. 94 354–94 362, 2022.
  * [27] Q. Li, S. Liu, L. Hu, and X. Liu, “Shape correspondence using anisotropic chebyshev spectral cnns,” in _Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition_ , 2020, pp. 14 658–14 667.
  * [28] A. Prados-Torreblanca, J. M. Buenaposada, and L. Baumela, “Shape preserving facial landmarks with graph attention networks,” _arXiv preprint arXiv:2210.07233_ , 2022.
  * [29] W. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [30] Y. Liu, X. Zhang, Y. Li, J. Zhou, X. Li, and G. Zhao, “Graph-based facial affect analysis: A review,” _IEEE Transactions on Affective Computing_ , 2022\.
  * [31] J. Han, L. Du, X. Ye, L. Zhang, and J. Feng, “The devil is in the face: Exploiting harmonious representations for facial expression recognition,” _Neurocomputing_ , vol. 486, pp. 104–113, 2022.
  * [32] J. Bai, W. Yu, Z. Xiao, V. Havyarimana, A. C. Regan, H. Jiang, and L. Jiao, “Two-stream spatial–temporal graph convolutional networks for driver drowsiness detection,” _IEEE Transactions on Cybernetics_ , vol. 52, no. 12, pp. 13 821–13 833, 2021.
  * [33] K. Xu, W. Hu, J. Leskovec, and S. Jegelka, “How powerful are graph neural networks?” _arXiv preprint arXiv:1810.00826_ , 2018.
  * [34] H. Palo, M. N. Mohanty, and M. Chandra, “Use of different features for emotion recognition using mlp network,” in _Computational Vision and Robotics: Proceedings of ICCVR 2014_. Springer, 2015, pp. 7–15.
  * [35] L. Rampášek, M. Galkin, V. P. Dwivedi, A. T. Luu, G. Wolf, and D. Beaini, “Recipe for a General, Powerful, Scalable Graph Transformer,” _Advances in Neural Information Processing Systems_ , vol. 35, 2022.