11institutetext: Faculty of Sciences - University of Porto, Porto, Portugal
11email<EMAIL_ADDRESS><EMAIL_ADDRESS>22institutetext: INESC TEC,
Porto, Portugal
22email<EMAIL_ADDRESS>
# Model Optimization in Imbalanced Regression

Aníbal Silva 11 Rita P. Ribeiro 1122 Nuno Moniz 2211

###### Abstract

Imbalanced domain learning aims to produce accurate models in predicting
instances that, though underrepresented, are of utmost importance for the
domain. Research in this field has been mainly focused on classification
tasks. Comparatively, the number of studies carried out in the context of
regression tasks is negligible. One of the main reasons for this is the lack
of loss functions capable of focusing on minimizing the errors of extreme
(rare) values. Recently, an evaluation metric was introduced: Squared Error
Relevance Area ($SERA$). This metric posits a bigger emphasis on the errors
committed at extreme values while also accounting for the performance in the
overall target variable domain, thus preventing severe bias. However, its
effectiveness as an optimization metric is unknown. In this paper, our goal is
to study the impacts of using $SERA$ as an optimization criterion in
imbalanced regression tasks. Using gradient boosting algorithms as proof of
concept, we perform an experimental study with 36 data sets of different
domains and sizes. Results show that models that used $SERA$ as an objective
function are practically better than the models produced by their respective
standard boosting algorithms at the prediction of extreme values. This
confirms that $SERA$ can be embedded as a loss function into optimization-
based learning algorithms for imbalanced regression scenarios.

###### Keywords:

Imbalanced Regression . Asymmetric Loss Functions . Model Optimization
Boosting .

##  1 Introduction

Supervised learning assumes there is an unknown function mapping a set of
independent to one or more dependent variables. Learning algorithms aim to
approximate such an unknown function through optimization processes. A key
decision rests on choosing which preference criterion, e.g. a loss function,
should be used. Such a decision entails critical definitions and assumptions
on what should be considered a successful approximation. Most importantly, we
should stress that, these decisions are commonly aimed at minimizing the
overall error across the entire domain of the target (dependent) variable of a
given data set. By assuming that all values are equally important, traditional
optimization processes tend to produce models that have a particular focus on
the most common values of the target variable. This is not the goal of many
real-world applications that configure imbalanced domain learning tasks.

In imbalanced learning, the following holds: i) the target variable has a non-
uniform or skewed distribution; ii) the values across the domain of the target
variable are not equally important; and iii) the focus is on the rare cases,
i.e. values that are poorly represented in the data set. Examples of this type
of predictive task spread from classification to regression. They include
multiple real-world applications in different areas, such as finance, where
the user might be interested in fraud detection, and environmental sciences,
to mitigate the occurrence of natural catastrophes, such as floods and
hurricanes.

Focusing on Imbalanced Regression, several challenges impose the non-
triviality of predicting extreme values. From a supervised learning
perspective, these include two main ones: 1) the definition of suitable and
non-uniform preferences over a continuous and possibly infinite domain of the
target variable; 2) map such preference regarding the extreme values into an
evaluation metric that would adequately allow model selection and, possibly,
optimization. Regarding the first challenge, a proposal [26, 22] exists that
suggests a mapping of the target variable domain into a well-defined space
(the relevance space), which gives information about the relevance of a given
instance based on its target value. As for the second challenge, while there
are some proposals for specially tailored evaluation metrics [27, 8, 13] in an
imbalanced regression scenario, only a very few works exist on including such
metrics in the optimization process. We focus on this second challenge. In
particular, we build on recent work that introduced the Squared Error
Relevance Area ($SERA$) [22] metric. This metric allows for errors of equal
magnitude to have different impacts depending on the relevance of the target
values. Moreover, while it focuses on errors in cases with extreme target
values, it also accounts for the errors committed across all the rest of the
target values, preventing a severe bias towards the extreme values. However,
despite its demonstrated interest in model selection tasks, it is unclear if
it is possible to use it directly in optimization processes.

In this work, our main contribution is to show that $SERA$ can be used as an
optimization loss function in machine learning algorithms, with the ability to
generalize its predictive power for both average and extreme target value
instances. Our demonstration efforts consist of empirical evaluation using
gradient boosting algorithms and a test bed of 36 data sets. Results show us
that, overall, $SERA$ can be used as an optimization loss function. In
addition, when these models optimized with $SERA$ under-perform w.r.t. models
optimized via standard loss functions (e.g. $MSE$), the former still have the
ability to outperform on extreme values, opening horizons to a broader set of
applications in the realm of Imbalance Regression (e.g. Deep Learning).

The paper is organized as follows. In Section 2, we provide a review of recent
related work regarding imbalanced domain learning. In Section 3, we formulate
the problem of Imbalanced Regression, introducing $SERA$, the loss function
that we will use to optimize our models. In addition, we also provide the
details needed to embed this loss function in gradient boosting models, which
will be our baselines. In Section 4, we demonstrate how $SERA$ can be
integrated as a custom optimization metric. In Section 5, we provide the
experimental study and discuss the obtained results. Finally, in Section 6 we
conclude our work with further research directions.

##  2 Related Work

The study of imbalanced learning has been advocated over the years, as it
poses well-known challenges to standard predictive learning tasks [2]. There
are three main strategies to cope with imbalanced domain learning problems:
data-level, algorithm-level and hybrid.

Data-level approaches are the most common ones. They allow for any standard
machine learning to be used, as they act in a pre-processing stage by changing
the data distribution to reduce the imbalance. Generally speaking, we can
group them into under-sampling, over-sampling, generation of synthetic
examples, or their combination. Even though far more data-level methods have
been proposed for classification, few exist for regression (e.g. [3]). An
adaptation of the Synthetic Minority Over-sampling Technique (SMOTE) [4],
initially proposed for classification, has been made for regression and named
SMOTEr [25]. More recently, in the context of Deep Learning, a method to deal
with missing data in an imbalanced regression domain was proposed in [28]:
Deep Imbalanced Regression (DIR). The specificity of this method lies in the
fact that there may be missing values close to a high (low)-representative
neighborhood in the target variable distribution. The distribution of the
target variable is smoothed across the entire domain, considering a similarity
kernel based on statistical properties of the data to estimate missing values.
However, there is a caveat from these approaches – they add artificial
instances that may not represent the reality with which we are faced or remove
common cases that can represent a crucial discriminating aspect for the
predictive task. Moreover, the models are not specifically optimized toward
predicting those rare cases. Thus, it is not easy to assess whether the change
made to the data distribution would effectively map to the intended predictive
focus [18].

Regarding algorithm-level approaches, one of the most popular methods is cost-
sensitive learning [24, 11, 10]. These methods use costs to emphasize/relax
errors committed by predictions at specific target values. An error committed
at a rare or extreme value in imbalanced domains should have a higher cost
than a common value. A problem linked to these methods is that assessing the
exact cost of a given error is highly domain-dependent and not straightforward
[10]. Other methods include the optimization of an asymmetric loss function in
standard learning algorithms [21, 9]. In the context of regression, few
contributions have been made regarding optimization techniques. In [9], the
authors focused on the prediction of extreme values by defining a branched
asymmetric loss function in the residual space, using the Gradient Boosting
algorithm as a training model. Here, the loss function branches into a
quadratic function - for values around the mean, so-called normal; and
exponential function - for extreme values. However, this loss function has the
caveat of a precise definition of normality and thus the requirement of a pre-
defined threshold defined in the residual space.

Recently, an ensemble model that consists of embedding SMOTE in several
variations of the AdaBoost algorithm was proposed, both in classification [5],
and regression tasks [19]. There is, however, a caveat to these models, as
mentioned before - they add artificial instances that may not represent the
reality with which we are faced. This is especially critical when we are
tackling ecological or health domains, where there is no guarantee that the
generated instances may be valid observations.

In standard regression tasks, a given model’s quality or predictive power is
typically assessed by metrics such as the Mean Squared Error ($MSE$) or the
Mean Absolute Error ($MAE$). These metrics have one property in common: the
importance attributed to each observation is uniform, which is not adequate if
we are facing a problem of imbalanced regression. Several metrics were
proposed for regression with the same goal of assigning uneven importance to
instances. The Linear Exponential (LINEX) [27] loss function, which,
controlled by a parameter, differentiates over and underestimations. The
Relevance-Weighted Root Mean Squared Error ($RW\text{-}RMSE$) [17], a modified
version of $RMSE$ which takes into the account the relevance of a given
observation. This metric has the caveat of neglecting values which have a low
relevance. The utility-based F-measure $F_{\beta}^{u}$ [26], is a function
that depends on variations of both the well-known precision and recall,
implemented in the context of regression. It relies on the values of relevance
and utility assigned to a prediction for a given true value. Nevertheless,
both values depend on a given threshold defined in the relevance and utility
space. All the metrics mentioned above share the same limitation: they are
threshold dependent.

In this work, we follow the same principle as in [9], but with another loss
function - Squared Error Relevance Area ($SERA$), a metric recently presented
in the context of imbalanced regression [22]. Using the definition of
relevance associated with the target variable domain, this metric explicitly
gives the notion of asymmetry regarding the loss in different ranges of the
target variable. This metric is not dependent on any threshold and thus does
not face the problems referred to in the above metrics. Due to such
characteristics, it presents the best option for exploring the possibility of
model optimization in imbalanced regression tasks.

##  3 Imbalanced Regression

Consider $\mathcal{D}$ a training set defined as
$\mathcal{D}=\\{\langle\boldsymbol{x}_{i},y_{i}\rangle\\}_{i=1}^{N}$, where
$\boldsymbol{x}_{i}$ is a feature vector of the feature space $\mathcal{X}$
composed by $m$ independent variables and $y_{i}$ an instance of the feature
space $\mathcal{Y}$ that depends on the feature space $\mathcal{X}$. In a
supervised learning setting, our aim is to find the function $f$ that maps the
feature space $\mathcal{X}$ onto $\mathcal{Y}$, $f:\mathcal{X}\to\mathcal{Y}$.
Depending on the nature of $\mathcal{Y}$, we can face a classification (if
$\mathcal{Y}$ is discrete) or a regression problem (if $\mathcal{Y}$ is
continuous). To obtain the best approximation function of $f$, $h$, the
standard approach in supervised learning is to consider a loss function
$\mathcal{L}$, responsible for the optimization of a set of parameters
$\boldsymbol{\Theta}$ which tune a model to extract predictions that better
describes new instances from the feature space $\mathcal{Y}$.

Here, we will focus on the problem of imbalanced regression, i.e., when the
target variable $\mathcal{Y}\in\mathbb{R}$ presents a skewed distribution and
the most important values for the prediction task are extreme (rare) values.
The most commonly used loss function in regression is the Mean Squared Error
($MSE$). However, this metric is not adequate for our prediction task. The
constant which minimizes $MSE$ is the mean of the target variable, $\bar{y}$,
which is counter intuitive for our predictive focus: the extreme values. In an
imbalanced regression scenario, an appropriate loss function should search the
parameter space $\boldsymbol{\Theta}$ such that it encompasses a good
predictive power for both common (around the mean) and uncommon (extremes)
instances of our target domain $\mathcal{Y}$. However, this is not a trivial
task to accomplish.

###  3.1 Relevance Function

In this study, we define an extreme value based on the notion of relevance
introduced by [26]. The authors define a relevance function
$\phi:\mathcal{Y}\to[0,1]$ as a continuous function that expresses the
application-specific bias concerning the target variable domain $\mathcal{Y}$
by mapping it into a $[0,1]$ scale of relevance, where 0 and 1 represent the
minimum and maximum relevance, respectively. With the assumption that extreme
values are the values of interest, authors have also proposed a method that
automatically constructs the $\phi(.)$ function. It achieves that by
interpolating a set of control points provided by the adjusted boxplot, a non-
parametric modification to Tukey’s boxplot, proposed by [14]. In particular,
this method uses the median and the whiskers as the set of key points to
interpolate. In Figure 1 we depict the result of the automatic mapping of the
adjusted boxplot to the relevance space, as introduced by [26], for three
different scenarios based on the type of extremes indicated to be of interest:
low, high or both (default).

Figure 1: The adjusted boxplot (top) of a target variable $y$ and its
automatically inferred relevance function $\phi(y)$ (bottom) for three data
sets with different type of extreme values: low (left), high (middle) and both
(right).

###  3.2 Squared Error Relevance Area (SERA)

Once we have the domain of the target variable mapped into a relevance space,
we now present the asymmetric loss function $SERA$. We will use $SERA$ to
improve the predictive power of extreme values in Gradient Boosting
algorithms, emphasizing that any model which relies on optimization in the
parameter space $\boldsymbol{\Theta}$ could be used to perform this
improvement.

Let $\mathcal{D}=\\{\langle\boldsymbol{x}_{i},y_{i}\rangle\\}_{i=1}^{N}$ be a
data set and $\phi:\mathcal{Y}\to\\{0,1\\}$ a relevance function defined for
the target variable $Y$. Considering the subset
$\mathcal{D}^{t}\subseteq\mathcal{D}$ of instances such that
$\mathcal{D}^{t}=\\{\langle\boldsymbol{x}_{i},y_{i}\rangle\in\mathcal{D}\,|\,\phi(y_{i})\geq
t\\}$, we can define a Squared Error-Relevance ($SER_{t}$) to estimate the
error of a given model with respect to a given cutoff $t$ as

$SER_{t}=\sum_{y_{i}\in\mathcal{D}^{t}}(\hat{y}_{i}-y_{i})^{2}~{}$ (1)

This is the sum of squared errors for all the instances such that the
relevance of the target value is bounded by a given threshold $t$. Since this
metric only depends on instances such that $\phi(y_{i})\geq t$, we will have
that, for any given $\delta\in\mathbb{R}^{+}$, s.t. $t+\delta\leq 1$:
$SER_{t+\delta}\leq SER_{t}$. Finally, its maximum and minimum value are
ascertained when $t=0$ and $t=1$, respectively.

In the same work, the authors took a step further and integrated this estimate
w.r.t. all possible cutoff values (i.e., between 0 and 1). There, they defined
this area as the Squared Error Relevance Area ($SERA$), and it is given by

$SERA=\int_{0}^{1}SER_{t}~{}dt=\int_{0}^{1}\sum_{y_{i}\in\mathcal{D}^{t}}(\hat{y}_{i}-y_{i})^{2}~{}dt$
(2)

This area has some important properties that can help us better understand the
performance of models in an imbalanced regression setting. First, it
encompasses all the possible relevance thresholds constrained in the
definition of $SER_{t}$, removing the need to explicitly define a threshold.
Secondly, it is a decreasing and monotonic function. Note that we are
integrating along with all possible relevance values. Since we know by
definition that $\mathcal{D}^{t+\delta}\subseteq\mathcal{D}^{t}$, the higher
the relevance threshold is, the lower will be the number of instances
considered, but also more relevant. Thus, on the one hand, values of $SER_{t}$
which have a high relevance will have a greater contribution to this area when
compared with instances where the relevance is small. The squared errors of
these latter instances are accounted less times for $SERA$, when compared to
the high relevance instances. On the other hand, the area will be smaller at
points where the relevance is high (since we are only considering the
observations that have a high relevance). In this sense, we are explicitly
penalizing high relevance errors, which are usually harder to optimize while
keeping the entire data domain. Finally, since this metric is built by
integrating $SER_{t}$ over all the relevance domain, which is convex,
convexity is also preserved. Also, this metric must be differentiable. By the
same token, given that $SER_{t}$ is differentiable, so is $SERA$.

##  4 Optimization Loss Function for Imbalanced Regression

We aim to study the possibility and impact of embedding $SERA$ as an
optimization loss function in supervised learning algorithms. Its use in
already implemented learning algorithms only requires the proposal of a custom
loss function, where the only thing we need to provide is the first and
second-order derivatives.

The first-order derivative of $SERA$ is obtained by evaluating the first
derivative w.r.t. a given prediction $\hat{y}_{j}$, as follows

$\displaystyle\frac{\partial SERA}{\partial\hat{y}_{j}}$
$\displaystyle=\frac{\partial}{\partial\hat{y}_{j}}\int_{0}^{1}\sum_{y_{i}\in\mathcal{D}^{t}}(\hat{y}_{i}-y_{i})^{2}~{}dt$
(3)
$\displaystyle=2\int_{0}^{1}\sum_{y_{i}\in\mathcal{D}^{t}}(\hat{y}_{i}-y_{i})~{}\delta_{ij}~{}dt$

where $\delta_{ij}$ is the Kronecker’s delta, which takes value of 1 if $i=j$
and 0 otherwise. Since we need to take into the account all the possible
relevance values a given observation is encompassed in, we can write the
expression above as

$\frac{\partial
SERA}{\partial\hat{y}_{j}}=2\int_{0}^{1}(\hat{y}_{j}-y_{j})\bigg{|}_{y_{j}\in\mathcal{D}^{t}}~{}dt$
(4)

The second derivative w.r.t. a given prediction $\hat{y}_{j}$ is obtained by

$\frac{\partial^{2}SERA}{\partial\hat{y}_{j}^{2}}=2\int_{0}^{1}\boldsymbol{1}(y_{j}\in\mathcal{D}^{t})~{}dt$
(5)

where $\boldsymbol{1}(.)$ is an indicator which takes the value of $1$ if the
argument holds, and $0$ otherwise.

In this paper, we will drive our efforts using Gradient Boosting algorithms,
namely two well-known variants of Gradient Boosting Regression Trees (GBRT)
[12], XGBoost [6] and LGBM [15]. For that, we resort to the implementations
found in R [20] from the packages xgboost [7] and lightgbm [23], respectively.

To approximate the values of the two derivatives, we resort to the trapezoidal
rule with a uniform grid of $T$ equally spaced intervals between $[0,1]$. In
this approximation, we can expand the summations and deduce the following
expressions (see Appendix 0.A for all derivation steps).

$\displaystyle\frac{\partial SERA}{\partial\hat{y}_{j}}$
$\displaystyle\approx\frac{1}{T}\left(1+2n_{j}+\boldsymbol{1}\left(y_{j}\in\mathcal{D}^{t_{T}}\right)\right)(\hat{y}_{j}-y_{j})~{}~{},$
$\displaystyle\frac{\partial^{2}SERA}{\partial\hat{y}_{j}^{2}}$
$\displaystyle\approx\frac{1}{T}\left(1+2n_{j}+\boldsymbol{1}\left(y_{j}\in\mathcal{D}^{t_{T}}\right)\right)$
(6)

where $n_{j}\in[1,T-1]$ is the number of times the instance $y_{j}$
contributes to $SERA$ derivative. From this, we can infer that, for a given
prediction, the first and second-order derivatives will be greater (assuming a
greater error for extreme values) as the relevance increases, as there will be
a higher contribution from $n_{j}$. These derivatives, in addition with
$SERA$, were implemented in R [20].

From now on, we will designate XGBoost and LGBM models optimized with $SERA$
as $\text{XGBoost}^{S}$and $\text{LGBM}^{S}$, while models optimized with
$MSE$ $\text{XGBoost}^{M}$and $\text{LGBM}^{M}$, respectively.

####  4.0.1 Computational Complexity

Another important aspect is the computational complexity introduced by $SERA$.
The trapezoidal rule has a computational complexity of $O(T)$, where $T$ is
the number of steps taken to discretize an integral. $SER_{t}$ has a
computational complexity of $O(|\mathcal{D}^{t}|)$, where $|\mathcal{D}^{t}|$
is the number of instances with relevance higher or equal to a given threshold
$t$. $SERA$ will consider
$|\mathcal{D}^{t_{0}}|+|\mathcal{D}^{t_{1}}|+...+|\mathcal{D}^{t_{T}}|$
instances for all the $T$ steps of the Riemann’s sum. In the worst-case
scenario, all the target values have a constant and maximum relevance equal to
1. In that case, $|\mathcal{D}|$ is the number of instances accounted for all
steps. Thus, $SERA$ will have a computational complexity of
$O(T\times|\mathcal{D}|)$. Regarding the computational complexity introduced
in XGBoost and LGBM, we only need to take into consideration the additional
complexity of the first and second order derivatives. Thus, using the
approximation found in Appendix 0.A, the computational complexity will be
again $O(T\times|\mathcal{D}|)$.

##  5 Experimental Study

Our goal is to answer the research question (Q) that motivated this work: can
$SERA$ be used as an optimization loss function to reduce errors for both
extreme and common values?

In this section, we take into consideration a group of 36 regression data sets
from several domains with an imbalanced distribution on the target variable.
Given these data sets and the models described in Section 4, we will start by
describing our experimental setup in Section 5.1. Namely, the considered data
sets with an imbalanced domain, and the grid-search procedure for parameter
tuning of models. In Section 5.2 we refer to the Bayes Sign Test used to
assess the statistical significance of the results. Given the best parameters
for each model, we present and discuss the obtained results for all data sets
in Section 5.3.

###  5.1 Experimental Setup

To study the effects of using $SERA$ as an optimization loss function, a wide
range of data sets from several domains in the context of imbalanced
regression is used. These data sets, with their respective main properties,
are presented in Table 1. From them, we extracted the number of instances
$|\mathcal{D}|$, the number of nominal (Nom) and numerical (Num) variables. In
addition, and to give a notion of the imbalance present in the target
variable, we resort to the automatic method proposed in [22] that is based on
the adjusted boxplot. We calculated the number of instances such that
$\phi(y)=1$, as representative of the number of extreme (rare) target value
instances $(|\mathcal{D}_{R}|)$ and the Imbalance Ratio ($IR$), calculated as
$|\mathcal{D}_{R}|/|\mathcal{D}|\times 100\%$. Finally, we also include the
type of imbalance for each target variable as follows: if the adjusted boxplot
only presents outliers below or above the respective fence, the type of
extremes is low (L) or high (H), respectively, while if it presents outliers
below and above the fences, the type is both (B).

Table 1: Data sets description: $|\mathcal{D}|$ \- nr of instances, Nom \- nr.
of nominal attributes, Num \- nr. of numeric attributes, $|\mathcal{D}_{R}|$
\- nr. of extreme (rare) instances, i.e. $\phi(y)=1$ $IR$ \- imbalance ratio
and Type - type of extremes.

id | dataset | $|\mathcal{D}|$ | Nom | Num | $|\mathcal{D}_{R}|$ | $IR$ | Type
---|---|---|---|---|---|---|---
1 | diabetes | 35 | 0 | 3 | 4 | 12.90 | H
2 | triazines | 151 | 0 | 61 | 4 | 2.72 | B
3 | a7 | 160 | 3 | 9 | 7 | 4.58 | H
4 | autoPrice | 165 | 10 | 16 | 3 | 1.85 | L
5 | elecLen1 | 399 | 0 | 3 | 4 | 1.01 | H
6 | housingBoston | 407 | 0 | 14 | 40 | 10.90 | B
7 | forestFires | 416 | 0 | 13 | 7 | 1.71 | H
8 | wages | 429 | 7 | 4 | 1 | 0.23 | B
9 | strikes | 501 | 0 | 7 | 1 | 0.20 | H
10 | mortgage | 841 | 0 | 16 | 60 | 7.68 | L
11 | treasury | 841 | 0 | 16 | 79 | 10.37 | L
12 | musicorigin | 848 | 0 | 118 | 15 | 1.80 | B
13 | airfoild | 1203 | 0 | 6 | 11 | 0.92 | H
14 | acceleration | 1387 | 3 | 12 | 30 | 2.21 | B
15 | fuelConsumption | 1413 | 12 | 26 | 27 | 1.95 | B
16 | availablePower | 1443 | 7 | 9 | 75 | 5.48 | B
17 | maxTorque | 1442 | 13 | 20 | 43 | 3.07 | B
18 | debutenizer | 1918 | 0 | 8 | 90 | 4.92 | H

id | data set | $|\mathcal{D}|$ | Nom | Num | $|\mathcal{D}^{R}|$ | $IR$ | Type
---|---|---|---|---|---|---|---
19 | space_ga | 2487 | 0 | 7 | 21 | 0.85 | B
20 | pollen | 3080 | 0 | 5 | 32 | 1.05 | B
21 | abalone | 3343 | 1 | 8 | 374 | 12.60 | B
22 | wine | 5199 | 0 | 12 | 1022 | 24.47 | H
23 | deltaAilerons | 5705 | 0 | 6 | 528 | 10.20 | B
24 | heat | 5922 | 3 | 9 | 39 | 0.66 | B
25 | cpuAct | 6555 | 0 | 22 | 227 | 3.59 | L
26 | kinematics8fh | 6556 | 0 | 9 | 50 | 0.77 | B
27 | kinematics32fh | 6556 | 0 | 33 | 53 | 0.82 | B
28 | pumaRobot | 6556 | 0 | 33 | 91 | 1.41 | B
29 | deltaElevation | 7615 | 0 | 7 | 1802 | 31 | H
30 | sulfur | 8065 | 0 | 6 | 606 | 8.12 | B
31 | ailerons | 11003 | 0 | 41 | 186 | 1.72 | B
32 | elevators | 13280 | 0 | 18 | 1598 | 13.68 | B
33 | calHousing | 16513 | 0 | 9 | 23 | 0.14 | L
34 | house8H | 18229 | 0 | 9 | 305 | 1.70 | B
35 | house16H | 18229 | 0 | 17 | 303 | 1.69 | B
36 | onlineNewsPopRegr | 31716 | 0 | 60 | 2879 | 9.98 | B

To assess the effectiveness of each model, we performed a random partition for
each data set, where 80% will be used to tune the models while the remaining
20% to make predictions under the best model configuration found in a given
data set. To tune the parameters for each model, we will use a grid-search
approach with a $10$-fold stratified cross-validation. We define a workflow of
a given algorithm $j$ as the tuple
$\boldsymbol{W}^{(j)}=(\boldsymbol{M}_{j},\boldsymbol{\Theta}^{(j)})=\\{\mathcal{W}_{q}^{(j)}\\}_{q=1}^{e}$,
where $e$ is the number of different workflows considered for a given tuple,
$\boldsymbol{M}_{j}$ denote the algorithm used, $\boldsymbol{\Theta}^{(j)}$
the respective set of parameters, which are described in Table 2.

Table 2: Models parameters considered for grid search. Model | R Package | Parameters
---|---|---
XGBoost LGBM | xgboost [7] lightgbm [15] | nrounds = $\\{250,500\\}$ max_depth = $\\{3,5,7\\}$ $\eta=\\{10^{-3},10^{-2},10^{-1}\\}$

Given the workflows obtained from the grid-search, we start by providing a
methodology to answer the question that motivated this work, Q. It consists on
the following tasks.

  1. T1:

For each data set, and for each model in $\boldsymbol{M}$, we select the
workflow that had the lowest score according to $SERA$. This score is
calculated by averaging the results obtained by cross-validation on the 80%
partition.

  2. T2:

Given the best workflows, we compare them using the Bayes Sign Test [1]
(Section 5.2). The designation we give to Gradient Boosting models is
$\mathcal{W}^{M}$, in case they are optimized using a standard loss function
($MSE$) and $\mathcal{W}^{S}$ in case they are optimized using $SERA$.

  3. T3:

Next, we train our best workflows for each data set with the partitioned
$80\%$ and, with the remaining $20\%$, we assess the quality of their
predictions (Section 5.3). This quality will also be evaluated by plotting
$SERA$ curves.

###  5.2 Results on Model Optimization

With the top workflows from each model obtained by T1, we can assess the
performance of our models in task T2. For that, we resort to the Bayes Sign
Test. Briefly, this test compares two models on a multi data set scenario by
measuring their score difference (a prior probability) for all data sets,
returning a probability measure (the posterior) hinting if a model is
practically better than another, or if they are equivalent. This equivalence
is measured in a given interval and is defined as the Region Of Practical
Equivalence (ROPE) [16]. The prior $z_{i}$, where $i$ indicates a given data
set, is determined by averaging the normalized difference below for all
$k$-folds

$z_{i}=\frac{1}{10}\sum_{k=1}^{10}\frac{\mathcal{L}_{k}(\mathcal{W}^{S})-\mathcal{L}_{k}(\mathcal{W}^{M})}{\mathcal{L}_{k}(\mathcal{W}^{M})}~{}~{},$
(7)

and taking $\mathcal{L}$ as $SERA$ or $MSE$. After determining this mean
difference for all data sets, we feed into the Bayes Sign Test the vector
$\boldsymbol{z}$ and a ROPE between $[-1\%,1\%]$, returning the posterior
probability $p(z)$ that a given model is practically better or equivalent than
the other.

Figure 2: Comparison between our models optimized with $SERA$,
$\text{LGBM}^{S}$and $\text{XGBoost}^{S}$, against the standard models
$\text{LGBM}^{M}$and $\text{XGBoost}^{M}$. Each color denotes the probability
of our implementation (green) or standard (red) being practically better or
equivalent (blue) to one another according with the Bayes Sign Test with the
ROPE interval $[-1\%,1\%]$. The left and right column denote the results of
the Bayes Sign Test with $MSE$ and $SERA$, respectively.

The results from this evaluation are depicted in Figure 2 and provide us with
two perspectives according to the considered error metrics. Regarding $MSE$
(left column), the standard models are practically better (with a $p(z)$ of
0.92 for $\text{XGBoost}^{M}$and a $p(z)$ of 0.7 for $\text{LGBM}^{M}$).
Concerning $SERA$ (right column), results tell us that both algorithms with
our optimization are practically better (with a $p(z)$ of 0.65 for
$\text{XGBoost}^{S}$and a $p(z)$ of 0.76 for $\text{LGBM}^{S}$).

Thus, from a statistical points of view and in a model optimization scenario,
there is a clear trade-off between the standard and our models when assessing
their scores with different metrics. This was somewhat expected as these
metrics have a different predictive focus as it was already mentioned above.
Nevertheless, from this test we are able to infer the ability of $SERA$ as a
loss function to lower the errors obtained in a problem of Imbalance
Regression. With this, we finish our second task T2 and partially answered our
main question Q. Next, we aim to show that $SERA$ do in fact improve the
predictive power for both common and extreme values in an out-of-sample
scenario (i.e., using our test data).

###  5.3 Results in Out-of-Sample

Using the parameters found in the best workflows obtained for each model and
for each data set in task T1, we train our models and assess their predictions
with the ($20\%$) out-of-sample data. Given these predictions, we calculated
$SERA$ and $MSE$ for all the models in the considered data sets (cf. Tables 3
and 4 in Appendix 0.B).

From the obtained results, we initially take into consideration a rank
evaluation of our models. For that purpose, and for a given data set, the rank
of 1 is assigned to the model which provided the lowest score. Figure 3
depicts the rank distribution for each model over all the considered data
sets. In $MSE$-based ranking (left column), $\text{XGBoost}^{M}$ had a median
rank of 1, followed by $\text{LGBM}^{M}$ with a rank of 2, and finally
$\text{XGBoost}^{S}$ and $\text{LGBM}^{S}$, both with a median rank of 3. In
$SERA$-based ranking (right column), both $\text{XGBoost}^{M}$and
$\text{XGBoost}^{S}$ had a median rank of two, followed by $\text{LGBM}^{S}$
with a median rank of 3, and finally $\text{LGBM}^{M}$ with a rank of 4. From
these ranking distributions, we can take a somewhat expected conclusion:
standard models were the top performers when evaluated under $MSE$, and an
unexpected one: under $SERA$, $\text{XGBoost}^{M}$ and $\text{XGBoost}^{S}$
compete with each other.

Figure 3: Rank distribution of models by $MSE$ and $SERA$ results in out-of-
sample.

Next, we focus on each algorithm independently. While for XGBoost,
$\text{XGBoost}^{S}$ had the lowest score in 3 and 18 of the data sets when
evaluated by $MSE$ and $SERA$, respectively, for LGBM, $\text{LGBM}^{S}$ had a
better score in 11 and 26 of the data sets when evaluated by $MSE$ and $SERA$.

Although the standard models had the lowest score in more data sets than ours,
it does not mean that, for a given data set, those errors were the lowest
across all the relevance domain. To study this statement, we follow [22] and
show $SERA$ curves for six selected data sets. These curves are built by
calculating the error $SER_{t}$ as the relevance threshold $t$ for $\phi(y)$
increases and are shown in Figure 4.

In addition to $SERA$ curves, we also define a turning point for each data
set. That point is the minimum relevance value $\varphi$ for which a model
optimized with $SERA$ has a $SERA$ estimate for all the values with a
relevance greater or equal to $\varphi$, i.e.
$SERA_{\phi(.)\geq\varphi}(\mathcal{M}^{S})$, lower than the $SERA$ estimate
obtained by the standard model in the same conditions, i.e. for all the values
with a relevance greater or equal to $\varphi$, i.e.
$SERA_{\phi(.)\geq\varphi}(\mathcal{M}^{M})$. More formally, and for a
specific data set, the turning point is then a threshold $\phi_{t}$ obtained
by

$\phi_{t}=\text{min}\\{\varphi\in[0,1]~{}|~{}SERA_{\phi(.)\geq\varphi}(\mathcal{M}^{S})<SERA_{\phi(.)\geq\varphi}(\mathcal{M}^{M})\\}.$
(8)

In the curves of Figure 4, the turning points are represented by dashed lines,
and the shadowed regions represent the relevance domain for which the
condition above holds.

Figure 4: $SERA$ curves for six selected data sets. The first row provides
data sets where our models had the lowest error, while the second row provides
data sets where $\text{XGBoost}^{M}$ had the lowest error. The highlighted
region in each graph depicts the turning point where models optimized with
$SERA$ started to have a lower error w.r.t. standard models.

The plots from the first row of Figure 4 show us $SERA$ curves where models
that were optimized with $SERA$ had the lowest score. As depicted, for house8H
data set, $\text{XGBoost}^{S}$ had the lowest score in most of the relevance
domain ($\phi_{t}=0.09$). For housingBoston data set, although
$\text{XGBoost}^{S}$ has the lowest $SERA$, its turning point occurs at a
higher threshold ($\phi_{t}=0.48)$. Finally, we show the curves for
musicorigin data set for which $\text{LGBM}^{S}$ had the lowest score. Here,
the turning point also occurs at a very low relevance ($\phi_{t}=0.01$).

Regarding the second row of plots of Figure 4, we show three data sets where
$\text{XGBoost}^{M}$ had the lowest $SERA$ score. From these examples, we see
that although $\text{XGBoost}^{M}$ was the top performer, there are several
turning points in the relevance domain for which our models surpassed
$\text{XGBoost}^{M}$ ($\phi_{t}=0.22$, $\phi_{t}=0.23$, $\phi_{t}=0.78$ for
deltaElevation, space_ga and strikes data sets, respectively).

From this analysis we can conclude that $SERA$ can be used as a loss function
to reduce errors in both extreme and common values (first row of Figure 4).
Even when models optimized with $SERA$ do not provide the best score across
the whole relevance domain, they can still perform better for different
relevance domains (second row from Figure 4). With this, we conclude our task
T3 and answered the question that motivated this work Q.

##  6 Conclusions

In this work, we addressed the problem of embedding $SERA$ as an optimization
loss function to improve the predictive power for both extreme and normal
values. We used as our baselines two well-known variations of Gradient
Boosting models, XGBoost and LGBM. Results showed that the embedment of $SERA$
as an optimization loss function in our algorithms provided the ability of
reducing errors for both extreme and normal values. In addition, when
underperformed by standard models, our algorithms were able to provide a more
accurate prediction in a high relevance domain. For the sake of
reproducibility, all the experiments are available in
github111https://github.com/anibalsilva1/IRModelOptimization.

Finally, we highlight that hyper-parameter optimization, due to the
construction of $SERA$, is not restricted to the considered algorithms,
opening the use of this loss function to several domains of Machine Learning
(e.g. Deep Learning) in the realm of Imbalanced Regression.

## Acknowledgements

This work was supported by the CHIST-ERA grant CHIST-ERA-19-XAI-012, and
project CHIST-ERA/0004/2019 funded by FCT.

## References

  * [1] Benavoli, A., Corani, G., Demšar, J., Zaffalon, M.: Time for a change: a tutorial for comparing multiple classifiers through bayesian analysis. Journal of Machine Learning Research 18(77), 1–36 (2017)
  * [2] Branco, P., Torgo, L., Ribeiro, R.P.: A survey of predictive modeling on imbalanced domains. ACM Comput. Surv. 49(2) (2016). https://doi.org/10.1145/2907070
  * [3] Branco, P., Torgo, L., Ribeiro, R.P.: Pre-processing approaches for imbalanced distributions in regression. Neurocomputing 343, 76–99 (2019). https://doi.org/10.1016/j.neucom.2018.11.100
  * [4] Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: Smote: Synthetic minority over-sampling technique. J. Artif. Int. Res. 16(1), 321–357 (2002)
  * [5] Chawla, N.V., Lazarevic, A., Hall, L.O., Bowyer, K.W.: Smoteboost: Improving prediction of the minority class in boosting. In: Knowledge Discovery in Databases: PKDD 2003. pp. 107–119. Springer Berlin Heidelberg, Berlin, Heidelberg (2003)
  * [6] Chen, T., Guestrin, C.: Xgboost: A scalable tree boosting system. In: 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. p. 785–794. KDD ’16, ACM (2016). https://doi.org/10.1145/2939672.2939785
  * [7] Chen, T., He, T., Benesty, M., Khotilovich, V., Tang, Y., Cho, H., Chen, K., Mitchell, R., Cano, I., Zhou, T., Li, M., Xie, J., Lin, M., Yifeng, G., Li, Y., Yuan, J.: xgboost: Extreme Gradient Boosting (2022), https://CRAN.R-project.org/package=xgboost
  * [8] Christoffersen, P.F., Diebold, F.X.: Further results on forecasting and model selection under asymmetric loss. Journal of Applied Econometrics 11(5), 561–571 (1996)
  * [9] Ehrig, L., Atzberger, D., Hagedorn, B., Klimke, J., Döllner, J.: Customizable asymmetric loss functions for machine learning-based predictive maintenance. In: 2020 8th International Conference on Condition Monitoring and Diagnosis (CMD). pp. 250–253 (2020). https://doi.org/10.1109/CMD48350.2020.9287246
  * [10] Elkan, C.: The foundations of cost-sensitive learning. 17th International Conference on Artificial Intelligence 1, 973–978 (2001)
  * [11] Fan, W., Stolfo, S.J., Zhang, J., Chan, P.K.: Adacost: Misclassification cost-sensitive boosting. In: 16th International Conference on Machine Learning. p. 97–105. ICML ’99, Morgan Kaufmann Publishers Inc. (1999)
  * [12] Friedman, J.H.: Greedy function approximation: A gradient boosting machine. The Annals of Statistics 29(5), 1189–1232 (2001), http://www.jstor.org/stable/2699986
  * [13] Granger, C.W.J.: Outline of forecast theory using generalized cost functions. Spanish Economic Review 1(2), 161–173 (1999). https://doi.org/10.1007/s101080050007
  * [14] Hubert, M., Vandervieren, E.: An adjusted boxplot for skewed distributions. Computational Statistics & Data Analysis 52, 5186–5201 (2008). https://doi.org/10.1016/j.csda.2007.11.008
  * [15] Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., Liu, T.Y.: Lightgbm: A highly efficient gradient boosting decision tree. In: 31st International Conference on Neural Information Processing Systems. p. 3149–3157. NIPS’17, Curran Associates Inc. (2017)
  * [16] Kruschke, J., Liddell, T.: The bayesian new statistics: Two historical trends converge. SSRN Electronic Journal (2015). https://doi.org/10.2139/ssrn.2606016
  * [17] Moniz, N.: Prediction and Ranking of Highly Popular Web Content. Ph.D. thesis, Faculty of Sciences - University of Porto (2017)
  * [18] Moniz, N., Monteiro, H.: No free lunch in imbalanced learning. Knowledge-Based Systems 227, 107222 (2021). https://doi.org/10.1016/j.knosys.2021.107222
  * [19] Moniz, N., Ribeiro, R., Cerqueira, V., Chawla, N.: Smoteboost for regression: Improving the prediction of extreme values. In: IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA). pp. 150–159 (2018). https://doi.org/10.1109/DSAA.2018.00025
  * [20] R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2020), https://www.R-project.org/
  * [21] Rengasamy, D., Rothwell, B., Figueredo, G.P.: Asymmetric loss functions for deep learning early predictions of remaining useful life in aerospace gas turbine engines. In: International Joint Conference on Neural Networks (IJCNN). pp. 1–7 (2020). https://doi.org/10.1109/IJCNN48605.2020.9207051
  * [22] Ribeiro, R., Moniz, N.: Imbalanced regression and extreme value prediction. Machine Learning 109, 1–33 (2020). https://doi.org/10.1007/s10994-020-05900-9
  * [23] Shi, Y., Ke, G., Soukhavong, D., Lamb, J., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., Liu, T.Y., Titov, N.: lightgbm: Light Gradient Boosting Machine (2022), https://CRAN.R-project.org/package=lightgbm
  * [24] Sun, Y., Kamel, M.S., Wong, A.K., Wang, Y.: Cost-sensitive boosting for classification of imbalanced data. Pattern Recognition 40(12), 3358–3378 (2007). https://doi.org/10.1016/j.patcog.2007.04.009
  * [25] Torgo, L., Ribeiro, R.P., Pfahringer, B., Branco, P.: Smote for regression. In: Progress in Artificial Intelligence. pp. 378–389. Springer (2013)
  * [26] Torgo, L., Ribeiro, R.: Utility-based regression. In: Knowledge Discovery in Databases: PKDD 2007. pp. 597–604. Springer Berlin Heidelberg (2007)
  * [27] Varian, H.R.: A bayesian approach to real estate assessment. Studies in Bayesian econometric and statistics in Honor of Leonard J. Savage pp. 195–208 (1975)
  * [28] Yang, Y., Zha, K., Chen, Y., Wang, H., Katabi, D.: Delving into deep imbalanced regression. CoRR abs/2102.09554 (2021), https://arxiv.org/abs/2102.09554

##  Appendix 0.A $SERA$ numerical approximation

$SERA$ and its derivatives are approximated by the trapezoidal rule with a
uniform grid of $T$ equally spaced intervals with a step of 0.001, as follows.

$\begin{split}SERA&=\int_{0}^{1}SER_{t}\,dt\\\
&\approx\frac{1}{2T}\sum_{k=1}^{T}\left(SER_{t_{k-1}}+SER_{t_{k}}\right)\\\
&=\frac{1}{2T}\left(SER_{t_{0}}+2SER_{t_{1}}+...+2SER_{t_{T-1}}+SER_{t_{T}}\right)\\\
&=\frac{1}{T}\bigg{(}\sum_{k=1}^{T-1}SER_{t_{k}}+\frac{SER_{t_{0}}+SER_{t_{T}}}{2}\bigg{)}\\\
&=\frac{1}{T}\bigg{(}\frac{1}{2}\\!\sum_{y_{i}\in\mathcal{D}^{t_{0}}}(\hat{y}_{i}-y_{i})^{2}+\\!\sum_{y_{i}\in\mathcal{D}^{t_{1}}}(\hat{y}_{i}-y_{i})^{2}+...\\\
&\qquad+\\!\sum_{y_{i}\in\mathcal{D}^{t_{T-1}}}(\hat{y}_{i}-y_{i})^{2}+\frac{1}{2}\\!\sum_{y_{i}\in\mathcal{D}^{t_{T}}}(\hat{y}_{i}-y_{i})^{2}\bigg{)}\end{split}$
(9)

Similarly, the derivative of $SERA$ w.r.t. a given prediction $\hat{y}_{j}$ is
obtained by

$\begin{split}\frac{\partial
SERA}{\partial\hat{y}_{j}}&\approx\frac{1}{T}\frac{\partial}{\partial\hat{y}_{j}}\bigg{(}\frac{1}{2}\\!\sum_{y_{i}\in\mathcal{D}^{t_{0}}}(\hat{y}_{i}-y_{i})^{2}+\\!\sum_{y_{i}\in\mathcal{D}^{t_{1}}}(\hat{y}_{i}-y_{i})^{2}+...\\\
&\qquad+\\!\sum_{y_{i}\in\mathcal{D}^{t_{T-1}}}(\hat{y}_{i}-y_{i})^{2}+\frac{1}{2}\\!\sum_{y_{i}\in\mathcal{D}^{t_{T}}}(\hat{y}_{i}-y_{i})^{2}\bigg{)}\\\
&=\frac{1}{T}\bigg{(}\sum_{y_{i}\in\mathcal{D}^{t_{0}}}(\hat{y}_{i}-y_{i})\delta_{ij}+2\sum_{y_{i}\in\mathcal{D}^{t_{1}}}(\hat{y}_{i}-y_{i})\delta_{ij}+...\\\
&\qquad+\\!\sum_{y_{i}\in\mathcal{D}^{t_{T-1}}}(\hat{y}_{i}-y_{i})\delta_{ij}+\frac{1}{2}\\!\sum_{y_{i}\in\mathcal{D}^{t_{T}}}(\hat{y}_{i}-y_{i})\delta_{ij}\bigg{)}\\\
&=\frac{1}{T}\bigg{(}(\hat{y}_{j}-y_{j})\bigg{|}_{y_{j}\in\mathcal{D}^{t_{0}}}+2(\hat{y}_{j}-y_{j})\bigg{|}_{y_{j}\in\mathcal{D}^{t_{1}}}+...\\\
&\qquad+2(\hat{y}_{j}-y_{j})\bigg{|}_{y_{j}\in\mathcal{D}^{t_{T-1}}}+(\hat{y}_{j}-y_{j})\bigg{|}_{y_{j}\in\mathcal{D}^{t_{T}}}\bigg{)}\\\
&=\frac{1}{T}\bigg{(}(\hat{y}_{j}-y_{j})\bigg{|}_{y_{j}\in\mathcal{D}^{t_{0}}}+2\sum_{k=1}^{T-1}(\hat{y}_{j}-y_{j})\bigg{|}_{y_{j}\in\mathcal{D}^{t_{k}}}+(\hat{y}_{j}-y_{j})\bigg{|}_{y_{j}\in\mathcal{D}^{t_{T}}}\bigg{)}\\\
&=\frac{1}{T}\bigg{(}\boldsymbol{1}\left(y_{j}\in\mathcal{D}^{t_{0}}\right)+2\sum_{k=1}^{T-1}\boldsymbol{1}\left(y_{j}\in\mathcal{D}^{t_{k}}\right)+\boldsymbol{1}\left(y_{j}\in\mathcal{D}^{t_{T}}\right)\bigg{)}(\hat{y}_{j}-y_{j})\end{split}$
(10)

Note that any given instance $y_{j}$ will always have at least zero relevance,
i.e. $\phi(y_{j})\geq 0$, so the first term of Equation (10) will always be
taken into account. Nevertheless, not all the summation terms will be
considered for cases where $\phi(y_{j})<1$. With this in mind, we define

$n_{j}=\sum_{k=1}^{K_{j}}\boldsymbol{1}\left(y_{j}\in\mathcal{D}^{t_{k}}\right)~{}~{}$
(11)

where $n_{j}$ is the number of times the instance $y_{j}$ contributes to
$SERA$ derivative, where $K_{j}\in\left[1,T-1\right]$. Equation (10) becomes
then

$\frac{\partial
SERA}{\partial\hat{y}_{j}}\approx\frac{1}{T}\left(1+2n_{j}+\boldsymbol{1}\left(y_{j}\in\mathcal{D}^{t_{T}}\right)\right)(\hat{y}_{j}-y_{j})~{}~{}$
(12)

In this context, the second derivative for a given prediction $\hat{y}_{j}$ is
obtained by

$\frac{\partial^{2}SERA}{\partial\hat{y}_{j}^{2}}\approx\frac{1}{T}\left(1+2n_{j}+\boldsymbol{1}\left(y_{j}\in\mathcal{D}^{t_{T}}\right)\right)~{}~{}$
(13)

We now proceed to a study on the degree of error committed by using the
approximations above (Eqs. (12), (13)) against the use of the trapezoidal rule
directly on Eqs. (4) and (5). For that, we resort to the predictions obtained
for $\text{XGBoost}^{S}$. The rationale is the following: 1) for each data
set, we compute the first and second derivatives using both methods; 2) we
calculate the absolute difference between the results obtained from both
methods; 3) we average these differences over all instances.

The results obtained using this evaluation are depicted in the left box plot
of Figure 5.

Figure 5: Left: Absolute error differences for the first and second
derivatives between our approximations and the trapezoidal rule. Right:
Execution time (in seconds) of the first and second derivative for a given
data set, under our approximation and the trapezoidal rule.

Results show that both first and second derivative approximations have a minor
error ($\approx 10^{-12}$). Sim On the right box plot of Figure 5, we also
show the difference in execution time using both methods for each data set.
Here, the execution time is measured as the time taken to evaluate both the
first and second derivatives. As we can see, there is a non-negligible
difference between our approximation and the trapezoidal rule as the size of a
data set increases.

##  Appendix 0.B Tables of Results

In this section, we report the $MSE$ and $SERA$ results obtained in out-of-
sample of each data set.

Table 3: $MSE$ results in out-of-sample, with the best models per data set in
bold. Model (#wins): $\text{XGBoost}^{M}$ (27), $\text{XGBoost}^{S}$ (2),
$\text{LGBM}^{M}$ (6), $\text{LGBM}^{S}$ (1).

id | $\text{XGBoost}^{S}$ | $\text{XGBoost}^{M}$ | $\text{LGBM}^{S}$ | $\text{LGBM}^{M}$
---|---|---|---|---
1 | 6.41e-01 | 3.36e-01 | 5.56e-01 | 5.15e-01
2 | 1.00e-02 | 1.41e-02 | 1.42e-02 | 1.55e-02
3 | 4.60e+01 | 4.76e+01 | 4.67e+01 | 4.56e+01
4 | 8.64e+07 | 2.39e+07 | 6.94e+07 | 1.60e+07
5 | 9.81e+05 | 3.34e+05 | 9.18e+05 | 3.75e+05
6 | 1.44e+01 | 1.08e+01 | 2.25e+01 | 1.84e+01
7 | 1.68e+04 | 4.62e+03 | 1.82e+02 | 4.29e+02
8 | 2.30e+01 | 1.78e+01 | 2.36e+01 | 1.69e+01
9 | 1.23e+06 | 4.88e+05 | 1.32e+06 | 4.80e+05
10 | 1.98e+00 | 2.57e-02 | 6.84e-01 | 7.49e-01
11 | 3.42e+00 | 8.19e-02 | 1.28e+00 | 9.15e-01
12 | 3.46e-01 | 3.00e-01 | 3.24e-01 | 3.10e-01
13 | 1.20e+01 | 2.17e+00 | 7.59e+01 | 1.65e+02
14 | 7.72e-01 | 4.87e-01 | 2.13e+00 | 2.14e+00
15 | 2.37e-01 | 1.11e-01 | 8.85e-01 | 7.62e-01
16 | 5.27e+01 | 2.53e+01 | 2.23e+02 | 2.53e+02
17 | 4.21e+01 | 1.33e+01 | 7.23e+02 | 1.02e+03
18 | 2.24e-02 | 4.66e-03 | 2.16e-02 | 8.45e-03

id | $\text{XGBoost}^{S}$ | $\text{XGBoost}^{M}$ | $\text{LGBM}^{S}$ | $\text{LGBM}^{M}$
---|---|---|---|---
19 | 1.46e-02 | 9.74e-03 | 1.59e-02 | 1.41e-02
20 | 2.93e+00 | 2.35e+00 | 2.72e+00 | 2.45e+00
21 | 6.66e+00 | 4.87e+00 | 5.96e+00 | 5.59e+00
22 | 2.02e+00 | 3.48e-01 | 9.53e-01 | 7.85e-01
23 | 4.78e-08 | 2.67e-08 | 3.36e-08 | 2.47e-08
24 | 2.06e+00 | 6.77e-01 | 4.43e+01 | 4.18e+01
25 | 2.00e+01 | 4.52e+00 | 1.25e+02 | 8.32e+01
26 | 2.36e-03 | 2.00e-03 | 1.01e-02 | 1.06e-02
27 | 7.12e-02 | 6.56e-02 | 1.00e-01 | 1.13e-01
28 | 8.44e-05 | 6.40e-05 | 8.02e-05 | 6.76e-05
29 | 7.93e-06 | 2.01e-06 | 9.01e-06 | 1.99e-06
30 | 2.07e-03 | 1.93e-03 | 2.64e-03 | 2.89e-03
31 | 3.89e+02 | 2.16e+02 | 1.37e+03 | 1.23e+03
32 | 5.85e-06 | 4.37e-06 | 1.65e-05 | 1.39e-05
33 | 8.39e+09 | 3.38e+09 | 1.19e+10 | 1.47e+10
34 | 1.49e+09 | 1.09e+09 | 1.38e+09 | 1.13e+09
35 | 1.23e+09 | 8.63e+08 | 1.12e+09 | 9.48e+08
36 | 9.89e+07 | 1.01e+08 | 1.07e+08 | 9.95e+07

Table 4: $SERA$ results in out-of-sample, with the best models per data set in
bold. Model (#wins): $\text{XGBoost}^{M}$ (16), $\text{XGBoost}^{S}$ (14),
$\text{LGBM}^{S}$ (5), $\text{LGBM}^{M}$ (1).

id | $\text{XGBoost}^{S}$ | $\text{XGBoost}^{M}$ | $\text{LGBM}^{S}$ | $\text{LGBM}^{M}$
---|---|---|---|---
1 | 3.86e+00 | 1.48e+00 | 3.36e+00 | 2.13e+00
2 | 1.47e-01 | 2.50e-01 | 2.03e-01 | 2.66e-01
3 | 1.70e+03 | 1.75e+03 | 1.58e+03 | 1.61e+03
4 | 1.18e+09 | 3.63e+08 | 9.99e+08 | 1.87e+08
5 | 3.50e+07 | 1.34e+07 | 3.59e+07 | 1.76e+07
6 | 6.39e+02 | 6.47e+02 | 9.58e+02 | 1.15e+03
7 | 5.82e+05 | 2.27e+05 | 4.66e+03 | 1.41e+04
8 | 7.06e+02 | 9.11e+02 | 9.06e+02 | 8.94e+02
9 | 8.70e+07 | 5.47e+07 | 8.47e+07 | 5.48e+07
10 | 1.90e+02 | 1.91e+00 | 4.43e+01 | 4.86e+01
11 | 4.22e+02 | 1.12e+01 | 1.32e+02 | 8.08e+01
12 | 3.03e+01 | 3.24e+01 | 2.32e+01 | 3.57e+01
13 | 1.07e+03 | 2.47e+02 | 1.05e+04 | 1.59e+04
14 | 5.11e+01 | 4.35e+01 | 2.03e+02 | 2.34e+02
15 | 1.63e+01 | 1.46e+01 | 9.51e+01 | 9.17e+01
16 | 5.77e+03 | 6.72e+03 | 4.26e+04 | 5.02e+04
17 | 3.52e+03 | 3.09e+03 | 1.29e+05 | 2.12e+05
18 | 4.71e+00 | 1.14e+00 | 4.94e+00 | 2.59e+00

id | $\text{XGBoost}^{S}$ | $\text{XGBoost}^{M}$ | $\text{LGBM}^{S}$ | $\text{LGBM}^{M}$
---|---|---|---|---
19 | 2.38e+00 | 2.30e+00 | 2.90e+00 | 3.58e+00
20 | 4.84e+02 | 6.05e+02 | 5.47e+02 | 7.81e+02
21 | 1.92e+03 | 2.26e+03 | 2.25e+03 | 2.86e+03
22 | 4.30e+01 | 1.54e+02 | 1.70e+02 | 5.50e+02
23 | 1.14e-05 | 1.65e-05 | 1.11e-05 | 1.62e-05
24 | 4.05e+02 | 3.48e+02 | 2.25e+04 | 2.63e+04
25 | 1.01e+04 | 1.58e+03 | 5.71e+04 | 4.39e+04
26 | 9.31e-01 | 1.12e+00 | 4.27e+00 | 5.02e+00
27 | 2.90e+01 | 3.82e+01 | 5.48e+01 | 7.93e+01
28 | 2.24e-02 | 2.82e-02 | 3.13e-02 | 4.05e-02
29 | 1.78e-03 | 1.33e-03 | 2.04e-03 | 1.47e-03
30 | 2.68e+00 | 2.93e+00 | 3.66e+00 | 4.61e+00
31 | 2.77e+05 | 2.23e+05 | 1.93e+06 | 2.21e+06
32 | 5.84e-03 | 6.20e-03 | 1.87e-02 | 1.98e-02
33 | 1.21e+13 | 5.64e+12 | 1.61e+13 | 2.36e+13
34 | 2.82e+12 | 3.25e+12 | 3.08e+12 | 3.85e+12
35 | 2.24e+12 | 2.33e+12 | 2.30e+12 | 3.07e+12
36 | 7.57e+11 | 7.81e+11 | 7.24e+11 | 7.46e+11


# SigScatNet: A Siamese + Scattering based Deep Learning Approach for
Signature Forgery Detection and Similarity Assessment

Anmol Chokshi*, Vansh Jain*, Rajas Bhope*, Sudhir Dhage  Sardar Patel
Institute of Technology, All authors contributed equally*

###### Abstract

The surge in counterfeit signatures has inflicted widespread inconveniences
and formidable challenges for both individuals and organizations. This
groundbreaking research paper introduces SigScatNet, an innovative solution to
combat this issue by harnessing the potential of a Siamese deep learning
network, bolstered by Scattering wavelets, to detect signature forgery and
assess signature similarity. The Siamese Network empowers us to ascertain the
authenticity of signatures through a comprehensive similarity index, enabling
precise validation and comparison. Remarkably, the integration of Scattering
wavelets endows our model with exceptional efficiency, rendering it light
enough to operate seamlessly on cost-effective hardware systems. To validate
the efficacy of our approach, extensive experimentation was conducted on two
open-sourced datasets: the ICDAR SigComp Dutch dataset and the CEDAR dataset.
The experimental results demonstrate the practicality and resounding success
of our proposed SigScatNet, yielding an unparalleled Equal Error Rate of
3.689% with the ICDAR SigComp Dutch dataset and an astonishing 0.0578% with
the CEDAR dataset. Through the implementation of SigScatNet, our research
spearheads a new state-of-the-art in signature analysis in terms of EER scores
and computational efficiency, offering an advanced and accessible solution for
detecting forgery and quantifying signature similarities. By employing
cutting-edge Siamese deep learning and Scattering wavelets, we provide a
robust framework that paves the way for secure and efficient signature
verification systems.

###### Index Terms:

Signatures, Siamese deep learning, Scattering wavelets, forgery detection,
similarity assessment

##  I Introduction

With the rising digitization of papers and transactions, signature forgery has
become a major concern. Traditional manual signature verification procedures
are frequently subjective, time-consuming, and error-prone. As a result, there
is a growing demand for automated systems and technologies that successfully
identify and prevent signature fraud using computer vision, machine learning,
and biometric identification approaches. A number of papers from the research
community have worked on using machine learning for detecting forgery in
signatures. While their work has provided useful insights into their
methodology, there are a few areas where improvements can be made in order to
make this system more robust.

Most papers from the research community use a basic CNN network to identify if
the signature is fraudulent or not. For employing this, they use a dataset
containing signature images and then apply pre-processing techniques and feed
this into the network. Now for testing, they provide an image that is then
compared with the images from the dataset. The output of this testing comes
out as a prediction which is then compared with a threshold to verify if the
image is a forgery or not. While this system works, it has its drawbacks.
Firstly, signatures can vary naturally due to factors like the signer’s
attitude, exhaustion, or writing pen type, making it difficult to set a
general threshold that successfully distinguishes authentic from faked
signatures. As a result, this strategy may produce more false negatives or
false positives, compromising the system’s reliability. Furthermore, the
success of CNN-based signature forgery detection is largely dependent on the
quality and variety of the training dataset. Building a comprehensive and
representative dataset is a difficult endeavor that necessitates great effort
in collecting a diverse range of authentic and counterfeit signatures.
Additionally, CNN-based signature forgery detection systems often struggle to
account for sophisticated forgery techniques, such as skilled imitations or
forged signatures created using advanced tools. These techniques can
potentially deceive the CNN network, leading to incorrect predictions and
reduced accuracy. Furthermore, the reliance on pre-processing techniques
introduces additional complexity and computational overhead, impacting the
overall efficiency of the system.

The SigScatNet employs a more complex form of CNN. The Siamese model with
Scattering wavelets counteracts the limitations discussed by leveraging its
unique architecture and comparison-based approach. The Siamese model, as
opposed to a simple CNN’s basic threshold-based decision-making, learns a
similarity measure that allows for a more flexible and comprehensive
evaluation of authentic or forged signatures. The Siamese model can reflect
the intrinsic complexity and diversity in signatures by examining pairwise
interactions between signatures, minimizing false negatives and false
positives. Furthermore, the model achieves lightweight efficiency while
preserving critical spatial and temporal information by using Scattering
wavelets. This improves performance in capturing dynamic elements of
signatures and adjusting to evolving forgery tactics, overcoming the limits of
standard CNN-based methods.

##  II Related Work

The identification of signature forgery using machine learning techniques has
seen significant advancements, as evidenced by several research papers.
Convolutional neural networks (CNNs) combined with image processing techniques
have been commonly employed in many papers. [1] developed a deep learning-
based off-line signature verification system using a Convolutional Neural
Network (CNN) and a novel method for extracting local features to distinguish
between genuine and forged signatures, with potential applications in
organizations with a limited number of individuals for training and future
forgery detection. [2] proposes a system for signature verification and
forgery detection wherein this paper uses CNN and deep learning to extract
unique features from pre-processed signatures, which are then compared to the
signatures stored in the system to determine whether the signature is real or
fake. The system was tested on a dataset containing 750 signatures from 150
individuals, and the results showcased how the proposed system works and then
stated that their method is more accurate and faster in the detection of
forged signatures than a trained professional. [3] developed a solution using
Convolutional Neural Network (CNN) to detect forgery in handwritten signatures
by training the model on a dataset of signatures and predicting the
authenticity of provided signatures. [4] developed an interactive software
implementation of signature verification that includes both general learning
from a population of signatures and special learning from multiple samples of
an individual’s signature, resulting in improved accuracy. [5] proposes a
signature verification system that uses pre-processing techniques such as RGB
to Gray Scale Conversion, image binarization, and Gaussian blurring. The
system uses contour and SIFT features for feature extraction and SVM and
K-Means algorithms for model training.

Siamese networks have gained attention in recent research. In [6], the
methodology used involves the CEDAR dataset. The signatures are processed
using various techniques such as median filter, Otsu method, and morphological
operations. The Siamese neural network is then trained and tested on this
dataset, resulting in high accuracy rates. [7] conducted offline signature
verification using a Siamese neural network based on one-shot learning, which
achieved successful classification with fewer signature images. By utilizing
the one-shot learning method, the model distinguished between genuine and
forged signatures without the need for a large amount of labeled data.
Experimental results on signature datasets demonstrated across 4NSigComp2012,
SigComp2011, 4NSigComp2010, and BHsig260 datasets, showcasing the
effectiveness of their proposed approach. [8] uses a Siamese Neural Network
(SNN) with a contrastive loss function and robust embedding to verify genuine
and forged offline signatures. The experiments were conducted on challenging
datasets like MCYT-75, GPDS, and CEDAR, and the proposed method outperformed
other state-of-the-art methods in terms of accuracy. Their results showcased
that the use of inter-quartile range and median absolute deviation measures in
the embedding vector increases the accuracy of the algorithm for GPDS and
MCYT-75 databases relating to the signature forgery domain. However, existing
approaches often face limitations in handling natural variations that
contribute to increased false positives and false negatives, or they require
computationally expensive networks with high hardware requirements.

This research paper aims to address these limitations by striking a balance
between the robustness of the system and the canonical form in signature
forgery detection. The objective is to develop an approach that effectively
captures natural factors influencing signatures while maintaining
computational efficiency. By employing a Siamese network and incorporating
novel techniques such as Scattering wavelets, the SigScatNet aims to achieve
accurate and lightweight signature forgery detection. This would contribute to
enhancing the reliability and effectiveness of signature verification systems,
improving overall security and trust in various domains where signatures play
a crucial role.

##  III Proposed Methodology

###  III-A SigScatNet Architecture

Figure 1: SigScatNet Architecture

Fig. 1 illustrates the SigScatNet architecture, showcasing the integration of
Scattering wavelets and the Siamese network. This architecture forms the core
of SigScatNet, enabling the robust detection of signature forgery and precise
assessment of signature similarity.

The SigScatNet architecture can be divided into two essential parts: the
Scattering wavelet component and the Siamese network component.

####  III-A1 Scattering Wavelets

Scattering wavelets are an effective technique for extracting useful
characteristics from signals or images in signal processing and image
analysis. Both local and global information is captured in a multiscale,
translation-invariant representation provided by this method. The wavelet
transform, which breaks down a signal into several frequency components at
various scales, serves as the foundation for the scattering transform. The
scattering transform is better suited for tasks like pattern identification
and signal classification since it keeps the information about phase and
magnitude in contrast to conventional wavelet transforms. The scattering
transform works via cascading wavelet transformations and modulus non-
linearities. A series of wavelet filters are convolved with the input signal
or image at various scales. Insight into local frequency is captured by the
resulting wavelet coefficients. The magnitude data is then captured by
applying the modulus non-linearity to the wavelet coefficients. A depiction of
multiscale scattering is created by repeating this procedure at several
scales.

The scattering transform is incredibly helpful when it comes to signature
analysis. The level of depth in a signature can range from broad strokes to
finer details. We can record these various informational scales by applying
the scattering transform on signature images. The signatures’ global and low-
frequency components, as represented by the coefficients determined from lower
scales, are what give the signatures their overall shape and structure. On the
other hand, finer details and high-frequency components, such as minor texture
patterns and variation in stroke, are captured by the coefficients from higher
scales. We consider a handwritten signature as an illustration in Fig. 2. The
scattering transform produces a scattering representation with several scales
of coefficients when applied to the signature image. Lower scale coefficients
encode the signature’s overall shape and structure, encapsulating its general
properties. The factors pick up finer features at larger scales, including
differences in the thickness of the signature’s strokes, their curvature, and
their texture patterns. In this process, the selection of scales and
scattering factors is critical. We may regulate the level of detail recorded
in the scattering representation by choosing the right scales. Finer features
can be captured at higher scales, however doing so may also result in an
increase in noise or artifacts. It’s crucial to find a balance and pick scales
that accurately depict the pertinent distinctive traits without obscuring the
representation. The scattering transform is further influenced by the choice
of wavelet filters, their number per scale, and their sizes. These variables
affect how the transform behaves and how many features are extracted. These
parameters need to be carefully thought out in order to guarantee ideal
feature extraction and discriminative power for signature analysis jobs.

Figure 2: Signatures at J=2 on performing Scattering Wavelet Operation.

####  III-A2 Siamese Model

The Siamese model is a potent method for comparing two signatures to assess
their similarity. In our scenario, the Siamese model is used to compare a
signature under scrutiny with a base or reference signature, allowing us to
determine whether the signature is genuine or not. The Siamese model comprises
of two identical neural networks with the same architecture and weights. Each
network receives an input signature, processes it through a series of
convolutional and fully connected layers to extract relevant features. The
Conv2D layers in the model use filters with the following configurations: 16
filters in the first two layers and 32 filters in the last two layers. The
kernel size for all Conv2D blocks is set to $3\times 3$, and the activation
function used is ReLU. The features from the two networks are concatenated and
passed into a similarity function.

The model takes an input image of size $180\times 300\times 3$. Before the
scattering network’s operation, the image is converted to grayscale. It then
goes through the convolutional blocks, resulting in an output array of size
128, which represents the signature’s features. A similarity score is
generated by comparing the feature embeddings of the two signatures using
these 128-sized arrays. One commonly used similarity metric is the Euclidean
distance, also known as the L2 distance.

The Euclidean distance $D$ between the feature embeddings of input image 1 and
input image 2 is computed as seen in equation (5), where $n$ denotes the
number of features (128 in our case):

$D=\sqrt{\sum_{i=1}^{n}(x_{i}-x_{i})^{2}}\hskip 30.0pt\text{(5)}$

This Euclidean distance is normalized between 0 and 1, where a lower $D$
denotes a higher degree of similarity between the signatures.

The Siamese model is trained using a triplet loss function during the training
phase [9]. The triplet loss function is a distance-based loss function
designed to learn embeddings by minimizing the distance between the anchor and
positive samples while maximizing the distance between the anchor and negative
samples. The triplet loss function plays a crucial role in learning
discriminative embeddings by encouraging the model to bring genuine signatures
(positive samples) closer to the anchor signature and push forged signatures
(negative samples) farther away from the anchor. This enables effective
differentiation between genuine and forged signatures based on their feature
representations.

We utilize the trained Siamese model to determine a signature’s similarity
score to the reference signature for validation. If the similarity score
exceeds a predefined threshold, the questioned signature is considered
authentic; otherwise, it is labeled as a forgery.

The formula for the Triplet loss is shown in equation (6):

$L=\max(d(a,p)-d(a,n)+\text{$margin$},0)\hskip 30.0pt\text{(6)}$

Where $L$ represents the Triplet loss, $d(a,p)$ represents the distance
between the anchor sample $a$ and the positive sample $p$, $d(a,n)$ represents
the distance between the anchor sample $a$ and the negative sample $n$. The
$margin$ is a hyperparameter that specifies a margin or threshold value,
$max(x,0)$ ensures that the loss is non-negative, as the loss is only
calculated when the positive distance plus the margin is greater than the
negative distance. The model was trained with a learning rate of 0.0005, a
batch size of 32, and it underwent 100 epochs of training.

###  III-B Dataset

In our study, we addressed the task of signature verification and forgery
detection using two different datasets: the ICDAR 2011 SigComp Dutch Signature
Dataset and the CEDAR Dataset.

There are two folders in the ICDAR 2011 SigComp Dutch Signature Dataset:
”train” and ”test.” The ”train” folder contains samples from 64 people, each
of whom has at least 24 genuine and 8 forged signatures. This dataset serves
as the primary training set for our model. The ”test” folder, on the other
hand, contains samples from 42 different individuals, each with a minimum of
12 authentic and 12 forged signatures. We use this dataset to evaluate the
effectiveness and generalizability of our trained model.

The CEDAR dataset comprises 1320 forgeries, with 24 forgeries for each of the
55 writers. Additionally, it contains 1320 genuine signatures, with 24
original signatures from the same 55 writers. Therefore, for each person, we
have access to 24 original signatures and 24 forgeries. Each image is resized
to 300x180 pixels before being fed into the model.

Out of the 55 individuals, 40 are utilized to train the model, while the
remaining 15 people are reserved for testing. This approach ensures that the
trained model is tested on entirely new signatures that it has not encountered
during the training phase.

##  IV Experiments and Results

The evaluation of our SigScatNet model on the ICDAR SigComp Dutch dataset and
the CEDAR dataset resulted in impressive performance metrics. The Receiver
Operating Characteristic (ROC) curve, which plots True Positive Rate (TPR)
against False Positive Rate (FPR), exhibited an exceptional Area Under the
Curve (AUC) of 99.6% on the ICDAR SigComp Dutch dataset and 99.9% on the CEDAR
dataset. This indicates the model’s ability to achieve high true positive
rates while maintaining low false positive rates, a crucial aspect for
accurate signature forgery detection. Additionally, the Precision-Recall (PR)
curve demonstrated an outstanding Average Precision (AUPR) of 99.4% on the
ICDAR SigComp Dutch dataset and 99.9% on the CEDAR dataset, further
reinforcing the model’s effectiveness in correctly identifying genuine and
forged signature pairs. These results validate the discriminative power of our
SigScatNet model and its robustness in capturing signature characteristics.
Fig. 3 and Fig. 4 showcases the ROC and PR curve on the two datasets.

Figure 3: (i)ROC and (ii)PR curve on the Testing set of ICDAR SigComp Dutch
dataset Figure 4: (i)ROC and (ii)PR curve on the Testing set of CEDAR dataset

We examined two histograms showing the Density vs. Euclidean distances and
Count vs. Euclidean distances between embeddings to acquire additional
insights into the model’s performance. These histograms visually represent the
distribution of distances between genuine pairings and imposter pairs,
allowing us to assess the model’s capability to distinguish between authentic
and forged signatures based on the estimated embeddings. An important factor
in successfully categorizing signatures is the optimal threshold.

Figure 5: Distance histograms for the Testing set on the ICDAR SigComp Dutch
dataset Figure 6: Distance histograms for the Testing set on the CEDAR dataset

Fig. 5 and Fig. 6 showcase the distance histograms for the Testing sets on the
ICDAR SigComp Dutch dataset and the CEDAR dataset, respectively.

The False Match Rate (FMR) and False Non-Match Rate (FNMR) curve, which
depicts the relationship between the error rates and the threshold value,
provided another tool we used to assess the model’s effectiveness. The Equal
Error Rate (EER), a crucial measure in the detection of signature fraud, is
represented at the junction of these two curves. In our instance, the EER at
the threshold coordinate came out to be 3.22% for the SigComp Dutch dataset
and 0.058% for the CEDAR dataset. This balance between false match and false
non-match errors denotes the point at which the model performs at its best.

Fig. 7 and Fig. 8 showcase the FMR and FNMR plot with the EER representation
for the ICDAR SigComp Dutch dataset and the CEDAR dataset, respectively.

These assessment metrics—the ROC and PR curves, distance histograms, and the
False Error Rate versus Threshold curve—all demonstrate the effectiveness of
our Siamese model in detecting signature forgeries. The model’s capacity to
distinguish between authentic and forged signatures is supported by its strong
AUC and AUPR values, and the distance histograms draw attention to the
distinct differences between the two groups.

A quantitative evaluation of the model’s performance at the best operating
point is also provided by the EER computation. These results demonstrate the
accuracy and dependability of the SigScatNet model in addressing the
challenges related to signature forgery detection.

Figure 7: False Match Rate and False Non-Match Rate Plot with EER
representation for the ICDAR SigComp Dutch dataset Figure 8: False Match Rate
and False Non-Match Rate Plot with EER representation for the CEDAR dataset

##  V Comparison with Other Models

To evaluate the efficacy of our SigScatNet model, it is imperative to compare
its performance against other state-of-the-art verification models in the
research community, providing valuable insights into the strengths and
weaknesses of our model and highlighting the advancements achieved by our
proposed method. The results of these comparisons are presented in Table I and
Table II.

TABLE I: Comparison of proposed SigScatNet with other methods on the SigComp Dutch Dataset used in the research community Method Used | Feature Extraction | Classifier | Accuracy | EER
---|---|---|---|---
Cozzens et al. (2018) [10] | CNN | CNN | 84.74 | -
Pham et al. (2014) [11] | Global Features | Distance Based | 87.80 | -
Khan et al. (2015) [12] | Local and Global Features | DTW | - | 5.27%
Alvarez et al. (2016) [13] | CNN | VGG-16 | 97.00 | -
Chen et al. (2018) [14] | Correlation Coefficients | Score based Likelihood | 96.17 | -
Parodi and Gomez (2014) [15] | - | Random Forest | - | 5.5%
Proposed SigScatNet | Scattering + Siamese CNN | Distance Based | 97.76 | 3.22%
TABLE II: Comparison of proposed SigScatNet with other methods on the CEDAR Dataset used in the research community Method Used | Feature Extraction | Classifier | Accuracy | EER
---|---|---|---|---
Zois et al. [16] | K-SVD | SVM | 84.74 | 2.78%
Hadjadji et al. [17] | Curvelet Transform | Bi-SVM, OC-PCA | 97.99 | -
Çalik et al. [18] | CNN | 1-NN, CNN | 98.3 | -
Okawa [19] | VLAD with KAZE | SVM | - | 1.0%
Rateria et al. [20] | CNN | CNN - SVM | 99.87 | -
Zois et al. [21] | Poset Grid Features | SVM | - | 3.02%
Eskander et al. [22] | Feature learning | SVM | - | 4.63%
Kumar et al. [23] | Surrounding pixels | RBF-SVM | 91.67 | -
Hafemann et al. [24] | CNN (Signet-SPP) | SVM | - | 2.33%
Proposed SigScatNet | Scattering + Siamese CNN | Distance Based | 99.91 | 0.058%

Another important comparison to be done was to compare the SigScatNet model in
two ways: one with Scattering parameters and one without Scattering
parameters. Table III showcases the advantages of using the Scattering
parameters.

TABLE III: Comparison of proposed SigScatNet with and without Scattering parameters on SigComp Dutch and on CEDAR Dataset Methodology Used | Number of Parameters | EER (For SigComp) | EER (For CEDAR)
---|---|---|---
Only Siamese Network | 3,327,056 | 3.22% | 0.069%
SigScatNet (Siamese CNN + Scattering) | 257,232 | 3.87% | 0.058%

The number of parameters in the plain Siamese network rose to around 3.3
million as illustrated in Table III to give comparable results. On the other
hand, while incorporating the Scattering parameters in the network, we were
able to reduce the parameters down to 257,232, which is a 92.27% decrease in
the number of parameters, still yielding a comparable or even better EER. This
further showcases that incorporating wavelets into the network drastically
reduces the complexity and dramatically increases computational efficiency.

##  VI Conclusion and Future Work

This research paper presents SigScatNet: a comprehensive and effective
solution for signature forgery detection and similarity assessment by
combining a Siamese deep learning network with Scattering wavelets. This
system proves to be a reliable and effective model that can accurately
differentiate between authentic and fake signatures while also offering a
measure of similarity by utilizing the strength of deep learning and the
distinctive characteristics of scattering wavelets. Furthermore, the addition
of Scattering wavelets to our model helped us achieve excellent efficiency
without sacrificing accuracy. In order to help the model make wise decisions,
the wavelet-based feature extraction procedure extracted crucial structural
and textural data from the signatures. The experimental evaluation of the
SigComp Dutch and CEDAR datasets proved the viability of our suggested
methodology. The SigScatNet model demonstrated excellent performance with an
Equal Error Rate of 3.689% for the SigComp Dutch Dataset and 0.05787% for the
CEDAR dataset after being trained on a wide range of real and fake signatures.
This demonstrates the model’s capacity to balance false match and false non-
match errors, resulting in reliable and accurate detection of signature fraud.

The proposed approach presents a practical solution to address the challenges
associated with signature forgery. Our model illustrates its potential for
real-world deployment, enabling individuals and organizations to verify
signatures quickly and accurately. It does this by utilizing cutting-edge
deep-learning techniques and harnessing the special properties of scattering
wavelets. By offering a strong and effective framework that combines the
strength of Siamese deep learning networks and Scattering wavelets, this
research makes a contribution to the field of signature fraud detection and
paves the way for increased security and confidence in applications for
signature verification. Since this model is extremely lightweight, it can in
theory be implemented into an app that can make the signature verification
task in the palm of everyone’s hands.

All other authors have no conflicts of interest. SigComp Dutch dataset
analyzed during the current study is available in the ICDAR 2011 Signature
Verification Competition (SigComp2011) repository. CEDAR dataset analyzed
during the current study is available in the CEDAR from University of Buffalo
repository.

## References

  * [1] Gowri P, Sivapriya G, Karthick Raja K, Kamaleshwar N K J, and Kesavaraj N.  Real time signature forgery detection using machine learning.  In 2022 Second International Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies (ICAECT), pages 1–5, 2022.
  * [2] Subramaniam M, Teja E, and Mathew A.  Signature forgery detection using machine learning.  In International Research Journal of Modernization in Engineering Technology and Science, volume 04, 2022.
  * [3] S Jerome Gideon, Anurag Kandulna, Aron Abhishek Kujur, A Diana, and Kumudha Raimond.  Handwritten signature forgery detection using convolutional neural networks.  volume 143, pages 978–987, 2018.  8th International Conference on Advances in Computing & Communications (ICACC-2018).
  * [4] Harish Srinivasan, Sargur N. Srihari, and Matthew J. Beal.  Machine learning for signature verification.  In Prem K. Kalra and Shmuel Peleg, editors, Computer Vision, Graphics and Image Processing, pages 761–775, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg.
  * [5] Dr. Chayadevi M L, Anagha R, and Chandan Kumar G.  Signature Recognition and Forgery Detection.  11 2022.
  * [6] A. Agarwal and A. BS.  International journal of creative research thoughts (ijcrt), 9(8).  volume 9, pages 659–662, 2021.
  * [7] M. Varol Arısoy.  Signature verification using siamese neural network one-shot learning.  volume 3, pages 248–260, 2021.
  * [8] Amruta Jagtap, Dattatray Sawat, Rajendra Hegadi, and Ravindra Hegadi.  Verification of genuine and forged offline signatures using siamese neural network (snn).  volume 79, 12 2020.
  * [9] Soumitri Chattopadhyay, Siladittya Manna, Saumik Bhattacharya, and Umapada Pal.  Surds: Self-supervised attention-guided reconstruction and dual triplet loss for writer independent offline signature verification.  01 2022.
  * [10] Brittany Cozzens, Richard Huang, Maxwell Jay, Kyle Khembunjong, Sahan Paliskara, Felix Zhan, Mark Zhang, Shahab Tayeb, Aeop Unite, and Unlv Stem.  Signature verification using a convolutional neural network.  2017\.
  * [11] T.-A. Pham, H.-H. Le, and N.-T. Do.  Offline handwritten signature verification using local and global features.  volume 75, page 231–247, 2014.
  * [12] S. Khan, M. Ali Akbar, F. Shahzad, M. Farooq, and Z. Khan.  Secure biometric template generation for multi-factor authentication.  volume 48, pages 458–472. Elsevier, 2015.
  * [13] G. Alvarez, B. Sheffer, and M. Bryant.  Offline signature verification with convolutional neural networks: Technical report, stanford university.  2016\.
  * [14] X. h. Chen, C. Champod, X. Yang, S. pei Shi, Y. wen Luo, N. Wang, Y. chen Wang, and Q. meng Lu.  Assessment of signature handwriting evidence via score-based likelihood ratio based on comparative measurement of relevant dynamic features.  volume 282, page 101–110, 2018.
  * [15] Marianela Parodi and Juan Gómez.  Legendre polynomials based feature extraction for online signature verification. consistency analysis of feature combinations.  volume 47, pages 128–140, 01 2014.
  * [16] Elias Zois, Ilias Theodorakopoulos, Dimitrios Tsourounis, and George Economou.  Parsimonious coding and verification of offline handwritten signatures.  pages 636–645, 07 2017.
  * [17] Bilal Hadjadji, Youcef Chibani, and Hassiba Nemmour.  An efficient open system for offline handwritten signature identification based on curvelet transform and one-class principal component analysis.  volume 265, pages 66–77, 2017.  New Trends for Pattern Recognition: Theory & Applications.
  * [18] Nurullah Calik, Onur Kurban, Ali Yilmaz, Tülay Yildirim, and Lutfiye Durak Ata.  Large-scale offline signature recognition via deep neural networks and feature embedding.  volume 359, 05 2019.
  * [19] Manabu Okawa.  From bovw to vlad with kaze features: Offline signature verification considering cognitive processes of forensic experts.  volume 113, pages 75–82, 2018.  Integrating Biometrics and Forensics.
  * [20] Avani Rateria and Suneeta Agarwal.  Off-line signature verification through machine learning.  In 2018 5th IEEE Uttar Pradesh Section International Conference on Electrical, Electronics and Computer Engineering (UPCON), pages 1–7, 2018.
  * [21] Elias N. Zois, Linda Alewijnse, and George Economou.  Offline signature verification and quality characterization using poset-oriented grid features.  volume 54, pages 162–177, 2016.
  * [22] George S. Eskander, Robert Sabourin, and Eric Granger.  Hybrid writer-independent–writer-dependent offline signature verification system.  volume 2, pages 169–181, 2013.
  * [23] Rajesh Kumar, J.D. Sharma, and Bhabatosh Chanda.  Writer-independent off-line signature verification using surroundedness feature.  volume 33, pages 301–308, 2012.
  * [24] Luiz Gustavo Hafemann and Luiz Soares de Oliveira.  Fixed-sized representation learning from offline handwritten signatures of different sizes.  volume 21, pages 219–232, 09 2018.