# Reduced Jeffries-Matusita distance: A Novel Loss Function to Improve
Generalization Performance of Deep Classification Models

Mohammad Lashkari 11footnotemark: 1222Corresponding author,
<EMAIL_ADDRESS>Amin Gheibi 11footnotemark<EMAIL_ADDRESS>Department of Computer Science, Amirkabir University of Technology (Tehran
polytechnic), Iran

###### Abstract

The generalization performance of deep neural networks in classification tasks
is a major concern in machine learning research. Despite widespread techniques
used to diminish the over-fitting issue such as data augmentation, pseudo-
labeling, regularization, and ensemble learning, this performance still needs
to be enhanced with other approaches. In recent years, it has been
theoretically demonstrated that the loss function characteristics i.e. its
Lipschitzness and maximum value affect the generalization performance of deep
neural networks which can be utilized as a guidance to propose novel distance
measures. In this paper, by analyzing the aforementioned characteristics, we
introduce a distance called Reduced Jeffries-Matusita as a loss function for
training deep classification models to reduce the over-fitting issue. In our
experiments, we evaluate the new loss function in two different problems:
image classification in computer vision and node classification in the context
of graph learning. The results show that the new distance measure stabilizes
the training process significantly, enhances the generalization ability, and
improves the performance of the models in the Accuracy and F1-score metrics,
even if the training set size is small.

###### keywords:

classification , generalization , deep neural networks , loss function ,
Jeffries-Matusita distance

###### MSC:

[2010] 68Q32 , 68T05 , 68T45 , 68R10

††journal: Neural Networks

##  1 Introduction

Classification tasks are ubiquitous in many applications such as face
recognition, topic modeling, recommender systems, cancer prognosis, etc. One
of the main challenges in these tasks is to improve the generalization
performance and prevent the over-fitting issue of the output model, designed
based on deep neural networks (DNNs). Several methods have been proposed to
address this issue [1, 2, 3]. However, the over-fitting of classification
models has not been alleviated completely yet. Therefore, new fundamental
propositions, derived from theoretical analyses are needed to improve the
generalization ability of deep classification models that can be exploited in
any subsequent research on deep learning-based classification including those
that address the over-fitting issue. In this paper, we focus on loss
functions.

Loss functions have a significant role in the training phase of a deep
learning model. Theoretically, it has been deduced that the Lipschitz constant
and magnitude value of loss functions are related to the generalization
ability of DNNs [4]. Furthermore, a novel loss function called the Generalized
Jeffries-Matusita (GJM) distance was proposed for label distribution learning
(LDL) as an alternative to the Kullback-Leibler (KL) divergence to overcome
the over-fitting issue of the output models [4]. The experiments of [4] show
that GJM can stabilize the training process of LDL models and increase
accuracy. However, their theoretical results are only valid for deep learning
models trained by stochastic gradient descent (SGD). Also, according to the
definition of GJM, it cannot be used in single-label classification problems.
Recently, in the work of [5], we have theoretically demonstrated that the same
characteristics of loss functions are also attributed to the generalization
performance of the output model obtained by the Adam [6] and AdamW [7]
optimizers which are widespread in the recent years for training DNNs,
especially in classification tasks. A well-known loss function to train
classification models is Cross-Entropy (CE) which is logarithmic and
unbounded. In the present work, using the theoretical results of [4, 5], we
propose a bounded loss function named the Reduced Jeffries-Matusita (RJM)
distance which provides better generalization performance for deep
classification models, compared to CE.

In the experiments, we focus on image classification in computer vision and
node classification in the graph learning era because there are appropriate
benchmarks in these problems challenging the models in the over-fitting issue.
We train and test common deep learning architectures in the aforementioned
tasks using CE and RJM to analyze the difference between the generalization
performance of the output models. For assessing the time cost of RJM, we
measure how long the models take to train. The results show how RJM can
perform better than CE to improve the generalization performance and increase
the accuracy of classification models.

##  2 Related Work

One of the major criteria to measure the generalization performance is the
generalization error, which is defined as the difference between the true
error i.e. the expected value of the loss function over the whole input space
and the training error i.e. the expected loss value over the training data. A
common technique to diminish the generalization error of DNNs is to upper-
bound it theoretically. We call these upper bounds generalization bounds.

There are many approaches for deriving generalization bounds such as
robustness [8, 9], PAC-Bayesian theory [10, 11], Vapnik-Chervonenkis (VC)
dimension [12, 13, 14], and uniform stability [15, 16, 17, 4, 5]. In the
robustness approach, a learning algorithm is robust if the slight changes in
the training set cannot cause noticeable changes in the training error. It has
been proved that the more robust an algorithm is, the less the generalization
error of the output model [18]. The PAC-Bayesian theory is an approach for
analyzing Bayesian learning algorithms where the hypothesis space has a prior
distribution and the output model is a distribution over this space. VC-
dimension is a measure to evaluate complexity, flexibility, and the
generalization error of classification models whose architecture is a feed-
forward or recurrent neural network. Subsequently, it was extended to graph
and recursive neural networks [12]. In the previous works conducted under the
approaches explained above, generalization bounds in terms of some hyper-
parameters, the $L_{2}$ and the Frobenius norms of the DNNs’ weights, and the
size of the training set have been derived. However, the particulars of
optimization learning algorithms were not considered and the question of which
characteristics of loss functions can affect the generalization error of DNNs
remains unanswered. Hence, we follow the notion of uniform stability, which is
defined for any learning algorithm based on the true error, where the
researchers could take steps forward to achieve generalization bounds related
to the loss function properties.

Intuitively, if a learning algorithm is uniformly stable, then the true error
of the output model is not sensitive to the noise of training samples. In the
work of [15], Bousquet and Elisseeff introduced this notion for deterministic
learning algorithms. Hardt et al. extended it to randomized learning
algorithms [17]. They found an expected generalization bound directly related
to the number of iterations of a learning algorithm. Ali Akbari et al. derived
a high probability generalization bound for output models obtained by SGD,
which is a vanishing function directly related to the Lipschitz constant and
the maximum value of a loss function [4]. With the same idea, an analogous
high probability generalization bound i.e. directly related to the above
characteristics has been derived for DNNs trained by Adam or AdamW [5]. The
theoretical results of [4, 5] can be used as an instruction to choose or
create a loss function when the optimizer is SGD or Adam or AdamW. In this
paper, by analyzing Lipschitzness and the magnitude value of loss functions,
we create RJM and compare it to CE.

##  3 Preliminaries

Let $X\subseteq\mathbb{R}^{m\times n}$ be the input space of a classification
problem and $C\in\mathbb{N}$ be the number of labels. Let $\mathbb{P}_{C}$
containing probability vectors be the output space and $\mathbbm{1}_{C}$
containing one-hot encoded vectors be the target space. A deep classification
model is $f^{\theta}:X\rightarrow\mathbb{P}_{C}$ parameterized by $\theta\in
H$ where $H\subset\mathbb{R}^{K}$ is a bounded set representing the hypothesis
space of the model. The format of classification loss functions is
$\ell:\mathbb{P}_{C}\times\mathbbm{1}_{C}\rightarrow\mathbb{R}^{+}$ which
compares the predicted probability distribution to the target, having a
substantial role in the DNNs training phase. A learning problem is to minimize
the true error, defined as
$E_{true}(f^{\theta})\coloneqq\mathbb{E}_{(\mathrm{x},\mathrm{y})\sim\mathbb{Q}}\left[\ell(f^{\theta}(\mathrm{x}),\mathrm{y})\right]$:

$\min_{\theta\in H}E_{true}(f^{\theta}).$ (1)

Since $\mathbb{Q}$ is unknown, the minimization problem (1) is intractable.
Therefore, we estimate $E_{true}(f^{\theta})$ by the training error
$E_{train}(f^{\theta})\coloneqq\frac{1}{N}\sum_{i=1}^{N}\ell(f^{\theta}(\mathrm{x}_{i}),\mathrm{y}_{i})$
where $(\mathrm{x}_{i},\mathrm{y}_{i})$ is a training sample belonging to
$S\in(X\times\mathbbm{1}_{C})^{N}$. To define the generalization error, we
also need some definitions and notes.

######  Definition 1 (Partition).

Given a training set $S$ of size $N$, $P_{S}=\\{P_{1},P_{2},\ldots,P_{k}\\}$
is a partition for $S$ of size $k$ if each member of $S$ is in exactly one
element of $P_{S}$ and $\forall i\ |P_{i}|=\frac{N}{k}$.

In Definition 1, we assumed $N$ is divisible by $k$. If it is not possible, we
repeat a sample enough to make that happen. Each element of $P_{S}$ represents
a mini-batch which is selected in each iteration of the training process to
update the parameters.

To train a DNN, we need an iterative optimization algorithm e.g. first-order
gradient-based methods. In each iteration of the algorithm, we have to select
a mini-batch of the training examples. Suppose that
$P_{S}=\\{P_{1},P_{2},\ldots,P_{k}\\}$ is a partition of the training set $S$.
We use a random sequence $R=\left(r_{1},r_{2},\ldots,r_{T}\right)$ from
$\\{1,2,\ldots,k\\}$ to specify the selected mini-batch in the arbitrary
iteration $1\leq t\leq T$ for updating the model parameters. In the following,
$f^{\theta}_{P_{S},R}$ denotes the output model obtained by an optimization
algorithm using a partition $P_{S}$ and a random sequence $R$.

######  Definition 2 (Generalization Error).

Suppose that $S$ is a training set and $P_{S}$ is a partition for it of size
$k$. Consider a random sequence $R$ from $\\{1,2,\ldots,k\\}$. The
generalization error of $f^{\theta}_{P_{S},R}$ obtained by an optimization
algorithm is defined as

$GE(f^{\theta}_{P_{S},R})=E_{true}(f^{\theta}_{P_{S},R})-E_{train}(f^{\theta}_{P_{S},R}).$

######  Notation.

For simplicity, moving forward, we denote a loss function by
$\bm{\ell(\mathrm{\hat{y}},\mathrm{y})}$ in which the first argument is
predicted probability vector and the second argument is the target vector.

######  Definition 3 (Lipschitzness).

Let $C\in\mathbb{N}$ be the number of labels of a classification problem.
$\ell(\mathrm{\hat{y}},\mathrm{y})$ is $\gamma$-Lipschitz with regard to its
first argument if $\gamma\geq 0$ exists such that
$\,\forall\mathrm{\hat{y}_{1}},\mathrm{\hat{y}_{2}}\in\mathbb{P}_{C}$, we have

$\mathrm{|\ell(\hat{y}_{1},y)-\ell(\hat{y}_{2},y)|}\leq\gamma\left\|\mathrm{\hat{y}}_{1}-\mathrm{\hat{y}}_{2}\right\|,$
(2)

where $\left\|.\right\|$ is the $L_{2}$-norm.

######  Definition 4 (Smoothness).

Given $C\in\mathbb{N}$, $\ell(\mathrm{\hat{y}},\mathrm{y})$ is $\zeta$-smooth
with regard to its first argument if for every
$\mathrm{\hat{y}_{1}},\mathrm{\hat{y}_{2}}\in\mathbb{P}_{C}$, the gradient
$\nabla_{\hat{\mathrm{y}}}\ell(\mathrm{\hat{y}},\mathrm{y})$ holds the
inequality (2) for $\zeta\geq 0$:

$\mathrm{\left\|\nabla_{\hat{y}_{1}},\ell(\hat{y}_{1},y)-\nabla_{\hat{y}_{2}}\ell(\hat{y}_{2},y)\right\|}\leq\zeta\left\|\mathrm{\hat{y}}_{1}-\mathrm{\hat{y}}_{2}\right\|.$

As mentioned in Section 2, we use the notion of uniform stability to link the
generalization error with the loss function properties including its
Lipschitzness. To define the uniform stability we follow [17, 4]:

######  Definition 5 (Uniform Stability).

Let $S$ and $S^{\prime}$ be two training sets. Consider two partitions $P_{S}$
and $P_{S^{\prime}}$ of size $k$ which differ in one element. Consider a
random sequence $R$ from $\\{1,2,\ldots,k\\}$. Let $f^{\theta}_{P_{S},R}$ and
$f^{\theta}_{P_{S^{\prime}},R}$ be the output models obtained by an arbitrary
optimization algorithm, $A_{opt}$ with the same initialization. Then,
$A_{opt}$ is $\beta$-uniform stable with regard to
$\ell(\mathrm{\hat{y}},\mathrm{y})$ if:

$\forall
S,S^{\prime}\;\;\sup_{\mathrm{(x,y)}}\mathbb{E}_{R}\left[|\ell(f_{P_{S^{\prime}},R}(\mathrm{x}),\mathrm{y})-\ell(f_{P_{S},R}(\mathrm{x}),\mathrm{y})|\right]\leq\beta.$

Definition 5 states that if an algorithm is uniformly stable with the constant
$\beta$, then changing one mini-batch alters the expected value of the loss
function over the random sequences at most $\beta$ for any sample of
$(\mathrm{x},\mathrm{y})$. In addition to uniform stability, another factor
that affects the generalization error directly is the bounded difference
condition [4]:

######  Definition 6 (BDC).

Let $k,T\in\mathbb{N}$. $G:\\{1,2,\ldots,k\\}^{T}\rightarrow\mathbb{R}^{+}$
satisfies the bounded difference condition (BDC) with the constant $\rho$ if
$\,\forall R,R^{\prime}\in Dom(G)$ which differ in two elements, we have

$\sup_{R,R^{\prime}}|G(R^{\prime})-G(R)|\leq\rho.$

##  4 Generalization Bounds

###  4.1 SGD Optimizer

SGD is the simplest gradient-based optimization algorithm that is used to
train DNNs i.e. minimizing the training error. Roughly speaking, in $t$-th
iteration of SGD, the current parameters $\theta_{t-1}$ are updated by moving
against the direction of the gradient vector with a specified step size:

$\theta_{t}\leftarrow\theta_{t-1}-\eta\nabla_{\theta_{t-1}}\ell(f^{\theta_{t-1}}(\mathrm{x_{i}}),\mathrm{y}_{i}),$
(3)

where $\eta$ is the step size which we further denote by the learning rate. If
the learning rate is too small, the model remains under-fitted. However, with
a large learning rate, the exploding gradient issue occurs. Besides, the
performance of the output model is sensitive to the number of iterations.
Training the model with too many iterations causes over-fitting even if the
learning rate is precisely tuned.

In the following, we discuss the uniform stability and the generalization
error of DNNs trained by SGD. In Theorem 1, the relationship of the Lipschitz
constant of a loss function with the uniform stability and BDC is shown [4].
Using Theorem 1, a generalization bound for DNNs trained by SGD has been
derived [4], clarified in Theorem 2. The proofs are available in the
supplementary of [4].

######  Notation.

As indicated in the statement (3), SGD uses a specific partition of the
training set in which each element has only one sample. In other words, the
partition is a set containing the singleton of each training sample.
Therefore, we use $f^{\theta}_{\\{S\\},R}$ to denote the output model obtained
by SGD in which
$\\{S\\}=\\{\\{(\mathrm{x},\mathrm{y})\\}\\}_{(\mathrm{x},\mathrm{y})\in S}$.

######  Theorem 1.

[4] Assume SGD runs for $T$ iterations with an annealing learning rate
$\eta_{t}$ to minimize the training error computed on $N$ samples. Let
$\ell(\mathrm{\hat{y}},\mathrm{y})$ be $\gamma$-Lipschitz, $\zeta$-smooth, and
convex. Then SGD is $\beta$-uniformly stable and for every
$(\mathrm{x},\mathrm{y})$,
$\ell(f^{\theta}_{\\{S\\},R}(\mathrm{x}),\mathrm{y})$ satisfies $\rho$-BDC
with respect to $R$. Accordingly,

$\beta\leq\frac{2\gamma^{2}}{N}\sum_{t=1}^{T}\eta_{t},\ \ \
\rho\leq\frac{4\gamma^{2}}{T}\sum_{t=1}^{T}\eta_{t}.$

######  Theorem 2.

[4] Consider a loss function $\ell(\mathrm{\hat{y}},\mathrm{y})$ with a
maximum value of $L$ which is $\gamma$-Lipschitz, $\zeta$-smooth, and convex.
Suppose that SGD is executed for $T$ iterations with an annealing learning
rate $\eta_{t}$ to obtain $f^{\theta}_{\\{S\\},R}$ by minimizing the training
error achieved from $N$ samples. Then we have the following inequality with
probability at least $1-\delta$:

$GE(f^{\theta}_{\\{S\\},R})\leq
2\gamma^{2}\sum_{t=1}^{T}\eta_{t}\left(2\sqrt{\frac{\log(2/\delta)}{T}}+\sqrt{\frac{2\log(2/\delta)}{N}}+\frac{1}{N}\right)+L\sqrt{\frac{\log(2/\delta)}{2N}}.$
(4)

######  Note.

The inequality (4) demonstrates that choosing a loss function with a lower
Lipschitz constant and maximum value reduces the generalization error of an
output model obtained by SGD, which leads to overcoming the over-fitting
issue.

###  4.2 Adam Optimizer

SGD has two major disadvantages. First, in each iteration, it takes just one
sample of the training set to update the parameters. Second, it is sensitive
to the norm of the gradient vector which makes the process of learning rate
tuning more involved. This can result in the problem vanishing or exploding
gradient. To address these issues, adaptive gradient algorithms were
introduced [19, 20, 21, 6, 7]. One of the most useful of these optimizers is
the adaptive moment estimation (Adam). Intuitively, Adam utilizes the first-
moment estimate of the gradient for determining the appropriate direction and
also uses the second-moment estimate to neutralize the effect of the gradient
norm.

Let $\ell(f^{\theta};P)$ be the loss function value, computed on an arbitrary
mini-batch $P=\\{(\mathrm{x}_{i},\mathrm{y}_{i})\\}_{i=1}^{b}$:

$\ell(f^{\theta};P)=\frac{1}{b}\sum_{i=1}^{b}\ell(f^{\theta}(\mathrm{x}_{i}),\mathrm{y}_{i}),$

where $b$ is the size of $P$ denoting the batch size. In the following, we get
to the particulars of Adam. All operators in (5)-(9) are element-wise. Let
$m_{t}$ and $v_{t}$ be the first and second moment estimates respectively
which are defined as:

$\displaystyle m_{t}$ $\displaystyle\leftarrow\beta_{1}\cdot
m_{t-1}+(1-\beta_{1})\cdot g(\theta_{t-1}),$ (5) $\displaystyle v_{t}$
$\displaystyle\leftarrow\beta_{2}\cdot v_{t-1}+(1-\beta_{2})\cdot
g^{2}(\theta_{t-1}),$ (6)

where $\beta_{1},\beta_{2}\in(0,1),m_{0}=0,v_{0}=0$, and
$g(\theta)=\nabla_{\theta}\ell(f^{\theta};P)$. The bias-corrected versions of
$m_{t}$ and $v_{t}$ are:

$\displaystyle\widehat{m}_{t}$
$\displaystyle\leftarrow\frac{m_{t}}{1-\beta_{1}^{t}},$ (7)
$\displaystyle\widehat{v}_{t}$
$\displaystyle\leftarrow\frac{v_{t}}{1-\beta_{2}^{t}}.$ (8)

Adam updates the parameters as

$\theta_{t}\leftarrow\theta_{t-1}-\eta\cdot\frac{\widehat{m}_{t}}{(\sqrt{\widehat{v}_{t}}+\epsilon)},$
(9)

where $\eta$ is the learning rate and $\epsilon=10^{-8}$. Now, we analyze the
generalization error. In Theorem 3, we discuss the uniform stability of Adam.
In Theorem 4, a generalization bound for a DNN trained by Adam is argued,
which gives us the same information about the effect of loss functions on the
generalization error of SGD that we clarified in Subsection 4.1:

######  Theorem 3.

[5] Assume Adam runs for $T$ iterations with a learning rate $\eta$ and batch
size $b$ to minimize the training error computed on $N$ samples. Consider a
convex and $\gamma$-Lipschitz loss function
$\ell(\mathrm{\hat{y}},\mathrm{y})$. Then Adam is $\beta$-uniformly stable and
for every $(\mathrm{x},\mathrm{y})$,
$\ell(f^{\theta}_{P_{S},R}(\mathrm{x}),\mathrm{y})$ satisfies $\rho$-BDC with
respect to $R$. Additionally, we have

$\beta\leq\frac{2\eta}{c}\cdot\frac{bT\gamma^{2}}{N},\ \ \
\rho\leq\frac{8\eta}{c}\cdot\left(\frac{b\gamma}{N}\right)^{2},$

where $c\in(0,1)$ is constant.

######  Theorem 4.

[5] Consider a loss function $\ell(\mathrm{\hat{y}},\mathrm{y})$ with a
maximum value of $L$ which is $\gamma$-Lipschitz and convex. Suppose that Adam
is executed for $T$ iterations with a learning rate $\eta$ and batch size $b$
to obtain $f^{\theta}_{P_{S},R}$ by minimizing the training error achieved
from $N$ samples. Then with probability at least $1-\delta$, we have

$GE(f^{\theta}_{P_{S},R})\leq\frac{2\eta}{c}\left(4\left(\frac{b\gamma}{N}\right)^{2}\sqrt{T\
log(2/\delta)}+\frac{bT\gamma^{2}}{N}\left(1+\sqrt{2N\log(2/\delta)}\right)\right)+L\sqrt{\frac{\log(2/\delta)}{2N}},$
(10)

where $c\in(0,1)$ is constant.

###  4.3 AdamW Optimizer

A common technique to improve the generalization performance of deep learning
models is to exploit a regularization parameter in loss functions. This
parameter makes the hypothesis space smaller and reduces the over-fitting
issue. However, when we train the models by Adam, regularization does not
decay the weight of the model parameters [7]. The idea of AdamW aaaAdam with
decoupled weight decay (AdamW) is to remove the regularizer from the loss
function and use it in the update statement. Let $\ell^{reg}(f^{\theta};P)$
represents the regularized version of $\ell(f^{\theta};P)$:

$\ell^{reg}(f^{\theta};P)=\ell(f^{\theta};P)+\frac{\lambda}{2b}\left\|\theta\right\|^{2},$
(11)

where $\lambda\in\mathbb{R}^{+}$ is the regularizer (weight decay) and $b$ is
the batch size.

Using the weight decay as the equation (11) is not effective in Adam [7].
Therefore, AdamW decouples this parameter and uses $\ell(f^{\theta};P)$ to
compute the gradient same as Adam. It adds the weight decay to the update
statement directly. Consider $\widehat{m}_{t}$ and $\widehat{v}_{t}$ in the
statements (7) and (8). Given a learning rate $\eta$ and schedule multiplier
$\alpha_{t}$, the AdamW’s update statement is:

$\theta_{t}\leftarrow\theta_{t-1}-\alpha_{t}\left(\eta\cdot\frac{\widehat{m}_{t}}{(\sqrt{\widehat{v}_{t}}+\epsilon)}+\lambda\theta_{t-1}\right).$

Consider a hypothesis space, $H$. Note that $H$ is bounded. Let

$\left\|\theta\right\|_{\sup}\vcentcolon=\sup_{\theta\in
H}\left\|\theta\right\|.$

Now, we can state the theorems:

######  Theorem 5.

[5] Assume AdamW runs for $T$ iterations with a learning rate $\eta$, schedule
multiplier $\alpha_{t}$, weight decay $\lambda$, and batch size $b$ to
minimize the training error computed on $N$ samples. Consider a convex and
$\gamma$-Lipschitz loss function $\ell(\mathrm{\hat{y}},\mathrm{y})$. Then
AdamW is $\beta$-uniformly stable and for every $(\mathrm{x},\mathrm{y})$,
$\ell(f^{\theta}_{P_{S},R}(\mathrm{x}),\mathrm{y})$ satisfies $\rho$-BDC with
respect to $R$. Besides,

$\beta\leq\frac{2bT}{N}\left(\frac{\eta\gamma^{2}}{c}+\gamma\lambda\left\|\theta\right\|_{\sup}\right)\sum_{t=1}^{T}\alpha_{t},\
\ \
\rho\leq\frac{8b^{2}}{N^{2}}\left(\frac{\eta\gamma^{2}}{c}+\gamma\lambda\left\|\theta\right\|_{\sup}\right)\sum_{t=1}^{T}\alpha_{t},$

where $c\in(0,1)$ is constant.

######  Theorem 6.

[5] Consider a loss function $\ell(\mathrm{\hat{y}},\mathrm{y})$ with a
maximum value of $L$ which is $\gamma$-Lipschitz and convex. Suppose that
AdamW is executed for $T$ iterations with a learning rate $\eta$, schedule
multiplier $\alpha_{t}$, weight decay $\lambda$, and batch size $b$ to obtain
$f^{\theta}_{P_{S},R}$ by minimizing the training error achieved from $N$
samples. Then with probability at least $1-\delta$, we have

$GE(f^{\theta}_{P_{S},R})\leq\frac{2b}{N}\left(\frac{\eta\gamma^{2}}{c}+\gamma\lambda\left\|\theta\right\|_{\sup}\right)\left(\frac{4b}{N}\sqrt{T\log(2/\delta)}+T\sqrt{2N\log(2/\delta)}\right)\sum_{t=1}^{T}\alpha_{t}+L\sqrt{\frac{\log(2/\delta)}{2N}},$

where $c\in(0,1)$ is constant.

##  5 Loss Functions

In this section, we analyze the characteristics of classification loss
functions, based on the generalization bounds previously mentioned. CE is a
logarithmic function that is derived from the maximum log-likelihood
estimation to classify samples in a label set of size $C$:

$\ell_{CE}(\mathrm{\hat{y}},\mathrm{y})=-\sum_{c=1}^{C}y_{c}\log(\hat{y}_{c}).$

Our proposed loss function, RJM is a reduced version of Jeffries-Matusita
distance [22]:

$\ell_{RJM}(\mathrm{\hat{y}},\mathrm{y})=\sum_{c=1}^{C}y_{c}(1-\sqrt{\hat{y}_{c}}).$

Based on Theorems 2, 4, and 6, we should prove that the Lipschitz constant and
maximum value of RJM are less than CE. In Lemma 1, we prove general properties
for classification loss functions. Lemma 2 shows the relationship between the
absolute values of a real function and its Lipschitzness. Corollary 1
demonstrates the convexity of CE and RJM. Finally, using Lemmas 1 and 2, we
state Theorem 7, showing the relationship between the Lipschitz constant and
the maximum value of CE and RJM.

######  Lemma 1.

Consider a function $h:(0,1)\rightarrow\mathbb{R}^{+}$. The identity loss
function for classification problems is defined as:

$I(\mathrm{\hat{y}},\mathrm{y})=\sum_{c=1}^{C}y_{c}h(\hat{y}_{c}).$

If $h(.)$ is $\gamma$-Lipschitz i.e.

$\forall u,v\in Dom(h)\;\;\left|h(u)-h(v)\right|\leq\gamma\left|u-v\right|,$
(12)

Then $I(\mathrm{\hat{y}},\mathrm{y})$ is $\gamma\sqrt{C}$-Lipschitz.
Furthermore, $I(\mathrm{\hat{y}},\mathrm{y})$ is convex with regard to its
first argument if $h(.)$ is convex.

######  Proof 1.

Consider two probability vectors $\mathrm{u}=\left[u_{c}\right]_{c=1}^{C}$,
$\mathrm{v}=\left[v_{c}\right]_{c=1}^{C}$, and the target vector
$\mathrm{y}\in\mathbbm{1}_{C}$. According to Definition 3 we have

$\displaystyle\left|I(\mathrm{u},\mathrm{y})-I(\mathrm{v},\mathrm{y})\right|$
$\displaystyle=\left|\sum_{c=1}^{C}y_{c}h(u_{c})-\sum_{c=1}^{C}y_{c}h(v_{c})\right|$
$\displaystyle\leq\sum_{c=1}^{C}y_{c}\left|h(u_{c})-h(v_{c})\right|$
$\displaystyle\leq\gamma\sum_{c=1}^{C}y_{c}\left|u_{c}-v_{c}\right|$
$\displaystyle\leq\gamma\sqrt{C}\left\|\mathrm{u}-\mathrm{v}\right\|.$ (13)

In the inequality (13), Cauchy-Schwartz is applied.

Now we prove the convexity of $I(\mathrm{\hat{y}},\mathrm{y})$ subject to the
convexity of $h(.)$. For $t\in[0,1]$ we have

$\displaystyle I(t\mathrm{u}+(1-t)\mathrm{v},\mathrm{y})$
$\displaystyle=\sum_{c=1}^{C}y_{c}h(tu_{c}+(1-t)v_{c})$
$\displaystyle\leq\sum_{c=1}^{C}y_{c}(th(u_{c})+(1-t)h(v_{c}))$
$\displaystyle=t\sum_{c=1}^{C}y_{c}h(u_{c})+(1-t)\sum_{c=1}^{C}y_{c}h(v_{c})$
$\displaystyle=tI(\mathrm{u},\mathrm{y})+(1-t)I(\mathrm{v},\mathrm{y}).$

######  Corollary 1.

$\ell_{CE}(\mathrm{\hat{y}},\mathrm{y})$ and
$\ell_{RJM}(\mathrm{\hat{y}},\mathrm{y})$ are convex.

######  Proof 2.

According to the definition of $I(\mathrm{\hat{y}},\mathrm{y})$ in Lemma 1, we
have $h_{CE}(x)=-\log(x)$ and $h_{RJM}(x)=1-\sqrt{x}$. Note that $\log(x)$ and
$\sqrt{x}$ are convex. Thus, the proposition is concluded.

######  Note.

In Lemma 1, $h(.)$ should be chosen in such a way that $\lim_{x\rightarrow
1^{-}}h(x)=0$. Otherwise, the training process will not proceed properly and
by minimizing the loss function the performance of deep learning models may
decrease. Both $h_{CE}(x)$ and $h_{RJM}(x)$ have this property.

######  Lemma 2.

Let $h:(0,1)\rightarrow\mathbb{R}^{+}$ be a derivative function. If a constant
$\gamma$ exists such that

$\gamma=\sup_{x}\left|h^{\prime}(x)\right|,$

Then $h(.)$ is $\gamma$-Lipschitz.

######  Proof 3.

This lemma is proved from the Lipschitzness definition of $h(.)$ by the Mean
Value Theorem.

######  Theorem 7.

Let $\gamma_{CE}$, $\gamma_{RJM}$ be the Lipschitz constants of
$\ell_{CE}(\mathrm{\hat{y}},\mathrm{y})$ and
$\ell_{RJM}(\mathrm{\hat{y}},\mathrm{y})$ respectively. We have

$\gamma_{CE}\leq\gamma_{RJM}.$ (14)

In addition, RJM is upper-bounded by CE:

$\ell_{RJM}(\mathrm{\hat{y}},\mathrm{y})\leq\ell_{CE}(\mathrm{\hat{y}},\mathrm{y}).$
(15)

######  Proof 4.

We first prove the inequality (14). According to Lemmas 1 and 2, it is enough
to prove that

$\left|h^{\prime}_{RJM}(x)\right|\leq\left|h^{\prime}_{CE}(x)\right|.$ (16)

Simplifying the inequality (16) gives

$0<x\leq 2\sqrt{x}.$ (17)

The inequality (17) holds for $0<x\leq 4$. Note that $Dom(h)=(0,1)$.
Therefore, the inequality (17) is satisfied.

Let’s prove the second part. Using Bernoulli’s inequality $\exp(x)\geq 1+x$,
we have

$\displaystyle\ell_{RJM}(\mathrm{\hat{y}},\mathrm{y})$
$\displaystyle=\sum_{c=1}^{C}y_{c}(1-\sqrt{\hat{y}_{c}})$
$\displaystyle=\sum_{c=1}^{C}y_{c}(1-\exp(\frac{1}{2}\log(\hat{y}_{c})))$
$\displaystyle\leq\sum_{c=1}^{C}y_{c}(-\frac{1}{2}\log(\hat{y}_{c}))$ (18)
$\displaystyle\leq\ell_{CE}(\mathrm{\hat{y}},\mathrm{y}).$

By Bernoulli’s inequality, we argued the inequality (18).

######  Corollary 2.

Let $f^{\theta,CE}_{P_{S},R}$ and $f^{\theta,RJM}_{P_{S},R}$ be the output
models obtained by SGD or Adam or AdamW using CE and RJM respectively under
the same settings for hyper-parameters. Then, the upper bound of
$GE(f^{\theta,RJM}_{P_{S},R})$ is less than the upper bound of
$GE(f^{\theta,CE}_{P_{S},R})$.

######  Proof 5.

Let $L_{CE}$ and $L_{RJM}$ be the maximum values of CE and RJM respectively.
From Theorem 7, we have $\gamma_{CE}\leq\gamma_{RJM}$ and $L_{CE}\leq
L_{RJM}$. By applying these inequalities to generalization bounds stated in
Theorems 2, 4 and 6, we argue the preposition.

##  6 Experiments

In our experimental evaluation, we do not compete with the state-of-the-art
classification models. The goal is to show the effect of RJM on the
generalization error of DNNs in node and image classification tasks. We train
the models by Adam, AdamW, and SGD using CE and RJM under the same settings
for hyper-parameters to fairly compare our loss function with the previous
one.

###  6.1 Image Classification

####  6.1.1 Problem Formulation

Consider a label set $\\{1,2,\ldots,C\\}$. Let $(\mathrm{x},\mathrm{y})$
represents a sample of this problem where $\mathrm{x}$ is the input image and
$\mathrm{y}\in\mathbbm{1}_{C}$ is the true label. Then,
$\hat{\mathrm{y}}=f^{\theta}(\mathrm{x})$ indicates the output of a deep
learning model $f^{\theta}$ given the input $\mathrm{x}$ which is a
probability vector of size $C$. The predicted label corresponding to
$\mathrm{x}$ is the index of the largest element of $\hat{\mathrm{y}}$.

To obtain $f^{\theta}$ we choose the ResNet50 [23] and VGG16 [24]
architectures. These convolution-based models were pre-trained on large
computer vision datasets and provided adequate results in several
classification tasks. We train these models using both CE and RJM to show how
much RJM can reduce the generalization error of an image classifier in
practice.

####  6.1.2 Dataset and Settings

We utilize the Intel [25] dataset containing images of natural scenes around
the world. There are $14034$ training samples and $3000$ test samples
distributed almost uniformly under $6$ different classes of building, forest,
glacier, mountain, sea, and street in various lights and colors of size
$150\times 150$. To estimate the generalization error properly in each step of
the training phase, it is necessary to have an adequate number of validation
samples. Hence, by separating $4034$ samples randomly from the training set,
we get to the training, validation, and test sets called Intel-Train, Intel-
Val, and Intel-Test respectively. We augment the samples by random cropping
and flipping horizontally.

We use ResNet50 and VGG16 pre-trained on ImageNet [26]. The last layer of
ResNet50 is replaced by two dense layers of sizes $512$ and $6$, respectively,
and the last layer of VGG16 is replaced with one dense layer of size $6$.
AutoGrad does not run over the other layers. We use the same random seed to
initialize the new parameters. ResNet50 is trained by Adam and AdamW. SGD is
used to train VGG16. The batch size and weight decay are set to $64$ and $0.1$
respectively. The models are trained in $20$ epochs. The learning rate value
in each epoch is reported in Table 1 bbbIn AdamW, scheduling the learning is
handled leveraging the scheduling multiplier parameter. However, In our
theoretical results for Adam, the learning rate in the training process is
fixed, but it does not affect the correctness of our theorems because we can
replace $\eta$ by the largest value for the learning rate in the
generalization bound (10).. The parameters $\beta_{1}$ and $\beta_{2}$ are set
to $0.9$ and $0.999$ respectively which are suggested by [6, 7].

Table 1: Learning Rate Settings Optimizer | Learning rate
---|---
Adam |  $10^{-4}$ in epochs $1$ to $9$; $10^{-5}$ in epochs $10$ to $20$
AdamW | same as Adam
SGD |  $10^{-3}$ in epochs $1$ to $9$; $2\times 10^{-4}$ in epochs $10$ to $14$; $4\times 10^{-5}$ in epochs $15$ to $20$

####  6.1.3 Evaluation

As our first observation, we evaluate the generalization performance of our
models by monitoring the generalization error estimate in every epoch:

$\hat{GE}(f^{\theta}_{P_{S},R})=|E_{train}(f^{\theta}_{P_{S},R})-E_{val}(f^{\theta}_{P_{S},R})|,$

where $E_{val}(f^{\theta}_{P_{S},R})$ is the average loss value on the
validation set, and $E_{train}(f^{\theta}_{P_{S},R})$ is the training error of
the output model defined in Section 3. Figures 1(a), 2(a), and 3(a) show the
effectiveness of RJM in preventing the over-fitting issue where the training
and validation sets are Intel-Train and Intel-Val respectively. The plots
demonstrate that training DNNs with RJM can reduce the generalization error
and improve confidence in the output models. Comparing Figures 3(b) and 3(c)
to Figures 1(b), 1(c), 2(b), and 2(c), we realize that the output models
obtained by SGD have been under-fitted. However, RJM was still effective in
diminishing the generalization error.

We also evaluate the generalization performance in terms of Accuracy and
F1-score which are the specific metrics for classification tasks. The results
are reported in Table 2. RJM increases the Accuracy and F1-score of the models
trained by Adam, AdamW, and SGD on the test set. These metrics for models
obtained by SGD are lower than others because it does not use the exponential
moving average of the gradient vector to adapt it, leading to the under-
fitting issue in our case. Furthermore, we report the training times in Table
2 to show that the computation time of RJM is as much as CE.

Table 2:  Accuracy and F1-score on Intel-Test Optimizer | Arch. | Loss function | Accuracy | F1-score | Training time
---|---|---|---|---|---
Adam | ResNet50 | CE | $93.03$ | $93.16$ | $19\text{m}\ 19\text{s}$
RJM | $\mathbf{93.33}$ | $\mathbf{93.44}$ | $19\text{m}\ 15\text{s}$
AdamW | ResNet50 | CE | $92.40$ | $92.50$ | $20\text{m}\ 31\text{s}$
RJM | $\mathbf{93.27}$ | $\mathbf{93.38}$ | $20\text{m}\ 25\text{s}$
SGD | VGG16 | CE | $79.55$ | $79.59$ | $25\text{m}\ 35\text{s}$
RJM | $\mathbf{79.65}$ | $\mathbf{79.79}$ | $25\text{m}\ 36\text{s}$

(a) Generalization error estimate

(b) CE on the training and validation sets

(c) RJM on the training and validation sets

Figure 1: Evaluation in terms of the generalization error estimate and loss
values (Model: ResNet50, Optimizer: Adam)

(a) Generalization error estimate

(b) CE on the training and validation sets

(c) RJM on the training and validation sets

Figure 2: Evaluation in terms of the generalization error estimate and loss
values (Model: ResNet50, Optimizer: AdamW)

(a) Generalization error estimate

(b) CE on the training and validation sets

(c) RJM on the training and validation sets

Figure 3: Evaluation in terms of the generalization error estimate and loss
values (Model: VGG16, Optimizer: SGD)

###  6.2 Node Classification

####  6.2.1 Problem Formulation

Node classification is a single-label learning problem in the graph learning
domain. Consider a graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ where
$\mathcal{V}$ is the node set and
$\mathcal{E}\subseteq\mathcal{V}\times\mathcal{V}$ is the set of edges.
$\mathcal{E}$ indicates the connection between the nodes which is used to pass
messages throughout the graph in the learning process. Let $(v,c)$ be a sample
of this problem such that $v\in\mathcal{V}$ and $c\in\\{1,2,\ldots,C\\}$.
Given a mapping function $f^{\theta}_{\mathcal{V},\mathcal{E}}$, the predicted
probability vector of size $C$ corresponding to $v$ is
$\hat{\mathrm{y}}=f^{\theta}_{\mathcal{V},\mathcal{E}}(v)$. The class with the
highest probability is the predicted label.

In recent years, graph neural networks (GNNs) have been widely exploited to
solve this problem. GNNs learn representations of nodes and predict the
corresponding label to the input from end to end. Two major components of
these architectures in each layer are message transformation and aggregation
which create computation trees for each node. We study pioneering models GCN
[27], GraphSAGE [28], and GAT [29] to assess the new loss function in the
graph learning context.

####  6.2.2 Dataset and Settings

We use a specific version of the CiteSeer dataset [30] containing $3327$
articles, classified into $6$ classes, and form a citation network. The number
of training, validation, and test nodes are $120$, $500$, and $1000$
respectively. Other nodes are isolated. Due to the small number of training
nodes, we can evaluate the models in the over-fitting issue appropriately.
There are $9104$ edges in this graph and all nodes have $3703$ features.

Now we explain our settings for the GNNs. GCN has two hidden layers. Each
GraphSAGE and GAT has one hidden layer. The number of hidden channels of GCN,
GraphSAGE, and GAT are $8$, $16$, and $64$ respectively. We only use Adam to
train the models because based on our observations, it is the most widely used
optimizer in graph learning. The learning rate is set to $0.001$, as suggested
by [6]. The values of $\beta_{1}$ and $\beta_{2}$ are set to $0.9$ and
$0.999$, respectively. We train the GCN model in $100$ epochs. GraphSAGE and
GAT are trained in $200$ epochs. We save the models at the epoch they have the
minimum validation loss.

####  6.2.3 Evaluation

As shown in Figures 4(a), 5(a), and 6(a), the over-fitting issue is alleviated
for the models trained using RJM. The figures illustrate that the
generalization error estimate of all three models becomes closer to zero
significantly when the loss function is RJM.

The Accuracy and F1-score metrics on Test and Validation sets are reported in
Table 3. The results show that RJM can perform better than CE in the domain of
node classification. GAT models outperform the others because, in this
architecture, attention parameters are used to aggregate messages. The best
model is GAT, trained using RJM. The number of parameters of the models and
training samples is very low. Hence, the models take a negligible amount of
time in the training phase, having no value to report.

Table 3: Accuracy and F1-score on Validation and Test Sets Arch. | Loss function | Validation Accuracy | Validation F1-score | Test Accuracy | Test F1-score
---|---|---|---|---|---
GCN | CE | $62.00$ | $57.16$ | $60.70$ | $57.73$
RJM | $\mathbf{63.80}$ | $\mathbf{58.84}$ | $\mathbf{61.30}$ | $\mathbf{58.21}$
GraphSAGE | CE | $60.60$ | $57.76$ | $58.80$ | $55.99$
RJM | $\mathbf{61.80}$ | $\mathbf{59.03}$ | $\mathbf{59.20}$ | $\mathbf{56.33}$
GAT | CE | $62.40$ | $60.20$ | $62.10$ | $59.72$
RJM | $\mathbf{63.00}$ | $\mathbf{60.96}$ | $\mathbf{62.40}$ | $\mathbf{59.91}$

(a) Generalization error estimate

(b) CE on the training and validation sets

(c) RJM on the training and validation sets

Figure 4: Evaluation in terms of the generalization error estimate and loss
values (Model: GCN)

(a) Generalization error estimate

(b) CE on the training and validation set

(c) RJM on the training and validation sets

Figure 5: Evaluation in terms of the generalization error estimate and loss
values (Model: GraphSAGE)

(a) Generalization error estimate

(b) CE on the training and validation set

(c) RJM on the training and validation sets

Figure 6: Evaluation in terms of the generalization error estimate and loss
values (Model: GAT)

##  7 Conclusion and Future Work

In this paper, we proposed the RJM loss function to diminish the
generalization error of DNNs in classification tasks using generalization
bounds previously found under the uniform stability approach, distinguishing
the role of loss functions in improving the generalization of deep learning
models. Comparing RJM to CE in image and node classification problems, we
conclude that RJM reduces over-fitting and increases the value of Accuracy and
F1-score.

RJM can also prevent the over-fitting issue of probabilistic models in other
machine learning tasks (e.g. image segmentation, part of speech tagging, named
entity recognition, recommender systems). Additionally, RJM is applicable in
the multi-label learning framework i.e. single-label classification on each
feature. Note that Lemma 1 can be utilized to create different novel loss
functions in classification tasks.

As a future work, we suggest upper-bounding the difference between the true
and training values of specific evaluation metrics for classification models
(e.g. Accuracy and F1-score) in terms of loss function characteristics and
hyper-parameters of a deep learning problem because the relationship between
loss functions and these metrics is not distinguished by generalization bounds
directly.

## References

  * [1] Z. Zhang, B. Chen, J. Sun, and Y. Luo, “A bagging dynamic deep learning network for diagnosing covid-19,” Scientific Reports, vol. 11, no. 1, p. 16280, 2021.
  * [2] C. F. G. D. Santos and J. P. Papa, “Avoiding overfitting: A survey on regularization methods for convolutional neural networks,” ACM Computing Surveys (CSUR), vol. 54, no. 10s, pp. 1–25, 2022.
  * [3] W. Zhang, Z. Yin, Z. Sheng, Y. Li, W. Ouyang, X. Li, Y. Tao, Z. Yang, and B. Cui, “Graph attention multi-layer perceptron,” in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4560–4570, 2022.
  * [4] A. Akbari, M. Awais, M. Bashar, and J. Kittler, “How does loss function affect generalization performance of deep learning? application to human age estimation,” in International Conference on Machine Learning, pp. 141–151, PMLR, 2021.
  * [5] M. Lashkari and A. Gheibi, “Lipschitzness effect of a loss function on generalization performance of deep neural networks trained by adam and adamw optimizers,” AUT Journal of Mathematics and Computing, 2023.  URL https://doi.org/10.22060/ajmc.2023.22182.1139.
  * [6] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  * [7] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017.
  * [8] T. Zahavy, B. Kang, A. Sivak, J. Feng, H. Xu, and S. Mannor, “Ensemble robustness and generalization of stochastic deep learning algorithms,” arXiv preprint arXiv:1602.02389, 2016.
  * [9] M. Ren, W. Zeng, B. Yang, and R. Urtasun, “Learning to reweight examples for robust deep learning,” in International conference on machine learning, pp. 4334–4343, PMLR, 2018.
  * [10] B. Neyshabur, S. Bhojanapalli, and N. Srebro, “A pac-bayesian approach to spectrally-normalized margin bounds for neural networks,” in International Conference on Learning Representations, 2018.
  * [11] J. Guan and Z. Lu, “Fast-rate pac-bayesian generalization bounds for meta-learning,” in International Conference on Machine Learning, pp. 7930–7948, PMLR, 2022.
  * [12] F. Scarselli, A. C. Tsoi, and M. Hagenbuchner, “The vapnik–chervonenkis dimension of graph and recursive neural networks,” Neural Networks, vol. 108, pp. 248–259, 2018.
  * [13] S. Basu, S. Mukhopadhyay, M. Karki, R. DiBiano, S. Ganguly, R. Nemani, and S. Gayaka, “Deep neural networks for texture classification—a theoretical analysis,” Neural Networks, vol. 97, pp. 173–182, 2018.
  * [14] N. Harvey, C. Liaw, and A. Mehrabian, “Nearly-tight vc-dimension bounds for piecewise linear neural networks,” in Conference on learning theory, pp. 1064–1068, PMLR, 2017.
  * [15] O. Bousquet and A. Elisseeff, “Stability and generalization,” The Journal of Machine Learning Research, vol. 2, pp. 499–526, 2002.
  * [16] S. Shalev-Shwartz, O. Shamir, N. Srebro, and K. Sridharan, “Learnability, stability and uniform convergence,” The Journal of Machine Learning Research, vol. 11, pp. 2635–2670, 2010.
  * [17] M. Hardt, B. Recht, and Y. Singer, “Train faster, generalize better: Stability of stochastic gradient descent,” in International conference on machine learning, pp. 1225–1234, PMLR, 2016.
  * [18] H. Xu and S. Mannor, “Robustness and generalization,” Machine learning, vol. 86, pp. 391–423, 2012.
  * [19] E. Hazan, K. Levy, and S. Shalev-Shwartz, “Beyond convexity: Stochastic quasi-convex optimization,” Advances in neural information processing systems, vol. 28, 2015.
  * [20] T. Tieleman, G. Hinton, et al., “Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude,” COURSERA: Neural networks for machine learning, vol. 4, no. 2, pp. 26–31, 2012.
  * [21] J. Duchi, E. Hazan, and Y. Singer, “Adaptive subgradient methods for online learning and stochastic optimization.,” Journal of machine learning research, vol. 12, no. 7, 2011.
  * [22] K. Matusita, “Decision rules, based on the distance, for problems of fit, two samples, and estimation,” The Annals of Mathematical Statistics, pp. 631–640, 1955.
  * [23] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132–7141, 2018.
  * [24] O. M. Parkhi, A. Vedaldi, and A. Zisserman, “Deep face recognition,” in Proceedings of the British Machine Vision Conference (BMVC) (M. W. J. Xianghua Xie and G. K. L. Tam, eds.), pp. 41.1–41.12, BMVA Press, September 2015\.
  * [25] P. Bansal, “Intel image classification,” 2018.  https://www.kaggle.com/datasets/puneet6060/intel-image-classification?resource=download [Accessed: 12-10-2023].
  * [26] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255, Ieee, 2009.
  * [27] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” arXiv preprint arXiv:1609.02907, 2016.
  * [28] W. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” Advances in neural information processing systems, vol. 30, 2017.
  * [29] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, “Graph attention networks,” arXiv preprint arXiv:1710.10903, 2017.
  * [30] Z. Yang, W. W. Cohen, and R. Salakhutdinov, “Revisiting semi-supervised learning with graph embeddings,” CoRR, vol. abs/1603.08861, 2016.