#  See Through the Fog: Curriculum Learning with Progressive Occlusion in
Medical Imaging

Pradeep Singh Kishore Babu Nampalle  Uppala Vivek Narayan  Balasubramanian
Raman
Department of Computer Science and Engineering
Indian Institute of Technology Roorkee

###### Abstract

In recent years, deep learning models have revolutionized medical image
interpretation, offering substantial improvements in diagnostic accuracy.
However, these models often struggle with challenging images where critical
features are partially or fully occluded, which is a common scenario in
clinical practice. In this paper, we propose a novel curriculum learning-based
approach to train deep learning models to handle occluded medical images
effectively. Our method progressively introduces occlusion, starting from
clear, unobstructed images and gradually moving to images with increasing
occlusion levels. This ordered learning process, akin to human learning,
allows the model to first grasp simple, discernable patterns and subsequently
build upon this knowledge to understand more complicated, occluded scenarios.
Furthermore, we present three novel occlusion synthesis methods, namely
Wasserstein Curriculum Learning (WCL), Information Adaptive Learning (IAL),
and Geodesic Curriculum Learning (GCL). Our extensive experiments on diverse
medical image datasets demonstrate substantial improvements in model
robustness and diagnostic accuracy over conventional training methodologies.

##  1 Introduction

Medical imaging plays a pivotal role in modern healthcare, providing critical
information for diagnosis, treatment planning, and disease monitoring.
However, accurate interpretation of these images remains a challenging task,
largely due to their complex nature and the extensive variation observed among
patients. In recent years, deep learning has emerged as a promising tool to
augment the capabilities of medical practitioners, facilitating better and
faster interpretation of medical images.

Deep learning models, particularly convolutional neural networks (CNNs), have
shown impressive performance in tasks such as image classification, object
detection, and semantic segmentation [21]. Their ability to automatically
learn hierarchical representations from raw data makes them ideally suited for
medical image analysis [22]. Despite their potential, however, the performance
of these models can be significantly affected by the presence of occlusion in
images - where important features or objects are partially or fully obscured
[14]. This problem is of particular concern in the medical domain, where
images often contain overlapping structures or may be occluded by medical
instruments, implants, or artifacts.

Curriculum learning is an approach inspired by the way humans and animals
learn, where the learning process starts from easy examples and gradually
moves to more complex ones. This is seen in our educational curriculum where
we learn numbers before algebra, sentences before essays. This idea can be
applied to machine learning, to train a model on simpler tasks or examples
before more complex ones can make the learning process more effective. This
notion was introduced by Bengio et al. in 2009 [2], postulating that a model
could learn more effectively and efficiently if it first learns to recognize
easily distinguishable patterns and progressively handles more difficult
concepts. The authors investigate the value of a structured, incremental
approach to training machine learning models, particularly neural networks.
This approach can help the model to find better or more appropriate local
minima in the error surface. The training data is sorted in a meaningful order
that presents simpler concepts before more complex ones. This is opposed to
the traditional method of presenting training examples randomly. This
structured presentation of data could lead to a sort of scaffolding where
knowledge is built incrementally and complexities are added progressively.

In the experiments presented in the paper [2], Bengio and his team demonstrate
the potential benefits of curriculum learning in several contexts, including
learning to recognize shapes, language modeling, and other tasks. They show
that a curriculum can help improve generalization and speed up training,
suggesting that this kind of structured learning can be a valuable tool in
training deep learning models. However, _one of the main challenges that the
paper highlights is how to define and design a curriculum for a given
problem_. It remains an open problem and a potential area of research.

In this paper, we propose a novel application of curriculum learning to tackle
the challenge of occlusion in medical images. We argue that, by progressively
increasing the complexity of training examples in terms of occlusion, deep
learning models can learn more robust and accurate representations. We start
by training the model with clear, unobstructed images, and then gradually
introduce images with varying levels of occlusion. This staged learning
process enables the model to initially grasp the simple, discernable patterns
in the data and subsequently apply this foundational knowledge to understand
more complicated, occluded scenarios. Our approach aims to improve the
robustness and diagnostic accuracy of deep learning models in real-world
medical applications, where images are often not perfect and occlusion is
frequently encountered. Through a series of experiments on various medical
image datasets, we demonstrate the competency of our proposed approach over
traditional training methods. Our approach to progressively introducing
occlusion challenges draws inspiration from the methodologies of problem-
solving in physics, which often involve starting with simpler cases before
moving on to more complex scenarios [18, 20]. We believe this work paves the
way for a new line of research in making artificial intelligence more reliable
and effective in the realm of healthcare. Our primary contributions are:

  * •

We develop a novel curriculum learning strategy for deep learning models that
adaptively incorporates increasing levels of occlusion, providing a robust
solution for handling occluded medical images in classification tasks.

  * •

We introduce three novel occlusion synthesis methods based on optimal
transport principles, information theory and exploring the high-dimensional
space of occluded images from a geometric perspective to optimize the model
training process. We name them Wasserstein Curriculum Learning (WCL),
Information Adaptive Learning (IAL) and Geodesic Curriculum Learning (GCL).

  * •

We demonstrate through extensive experiments on real-world medical image
datasets, the effectiveness of our proposed methodology in significantly
improving the classification performance over baseline models.

The rest of the paper is organized as follows: Section 2 provides a
comprehensive review of related works in the fields of deep learning for
medical imaging and curriculum learning. Section 3 details the methodology of
our curriculum learning approach with progressive occlusion. In Section 4, we
present our experimental setup, including the datasets used, the evaluation
metrics, and the baseline models for comparison. Finally, we conclude the
paper and discuss future research directions in Section 5.

##  2 Background

Medical image analysis has been a central focus of artificial intelligence
research over the past decades. Specifically, deep learning techniques have
shown promising results in a wide array of applications, ranging from disease
diagnosis to anatomical structure segmentation. However, occlusions present in
medical images introduce added complexity and ambiguity, challenging these
techniques significantly.

Traditional approaches for medical image analysis primarily relied on
handcrafted features, including textural, morphological, and statistical
properties of the images [24]. However, such methods often grapple with
variability across different patients, modalities, and institutions.
Convolutional neural networks (CNNs), in particular, have revolutionised this
field by allowing the automatic extraction of discriminative features straight
from raw data [3]. These models have achieved state-of-the-art performance in
many medical imaging tasks, such as diagnosing diabetic retinopathy from
retinal images [15], detecting lung nodules from CT scans [7], and classifying
skin lesions from dermoscopic images [11].

The concept of curriculum learning, introduced by Bengio et al., draws
inspiration from the learning progression in humans and animals. The
fundamental idea is to initiate the training process with simpler examples,
gradually escalating the complexity. This strategy has proven its efficacy in
various domains, ranging from object recognition to natural language
processing [23]. However, its application to medical image analysis remains an
under explored area.

Handling occlusions in images has been a long-standing challenge in computer
vision. Occlusions, caused by various factors like overlapping structures,
foreign objects, or missing data, lead to incomplete or ambiguous visual
information [16]. Several methods have been proposed to counter occlusions,
from occlusion-aware models, such as part-based models and deformable models
[12], to occlusion synthesis techniques for data augmentation [19]. Despite
these efforts, occlusion remains a significant hurdle, especially in medical
images, where visual information is often intricate, and the repercussions of
misinterpretation are severe.

Recent advances in machine learning have begun to exploit more advanced
mathematical concepts, such as optimal transport and differential geometry.
Optimal transport offers a potent tool for comparing and transforming
probability distributions, finding a multitude of applications in machine
learning, from domain adaptation to generative models [9]. On the other hand,
differential geometry provides a framework for understanding high-dimensional
spaces and has been employed to investigate the properties of neural networks
and the dynamics of their training process [5]. In this work, we propose a
novel fusion of these diverse research areas to tackle the challenge of
occlusions in medical images. By integrating the principles of curriculum
learning with occlusion synthesis techniques, and employing the mathematical
tools of optimal transport and differential geometry, we aim to develop a
robust and efficient training strategy for deep learning models applied to
occluded medical images. In the subsequent sections, we will detail our
methodology and present our experimental results.

##  3 Methodology

In this section, we detail the methodology of our curriculum learning approach
with progressive occlusion. This approach encompasses two main steps:
occlusion generation (refer figure 1) and training schedule design (refer
figure 2).

Figure 1: Progressive occlusion strategy showing areal occlusions Figure 2:
Schematic representation of the operation of curriculum learning technique

###  3.1 Occlusion Generation

Let $X=\\{x_{1},x_{2},\ldots,x_{n}\\}$ represent our dataset of $n$ images,
and $x_{i}\in\mathbb{R}^{h\times w\times c}$ denote an individual image of
height $h$, width $w$, and $c$ color channels. To generate occlusion, we
introduce a binary mask $M=\\{m_{1},m_{2},\ldots,m_{n}\\}$, where each
$m_{i}\in\\{0,1\\}^{h\times w}$. We define the operation $\odot$ to denote
element-wise multiplication. The occluded image $x_{i}^{\prime}$ is then given
by $x_{i}^{\prime}=x_{i}\odot m_{i}$. The mask $m_{i}$ is generated by
randomly selecting a region within the image and setting the corresponding
pixels to 0, effectively occluding that region.

###  3.2 Training Schedule Design

With the occluded images, we now aim to design a learning schedule following
the principles of curriculum learning. In our approach, we define a function
$f:X\rightarrow\mathbb{R}$ that assigns a difficulty score to each image. The
difficulty of an image $x_{i}$ is proportional to the size of the occlusion,
denoted as $|m_{i}|$. Therefore, we can express this as $f(x_{i})=|m_{i}|$. We
sort the dataset $X$ based on the difficulty scores to obtain a new ordered
dataset
$X^{\prime}=\\{x^{\prime}_{1},x^{\prime}_{2},\ldots,x^{\prime}_{n}\\}$, such
that $f(x^{\prime}_{i})\leq f(x^{\prime}_{j})$ for all $1\leq i<j\leq n$.
Next, we divide the learning process into $T$ stages. At each stage $t$, we
train the model using a subset of the ordered dataset
$S_{t}=\\{x^{\prime}_{i}|\ i\leq n_{t}\\}$, where $n_{t}=\lceil t\cdot
n/T\rceil$. This implies that the model is initially trained with the least
occluded images and progressively exposed to more occluded examples as the
stages advance. For the smaller datasets we can induce different levels of
occlusion per sample. Let $\delta$ represent the number of times each sample
is used for generating its occluded representations. Then the new ordered
dataset is given by $X^{*}=\coprod\limits_{j=0}^{\delta}X^{(j)}$, where
$X^{(j)}$ represents the $j$th level occluded dataset ($X^{(0)}$ represents
the original dataset). The definition of order on $X^{\prime}$ is carried
forward onto $X^{*}$.

Let $y=\\{y_{1},y_{2},\ldots,y_{n}\\}$ represent the ground truth labels
corresponding to the images in $X$. Our model $M:\mathbb{R}^{h\times w\times
c}\rightarrow\mathbb{R}^{k}$ outputs a $k$-dimensional vector for each image,
representing the predicted probabilities for $k$ classes. We employ the
standard cross-entropy loss function
$L:\mathbb{R}^{k}\times\mathbb{R}^{k}\rightarrow\mathbb{R}$, defined as
$L(y,\hat{y})=-\sum_{i=1}^{k}y_{i}\log(\hat{y}_{i})$, where $y$ and $\hat{y}$
represent the ground truth and predicted probability vectors, respectively.
The total loss for the dataset $X$ is then
$\mathcal{L}(X,y)=\frac{1}{n}\sum_{i=1}^{n}L(y_{i},M(x_{i}^{\prime}))$, where
$n$ is the number of images in $X$. During training, we aim to minimize the
loss function using stochastic gradient descent, adjusting the model
parameters to better fit the training data.

###  3.3 Wasserstein Curriculum Learning (WCL)

Historically, the concept of Wasserstein distance has been a fundamental
element in the field of generative models, particularly in the training of
Generative Adversarial Networks (GANs). The introduction of the Wasserstein
GAN (WGAN) [30] aimed to tackle the common issues associated with traditional
GANs, such as unstable training, mode collapse, and vanishing gradients. The
application of the Wasserstein distance in this context allowed for a more
meaningful and smoother gradient during training, resulting in improved model
stability and performance. Furthermore, the Wasserstein distance has been
applied in the domain of adaptation and optimal transport [32]. It provides a
measure to compute the discrepancy between source and target domain
distributions, enabling a geometrically meaningful way of transporting samples
from one distribution to the other. Drawing upon this historical usage of the
Wasserstein distance in deep learning, we leverage its power to compare
probability distributions in our WCL framework. This distance provides an
effective means to ensure a smooth transition of occlusion distributions from
one level of complexity to the next, facilitating more efficient and effective
learning. This will enable a more efficient and effective way of increasing
the complexity of our training examples in a continuous, rather than discrete,
manner. The central idea here is to devise a mechanism by which the transition
from less occluded to more occluded images becomes more fluid, in turn,
offering the model a smoother learning trajectory. Wasserstein distances, a
fundamental notion in optimal transport, to measure the distance or
discrepancy between different distributions. For a smoother curriculum
learning experience, we devise the notion of Wasserstein Curriculum Learning.
The goal is to match the occlusion distributions of the training data from one
complexity level to the next. This approach leverages the first Wasserstein
distance, also known as the Earth Mover’s distance, to guide the gradual
transition from less occluded to more occluded images.

The Wasserstein distance is a measure of the difference between two
probability distributions that considers the underlying geometry of the data
space. It is defined in the context of optimal transport theory, a branch of
mathematics that deals with transporting mass in an optimal way. The $p$-th
Wasserstein distance between two probability measures $\mu$ and $\nu$ on a
metric space $(X,d)$ is defined as follows: Given two probability measures
$\mu$ and $\nu$ on a metric space $(X,d)$, the $p$-th Wasserstein distance
$W_{p}(\mu,\nu)$ is defined as:

$W_{p}(\mu,\nu)=\left(\inf_{\gamma\in\Gamma(\mu,\nu)}\int_{X\times
X}(d(x,y))^{p}d(\gamma(x,y))\right)^{1/p},$

where $p\geq 1$ and $\Gamma(\mu,\nu)$ is the set of all joint distributions
$\gamma$ on $X\times X$ with marginals $\mu$ and $\nu$ on the first and second
factors respectively. For the commonly used first Wasserstein distance
($p=1$), it is often referred to as the earth mover’s distance, as It might be
compared to the least expensive way to move and change a mound of dirt that is
in the form of one probability distribution into that of another. This
definition includes a minimization over all possible joint distributions
between $\mu$ and $\nu$. Solving this problem is computationally expensive and
is typically approximated in practice.

Let us represent the occlusion of each image $x_{i}$ as a normalized histogram
$h_{i}\in\mathbb{R}^{b}$, where $b$ is the number of bins. The occlusion
histogram $h_{i}$ can be thought of as a discrete probability distribution
over the occlusion levels. Given two successive stages $t$ and $t+1$, we aim
to minimize the Wasserstein distance between the occlusion distributions of
the corresponding data subsets, $S_{t}$ and $S_{t+1}$. The first Wasserstein
distance $W_{1}$ between two probability distributions $P$ and $Q$ is defined
as $W_{1}(P,Q)=\min_{\gamma\in\Gamma(P,Q)}\sum_{i,j}|i-j|\cdot\gamma_{i,j}$,
where $\Gamma(P,Q)$ is the set of all joint distributions $\gamma$ whose
marginals are $P$ and $Q$. In our case, $P$ and $Q$ correspond to the
occlusion histograms of $S_{t}$ and $S_{t+1}$. The Wasserstein distance offers
an effective way of comparing probability distributions, taking into account
not only the discrepancies in the distribution values but also their
locations.

By using this method, the model experiences a smooth increase in complexity,
since the occlusion levels between two successive stages have minimal
distance, and no sudden jumps are experienced. This, in turn, could facilitate
a more effective learning process, allowing the model to adjust more easily to
the new complexity level. Here too our training objective remains to minimize
the cross-entropy loss. However, we introduce a regularization term to
encourage a smooth transition between successive stages. Thus, our new loss
function becomes

$\displaystyle{\mathcal{L}}^{(W)}(X,y)=\frac{1}{n}\sum_{i=1}^{n}L(y_{i},M(x_{i}^{\prime}))+\lambda
W_{1}(S_{t},S_{t+1}),$

where $\lambda>0$ is a hyperparameter that controls the importance of the
Wasserstein distance in the loss function.

At each stage $t$, we have a set of training data, and the occlusion histogram
of this data forms a discrete probability distribution $S_{t}$. As the stages
advance (i.e., as $t$ increases), the complexity of the tasks also increases,
here characterized by the level of occlusion in the images. In essence,
$S_{t}$ and $S_{t+1}$ represent the discrete probability distributions of
occlusion levels for two consecutive stages in the learning process. The goal
of WCL, as defined in the loss function, is to minimize the cross-entropy loss
while encouraging a smooth transition between these successive stages, as
measured by the Wasserstein distance between $S_{t}$ and $S_{t+1}$.

###  3.4 Information Adaptive Learning (IAL)

We devise a strategy called Information Adaptive Learning for adaptively
determining the optimal level of occlusion to be introduced at each stage of
training. This strategy involves formulating an auxiliary optimization
problem, which aims to maximize the mutual information between the model’s
outputs and the true labels, subject to the constraint of a maximum allowed
occlusion. A notion from information theory called mutual information
quantifies the amount of knowledge one random variable can learn from
observing another random variable [8]. In our case, we are interested in the
mutual information between the true labels $Y={y_{1},y_{2},\ldots,y_{n}}$ and
the model’s outputs $\hat{Y}={\hat{y}_{1},\hat{y}_{2},\ldots,\hat{y}_{n}}$,
which we denote as $I(Y;\hat{Y})$.

Given a maximum allowed occlusion $\alpha$, we aim to find the occlusion level
that maximizes the mutual information. This can be expressed as the following
optimization problem:

$\displaystyle\max_{|m_{i}|\leq\alpha}$ $\displaystyle\quad I(Y;\hat{Y})$
subject to $\displaystyle\quad|m_{i}|\leq\alpha,\quad\forall i\in{1,\ldots,n}$

In this optimization problem, $|m_{i}|$ represents the level of occlusion
introduced to the i-th sample [13]. The constraint $|m_{i}|\leq\alpha$ ensures
that the occlusion level does not exceed the maximum limit specified by
$\alpha$. The mutual information $I(Y;\hat{Y})$ can be estimated using various
techniques [4], such as non-parametric methods based on k-nearest neighbors,
or parametric methods assuming specific distributions of $Y$ and $\hat{Y}$.
This optimization problem can be solved using gradient-based methods, where
the gradient of $I(Y;\hat{Y})$ with respect to $m_{i}$ can be approximated
using backpropagation.

We modify our training objective again, where our loss function becomes a
weighted combination of the cross-entropy loss, the Wasserstein distance [1],
and the negative mutual information (since we aim to maximize the mutual
information):

$\displaystyle{\mathcal{L}}^{(I)}(X,y)=\frac{1}{n}\sum_{i=1}^{n}L(y_{i},M(x_{i}^{\prime}))+\lambda_{1}W_{1}(S_{t},S_{t+1})-\lambda_{2}I(Y;\hat{Y}),$

where $\lambda_{1},\lambda_{2}>0$ are hyperparameters controlling the
importance of each term.

This adaptive occlusion optimization strategy adds another layer of
sophistication to our curriculum learning approach. By dynamically adjusting
the level of occlusion based on the model’s current performance, we can ensure
that the model is always presented with the right amount of challenge, thereby
fostering more effective learning. This approach, combined with the
Wasserstein curriculum learning strategy, provides a comprehensive framework
for robust training of deep learning models on occluded medical images.

###  3.5 Geodesic Curriculum Learning (GCL)

The concept of viewing the model’s state during training as a point in a high-
dimensional vector space, or more formally, a Riemannian manifold, has been
explored in several areas of machine learning. This approach leverages the
power of differential geometry to handle complex learning trajectories,
adapting and evolving model parameters based on the underlying geometric
structure of the data. In [27], the authors demonstrated the application of
geometric optimization on the manifold of positive definite matrices,
introducing a way to handle constraints and structure in the optimization
process.

The state of our model during training can be characterized by the weights of
its layers, which form a high-dimensional vector space. We can view this
vector space as a Riemannian manifold [28, 29], a mathematical structure that
generalizes the notion of curved surfaces to high dimensions. In a Riemannian
manifold, the distance between two points (or states of our model) is
determined by a metric tensor. In our case, we define the metric tensor based
on the cross-entropy loss function and the Wasserstein distance between
successive stages. This leads to an adaptive representation of our model’s
learning trajectory, where the curvature of the learning path is determined by
the complexity of the training data at each stage.

In this framework, the optimal learning trajectory becomes the geodesic path
on this manifold. We name it Geodesic Curriculum Learning. A geodesic is the
shortest path between two points on a curved surface, or more generally, a
Riemannian manifold. By following this path, our model can adapt more
efficiently to the increasing complexity of the training data, leading to
faster convergence and improved performance. Given two successive stages $t$
and $t+1$, the geodesic path connecting the corresponding model states is the
solution to the geodesic equation, which in general can be written as:

$\displaystyle\frac{d^{2}x^{\lambda}}{dt^{2}}+\Gamma^{\lambda}_{\mu\nu}\frac{dx^{\mu}}{dt}\frac{dx^{\nu}}{dt}=0,$

where $\Gamma^{\lambda}_{\mu\nu}$ are the Christoffel symbols, which depend on
the metric of the space and hence on the loss function and Wasserstein
distance, and $x^{\lambda}$ are the coordinates on the manifold, representing
the model parameters at a particular stage. The indices $\lambda$, $\mu$, and
$\nu$ run over all dimensions of the model parameter space. This is a system
of second-order differential equations that describe the evolution of the
model’s weights along the geodesic path. In practice, we can approximate the
solution to these equations using numerical integration methods, such as the
Euler method or the Runge-Kutta method.

We further refine our training objective. In addition to the cross-entropy
loss, the Wasserstein distance, and the mutual information, we introduce a
regularization term based on the length of the geodesic path. The new loss
function becomes:

$\displaystyle{\mathcal{L}}^{(G)}(X,y)=\frac{1}{n}\sum_{i=1}^{n}L(y_{i},M(x_{i}^{\prime}))+\lambda_{1}W_{1}(S_{t},S_{t+1})\
-\lambda_{2}I(Y;\hat{Y})+\lambda_{3}L_{geo}(M_{t},M_{t+1}),$

where $L_{geo}(M_{t},M_{t+1})$ represents the length of the geodesic path
between the model states at stages $t$ and $t+1$, and $\lambda_{3}>0$ is a
hyperparameter. It can be written as:

$\displaystyle
L_{geo}(M_{t},M_{t+1})=\int_{t}^{t+1}\sqrt{g_{ij}\frac{dM^{i}}{dt}\frac{dM^{j}}{dt}}dt,$

where $g_{ij}$ is the metric tensor which encodes the distance between two
infinitesimally close points in the parameter space, and $M^{i}$ represents
the coordinates in the parameter space (i.e., the weights of the model). This
equation essentially sums up (or integrates) all the infinitesimal distances
along the geodesic path to get the total length of the path. Casting the
learning process in the framework of differential geometry provides a
geometric interpretation to curriculum learning.

###  3.6 Explainability induced by WCL, IAL and GCL

As the field of AI progresses, the need for transparency and understandability
in machine learning models becomes more and more important, especially in
fields like healthcare, where interpretability of model decisions can have
critical consequences [25]. WCL, IAL, and GCL methods offer unique pathways
towards better explainability of AI models.

The Wasserstein Curriculum Learning approach facilitates explainability by
providing insights into how the model copes with the changing complexity of
the data. By leveraging Wasserstein distances, we can measure the difference
in complexity levels between different stages of learning. This can be
interpreted as the model’s journey of learning and adaptation as it
transitions from less occluded to more occluded images, providing a
quantitative and interpretable narrative of the learning process.

Information Adaptive Learning introduces a criterion based on maximizing
mutual information between the true labels and the model’s predictions. The
mutual information is a measure of the statistical dependence between two
variables, giving a clear and intuitive quantification of how much the model’s
output depends on the input. Therefore, a high mutual information implies that
the model has learned significant features from the input data. This becomes a
quantifiable measure of interpretability of what the model has learned.

Geodesic Curriculum Learning provides a geometric perspective on the model’s
learning trajectory. By representing the learning process as a path on a high-
dimensional Riemannian manifold, we provide a geometric visualization of the
learning process. This visualization can be used to explain how the model
evolves over time, what changes in the data affect its evolution, and how the
model reaches its final state. Additionally, the length of the geodesic path
represents a measure of the difficulty or complexity of the learning process
from one stage to another. Shorter paths correspond to easier transitions,
indicating that the model is able to adapt more efficiently to the new
complexity level. Conversely, longer paths signify more challenging
transitions. This can provide insights into how the model handles different
complexities and how it adjusts its parameters accordingly, providing an
interpretable measure of the model’s adaptability.

These three methods combined provide a comprehensive framework for enhancing
the transparency and interpretability of AI models. By using these approaches,
we can better understand and explain how our model learns, adapts, and makes
decisions, making the black-box nature of deep learning models a bit more
interpretable.

In the following section, we will discuss the implementation details and
present our experimental results, which showcase the effectiveness of this
Wasserstein Curriculum Learning strategy in dealing with occlusion in medical
image analysis.

##  4 Experiments & Results

In order to empirically evaluate the efficacy of our proposed methodology, we
conducted a series of experiments using various datasets of medical images.
For the purpose of our experiment, we selected a pre-trained MobileNetV2
architecture as our baseline model, which has achieved notable success in
various image classification tasks. We appended it with customized top layers
to tailor the network towards our specific binary and multi-class
classification tasks.

Figure 3:  The architectural design of the suggested medical image
classification system.

###  4.1 Architecture

Our model architecture (as shown in figure 3) is founded on MobileNetV2 [26],
a highly effective neural network known for its efficiency in image
classification tasks. The MobileNetV2 model we employ is pre-trained on the
ImageNet dataset [10], which allows us to leverage the learned feature
representations for our specific medical image classification tasks. The base
architecture, $\mathcal{M}$, is expressed as:

$\mathcal{M}:\mathbb{R}^{H\times W\times C}\rightarrow\mathbb{R}^{D}$

where $\mathbb{R}^{H\times W\times C}$ and $\mathbb{R}^{D}$ represent the
input and output spaces respectively. Here, $H$, $W$, and $C$ denote the
height, width, and the number of color channels of the input image,
respectively, and $D$ is the output dimension after the last layer of the base
model (which is flattened in our case).

The base model’s output, $\mathcal{M}(x)$, is a $D$-dimensional feature vector
that contains the learned representations of the input image $x$. This output
is then passed through a series of transformations to make it suitable for our
specific classification task. The transformations include dropout layers for
regularization, Dense layers for non-linear transformations, and Batch
Normalization layers for normalization. These transformations are collectively
denoted by $\mathcal{T}$, and can be expressed as:

$\mathcal{T}:\mathbb{R}^{D}\rightarrow\mathbb{R}^{D^{\prime}}$

where $\mathbb{R}^{D^{\prime}}$ is the output space after the transformations,
with $D^{\prime}$ being the output dimension after the final transformation
layer.

Lastly, the output of the transformations, $\mathcal{T}(\mathcal{M}(x))$, is
fed into a fully-connected (Dense) layer with sigmoid activation function for
binary classification tasks (or softmax for multi-class tasks). This layer,
denoted by $\mathcal{F}$, can be defined as:

$\mathcal{F}:\mathbb{R}^{D^{\prime}}\rightarrow\mathbb{R}^{K}$

where $\mathbb{R}^{K}$ is the output space after the final fully-connected
layer, and $K$ is the number of classes in our classification task. Our
complete model can thus be written as the composition of these functions:

$\mathcal{F}\circ\mathcal{T}\circ\mathcal{M}(x)$

where the circle symbol $\circ$ denotes function composition. The entire
process, from input image to class probabilities, is expressed by this
function composition.

The objective during the training process is to learn the parameters of the
model that minimize the loss function, which in our case is a weighted sum of
Binary Crossentropy (or Categorical Crossentropy for multi-class tasks) and
the Wasserstein distance between occlusion histograms. The loss function
$\mathcal{L}$ can be defined as:

${\mathcal{L}}^{(W)}(y,\hat{y})=\frac{1}{n}\sum_{i=1}^{n}L(y,\hat{y})+\lambda
W_{1}(p,q)$

where $y$ and $\hat{y}$ are the true and predicted labels, $p$ and $q$ are the
occlusion histograms, and $\lambda$ is a hyperparameter that controls the
contribution of the Wasserstein distance to the overall loss.

###  4.2 Datasets and Preprocessing

We have employed two datasets for our experiments. For the binary
classification task, the Br35H dataset was utilized [17]. This dataset
comprises tumor and non-tumor images, providing a binary classification
challenge. The second dataset used was the Brain Multi-Class (brain-multi)
dataset [6], offering a multi-class classification task with four classes -
glioma, meningioma, pituitary, and non-tumor. Image preprocessing involved
resizing the images to match the input size of the MobileNetV2 model and
normalizing pixel intensities. Furthermore, to simulate increasing levels of
occlusion and add more diversity to our training samples, we employed the
occlusion synthesis method mentioned in 3.3, which modifies images to mimic
occluded conditions. This synthesis helps the model better handle real-world
occlusions, providing a broader and more challenging training scope.

###  4.3 Results

We evaluated the models’ performance using standard metrics, including
accuracy, precision, recall, and F1-score. For the multi-class task, we
computed the macro averages of these metrics. Additionally, the Area Under the
Receiver Operating Characteristic Curve (AUC-ROC) was computed to assess the
models’ ability to distinguish between the classes under different thresholds.

In Table1, Baseline refers to the pre-trained MobileNetV2 with a single
sigmoid neuron to classify in the case of binary and 4 neurons with softmax
activation in case of 4-class classification of brain tumors. PROS refers to
Progressive Random Occlusion Strategy, where random occlusions were applied
progressively and PBOS refers to Progressive Border Occlusion Strategy where
occlusions were applied progressively as hollow rectangles of width 3 pixels.

Table 1: Quantitative Results Analysis

Strategy |  Dataset |  Precision |  Recall |  F1-Score |  ROC-AUC |  Accuracy
---|---|---|---|---|---|---
PROS |  Br35H |  100 |  99.67 |  99.83 |  100 |  99.83
Baseline |  Br35H |  100 |  99.00 |  99.50 |  99.67 |  99.50
PBOS |  Br35H |  100 |  98.67 |  99.33 |  99.67 |  99.33
PBOS |  brain-multi |  97.99 |  97.94 |  97.96 |  98.64 |  98.02
PROS |  brain-multi |  97.52 |  97.27 |  97.37 |  98.19 |  97.41
Baseline |  brain-multi |  96.26 |  95.91 |  96.03 |  96.88 |  96.11

These results indicate the effectiveness of our strategy to blend curriculum
learning with adaptive occlusion optimization. This integrated approach has
proven to handle occlusions in medical images adeptly and improve the
performance of deep learning models on complex classification tasks. The
robustness of the model performance, as evident from the metrics, highlights
the suitability of our approach for real-world medical imaging applications
where occlusions are common.

##  5 Discussion and Ending Remarks

In this work, we proposed a novel training methodology integrating curriculum
learning with adaptive occlusion optimization for deep learning models applied
to medical image classification tasks. The choice of MobileNetV2 as our base
model, equipped with our custom-built top layers, proved to be well-suited for
both binary and multi-class medical image classification tasks. Our
experiments demonstrated a significant improvement in model performance when
comparing our proposed methodology with the baseline model. This lends
credence to our hypothesis that by gradually introducing occlusion challenges
into the training process, much like the principles of curriculum learning, we
can enhance the model’s ability to handle occluded objects effectively.

The application of optimal transport in the occlusion synthesis process
allowed for a smoother transition between different occlusion levels, making
it easier for the model to adapt and learn the complex structures underlying
the data. Moreover, the differential geometry perspective helped us better
understand the landscape of the high-dimensional space formed by the occluded
images, potentially offering clues on how to further optimize the training
process. This work opens up a number of interesting avenues for future
research. The occlusion synthesis process could be further refined by
incorporating more sophisticated occlusion models or by using generative
models like GANs to create more diverse and realistic occlusions.
Additionally, other forms of curriculum learning, such as self-paced learning
or task difficulty estimation, could be integrated into our framework to
further enhance its effectiveness. From the perspective of optimal transport
and differential geometry, exploring other applications of these powerful
mathematical tools in the context of deep learning is a promising direction.
Particularly, their potential roles in other challenging issues in medical
image analysis, such as noise reduction, outlier detection, or multi-modal
data integration, could be investigated.

While our results are promising, it is important to note that the ultimate
measure of success is the practical impact of these methods in real-world
clinical settings. This would involve not only technical challenges such as
system integration, scalability, and real-time processing but also non-
technical issues such as user acceptance, regulatory compliance, and ethical
considerations. Future work should, therefore, aim at more extensive
validation on diverse and larger datasets, potentially including different
types of imaging modalities, diseases, and occlusion levels. Furthermore, it
would be interesting to examine how our methodology performs when integrated
into a full-fledged computer-aided diagnosis system, and whether it can help
improve the diagnostic accuracy and efficiency of healthcare professionals.
Additionally, our methodology requires a substantial amount of computational
resources due to the complexity of the optimal transport computation and the
large number of training iterations required by the curriculum learning
approach. This could be a constraint in scenarios with limited computational
resources or require real-time processing. It would be worthwhile to
investigate more efficient implementations or approximations of the optimal
transport computation and the curriculum learning process. Potential solutions
could involve using more efficient optimal transport algorithms, parallel
computing techniques, or hardware accelerators, or by developing more
sophisticated curriculum learning strategies that can achieve similar
performance improvements with fewer training stages or less severe occlusions.

## Acknowledgements

This research has been funded in part by the Department of Atomic Energy,
India under grant number 0204/18/2022/R&D-II/13979 and the Ministry of
Education, India under grant reference number OH-31-24-200-428.

## References

  * [1] M. Arjovsky, S. Chintala, and L. Bottou, Wasserstein GAN, Proceedings of the 34th International Conference on Machine Learning, pp. 214-223, 2017.
  * [2] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, Curriculum learning, Proceedings of the 26th annual international conference on machine learning, 41–48, 2009.
  * [3] Y. Bengio, Y. LeCun, and others, Scaling learning algorithms towards AI, Large-scale kernel machines, 34(5):1–41, 2007.
  * [4] M.I. Belghazi, A. Baratin, S. Rajeshwar, S. Ozair, Y. Bengio, A. Courville, and D. Hjelm, Mutual Information Neural Estimation, Proceedings of the 35th International Conference on Machine Learning, pp. 531-540, 2018.
  * [5] M.M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst, Geometric deep learning: going beyond Euclidean data, IEEE Signal Processing Magazine, 34(4):18–42, 2017.
  * [6] J. Cheng, Brain magnetic resonance imaging tumor dataset, Figshare MRI Dataset Version, 5, 2017.
  * [7] F. Ciompi, B. de Hoop, S.J. van Riel, K. Chung, E.Th. Scholten, M. Oudkerk, P.A. de Jong, M. Prokop, and B. van Ginneken, Automatic classification of pulmonary peri-fissural nodules in computed tomography using an ensemble of 2D views and a convolutional neural network out-of-the-box, Medical image analysis, 26(1):195–202, 2015.
  * [8] T.M. Cover and J.A. Thomas, Elements of Information Theory, Wiley, 2006.
  * [9] M. Cuturi, Sinkhorn distances: Lightspeed computation of optimal transport, Advances in Neural Information Processing Systems, 26, 2013.
  * [10] J. Deng, W. Dong, R. Socher, L.J. Li, K. Li, and L. Fei-Fei, Imagenet: A large-scale hierarchical image database, 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, 2009.
  * [11] A. Esteva, B. Kuprel, R.A. Novoa, J. Ko, S.M. Swetter, H.M. Blau, and S. Thrun, Dermatologist-level classification of skin cancer with deep neural networks, nature, 542(7639):115–118, 2017.
  * [12] P.F. Felzenszwalb, R.B. Girshick, D. McAllester, and D. Ramanan, Object detection with discriminatively trained part-based models, IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(9):1627–1645, 2009.
  * [13] Y. Gal and Z. Ghahramani, Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning, Proceedings of The 33rd International Conference on Machine Learning, 2016.
  * [14] R. Girshick, J. Donahue, T. Darrell, and J. Malik, Rich feature hierarchies for accurate object detection and semantic segmentation, Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580-587, 2014.
  * [15] V. Gulshan, L. Peng, M. Coram, M.C. Stumpe, D. Wu, A. Narayanaswamy, S. Venugopalan, K. Widner, T. Madams, J. Cuadros, and others, Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs, Jama, 316(22):2402–2410, 2016.
  * [16] G.S. Halford, Children’s understanding: The development of mental models, Psychology Press, 2014.
  * [17] HASHIRA, Br35H :: Brain Tumor Detection 2020 Dataset, Roboflow Universe, 2022, https://universe.roboflow.com/hashira-fhxpj/br35h-::-brain-tumor-detection-2020, visited on 2023-06-08.
  * [18] I.E. Irodov, Problems in general physics, Moscow: Mir Publishers, 1981.
  * [19] O.F. Kar, T. Yeo, A. Atanov, and A. Zamir, 3D common corruptions and data augmentation, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18963–18974, 2022.
  * [20] S.S. Krotov, Aptitude Test Problems in Physics, Mir Publishers, 1990.
  * [21] Y. LeCun, Y. Bengio, and G. Hinton, Deep learning, Nature, 521(7553):436-444, 2015.
  * [22] G. Litjens, T. Kooi, B. Ehteshami Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. W. M. van der Laak, B. van Ginneken, and C. I. Sánchez, A survey on deep learning in medical image analysis, Medical Image Analysis, 42:60-88, 2017.
  * [23] L. Liu, W. Ouyang, X. Wang, P. Fieguth, J. Chen, X. Liu, and M. Pietikäinen, Deep learning for generic object detection: A survey, International journal of computer vision, 128:261–318, 2020.
  * [24] A. Madabhushi and G. Lee, Image analysis and machine learning in digital pathology: Challenges and opportunities, Medical image analysis, 33:170–175, 2016.
  * [25] M.T. Ribeiro, S. Singh, and C. Guestrin, Why Should I Trust You?: Explaining the Predictions of Any Classifier, Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
  * [26] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.C. Chen, MobileNetV2: Inverted Residuals and Linear Bottlenecks, Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
  * [27] S. Sra and R. Hosseini, Conic Geometric Optimization on the Manifold of Positive Definite Matrices, SIAM Journal on Optimization, 25(1):713-739, 2015. [Online]. Available: https://doi.org/10.1137/140978168
  * [28] L.W. Tu, An Introduction to Manifolds, Springer, 2008.
  * [29] L.W. Tu, Differential Geometry: Connections, Curvature, and Characteristic Classes, Springer, 2017.
  * [30] C. Wang, C. Xu, X. Yao, and D. Tao, Evolutionary generative adversarial networks, IEEE Transactions on Evolutionary Computation, 23(6):921–934, 2019.
  * [31] G. Yang, Q. Ye, and J. Xia, Unbox the black-box for the medical explainable ai via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond, Information Fusion, 77:29–52, 2022.
  * [32] Z. Zhang, M. Wang, and A. Nehorai, Optimal transport in reproducing kernel hilbert spaces: Theory and applications, IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(7):1741–1754, 2019.

##  6 Appendix

######  Definition 6.1.

Let $\mathcal{P}_{p}(\mathbb{R}^{n})$ be the set of all Borel probability
measures on $\mathbb{R}^{n}$ with finite $p$th moment. Then the
$p$-Wasserstein distance
$W_{p}:\mathcal{P}_{p}(\mathbb{R}^{n})\times\mathcal{P}_{p}(\mathbb{R}^{n})\rightarrow\mathbb{R}$
is defined as follows:

$\displaystyle
W_{p}(\mu,\nu)=\left(\inf_{\gamma\in\Gamma(\mu,\nu)}\int_{\mathbb{R}^{n}\times\mathbb{R}^{n}}|x-y|^{p}d\gamma(x,y)\right)^{1/p},$

where $\Gamma(\mu,\nu)$ denotes the set of all couplings of $\mu$ and $\nu$,
i.e., all probability measures on $\mathbb{R}^{n}\times\mathbb{R}^{n}$ with
marginals $\mu$ and $\nu$.

######  Proposition 6.1.

The Wasserstein distance $W_{p}$ defines a metric on
$\mathcal{P}_{p}(\mathbb{R}^{n})$.

###### Proof.

We need to verify the properties of non-negativity, identity of
indiscernibles, symmetry, and the triangle inequality. The non-negativity,
identity of indiscernibles, and symmetry are clear from the definition of
$W_{p}$. For the triangle inequality, let
$\mu,\nu,\xi\in\mathcal{P}p(\mathbb{R}^{n})$. Then, for any
$\gamma\in\Gamma(\mu,\nu)$ and $\gamma^{\prime}\in\Gamma(\nu,\xi)$, we can
construct a coupling $\bar{\gamma}$ of $\mu$ and $\xi$ such that
$\int_{\mathbb{R}^{n}\times\mathbb{R}^{n}}|x-z|^{p}d\bar{\gamma}(x,z)\leq\int_{\mathbb{R}^{n}\times\mathbb{R}^{n}}|x-y|^{p}d\gamma(x,y)+\int_{\mathbb{R}^{n}\times\mathbb{R}^{n}}|y-z|^{p}d\gamma^{\prime}(y,z)$.
Hence, $W_{p}(\mu,\xi)\leq W_{p}(\mu,\nu)+W_{p}(\nu,\xi)$.

The coupling $\bar{\gamma}$ can be constructed using a gluing procedure on the
couplings $\gamma$ and $\gamma^{\prime}$. Suppose that $\gamma$ and
$\gamma^{\prime}$ are optimal couplings of $\mu$ to $\nu$ and $\nu$ to $\xi$,
respectively. For $\gamma$, consider the joint distribution on
$\mathbb{R}^{n}\times\mathbb{R}^{n}$, where $(x,y)$ represent a point in the
source domain and a point in the target domain, respectively. Similarly, for
$\gamma^{\prime}$, $(y,z)$ represents a point in the source domain (the same
as the target of $\gamma$) and a point in the target domain. Now, consider a
random variable $Y$ that has distribution $\nu$. By the definition of
couplings, we can find random variables $X$ and $Z$ with distributions $\mu$
and $\xi$, respectively, and the joint distributions of $(X,Y)$ and $(Y,Z)$
are given by $\gamma$ and $\gamma^{\prime}$. The coupling $\bar{\gamma}$ of
$\mu$ and $\xi$ is then the joint distribution of $(X,Z)$, which can be seen
as gluing together the couplings $\gamma$ and $\gamma^{\prime}$ through the
common variable $Y$. In other words, for each transport plan from $\mu$ to
$\nu$ (represented by $x$ to $y$) and from $\nu$ to $\xi$ (represented by $y$
to $z$), we construct a direct transport plan from $\mu$ to $\xi$ (represented
by $x$ to $z$) by bypassing the intermediate step $y$.

∎

Note that in general, constructing this coupling in a deterministic way may
not be possible or straightforward, especially in infinite-dimensional spaces.
The process may require certain regularity conditions or a probabilistic
framework to make the gluing well-defined. However, this construction is
typical in the proofs of various properties of optimal transport distances,
and it helps illustrate the fundamental ideas behind these distances.

######  Definition 6.2.

Given a joint probability distribution $P_{X,Y}$ for random variables $X$ and
$Y$, the mutual information $I(X;Y)$ is defined as follows:

$\displaystyle
I(X;Y)=\sum_{x\in\mathcal{X}}\sum_{y\in\mathcal{Y}}P_{X,Y}(x,y)\log\left(\frac{P_{X,Y}(x,y)}{P_{X}(x)P_{Y}(y)}\right),$

where $\mathcal{X}$ and $\mathcal{Y}$ denote the supports of $X$ and $Y$, and
$P_{X}$ and $P_{Y}$ are the marginal distributions of $X$ and $Y$.

######  Definition 6.3 (Shannon Entropy).

Let $X$ be a discrete random variable with a finite set of possible outcomes
${x_{1},x_{2},\ldots,x_{n}}$, where each outcome $x_{i}$ occurs with
probability $P(x_{i})$. The Shannon entropy $H(X)$ of $X$ is defined as:

$\displaystyle H(X)=-\sum_{i=1}^{n}P(x_{i})\log_{2}P(x_{i}),$

where the base of the logarithm is 2 if the measure of information is in bits
(it could also be base $e$ for nats, or base 10 for Hartleys, but bits are the
most common measure).

The Shannon entropy essentially quantifies the amount of uncertainty or
information inherent in the variable’s possible outcomes. For example, if all
outcomes are equally likely (i.e., $X$ has a uniform distribution), then the
entropy is maximized and equals $\log_{2}n$. This represents a state of
maximum uncertainty, as we have no prior information that would allow us to
predict the outcome. Conversely, if one outcome has a probability of 1 (i.e.,
$X$ is a constant), then the entropy is 0, reflecting that we have complete
certainty about the outcome. In the case of mutual information, it provides a
measure of the amount of information that knowing the outcome of one random
variable provides about the outcome of the other. Therefore, non-negativity of
mutual information as per the proposition stated, tells us that knowing one
variable always either increases our knowledge of the other or leaves it
unchanged, but never decreases it.

######  Proposition 6.2.

The mutual information $I(X;Y)$ is non-negative and symmetric, i.e.,
$I(X;Y)=I(Y;X)$.

###### Proof.

Non-negativity follows from the log-sum inequality. To show symmetry, we
observe that $I(X;Y)=H(X)-H(X|Y)=H(Y)-H(Y|X)=I(Y;X)$, where $H$ denotes the
Shannon entropy. ∎

In the paper, we proposed a new optimization problem that seeks to maximize
the mutual information $I(Y;\hat{Y})$ subject to a constraint on the maximum
level of occlusion. This problem can be formulated mathematically as follows:

$\displaystyle\max_{\hat{Y}};$ $\displaystyle I(Y;\hat{Y})$ subject to
$\displaystyle E[\text{Occlusion}]\leq\theta,$

where $Y$ and $\hat{Y}$ denote the true and predicted labels, Occlusion is a
random variable representing the level of occlusion, $E$ denotes expectation,
and $\theta$ is a pre-specified threshold.

In the manifold view of our learning process, we model the parameter space as
a Riemannian manifold and construct a metric tensor using the Wasserstein
distance and cross-entropy loss. This approach allows us to define geodesics,
or the shortest paths, in the manifold that represent the learning trajectory
of our model.

######  Definition 6.4.

A geodesic $\gamma:[0,1]\rightarrow\mathcal{M}$ in a Riemannian manifold
$(\mathcal{M},g)$ is a curve that locally minimizes distance, i.e., for any
$t\in[0,1]$ there exists a neighborhood $U$ of $t$ such that $\gamma|_{U}$
minimizes distance from $\gamma(t_{1})$ to $\gamma(t_{2})$ for all
$t_{1},t_{2}\in U$.

In our setting, the geodesic is described by the second-order ordinary
differential equation:

$\displaystyle\frac{d^{2}x^{\lambda}}{dt^{2}}+\Gamma^{\lambda}_{\mu\nu}\frac{dx^{\mu}}{dt}\frac{dx^{\nu}}{dt}=0,$

where $x^{\lambda}$ are the coordinates in the parameter space and
$\Gamma^{\lambda}_{\mu\nu}$ are the Christoffel symbols, defined as:

$\displaystyle\Gamma^{\lambda}_{\mu\nu}=\frac{1}{2}g^{\lambda\sigma}\left(\frac{\partial
g_{\mu\sigma}}{\partial x^{\nu}}+\frac{\partial g_{\nu\sigma}}{\partial
x^{\mu}}-\frac{\partial g_{\mu\nu}}{\partial x^{\sigma}}\right),$

where $g_{\mu\nu}$ are the components of the metric tensor and
$g^{\lambda\sigma}$ are the components of its inverse. This equation can be
solved numerically using techniques such as the Euler method or the Runge-
Kutta method.

The Euler method is a first-order numerical procedure for solving ordinary
differential equations (ODEs) with a given initial value. It is one of the
simplest methods to numerically solve the initial value problem, albeit the
error can build up rapidly if the step size isn’t small enough. Here we
formulate the solution using the Euler method.

Given a system of second-order ordinary differential equations (ODEs) like the
geodesic equation, we can rewrite it as a system of first-order ODEs. For our
geodesic equation, we define the velocity
$v^{\lambda}=\frac{dx^{\lambda}}{dt}$, and rewrite the geodesic equation as:

  1. 1.

$\frac{dx^{\lambda}}{dt}=v^{\lambda}$

  2. 2.

$\frac{dv^{\lambda}}{dt}=-\Gamma^{\lambda}_{\mu\nu}v^{\mu}v^{\nu}$

For the Euler method, we update the parameters and velocities iteratively as
follows:

  1. 1.

$x^{\lambda}_{k+1}=x^{\lambda}_{k}+hv^{\lambda}_{k}$,

  2. 2.

$v^{\lambda}_{k+1}=v^{\lambda}_{k}-h\Gamma^{\lambda}_{\mu\nu}v^{\mu}_{k}v^{\nu}_{k}$,

where $h$ is the step size (related to the learning rate in the training
process), and the subscript $k$ indicates the iteration step. This update
scheme is performed for each dimension in the parameter space, which is
equivalent to updating each model parameter iteratively based on the current
velocity and the geodesic equation’s acceleration term. Note that the
Christoffel symbols $\Gamma^{\lambda}_{\mu\nu}$ depend on the metric of the
space, which in our case is defined based on the cross-entropy loss function
and the Wasserstein distance. These quantities should be recomputed at each
step as the model parameters evolve. Also, the selection of a suitable step
size $h$ is critical in ensuring the numerical stability and accuracy of the
Euler method.

######  Proposition 6.3.

Given two states $M_{t}$ and $M_{t+1}$ in the model’s parameter space, the
geodesic path that connects them is unique.

###### Proof.

This statement is a direct consequence of the Hopf-Rinow theorem, which states
that any two points on a complete Riemannian manifold can be connected by a
minimizing geodesic. The parameter space of a deep learning model can be seen
as a subset of a Euclidean space, which is a complete Riemannian manifold.
Ergo, any two states of the model can be connected by a unique geodesic path.
∎

The length of the geodesic path between two model states $M_{t}$ and $M_{t+1}$
provides a measure of the difficulty of the learning transition from state
$M_{t}$ to state $M_{t+1}$. Indeed, the geodesic path is the shortest path
between two states in the model’s parameter space, according to the metric
induced by the loss function and the Wasserstein distance. Therefore, its
length represents the minimal amount of change needed to transition from state
$M_{t}$ to state $M_{t+1}$. This can be interpreted as the effort required by
the model to adapt to the new complexity level. Hence, longer paths correspond
to more challenging transitions, while shorter paths represent easier
transitions.

The geodesic path between two states $M_{t}$ and $M_{t+1}$ in the model’s
parameter space minimizes the rate of change in the model’s parameters. The
rate of change in the model’s parameters can be quantified as the derivative
of the parameters with respect to the training stage. We denote this
derivative by $\frac{dM}{dt}$. The length of the geodesic path is given by

$\displaystyle
L_{geo}(M_{t},M_{t+1})=\int_{t}^{t+1}\sqrt{g_{ij}\frac{dM^{i}}{dt}\frac{dM^{j}}{dt}}dt,$

where $g_{ij}$ is the metric tensor, which encodes the distance between two
infinitesimally close points in the parameter space. The term under the square
root can be interpreted as the square of the norm of the rate of change in the
model’s parameters, according to the metric $g_{ij}$. Therefore, the length of
the geodesic path is a measure of the rate of change in the model’s
parameters. By definition, the geodesic path is the shortest path between
$M_{t}$ and $M_{t+1}$, according to the metric $g_{ij}$. Hence, it minimizes
the length of the path, and thereby, it also minimizes the rate of change in
the model’s parameters.

######  Proposition 6.4.

Assuming the learning rate $\eta$ is sufficiently small, the model’s learning
trajectory under GCL converges to the minimum of the loss function at a
geometric rate, i.e., there exists a constant $0\leq r<1$ such that for the
error $\epsilon_{n}=||M_{n}-M^{*}||$ between the model state $M_{n}$ at stage
$n$ and the optimal model state $M^{*}$ (that minimizes the loss function), we
have

$\frac{\epsilon_{n+1}}{\epsilon_{n}}\leq r,$

for all sufficiently large $n$, where $||\cdot||$ is a norm in the model’s
parameter space representing the distance between two states.

###### Proof.

Let $L(M)$ denote the loss function, where $M$ represents the state of the
model in the high-dimensional parameter space. We aim to show that the
sequence of states $\\{M_{k}\\}_{k=1}^{\infty}$ generated by the GCL algorithm
converges to a minimum $M^{*}$ of $L(M)$ at a geometric rate. Consider the
geodesic path $\gamma:[0,1]\rightarrow\mathcal{M}$ on the Riemannian manifold
$\mathcal{M}$ of the model parameters, connecting the state $M_{k}$ at stage
$k$ to the state $M_{k+1}$ at stage $k+1$. The GCL algorithm ensures that this
path follows the direction of steepest descent in the loss function, which,
combined with a sufficiently small learning rate $\eta$, guarantees that the
sequence $\\{M_{k}\\}_{k=1}^{\infty}$ converges to a minimum $M^{*}$ of
$L(M)$.

To show that this convergence is at a geometric rate, we will use the fact
that, along the geodesic path, the Riemannian distance $d(M_{k},M^{*})$
decreases exponentially. More specifically, from the properties of geodesics
and assuming a small learning rate $\eta$, we can write the following
inequality:

$d(M_{k+1},M^{*})\leq(1-\eta\cdot\rho)d(M_{k},M^{*}),$

where $\rho>0$ is a constant that depends on the geometry of the loss function
and the learning rate $\eta$. This inequality shows that the ratio between the
Riemannian distances $d(M_{k+1},M^{*})$ and $d(M_{k},M^{*})$ is bounded by
$(1-\eta\cdot\rho)$, which is strictly less than 1 due to $\eta\cdot\rho>0$.
Ergo, the sequence $\\{d(M_{k},M^{*})\\}_{k=1}^{\infty}$ of distances from the
model state $M_{k}$ to the minimum $M^{*}$ of the loss function decreases at a
geometric rate, which implies that the sequence $\\{M_{k}\\}_{k=1}^{\infty}$
of model states converges to $M^{*}$ at a geometric rate as well. This
completes the proof. ∎