[orcid=0000-0002-9578-5706]

[1]

1] addressline=School of Computer Science, Queensland University of
Technology, city=Brisbane, postcode=4000, state=QLD, country=Australia

2] addressline=Centre for Data Science, Queensland University of Technology,
city=Brisbane, postcode=4000, state=QLD, country=Australia

3] addressline=School of Mechanical, Medical, and Process Engineering,
Queensland University of Technology, city=Brisbane, postcode=4000, state=QLD,
country=Australia

4] addressline= School of Computing, Clemson University, city=Clemson,
postcode=29631, state=SC, country=USA

5] addressline=School of Computing, Macquarie University, city=Sydney,
postcode=2109, state=NSW, country=Australia

[cor1]Corresponding author

# An Experimental Comparison of Transfer Learning against Self-supervised
Learning

Zehui Zhao<EMAIL_ADDRESS>Laith Alzubaidi<EMAIL_ADDRESS>Jinglan Zhang<EMAIL_ADDRESS>Ye Duan<EMAIL_ADDRESS>Usman Naseem
<EMAIL_ADDRESS>Yuantong Gu<EMAIL_ADDRESS>[  [  [  [  [

###### Abstract

Recently, transfer learning and self-supervised learning have gained
significant attention within the medical field due to their ability to
mitigate the challenges posed by limited data availability, improve model
generalisation, and reduce computational expenses. Leveraging pre-trained
features from analogous domains, these methodologies prove instrumental in
improving model performance and addressing data scarcity issues stemming from
the privacy concerns associated with patient data sharing and the exorbitant
costs involved in collecting medical test data. Consequently, transfer
learning and self-supervised learning hold immense potential for advancing
medical research. However, it is crucial to recognise that transfer learning
and self-supervised learning architectures exhibit distinct advantages and
limitations, manifesting variations in accuracy, training speed, and
robustness. Asserting the superiority of one method over the other across all
contexts is irresponsible, leaving researchers grappling with the challenge of
selecting the most fitting method and configuration for their specific
scenarios. This not only engenders confusion during method selection, but also
leads to inefficient utilisation of computational resources, potentially
compromising the reliability of experimental outcomes. To address this
dilemma, we carried out a comparative study between transfer learning and
self-supervised learning methods through experimentation. Two widely adopted
deep learning models, Xception and ResNet, were employed to extract features
under transfer learning and self-supervised learning settings respectively.
Subsequently, these pre-trained models underwent fine-tuning and testing using
both colourful (Kvasirv2, EyePacs) and grey-scale (BusI, chest CT) medical
datasets. The findings of this study revealed that transfer learning setting
demonstrated better performance on colourful datasets, achieving an accuracy
of 96.4% on the Kvasirv2 dataset and 92.1% on the EyePacs dataset. In
contrast, the self-supervised learning method has shown superior performance
on grey-scale datasets, achieving an accuracy of 90% and 97.22% on BusI and
chest CT datasets, both exceeding recent state-of-the-art models. Furthermore,
considering the data imbalance problem and domain mismatch issue that occurs
frequently in the medical field, we conducted experiments using two data-
altering techniques (data augmentation and down-sampling) and the double fine-
tuning technique combined with proposed pre-trained transfer learning and
self-supervised learning models to compare their efficiency in handling those
issues. The experimental result reveals that the data augmentation method can
improve the robustness of pre-trained models towards imbalanced data, while
the down-sampling methods have minor improvements. The double fine-tuning
technique has been proved to effectively reduce the domain mismatch problem
without using a large number of pre-training data. This comprehensive analysis
investigates the influence of colour information, dataset size, and dissimilar
transfer issues in the medical field, and uses explainable artificial
intelligence techniques to evaluate the reliability of the pre-trained models.
The insights gleaned from this research provide valuable guidance in selecting
the appropriate pre-training settings in various scenarios, with potential
applications extending beyond medical research.

###### keywords:

Transfer learning Self-supervised learning Double fine-tuning Imbalanced
datasets XAI Medical images

##  1 Introduction

Transfer Learning (TL) and Self-supervised Learning (SSL) methodologies strive
to enhance the performance of Convolutional Neural Networks (CNNs) in target
domains by capitalising on preexisting knowledge derived from analogous
domains. In recent years, these techniques have emerged as pivotal
contributors to advances in computer vision, particularly within the medical
domain. Their significance lies in mitigating data dependence during the
training of high-performance deep learning (DL) models, while simultaneously
reducing computational costs and training time. Traditional supervised DL
architectures, exemplified by models such as AlexNet (Krizhevsky et al., 2012)
and ResNet (He et al., 2016), are based heavily on abundant, well-labelled
data for feature acquisition and subsequent task support. In the medical
realm, where data annotation requires expert intervention, the scarcity and
expense of labelled data make the architectures employing TL and SSL methods
increasingly appealing. These methods not only elevate model performance and
alleviate overfitting risks, but also enhance model reusability. The success
of pre-trained methods is largely due to the advent of ImageNet (Russakovsky
et al., 2015), a massive publicly available dataset comprising more than 1.2
million images classified into 1,000 classes. ImageNet’s rich data repertoire
facilitates seamless knowledge transfer to diverse fields, notably
contributing to the effectiveness of pre-trained models.

In the medical domain, transfer learning has undergone extensive refinement,
producing various designs and improvements that increase its efficacy in
various diseases (Niu et al., 2020). In contrast, self-supervised learning, a
more recent entrant, has gained prominence by achieving state-of-the-art
supervised representation learning model performance without relying on
additional manual annotations. Recognising the practical importance of TL and
SSL in computer vision tasks, researchers have diligently benchmarked their
performance, seeking information for future architectural advancements
(Hosseinzadeh Taher et al., 2021). Recent investigations encompass diverse
aspects such as the suitability of different architectures, the relevance of
source and target domains, the exploration of various pretext tasks, analysis
of the impact of the size of the pre-training dataset, and evaluation of the
model capacity on the performance of downstream tasks (Neyshabur et al., 2020;
Ericsson et al., 2022b). Despite the strides made in TL and SSL studies,
emerging research suggests that the ubiquitous ImageNet may not be optimal for
the medical field (Raghu et al., 2019). The inherent dissimilarity between
natural images in ImageNet and medical images raises concerns about domain
mismatch adversely affecting downstream task training. A ’medical version
ImageNet’ is conspicuously absent, compelling researchers to turn to SSL
methods that do not need manual labels during pre-training. However, there is
still a notable gap in understanding how well these pre-training methods
perform in various scenarios.

Based on our recent study (Zhao et al., 2023), we observed that the majority
of recent TL methods still tend to operate within a supervised learning
framework, allowing models to acquire features from meticulously annotated
data. In contrast, SSL methods are performed under the unsupervised learning
setting by leveraging data augmentation techniques to create corresponding
pseudo-labels, eliminating the necessity for conventional human supervision
during training. This methodological contrast yields distinctive strengths and
weaknesses in data types, sizes, and domains. However, the limited
understanding of these two pre-trained methods has resulted in adverse effects
on model performance and robustness across varied environments. This gap
places an increased burden on researchers, necessitating a nuanced
understanding of both methods and amplifying time and computational costs in
numerous research endeavours.

Some researchers have begun to take note of this knowledge gap between TL and
SSL methods and have endeavoured to address this issue. Yang et al. (2020)
have conducted a comparative experiment of the two pre-training methods to
analyse the impact of domain similarity on the two pre-training methods,
across various source and target domains, covering daily objects, natural
environments, and even medical data. However, their research did not consider
the difference and influence of image colour information, target dataset size,
and data imbalance issues, which are important conditions in the training
process and have often faced challenges in the medical field. Moreover, they
did not consider the robustness of their model, since it is vital for models
to provide explainable and reliable answers instead of simply providing the
correct answers in the medical field. Azizi et al. (2023) elucidated the
benefits of the SSL method, noting its ability to achieve comparable
performance with TL models while using less than 10% of the target domain
data. However, their investigation primarily focused on the development of
sophisticated SSL architectures rather than undertaking a comprehensive
comparative analysis of the two methodologies.

To solve the current knowledge gap between TL and SSL methods and provide
supporting evidence to our previous work, this study aims to dive deeper into
the factors that influence the performance of TL and SSL methods and provide
guidance to researchers in choosing suitable pre-training settings. A
comparative experiment using both pre-trained methods was designed to identify
the effectiveness of these pre-training methods in multiple imaging modalities
and recognising potential limitations is of utmost importance. Meanwhile, we
utilised two data-altering methods including data augmentation and down-
sampling to solve the data imbalance problem that occurred in the target
datasets and employed the double fine-tuning technique to solve the potential
domain mismatch issues between source data and target medical data.
Furthermore, one Explainable Artificial Intelligence (XAI) technique was used
to analyse the reliability and robustness of two pre-trained models, providing
evidence for the pre-trained models in learning useful features. The knowledge
gained from this study will streamline the experimental setup phase, saving
valuable time and resources, and pave the way for future advancements in both
the TL and SSL methods. Thus, we would like to emphasise the contributions of
this study to knowledge:
1\. To the best of our knowledge, this is the first comparative evaluation
between TL and SSL methods that compare their strengths and weaknesses on
different colourful and grey-scale medical datasets.
2\. A detailed review of the pre-trained TL and SSL methods across varied
target dataset sizes revealed nuanced performance dynamics, demonstrating
better stability of the pre-trained SSL method towards small datasets compared
to the TL method.
3\. The XAI technique has been used in analysing the two pre-trained methods,
elucidating the superior reliability and robustness of the SSL method compared
to the TL method.
4\. An examination of two data-altering techniques coupled with pre-trained
methodologies has been conducted to address data imbalance issues, elucidate
the pronounced effectiveness and performance superiority of the data-
augmentation technique.
5\. A strategic application and evaluation of the double fine-tuning technique
within the proposed pre-trained models underscored its instrumental role in
augmenting the models’ performance and fortifying their robustness.
6\. The proposition of two pre-trained models, instantiated using TL and SSL
methods, produced results that exceeded current state-of-the-art CNN models on
various publicly available medical datasets.

This introduction section provides background and contributions of our
research; the subsequent segments of the paper are organised as follows: The
second section delves into the current applications and research of DL, TL,
and SSL methods in the medical field and reveals several common limitations
within pre-trained methods. The third section then presents the details of the
datasets used and outlines the methods proposed in this study, elucidating the
experimental process. The fourth section compiles the results of the
experiment and performs a comparative analysis of the performance of the two
pre-trained methods, demonstrating their effectiveness in diverse medical
datasets. Finally, the last section summarised the findings of this study,
providing a guideline for selecting appropriate pre-trained methods in various
medical application scenarios.

##  2 Literature Review

In this section, we briefly review the applications of deep learning, transfer
learning, and self-supervised learning in the field of medical imaging.
Additionally, we highlight recent constraints encountered in the application
of transfer learning and self-supervised learning within medical contexts.

###  2.1 Deep Learning for Medical Imaging Tasks

Deep learning has emerged as a highly promising technique for various
analytical tasks, encompassing classification, segmentation, and generation.
Within the medical field, the primary objective of DL models is to detect and
locate diseases quickly and efficiently, assisting physicians in rapid
diagnoses. As these models demonstrate considerable prowess in medical tasks,
an increasing number of DL models are proposed and implemented in this domain.
However, the performance of DL models in medical imaging tasks is
fundamentally dependent on the volume of the dataset, and data scarcity
remains a persistent challenge in the medical field (Wang, 2021). This
limitation comes from various reasons, such as privacy concerns, that prevent
medical agencies from disseminating patient records, which is the most
prevalent (Razzak et al., 2018). In addition, financial costs and the need for
professional knowledge to collect high-quality medical data with annotations
contribute to this scarcity. Consequently, data scarcity in the medical field
has limited the development of DL, and researchers have devised technical
solutions, such as data augmentation and pre-trained methods like TL or SSL,
to address this issue.

###  2.2 Transfer Learning for Medical Imaging Tasks

Transfer learning, also known as knowledge transfer, is a machine learning
(ML) technique that utilises the knowledge acquired from one task to improve
performance on a related task. The objective of the TL method is to enrich the
learnt feature of the pre-trained model and reduce its data dependence (Pan &
Yang, 2010). With the proliferation of deep learning, transfer learning has
become integral to various applications, particularly in the medical field.
Numerous studies have explored the factors that contribute to the success of
transfer learning in CNNs. These factors include choosing an optimal deep
learning architecture for guaranteed model performance (Kornblith et al.,
2019), choosing source domains that share a close distribution to the target
domain to accelerate better transfer performance (Zhang et al., 2022), and
increasing the size of the source datasets and pre-trained model parameters
for richer knowledge transfer (Kolesnikov et al., 2020). In particular, a
central drawback of TL pre-training is the requirement for a large quantity of
costly manually supervised training (Huh et al., 2016), which requires a
substantial pretext dataset such as ImageNet for successful pre-trained
learning. To address this issue, researchers also designed fully unsupervised
TL architectures to avoid requiring additional labels, but the performance has
turned out to be too weak. The prevailing standard in transfer learning
architectures in medical imaging involves utilising ImageNet as a source
domain and fine-tuning it with medical-related target datasets. Despite the
positive results reported with pre-trained ImageNet models on colourful target
datasets (Morid et al., 2021; Patil & Sharma, 2024), questions have arisen
regarding the suitability of ImageNet for the grey-scale medical datasets
(Raghu et al., 2019). Evidence suggests that the transfer learning method does
not consistently produce favourable results in the medical domain (Alzubaidi
et al., 2020), as ImageNet representations do not align well with those of
medical images, particularly grey-scale ones.

###  2.3 Self-supervised Learning for Medical Imaging Tasks

Self-supervised learning is a recently appeared unsupervised pre-training
method that employs model-augmented data as pseudo-labels to provide
supervision signals during the pre-training process. Compared to the transfer
learning method, self-supervised learning appears particularly attractive in
the medical field, offering an effective solution to use large amounts of
unlabelled data (Shurrab & Duwairi, 2022). Notably, the self-supervised
learning model exhibits improved accuracy performance by incorporating more
complex auxiliary tasks and leveraging model-generated pseudo-labels to
facilitate the learning of powerful features from unlabelled data. Some
studies have indicated that self-supervised learning models, particularly
those employing contrastive methods, tend to learn invariant features by
maximising similarities between raw input data and self-generated pseudo
labels (Ericsson et al., 2022a). This suggests that contrastive self-
supervised learning methods are inclined to overlook changeable feature
information, resulting in increased stability when confronted with disease
images that have unaltered features. Despite certain self-supervised learning
studies that have utilised ImageNet with self-supervised pre-training methods
for medical imaging tasks and achieved positive results (Truong et al., 2021),
there remains a lack of conclusive evidence to ascertain whether pre-trained
models with self-supervised learning methods can consistently outperform
transfer learning methods in medical imaging tasks and potentially become a
new standard in this domain.

###  2.4 Recent Limitations of Transfer Learning and Self-supervised Learning
for Medical Imaging Tasks

The success of the TL and SSL methods in the medical domain is undeniable;
however, several challenges impede their further advancement and the full
realisation of their potential. Table 1 encapsulates some of these
limitations, and our study aims to dissect and address these challenges,
offering insights to guide future researchers in leveraging pre-trained
methods effectively.

Table 1: Limitations of recent pre-trained methods in the medical field.

Author Methods Descriptions Del Pup & Atzori (2023) Self-Supervised Learning
SSL lacks comparison with supervised learning methods when the amount of
supervision that can be provided for the downstream task is sufficiently high.
Wu et al. (2023) Self-Supervised Learning Some transfer learning studies just
treated SSL pretrain- -ing method as an alternative scheme to improve the
performance of downstream tasks. Zhao et al. (2023) TL & SSL Lacks specific
experiments in comparison of TL and SSL methods’ performance,
generalisability, and robustness. Limitation: Lacks systematic comparison
between pre-trained methods Romero et al. (2020) Transfer Learning Pre-trained
TL model with ImageNet weight does not bring advantages to X-ray based
classification tasks. Alzubaidi et al. (2021) Transfer Learning Pre-trained
features from ImageNet mismatch medical images. Atasever et al. (2023)
Transfer Learning Transferring information obtained from natural images are
less successful than lighter methods trained from scratch. Tan et al. (2024)
Self-Supervised Learning Significant differences exist between COVID-19
medical images and pre-trained natural samples, and need more information to
guide the reconstruction of target images. Limitation: Domain discrepancy
Muljo et al. (2023) Transfer Learning Pre-trained TL model only have limited
positive impacts when facing an imbalanced medical dataset. Zhang et al.
(2023) Self-Supervised Learning Data bias in both pre-training and target data
affects representation learning, leading to poor target performance.
Limitation: Imbalanced datasets Qayyum et al. (2020) Transfer Learning The
learning process of models is complex and fails to provide the knowledge
needed. explanations for learning behaviour and prediction-making process.
Schiappa et al. (2023) Self-Supervised Learning The lack of explainability has
prevented the understanding of pretext and downstream task learning process,
and constrained further improvement of the SSL model. Limitation: Lack of
additional interpretability

In summary, four overarching limitations hinder the progress of pre-trained
models. These encompass the lack of comprehension and comparison between pre-
trained methods, risks associated with domain discrepancy and imbalanced
datasets, and the dearth of interpretability of the model. Specifically, the
limited understanding and comparability between pre-trained methods stem
primarily from their architectural level. Despite the fact that TL and SSL
methods are both rooted in DL architectures, there has been a longstanding
tendency to perceive them as interchangeable. This overlooks their divergent
performance, generalisability across various data types and sizes, and
potential representation nuances. Moreover, the opaque nature of DL approaches
compounds the challenge of understanding how models extract salient features
and make predictions and is exacerbated by the intricate multi-staged training
process inherent in pre-trained methods, while many researchers also fail to
realise the importance of interpretability. However, raising the understanding
of the model’s prediction is particularly pertinent in the medical realm,
where diagnostic decisions must be rigorously accountable.

Conversely, domain mismatch and data imbalance issues manifest at the data
level. Domain mismatch arises when the representation and distribution of
domain data diverge from the target data, which is a common concern in pre-
trained methods but exacerbated in medical applications because of the
dissimilarity between publicly available pre-training datasets (e.g.,
ImageNet) and medical contexts. Also, data imbalance poses a significant
challenge, especially in medical scenarios where target diseases are
infrequent and difficult to accumulate in sample sizes equivalent to normal
images. These issues have constrained the quality of the features extracted
from the pre-training learning process, further leading to inefficient
resource utilisation and limited model performance.

Despite the strides made by pre-trained methods in the medical domain, urgent
attention is warranted to surmount these challenges and propel their further
development, capitalising on their potential to revolutionise medical
applications.

##  3 Materials and Method

###  3.1 Datasets

This section gives a detailed overview of the source and target data used in
this study. As aforementioned, the TL and SSL methods have shown different
performances towards colourful and grey-scale datasets. The data utilised in
this study were also classified into grey-scale and colourful images based on
the number of colour channels they possess and the imaging technique used,
aiming to compare the strengths and weaknesses of the two pre-trained methods.
Table 2 below lists the size and categories of the datasets. In summary, three
grey-scale datasets (VinDr-CXR, OASIS MRI, and CT Kidney) were used to pre-
train the corresponding greyscale-related downstream tasks, while two
colourful datasets (ODIR and ISIC 2019) were selected for training colourful
pretext tasks. In addition to the source domain datasets, we selected several
publicly available target datasets from different types of commonly researched
diseases to ensure generalisability and convenience comparison with the state-
of-the-art models. To be specific, two grey-scale datasets (BusI and Lung
cancer CT) and two colourful datasets (KvasirV2 and EyePacs) were chosen as
the target datasets to assess the performance of pre-trained TL and SSL
models. Notably, only the two target grey-scale datasets here are imbalanced
datasets, which have a sample size bias within the class.

Table 2: Details of Datasets.

Datasets Image Count Image Category Grey-scale Image Datasets Source Domain
Datasets VinDr-CXR Nguyen et al. (2022) 18,000 X-ray CT Kidney Islam et al.
(2022) 12,446 CT OASIS MRI Marcus et al. (2007) 80,000 MRI Total number
110,446 Grey-scale Target Domain Datasets BusI Al-Dhabyani et al. (2020) 780
Ultrasound Lung Cancer CT Hany (2020) 900 CT Colourful Image Datasets Source
Domain Datasets ODIR-2019 14,400 Ophthalmic ISIC 2019 Combalia et al. (2019)
25,331 Dermoscopic Total number 39,731 Colourful Target Domain Datasets
Kvasirv2 Pogorelov et al. (2017) 8000 Gastrointestinal tract EyePacs Kiefer et
al. (2023) 6000 Fundus

####  3.1.1 Grey-scale Image Dataset

Five grey-scale medical datasets originating from diverse modalities and
disease backgrounds fulfil the purpose of pre-training and fine-tuning, with
the first three datasets used for pre-training, and the last two datasets
serving as target datasets for fine-tuning.
1\. VinDr-CXR (Nguyen et al., 2022) is an open dataset comprising 18,000 chest
X-ray images meticulously collected from two major hospitals in Vietnam.
Radiologists manually annotated all images, providing 22 local labels
delineating abnormalities and 6 global labels indicating suspected diseases.
The dataset encompasses diseases such as lung tumours, pneumonia,
tuberculosis, and chronic obstructive pulmonary disease.
2\. CT Kidney (Islam et al., 2022) is a comprehensive CT dataset designed to
facilitate automatic kidney tumour diagnosis. It contains a total of 12,446 CT
whole abdomen and urinary images, focusing on three primary renal disease
categories: kidney stones, cysts, and tumours. Notably, all classes exhibit a
consistent mean colour distribution.
3\. OASIS MRI is a widely used, publicly available dataset Marcus et al.
(2007) gaining popularity as a resource for in-depth analysis of brain MRI
images using deep neural networks. This dataset comprises 80,000 brain MRI
images categorized into four distinct classes representing different stages of
Alzheimer’s disease, primarily supporting tasks related to Alzheimer’s disease
detection.
4\. BusI (Al-Dhabyani et al., 2020) is a breast ultrasound image dataset for
grey-scale breast cancer detection images. The dataset consists of 780 images
that are separated into three categories: Benign, Malignant, and Normal, which
comprise 210 malignant images, 437 benign images, and 133 normal images along
with 433 masked ground truth images that segment the specific disease areas.
In this study, we have excluded all the masked images and focused on those
real-world image samples.
5\. Lung Cancer CT Dataset (Hany, 2020) is a publicly available dataset
collected from the Kaggle website, comprising 1000 grey-scaled CT images
belonging to four classes, representing three chest cancer types and a normal
type: adenocarcinoma, large cell carcinoma, squamous cell carcinoma, and
normal cell. The adenocarcinoma class possess 326 image samples, the large
cell carcinoma class holds 163 image samples, the normal class has 159 image
samples, and the squamous cell carcinoma class reserved 252 image samples.

####  3.1.2 Colourful Image Dataset

At the same time, four colourful medical datasets have been chosen to support
the comparison experiment, with the initial two datasets serving as pre-
training sets and the subsequent two datasets employed as target datasets.
1\. ODIR-2019 (Peking University International Competition on Ocular Disease
Intelligent Recognition) stands as an extensive ophthalmic database
encompassing 14,400 vividly coloured images captured from both the left and
right eyes. Accompanied by diagnostic keywords provided by medical
professionals, the dataset spans eight distinct classes, effectively
characterising various diseases and conditions depicted in the images.
2\. The ISIC 2019 dataset (Combalia et al., 2019) originated from the ISIC
2019 challenge and comprises 25,331 vibrant dermoscopic images. This dataset
encapsulates eight different diseases, including melanoma, melanocytic nevus,
basal cell carcinoma, actinic keratosis, benign keratosis, dermatofibroma,
vascular lesion, and squamous cell carcinoma. Each image within the dataset is
accompanied by its corresponding label, furnishing ground-truth information
for comprehensive analysis and training purposes.
3\. KvasirV2 dataset (Pogorelov et al., 2017) is a dataset containing 8000
images from inside the gastrointestinal (GI) tract. The data distribution of
this dataset is balanced and can be categorised into 8 classes related to
anatomical landmarks and endoscopic polyp removal, listed as ’dyed-lifted-
polyps’, ’dyed-resection-margins’, ’esophagitis’, ’normal-cecum’, ’normal-
pylorus’, ’normal-z-line’, ’polyps’, and ’ulc-erative-colitis’. Each class
consists of 1000 sample images and all of the images have been annotated and
sorted by experienced endoscopists.
4\. EyePacs dataset (Kiefer et al., 2023) is a generic glaucoma dataset
constituted of 6000 fundus images, splitting into two classes of referable
glaucoma (RG) and non-referable glaucoma (NRG). The data samples in each class
are balanced and have been resized to 256x256 pixels for quick employment to
machine learning algorithms and all the data from training and testing
directories have been annotated by experts.

###  3.2 Proposed Methodology

The objective of this experiment is to compare the effectiveness of TL and SSL
methods in the medical field and provide evidence for their strengths and
weaknesses. Considering the huge time consumption in pre-training models and
the limitation in computational resources, we selected two representative DL
models including Xception and ResNet to pre-train with TL and SSL methods and
compare their performance, stability, and reliability. The workflow diagram
for the pre-training stage is shown in Fig. 3 and the workflow diagram for the
downstream stage is presented in Fig. 5. The training options are shown in
Table 3.

Table 3: CNN model training options. Methods | Function | Value
---|---|---
TL & SSL | Execution environment | GPU
SSL | Pretraining epochs | 50
TL & SSL | Retraining epochs | 50
TL | Fine-tuning epochs | 30
TL & SSL | Batch size | 64
TL & SSL | Pretrain& Retrain learning rate | 1e-03
TL | Fine-tuning learning rate | 1e-05
TL & SSL | Shuffle | True
TL & SSL | Optimiser | Adam
TL & SSL | Learning rate schedule | Step-wise

For the TL setting, we selected the Xception architecture (Chollet, 2017) as
our base model, which is a widely used DL architecture known for its high
performance in various tasks. The Xception module is constructed as a series
of three flow structures, each incorporating batch normalisation (BN),
rectified linear units (ReLU) activation functions, and separable
convolutional kernels in depth. Each module in the Xception model operates
through sequential processing: first through the entry flow, then via the
middle flow, and ultimately undergoing further processing in the exit flow.
Within the Xception architecture, its 36 convolutional layers are organised
into 14 modules. In particular, all of these modules feature linear residual
connections, except for the initial and terminal ones. Fig. 1 shows the whole
Xception architecture as follows. Compared to other DL architectures, the
Xception model has shown advanced performance with a reduced time cost during
the training process and has been recorded as the state-of-the-art model in
many medical-related tasks (Kassani et al., 2019; Liu et al., 2022).

Figure 1: Xception module.

For the SSL setting, we employed ResNet50 architecture (He et al., 2016) since
it is also a well-performed model and has been tested with numerous
applications. ResNet50 is a 50-layer convolutional neural network (48
convolutional layers, one max pool layer, and one average pool layer). Each
convolutional layer uses a bottleneck design for the building block which
consists of two 1x1 convolutions as the bottleneck to process the input and
output features, reducing the number of parameters and matrix multiplications.
The identity mapping used in ResNet allows the model to bypass a CNN weight
layer if the current layer is not necessary, which also helps avoid
overfitting problems to the training set. The ResNet50 architecture is shown
in the following Fig. 2.

Figure 2: ResNet-50 module.

Both pre-trained methodologies incorporated the Adam optimiser, commencing
with an initial learning rate of 0.001 and undergoing 50 epochs during both
the pre-training and re-training phases. To enhance the robustness of the
learning process and introduce variability into the model, the input data were
shuffled to provide randomised samples. Furthermore, a stepwise learning rate
schedule was used for the optimiser, implemented to mitigate overfitting,
involving a gamma value of 0.2 to decrease the learning rate after 40 epochs.
It is noteworthy that, in the TL configuration, an additional fine-tuning step
was introduced, involving the unfreezing of the entire model’s layers to
extract more informative features. In this context, the Adam optimiser began
with a learning rate of 0.00001 to prevent rapid overfitting of the entire
model. This specific fine-tuning step was omitted in the SSL setting due to
the concatenated model’s extensive layer count, coupled with the limited size
of the target datasets, making it impractical to tune all model parameters and
potentially leading to overfitting.

####  3.2.1 Pretext Pretraining Design

For the pre-training stage, we followed a workflow as shown in Fig. 3. The
proposed methods start from processing input data and follow their
corresponding pre-training setting to get useful pre-trained knowledge before
the following retraining and fine-tuning steps. The detailed steps for the two
pre-trained methods are presented in Table 4.

Table 4: Pretext training steps.

TL setting SSL setting \- Step 1: Loading empty Xception model. Loading the
source domain images. \- Step 2: Loading pretrained ImageNet weight. Image
pre-processing. \- Step 3: Removing original FC layers. Loading two empty
ResNet50 model. \- Step 4: Adding new FC layers. Concatenate two ResNet50
models. \- Step 5: Loading pretrained ImageNet weight to the first ResNet50
model. \- Step 6: Freeze the first ResNet model layers. \- Step 7: Adding a
new projection head. \- Step 8: Training the whole model.

Figure 3: Pretext training workflow diagram.

In the TL setting, we used the Keras library to deploy the Xception model, as
it is a high-level API that provides ML and DL models for convenient
employment. Additionally, we used the pre-trained ImageNet weight provided by
Keras for our Xception model because the ImageNet dataset is one of the
representative pre-training datasets and the first choice when applying the TL
method. The Xception network import from the Keras library is composed of
entry flow, middle flow, exit flow, separable convolution in-depth, and Fully
Connected (FC) layers. We also removed the original fully connected layers
since it is designed for ImageNet classification tasks and is not useful for
our following downstream tasks. Five additional dense layers and one dropout
layer were then used to build a custom projection head to prepare for the
following training of the downstream task. The dense layers facilitated linear
mapping from input to output, incorporating a ReLU activation function to
enhance computational efficiency and effectiveness. The dropout layer was
designed to randomly set a fraction of input units to zero during forward and
backward passes, mitigating overfitting, particularly vital when working with
small datasets or complex models. Notably, the final dense layer was combined
with a softmax layer, which produced predictions by assigning the class with
the highest probability as the model’s output. By employing these additional
fully connected layers, we aim to provide better network performance and
reduce overfitting risks.

For the SSL setting, the method we choose to pre-train the model is the Simple
Framework for Contrastive Learning of Visual Representations (SimCLR) method
(Chen et al., 2020). At the same time, Chen and his colleagues have used
ResNet50 as the base model. In our study, the base model is a concatenated
model that is combined with two ResNet50 networks. The reason for using a
hybrid model is that the selected Xception model for the TL setting has a
higher Top-1 and Top-5 accuracy performance than the normal ResNet-50 model on
the ImageNet classification task, which states that they might have a
performance advantage at the architecture level. We wanted to minimise the
performance difference due to architecture itself, since the main goal of this
study is to compare the effectiveness of two pre-trained methods. The two
ResNet networks were concatenated using two resize layers, as the first ResNet
is a pre-trained model using ImageNet weights and the second ResNet is an
empty network without any pre-trained weight. The first ResNet has been frozen
during the training process and will be treated as a feature extractor. The
reason for utilising a network as the feature extractor is to provide well-
extracted features as input data for the following network and improve the
performance of the whole architecture (Kim et al., 2022). The source data are
preprocessed by resizing the image size into 224 pixels by 224 pixels. Later,
the whole model was trained with source medical data to gain powerful features
from self-contrastive learning. In SimCLR setup, the image pairs are generated
internally by the model itself utilising data augmentation techniques
including resizing, flipping, zooming, rotating, and brightness and hue
changing, serving as pseudo labels for the training data and supporting
contrastive loss calculating. Fig. 4 gives a sample of the pseudo-labelled
generated.

Figure 4: Self-generated pseudo label sample.

The SimCLR method utilised a loss function called NT-Xent to calculate the
training loss by contrasting the representations of positive pairs (similar
images) and concurrently creating negative pairs (dissimilar images). The
equation of the loss function is as follows.

$l_{i,j}=-\log\frac{exp(sim(z_{i},z_{j})/\tau)}{\sum^{2N}_{k=1}1[k\neq
i]exp(sim(z_{i},z_{k})/\tau)}$

The positive pair of example (i,j) uses $1[k\neq i]$ as an indicator function
to evaluate if $k\neq i$, and $\tau$ denotes a temperature parameter while the
final loss is computed by summing all positive pairs and divide by 2 x N =
views x batch-size. The loss function encourages the model to bring positive
pairs closer within the embedding space, while pushing negative pairs further
apart, thereby enhancing the learning from input representations. Following
the SimcLR method, a projection head containing three linear layers was added
to the top of the base ResNet to be trained to maximise agreement using
contrast loss. Moreover, we trained two pre-trained SSL models using the grey-
scale and colourful medical image datasets, respectively, for the different
target domains to reduce the risks of domain mismatch and compare the
effectiveness by using ImageNet weight directly. Notably, we used image data
from different modalities to enlarge the size of the pre-training data,
providing the network with more disease characteristics to learn powerful
features.

####  3.2.2 Downstream training Design

Figure 5: Downstream training workflow diagram.

The downstream training process, depicted in Fig. 5, adheres to a systematic
workflow. Initially, the data from the target domain were divided into
training (80%) and testing (20%) subsets. Subsequently, all data were
preprocessed to align with the input specifications of the respective model.
The input data has an additional data-altering step using the down-sampling or
augmentation technique before the fine-tuning step to address the data
imbalance issue. Table 5 lists the working steps for each pre-trained method
setting.

Table 5: Downstream training steps.

TL setting SSL setting \- Step 1: Loading the target images. Loading the
target images. \- Step 2: Loading the pre-trained model. Loading the
pretrained model. \- Step 3: Data split. Data split. \- Step 4: Data
preprocessing. Data preprocessing. \- Step 5: Data downsampling or
augmentation. Data downsampling or augmentation. \- Step 6: Model retraining &
fine-tuning. Model retraining. \- Step 7: Model testing. Model testing. \-
Step 8: Class activation map building. Class activation map building.

Data preprocessing

To facilitate subsequent classification tasks, rigorous data pre-processing
was conducted to meet the model’s input requirements. The input data were
transformed into four dimensions, encompassing batch size, colour channels,
height, and width. The batch size determines the number of training examples
processed in each iteration, which have been set to 64 in the experiment to
prevent memory overflow. The colour channels represent the colour information
that the model holds and are based on three base colours ‘Red, Green, and
Blue. For grey-scale images, which only possess a single colour channel, two
pseudo channels were introduced by replicating the original channel.
Additionally, height and width dimensions were standardised to 224 pixels by
224 pixels to maintain the consistency of the image. Furthermore, the OneHot
encoding technique was applied to convert all labels into vector
representations based on their respective categories for the convenience of
the following evaluation calculations.

Data down-sampling and augmentation

Data imbalance is a prevalent challenge in the medical domain, manifested when
certain classes within the training set exhibit a significantly higher number
of examples than others. This occurrence stems from the varying frequencies of
different diseases, making it impractical to collect data evenly across all
classes in experiments. Previous studies have indicated that such a class
imbalance negatively affects the learning process and can decrease the
performance of CNN models (Buda et al., 2018). In our investigation, we
identified a substantial data imbalance in the BusI and ChestCT datasets. In
particular, BusI’s first-class benign featured 437 images, twice the count of
the second-class malignant (210 images), and three times that of the third-
class normal (133 images). To address the imbalance in our target dataset, we
implemented two data-altering techniques: data down-sampling and data
augmentation.

Data down-sampling operates at the data level, randomly removing examples from
majority classes until all classes share an equal number of instances
(Haixiang et al., 2017). This method aims to create a balance between classes
by ensuring that each class has an identical number of examples. In our study,
we randomly removed samples from target datasets with oversized classes,
equalising the sample count for each class. The sample number for each target
dataset down-sampled is listed in Table 6.

Table 6: Downsampled target datasets size.

Datasets Methods Categories Samples BusI Down-sampling Train 108 images each
class BusI Down-sampling Test 25 images each class Total: 399 images Chest CT
Down-sampling Train 127 images each class Chest CT Down-sampling Test 32
images each class Total: 636 images Kvasirv2 Down-sampling Train 100 images
each class Kvasirv2 Down-sampling Test 25 images each class Total: 1000 images
EyePacs Down-sampling Train 500 images each class EyePacs Down-sampling Test
100 images each class Total: 1200 images

Similarly, data augmentation seeks to balance the number of samples in each
class by augmenting the minority classes. This method involves randomly
selecting samples and replicating them, proven to enhance feature learning and
improve model generalisability (Shorten & Khoshgoftaar, 2019). For this
experiment, rotation, flip, zoom, and blur methods were applied to the
training data, guaranteeing a balanced representation for each class. Table 7
details the augmentation functions and values employed. The rotation function
rotates the input images with an angle from -15 to 15 degrees, while the flip
function reflects the input images horizontally on the X-axis. The zoom
function transforms the input images into 80% of the original size, and the
blur function uses the Gaussian function to give an average weight to all
pixels in a kernel of 9x9. The original images and the augmented samples are
illustrated in the following Fig. 6 to provide a visual representation of the
transformation process. Specifically, Table 8 furnishes the post-augmentation
size of each class within the target datasets. In the case of the BusI
dataset, samples within the malignant class underwent a single round of
augmentation utilizing the mixed function, while samples in the normal class
underwent three rounds of augmentation through rotation, blur, and mixed
functions to align with the sizes of other classes. The chest CT dataset,
characterised by an adenocarcinoma class with a surplus of samples compared to
other classes, underwent mixed augmentation functions for the remaining three
classes (large.cell.carcinoma, normal, and squamous.cell.carcinoma) to
harmonise their sizes. For the balanced Kvasirv2 and EyePacs datasets, each of
their classes was subjected to one round of mixed augmentation to maintain the
equilibrium of the dataset.

Table 7: Augmentation functions.

Function Value Action Rotation [-15 to 15] Rotating input data by -15 to 15
degrees Flip 1 Randomly flip input data horizontally and vertically Zoom 0.8
Zoom in by 0.8 distance from the middle Blur [9,9] Blur the input data with a
9x9 pixel kernel Mixed - Rotating and flipping input data at the same time

Table 8: Augmentated target datasets size.

Datasets Methods Categories Samples BusI Augmentation Train 397 images in
Benign class BusI Augmentation Train 340 images in Malignant class BusI
Augmentation Train 372 images in Normal class BusI Augmentation Test 40 images
each class Total: 1229 images Chest CT Augmentation Train 281 images in
Adenocarcinoma class Chest CT Augmentation Train 236 images in
LargeCellCarcinoma class Chest CT Augmentation Train 228 images in Normal
class Chest CT Augmentation Train 207 images in SquamousCellCarcinoma class
Chest CT Augmentation Test 45 images each class Total: 1132 images Kvasirv2
Augmentation Train 1600 images each class Kvasirv2 Augmentation Test 200
images each class Total: 14400 images EyePacs Augmentation Train 5000 images
each class EyePacs Augmentation Test 500 images each class Total: 11000 images

(a) Original image

(b) Rotated image

(c) Flipped image

(d) Zoomed image

(e) Blurred image

(f) Mixed image

Figure 6: The augmented image sample.

Although these data-altering methods have been proven to be effective in
addressing data imbalance, some concerns should be paid attention. Data down-
sampling risks the loss of valuable information and exacerbates the data
scarcity problem, particularly given the relatively small sizes of our target
datasets. Data augmentation, on the other hand, poses a risk of overfitting
due to the repetition of samples, potentially capturing certain features
excessively over time. To comprehensively evaluate the impact of these data-
altering methods across two pre-trained methods, we extended our experiment to
apply data down-sampling and augmentation universally, even in the dataset
absent of apparent data imbalance issues.

Model retraining and fine-tuning

In the retraining phase, the initial step involved retraining select layers,
particularly the fully connected layers, to leverage knowledge from pre-
trained parameters and learn from the new target data. The majority of the
model’s layers were kept frozen, with only the top layers and the fully
connected layer adapting to the target data. Balancing the retraining of
layers was crucial, as an excessive number of retrained layers might
compromise the model’s generalisation and reliability. Various settings of the
number of unfreezing layers were tested to strike a balance. This methodology
ensured the preservation of most transferred knowledge while allowing a few
layers to adapt to unique features from input images, assessing the
effectiveness of pre-trained weights from TL and SSL settings. Additionally,
the Adam optimiser and cross-entropy loss function were uniformly applied to
both models, ensuring comparability and alignment with nonbinary downstream
tasks.

During the fine-tuning phase, we systematically unfroze all layers in the
chosen model and fine-tuned the entire model with a conservative learning rate
for an additional 30 epochs. After each epoch, the model assessed its training
accuracy and loss, dynamically adjusting the learning rate if the training
loss plateaued for three consecutive epochs. This iterative process was
designed to meticulously optimise the model’s parameters throughout all
layers, enhancing the alignment of learnt features with the characteristics of
the target domain.

Model testing

After the retraining and fine-tuning stages, the pre-trained model acquired
substantial knowledge for subsequent downstream tasks. In the testing phase,
we used ground-truth images extracted from the target dataset to evaluate the
performance of the trained models. Given the focus of this study on medical
classification tasks, each test image underwent model predictions for their
respective categories. The accuracy, loss, sensitivity, precision, and F1
score metrics were computed for each prediction, serving as key indicators for
the subsequent analysis.

Class activation heat-map development

The final step involves the generation of feature visualisation heat-maps,
elucidating the decision-making process of the trained model, and assessing
its reliability. Feature visualisation is valuable for uncovering and
understanding the learnt features within DL models. Zhou et al. (2016)
introduced the Class Activation Mapping (CAM) method, which incorporates a
global average pooling layer into a standard CNN. This innovation facilitated
the identification of critical feature contributions linked to CNN’s specific
predictions, shedding light on the rationale behind the model’s decisions.
Furthermore, Selvaraju et al. (2017) introduced Gradient Weighted Class
Activation Mapping (Grad-CAM), using gradients from the network’s final
convolutional layer to generate a coarse localisation map. This map highlights
influential regions within an image that contribute to the prediction of
specific concepts or classes. In our study, we embraced the Grad-CAM technique
on the model’s final layer to generate class-specific heat-maps. The objective
is to unveil evidence of how the models arrive at predictions, with the heat-
maps delineating regions of significance in the model’s prediction rationale.

##  4 Experimental Evaluation

This section evaluates the test results of the fine-tuned model on the target
domain. The proposed methods are run in Python 3.10.12 on a Windows 11 pro
system, under a desktop PC equipped with an Intel (R) Core (TM) i5-12600K CPU
at 4.8 GHz, an NVIDIA Geforce RTX 4080 GPU, and 64 GB of RAM. In this study,
one TL model was downloaded and utilised pre-trained ImageNet weight from the
Keras library and two SSL models were pre-trained on two groups of colourful
and grey-scale medical image datasets.

###  4.1 Evaluation Metrics

The statistical metrics for evaluating classification performance can be
various, and we have used some of the most common methods including Accuracy,
Loss, Sensitivity (Recall), Precision and F1 score. The calculation equations
of these performance measurements are shown as follows. The TP here means true
positive, TN is true negative, and they represent the number of correctly
classified negative and positive instances, respectively. The FP is false
positive, and FN stands for false negative, which denotes the number of
misclassified positive and negative cases, respectively.

$Accuracy=\frac{TN+TP}{TN+FN+TP+FP}$

$CrossEntropy-Loss=-\sum_{c=1}^{M}y_{o,c}\log(p_{o,c})$

$Sensitivity=Recall=\frac{TP}{TP+FN}$

$Precision=\frac{TP}{TP+FP}$

$F1=2*\frac{Precision*Recall}{Precision+Recall}$

###  4.2 Experimental assessment

This section intricately elucidates the performance dynamics exhibited by the
proposed pre-trained CNN models, namely Xception and concatenated ResNet,
across both the training and testing phases. The comprehensive evaluation
entails a meticulous comparison of the two pre-trained models, encompassing
their performance across various training phases, multiple fine-tuning steps,
combined with data down-sampling, augmentation and double fine-tuning
techniques, juxtaposition with state-of-the-art models operating on the same
target datasets, and scrutiny under the lens of XAI techniques. Within the
scope of these comparative analyses, salient metrics such as performance
metrics, generalisability, temporal considerations, and robustness were
systematically scrutinised. This rigorous examination aims to present a
nuanced understanding of the inherent strengths and weaknesses encapsulated
within the two pre-trained methods.

Comparison in training phase assessment

Table 9: Results of TL model on training set.

Datasets Acc(%) Loss Sen(%) Pre(%) F1(%) BusI 84.23 0.41 74.69 82.00 77.02
BusI (Augmented) 96.94 0.09 97.72 97.63 97.67 BusI (Down-sampled) 63.85 0.84
56.78 64.13 55.05 Chest CT 96.72 0.10 95.58 95.63 95.44 Chest CT (Augmented)
98.42 0.07 98.83 98.88 98.86 Chest CT (Down-sampled) 88.30 0.48 88.28 88.14
87.92 Kvasirv2 95.17 0.13 96.71 96.77 96.70 Kvasirv2 (Augmented) 99.93 0.08
99.99 99.99 99.99 Kvasirv2 (Down-sampled) 100 0.01 100 100 100 EyePacs 99.78
0.01 99.72 99.71 99.71 EyePacs (Augmented) 99.95 0.01 99.96 99.94 99.94
EyePacs (Down-sampled) 99.36 0.01 100 99.50 99.50

Table 10: Results of SSL model on training set.

Datasets Acc(%) Loss Sen(%) Pre(%) F1(%) BusI 98.29 0.57 90.28 91.89 83.59
BusI (Augmented) 99.00 0.56 98.97 99.05 99.00 BusI (Down-sampled) 99.21 0.56
91.60 91.68 92.25 Chest CT 99.83 0.74 99.83 99.78 99.80 Chest CT (Augmented)
99.58 0.74 96.75 97.08 96.88 Chest CT (Down-sampled) 99.80 0.74 95.74 95.90
95.84 Kvasirv2 93.76 1.33 85.00 84.86 84.96 Kvasirv2 (Augmented) 91.00 1.36
84.35 84.41 84.30 Kvasirv2 (Down-sampled) 99.39 1.28 91.98 92.13 91.79 EyePacs
98.18 0.33 92.12 92.10 92.10 EyePacs (Augmented) 98.43 0.32 93.00 93.03 93.02
EyePacs (Down-sampled) 98.53 0.32 93.10 93.13 93.10

The performance of the training stage for each model is presented and
tabulated in Tables 9 and 10. When faced with grey-scale datasets, the TL
model achieved an accuracy of 84.23% and 96.72% on the base BusI and Chest CT
datasets, respectively, which is relatively lower than the performance of the
SSL model at 98. 29% and 99. 83%. The SSL model exhibited higher sensitivity,
precision, and F1 scores on BusI (90.28%, 91.89%, and 83.59%) and Chest CT
(99.83%, 99.78%, and 99.8%), showcasing a 20% improvement over the TL model on
these two datasets. Additionally, the performance of the TL model in the down-
sampled BusI and chest CT datasets experienced a significant drop from 84. 23%
to 63. 85% and from 96. 72% to 88. 3%, respectively. In contrast, the SSL
model maintained stable results even with reduced training samples, achieving
training accuracies of 99.21% and 99.8% on down-sampled datasets, without any
reduction from the base datasets. Notably, the augmentation method salvaged
the TL model from this unfavourable situation, significantly improving
accuracy from 84.23% to 96.94% and 96.72% to 98.42%, aligning sensitivity,
precision, and F1 scores with the SSL model. At the same time, the performance
of the SSL model on augmented datasets remained high at 99% and 99.58%,
maintaining parity with the base dataset.

Conversely, the situation reversed in colourful datasets. The TL model
demonstrated commendable performance, achieving 95.17% accuracy, 96.71%
sensitivity, 96.77% precision, and 96.7% F1 score on the base KvasirV2
dataset. In comparison, the SSL model exhibited lower accuracy at 93.76% and
poorer performance in sensitivity (85%), precision (84.86%), and F1 score
(84.96%). Surprisingly, the TL model achieved perfect scores of 100% accuracy,
sensitivity, precision, and F1 score with the down-sampled dataset, while the
SSL model showed a consistent trend in each evaluation metric. This scenario
is attributed to the larger scale of the KvasirV2 dataset compared to grey-
scale datasets, requiring more time for the model to assimilate features and
can not reach a suitable gradient within 50 epochs. Thus, the down-sampling
method expedited the model’s adaptation to target representations by reducing
the total dataset size. Furthermore, the performance of the SSL model in the
augmented dataset supported this assumption, with a drop in accuracy from
93.76% to 91% compared to the original dataset due to the increasing amount of
learning samples. For the EyePacs dataset, the TL model also demonstrates
superior performance compared to the SSL model. The TL model achieves 99.78%,
99.95%, and 99.36% accuracy across the base, augmented, and down-sampled
versions of the EyePacs dataset, which is 1.5% higher than the SSL model in
base and augmented versions, and 0.8% higher in the downsampled dataset.
Neither the augmentation nor down-sampling techniques have significantly
impacted the performance of the TL and SSL models, as the general performance
of both models in the EyePacs dataset remains high and stable. Compared to the
other three datasets, the main difference lies in the EyePacs dataset having
only two classes, and the downstream task for this dataset is binary
classification, making it easier to learn enough features than other datasets.
The experimental results show a small performance advantage of the TL method
compared to the SSL method in colourful target datasets.

Summarising the training stage performance of both pre-trained models, the SSL
model demonstrates superior performance with grey-scale datasets, while the TL
model excels with colourful datasets. Additionally, the results underscore the
positive impact of the augmentation technique on pre-trained model
performance, but it is noteworthy that the training time and computational
resource costs of the augmented datasets also raised as the training sample
number increased. In contrast, the down-sampling technique demonstrates
opposing effects mainly on grey-scale datasets. It enhances the TL model’s
performance on colourful datasets but diminishes it on grey-scale datasets,
revealing the sensitivity of the TL method to dataset size reduction,
particularly in smaller datasets like BusI and Chest CT (with less than 800
samples). The performance improvement observed in the relatively large
colourful KvasirV2 dataset (with 8000 samples) and EyePacs (with 6000 samples)
suggests that the pre-trained TL model could perform stably when the target
dataset is reasonably substantial. However, the poor performance of the down-
sampling method poses a risk of losing critical information and increases
susceptibility to overfitting, which should raise attention in subsequent
experiments.

Comparison in testing phase assessment

The performance evaluation of the two pre-trained models during the testing
phase is encapsulated in Table 11 and Table 12. Moreover, Fig. 7, Fig. 8, Fig.
9, and Fig. 10 provide a visual representation of the confusion matrix for the
original, augmented, and down-sampled target datasets, facilitating a
comparison between predicted and true labels across each class. The assessment
metrics were derived from the confusion matrix values, which provided a
detailed breakdown of the model classifications. From the table, the testing
outcomes of the grey-scale datasets follow the trends observed in the training
phase, revealing the SSL model’s superior performance on these datasets.
Specifically, the TL model achieved an accuracy of 69.2% on the BusI dataset
and 74% on the Chest CT dataset. In contrast, the SSL model attained accuracy
rates of 74.16% and 93.02% and showcased a 10% advantage in each sensitivity,
precision, and F1 score metric. Also, the SSL model combined with both
augmentation and down-sampling techniques showed a more balanced performance
distribution between each class from the confusion matrix, demonstrating their
ability to address data imbalance issues, while the TL model only had
improvements when combined with the augmentation technique.

Table 11: Results of TL model on testing set.

Datasets Acc(%) Loss Sen(%) Pre(%) F1(%) BusI 69.20 0.75 69.16 79.21 69.28
BusI (Augmented) 78.30 1.18 82.49 82.66 77.51 BusI (Down-sampled) 64.00 0.91
64.00 69.17 64.19 Chest CT 87.30 0.67 86.60 90.20 88.11 Chest CT (Augmented)
93.30 0.26 93.32 93.69 93.39 Chest CT (Down-sampled) 62.50 1.43 88.28 88.14
87.92 Kvasirv2 91.50 0.22 90.90 91.00 90.87 Kvasirv2 (Augmented) 93.40 0.38
93.37 93.37 93.35 Kvasirv2 (Down-sampled) 70.50 1.89 70.50 71.28 70.65 EyePacs
92.10 0.30 92.10 92.00 92.00 EyePacs (Augmented) 91.20 0.80 91.10 91.10 91.00
EyePacs (Downsampled) 88.00 0.66 88.00 88.38 87.90

Table 12: Results of SSL model on testing set.

Datasets Acc(%) Loss Sen(%) Pre(%) F1(%) BusI 74.16 0.80 74.16 79.58 73.71
BusI (Augmented) 81.66 0.73 81.66 84.05 81.72 BusI (Down-sampled) 72.00 0.79
72.00 73.48 71.88 Chest CT 93.02 0.81 94.18 93.03 93.42 Chest CT (Augmented)
97.22 0.77 97.21 97.22 97.20 Chest CT (Down-sampled) 91.40 0.82 91.40 91.49
91.23 Kvasirv2 82.93 1.44 82.93 83.03 82.89 Kvasirv2 (Augmented) 85.81 1.41
85.81 82.96 84.36 Kvasirv2 (Down-sampled) 78.00 1.47 78.00 77.80 77.76 EyePacs
83.10 0.47 83.10 83.10 83.00 EyePacs (Augmented) 83.20 0.48 83.20 83.20 83.20
EyePacs (Downsampled) 75.50 0.54 75.50 75.50 75.50

However, same as the training phase, the down-sampling technique led to a
reduction in performance for both pre-trained models on the grey-scale testing
set. The decrease for the TL model on down-sampled datasets ranged from 5% to
10%, emphasizing a noteworthy performance decline of the down-sampling
technique when facing imbalanced small datasets. Comparatively, the SSL model
also experienced a performance decrease, albeit at a smaller rate of about 2%
compared to the base dataset performance. Given the SSL model’s stable
performance when facing reduced input samples during the training stage, it
exhibits greater resilience to sample reduction, demonstrating superior
efficacy in handling small datasets compared to the TL model. On the other
hand, the augmentation technique proved beneficial for both pre-trained
models, with the TL model displaying more substantial improvement, ranging
from 69.2% accuracy to 77.5% on BusI and 74% to 93.3% on Chest CT. This
observation aligns with the earlier finding that the TL model is more
sensitive to the target dataset size, while the SSL model is less sensitive to
the dataset size but also benefits from its increase. Furthermore, the
correctly predicted ratio of the augmented BusI dataset’s Malignant and Normal
class increased from 53% and 60% to 72%, and the prediction performance of the
Large.cell and Squamous.cell classes of the augmented Chest CT dataset
improved from 80% to 90%. This underscores the augmentation technique’s
benefits in facilitating a more reasonable result and reducing the impact of
imbalanced datasets.

In the context of colourful datasets, the TL model demonstrated superior
performance across all evaluation metrics, achieving an impressive 91.5%
accuracy on the KvasirV2 dataset and 92.1% accuracy on the EyePacs dataset,
maintaining a 10% performance advantage over the SSL model. However, the TL
model experienced a notable decline on the KvasirV2 dataset when employing the
down-sampling technique, resulting in a decrease of about 20% in accuracy,
sensitivity, precision, and F1 score compared to the performance on the base
KvasirV2 dataset. In contrast, the SSL model exhibited a slight decrease of 5%
across each evaluation metric. This situation likely arose due to the greater
reduction range of the KvasirV2 dataset compared to the other two grey-scale
datasets and the TL model’s sensitivity towards dataset size. However, the
decrease in performance of the TL model on EyePacs is much smaller, which can
be attributed to the complexity difference between the classification tasks in
the KvasirV2 and EyePacs datasets.

(a) TL model on base BusI dataset

(b) TL model on augmented BusI dataset

(c) TL model on downsampled BusI dataset

(d) SSL model on base BusI dataset

(e) SSL model on augmented BusI dataset

(f) SSL model on downsampled BusI dataset

Figure 7: Confusion matrix (a), (b), (c) represent TL model on BusI dataset,
Confusion matrix (d), (e), (f) represent SSL model on BusI dataset.

(a) TL model on base ChestCT dataset

(b) TL model on augmented ChestCT dataset

(c) TL model on downsampled ChestCT dataset

(d) SSL model on base ChestCT dataset

(e) SSL model on augmented ChestCT dataset

(f) SSL model on downsampled ChestCT dataset

Figure 8: Confusion matrix (a), (b), (c) represent TL model on ChestCT
dataset, Confusion matrix (d), (e), (f) represent SSL model on ChestCT
dataset.

(a) TL model on base Kvasirv2 dataset

(b) TL model on augmented Kvasirv2 dataset

(c) TL model on downsampled Kvasirv2 dataset

(d) SSL model on base Kvasirv2 dataset

(e) SSL model on augmented Kvasirv2 dataset

(f) SSL model on downsampled Kvasirv2 dataset

Figure 9: Confusion matrix (a), (b), (c) represent TL model on Kvasirv2
dataset, Confusion matrix (d), (e), (f) represent SSL model on Kvasirv2
dataset.

(a) TL model on base EyePacs dataset

(b) TL model on augmented EyePacs dataset

(c) TL model on downsampled EyePacs dataset

(d) SSL model on base EyePacs dataset

(e) SSL model on augmented EyePacs dataset

(f) SSL model on downsampled EyePacs dataset

Figure 10: Confusion matrix (a), (b), (c) represent TL model on EyePacs
dataset, Confusion matrix (d), (e), (f) represent SSL model on EyePacs
dataset.

Notably, there is no indication that the down-sampling technique benefits in
narrowing the performance gap between the classes, and it may even deepen the
gap, especially in datasets like KvasirV2 due to reduced information from the
samples. Compared to the significant performance improvement in grey-scale
datasets, the augmentation technique contributed only modest improvements to
the test results of both the KvasirV2 and EyePacs datasets, with the TL model
improving by 2% and the SSL model by 3% within each evaluation metric on
KvasirV2, and no improvement for either TL or SSL models on EyePacs. This
discrepancy may stem from the original size of the KvasirV2 and EyePacs
datasets being sufficiently large for the models to learn robust features,
thereby limiting the incremental value of further dataset size augmentation.
However, despite the limited improvement in the augmentation technique on the
TL model in general, it has still shown benefits in reducing the imbalance
between each class’s performance. For instance, the positive prediction ratio
of Esophagitis and Normal-z-line in the original KvasirV2 dataset is 88% and
80%, resulting in an 8% performance gap, highlighting the challenge for the TL
model in distinguishing them. Conversely, the performance in the augmented
KvasirV2 dataset is 84% against 86%, demonstrating a reduced 2% gap and
indicating improvement in the pre-trained model’s ability to recognize
representations from two classes. A similar trend is observed in the confusion
matrix of the SSL model on the base and augmented KvasirV2 dataset, with a
reduction in the performance gap between the first and second classes, and the
third and sixth classes.

In summary, the performance trends of the two pre-trained models during the
testing stage align with the observations made in the training stage. The TL
method excels at handling colourful datasets, while the SSL method
demonstrates superior performance in grey-scale datasets. One primary reason
for this divergence lies in the utilisation of ImageNet weight in the pre-
trained TL model, while leveraging the colourful dataset’s knowledge can
improve the ability to handle colourful target datasets. However, this
explanation does not fully account for the SSL model’s comparatively poorer
performance on colourful datasets, despite utilizing the pretext colourful
medical datasets in the pre-training stage. We posit that the distinctive
learning methods of the TL and SSL approaches contribute to this outcome, with
recent studies (Ericsson et al., 2021) suggesting that supervised learning
settings adeptly glean knowledge from colourful features and outperform
unsupervised learning settings. Furthermore, the testing stage results
underscore that the SSL model outperforms the TL model in handling smaller
datasets, a critical consideration in the medical field where data scarcity
issues are prevalent. Importantly, the data down-sampling technique can harm
the pre-trained model’s performance, especially when the target dataset size
is limited, while the data augmentation technique can bring benefits to pre-
trained models. However, there is a diminishing effect when applying the
augmentation technique to relatively large target datasets.

Comparison with double fine-tuned model

The preceding section provided a comparative analysis of TL and SSL models
across both training and testing phases, revealing distinct advantages
inherent in each method when applied to grey-scale and colourful medical
datasets. Despite leveraging pre-trained knowledge to enhance downstream task
learning, the test results sometimes fell short of expectations with the fine-
tuned models. For instance, the SSL model failed to achieve satisfactory
performance on the BusI dataset initially when a significant portion of the
pre-trained model layers was frozen, with only 10% of the top layers being
retrained. This scenario improved considerably when most of the layers were
unfrozen. When leaving only 25% of the bottom layers’ parameters untouched,
there is an enhancement across all evaluation metrics. Similar situations
occurred with the TL model on the KvasirV2 dataset. By using a smaller portion
of transferred parameters and retraining more layers, the TL model exhibited
improved performance on the target dataset. Notably, considering both the BusI
and Chest CT datasets being grey-scale and multi-class, the proposed SSL model
only achieved optimal performance on the Chest CT dataset. This discrepancy
may be attributed to differences in feature distribution between the learned
target samples and the transferred weights, indicating a significant role of
positive knowledge transfer in the downstream task performances. The initial
experimental setup for the pre-trained SSL model involved selecting three
grey-scale medical datasets from diverse modalities (MRI, X-ray, and CT) as
source data to augment the feature set available to the pre-trained model. The
Chest CT dataset showed closer alignment with the source data in feature
distribution, while the BusI dataset samples exhibited distinct
representations from the source domain data. Furthermore, the pre-trained
ImageNet weights did not share similar features with gastrointestinal images,
possibly contributing to the performance drop for pre-trained models in both
the BusI and KvasirV2 datasets.

As supported by prior pre-training studies (Krishnan et al., 2022), the
disparity in representations between the source and target domains
significantly influences the success of knowledge transfer. To address the
domain mismatch issue to mitigate the impact of domain discrepancy and enhance
the SSL and TL models’ performance on the BusI and KvasirV2 datasets, we
implemented a solution employing the double fine-tuning technique. This
approach entails fine-tuning the pre-trained foundational model on an
extensive training dataset, followed by further fine-tuning on the specific
domain/task with limited training data. For the SSL setting, the pre-trained
SSL model using grey-scale knowledge underwent double fine-tuning on an
ultrasound dataset called Polycystic Ovary Syndrome (PCOS) (Anagha &
Aishwarya, 2021). The PCOS dataset has 3856 ultrasound images and was
categorized into infected and non-infected classes. This dataset shares the
same imaging technique as the BusI dataset but involves different disease
types for detection. Initial fine-tuning involved unfreezing 90% of the pre-
trained SSL model layers, followed by using the SimCLR self-supervised
learning method to fine-tune the model. Subsequently, an additional fine-
tuning phase was conducted on the target BusI dataset, using the same learning
settings as the aforementioned experiment to assess performance.

The results, detailed in Table 13, and the evaluation metrics depicted in Fig
11, indicate a significant enhancement in the performance of the double fine-
tuned SSL model across all versions of the target dataset compared to the
single fine-tuned model. Accuracy and sensitivity on the base dataset notably
increased from 74.16% to 79.16%, with precision and F1 score performances
improving by 4% and 5.5%, respectively. Similar improvements were observed in
both augmented and down-sampled datasets, maintaining a consistent enhancement
of approximately 5% across all evaluation metrics. Furthermore, the confusion
matrix maps demonstrate improvements in each class and a reduced performance
gap between classes, indicating that the pre-trained model has improved its
ability to recognize breast tumours from ultrasound images. This outcome
highlights that leveraging a related domain dataset for double fine-tuning can
significantly enhance the pre-trained model’s performance, negating the need
for an extensive sample size. Moreover, the conclusion emphasizes that merely
enriching the modalities and size of the pre-trained datasets during the pre-
training stage may not yield benefits and could potentially waste
computational resources. The pivotal factor lies in transferring useful and
pertinent knowledge to the target domain to sustain high-performance levels of
the pre-trained model.

Table 13: Results of double fine-tuned SSL model on BusI testing set.

Datasets Acc(%) Loss Sen(%) Pre(%) F1(%) BusI 79.16 0.76 79.16 83.36 79.23
BusI (Augmented) 90.00 0.65 90.00 90.21 90.00 BusI (Down-sampled) 77.33 0.79
77.33 80.37 76.76

Table 14: Results of double fine-tuned TL model on KvasirV2 testing set.

Datasets Acc(%) Loss Sen(%) Pre(%) F1(%) KvasirV2 96.40 0.40 96.43 96.44 96.43
KvasirV2 (Augmented) 95.90 0.41 95.87 95.80 95.87 KvasirV2 (Down-sampled)
93.50 0.52 93.50 93.67 93.48

(a) Double fine-tuned SSL model on base BusI dataset

(b) Double fine-tuned SSL model on augmented BusI dataset

(c) Double fine-tuned SSL model on downsampled BusI dataset

(d) Double fine-tuned TL model on base KvasirV2 dataset

(e) Double fine-tuned TL model on augmented KvasirV2 dataset

(f) Double fine-tuned TL model on downsampled KvasirV2 dataset

Figure 11: Confusion matrix (a), (b), (c) represent double fine-tuned SSL
model on BusI dataset, Confusion matrix (d), (e), (f) represent double fine-
tuned TL model on KvasirV2 dataset.

To assess and compare the effectiveness of the double fine-tuned technique
with the TL method, we replicated the process using our pre-trained TL model.
The dataset selected for double fine-tuning is known as Medico-2018 (Pogorelov
et al., 2018), comprising 14,033 gastrointestinal images. Among these, 5,293
images from the training set were annotated by experienced endoscopists into
16 classes of gastrointestinal diseases, while the remaining 8,740 images from
the testing set remained unlabeled. Notably, the Medico-2018 dataset displays
significant class imbalances, with notable differences in sample counts across
each class. During the first fine-tuning phase, we exclusively utilized the
5,293 annotated images due to the supervised learning approach employed in the
pre-trained TL model. We integrated the same projection head, consisting of
four dense layers and one dropout layer, as in the previous experiment setting
for the TL model to facilitate the supervised learning process. Upon
completing the initial training step, the projection head was removed, and the
fine-tuned Xception model was applied to the target Kvasirv2 dataset for the
second fine-tuning step. This second fine-tuning step followed the same
process as before, and the double fine-tuned model underwent testing on the
Kvasirv2 dataset for subsequent analysis.

The results of the double fine-tuned TL model are detailed in Table 14, with
the corresponding confusion matrix outlined in Fig 11. A noticeable
performance improvement is observed in the double fine-tuned TL model compared
to the pre-trained TL model. Specifically, the double fine-tuned model
demonstrated a 5% enhancement on the base dataset and a 3% improvement on the
augmented dataset across metrics such as accuracy, sensitivity, precision, and
F1 score. Furthermore, the double fine-tuned model on the base dataset
outperformed its counterpart on the augmented dataset. Given the size of the
Kvasirv2 dataset, we suggest that this phenomenon occurs because the enriched
parameters transferred from the source domain have reduced the requirement of
needed features to be learned in the target dataset, thereby providing
satisfactory performance. Importantly, the performance of the double fine-
tuned TL model on the down-sampled Kvasirv2 dataset witnessed a substantial
improvement of more than 20% across each evaluation metric, underscoring the
significant value of the double fine-tuned technique in scenarios where the
target dataset is limited. Additionally, the evaluation matrix maps indicate a
more balanced distribution of true prediction samples compared to the single
fine-tuned pre-trained model, demonstrating that the double fine-tuning
technique can equally enhance the pre-trained model’s robustness and reduce
the risks of bias learning in multi-classification tasks.

Comparison with state-of-the-art Methods

Table 15: The proposed models versus state-of-the-art models.

Author Models Pretrained methods Datasets Acc(%) Loss Gheflati & Rivaz (2022)
ResNet-50 Transfer learning BusI 83.00 - Deb & Jha (2023) Ensemble Transfer
learning BusI 85.23 - Proposed TL method Xception Transfer learning BusI 77.50
1.18 Proposed SSL method ResNet-50 Self-supervised learning BusI 90.00 0.65
Dadgar & Neshat (2022) InceptionResNetV2 Transfer learning ChestCT 91.10 0.063
Mamun et al. (2023) CNN - ChestCT 92.00 0.33 Proposed TL method Xception
Transfer learning ChestCT 93.30 0.26 Proposed SSL method ResNet-50 Self-
supervised learning ChestCT 97.22 0.77 Dong et al. (2023) Ensemble Transfer
learning Kavisirv2 90.60 - Mukhtorov et al. (2023) ResNet-152 - Kavisirv2
93.46 0.11 Wang et al. (2023) Hybrid VIT - Kavisirv2 95.42 - Proposed TL
method Xception Transfer learning Kavisirv2 96.40 0.40 Proposed SSL method
ResNet-50 Self-supervised learning Kavisirv2 85.81 1.41 Kiefer et al. (2023)
MobileNetV3 Transfer Learning EyePacs 88.30 - Proposed TL method Xception
Transfer learning EyePacs 92.10 0.80 Proposed SSL method ResNet-50 Self-
supervised learning EyePacs 83.20 0.48

In this subsection, we conducted a comparative analysis, juxtaposing the
optimal outcomes from both pre-trained methods against the performances of
several contemporary state-of-the-art models. The objective is to elucidate
the efficacy of the proposed pre-trained models on the selected datasets, and
the results are delineated in Table 15. A discerning examination of the table
reveals the superior performance of our proposed TL and SSL models compared to
state-of-the-art models on four datasets. The proposed SSL model has achieved
advantages of 4.8% and 5.2% in accuracy compared to those state-of-the-art DL
models that were either trained from scratch or employed transfer learning
methodologies. This outcome aligns with the earlier experimental observation,
indicating that the SSL method has more advantages in grey-scale medical image
learning than the TL method. Concurrently, the proposed TL model demonstrated
competitive prowess when compared to state-of-the-art DL models, showing a 1%
improvement in the KvasirV2 dataset and a 4% improvement in the EyePacs
dataset. This observation underscores the generalization ability and advanced
performance of pre-trained TL models with ImageNet weight, showcasing that
even the pre-trained ImageNet model is constrained in some scenarios but still
has value in medical application learning.

Furthermore, by comparing with some DL studies trained from scratch, the TL
model has shown additional advantages in training speed. For instance,
Mukhtorov et al. (2023) reported that their CNN model, developed without the
use of pre-training, necessitated a training duration of 6-12 hours from
inception on the target Kvasirv2 dataset. In stark contrast, the pre-trained
TL model required a mere 40 minutes for fine-tuning on the identical dataset,
concurrently maintaining a better performance level during the testing phase.
However, the pre-trained SSL model, on average, required double or triple the
time compared to the TL model, consuming more time and computational resources
in the training process. This high time consumption of SSL training has raised
attention from researchers in achieving a balance between high model
performance and reasonable time cost.

Comparison using XAI techniques

As emphasized in the literature review and recent studies (Albahri et al.,
2023; Quinn et al., 2022), the reliability of AI applications in bio-medicine
and healthcare has become a focal point of concern among researchers and
medical institutions. This emphasis on trustworthiness holds pivotal
significance for the widespread acceptance and integration of AI within the
medical domain. However, many studies have overlooked the importance of
interpretability and robustness in their proposed models, potentially leaving
researchers at risk when applying their work to real-world scenarios. In this
section, we aim to test the robustness of our pre-trained models and
demonstrate whether the proposed models have made reasonable predictions based
on disease-type representations. To achieve this goal, we utilized Grad-CAM
heatmaps to gain insights into the focus of the pre-trained models and to
compare the predictions of the two pre-trained models. Understanding why one
pre-trained model made correct predictions while the other did not is of
significant value to the research. Specifically, we selected samples that were
not classified into the true labels by one pre-trained model but were
classified correctly by the other. We then employed the Grad-CAM technique to
generate corresponding heatmaps and compared them to determine the reliability
of correct predictions and identify areas for improvement in incorrect
predictions.

Table 16: The comparison of TL and SSL models’ robustness towards grey-scale
target datasets.

Datasets Pretrained TL Model Pretrained SSL Model True Label BusI Pred: Benign
Pred: Malignant Malignant BusI (Augmented) Pred: Benign Pred: Malignant
Malignant BusI (Downsampled) Pred: Benign Pred: Malignant Malignant ChestCT
Pred: Adenocarcinoma Pred: Squamous.Cell Squamous.Cell ChestCT (Augmented)
Pred: Adenocarcinoma Pred: Large.Cell Large.Cell ChestCT (Downsampled) Pred:
Squamous.Cell Pred: Large.Cell Large.Cell

Table 17: The comparison of TL and SSL models’ robustness towards colourful
target datasets.

Datasets Pretrained TL Model Pretrained SSL Model True Label KvasirV2 Pred:
Polyps Pred: Ulcerative-colitis Polyps KvasirV2 (Augmented) Pred: Normal-z-
line Pred: Normal-pylorus Normal-z-line KvasirV2 (Downsampled) Pred:
Ulcerative-colitis Pred: Polyps Polyps EyePacs Pred: RG Pred: NRG RG EyePacs
(Augmented) Pred: RG Pred: NRG RG EyePacs (Downsampled) Pred: NRG Pred: RG NRG

In Table 16, we conducted a comparative analysis of heatmaps generated from
predicted samples sourced from grey-scale datasets including BusI and Chest
CT. Each set includes the original sample with marked disease regions,
followed by two heatmaps predicted by the pre-trained TL and SSL models,
respectively. For the BusI dataset, we focused on samples from the Malignant
class, as the primary objective of this dataset is to distinguish malignant
tumours that pose a threat to patient health. The corresponding confusion
matrices indicated that the pre-trained TL model struggled to classify this
disease accurately. Malignant cancer typically presents with large tumour
segments, shadows, and multiple projections from nodules. In the first BusI
dataset sample, the TL model misclassified it as Benign, with its attention
focused only on a fraction of the disease region, failing to encompass the
entire area. In contrast, the SSL model correctly identified the disease
region, indicating its ability to discern malignant tumours accurately. This
trend persisted in the augmented BusI dataset, where the TL model again failed
to fully capture the disease region, while the SSL model made the correct
prediction. However, in the down-sampled BusI dataset, the SSL model, while
correctly predicting the disease, lost focus on the true disease region, which
highlights the potential negative impact of the down-sampling technique on
model performance and robustness, particularly with limited target dataset
sizes.

For the Chest CT dataset, samples were selected from the
Squamous.cell.carcinoma and Large.cell.carcinoma classes, representing the
main symptoms of lung cancer diseases. Squamous.cell.carcinoma typically
exhibits cavitation and lung collapse, while Large.cell.carcinoma presents as
large peripheral masses with irregular margins in lung segments. Accordingly,
the pre-trained models should focus on the lung sides rather than soft
tissues. In the Chest CT dataset, we isolated the lung sides from the original
samples for comparison and heatmap evaluation. In the first image from the
base ChestCT dataset, the TL model misclassified the disease as
Adenocarcinoma, focusing its attention on the bottom segment of the right
lung. In contrast, the SSL model correctly classified the sample, directing
its attention to the right lung side and middle aorta and cavity, aligning
more closely with the symptoms. In the second sample from the
Large.cell.carcinoma class, the TL model improved its focus but still
mispredicted the label, whereas the SSL model correctly classified the sample
by widening its focus to both lung sides and middle tissue. In the down-
sampled ChestCT sample, the TL model remained fixated on the right lung
segment, focusing on white bubbles and nerves at the bottom, resulting in an
incorrect prediction of Squamous.cell.carcinoma. In contrast, the SSL model
correctly focused on the entire left lung segment and middle cavity, leading
to an accurate prediction.

Table 17 presents a series of heat maps generated from colourful target
datasets, specifically KvasirV2 and EyePacs. Within the KvasirV2 dataset, our
analysis revealed challenges in classifying three pairs of classes: Dyed-
lifted polyps versus Dyed-resection-margins; Esophagitis versus Normal z-line;
Polyps versus Ulcerative-colitis. Among these pairs, Esophagitis and Polyps
represent common esop-hageal diseases, prompting our focus on samples from the
normal z-line and polyps to discern how TL and SSL models differentiate
between them. The normal z-line typically features a clear demarcation at the
oesophagus-stomach junction with uniform colour and smooth texture, similar to
surrounding tissue. In contrast, Esophagitis often manifests as inflammation,
erosion, or ulceration along the esophageal lining. Polyps exhibit discrete
raised lesions with a smooth or lobulated surface, while Ulcerative Colitis
presents diffuse inflammation, erythema, loss of vascular pattern, and
friability in a continuous pattern. Taking advantage of these distinctions, we
annotated the disease regions in the original images for comparative analysis.
In the KvasirV2 dataset, the first Polyps sample shows substantial caking on
the intestinal surface. The TL model correctly directed its attention around
the caking, while the SSL model erroneously fixated on the caking itself,
resulting in an incorrect prediction. In another instance from the augmented
KvasirV2 dataset, the TL model effectively discerned the demarcation between
tissue and texture, accurately classifying the sample. Conversely, the SSL
model misidentified the background, leading to an incorrect prediction of
Normal-pylorus. In the downsampled KvasirV2 dataset, the attention of the TL
model was dispersed between intestinal pores and caking tissue, resulting in
misclassification. In contrast, the SSL model accurately identified Polyps by
focusing on the caking tissue and part of the intestinal lining.

In the EyePacs dataset, the task revolves around distinguishing referable
glaucoma (RG) and non-referable glaucoma (NRG) from colourful fundus images.
Samples from both RG and NRG classes were examined. RG samples typically
exhibit pronounced optic disc features such as cup-to-disc ratio, neuroretinal
rim thinning, disc hemorrhages, and notching compared to NRG. The TL model
effectively directed attention to the disc area, correctly classifying RG
images. In contrast, the SSL model was fixed on the background of the image,
leading to misclassifications. In an RG sample from the augmented EyePacs
dataset, the TL model maintained accurate predictions by focussing on the disc
area, while the SSL model failed to improve despite the augmentation, and
fixed on the background. In the last sample with few optic disc features, the
TL model correctly identified the white representation area, while the
attention of the SSL model was divided between the eyecup and background,
indicating a lack of true feature comprehension in NRG representations.

In summary, the heatmap samples generated through the Grad-CAM technique offer
further evidence that both TL and SSL models not only exhibit distinct
advantages in handling grey-scale and colourful medical images but also
demonstrate superior robustness towards the specific image types in which they
excel. These findings align closely with observations made during the training
and testing phases, highlighting the efficacy of the pre-trained TL model in
managing colourful medical images and the SSL model’s remarkable capability in
learning from grey-scale medical images.

##  5 Conclusion and Recommendation

In the realm of real-world medical diagnosis, advanced artificial intelligence
techniques have assumed a pivotal role, with pre-trained methods that show
significant potential in facilitating the integration of AI models into this
domain. This study undertakes a comprehensive exploration of the strengths and
limitations inherent in the two predominant pre-trained methods: transfer
learning and self-supervised learning. The findings serve as a guiding
resource for researchers, helping them select the most suitable pre-trained
methods tailored to specific scenarios. The experimental analysis reveals that
the TL method, coupled with widely employed ImageNet-pre-trained weights,
excels in handling colourful medical datasets, but exhibits comparatively
reduced efficacy in managing grey-scale medical datasets. Conversely, the SSL
method demonstrates superior performance when confronted with grey-scale
medical datasets but lags behind the TL method in handling their colourful
counterparts. Significantly, despite their shared objective of addressing data
scarcity, models pre-trained using the SSL method exhibit heightened stability
in the face of smaller datasets compared to those trained using the TL method.
Additionally, the study examines the efficacy of data-altering methods in
conjunction with pre-trained models for managing imbalanced data, which is an
additional prevalent challenge in the medical field. The augmentation
technique, when combined with pre-trained methods, not only enhances
performance, but also fortifies the robustness of pre-trained models.
Conversely, the down-sampling technique proves counterproductive, mitigating
the influence of imbalanced class samples but compromising the overall
performance and stability of the model.

A novel aspect of this study involves the evaluation of the double fine-tuning
technique on both the pre-trained TL and SSL models. The observations affirm
the utility of double fine-tuning in enhancing pre-trained models’ performance
and robustness, particularly in scenarios where it is hard to obtain extensive
pre-trained data that matches the target data domain. The study introduced a
modified Xception model and a concatenated ResNet-50 model, pre-trained using
TL and SSL methods, respectively. The proposed SSL model achieves an accuracy
of 90% and 97.22% in the multiclass classification of the augmented BusI and
chest CT datasets. In contrast, the TL model achieves 96.4% and 92.1% accuracy
in classifying the Kvasirv2 and EyePacs dataset image classes. These models,
when fine-tuned with image-based AI technology, demonstrate efficacy in
supporting disease diagnosis and classification.

The insights of this study have the potential to go beyond the realm of
medical research, providing valuable considerations for addressing data
scarcity through pre-trained methods in diverse fields. In addition,
researchers who intend to utilise pre-trained techniques in their studies
should consider the following conditions.

1\. The SSL method is recommended for small target datasets (for example: less
than 1000 samples), as it imparts superior performance and stability compared
to the TL method.

2\. The nature of the colour information within the target data samples
dictates the choice between the TL and SSL methods. TL methods excel with
colourful images, while SSL methods exhibit proficiency with grey-scale
images.

3\. When confronting imbalanced data, it is advisable to employ the data
augmentation technique with pre-trained methods. However, researchers need to
pay more attention to the associated increase in training time and
computational resource consumption, as well as to the risk of overfitting.

4\. Avoid using data down-sampling when the target dataset is small, as it may
compromise model performance and robustness, leading to a loss of crucial
target information.

5\. In the absence of extensive source data that mirror target data features,
consider a two-step approach: pre-training the model in a less related domain
with more samples, followed by fine-tuning it in a closely related domain with
smaller samples. This strategy bridges the gap between pre-trained models and
target datasets without requiring abundant similar training samples.

6\. Pre-trained methods extend their utility beyond small datasets; Even
relatively large datasets benefit from improved performance, robustness,
training speed, and avoidance of overfitting. Researchers are encouraged to
explore pre-trained methods within and beyond the medical field.

CRediT authorship contribution statement

Zehui Zhao:  Conceptualisation, Validation, Writing – original draft, Laith
Alzubaidi:  Conceptualisation, Resources, Formal analysis, Writing – review &
editing, Validation, Supervision, Jinglan Zhang: Writing – review & editing,
Formal analysis, Validation, Supervision Ye Duan:  Formal analysis, Writing –
review & editing, Usman Naseem:  Formal analysis, Writing – review & editing,
Yuantong Gu:  Writing – review & editing, Validation, Funding acquisition.

Consent for publication

All authors have given their consent for publication.

Declaration of competing interest

All authors have declared that they have no conflict of interest.

Acknowledgements

I would like to thank my supervisors, colleagues, and reviewers for supporting
my work and providing me with valuable feedback and suggestions.

Funding

Laith Alzubaidi and Yuantong Gu acknowledge the support received through the
following funding schemes of the Australian Government: Australian Research
Council (ARC) Industrial Transformation Training Centre (ITTC) for Joint
Biomechanics under grant (IC190100020).

## References

  * Al-Dhabyani et al. (2020) Al-Dhabyani, W., Gomaa, M., Khaled, H., Fahmy, A., (2020). Dataset of breast ultrasound images. Data in brief 28, 104863. https://doi.org/10.1016/j.dib.2019.104863
  * Albahri et al. (2023) Albahri, A., Duhaim, A. M., Fadhel, M. A., Alnoor, A., Baqer, N. S., Alzubaidi, L., Albahri, O., Alamoodi, A., Bai, J., Salhi, A., et al., (2023). A systematic review of trustworthy and explainable artificial intelligence in healthcare: Assessment of quality, bias risk, and data fusion. Information Fusion. https://doi.org/10.1016/j.inffus.2023.03.008
  * Alzubaidi et al. (2021) Alzubaidi, L., Al-Amidie, M., Al-Asadi, A., Humaidi, A. J., Al-Shamma, O., Fadhel, M. A., Zhang, J., Santamaría, J., Duan, Y., (2021). Novel transfer learning approach for medical imaging with limited labeled data. Cancers 13 (7), 1590. https://doi.org/10.3390/cancers13071590
  * Alzubaidi et al. (2020) Alzubaidi, L., Fadhel, M. A., Al-Shamma, O., Zhang, J., Santamaría, J., Duan, Y., R. Oleiwi, S., (2020). Towards a better understanding of transfer learning for medical imaging: a case study. Applied Sciences 10 (13), 4523. https://doi.org/10.3390/app10134523
  * Anagha & Aishwarya (2021) Anagha, C., Aishwarya, K., (2021). Pcos detection using ultrasound images. URL: https://www.kaggle.com/datasets/anaghachoudhari/pcos-detection-using-ultrasound-images
  * Atasever et al. (2023) Atasever, S., Azginoglu, N., Terzi, D. S., Terzi, R., (2023). A comprehensive survey of deep learning research on medical image analysis with focus on transfer learning. Clinical Imaging 94, 18–41. https://doi.org/10.1016/j.clinimag.2022.11.003
  * Azizi et al. (2023) Azizi, S., Culp, L., Freyberg, J., Mustafa, B., Baur, S., Kornblith, S., Chen, T., Tomasev, N., Mitrović, J., Strachan, P., et al., (2023). Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging. Nature Biomedical Engineering. https://doi.org/10.1038/s41551-023-01049-7
  * Buda et al. (2018) Buda, M., Maki, A., Mazurowski, M. A., (2018). A systematic study of the class imbalance problem in convolutional neural networks. Neural networks 106, 249–259. https://doi.org/10.1016/j.neunet.2018.07.011
  * Chen et al. (2020) Chen, T., Kornblith, S., Norouzi, M., Hinton, G., (2020). A simple framework for contrastive learning of visual representations. International conference on machine learning. PMLR, pp. 1597–1607.
  * Chollet (2017) Chollet, F., (2017). Xception: Deep learning with depthwise separable convolutions. Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1251–1258.
  * Combalia et al. (2019) Combalia, M., Codella, N. C., Rotemberg, V., Helba, B., Vilaplana, V., Reiter, O., Carrera, C., Barreiro, A., Halpern, A. C., Puig, S., et al., (2019). Bcn20000: Dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288. https://doi.org/10.48550/arXiv.1908.02288
  * Dadgar & Neshat (2022) Dadgar, S., Neshat, M., (2022). Comparative hybrid deep convolutional learning framework with transfer learning for diagnosis of lung cancer. International Conference on Soft Computing and Pattern Recognition. Springer, pp. 296–305.
  * Deb & Jha (2023) Deb, S. D., Jha, R. K., (2023). Breast ultrasound image classification using fuzzy-rank-based ensemble network. Biomedical Signal Processing and Control 85, 104871. https://doi.org/10.1016/j.bspc.2023.104871
  * Del Pup & Atzori (2023) Del Pup, F., Atzori, M., (2023). Applications of self-supervised learning to biomedical signals: a survey. IEEE Access. https://doi.org/10.1109/ACCESS.2023.3344531
  * Dong et al. (2023) Dong, Z., Xu, B., Shi, J., Zheng, L., (2023). Local and global feature interaction network for endoscope image classification. International Conference on Image and Graphics. Springer, pp. 412–424.
  * Ericsson et al. (2021) Ericsson, L., Gouk, H., Hospedales, T. M., (2021). How well do self-supervised models transfer? Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5414–5423.
  * Ericsson et al. (2022a) Ericsson, L., Gouk, H., Hospedales, T. M., (2022a). Why do self-supervised models transfer? on the impact of invariance on downstream tasks. 33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 21-24. BMVA Press, p. 14.
  * Ericsson et al. (2022b) Ericsson, L., Gouk, H., Loy, C. C., Hospedales, T. M., (2022b). Self-supervised representation learning: Introduction, advances, and challenges. IEEE Signal Processing Magazine 39 (3), 42–62. https://doi.org/10.1109/MSP.2021.3134634
  * Gheflati & Rivaz (2022) Gheflati, B., Rivaz, H., (2022). Vision transformers for classification of breast ultrasound images. 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, pp. 480–483. https://doi.org/10.1109/EMBC48229.2022.9871809
  * Haixiang et al. (2017) Haixiang, G., Yijing, L., Shang, J., Mingyun, G., Yuanyue, H., Bing, G., (2017). Learning from class-imbalanced data: Review of methods and applications. Expert systems with applications 73, 220–239. https://doi.org/10.1016/j.eswa.2016.12.035
  * Hany (2020) Hany, M., (2020). Chest ct-scan images dataset. URL: https://www.kaggle.com/datasets/mohamedhanyyy/chest-ctscan-images
  * He et al. (2016) He, K., Zhang, X., Ren, S., Sun, J., (2016). Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778.
  * Hosseinzadeh Taher et al. (2021) Hosseinzadeh Taher, M. R., Haghighi, F., Feng, R., Gotway, M. B., Liang, J., (2021). A systematic benchmarking analysis of transfer learning for medical image analysis. Domain Adaptation and Representation Transfer, and Affordable Healthcare and AI for Resource Diverse Global Health: Third MICCAI Workshop, DART 2021, and First MICCAI Workshop, FAIR 2021, Held in Conjunction with MICCAI 2021, Strasbourg, France, September 27 and October 1, 2021, Proceedings 3. Springer, pp. 3–13.
  * Huh et al. (2016) Huh, M., Agrawal, P., Efros, A. A., (2016). What makes imagenet good for transfer learning? arXiv preprint arXiv:1608.08614. https://doi.org/10.48550/arXiv.1608.08614
  * Islam et al. (2022) Islam, M. N., Hasan, M., Hossain, M. K., Alam, M. G. R., Uddin, M. Z., Soylu, A., (2022). Vision transformer and explainable transfer learning models for auto detection of kidney cyst, stone and tumor from ct-radiography. Scientific Reports 12 (1), 11440. https://doi.org/10.1038/s41598-022-15634-4
  * Kassani et al. (2019) Kassani, S. H., Kassani, P. H., Khazaeinezhad, R., Wesolowski, M. J., Schneider, K. A., Deters, R., (2019). Diabetic retinopathy classification using a modified xception architecture. 2019 IEEE international symposium on signal processing and information technology (ISSPIT). IEEE, pp. 1–6. https://doi.org/10.1109/ISSPIT47144.2019.9001846
  * Kiefer et al. (2023) Kiefer, R., Abid, M., Ardali, M. R., Steen, J., Amjadian, E., (2023). Automated fundus image standardization using a dynamic global foreground threshold algorithm. 2023 8th International Conference on Image, Vision and Computing (ICIVC). IEEE, pp. 460–465. https://doi.org/10.1109/ICIVC58118.2023.10270429
  * Kim et al. (2022) Kim, H. E., Cosa-Linan, A., Santhanam, N., Jannesari, M., Maros, M. E., Ganslandt, T., (2022). Transfer learning for medical image classification: a literature review. BMC medical imaging 22 (1), 69. https://doi.org/10.1186/s12880-022-00793-7
  * Kolesnikov et al. (2020) Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., Houlsby, N., (2020). Big transfer (bit): General visual representation learning. Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16. Springer, pp. 491–507.
  * Kornblith et al. (2019) Kornblith, S., Shlens, J., Le, Q. V., (2019). Do better imagenet models transfer better? Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2661–2671.
  * Krishnan et al. (2022) Krishnan, R., Rajpurkar, P., Topol, E. J., (2022). Self-supervised learning in medicine and healthcare. Nature Biomedical Engineering 6 (12), 1346–1352. https://doi.org/10.1038/s41551-022-00914-1
  * Krizhevsky et al. (2012) Krizhevsky, A., Sutskever, I., Hinton, G. E., (2012). Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25.
  * Liu et al. (2022) Liu, Y., Zhang, L., Hao, Z., Yang, Z., Wang, S., Zhou, X., Chang, Q., (2022). An xception model based on residual attention mechanism for the classification of benign and malignant gastric ulcers. Scientific Reports 12 (1), 15365. https://doi.org/10.1038/s41598-022-19639-x
  * Mamun et al. (2023) Mamun, M., Mahmud, M. I., Meherin, M., Abdelgawad, A., (2023). Lcdctcnn: Lung cancer diagnosis of ct scan images using cnn based model. 2023 10th International Conference on Signal Processing and Integrated Networks (SPIN). IEEE, pp. 205–212. https://doi.org/10.1109/SPIN57001.2023.10116075
  * Marcus et al. (2007) Marcus, D. S., Wang, T. H., Parker, J., Csernansky, J. G., Morris, J. C., Buckner, R. L., (2007). Open access series of imaging studies (oasis): cross-sectional mri data in young, middle aged, nondemented, and demented older adults. Journal of cognitive neuroscience 19 (9), 1498–1507. https://doi.org/10.1162/jocn.2007.19.9.1498
  * Morid et al. (2021) Morid, M. A., Borjali, A., Del Fiol, G., (2021). A scoping review of transfer learning research on medical image analysis using imagenet. Computers in biology and medicine 128, 104115. https://doi.org/10.1016/j.compbiomed.2020.104115
  * Mukhtorov et al. (2023) Mukhtorov, D., Rakhmonova, M., Muksimova, S., Cho, Y.-I., (2023). Endoscopic image classification based on explainable deep learning. Sensors 23 (6), 3176. https://doi.org/10.3390/s23063176
  * Muljo et al. (2023) Muljo, H. H., Pardamean, B., Elwirehardja, G. N., Hidayat, A. A., Sudigyo, D., Rahutomo, R., Cenggoro, T. W., (2023). Handling severe data imbalance in chest x-ray image classification with transfer learning using swav self-supervised pre-training. Commun. Math. Biol. Neurosci. 2023, Article–ID.
  * Neyshabur et al. (2020) Neyshabur, B., Sedghi, H., Zhang, C., (2020). What is being transferred in transfer learning? Advances in neural information processing systems 33, 512–523.
  * Nguyen et al. (2022) Nguyen, H. Q., Lam, K., Le, L. T., Pham, H. H., Tran, D. Q., Nguyen, D. B., Le, D. D., Pham, C. M., Tong, H. T., Dinh, D. H., et al., (2022). Vindr-cxr: An open dataset of chest x-rays with radiologist’s annotations. Scientific Data 9 (1), 429. https://doi.org/10.1038/s41597-022-01498-w
  * Niu et al. (2020) Niu, S., Liu, Y., Wang, J., Song, H., (2020). A decade survey of transfer learning (2010–2020). IEEE Transactions on Artificial Intelligence 1 (2), 151–166. https://doi.org/10.1109/TAI.2021.3054609
  * Pan & Yang (2010) Pan, S. J., Yang, Q., (2010). A survey on transfer learning. IEEE Transactions on knowledge and data engineering 22 (10), 1345–1359. https://doi.org/10.1109/TKDE.2009.191
  * Patil & Sharma (2024) Patil, R., Sharma, S., (2024). Automatic glaucoma detection from fundus images using transfer learning. Multimedia Tools and Applications, 1–20. https://doi.org/10.1007/s11042-024-18242-8
  * Pogorelov et al. (2017) Pogorelov, K., Randel, K. R., Griwodz, C., Eskeland, S. L., de Lange, T., Johansen, D., Spampinato, C., Dang-Nguyen, D.-T., Lux, M., Schmidt, P. T., et al., (2017). Kvasir: A multi-class image dataset for computer aided gastrointestinal disease detection. Proceedings of the 8th ACM on Multimedia Systems Conference. pp. 164–169. https://doi.org/10.1145/3083187.3083212
  * Pogorelov et al. (2018) Pogorelov, K., Riegler, M., Halvorsen, P., Hicks, S., Randel, K. R., Dang Nguyen, D. T., Lux, M., Ostroukhova, O., de Lange, T., (2018). Medico multimedia task at mediaeval 2018. CEUR Workshop Proceedings. Vol. 2283. Technical University of Aachen, pp. 1–4.
  * Qayyum et al. (2020) Qayyum, A., Qadir, J., Bilal, M., Al-Fuqaha, A., (2020). Secure and robust machine learning for healthcare: A survey. IEEE Reviews in Biomedical Engineering 14, 156–180. https://doi.org/10.1109/RBME.2020.3013489
  * Quinn et al. (2022) Quinn, T. P., Jacobs, S., Senadeera, M., Le, V., Coghlan, S., (2022). The three ghosts of medical ai: Can the black-box present deliver? Artificial intelligence in medicine 124, 102158. https://doi.org/10.1016/j.artmed.2021.102158
  * Raghu et al. (2019) Raghu, M., Zhang, C., Kleinberg, J., Bengio, S., (2019). Transfusion: Understanding transfer learning for medical imaging. Advances in neural information processing systems 32.
  * Razzak et al. (2018) Razzak, M. I., Naz, S., Zaib, A., (2018). Deep learning for medical image processing: Overview, challenges and the future. Classification in BioApps: Automation of Decision Making, 323–350.
  * Romero et al. (2020) Romero, M., Interian, Y., Solberg, T., Valdes, G., (2020). Targeted transfer learning to improve performance in small medical physics datasets. Medical physics 47 (12), 6246–6256. https://doi.org/10.1002/mp.14507
  * Russakovsky et al. (2015) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al., (2015). Imagenet large scale visual recognition challenge. International journal of computer vision 115, 211–252. https://doi.org/10.1007/s11263-015-0816-y
  * Schiappa et al. (2023) Schiappa, M. C., Rawat, Y. S., Shah, M., (2023). Self-supervised learning for videos: A survey. ACM Computing Surveys 55 (13s), 1–37. https://doi.org/10.1145/3577925
  * Selvaraju et al. (2017) Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D., (2017). Grad-cam: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE international conference on computer vision. pp. 618–626.
  * Shorten & Khoshgoftaar (2019) Shorten, C., Khoshgoftaar, T. M., (2019). A survey on image data augmentation for deep learning. Journal of big data 6 (1), 1–48. https://doi.org/10.1186/s40537-019-0197-0
  * Shurrab & Duwairi (2022) Shurrab, S., Duwairi, R., (2022). Self-supervised learning methods and applications in medical imaging analysis: A survey. PeerJ Computer Science 8, e1045. https://doi.org/10.7717/peerj-cs.1045
  * Tan et al. (2024) Tan, Z., Yu, Y., Meng, J., Liu, S., Li, W., (2024). Self-supervised learning with self-distillation on covid-19 medical image classification. Computer Methods and Programs in Biomedicine 243, 107876. https://doi.org/10.1016/j.cmpb.2023.107876
  * Truong et al. (2021) Truong, T., Mohammadi, S., Lenga, M., (2021). How transferable are self-supervised features in medical image classification tasks? Machine Learning for Health. PMLR, pp. 54–74.
  * Wang (2021) Wang, K., (2021). An overview of deep learning based small sample medical imaging classification. 2021 International Conference on Signal Processing and Machine Learning (CONF-SPML). IEEE, pp. 278–281. https://doi.org/10.1109/CONF-SPML54095.2021.00060
  * Wang et al. (2023) Wang, W., Yang, X., Tang, J., (2023). Vision transformer with hybrid shifted windows for gastrointestinal endoscopy image classification. IEEE Transactions on Circuits and Systems for Video Technology. https://doi.org/10.1109/TCSVT.2023.3277462
  * Wu et al. (2023) Wu, R., Liang, C., Li, Y., Shi, X., Zhang, J., Huang, H., (2023). Self-supervised transfer learning framework driven by visual attention for benign–malignant lung nodule classification on chest ct. Expert Systems with Applications 215, 119339. https://doi.org/10.1016/j.eswa.2022.119339
  * Yang et al. (2020) Yang, X., He, X., Liang, Y., Yang, Y., Zhang, S., Xie, P., (2020). Transfer learning or self-supervised learning? a tale of two pretraining paradigms. arXiv preprint arXiv:2007.04234. https://doi.org/10.48550/arXiv.2007.04234
  * Zhang et al. (2023) Zhang, C., Zheng, H., Gu, Y., (2023). Dive into the details of self-supervised learning for medical image analysis. Medical Image Analysis 89, 102879. https://doi.org/10.1016/j.media.2023.102879
  * Zhang et al. (2022) Zhang, W., Deng, L., Zhang, L., Wu, D., (2022). A survey on negative transfer. IEEE/CAA Journal of Automatica Sinica 10 (2), 305–329. https://doi.org/10.1109/JAS.2022.106004
  * Zhao et al. (2023) Zhao, Z., Alzubaidi, L., Zhang, J., Duan, Y., Gu, Y., (2023). A comparison review of transfer learning and self-supervised learning: Definitions, applications, advantages and limitations. Expert Systems with Applications, 122807. https://doi.org/10.1016/j.eswa.2023.122807
  * Zhou et al. (2016) Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A., (2016). Learning deep features for discriminative localization. Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2921–2929.