# Reaching Diverse Groups in Long-Term Astronomy Public Engagement Efforts

Melanie Archipley,1,5 Hannah S. Dalgleish,2,5 Eva-Maria Ahrer,3,5 Daniel
Mortimer4,5 1Dept. of Astronomy, University of Illinois Urbana-Champaign, 1002
West Green Street, Urbana, IL, 61801, USA<EMAIL_ADDRESS>2Dept. of Physics,
University of Namibia, Pionierspark, Windhoek, Namibia; 3Dept. of Physics,
University of Warwick, Gibbet Hill Road, Coventry, CV4 7AL, UK; 4Cavendish
Laboratory, University of Cambridge, 19 J.J. Thomson Avenue, Cambridge, CB3
0HE, UK; 5The International Workshop for Astronomy e.V., Türkenstraße 12,
Garching, Munich, 85748, DE

###### Abstract

Professional astronomy is historically not an environment of diverse
identities. In recognizing that public outreach efforts affect career outcomes
for young people, it is important to assess the demographics of those being
reached and continually consider strategies for successfully engaging
underrepresented groups. One such outreach event, the International
Astronomical Youth Camp (IAYC), has a 50-year history and has reached
$\sim$1700 participants from around the world. We find that the IAYC is doing
well in terms of gender (59% female, 4.7% non-binary at the most recent camp)
and LGBT+ representation, whereas black and ethnic minorities are lacking. In
this proceeding, we report the current landscape of demographics applying to
and attending the IAYC; the efforts we are making to increase diversity
amongst participants; the challenges we face; and our future plans to bridge
these gaps, not only for the benefit of the camp but for society overall.

##  1 Introduction

Motivating young people to pursue careers in science has become the mission of
many outreach programs today. Several studies have shown that informal
learning environments are key drivers behind improving attitudes towards
science (e.g. Vennix et al. 2017; Kitchen et al. 2018; Roberts et al. 2018).
Looking at summer science camps in particular, the International Astronomical
Youth Camp (IAYC) is a 50-year old event which uses astronomy as a medium for
scientific inquiry. The first camp took place in 1969 in West Germany and the
most recent in 2019 was also in Germany — no in-person 2020 iteration took
place due to the worldwide COVID-19 pandemic. There have been 15 different
host countries throughout Europe, northern Africa, and the Middle East in the
IAYC’s history. In 1979, The International Workshop for Astronomy e.V. (IWA)
was founded as the official organization behind the camp.

Once a year, around 65 participants aged 16-24 spend three weeks working on an
independent project of their choosing. They usually work in pairs or small
groups with the guidance of their group leader. There are no lectures or
exams. At the end of the camp, the participants write up their findings in a
4-8 page report using L$\scriptstyle\kern-2.10002ptA$TeX, which is published
in a report book distributed to all attendees of that year’s camp.

Participants come from a great range of backgrounds — academically and
culturally — and $\sim$80% have never studied astronomy before
attending.111The IAYC survey comprised of 54 questions and was completed by
307 former participants between 16 August 2017 and 21 September 2019. The
results were compiled by IWA and many of the survey’s findings are published
in Dalgleish & Veitch-Michaelis (2018, 2019). The application process reflects
this expectation: applicants do not submit transcripts, grades, a resume, or
letters of recommendation. Applicants are prompted to discuss their
experiences and inspirations in science and international environments in a
“letter of motivation.” Since 2018, particular effort has been made to better
understand our demographics and how they have changed over the IAYC’s long
history. Using self-reported data from the IAYC survey,1 internal (generalized
and anonymized) data, and anecdotal experience as organizers of the camp, we
have identified specific areas of strength (gender, LGBT+) and weakness
(racial and ethnic diversity) in terms of diversity of the camp.

In Section 2, we present data describing demographics of IAYC participants and
leaders from 1969 to 2019. In Section 3, we discuss recent efforts to improve
our reach to, and welcoming of, underrepresented groups and to what end those
efforts have been successful. Finally, in Section 4 we assess further gaps to
be filled, such as increased funding for low-income participants, to help us
achieve our goals in the future.

##  2 Quantitative History and Status of Diversity

For our assessments, we identify four broad categories of diversity in which
we are interested: gender, LGBT+, nationality, and ethnicity. We later plan to
extend our studies to assess our reach and inclusion of neurodiverse people;
people with disabilities; people from varying socioeconomic backgrounds; and
more.

###  2.1 Gender and LGBT+

Our earliest records on gender comes from IAYC 1984, with data for every year
since then except 1995. Figure 1 shows the gender representation for these 33
camps for participants, alongside 24 years of data for leaders, with the
notable exception that an openly non-binary individual attended the camp in
2017 for the first time.

Figure 1.: Data showing the trend of gender representation at the IAYC since
1984. Orange, blue, and green points represent male, female, and non-binary
people, respectively. Each straight line is the linear best fit to the data,
showing the overall trend of increasing females and decreasing males, even
beyond gender parity for participants (top panel). The leader data (bottom
panel) is represented as raw numbers instead of percentages due to the small
size of leaders per camp. From this, we note that the trend of fewer male
leaders at the camp over the years is due to a marginal increase in female
leaders and a decrease in leader team size overall.

We infer that the gender gap trended towards parity for most of the IAYC’s
history, and the gap was finally closed for participants around 2012. Somewhat
surprisingly, the gender representation has not stopped trending towards
increasing females and decreasing males in the past decade. It is possible
that the trend towards gender balance on the leader team had a ripple effect
to participants (bottom panel of Figure 1), however, the leader team has not
seen as drastic an increase in female representation.

We would be remiss to not also note that a wide range of gender identities are
absent in our limited and statistical discussion of gender. We strive to make
the camp atmosphere as open and welcoming as possible to people beyond these
limits — see Section 3. Given the high number of nationalities at each camp,
we have a unique opportunity to examine LGBT+ perceptions across many
cultures. We are interested in the impact of the camp on LGBT+ and non-LGBT+
participants, as individuals and as youth astronomers and scientists. The
beginnings of these efforts exist in the form of a qualitative survey with
quotes and feedback presented in Miller (2021). A quantitative survey is now
being developed to better assess the LGBT+ population at the IAYC. Overall, we
feel that our current representations of gender and LGBT+ people indicate that
the IAYC is a welcoming and accessible environment to these diverse groups.

###  2.2 Nationality and Ethnicity

The number of nationalities at each IAYC has steadily and significantly
increased since its beginnings (see Figure 2). However, unique ethnicities
have not been represented equitably, and the majority of participants have
been European whites. Of the 307 respondents to the IAYC survey (see Footnote
1), 84.7% were white and not a single respondent was Black (other responses
included Mixed, 5.9%; Asian, 4.2%; Other, 3.3%; and Arab, 2.0%). Participants
are aware of, and concerned about, the lack of ethnic diversity at the camp,
as the topic has been raised to IWA in recent years.

Figure 2.: Data showing the number of nationalities represented at every IAYC
since its inception, including the four years that two camps took place. The
straight line is the best linear fit to the data. Roughly every decade, five
more nationalities are represented at the camp, without appreciably increasing
the total number of participants accepted to each camp.

Even though our demographic is largely European — and predictably so, given
that the last time the camp was outside of Europe was in 1984 — the lack of
Black and minority ethnic (BME) participants is neither representative of the
European population as a whole, nor of the professional astronomy
community.222While ethnicity data are hard to come by, we are aware of this
demographic survey carried out by the Royal Astronomical Society (McWhinnie
2017) which gives some indication. Because of our success in increasing number
of nationalities over the years, we know it is possible to extend the IAYC’s
reach to new countries, but we are nonetheless struggling to reach non-whites.
Some of the ways we are strategizing to reach ethnically diverse groups are
described in Section 3.

##  3 Recent Developments to Improve Access

We recognize the importance of being efficient and prudent in how we tackle
the many steps that a participant takes to come to the IAYC. Therefore, we
have made adjustments both in the recruitment and application phases as well
as during the camp itself. Here we highlight some of the ways we have
approached increasing and welcoming diversity.

###  3.1 Application Evaluation

We are not alone in exploring different methods to improve inclusivity through
our application process (Johnson & Kirk 2020; Aloisi & Reid 2020). In our
initial review process, we experimented with blinding identifying and
demographic information — name, nationality, gender, and age — from the
evaluators with the goal of reducing implicit bias. It is difficult to assess
if we achieved that goal explicitly, but some evaluators reported that they
felt they were not getting the full picture of applicants without those
details. We are exploring ways to fill in those gaps without exposing
ourselves to pitfalls of internally-held stereotypes.

After accepting a first round of applicants, we create a waiting list and
accept new people as people drop out. Inspired by the Entrofy algorithm for
candidate selection (Huppenkothen et al. 2019), IWA developed an algorithm
that assessed the waiting list for an underrepresented nationality each time a
spot opened up if the next few candidates were nearly equally ranked. An
example is as follows:

  * Someone declines their acceptance. The next person on the waiting list is from country A. Many confirmed participants are also from country A. The person after them is from country B. There are 0 confirmed participants from country B. The algorithm recommends filling the open spot with the person from country B.

The algorithm was not used on the first round of acceptances and was not a
replacement for meritocratic evaluation of applications. In the end, the
algorithm was used so rarely and conservatively that no participant who
attended the camp was affected by it. Though we are unlikely to employ it in
future rounds of evaluation, developing it sparked important discussion and
ideas about our application process through this lens.

###  3.2 LGBT+ Focused Efforts

It would not be enough to stop at the recruitment and application stage —
therefore we take deliberate actions to welcome diverse groups once at the
camp. In 2017, leaders began to introduce themselves alongside their pronouns
and encouraged participants to do the same. In 2019, we added an “other (if
gender is other, please specify)” option on the application for the “gender”
category as well as providing the possibility for mixed gender dorm rooms,
which the majority of participants chose. These changes were designed
specifically to remove any pressure upon participants to slot themselves into
a gender identity with which they did not feel wholly comfortable.

The camp has been rooted in celebrations of cultural and ethnic diversity
since it originated, with specific time carved out to share and exchange food,
music, dance, language, and history across nations. Since 2017, many
participants and leaders have taken part in a “IAYC pride” celebration during
the camp, as an added dimension to the kinds of exchange already taking place.
Some annual events include a short informational talk about sexual orientation
and gender, a panel discussion, collaborative art and music, and lessons on
historical events and people related to the struggle for equity and
recognition in the LGBT+ community.

##  4 Reflections and Future Plans

Addressing diversity in STEM outreach is a continuous effort of self-
evaluation, trial and error, data collection and analysis, and sometimes,
fundraising. The IAYC has been oversubscribed only in the past $\sim 7$ years,
significantly so for the two most recent camps which were oversubscribed by a
factor of $\sim$2.5. This recent oversubscription calls upon the organizers to
ensure that people of all identities and backgrounds are able to attend the
camp going forward. Thinking critically about the entire process from the
participant’s perspective — deciding to apply, writing a successful
application, traveling to the camp, thriving while there, and finally,
embracing lessons learned and the IAYC community after the camp — is crucial
to developing equitable outcomes.

With little effort, we have achieved gender parity at the IAYC, as well as a
significant upward trend in number of unique nationalities at each camp. LGBT+
identities are now not only well represented but also celebrated by all during
the camp. However, we have a poor record when it comes to ethnicity and
serving non-white participants. Part of addressing this gap requires financial
means we do not currently possess, such as to fund participants’ travel from
countries outside of Europe. While the actions we described in Section 3 were
focused on aspects like implicit bias and accommodation, we recognize that
some solutions will come from simple fundraising.

For the 2020 camp (cancelled due to COVID-19), we included a new optional
diversity monitoring form. Questions included parents’ level of education,
physical or learning disabilities, and sexual orientation in addition to the
standard questions about age, gender, and so on. This is a first step to
assessing the pipeline from application to acceptance and participation. We
intend to monitor these data going forward to inform our choices when
organizing the camp to ensure its accessibility to all.

#### Acknowledgments

MA acknowledges support from the Illinois Graduate College’s Conference
Presentation Award. HD acknowledges support from the UKRI STFC Global
Challenges Research Fund project ST/S002952/1. Figures are produced using the
python package matplotlib (Hunter 2007).

## References

  * Aloisi & Reid (2020) Aloisi, A., & Reid, N. 2020, arXiv e-prints, arXiv:2006.13685.  https://arxiv.org/abs/2006.13685
  * Dalgleish & Veitch-Michaelis (2018) Dalgleish, H. S., & Veitch-Michaelis, J. L. 2018, in Communicating Astronomy with the Public Conference 2018 2nd Edition, 206–207
  * Dalgleish & Veitch-Michaelis (2019) Dalgleish, H. S., & Veitch-Michaelis, J. L. 2019, Nature Astronomy, 3, 1043, doi: 10.1038/s41550-019-0965-y
  * Hunter (2007) Hunter, J. D. 2007, Computing in Science & Engineering, 9, 90, doi: 10.1109/MCSE.2007.55
  * Huppenkothen et al. (2019) Huppenkothen, D., McFee, B., & Norén, L. 2019, arXiv e-prints, arXiv:1905.03314.  https://arxiv.org/abs/1905.03314
  * Johnson & Kirk (2020) Johnson, S. K., & Kirk, J. F. 2020, Publications of the Astronomical Society of the Pacific, 132, 034503, doi: 10.1088/1538-3873/ab6ce0
  * Kitchen et al. (2018) Kitchen, J. A., Sonnert, G., & Sadler, P. M. 2018, Science Education, 102, 529, doi: https://doi.org/10.1002/sce.21332
  * McWhinnie (2017) McWhinnie, S. 2017.  https://ras.ac.uk/ras-policy/community-demographics/demographic-survey-2017
  * Miller (2021) Miller, N. 2021, The International Astronomical Youth Camp: Diversity and Inclusion in the Context of Astronomy Education, Zenodo, doi: 10.5281/zenodo.4554514
  * Roberts et al. (2018) Roberts, T., Jackson, C., Mohr-Schroeder, M. J., et al. 2018, International Journal of STEM Education, 5, doi: 10.1007/s10984-016-9217-6
  * Vennix et al. (2017) Vennix, J., den Brok, P., & Taconis, R. 2017, Learning Environments Research, 20, 21, doi: 10.1007/s10984-016-9217-6


# DACB-Net: Dual Attention Guided Compact Bilinear Convolution Neural Network
for Skin Disease Classification

Belal Ahmad
School of Computer Science and Technology,
Huazhong University of Science and Technology,
Wuhan, China
<EMAIL_ADDRESS>
&Mohd Usama
Departments of Diagnostics and Intervention,
and Biomedical Engineering,
Umea University, Sweden
<EMAIL_ADDRESS>
&Tanvir Ahmad
School of Information and Control Engineering,
Xi’an University of Architecture and Technology,
Xian, China
<EMAIL_ADDRESS>
&Adnan Saeed
School of Civil and Hydraulic Engineering,
Huazhong University of Science and Technology,
Wuhan, China
<EMAIL_ADDRESS>
&Shabnam Khatoon
School of Management Science and Engineering,
China University of Geosciences,
Wuhan, China
<EMAIL_ADDRESS>
&Min Chen
School of Computer Science and Technology,
Huazhong University of Science and Technology,
Wuhan, China
<EMAIL_ADDRESS>

###### Abstract

This paper introduces the three-branch Dual Attention-Guided Compact Bilinear
CNN (DACB-Net) by focusing on learning from disease-specific regions to
enhance accuracy and alignment. A global branch compensates for lost
discriminative features, generating Attention Heat Maps (AHM) for relevant
cropped regions. Finally, the last pooling layers of global and local branches
are concatenated for fine-tuning, which offers a comprehensive solution to the
challenges posed by skin disease diagnosis. Although current CNNs employ
Stochastic Gradient Descent (SGD) for discriminative feature learning, using
distinct pairs of local image patches to compute gradients and incorporating a
modulation factor in the loss for focusing on complex data during training.
However, this approach can lead to dataset imbalance, weight adjustments, and
vulnerability to overfitting. The proposed solution combines two supervision
branches and a novel loss function to address these issues, enhancing
performance and interpretability. The framework integrates data augmentation,
transfer learning, and fine-tuning to tackle data imbalance to improve
classification performance, and reduce computational costs. Simulations on the
HAM10000 and ISIC2019 datasets demonstrate the effectiveness of this approach,
showcasing a 2.59% increase in accuracy compared to the state-of-the-art.

_K_ eywords Skin Disease Classification $\cdot$ Bilinear Convolutional Neural
Networks $\cdot$ Attention Heat Maps $\cdot$ Stochastic Gradient Descent.

##  1 Introduction

####

Skin lesions are deviations in the dermal structure that show abnormal growth
or appearance compared to the surrounding skin area. They are categorized as
either primary or secondary lesions based on their origin. Primary lesions
manifest as abnormal skin conditions present at birth, acquired during one’s
lifetime, or at the onset of an illness [1]. On the other hand, secondary
lesions result from modifications to a primary lesion caused by treatment,
disease progression, or manual intervention such as scratching, pinching, or
rubbing [2]. Depending on the type of skin lesion, dermatologists recommend
various treatments, including surgery, medications, or home care. Despite
their seemingly innocuous nature, some skin lesions may pose hazards,
signalling the potential presence of skin cancer and necessitating surgical
removal. Melanoma, in particular, stands out as the riskiest form of skin
cancer. While deadly when it spreads, melanoma is curable in its early stages.
Hence, accurate diagnosis of skin lesions is crucial to ensure timely
treatment.

####

Globally, approximately 123,000 melanoma cases and 3,000,000 non-melanoma
cases are reported annually. Factors like ultraviolet radiation and sun
overexposure are the main reasons for an annual increase in skin cancer cases
by 3% to 7%, translating to an additional 4500 melanoma cases [3]. A study
revealed that dermatologists with over ten years of experience achieved an 80%
accuracy in skin lesion diagnosis, while those with 3–5 years of experience
correctly diagnosed 62% of cases [4]. Various techniques are employed to
identify potentially dangerous lesions, including dermoscopy, complete
physical exams, biopsies, medical history analysis, skin exams, and monitoring
changes in lesion time, size, shape, and associated symptoms. Despite
stabilized mortality rates, skin cancer still imposes significant morbidity
and treatment costs [5].

####

To address these challenges, developing a Computer-Aided Diagnosis (CAD)
system becomes imperative, utilizing a unified measurement standard to assess
lesion severity. In recent years, integrating artificial intelligence and
computer technology has played a pivotal role in various disciplines. Medical
image processing, a multidisciplinary subject bridging medicine and computer
technology, has leveraged image processing technology, computer vision, and
other methods to standardize the analysis of medical images, contributing to
unified diagnostic results. DL-based classification methods have gained
prominence in the classification of skin diseases, offering a potential
solution to the complexities associated with skin lesion diagnosis. In recent
years, CNNs have played a crucial role in object classification [6] and
feature learning [7], significantly improving the performance of natural image
classification [5] and skin disease classification [8]. Nevertheless, current
methods often depend solely on global features extracted by the FC layer,
overlooking valuable local pattern features found in the intermediate layers
of Deep CNN (DCNN). Because of the evident intra-class variation and inter-
class similarity, the subtle differences in local patterns in the skin lesion
regions are critical for skin disease classification [9] [10].

####

Traditionally, two-stage approaches have been employed to enhance the
extraction of local skin lesion features. For instance, Moeskops et al. [8]
introduced an additional segmentation mask to train a segmentation network,
which was utilized to crop skin lesion regions from the entire image.
Subsequently, the classification network focused on the cropped images during
training, improving performance by emphasizing skin lesion region features.
However, this method faced limitations, such as the scarcity and expense of
segmentation mask annotations, demanding considerable professional knowledge
and labor. In another approach, Yu et al. [11] utilized the Fisher-Vector (FV)
encoding method to aggregate DCNN features from re-scaled input images. The FV
encoding features were then employed to train a chi-squared kernel-based SVM
for melanoma recognition. Yan et al [12] proposed an end-to-end network for
melanoma recognition. They incorporated an attention module to focus on skin
lesion local features within intermediate layers [13]. They employed an
additional segmentation mask as prior information to refine the attention map
decisively. However, using the segmentation mask for the entire skin lesion
region in the segmentation task while introducing intense supervision proved
effective in refining local pattern features due to significant variations in
scale, shape, and site across different stages and degrees of skin cancer.
Building on insights from Yosinski et al. [14], who demonstrated that
different feature channels of extracted CNN features can represent various
local pattern features in the original input image; a recent development is
GCNet [15]. GCNet introduced a lightweight non-local block to capture the non-
local-based global context in the feature map, facilitating the capture of
long-distance dependencies between pixels. Consequently, this approach
highlights discriminative local pattern features, addressing the limitations
of traditional convolution layers with limited receptive fields, especially in
capturing the evolving appearance of local patterns within changing skin
lesion regions.

####

To address the challenges, this paper introduces a novel Dual Attention
Mechanism (DAM)-based network with a novel loss function that combines
complementary entropy and cross-entropy for improved skin disease
classification. The novel DAM aims to capture meaningful local patterns within
the skin lesion region. In detail, the DAM spatially focuses on the multi-
scale skin lesion region features present in the intermediate layers of the
backbone network. Importantly, it achieves this without requiring complicated
pre-processing and automatically reduces irrelevant artifact features. Given
the substantial changes in skin lesions, the spatially filtered multi-scale
skin lesion region features are concatenated to form the output feature. To
further refine this output feature and extract meaningful local pattern
features within the skin lesion regions, the proposed approach deviates from
using an extra segmentation mask as intense supervision [12]. Inspired by the
research [14] [15], a Channel Attention (CA) module is introduced. This module
employs a lightweight non-local block to extract the non-local-based global
feature of the Spatial Attention (SA). The SA captures long-distance
dependencies between each pair of pixels in the SA output feature (SA_O).
Subsequently, the non-local-based global feature is utilized to generate a
feature channel reweighting vector. This vector is then applied to reweight
the different feature channels of the SA_O, forming the output feature of the
CA_O.

####

Consequently, the CA_O automatically highlights meaningful local pattern
features critical for skin disease classification. This not only enhances the
interpretability of the proposed network but also contributes to the efficient
utilization of small training data and optimization of the entire network.
Integrating the DAM and Channel Attention Module (CAM) addresses the
challenges posed by significant changes in skin lesions, providing a robust
framework for accurate and interpretable skin disease classification.

####

The proposed network combines two supervision branches and a new loss
function: CCE. The two branches employ the CA_O and the global feature
captured by the last convolutional layer as input, respectively. The image-
level labels for classification supervise both of these two branches. SGD is
mainly used in the discriminative feature learning for skin lesions, which
computes the gradient from distinct pairs of local image patches and the
modulation factor in the loss to focus the model on the training of complex
samples that also affect the imbalance of the dataset by adjusting the weight.
As a result, it fails to address the issue of achieving high accuracy in both
simple and complex models and may lead to vulnerability to overfitting.

####

This paper introduces a novel loss function based on complementary and cross-
entropy to address the earlier challenges. By leveraging training samples from
non-labeled categories, the method harnesses incorrect class information to
train a robust classification model, particularly adept at handling unbalanced
class distributions. This approach effectively mitigates the impact of uneven
data distribution on the network model’s predictive ability without increasing
the number of samples for specific categories. Consequently, the strategy
enhances learning opportunities for underrepresented sample types, compelling
the network model to prioritize diseased samples and improving overall
classification ability. Extensive experiments are conducted on the HAM10000
[16] and ISIC2019 [17] datasets to evaluate the performance of the proposed
network, demonstrating state-of-the-art results across both datasets. The
primary contributions of the proposed method are summarized as follows:

  * •

A novel DAM is proposed to highlight meaningful local patterns within skin
lesion regions. This mechanism enhances feature representation and
interpretability concurrently, contributing to the robustness of the proposed
network.

  * •

The DAM is combined with non-homologous BCNN and CCE. The DAM guides the
network to extract useful local pattern features in a weakly supervised
manner, providing strong regularization to prevent overfitting when training
on limited data.

  * •

Thorough experiments are conducted on the HAM10000 and ISIC2019 datasets
without additional training data. Results showcase the proposed network’s
state-of-the-art performance in skin disease classification, demonstrating
robustness and interpretability. These characteristics enhance the potential
clinical application of the proposed network.

##  2 Literature Review

####

Existing literature on automated skin disease classification primarily focuses
on exploring various CNN architectures, feature extraction techniques, and
attention mechanisms. Several studies have proposed innovative CNN-based
models designed specifically for skin disease diagnosis, utilizing transfer
learning, data augmentation, and ensemble methods to improve classification
performance. For instance, Josphineleela et al. [18] proposed a multi-stage
faster RCNN-based iSPLInception network to effectively classify benign and
malignant tumors with addressing overfitting issues. The method utilizes
faster RCNN for image classification, augmented with the iSPLInception model
to reduce computation time and network complexity. Inception-ResNet is applied
in iSPLInception for multi-stage classification. The prairie dog optimization
algorithm is evaluated to detect the candidate box. ISIC 2019 and HAM10000
datasets are used to conduct the experimental results. Compared the accuracy,
precision, recall, and F1-score with existing methods like CNN, hybrid DL,
Inception v3, and VGG19. The proposed method achieves 95.82% accuracy, 96.85%
precision, 96.52% recall, and 0.95 F1 score, demonstrating effective
prediction and classification. A bilinear CNN method is presented to classify
seven skin lesion classes with high accuracy and low computational cost [19].
The method includes data augmentation to address data imbalance, transfer
learning, and fine-tuning to enhance classification performance while reducing
computational load. Using the HAM10000 dataset, simulations demonstrated that
combining ResNet50 and VGG16 architectures improved accuracy by 2.7% over the
state-of-the-art methods. The proposed method achieved the best mean accuracy
of 93.21%, precision of 92.92%, recall of 93.00%, F1-score of 93.21%, and AUC
of 98.10% by 239 minutes of training.

####

Pundir et al. [20] introduced a skin cancer detection technique using
Bilinear-CNN-SA to detect malignant moles early which provides significant
patient benefits. Preprocessing steps include noise removal, grayscale
conversion, and image enhancement. Otsu segmentation is applied after data
cleanup. Feature extraction is done using the ABCD Rule, followed by
classification using the Bilinear CNN-SA model. Comparative analysis shows
superior performance compared to CNNs and Support Vector Machines (SVM) with
the proposed method. In [21], proposed a method based on Multi-Modal Bilinear
Fusion (MBF) with a Hybrid Attention (HA) mechanism for skin lesion
classification. MBF-HA employs a common representation learning framework to
capture correlated features between modalities. Additionally, it utilizes a
HA-based reconstruction module to enhance feature extraction and localize
lesion regions. Experiments on the 7-point Checklist database show MBF-HA
achieves an average accuracy of 76.3% in multi-classification tasks and 76.0%
in diagnostic tasks. Nigat et al. [22] employed CNN to classify four common
fungal skin diseases: tinea pedis, tinea capitis, tinea corporis, and tinea
unguium. They used a dataset comprising 407 fungal skin lesions from Gerbi
Medium Clinic, Jimma, Ethiopia. Image preprocessing included normalization,
RGB to grayscale conversion and intensity balancing. Images were resized to
$120\times 120$, $150\times 150$, and $224\times 224$, followed by
augmentation. Their model achieved a 93.3% accuracy in disease classification.
The proposed model achieved better results than MobileNetV2 and ResNet 50.

####

A CNN-based fruit classification system [23] is introduced which employs
bilinear pooling with heterogeneous streams. This framed fruit classification
as a fine-grained visual classification and developed a heterogeneous bilinear
network, then compared it with conventional approaches. The CNN network was
initialized with ImageNet weights and pre-trained networks as components in
the BCNN. These CNNs are used as feature extractors and combined with bilinear
pooling functions. The BCNN classifier is trained and tested with Fruits-360,
ImageNet, and VegFru datasets. The performance of the proposed model is
validated using various metrics by comparing some existing methods, the
proposed method achieved a classification accuracy of 99.69% and an F1 score
of 99.68%, outperforming other CNN models. A method [24] based on model fusion
with bilinear pooling by combining EfficientNet and DenseNet is proposed.
Image segmentation and feature extraction are performed using these two models
separately. Bilinear features are obtained by calculating the outer product
and average confluence of different spatial locations. The monkeypox skin
lesion dataset is used for the binary classification of monkeypox and non-
monkeypox. The model achieved the best accuracy of 94.57%, with a recall of
94.00% and an F1-score of 94.50%. Detection and classification can assist in
identifying monkeypox patients, which reduces the pressure of early diagnosis
and prevents the monkeypox virus from spreading too fast.

####

The authors [25] introduced a BCNN-based architecture for fine-grained
recognition, which represents images as pooled outer products of features from
two CNNs. This captures localized feature interactions in a translationally
invariant manner. BCNNs are orderless texture representations that can be
trained in an end-to-end manner. The most accurate models achieved per-image
accuracies of 84.1%, 79.4%, 84.5%, and 91.3% on the Caltech-UCSD Birds,
NABirds, FGVC Aircraft, and Stanford Cars datasets. Additionally, the method
systematically analyzed these networks and showed that: a) the size of the
bilinear features can be significantly reduced without loss of accuracy, b)
these are also effective for other tasks such as texture and scene
recognition, and c) it can be trained from scratch on ImageNet with consistent
improvements over the baseline. Visualizations of these models on various
datasets are presented using top activation and gradient-based inversion
techniques. Inthiyaz et al. [5] proposed an automated image-based method
utilizing machine learning to diagnose and categorize skin disease problems.
Computational techniques are employed to analyze, process, and organize
picture data, considering various image characteristics. Skin images undergo
noise removal and quality enhancement processes. Features are extracted using
CNN, images are classified using the Softmax classifier algorithm, and a
diagnostic report is generated. This system could serve as a dependable real-
time educational tool for dermatology researchers.

####

In [26], a novel approach, eDermaCare is introduced, which utilizes ResNet-50
for skin disease classification. In contrast, other CNN architectures produced
unsatisfactory results, ResNet-50 captures complex skin features well. The
focus was on enhancing classification through DL capabilities. The study
explores training and retention challenges, highlighting block balances to
prevent performance degradation. Ultimately, the model achieves 76.07%
classification accuracy. Sazzadul et al. [27] aimed to diagnose Eczema and
Psoriasis using deep CNN architectures. Five state-of-the-art CNN
architectures were employed, and their performance was assessed using 10-fold
cross-validation. Inception ResNet v2 with the Adam optimizer achieved a
maximum validation accuracy of 97.1%. The study presents two practical
application approaches: a) a smartphone-oriented approach, integrating CNN
models with a mobile application; and b) a web server-oriented approach,
integrating the CNN model with a web server for real-time skin disease
classification. Gautam et al. [28] attempted various techniques to detect skin
diseases early but obtained unsatisfactory results. Their study aims to
classify, detect, and provide accurate information about skin diseases. They
propose a high-performance CNN to achieve this goal. The methodology involves
preprocessing skin disease images, extracting important features, and
analyzing them using a deep CNN. Their approach achieves up to 98% accuracy in
detecting six different diseases, being simple, fast, and accurate.

##  3 Materials and Methods

####

This paper introduces a non-homologous BCNN for skin disease classification.
The model comprises three key components: a feature extraction module, a DAM
module, and a compact bilinear pooling module. The feature extraction module
is built upon ResNet-50 [29] and Xception [30]. Image features extracted by
these networks form a comprehensive data feature map, further enhanced by the
DAM. The resulting data feature maps from both CNNs are then directed to the
FC layer for classification following Compact Bilinear Pooling (CBP). The
network notably enhances feature extraction capability and classification
performance by combining two CNNs with distinct architectures and integrating
a DAM module.

###  3.1 Bilinear-CNN Architecture

####

A BCNN architecture consists of two equal or different CNNs (streams A, and B)
in a parallel manner. An input feeds both CNNs and a training step computes
the weights for each neuron and the tensor output of two models for the
features at each image location and then pooling to obtain the image
descriptor. The outer product and sum pooling for two CNN tensor outputs
combine feature vector at each location, i.e., for any location l of $i^{th}$
image is denoted as:

$\phi\left(c,\ f_{A},f_{B}\right)=\sum_{i=1}^{p,q}{U_{c,p,q,f_{A}}{\
V}_{c,p,q,f_{B}}}$ (1)

where $U\in\mathbb{R}^{c\times p\times q\times f_{A}}$ and
$V\in\mathbb{R}^{c\times p\times q\times f_{B}}$ are the result matrix from
$f_{A}$ and $f_{B}$, and c denotes the batch size, _p_ , and _q_ the spatial
dimensions of $f_{A}$ and $f_{B}$. The resulting matrix $\phi\in U^{(c\times
f_{A}\times f_{B})}$ captures pairwise correlations between feature channels.
The matrix offers richer representations compared to linear models. After
reshaping the output matrix $\phi$ to $c\times f_{A}\times f_{B}$, it is
normalized by signed square root normalization as given below:

$R=\text{sign}(\phi)\sqrt{|\phi|}$ (2)

where R is the signed square root matrix which is normalized by the _L2_ norm
to obtain the final value that passes to the FC layer as:

$S=\frac{R}{\left\|R\right\|^{2}}$ (3)

####

Finally, the output of the model passes through an FC layer with softmax
activation to derive predictions. The lack of large-scale labeled datasets can
make it challenging to train a DCNN from scratch, especially in the medical
image classification domain. Transfer learning is an alternate method that
uses a pre-trained model to start a new task. This method enables quick
development and improved modeling performance for the next task.

####

The BCNN adopted in this study involves integrating a ResNet-50 [29] in one
arm and the Xception network [30] in the other. FC layers are removed from
each model before their combination into the BCNN. Following an outer product
operation, a single FC layer is adjusted for the classification, incorporating
seven neurons instead of the original 1000.

####

ResNet-50 and Xception models, pre-trained on ImageNet [31] and available in
the Keras library, are employed for this study. A seven-neuron layer replaces
the FC layer with 1000 neurons, and to retrain the weights for the target task
without random initialization, the average pooling layer is removed.
ResNet-50, a DCNN, has demonstrated high performance in computer vision tasks,
clinching the ImageNet challenge in 2015. Noteworthy for learning residual
representation functions instead of direct signal representation, ResNet
employs skip connections to combat vanishing gradient issues, promoting model
convergence. Specifically chosen for its superior performance in this context,
ResNet-50 initiates with a 7 × 7 kernel size for the initial convolution,
followed by Max-pooling with a 3 × 3 kernel size. The model consists of four
stages, each with 2, 4, 6, and 3 residual blocks. Each residual block
comprises three convolutional layers with kernel sizes of 1 × 1, 3 × 3, and 1
× 1. The 1 × 1 convolution layer serves for dimension reduction and
restoration, while the 3 × 3 layer acts as a bottleneck with smaller
input/output dimensions. Concluding the architecture is an average pooling
layer followed by an FC layer with 1000 neurons. In total, ResNet-50 boasts
over 23 million trainable parameters.

####

Conversely, Xception is a deep CNN boasting 36 convolutional layers,
surpassing ResNet in the ImageNet challenge. With a total of up to 22 million
parameters, the Xception model consists of 14 modules arranged into three
stages: the entry flow, the middle flow, and the exit flow. The entry flow
starts with four modules. The first module has an input layer with dimensions
of 299 × 299, and then there are two convolutional layers with kernel sizes of
3 × 3. Modules that come after that have two separable convolutional layers
with a 3 × 3 kernel size, a 3 × 3 max pooling layer, and a linear residual
connection coating every module. A middle flow iterates eight times over a
module with three separable convolutional layers, each time using a 3 × 3
kernel size and incorporating a residual connection around the module. Two
separable convolutional layer modules with a 3 × 3 kernel size, a 3 × 3 max
pooling layer, and a residual connection are the first components of the exit
flow. Finally, a Global Average Pooling (GAP) layer is used before two
separable convolutional layers. Fine-tuning is applied to enhance performance
by freezing 70% of layers for both the ResNet-50 and Xception models. The
weights of the remaining layers are adjusted, allowing for meaningful
improvements by iteratively adapting pre-trained features to the new data.

###  3.2 Compact Bilinear Pooling and Non-Homologous BCNN

####

Traditional CNNs are unable to extract distinct features from the input skin
disease images due to their fine-grained nature. For this reason, the bilinear
pooling [25] described in this paper works well for classifying skin disease
images. The BCNN treats features from two different information streams as
unique and combines high-order statistical data. The model effectively
captures connections between the two streams by performing outer product and
pooling operations on these features, yielding global features. This approach
leverages image translation invariance to model local dual features
dynamically. The discriminative features obtained through BP are then used for
image classification for better results. Two CNNs with differing structures
are ultimately employed to extract various image features.

####

While bilinear pooling performs well in image classification, the bilinear
features generated are often high-dimensional, ranging from hundreds of
thousands to millions. Such ineffective representations significantly increase
memory usage, rendering them unsuitable for subsequent analysis. Compact BCNN
is employed to conduct core analysis on bilinear features to reduce the
dimension. Compact BCNN retains the discriminative capacity of Bilinear
Pooling (BP) but with reduced dimensions in the thousands. It enables the
backpropagation of misclassified samples, facilitating end-to-end optimization
of image classification. A compact bilinear representation is derived through
core analysis, relying on low-dimensional feature maps of the kernel function.
The bilinear pooling model forms a global image descriptor through a sequence
of the following calculations:

$blinear\left(i,l,f_{A},f_{B}\right)=f_{A}(i,l)^{T}f_{B}(i,l)$ (4)

####

It is essential to appropriate feature representation for a specific model,
which requires feature vectors $f_{A}$, and $f_{B}$ to be equal feature
dimensions. The pooling function generates the global feature representation
$\phi(l)$ for an image by accumulating the bilinear feature combinations at
every location of an image. In our method, we used pooling sum as shown below:

$\phi(l)=\sum_{l\in L}{bilinear\left(i,l,f_{A},f_{B}\right)}$ (5)

####

The global feature representation $\phi(l)$ for an image is order-less due to
ignoring the feature locations by pooling; hence, for every spatial location
$l$ in a location space $L$ of image $i$, we compute the bilinear feature
vector. If $f_{A}$ and $f_{B}$ extract corresponding local feature vectors of
size $Y\times N$ and $Y\times M$, then their bilinear combination $\phi(l)$
will have size $N\times M$. The similarity function $S$ uses the general-
purpose bilinear vector representation. Instinctively, the outer product
indicates the pairwise interactions of the feature extractors $f_{A}$ and
$f_{B}$ on each other. To appropriate vector representation of a particular
model, $f_{A}$ and $f_{B}$ need to be equal feature dimension $Y$ as
$f:l\times L\rightarrow R^{Y\times D}$ of size $Y\times D$, where $l\in L$
represents the location of any image $i\in l$. The pooling function generates
the global feature representation $\phi(l)$ of image $i$ by accumulating the
bilinear feature combinations at every location of an image as formulated by
Equation 5. $\phi(l)$ symbolizes a vector projection with low dimensions;
$f_{A}$ and $f_{B}$ represent feature vectors. If we represent the second-
order polynomial kernel function using $\mathit{k}\left(f_{A},f_{B}\right)$,
then, to make the value of $d$ considerably lower than $c^{2}$, we can find a
low-dimensional projection function $\phi\left(f_{A}\right)\in R^{d}$ that
fulfills Equation 6:

$\phi\left(f_{A}\right),\
\phi(f_{B})\approx\mathit{k}\left(f_{A},f_{B}\right)$ (6)

where $\mathit{k}\left(f_{A},f_{B}\right)$ denotes the low-dimensional vector,
and $\phi\left(f_{A}\right)\in R^{d}$ denotes a projection function. The
$c^{2}$-dimensional inner product of the original is approximately equal to
the inner product reduced to the $d$-dimension. The representation that
follows can be obtained by:

$\begin{split}\left(\mathfrak{b}(A),\mathfrak{b}(B)\right)=\sum_{l\in
L}{{{{(f}_{A}(i,l),\ f}_{A}(i,l)}^{T})}\sum_{l\in L}{{{(f_{B}(i,l),\
f}_{B}(i,l)}^{T})}\\\ =\sum_{i\in l}{\sum_{l\in
L}\left(\phi\left(f_{A}\right),\ \phi\left(f_{B}\right)\right)}=(\upsilon(A),\
\upsilon(B))\end{split}$ (7)

where $A$ and $B$ are the set of local descriptors, $l$, and $L$ is the set of
spatial positions, $A_{i}$ and $B_{l}$ represent local descriptors, and
$\upsilon(A)$ and $\upsilon(B)$ represent low-dimensional vectors after
$\mathfrak{b}(A)$ and $\mathfrak{b}(B)$ are mapped by the mapping function;
where $\upsilon(A)=\sum_{i\in l}{\phi(A_{i}),}$ and $\upsilon(B)=\sum_{l\in
L}{\phi(B_{l})}$ are the compact bilinear features, $i$ and $l$ are the number
of channels of the vector $\mathfrak{b}(A)=\sum_{i\in l}{{f_{A}(i,l),\
f}_{A}(i,l)}^{T}$, and $\mathfrak{b}(B)=\sum_{l\in L}{{f_{B}(i,l),\
f}_{B}(i,l)}^{T}$. Any polynomial kernel can provide a low-dimensional
approximation function to reduce dimensionality.

###  3.3 Dual Attention Mechanism

####

In addition to embedding an attention mechanism with ResNet-50 and Xception
for the weight allocation of the input features, the Convolutional Block
Attention Module (CBAM) [32] is further improved in this paper. It proposes a
new attention mechanism to enhance the accuracy of ResNet-50 with Xception in
performing classification to the fullest extent. This section introduces the
improved CBAM in detail.

####

Building upon the BCNN, this paper introduces an enhancement to the CBAM by
incorporating bi-linearity principles, resulting in a bilinear CBAM. Moreover,
the feature information from channels and spatial dimensions is integrated to
reinforce the CAM within the CBAM. To further refine the algorithm, the
linearity of the BCNN is considered, retaining the original maximum pooling
and average pooling in parallel to preserve the initial map’s feature
information comprehensively. The output result concatenation is subsequently
executed based on the channels. The FC layer in the original structure
transforms into a $1\times 1$ convolutional layer with layer normalization
applied for batch data processing. Given that both ResNet-50 and Xception
utilize the ReLU activation function consistently, the CAM’s ReLU function is
incorporated with random regularity. The structural representation of this
configuration is depicted in Figure 1.

Figure 1: Improved CAM mechanism.

####

The input data processing by the above-mentioned network is shown in Equation
8:

$\begin{split}F_{A1}=\ \mu(Conv(ReLU(LN(Conv(Concat(AP(F),MP(F)))))))\\\
=\mu(W1(ReLU(LN(W_{0}(Concat(F_{avg}^{c},\ F_{\max}^{c}))))))\end{split}$ (8)

where $\mu$ and ReLU are activation functions, $Conv$ is the convolution, $AP$
is the Average Pool, $MP$ denotes the Max Pool, and $F_{A1}$ is the feature
map produced by the CAM. $F\in R^{(B,2C/r,W,H)}$ follows the application of
the ReLU activation function, while $F\in R^{(B,C,W,H)}$ follows the
application of the sigmoid activation function.

####

Moving forward, the Spatial Attention Module (SAM) is infused with linearity.
He et al. [33] introduced a spatial attention method utilizing $1\times 1$ and
$3\times 3$ convolutions to extract diverse and abundant feature information.
In this paper, we build upon this method for enhancement. Initially, $1\times
1$ and $3\times 3$ convolution operations are conducted on the input feature
map to obtain spatial features of varying scales. Subsequently, layer
normalization and ReLU mappings are applied to the two features. The
dimensions of the features are then reduced to single channels using $1\times
1$ convolution. Multiplying these two single-channel features produces multi-
scale spatial attention features. These features are further multiplied by the
feature map with channel attention weight, resulting in a feature map
encompassing both channel weight and spatial features. Batch processing is
implemented for data using layer normalization for two primary reasons. First,
this approach aligns with the data processing methodology of ResNet-50 with
Xception, building upon their research results. Second, it follows the
consistent use of batch normalization in CNNs, with the added benefit of
independence from the hardware constraints associated with batch size.

####

In selecting convolution kernel sizes, $1\times 1$ and $3\times 3$
convolutions are employed. Beyond capturing spatial features at different
scales, using $1\times 1$ convolution reduces the overall network’s complexity
and computational load. Furthermore, GPUs often optimize the analysis of
$3\times 3$ convolutions, contributing to improved efficiency. Optimizations
and modifications are executed based on the considerations outlined above. The
improved structure chart of the SAM is depicted in Figure 2.

Figure 2: Improved SAM mechanism.

####

The input data is processed by the network described above as Equations 9-11:

${F}_{A2}=Sigmoid\left(F_{1\ }\oplus\ F_{2}\right)$ (9)

${F}_{1\ }={Conv}_{\\_(1\times 1)}(ReLU(LN({Conv}_{\\_(1\times 1)}(F))))$ (10)

$F_{2\ }={Conv}_{\\_(1\times 1)}(ReLU(LN({Conv}_{\\_(3\times 3)}(F))))$ (11)

where $F_{A2}$ is the feature map generated by the SAM; $Conv_{(1\times 1)}$
is a convolution with a $1\times 1$ convolution kernel; $Conv_{(3\times 3)}$
is a convolution with a $3\times 3$ convolution kernel; $F\in R^{(B,C/r,W,H)}$
follows the ReLU, $F\in R^{(B,1,W,H)}$ follows the sigmoid, and $\oplus$
denotes the concatenation operation.

####

The conventional attention mechanism exclusively assigns weights between
channels without considering the spatial dimension, so the DAM is constructed
with two branches. The first branch provides channel weights, while the second
offers spatial weights (aimed at giving channel weights). The initial
convolutional DAM employs a serial processing method. In improving this
mechanism, particularly for classification tasks, the network is more precise
by extracting feature attention. Consequently, the serial processing approach
replaces parallel processing, enabling simultaneous channel and spatial
attention training. This adjustment ensures the retention of sufficient
semantic information while acquiring multi-scale features for the target.
Figure 3 shows the specific structure of the attention network The DAM, as
mentioned above, can be summarized by:

$F_{o}\ =\ (F_{A1}\ \oplus\ F_{in})\ \oplus\ F_{A2}\ \oplus\ F_{in}$ (12)

where $F_{in}$ represents the input feature map of BCDAM, $F_{o}$ represents
the feature map of the output BCDAM, and $F_{A1}$ and $F_{A2}$ represent the
feature map generated by the CAM and SAM, and $\oplus$ denotes the
concatenation operation.

Figure 3: Structure of dual attention mechanism.

###  3.4 DACB-Net with CCE for Skin Disease Classification

####

A method of two embedding attention mechanisms with ResNet-50 and Xception is
proposed in this paper. A detailed implementation is conducted on the specific
attention framework for this position to thoroughly investigate the embedding
position of the attention mechanism within the entire network. This section
introduces a novel multi-perspective attention framework. The shortcut
attention branch incorporates the DAM proposed in this paper and the feature
map size passing through each module.

Figure 4: Bilinear attention mechanism model.

####

The Efficient Channel Attention (ECA) combined with the attention architecture
shown in Figure 4 considers each channel, the spatial dimension of the feature
map, and the information interaction between channels \cite{Wang2020}. Due to
the dimension changes in convolution, ECA covers up for the lost interaction
information between channels. Using a feature map that does not cycle network
blocks, this method balances relationships along various feature map
directions. It retains a significant degree of channel interaction information
from the original feature map, enabling the attention mechanism to address
deficiencies during training and achieve comprehensive and multi-perspective
attention parameter training.

Figure 5: Proposed DACB-Net architecture.

####

Furthermore, when elevating the original ResNet-50 with Xception to construct
a deeper architecture, this architecture aggregates the feature map with the
attention output before the input block, which takes shape in channel
relationships. This aids the network in extracting and focusing on sample
features. Additionally, the ReLU activation function is omitted after adding
the original feature map and attention feature map, contributing to an
enhancement in the final classification accuracy of the network. The ReLU
activation function is also removed in the subsequent comparison experiment
network. Figure 5 indicates how the DACB-Net proposed in this paper is
embedded into the ResNet-50 with the Xception network through SBA.

###  3.5 Unbalanced Sample Distribution Problem

####

In clinical medicine, a minuscule proportion of patients exhibit serious
illnesses compared to the overall number of individuals examined. This leads
to a data set with a pronounced imbalance, presenting a significant
distribution challenge. The class with more samples dominates the training
process when using imbalanced data for network learning [5]. Consequently, the
trained classification models often perform well in classes with a large
sample size but poorly in classes with a smaller sample size. To address the
degradation of network performance due to data imbalance in datasets, the
Complement Cross Entropy (CCE) is used. CCE tackles the problem by leveraging
the training of non-labelled category samples. It eliminates the need to
increase the sample count or the size of minority losses. Instead, this method
utilizes information from incorrect classes to robustly train a classification
model under unbalanced class distribution. Consequently, this strategy offers
improved learning opportunities for categories with a limited sample size. It
achieves this by encouraging the correct category (including a few categories)
to have a higher Softmax score than all other incorrect categories. The CCE
loss function is calculated from the average value of each batch’s sample
entropy of the wrong class. The CCE is defined as Equation 13:

$L(y,\widehat{y})=-\frac{1}{N}\sum_{i=1}^{N}\mspace{2.0mu}\sum_{j=1,j\neq
g}^{K}\mspace{2.0mu}\frac{{\widehat{s}}^{(i)[j]}}{1-{\widehat{s}}^{(i)[g]}}\log\frac{{\widehat{s}}^{(i)[j]}}{1-{\widehat{s}}^{(i)[g]}}$
(13)

where $g$ is the sequence number of the ground truth category,
$\widehat{s}(i){[}j{]}$ is the prediction of the $i^{th}$ sample in class $g$,
$N$ is the number of samples in the mini-batch, $P$ is the total number of
categories, $s$, and $\widehat{s}$ are the feature map and its estimated
probability vectors of the given sample. This entropy aims to predict the
probability that the ground truth class will have greater significance than
the wrong classes. This is achieved by distributing the Softmax scores among
the incorrect classes. In other words, there is higher confidence in guessing
the true class when there is a more neutral prediction probability
distribution for the false class. Since the complement entropy achieves its
maximum when the probability distribution is uniform, the optimizer aims to
maximize it. The complementary entropy and cross-entropy scales align with the
balanced complement entropy is defined as:

$\widehat{L}\left(s,\widehat{s}\right)=\frac{1}{P-1}L\left(s,\widehat{s}\right)$
(14)

where the balancing coefficient is $1/(P-1)$, the adjustment factor $g$ is
added to balance the cross entropy and complement entropy. Equation 15 shows
the final loss function:

$C=\widehat{H}\left(s,\widehat{s}\right)+\frac{\beta}{P-1}L(s,\widehat{s}).$
(15)

####

The advantage of loss over focal loss is that its internal connection may be
balanced without the need to establish weights. The loss function can solve
the problem of sample imbalance by examining the relationship within the
sample.

##  4 Dataset and Evaluation

###  4.1 Dataset Acquisition and Description

####

The proposed method is evaluated on two publicly available benchmark datasets:
HAM10000 [16] and ISIC2019 [17]. The details of the datasets are described in
the following.

#### HAM10000:

It consists of a total of $10015$ dermoscopic images. Seven types of skin
lesions are included in this dataset as shown in Figure 6: $BCC$, $BKL$, $DF$,
$MEL$, $NV$, $VASC$, and $AKIEC$. However, all classes are not balanced;
therefore, we performed data augmentation for the final classification. This
dataset consists of $514$ in $BCC$, $1099$ in $BKL$, $115$ in $DF$, $1113$ in
$MEL$, $327$, $6705$ in $NV$, $142$ in $VASC$, and $327$ images in $AKIEC$
class. The number of images shows that the dataset is unbalanced. Therefore,
we augmented the input data before training. In this work, we use 70% of the
images for training, the rest 15% for validation, and 15% for testing. Select
all the images in a randomized process.

Figure 6: Seven skin lesion classes in the HAM10000 dataset.

#### ISIC2019:

There are $25331$ labelled images of $8$ categories in the ISIC2019, which
split into $867$ Actinic keratosis (AK), $3323$ Basal cell carcinoma (BCC),
$2624$ Benign keratosis (BKL), $239$ Dermatofibroma (DF), $4522$ Melanoma
(MEL), $12875$ Melanocytic nevus (NV), $628$ Squamous cell carcinoma (SCC),
and $253$ Vascular lesion (VASC). We divide each class of the ISIC2019 into
three parts with identical data distributions, in terms of 80% for training
data, 10% for validation data, and 10% for test data. So the training,
validation, and testing parts own $20269$, $2531$, and $2531$ images,
respectively.

Table 1: Dataset information of HAM10000 and ISIC2019. Dataset | Class | Before (Data Augmentation) | After (Data Augmentation)
---|---|---|---
HAM10000 | Basal Cell Carcinoma (BCC) | 514 | 1750
Benign Keratosis (BKL) | 1099 | 1750
Dermatofibroma (DF) | 115 | 1750
Melanoma (MEL) | 1113 | 1750
Melanocytic Nevus (NV) | 6705 | 1750
Vascular Lesion (VASC) | 142 | 1750
Actinic Keratosis (AKIEC) | 327 | 1750
| Total | 10015 | 12250
ISIC2019 | Actinic Keratosis (AKIEC) | 867 | 1750
Basal Cell Carcinoma (BCC) | 3323 | 1750
Benign Keratosis (BKL) | 2624 | 1750
Dermatofibroma (DF) | 239 | 1750
Melanoma (MEL) | 4522 | 1750
Melanocytic Nevus (NV) | 12875 | 1750
Squamous Cell Carcinoma (SCC) | 253 | 1750
Vascular Lesion (VASC) | 628 | 1750
| Total | 25331 | 14000

###  4.2 Dataset Pre-Processing and Augmentations

####

As previously mentioned, some classes in the dataset have low sample sizes.
Therefore, data augmentation is needed to solve this problem. However, before
that, the dataset was split into a train-test-split, and augmented samples
were not included in the test set.

####

For training, 80% of the data was used, and the remaining 80% for testing. A
total of 6 augmentation techniques, which were Rotation, Zoom, Vertical Flip,
Horizontal Flip, Brightness, and Shear, were performed for the training set as
shown in Figure 7. Table 1 shows the number of input samples taken from each
class before and after augmentation for the HAM10000 and ISIC2019 datasets,
and the total number of samples used for each splitting technique is shown in
Table 2.

Figure 7: Data augmentation results with (a) Zoom, (b) Rotation, (c) Width Shift, (d) Height Shift, (e) Horizontal Flip, (f) Vertical Flip of the HAM10000 dataset. Table 2: Number of samples for all splits. Splitting and Augmentation | No. of Train Samples | No. of Test Samples
---|---|---
HAM10000 | ISIC2019 | HAM10000 | ISIC2019
(Without Augmentation) | 7010 | 17732 | 3305 | 7599
(With Augmentation) | 8575 | 9800 | 3675 | 4200
(Without Augmentation) | 8012 | 20265 | 2003 | 5066
(With Augmentation) | 9800 | 11200 | 2450 | 2800

###  4.3 Model Training

####

Six models trained on two dataset splits were collected in the previous steps
for the present study. The proposed model reached convergence in about 320
minutes on the NVIDIA GTX 980 Ti GPU occupying less than 4GBs of GPU memory to
accelerate the computation for full implementations Each model was trained to
50 epochs to evaluate which model performed well. Learning rate and weight
decay change depending on the model’s architecture. We use the Adam [34]
optimizer with default momentum values (0.9, 0.999) for $\beta 1$ and $\beta
2$. The weight decay is 0.0001. The learning rate initializes to $1\times
10^{-3}$ for both HAM10000 and ISIC2019 datasets. The proposed model was the
best-scoring model on 50 epochs. The extensive training took the previous
testing accuracy of 50 epochs from 92.86 % to 94.24 %.

###  4.4 Classification Metrics

####

Initially, we computed the confusion matrix to get all classes TP, TN, FP, and
FN scores. All four measures are calculated as one against other categories.
Six out of the seven behaved as negative class for HAM10000. Similarly, seven
out of the eight are considered to be the negative class for ISIC 2019. If the
score is computed for any class. TP denotes the true positive and accurate
prediction for any class the model makes. FP denotes the false positive and
false prediction for any class. Similarly, TN and FN represent true negative
and false negative. These metrics are computed and combined to comprehensively
evaluate the model’s performance, such as precision, recall, F1-score, and
accuracy.

####

Evaluate the performance using early-discussed measures, such as precision,
recall, F1 score, and accuracy. The total counts of true positive are divided
by the overall true positive + false positive and calculated by Equation 16:

$\ \ \ \ \ Precison=\frac{TP}{TP+FP}.$ (16)

####

To compute recall, divide the number of true positives by the total number of
true positives and false negatives, as computed in Equation 17:

$\ \ Recall=\frac{TP}{TP+FN}.$ (17)

####

The F1-score is the mean of precision and recall when both metrics are used is
shown as:

$\ F1-score=\frac{2\times(Precision\times Recall)}{Precision\ +\ Recall}.$
(18)

##  5 Experiments and Results

####

This section assesses the effectiveness of the skin lesion classification
method. Specifically, we categorize lesions from the HAM10000 and ISIC2019
datasets, apply data augmentation techniques, and quantitatively assess
performance using six standard metrics in skin disease classification.
Ultimately, we compare the results we obtained against other state-of-the-art
methods [35] [17] [36] [37] [38] [39] [40] [41] [42] [43] [44] [45] [46] [47]
[48] [49] [50] as shown in Table 6.

###  5.1 Ablation Study

####

The BCNN has proven highly effective in image classification [51] [52] [53].
This paper introduces a method based on bilinear pooling operation for skin
disease classification from the perspective of fine-grained image
classification. To assess the performance improvement of the model, we applied
compact BCNN with DAM and CCE on two benchmark skin disease image datasets.
All the methods used the same experimental steps in the experiment, employing
the same data stream except for one variation. The experimental results in
Table 3 and Table 4 indicate that compact BCNN with and without attention
enhances the classification accuracy of the model. The results suggest that
compact BCNN, with attention, is a notably effective method in medical image
classification and well-suited for skin disease classification.

Table 3: Classification results (Average) of the proposed model on HAM10000. Method | Precision (%) | Recall (%) | F1-score (%) | Accuracy (%)
---|---|---|---|---
ResNet-50 + CCE | 89.80 | 92.78 | 90.61 | 87.69
Xception + CCE | 89.14 | 89.23 | 89.48 | 89.08
ResNet-50 + ResNet-50 + CBP + DAM + CCE | 90.08 | 90.45 | 91.54 | 89.74
Xception + Xception + CBP + DAM + CCE | 91.92 | 93.83 | 94.88 | 91.36
ResNet-50 + Xception + CBP + CCE | 89.86 | 89.12 | 93.93 | 91.47
Proposed | 91.72 | 92.34 | 93.03 | 92.86
Table 4: Classification results (Average) of the proposed model on ISIC2019. Method | Precision (%) | Recall (%) | F1-score (%) | Accuracy (%)
---|---|---|---|---
ResNet-50 + CCE | 88.67 | 87.92 | 92.21 | 87.76
Xception + CCE | 88.38 | 89.54 | 91.40 | 88.43
ResNet-50 + ResNet-50 + CBP + DAM + CCE | 91.23 | 89.76 | 92.08 | 88.63
Xception + Xception + CBP + DAM + CCE | 89.22 | 94.65 | 93.67 | 89.20
ResNet-50 + Xception + CBP + CCE | 95.64 | 91.36 | 89.11 | 90.14
Proposed | 92.56 | 99.2 | 95.38 | 94.24
Table 5: DACB-Net trained with the different loss functions. Loss Function | Accuracy (%)
---|---
HAM10000 | ISIC2019
BCE | 91.68 | 91.29
Focal Loss | 92.62 | 92.77
CCE | 93.86 | 94.24

####

Following the same experimental steps, further experiments were conducted on
the dataset using homologous and non-homologous BCNN. The results show that
non-homologous BCNN achieves better classification accuracy than non-
homologous BCNN by 1.5% for HAM10000 and 0.90% for the ISIC2019 dataset. This
suggests that non-homologous BCNN is a superior method in the grading
experiment of skin disease images. Non-homologous BCNN concurrently trains two
CNNs with different parameter weights, utilizing various data streams to
recognize distinct image features. This approach proves more beneficial in
extracting specific disease image features.

####

We explored by replacing one of the two ResNet-50 networks in the non-
homologous BCNN with the Xception. The comparative analysis under a similar
experimental setup is detailed in Tables 3 and 4. The results indicate that
combining different CNNs in BCNN demonstrates their ability to extract more
distinctive features from the dataset images, producing better results. Our
investigation compared cross-entropy loss [44] and focal loss [44] with CCE
loss. All three experiments were performed in the same setup, except for the
different loss functions and results outlined in Table 5. The results show
that CCE loss surpasses focal loss and BCE in classifying uneven distribution
in datasets.

####

We evaluated the proposed network on the HAM10000 and ISIC2019 datasets. In
addition, these two experiments use the same experimental procedures. It
improves the accuracy of the network from 92.86% to 94.24%. The performance of
the skin disease classification is significantly affected by the attention
mechanism. As previously mentioned, the attention mechanism can classify
features extracted by the CNN channel by channel, determining dependencies
between each channel. Implementing two attention mechanisms enables the
network to allocate varying degrees of attention to different image features.
The image features enhance the expressive ability of the model to obtain
effective results.

####

Figure 8 visually represents the attention of the proposed model for different
regions in the input image after utilizing the attention mechanism. An
original image from the input dataset is shown on the left. In contrast, the
images on the right visualize the attention of the network to different
regions of the images. In the AHM by Grad-CAM, red denotes a feature in the
relevant region (higher weight), while blue indicates the feature’s low weight
in the corresponding area.

Figure 8: The visualization of the ablation test with the AHM by Grad-CAM on
the test set. In the Figure, from left to right, the first column is for the
original images, and the second, third, and fourth columns are for ResNet-50 +
DAM, Xception + DAM, and Proposed model (DACB-Net), respectively. The higher
weight shows a higher thermal value that is classified more correctly.

###  5.2 Analysis of DACB-Net

####

The proposed model was trained and evaluated with five folds to check its
effectiveness for classification. The architecture details are described in
the above section. The model was trained with an Adam optimizer for 50 epochs
on each fold. The accuracy and loss curves for the training and validation
data are shown in Figures 9 \- 14.

####

Although the training accuracy increased gradually during the training, it
started initially faster and later slowed significantly. With the training,
the validation accuracy fluctuated while it was increasing. The validation
curves of the proposed model are relatively less staggered than the compact
bilinear model without an attention mechanism.

####

The validation curve is relatively less staged than the compact BCNN without
DAM. It reaches the saturation points at around 50 epochs. The margin between
training and validation curves was minimized compared to the compact BCNN
without DAM. Using the validation data, the proposed model achieved a
validation accuracy of 94.24%, with a variance of ± 3.4% over the five folds.
This confirms that when test data is unknown, the model fitting performs
better at generalizing than compact BCNN without DAM.

Figure 9: Trend of loss and accuracy for BCNN without attention. Figure 10:
Trend of loss and accuracy for BCNN with attention. Figure 11: Trend of loss
and accuracy for DACB-Net for HAM10000. Figure 12: Trend of loss and accuracy
for BCNN without attention on the ISIC2019 dataset. Figure 13: Trend of loss
and accuracy for BCNN with attention. Figure 14: Trend of loss and accuracy
for DACB-Net for ISIC2019.

###  5.3 Performance Analysis

####

The ROC curves depicting the classification of all classes in the HAM10000 and
ISIC2019 datasets are shown in Figure 15 \- 17. The AUC values corresponding
to each curve are indicated in the labels, along with micro and macro
averages. The micro-average considers contributions from all classes to
compute the average metric, while the macro-average calculates the metric
independently for each category and then determines the average. Notably, in
both bilinear methods, the extreme points of curvature in the graphs are
positioned close to the upper left corner, suggesting a good separation
capacity of the evaluated model. Substituting the ECA block in the proposed
architecture significantly reduced model complexity and trainable parameters.
The proposed model achieved 92.86 % and 94.24% accuracy on the HAM10000 and
ISIC2019 datasets, respectively.

####

The confusion matrix, a quantitative-visual representation of classification
results, is displayed for the proposed model in Figure 18 for (a) HAM10000 and
(b) ISIC2019 datasets. The vertical axis represents true labels in these
matrices, and the horizontal axis denotes predicted labels. Ideally, identity
matrices are expected, indicating the perfect classification of all images.
Errors in classification are reflected outside the main diagonal. For
instance, in Figure 18 (a), a value of 339 was obtained for the "BCC" class
due to some images being predicted as "BKL". Similar confusion between the
"AKIEC" and "MEL" classes is evident in Figure 18 (b). Additionally, there
were labelling errors for "BKL", predicted as "BCC" and "BKL". Despite these
errors, both confusion matrices indicate highly accurate results.

Table 6: Comparison with state-of-the-art methods. Method | Precision (%) | Sensitivity (%) | Specificity (%) | F1-score (%) | Accuracy (%)
---|---|---|---|---|---
MobileNetV2-LSTM [35] | - | 88.24 | 92.00 | - | 85.34
Kassem et al. [17] | - | 74.00 | 84.00 | - | 81.00
Raju et al. [36] | 91.18 | 92.20 | 91.92 | 92.28 | 92.81
Bharat et al. [37] | 94.46 | 94.92 | 94.83 | 93.13 | 93.95
Kumar et al. [38] | 95.72 | 95.71 | 95.21 | 95.25 | 94.19
Brinker et al. [39] | - | 88.88 | 91.77 | - | 90.33
MKhan et al.[40] | - | 88.50 | 91.00 | - | 88.20
MobileNetV2 [41] | - | 85.00 | 85.00 | - | 85.00
Prathiba et al. [42] | - | 85.30 | 99.30 | - | 85.50
MWNL-CLS [43] | - | 56.40 | 76.00 | - | 76.30
InceptionV3 [44] | 88.00 | 75.00 | - | 81.00 | 89.81
Semi-supervised [45] | 88.00 | 71.47 | - | 60.68 | 92.54
MobileNet [46] | 87.00 | 81.00 | - | 84.00 | 92.70
EfficientNetB1[47] | 94.00 | 94.00 | - | 94.00 | 94.00
CSLNet [48] | 92.56 | 99.20 | 97.57 | 89.75 | 89.58
Reddy et al. [49] | 96.15 | 91.83 | 96.47 | 93.94 | 94.20
DBN Small [50] | 85.89 | 84.20 | 97.58 | 85.03 | 91.58
Proposed | 94.56 | 99.20 | - | 95.38 | 94.24

###  5.4 Comparison

####

To examine the concern of the proposed model, we compared the results of the
proposed method with some existing methods for skin disease classification.
Performance comparison metrics included precision, recall, F1-score, and model
accuracy. Table 6 presents the resulting observations, and the accuracy and
F1-score of the proposed method achieved the best results compared to the
existing methods, whereas the precision of [49], and specificity of [42] is
more than the DACB-Net, but their sensitivity and accuracy is comparatively
very low. So, it was concluded that the proposed DACB-Net achieved the best
results compared to the existing methods.

Figure 15: ROC Curve of ResNet-50 and Xception without attention for (a)
HAM10000 and (b) ISIC2019. Figure 16: ROC Curve of ResNet-50 and Xception with
attention for (a) HAM10000 and (b) ISIC2019. Figure 17: ROC Curve of DACB-Net
for (a) HAM10000 and (b) ISIC2019 dataset. Figure 18: Confusion matrices of
the proposed model on (a) HAM10000 and (b) ISIC2019.

##  6 Conclusion and Future Work

####

In this paper, we address the skin disease image classification problem. The
final classification accuracy of the proposed network reached a state-of-the-
art level. In this network, we used a compact bilinear attention mechanism by
improving the CAM and SAM design and restructuring with the bilinear concept,
which is more effective in extracting discriminative features. Furthermore,
the network employed two supervision branches along with a new loss function.
This approach calculated gradients from distinct pairs of local image patches
and incorporated a modulation factor in the loss function to prioritize
training on complex samples. It handles the data imbalance problem by
adjusting the weight, making it vulnerable to overfitting and enhancing the
proposed network’s performance and interpretability. Additionally, using the
ECA block instead of the SE block in the proposed network reduces the model
complexity and trainable parameters. BCNN is high-dimensional and affects the
computational cost and memory use for further feature analysis. To address
this, we used a compact bilinear pooling network to perform core analysis on
bilinear features, significantly reducing dimensionality. This operation
allows the proposed model to optimize memory usage while maintaining
significant classification accuracy.

####

Furthermore, we visually analyzed the AHM of the two mechanisms using Grad-CAM
to evaluate the classification accuracy. Second, we conducted an ablation test
to verify the importance of each component in the attention framework. The
effectiveness of the framework structure in enhancing the network feature
extraction ability was confirmed. Finally, we evaluated the proposed network’s
superiority and robustness by comparing its results with the state-of-the-art
methods for skin disease classification. The results demonstrated that the
proposed network exhibits higher accuracy and better robustness in
implementing skin disease classification.

####

In the future, DACB-Net may be combined or used with other network structures
for other tasks to show the generality. However, due to the limitations of
hardware resources, it has not been tested with different specifications. The
research results of the proposed network show that its accuracy increases
continuously with the expansion of specification. Therefore, theoretically, if
other network specifications are used, the accuracy and their degree of
optimization can be further improved. Furthermore, since the proposed
algorithm and the optimization approach are experimentally validated on the
existing dataset, the model can be subsequently used to accomplish fine-
grained classification tasks in practical applications.

## References

  * [1] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115:211–252, 2015.
  * [2] Antony R Young.  Tanning devices–fast track to skin cancer?  Pigment Cell Research, 17(1):2–9, 2004.
  * [3] L Li, Q Zhang, Y Ding, H Jiang, BH Thiers, and JZ Wang.  Automatic diagnosis of melanoma using machine learning methods on a spectroscopic system. 14: 1–12, 2014.
  * [4] Saket S Chaturvedi, Jitendra V Tembhurne, and Tausif Diwan.  A multi-class skin cancer classification using deep convolutional neural networks.  Multimedia Tools and Applications, 79(39):28477–28498, 2020.
  * [5] Syed Inthiyaz, Baraa Riyadh Altahan, Sk Hasane Ahammad, V Rajesh, Ruth Ramya Kalangi, Lassaad K Smirani, Md Amzad Hossain, and Ahmed Nabih Zaki Rashed.  Skin disease detection using deep learning.  Advances in Engineering Software, 175:103361, 2023.
  * [6] RB Sushma and GR Manjula.  Stegvrn: Enhancing quality of video steganography using cnn-based object selection.  SN Computer Science, 5(2):227, 2024.
  * [7] Guannan Li, Liang Chen, Cheng Fan, Jiajia Gao, Chengliang Xu, and Xi Fang.  Improved convolutional neural network chiller early fault diagnosis by gradient-based feature-level model interpretation and feature learning.  Applied Thermal Engineering, 236:121549, 2024.
  * [8] Pim Moeskops, Max A Viergever, Adriënne M Mendrik, Linda S De Vries, Manon JNL Benders, and Ivana Išgum.  Automatic segmentation of mr brain images with a convolutional neural network.  IEEE transactions on medical imaging, 35(5):1252–1261, 2016.
  * [9] Noel Codella, Junjie Cai, Mani Abedini, Rahil Garnavi, Alan Halpern, and John R Smith.  Deep learning, sparse coding, and svm for melanoma recognition in dermoscopy images.  In International workshop on machine learning in medical imaging, pages 118–126. Springer, 2015.
  * [10] Sergey Demyanov, Rajib Chakravorty, Mani Abedini, Alan Halpern, and Rahil Garnavi.  Classification of dermoscopy patterns using deep convolutional neural networks.  In 2016 IEEE 13th International symposium on biomedical imaging (ISBI), pages 364–368. IEEE, 2016.
  * [11] Laith Alzubaidi, Jinglan Zhang, Amjad J Humaidi, Ayad Al-Dujaili, Ye Duan, Omran Al-Shamma, José Santamaría, Mohammed A Fadhel, Muthana Al-Amidie, and Laith Farhan.  Review of deep learning: concepts, cnn architectures, challenges, applications, future directions.  Journal of big Data, 8:1–74, 2021.
  * [12] Yiqi Yan, Jeremy Kawahara, and Ghassan Hamarneh.  Melanoma recognition via visual attention.  In Information Processing in Medical Imaging: 26th International Conference, IPMI 2019, Hong Kong, China, June 2–7, 2019, Proceedings 26, pages 793–804. Springer, 2019.
  * [13] J Schlemper, O Oktay, L Chen, J Matthew, C Knight, B Kainz, B Glocker, and D Rueckert.  Attention-gated networks for improving ultrasound scan plane detection. arxiv.  arXiv preprint arXiv:1804.05338, 2018.
  * [14] Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson.  Understanding neural networks through deep visualization.  arXiv preprint arXiv:1506.06579, 2015.
  * [15] Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, and Han Hu.  Gcnet: Non-local networks meet squeeze-excitation networks and beyond.  In Proceedings of the IEEE/CVF international conference on computer vision workshops, pages 0–0, 2019.
  * [16] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler.  The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. scientific data. 2018; 5: 180161.  Search in, 2, 2018.
  * [17] Mohamed A Kassem, Khalid M Hosny, and Mohamed M Fouad.  Skin lesions classification into eight classes for isic 2019 using deep convolutional neural network and transfer learning.  IEEE Access, 8:114822–114832, 2020.
  * [18] R Josphineleela, PBV Raja Rao, Amir Shaikh, and K Sudhakar.  A multi-stage faster rcnn-based isplinception for skin disease classification using novel optimization.  Journal of Digital Imaging, 36(5):2210–2226, 2023.
  * [19] Camilo Calderón, Karen Sanchez, Sergio Castillo, and Henry Arguello.  Bilsk: A bilinear convolutional neural network approach for skin lesion classification.  Computer Methods and Programs in Biomedicine Update, 1:100036, 2021.
  * [20] Aditya Kumar Singh Pundir, Md Abul Ala Walid, Pravin Adivarekar, Arepalli Gopi, V Malathy, and Kamlesh Singh.  Intelligent system for early diagnosis of skin cancer using bilinear cnn and soft attention approach.  In 2023 4th International Conference on Electronics and Sustainable Communication Systems (ICESC), pages 1146–1152. IEEE, 2023.
  * [21] Yun Wei and Lin Ji.  Multi-modal bilinear fusion with hybrid attention mechanism for multi-label skin lesion classification.  Multimedia Tools and Applications, pages 1–27, 2024.
  * [22] Tsedenya Debebe Nigat, Tilahun Melak Sitote, and Berihun Molla Gedefaw.  Fungal skin disease classification using the convolutional neural network.  Journal of Healthcare Engineering, 2023(1):6370416, 2023.
  * [23] Achanta Jyothi Prakash and P Prakasam.  An intelligent fruits classification in precision agriculture using bilinear pooling convolutional neural networks.  The Visual Computer, 39(5):1765–1781, 2023.
  * [24] Wenshu Liu.  Implementation of detection of skin lesions in monkeypox based on a deep learning model: using an improved bilinear pooling model.  In Second International Conference on Biological Engineering and Medical Science (ICBioMed 2022), volume 12611, pages 212–219. SPIE, 2023.
  * [25] Tsung-Yu Lin, Aruni RoyChowdhury, and Subhransu Maji.  Bilinear cnn models for fine-grained visual recognition.  In Proceedings of the IEEE international conference on computer vision, pages 1449–1457, 2015.
  * [26] Aakanksha Karambele, Mrunal Tamhane, Pranati Jayanty, Rutuja Patil, and Kavita Shirsat.  edermacare: Enhancing skin disease classification with resnet50.  In 2024 11th International Conference on Computing for Sustainable Global Development (INDIACom), pages 1008–1014. IEEE, 2024.
  * [27] Md Sazzadul Islam Prottasha, Sanjan Mahjabin Farin, Md Bulbul Ahmed, Md Zihadur Rahman, ABM Kabir Hossain, and M Shamim Kaiser.  Deep learning-based skin disease detection using convolutional neural networks (cnn).  In The Fourth Industrial Revolution and Beyond: Select Proceedings of IC4IR+, pages 551–564. Springer, 2023.
  * [28] Vinay Gautam, Naresh Kumar Trivedi, Abhineet Anand, Rajeev Tiwari, Atef Zaguia, Deepika Koundal, and Sachin Jain.  Early skin disease identification using eep neural network.  Comput. Syst. Sci. Eng., 44(3):2259–2275, 2023.
  * [29] Sinno Jialin Pan and Qiang Yang.  A survey on transfer learning.  IEEE Transactions on knowledge and data engineering, 22(10):1345–1359, 2009.
  * [30] François Chollet.  Xception: Deep learning with depthwise separable convolutions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1251–1258, 2017.
  * [31] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015.
  * [32] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon.  Cbam: Convolutional block attention module.  In Proceedings of the European conference on computer vision (ECCV), pages 3–19, 2018.
  * [33] K He, X Feng, and SN Gao.  A fine-grained image classification algorithm based on multi-scale feature fusion and repeated attention mechanism.  J. Tianjin Univ.(Nat. Sci. Eng. Technol. Ed.), 53:1077–1085, 2020.
  * [34] D Kinga, Jimmy Ba Adam, et al.  A method for stochastic optimization.  In International conference on learning representations (ICLR), volume 5, page 6. San Diego, California;, 2015.
  * [35] Parvathaneni Naga Srinivasu, Jalluri Gnana SivaSai, Muhammad Fazal Ijaz, Akash Kumar Bhoi, Wonjoon Kim, and James Jin Kang.  Classification of skin disease using deep learning neural networks with mobilenet v2 and lstm.  Sensors, 21(8):2852, 2021.
  * [36] VSSP Raju Gottumukkala, N Kumaran, and V Chandra Sekhar.  Blsnet: Skin lesion detection and classification using broad learning system with incremental learning algorithm.  Expert Systems, 39(9):e12938, 2022.
  * [37] P Bharat Siva Varma, Siddartha Paturu, Suman Mishra, B Srinivasa Rao, Pala Mahesh Kumar, and Namani Vamshi Krishna.  Sldcnet: Skin lesion detection and classification using full resolution convolutional network-based deep learning cnn with transfer learning.  Expert Systems, 39(9):e12944, 2022.
  * [38] K Suresh Kumar, N Suganthi, Satish Muppidi, and B Santhosh Kumar.  Fspbo-dqn: Segan based segmentation and fractional student psychology optimization enabled deep q network for skin cancer detection in iot applications.  Artificial intelligence in medicine, 129:102299, 2022.
  * [39] Titus J Brinker, Max Schmitt, Eva I Krieghoff-Henning, Raymond Barnhill, Helmut Beltraminelli, Stephan A Braun, Richard Carr, Maria-Teresa Fernandez-Figueras, Gerardo Ferrara, Sylvie Fraitag, et al.  Diagnostic performance of artificial intelligence for histologic melanoma recognition compared to 18 international expert pathologists.  Journal of the American Academy of Dermatology, 86(3):640–642, 2022.
  * [40] M Attique Khan, Tallha Akram, Muhammad Sharif, Aamir Shahzad, Khursheed Aurangzeb, Musaed Alhussein, Syed Irtaza Haider, and Abdualziz Altamrah.  An implementation of normal distribution based segmentation and entropy controlled features selection for skin lesion detection and classification.  BMC cancer, 18:1–20, 2018.
  * [41] Rarasmaya Indraswari, Rika Rokhana, and Wiwiet Herulambang.  Melanoma image classification based on mobilenetv2 network.  Procedia computer science, 197:198–207, 2022.
  * [42] M Prathiba, Deepa Jose, R Saranya, et al.  Automated melanoma recognition in dermoscopy images via very deep residual networks.  In IOP Conference series: materials science and engineering, volume 561, page 012107. IOP Publishing, 2019.
  * [43] Peng Yao, Shuwei Shen, Mengjuan Xu, Peng Liu, Fan Zhang, Jinyu Xing, Pengfei Shao, Benjamin Kaffenberger, and Ronald X Xu.  Single model deep learning on imbalanced small datasets for skin lesion classification.  IEEE transactions on medical imaging, 41(5):1242–1254, 2021.
  * [44] Mohanad A Deif and Rania E Hammam.  Skin lesions classification based on deep learning approach.  Journal of Clinical Engineering, 45(3):155–161, 2020.
  * [45] Quande Liu, Lequan Yu, Luyang Luo, Qi Dou, and Pheng Ann Heng.  Semi-supervised medical image classification with relation-driven self-ensembling model.  IEEE transactions on medical imaging, 39(11):3429–3440, 2020.
  * [46] Ensaf Hussein Mohamed and Wessam H El-Behaidy.  Enhanced skin lesions classification using deep convolutional networks.  In 2019 Ninth international conference on intelligent computing and information systems (ICICIS), pages 180–188. IEEE, 2019.
  * [47] Harsh Gupta, Hartik Bhatia, Dheeraj Giri, Rishabh Saxena, and Raghuraj Singh.  Comparison and analysis of skin lesion on pretrained architectures.  International Research Journal of Engineering and Technology (IRJET), 7(7):2704–2707, 2020.
  * [48] Imran Iqbal, Muhammad Younus, Khuram Walayat, Mohib Ullah Kakar, and Jinwen Ma.  Automated multi-class classification of skin lesions through deep convolutional neural network with dermoscopic images.  Computerized medical imaging and graphics, 88:101843, 2021.
  * [49] Dasari Anantha Reddy, Swarup Roy, Sanjay Kumar, and Rakesh Tripathi.  A scheme for effective skin disease detection using optimized region growing segmentation and autoencoder based classification.  Procedia Computer Science, 218:274–282, 2023.
  * [50] Andre GC Pacheco, Gustavo R Lima, Amanda S Salomao, Breno Krohling, Igor P Biral, Gabriel G de Angelo, Fábio CR Alves Jr, José GM Esgario, Alana C Simora, Pedro BC Castro, et al.  Pad-ufes-20: A skin lesion dataset composed of patient data and clinical images collected from smartphones.  Data in brief, 32:106221, 2020.
  * [51] Tianyu Fang, Jialin Zhang, Dawei Qi, and Mingyu Gao.  Blsenet: A novel lightweight bilinear convolutional neural network based on attention mechanism and feature fusion strategy for apple leaf disease classification.  Journal of Food Quality, 2024(1):5561625, 2024.
  * [52] Ying Li, Lingkun Ma, and Nan Sun.  A bilinear transformer interactive neural networks-based approach to fine-grained recognition and protection of plant diseases for gardening design.  Crop Protection, 180:106660, 2024.
  * [53] Lei Cai, Jingyang Gao, and Di Zhao.  A review of the application of deep learning in medical image classification and segmentation.  Annals of translational medicine, 8(11), 2020.