Corresponding author: Dorit Merhof (e-mail: dorit.merhof@lfb.rwth-aachen.de).

# TransNorm: Transformer Provides a Strong Spatial Normalization Mechanism for
a Deep Segmentation Model

REZA AZAD1  MOHAMMAD T. AL-ANTARY2, MOEIN HEIDARI3  DORIT MERHOF4  Institute
of Imaging and Computer Vision, RWTH Aachen University, Germany. (e-mail:
<EMAIL_ADDRESS> School of Computing and Mathematical Sciences,
University of Greenwich, London SE10 9LS, U.K. (e-mail<EMAIL_ADDRESS>School of Electrical Engineering, Iran University of Science and Technology,
Iran. (e-mail<EMAIL_ADDRESS> Institute of Imaging and
Computer Vision, RWTH Aachen University, Germany. (e-mail:
<EMAIL_ADDRESS>
###### Abstract

In the past few years, convolutional neural networks (CNNs), particularly
U-Net, have been the prevailing technique in the medical image processing era.
Specifically, the seminal U-Net, as well as its alternatives, have
successfully managed to address a wide variety of medical image segmentation
tasks. However, these architectures are intrinsically imperfect as they fail
to exhibit long-range interactions and spatial dependencies leading to a
severe performance drop in the segmentation of medical images with variable
shapes and structures. Transformers, preliminary proposed for sequence-to-
sequence prediction, have arisen as surrogate architectures to precisely model
global information assisted by the self-attention mechanism. Despite being
feasibly designed, utilizing a pure Transformer for image segmentation
purposes can result in limited localization capacity stemming from inadequate
low-level features. Thus, a line of research strives to design robust variants
of Transformer-based U-Net. In this paper, we propose Trans-Norm, a novel deep
segmentation framework which concomitantly consolidates a Transformer module
into both encoder and skip-connections of the standard U-Net. We argue that
the expedient design of skip-connections can be crucial for accurate
segmentation as it can assist feature fusion between the expanding and
contracting paths. In this respect, we derive a Spatial Normalization
mechanism from the Transformer module to adaptively recalibrate the skip
connection path. Extensive experiments across three typical tasks for medical
image segmentation demonstrate the effectiveness of TransNorm. The codes and
trained models are publicly available at github.

###### Index Terms:

Transformer, Semantic segmentation, Attention, Medical image analysis

=-15pt

##  I Introduction

In the healthcare field, early diagnosis of diseases is a crucial step as it
can aid in detecting the severity and spread of disorders in early stages.
Medical image segmentation plays an integral role in computer-aided disease
diagnosis (CAD), treatment planning and surgical pre-assessment. It mainly
consists of partitioning the shapes and volumes of target organs and tissues
by pixel-wise classification [1]. Manual annotation in clinical applications
is labour-intensive and time-consuming [2] and may be prone to human error.
Thus, to preclude the burden of manual annotation, automated medical image
segmentation has become a direction of research for many years. The recent
growth of deep learning in computer vision has prompted researchers to
reconsider its potential in CAD. Based on this line of research, deep learning
has attained immense success in a wide range of medical domains ranging from
abdominal organ segmentation from CT images [3, 4, 5], and skin lesion
segmentation from dermoscopy [6] to multiple Mylomia segmentation [7] from
microscopic images, MRI images [8, 9, 10, 11] and etc. Figure 1 shows samples
of the Synapse multi-organ segmentation dataset [12] which exhibits a diverse
set of challenging samples for a medical image segmentation task.

Figure 1: Sample 2D slices of the Synapse multi-organ segmentation dataset and
the multi-organ annotation mask.

Convolutional neural networks (CNNs), and more specifically fully
convolutional networks (FCNs) [13] have been the de facto standard in medical
image segmentation for many years. However, these architectures face a major
issue, namely that essential details are lost at deeper layers of the network.
To solve this problem, and inspired by the seminal U-Net structure [14], a
family of U-shaped networks were devised. The U-shaped structure, designated
as U-Net, and consisting of symmetric encoder-decoder with skip connections
has rendered remarkable performances in recent years. The contracting path
(encoder) follows the typical structure of a convolutional neural network
which encodes the input image into feature representations at multiple levels.
The CNN-based models, despite being able to improve context modelling to some
degree, have an unavoidable confined receptive field following from the nature
of the convolution operation. In natural language processing (NLP), the
ubiquitous architecture _Transformer_ has been seen as capable of learning
long-term features [15, 16]. Transformers, unlike CNNs, are not only
significant at modelling global contexts but are also a promising tool for
localizing local details. Inspired by their success in the NLP domain,
researchers have inspected leveraging the Transformer network in computer
vision. The pioneering architecture, based purely on the self-attention
mechanism, was the Vision Transformer (ViT) [17] which attained high
performance compared to SOTA in image recognition tasks. Since then,
Transformers have proven their prodigious power in diverse computer vision
tasks [18, 19, 6]. To concurrently benefit from the global context modelling
of Transformers as well as the capability of CNNs in learning the rich
detailed information, many cohort studies have investigated the amalgamation
of U-Net and Transformer [4, 20]. Inspired by this line of research, in this
paper, we propose TransNorm, a novel architecture that provides a strong
Spatial Normalization mechanism for a deep segmentation model and applies it
to medical image segmentation. Our TransNorm model initially applies the
convolution block to encode the input image into a latent space. Next, on top
of the generated features, we incorporate the vision Transformer module to
capture global contextual dependency from the latent space. The combination of
local features attained from the CNN module and the global dependency from the
Transformer module provides a complementary representation to enrich the
encoder representation. This design is similar to the recently proposed
TransUnet approach [4]. However, we assert that a feasible design of skip-
connections can be crucial for accurate segmentation. Thus, to further enhance
the aforementioned design and increase the benefit from the Transformer module
we extend this strategy to normalize the skip connection sections of the
decoding path. To this end, we define a two-level attention mechanism to
adaptively recalibrate the feature combination on the skip connection path.
First, by applying the channel attention mechanism, we normalize the feature
representation to emphasize the more informative channels. Next, by utilizing
the spatial coefficient stemming from the Transformer module, we apply a
second level attention mechanism to strengthen the contribution of important
regions to the segmentation process. The experimental results on the multi-
organ segmentation task show that incorporating a spatial attention module
enhances model strength in recovering object boundaries from an overlapped
background. Our contributions are as follows :

$\bullet$ Incorporting an attention module on the skip connection

$\bullet$ Spatial attention module derived from a Transformer model

$\bullet$ Improved results on the public datasets as well as publicly
available implementation source code.

##  II Literature review

###  II-A CNN-based Segmentation Networks

Convolutional neural networks have been the most commonly used approach for
image segmentation tasks in recent years. Following the advent of the
monumental U-Net [14], this elegant design has been incorporated for diverse
medical image segmentation tasks. The main contribution of U-Net in this sense
is that while upsampling in the network it also concatenates the higher
resolution feature maps from the encoder network with the upsampled features
in order to more accurately learn representations. Meanwhile, persuaded by the
success of U-Net, a line of research attempted to extend this architecture for
the purpose of even more accurate segmentation such as Res-UNet [21], Dense-
UNet [22], U-Net++ [23] and UNet3+ [24], style matching U-Net [11] few-shot
U-Net [25, 26]. To this end, 3D U-net [27] was proposed as an augmentation of
U-Net which deals with 3D volumetric segmentation. Oktay et al. [28] proposed
to use _attention gates_ in the basic U-Net architecture for pancreas
segmentation to enable the network to focus on specific objects of greater
importance while ignoring superfluous areas. Furthermore, U-Net++ [23]
proposed to bridge the semantic gap between the feature maps of the encoder
and decoder using nested and dense skip connections instead of directly
fetching high-resolution feature maps from the encoder into the decoder. The
common point of these CNN-based methods is that they inevitably have
limitations in capturing long-range dependencies due to the inherited property
of the confined receptive field. In fact, the locality and weight-sharing
property of convolution operations make them incapable of apprehending global
contexts. To date, various methods have been developed to solve this problem
of CNN’s restricted receptive field. As such, Yu et al. [29] proposed to use
atrous convolution with a dilation rate that expands the receptive field
without loss of resolution. Zhao et al. [30] exploited pyramid pooling at
different feature scales so as to agglomerate global information. Wang et al.
[31] proposed a non-local neural network for capturing long-range dependencies
by computing the response at a position as a weighted sum of the features at
all positions in the input feature maps. Incorporating self-attention modules
into convolutional layers is another attempt to restrain the deficiency of
CNNs in non-local modelling capability. To achieve this, Fu et al. [32]
proposed a dual attention network that models the semantic interdependencies
in spatial and channel dimensions using the self-attention mechanism
respectively. Despite the efforts made to mitigate the aforementioned problem
of CNNs, they still cannot fully meet the clinical application requirements
and exacerbate a notorious problem in the segmentation of medical images with
various shapes and scales. Specifically, although these methods have been able
to boost the performance of segmentation to some extent, the potential of
capturing long-range semantic dependencies still needs to be addressed.

###  II-B Transformers

Following the consensus of exploiting Transformer as a de facto operator in
the NLP era, more and more Transformer-based methods appear in CV tasks.
Specifically, ViT [17] was the primary medium of the Transformer-based methods
to surpass the traditional CNN-based architectures in image recognition tasks.
Intuitively, ViT divides the input image into multiple partitions (patches),
which are then fed into a Transformer encoder followed by an MLP layer to
perform the classification. Subsequently, different alternatives of ViT were
proposed in the literature such as Swin Transformer [33], LeViT [34], and
Twins [35]. To solve the drawback of model complexity in regular Transformers,
Swin Transformer [33] proposes to split image patches into windows, and apply
Transformer only within patches inside each window. Additionally, inspired by
the basic idea of CNNs, the authors suggest shifting the window and then
applying the Transformer module again allowing adjacent windows to interact
with each other.

###  II-C Combining CNNs with Transformer architecture

The pioneering Transformer-based medical image segmentation approaches involve
exploiting Transformer layers in the encoder of the architecture. The first
work to do this was the TransUNet paper [4]. As opposed to other ViT-based
methods, the authors propose to first convert the input image to a set of low-
resolution feature maps by initially inputting the image to a series of
convolution layers (using a Res-Net50 backbone) which they then encode with a
ViT. The resulting encoded features are then upsampled via so-called Cascaded
Upsampler layers in the decoder to output the final segmentation map. To
restrain the quadratic complexity of using a purely Transformer-based model
which can hinder the performance in segmentation of high-resolution medical
images, Cao et al. [36] propose Swin-UNet. Inspired by Swin-Transformer they
compute the attention within a fixed window. Furthermore, Swin-UNet contains a
patch expanding layer for upsampling the decoder’s feature maps thereby
reshaping the feature maps of adjacent dimensions into a higher resolution
feature map. In another attempt to incorporate the Transformer module into
encoding layers, Wu et al. [37] replace the long-established single branch
encoder architecture with a dual encoder containing CNNs and a Transformer
branch. Moreover, to attain an adaptive feature fusion from these branches to
the decoding part, they devise a feature adaptation module (FAM) and a memory-
efficient decoder to circumvent the burden of computational inefficiency.

The CNN-based segmentation methods usually suffer from poor global contextual
encoding and render a poor prediction of the object boundary. However, the
Transformer-based methods are highly capable of encoding the long-range
connectively and exhibit a strong feature learning strategy. However, this
strategy usually lacks the learning of local information and suffers from poor
generalization performance. Thus, combining these two descriptors might
provide an efficient feature representation, which is at the heart of our
research in this paper. Following the aforementioned TransUnet method
(combining CNN and Transformer modules), we argue that this method mainly
suffers from a weak construction on the skip connection section, hence, we
propose an attention module to boost the performance of this pipeline. To this
end, we design a two-level attention mechanism based on the Transformer module
to adaptively recalibrate the feature combination on the skip connection path.
In the next section, we will present our method comprehensively.

##  III Proposed Method

A conceptual overview of the proposed TransNorm is depicted in Figure 2. Given
an image $\mathbf{I}\in\mathbb{R}^{H\times W\times C}$ with a spatial
resolution of $H\times W$ and $C$ number of channels, we aim to predict the
corresponding segmentation map of size $H\times W$. As shown in Figure 2,
TransNorm consists of several components processing information differently:
1) encoder, which is mainly a CNN that encodes semantic and high-level
features; 2) Transformer branch, whose goal is to learn the long-range
contextual dependency in the bottom stream of our network, within which we
propose a so-called _Spatial Normalizer_ shown in Figure 3; 3) decoder, to
gradually output the final labelmap. We will elaborate each part in more
detail in the next subsections.

Figure 2: The structure of the proposed method for medical image segmentation.
Our approach utilizes a Transformer module on the network bottleneck to learn
long-range contextual dependency and produces a spatial normalization
coefficient for the attention module.

###  III-A Encoder

The first module that is incorporated into our design is the encoder module.
Generally, the design choice for the encoder module can follow any well-known
network, whereas a deeper network may increase the feature representation
power and, consequently, offer a better generalization performance. However,
the main objective of this study is to investigate the effectiveness and
importance of the skip connection in the Transformer-based model. Thus, to
squeeze the CNN representation, we exploit the semi U-Net [14] structure to
obtain the semantic and global context of features. Given an input image
$\mathbf{I}\in\mathbb{R}^{H\times W\times C}$, the feature maps produced by
the encoder $x\in R^{H^{\prime}\times W^{\prime}\times C^{\prime}}$ can be
formally expressed as

$x=E_{\theta}(\mathbf{I}),$ (1)

where $E$ denotes the encoder with parameters $\theta$.

###  III-B Long-range Contextual Representation

To exploit the supremacy of learning long-range correlation and contextual
dependency, we integrate the seminal Transformer module following the encoding
part of our design. To this end, we perform sequentialization by reshaping the
input $x\in R^{H^{\prime}\times W^{\prime}\times C^{\prime}}$ into a sequence
of flattened non-overlapping patches
$\mathbf{x}_{p}\in\mathcal{R}^{N\times\left(p^{2}\cdot C^{\prime}\right)}$,
where each patch is of size $P\times P$ and
$N=\frac{H^{\prime}W^{\prime}}{P^{2}}$ denotes the input sequence length.
Subsequently, we map the patches into a $D$-dimensional embedding space using
a linear projection. To retain the spatial information of each patch, as the
attention mechanism is intrinsically permutation-invariant, we learn a
position embedding which is later added to the patch embedding as follows:

$\mathbf{z}_{0}=\left[\mathbf{x}_{p}^{1}\mathbf{E};\mathbf{x}_{p}^{2}\mathbf{E};\cdots;\mathbf{x}_{p}^{N}\mathbf{E}\right]+\mathbf{E}_{pos},$
(2)

where $\mathbf{E}\in\mathbb{R}^{\left(P^{2}\cdot C^{\prime}\right)\times D}$
is the patch embedding projection, and $\mathbf{E}_{\text{pos
}}\in\mathbb{R}^{N\times D}$ intends to learn the positional encoding.
Ultimately, the Transformer encoder is comprised of $K$ stages, each
consisting of a Multihead Self-Attention (MSA) and a Multi-Layer Perceptron
(MLP) block. Therefore, the output of the $k$-th layer is modeled as:

$\mathbf{z}_{\emph{k}}^{\prime}=\operatorname{MSA}\left(\mathrm{LN}\left(\mathbf{z}_{\emph{k}-1}\right)\right)+\mathbf{z}_{\emph{k}-1}$
(3)
$\mathbf{z}_{\emph{k}}=\mathrm{MLP}\left(\mathrm{LN}\left(\mathbf{z}_{\emph{k}}^{\prime}\right)\right)+\mathbf{z}_{\emph{k}}^{\prime}$
(4)

where $\mathrm{LN}()$ denotes the layer normalization operator [38]. We
concatenate the generated features with the CNN-based features derived from
the encoder to form a complementary (semantic and long-range contextual)
feature set. To further benefit from the Transformer module we propose an
attention module to provide a Spatial Normalization mechanism for the skip
connections in the decoding path.

###  III-C Attention Gate

The adaptation of attention mechanisms with the aim of diverting focus to
salient regions of an image while disregarding irrelevant parts has long been
established in the computer vision era. Inspired by its success, we devise a
two-level attention mechanism into the skip connection of the decoding path as
shown in Figure 2. Given an encoded CNN feature $x\in R^{H^{\prime}\times
W^{\prime}\times C^{\prime}}$ and the transformer feature $z\in
R^{H^{\prime}\times W^{\prime}\times C^{\prime}}$, our two-level attention
module first calculates the 1D channel attention map
$\mathbf{W_{c}}\in\mathbb{R}^{C\times 1\times 1}$ to adaptively recalibrate
the object recognition features and then utilizes a 2D Spatial Normalization
map $\mathbf{W_{s}}\in\mathbb{R}^{1\times H^{\prime}\times W^{\prime}}$
generated by the Transformer module to perform the Spatial Normalization
process. The overall attention process can be written as:

$\displaystyle\mathbf{f}_{cn}$
$\displaystyle=\mathbf{W}_{\mathrm{c}}(\mathbf{f})\otimes\mathbf{f},$ (5)
$\displaystyle\mathbf{f}{sn}$
$\displaystyle=\mathbf{W}_{\mathbf{s}}\otimes\mathbf{f}_{cn},$

Where $\otimes$ shows the element-wise multiplication and $f$ represents the
concatenation of the CNN and Transformer features. More specifically, we
formulate the channel normalization $f_{cn}$ by adaptively recalibrating the
weight of each channel, to feasibly exploit the inter-channel correspondence
of feature maps. To this end, we first calculate the global average pooling of
each layer. Then, by utilizing a fully-connected layer, we scale each channel
accordingly

$\displaystyle\mathbf{f}_{\mathbf{cn}}$
$\displaystyle=\sigma(MLP(\operatorname{Avg}\operatorname{Pool}(\mathbf{F})))$
(6)
$\displaystyle=\sigma\left(\mathbf{W}_{\mathbf{2}}\left(\mathbf{W}_{\mathbf{1}}\left(\mathbf{F}_{\mathbf{avg}}^{\mathbf{c}}\right)\right)\right),$

where $\sigma$ represents the sigmoid activation function, and $W_{1}$ and
$W_{2}$ show the MLP weights. Next, for the purpose of an adaptive spatial
region selection mechanism, we devise a spatial attention map to focus on
informative parts of a single feature map. We assert that utilizing the
spatial coefficients derived from the Transformer module will lead to a better
conclusion for the segmentation model as to where to emphasize or suppress
while generating the segmentation map. In this respect, we use the attention
probability map $\mathbf{W_{s}}\in\mathbb{R}^{1\times H^{\prime}\times
W^{\prime}}$ derived from the Transformer module
$W_{s}=\operatorname{softmax}\left(\mathbf{qk}^{\top}/\sqrt{d}\right)$ to
perform an attention mechanism:

$\displaystyle\mathbf{f}{sn}$
$\displaystyle=\mathbf{W}_{\mathbf{s}}\otimes\mathbf{f}_{cn},$ (7)

where $q$, $k$, and $d$ represent the query, key and normalization values,
respectively. To impose an expedient feature fusion scheme into the decoding
part of our proposal, we reconsider the feature recalibration on the skip-
connection path. Intuitively, in order to guarantee the aggregation scheme to
exploit both the high-level and the with low-level feature maps, we make use
of generated attention maps into the skip-connection configuration.

###  III-D Decoder

The decoder, a pure CNN architecture, progressively upsamples the feature maps
to the original image space (i.e., $\mathbb{R}^{H\times W\times C}$). The
cascaded upsampling strategy is used to recover the resolution from the
previous layer using transpose convolution to perform pixel-level
segmentation. Moreover, to effectively promote global information from the
Spatial Normalizer module, we utilize the feature maps obtained from the
Spatial Normalizer which contain high-level semantic information in the
upsample layers using attention gates as shown in Figure 2. We claim that the
idea of prominently designing skip-connections leads to finer feature fusion
without the severe cost of computational burden. To support our claim, we
performed an ablation study on the effect of the afrementioned idea in the
following sections.

Figure 3: Our proposed two-level attention gate to normalize both channel and
spatial information on each decoding path.

##  IV Experimental Results

To validate our method for different medical image segmentation tasks,
extensive experiments on three different tasks were designed. In this section,
first, we briefly present the dataset utilized to evaluate the effectiveness
of the designed network. Next, the training hyperparameters and the training
process are described in more detail. To validate the performance of the
proposed approach compared to the baseline and the counterpart methods, we
first describe the metrics and the experimental setting. Then, for each
dataset quantitative and visualization results are provided to discuss the
obtained results in a comprehensive manner. Furthermore, the ablation study is
included to evaluate the contribution and effects of each proposed module
separately. In the next subsections, we will discuss each section in more
detail.

###  IV-A Dataset

####  IV-A1 Synapse multi-organ segmentation

The Beyond the Cranial Vault (BTCV) abdomen challenge dataset [12] comprises
30 abdominal CT scans with, in total, 3779 axial contrast-enhanced abdominal
clinical CT images. In each instance, 13 organs were annotated by interpreters
including the spleen, the right kidney, the left kidney, the gallbladder, the
esophagus, the liver, the stomach, the aorta, the inferior vena cava (IVC),
the portal vein, the splenic vein, the pancreas, the left adrenal gland, and
the right adrenal gland. Each CT scan is acquired with contrast enhancement
leading to volumes in the range of $85\sim 198$ slices of $512\times 512$
pixels, with a voxel spatial resolution of $([0.54\sim 0.54]\times[0.98\sim
0.98]\times[2.5\sim 5.0])\mathrm{mm}^{3}$. Following the splitting strategy of
[4], 18 samples were divided into the training set and 12 samples into the
testing set. Moreover, as for the pre-processing pipeline, each scan value was
normalized to values between 0 an 1 followed by random sampling of $96\times
96\times 96$ voxels [39].

####  IV-A2 Skin lesion segmentation

ISIC 2017 Dataset. The ISIC 2017 dataset [40] was published by the
International Skin Imaging Collaboration (ISIC) as a large-scale dataset of
dermoscopy images which was primarily rendered for three tasks including
lesion segmentation (considered in this work), dermoscopic feature detection,
and disease classification. The ISIC 2017 dataset contains a training set of
2000 images with corresponding ground truth images. The size of each sample is
576 × 767 pixels. Following the settings of [41], we use 1250 samples for
training, 150 samples for validation data, and 600 samples as a test set. As
for the pre-processing stage, we resize images to 256 × 256 pixels following
the lead of [42].

ISIC 2018 Dataset. In 2018 the ISIC foundation developed this dataset [43] as
a large-scale dataset of dermoscopy images with the aim of further supporting
clinical research which yields automated algorithmic analysis. The database
comprises more than 10,000 dermoscopic images of 7 types of diseases:
(melanoma, nevi, seborrheic keratosis, BCC, Bowen’s disease and actinic
keratosis, vascular lesions, and dermatofibromas) as well as their
corresponding ground truth annotations. We alter the size of each sample i.e.,
2016 × 3024, to a fixed size of 256 × 256 pixels.

$\mathbf{PH}^{2}$ Dataset. The $\mathrm{PH}^{2}$ dataset [44] comprises 200
dermatoscopic images of melanocytic skin lesions including 80 common nevi, 80
atypical nevi, and 40 melanomas. The resolution of each input image is 768 ×
560 pixels. Following the same strategy as [45], we divide the dataset into
train and test sets (100 samples per each set) and resize the images into 256
× 256 pixels. Furthermore, similar to [45] we pre-train the model on ISIC 2017
to later tune it for this dataset.

####  IV-A3 Multiple Mylomia Segmentation

The Multiple Myeloma (MM) challenge was held by the SegPC grand challenge in
conjunction with the ISBI 2021 symposium to evaluate the effectiveness of
machine learning methods for multiple myeloma cell segmentation [46]. This
challenge contains several images captured from the bone marrow slides of
patients diagnosed with Multiple Myeloma (MM) cancer, which cancer arises from
a white blood cell. The annotated train dataset provided by this challenge
contains 290 images with an MM instance in each image. To evaluate our method
on this dataset, we follow the same strategy presented in [46] and divide the
original train set into train and validation sets. Our inference is on the
original validation set provided by this challenge.

###  IV-B Evaluation Metrics

We evaluated the performance of Synapse multi-organ segmentation with the
average Dice Similarity Coefficient (DSC) and the average Hausdorff Distance
(HD). To validate the results obtained on skin lesion segmentation and MM we
used several well-known metrics including accuracy (AC), sensitivity (SE),
specificity (SP), F1-Score, and mean Intersection over Union (mIoU). The
equation of each metric and the brief description is provided as follows: TP
indicates a sample that is correctly predicted as a true sample; TN shows a
sample with a negative label that is correctly predicted as a negative sample;
FP indicates that the model predicted a negative sample as a true class while
the correct label was the negative class; FN similarly shows a wrong
classification for a true positive sample.

Accuracy represents the ratio of the correct classification,

$ACC=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}$
(8)

Specificity calculates the percentage of the FP that are identified by the
model,

$\text{ Specificity }=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}$ (9)

Sensitivity calculates the percentage of the TP that are identified by the
model,

$\text{ Sensitivity/ Recall }=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$
(10)

F1 score calculates the weighted average of the precision vs recall,

$\text{ F1 score
}=\frac{2*\mathrm{TP}}{2*\mathrm{TP}+\mathrm{FP}+\mathrm{FN}}$ (11)

mIoU measures the overlapped area of the predicted mask with the grand truth
divided by the union of the grand truth and predicted mask.

$mIOU=\frac{TP}{TP+FP+FN}$ (12)

DSC measures the similarity of the predicted mask and the grand truth.

$DSC=\frac{2TP}{2TP+FP+FN}$ (13)

###  IV-C Implementation details

To implement the proposed method we used the Pytorch library and carried out
the results on a single GPU RTX 3090 equipped System. Both the baseline model
and the proposed one are trained with a batch size 16 with an initial learning
rate $1e-3$ and the decay rate $1e-4$ for 100 epochs. All layers are
initialized using the normal distribution and data augmentation is not used
during the training process. To facilitate the training convergence we
included the batch normalization layer in each block of the encoding and
decoding paths. During the training the validation loss is monitored to stop
the training process if the validation loss did not decrease in ten successive
epochs. For the baseline model, we followed the common implementation of the
U-Net network and incorporated the proposed modules to present our network. It
is worthwhile to mention that the training process for all datasets converged
smoothly without any instability.

###  IV-D Comparison Results

In this section, the experimental results of each dataset are presented. To
provide a fair evaluation we followed the recent work and only included the
approaches which use the same settings for their performance evaluation.

####  IV-D1 Synapse multi-organ segmentation

Table 1 presents the comparison results of the proposed method against the
SOTA approaches. First, it is clear that our TransNorm achieves the best
results, with DSC and HD of 78.40% and 30.25%, respectively. Compared to the
recent TransU-Net variants, our method improves the average DSC and Hausdorff
distance by $0.98\%$ and 1.44 mm respectively. This may explain the importance
of the attention mechanism we incorporated on the skip connections of the
decoding path.
Second, the results presented demonstrate that the TransNorm model improves
both CNN and pure Transformer models by a large margin. The CNN models are
less capable of capturing the global contextual information and, consequently,
less precise in the boundary area. However, the pure Transformer model renders
a pool representation of the local information and yields a weak segmentation
mask. Combining both these networks and incorporating an attention mechanism,
our proposed method results in learning both semantic and global contextual
information with an additional adaptation (due to the attention mechanism)
which is crucial for the segmentation task.

Furthermore, for visual assessment, we have provided Figure 4 in which the
proposed method’s performance is closer to the ground truth, and in line with
the real situation. This visualization reveals the importance of the attention
mechanism we added to the skip connection module to effectively reconstruct
the object boundary on a highly overlapped background.

Figure 4: Segmentation result of the proposed method on Synaps dataset. From
Left to right, the images indicate the input image, grand truth and the
predicted mask. TABLE I: Comparision results of the proposed method on Synaps
multi-organ segmentation dataset.

Methods | DSC $\uparrow$ | HD $\downarrow$ | Aorta | Gallbladder | Kidney(L) | Kidney(R) | Liver | Pancreas | Spleen | Stomach
---|---|---|---|---|---|---|---|---|---|---
V-Net [47] | $68.81$ | $-$ | $75.34$ | $51.87$ | $77.10$ | $80.75$ | $87.84$ | $40.05$ | $80.56$ | $56.98$
R50 U-Net [4] | $74.68$ | $36.87$ | $87.74$ | $63.66$ | $80.60$ | $78.19$ | $93.74$ | $56.90$ | $85.87$ | $74.16$
R50 Att-UNet [4] | $75.57$ | $36.97$ | $55.92$ | $63.91$ | $79.20$ | $72.71$ | $93.56$ | $49.37$ | $87.19$ | $74.95$
Att-UNet [28] | $77.77$ | $36.02$ | $89.55$ | $68.88$ | $77.98$ | $71.11$ | $93.57$ | $58.04$ | $87.30$ | $75.75$
R50 ViT [4] | $71.29$ | $32.87$ | $73.73$ | $55.13$ | $75.80$ | $72.20$ | $91.51$ | $45.99$ | $81.99$ | $73.95$
TransUnet [4] | $77.48$ | $31.69$ | $87.23$ | $63.13$ | $81.87$ | $77.02$ | $94.08$ | $55.86$ | $85.08$ | $75.62$
Baseline | $76.85$ | $39.70$ | $89.07$ | $69.72$ | $77.77$ | $68.60$ | $93.43$ | $53.98$ | $86.67$ | $75.58$
TransNorm (Proposed) | $78.40$ | $30.25$ | $86.23$ | $65.10$ | $82.18$ | $78.63$ | $94.22$ | $55.34$ | $89.50$ | $76.01$

####  IV-D2 Skin Lesion Segmentation

In this section, we present the experimental results for skin lesion
segmentation. First, in Table 2 we present the comparison results for ISIC
2017. It can be seen that the proposed method outperforms the SOTA approaches
in almost all metrics. Compared to the U-Net model (the best CNN model for
this dataset), our method produces a better generalization performance and
improves the F1 score by 8$\%$. Furthermore, in comparison with the TransU-Net
model, our method exhibits a better performance which further proves the
effectiveness of the attention module we incorporated on the skip connection
parts.

TABLE II: Quantitative analysis of the proposed method against the SOTA
approaches for skin lesion segmentation on ISIC 2017.

Methods | DSC | SE | SP | ACC
---|---|---|---|---
U-Net [14] | 0.8159 | 0.8172 | 0.9680 | 0.9164
Att U-Net [28] | 0.8082 | 0.7998 | 0.9776 | 0.9145
DAGAN [48] | 0.8425 | 0.8363 | 0.9716 | 0.9304
TransUNet [4] | 0.8123 | 0.8263 | 0.9577 | 0.9207
MCGU-Net [49] | 0.8927 | 0.8502 | 0.9855 | 0.9570
MedT [50] | 0.8037 | 0.8064 | 0.9546 | 0.9090
FAT-Net [37] | 0.8500 | 0.8392 | 0.9725 | 0.9326
Proposed Method | 0.8933 | 0.8532 | 0.9859 | 0.9582

In Figure 5 we illustrate visual output for the method to further deliver a
clear view of the smooth segmentation results provided by our approach.

Figure 5: Segmentation results of the proposed method on ISIC 2017. We can
observe that the proposed method produces a precise segmentation map and a
smooth boundary for the skin lesion area.

In Table 3 we present the quantitative results of the proposed method against
the SOTA approaches. Similar to the ISIC 2018 dataset, our method outperforms
both CNN and Transformer-based approaches. To further investigate the
qualitative results we depict some segmentation results of the method in
Figure 6. The method produces precise segmentation results even when the
background has a high overlap with the skin lesion class.

TABLE III: Performance comparison of the suggested network against the SOTA
counterparts for skin lesion segmentation on the ISIC 2018 dataset.

Methods | DSC | SE | SP | ACC
---|---|---|---|---
U-Net [14] | 0.8545 | 0.8800 | 0.9697 | 0.9404
Att U-Net [28] | 0.8566 | 0.8674 | 0.9863 | 0.9376
DAGAN [48] | 0.8807 | 0.9072 | 0.9588 | 0.9324
TransUNet [4] | 0.8499 | 0.8578 | 0.9653 | 0.9452
MCGU-Net [49] | 0.895 | 0.848 | 0.986 | 0.955
MedT [50] | 0.8389 | 0.8252 | 0.9637 | 0.9358
FAT-Net [37] | 0.8903 | 0.9100 | 0.9699 | 0.9578
Proposed Method | 0.8951 | 0.8750 | 0.9790 | 0.9580

Figure 6: Segmentation samples of the proposed method on ISIC 2018. The method
separates the skin lesion from the overlapped background with high precision.

Finally, Table 4 shows the quantitative results for the PH2 dataset. The table
shows that, except for the sensitivity metric, the method exhibits a better
performance compared to the SOTA models.

TABLE IV: Performance comparison on $PH^{2}$ dataset.

Methods | DSC | SE | SP | ACC
---|---|---|---|---
U-Net [14] | 0.8936 | 0.9125 | 0.9588 | 0.9233
Att U-Net [28] | 0.9003 | 0.9205 | 0.9640 | 0.9276
DAGAN [48] | 0.9201 | 0.8320 | 0.9640 | 0.9425
TransUNet [4] | 0.8840 | 0.9063 | 0.9427 | 0.9200
MCGU-Net [49] | 0.9263 | 0.8322 | 0.9714 | 0.9537
MedT [50] | 0.9122 | 0.8472 | 0.9657 | 0.9416
FAT-Net [37] | 0.9440 | 0.9441 | 0.9741 | 0.9703
Proposed Method | 0.9437 | 0.9438 | 0.9810 | 0.9723

Figure 7 shows segmentation results of the proposed method where the skin
lesion area is predicted precisely. Notably, the method does not produce
isolated FP pixels and, remarkably, it separates the background from the skin
lesion area.

Figure 7: Segmentation samples of the proposed method on PH2. Segmentation
results shows that the model is highly capable in separating skin lesion area.

####  IV-D3 Multiple Mylomia Segmentation

We present the visual segmentation results in Figure 8.One can see in Figure 8
that TransNorm can describe mylomia precisely and generate more fitting
segmentation masks. This stems from its ability in modeling long-range spacial
correlations between each volume. Furthermore, we conduct quantitative
comparisons in terms of the mIOU of the proposed network with the winners of
the SegPC 2021 Challenge that utilized the mylomia dataset. As shown in Table
5, the proposed method obtained the best overall performance. It is worth
mentioning that the leading team (XLAB Insights [7]) propose to use a
combination of three instance segmentation networks (SCNet [51], ResNeSt [52],
and Mask-RCNN [53]). The second team (DSC-IITISM [7]) employs the Mask-RCNN
model with substantial data augmentation techniques while the third team,
(bmdeep [7]) uses an attention deeplabv3+ method [45] with a multi-scale
region-based training procedure. All these approaches require a large number
of parameters compared to our model. However, even with such merit those
methods render a poorer performance compared to our novel approach.

TABLE V: Performance evaluation on the SegPC challenge (best result is
highlighted).

Methods | mIOU
---|---
Frequency recalibration U-Net [41] | 0.9392
XLAB Insights [7] | 0.9360
DSC-IITISM [7] | 0.9356
Multi-scale attention deeplabv3+ [7] | 0.9065
U-Net [14] | 0.7665
Contexual attention [6] | 0.9395
Baseline | 0.9172
Proposed | 0.9399

Figure 8: Segmentation results of the proposed method on the SegPC challenge
dataset. The method performs well on the highly overlapped background.

###  IV-E Ablation Study

In this section, we discuss the influence of different design choices on the
model’s overall performance. To this end, we conducted a comprehensive
ablation study on the effect of the attention mechanism, skip connections,
input resolution, and model scaling using the ISIC 2018 dataset. In the next
subsections, we will elaborate on each factor in detail.

####  IV-E1 Attention Mechanism

We conducted ablation experiments to demonstrate the values of each
contribution and justify the rationale of its design choices towards overall
performance. The results, recorded on five metrics are presented in Table 6.
Our results reveal the significance of the proposed two-level attention
mechanism. In particular, we noticed that integrating both channel and spatial
information on the decoding paths would leverage the Transformer to model the
global relationships, while exploiting each of them solely lead to less
performance boost.

TABLE VI: Performance comparison on ISIC 2018 dataset.

Methods | DSC | AC
---|---|---
Baseline | 85.45 | 94.04
Baseline+Transformer | 86.64 | 94.74
Baseline+Transformer+Channel attention | 0.8771 | 0.9492
Baseline+Transformer+Spatial attention | 0.8821 | 0.9503
(proposed method) | 0.8951 | 0.9580

####  IV-E2 Skip Connection Influence

The objective of the skip connection in our model is to enhance the
segmentation performance by providing a low-level spatial information for the
decoding path. We argue that the number of skip connections incorporated in
our design can remarkably effect the model performance. To analyze such
influence, we have conducted several experiments using a varying number of
skip connections in the model structure. Table 7 shows the experimental
results. It can be seen that the model with three skip connections in 1/4,
1/8, and 1/16 resolution scales provides a better performance compared to the
other settings.

TABLE VII: The influence of the number of skip connections on the model
performance. Experiments on the ISIC 2018 dataset.

Skip Connection count | DSC | AC
---|---|---
0 | 0.8635 | 0.9440
1 | 0.8714 | 0.9494
2 | 0.8861 | 0.9522
3 | 0.8951 | 0.9580

####  IV-E3 Input Resolution Influence

The input resolutions utilized in our experiments for the skin lesion
segmentation follows the literature work [41, 48] and uses the input
resolution 256*256 pixels. However, the high-resolution input ($512\times
512$) samples can effectively influence the segmentation results as they
provide a finer details of the object of interest. In Table 8 we conducted an
experiment using higher image resolution to demonstrate the impact of this
factor on the model performance. Although the higher resolution provides a
better DSC score and justifies the model capability in learning a finer
segmentation map, it largely increases the computation complexity. Hence, the
performance of our model could be further increased, but this would demand
extra computational cost.

TABLE VIII: The influence of the input resolution on the model performance.
Experiments on ISIC 2018 dataset.

Input Resolution size | DSC | AC
---|---|---
$256\times 256$ | 0.8951 | 0.9580
($512\times 512$) | 0.9131 | 0.9650

####  IV-E4 Model Scaling Influence

In another setting, we evaluated the effect of network deepeing on the overall
performance. To this end, a larger version of the model with a higher number
of parameters is created to gain a better performance. However, the
experimental results (Table 9) demonstrated that increasing the number of
parameters only provides a better performance gain for the training set
(overfitting) and, consequently, little improvement is achieved at inference
time. Thus, model scaling is not efficient in terms of the computational
burden for our design.

TABLE IX: The influence of the input resolution on the model performance.
Experiments on the ISIC 2018 dataset.

Model Scale selection | DSC | AC
---|---|---
Base | 0.8951 | 0.9580
Larger version | 0.8963 | 0.9588

##  V Conclusion

In this paper, we presented our Transnorm model for the medical image
segmentation task. Our design uses the benefit of combining Transformer and
CNN features to encode both semantic and long-range contextual features. It
further utilizes an attention mechanism on the skip connection sections to
adaptively recalibrate the feature representation power and, thus, boost the
generalization performance. Experimental results on several datasets
demonstrate the effectiveness of our approach for smooth and precise
segmentation results which is crucial for a clinical application.

## References

  * [1] Yixuan Wu, Kuanlun Liao, Jintai Chen, Danny Z Chen, Jinhong Wang, Honghao Gao, and Jian Wu.  D-former: A u-shaped dilated transformer for 3d medical image segmentation.  arXiv preprint arXiv:2201.00462, 2022.
  * [2] Zhuangzhuang Zhang, Baozhou Sun, and Weixiong Zhang.  Pyramid medical transformer for medical image segmentation.  arXiv preprint arXiv:2104.14702, 2021.
  * [3] Reza Azad, Maryam Asadi-Aghbolaghi, Mahmood Fathy, and Sergio Escalera.  Bi-directional convlstm u-net with densley connected convolutions.  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [4] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou.  Transunet: Transformers make strong encoders for medical image segmentation.  arXiv preprint arXiv:2102.04306, 2021.
  * [5] Moein Heidari, Amirhossein Kazerouni, Milad Soltany, Reza Azad, Ehsan Khodapanah Aghdam, Julien Cohen-Adad, and Dorit Merhof.  Hiformer: Hierarchical multi-scale representations using transformers for medical image segmentation.  arXiv preprint arXiv:2207.08518, 2022.
  * [6] Azad Reza, Heidari Moein, Wu Yuli, and Merhof Dorit.  Contextual attention network: Transformer meets u-net.  arXiv preprint arXiv:2203.01932, 2022.
  * [7] Afshin Bozorgpour, Reza Azad, Eman Showkatian, and Alaa Sulaiman.  Multi-scale regional attention deeplab3+: Multiple myeloma plasma cells segmentation in microscopic images.  arXiv preprint arXiv:2105.06238, 2021.
  * [8] Reza Azad, Nika Khosravi, Mohammad Dehghanmanshadi, Julien Cohen-Adad, and Dorit Merhof.  Medical image segmentation on mri images with missing modalities: A review.  arXiv preprint arXiv:2203.06217, 2022.
  * [9] Reza Azad, Lucas Rouhier, and Julien Cohen-Adad.  Stacked hourglass network with a multi-level attention mechanism: Where to look for intervertebral disc labeling.  In International Workshop on Machine Learning in Medical Imaging, pages 406–415. Springer, 2021.
  * [10] Reza Azad, Moein Heidari, Julien Cohen-Adad, Ehsan Adeli, and Dorit Merhof.  Intervertebral disc labeling with learning shape information, a look once approach.  arXiv preprint arXiv:2204.02943, 2022.
  * [11] Reza Azad, Nika Khosravi, and Dorit Merhof.  Smu-net: Style matching u-net for brain tumor segmentation with missing modalities.  arXiv preprint arXiv:2204.02961, 2022.
  * [12] Bennett Landman, Zhoubing Xu, J Igelsias, Martin Styner, T Langerak, and Arno Klein.  Miccai multi-atlas labeling beyond the cranial vault–workshop and challenge.  In Proc. MICCAI Multi-Atlas Labeling Beyond Cranial Vault—Workshop Challenge, volume 5, page 12, 2015.
  * [13] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * [14] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [15] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [16] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving language understanding by generative pre-training.  2018\.
  * [17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [18] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * [19] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever.  Generative pretraining from pixels.  In International Conference on Machine Learning, pages 1691–1703. PMLR, 2020.
  * [20] Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R Roth, and Daguang Xu.  Unetr: Transformers for 3d medical image segmentation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 574–584, 2022.
  * [21] Foivos I Diakogiannis, François Waldner, Peter Caccetta, and Chen Wu.  Resunet-a: A deep learning framework for semantic segmentation of remotely sensed data.  ISPRS Journal of Photogrammetry and Remote Sensing, 162:94–114, 2020.
  * [22] Yue Cao, Shigang Liu, Yali Peng, and Jun Li.  Denseunet: densely connected unet for electron microscopy image segmentation.  IET Image Processing, 14(12):2682–2689, 2020.
  * [23] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang.  Unet++: A nested u-net architecture for medical image segmentation.  In Deep learning in medical image analysis and multimodal learning for clinical decision support, pages 3–11. Springer, 2018.
  * [24] Huimin Huang, Lanfen Lin, Ruofeng Tong, Hongjie Hu, Qiaowei Zhang, Yutaro Iwamoto, Xianhua Han, Yen-Wei Chen, and Jian Wu.  Unet 3+: A full-scale connected unet for medical image segmentation.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1055–1059. IEEE, 2020.
  * [25] Abdur R Feyjie, Reza Azad, Marco Pedersoli, Claude Kauffman, Ismail Ben Ayed, and Jose Dolz.  Semi-supervised few-shot learning for medical image segmentation.  arXiv preprint arXiv:2003.08462, 2020.
  * [26] Reza Azad, Abdur R Fayjie, Claude Kauffmann, Ismail Ben Ayed, Marco Pedersoli, and Jose Dolz.  On the texture bias for few-shot cnn segmentation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2674–2683, 2021.
  * [27] Özgün Çiçek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger.  3d u-net: learning dense volumetric segmentation from sparse annotation.  In International conference on medical image computing and computer-assisted intervention, pages 424–432. Springer, 2016.
  * [28] Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al.  Attention u-net: Learning where to look for the pancreas.  arXiv preprint arXiv:1804.03999, 2018.
  * [29] Fisher Yu and Vladlen Koltun.  Multi-scale context aggregation by dilated convolutions.  arXiv preprint arXiv:1511.07122, 2015.
  * [30] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881–2890, 2017.
  * [31] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794–7803, 2018.
  * [32] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu.  Dual attention network for scene segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3146–3154, 2019.
  * [33] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021.
  * [34] Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze.  Levit: a vision transformer in convnet’s clothing for faster inference.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12259–12269, 2021.
  * [35] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen.  Twins: Revisiting the design of spatial attention in vision transformers.  Advances in Neural Information Processing Systems, 34, 2021.
  * [36] Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang.  Swin-unet: Unet-like pure transformer for medical image segmentation.  arXiv preprint arXiv:2105.05537, 2021.
  * [37] Huisi Wu, Shihuai Chen, Guilian Chen, Wei Wang, Baiying Lei, and Zhenkun Wen.  Fat-net: Feature adaptive transformers for automated skin lesion segmentation.  Medical Image Analysis, 76:102327, 2022.
  * [38] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [39] Yucheng Tang, Dong Yang, Wenqi Li, Holger Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh.  Self-supervised pre-training of swin transformers for 3d medical image analysis.  arXiv preprint arXiv:2111.14791, 2021.
  * [40] Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al.  Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic).  In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pages 168–172. IEEE, 2018.
  * [41] Reza Azad, Afshin Bozorgpour, Maryam Asadi-Aghbolaghi, Dorit Merhof, and Sergio Escalera.  Deep frequency re-calibration u-net for medical image segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3274–3283, 2021.
  * [42] Md Zahangir Alom, Mahmudul Hasan, Chris Yakopcic, Tarek M Taha, and Vijayan K Asari.  Recurrent residual convolutional neural network based on u-net (r2u-net) for medical image segmentation.  arXiv preprint arXiv:1802.06955, 2018.
  * [43] Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al.  Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic).  arXiv preprint arXiv:1902.03368, 2019.
  * [44] Teresa Mendonça, Pedro M Ferreira, Jorge S Marques, André RS Marcal, and Jorge Rozeira.  Ph 2-a dermoscopic image database for research and benchmarking.  In 2013 35th annual international conference of the IEEE engineering in medicine and biology society (EMBC), pages 5437–5440. IEEE, 2013\.
  * [45] Reza Azad, Maryam Asadi-Aghbolaghi, Mahmood Fathy, and Sergio Escalera.  Attention deeplabv3+: Multi-level context attention mechanism for skin lesion segmentation.  In European Conference on Computer Vision, pages 251–266. Springer, 2020.
  * [46] Anubha Gupta, Pramit Mallick, Ojaswa Sharma, Ritu Gupta, and Rahul Duggal.  Pcseg: Color model driven probabilistic multiphase level set based tool for plasma cell segmentation in multiple myeloma.  PloS one, 13(12):e0207908, 2018.
  * [47] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi.  V-net: Fully convolutional neural networks for volumetric medical image segmentation.  In 2016 fourth international conference on 3D vision (3DV), pages 565–571. IEEE, 2016.
  * [48] Baiying Lei, Zaimin Xia, Feng Jiang, Xudong Jiang, Zongyuan Ge, Yanwu Xu, Jing Qin, Siping Chen, Tianfu Wang, and Shuqiang Wang.  Skin lesion segmentation via generative adversarial networks with dual discriminators.  Medical Image Analysis, 64:101716, 2020.
  * [49] Maryam Asadi-Aghbolaghi, Reza Azad, Mahmood Fathy, and Sergio Escalera.  Multi-level context gating of embedded collective knowledge for medical image segmentation.  arXiv preprint arXiv:2003.05056, 2020.
  * [50] Jeya Maria Jose Valanarasu, Poojan Oza, Ilker Hacihaliloglu, and Vishal M Patel.  Medical transformer: Gated axial-attention for medical image segmentation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 36–46. Springer, 2021.
  * [51] Thang Vu, Haeyong Kang, and Chang D Yoo.  Scnet: Training inference sample consistency for instance segmentation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2701–2709, 2021.
  * [52] Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Haibin Lin, Zhi Zhang, Yue Sun, Tong He, Jonas Mueller, R Manmatha, et al.  Resnest: Split-attention networks.  arXiv preprint arXiv:2004.08955, 2020.
  * [53] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.