.jpg

[1]Zhenguo Gao

[1]College of Computer Science and Technology, Huaqiao University, No.668
Jimei Avenue, Xiamen, 361021, Fujian, China

# Few-Shot Medical Image Segmentation with Large Kernel Attention

Xiaoxiao Wu<EMAIL_ADDRESS>Xiaowei Chen<EMAIL_ADDRESS><EMAIL_ADDRESS>Shulei Qu<EMAIL_ADDRESS>Yuanyuan Qiu<EMAIL_ADDRESS>*

###### Abstract

Medical image segmentation has witnessed significant advancements with the
emergence of deep learning. However, the reliance of most neural network
models on a substantial amount of annotated data remains a challenge for
medical image segmentation. To address this issue, few-shot segmentation
methods based on meta-learning have been employed. Presently, the methods
primarily focus on aligning the support set and query set to enhance
performance, but this approach hinders further improvement of the model’s
effectiveness. In this paper, our objective is to propose a few-shot medical
segmentation model that acquire comprehensive feature representation
capabilities, which will boost segmentation accuracy by capturing both local
and long-range features. To achieve this, we introduce a plug-and-play
attention module that dynamically enhances both query and support features,
thereby improving the representativeness of the extracted features. Our model
comprises four key modules: a dual-path feature extractor, an attention
module, an adaptive prototype prediction module, and a multi-scale prediction
fusion module. Specifically, the dual-path feature extractor acquires multi-
scale features by obtaining features of 32*32 size and 64*64 size. The
attention module follows the feature extractor and captures local and long-
range information. The adaptive prototype prediction module automatically
adjusts the anomaly score threshold to predict prototypes, while the multi-
scale fusion prediction module integrates prediction masks of various scales
to produce the final segmentation result. We conducted experiments on publicly
available MRI datasets, namely CHAOS and CMR, and compared our method with
other advanced techniques. The results demonstrate that our method achieves
state-of-the-art performance.

###### keywords:

Few-shot Segmentation, Attention Mechanism, Deep Learning

##  1 Introduction

Convolutional neural networks (CNNs) have been widely used in deep learning
for a variety of image segmentation tasks, particularly within the medical
field. Medical research, clinical diagnosis, and case analysis all benefit
greatly from the use of medical image segmentation. Quantitative measurement
and analysis of relevant imaging indicators before and after therapy are
crucial for medical diagnosis and patient treatment plan revision. Although
CNNs have proven successful for segmentation tasks[1, 2, 3, 4], most current
models rely heavily on annotated data. However, in medical filed, data
labeling often requires manual work by medical experts, making it expensive
and time-consuming[5, 6]. Therefore, scholars have been exploring ways to
complete the image segmentation task with limited amount of annotation data.

Few-shot learning(FSS), also known as learn to learn, is a way to learn from a
few annotated examples and generalize to new classes. Most models of few-shot
segmentation are constructed by meta-learning techniques (Meta-FSS) [7, 8, 9,
10, 11, 12]. In the meta-training stage, the model can learn task-independent
generation ability by constructing different meta-tasks. Each task (also known
as a scenario) has a support set and a query set. After learning the
representation of each class on the annotated support set, the query set is
segmented using the previously acquired information. With this paradigm, it is
possible to segment a previous “unseen” organ with a few annotated images in
the test stage. However, in order to avoid overfitting, the Meta-FSS methods
rely on a large number of annotated data, which is unrealistic in the medical
field.

To solve this problem, Ouyang et al. [7] creatively proposed a self-supervised
training approach to bypass model’s need for large amounts of labeled data.
Specifically, they created image pseudo labels using superpixels and trained
with those labels. Superpixel is a type of short and compact image fragment,
and the pseudo labels produced by superpixel typically contain real semantic
information, which is helpful to the training of the model. Based on Ouyang’s
work, Hansen et al.[8] further proposed the concept of supervoxel. By using
supervoxel to generate pseudo labels, local information and volume information
of the images can be both used. Supervoxel segmentation has gradually gained
popularity in the medical image segmentation in recent years[13, 14, 15].
Their proposed approaches effectively address the issue of data annotation,
yet the challenge of data scarcity persists.

There are two major factors that hinder model performance: intra-class and
inter-class gaps[16].The former arises from insufficient data in the support
set, resulting in disparities between the support set and query set. The
latter stems from non-overlapping categories between the training and test
sets. To minimum these gaps, previous studies have introduced attention
mechanisms[17, 18, 19, 32], while others have adopted multi-scale methods[9,
20, 21]. Prior work has utilized attention mechanisms to learn common features
between query sets and support sets for reducing intra-class gaps. Although
this method has shown some effectiveness, it struggles to adapt to complex and
variable medical sections encountered in clinical settings where even
identical organ slices can vary significantly. Moreover, scarce medical data
can bias model focus due to subtle differences between support and query
images, particularly in 1-shot scenarios. In this study, we propose a multi-
scale approach based on the large kernel attention(LKA) mechanism by combining
these two methods. Different from previous approaches, our method employs
attention mechanisms to extract more comprehensive features, allowing the
model itself to determine their importance. Additionally, we emphasize the
importance of constructing a high-quality foreground in medical image
segmentation, given the pronounced foreground-background imbalance. By
integrating multi-scale information, local information, and long-range
information, our method achieves a comprehensive feature representation,
thereby attaining state-of-the-art performance.

In summary, we propose a multi-scale model based on the large kernel attention
mechanism, as shown in Figure 1. The model consists of four modules: dual-path
feature extractor, attention module, adaptive prototype prediction module and
multi-scale prediction fusion module. Among them, the attention module follows
the feature extractor, which can not only enhance the semantic representation
of feature vectors, but also capture local features and long-range semantic
features. The adaptive prototype prediction module can predict the prototype
by automatically adjusting the threshold of anomaly scores, and the multi-
scale fusion prediction module can combine the prediction masks of different
scales to get the final prediction segmentation result. We test the model on
the public datasets.

The main contributions of this paper are as follows:

(1)We propose a large kernel attention module to enhance support features and
query features, which can capture both local features and long-range semantic
features.

(2)By integrating the large kernel attention mechanism with multi-scale
analysis, our approach enables the extraction of features encompassing local
and long-range information as well as multi-scale information, thereby
achieving a comprehensive representation that surpasses existing methods.

(3)We propose an effective few-shot segmentation model and obtain state-of-
the-art performance on two widely used organ segmentation datasets.

Figure 1: The model we proposed.

##  2 Related Work

We first review the development of convolutional neural networks in the
medical field. Then we focus on the few-shot learning and introduce the recent
development of few-shot medical segmentation model. We also introduce the
application of attention mechanisms in medical image segmentation, focusing on
more widely used attention methods. Finally, we introduce the segmentation
method based on supervoxels and its advantages briefly.

###  2.1 Medical Image Segmentation

In recent years, deep learning technology has attracted more and more
researchers’attention. In the medical field, it is common to use convolutional
neural networks to perform segmentation tasks. Full convolutional network
(FCN) [3] is a kind of network with strong learning ability, and it is also
the first network to realize end-to-end pixel level segmentation. U-Net,
proposed by Ronneberger et al.[22], uses a symmetrical encoder-decoder
structure. In the contraction path, features are extracted using convolution.
In the expansion path, the expression ability of the feature map is enhanced
by upsampling the feature map and concatenating it with the corresponding
feature map from the contraction path. The final feature map is resized to the
original image size for segmentation. U-Net has inspired further research in
this field. U-Net’s variants include Res-Unet[25], Dense-Unet[26] and
MultiRes-UNet[27] et al. Çiçek et al.[23] propose 3D U-Net, replacing all
2-dimensional operations in the original U-Net with 3-dimensional operations
to achieve volume segmentation. Milletari et al.[24] propose the V-net
structure. In addition, they propose a new objective function to deal with the
severe imbalance between foreground and background voxels.

###  2.2 Few-shot Semantic Segmentation

Few-shot semantic segmentation extends few-shot learning from classification
task to segmentation task. Specifically, given a set of labeled data, the task
of few-shot segmentation is to segment objects on a new class. In some recent
studies, Wang et al.[10] propose a prototype alignment network (PANet) from
the perspective of metric learning to better utilize the information of
support sets. Based on PANet, Ouyang[7] and Hansen[8] propose segmentation
models based on superpixels and supervoxels by using self-supervised meta-
learning method. Among them, Hansen et al.[8] believe that since the
foreground types of medical images (such as organs) are relatively
homogeneous, the imbalance between foreground and background is prominent.
Therefore, they can avoid modeling the background and only build a prototype
for the same structure of the foreground class. Specifically, only one class
prototype is built in each ”scenario” and an anomaly score is introduced to
measure the difference between the foreground prototype and the query feature.
Shen et al.[9], inspired by the learning mechanism of clinical experts,
believes that the class prototype should be dynamically adjusted during the
test stage to adapt to the differences between the training set and the test
set. Protonotarios et al.[11] use the UNet architecture to solve lung cancer
lesion segmentation problems, the difference is that it can continuously
adjust the weights of the model based on user feedback, aiming to achieve the
best performance. Li et al.[12] utilize the feature pyramid technique, the
sensitivity of the multi-scale feature space is enhanced for better
performance in segmenting lung cancer lesions. The method proposed by Ding et
al.[32] is modified on the basis of non-local block. By intensively comparing
the similarity between query pixels and support pixels, the query features and
support features are selectively enhanced.

###  2.3 Attention Mechanisms

Attention mechanism has been widely used since it was proposed. In a nutshell,
the main idea of the attention mechanism is to highlight important features
and ignore irrelevant ones. Recently, some studies have proposed attentional
mechanisms that can enhance the network’s focus on segmented objects at little
cost. For example, the CBAM module proposed by Woo et al.[17] can infer the
attention maps along the two dimensions of channel and space, and then carry
out adaptive feature fusion. Due to its low cost, it can be easily embedded
into other networks to achieve end-to-end training. Attention Gate is first
used in natural image analysis, knowledge graph and classification tasks. Roy
et al.[18] use dense connections with squeeze and excitation blocks to make
the interaction between the two branches stronger and achieve better
performance. Sinha et al.[20] believe that the multi-scale method applied in
the previous model is not effective because of the redundancy of information
caused by the constant extraction of similar low-level features. However, the
application of attention mechanism can well highlight the features of the
segmented area and suppress other irrelevant parts. Therefore, the author uses
a relatively perfect attention mechanism to make the model learn richer
context information.

###  2.4 Supervoxel Segmentation

Both superpixels and supervoxels are collections of local pixels and voxels in
an image. Specifically, a superpixel is a small, compact fragment of an image,
usually obtained by clustering a local pixel by applying a statistical model.
The supervoxel is the subvolume in the 3D image, which represents the similar
voxel group in the local area of the image volume. The advantage of three-
dimensional voxel is that it can make better use of the volumetric properties
of medical images. In recent years, the use of supervoxel segmentation has
gradually become a common method in the field of medical image
segmentation[13, 14, 15].

##  3 Methods

In this work, we propose a prototypical network based on large kernel
attention mechanisms.

The model consists of these main parts: (1) Dual-path feature extractor, which
extracts features of 64*64 and 32*32 scale for both support images and query
images; (2) attention module, which captures both local and long-range
characteristics; (3) Adaptive prototype prediction layer, which uses class
prototypes and adaptive anomaly scores to separate foreground and background;
and (4) multi-scale prediction fusion layer, which fuses prediction masks of
different scales for final segmentation results. See Figure 2 for a detailed
illustration of the model.

###  3.1 Problem Definition

In few-shot segmentation, a segmentation model is initially trained on the
training dataset $D_{{train}}$ and subsequently evaluated on the test dataset
$D_{{test}}$ in episodes. The training dataset $D_{{train}}$ comprises
training classes $C_{{train}}$ , while the test dataset $D_{{test}}$ includes
test classes $C_{{test}}$ . It’s worth noting that the training classes
${C_{{train}}}$ and the test classes ${C_{{test}}}$ are disjoint, denoted as
${C_{train}}\cap{C_{test}}$ = $\emptyset$.

Each episode consists of a support set
$S{}_{i}=\\{(x_{1}^{j},m_{1}^{j}),(x_{2}^{j},m_{2}^{j}),\ldots,(x_{k}^{j},m_{k}^{j})\\}$
and a query set $Q=\\{{x_{q}^{j}},{m_{q}^{j}}\\}$ with the same class
${c_{j}}\in{C_{train}},j=1,2,\ldots,N$. Let image
$X=\\{x_{1}^{j},x_{2}^{j},...,x_{k}^{j},x_{q}^{j},x_{s},x_{q}\\}$, $x\in X$,
and mask $M=\\{m_{1}^{j},m_{2}^{j},...,m_{k}^{j},m_{q}^{j},m_{s},m_{q}\\}$,
$m\in M$, where the image ${x}\in\mathbb{R}{{}^{H\times W\times 1}}$ is a
single-channel grayscale image, and the binary mask of the corresponding
category is $m\in{\\{0,1\\}^{H\times W}}$. Since each support set ${S_{i}}$
contains K annotated images and the number of object classes being segmented
in query set Q is N, this is also called N-way K-shot. In general, the model
first learns semantic information about each category from the support set and
then uses the learned knowledge to segment the query set. The query mask is
invisible to the model and will only be used to evaluate the model in the test
stage. Specifically for our experiment, we adopt 1-way 1-shot meta-learning
strategy.

Figure 2: Illustration of the model. The model integrates feature information
from two scales, 64*64 and 32*32. The feature flow of the 64*64 scale is
illustrated here. The model consists of three layers: feature extractor and
attention layer, adaptive prototype prediction layer, and multi-scale
prediction fusion layer. The feature extractor and attention module extract
comprehensive feature representations of query and support images. The
adaptive prototype prediction layer generates predicted segmentation masks by
calculating cosine similarity between foreground prototype and query feature
and obtaining an adaptive threshold T. The multi-scale prediction fusion layer
upsamples masks of different scales and fuses them to produce the final mask
prediction result.

###  3.2 Dual-path Feature Extractor

We employ a dual-scale feature extractor ${f_{\theta}}$ to extract features at
both 64*64 and 32*32 scales from both support and query images. The extracted
support image features ${F_{s}}={f_{\theta}}({x^{s}})$,
${F_{s}}\in\mathbb{R}{{}^{H^{\prime}\times W^{\prime}\times Z}}$,
${H^{\prime}}$ represents the height of the extracted feature map,
${W^{\prime}}$ represents the width of the extracted feature map, and Z
represents the number of channels of the feature map. Corresponding, query
image features ${F_{q}}={f_{\theta}}({x^{q}})$. We extracted 64*64 (1/4 the
size of the original picture) and 32*32 (1/8 the size of the original image)
feature maps from the Resnet-101 pre-trained on Coco dataset network,
respectively, then fused them to produce multi-scale features.

Figure 3: Attention module. Where, DW-Conv represents the depth-wise
convolution, DW-D-Conv represents the depth-wise-dilation convolution, and 1×1
Conv represents the 1*1 convolution.

###  3.3 Attention Module

We aim to extract more comprehensive semantic features by integrating local
and long-range features in our dual-path feature extractor. Inspired by the
work of Park et al.[28] and Guo et al.[29], we use attention methods at early
layers to focus on local information and decompose large convolution kernels
to capture long-range relationships. This results in enhanced support and
query features, noted as ${F_{s}^{{}^{\prime}}}$ and ${{F_{q}}^{{}^{\prime}}}$
respectively. ${F_{s}^{{}^{\prime}}}$ and ${{F_{q}}^{{}^{\prime}}}$ are
obtained by subjecting support and query features to a 1*1 convolution
followed by a GELU activation function. This activation function has the form:

$\text{GELU}(x)=\frac{1}{2}\cdot
x\left(1+\text{erf}\left(\frac{x}{\sqrt{2}}\right)\right)$ (1)

The error function can be expressed as:

$\text{erf}(x)=\frac{2}{\sqrt{\pi}}\int_{0}^{x}e^{-t^{2}}dt$ (2)

The article describes a method to capture local, long-range, and channel
features using a large kernel attention (LKA) module comprising a depth-wise
convolution, a depth-wise-dilation convolution, and a 1*1 general convolution.
Unlike traditional convolution methods, each input layer channel is
independently convolved using depth-wise convolution, while depth-wise-
dilation expands the receptive field. The 1*1 convolution is used for self-
adaptability of channel dimension. The LKA mechanism combines local, long-
range, and channel information to create a comprehensive feature
representation.[29] It can be expressed as:

$LKA(F)=Conv{{}_{1\times 1}}(\rm DW-D-Conv(\rm DW-Conv(F)))$ (3)
$Output=LKA\otimes F$ (4)

Where, $F\in\mathbb{R}{{}^{C\times H\times W}}$ represents the input features,
$LKA(F)\in\mathbb{R}{{}^{C\times H\times W}}$ represents the attention feature
map of $F$. $\otimes$ represents element-level multiplication. After element-
level multiplication between attention feature map and support features or
query features, the enhanced support features ${F_{s}^{{}^{\prime}}}$ and
query features ${F_{q}^{{}^{\prime}}}$ are obtained. By using this attention
module, not only the spatial dimension but also the channel dimension can be
self-adaptive. Finally, the enhanced feature vectors are spliced into the
prototype segmentation module. The attention module is shown in Figure 3.

###  3.4 Adaptive Prototype Prediction Module

To deal with the high homogeneity of medical images, a single prototype is
constructed in each episode, as seen in Hansen[8] and Shen et al[9]. Metric
learning techniques are employed to learn prototype and segment the image. The
prototype for a class is calculated using masked averaging pooling between
support feature ${F_{s}^{{}^{\prime}}}$ and support masks ${m_{s}}$:

${\rm{p}}=\frac{{\sum{{}_{x,y}F_{\rm{s}}^{{}^{\prime}}(x,y)}\odot{m_{s}}(x,y)}}{{\sum{{}_{x,y}{m_{s}}(x,y)}}}$
(5)

Where ${F_{s}^{{}^{\prime}}}$ is scaled to the mask ${m_{s}}$ size (H, W), and
$F_{\rm{s}}^{{}^{\prime}}(x,y)$ represents the pixels in the support feature
${F_{s}^{{}^{\prime}}}$. And ${\odot}$ represents the Hadamard product.

Then, we use the obtained class prototype to segment the query features. By
computing the negative cosine similarity between the class prototype p and the
query feature ${F_{q}^{{}^{\prime}}}$, we created the anomaly score S:

${\rm{S(x,y)=-}}\alpha\frac{{F_{q}^{{}^{\prime}}(x,y)\cdot
p}}{{\left\|{F_{q}^{{}^{\prime}}(x,y)}\right\|\left\|p\right\|}}$ (6)

Where ${F_{q}^{{}^{\prime}}(x,y)}$ represents pixels in the query features,
and ${\alpha=20}$, is a scaling factor. The anomaly score is thresholded by
the parameter T to obtain predicted foreground and background masks,
${\tilde{m}_{f}^{q}}$ and ${\tilde{m}_{b}^{q}}$.

$\tilde{m}_{f}^{q}=(1-\sigma(S(x,y)-T))$ (7)
$\tilde{m}_{b}^{q}=1-\tilde{m}_{f}^{q}$ (8)

Where ${\sigma}$ represents a shifted sigmoid function, the subscript f and b
represent the foreground and background. T is the adaptive threshold learned
by the model according to the query feature ${F_{q}^{{}^{\prime}}}$. T is
obtained by adding a fully connected layer attached to the top of the feature
extractor ${{f_{\theta}}}$.

###  3.5 Multi-scale Prediction Fusion Module

The model predicts foreground and background masks through two paths based on
64*64 and 32*32 feature maps. The predicted masks ${\tilde{m}_{64}^{q}}$ and
${\tilde{m}_{32}^{q}}$ are up-sampled to the original image size (H,W) to
obtain ${\hat{m}_{64}^{q}}$ and ${\hat{m}_{32}^{q}}$. The final mask
${\hat{m}^{q}}$ is combined with an equilibrium factor ${\alpha}$:

${\hat{m}_{q}}=\alpha\cdot{\hat{m}^{q}}_{64}+(1-\alpha)\cdot\hat{m}_{32}^{q}$
(9)

Where the subscripts 64 and 32 represent the dimensions are 64*64 and 32*32
respectively. ${\alpha\in(0,1.0)}$, its value can be adjusted manually. In the
experimental section, we did several experiments to get the optimal
${\alpha}$.

###  3.6 Loss Function

We used the cross entropy loss of the predicted mask and the true mask as the
segmentation loss during the training stage:

$\displaystyle\begin{split}{L_{seg}}=-\frac{1}{{HW}}\sum\limits_{x,y}{m_{f}^{q}}(x,y)\log(\hat{m}_{f}^{q}(x,y))+\\\
m_{b}^{q}(x,y)\log(\hat{m}_{b}^{q}(x,y))\end{split}$ (10)

To align query and support images, we introduce an alignment regularization
loss based on Wang et al.’s[10] proposal. It involves segmenting support
images based on prototype information from query images. The alignment loss is
calculated as shown below:

$\displaystyle\begin{split}{L_{reg}}=-\frac{1}{{HW}}\sum\limits_{x,y}{m_{f}^{s}}(x,y)\log(\hat{m}_{f}^{s}(x,y))+\\\
m_{b}^{s}(x,y)\log(\hat{m}_{b}^{s}(x,y))\end{split}$ (11)

Finally, the Loss can be expressed as:

$Loss={L_{seg}}+{L_{reg}}$ (12)

Table 1: Comparison of our proposed method with other methods on the CMR
dataset.

Settings | Method | LV-BP | LV-MYO | RV | Mean Dice
---|---|---|---|---|---
Setting 1 | ALPNet[7] | 83.99±1.43 | 66.74±3.37 | 79.96±5.89 | 76.90±3.56
| ADNet[8] | 86.62±1.72 | 62.41±3.35 | 74.93±4.64 | 74.65±3.24
| Q-Net[9] | 89.91±0.88 | 64.42±1.50 | 75.70±1.61 | 76.68±1.33
| CRAPNet[32] | 86.74±2.12 | 59.14±1.38 | 73.74±2.10 | 73.21±1.87
| Proposed | 89.95±0.83 | 66.47±1.06 | 78.15±1.46 | 78.19±1.12

Table 2: Comparison of our proposed method with other methods on the CHAOS
dataset.

Settings | Method | Liver | R.kidney | L.kidney | Spleen | Mean Dice
---|---|---|---|---|---|---
Setting 1 | ALPNet[7] | 73.98±3.03 | 83.59±2.41 | 81.92±2.93 | 71.26±7.79 | 77.69±4.04
| ADNet[8] | 81.88±0.77 | 79.84±14.87 | 66.24±14.92 | 71.87±13.19 | 74.96±10.94
| Q-Net[9] | 84.41±1.58 | 88.92±1.14 | 77.46±8.04 | 70.02±9.65 | 80.20±5.94
| CRAPNet[32] | 82.63±2.01 | 85.04±1.36 | 78.23±6.34 | 70.52±7.94 | 79.10±4.41
| Proposed | 84.05±2.12 | 89.43±0.71 | 80.72±7.79 | 74.13±6.82 | 82.08±4.36
Setting 2 | ALPNet[7] | 71.74±2.34 | 73.30±7.55 | 61.30±5.13 | 62.98±4.64 | 67.33±4.92
| ADNet[8] | 77.12±4.43 | 61.11±5.91 | 65.93±6.41 | 62.65±6.47 | 66.70±5.81
| Q-Net[9] | 78.25±4.81 | 48.98±4.11 | 55.44±4.26 | 62.57±8.73 | 61.31±5.48
| CRAPNet[32] | 80.09±3.52 | 57.83±3.16 | 63.64±3.48 | 60.97±7.81 | 65.63±4.49
| Proposed | 80.44±3.54 | 71.39±4.36 | 72.89±6.47 | 66.00±7.10 | 72.68±5.37

##  4 Results

###  4.1 Datasets

We conducted experiments on representative public datasets CHAOS [30] and CMR
[31]:

(1) The 2019 ISBI Combined Healthy Abdominal Organ Segmentation Challenge
(CHAOS) task 5 was launched for abdominal organ segmentation. 20 3D T2
sequence scans from the liver, left kidney, right kidney, and spleen were
included, with each scan averaging 36 slices.

(2) The MICCAAI 2019 Multi-sequence Heart MRI Segmentation Challenge published
the CMR dataset, which was an MRI dataset of heart organ segmentation. It
consisted of 35 heart MRI scans, each with around 13 slices.

To ensure fairness, we used the same pre-processing scheme as Ouyang [7],
which involved deleting the bright end of the histogram to solve resonance
issues, resampling images to the same resolution, and unifying all images to
256x256 pixels. Additionally, since the pre-training network required three-
channel input, we repeated each single-channel medical image slice three times
in the channel dimension.

Models undergo five-fold cross-validation after self-supervised training using
supervoxel-based pseudo labels created offline. Minimum supervoxel size is
controlled by parameter during supervoxel generation, affecting segmentation
outcome. CHAOS set to 5000 and CMR set to 1000 based on Hansen’s advice.
Three-dimensional image is sliced and segmented during evaluation with one
volume chosen as support and the rest as queries. Each fold is repeated three
times to account for randomness.

###  4.2 Evaluation Metric

We used the mean dice score as the evaluation criterion in line with standard
medical segmentation procedures. It contrasts the projected segmentation A
with the actual segmentation B’s overlap ratio. The equation reads as follows:

${\rm{D}}(A,B)=2\frac{{|A\cap B|}}{{|A|+|B|}}\cdot 100\%$ (13)

Where the higher the Dice score, the better the segmentation effect.

ADNet[8]

Q-Net[9]

CRAPNet[32]

Proposed

GT

Figure 4: Comparison of segmentation results from Setting1 on CMR dataset.
From left to right: ADNet, Q-Net, CRAPNet, our proposed method and GT. From
top to bottom: LV-MYO, LY-BP and RV.

ALPNet[7]

ADNet[8]

Q-Net[9]

CRAPNet[32]

Proposed

GT

Figure 5: Comparison of segmentation results from Setting1 on CHAOS dataset.
From left to right: ALPNet, ADNet, Q-Net, CRAPNet, proposed model and GT. From
top to bottom: Liver, R.kidney, L.kidney, and Spleen.

ALPNet[7]

ADNet[8]

Q-Net[9]

CRAPNet[32]

Proposed

GT

Figure 6: Comparison of segmentation results from Setting2 on CHAOS dataset.
From left to right: ALPNet, ADNet, Q-Net, CRAPNet, proposed model and GT. From
top to bottom: Liver, R.kidney, L.kidney, and Spleen.

###  4.3 Comparison to state-of-the-art

We compare our method with the following advanced models: ALPNet[7], ADNet[8],
Q-Net[9] and CRAPNet[32]. Table 1 and Table 2 display the results.

To evaluate the performance of the model, two settings were used. In setting
1, test class objects may already appear in the training set’s background as
supervoxels unavoidably contain various information. Thus, the test classes in
setting 1 are not entirely “unseen” and evaluation was higher than in setting
2. In setting 2, to ensure the test classes are truly unseen, the slice
containing their images was deleted from the training set. The abdominal organ
segmentation dataset was split into upper and lower abdomen organs, following
Ouyang’s approach.

For CMR dataset, we only consider setting 1 because setting 2 is difficult to
implement. The results are shown in Table 1 and Figure 4. In general, our
proposed method is superior than others. For CHAOS dataset, we consider both
setting 1 and setting 2. From the results, we can see the performance of our
proposed model is higher than others on CHAOS and CMR datasets. Compared with
Q-Net, our mean dice score on CHAOS dataset achieves about 82%. The
segmentation results of the model are shown in Figure 5. In setting 2, the
slices of the objects containing the test classes are all removed at training
stage, and our model still performs well. As can be seen from the Table 2, the
segmentation effect of the model on the right kidney is slightly worse than
that of ALPNet, but overall, we achieve the best performance. The specific
segmentation results are shown in Figure 6. As can be seen from the figure,
other methods have excessive segmentation, but ours does not. Our segmentation
results are more accurate and reduce redundant segmentation.

Table 3: Ablation study of different modules on CHAOS dataset under setting 2.
DP is Dual-path extractor, and AM is an Attention Module.

Experiment | Module | Liver | R.kidney | L.kidney | Spleen | Mean Dice
---|---|---|---|---|---|---
1 | 32×32 | 78.67 | 67.62 | 64.38 | 62.20 | 68.22±5.19
2 | 64×64 | 79.07 | 68.32 | 65.69 | 63.35 | 69.11±8.36
3 | DP | 79.84 | 69.73 | 69.15 | 64.74 | 70.87±7.65
4 | DP+AM | 80.44 | 71.39 | 72.89 | 66.00 | 72.68±6.63

Table 4: Ablation study of ${\alpha}$. ${\alpha}$ | Setting2
---|---
0.9 | 72.21±7.06
0.8 | 72.68±6.63
0.6 | 67.36±6.82
0.4 | 72.30±7.40
0.2 | 66.70±8.68

###  4.4 Ablation Study

Firstly, we conduct experiments to verify the effectiveness of the dual-path
feature extractor, the attention module and the adaptive prototype prediction
module. We use CHAOS dataset to conduct the experiment in setting 2. The
results are shown in Table 3. As you can see from the table, each module does
contribute to improving the final performance of the model.

Then, we test the effect of the balance factor ${\alpha}$ on the model. At the
last layer of the model, namely the multi-scale prediction fusion layer, the
prediction results obtained from the feature maps of different sizes will be
fused. Where, ${\alpha}$ represents the proportion of mask predicted according
to the feature map size 64*64 in the final prediction result. Considering the
different sizes of the partitioned organs, which may lead to different results
of the model, we conducted many experiments to verify the effect under setting
2, and the results are shown in Table 4. As we can see from the table, the
optimal value is 0.8. Specifically, the model works well when ${\alpha}$ is
0.4 or 0.8, and 0.8 has the best performance. Of the four organs to be
segmented, the upper abdominal organs (liver and spleen) are both larger,
while the lower abdominal organs (left and right kidneys) are smaller. As the
value of ${\alpha}$ rises, the segmentation impact of the upper abdominal
organs is enhanced by the model’s usage of a larger segmentation mask, while
the segmentation effect of the lower abdominal organs is diminished. The model
works best at balancing different organs when ${\alpha}$ is 0.8. Besides, the
model’s performance improved by setting the kernel size of the attention
module’s DW convolution and DW-D convolution to 3, instead of the original 5
based on guo et al.’s[29] experience. Large scale convolution caused the model
to focus excessively on larger organs like the liver, leading to a good
performance in the upper abdomen but a poor performance in the lower abdomen,
ultimately affecting the overall performance negatively.

##  5 Conclusion

We presented a plug-and-play attention module utilizing a prototype network
technique based on the work of Shen et al. This attention module is capable of
adaptive enhancement of support features and query features to capture not
only local features but also long-range features to obtain comprehensive
feature representation. In general, our model starts from the aspects of
multi-scale and multi-dimension, making the extracted prototype more
representative and achieving the most advanced performance on the CHAOS and
CMR datasets. However, some difficulties still exist. First, attention
mechanisms may not be as effective for fine segmentation since they frequently
concentrate on the more crucial components. For smaller organs and the
division of organ margins, for instance, using attention mechanisms alone is
insufficient. Secondly, for the few-shot image segmentation, the most urgent
problem to be solved should be the scarcity of medical annotated data.
Subsequent work could consider data enhancement strategies or the use of GANs
to generate high quality images. In addition, Data Heterogeneity, Model
Interpretability and Generalizability and Robustness are also important
issues. Addressing these challenges requires collaboration between
researchers, hospitals, and other institutions.

##  6 Declarations

###  6.1 Conflict of interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

###  6.2 Ethical Approval

Not applicable.

###  6.3 Funding

No applicable.

###  6.4 Availability of data and materials

The datasets we use are all public open-source data, they can be obtained from
[30, 31]

## References

  * [1] Chen L C, Papandreou G, Kokkinos I, et al. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. _IEEE transactions on pattern analysis and machine intelligence_. 2018; 40(4): 834–848. https://arxiv.org/abs/1606.00915v2
  * [2] Zhao H, Shi J, Qi X, et al. Pyramid scene parsing network. in _Proceedings of the IEEE conference on computer vision and pattern recognition._ 2017, pp. 2881-2890. https://ieeexplore.ieee.org/document/8100143
  * [3] Long J, Shelhamer E, and Darrell T, Fully convolutional networks for semantic segmentation. in _Proceedings of the IEEE conference on computer vision and pattern recognition,_ 2015, pp. 3431–3440. https://arxiv.org/abs/1411.4038
  * [4] Lin G, Milan A, Shen C, et al. Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. in _Proceedings of the IEEE conference on computer vision and pattern recognition._ 2017, pp. 1925-1934. https://arxiv.org/abs/1611.06612v3
  * [5] Byoung Chul Ko, JiHyeon Lee, and Jae-Yeal Nam. Automatic medical image annotation and keyword-based image retrieval using relevance feedback. _Journal of digital imaging_ , 2012; 25(4): 454–465. https://link.springer.com/article/10.1007/s102
78-011-9443-5

  * [6] Samuel Budd, Emma C Robinson, and Bernhard Kainz. A survey on active learning and human-in-the-loop deep learning for medical image analysis. _Medical Image Analysis_ , 2021; 71: 102062. https://arxiv.org/abs/1910.02923
  * [7] Ouyang C, Biffi C, Chen C, et al, Self-supervision with superpixels: Training few-shot medical image segmentation without annotation. in _European Conference on Computer Vision._ Springer, 2020, pp. 762-780. https://arxiv.org/abs/2007.09886
  * [8] Hansen S, Gautam S, Jenssen R, et al, Anomaly detection-inspired few-shot medical image segmentation through self-supervision with supervoxels. _Medical Image Analysis,_ 2022; 78:102385. https://arxiv.org/abs/2203.02048v1
  * [9] Shen Q, Li Y, Jin J, et al. Q-Net: Query-Informed Few-Shot Medical Image Segmentation. _arXiv preprint arXiv:2208.11451_ , 2022. https://arxiv.org/abs/2208.11451
  * [10] Wang K, Liew J. H, Zou Y, et al, PANet: Few-shot image semantic segmentation with prototype alignment, in _IEEE International Conference on Computer Vision,_ 2019, pp. 9197–9206. https://arxiv.org/abs/1908.06391
  * [11] Protonotarios N E, Katsamenis I, Sykiotis S, et al. A few-shot U-Net deep learning model for lung cancer lesion segmentation via PET/CT imaging. _Biomedical Physics & Engineering Express_, 2022; 8(2): 025019. https://iopscience.iop.org/article/10.1088/2057-1976/ac53bd
  * [12] Li C, Zhang D, Tian Z. Q, et al. Few-shot learning with deformable convolution for multiscale lesion detection in mammography. _Medical physics_ , 2020; 47: 7. https://pubmed.ncbi.nlm.nih.gov/32160321/
  * [13] Huang Q, Huang Y, Luo Y, et al, Segmentation of breast ultrasound image with semantic classification of superpixels, _Medical image analysis,_ 2020; 61: 101657. https://pubmed.ncbi.nlm.nih.gov/32032899/
  * [14] Irving B, Franklin J. M, Papież B. W, et al, Pieces-of-parts for supervoxel segmentation with global context: Application to DCE-MRI tumour delineation, _Medical image analysis,_ 2016; 32: 69-83. https://arxiv.org/pdf/1604.05210v1.pdf
  * [15] Stutz D, Hermans A, and Leibe B, Superpixels: An evaluation of the state-of-the-art, _Computer Vision and Image Understanding,_ 2018; 166: 1–27. https://arxiv.org/abs/1612.01601
  * [16] Yang X. H, Wang B. R, Zhou X. C, et al, BriNet: Towards Bridging the Intra-class and Inter-class Gaps in One-Shot Segmentation, _31st British Machine Vision Conference,_ 2020\. https://doi.org/10.48550/arXiv.2008.06226
  * [17] Woo S, Park J, Lee J. Y, et al, CBAM: Convolutional block attention module, in _Proceedings of the European conference on computer vision ECCV,_ 2018, pp. 3-19. https://arxiv.org/abs/1807.06521
  * [18] Roy A. G, Siddiqui S, Pölsterl S, et al, Squeeze & excite guided few-shot segmentation of volumetric images, _Medical Image Analysis,_ 2020; 59: 101587. https://arxiv.org/abs/1902.01314
  * [19] Cai Y. T and Wang Y, MA-UNet: An improved version of UNet based on multi-scale and attention mechanism for medical image segmentation, in _Third International Conference on Electronics and Communication Network; and Computer Technology (ECNCT 2021),_ SPIE, 2022; 12167: 205-211. https://arxiv.org/abs/2012.10952
  * [20] Sinha A, and Dolz J, Multi-scale self-guided attention for medical image segmentation, _IEEE Journal of Biomedical and Health Informatics_ , 2021; 25(1): 121-130. https://arxiv.org/abs/1906.02849
  * [21] Chen L C, Zhu Y, Papandreou G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation, in _Proceedings of the European conference on computer vision (ECCV),_ 2018, pp. 801-818. https://arxiv.org/abs/1802.02611
  * [22] Ronneberger O, Fischer P, and Brox T, U-Net: Convolutional networks for biomedical image segmentation, in _International Conference on Medical Image Computing and Computer-Assisted Intervention,_ Springer, 2015, pp. 234–241. https://doi.org/10.48550/arXiv.1505.04597
  * [23] Çiçek Ö, Abdulkadir A, Lienkamp S S, et al. 3D U-Net: learning dense volumetric segmentation from sparse annotation, in _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2016: 19 th International Conference,_ Athens, Greece, October 17-21, 2016, Proceedings, Part II 19: pp. 424-432. Springer International Publishing. https://doi.org/10.48550/arXiv.1606.06650
  * [24] Milletari F, Navab N, and Ahmadi S. A, V-Net: Fully convolutional neural networks for volumetric medical image segmentation, in _2016 4 th International Conference on 3D Vision (3DV),_ IEEE, 2016, pp. 565–571. https://ieeexplore.ieee.org/abstract/document/
7785132

  * [25] Xiao X, Lian S, Luo Z, et al, Weighted RES-Unet for high-quality retina vessel segmentation, in _2018 9 th international conference on information technology in medicine and education (ITME)_, IEEE, 2018, pp. 327-331. https://ieeexplore.ieee.org/abstract/document/
8589312

  * [26] Li X, Chen H, Qi X, et al. H-DenseUNet: Hybrid densely connected UNet for liver and tumor segmentation from CT volumes, _IEEE Transactions on Medical Imaging_ , 2018; 37(12): 2663–2674. https://ieeexplore.ieee.org/document/8379359
  * [27] Ibtehaz N, and Rahman M. S, MultiResUNet: Rethinking the U-Net architecture for multimodal biomedical image segmentation. _Neural networks,_ 2020; 121: 74-87. https://arxiv.org/abs/1902.04049
  * [28] Park D and Lee J. Hierarchical Attention Network for Few-Shot Object Detection via Meta-Contrastive Learning. _arXiv preprint arXiv: 2208.07039,_ 2022\. https://arxiv.org/abs/2208.07039v1
  * [29] Guo M, Lu C, Liu Z, et al. Visual attention network. Computational Visual Media. 2023; 9(4): 733-752. https://arxiv.org/abs/2202.09741
  * [30] Kavur A. E, Gezer N. S, Barış M, et al, CHAOS challenge-combined (CT-MR) healthy abdominal organ segmentation, _Medical Image Analysis,_ 2021; 69: 101950. https://arxiv.org/abs/2001.06535
  * [31] Zhuang X, Multivariate mixture model for cardiac segmentation from multi-sequence MRI, in _International Conference on Medical Image Computing and Computer-Assisted Intervention,_ Springer, 2016, pp. 581–588. https://arxiv.org/pdf/1612.08820.pdf
  * [32] Ding H, Sun C, Tang H, et al. Few-shot Medical Image Segmentation with Cycle-resemblance Attention, in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision._ 2023, pp. 2488-2497. https://ieeexplore.ieee.org/document/10030099