# SACNet: A Spatially Adaptive Convolution Network for 2D Multi-organ Medical
Segmentation
††thanks: *Corresponding author.This work is supported by National Natural
Science Foundation of China [No. 62371156] and Natural Science Foundation of
Guangdong Province [No. 2022A1515011629]

Lin Zhang, Wenbo Gao, Jie Yi, Yunyun Yang∗ School of Science
Harbin Institute of Technology, Shenzhen
Shenzhen, China
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Multi-organ segmentation in medical image analysis is crucial for diagnosis
and treatment planning. However, many factors complicate the task, including
variability in different target categories and interference from complex
backgrounds. In this paper, we utilize the knowledge of Deformable Convolution
V3 (DCNv3) and multi-object segmentation to optimize our Spatially Adaptive
Convolution Network (SACNet) in three aspects: feature extraction, model
architecture, and loss constraint, simultaneously enhancing the perception of
different segmentation targets. Firstly, we propose the Adaptive Receptive
Field Module (ARFM), which combines DCNv3 with a series of customized block-
level and architecture-level designs similar to transformers. This module can
capture the unique features of different organs by adaptively adjusting the
receptive field according to various targets. Secondly, we utilize ARFM as
building blocks to construct the encoder-decoder of SACNet and partially share
parameters between the encoder and decoder, making the network wider rather
than deeper. This design achieves a shared lightweight decoder and a more
parameter-efficient and effective framework. Lastly, we propose a novel
continuity dynamic adjustment loss function, based on t-vMF dice loss and
cross-entropy loss, to better balance easy and complex classes in
segmentation. Experiments on 3D slice datasets from ACDC and Synapse
demonstrate that SACNet delivers superior segmentation performance in multi-
organ segmentation tasks compared to several existing methods.

###### Index Terms:

Multi-organ segmentation, Adaptive spatial aggregation, Category balancing

##  I Introduction

Multi-organ medical image segmentation, which simultaneously provides clearer
visualization of anatomical and pathological structures across multiple
organs, greatly enhances the efficiency and accuracy of diagnosis [1]-[2].

However, it remains challenging due to the following difficulties: (1)
Variability in different target categories. As shown in Fig. 1, not only do
different organs in the same slice dataset vary significantly in size, shape,
and texture, but different slices of the same organ also differ greatly. This
variability makes it difficult for segmentation models to generalize
effectively, as they may overfit to specific features seen during training and
struggle with unseen morphological variations. (2) Interference from complex
backgrounds. Fig. 1 shows the complexity of the surrounding anatomical
structures and the presence of varying textures, intensities, and overlapping
tissues, which introduce substantial difficulties. These intricate backgrounds
can lead to ambiguous boundaries and obscure the precise contours of the
target organs. Consequently, the model’s ability to precisely identify and
segment the organs is compromised, often resulting in incomplete or inaccurate
segmentation outcomes.

Figure 1:  Overview of challenges in multi-organ segmentation tasks and the
effect of different convolutional techniques. The above figure: challenges
faced in multi-organ segmentation tasks. The four images below: effects of
standard convolutional kernel, dilated convolutional kernel, large-kernel
convolutional kernel, and standard deformable convolutional kernel.

Recently, most existing studies [3]-[16] have utilized standard convolutional
kernels, dilated convolutional kernels, large-kernel convolutional kernels,
transformers, or hybrid methods as building blocks to design specific network
architectures to improve segmentation accuracy. As shown in Fig. 1, dilated
and large-kernel convolutional kernels have a larger receptive field compared
to standard convolutional kernels. The transformers [9] capture long-range
dependencies more effectively through self-attention mechanisms. However, due
to significant differences between different organs, these models lack an
inherent mechanism to adjust the specific receptive field according to
different segmentation targets, inevitably losing the perception of the
corresponding structures. In this context, some researchers [17]-[20] have
started to introduce deformable convolutions and deformable attention
transformers for multi-organ medical image segmentation. However, deformable
convolutions [17] have limited flexibility in sampling points and high
computational complexity. Deformable attention transformers [19] perform
poorly when handling imbalanced label data, and the need for large amounts of
data for generalization poses significant challenges for medical image tasks,
where data is relatively scarce. Some researchers have also attempted to
improve loss functions to better adapt to multi-organ medical image
segmentation tasks. The standard Dice loss, due to cosine similarity’s
disregard for data distribution characteristics [26], treats each class
equally, which tends to result in imbalanced segmentation. Various loss
functions [21]-[26] based on dice loss have been proposed to further reinforce
the stringent constraints on segmentation. Building on this foundation,
structured losses combined with continuous dynamic adjustment mechanisms will
potentially further improve the accuracy of segmentation.

To tackle the above obstacles, we propose a novel framework, SACNet, involving
an Adaptive Receptive Field Module (ARFM), widenet strategy, and dynamic
continuity adjustment loss function. (1) To address the variability in
different target categories and the interference from complex backgrounds, we
propose the ARFM, which combines DCNv3 with a series of tailored block-level
and architecture-level designs similar to transformers to enhance the
perception of the geometric structure by adaptively focusing on different
segmentation targets. Consider that DCNv3 allows the network to freely learn
geometrical changes, which can cause perceptual regions to wander; therefore,
our ARFM adopts the strategy of grouped convolutions. It divides the bias
generation process of DCNv3 into groups corresponding to the segmentation
target categories, with each group adaptively adjusting the receptive field
according to different targets. Additionally, ARFM introduces the Feed-Forward
block to capture specific patterns in the different segmentation targets. It
also introduces the Layer Scale block and the DropPath block to prevent the
model from excessively freely learning geometric changes. (2) To address the
challenge of high memory and computation complexity, we adopt the WideNet
strategy [31]. We share the DCNv3 block projection weights of ARFM modules
between the encoder and decoder when generating the original location-aware
modulation scalars and offsets. Through the FFN block, SACNet can train more
parameters in width; by sharing parameters between DCNv3 blocks, SACNet can
train fewer parameters in depth. In this method, the network goes wider
instead of deeper, achieving a shared lightweight decoder and a more
parameter-efficient and effective framework. (3) To address the problem of
imbalanced segmentation caused by imbalanced label data, we propose a novel
Continuity Dynamic Adjustment Loss Function (CTLoss). Based on t-vMF Dice loss
and cross-entropy loss, CTLoss adaptively adjusts using the Intersection Over
Union (IOU) of each segmentation category on the validation set after each
epoch. Our CTLoss automatically adjusts to use more compact similarities for
easy classes and wider similarities for difficult classes during training.

To summarize, our contributions are as follows:

  * •

We propose an Adaptive Receptive Field Module to enhance the perception of
geometric structures by adaptively focusing on different segmentation targets.

  * •

We design a novel convolutional network that expands the network’s width
rather than its depth, achieving a shared lightweight decoder and a more
parameter-efficient and effective framework.

  * •

We propose a novel continuity dynamic adjustment loss function based on IOU,
which better balances easy classes and difficult classes.

  * •

Our proposed SACNet method achieves state-of-the-art performance on two
different multi-organ medical image segmentation datasets, as demonstrated by
various ablation studies highlighting its effectiveness.

##  II RELATED WORK

###  II-A Methods based on Network Design

Various methods [3]-[16] have been proposed to achieve better performance by
designing specific network structures and modules for multi-organ
segmentation. (1) Methods based on the convolutional kernel and transformers:
Since the introduction of UNet [3], Convolutional Neural Networks (CNNs) have
become the most common choice. To enhance the limited receptive field,
researchers [5] have attempted to replace standard convolutions with dilated
convolutions and large-kernel convolutions. With the success of vision
transformers, TransUNET [9] and similar methods [10]-[12] integrate
transformers with convolutions as building blocks for the encoder. SwinUNet
[13] and similar methods [14]-[16] aim to create a fully transformer-based
model. (2) Methods based on deformable convolution and deformable
transformers: Recently, deformable convolutions have been applied to medical
image segmentation [17]-[18]. [27] extended standard deformable convolutions
to DCNv2, and DCNv3 [28] further enhanced feature extraction at different
scales by building on DCNv2. AgileFormer [20] introduced the deformable
attention transformer, integrating deformable convolutions into patch
embedding, self-attention, and positional encoding within the model.
Considering the limitations of convergence difficulty and perceptual region
wander caused by excessive randomness, our work integrates the knowledge of
deformable convolution v3 and designs the ARFM as a building block for the
encoder. The ARFM can adaptively adjust the receptive field according to
different segmentation targets and mitigate the excessive flexibility of
DCNv3, stably enhancing the perception of different segmentation regions in
the feature extraction process.

Figure 2: Overview of the SACNet architecture. The left side of the network
represents the encoder, which uses four stages of adaptive spatial aggregation
feature extraction, while the right side shows the shared lightweight decoder
that generates the segmentation maps. Each block of the ARFM employs pre-
trained DCNv3 that adaptively aggregates spatial features. The Feed-Forward
block captures specific patterns in the different segmentation targets and
introduces the Layer Scale block and the DropPath block to prevent the model
from excessively freely learning geometric changes.

###  II-B Methods based on Loss Function

In recent studies, various loss functions based on Dice loss have been
proposed to further improve the Dice Similarity Coefficient (DSC) [21]-[25].
Li et al. [21] proposed a generalized Dice loss that utilizes the class re-
balancing properties of the generalized Dice overlap, achieving a robust and
accurate deep learning loss function for unbalanced segmentation. Shit et al.
[22] introduced a similarity measurement termed centerline Dice, which is
calculated based on the intersection of the segmentation masks and the
skeleton. Wang et al. [23] improved Dice loss by mining the information of
negative areas. Kato and Hotta [25] applied the t-vMF similarity to the Dice
loss, adjusting the compactness of the similarity for each class. Drawing
inspiration from this, our work proposes a novel Continuity Dynamic Adjustment
Loss Function (CTLoss), which combines t-vMF dice loss and cross-entropy loss.
Our CTLoss introduces a new dynamic updating mechanism, updating with the
average IOU of each class on the validation set after each epoch, better
guiding the network to balance easy classes and difficult classes.

##  III METHOD

###  III-A Encoder

Recent advancements in large-scale models [27] based on Convolutional Neural
Networks (CNNs) have shown the potential of Deformable Convolution v3 (DCNv3),
which has demonstrated greater effectiveness in various downstream visual
tasks. In our SACNet model, we primarily utilize the ARFM module, which
employs the pre-trained DCNv3 block as its core operation. This module
integrates the DCNv3 block with a series of tailored block-level and
architecture-level designs similar to Transformers to serve as the building
blocks of the spatially adaptive pyramid feature encoder. Specifically, our
input is an image size of 3×224×224, starting with a Stem Layer, which applies
convolution followed by layer normalization and activation functions. Next,
the input passes through four stages with embedding sizes of 112, 224, 448,
and 896, respectively. Each stage consists of a different number of ARFM
modules and a down-sampling module, with each ARFM module composed of five
separate blocks: the DCNv3 block, which provides the model with the capability
to adjust specific adaptive receptive fields according to different
segmentation targets; the Layer Norm block, which performs normalization on
the input samples; the Feed-Forward block, which captures specific patterns in
the different segmentation targets; the Layer Scale block and DropPath block,
which prevent the model from excessively freely learning geometric changes.
The implementation of ARFM is shown in Equations (1)-(3) as follows:

$DCN(p_{0})=\sum_{g=1}^{G}\sum_{k=1}^{K}w_{g}m_{gk}x_{g}(p_{0}+p_{k}+\Delta
p_{gk}),$ (1)

$z_{l}^{\prime}=DP\left(\gamma_{1}\cdot
N\left(DCN\left(z_{l-1}\right)\right)+z_{l-1}\right),$ (2)

$z_{l}=DP\left(\gamma_{2}\cdot
N\left(FFN\left(z_{l}^{\prime}\right)\right)+z_{l}^{\prime}\right),$ (3)

where $K$ represents the total number of sampling points, and $k$ enumerates
each sampling point. $G$ denotes the total number of aggregation groups. For
the $g$-th group, $\mathbf{w}_{g}\in\mathbb{R}^{C\times C^{\prime}}$
represents the location-irrelevant projection weights, where
$C^{\prime}=\frac{C}{G}$ defines the group dimension. $m_{gk}\in\mathbb{R}$
denotes the modulation scalar of the $k$-th sampling point in the $g$-th
group, normalized by the softmax function along dimension $K$.
$\mathbf{x}_{g}\in\mathbb{R}^{C^{\prime}\times H\times W}$ represents the
sliced input feature map. $p_{k}$ denotes the $k$-th location of the pre-
defined grid sampling $\\{(-1,-1),(-1,0),\ldots,(0,+1),\ldots,(+1,+1)\\}$ as
in regular convolutions. $\Delta\mathbf{p}_{gk}$ is the offset corresponding
to the grid sampling location $p_{k}$ in the $g$-th group. $DCN$ stands for
deformable convolution v3. $\gamma_{1}$ and $\gamma_{2}$ are the inter-layer
scaling factors, $N$ denote layer normalization, and $DP$ denotes the droppath
function.

The third layer of the encoder typically captures better feature
representations than the others [28]. Through extensive experiments, we
ultimately distribute more computation to the third encoder stage with an ARFM
module ratio of [4, 4, 21, 4], respectively. A skip connection is utilized
inside each stage to address the issue of vanishing gradients and facilitate
information flow. Our encoder generates four feature representations of size
$\\{1\times(112\times 2^{i})\times\frac{H}{2^{2+i}}\times\frac{W}{2^{2+i}}\\}$
in four different stages where $i=\\{0,1,2,3\\}$, which are passed to the
decoder through skip connections. All modules are connected in the sequence
shown in Fig. 2.

###  III-B Decoder

The SACNet decoder has four consecutive decoder blocks and one bottleneck
block. Each decoder block starts with layer normalization, and then upsamples
the inputs by a factor of two to match the size of the encoder outputs. Next,
we apply the ARFM module to reshape the features, followed by layer
normalization while keeping the input dimension the same. We then concatenate
it with the encoder output through the skip connection. Similar to regular
convolution, when the DCNv3 block generates offsets $\Delta{p}_{gk}$ and
modulation scalars $m_{gk}$, different convolutional neurons in different
groups have independent linear projection weights. Therefore, as the total
number of sampling points increases, the model’s parameter and memory
complexity grow rapidly, significantly limiting the model’s efficiency. To
address this problem, we borrow the idea from WideNet [31], which shares some
model parameters in depth. We share the DCNv3 block projection weights of ARFM
modules between the encoder and decoder when generating the original location-
aware modulation scalars and offsets. Through the Feed-Forward blocks, SACNet
can train more parameters in width; by sharing parameters between DCNv3
blocks, SACNet can train fewer parameters in depth. Then, we apply bilinear
interpolation to resize the images to $H\times W\times N$, where $N$ is the
number of classes, used as the final segmented maps of the decoder, and pass
through Softmax to classify each pixel. The operation of the bottleneck layer
is the same as the other four decoder blocks, except that the bottleneck
gathers more information in the channel of the image while the image size
remains the same. Additionally, our decoder does not incorporate any extra
techniques or strategies such as residual connections [29] or deep supervision
[30].

Figure 3: Qualitative comparisons of the segmentation performance of our
approach alongside other segmentation methods. The first and second rows show
the results of the Synapse dataset, where the color codes are: yellow for the
aorta, green for the gallbladder, orange for the left kidney, cyan for the
right kidney, plum for the liver, purple for the pancreas, indigo for the
spleen, and crimson for the stomach. The third and fourth rows show the
results of the ACDC dataset, where the color codes are: red for the right
ventricle, green for the myocardium, and blue for the left ventricle. From
left to right, we show the results from TransUNet, Swin-UNet, AgileFormer, our
SACNet with $L_{CD}$ (a combination of cross-entropy loss and Dice loss), our
SACNet with $L_{CT}$ (our proposed dynamic loss), and the ground truth.

###  III-C Model Architectures

The overall architecture of our model is illustrated in Fig. 2. Like most
medical image segmentation modeling methods [8]–[10], an encoder-decoder
design is employed. Our framework contains four stages of adaptive spatial
aggregation feature extraction encoders and a shared lightweight decoder. For
an input RGB medical image with the shape of $H\times W\times 3$, we first use
stem layers to reduce the spatial resolution. Subsequently, four feature maps
($X_{1}$, $X_{2}$, $X_{3}$, and $X_{4}$) in four different stages are
generated and concatenated with the up-sampled features of the decoder through
skip connections. The extracted features from the last stage ($X_{4}$) are
also passed through a bottleneck layer, where the channel size is adjusted to
gather more information while keeping the image size the same. The features
from the bottleneck layer are propagated upward, up-sampled by a scale of two,
and concatenated with the features from the encoder. We apply bilinear
interpolation to generate the final segmentation maps, which pass through a
softmax layer to classify each pixel.

###  III-D Training Loss

Recent 2D medical image segmentation methods combine the Dice loss with the
cross-entropy loss as the final loss function. However, for multi-object
segmentation tasks, the standard Dice loss, due to cosine similarity’s
disregard for data distribution characteristics [25], treats each class
equally, which tends to result in imbalanced segmentation. In this study, we
tested most of the proposed loss functions for multi-class medical image
segmentation and comprehensively considered the loss aggregation strategy. We
propose a novel Continuity Dynamic Adjustment Loss Function (CTLoss) based on
t-vMF Dice loss [25] and the cross-entropy loss function. This new loss
function adaptively uses more compact similarities for easy classes and wider
similarities for difficult classes, making it more effective in helping the
model converge better and more efficiently for multi-object image
segmentation.

The t-vMF Dice Loss is formulated as:

$\cos\theta_{i}=\sum_{i=1}^{n}A_{in}B_{in},$ (4)
$\phi_{\kappa}(\cos\theta;\kappa)=\frac{1+\cos\theta}{1+\kappa(1-\cos\theta)}-1,$
(5)
$L_{t-vMF}=\frac{1}{C}\sum_{i=1}^{C}\left(1-\phi_{\kappa}(\cos\theta_{i};\kappa)\right)^{2},$
(6)

where $n$ indicates the number of class samples, $i$ represents the category
index, $A_{in}$ indicates the vectors containing all positive examples
predicted by the model, and $B_{in}$ indicates the vectors containing all
positive examples of the ground truth in the dataset. $\kappa$ is a
concentration parameter that adjusts the shape of the similarity function, and
$C$ is the number of classes. We propose an effective algorithm that uses the
Intersection Over Union (IOU) of the model on the validation set to update
$\kappa$ in the t-vMF Dice loss after each epoch. This approach adaptively
determines $\kappa$ to achieve more compact similarities for easy classes and
wider similarities for difficult classes.

TABLE I: Comparative analysis of model performance with 2D methods in multi-organ segmentation using the synapse dataset. The best result within each column is highlighted in bold, and the second-best is underlined. Models marked with an asterisk (∗) were implemented by us. Model | Year | Aorta | Gallbladder | Kidney(L) | Kidney(R) | Liver | Pancreas | Spleen | Stomach | Average
---|---|---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |  |  | DSC$\uparrow$ | HD95$\downarrow$
U-net [3] | 2015 | 89.07 | 69.72 | 77.77 | 68.6 | 93.43 | 53.98 | 86.67 | 75.58 | 76.85% | 39.70
U-Net++ [4] | 2018 | 88.19 | 68.89 | 81.76 | 75.27 | 93.01 | 58.20 | 83.44 | 70.52 | 76.91% | 36.93
MultiResUNet [5] | 2020 | 87.33 | 65.67 | 82.08 | 73.82 | 93.68 | 52.85 | 85.23 | 75.66 | 77.42% | 36.84
TransUNet [9] | 2021 | 87.23 | 63.13 | 81.87 | 77.02 | 94.08 | 55.86 | 85.08 | 75.62 | 77.48% | 31.69
TransNorm [10] | 2022 | 86.16 | 65.11 | 82.18 | 78.64 | 93.42 | 55.34 | 89.50 | 76.01 | 78.40% | 30.25
Swin-UNet [13] | 2022 | 85.47 | 66.53 | 83.28 | 79.61 | 94.29 | 56.58 | 90.66 | 76.60 | 79.13% | 21.55
DA-TransUNet∗ [11] | 2023 | 87.13 | 58.75 | 79.39 | 78.41 | 94.82 | 62.62 | 87.84 | 79.72 | 78.59% | 29.20
DAE-Former [16] | 2023 | 88.96 | 72.30 | 86.08 | 80.88 | 94.98 | 65.12 | 91.94 | 79.19 | 82.43% | 17.46
ParaTransCNN [12] | 2024 | 88.12 | 68.97 | 87.99 | 83.84 | 95.01 | 69.79 | 92.71 | 84.43 |  83.86% | 15.86
AgileFormer∗ [20] | 2024 | 88.52 | 70.37 | 85.69 | 81.14 | 95.65 | 70.78 | 91.19 | 86.17 | 83.69% | 19.37
SACNet+$L_{CD}$(Ours) | 2024 | 88.46 | 71.05 | 83.68 | 80.99 | 95.74 | 71.40 | 91.27 | 86.56 | 83.64% | 22.01
SACNet+$L_{CT}$(Ours) | 2024 | 88.88 | 70.45 | 89.36 | 85.41 | 95.98 | 70.38 | 91.26 | 87.64 | 84.92% | 15.13

The multi-class cross-entropy loss function is defined as follows:

$L_{CE}=-\sum_{i=1}^{N}\sum_{c=1}^{C}y_{ic}\log(\hat{y}_{ic}),$ (7)

where $N$ represents the total number of samples, and $C$ is the total number
of classes. $y_{ic}$ denotes the true label for sample $i$ and class $c$ (if
the sample belongs to class $c$, then $y_{ic}=1$, otherwise $y_{ic}=0$).
$\hat{y}_{ic}$ is the predicted probability that sample $i$ belongs to class
$c$.

The final loss function combines the adaptive t-vMF Dice Loss and the cross-
entropy loss:

$L_{CT}=\gamma L_{t-vMF}+(1-\gamma)L_{CE},$ (8)

where $\gamma$ is the weight coefficient, with $\gamma=0.6$ and $1-\gamma=0.4$
representing the weights for the adaptive t-vMF Dice loss and the cross-
entropy loss, respectively.

##  IV Experiment

###  IV-A Dataset

Consistent with previous work, we validated the proposed method on two public
medical image segmentation datasets: the Synapse multi-organ segmentation
dataset and the Automated Cardiac Diagnosis Challenge (ACDC) dataset.

The Synapse dataset consists of 30 scans of eight abdominal organs from the
MICCAI 2015 Multi-Atlas Abdominal Labeling Challenge. These eight organs
include the left kidney, right kidney, aorta, spleen, gallbladder, liver,
stomach, and pancreas. There are a total of 3779 axially enhanced abdominal
clinical CT images. Following previous studies, we report the mean Dice
Similarity Coefficient (DSC) and the 95% Hausdorff Distance (HD95) for the
segmentation of abdominal organs using 18 training cases (2212 axial slices)
and 12 validation cases.

The ACDC dataset consists of cine MR images acquired during breath-hold,
covering the heart from the base to the apex of the left ventricle. Each scan
was annotated with ground truth for the Left Ventricle (LV), Right Ventricle
(RV), and Myocardium (Myo). Following previous studies, we report the mean DSC
for 70 training cases (1930 axial slices), 10 validation cases, and 20 test
cases.

###  IV-B Evaluation

We evaluate our model’s performance on the ACDC and Synapse multi-organ
datasets using the DSC and the HD95 for the Synapse dataset. The DSC and HD95
are calculated according to the formulas as follows:

$DSC(X,Y)=\frac{2\cdot|X\cap Y|}{|X|+|Y|},$ (9)
$HD95(A,B)=\max\left\\{\max_{a\in A}\min_{b\in B}\|a-b\|,\max_{b\in
B}\min_{a\in A}\|b-a\|\right\\},\small$ (10)

where $X$ represents the predicted segmentation mask, and $Y$ represents the
ground truth segmentation mask. $a$ and $b$ are points in sets $A$ and $B$
respectively. $A$ represents the set of points in the predicted segmentation
mask, and $B$ represents the set of points in the ground truth segmentation
mask.

###  IV-C Setups

The model and experiments were implemented using PyTorch 2.2.0 and the CUDA
toolkit 12.1, with all experiments conducted on an NVIDIA 3090 24G GPU. To
improve computational efficiency, the resolution of all training, validation,
and test images was resized to 224×224. We used a batch size of 18, an initial
learning rate of $3\times 10^{-4}$, and trained the models for 400 epochs. All
models were trained using a combination of t-vMF Dice Loss and cross-entropy
losses, incorporating our proposed continuity dynamic adaptive algorithm. The
AdamW optimizer was used, employing cosine learning rate decay. For better
model initialization, we initialized the model parameters with weights pre-
trained on ImageNet-1k. The DSC and the HD95 were used as metrics to evaluate
segmentation accuracy. During training, the best validation weights were
recorded to test the performance of our model.

TABLE II: Performance comparison with 2D methods in cardiac MRI segmentation
using the ACDC dataset. The best result within each column is highlighted in
bold, and the second-best is underlined. Models marked with an asterisk (∗)
were implemented by us.

Model | Year | RV | Myo | LV | DSC$\uparrow$
---|---|---|---|---|---
R50+UNet [9] | 2021 | 87.10 | 80.63 | 94.92 | 87.55%
TransUNet [9] | 2021 | 86.67 | 87.27 | 95.18 | 89.71%
MISSFormer [14] | 2021 | 89.55 | 88.04 | 94.99 | 90.86%
Swin-UNet [13] | 2022 | 85.77 | 84.42 | 94.03 | 88.07%
MT-UNet [32] | 2022 | 86.64 | 89.04 | 95.62 | 90.43%
TransCASCADE [33] | 2023 | 89.14 | 90.25 | 95.50 | 91.63%
Parallel MERIT [34] | 2024 | 90.87 | 90.00 | 96.08 | 92.32%
AgileFormer∗ [20] | 2024 | 90.93 | 90.29 | 95.99 | 92.41%
SACNet+$L_{CD}$(Ours) | 2024 | 91.05 | 89.96 | 96.01 | 92.34%
SACNet+$L_{CT}$(Ours) | 2024 | 91.38 | 90.11 | 96.10 | 92.53%

TABLE III: Comparison of Ablation Studies: Only a single parameter is altered while keeping the others constant to observe the impact of each parameter on performance. D represents the DCNv3 block, F represents the FFN block, and S represents the Layer Scale and DropPath block. D | F | S | CFLoss | Aorta | Gallbladder | Kidney(L) | Kidney(R) | Liver | Pancreas | Spleen | Stomach | DSC$\uparrow$ | HD95$\downarrow$
---|---|---|---|---|---|---|---|---|---|---|---|---|---
| ✓ | ✓ | ✓ | 84.33 | 60.83 | 85.73 | 77.54 | 93.54 | 50.12 | 88.20 | 75.62 | 76.99% | 31.05
✓ | ✓ |  | ✓ | 85.81 | 66.21 | 85.58 | 81.09 | 94.98 | 58.81 | 92.16 | 77.12 | 80.22% | 16.90
✓ |  | ✓ | ✓ | 87.23 | 64.81 | 86.97 | 83.35 | 94.74 | 62.50 | 93.25 | 81.58 | 81.81% | 15.84
✓ | ✓ | ✓ |  | 88.46 | 71.05 | 83.68 | 80.99 | 95.74 | 71.40 | 91.27 | 86.56 | 83.64% | 22.01
✓ | ✓ | ✓ | ✓ | 88.88 | 70.45 | 89.36 | 85.41 | 95.98 | 70.38 | 91.26 | 87.64 | 84.92% | 15.13

##  V Result

###  V-A Performance on Synapse Dataset

On the Synapse dataset, as shown in Table I, our SACNet stands out with the
highest scores for most organs, achieving 89.36% for the left kidney, 85.41%
for the right kidney, 95.98% for the liver, and 87.64% for the stomach. For
other organs, except the spleen, our SACNet also achieves the second-best
scores. Additionally, our DSC reaches 84.92% and HD95 is 15.13, both being the
best results compared to other benchmarks. Figure 3 provides visual examples
of the segmentation outputs produced by our proposed model and other state-of-
the-art methods. As shown, our predictions have clearer boundaries between
different segmentation targets: the boundaries between the pancreas and spleen
are clear and smooth. Our model is more accurate for small objects: Swin-UNet
and AgileFormer miss the gallbladder in their segmentations, and TransUNet
only partially segments it, but our model segments it completely. As shown in
the last two rows of Table I, replacing CDLoss with CTLoss significantly
improves the results for DSC and HD95, as well as for most organs,
particularly the left and right kidneys. In most slice data, the left and
right kidneys have complex boundaries and overlap with surrounding organs,
making them relatively difficult to segment, as shown in the orange and cyan
regions in Figure 3. This demonstrates that our CTLoss is highly effective in
balancing easy and difficult classes.

###  V-B Performance on ACDC Dataset

On the ACDC dataset, our SACNet outperforms other models in terms of
segmentation accuracy. As shown in Table II, our proposed SACNet achieves the
best segmentation results compared to other benchmarks, with a Dice score of
92.53%, representing a significant improvement of 4.46% over Swin-UNet and a
2.82% increase over TransUNet. Furthermore, in Figure 3, we provide visual
illustrations of the segmentation outputs produced by our proposed model and
other state-of-the-art approaches. As demonstrated, our predictions have
clearer boundaries between different segmentation targets and align closely
with the provided ground truth masks, indicating the effectiveness of our
approach. One notable advantage of SACNet is its capability to model adaptive
receptive fields according to different segmentation targets. Additionally,
the inclusion of our CTLoss results in a significant improvement in
segmentation accuracy for difficult classes compared to easy classes, such as
the right ventricle. In most slice data, the RV is smaller in volume and has a
higher overlap with other organs, making it a difficult class, as shown in the
red portion of Figure 3. This result demonstrates the effectiveness of our
loss function in balancing categories.

###  V-C Ablation Study

Taking the Synapse dataset as an example, we ablate the key components in our
SACNet to verify the effectiveness of these designs. As shown in Table III, we
evaluate the effectiveness of our pre-trained DCNv3 blocks, FFN blocks, Layer
Scale, and DropPath blocks respectively. When these key modules are removed,
the model’s performance decreases to varying degrees. Without DCNv3, the
model’s DSC drops from 84.92% to 76.99%, and HD95 increases from 15.13 to
31.05, with performance declining the most severely. This demonstrates that
DCNv3 plays a vital role in our model. We also compare our model using two
different loss functions: one combining cross-entropy and Dice loss, and the
other using our proposed continuity dynamic adjustment loss function. The
results show that using CTLoss for updating the model achieves better
performance, and CTLoss is highly effective in balancing easy classes and
difficult classes. Based on our comparisons, it is evident that the ARFM we
designed, which integrates pre-trained DCNv3 blocks, FFN blocks, Layer Scale
blocks, and DropPath blocks, can significantly enhance accuracy and establish
SACNet as a new state-of-the-art model for the 2D multi-organ medical image
segmentation task.

##  VI Conclusion

In this study, we focus on the 2D multi-organ medical segmentation task and
utilize the knowledge of deformable convolution v3 and multi-object
segmentation to optimize our SACNet in three aspects: feature extraction,
model architecture, and loss constraint, simultaneously enhancing perception
for different segmentation targets. Firstly, we propose the Adaptive Receptive
Field Module (ARFM), which combines (DCNv3) with a series of customized block-
level and architecture-level designs similar to transformers. This module can
capture the unique features of different organs by adaptively adjusting the
receptive field according to various targets. Secondly, we utilize ARFM as
building blocks to construct the encoder-decoder of SACNet and partially share
parameters between the encoder and decoder, making the network wider rather
than deeper, thereby achieving a more parameter-efficient and effective
framework. Lastly, we propose a novel continuity dynamic adjustment loss
function, based on t-vMF Dice loss and cross-entropy loss, to better balance
easy and complex classes in segmentation. Our method is verified on 3D slice
datasets from ACDC and Synapse, and the results show that our method provides
better accuracy in multi-organ segmentation tasks compared to several existing
methods.

However, there are limitations to our work compared to 3D segmentation
methods. 3D segmentation can utilize the complete information from volumetric
data, providing higher accuracy, but its computational and memory complexity
is also significantly higher. In the future, we aspire to extend our method to
3D segmentation tasks to further enhance performance.

## References

  * [1] R. Wang, T. Lei, R. Cui, et al., ”Medical image segmentation using deep learning: A survey,” IET Image Processing, vol. 16, no. 5, pp. 1243-1267, 2022.
  * [2] X. Liu, L. Song, S. Liu, et al., ”A review of deep-learning-based medical image segmentation methods,” Sustainability, vol. 13, no. 3, p. 1224, 2021.
  * [3] O. Ronneberger, P. Fischer, and T. Brox, ”U-net: Convolutional networks for biomedical image segmentation,” in Proc. Medical Image Computing and Computer-Assisted Intervention (MICCAI), Munich, Germany, 2015, pp. 234-241.
  * [4] Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, et al., ”Unet++: A nested u-net architecture for medical image segmentation,” in Proc. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support (DLMIA), Granada, Spain, 2018, pp. 3-11.
  * [5] N. Ibtehaz and M. S. Rahman, ”MultiResUNet: Rethinking the U-Net architecture for multimodal biomedical image segmentation,” Neural Networks, vol. 121, pp. 74-87, 2020.
  * [6] Y. Chen, T. Zhou, Y. Chen, et al., ”HADCNet: Automatic segmentation of COVID-19 infection based on a hybrid attention dense connected network with dilated convolution,” Computers in Biology and Medicine, vol. 149, p. 105981, 2022.
  * [7] X. Shu, Y. Gu, X. Zhang, et al., ”FCRB U-Net: A novel fully connected residual block U-Net for fetal cerebellum ultrasound image segmentation,” Computers in Biology and Medicine, vol. 148, p. 105693, 2022.
  * [8] F. Isensee, P. F. Jaeger, S. A. A. Kohl, et al., ”nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation,” Nature Methods, vol. 18, no. 2, pp. 203-211, 2021.
  * [9] J. Chen, Y. Lu, Q. Yu, et al., ”Transunet: Transformers make strong encoders for medical image segmentation,” arXiv preprint arXiv:2102.04306, 2021.
  * [10] R. Azad, M. T. Al-Antary, M. Heidari, et al., ”Transnorm: Transformer provides a strong spatial normalization mechanism for a deep segmentation model,” IEEE Access, vol. 10, pp. 108205-108215, 2022.
  * [11] G. Sun, Y. Pan, W. Kong, et al., ”DA-TransUNet: Integrating Spatial and Channel Dual Attention with Transformer U-Net for Medical Image Segmentation,” arXiv preprint arXiv:2310.12570, 2023.
  * [12] H. Sun, J. Xu, and Y. Duan, ”ParaTransCNN: Parallelized TransCNN Encoder for Medical Image Segmentation,” arXiv preprint arXiv:2401.15307, 2024.
  * [13] H. Cao, Y. Wang, J. Chen, et al., ”Swin-unet: Unet-like pure transformer for medical image segmentation,” in Proc. European Conference on Computer Vision (ECCV), Cham, Switzerland: Springer Nature Switzerland, 2022, pp. 205-218.
  * [14] X. Huang, Z. Deng, D. Li, et al., ”MISSFormer: an effective transformer for 2D medical image segmentation,” IEEE Transactions on Medical Imaging, 2022.
  * [15] R. Azad, M. Heidari, M. Shariatnia, et al., ”Transdeeplab: Convolution-free transformer-based deeplab v3+ for medical image segmentation,” in Proc. International Workshop on PRedictive Intelligence In MEdicine (PRIME), Cham, Switzerland: Springer Nature Switzerland, 2022, pp. 91-102.
  * [16] R. Azad, R. Arimond, E. K. Aghdam, et al., ”Dae-former: Dual attention-guided efficient transformer for medical image segmentation,” in Proc. International Workshop on PRedictive Intelligence In MEdicine (PRIME), Cham, Switzerland: Springer Nature Switzerland, 2023, pp. 83-95.
  * [17] J. Dai, H. Qi, Y. Xiong, et al., ”Deformable convolutional networks,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017, pp. 764-773.
  * [18] H. H. Lee, Q. Liu, Q. Yang, et al., ”DeformUX-Net: Exploring a 3D Foundation Backbone for Medical Image Segmentation with Depthwise Deformable Convolution,” arXiv preprint arXiv:2310.00199, 2023.
  * [19] Z. Xia, X. Pan, S. Song, et al., ”Vision transformer with deformable attention,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 4794-4803.
  * [20] P. Qiu, J. Yang, S. Kumar, et al., ”AgileFormer: Spatially Agile Transformer UNet for Medical Image Segmentation,” arXiv preprint arXiv:2404.00122, 2024.
  * [21] C. H. Sudre, W. Li, T. Vercauteren, et al., ”Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations,” in Proc. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support (DLMIA), Québec City, QC, Canada, 2017, pp. 240-248.
  * [22] S. Shit, J. C. Paetzold, A. Sekuboyina, et al., ”clDice-a novel topology-preserving loss function for tubular structure segmentation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 16560-16569.
  * [23] L. Wang, C. Wang, Z. Sun, et al., ”An improved dice loss for pneumothorax segmentation by mining the information of negative areas,” IEEE Access, vol. 8, pp. 167939-167949, 2020.
  * [24] P. Wang and A. C. S. Chung, ”Focal dice loss and image dilation for brain tumor segmentation,” in Proc. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support (DLMIA), Granada, Spain, 2018, pp. 119-127.
  * [25] S. Kato and K. Hotta, ”Adaptive t-vmf dice loss for multi-class medical image segmentation,” arXiv preprint arXiv:2207.07842, 2022.
  * [26] X. Zhu, H. Hu, S. Lin, et al., ”Deformable convnets v2: More deformable, better results,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 9308-9316.
  * [27] W. Wang, J. Dai, Z. Chen, et al., ”Internimage: Exploring large-scale vision foundation models with deformable convolutions,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 14408-14419.
  * [28] H. Wang, P. Cao, J. Wang, et al., ”Uctransnet: rethinking the skip connections in u-net from a channel-wise perspective with transformer,” in Proc. AAAI Conference on Artificial Intelligence (AAAI), 2022, vol. 36, no. 3, pp. 2441-2449.
  * [29] A. Hatamizadeh, Y. Tang, V. Nath, et al., ”Unetr: Transformers for 3d medical image segmentation,” in Proc. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2022, pp. 574-584.
  * [30] B. Dong, W. Wang, D. P. Fan, et al., ”Polyp-pvt: Polyp segmentation with pyramid vision transformers,” arXiv preprint arXiv:2108.06932, 2021.
  * [31] F. Xue, Z. Shi, F. Wei, et al., ”Go wider instead of deeper,” in Proc. AAAI Conference on Artificial Intelligence (AAAI), 2022, vol. 36, no. 8, pp. 8779-8787.
  * [32] H. Wang, S. Xie, L. Lin, et al., ”Mixed Transformer U-Net for Medical Image Segmentation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 2390-2394.
  * [33] M. M. Rahman and R. Marculescu, ”Medical image segmentation via cascaded attention decoding,” in Proc. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023, pp. 6222-6231.
  * [34] M. M. Rahman and R. Marculescu, ”Multi-scale hierarchical vision transformer with cascaded attention decoding for medical image segmentation,” in Proc. Medical Imaging with Deep Learning (MIDL), 2024, pp. 1526-1544.