TEC-Net: Vision Transformer Embrace Convolutional Neural Networks for
  Medical Image Segmentation
The hybrid architecture of convolution neural networks (CNN) and Transformer
has been the most popular method for medical image segmentation. However, the
existing networks based on the hybrid architecture suffer from two problems.
First, although the CNN branch can capture image local features by using
convolution operation, the vanilla convolution is unable to achieve adaptive
extraction of image features. Second, although the Transformer branch can model
the global information of images, the conventional self-attention only focuses
on the spatial self-attention of images and ignores the channel and
cross-dimensional self-attention leading to low segmentation accuracy for
medical images with complex backgrounds. To solve these problems, we propose
vision Transformer embrace convolutional neural networks for medical image
segmentation (TEC-Net). Our network has two advantages. First, dynamic
deformable convolution (DDConv) is designed in the CNN branch, which not only
overcomes the difficulty of adaptive feature extraction using fixed-size
convolution kernels, but also solves the defect that different inputs share the
same convolution kernel parameters, effectively improving the feature
expression ability of CNN branch. Second, in the Transformer branch, a
(shifted)-window adaptive complementary attention module ((S)W-ACAM) and
compact convolutional projection are designed to enable the network to fully
learn the cross-dimensional long-range dependency of medical images with few
parameters and calculations. Experimental results show that the proposed
TEC-Net provides better medical image segmentation results than SOTA methods
including CNN and Transformer networks. In addition, our TEC-Net requires fewer
parameters and computational costs and does not rely on pre-training. The code
is publicly available at https://github.com/SR0920/TEC-Net.
Rui Sun, Tao Lei, Weichuan Zhang, Yong Wan, Yong Xia, Asoke K. Nandi
# TEC-Net: Vision Transformer Embrace Convolutional Neural Networks for Medical Image Segmentation

###### Abstract

The hybrid architecture of convolution neural networks (CNN) and Transformer has been the most popular method for medical image segmentation. However, the existing networks based on the hybrid architecture suffer from two problems. First, although the CNN branch can capture image local features by using convolution operation, the vanilla convolution is unable to achieve adaptive extraction of image features. Second, although the Transformer branch can model the global information of images, the conventional self-attention only focuses on the spatial self-attention of images and ignores the channel and cross-dimensional self-attention leading to low segmentation accuracy for medical images with complex backgrounds. To solve these problems, we propose vision Transformer embrace convolutional neural networks for medical image segmentation (TEC-Net). Our network has two advantages. First, dynamic deformable convolution (DDConv) is designed in the CNN branch, which not only overcomes the difficulty of adaptive feature extraction using fixed-size convolution kernels, but also solves the defect that different inputs share the same convolution kernel parameters, effectively improving the feature expression ability of CNN branch. Second, in the Transformer branch, a (shifted)-window adaptive complementary attention module ((SW)-ACAM) and compact convolutional projection are designed to enable the network to fully learn the cross-dimensional long-range dependency of medical images with few parameters and calculations. Experimental results show that the proposed TEC-Net provides better medical image segmentation results than SOTA methods including CNN and Transformer networks. In addition, our TEC-Net requires fewer parameters and computational costs and does not rely on pre-training. The code is publicly available at [https://github.com/SR0920/TEC-Net](https://github.com/SR0920/TEC-Net).

 Deep learning, Medical image segmentation, Dynamic deformable convolution, (Shifted)-window adaptive complementary attention.

## I Introduction

Medical image segmentation refers to dividing a medical image into several specific regions with unique properties. Medical image segmentation results can not only achieve abnormal detection of human body regions but also be used to guide clinicians. Therefore, accurate medical image segmentation has become a key component of computer-aided diagnosis and treatment, patient condition analysis, image-guided surgery, tissue and organ reconstruction, and treatment planning. Compared with common RGB images, medical images usually suffer from the problems such as high-density noise, low contrast and blurred edges. So how to quickly and accurately segment specific human organs and lesions from medical images has always been a huge challenge in the field of smart medicine.

The early traditional medical image segmentation algorithms are based on manual features designed by medical experts using professional knowledge [1]. These methods have a strong mathematical basis and theoretical support, but these algorithms have poor generalization abilities for different organs or lesions of human body. Later, inspired by the full convolutional networks (FCN) [2] and the idea of encoder-decoder, Ronneberger et al. designed U-Net [3] network that was first applied to medical image segmentation. After that the U-shaped encoder-decoder structure receives widespread attention. At the same time, due to the small number of parameters and the good segmentation effect of U-Net, deep learning has made a breakthrough in medical image segmentation. Then a series of improved medical image segmentation networks are reported, such as 2D U-Net++ [4], ResDO-UNet [5], SGU-Net [6], 2.5D RIU-Net [7], 3D Unet [8] and V-Net [9]. The rapid development of CNN in the field of medical image segmentation is largely due to the scale invariance and inductive bias of convolution operation. Although this
fixed receptive field improves the computational efficiency of CNN, it limits the ability of CNN to capture the long-range dependency relationship.

Aiming at the shortcomings of CNN in obtaining global features of images, Vaswani et al. [10] proposed the vision Transformer architecture for image classification. The Transformer achieves a global representation of image information through complex spatial transformations and long-range dependency modeling, effectively solving the problem of CNN being only able to obtain local features of images. Currently, many methods based on Transformer have been applied to medical image segmentation, representative methods such as Swin-Unet [11], BAT [12], Swin UNETR [13], and UC-TransNet [14]. These methods can be roughly divided into the pure Transformer architecture and the hybrid architecture of CNN and Transformer. The pure Transformer architecture realizes the long-range dependency modeling using self-attention. However, due to the lack of inductive bias of the Transformer itself, the traditional Transformer cannot be widely used on small-scale datasets like medical images [15]. At the same time, the Transformer architecture is prone to ignore local detaile features, which reduces the separability between the background and the foreground of small lesions or objects with large-scale shape changes in medical images. The hybrid architecture of CNN and Transformer realizes both local and global information modeling of medical images by taking the complementary advantages of CNN and Transformer, thus achieving a better medical image segmentation effect. However, these hybrid architectures still suffer from the following two problems. First, these networks ignore the problems of organ deformation and lesion irregularities when modeling local features, resulting in weak local feature expression. Second, these networks ignore the correlation between spacial and channels when modeling the global feature, resulting in inadequate expression of self-attention. To address the above problems, our main contributions are as follows:

* _A novel dynamic deformable convolution (DDConv) is proposed. Through task adaptive learning, the DDConv can flexibly change the weight coefficient and deformation offset of convolution itself. The DDConv can overcome the problems of fixation of the narrow reception field and non-adaptive convolution kernel parameters existing in the vanilla convolution and its variants, such as Atrous convolution and Involution, etc. The DDConv can improve the ability to perceive tiny lesions and targets with large-scale shape changes in medical images._
* _A new (shifted)-window adaptive complementary attention module ((S)W-ACAM) is proposed. The (S)W-ACAM realizes the cross-dimensional global modeling of medical images through four parallel branches of weight coefficient adaptive learning. Compared with popular attention mechanisms, such as CBAM and Non-Local, the (S)W-ACAM fully makes up for the deficiency of the conventional attention mechanism in modeling the cross-dimensional relationship between spatial and channels, and thus enhances the separability between the segmented object and the background in medical images._
* _A new parallel network structure based on dynamically adaptive CNN and cross-dimensional feature fusion Transformer are proposed for medical image segmentation, called TEC-Net. Compared with popular hybrid architectures of CNN and Transformer, like Swin-Unet [11] and Swin UNETR [13]. TEC-Net enhances representation learning by tightly combining local and global features at different resolutions through parallel interaction between CNN and Transformer. However, TEC-Net not only abandons pre-training but also requires fewer parameters and fewer computational costs, which are 11.58 M and 4.53 GFLOPs respectively._

## 2 Related Work

Medical image segmentation plays a very important role in the field of medical image processing, and is also one of the core techniques of computer-aided diagnosis and treatment systems. Because it's tedious and complex to label manually medical images, and also it's difficult to guarantee the efficiency and accuracy of manual labeling, the rapid and accurate segmentation of medical images is of great significance for clinical treatment. In recent years, with the rapid development of deep learning techniques, researchers have continuously developed many deep network models for medical image segmentation. These medical image segmentation networks can be coarsely divided into two categories: CNN and Transformer networks.

### CNN-based Methods

Different from traditional medical image segmentation algorithms, the algorithms based on deep learning can learn the high-dimensional feature information of medical images through a multi-layer network structure. Among various deep-learning networks related to medical image segmentation, CNN perform extremely well. CNN can effectively learn from large-scale medical datasets to distinguish features and extract the prior knowledge, making them an important part of smart medical image analysis systems.

In 2015, Ronneberger et al. were inspired by the FCN [2] network and designed the first end-to-end network U-Net [3] for medical image segmentation in the ISBI cell tracking challenge. U-Net adopts a symmetric encoder and decoder structure, which can make full use of the local details of medical images and reduce the dependence on training datasets. Therefore, on the case of small datasets, U-Net can still achieve good medical image segmentation results. Based on U-Net, Alom et al. designed R2U-Net [16] by combining U-Net, ResNet [17], and recurrent neural network (RCNN) [18], which has achieved good performance on multiple medical image segmentation datasets such as blood vessels and retinas. To further improve the performance of U-Net, Gu et al. introduced dynamic convolution [19] into U-Net and proposed CA-Net [20]. Experiments on medical datasets show that CA-Net can not only improve the segmentation accuracy of medical images but also reduce the training time of the network. Inspired by the idea of residual connection and deformable convolution [21], Yang et al. added a residual deformable
convolution to U-Net, and proposed DCU-Net [22]. DCU-Net shows a more advanced segmentation effect than U-Net on DRIVE medical dataset. Lei et al. designed SGU-Net [6] based on U-Net, and proposed an ultralight convolution module and additional adversarial shape-constraint that can significantly improve the segmentation accuracy of abdominal medical images through self-supervised training. Although CNN have made great progress in network structure, the main reason for their success is due to the invariance in dealing with different scales and the inductive bias in local modeling. Although this fixed receptive field improves the computational efficiency of CNN, it also limits the ability to capture the relationship between distant pixels in medical images and lacks the ability to model medical images in a long-range.

### Transformer-based Methods

In 2017, Vaswani et al. [10] proposed the first Transformer network. Because of its unique structure, Transformer obtains the ability of processing indefinite-length input, establishs long-range dependency relationship, and captures global features of input data. Transformer's success is mainly attributed to the self-attention (SA) mechanism because it can capture long-range dependency.

With the excellent performance of the Transformer in NLP fields, ViT [23] firstly applies Transformer to the field of image processing, capturing the global context information of input images through multiple cascaded Transformer layers, making Transformer a great success in image classification tasks. Then, Chen et al. proposed TransUNet [24], which opened a new era of Transformer in the field of medical image segmentation. As TransUNet directly uses the Transformer network designed for NLP in the field of image segmentation, the size of the input image block is fixed and the calculation is massive. To solve this problem, Valanarasu et al. proposed MedT [25] for medical image segmentation. It adds the gating mechanism to the network, and the gating parameters can be automatically adjusted to obtain the position embedding weight suitable for datasets of different sizes. Since images are more diverse than text and have high resolution, Cao et al. proposed a pure Transformer network Swin-Unet [11] for medical image segmentation by combining the shifted window multi-head self-attention (SW-MSA) in Swin Transformer [26]. Swin-Unet achieved the most advanced segmentation performance at that time on Synapse and ACDC multi-organ segmentation datasets. In order to better use Transformer to process dermoscopic image data, Wang et al. designed BAT [12] network based on the edge detection idea. The proposed boundary-wise attention gate (BAG) can fully utilize the prior knowledge of image boundaries to capture more details of medical images. BAT achieves amazing segmentation Dice value on the skin lesion datasets and surpasses many of the latest medical image segmentation networks.

Compared with the previous vanilla convolution [3], dynamic convolution [19][27] and deformable convolution [21], our DDConv can not only adaptively change the weight coefficient and deformation offset of the convolution according to the medical image task, but also better adapt to the shape of organs and small lesions with large-scale changes in medical images, and additionally, it can improve the local feature expression ability of the segmentation network. Compared with the self-attention mechanism in existing Transformer architectures [11][12], our (S)W-ACAM requires fewer parameters and less computation while it is capable of capturing the global cross-dimensional long-range dependency in the medical images, and improving the global feature expression ability of the segmentation network. Our TEC-Net does not require a large amount of labeled data for pre-training, but it can maximize the retention of local details and global semantic information in medical images. It achieves the best segmentation performance on dermoscopic images, liver datasets, and cardiac multi-organ datasets.

## 3 Method

### Overall Architecture

The fusion of local and global features are clearly helpful for improving medical image segmentation. CNN capture local features of medical images through convolution operation and hierarchical feature representation. In contrast, the Transformer network realizes the extraction of global features in medical images through the cascaded self-attention mechanism and the matrix operation with context interaction. In order to make full use of local details and global semantic features of medical images, we design a parallel interactive network architecture TEC-Net. The overall architecture of the network is shown in Fig. 1(a).

TEC-Net fully considers the complementary properties of CNN and Transformer. During the forward propagation process, TEC-Net continuously feeds the local details extracted by the CNN branch to the decoder of the Transformer branch. Similarly, TEC-Net also feeds the global long-range relationships captured by the Transformer branch to the decoder of the CNN branch. Obviously, the proposed TEC-Net provides better local and global feature representation than pure CNN or Transformer networks, and it shows great potential in the field of medical image segmentation.

Specifically, TEC-Net consists of four components: a patch embedding model, a dynamically adaptive CNN branch, a cross-dimensional fusion Transformer branch, and a feature fusion module. Among them, the dynamically adaptive CNN branch and the cross-dimensional fusion Transformer branch follow the design of U-Net and Swin-Unet, respectively. The dynamically adaptive CNN branch consists of seven main stages. By using the weight coefficient and deformation offset adaptive DDConv in each stage, the segmentation network can better understand the local semantic features of medical images, better perceives the subtle changes of human organs or lesions, and improves the ability to extract multi-scale change targets in medical images. Similarly, the cross-dimensional fusion Transformer branch also consists of seven main stages. By using (S)W-ACAM attention in each stage, as shown in Fig. 1(b), the segmentation network can better understand the global dependency of medical images to capture the position information between different organs, and improves the separability of segmented objects and the background in medical images.
Although our TEC-Net can effectively improve the feature representation of medical images, it requires a large number of training data and network parameters due to the dual-branch structure. As the conventional Transformer network contains a lot of multi-layer perceptron (MLP) layers, which not only aggravates the training burden of the network but also causes the number of model parameters rise sharply, resulting in the slow training for the model. Inspired by the idea of the Ghost network [28], we redesign the MLP layer in the original Transformer and proposed a lightweight perceptron module (LPM). The LPM can help our TEC-Net not only achieve better medical image segmentation results than MLP but also greatly reduces the number of parameters and computational costs, even the Transformer can achieve good results without a lot of labeled data training. It is worth mentioning that the dual-branch structure involves mutually symmetric encoders and decoders so that the parallel interaction network structure can maximize the preservation of local features and global features in medical images.

### Dynamic Deformable Convolution

Vanilla convolution has spatial invariance and channel specificity, so it has a limited ability to change different visual modalities when dealing with different spatial locations. At the same time, due to the limitations of the receptive field, it is difficult for vanilla convolution to extract features of small targets or targets with blurred edges. Therefore, vanilla convolution inevitably has poor adaptability and weak generalization ability for feature representation of complex medical images. Although the existing deformable convolution [21] and dynamic convolution [19][27] outperforms vanilla convolution to a certain extent, they still face a problem of balancing the performance and size of networks when dealing with medical image segmentation.

In order to overcome the shortcomings of popular convolution operations, this paper proposes a new convolution strategy namely DDConv, as shown in Fig. 2. It can be seen that DDConv can adaptively learn the kernel deformation offset and weight coefficients according to the specific task and data distribution, so as to realize the change of both the shapes and the values of convolution kernels. It can effectively deal with the problems of large data distribution differences and large target deformation in medical image segmentation. Also, DDConv is plug-and-play and can be embedded in any network structure.

The shape change of the convolutional kernel in DDConv is based on the network learning of the deformation offsets. The segmentation network first samples the input feature map \(X\) using a square convolutional kernel \(S\), and then performs a weighted sum with a weight matrix \(M\). The square convolution kernel \(S\) determines the range of the receptive field, e.g., a \(3\times 3\) convolution kernel can be expressed as:

\[S=\{(0,0),(0,1),(0,2),...,(2,1),(2,2)\}, \tag{1}\]

then the output feature map \(Y\) at the coordinate \(\varphi_{n}\) can be

Figure 1: (**a**) The architecture of TEC-Net. TEC-Net consists of a dual-branch interaction between dynamically adaptive CNN and cross-dimensional feature fusion Transformer. The DDConv in the CNN branch can adaptively change the weight coefficient and deformation offset of the convolution itself, which improves the segmentation accuracy of irregular objects in medical images. The (S)W-ACAM in the Transformer branch can capture the cross-dimensional long-range dependency in medical images, improving the separability of segmented objects and backgrounds in medical images. The lightweight perceptron module (LPM) greatly reduces the parameters and calculations of the original Transformer network by using the Ghost strategy. (b) Two successive Transformer blocks. W-ACAM and SW-ACAM are cross-dimensional self-attention modules with shifted windows and compact convolutional projection configurations.

Figure 2: The module of the proposed DDConv. Compared with the current popular convolution strategy, DDConv can dynamically adjust the weight coefficient and deformation offset of the convolution itself during the training process, which is conducive to the feature capture and extraction of irregular targets in medical images. \(\alpha\) and \(\beta\) represent the different weight values of DDConv in different states.
expressed as:

\[Y\left(\varphi_{n}\right)=\sum_{\varphi_{n\in S}}S\left(\varphi_{m}\right)\cdot X \left(\varphi_{n}+\varphi_{m}\right), \tag{2}\]

when the deformation offset \(\triangle\varphi_{m}=\left\{m=1,2,3,\ldots,N\right\}\) is introduced in the weight matrix \(M\), \(N\) is the total length of \(S\). Thus the Equation (2) can be expressed as:

\[Y\left(\varphi_{n}\right)=\sum_{\varphi_{m\in S}}S\left(\varphi_{m}\right) \cdot X\left(\varphi_{n}+\varphi_{m}+\triangle\varphi_{m}\right). \tag{3}\]

Through network learning, an offset matrix with the same size as the input feature map can be finally obtained, and the matrix dimension is twice that of the input feature map.

To show the convolution kernel of DDConv is dynamic, we first present the output feature map of vanilla convolution:

\[y=\sigma(W\cdot x), \tag{4}\]

where \(\sigma\) is the activation function, \(W\) is the convolutional kernel weight matrix and \(y\) is the output feature map. In contrast, the output of the feature map of DDConv is:

\[\hat{y}=\sigma\left(\left(\alpha_{1}\cdot W_{1}+\ldots+\alpha_{n}\cdot W_{n} \right)\cdot x\right), \tag{5}\]

where \(n\) is the number of weight coefficients, \(\alpha_{n}\) is the weight coefficients with learnable parameters and \(\hat{y}\) is the output feature map generated by DDConv. DDConv achieves dynamic adjustment of the convolution kernel weights by linearly combining different weight matrices according to the corresponding weight coefficients before performing the convolution operation.

According to the above analysis, we can see that DDConv realizes the dynamic adjustment of the shape and weights of the convolution kernel. Compared with directly increasing the number and size of convolution kernels, the DDConv is simpler and more efficient. The proposed DDConv not only solves the problem of poor adaptive feature extraction ability of fixed-size convolution kernels but also overcomes the defect that different inputs share the same convolution kernel parameters. Consequently, our DDConv can be used to improve the segmentation accuracy of small targets and large targets with blurred edges in medical images.

### Shifted Window Adaptive Complementary Attention Module

The self-attention mechanism is the core computing unit in Transformer networks, which realizes the capture of long-range dependency of feature maps by utilizing matrix operations. However, the self-attention mechanism only considers the dependency in the spatial dimension but not the cross-dimensional dependency between spatial and channels [29]. Therefore, when dealing with medical image segmentation with low contrast and high-density noise, the self-attention mechanism is easy to confuse segmentation targets with their background, resulting in poor segmentation results.

To solve the problems mentioned above, we propose a new cross-dimensional self-attention module called (S)W-ACAM. As shown in Fig. 3, (S)W-ACAM consists of four parallel branches, the top two branches are the conventional dual attention module and the bottom two branches are cross-dimensional attention modules. Compared to popular self-attention modules such as spatial self-attention, channel self-attention, and dual self-attention, our proposed (S)W-ACAM can not only fully extract the long-range dependency of both spatial and channels, but also capture the cross-dimensional long-range dependency between spatial and channels. These four branches complement each other, provide richer long-range dependency relationships, enhance the separability between the foreground and background, and thus improve the segmentation results for medical images.

The standard Transformer architecture [23] uses the global self-attention method to calculate the relationship between one token and all other tokens. This calculation method is complex since the computational costs will increase exponentially with the increase of image size. In order to improve the calculation efficiency, we use the shifted window calculation method similar to that in Swin Transformer [26], which only calculates the self-attention in the local window. However, in the face of our (S)W-ACAM four branches module, using the shifted window method to calculate self-attention does not reduce the overall computational complexity of the module. Therefore, we also designed the compact convolutional projection. First, we reduce the local size of the medical image through the shifted window operation, then we compress the channel dimension of feature maps through the compact convolutional projection, and finally calculate the self-attention. It is worth mentioning that this method can not only better capture the global information of medical images but also significantly reduce the computational costs of the module.

Suppose an image contains \(h\times w\) windows, each window size is \(M\times M\), then the complexity of the (S)W-ACAM, the global MSA in the original Transformer, and the (S)W-MSA in the Swin Transformer are compared as follows:

\[\Omega\left(MSA\right)=4hwC^{2}+2(hw)^{2}C, \tag{6}\]

Figure 3: The module of the proposed (SW)W-ACAM. Unlike conventional self-attention, (SW)W-ACAM has the advantages of spatial and channel attention, and can also capture long-distance correlation features between spatial and channels. Through the shifted window operation, the spatial resolution of images is significantly reduced, and through the compact convolutional projection operation, the channel dimension of images is also significantly reduced. Thus, the overall computational costs and complexity of our proposed network are reduced. \(\mathbf{\lambda_{1}}\), \(\mathbf{\lambda_{2}}\), \(\mathbf{\lambda_{3}}\) and \(\mathbf{\lambda_{4}}\) are learnable weight parameters.
\[\Omega\left((S)W\text{-}MSA\right)=4hwC^{2}+2M^{2}hwC, \tag{7}\]

\[\Omega\left((S)W\text{-}ACAM\right)=\frac{hwC^{2}}{4}+M^{2}hwC, \tag{8}\]

if the former term of each formula is a quadratic function of the number of patches \(h\cdot w\), the latter term is linear when \(M\) is fixed (the default is 7). Then the computational cost of (S)W-ACAM is smaller than MSA and (S)W-MSA.

Among the four parallel branches of (S)W-ACAM, two branches are used to capture channel correlation and spatial correlation, respectively, and the remaining two branches are used to capture the correlation between channel dimension \(C\) and space dimension \(H\) and vice versa (between channel dimension \(C\) and space dimension \(W\)). After adopting the shifted window partitioning method, as shown in Fig. 1(b), the calculation process of continuous Transformer blocks is as follows:

\[\hat{T}^{l}=W\text{-}ACAM\left(LN\left(T^{l-1}\right)\right)+T^{l-1}, \tag{9}\]

\[T^{l}=LPM\left(LN\left(\hat{T}^{l}\right)\right)+\hat{T}^{l}, \tag{10}\]

\[\hat{T}^{l+1}=SW\text{-}ACAM\left(LN\left(T^{l}\right)\right)+T^{l}, \tag{11}\]

\[T^{l+1}=LPM\left(LN\left(\hat{T}^{l+1}\right)\right)+\hat{T}^{l+1}, \tag{12}\]

where \(\hat{T}^{l}\) and \(T^{l}\) represent the output features of (S)W-ACAM and LPM, respectively. W-ACAM represents window adaptive complementary attention, SW-ACAM represents shifted window adaptive complementary attention, and LPM represents lightweight perceptron module. For the specific attention calculation process of each branch, we follow the same principle in Swin Transformer as follows:

\[Attention\left(Q,K,V\right)=SoftMax\left(\frac{QK^{T}}{\sqrt{C/8}}+B\right)V, \tag{13}\]

where relative position bias \(B\in\mathbb{R}^{M^{2}\times M^{2}}\), \(Q,K,V\in\mathbb{R}^{M^{2}\times\frac{Q}{8}}\) are query, key, and value matrices respectively. \(\frac{C}{8}\) represents the dimension of query/key, and \(M^{2}\) represents the number of patches.

After obtaining \(Out_{1}\), \(Out_{2}\), \(Out_{3}\) and \(Out_{4}\), the final feature fusion output is:

\[Out=\lambda_{1}\cdot Out_{1}+\lambda_{2}\cdot Out_{2}+\lambda_{3}\cdot Out_{ 3}+\lambda_{4}\cdot Out_{4}, \tag{14}\]

where \(\lambda_{1}\), \(\lambda_{2}\), \(\lambda_{3}\) and \(\lambda_{4}\) are learnable parameters that enable adaptive control of the importance of each attention branch.

Different from other self-attention mechanisms, our proposed (S)W-ACAM can fully capture the correlation between spatial and channels, and reasonably use the context information of medical images to achieve long-range dependency modeling. Since our (S)W-ACAM effectively overcomes the defect that the conventional self-attention only focuses on the spatial self-attention of images and ignores the channel and cross-dimensional self-attention, it achieves better feature representation for medical images with high-density noise, low contrast and complex background.

### Loss Function

In our task, three loss functions are used for model training, namely, the overall loss \(L_{TEC}\) of TEC-Net network, the loss \(L_{CNN}\) of CNN branch and the loss \(L_{Trans}\) of Transformer branch.

\[L_{TEC}=L_{MSE}\left(y_{i}^{TEC},y_{i}^{label}\right)+L_{Dice}\left(y_{i}^{TEC},y_{i}^{label}\right), \tag{15}\]

\[L_{CNN}=L_{MSE}\left(y_{i}^{CNN},y_{i}^{label}\right)+L_{Dice}\left(y_{i}^{CNN },y_{i}^{label}\right), \tag{16}\]

\[L_{Trans}=L_{MSE}\left(y_{i}^{Trans},y_{i}^{label}\right)+L_{Dice}\left(y_{i}^{ Trans},y_{i}^{label}\right), \tag{17}\]

where \(L_{MSE}(\bullet)\) represents mean squared error loss, and \(L_{Dice}(\bullet)\) represents Dice loss. \(y_{i}^{TEC}\), \(y_{i}^{CNN}\), \(y_{i}^{Trans}\) and \(y_{i}^{label}\) represent the final predicted image using our TEC-Net network, the predicted image from the CNN branch output, the predicted image from the Transformer branch output, and the label, respectively. The total loss function of TEC-Net network can be expressed as:

\[L_{Total}=\lambda L_{TEC}+\left(\left(1-\lambda\right)/2\right)L_{CNN}+\left( \left(1-\lambda\right)/2\right)L_{Trans}, \tag{18}\]

where \(\lambda=\delta e^{-5\left(1-k\right)^{2}}\), \(\lambda\) is a Gaussian ramp-up curve, \(k\) represents the number of epochs.

### Architecture Variants

We have built a TEC-Net-T as a base network with a model size of 11.58 M and a computing capacity of 4.53 GFLOPs. In addition, we built the TEC-Net-B network to make a fair comparison with the latest networks such as CvT [33] and PVT [34]. The window size is set to 7, and the input image size is \(224\times 224\). Other network parameters are set as follows:

* TEC-Net-T: \(layer\)\(number=\{2,\ 2,\ 6,\ 2,\ 6,\ 2,\ 2\}\), \(H=\{3,\ 6,\ 12,\ 24,\ 12,\ 6,\ 3\}\), \(D=96\)
* TEC-Net-B: \(layer\)\(number=\{2,\ 2,\ 18,\ 2,\ 18,\ 2,\ 2\}\), \(H=\{4,\ 8,\ 16,\ 32,\ 16,\ 8,\ 4\}\), \(D=96\)

where \(D\) represents the number of image channels when entering the first layer of the dynamically adaptive CNN branch and the cross-dimensional fusion Transformer branch, \(layer\)\(number\) represents the number of Transformer blocks used in each stage, and \(H\) represents the number of multiple heads in the Transformer branch.

## IV Experiment and Results

### Datasets

We conducted experiments on the skin lesion segmentation dataset (ISIC2018) from the International Symposium on Biomedical Imaging (ISBI) [36], the Liver Tumor Segmentation Challenge dataset (LiTS) from the Medical Image Computing and Computer Assisted Intervention Society (MICCAI) [37] and the Automated Cardiac Diagnosis Challenge dataset (ACDC) from the University Hospital of Dijon (France) [38]. These three datasets have different data types and data distributions. Among them, the ISIC2018 dataset is electron microscope images, which contain 2,594 dermoscopic images for training, but the ground truth images of the testing set have not been released, so we performed a five-fold cross-validation on the training set for a fair comparison. The LiTS
dataset is a CT image of the human abdomen, containing 131 3D CT liver scans, where 100 scans of which are used for training, and the remaining 31 scans are used for testing. The ACDC dataset is an MRI image of the human heart, which contains cardiac short-axis Cine MRI data from 100 patients. The ACDC dataset includes healthy patients, patients with previous myocardial infarction, dilated cardiomyopathy, hypertrophic cardiomyopathy, and abnormal right ventricle, with 20 scans for each group. These data are obtained over a 6 years period using two MRI scanners of two magnetic strengths (1.5T and 3.0T). In addition, all images are empirically resized to 224x224 for efficiency.

### Implementation Details and Evaluation Indicators

All the networks are implemented on NVIDIA GeForce RTX 3090 24GB and PyTorch 1.7. We utilized Adam with an initial learning rate of 0.001 to optimize the networks. The learning rate decreases in half when the loss on the validation set has not dropped by 10 epochs. We used mean squared error loss (MSE) and Dice loss as loss functions in our experiment.

In the experiment on the ISIC2018 dataset, we conducted an overall evaluation for SOTA networks and the proposed TEC-Net using five indicators: Dice (DI), Jaccard (JA), Sensitivity (SE), Accuracy (AC), and Specificity (SP) [39]. In the experiment on the LiTS-Liver dataset, we conducted an overall evaluation for SOTA networks and the proposed TEC-Net using five indicators: DI, VOE, RVD, ASD, and RMSD. In the experiment on the ACDC dataset, we conducted an overall evaluation for SOTA networks and the proposed TEC-Net using DI and 95HD [40].

The values of DI, JA, AC, SE, and SP range from 0 to 100, and better segmentation results mean that the values of evaluation indicators such as DI, JA, AC, SE, and SP will be higher, while the values of evaluation indicators such as VOE, RVD, ASD, RMSD, and 95HD will be lower [41].The 95HD is defined as the 95th quantile of Hausdorff distances (HD) instead of the maximum [42].

### Evaluation and Results

In this paper, we selected popular SOTA networks for medical image segmentation networks U-Net [3], R2UNet [16], Attention Unet [30], CENet [31], 3D Unet [8], V-Net [9], Swin-Unet [11], TransUNet [24], CVT [33], PVT [34], CrossForm [35] and the proposed TEC-Net to conduct a comprehensive comparison of the three different modalities datasets, the ISIC2018, the LiTS-Liver and the ACDC.

TABLE 1 shows the quantitative analysis results of the proposed TEC-Net and the competitive CNN and Transformer
networks on the ISIC2018 dataset. From the experimental results, we can conclude that our TEC-Net needs the minimum number of parameters and the lowest computational costs, and can obtain the best segmentation effect on the dermoscopic images without adding pre-training. Moreover, our TEC-Net-T network requires only 11.58 M of parameters and 4.53 GFLOPs of computational costs, but still achieves the second-best segmentation effect. Our TEC-Net-B network, BAT, CvT, and CrossForm have similar parameters or computational costs, but on the ISIC2018 dataset, the division Dice value of our TEC-Net-B is 1.02%, 3.00%, and 3.79% higher than that of the BAT, CvT, and CrossForm network respectively. In terms of other evaluation indicators, our TEC-Net-B is also significantly better than other competitive networks.

TABLE II shows the quantitative analysis results of the proposed TEC-Net and the competitive networks on the LiTS-Liver dataset. It can be seen from the experimental results that our TEC-Net shows great advantages in medical image segmentation, which further verifies the integrity of TEC-Net in extracting local and global features for medical images. It is worth noting that the TEC-Net-B and TEC-Net-T networks achieve good results in medical image segmentation in the first and second place, with the least number of model parameters and computational costs. The division Dice value of our TEC-Net-B network without pre-training is 1.20%, 1.03%, and 1.01% higher than that of the Swin-Unet, TransUNet, and CvT network with pre-training. In terms of other evaluation indicators, our TEC-Net-B is also significantly better than other competitive networks.

TABLE III shows the quantitative analysis results of the proposed TEC-Net and the competitive networks on the ACDC dataset. From the experimental results, it can be seen that the proposed TEC-Net still exhibits significant advantages on MRI type multi-organ segmentation datasets. Both TEC-Net-T and TEC-Net-B provide state-of-the-art segmentation effects for the left ventricle (LV), right ventricle (RV), and left ventricular myocardium (MYO). Among them, LV provides the best segmentation effect, while MYO provides a poor segmentation effect. Compared with the latest CvT, PVT, and CrossForm, the average segmentation performance of TEC-Net-B improves by 0.98%, 1.45%, and 1.38%, respectively, while the average 95HD decreases by 0.72%, 0.78%, and 0.74%, respectively. It also proves that TEC-Net has strong generalization performance on different datasets and can be flexibly applied to medical image segmentation tasks in different modalities and collection environments.

From the visualization in Fig. 4, it can be clearly seen that the TEC-Net network can effectively extract local detail features and global semantic features in medical images to the maximum extent possible. Therefore, it provides accurate segmentation results for segmentation targets with irregular edges and large deformation scales. Fig. 4(a) shows the final output result from the CNN branch, and Fig. 4(b) shows

Fig. 4: Visualization effect display of TEC-Net network on ISIC2018 dataset and LTS-Liver dataset. (a) The predicted image output by the CNN branch, (b) the predicted image output by the Transformer branch, (c) the predicted image output by the TEC-Net network, (d) the corresponding label, (e) the corresponding original image.
the final output result from the Transformer branch. We can find that the CNN branch captures more accurate local detail information of segmented targets. However, due to the fact that CNN branches mainly capture local features of images, they are more susceptible to noise interference. However, the Transformer branch captures more accurate position information of segmented targets. Due to the fact that the Transformer branch mainly captures the global features of the image, it is easy to overlook the detailed information of the segmented target. So, after integrating the CNN branch with the Transformer branch, the TEC-Net network fully inherits the structural and generalization advantages of CNN and Transformer, providing better local and global feature representations for medical images, demonstrating great potential in the field of medical image segmentation.

### Ablation Study

In order to fully prove the effectiveness of different modules in our TEC-Net, we conducted a series of ablation experiments on the ISIC2018 dataset. As shown in TABLE IV, we can see that the DDConv and (S)W-ACAM proposed in this paper show good performance, and the combination of these two modules, TEC-Net shows the best medical image segmentation effect.

### Network Visualization

In order to better understand the TEC-Net network, we visualized the feature maps of each stage of the network, as shown in Fig. 5. We used the linear interpolation method to restore the deep feature map with the low spatial resolution to the same size as the input and output images. In the visualization, red represents areas that the network is more concerned about, while blue represents areas that the network is less concerned about.

Through visualization images, we can clearly see the entire process of feature extraction, feature fusion, and other stages that the image undergoes after being input into the network. In the two branches of the encoder, the model can analyze and identify the semantic information of images. In the two branches of the decoder, after skip connections and feature fusion operations, the model began to pay more attention to the semantic features of target regions. During this process, the information exchange and fusion between CNN and Transformer branches play an important role in the accurate segmentation of target regions.

As a whole structure, as the number of network layers continues to deepen, the TEC-Net network gradually refines the localization and contour segmentation of semantic objects in medical images, proving the effectiveness of our proposed TEC-Net network for global information modeling and accurate segmentation of medical images.

## V Conclusion

In this study, we have proposed a new architecture TEC-Net that combined dynamically adaptive CNN and cross-dimensional fusion Transformer in parallel for medical image segmentation. The proposed TEC-Net integrates the advantages of both CNN and Transformer, and retains the local details and global semantic features of medical images using local relationship modeling and long-range dependency modeling. The proposed DDConv overcomes the problems of fixed receptive field and parameter sharing in vanilla convolution, enhances the ability to express local features, and realizes adaptive extraction of spatial features. The proposed (S)W-ACAM self-attention mechanism can fully capture the cross-dimensional correlation between spatial and channels, and adaptively learn the important information between spatial and channels through network training. In addition, by using the LPM to replace the MLP in the traditional Transformer, our TEC-Net significantly reduces the number of parameters, gets rid of the dependence of the network on pre-training, avoids the challenge of lacking labeled medical images and easy suffering from over-fitting. Compared with popular CNN and Transformer medical image segmentation networks, our TEC-Net shows significant advantages in terms of operational efficiency and segmentation effect.
