11institutetext:  Wellcome Centre for Human Neuroimaging, University College
London 22institutetext: NVIDIA 33institutetext: Department of Neurology,
University of California, San Francisco 44institutetext: Helen Wills
Neuroscience Institute, University of California, Berkeley 55institutetext:
Stevens Institute for Neuroimaging and Informatics, University of Southern
California
55email<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Large-kernel Attention for Efficient and Robust Brain Lesion Segmentation

Liam Chalcroft 11 Ruben Lourenço Pereira 11 Mikael Brudfors 1122 Andrew S.
Kayser 3344 Mark D’Esposito 44 Cathy J. Price 11 Ioannis Pappas∗ 4455 John
Ashburner These authors contributed equally to this work11

###### Abstract

Vision transformers are effective deep learning models for vision tasks,
including medical image segmentation. However, they lack efficiency and
translational invariance, unlike convolutional neural networks (CNNs). To
model long-range interactions in 3D brain lesion segmentation, we propose an
all-convolutional transformer block variant of the U-Net architecture. We
demonstrate that our model provides the greatest compromise in three factors:
performance competitive with the state-of-the-art; parameter efficiency of a
CNN; and the favourable inductive biases of a transformer. Our public
implementation is available at github.com/liamchalcroft/MDUNet.

##  1 Introduction

Image segmentation is a common processing step in neuroimaging, and the
delineation of pathology from healthy tissue has use in downstream tasks in
both research and clinical settings. Labelling of the fine structure of a
tumour is often performed for radiotherapy planning [30], and details of the
size and location of a stroke lesion may be used for prognosis of associated
conditions like aphasia [32]. Whilst labelling healthy brain structures has
long been performed using atlas-based approaches, the heterogeneity of
pathologies means that they cannot be effectively modelled in an atlas and so
were traditionally delineated through anomaly detection algorithms [37, 24].
Generative models (e.g. a mixture model) are fitted with atlas-based healthy
tissue priors, adding an additional class to identify pixels that do not fit
appropriately into the available tissue classes.

Prior to deep learning, discriminative methods were introduced to classify
each pixel based on its neighbours. Asymmetry maps (the difference between an
image and its co-registered mirror image) have been used as additional input
channels to exploit the asymmetric nature of pathologies [33]. Alternatively,
features are extracted based on symmetric and asymmetric templates [41]. All
of the above methods require images to be skull-stripped and aligned with a
healthy template brain.

The introduction of neural-network-based computer vision architectures to
medical imaging has led to the success of discriminative models trained
through supervised learning. These employ convolutional neural network (CNN)
architectures such as U-Net [36]. CNNs learn a series of small kernels
(typically of size $3^{n}$ for $n$-dimensional data), with intermediate
pooling operations allowing an increasing receptive field by which to classify
each pixel. The effect of this is that networks focus on local interactions,
particularly in the higher-resolution layers. This is at odds with the typical
clinical process that involves comparing pixels on opposite sides of a scan
[12]. It is therefore desirable to develop a segmentation framework that can
better leverage global interactions - in addition to local interactions like
textures - to provably identify pathologies with the same visual cues
available to a human clinician. It is expected that a model that can learn
shape-based features will further be more robust to domain shifts in
appearance caused by variation in MRI scanner or sequence used.

Transformers - Recent computer vision research proposes using attention-based
transformer models for tasks such as classification and segmentation. These
models split images into one-dimensional input tokens and use a spatial token
mixing operation and a channel-wise feed-forward network [9, 44]. While
transformers perform well in modeling long-range interactions [34], their
attention mechanism can be expensive and they lack an inductive bias towards
local interactions. Window-based mechanisms like Swin [27, 16] have been
proposed to address this, but they add computational overhead. Alternatives
include applying transformers to bottleneck features [7] or applying the
transformer layout to CNN blocks [23], or using a differentiable matrix
factorization [1].

The key benefit of transformer-based models is believed to lie in their global
receptive field, and recent works have shown that CNNs with sufficiently large
receptive fields can be competitive with state-of-the-art transformers [8,
26]. Due to the importance of long-range features such as contralateral
asymmetry, it is expected that brain lesion segmentation will benefit from a
larger receptive field.

Texure vs. Shape Bias - In addition to the aforementioned properties, the
differences in inductive biases of models favouring textures (high-frequency,
local features) versus shapes (low-frequency, global features) has been
explored in detail for CNNs. It has been found that CNNs are biased towards
texture, and that enforcing a stronger shape bias can improve robustness [11].
This has also been shown to hurt performance in low-data regimes, such as few-
shot learning [35]. Transformers have been shown to have a stronger inductive
bias towards shape [39]. It can be argued that, due to the heterogeneity of
lesions in terms of position, texture-biased local features may lead to more
spurious correlations between lesions and their neighbouring healthy tissue,
compared to a shape-biased model. Methods such as SynthSeg [6] may be argued
to be relaxing the texture bias by extreme augmentation, and this may
contribute to their superior robustness in out-of-distribution (OOD)
scenarios. It is therefore expected that a model with greater shape bias
should perform better on OOD data.

Large-Kernel Attention - A recently proposed alternative to windowed attention
is large-kernel attention (LKA) [14], where a large-kernel convolution is made
feasible by decomposing the transform into a series of convolutions. This is
achieved by a series of a depth-wise convolution (DWConv), a dilated depth-
wise convolution (dDWConv) and a pointwise convolution (PConv). A DWConv is a
filter where the weights are shared across all channels, whilst a PConv is a
$1\times 1$ filter that will rescale the number of channels. This series of
kernels factorises a large kernel by separating the spatial and channel mixing
aspects, and then further separating the spatial mixing into two small
kernels:

$\begin{array}[]{r c
l}\mathbf{X_{1}}&=&\mathrm{dDWConv}(\mathrm{DWConv}(\mathbf{X})),\\\
\mathbf{X_{2}}&=&\mathrm{PConv}(\mathbf{X_{1}}),\end{array}$ (1)

where $\mathbf{X_{1}}$ contains the result from spatial mixing and
$\mathbf{X_{2}}$ contains the result from channel mixing. This then makes up
the token-mixing component of a transformer block. The spatial mixing may be
further separated into a local context (DWConv) and global context (dDWConv).

Results on natural images have indicated that this method attains greater
accuracy than other vision models, and has a favourable accuracy-efficiency
tradeoff when compared to the Swin architecture. Similar results have been
reported for medical image registration when using a CNN with large kernels
[22]. In our work, a novel architecture is proposed that aims to bridge the
trade-off between receptive field and efficiency by extending LKA to 3D and
incorporating attention blocks into a U-Net-style architecture.

##  2 Methods

The core of the architecture employed in this work is summarised in Fig. 1.
The LKA unit, shown in Fig. 1(a), generates an attention mask to multiply over
the input features, through a set of learned convolutions described in Eq. 1.
This series of layers is equivalent to a matrix decomposition of a larger
kernel [14]. In all experiments, DWConv was used with a kernel size of
$5^{3}$, dDWConv with a kernel size of $7^{3}$ and dilation of $3^{3}$ to give
an attention mechanism equivalent to a kernel size of $21^{3}$. This is then
used within the overall attention module in Fig. 1(b), with a series of a
PConv, GELU [17], LKA and PConv with a residual connection between the input
of the module and the output of the PConv layer.

Transformer layers typically contain both an attention module and a feed-
forward module - for this work a convolutional feed-forward (ConvFF) module is
used [42]. Shown in 1(c), this consists of a PConv, DWConv (kernel size
$3^{3}$), GELU and a final PConv. These modules are combined to create the
overall LKA block in Fig. 1(d), consisting of an overlapping patch embedding
[43] with a stride of $2^{3}$ and patch size of $3^{3}$ \- this is used
directly as a replacement for the strided convolution used for downscaling in
convolutional U-Nets. Following the embedding, $N$ repeating units are used,
consisting of Batch Norm (BN) [19], attention module, BN, ConvFF, with
residual connections between the pre-BN and post-(attention/ConvFF) features.
An additional scaling parameter is learnt for the channel-wise magnitude of
the post-(attention/ConvFF) layer, which has been shown previously to improve
training stability [38]. The final output is normalised using a Layer Norm
unit [4].

The layers are constructed in a U-Net style design with six stages, using
feature maps with ($32$, $64$, $128$, $256$, $320$, $320$) channels
respectively in order to match the implementation of the popular nnUNet [20].
For the repeating LKA units, a size of $N=1$ is used for all stages except the
$5$th, which uses $N=2$.

(a) LKA

(b) Attention

(c) ConvFF

(d) LKA Block

Figure 1: Summary of LKA block 1(d) used as direct drop-in for convolution
block in U-Net, and its components LKA 1(a), attention 1(b) and ConvFF 1(c).

Decoder design - The authors of the original Visual Attention Network paper
[14] recently extended their model with a design tailored to semantic
segmentation [13], replacing LKA with a Multi-Scale Convolutional Attention
(MSCA). The choice of decoder is also discussed in their work, looking at
other transformer decoders used in recent works such as SegFormer [43]. We
also propose a model using attention layers in the decoder, but opt for a
design consistent with the proven U-Net decoder layout. We thus evaluate two
models: LKA-E, which uses the default convolutional U-Net decoder; and LKA-ED,
which uses LKA blocks in the decoder. In LKA-ED, upsampling is performed by
replacing the convolutional layer of the patch embedding with a transpose
convolution - this can be thought of as performing a learnt upsampling for
each patch before the subsequent layer.

##  3 Experiments

A number of tasks were performed to ascertain model performance in a variety
of scenarios. The first task, ISLES [31], involved segmentation of multi-modal
MRI into binary stroke lesion labels. The second, ATLAS [25], segmented
T1-weighted MRI into binary stroke lesion labels. Finally, the BraTS task
segmented multi-modal MRI into multi-class labels of glioblastoma tumours [5].

Baseline methods - For all experiments, the LKA-E and LKA-ED models are
compared to a U-Net [36, 20] and a Swin-UNETR [16]. The U-Net is configured
with the same number of channels per stage as LKA-E. The Swin-UNETR model is
configured with $4$ stages of $N=2$ repeating units and ($3$, $6$, $12$, $24$)
heads respectively, with a feature size of $48$. Computational resources for
each model are listed in Table 1, calculated using the FVCore library. Table 1
shows that both LKA-E and LKA-ED are significantly more efficient than Swin-
UNETR and that LKA-ED even requires less compute than the baseline U-Net when
measured in GFLOPS.

Table 1: Model parameters (in millions) of LKA and popular baselines, in addition to the estimated compute (in GFLOPS). Model | # Params (M) | GFLOPS
---|---|---
nnUNet | 31.2 | 479.0
Swin-UNETR | 62.2 | 766.0
LKA-E (Ours) | 33.7 | 496.7
LKA-ED (ours) | 32.9 | 413.2

ISLES - Co-registered ADC/DWI/FLAIR images were resliced to the FLAIR image
using ANTs [2] and stacked to create a 4D volume using FSL [21]. Both of these
steps were performed at test-time using SimpleITK [28] and NumPy [15]. Data
were resliced to $1$ mm3 and cropped to the brain foreground using MONAI. In
training, the DALI library was used for loading with augmentations following
the winner of the BraTS 2021 competition [10] to produce an augmented crop of
size $128^{3}$, and a foreground mask (voxels with positive nonzero intensity)
was appended to the input as an extra data channel. Models were trained using
an $80:20$ data split of $200$ training and $50$ validation samples. A
combination of Dice and cross entropy loss was used, and optimized using the
Apex implementation of Adam with a learning rate of $0.0002$ and a batch size
of $2$ for $1,000$ epochs. Gradient clipping [45] was applied for a norm value
of $12$. For LKA-E and LKA-ED, learning rate was reduced to $0.0001$ and a
gradient clipping norm of $1$ was adopted to ensure stability during training.

ATLAS - T1w images were resliced to $1$ mm3 resolution, skull-stripped using
Robex [18] and bias-corrected using the SimpleITK [28] implementation of N4
[40]. Data were resliced to $1$ mm3 and cropped to the brain foreground using
MONAI. All training and inference parameters were otherwise identical to those
used in ISLES, with $524$ training samples and $131$ validation samples.

BraTS - Data were obtained in a preprocessed format with 4 co-registered
modalities of T1, contrast-enhanced T1 (ceT1), T2 and FLAIR. All images were
skull-stripped and resliced to $1$ mm3, and so the only preprocessing
performed was cropping to the brain foreground using MONAI. All training and
inference parameters were otherwise identical to those used in ISLES, with
$1,000$ training samples and $251$ validation samples.

Compute - Training was performed on a mix of an NVIDIA RTX A6000 GPU, an
NVIDIA RTX A100 GPU (via Google Cloud) and a Google v3-8 TPU (via Google
Cloud). Test-set inference was performed for all models using an NVIDIA RTX
A6000 GPU for public test data, or an NVIDIA T4 GPU for private test sets via
Grand Challenge.

##  4 Results

A number of metrics were used in the varied datasets. The Dice and 95th-
percentile Hausdorff Distance (HD95) were used in the BraTS experiments,
whilst Dice was used in ATLAS and ISLES along with three other metrics
detailed below. Lesion-wise F1: Lesion-wise accuracy is calculated using the
F1/Dice equation, however with a single positive/negative result per lesion,
i.e. a single positive voxel overlapping with the ground truth will return a
true positive.
LCD: The lesion count difference (LCD) is a measure of the difference in
number of individual lesions compared to the ground truth label.
AVD: The average volume difference (AVD) is a measure of the difference in
total lesion volume compared to the ground truth label.

ISLES - Final inference was performed on a private test set of 150 subjects
using a Docker container. The model and inputs were used in half-float
precision with test-time augmentation used for each model’s predictions to
average logits over all possible orientations. Inference of the model in all
orientations was performed using a sliding window of size $128^{3}$, with an
overlap of $0.5$ and a Gaussian weighting to merge windowed predictions. For
the Swin-UNETR model, the original model required too much memory in inference
and so was compiled via TorchScript. For all tests, case-level metrics were
not provided and so it was not possible to perform paired statistical tests.
Results in Table 2 indicate that both versions of LKA improve over both the
CNN and transformer baselines, with a significant improvement in both cases
for the F1 score.

Table 2: Performance of models on ISLES hidden test set. Scores reported are mean values. *: $p<0.05$, **: $p<0.0001$ for an unpaired t-test compared to both baseline models. Model | Dice ($\uparrow$) | F1 ($\uparrow$) | LCD ($\downarrow$) | AVD ($\downarrow$) | Demo
---|---|---|---|---|---
nnUNet | $0.644$ | $0.579$ | $4.560$ | $6.554$ | Link
Swin-UNETR | $0.644$ | $0.580$ | $4.580$ | $6.567$ | Link
LKA-E (Ours) | $0.692$ | $\mathbf{0.681^{**}}$ | $\mathbf{3.733}$ | $6.108$ | Link
LKA-ED (Ours) | $\mathbf{0.693}$ | $0.657^{*}$ | $3.993$ | $\mathbf{6.076}$ | Link

ATLAS - Models were tested on a public test set of $300$ subjects, with
predictions performed locally using half-float precision, test-time
augmentation and a Gaussian-weighted sliding window. For all tests, case-level
metrics were not provided and so it was not possible to perform paired
statistical tests. Table 3 shows that in this task the CNN outperforms other
models. The LKA models both outperform the transformer on this task, and in
some metrics match the CNN, providing a middle ground between the two
architectures.

Table 3: Performance of models on ATLAS public test set. Scores reported are median values. Model | Dice ($\uparrow$) | F1 ($\uparrow$) | LCD ($\downarrow$) | AVD ($\downarrow$) | Demo
---|---|---|---|---|---
nnUNet | $\mathbf{0.697}$ | $\mathbf{0.500}$ | $2.000$ | $2706.5$ | Link
Swin-UNETR | $0.676$ | $0.444$ | $2.000$ | $2706.5$ | Link
LKA-E (Ours) | $0.682$ | $\mathbf{0.500}$ | $2.000$ | $2597.0$ | Link
LKA-ED (Ours) | $0.678$ | $0.474$ | $2.000$ | $\mathbf{2437.5}$ | Link

Receptive Field - The effective receptive fields (ERFs) of the different
models were compared empirically to observe qualitative differences in the
features used by different model architectures. ERFs are acquired for a given
feature map by calculating the average scaled gradient contributing to the
central voxel for 100 subjects in the validation hold-out set. As shown in
Fig. 2, both the LKA models show a noticable increase in ERF in the early
layers over both nnUNet and Swin-UNETR.

(a) nnUNet, Random init.

(b) nnUNet, Trained

(c) Swin-UNETR, Random init.

(d) Swin-UNETR, Trained

(e) LKA-E, Random init.

(f) LKA-E, Trained

(g) LKA-ED, Random init.

(h) LKA-ED, Trained

Figure 2: ERFs for models with the ATLAS validation set. Subfigures show the
ERFs of the final pre-normalisation layer of each encoder stage.

Domain shift - It is posited that the stronger texture bias in CNNs will lead
to poorer robustness against shifts in MRI contrast. This is validated in Fig.
3 where the pre-trained ATLAS models are tested on new T1, T2, DWI and FLAIR
images from the ISLES 2015 dataset [29] for 28 subjects of each sequence.
Results show that the nnUNet model performs significantly worse than all
others, with both LKA variations remaining competitive with the Swin-based
model.

Figure 3: Performance on new MRI sequences for LKA and baseline models.

BraTS - Models were tested on a public test set of $219$ subjects, with
predictions performed locally using half-float precision, test-time
augmentation and a Gaussian-weighted sliding window. Results in Table 4 show
that in this task, the CNN is the lowest performing model and LKA outperforms
even the transformer in all but one metric.

Table 4: Performance of models on BraTS public test set. Scores reported are median values. *: $p<0.0001$ against both baselines with a Wilcoxon signed-ranked test. Model | Dice ($\uparrow$) | HD95 ($\downarrow$)
---|---|---
ET | TC | WT | ET | TC | WT
nnUNet | $0.875$ | $0.921$ | $0.902$ | $1.414$ | $2.236$ | $3.162$
Swin-UNETR | $\mathbf{0.894}$ | $\mathbf{0.932}$ | $0.938$ | $1.414$ | $2.236$ | $3.606$
LKA-E (Ours) | $0.888$ | $\mathbf{0.932}$ | $\mathbf{0.939}$ | $1.414$ | $\mathbf{2.000}$ | $\mathbf{2.449^{*}}$

Texture bias - To further compare the behaviour of the different models, the
inductive biases were probed by using the response to Gaussian smoothing as a
measure of a bias towards texture features [3]. This is expected to be more
dominant in the all-CNN nnUNet than the all-attention Swin encoder, and so it
is of interest to observe how the behaviour of the hybrid proposed in LKA
compares to these. To compare the invariance of the different models to image
texture, the images were blurred with varied kernel sizes (denoted by the
standard deviation $\sigma$), and the models’ prediction for the GD-enhanced
tumour core label compared to the prediction made on the unaugmented image.
The Dice and HD95 metrics were then calculated for these labels, with a
greater accuracy against the original label indicating a stronger bias towards
shape than texture.

Fig. 4 indicates a greater invariance to blurring in both Swin-UNETR and
LKA-E, for measures of overlap (4(a)) and surface distance (4(b)). This
suggests that LKA-E more closely follows the inductive shape bias of the Swin-
UNETR, further supporting the observation made for improved robustness in Fig.
3.

(a)

(b)

Figure 4: Label degradation for GD-enhanced tumour (label ET in tables) for
nnUNet, Swin-UNETR and LKA-E. Central lines on boxplot figures show median
values.

##  5 Conclusions

This paper proposes a hybrid convolutional-attention model for medical image
segmentation, which combines the efficiency of CNNs with the global receptive
field of attention-based transformers. The model is evaluated on three
datasets for 3D brain lesion segmentation, with the LKA module outperforming
traditional CNN and attention-based SOTA models on the first dataset. In the
second experiment, the CNN baseline outperforms attention-based and LKA-based
models in Dice for lesions identified in isotropic T1w MRI, but the LKA model
outperforms both in AVD. Evaluation on OOD data also indicates LKA- and
attention-based models’ superior robustness to unseen MRI sequences. Finally,
the LKA model outperforms both CNN-based and attention-based models in a
number of metrics for tumour sub-sections identified in multi-modal MRI.
Empirical results suggest that LKA models may provide similar inductive biases
to attention models at the computational cost of a CNN. Varying the kernel
size and dilation of LKA ought to make possible the fine-tuning of inductive
bias, which would allow for task-specific tailoring of models on a spectrum
between CNNs and transformers.

Although LKA models have demonstrated good performance compared to SOTA CNNs
and transformers, there are still areas for improvement. Hyperparameters like
kernel size and dilation used in LKA need further exploration, and the value
of patch embeddings in an all-convolution model has not been studied.

##  6 Acknowledgement

LC is supported by the EPSRC-funded UCL Centre for Doctoral Training in
Intelligent, Integrated Imaging in Healthcare (i4health) (EP/S021930/1), and
the Wellcome Trust (203147/Z/16/Z and 205103/Z/16/Z). This research was
supported by NVIDIA and utilized NVIDIA RTX A6000 48GB. This research was also
supported by Google Cloud through the Google Cloud Research Credits program
(GCP19980904) and the Google TPU Research Cloud (TRC).

## References

  * [1] Ashtari, P., Sima, D.M., Lathauwer, L.D., et al.: Factorizer: A scalable interpretable approach to context modeling for medical image segmentation. Medical Image Analysis p. 102706 (Nov 2022)
  * [2] Avants, B.B., Tustison, N., Song, G., et al.: Advanced normalization tools (ants). Insight j 2(365), 1–35 (2009)
  * [3] Azad, R., Fayjie, A.R., Kauffman, C., et al.: On the texture bias for few-shot cnn segmentation (2020)
  * [4] Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization (2016)
  * [5] Baid, U., Ghodasara, S., Mohan, S., et al.: The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification (2021)
  * [6] Billot, B., Greve, D.N., Puonti, O., et al.: Synthseg: Domain randomisation for segmentation of brain scans of any contrast and resolution (2021)
  * [7] Chen, J., Lu, Y., Yu, Q., et al.: Transunet: Transformers make strong encoders for medical image segmentation (2021)
  * [8] Ding, X., Zhang, X., Zhou, Y., et al.: Scaling up your kernels to 31x31: Revisiting large kernel design in cnns (2022)
  * [9] Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al.: An image is worth 16x16 words: Transformers for image recognition at scale (2020)
  * [10] Futrega, M., Milesi, A., Marcinkiewicz, M., et al.: Optimized u-net for brain tumor segmentation (2021)
  * [11] Geirhos, R., Rubisch, P., Michaelis, C., et al.: Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness (2018)
  * [12] González, R.G.: Clinical MRI of acute ischemic stroke. Journal of Magnetic Resonance Imaging 36(2), 259–271 (Jul 2012)
  * [13] Guo, M.H., Lu, C.Z., Hou, Q., , et al.: Segnext: Rethinking convolutional attention design for semantic segmentation (2022)
  * [14] Guo, M.H., Lu, C.Z., Liu, Z.N., et al.: Visual attention network (2022)
  * [15] Harris, C.R., Millman, K.J., van der Walt, S.J., et al.: Array programming with NumPy. Nature 585(7825), 357–362 (Sep 2020)
  * [16] Hatamizadeh, A., Nath, V., Tang, Y., et al.: Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images (2022)
  * [17] Hendrycks, D., Gimpel, K.: Gaussian error linear units (gelus) (2016)
  * [18] Iglesias, J.E., Liu, C.Y., Thompson, P.M., et al.: Robust brain extraction across datasets and comparison with publicly available methods. IEEE transactions on medical imaging 30(9), 1617–1634 (2011)
  * [19] Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift (2015)
  * [20] Isensee, F., Jaeger, P.F., Kohl, S.A.A., et al.: nnU-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods 18(2), 203–211 (Dec 2020)
  * [21] Jenkinson, M., Beckmann, C.F., Behrens, T.E.J., et al.: FSL. Neuroimage 62(2), 782–790 (Aug 2012)
  * [22] Jia, X., Bartlett, J., Zhang, T., et al.: U-net vs transformer: Is u-net outdated in medical image registration? (2022)
  * [23] Lee, H.H., Bao, S., Huo, Y., et al.: 3d ux-net: A large kernel volumetric convnet modernizing hierarchical transformer for medical image segmentation (2022)
  * [24] Leemput, K.V., Maes, F., Bello, F., et al.: Automated segmentation of ms lesions from multi-channel mr images. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 11–21. Springer (1999)
  * [25] Liew, S.L., Lo, B.P., Donnelly, M.R., et al.: A large, curated, open-source stroke neuroimaging dataset to improve lesion segmentation algorithms. Scientific Data 9(1) (Jun 2022)
  * [26] Liu, S., Chen, T., Chen, X., et al.: More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity (2022)
  * [27] Liu, Z., Lin, Y., Cao, Y., et al.: Swin transformer: Hierarchical vision transformer using shifted windows (2021)
  * [28] Lowekamp, B.C., Chen, D.T., Ibáñez, L., et al.: The design of SimpleITK. Front. Neuroinform. 7, 45 (Dec 2013)
  * [29] Maier, O., Menze, B.H., von der Gablentz, J., et al.: ISLES 2015 - a public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI. Medical Image Analysis 35, 250–269 (Jan 2017)
  * [30] Meyer, F., Hoover, J., Morris, J.: Use of preoperative magnetic resonance imaging t1 and t2 sequences to determine intraoperative meningioma consistency. Surgical Neurology International 2(1), 142 (2011)
  * [31] Petzsche, M.R.H., de la Rosa, E., Hanning, U., et al.: Isles 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset (2022)
  * [32] Price, C.J., Hope, T.M., Seghier, M.L.: Ten problems and solutions when predicting individual outcome from lesion site after stroke. NeuroImage 145, 200–208 (Jan 2017)
  * [33] Pustina, D., Coslett, H.B., Turkeltaub, P.E., et al.: Automated segmentation of chronic stroke lesions using LINDA: Lesion identification with neighborhood data analysis. Human Brain Mapping 37(4), 1405–1421 (Jan 2016)
  * [34] Raghu, M., Unterthiner, T., Kornblith, S., et al.: Do vision transformers see like convolutional neural networks? (2021)
  * [35] Ringer, S., Williams, W., Ash, T., et al.: Texture bias of cnns limits few-shot classification performance (2019)
  * [36] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation (2015)
  * [37] Seghier, M.L., Ramlackhansingh, A., Crinion, J., et al.: Lesion identification using unified segmentation-normalisation models and fuzzy clustering. NeuroImage 41(4), 1253–1266 (Jul 2008)
  * [38] Touvron, H., Cord, M., Sablayrolles, A., et al.: Going deeper with image transformers (2021)
  * [39] Tuli, S., Dasgupta, I., Grant, E., et al.: Are convolutional neural networks or transformers more like human vision? (2021)
  * [40] Tustison, N.J., Avants, B.B., Cook, P.A., et al.: N4itk: Improved n3 bias correction. IEEE Transactions on Medical Imaging 29(6), 1310–1320 (Jun 2010)
  * [41] Tustison, N.J., Shrinidhi, K.L., Wintermark, M., et al.: Optimal symmetric multimodal templates and concatenated random forests for supervised brain tumor segmentation (simplified) with ANTsR. Neuroinformatics 13(2), 209–225 (Nov 2014)
  * [42] Wang, W., Xie, E., Li, X., et al.: Pvtv2: Improved baselines with pyramid vision transformer. CoRR abs/2106.13797 (2021)
  * [43] Xie, E., Wang, W., Yu, Z., , et al.: Segformer: Simple and efficient design for semantic segmentation with transformers (2021)
  * [44] Yu, W., Luo, M., Zhou, P., et al.: Metaformer is actually what you need for vision (2021)
  * [45] Zhang, J., He, T., Sra, S., et al.: Why gradient clipping accelerates training: A theoretical justification for adaptivity (2019)