The authors are with the Department of Engineering Cybernetics, Norwegian University of Science and Technology (NTNU), O. S. Bragstads plass 2D, NO-7491 Trondheim, Norway E-mails:{andreas.b.martinsen, anastasios.lekkas<EMAIL_ADDRESS>
Missing 'biblatex' package
The bibliography requires the 'biblatex' package.

booktitleSensing and Control for Autonomous Vehicles
titleModel-based path planning

titleEnergy-Optimized Path Planning for Autonomous Ferries

Cambridge university press

titlePlanning algorithms

booktitleProceedings of the IJCAI workshop on reasoning, representation, and learning in computer games
titlePath planning in triangulations

journaltitleIEEE transactions on Systems Science and Cybernetics
titleA formal basis for the heuristic determination of minimum cost paths

journaltitleControl Engineering Practice
titleA Voronoi-diagram-based dynamic path-planning system for underactuated marine vessels

journaltitleIEEE transactions on systems, man, and cybernetics
titleNumerical potential field techniques for robot path planning

Norsk Forening for Automatisering (NFA)

journaltitleModeling, Identification and Control
titleContinuous-Curvature Path Generation Using Fermat's Spiral

booktitleNonholonomic Motion Planning
titlePlanning smooth paths for mobile robots

journaltitleIEEE transactions on robotics and automation
titlePrimitives for smoothing mobile robot trajectories

booktitleAIAA Guidance, Navigation, and Control Conference and Exhibit
titleSpline based path planning for unmanned air vehicles

MIT Press

journaltitleRobotics: Science and Systems VII
titleCollision-free and curvature-continuous path smoothing in cluttered environments

journaltitleIEEE Transactions on Control Systems Technology
titleIntegral LOS path following for curved paths based on a monotone cubic Hermite spline parametrization

journaltitleIEEE Transactions on Control Systems Technology
titlePath planning for autonomous vehicles by trajectory smoothing using motion primitives

journaltitleIEEE transactions on Robotics and Automation
titleProbabilistic roadmaps for path planning in high-dimensional configuration spaces

titleRapidly-exploring random trees: A new tool for path planning

SAGE Publications

journaltitleThe international journal of robotics research
titleRandomized kinodynamic planning

journaltitleRobotics Science and Systems VI
titleIncremental sampling-based algorithms for optimal motion planning

journaltitleIEEE Transactions on Robotics
titleMotion planning using adaptive random walks

booktitleProceedings of the Fifth International Workshop on Robot Motion and Control, 2005. RoMoCo'05.
titleMerging the adaptive random walks planner with the randomized potential field planner

journaltitleAmerican Journal of mathematics
titleOn curves of minimal length with a constraint on average curvature, and with prescribed initial and terminal positions and tangents

Mathematical Sciences Publishers

journaltitlePacific journal of mathematics
titleOptimal paths for a car that goes both forwards and backwards

American Association for the Advancement of Science

titleDynamic programming

booktitleProceedings of the IEEE international conference on neural networks
titleParticle swarm optimization

booktitle1997 IEEE International conference on systems, man, and cybernetics. Computational cybernetics and simulation
titleA discrete binary version of the particle swarm algorithm

journaltitleIFAC Proceedings Volumes
titleA multiple shooting algorithm for direct solution of optimal control problems

journaltitleJournal of guidance, control, and dynamics
titleDirect trajectory optimization using nonlinear programming and collocation

journaltitleJournal of Guidance, Control, and Dynamics
titleDirect trajectory optimization by a Chebyshev pseudospectral method

booktitle2017 IEEE Intelligent Vehicles Symposium (IV)
titleLattice-based motion planning for a general 2-trailer system

journaltitlearXiv preprint arXiv:2005.02674
titleAn Optimization-Based Motion Planner for Autonomous Maneuvering of Marine Vessels in Complex Environments

titleWarm-started optimized trajectory planning for ASVs

journaltitleIEEE Transactions on Robotics
titleChance-constrained optimal path planning with obstacles

title$\text{CIAO}^* $: MPC-based Safe Motion Planning in Predictable Dynamic Environments

titleAutonomous docking using direct optimal control

booktitle2008 7th World Congress on Intelligent Control and Automation
titlePath planning based on constrained delaunay triangulation

journaltitleIEEE Transactions on Control Systems Technology
titleSpline-based motion planning for autonomous guided vehicles in a dynamic environment

titleConstrained delaunay triangulations

titleComputational geometry: an introduction

Taylor & Francis

journaltitleInternational Journal of Control
titleOptimal control via collocation and non-linear programming

booktitle2019 International Symposium ELMAR
titlePath Planning for Mobile Robot using Dubins-curve based RRT Algorithm with Differential Constraints

Sage Publications Sage UK: London, England

journaltitleThe international journal of robotics research
titleSampling-based algorithms for optimal motion planning

John Wiley & Sons

titleHandbook of marine craft hydrodynamics and motion control


11institutetext: Chongqing University of Technology, Chongqing, China
22institutetext: University of Electronic Science and Technology of China,
Chengdu, China
33institutetext: Capital Normal University, Beijing, China 44institutetext:
University of Liverpool, Liverpool, United Kingdom
44email:
m17692405449@gmail.com;zengboheng<EMAIL_ADDRESS>
{shipengbo;chenqingzi;liujirui}<EMAIL_ADDRESS>

# LKCA: Large Kernel Convolutional Attention

Chenghao Li 11 Boheng Zeng 22 Yi Lu 3344
Pengbo Shi 11 Qingzi Chen 11 Jirui Liu 11 Lingyun Zhu 11

###### Abstract

We revisit the relationship between attention mechanisms and large kernel
ConvNets in visual transformers and propose a new spatial attention named
Large Kernel Convolutional Attention (LKCA). It simplifies the attention
operation by replacing it with a single large kernel convolution. LKCA
combines the advantages of convolutional neural networks and visual
transformers, possessing a large receptive field, locality, and parameter
sharing. We explained the superiority of LKCA from both convolution and
attention perspectives, providing equivalent code implementations for each
view. Experiments confirm that LKCA implemented from both the convolutional
and attention perspectives exhibit equivalent performance. We extensively
experimented with the LKCA variant of ViT in both classification and
segmentation tasks. The experiments demonstrated that LKCA exhibits
competitive performance in visual tasks. Our code will be made publicly
available at https://github.com/CatworldLee/LKCA.

###### Keywords:

ConvNet, Vision Transformer, Large Kernel CNN, Attention Mechanism

##  1 Introduction

Convolutional Neural Networks (CNNs) [1, 2] have been a common choice for
visual encoders in modern visual models. However, the emergence of Vision
Transformers (ViTs) [3] has expanded the options for the backbone network of
models. ViTs have demonstrated excellent performance in various visual tasks
[4, 5, 6, 7].

Figure 1: Two views to interpret LKCA. The Large Kernel Convolutional
Attention (LKCA) can be understood From the perspective of convolution on the
left and attention on the right. The effects of the two approaches are
equivalent.

CNN and ViT each possess unique advantages. CNN is fundamentally translation
equivariance and shifting window invariance [8], providing robustness to
subtle changes in object positions and orientations in images, leading to
significant success in various computer vision tasks. In contrast, ViT
effectively captures global context information through its self-attention
mechanism [9], enabling the modeling of long-range dependencies [10] and
contextual relationships [11]. Despite the significant progress made by CNN
and ViT over the years, they still have some drawbacks and challenges. CNN may
face difficulties in capturing remote contextual information [12], as its
primary focus is on local feature extraction, potentially limiting its ability
to understand global context [13]. This limitation can result in performance
degradation in tasks that require global context information, such as scene
understanding, image captioning, or fine-grained recognition. On the other
hand, ViT lacks the inherent spatial inductive bias that CNN possesses [14],
which is specifically designed to leverage the spatial locality of images.

In order to integrate the strengths of both CNN and ViT, we propose a Large
Kernel Convolutional Attention (LKCA) mechanism. The implementation of LKCA is
straightforward, replacing the self-attention operation with a single large
kernel convolution operation. LKCA not only inherits the translational
invariance and spatial inductive bias advantages of ConvNets but also
leverages the global modeling and long-range dependency capabilities of ViT.
Deconstructing LKCA from the perspective of attention can be considered as a
novel approach of spatial attention. The approach closest to our work is
Visual Attention Network (VAN) [15], where they replace the multi-head self-
attention mechanism with three types of small kernel convolutions: DW-Conv,
DW-DConv, and 1$\times$1-Conv. In contrast, our method is simpler, directly
employing one large kernel convolution [12] to replace MHSA.

We extensively validated the effectiveness of LKCA in two visual tasks: image
classification and semantic segmentation. In image classification, we
conducted experiments on classic datasets including CIFAR10, CIFAR100, SVHN,
and Tiny-ImageNet, demonstrating the efficacy of LKCA across these datasets.
Notably, in the case of Tiny-ImageNet, we performed comparative experiments
against prominent variants of Vision Transformers ViT and achieved compelling
and competitive results. For semantic segmentation, we selected the ADE20K
dataset and conducted ablation experiments under different framework
configurations. Through comparisons with multiple backbone networks, we
verified the superior performance of LKCA in terms of both parameter
efficiency and accuracy. This further underscores the excellence of LKCA in
addressing semantic segmentation tasks.

##  2 Related Work

###  2.1 Visual Attentions

Attention mechanisms [16, 17, 18, 19, 20, 21] are inspired by the way humans
naturally focus on important elements in their subconscious. Visual attention
mechanisms [20, 21] can be viewed as a dynamic weighting process based on
input image features. Visual attention methods can be broadly categorized into
four types: channel attention [22, 23, 24, 25], spatial attention [26, 27,
28], temporal attention [29, 30], and branch attention [31, 32, 33], as well
as other mixed combinations [34, 35, 36, 37, 38, 39, 40]. Non-Local Network
[41] successfully employs self-attention for the first time to simulate non-
local relationships in computer vision. CBAM [42] infers attention weights
sequentially along both spatial and channel dimensions and can seamlessly
integrate into any CNN architecture. ECA [22] module introduces a non-
reductive local cross-channel interaction strategy, which efficiently enhances
the attention module’s effectiveness through the effective implementation of
1D convolution. EncNet [23] significantly improves the spatial resolution of
pixel-level semantic segmentation labels by introducing a context encoding
module. OCNet [26] combines an interleaved sparse self-attention scheme with a
traditional multi-scale context approach, effectively modeling dense
relationships between pixels and providing richer contextual information.
SAGAN [27] employs spatial attention to model long-range dependencies in image
generation tasks. SCAN [29] achieves significantly superior performance in
Video Person Re-Identification by refining both intra-sequence and inter-
sequence feature representations through the introduction of temporal
attention to video sequences. SKNet [31] enhances the model’s capability to
capture objects at different scales by introducing branch attention to merge
multiple branches with different kernel sizes.

###  2.2 Large Kernel ConvNets

Compared to small-kernel CNNs [1, 43], large-kernel CNNs [44, 45, 46, 47, 48,
49, 50, 12, 15] possess a broader effective receptive field and stronger shape
preferences. However, older models like Inception [44, 45, 46] fell out of
favor after VGG-Net [47], leading to a decline in the popularity of large-
kernel models. A notable research outcome is the Global Convolutional Network
(GCN) [48], which employs very large convolutional kernels of 1$\times$K and
K$\times$1 to optimize semantic segmentation tasks. However, there are reports
suggesting that large kernels may compromise ImageNet performance. On the
other hand, the Local Relation Network (LRNet) [49] introduces a spatial
aggregation operator (LRLayer) as a replacement for standard convolution,
which can be seen as a dynamic convolution. ConvMixer [50], on the contrary,
uses convolutional kernels as large as 9$\times$9 to replace the ”mixer”
components of ViT or MLP. Inspired by ViT, RepLKNet [12] utilizes a large
31$\times$31 convolutional kernel, and its outstanding performance is
primarily attributed to the effective receptive fields constructed through
large kernels [65]. It has been demonstrated that RepLKNet better exploits
shape information compared to traditional CNNs. The Visual Attention Network
(VAN) [15] employs three types of convolutions—DW-Conv, DW-DConv, and
1$\times$1 Conv—to expand the receptive field of convolutions and integrate
them into the attention mechanism.

###  2.3 Large ERF ViTs

The strength of ViTs lies in its large Effective Receptive Fields (ERFs).
Transformer [51] was initially applied in the field of Natural Language
Processing, and the first work to migrate its powerful representation
capabilities to the Computer Vision domain was ViT [3]. Following the paradigm
of ViT, a series of ViT variants [52, 53, 54, 55, 56, 57, 58, 59, 60, 61] have
been proposed to enhance the performance of visual tasks. Visual Transformers
excel at capturing long-range dependencies between patches, representing large
receptive fields. However, ViT lacks the dependence on short-term
relationships for 2D image local dependencies that CNNs inherently possess.
Researchers have primarily focused on improving ViT’s modeling capability for
local information. Transformer in Transformer (TNT) [52] enhances the feature
representation capability of local patches through internal attention. Swin
Transformer [53, 54] addresses local modeling challenges in adapting
transformers from language to vision by introducing a hierarchical
architecture with shift windowing. Twins [55] and CAT [56] alternately perform
local and global attention layers layer-wise. Shuffle Transformer [57, 58]
efficiently establishes inter-window connections through spatial shuffling
with deep convolution and enhances short-term dependency modeling with
neighboring window connections. RegionViT [59] introduces regions and local
tokens with different patch sizes, enhancing local modeling through a region-
to-local attention mechanism.

##  3 Method

###  3.1 Preliminary

####  3.1.1 Review of Large Kernel Convolution.

Ding et al. [12] revisit the paradigm of large kernel design in modern CNNs,
drawing inspiration from the ViT. Ding et al. assert that employing a few
large convolutional kernels instead of multiple smaller ones constitutes a
more powerful design paradigm. Building upon this notion, the author
introduces a novel CNN architecture named RepLKNet [12], featuring kernel
sizes as large as 31$\times$31\. In comparison to commonly used 3$\times$3
kernels, RepLKNet achieves performance results equivalent to or better than
the Swin Transformer [53], with lower latency. The difference between small
kernel convolution and large kernel convolution is shown in Figure 2

####  3.1.2 Comparision of LKC and MHSA.

Convolution possesses a natural advantage wherein parameters are shared based
on geographical location. Specifically, identical parameters of the same
convolutional kernel act on every position within the receptive field during
the sliding window process. This introduces equivariance and translation
invariance to the convolutional operation. However, in Vision Transformer, the
attention scores generated by the self-attention mechanism do not explicitly
exhibit positional invariance. Attention weights are determined by the feature
vectors themselves and the introduced positional encodings. While this
enhances the model’s fitting capability, it significantly increases the
complexity of training. Introducing a mechanism for sharing parameters in the
attention mechanism of ViT may prove beneficial in alleviating training
challenges.

Figure 2: Illustration of the difference between small kernel convolution and
large kernel convolution by constructing a 5x5 feature map, a 3x3 convolution
kernel smaller than the feature map, and a 7x7 convolution kernel larger than
the feature map. The distinction between small kernel convolution and large
kernel convolution lies in the fact that, in small kernel convolution, all
parameters of the kernel are involved in each correlation operation, while
only a subset of feature map parameters participates in the computation. In
the case of large kernel convolution, during each correlation operation, all
parameters of the feature map are involved, and only a subset of the
convolution kernel parameters participates in the computation.

###  3.2 Large Kernel Convolution Attention (LKCA)

####  3.2.1 Shared Weight Position Operation.

Firstly, we initialize a square matrix of size $N^{\frac{1}{2}}\times 2-1$ as
learnable parameters, denoted as the shared parameter weights, where $N$
represents the number of patches, akin to the initialization of convolutional
kernels. Subsequently, we initialize a window of size $N^{\frac{1}{2}}$ and
traverse this window as it slides over the shared parameter weights matrix
from the bottom-right corner to the top-right corner. At each sliding step,
the matrix within the window is flattened row-wise into a row vector of size
$(1,N)$ and stored. This process is iterated as the window slides,
concatenating the row vectors obtained from each step. In total, the window
slides $N$ times. Consequently, the concatenated matrix assumes a shape of
$(N,N)$, aligning with the shape of the attention weight matrix. This
concatenated matrix serves as the new attention score, undergoing matrix
multiplication with the linearly mapped Value matrix. This step is referred to
as the Shared Weight Matrix Attention. Listing 1.1 presents pseudocode for
implementing LKCA from the attention perspective.

⬇

class Attention_ViT(nn.Module):

def __init__(num_patches):

super().__init__()

n = num_patches

H, W = n**0.5

weight = nn.Parameter(torch.zeros(H*2-1, W*2-1))

def forward(X):

b, n, d = X.shape

V = nn.Linear(X)

for i in range(H):

for j in range(W):

start_row = H-1-i

start_col = W-1-j

window = weight[start_row:start_row+H, start_col:start_col+W]

window_vector = window.flatten()

attn_score[i*W+j, :] = window_vector

Out = attn_score @ V

return Out

Listing 1: Pseudocode from attention perspective

####  3.2.2 From the Perspectives of Convolution: Implementing LKCA.

Large Kernel Convolutional Attention (LKCA) can be expressed equivalently as
the attention operation with shared parameters described above in terms of
code and mathematical implementation. Listing 1.2 showcases pseudocode for
implementing LKCA from the perspective of convolution. The convolutional
kernel is a learnable weight matrix of size $N^{\frac{1}{2}}\times 2-1$, and
the input feature map is a matrix $V$ of size $N^{\frac{1}{2}}$ with a channel
count equal to the hidden dimension. After zero-padding of size
$N^{\frac{1}{2}}-1$, the convolution operation is applied to the two matrices.

$\mathbf{LKCA(x)}\mathbf{=}\mathbf{Conv2d}(\mathbf{MLP(x)},\mathbf{LargeKernel})$

⬇

class Attention_CNN(nn.Module):

def __init__(num_patches):

super().__init__()

n = num_patches

H, W = n**0.5

weight = nn.Parameter(torch.zeros(H*2-1, W*2-1))

def forward(X):

b, n, d = X.shape

V= nn.Linear(X)

V = rearrange(V, ’b (H W) d -> (b d) H W’)

Out = F.conv2d(V, weight, zero_padding=(H-1, W-1))

Out = rearrange(Out, ’(b d) H W -> b (H W) d’)

return Out

Listing 2: Pseudocode from convolution perspective

####  3.2.3 Overall Architecture.

The incorporation of large-kernel convolution attention enhances the model’s
ability to capture positional information. However, it simultaneously weakens
the relationships between patches and within individual patches. Therefore, we
intersperse the large-kernel convolution attention modules within the original
ViT architecture while retaining the self-attention modules.

The model overview is represented by the following formulas. An image
$x\in\mathbb{R}^{H\times W\times C}$, where $(H,W)$ is the resolution of the
original image, and $C$ is the number of channels. A series of flattened 2D
patches $x_{p}\in\mathbb{R}^{N\times(P^{2}\cdot C)}$, where $(P,P)$ is the
resolution of each image block, and $N=\frac{HW}{P^{2}}$ is the number of
generated blocks. The patches are linearly projected by weight matrix
$\mathbf{E}$ to a $D$-dimensional space and added with position embedding
$\mathbf{E}_{pos}$. Differing from the original ViT, we discard the use of the
[class] token. Instead of extracting the [class] token from the final linear
layer as classification information, we adopt the mean of the last layer’s
feature vectors for classification. Additionally, we insert the Large Kernel
Convolution Attention (LKCA) module before the Multi-Layer Perceptron (MLP)
module and apply Layer Normalization (LN) before each module.

$\displaystyle\mathbf{z}_{0}$
$\displaystyle=\left[\mathbf{x}_{p}^{1}\mathbf{E};\mathbf{x}_{p}^{2}\mathbf{E};\cdots;\mathbf{x}_{p}^{N}\mathbf{E}\right]+\mathbf{E}_{pos},$
$\displaystyle\mathbf{E}\in\mathbb{R}^{\left(P^{2}\cdot C\right)\times
D},\mathbf{E}_{pos}\in\mathbb{R}^{N\times D}$ (1)
$\displaystyle\mathbf{z}_{\ell}^{\prime}$
$\displaystyle=\operatorname{LKCA}\left(\operatorname{LN}\left(\mathbf{z}_{\ell-1}\right)\right)+\mathbf{z}_{\ell-1},$
$\displaystyle\ell=1\ldots L$ (2) $\displaystyle\mathbf{z}_{\ell}$
$\displaystyle=\operatorname{MLP}\left(\operatorname{LN}\left(\mathbf{z}_{\ell}^{\prime}\right)\right)+\mathbf{z}_{\ell}^{\prime},$
$\displaystyle\ell=1\ldots L$ (3) $\displaystyle\mathbf{y}$
$\displaystyle=\operatorname{LN}\left(\overline{\mathbf{z}_{L}}\right)$ (4)

##  4 Experiments

###  4.1 Image Classification

Experimental Setting. In our experimental setup, we conducted extensive image
classification experiments on four prominent datasets: CIFAR-10, CIFAR-100,
SVHN, and Tiny-ImageNet. All four datasets are characterized by low
resolution, with image sizes of 32 for CIFAR-10, CIFAR-100, and SVHN, and 64
for Tiny-ImageNet. The training regime was consistently maintained across all
experiments, conducted on a single RTX 2080ti GPU. The training spanned 100
epochs, employing 4 data loading workers, a batch size of 128. For optimizer,
we choose AdamW [62]. The default learning rate was set to 0.001, with a
weight decay of 0.05. We utilized cosine annealing with a warm-up period of 10
epochs. During the training phase, we employed Stochastic Depth [63], Label
Smoothing [45], and Random Erasing [64] with a probability of 0.25, a maximum
erasing area of 0.4, and an aspect of erasing area of 0.3 as regularization
strategies to enhance the robustness and generalization capabilities of our
model. Regarding data augmentation, we employed various techniques.
Specifically, we applied RandomHorizontalFlip, RandomCrop [65], Mixup [66],
and Cutmix [67]. Additionally, we incorporated Autoaugmentation [68],
utilizing the CIFAR10 Policy for CIFAR-10 and CIFAR-100, the SVHN Policy for
SVHN, and the ImageNet Policy for Tiny-ImageNet.

Comparison of ViT and LKCA. Initially, we conducted extensive experiments on
CIFAR-10, CIFAR-100, and Tiny-ImageNet to compare the performance of the
original ViT with an LKCA-modified version of ViT having a similar order of
magnitude in terms of parameters. The purpose of these experiments was to
validate the effectiveness of LKCA across various parameter magnitudes and
datasets. The parameter sizes of both the ViT and the LKCA-modified ViT varied
across a range of magnitudes, spanning from small-scale models with 0.5
million parameters to large-scale models with 10 million parameters.
Specifically, the parameter magnitudes were explored at levels of 0.5M, 1M,
2M, 4M, and 8M. It is important to note that the benchmark representing the
model size across all variants is fixed at 2.69 million parameters, and it is
independent of any subjective categorization as either large or small. This
standardized model size serves as the reference point for assessing the
performance of the different variants, ensuring a consistent comparison across
the experiment.

Table 1: Performance Comparison of Vision Transformer and LKCA Variants on
Image Classification Tasks on CIFAR-10 (% Top-1 accuracy)

Model | Acc | # Params | Flops
---|---|---|---
ViT | 93.36 | 2.69M | 174.25M
ViT-LKCA | 94.11 (+0.75) | 2.69M | 170.81M
VIT/0.5M | 87.83 | 0.48M | 30.58M
ViT/0.5M-LKCA | 90.94 (+3.11) | 0.50M | 31.34M
VIT/1M | 91.16 | 1.06M | 68.36M
ViT/1M-LKCA | 92.49 (+1.33) | 1.03M | 64.82M
VIT/2M | 92.32 | 1.96M | 126.09M
ViT/2M-LKCA | 93.55 (+1.23) | 2.03M | 128.66M
VIT/4M | 93.64 | 3.95M | 255.47M
ViT/4M-LKCA | 94.71 (+1.07) | 4.05M | 257.43M
VIT/8M | 94.46 | 7.85M | 508.72M
ViT/8M-LKCA | 94.88 (+0.42) | 7.98M | 506.00M

Table 1 presents a performance comparison of ViT and its LKCA variants on the
CIFAR-10 image classification task, along with their performance across
different orders of magnitude in terms of parameters. The table includes the
Top-1 accuracy (Acc), the number of parameters (# Params), and Floating Point
Operations (Flops) for each model. ViT and ViT-LKCA: The ViT model achieves a
Top-1 accuracy of 93.36%, with 2.69M parameters and 174.25M Flops. The ViT-
LKCA outperforms with a Top-1 accuracy of 94.11% (an improvement of 0.75
percentage points) and has 2.69M parameters and 170.81M Flops. Small-scale
model (0.5M parameter magnitude): The ViT/0.5M model has a Top-1 accuracy of
87.83%, 0.48M parameters, and 30.58M Flops. The ViT/0.5M-LKCA, with LKCA
improvement, significantly improves accuracy to 90.94% (an increase of 3.11
percentage points) with 0.50M parameters and 31.34M Flops. Medium-scale models
(1M, 2M, 4M parameter magnitudes): For ViT and ViT-LKCA models with 1M, 2M,
and 4M parameters, LKCA consistently yields higher Top-1 accuracy. For
instance, ViT/1M-LKCA achieves a 1.33 percentage point improvement, reaching
an accuracy of 92.49%. Large-scale model (8M parameter magnitude): For ViT and
ViT-LKCA models with 8M parameters, the performance gap is narrower.
ViT/8M-LKCA slightly improves Top-1 accuracy by 0.42 percentage points,
reaching 94.88%.

Table 2: Performance Comparison of Vision Transformer and LKCA Variants on
Image Classification Tasks on CIFAR-100 (% Top-1 accuracy)

Model | Acc | # Params | Flops
---|---|---|---
ViT | 72.65 | 2.69M | 174.25M
ViT-LKCA | 76.50 (+3.85) | 2.69M | 170.81M
VIT/0.5M | 64.68 | 0.48M | 30.58M
ViT/0.5M-LKCA | 69.35 (+4.67) | 0.50M | 31.34M
VIT/1M | 70.86 | 1.06M | 68.36M
ViT/1M-LKCA | 73.40 (+2.54) | 1.03M | 64.82M
VIT/2M | 71.17 | 1.96M | 126.09M
ViT/2M-LKCA | 75.94 (+4.77) | 2.03M | 128.66M
VIT/4M | 72.82 | 3.95M | 255.47M
ViT/4M-LKCA | 76.95 (+4.13) | 4.05M | 257.43M
VIT/8M | 73.39 | 7.85M | 508.72M
ViT/8M-LKCA | 78.57 (+5.18) | 7.98M | 506.00M

Table 2 illustrates Acomparison between ViT and LKCA in the context of the
CIFAR-100. ViT and ViT-LKCA: The base ViT model achieves a Top-1 accuracy of
72.65% with 2.69M parameters and 174.25M Flops. In contrast, ViT-LKCA exhibits
superior performance, attaining a Top-1 accuracy of 76.50% (an increase of
3.85 percentage points) with the same 2.69M parameters and 170.81M Flops.
Small-scale model (0.5M parameter magnitude): The ViT/0.5M model achieves a
Top-1 accuracy of 64.68% with 0.48M parameters and 30.58M Flops. The
ViT/0.5M-LKCA, featuring LKCA enhancements, significantly improves accuracy to
69.35% (a notable increase of 4.67 percentage points) with 0.50M parameters
and 31.34M Flops. Medium-scale models (1M, 2M, 4M parameter magnitudes): ViT
and ViT-LKCA models with 1M, 2M, and 4M parameters consistently benefit from
LKCA improvements. For instance, ViT/2M-LKCA achieves a noteworthy 4.77
percentage point improvement and reaches an accuracy of 75.94%. Large-scale
model (8M parameter magnitude): For ViT and ViT-LKCA models with 8M
parameters, the performance difference is more subtle. ViT/8M-LKCA exhibits a
huge improvement of 5.18 percentage points, reaching a Top-1 accuracy of
78.57%.

Table 3: Performance Comparison of Vision Transformer and LKCA Variants on
Image Classification Tasks on Tiny-ImageNet (% Top-1 accuracy)

Model | Acc | # Params | Flops
---|---|---|---
ViT | 55.74 | 2.69M | 174.25M
ViT-LKCA | 60.95 (+5.21) | 2.69M | 170.81M
VIT/0.5M | 46.53 | 0.48M | 30.58M
ViT/0.5M-LKCA | 50.59 (+4.06) | 0.50M | 31.34M
VIT/1M | 53.46 | 1.06M | 68.36M
ViT/1M-LKCA | 57.29 (+3.83) | 1.03M | 64.82M
VIT/2M | 54.58 | 1.96M | 126.09M
ViT/2M-LKCA | 60.70 (+6.12) | 2.03M | 128.66M
VIT/4M | 56.16 | 3.95M | 255.47M
ViT/4M-LKCA | 62.30 (+6.14) | 4.05M | 257.43M
VIT/8M | 57.39 | 7.85M | 508.72M
ViT/8M-LKCA | 63.38 (+5.99) | 7.98M | 506.00M

Table 3 provides the comparison on Tiny-ImageNet. ViT and ViT-LKCA: The
baseline ViT model achieves a Top-1 accuracy of 55.74% with 2.69M parameters
and 174.25M Flops. In contrast, the ViT-LKCA model outperforms significantly
with a Top-1 accuracy of 60.95% (an increase of 5.21 percentage points) and
has the same 2.69M parameters and 170.81M Flops. Small-scale model (0.5M
parameter magnitude): The ViT/0.5M model achieves a Top-1 accuracy of 46.53%
with 0.48M parameters and 30.58M Flops. The ViT/0.5M-LKCA, incorporating LKCA
enhancements, significantly improves accuracy to 50.59% (an increase of 4.06
percentage points) with 0.50M parameters and 31.34M Flops. Medium-scale models
(1M, 2M, 4M parameter magnitudes): With ViT/4M-LKCA achieving a remarkable
6.14 percentage point improvement and reaching an accuracy of 62.30%. Large-
scale model (8M parameter magnitude): For ViT and ViT-LKCA models with 8M
parameters, ViT/8M-LKCA exhibits a significant improvement of 5.99 percentage
points, achieving a Top-1 accuracy of 63.38%.

In summary, the LKCA improvement demonstrates significant performance gains
across various parameter magnitudes, confirming its effectiveness on ViT
models.

Comparison of ViT Variants and LKCA. Next, we compared LKCA with other
mainstream variants of Vision Transformer.

First, we provide a comprehensive comparison of LKCA with various mainstream
ViT variants on different datasets, including CIFAR-10, CIFAR-100, and SVHN.
Each model’s Top-1 accuracy, number of parameters, and Floating Point
Operations are presented. In the deep ViT, localized ViT, and lightweight ViT
categories, we selected representative models such as Swin-T [53], CaiT-T [69]
and MobileViTv2 [70], to compare with standard ViT [3] and our proposed LKCA.

Table 4: Comparison of LKCA with Several Mainstream ViT Variants on Different
Datasets (% Top-1 accuracy)

Model | CIFAR10 | CIFAR100 | SVHN | # Params | Flops
---|---|---|---|---|---
ViT-Lite [3] | 91.16 | 70.86 | 97.53 | 1.06M | 68.36M
Swin-T [53] | 91.21 | 70.37 | 96.40 | 1.01M | 38.26M
CaiT-T [69] | 91.99 | 70.65 | 97.53 | 1.00M | 61.16M
MobileViTv2-T [70] | 91.46 | 67.88 | 96.60 | 1.00M | 7.43M
LKCA-T | 92.49 (Ours) | 73.40 (Ours) | 97.15 (Ours) | 1.03M | 64.82M
ViT-Small [71] | 93.36 | 72.65 | 97.90 | 2.69M | 174.25M
Swin-S [53] | 93.99 | 75.34 | 97.59 | 2.87M | 94.72M
CaiT-S [69] | 94.02 | 74.70 | 97.79 | 2.74M | 163.55M
MobileViTv2-S [70] | 93.48 | 71.70 | 96.98 | 3.92M | 24.55M
LKCA-B | 94.11 (Ours) | 76.50 (Ours) | 97.68 (Ours) | 2.69M | 170.81M

From the information in the table 4. In the context of small-scale models,
Swin-T and CaiT-T perform similarly in terms of Top-1 accuracy, while
MobileViTv2 lags slightly behind but possesses a significant advantage in
computational efficiency. Compared to other small-scale models, LKCA-T
achieves a higher Top-1 accuracy. Moving to large-scale models, it is a common
trend that they generally outperform their small-scale counterparts. LKCA-B
stands out by significantly surpassing ViT and MobileViTv2 in Top-1 accuracy,
and it slightly outperforms CaiT-S and Swin-S. Overall, in CIFAR10 and
CIFAR100, LKCA demonstrates higher Top-1 accuracy compared to other models.
However, there is no clear advantage observed in the SVHN dataset.

Finally, we have validated the performance of various ViT variants and the
LKCA model on the Tiny-ImageNet dataset, while also providing details on the
models’ parameter counts and computational complexities in Table 5.

In the category of small-scale models, LKCA-T achieves a Top-1 accuracy of
57.29%, demonstrating excellent performance compared to other models. Its
performance surpasses that of several common small-scale models. Moving to the
medium-scale models, LKCA-B achieves an impressive Top-1 accuracy of 60.95%,
once again showcasing outstanding performance. In the realm of large-scale
models, LKCA-L attains a Top-1 accuracy of 63.43%, exhibiting a significant
advantage in accuracy compared to models of the same scale. Overall, across
various scales, LKCA demonstrates outstanding Top-1 accuracy on the Tiny-
ImageNet dataset compared to other competitors.

Table 5: Comparison of Various ViT Variants with LKCA on Tiny-ImageNet Dataset
(% Top-1 accuracy)

Model | Tiny-ImageNet | # Params | Flops
---|---|---|---
T2T-T [72] | 53.92 | 1.07M | 78.23M
RvT-T [73] | 50.65 | 1.07M | 57.59M
Swin-T [53] | 54.93 | 1.01M | 38.26M
CaiT-T [69] | 54.76 | 1.00M | 61.16M
XCiT-T [74] | 56.78 | 0.95M | 51.44M
ViT-Lite [3] | 53.46 | 1.06M | 68.36M
DeepViT-T [60] | 34.64 | 0.99M | 62.96M
RegionViT-T [59] | 54.32 | 0.97M | 29.38M
CrossViT-T [75] | 47.03 | 1.04M | 57.59M
LKCA-T | 57.29 (Ours) | 1.03M | 64.82M
T2T-S [72] | 41.25 | 2.56M | 52.96M
RvT-S [73] | 55.51 | 2.70M | 145.07M
Swin-S [53] | 58.61 | 2.87M | 94.72M
CaiT-S [69] | 59.21 | 2.74M | 163.55M
XCiT-S [74] | 60.09 | 2.81M | 157.54M
ViT-Small [3] | 55.74 | 2.69M | 174.25M
DeepViT-S [60] | 44.45 | 2.53M | 162.85M
Twins_SVT-S [55] | 37.13 | 2.76M | 197.00M
RegionViT-S [59] | 53.96 | 2.86M | 53.82M
CrossViT-S [75] | 52.70 | 2.40M | 126.11M
LKCA-B | 60.95 (Ours) | 2.69M | 170.81M
T2T-B [72] | 58.46 | 13.45M | 853.02M
CvT-B [71] | 55.88 | 6.48M | 25.63M
MobileViTv2 [70] | 58.28 | 8.17M | 189.77M
Twins_SVT-B [55] | 49.24 | 9.04M | 308.74M
RegionViT-B [59] | 57.83 | 12.39M | 195.02M
LKCA-L | 63.43 (Ours) | 12.53M | 799.61M

###  4.2 Semantic Segmentation

Experimental Setting. We use MMSegmentation [76] to conduct our experiments on
semantic segmentation. All our experiments are carried out on the ADE20K [77]
dataset, testing two algorithms: UperNet [78] and FPN [79]. We refrain from
using pre-training and instead train the models from scratch on the dataset.
Regarding evaluation metrics, we assess the models based on aAcc and mIoU in
10 high-score categories. In the ablation study of the backbone network, we
examine various popular variants of ViT and CNN and explore their performance
during 0 - 20,000 iterations.

Segment on ADE20K. We conducted experiments on multiple backbone networks on
ADE20K

Table 6: Semantic Segmentation of Vanilla ViT and LKCA on ADE20K Dataset

Backbone | Iteration | Algorithm | aAcc(%) | mIoU(%) | # Params | Flops
---|---|---|---|---|---|---
ViT-B | 2000 | UPerNet | 40.90 | 22.28 | 0.144G | 0.108T
LKCA-B | 2000 | UPerNet | 45.48 | 24.65 | 0.102G | 0.115T
ViT-B | 4000 | UPerNet | 47.41 | 28.08 | 0.144G | 0.108T
LKCA-B | 4000 | UPerNet | 52.67 | 30.00 | 0.102G | 0.115T
ViT-B | 8000 | UPerNet | 53.43 | 31.73 | 0.144G | 0.108T
LKCA-B | 8000 | UPerNet | 50.79 | 33.73 | 0.102G | 0.115T
ViT-B | 16000 | UPerNet | 61.55 | 39.33 | 0.144G | 0.108T
LKCA-B | 16000 | UPerNet | 65.49 | 40.60 | 0.102G | 0.115T

Table 6 provides performance metrics for the UPerNet algorithm using different
backbones (ViT and LKCA), iteration algorithms, and iteration counts (2000,
4000, 8000, 16000). At each iteration count, LKCA-B consistently outperforms
ViT-B in terms of aAcc and mIoU, indicating superior performance of LKCA-B in
the UPerNet task. As the iteration count increases, both models show an
improvement in performance, but at each iteration count, LKCA-B maintains a
leading position. Overall, LKCA-B demonstrates superior performance compared
to ViT-B at different iteration counts, with higher pixel accuracy and mean
Intersection over Union. Additionally, LKCA-B exhibits advantages in terms of
parameter count and computational complexity.

Table 7: Semantic Segmentation of Various ViT Variants and LKCA on ADE20K
Dataset

Backbone | Iteration | Algorithm | aAcc(%) | mIoU(%) | # Params | Flops
---|---|---|---|---|---|---
Vit-S | 10000 | UPerNet | 55.55 | 32.92 | 57.66M | 67.25G
ResNet-50 | 10000 | UPerNet | 58.12 | 35.35 | 64.12M | 0.238T
Twins_PCPVT-S | 10000 | UPerNet | 52.56 | 29.64 | 53.30M | 0.234T
Swin-S | 10000 | UPerNet | 57.18 | 35.30 | 51.90M | 0.231T
LKCA-S | 10000 | UPerNet | 60.16 | 37.16 | 50.58M | 65.94G
Vit-S | 10000 | FPN | 56.56 | 35.84 | 37.23M | 76.14G
Twins_PCPVT-S | 10000 | FPN | 56.74 | 33.09 | 33.43M | 49.74G
LKCA-S | 10000 | FPN | 62.55 | 39.03 | 30.18M | 77.92G
Vit-S | 20000 | UPerNet | 58.51 | 36.61 | 57.66M | 67.25G
ResNet-50 | 20000 | UPerNet | 62.76 | 37.84 | 64.12M | 0.238T
Twins_PCPVT-S | 20000 | UPerNet | 58.91 | 38.00 | 53.30M | 0.234T
Swin-S | 20000 | UPerNet | 58.00 | 37.57 | 51.90M | 0.231T
LKCA-S | 20000 | UPerNet | 64.37 | 40.09 | 50.58M | 65.94G
Vit-S | 20000 | FPN | 63.36 | 41.64 | 37.23M | 76.14G
Twins_PCPVT-S | 20000 | FPN | 58.51 | 36.61 | 33.43M | 49.74G
LKCA-S | 20000 | FPN | 67.21 | 41.81 | 30.18M | 77.92G

Table 7 presents a performance comparison among different backbones (ViT-S,
ResNet-50, Swin-S, Twins PCPVT-S, LKCA-S) under various iteration counts and
across two different task settings (UPerNet and FPN). In the UPerNet task,
LKCA-S consistently outperforms other models at different iteration counts,
exhibiting higher aAcc and mIoU. It achieves superior performance compared to
other models. Similarly, in the FPN task, LKCA-S demonstrates excellent
performance across different iteration counts, showcasing higher aAcc and
mIoU, making it competitive against other models. LKCA-S consistently exhibits
outstanding performance across different tasks and iteration settings, while
maintaining relatively lower computational complexity and parameter counts.

##  5 Conclusion

In this study, we re-examine the relationship between the attention mechanism
in visual transformers and the large kernel convolutional networks, proposing
a new visual attention called Large Kernel Convolutional Attention (LKCA). It
employs a single large kernel convolution to simplify attention operations. In
comparison to the previous approach, Large Kernel Attention (LKA), which uses
multiple small kernel convolutions, our method achieves the same effects but
is more straightforward and simpler. LKCA combines the advantages of
convolutional neural networks and visual transformers, featuring a larger
receptive field, locality, and parameter sharing. We explained the superiority
of LKCA from the perspectives of both convolution and attention, providing
equivalent code implementations for each view. Experimental results
demonstrate that LKCA, implemented from both convolutional and attention
perspectives, exhibits comparable performance. We extensively experimented
with LKCA variants of ViT on multiple benchmark datasets, including CIFAR-10,
CIFAR-100, SVHN, Tiny-ImageNet, and ADE20K, for classification and
segmentation tasks. The experimental outcomes reveal that LKCA demonstrates
competitive performance in visual tasks.

## References

  * [1] He, K., Zhang, X., Ren, S., Sun, J.:  Deep residual learning for image recognition.  In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2016) 770–778
  * [2] Krizhevsky, A., Sutskever, I., Hinton, G.E.:  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems 25 (2012)
  * [3] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.:  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929 (2020)
  * [4] Dai, X., Chen, Y., Xiao, B., Chen, D., Liu, M., Yuan, L., Zhang, L.:  Dynamic head: Unifying object detection heads with attentions.  In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. (2021) 7373–7382
  * [5] Yuan, L., Hou, Q., Jiang, Z., Feng, J., Yan, S.:  Volo: Vision outlooker for visual recognition.  IEEE transactions on pattern analysis and machine intelligence 45(5) (2022) 6575–6586
  * [6] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.:  Masked autoencoders are scalable vision learners.  In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. (2022) 16000–16009
  * [7] Wang, W., Xie, E., Li, X., Fan, D.P., Song, K., Liang, D., Lu, T., Luo, P., Shao, L.:  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  In: Proceedings of the IEEE/CVF international conference on computer vision. (2021) 568–578
  * [8] Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., LeCun, Y.:  Overfeat: Integrated recognition, localization and detection using convolutional networks.  arXiv preprint arXiv:1312.6229 (2013)
  * [9] Cordonnier, J.B., Loukas, A., Jaggi, M.:  On the relationship between self-attention and convolutional layers.  arXiv preprint arXiv:1911.03584 (2019)
  * [10] Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., Dosovitskiy, A.:  Do vision transformers see like convolutional neural networks?  Advances in Neural Information Processing Systems 34 (2021) 12116–12128
  * [11] Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H., Li, Z., Duerig, T.:  Scaling up visual and vision-language representation learning with noisy text supervision.  In: International conference on machine learning, PMLR (2021) 4904–4916
  * [12] Ding, X., Zhang, X., Han, J., Ding, G.:  Scaling up your kernels to 31x31: Revisiting large kernel design in cnns.  In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. (2022) 11963–11975
  * [13] Zhang, R.:  Making convolutional networks shift-invariant again.  In: International conference on machine learning, PMLR (2019) 7324–7334
  * [14] Hassani, A., Walton, S., Shah, N., Abuduweili, A., Li, J., Shi, H.:  Escaping the big data paradigm with compact transformers.  arXiv preprint arXiv:2104.05704 (2021)
  * [15] Guo, M.H., Lu, C.Z., Liu, Z.N., Cheng, M.M., Hu, S.M.:  Visual attention network.  Computational Visual Media 9(4) (2023) 733–752
  * [16] Nadaraya, E.A.:  On estimating regression.  Theory of Probability & Its Applications 9(1) (1964) 141–142
  * [17] Watson, G.S.:  Smooth regression analysis.  Sankhyā: The Indian Journal of Statistics, Series A (1964) 359–372
  * [18] Galassi, A., Lippi, M., Torroni, P.:  Attention in natural language processing.  IEEE transactions on neural networks and learning systems 32(10) (2020) 4291–4308
  * [19] Cho, K., Courville, A., Bengio, Y.:  Describing multimedia content using attention-based encoder-decoder networks.  IEEE Transactions on Multimedia 17(11) (2015) 1875–1886
  * [20] Wang, F., Tax, D.M.:  Survey on the attention based rnn model and its applications in computer vision.  arXiv preprint arXiv:1601.06823 (2016)
  * [21] Guo, M.H., Xu, T.X., Liu, J.J., Liu, Z.N., Jiang, P.T., Mu, T.J., Zhang, S.H., Martin, R.R., Cheng, M.M., Hu, S.M.:  Attention mechanisms in computer vision: A survey.  Computational visual media 8(3) (2022) 331–368
  * [22] Wang, Q., Wu, B., Zhu, P., Li, P., Zuo, W., Hu, Q.:  Eca-net: Efficient channel attention for deep convolutional neural networks.  In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. (2020) 11534–11542
  * [23] Zhang, H., Dana, K., Shi, J., Zhang, Z., Wang, X., Tyagi, A., Agrawal, A.:  Context encoding for semantic segmentation.  In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. (2018) 7151–7160
  * [24] Chen, Z., Li, Y., Bengio, S., Si, S.:  You look twice: Gaternet for dynamic filter selection in cnns.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. (2019) 9172–9180
  * [25] Shi, H., Lin, G., Wang, H., Hung, T.Y., Wang, Z.:  Spsequencenet: Semantic segmentation network on 4d point clouds.  In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. (2020) 4574–4583
  * [26] Yuan, Y., Huang, L., Guo, J., Zhang, C., Chen, X., Wang, J.:  Ocnet: Object context network for scene parsing.  arXiv preprint arXiv:1809.00916 (2018)
  * [27] Zhang, H., Goodfellow, I., Metaxas, D., Odena, A.:  Self-attention generative adversarial networks.  In: International conference on machine learning, PMLR (2019) 7354–7363
  * [28] Mnih, V., Heess, N., Graves, A., et al.:  Recurrent models of visual attention.  Advances in neural information processing systems 27 (2014)
  * [29] Zhang, R., Li, J., Sun, H., Ge, Y., Luo, P., Wang, X., Lin, L.:  Scan: Self-and-collaborative attention network for video person re-identification.  IEEE Transactions on Image Processing 28(10) (2019) 4870–4882
  * [30] Chen, D., Li, H., Xiao, T., Yi, S., Wang, X.:  Video person re-identification with competitive snippet-similarity aggregation and co-attentive snippet embedding.  In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2018) 1169–1178
  * [31] Li, X., Wang, W., Hu, X., Yang, J.:  Selective kernel networks.  In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. (2019) 510–519
  * [32] Zhang, H., Wu, C., Zhang, Z., Zhu, Y., Lin, H., Zhang, Z., Sun, Y., He, T., Mueller, J., Manmatha, R., et al.:  Resnest: Split-attention networks.  In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. (2022) 2736–2746
  * [33] Chen, Y., Dai, X., Liu, M., Chen, D., Yuan, L., Liu, Z.:  Dynamic convolution: Attention over convolution kernels.  In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. (2020) 11030–11039
  * [34] Chen, L., Zhang, H., Xiao, J., Nie, L., Shao, J., Liu, W., Chua, T.S.:  Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning.  In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2017) 5659–5667
  * [35] Park, J., Woo, S., Lee, J.Y., Kweon, I.S.:  Bam: Bottleneck attention module.  arXiv preprint arXiv:1807.06514 (2018)
  * [36] Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., Tang, X.:  Residual attention network for image classification.  In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2017) 3156–3164
  * [37] Misra, D., Nalamada, T., Arasanipalai, A.U., Hou, Q.:  Rotate to attend: Convolutional triplet attention module.  In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. (2021) 3139–3148
  * [38] Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., Lu, H.:  Dual attention network for scene segmentation.  In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. (2019) 3146–3154
  * [39] Li, W., Zhu, X., Gong, S.:  Harmonious attention network for person re-identification.  In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2018) 2285–2294
  * [40] Yan, C., Tu, Y., Wang, X., Zhang, Y., Hao, X., Zhang, Y., Dai, Q.:  Stat: Spatial-temporal attention mechanism for video captioning.  IEEE transactions on multimedia 22(1) (2019) 229–241
  * [41] Wang, X., Girshick, R., Gupta, A., He, K.:  Non-local neural networks.  In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2018) 7794–7803
  * [42] Woo, S., Park, J., Lee, J.Y., Kweon, I.S.:  Cbam: Convolutional block attention module.  In: Proceedings of the European conference on computer vision (ECCV). (2018) 3–19
  * [43] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.:  Densely connected convolutional networks.  In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2017) 4700–4708
  * [44] Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.:  Inception-v4, inception-resnet and the impact of residual connections on learning.  In: Proceedings of the AAAI conference on artificial intelligence. Volume 31. (2017)
  * [45] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.:  Rethinking the inception architecture for computer vision.  In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2016) 2818–2826
  * [46] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.:  Going deeper with convolutions.  In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2015) 1–9
  * [47] Simonyan, K., Zisserman, A.:  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556 (2014)
  * [48] Peng, C., Zhang, X., Yu, G., Luo, G., Sun, J.:  Large kernel matters–improve semantic segmentation by global convolutional network.  In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2017) 4353–4361
  * [49] Hu, H., Zhang, Z., Xie, Z., Lin, S.:  Local relation networks for image recognition.  In: Proceedings of the IEEE/CVF International Conference on Computer Vision. (2019) 3464–3473
  * [50] Trockman, A., Kolter, J.Z.:  Patches are all you need?  arXiv preprint arXiv:2201.09792 (2022)
  * [51] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.:  Attention is all you need.  Advances in neural information processing systems 30 (2017)
  * [52] Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., Wang, Y.:  Transformer in transformer.  Advances in Neural Information Processing Systems 34 (2021) 15908–15919
  * [53] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.:  Swin transformer: Hierarchical vision transformer using shifted windows.  In: Proceedings of the IEEE/CVF international conference on computer vision. (2021) 10012–10022
  * [54] Dong, X., Bao, J., Chen, D., Zhang, W., Yu, N., Yuan, L., Chen, D., Guo, B.:  Cswin transformer: A general vision transformer backbone with cross-shaped windows.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. (2022) 12124–12134
  * [55] Chu, X., Tian, Z., Wang, Y., Zhang, B., Ren, H., Wei, X., Xia, H., Shen, C.:  Twins: Revisiting the design of spatial attention in vision transformers.  Advances in Neural Information Processing Systems 34 (2021) 9355–9366
  * [56] Lin, H., Cheng, X., Wu, X., Shen, D.:  Cat: Cross attention in vision transformer.  In: 2022 IEEE International Conference on Multimedia and Expo (ICME), IEEE (2022) 1–6
  * [57] Huang, Z., Ben, Y., Luo, G., Cheng, P., Yu, G., Fu, B.:  Shuffle transformer: Rethinking spatial shuffle for vision transformer.  arXiv preprint arXiv:2106.03650 (2021)
  * [58] Fang, J., Xie, L., Wang, X., Zhang, X., Liu, W., Tian, Q.:  Msg-transformer: Exchanging local spatial information by manipulating messenger tokens.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. (2022) 12063–12072
  * [59] Chen, C.F., Panda, R., Fan, Q.:  Regionvit: Regional-to-local attention for vision transformers.  arXiv preprint arXiv:2106.02689 (2021)
  * [60] Zhou, D., Kang, B., Jin, X., Yang, L., Lian, X., Jiang, Z., Hou, Q., Feng, J.:  Deepvit: Towards deeper vision transformer.  arXiv preprint arXiv:2103.11886 (2021)
  * [61] Wang, P., Wang, X., Wang, F., Lin, M., Chang, S., Li, H., Jin, R.:  Kvt: k-nn attention for boosting vision transformers.  In: European conference on computer vision, Springer (2022) 285–302
  * [62] Kingma, D.P., Ba, J.:  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980 (2014)
  * [63] Huang, G., Sun, Y., Liu, Z., Sedra, D., Weinberger, K.Q.:  Deep networks with stochastic depth.  In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, Springer (2016) 646–661
  * [64] Zhong, Z., Zheng, L., Kang, G., Li, S., Yang, Y.:  Random erasing data augmentation.  In: Proceedings of the AAAI conference on artificial intelligence. Volume 34. (2020) 13001–13008
  * [65] Zagoruyko, S., Komodakis, N.:  Wide residual networks.  arXiv preprint arXiv:1605.07146 (2016)
  * [66] Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.:  mixup: Beyond empirical risk minimization.  arXiv preprint arXiv:1710.09412 (2017)
  * [67] Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.:  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In: Proceedings of the IEEE/CVF international conference on computer vision. (2019) 6023–6032
  * [68] Cubuk, E.D., Zoph, B., Mane, D., Vasudevan, V., Le, Q.V.:  Autoaugment: Learning augmentation strategies from data.  In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. (2019) 113–123
  * [69] Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., Jégou, H.:  Going deeper with image transformers.  In: Proceedings of the IEEE/CVF international conference on computer vision. (2021) 32–42
  * [70] Mehta, S., Rastegari, M.:  Separable self-attention for mobile vision transformers.  arXiv preprint arXiv:2206.02680 (2022)
  * [71] Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., Zhang, L.:  Cvt: Introducing convolutions to vision transformers.  In: Proceedings of the IEEE/CVF international conference on computer vision. (2021) 22–31
  * [72] Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z.H., Tay, F.E., Feng, J., Yan, S.:  Tokens-to-token vit: Training vision transformers from scratch on imagenet.  In: Proceedings of the IEEE/CVF international conference on computer vision. (2021) 558–567
  * [73] Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., Liu, Y.:  Roformer: Enhanced transformer with rotary position embedding.  Neurocomputing 568 (2024) 127063
  * [74] Ali, A., Touvron, H., Caron, M., Bojanowski, P., Douze, M., Joulin, A., Laptev, I., Neverova, N., Synnaeve, G., Verbeek, J., et al.:  Xcit: Cross-covariance image transformers.  Advances in neural information processing systems 34 (2021) 20014–20027
  * [75] Chen, C.F.R., Fan, Q., Panda, R.:  Crossvit: Cross-attention multi-scale vision transformer for image classification.  In: Proceedings of the IEEE/CVF international conference on computer vision. (2021) 357–366
  * [76] Contributors, M.:  MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark.  https://github.com/open-mmlab/mmsegmentation (2020)
  * [77] Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., Torralba, A.:  Semantic understanding of scenes through the ade20k dataset.  International Journal of Computer Vision 127 (2019) 302–321
  * [78] Xiao, T., Liu, Y., Zhou, B., Jiang, Y., Sun, J.:  Unified perceptual parsing for scene understanding.  In: Proceedings of the European conference on computer vision (ECCV). (2018) 418–434
  * [79] Kirillov, A., Girshick, R., He, K., Dollár, P.:  Panoptic feature pyramid networks.  In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. (2019) 6399–6408