∎

11institutetext: Henry Hengyuan Zhao 22institutetext: Show Lab, National
University of Singapore, Singapore
22email<EMAIL_ADDRESS>33institutetext: Pichao Wang 44institutetext:
Alibaba Group, USA
44email<EMAIL_ADDRESS>55institutetext: Yuyang Zhao 66institutetext:
National University of Singapore, Singapore
66email<EMAIL_ADDRESS>77institutetext: Hao Luo 88institutetext:
Alibaba Group, China
88email<EMAIL_ADDRESS>99institutetext: Fan Wang
1010institutetext: Alibaba Group, USA
1010email<EMAIL_ADDRESS>1111institutetext: Mike Zheng Shou
(Corresponding author)1212institutetext: Show Lab, National University of
Singapore, Singapore
1212email<EMAIL_ADDRESS>
# SCT: A Simple Baseline for Parameter-Efficient Fine-Tuning via Salient
Channels

Henry Hengyuan Zhao  Pichao Wang  Yuyang Zhao  Hao Luo  Fan Wang  Mike Zheng
Shou

(Received: date / Accepted: date)

###### Abstract

Pre-trained vision transformers have strong representation benefits to various
downstream tasks. Recently, many parameter-efficient fine-tuning (PEFT)
methods have been proposed, and their experiments demonstrate that tuning only
1% extra parameters could surpass full fine-tuning in low-data resource
scenarios. However, these methods overlook the task-specific information when
fine-tuning diverse downstream tasks. In this paper, we propose a simple yet
effective method called “Salient Channel Tuning” (SCT) to leverage the task-
specific information by forwarding the model with the task images to select
partial channels in a feature map that enables us to tune only 1/8 channels
leading to significantly lower parameter costs. Experiments outperform full
fine-tuning on 18 out of 19 tasks in the VTAB-1K benchmark by adding only
0.11M parameters of the ViT-B, which is 780$\times$ fewer than its full fine-
tuning counterpart. Furthermore, experiments on domain generalization and few-
shot learning surpass other PEFT methods with lower parameter costs,
demonstrating our proposed tuning technique’s strong capability and
effectiveness in the low-data regime. The code will be available at
https://github.com/showlab/SCT

##  1 Introduction

Figure 1: The comparison of parameters and top-1 accuracy on VTAB-1K benchmark
with different baselines. We only tune 96 channels in 768 channels of
ViT-B/16, obtaining the best results compared with other methods.

Large vision transformers (ViT) have achieved remarkable success in computer
vision tasks (Dosovitskiy et al., 2020; Liu et al., 2021; Yuan et al., 2022;
Zhou et al., 2021a; Carion et al., 2020; Li et al., 2021; Strudel et al.,
2021) by using large-scale training data, such as ImageNet21K and JFT-300M, to
create strong representations. However, downstream recognition tasks often
lack sufficient data to train from scratch. Therefore, transferring knowledge
from pre-trained ViT models can significantly reduce training difficulty and
produce promising results. End-to-end full fine-tuning is commonly used to
inherit these robust representations, but it faces two challenges. First,
large models are prone to overfitting when tuning their massive weights on
small downstream training data. Second, ViT models are too large to store all
weights for each downstream task, making it infeasible to deploy fine-tuned
models on resource-limited devices.

Figure 2: Illustration of Adapter and our SCT. “Downsample” and “Upsample”
represent the channel downsampling and upsampling operations. $D$ represents
the number of channel dimensions.

(a) Visualizations of the extracted feature maps on the Caltech101 dataset at
each transformer layer. Y-axis represents the class indices, and X-axis
represents channel indices, _i.e_., 768 in total. We categorize the total
image features with the class label and then calculate each channel’s $L_{2}$
values. Thus, we obtain the 768 dimension vector for each class and find that
some channels have higher activation values than others in all classes, as the
vertical lines are shown in this figure.

(b) The top-36 selected salient channel indices on the Caltech101 dataset at
each transformer layer _i.e_., Layer-1, Layer-2. Each layer selects different
salient channels. As the layer goes deeper, the salient channels appear more
concentrated. Darker color represents a larger activation value.

(c) We select the salient channels from the same layer on four downstream
tasks. The results suggest that channel bias exists in various tasks. Figure
3: The visualizations of feature maps and the selected salient channel
indices. All the results are obtained by ViT-B/16 pre-trained on ImageNet21K.

To mitigate the above two challenges, tuning a subset of parameters or
adopting an external trainable module to preserve the knowledge learned from
pre-trained models and adapt to the downstream tasks have been proposed. As
for tuning a subset of full parameters, there are two common methods: tuning
the classification head (Mahajan et al., 2018; Jia et al., 2021; Chen et al.,
2021b) and bias term (Cai et al., 2020). Tuning the classification head
freezes the weights of the backbone network and only updates the
classification head, while tuning the bias term means unfreezing the bias term
in the backbone network. Both methods lead to inferior performance.

As for adopting an external trainable module, two lines of Parameter-Efficient
Fine-Tuning (PEFT) methods are proposed. Firstly, adopt learnable prompts for
each downstream task. VPT (Jia et al., 2022) is one representative method. One
drawback of VPT is that its task-specific prompts are obtained through
supervised learning, different from the text prompts used in NLP fields that
users give. Another drawback is that it individually searches the prompt
length for each task. It is not flexible in applying to a new task. Another
direction (Chen et al., 2022b; Houlsby et al., 2019; Jie and Deng, 2022) is to
add an adapter alongside the multi-head self-attention (MHSA) or MLP block in
ViT (Dosovitskiy et al., 2020). Regardless of whether the structure of the
adapter is MLP-like (Houlsby et al., 2019; Chen et al., 2022b) or Convolution-
like (Jie and Deng, 2022), these modules overlook leveraging the task-specific
information in their efficient module, and the number of trainable parameters
is not quite small and offers suboptimal performance.

Though many works are proposed to explore the parameter-efficient tuning
techniques, there still lacks the exploration of utilizing the task-specific
information in the efficient module design as it is evident that channel bias
exists in diverse downstream tasks(Luo et al., 2022). In this paper, we
propose a simple and straightforward baseline that aims to find partial
channels in the feature map by forwarding the pre-trained model with task-
specific images. Such partial tuning enables us only fine-tune a small portion
of channels which reduces the tunable parameters heavily. Our experiments
surprisingly demonstrate that tuning only a small portion of task-specific
channels is sufficient for downstream task adaptation in the low-data regime.
As for selecting the task-specific channels, we propose the Class-Aware
Importance Score (CAIS) by adopting $L_{2}$ norm as the criteria to evaluate
the channel importance inspired by the classic pruning method (Han et al.,
2015). We refer to the selected channels as “salient channels” (SC) since they
have higher activation values and are crucial for task performance. This
channel selection procedure helps to avoid the training costs by selecting the
best structure as in NOAH(Zhang et al., 2022) or the best prompt length as in
VPT(Jia et al., 2022). Compared with adapter-based methods (Houlsby et al.,
2019; Chen et al., 2022b), selecting partial task-specific channels enables us
to achieve lower parameter costs by removing the operations of “downsample”
and “nonlinearity” as shown in Fig. 2. We conclude the contributions of our
SCT as follows:

Contributions

  * •

We propose a simple baseline with the perspective in partial channel tuning
(_i.e_., salient channel tuning) for addressing the PEFT problem with
leveraging task-specific information, achieving promising performance on 19
downstream tasks, domain generalization, and few-shot learning. The
experiments demonstrate that tuning a subset of channels in a feature map
efficiently for fine-tuning downstream tasks.

  * •

It is the first time to reduce the extra parameters into 0.11M level of ViT-B
with such competitive performance, which is 780$\times$ fewer parameters than
the original model.

  * •

Our proposed task-specific information storage baseline (SCT) offers a simple
solution that requires minimal parameters while preventing the degradation of
the original model in downstream domains (full fine-tuning). The low parameter
requirements and the ability to adapt with few samples make it a potential
task knowledge storage unit for real-world applications.

##  2 Related Work

###  2.1 Vision Transformers

Transformer (Vaswani et al., 2017) has demonstrated outstanding results on
natural language processing and computer vision tasks. Lots of vision
transformers (Chen et al., 2021a; d’Ascoli et al., 2021; Dong et al., 2022;
Ali et al., 2021; Fan et al., 2021; Han et al., 2021; Rao et al., 2021; Yuan
et al., 2021; Touvron et al., 2021; Liu et al., 2021; Wang et al., 2021; Zhou
et al., 2021a) are proposed after the pioneering work ViT (Dosovitskiy et al.,
2020). Many of them increase the model size gradually for state-of-the-art
results and learn the rich representations by various architectural designs.
Noteworthy, most of them are trained on the natural dataset and have the
strong potential to be transferred to other domains/tasks. Adopting these
models to the downstream tasks alleviates the training difficulty obviously
and achieves promising results rapidly.

###  2.2 Parameter-Efficient Fine-Tuning Methods

PEFT focuses on adopting a trainable module with a few parameters for fine-
tuning. Two lines of PEFT have been proposed recently. On the one hand,
applying prompts (Jia et al., 2022; Liu et al., 2022; Xing et al., 2022; Zheng
et al., 2022; Nie et al., 2022; Wang et al., 2022; Zhou et al., 2022a, b; Liao
et al., 2023; Manli et al., 2022; Zhang et al., 2023b; Zang et al., 2022; Bar
et al., 2022) to the backbone networks shows success on several vision tasks.
VPT (Jia et al., 2022) first proposes the prompt-based method in computer
vision fields. VPT injects the prompts into the transformer layer with a small
number of extra parameters. However, one main limitation of VPT is that it
relies on hand-crafted selection to determine the optimal prompt length for
each task. This can be inflexible when applying the method to new tasks.
Another limitation is that the task-specific prompts of VPT are obtained
through the training procedure and cannot be given by the user as in the NLP
field. In our work, to explicitly reduce the computations of searching the
task-specific prompt length, we pass the training images into the backbone
network and leverage the task-specific information by determining the salient
channels. This procedure only needs one forward propagation without training
costs.

On the other hand, adding a residual module (Houlsby et al., 2019; Chen et
al., 2022b; Jie and Deng, 2022; Chen et al., 2022a) in the backbone networks
also acquires promising results for performance and efficiency. Adapter
(Houlsby et al., 2019), a widely used baseline in many tasks (Sung et al.,
2022; Pan et al., 2022; Zhang et al., 2023a), proposes an MLP-like module, a
successful design that first projects the original dimensional features into a
smaller dimension with one nonlinear layer and projects it back to the
original dimensions. It vastly reduces the number of parameters. Inspired by
its smallest intermediate dimensions, finding a small number of salient
channels in a feature map might be enough for the adaptation. Unlike injecting
trainable modules into the transformer blocks, LoRA (Hu et al., 2021)
optimizes a low-rank decomposition matrix with a low intrinsic dimension to
project the query, key, and value matrices used in multiheaded self-attention
in ViT. As for NOAH (Zhang et al., 2022), a neural architecture search
algorithm incorporates Adapter, LoRA, and VPT into its network search space.
NOAH provides a strong baseline for performing consistently well on different
datasets. SSF (Lian et al., 2022), a recently proposed strong baseline by only
scaling and shifting the features to implement the efficient model tuning.
Child-Tuning Xu et al. (2021) updates a subset of parameters of large pre-
trained language models via strategically masking out the gradients of the
non-child network during the backward process. AdaptFormer Chen et al. (2022b)
is an adapter-like method. It explores the efficiency of the adapter-like
architecture of vision transformers’ fine-tuning.

Unlike the above methods, our SCT handles the PEFT by leveraging the task-
specific information and then proposes the Salient Channel Tuning Module
(SCTM) from the new perspective of partial channel tuning, achieving low
parameter costs and higher performance than previous methods.

##  3 Method

Figure 4: The overview of proposed salient channel tuning module.

###  3.1 Not all channels are equal

To leverage task-specific information, this paper aims to find task-specific
channels for fine-tuning. Here we provide a simple observation highlighting
the “Salient Channels” that exist in the downstream tasks.

Previous works (Luo et al., 2017; Li et al., 2016; He et al., 2018; Liu et
al., 2018; Han et al., 2015; Li et al., 2017) demonstrate that pruning some
channels of deep neural networks has a marginal influence on the model
performance but can significantly reduce the parameter number and
computational cost. Such results reflect that the importance of different
channels is not the same, _i.e_., “Not all channels are equal”. Intuitively,
the channel importance is different in terms of the tasks, which motivates us
to investigate the impact of channel selection in model tuning.

To find task-specific channels, an intuitive way is to find some mutual
channels across different categories. Thus, We first illustrate an observation
between the pre-trained model and the downstream task. We choose Caltech101
(Fei-Fei et al., 2004) (one of the downstream tasks from the VTAB-1K
benchmark) as an example to evaluate the intermediate features of each
transformer layer of ViT. ViT-B pre-trained on ImageNet21K is the backbone,
and we feed the whole dataset to extract the features between multi-head self-
attention (“Attn”) and “MLP” blocks in all 12 transformer layers. Fig. 3 (a)
shows some vertical lines in each subfigure of transformer layers. It
indicates that when using a pre-trained model to extract the features on the
target dataset, some channels have higher activation values than others,
regardless of categories. We so-call these channels “salient channels”. To
investigate the position of salient channels, we also report the index of
selected top-36 salient channels after calculating the importance score of
each layer in Fig. 3 (b). The number represents the channel index of the total
of 768 channels. The darker color represents the higher activation values. We
could find different layers have different salient channel sets. As the layer
goes deeper, the salient channels appear more concentrated. We hypothesize
that deeper layers contain more abstract information and information
aggregated in a few channels. In contrast, the shallow layers extract more
low-level representation, making the information dense.

It naturally raises a question: Could we find salient channels in a feature
map and then only tune these salient channels for efficient tuning? To answer
this question, we design a simple class-aware importance score to identify
these salient channels and leverage a salient channel tuning module to tune
these channels efficiently.

###  3.2 Class-Aware Importance Score

The selection of salient channels is crucial for the adaptation to downstream
tasks, which should contain mutual information in all classes. Intuitively, we
can directly use the mean feature of all the data samples to represent the
downstream task distribution to determine the salient channels. However,
considering downstream datasets may not always be balanced, treating the whole
dataset as a unity may be biased to the head classes, leading to the selected
channels not representing the mutual information over all the classes.
Consequently, we propose the Class-Aware Importance Score (CAIS), where the
importance score calculation is conducted at each class and then averaged
across all the classes. Assume the intermediate feature maps as
$\\{f_{m}^{l}\in\mathbb{R}^{B_{m}\times N\times D}|\ l,m\in\mathbb{N},1\leq
l\leq L\ and\ 1\leq m\leq M\\}$, where the $N$ and $D$ represent the number of
tokens and amount of channel dimension, respectively. $L$ represents the total
layers of the ViT backbone. The $B_{m}$ is the volume of each class, and $M$
represents the number of categories of the downstream recognition task. We
first apply the $L_{2}$ norm regularization of each channel dimension of
feature $f_{m}^{l}$:

$\begin{split}\tilde{f}_{m}^{l}=&\\{{\|f_{m,1}^{l}\|_{2}},\
{\|f_{m,i}^{l}\|_{2}},...,\ {\|f_{m,D}^{l}\|_{2}}\\},\\\ &\
\|f_{m,i}^{l}\|_{2}\in\mathbb{R},\ i\in\mathbb{N},\ 1\leq i\leq D,\end{split}$
(1)

where $\tilde{f}_{m}^{l}\in\mathbb{R}^{1\times D}$ is the importance score
vector at $l$-th layer and $m$-th class. To investigate the salient channels,
we average the $\tilde{f}_{m}^{l}\in\mathbb{R}^{1\times D}$ over all classes
to get the importance score at $l$-th layer:

$Z^{l}=\frac{1}{M}\sum_{m=1}^{M}\tilde{f}_{m}^{l},\tilde{f}_{m}^{l}\in\mathbb{R}^{1\times
D},$ (2)

After getting the importance score vector $\\{Z^{l}\in\mathbb{R}^{1\times D}|\
l\in\mathbb{N},\ 1\leq l\leq L\\}$, we can choose the largest $K$ values of
$Z^{l}$ and then we can derive selected indices as $I^{l}=topK(Z^{l}),\
I^{l}\in\mathbb{N}^{1\times K}$ at $l$-th layer. Later, the salient channels
at each layer are different across different downstream tasks, and it could
explicitly involve task-specific information in the model fine-tuning. The
procedures for selecting salient channels are shown in Algorithm 1.

###  3.3 Adapting ViT via Salient Channel Tuning

Given the importance score of selecting top-$K$ salient channels, we propose a
Salient Channel Tuning Module (SCTM) in this paper. The overview of the SCTM
is depicted in Fig. 4. Unlike other PEFT methods, our SCTM only contains a
linear layer rather than an MLP-like adapter (Houlsby et al., 2019) involving
two learnable layers and one activation layer illustrated in Fig.2. Such a
simple structure is easy to reproduce and maintains relatively low parameter
costs. To inherit the original robust representations, we utilize a residual
shortcut to fuse the salient features with intermediate features and use the
$Scale\in\mathbb{R}$ (as shown in Fig. 4) to adjust the weights between both
features, which is a constant parameter. After obtaining the transformed
features, we insert the transformed features back into the input features at
the same position. The unselected channels we keep frozen during fine-tuning.
In Fig. 6, we present two forms of inserting the SCTM into the ViT. The “Ours-
MLP” represents that we insert the SCTM after the MLP block, and the “Our-
Attn” indicates that we put the SCTM after the MHSA block while before the MLP
block. In the following experiments, “Ours-Attn” is the default injection
position compared with other baselines.

(a) Evaluate the importance of salient channels on three datasets. We erase
all layers with randomly selected channel indices and salient channel indices.
To avoid the noise, we repeat the random selection process 10 times. The
results demonstrate the effectiveness of our salient channel selection.

(b) We systematically remove one layer at a time, identified by its index. The
resulting curve highlights the relative significance of the first four layers
compared to the last four in the context of the Caltech101 and Resisc45 tasks.
When considering the KITTI task, all layers are important for the final
output. Figure 5: Channel erasing test.

Overall. Our SCT first feeds the training set to a backbone network to extract
the intermediate features in different layers. Then, using the proposed CAIS
to determine the salient channels and save the channel indices. During fine-
tuning, we add our SCTM in each transformer layer and tune the selected
channels with a linear layer while freezing other unselected channels.

Discussion. Firstly, different downstream tasks have their peculiarities,
_i.e_., “Not all channels are equal”. Previous PEFT methods (_i.e_., Adapter)
lack the consideration of explicitly using task-specific information. Instead,
we propose the SCT explicitly leverage the task-specific information for
channel selection, which could enable us to save the computations and extra
parameters of the “Downsample” and “Nonlinearity” operations compared with
Adapter (Houlsby et al., 2019). Secondly, calculating the salient channels
offline could save the computations of searching the best prompt length (Jia
et al., 2022) or the best network structure (Zhang et al., 2022). Thirdly, our
results in Fig. 1 demonstrate that only adapting 12.5% channels could obtain
competitive results compared to other baselines. Simultaneously, transforming
a small portion of the features could significantly reduce the number of
learnable parameters (e.g., $12\times 96\times 96$ is relatively more minor
than the $12\times 768\times 768$). Fourthly, such a small amount of
parameters largely reduces the difficulty of storing task-specific knowledge
when we have many downstream tasks. To evaluate this simple baseline, we also
test its few-shot capacity and domain generalization ability to meet the
requirements of real-world applications.

###  3.4 Channel Erasing Test

To assess the channel selection configuration, we give two toy tests to
evaluate the effectiveness of salient channels as in Figure 5. The first test
in Figure 5 (a) demonstrates the effectiveness of salient channels compared
with random channel selection. The second test in Figure 5 (b) evaluates the
importance of each layer. We erase the salient channels layer by layer. The
curves suggest that for Natural type task Caltech101, the first four layers
are more important than the last four layers as the accuracy drops
significantly in the erasing first four layers. A similar trend is evident in
the Resisc45 task. As for the KITTI task, the last four layers are as
important as the first four.

Based on the aforementioned tests, we manipulate the number of fine-tuned
channels within the first four layers and the last eight layers, as outlined
in Table 1. Increasing the number of fine-tuned channels within the first four
layers (first and third rows) yielded enhancements of 0.1% for Caltech101 and
0.5% for KITTI, albeit with a 0.1% decline in Resisc45 performance. Notably,
the influence of the last four layers on the final results is slight.
Consequently, we explored the effect of decreasing the number of tuned
channels within the last eight layers. The results (fourth and fifth rows)
detailed in Table 1 indicate that reducing the $K$ from 96 to 32 leads to a
significant deterioration in performance. However, a slightly higher value
compared with setting $K=32$ across all 12 layers.

Algorithm 1 The procedures of channel selection via class-aware importance
score.

Inputs:

The pre-trained ViT model $\mathcal{F}$ and the number of layers $L$;
The number of selected channels $K$ and the total number of classes $M$;

1: for each $l\in[1,L]$ do

2: Extract the feature $f_{m}^{l}$ at each class $m$;

3: Apply the $L_{2}$ norm regularization on each channel to obtain
$\tilde{f}_{m}^{l}$ via Eq. 1;

4: Calculate the importance score $Z^{l}$ of $l$-th layer via Eq. 2;

5: Select top-$K$ largest channels $I^{l}=\text{topK}(Z^{l})$;

6: end for

Outputs: Selected channel indices of each transformer layer.

Channel configurations | Caltech101 | Resisc45 | KITTI | Params
---|---|---|---|---
[96, 96, 96, 96, 96, 96, 96, 96, 96, 96, 96, 96] | 91.6 | 86.3 | 80.2 | 0.11M
[192, 192, 192, 192, 192, 192, 192, 192, 192, 192, 192, 192] | 91.5 | 87.0 | 81.0 | 0.44M
[192, 192, 192, 192, 96, 96, 96, 96, 96, 96, 96, 96] | 91.7 | 86.2 | 80.7 | 0.22M
[96, 96, 96, 96, 32, 32, 32, 32, 32, 32, 32, 32] | 91.1 | 85.0 | 76.2 | 0.045M
[32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32] | 90.3 | 84.8 | 74.0 | 0.01M

Table 1: Different number of fine-tuned channels on three tasks.

##  4 Experiments

This section compares our SCT with other state-of-the-art PEFT baselines on
the VTAB-1K benchmark, using ViT and Swin Transformer backbones. In addition,
we analyze the channel selection strategy, class-aware importance score,
insert position, insert length, and the number of selected channels to verify
SCT’s effectiveness further. Beyond evaluating performance on various
downstream tasks, we also test SCT’s domain generalization capabilities using
four datasets with natural distribution shifts and evaluate its performance in
low-data regimes with five datasets in the few-shot scenario.

###  4.1 Training details

Following VPT (Jia et al., 2022), we utilize grid search to find the tuning-
specific hyper-parameters, including learning rate and weight decay, based on
the val set of each task as shown in Tab.2. For the hyper-parameter $Scale$,
which is a manually defined constant to balance the weight of transformed
features and salient features, we search it at the range {0.1, 0.2, 0.3, 0.4,
0.5, 0.6, 0.7, 0.8, 0.9, 1.0}. During training, we use a standard image
augmentation strategy: normalize with ImageNet means and standard deviation,
randomly resize crop to 224×224, and random horizontal flip as in VPT.

| VTAB-1K, Domain Generalization, and Few-shot Learning
---|---
Optimizer | AdamW (Loshchilov and Hutter, 2017)
$base_{lr}$ range | {0.1, 0.5, 0.01, 0.05 0.001, 0.005, 0.0001, 0.0005}
Weight decay range | {0.1 0.01, 0.05, 0.001, 0.005, 0.0005, 0.0001}
Learning rate schedule | cosine decay
Warm up epochs | 10
Total epochs | 100
Batch size | 64

Table 2: Training details on each visual task with ViT-B/16.

###  4.2 Experiments on VTAB-1K Benchmark

| Dataset | # Classes | Train | Val | Test
---|---|---|---|---|---
VTAB-1K (Zhai et al., 2019)
Natural | CIFAR100 (Krizhevsky et al., 2009) | 100 | 800/1000 | 200 | 10,000
Caltech101 (Fei-Fei et al., 2004) | 102 | 6,048
DTD (Cimpoi et al., 2014) | 47 | 1,880
Oxford-Flowers102 (Nilsback and Zisserman, 2006) | 192 | 6,149
Oxford-PetS (Parkhi et al., 2012) | 37 | 3,669
SVHN (Netzer et al., 2011) | 10 | 26,032
Sun397 (Xiao et al., 2010) | 397 | 21,750
Specialized | Patch Camelyon (Veeling et al., 2018) | 2 | 800/1000 | 200 | 32,768
EuroSAT (Helber et al., 2019) | 10 | 5,400
Resisc45 (Cheng et al., 2017) | 45 | 1,880
Retinopathy (Kaggle and EyePacs, 2015) | 5 | 42,670
Structured | Clevr/count (Johnson et al., 2017) | 8 | 800/1000 | 200 | 15,000
Clevr/distance (Johnson et al., 2017) | 6 | 15,000
DMLab (Beattie et al., 2016) | 6 | 22,735
KITTI-Dist (Geiger et al., 2013) | 4 | 711
dSprites/location (Matthey et al., 2017) | 16 | 73,728
dSprites/orientation (Matthey et al., 2017) | 16 | 73,728
SmallNORB/azimuth (LeCun et al., 2004) | 18 | 12,150
SmallNORB/elevation (LeCun et al., 2004) | 18 | 12,150
Domain generalization
| ImageNet-1K (Deng et al., 2009) | 1,000 | 16 per class | 50,000 | N/A
| ImageNet-V2 (Recht et al., 2019) | 1,000 | N/A | N/A | 10,000
| ImageNet-Sketch (Wang et al., 2019) | 1,000 | N/A | N/A | 50,889
| ImageNet-A (Hendrycks et al., 2021b) | 200 | N/A | N/A | 7,500
| ImageNet-R (Hendrycks et al., 2021a) | 200 | N/A | N/A | 30,000
Few-shot Learning
| Food101 (Bossard et al., 2014) | 101 | 1/2/4/8/16 per class | 20,200 | 30,300
| Stanford Cars (Krause et al., 2013) | 196 | 1,635 | 8,041
| Oxford-Flowers102 (Nilsback and Zisserman, 2006) | 102 | 1,633 | 2,463
| FGVC-Aircraft (Maji et al., 2013) | 100 | 3,333 | 3,333
| Oxford-Pets (Parkhi et al., 2012) | 37 | 736 | 3,669

Table 3: Specifications of used datasets on VTAB-1K benchmark, Domain
Generalization, and Few-shot Learning.

Dataset. VTAB-1K (Zhai et al., 2019) contains 19 visual classification tasks
which cover a broad spectrum of domains and semantics in three groups, _i.e_.,
Natural, Specialized, and Structured. The Natural group contains 7 classic
classification datasets of natural images. The Specialized group involves 4
datasets of two special scenarios: medical and remote-sensing. The Structured
group has 8 datasets, mainly focusing on understanding the structure of a
scene, such as object counting and depth prediction. Each task of VTAB-1K
contains 1000 training images. More details are available in Tab.3. Following
Zhang et al. (2022); Jia et al. (2022), we use the 800-200 train-val split to
determine the hyperparameters and the entire 1000 training data to train the
final model. We report the average top-1 accuracy on the test set.

Figure 6: Two types of structures when inserting SCTM into the backbone. Only
SCTM is trainable, and other modules are frozen.

Baselines. We compare our method with three baselines Full fine-tuning,
Linear, and Bias without external parameters and four baselines Adapter
(Houlsby et al., 2019), LoRA (Hu et al., 2021), VPT (Jia et al., 2022), and
NOAH (Zhang et al., 2022) with external parameters. Bias method only updates
all the bias terms in the pre-trained backbone. Adapter injects an additional
MLP module into each transformer layer. LoRA adopts an optimized low-rank
matrix to the MHSA module in the transformer layers. VPT is a visual prompt
algorithm to incorporate the prompts with tokens into the backbone. NOAH is a
neural architecture search algorithm incorporating the Adapter, LoRA, and VPT
into the network search. SSF propose two learnable factors to scale and shift
the features when fine-tuning the downstream tasks. AdaptFormer proposes an
adapter-like architecture for vision tasks. We directly use their released
results or run their code to generate them to provide a fair comparison.

|  | Natural | Specialized | Structured |
---|---|---|---|---|---
# Params (M) |  Cifar100 |  Caltech101 |  DTD |  Flower102 |  Pets |  SVHN |  Sun397 |  Camelyon |  EuroSAT |  Resisc45 |  Retinopathy |  Clevr-Count |  Clevr-Dist |  DMLab |  KITTI-Dist |  dSpr-Loc |  dSpr-Ori |  sNORB-Azim |  sNORB-Ele |  Average
Commonly used methods
Full | 85.8 | 68.9 | 87.7 | 64.3 | 97.2 | 86.9 | 87.4 | 38.8 | 79.7 | 95.7 | 84.2 | 73.9 | 56.3 | 58.6 | 41.7 | 65.5 | 57.5 | 46.7 | 25.7 | 29.1 | 65.6
Linear | 0 | 64.4 | 85.0 | 63.2 | 97.0 | 86.3 | 36.6 | 51.0 | 78.5 | 87.5 | 68.5 | 74.0 | 34.3 | 30.6 | 33.2 | 55.4 | 12.5 | 20.0 | 9.6 | 19.2 | 53.0
Bias | 0.10 | 72.8 | 87.0 | 59.2 | 97.5 | 85.3 | 59.9 | 51.4 | 78.7 | 91.6 | 72.9 | 69.8 | 61.5 | 55.6 | 32.4 | 55.9 | 66.6 | 40.0 | 15.7 | 25.1 | 62.1
PEFT methods
VPT-Deep | 0.53 | 78.8 | 90.8 | 65.8 | 98.0 | 88.3 | 78.1 | 49.6 | 81.8 | 96.1 | 83.4 | 68.4 | 68.5 | 60.0 | 46.5 | 72.8 | 73.6 | 47.9 | 32.9 | 37.8 | 69.4
NOAH | 0.43 | 69.6 | 92.7 | 70.2 | 99.1 | 90.4 | 86.1 | 53.7 | 84.4 | 95.4 | 83.9 | 75.8 | 82.8 | 68.9 | 49.9 | 81.7 | 81.8 | 48.3 | 32.8 | 44.2 | 73.3
LoRA | 0.29 | 67.1 | 91.4 | 69.4 | 98.8 | 90.4 | 85.3 | 54.0 | 84.9 | 95.3 | 84.4 | 73.6 | 82.9 | 69.2 | 49.8 | 78.5 | 75.7 | 47.1 | 31.0 | 44.0 | 72.3
SSF† | 0.24 | 67.4 | 91.1 | 73.3 | 99.4 | 92.0 | 91.5 | 52.0 | 86.6 | 95.9 | 87.4 | 75.6 | 76.9 | 62.3 | 53.6 | 77.7 | 77.3 | 54.5 | 30.1 | 36.0 | 72.6
Adapter-8† | 0.16 | 72.8 | 90.7 | 70.3 | 93.8 | 91.1 | 87.5 | 54.8 | 83.0 | 84.9 | 85.7 | 75.5 | 82.1 | 63.4 | 50.9 | 79.2 | 74.8 | 47.3 | 28.6 | 39.1 | 71.3
AdaptFormer-8† | 0.18 | 73.5 | 91.5 | 70.8 | 98.9 | 91.1 | 87.8 | 54.3 | 82.3 | 94.9 | 86.9 | 76.3 | 82.3 | 66.3 | 51.0 | 78.8 | 76.5 | 46.2 | 32.3 | 40.1 | 72.7
SCT | 0.11 | 75.3 | 91.6 | 72.2 | 99.2 | 91.1 | 91.2 | 55.0 | 85.0 | 96.1 | 86.3 | 76.2 | 81.5 | 65.1 | 51.7 | 80.2 | 75.4 | 46.2 | 33.2 | 45.7 | 73.6

Table 4: Comparisons with state-of-the-art methods on the VTAB-1K benchmark
with ViT-B/16. Average results are calculated across 19 datasets. “# Params”
denotes the backbone’s average number of extra trainable parameters. The best
performance and smallest parameter number are bolded in each column. Underline
notes the second-best result. $\dagger$ means the model we implemented and
trained on the same training setting, and the hyperparameters are selected by
standard grid search since Adapter Houlsby et al. (2019) is a strong baseline
for NLP tasks and Adaptformer Chen et al. (2022b) is not evaluated on VTAB-1K.
We implement the SSF with its released code and configurations. The results of
LoRA Hu et al. (2021) are from NOAH Zhang et al. (2022). The hidden dimensions
of the Adapter and AdaptFormer are 8 and 96. The reduction dimension of LoRA
we set is 8. Figure 7: Group-wise average results on VTAB-1K benchmark.

|  | Natural | Specialized | Structured |
---|---|---|---|---|---
# Params (M) |  Cifar100 |  Caltech101 |  DTD |  Flower102 |  Pets |  SVHN |  Sun397 |  Camelyon |  EuroSAT |  Resisc45 |  Retinopathy |  Clevr-Count |  Clevr-Dist |  DMLab |  KITTI-Dist |  dSpr-Loc |  dSpr-Ori |  sNORB-Azim |  sNORB-Ele |  Average
Commonly used methods
Full | 86.7 | 72.2 | 88.0 | 71.4 | 98.3 | 89.5 | 90.1 | 45.0 | 86.6 | 96.9 | 87.7 | 79.4 | 75.7 | 59.8 | 54.6 | 78.6 | 79.4 | 53.6 | 34.6 | 40.9 | 74.2
Linear | 0 | 61.4 | 90.2 | 74.8 | 99.5 | 90.2 | 42.7 | 55.8 | 81.5 | 90.1 | 82.1 | 69.4 | 39.1 | 35.9 | 40.1 | 65.0 | 20.3 | 26.0 | 14.3 | 27.8 | 56.4
Bias | 0.24 | 73.0 | 86.8 | 65.6 | 97.7 | 87.5 | 56.4 | 52.3 | 80.4 | 91.6 | 76.1 | 72.5 | 47.3 | 48.5 | 34.7 | 66.2 | 57.6 | 36.2 | 34.7 | 66.2 | 62.1
PEFT methods
VPT-Deep | 0.19 | 79.6 | 90.8 | 78.0 | 99.5 | 91.4 | 42.3 | 51.7 | 84.9 | 96.2 | 85.0 | 72.0 | 67.6 | 59.4 | 50.1 | 61.3 | 74.4 | 50.6 | 25.7 | 25.7 | 68.6
Adapter-8† | 0.11 | 74.2 | 90.7 | 75.9 | 99.5 | 92.5 | 38.0 | 55.6 | 88.2 | 95.0 | 88.9 | 76.9 | 71.4 | 56.7 | 54.1 | 84.5 | 79.6 | 35.8 | 21.9 | 35.3 | 69.2
LoRA† | 0.38 | 72.0 | 91.1 | 76.5 | 99.6 | 92.3 | 81.9 | 55.5 | 86.8 | 96.3 | 82.0 | 77.5 | 71.0 | 60.6 | 57.0 | 78.6 | 87.0 | 52.1 | 30.6 | 39.5 | 73.1
AdaptFormer-8† | 0.13 | 74.6 | 91.0 | 76.2 | 99.6 | 92.4 | 83.5 | 55.4 | 87.0 | 95.2 | 88.9 | 76.8 | 73.8 | 57.9 | 54.8 | 82.7 | 82.2 | 49.7 | 21.6 | 36.9 | 72.6
SSF† | 0.27 | 75.4 | 90.0 | 75.1 | 91.0 | 91.2 | 92.3 | 55.0 | 88.8 | 84.2 | 89.1 | 76.8 | 71.0 | 53.5 | 56.0 | 83.4 | 84.2 | 52.8 | 30.5 | 33.2 | 72.3
SCT (ours) | 0.10 | 75.7 | 92.6 | 76.5 | 99.7 | 91.7 | 87.2 | 55.5 | 87.6 | 96.5 | 89.4 | 76.6 | 82.5 | 63.1 | 53.7 | 85.9 | 86.7 | 46.1 | 26.8 | 40.0 | 74.4

Table 5: Per-task results on VTAB-1K benchmark with a pre-trained Swin-B. The
average result is calculated on all 19 datasets in the VTAB-1K benchmark.

Performance with ViT backbone. We compare our SCT with the above 7 baselines
in Tab. 4 and Fig. 7. We use ViT-B/16 as the backbone and insert SCTM in each
transformer layer. The default $K$ is set to 96, 1/8 of the total channels,
leading to the trainable parameter number being only 0.11M. First, our SCT
achieves the average accuracy of 73.6% on the 19 datasets, outperforming the
full fine-tuning (85.8M) on 18 out of 19 datasets and gains the improvement of
6.3%, 2.5%, and 12.3% in the three groups, respectively, with only additional
0.13% of the backbone parameters. Such results reflect that SCT can greatly
reduce the storage space and alleviate the overfitting problem common in fine-
tuning large models. Second, compared with Adapter (0.16M) (Houlsby et al.,
2019) that treats all the channels equally, selecting a part of salient
channels for each downstream task is more effective and efficient,
outperforming it by 2.3% in average accuracy. Third, compared with other PEFT
methods, our SCT surpasses NOAH (0.43M) (Zhang et al., 2022) by 0.3% with only
a quarter of its trainable parameters. When increasing the $K$ to 192 with
0.44M extra parameters, our SCT surpasses NOAH (0.43) by 0.7 % average
accuracy. Moreover, our SCT outperforms VPT (Jia et al., 2022) by 3.8%, 3.5%,
and 4.9% in the three groups, respectively. These results demonstrate that
instead of specially designed structures for each downstream dataset,
explicitly leveraging the task-specific information can reduce the
computational cost in the model search and improve the performance. Moreover,
compared with SSF (0.20M) (Lian et al., 2022) baseline, a method without an
additional trainable module, our SCT only needs almost its half parameters and
achieves an average improvement of 1.0% across 19 downstream tasks.

Performance with Swin Transformer Backbone. To verify the effectiveness of SCT
with different backbones, we apply SCT on hierarchical transformers, _i.e_.,
Swin-B. We use the same setting of inserting SCTM as in the ViT backbone:
inserting the SCTM after the “Attn” block in the transformer layer.
Considering deep layers contain more semantic information in the hierarchical
structure, instead of applying SCTM on all the transformer layers, we insert
it to the last half of the layers in the stage 3 and all layers of stage 4 of
the Swin-B to keep a similar level of trainable parameters. The results of
Tab. 5 show that SCT outperforms full fine-tuning in all 19 downstream tasks
with only 0.12% parameters while other methods cannot. In addition, compared
with the PEFT method, SCT outperforms VPT (Jia et al., 2022) by 5.9%, 3.0%,
and 7.2% in the three groups, respectively. All the results above suggest that
our SCT also applies to hierarchical transformers and can yield much more
improvement than other PEFT methods.

###  4.3 Evaluation

|  | Natural | Specialized | Structured |
---|---|---|---|---|---
# Params (M) |  Cifar100 |  Caltech101 |  DTD |  Flower102 |  Pets |  SVHN |  Sun397 |  Camelyon |  EuroSAT |  Resisc45 |  Retinopathy |  Clevr-Count |  Clevr-Dist |  DMLab |  KITTI-Dist |  dSpr-Loc |  dSpr-Ori |  sNORB-Azim |  sNORB-Ele |  Average
Full | 85.8 | 68.9 | 87.7 | 64.3 | 97.2 | 86.9 | 87.4 | 38.8 | 79.7 | 95.7 | 84.2 | 73.9 | 56.3 | 58.6 | 41.7 | 65.5 | 57.5 | 46.7 | 25.7 | 29.1 | 65.6
Linear | 0 | 64.4 | 85.0 | 63.2 | 97.0 | 86.3 | 36.6 | 51.0 | 78.5 | 87.5 | 68.5 | 74.0 | 34.3 | 30.6 | 33.2 | 55.4 | 12.5 | 20.0 | 9.6 | 19.2 | 53.0
IC | 0.11 | 68.8 | 91.5 | 71.5 | 99.0 | 91.1 | 89.0 | 54.3 | 85.3 | 95.8 | 86.6 | 75.2 | 78.1 | 61.2 | 50.3 | 79.0 | 73.1 | 40.0 | 27.4 | 36.7 | 71.1
RC-1 | 0.11 | 72.1 | 90.4 | 71.7 | 99.1 | 91.2 | 90.2 | 54.0 | 84.2 | 95.5 | 85.7 | 75.9 | 78.8 | 62.5 | 49.5 | 77.6 | 72.2 | 43.3 | 27.3 | 40.9 | 71.7
RC-2 | 0.11 | 74.1 | 91.4 | 72.1 | 99.2 | 90.9 | 90.2 | 55.1 | 83.0 | 95.7 | 86.3 | 75.0 | 79.3 | 62.5 | 50.4 | 78.2 | 72.5 | 43.2 | 28.1 | 39.8 | 72.0
RC-3 | 0.11 | 71.3 | 91.1 | 70.9 | 99.0 | 90.9 | 89.6 | 54.4 | 83.9 | 95.7 | 85.9 | 74.8 | 78.6 | 62.2 | 48.9 | 80.2 | 71.2 | 42.1 | 27.8 | 38.5 | 71.4
SCT | 0.11 | 75.3 | 91.6 | 72.2 | 99.2 | 91.1 | 91.2 | 55.0 | 85.0 | 96.1 | 86.3 | 76.2 | 81.5 | 65.1 | 51.7 | 80.2 | 75.4 | 46.2 | 33.2 | 45.7 | 73.6

Table 6: Per-task results on VTAB-1K benchmark with a pre-trained ViT-B/16.
The average result is calculated on all 19 datasets in the VTAB-1K benchmark.

|  | Natural | Specialized | Structured |
---|---|---|---|---|---
# Params (M) |  Cifar100 |  Caltech101 |  DTD |  Flower102 |  Pets |  SVHN |  Sun397 |  Camelyon |  EuroSAT |  Resisc45 |  Retinopathy |  Clevr-Count |  Clevr-Dist |  DMLab |  KITTI-Dist |  dSpr-Loc |  dSpr-Ori |  sNORB-Azim |  sNORB-Ele |  Average
Full | 85.8 | 68.9 | 87.7 | 64.3 | 97.2 | 86.9 | 87.4 | 38.8 | 79.7 | 95.7 | 84.2 | 73.9 | 56.3 | 58.6 | 41.7 | 65.5 | 57.5 | 46.7 | 25.7 | 29.1 | 65.6
Linear | 0 | 64.4 | 85.0 | 63.2 | 97.0 | 86.3 | 36.6 | 51.0 | 78.5 | 87.5 | 68.5 | 74.0 | 34.3 | 30.6 | 33.2 | 55.4 | 12.5 | 20.0 | 9.6 | 19.2 | 53.0
w/o CA | 0.11 | 75.1 | 91.8 | 71.6 | 99.1 | 90.9 | 91.2 | 55.1 | 84.4 | 95.6 | 85.6 | 75.1 | 81.1 | 64.7 | 51.2 | 78.3 | 75.0 | 44.8 | 31.3 | 40.4 | 72.8
w/ CA | 0.11 | 75.3 | 91.6 | 72.2 | 99.2 | 91.1 | 91.2 | 55.0 | 85.0 | 96.1 | 86.3 | 76.2 | 81.5 | 65.1 | 51.7 | 80.2 | 75.4 | 46.2 | 33.2 | 45.7 | 73.6

Table 7: Per-task results of evaluating the effectiveness of adopting the
Class-Aware Importance Score. The average result is calculated on all 19
datasets in the VTAB-1K benchmark.

Effectiveness of Salient Channel Tuning. We compare three channel selection
strategies to verify the effectiveness of selecting salient channels in Tab.
6. The selection strategies include Salient Channel Selection (SC),
Inconspicuous Channel Selection (IC), and Random Channel Selection (RC). IC
selects channels with the lowest $K$ values of $L_{2}$ norm, and RC denotes
selecting $K$ channels randomly. We randomly select three sets of channels
(RC-1/2/3) for random channel selection to alleviate the outliers. As shown in
Tab. 6, SC achieves the best results and outperforms IC by 2.4% in the average
accuracy. Random channel selection could obtain better results than full fine-
tuning, but all of them perform worse than SC. Interestingly, even IC can
perform better than full fine-tuning, demonstrating that selecting a small
subset of the channels can prevent the large model from overfitting to the
small training set.

Effectiveness of Class-Aware Calculation for importance score. In Sec. 3.2, we
introduced our method, which considers the impact of individual classes rather
than treating the entire dataset as a whole when estimating the importance
score (IS). This approach aims to identify the most significant channels based
on the mutual information between categories and reduce the effects of
imbalanced data. Our experimental results, presented in Tab. 7, demonstrate
that adopting a class-aware calculation approach improves performance across
all three groups.

|  | Natural | Specialized | Structured |
---|---|---|---|---|---
# Params (M) |  Cifar100 |  Caltech101 |  DTD |  Flower102 |  Pets |  SVHN |  Sun397 |  Camelyon |  EuroSAT |  Resisc45 |  Retinopathy |  Clevr-Count |  Clevr-Dist |  DMLab |  KITTI-Dist |  dSpr-Loc |  dSpr-Ori |  sNORB-Azim |  sNORB-Ele |  Average
Full | 85.8 | 68.9 | 87.7 | 64.3 | 97.2 | 86.9 | 87.4 | 38.8 | 79.7 | 95.7 | 84.2 | 73.9 | 56.3 | 58.6 | 41.7 | 65.5 | 57.5 | 46.7 | 25.7 | 29.1 | 65.6
Linear | 0 | 64.4 | 85.0 | 63.2 | 97.0 | 86.3 | 36.6 | 51.0 | 78.5 | 87.5 | 68.5 | 74.0 | 34.3 | 30.6 | 33.2 | 55.4 | 12.5 | 20.0 | 9.6 | 19.2 | 53.0
SCT${}_{\text{MLP}}$ | 0.11 | 72.9 | 92.0 | 71.6 | 99.1 | 90.6 | 90.5 | 54.9 | 86.2 | 95.9 | 85.9 | 76.0 | 81.3 | 64.7 | 50.7 | 78.9 | 77.2 | 43.3 | 29.3 | 43.6 | 72.9
SCT${}_{\text{Attn}}$ | 0.11 | 75.3 | 91.6 | 72.2 | 99.2 | 91.1 | 91.2 | 55.0 | 85.0 | 96.1 | 86.3 | 76.2 | 81.5 | 65.1 | 51.7 | 80.2 | 75.4 | 46.2 | 33.2 | 45.7 | 73.6

Table 8: Per-task results of evaluating the effectiveness of the insert
positions. The average result is calculated on all 19 datasets in the VTAB-1K
benchmark.

|  | Natural | Specialized | Structured |
---|---|---|---|---|---
# Params (M) |  Cifar100 |  Caltech101 |  DTD |  Flower102 |  Pets |  SVHN |  Sun397 |  Camelyon |  EuroSAT |  Resisc45 |  Retinopathy |  Clevr-Count |  Clevr-Dist |  DMLab |  KITTI-Dist |  dSpr-Loc |  dSpr-Ori |  sNORB-Azim |  sNORB-Ele |  Average
Full | 85.8 | 68.9 | 87.7 | 64.3 | 97.2 | 86.9 | 87.4 | 38.8 | 79.7 | 95.7 | 84.2 | 73.9 | 56.3 | 58.6 | 41.7 | 65.5 | 57.5 | 46.7 | 25.7 | 29.1 | 65.6
Linear | 0 | 64.4 | 85.0 | 63.2 | 97.0 | 86.3 | 36.6 | 51.0 | 78.5 | 87.5 | 68.5 | 74.0 | 34.3 | 30.6 | 33.2 | 55.4 | 12.5 | 20.0 | 9.6 | 19.2 | 53.0
Bias | 0.10 | 72.8 | 87.0 | 59.2 | 97.5 | 85.3 | 59.9 | 51.4 | 78.7 | 91.6 | 72.9 | 69.8 | 61.5 | 55.6 | 32.4 | 55.9 | 66.6 | 40.0 | 15.7 | 25.1 | 62.1
SCT${}_{\text{K=32}}$ | 0.01 | 71.7 | 90.3 | 70.2 | 99.0 | 90.1 | 85.8 | 53.1 | 82.2 | 95.6 | 84.8 | 75.7 | 71.6 | 62.1 | 48.3 | 74.0 | 59.1 | 42.0 | 24.5 | 33.6 | 69.1
SCT${}_{\text{K=96}}$ | 0.11 | 75.3 | 91.6 | 72.2 | 99.2 | 91.1 | 91.2 | 55.0 | 85.0 | 96.1 | 86.3 | 76.2 | 81.5 | 65.1 | 51.7 | 80.2 | 75.4 | 46.2 | 33.2 | 45.7 | 73.6
SCT${}_{\text{K=192}}$ | 0.44 | 76.0 | 91.5 | 72.3 | 99.1 | 91.2 | 92.2 | 55.2 | 86.2 | 95.7 | 87.0 | 76.1 | 82.5 | 64.0 | 52.1 | 81.0 | 75.5 | 47.5 | 34.7 | 45.6 | 74.0

Table 9: Per-task results of evaluating different $K$ values on VTAB-1K
benchmark.

Figure 8: Choosing different $K$ values on 19 downstream tasks.

| Cifar100 | Resisc45 | DMLab | Params
---|---|---|---|---
SCT${}_{\text{Attn}}$ | 75.3 | 86.3 | 51.7 | 0.11M
SCT${}_{\text{MLP}}$ | 72.9 | 85.9 | 50.9 | 0.11M
SCT${}_{\text{Attn}}$ \+ SCT${}_{\text{MLP}}$ | 75.7 | 86.5 | 51.0 | 0.22M

Table 10: Comparison of different insert positions.

Insert Position. As shown in Fig. 6, SCTM can be inserted after the MLP block
(SCT${}_{\text{MLP}}$) or between the MHSA block and MLP block
(SCT${}_{\text{Attn}}$). To investigate the influence of insert location, we
compare two forms on the VTAB-1K benchmark in Tab. 8. Both achieve promising
performances, and SCT${}_{\text{Attn}}$ outperforms SCT${}_{\text{MLP}}$ on
two of the three groups. We conjecture that after gathering long-range
dependencies with MHSA, the salient channels of features are more
representative, which can be better adapted to downstream tasks. We also
evaluate the performance of inserting SCTM after both MHSA and MLP blocks, as
shown in Tab. 10. Jointly inserting SCT${}_{\text{Attn}}$ and
SCT${}_{\text{MLP}}$ only brings small improvements while the number of
parameters doubles.

Figure 9: Results of inserting SCTM into different layers. Zoom in for the
best view.

Number of insert depth. The insert depth is an important factor that can
significantly influence the performance of a model. To investigate this, we
conducted experiments by inserting SCTM into the last $l$ layers of ViT-B and
evaluated the performance on three representative tasks from each group of
VTAB-1K. Our findings, presented in Fig. 9, reveal that increasing the insert
depth results in a gradual improvement in performance. For the Resisc45 and
DMLab tasks, we observed that competitive performance was achieved when the
SCTM was inserted into the last six layers. Further increases in the insert
depth resulted in a plateau in performance, indicating that additional layers
did not contribute significantly to the final performance.

Number of selected channels $K$. The most important hyperparameter of SCTM is
the number of selected channels $K$, which influences the model architecture
and the number of trainable parameters. Note that different from previous
works (Jia et al., 2022; Zhang et al., 2022) that select hyperparameters for
each dataset, we use the same $K$ for all the datasets. As shown in Tab.9,
SCT${}_{\text{K=32}}$ beats the full fine-tuning and bias tuning by 3.5% and
7.0%, respectively. Furthermore, SCT${}_{\text{K=32}}$ only adopts 0.01M
parameters while the tuning bias term adopts 0.10M. As shown in Fig. 8, the
performance generally improves along with the increase of $K$. However, the
improvement of $K=192$ over $K=96$ is marginal, while the number of parameters
is four times larger. Considering both the effectiveness and efficiency, we
set $K$ to 96 by default.

###  4.4 Computation Analysis

Our analysis considers a ViT-B backbone with $L$ layers and $D$ dimensions and
$N$ tokens for a single image. We also assume that the intermediate dimension
of Adapter (Houlsby et al., 2019) is $D^{\prime}$, that the prompt length of
VPT (Jia et al., 2022) is $n$, and that the total insert times of SSF (Lian et
al., 2022) is $m$. Finally, we compare our proposed SCT approach to Adapter,
VPT, and SSF in terms of parameters and FLOPs, as summarized in Tab. 12.
Notably, our selection of $K$ as $\frac{1}{8}D$ is fairly small compared to
$D$. When the reduction ratio of the Adapter is also defined as 8, the
parameters of the Adapter are $\frac{1}{4}LD^{2}$ while our parameters are
$\frac{1}{64}LD^{2}$. When we compare our approach to SSF, which inserts its
module $m$ times in the ViT-B backbone, the number of parameters for SSF is
$mLD$. Examining the ViT-B backbone, we find that $m=74$ and
$\frac{1}{64}D=12$, our parameters ($12LD<74LD$) and FLOPs ($12NLD<74NLD$) are
smaller than SSF. Overall, our analysis suggests that SCT may offer PEFT a
more efficient and effective baseline.

To assess the computational efficiency of our SCT approach, we choose to
compare it with Adapter (0.16M) and SSF (0.20M) since they have a similar
level of parameters and all of them are applied in the backbone network. We
measured the running time and memory usage for both the training and testing
phases using a batch size of 64 on the Cifar100 (VTAB-1K) dataset with the
ViT-B backbone. We conducted training experiments on a single NVIDIA V100-32GB
GPU and test experiments on a single NVIDIA A100-40GB GPU. As the usage of GPU
memory during the training and testing phases is different. During the
training phase, tensor activations, model parameters, gradients, and optimizer
states (model, gradient, momentum) are the main sources. During the testing
phase, tensor activations and model parameters are the main sources. Our
results, as shown in Fig. 10, indicate that our SCT approach outperforms
Adapter and SSF regarding training time and memory usage. As our tunable
parameters are smaller than Adapter and SSF, the cost of gradients and
optimizer states are small which helps us save the GPU memory during training.
Specifically, our method requires only 12 additional SCT computations, while
SSF requires 74 scaling and shifting operations. The training of SSF is not
quite efficient compared with our SCT. However, during testing, the test time
of our method is slightly higher than Adapter and SSF. Noted, SSF could merge
its extra parameters to the original backbone with no extra inference costs
while our method does not focus on reducing the computation costs during
inference. The reason for higher test costs compared with Adapter may be the
extra operations of querying the salient channels from a full feature and then
putting the projected salient channels back. Furthermore, we give exact test
time cost results in Tab. 11. Here LoRA and SSF could merge its parameter into
the backbone network thus we choose Full fine-tuning as the representative.

Methods | Test time | Memory | Params |
---|---|---|---|---
Full fine-tuning | 5.78 ms | 336 MB | 85.8 M |
Adapter-8 | 6.87 ms | 337 MB | 85.8+0.16 M |
AdaptFormer-8 | 7.88 ms | 337 MB | 85.8+0.18 M |
SCT (Ours) | 8.43 ms | 337 MB | 85.8+0.11 M |

Table 11: We evaluate the test time for 1 image with resolution 224. We run
200 times and yield the average result.

| Adapter | VPT-Deep | SSF | TTC-Module
---|---|---|---|---
# Extra Parameters | $2LDD^{\prime}$ | $nLD$ | $mLD$ | $LKK$
# Extra FLOPs | $2NLDD^{\prime}$ | $2n(2N+n)LD$ | $mNLD$ | $NLKK$

Table 12: A complexity analysis of Adapter (Houlsby et al., 2019), VPT (Jia et
al., 2022), SSF (Lian et al., 2022), and our proposed TTC-Module.

###  4.5 Experiments on Domain Generalization

| Source | Target
---|---|---
ImageNet | -V2 | -Sketch | -A | -R
Adapter (Houlsby et al., 2019) | 70.5 | 59.1 | 16.4 | 5.5 | 22.1
VPT (Jia et al., 2022) | 70.5 | 58.0 | 18.3 | 4.6 | 23.2
LoRA (Hu et al., 2021) | 70.8 | 59.3 | 20.0 | 6.9 | 23.3
NOAH (Zhang et al., 2022) | 71.5 | 66.1 | 24.8 | 11.9 | 28.5
SCT-B (ours) | 77.1 | 65.8 | 28.5 | 12.1 | 31.0

Table 13: Comparison with previous methods on domain generalization. The
backbone network is ViT-B/16. The number of salient channels is 192.

Dataset. In addition to evaluating the model on test data of the same
distribution, modern deep neural networks commonly suffer from performance
degradation when the testing distribution is different from that of the
training set, _i.e_., domain shift, which is inevitable in a real-world
application. To alleviate this problem, domain generalization (Zhou et al.,
2021b; Zhao et al., 2022; Zhou et al., 2022c; Yang et al., 2022a, b, 2021) is
investigated in the community, aiming to train a model with one or multiple
source domains that can perform well on other unseen target domains. To verify
the generalization ability of our SCT, we follow Zhang et al. (2022) to
conduct experiments on ImageNet and its variants. Specifically, we use the
ImageNet-1K (Deng et al., 2009) as the source domain with 16-shot per category
and evaluate our model on ImageNetV2 (Recht et al., 2019), ImageNet-Sketch
(Wang et al., 2019), ImageNet-A (Hendrycks et al., 2021b), and ImageNet-R
(Hendrycks et al., 2021a). ImageNetV2 (Recht et al., 2019) is collected from
different sources from ImageNet-1K with the same protocol, and ImageNet-Sketch
(Wang et al., 2019) contains the sketch images of ImageNet classes. Both of
them use the same classes as ImageNet-1K. ImageNet-A (Hendrycks et al., 2021b)
and ImageNet-R (Hendrycks et al., 2021a) contain the adversarially-filtered
images and renditions of ImageNet data of a 200-class subset, respectively. We
use a large version of SCT, _i.e_., SCT-B, containing comparable parameters
with NOAH (0.44M _vs_. 0.43M).

Results. In Tab. 13, we compare our SCT-B with Adapter (Houlsby et al., 2019),
VPT (Jia et al., 2022), LoRA (Hu et al., 2021), and NOAH (Zhang et al., 2022)
on the above datasets to verify the generalization ability. We can make two
observations. First, SCT-B outperforms the previous best method (NOAH) on
three of the four target datasets and achieves comparable performance on
ImageNetV2. Specifically, SCT-B yields an improvement of 2.5% on ImageNet-R
over NOAH. Second, our SCT-B achieves an accuracy of 77.1% on the source
domain, greatly outperforming previous methods by 6%. Since the backbone model
is pre-trained on ImageNet-21K, the results on ImageNet-1K show that SCT can
better enhance the knowledge transfer from superset to subset. The two
observations demonstrate the superiority of our SCT over previous fine-tuning
techniques on strong generalization ability.

Figure 10: The computational costs of training and test phases.

###  4.6 Experiments on Few-shot Learning

Figure 11: Results of few-shot learning on five fine-grained visual
recognition datasets.

Datasets. Following Zhang et al. (2022), we choose five fine-grained visual
recognition datasets, including Food101 (Bossard et al., 2014), Flowers102
(Nilsback and Zisserman, 2006), StandfordCars (Krause et al., 2013),
OxfordPets (Parkhi et al., 2012), and FGVCAircraft (Maji et al., 2013), to
investigate the effectiveness of SCT in few-shot learning. The categories in
these datasets cover various visual concepts closely related to our daily
life: food, plant, vehicle, and animal. Next, we follow existing studies
(Zhang et al., 2022; Jie and Deng, 2022) to evaluate our model under 1, 2, 4,
8, and 16 shots settings. The experimental setup is the same as in VTAB-1K.

Results. According to Fig. 11, our SCT surpasses other baselines in all
settings in terms of average performance. In addition, SCT largely obtains the
advantages at FGVCAircraft and OxfordPets datasets. For other datasets, SCT
achieves competitive performance compared with other baselines. Such results
demonstrate that SCT has a strong generalization ability that can be readily
transferred to downstream tasks only with a few samples.

##  5 Conclusion

This paper introduces a simple baseline for PEFT, which is both simple and
effective. Unlike previous methods, we adopt a channel selection perspective
and propose a straightforward importance score to identify task-specific
channels. By eliminating downsampling and nonlinearity operations, our method
outperforms the similar Adapter method, requiring fewer parameters. We
evaluate our algorithm on 19 downstream tasks and achieve the best results
with only 780$\times$ fewer parameters than the original backbone. We also
test our approach on four datasets with natural distribution shifts to assess
domain generalization capabilities and on five datasets in the few-shot
scenario to evaluate performance in low-data regimes. Our approach is
computationally efficient during both training and testing. Moreover, it is
independent to prompt-based and LoRa-like methods, allowing for further
exploration in future research. Although our channel selection technique is
straightforward, there is considerable potential to enhance its performance.
Our baseline can be particularly useful in resource-limited situations as it
requires fewer extra parameters and enables rapid adaptation to new tasks with
just a few samples, making the simple and small modules effective knowledge
storage units.

## Acknowledgement

This research is supported by National Research Foundation, Singapore and
A*STAR, under its RIE2020 Industry Alignment Fund – Industry Collaboration
Projects (IAF-ICP) grant call (Grant No. I2001E0059) – SIA-NUS Digital
Aviation Corp Lab. Mike Zheng Shou is supported by the National Research
Foundation, Singapore, under its NRFF Award NRF-NRFF13-2021-0008, and Mike
Zheng Shou’s Start-Up Grant from NUS.

## References

  * Ali et al. (2021) Ali A, Touvron H, Caron M, Bojanowski P, Douze M, Joulin A, Laptev I, Neverova N, Synnaeve G, Verbeek J, et al. (2021) Xcit: Cross-covariance image transformers. In: NeurIPS
  * Bar et al. (2022) Bar A, Gandelsman Y, Darrell T, Globerson A, Efros AA (2022) Visual prompting via image inpainting. arXiv preprint arXiv:220900647
  * Beattie et al. (2016) Beattie C, Leibo JZ, Teplyashin D, Ward T, Wainwright M, Küttler H, Lefrancq A, Green S, Valdés V, Sadik A, et al. (2016) Deepmind lab. arXiv preprint arXiv:161203801
  * Bossard et al. (2014) Bossard L, Guillaumin M, Gool LV (2014) Food-101–mining discriminative components with random forests. In: European conference on computer vision (ECCV), Springer, pp 446–461
  * Cai et al. (2020) Cai H, Gan C, Zhu L, Han S (2020) Tinytl: Reduce memory, not parameters for efficient on-device learning. In: NeurIPS
  * Carion et al. (2020) Carion N, Massa F, Synnaeve G, Usunier N, Kirillov A, Zagoruyko S (2020) End-to-end object detection with transformers. In: European conference on computer vision, Springer, pp 213–229
  * Chen et al. (2021a) Chen CFR, Fan Q, Panda R (2021a) Crossvit: Cross-attention multi-scale vision transformer for image classification. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 357–366
  * Chen et al. (2022a) Chen H, Tao R, Zhang H, Wang Y, Ye W, Wang J, Hu G, Savvides M (2022a) Conv-adapter: Exploring parameter efficient transfer learning for convnets. arXiv preprint arXiv:220807463
  * Chen et al. (2022b) Chen S, Ge C, Tong Z, Wang J, Song Y, Wang J, Luo P (2022b) Adaptformer: Adapting vision transformers for scalable visual recognition. arXiv preprint arXiv:220513535
  * Chen et al. (2021b) Chen X, Xie S, He K (2021b) An empirical study of training self-supervised vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 9640–9649
  * Cheng et al. (2017) Cheng G, Han J, Lu X (2017) Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE 105(10):1865–1883
  * Cimpoi et al. (2014) Cimpoi M, Maji S, Kokkinos I, Mohamed S, , Vedaldi A (2014) Describing textures in the wild. In: Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)
  * Deng et al. (2009) Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: A large-scale hierarchical image database. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Ieee, pp 248–255
  * Dong et al. (2022) Dong X, Bao J, Chen D, Zhang W, Yu N, Yuan L, Chen D, Guo B (2022) Cswin transformer: A general vision transformer backbone with cross-shaped windows. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 12124–12134
  * Dosovitskiy et al. (2020) Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, et al. (2020) An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:201011929
  * d’Ascoli et al. (2021) d’Ascoli S, Touvron H, Leavitt ML, Morcos AS, Biroli G, Sagun L (2021) Convit: Improving vision transformers with soft convolutional inductive biases. In: International Conference on Machine Learning, PMLR, pp 2286–2296
  * Fan et al. (2021) Fan H, Xiong B, Mangalam K, Li Y, Yan Z, Malik J, Feichtenhofer C (2021) Multiscale vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 6824–6835
  * Fei-Fei et al. (2004) Fei-Fei L, Fergus R, Perona P (2004) Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In: conference on computer vision and pattern recognition workshop, IEEE, pp 178–178
  * Geiger et al. (2013) Geiger A, Lenz P, Stiller C, Urtasun R (2013) Vision meets robotics: The kitti dataset. The International Journal of Robotics Research 32(11):1231–1237
  * Han et al. (2021) Han K, Xiao A, Wu E, Guo J, Xu C, Wang Y (2021) Transformer in transformer. In: NeurIPS
  * Han et al. (2015) Han S, Pool J, Tran J, Dally W (2015) Learning both weights and connections for efficient neural network. Advances in neural information processing systems 28
  * He et al. (2018) He Y, Kang G, Dong X, Fu Y, Yang Y (2018) Soft filter pruning for accelerating deep convolutional neural networks. In: IJCAI International Joint Conference on Artificial Intelligence
  * Helber et al. (2019) Helber P, Bischke B, Dengel A, Borth D (2019) Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
  * Hendrycks et al. (2021a) Hendrycks D, Basart S, Mu N, Kadavath S, Wang F, Dorundo E, Desai R, Zhu T, Parajuli S, Guo M, et al. (2021a) The many faces of robustness: A critical analysis of out-of-distribution generalization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp 8340–8349
  * Hendrycks et al. (2021b) Hendrycks D, Zhao K, Basart S, Steinhardt J, Song D (2021b) Natural adversarial examples. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 15262–15271
  * Houlsby et al. (2019) Houlsby N, Giurgiu A, Jastrzebski S, Morrone B, De Laroussilhe Q, Gesmundo A, Attariyan M, Gelly S (2019) Parameter-efficient transfer learning for nlp. In: International Conference on Machine Learning (ICML), PMLR, pp 2790–2799
  * Hu et al. (2021) Hu EJ, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, Wang L, Chen W (2021) Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:210609685
  * Jia et al. (2021) Jia M, Wu Z, Reiter A, Cardie C, Belongie S, Lim SN (2021) Exploring visual engagement signals for representation learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 4206–4217
  * Jia et al. (2022) Jia M, Tang L, Chen BC, Cardie C, Belongie S, Hariharan B, Lim SN (2022) Visual prompt tuning. In: ECCV
  * Jie and Deng (2022) Jie S, Deng ZH (2022) Convolutional bypasses are better vision transformer adapters. arXiv preprint arXiv:220707039
  * Johnson et al. (2017) Johnson J, Hariharan B, Van Der Maaten L, Fei-Fei L, Lawrence Zitnick C, Girshick R (2017) Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 2901–2910
  * Kaggle and EyePacs (2015) Kaggle, EyePacs (2015) Kaggle diabetic retinopathy detection URL https://www.kaggle.com/c/diabetic-retinopathy-detection/data
  * Krause et al. (2013) Krause J, Stark M, Deng J, Fei-Fei L (2013) 3d object representations for fine-grained categorization. In: Proceedings of the IEEE international conference on computer vision workshops, pp 554–561
  * Krizhevsky et al. (2009) Krizhevsky A, Hinton G, et al. (2009) Learning multiple layers of features from tiny images
  * LeCun et al. (2004) LeCun Y, Huang FJ, Bottou L (2004) Learning methods for generic object recognition with invariance to pose and lighting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, vol 2, pp II–104
  * Li et al. (2016) Li H, Kadav A, Durdanovic I, Samet H, Graf HP (2016) Pruning filters for efficient convnets. arXiv preprint arXiv:160808710
  * Li et al. (2017) Li H, Kadav A, Durdanovic I, Samet H, Graf HP (2017) Pruning filters for efficient convnets. In: International Conference on Learning Representations, URL https://openreview.net/forum?id=rJqFGTslg
  * Li et al. (2021) Li Y, Xie S, Chen X, Dollar P, He K, Girshick R (2021) Benchmarking detection transfer learning with vision transformers. arXiv preprint arXiv:211111429
  * Lian et al. (2022) Lian D, Zhou D, Feng J, Wang X (2022) Scaling & shifting your features: A new baseline for efficient model tuning. In: Advances in Neural Information Processing Systems (NeurIPS)
  * Liao et al. (2023) Liao N, Shi B, Cao M, Zhang X, Tian Q, Yan J (2023) Rethinking visual prompt learning as masked visual token modeling. arXiv preprint arXiv:230304998
  * Liu et al. (2022) Liu L, Yu BX, Chang J, Tian Q, Chen CW (2022) Prompt-matched semantic segmentation. arXiv preprint arXiv:220810159
  * Liu et al. (2018) Liu Z, Sun M, Zhou T, Huang G, Darrell T (2018) Rethinking the value of network pruning. In: International Conference on Learning Representations
  * Liu et al. (2021) Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 10012–10022
  * Loshchilov and Hutter (2017) Loshchilov I, Hutter F (2017) Decoupled weight decay regularization. arXiv preprint arXiv:171105101
  * Luo et al. (2017) Luo JH, Wu J, Lin W (2017) Thinet: A filter level pruning method for deep neural network compression. In: Proceedings of the IEEE international conference on computer vision, pp 5058–5066
  * Luo et al. (2022) Luo X, Xu J, Xu Z (2022) Channel importance matters in few-shot image classification. In: International conference on machine learning, PMLR, pp 14542–14559
  * Mahajan et al. (2018) Mahajan D, Girshick R, Ramanathan V, He K, Paluri M, Li Y, Bharambe A, Van Der Maaten L (2018) Exploring the limits of weakly supervised pretraining. In: Proceedings of the European conference on computer vision (ECCV), pp 181–196
  * Maji et al. (2013) Maji S, Rahtu E, Kannala J, Blaschko M, Vedaldi A (2013) Fine-grained visual classification of aircraft. arXiv preprint arXiv:13065151
  * Manli et al. (2022) Manli S, Weili N, De-An H, Zhiding Y, Tom G, Anima A, Chaowei X (2022) Test-time prompt tuning for zero-shot generalization in vision-language models. In: NeurIPS
  * Matthey et al. (2017) Matthey L, Higgins I, Hassabis D, Lerchner A (2017) dsprites: Disentanglement testing sprites dataset
  * Netzer et al. (2011) Netzer Y, Wang T, Coates A, Bissacco A, Wu B, Ng AY (2011) Reading digits in natural images with unsupervised feature learning
  * Nie et al. (2022) Nie X, Ni B, Chang J, Meng G, Huo C, Zhang Z, Xiang S, Tian Q, Pan C (2022) Pro-tuning: Unified prompt tuning for vision tasks. arXiv preprint arXiv:220714381
  * Nilsback and Zisserman (2006) Nilsback ME, Zisserman A (2006) A visual vocabulary for flower classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, vol 2, pp 1447–1454
  * Pan et al. (2022) Pan J, Lin Z, Zhu X, Shao J, Li H (2022) St-adapter: Parameter-efficient image-to-video transfer learning. Advances in Neural Information Processing Systems 35:26462–26477
  * Parkhi et al. (2012) Parkhi OM, Vedaldi A, Zisserman A, Jawahar C (2012) Cats and dogs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp 3498–3505
  * Rao et al. (2021) Rao Y, Zhao W, Liu B, Lu J, Zhou J, Hsieh CJ (2021) Dynamicvit: Efficient vision transformers with dynamic token sparsification. In: NeurIPS
  * Recht et al. (2019) Recht B, Roelofs R, Schmidt L, Shankar V (2019) Do imagenet classifiers generalize to imagenet? In: International Conference on Machine Learning (ICML), PMLR, pp 5389–5400
  * Strudel et al. (2021) Strudel R, Garcia R, Laptev I, Schmid C (2021) Segmenter: Transformer for semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 7262–7272
  * Sung et al. (2022) Sung YL, Cho J, Bansal M (2022) Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 5227–5237
  * Touvron et al. (2021) Touvron H, Cord M, Sablayrolles A, Synnaeve G, Jégou H (2021) Going deeper with image transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 32–42
  * Vaswani et al. (2017) Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: NeurIPS
  * Veeling et al. (2018) Veeling BS, Linmans J, Winkens J, Cohen T, Welling M (2018) Rotation equivariant cnns for digital pathology. In: International Conference on Medical image computing and computer-assisted intervention, Springer, pp 210–218
  * Wang et al. (2019) Wang H, Ge S, Lipton Z, Xing EP (2019) Learning robust global representations by penalizing local predictive power. In: NeurIPS
  * Wang et al. (2021) Wang P, Wang X, Wang F, Lin M, Chang S, Xie W, Li H, Jin R (2021) Kvt: k-nn attention for boosting vision transformers. arXiv preprint arXiv:210600515
  * Wang et al. (2022) Wang S, Chang J, Wang Z, Li H, Ouyang W, Tian Q (2022) Fine-grained retrieval prompt tuning. arXiv preprint arXiv:220714465
  * Xiao et al. (2010) Xiao J, Hays J, Ehinger KA, Oliva A, Torralba A (2010) Sun database: Large-scale scene recognition from abbey to zoo. In: 2010 IEEE computer society conference on computer vision and pattern recognition, IEEE, pp 3485–3492
  * Xing et al. (2022) Xing Y, Wu Q, Cheng D, Zhang S, Liang G, Zhang Y (2022) Class-aware visual prompt tuning for vision-language pre-trained model. arXiv preprint arXiv:220808340
  * Xu et al. (2021) Xu R, Luo F, Zhang Z, Tan C, Chang B, Huang S, Huang F (2021) Raise a child in large language model: Towards effective and generalizable fine-tuning. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics
  * Yang et al. (2021) Yang J, Zhou K, Li Y, Liu Z (2021) Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:211011334
  * Yang et al. (2022a) Yang J, Wang P, Zou D, Zhou Z, Ding K, Peng W, Wang H, Chen G, Li B, Sun Y, et al. (2022a) Openood: Benchmarking generalized out-of-distribution detection. arXiv preprint arXiv:221007242
  * Yang et al. (2022b) Yang J, Zhou K, Liu Z (2022b) Full-spectrum out-of-distribution detection. arXiv preprint arXiv:220405306
  * Yuan et al. (2021) Yuan L, Chen Y, Wang T, Yu W, Shi Y, Jiang ZH, Tay FE, Feng J, Yan S (2021) Tokens-to-token vit: Training vision transformers from scratch on imagenet. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 558–567
  * Yuan et al. (2022) Yuan L, Hou Q, Jiang Z, Feng J, Yan S (2022) Volo: Vision outlooker for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence
  * Zang et al. (2022) Zang Y, Li W, Zhou K, Huang C, Loy CC (2022) Unified vision and language prompt learning. arXiv preprint arXiv:221007225
  * Zhai et al. (2019) Zhai X, Puigcerver J, Kolesnikov A, Ruyssen P, Riquelme C, Lucic M, Djolonga J, Pinto AS, Neumann M, Dosovitskiy A, et al. (2019) A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:191004867
  * Zhang et al. (2023a) Zhang B, Jin X, Gong W, Xu K, Zhang Z, Wang P, Shen X, Feng J (2023a) Multimodal video adapter for parameter efficient video text retrieval. arXiv preprint arXiv:230107868
  * Zhang et al. (2022) Zhang Y, Zhou K, Liu Z (2022) Neural prompt search. arXiv preprint arXiv:220604673
  * Zhang et al. (2023b) Zhang Y, Zhou K, Liu Z (2023b) What makes good examples for visual in-context learning?
  * Zhao et al. (2022) Zhao Y, Zhong Z, Zhao N, Sebe N, Lee GH (2022) Style-hallucinated dual consistency learning for domain generalized semantic segmentation. In: ECCV
  * Zheng et al. (2022) Zheng Z, Yue X, Wang K, You Y (2022) Prompt vision transformer for domain generalization. arXiv preprint arXiv:220808914
  * Zhou et al. (2021a) Zhou J, Wang P, Wang F, Liu Q, Li H, Jin R (2021a) Elsa: Enhanced local self-attention for vision transformer. arXiv preprint arXiv:211212786
  * Zhou et al. (2021b) Zhou K, Yang Y, Qiao Y, Xiang T (2021b) Domain generalization with mixstyle. In: ICLR
  * Zhou et al. (2022a) Zhou K, Yang J, Loy CC, Liu Z (2022a) Conditional prompt learning for vision-language models. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  * Zhou et al. (2022b) Zhou K, Yang J, Loy CC, Liu Z (2022b) Learning to prompt for vision-language models. International Journal of Computer Vision (IJCV)
  * Zhou et al. (2022c) Zhou K, Zhang Y, Zang Y, Yang J, Loy CC, Liu Z (2022c) On-device domain generalization. arXiv preprint arXiv:220907521