# Sparse-Tuning: Adapting Vision Transformers with Efficient Fine-tuning and
Inference

Ting Liu1 Xuyang Liu2∗ Liangtao Shi3 Zunnan Xu4
Siteng Huang5 Yi Xin6 Quanjun Yin1

1National University of Defense Technology 2Sichuan University 3Hefei
University of Technology
4Tsinghua University 5Zhejiang University 6Nanjing University
<EMAIL_ADDRESS><EMAIL_ADDRESS>Equal contribution.

###### Abstract

Parameter-efficient fine-tuning (PEFT) has emerged as a popular approach for
adapting pre-trained Vision Transformer (ViT) models to downstream
applications. While current PEFT methods achieve parameter efficiency, they
overlook GPU memory and time efficiency during both fine-tuning and inference,
due to the repeated computation of redundant tokens in the ViT architecture.
This falls short of practical requirements for downstream task adaptation. In
this paper, we propose Sparse-Tuning, a novel tuning paradigm that
substantially enhances both fine-tuning and inference efficiency for pre-
trained ViT models. Sparse-Tuning efficiently fine-tunes the pre-trained ViT
by sparsely preserving the informative tokens and merging redundant ones,
enabling the ViT to focus on the foreground while reducing computational costs
on background regions in the images. To accurately distinguish informative
tokens from uninformative ones, we introduce a tailored Dense Adapter, which
establishes dense connections across different encoder layers in the ViT,
thereby enhancing the representational capacity and quality of token
sparsification. Empirical results on VTAB-1K, three complete image datasets,
and two complete video datasets demonstrate that Sparse-Tuning reduces the
GFLOPs to 62%-70% of the original ViT-B while achieving state-of-the-art
performance. Source code is available at https://github.com/liuting20/Sparse-
Tuning.

Figure 1: Comparisons of Sparse-Tuning with other mainstream PEFT methods on
CIFAR-100. Sparse-Tuning enhances performance while remarkably reducing
training /inference time, GPU memory consumption, and computational cost,
achieving both fine-tuning and inference efficiency.

##  1 Introduction

Large-scale Vision Transformer (ViT) models [10, 41, 40, 48, 30] have
demonstrated strong generalization capabilities across a wide range of
downstream vision tasks. The prevalent approach to adapt these models for
specific tasks follows the pretrain-then-finetune paradigm, where models are
initially pre-trained on large-scale datasets [8, 38, 32] and then fine-tuned
for each downstream task. However, as these pre-trained ViT models continue to
scale up [57, 7], fully fine-tuning them becomes more computationally
intensive. Additionally, there are risks of catastrophic forgetting and
overfitting when fine-tuning on limited downstream datasets [59, 24, 9].

Recently, various parameter-efficient fine-tuning (PEFT) methods [21, 34, 22,
25, 5] have been proposed to address the high computational costs and risks
associated with fully fine-tuning large models. By updating additional
parameters inserted into the model [21, 5] or appended to the input data [34,
25], PEFT methods can achieve similar or even better performance compared to
full fine-tuning. Current PEFT methods [29, 26, 12, 44] focus primarily on
efficiently fine-tuning a pre-trained ViT for downstream visual tasks.
However, despite achieving fine-tuning efficiency, they fall short in
inference efficiency due to the repeated computation of redundant tokens in
the Multi-Head Attention (MHA) and Feed-Forward Network (FFN) blocks, failing
to meet practical requirements.

Inspired by the model acceleration methods for ViT [49, 51, 36, 37], we aim to
reduce redundant tokens to enhance efficiency during both the fine-tuning and
inference stages. Since most acceleration methods progressively prune tokens
across encoder layers in ViT, we consider establishing interactions between
shallow encoder layers, which capture local information, and deep encoder
layers, which capture global information. By supplementing global information
with local details, we aim to accurately discriminate between informative and
uninformative tokens. To this end, in this paper, we introduce Sparse-Tuning,
a novel tuning paradigm that achieves efficient fine-tuning and inference for
ViT adaptation. Sparse-Tuning efficiently fine-tunes the pre-trained ViT by
sparsely adapting the tokens to enable the model to focus on informative
regions, thereby enhancing both computational and memory efficiency during the
fine-tuning and inference stages. Specifically, we design a novel parameter-
free Token Sparsification method that allows the pre-trained ViT to
progressively preserve informative tokens while integrating uninformative ones
into a representative token, thereby reducing redundant computational costs.
Moreover, to mitigate information loss from Token Sparsification and
efficiently fine-tune the pre-trained ViT, we propose Dense Adapters, which
take multiple features from different encoder layers as inputs to establish
dense connections between multiple Token Sparsification steps. With these non-
trivial designs, Sparse-Tuning enhances performance while reducing training
and inference time, GPU memory consumption, and computational cost for ViT
adaptation, compared to most PEFT methods [21, 22, 5, 25, 60], as shown in
Figure 1.

To fully evaluate the generalization capability, we conduct extensive
experiments on the common PEFT benchmark VTAB-1K [58], three complete image
datasets: CIFAR-100 [33], SVHN [13], and Food-101 [2], as well as two complete
video datasets: Kinetics-400 (K400) [4] and Something-Something V2 (SSv2)
[14]. Empirical results on VTAB-1K demonstrate that with only 11.65 GFLOPs,
approximately 66% of the computational cost of the original ViT-B, Sparse-
Tuning outperforms all state-of-the-art methods in performance, fine-tuning
and inference efficiency. Moreover, Sparse-Tuning achieves superior
performance in both image and video recognition on complete datasets while
significantly improving both fine-tuning and inference efficiency.

##  2 Related Work

###  2.1 Parameter-efficient Fine-tuning

With the trend of scaling up Vision Transformers (ViT) [10, 18, 57, 7] for
enhanced performance and generalization, adapting the entire model to
downstream tasks becomes increasingly computationally expensive. To mitigate
this, parameter-efficient fine-tuning (PEFT) [21, 34, 22, 25, 55, 54] emerges
as a strategic approach. PEFT methods updates only a small subset of
additional parameters while keeping the majority of the pre-trained model
frozen, thereby mitigating the risks of catastrophic forgetting and
overfitting. Most PEFT methods methods designed for Transformer [50] can be
classified into three types: (1) Partially Tuning [31, 56, 11, 17], that only
updates a small subset of inherent parameters while freezing most original
parameters. (2) Prompt Tuning [34, 25, 24, 53], that integrates a fixed length
of learnable tokens (i.e., prompts) appended with the input data. It only
updates the prompts during fine-tuning. (3) Adapter Tuning [21, 5, 52, 39],
that only updates additional parameters in the module inserted into the model
(i.e., Adapter) during fine-tuning.

While most PEFT methods improve parameter efficiency during the fine-tuning
stage of ViT, they often introduce new parameters that hinder inference
efficiency. Reparameterization methods, such as LoRA [22] and FacT [29],
introduce learnable parameters that can be integrated into the original model
through reparameterization during inference. Therefore, these methods can
maintain the original model’s inference efficiency. However, current PEFT
methods fail to enhance inference efficiency while achieving parameter-
efficient fine-tuning, which does not meet the practical needs for adapting
the large-size ViT (e.g., ViT-L [10]). In this paper, we aim to enhance the
efficiency of both the fine-tuning and inference processes for the pre-trained
ViT.

###  2.2 Model Acceleration for ViT

Recently, numerous works have explored to accelerate the inference efficiency
of ViT [15, 49, 36, 6, 37], with most of them aiming to reduce the token
redundancy in ViT to decrease computational complexity. For instance,
DynamicViT [49] efficiently sparsifies ViT by pruning less informative tokens
identified through prediction modules. DVT [51] enhances computational
efficiency by automatically adapting the token count for each input image.
SuperViT [37] handles diverse patch sizes with a single model, adaptively
adjusting token retention during inference.

Existing ViT acceleration methods generally require either fine-tuning all
pre-trained parameters [49, 51], or training models from scratch [36, 37].
Consequently, these approaches necessitate substantial training or fine-tuning
time to adapt ViT to downstream vision tasks. Recently, Dynamic Tuning (DyT)
[60] keeps the pre-trained ViT parameters frozen, updating only the adapters
and token dispatchers to enhance parameter efficiency and reduce redundant
computation during inference. Unlike DyT, which directly skips the
uninformative tokens, our method consolidates these tokens into a single
representative token to retain visual features beneficial for classification.
Additionally, unlike DyT, which necessitates computing all tokens to update
the parameters of the proposed token dispatchers, our approach does not
introduce any additional modules for token sparsification. Our method
efficiently fine-tunes the pre-trained ViT by selectively adapting the tokens,
thereby improving efficiency during both the fine-tuning and inference stages.

##  3 Method

In this section, we present our proposed Sparse-Tuning in detail. First, we
briefly review the preliminaries of Vision Transformer and Adapter Tuning. in
Section 3.1. Next, we provide a general introduction to the overall framework
of Sparse-Tuning in Section 3.2. Following this, we elaborate on the core
techniques of Sparse-Tuning: Token Sparsification and Dense Adapter.

###  3.1 Preliminaries

Vision Transformers (ViTs) [10] basically consist of a patch embedding layer
and a stack of transformer encoder layers. The patch embedding layer first
splits and flattens an input image $\mathit{x}\in\mathbb{R}^{H\times W\times
3}$ into a sequence of patches
$\mathit{x}_{p}\in\mathbb{R}^{N\times(P^{2}\cdot C)}$, where $(H,W)$ denotes
the size of the input image, $(P,P)$ represents the size of each image patch,
$C$ is the number of channels, and $N={H\cdot W}/{P^{2}}$ is the number of
image tokens. The patches $\mathit{x}_{p}$, prepended with a learnable [CLS]
token, are fed into a stack of transformer encoder layers, each of which
includes a Multi-Head Attention (MHA) block and a Feed-Forward Network (FFN).
In MHA, the tokens are linearly projected and packed into three vectors,
namely $\bm{Q}$, $\bm{K}$, and $\bm{V}$. The self-attention operation can be
written as:

${\rm Attention}(\bm{Q},\bm{K},\bm{V})={\rm
Softmax}(\frac{\bm{Q}\bm{K}^{\top}}{\sqrt{d}})\bm{V},$ (1)

${\rm Softmax}(\frac{\bm{Q}\bm{K}^{\top}}{\sqrt{d}})$ is the attention map,
where $\bm{Q}\bm{K}^{\top}$ indicates the attention from the [CLS] token to
all tokens and reflects the importance of each token. Subsequently, the output
tokens are sent to a Layer Normalization (LayerNorm) [1] and a FFN, which
consists of two fully-connected layers with a GELU activation function [20] in
between. After processing the tokens by a stack of encoder layers, the [CLS]
token is extracted and utilized for classification.

Adapter Tuning is a prevalent strategy for efficient fine-tuning of ViT [5,
45, 27], typically involving the insertion of an MLP in parallel with the FFN.
The adapters consist of a down-projection layer $\mathbf{W}_{\text{down}}$,
ReLU non-linear activation, and an up-projection layer
$\mathbf{W}_{\text{up}}$ in sequence. Given the input feature $x$, the
function of a standard adapter can be formally expressed as:

${\rm Adapter}(x)=x+s\cdot{{\rm
ReLU}(x\mathbf{W}_{\text{down}})}\mathbf{W}_{\text{up}},$ (2)

where $s$ denotes the scaling factor. Unlike the standard adapter, in this
paper, we introduce the Dense Adapter, which receives multiple adapted
features from different encoder layers to establish connections across the
encoder layers in ViT.

###  3.2 Sparse-Tuning for Efficient ViT adaptation

Existing works [49, 51, 36] have demonstrated that the final prediction in ViT
largely depends on a subset of the most informative tokens. Dynamic Tuning
(DyT) [60] keeps the pre-trained parameters frozen and updates the adapters
with the proposed token dispatcher to distinguish and discard uninformative
tokens. This design can improve the inference speed of ViT but suffer from two
main shortcomings: (1) Inefficient fine-tuning, as DyT requires gradients to
backpropagate through all tokens to update the parameters of the proposed
token dispatcher, thus leading to low efficiency in terms of GPU memory
consumption and fine-tuning speed. (2) Information loss, as the token
dispatcher directly removes those inactivated tokens, which can lead to a
direct loss of information, thereby deteriorating the classification accuracy.

Figure 2: Overall framework of Sparse-Tuning. We freeze the pre-trained
ViT-B/16 and update the proposed Dense Adapters (DAs) to efficiently fine-tune
the pre-trained ViT. By selectively adapting tokens to focus on informative
regions, Sparse-Tuning significantly reduces the computational cost of
redundant tokens, thereby enhancing efficiency during both fine-tuning and
inference stages.

Motivated by the above analysis, we introduce Sparse-Tuning with Dense
Adapters, efficiently fine-tuning pre-trained ViT by selectively adapting
tokens to focus on informative regions, enhancing efficiency during both fine-
tuning and inference stages. As shown in Figure 2, the overall framework
includes two parts: (1) a pre-trained ViT-B/16 [10] that consists of a patch
embedding layer and 12 transformer encoder layers with our carefully designed
Token Sparsification process, and (2) our Dense Adapters. During fine-tuning,
we freeze the pre-trained ViT and only update a series of Dense Adapters to
facilitate efficient adaptation to downstream tasks. In the 4th, 7th, and 10th
encoder layers (we conduct relevant analysis on Table 6), we implement Token
Sparsification (see Figure 3) to enable ViT focus more on the informative
tokens and reduce redundant computation cost.

Token Sparsification. The main idea of Sparse-Tuning is to decrease the
computation load on uninformative tokens, which in turn reduces the
computational cost for both ViT and Dense Adapters during fine-tuning and
inference, thereby improving overall efficiency and speed. An intriguing
question arises: how to distinguish the informative tokens from other tokens
with less information? Previous works [3, 47, 16] have demonstrated the strong
relationship between the [CLS] token and the class-specific tokens. In other
words, the attention scores between the [CLS] token and other tokens reflect
the contribution of the current token to the classification. Consequently,
tokens that exhibit higher/lower attention scores with the [CLS] token contain
more/less semantic information for classification, and thus can be viewed as
the attentive/inattentive tokens. Though the inattentive tokens show lower
attention scores, they may still influence the classification results in some
cases, such as the prediction of large objects which cover large regions of
the image. To this end, unlike DyT [60], our Token Sparsification
progressively preserves the attentive tokens and merges the inattentive ones
into one token during fine-tuning and inference to reduce the computation
cost. Specifically, as shown in Figure 2, we calculate the average attention
scores of all heads in MHA, and preserve the k largest (i.e., top-k) elements
corresponding to tokens (attentive tokens), and fuse the rest tokens
(inattentive tokens) by weighted average into a representative token to
supplement the attentive ones. With this design, Sparse-Tuning allows a pre-
trained ViT to concentrate on the most informative regions while discarding
the uninformative ones, consequently lowering redundant computational costs
during both fine-tuning and inference. Furthermore, by integrating the
inattentive tokens, Sparse-Tuning mitigates information loss resulting from
Token Sparsification.

Figure 3: Token Sparsification by Sparse-Tuning during fine-tuning and
inference on ViT-B/16. Sparse-Tuning enables a pre-trained ViT to gradually
focus on the objects in the images while merging the uninformative regions
(masked regions).

Dense Adapter. To further alleviate the information loss caused by Token
Sparsification and efficiently adapt the pre-trained ViT for downstream tasks,
we consider utilizing the adpter-tuning method. Most current adapter-tuning
methods for ViT [5, 26] follow the basic residual connection approach from
ResNet [19], which can only establish connections between two adjacent ViT
encoder layers, greatly limiting the propagation of adapted features during
fine-tuning. The transition from local features to global features across
encoder layers in ViT affects the effectiveness of Token Sparsification. Given
that Token Sparsification occurs across encoder layers in ViT, we introduce
the Dense Adapter (DA), inspired by DenseNet [23], to establish dense
connections across multiple encoder layers. As shown in Figure 2 (right),
unlike the standard adapter [21], DA takes multiple features from different
encoder layers as inputs to establish interactions between multiple Token
Sparsification steps, thereby compensating for the information loss caused by
Token Sparsification.

According to the position, DA consists of one to three down-projection layers
(i.e., $\mathbf{W}_{\text{down}_{1}}$, $\mathbf{W}_{\text{down}_{2}}$,
$\mathbf{W}_{\text{down}_{3}}$), ReLU non-linear activation, and an up-
projection layer $\mathbf{W}_{\text{up}}$. Specifically, we donate the N-th DA
as $\text{DA}_{\text{N}}$. The output of $\text{DA}_{\text{N}}$ can be
formulated as:

$x_{\text{DA}_{\text{N}}}=\begin{cases}{{\rm
ReLU}(x_{\text{MHA}_{\text{N}}}\mathbf{W}_{\text{down}_{1}})}\mathbf{W}_{\text{up}}&\text{
if }\text{N}=1\\\ {{\rm
ReLU}(x_{\text{MHA}_{\text{N}}}\mathbf{W}_{\text{down}_{1}}+x_{\text{DA}_{\text{N-1}}}\mathbf{W}_{\text{down}_{2}}})\mathbf{W}_{\text{up}}&\text{
if }1<\text{N}\leq 3\\\ {{\rm
ReLU}(x_{\text{MHA}_{\text{N}}}\mathbf{W}_{\text{down}_{1}}+x_{\text{DA}_{\text{N-1}}}\mathbf{W}_{\text{down}_{2}}}+x_{\text{DA}_{\text{N-3}}}\mathbf{W}_{\text{down}_{3}})\mathbf{W}_{\text{up}}&\text{
if }3<\text{N}\leq 12\\\ \end{cases},$ (3)

where $x_{\text{DA}_{\text{N}}}$ and $x_{\text{MHA}_{\text{N}}}$ represent the
outputs of the $\text{DA}_{\text{N}}$ and MHA at the N-th encoder layer,
respectively. It is noteworthy that when $3<N\leq 12$, the
$x_{\text{DA}_{\text{N-1}}}$ and $x_{\text{DA}_{\text{N-3}}}$ may be
sparsified by the corresponding Token Sparsification step to ensure
consistency in token length for $x_{\text{DA}_{\text{N}}}$,
$x_{\text{DA}_{\text{N-1}}}$, and $x_{\text{DA}_{\text{N-3}}}$. Dense Adapters
facilitate multiple interactions between the lower and higher layers of the
ViT encoder, thereby enhancing the representational capacity and quality of
Token Sparsification.

##  4 Experiments

###  4.1 Experimental Setup

Datasets. We compare our Sparse-Tuning with other state-of-the-art methods on
the common PEFT benchmark VTAB-1K [58] to evaluate the adaptation performance
when the training data is limited. For each downstream classification task,
the training data in VTAB-1K [58] is extremely scarce, comprising only 1,000
training samples. Thus, following [5, 60], we conduct experiments on three
complete image datasets: CIFAR-100 [33], SVHN [13], and Food-101 [2], as well
as two complete video datasets: Kinetics-400 (K400) [4] and Something-
Something V2 (SSv2) [14], to further evaluate the adaptation performance and
efficiency of our Sparse-Tuning.

Implementation Details. We utilize the ViT-Base (ViT-B/16) model [10] as our
backbone, which is pre-trained on the ImageNet21K dataset [8] under full
supervision. The bottleneck dimension $d$ of our Dense Adapter is set to 32 by
default, and we reduce $d$ to 8 on VTAB-1K, following most existing works [5,
60]. The scaling factor $s$ is set to 1. We set the keeping rate $r$ of
attentive tokens to 0.7 by default, unless otherwise specified. We adhere to
the same training schedule as reported in [5, 60]. For all the downstream
tasks, we employ top-1 accuracy as the primary evaluation metric. We conduct
all experiments on a A800 GPU. More details are provided in the Appendix C.

Table 1:  Comparison to state-of-the-art PEFT methods on VTAB-1K with ViT-B/16. Group Mean: the average accuracy of the three subgroups. Params.: the number of learnable parameters excluding the final classification layer. GFLOPs: the average GFLOPs across all datasets. $r$ denotes the keeping rate of attentive (activated) tokens. We highlight the best and the second-best results.  | Natural | Specialized | Structured |  |  |
---|---|---|---|---|---|---
|  CIFAR-100 |  Caltech101 |  DTD |  Flowers102 |  Pets |  SVHN |  Sun397 |  Camelyon |  EuroSAT |  Resisc45 |  Retinopathy |  Clevr-Count |  Clevr-Dist |  DMLab |  KITTI-Dist |  dSpr-Loc |  dSpr-Ori |  sNORB-Azim |  sNORB-Elev |  Group Mean |  Params. (M) $\downarrow$ |  GFLOPs $\downarrow$
_Traditional fine-tuning_
Full fine-tunig | 68.9 | 87.7 | 64.3 | 97.2 | 86.9 | 87.4 | 38.8 | 79.7 | 95.7 | 84.2 | 73.9 | 56.3 | 58.6 | 41.7 | 65.5 | 57.5 | 46.7 | 25.7 | 29.1 | 68.96 | 85.84 | 17.58
Linear | 63.4 | 85.0 | 63.2 | 97.0 | 86.3 | 36.6 | 51.0 | 78.5 | 87.5 | 68.6 | 74.0 | 34.3 | 30.6 | 33.2 | 55.4 | 12.5 | 20.0 | 9.6 | 19.2 | 57.64 | 0 | 17.58
_Parameter-efficient fine-tuning_
Adapter [21] | 69.2 | 90.1 | 68.0 | 98.8 | 89.9 | 82.8 | 54.3 | 84.0 | 94.9 | 81.9 | 75.5 | 80.9 | 65.3 | 48.6 | 78.3 | 74.8 | 48.5 | 29.9 | 41.6 | 73.85 | 0.16 | 17.61
BitFit [56] | 72.8 | 87.0 | 59.2 | 97.5 | 85.3 | 59.9 | 51.4 | 78.7 | 91.6 | 72.9 | 69.8 | 61.5 | 55.6 | 32.4 | 55.9 | 66.6 | 40.0 | 15.7 | 25.1 | 65.21 | 0.10 | 17.58
LoRA [22] | 67.1 | 91.4 | 69.4 | 98.8 | 90.4 | 85.3 | 54.0 | 84.9 | 95.3 | 84.4 | 73.6 | 82.9 | 69.2 | 49.8 | 78.5 | 75.7 | 47.1 | 31.0 | 44.0 | 74.60 | 0.29 | 17.58
VPT [25] | 78.8 | 90.8 | 65.8 | 98.0 | 88.3 | 78.1 | 49.6 | 81.8 | 96.1 | 83.4 | 68.4 | 68.5 | 60.0 | 46.5 | 72.8 | 73.6 | 47.9 | 32.9 | 37.8 | 71.96 | 0.53 | 18.30
SSF [35] | 69.0 | 92.6 | 75.1 | 99.4 | 91.8 | 90.2 | 52.9 | 87.4 | 95.9 | 87.4 | 75.5 | 75.9 | 62.3 | 53.3 | 80.6 | 77.3 | 54.9 | 29.5 | 37.9 | 75.69 | 0.20 | 17.58
NOAH [59] | 69.6 | 92.7 | 70.2 | 99.1 | 90.4 | 86.1 | 53.7 | 84.4 | 95.4 | 83.9 | 75.8 | 82.8 | 68.9 | 49.9 | 81.7 | 81.8 | 48.3 | 32.8 | 44.2 | 75.48 | 0.36 | 17.58
ConvPass [27] | 72.3 | 91.2 | 72.2 | 99.2 | 90.9 | 91.3 | 54.9 | 84.2 | 96.1 | 85.3 | 75.6 | 82.3 | 67.9 | 51.3 | 80.0 | 85.9 | 53.1 | 36.4 | 44.4 | 76.56 | 0.33 | 17.64
AdaptFormer [5] | 70.8 | 91.2 | 70.5 | 99.1 | 90.9 | 86.6 | 54.8 | 83.0 | 95.8 | 84.4 | 76.3 | 81.9 | 64.3 | 49.3 | 80.3 | 76.3 | 45.7 | 31.7 | 41.1 | 74.75 | 0.16 | 17.61
FacT [29] | 71.3 | 89.6 | 70.7 | 98.9 | 91.0 | 87.8 | 54.6 | 85.2 | 95.5 | 83.4 | 75.7 | 82.0 | 69.0 | 49.8 | 80.0 | 79.2 | 48.4 | 34.2 | 41.4 | 75.30 | 0.04 | 17.58
Res-Tuning [26] | 75.2 | 92.7 | 71.9 | 99.3 | 91.9 | 86.7 | 58.5 | 86.7 | 95.6 | 85.0 | 74.6 | 80.2 | 63.6 | 50.6 | 80.2 | 85.4 | 55.7 | 31.9 | 42.0 | 76.32 | 0.51 | 17.67
DyT $r=0.5$ [60] | 70.4 | 94.2 | 71.1 | 99.1 | 91.7 | 88.0 | 51.5 | 87.1 | 95.3 | 84.2 | 75.8 | 79.2 | 61.8 | 51.0 | 82.4 | 79.7 | 52.3 | 35.3 | 44.5 | 75.73 | 0.16 | 12.54
DyT $r=0.7$ [60] | 73.9 | 94.9 | 72.1 | 99.4 | 91.8 | 88.4 | 55.5 | 87.2 | 95.6 | 86.2 | 75.9 | 80.3 | 61.8 | 51.7 | 83.1 | 81.6 | 53.7 | 35.3 | 45.2 | 76.69 | 0.16 | 14.92
DyT $r=0.9$ [60] | 74.0 | 95.1 | 72.9 | 99.3 | 91.7 | 87.6 | 56.9 | 87.7 | 95.7 | 85.4 | 76.1 | 81.6 | 63.2 | 50.1 | 83.0 | 83.3 | 52.0 | 34.5 | 44.5 | 76.74 | 0.16 | 17.07
_The proposed Sparse-Tuning_
Sparse-Tuning $r=0.5$ | 70.6 | 94.6 | 71.5 | 99.3 | 91.9 | 88.5 | 51.9 | 87.6 | 95.7 | 84.7 | 75.6 | 79.9 | 62.3 | 51.9 | 82.7 | 80.1 | 52.9 | 35.8 | 44.7 | 76.54 | 0.32 | 8.94
Sparse-Tuning $r=0.7$ | 74.2 | 95.1 | 72.5 | 99.6 | 92.2 | 90.3 | 55.8 | 87.7 | 96.3 | 86.7 | 76.2 | 81.7 | 62.6 | 52.6 | 83.8 | 82.3 | 55.3 | 36.9 | 45.8 | 77.71 | 0.32 | 11.65
Sparse-Tuning $r=0.9$ | 74.8 | 95.5 | 73.2 | 99.4 | 91.7 | 88.1 | 58.7 | 88.2 | 96.4 | 85.8 | 76.4 | 82.9 | 64.7 | 50.7 | 83.4 | 83.9 | 53.7 | 35.2 | 45.2 | 77.92 | 0.32 | 15.62

###  4.2 Main Results

Comparisons on VTAB-1K. The comparison results with state-of-the-art (SOTA)
PEFT methods on VTAB-1K [58] are presented in Table 1, from which we can
observe that: (1) Sparse-Tuning outperforms all SOTA PEFT methods. Sparse-
Tuning achieves a 1.18% improvement in terms of the average accuracy across
the three subgroups, compared with the previous best model DyT [60]. (2)
Sparse-Tuning largely improves inference efficiency. With only 11.65 GFLOPs,
about 66% of the computational cost of the original ViT-B, Sparse-Tuning with
keeping rate $r=0.7$ has outperformed all state-of-the-art methods in terms of
both performance and inference efficiency. (3) Sparse-Tuning continues to
exhibit better performance when the keeping rate $r$ increases. While even the
Sparse-Tuning with $r=0.5$ can outperform recent strong methods, such as Res-
Tuning [26] and FacT [29], which validates the effectiveness and efficiency of
our Sparse-Tuning.

Table 2: Results on complete image and video datasets. Avg.: the mean value derived from the corresponding results across various image and video datasets. Params.: the number of learnable parameters excluding the final classification layer. The GFLOPs are evaluated on CIFAR-100 and K400. DyT${\dagger}$ $N=4$ represents DyT with four experts. Method |  Params. $\downarrow$ | Image Datasets | Video Datasets
---|---|---|---
(M) |  GFLOPs $\downarrow$ | CIFAR-100 | SVHN | Food-101 | Avg. |  GFLOPs $\downarrow$ | K400 | SSv2 | Avg.
Traditional fine-tuning
Full fine-tuning | 85.80 | 17.58 | 90.91 | 97.29 | 90.69 | 92.69 | 142.53 | 75.48 | 75.22 | 60.35
Linear | 0 | 17.58 | 85.87 | 56.29 | 88.07 | 76.74 | 142.53 | 69.04 | 27.64 | 48.34
Parameter-efficient fine-tuning
Adapter [21] | 1.19 | 17.81 | 91.76 | 96.88 | 89.91 | 92.76 | 144.39 | 74.72 | 44.58 | 59.75
AdaptFormer [5] | 1.19 | 17.81 | 92.03 | 97.23 | 90.84 | 93.36 | 144.39 | 75.53 | 45.36 | 60.45
LoRA [22] | 1.19 | 17.58 | 91.42 | 97.36 | 90.48 | 93.08 | 142.53 | 75.48 | 45.62 | 60.55
VPT [25] | 0.07 | 18.32 | 91.64 | 95.72 | 90.41 | 92.59 | 148.44 | 73.46 | 38.17 | 55.82
DyT [60] | 1.19 | 12.21 | 91.37 | 97.08 | 90.32 | 92.92 | 108.31 | 74.39 | 45.34 | 59.87
DyT${\dagger}$ $N=4$ [60] | 4.80 | 12.29 | 91.01 | 96.90 | 89.77 | 92.56 | 105.45 | 75.00 | 46.56 | 60.78
The proposed Sparse-Tuning
Sparse-Tuning | 1.10 | 11.70 | 92.31 | 97.47 | 90.72 | 93.50 | 99.8 | 75.55 | 46.67 | 61.11

Comparisons on Complete Datasets. We conduct experiments on comprehensive
image and video datasets to evaluate the adaptation performance with abundant
training data. The results on complete image and video datasets are shown in
Table 2, from which we find that: (1) Sparse-Tuning outperforms all baseline
methods on both image and video datasets, demonstrating its strong
transferability on complete datasets. (2) Sparse-Tuning demonstrates
exceptional inference efficiency on both image and video datasets.
Particularly on video datasets, Sparse-Tuning reduces the computational
complexity of the original ViT-B by around 30%, highlighting its strong
efficiency in video applications. With only 1.11M updated parameters, our
Sparse-Tuning achieves superior performance in image and video recognition,
while significantly improving inference efficiency.

Table 3: Ablation on different components of Sparse-Tuning. Without any component of Sparse-Tuning, it can be viewed as freezing the pre-trained ViT, and only fine-tuning the final classification layer. Params.: learnable parameters excluding the final classification layer. # | Token | Dense | Params. (M)$\downarrow$ | GFLOPs$\downarrow$ | CIFAR-100 | SVHN | Food-101 | Avg.
---|---|---|---|---|---|---|---|---
Sparsification | Adapters
(a) |  |  | 0 | 17.58 | 85.87 | 56.29 | 88.07 | 76.74
(b) | ✓ |  | 0 | 11.81 | 76.59 | 48.81 | 78.50 | 67.97
(c) |  | ✓ | 1.10 | 17.89 | 92.66 | 97.93 | 91.34 | 93.98
(d) | ✓ | ✓ | 1.10 | 11.70 | 92.31 | 97.47 | 90.72 | 93.50

###  4.3 Ablation Studies

In this subsection, we first analyze the effectiveness of Token Sparsification
and Dense Adapters. We then provide an in-depth analysis of the feature inputs
and their fusion methods in our Dense Adapters. Subsequently, we investigate
the impact of different positions of Token Sparsification to achieve optimal
performance. Finally, we verify the effectiveness of Sparse-Tuning when the
pre-trained ViT is scaled up. We conduct all ablation studies on three
complete image datasets.

Components Effectiveness. In Table 3, we report the performance of using
different components of Sparse-Tuning to investigate the effectiveness of
Token Sparsification and Dense Adapters. We can observe the following: (1)
Token Sparsification can reduce the computational complexity, but it leads to
a significant performance degradation, resulting in a 7% decrease in average
accuracy (Table 3 (a,b)). (2) Dense Adapters can significantly improve the
performance across three datasets (Table 3 (a,c)), which demonstrates their
effectiveness in ViT adaptation. (3) Sparse-Tuning incorporates Token
Sparsification and Dense Adapters into the pre-trained ViT, achieving the best
trade-off between performance and fine-tuning and inference efficiency (Table
3 (a,b,c,d)). Compared to using only Dense Adapters for efficient ViT
adaptation (Table 3 (b)), Sparse-Tuning sacrifices only 0.48% average accuracy
while significantly reducing the computational cost from 17.89 GFLOPs to 11.70
GFLOPs, highlighting its strong adaptation performance and efficiency.

Table 4: Comparison of different feature inputs. $\text{MHA}_{\text{N}}$, $\text{DA}_{\text{N-1}}$, and $\text{DA}_{\text{N-3}}$ represent the outputs of the MHA at the N-th encoder layer, the outputs of the $\text{DA}_{\text{N-1}}$ and $\text{DA}_{\text{N-3}}$, respectively. # | $\text{MHA}_{\text{N}}$ | $\text{DA}_{\text{N-1}}$ | $\text{DA}_{\text{N-3}}$ |  Params. (M)$\downarrow$ |  GFLOPs$\downarrow$ | CIFAR-100 | SVHN | Food-101 | Avg.
---|---|---|---|---|---|---|---|---|---
(a) | ✓ |  |  | 0.69 | 11.19 | 91.54 | 96.67 | 89.94 | 92.72
(b) | ✓ | ✓ |  | 0.95 | 11.67 | 91.95 | 97.03 | 90.65 | 93.21
(c) | ✓ |  | ✓ | 0.95 | 11.67 | 91.61 | 96.96 | 90.71 | 93.09
(d) | ✓ | ✓ | ✓ | 1.10 | 11.70 | 92.31 | 97.47 | 90.72 | 93.50

Effects of Different Feature Inputs. To investigate the effectiveness of dense
connections, we compare different inputs to our Dense Adapters. As shown in
Table 4, when feeding multiple features from different encoder layers into the
Dense Adapters, the performance increases. This suggests that our Dense
Adapters effectively facilitate dense interactions between the lower and
higher layers of the ViT to enhance the representational capability, thereby
improving performance compared to standard adapter-tuning (Table 4 (a)). It is
worth noting that while our Sparse-Tuning introduces more feature interactions
requiring computation, the GFLOPs are still reduced compared to adapter-
tuning, demonstrating that Token Sparsification also alleviates the
computation cost in Dense Adapters.

Figure 4: Variants of Dense Adapters. We present three variants that integrate multi-level features from different encoder layers at (a) the input stage, (b) within the adapter, and (c) the output stage. Table 5: Comparison of different feature fusion methods. The positions "Input", "Inner", and "Output" correspond to (a), (b), and (c) in Figure 4, respectively. # | Position |  Params. (M)$\downarrow$ |  GFLOPs$\downarrow$ | CIFAR-100 | SVHN | Food-101 | Avg.
---|---|---|---|---|---|---|---
(a) | Input | 0.69 | 11.19 | 91.05 | 96.55 | 89.72 | 92.44
(b) | Inner | 1.10 | 11.70 | 92.31 | 97.47 | 90.72 | 93.50
(c) | Output | 1.68 | 11.81 | 91.27 | 97.14 | 90.19 | 92.87

Effects of Different Feature Fusion Methods. Since Dense Adapters take
multiple features as inputs, we consider three variants of Dense Adapters that
can fuse these multi-level features, as shown in Figure 4. We report the
performance of different feature fusion methods in Table 5. Fusing the multi-
level features before feeding them into the Dense Adapters (Figure 4 (a))
requires fewer trainable parameters but deteriorates performance. This occurs
because this fusion method leads to information loss; features from different
layers may contain complementary information, and simple addition may not
effectively integrate this information. Fusing the features after feeding them
into the Dense Adapters (Figure 4 (c)) also deteriorates performance. This is
due to the fact that multi-level features are mapped into different spaces,
and directly fusing them may obscure important information, thereby reducing
classification performance. Our Dense Adapters first project multi-level
features into the same space, then fuse them, and finally up-project the fused
features back into their original shape (Figure 4 (b)). This ensures that the
dense interaction process occurs within the same feature space, which leads to
better performance.

Effects of Different Positions of Token Sparsification. Since Token
Sparsification occurs across different encoder layers in the ViT, we
investigate its effects at various positions to achieve the best trade-off
between performance and computational cost. As shown in Table 6, the shallower
the position of the first Token Sparsification, the fewer encoder layers with
full tokens need to be processed, hence the lower the computational cost.
However, in the early stages, ViT cannot reliably identify important tokens,
so merging tokens based on unreliable attention maps may result in the loss of
important information, leading to decreased performance (Table 6 (a, b)). In
contrast, as shown in Table 6 (d, e), shallower-layer tokens tokens processed
later by Dense Adapters may have lost local features, resulting in better
overall performance compared to Table 6 (a, b) but still not optimal. We find
that adopting Token Sparsification in the 4th, 7th, and 10th encoder layers
yields the best performance. This suggests that performing multiple dense
interactions in the relatively middle encoder layers of ViT balances local and
global features more effectively during Token Sparsification. Therefore, we
select the 4th, 7th, and 10th encoder layers in ViT for Token Sparsification
to achieve the best trade-off between performance and computational cost.

Scaling up ViT with Sparse-Tuning. We apply Sparse-Tuning to ViT-L [10] to
evaluate its performance and efficiency when scaling up the pre-trained model.
As shown in Table 7, Sparse-Tuning reduces tunable parameters by 99.03% and
decreases GFLOPs by 7.82-30.97 compared to full fine-tuning, while also
surpassing its performance. Additionally, Sparse-Tuning outperforms DyT [60]
in both performance and efficiency, demonstrating its effectiveness for larger
pre-trained models.

Table 6: Comparison of different positions of Token Sparsification. For instance, "[4, 7, 10]" represents conducting Token Sparsification in the 4th, 7th, and 10th encoder layers of ViT. # | Position |  Params. (M)$\downarrow$ |  GFLOPs$\downarrow$ | CIFAR-100 | SVHN | Food-101 | Avg.
---|---|---|---|---|---|---|---
(a) | [2, 5, 8] | 1.10 | 10.78 | 89.77 | 94.65 | 88.71 | 91.04
(b) | [3, 6, 9] | 1.10 | 11.35 | 91.03 | 96.21 | 89.76 | 92.33
(c) | [4, 7, 10] | 1.10 | 11.70 | 92.31 | 97.47 | 90.72 | 93.50
(d) | [5, 8, 11] | 1.10 | 12.73 | 92.12 | 96.69 | 90.24 | 92.68
(e) | [6, 9, 12] | 1.10 | 13.68 | 91.38 | 96.10 | 89.93 | 92.47
Table 7: Comparison when scaling up the model size to ViT-L [10]. $r$ represents the keeping rate of the attentive (activated) tokens. Method |  Params. (M)$\downarrow$ |  GFLOPs$\downarrow$ | CIFAR-100 | SVHN | Food-101 | Avg.
---|---|---|---|---|---|---
Full fine-tuning | 303.3 | 61.60 | 92.05 | 97.44 | 90.62 | 93.04
DyT $r=0.5$ [60] | 3.17 | 43.79 | 93.49 | 97.38 | 91.49 | 94.12
DyT $r=0.7$ [60] | 3.17 | 51.11 | 93.28 | 97.25 | 91.60 | 94.04
DyT $r=0.9$ [60] | 3.17 | 60.05 | 93.44 | 97.23 | 91.59 | 94.09
Sparse-Tuning $r=0.5$ | 2.93 | 30.63 | 93.56 | 97.31 | 91.46 | 94.11
Sparse-Tuning $r=0.7$ | 2.93 | 40.08 | 93.97 | 98.23 | 91.98 | 94.73
Sparse-Tuning $r=0.9$ | 2.93 | 53.78 | 93.45 | 98.15 | 92.77 | 94.79

##  5 Conclusion

In this work, we aim to enhance efficiency during both fine-tuning and
inference stages when adapting the pre-trained ViT. To this end, we propose a
novel tuning method called Sparse-Tuning, which selectively adapts tokens to
enable the pre-trained ViT to focus more on the foreground and less on
background regions during the fine-tuning stage. By gradually preserving
informative tokens and merging uninformative ones into one representative
token, our Sparse-Tuning significantly reduces redundant computational costs,
achieving both fine-tuning and inference efficiency for ViT adaptation. We
conduct empirical experiments on the VTAB-1K benchmark, three complete image
datasets, and two complete video datasets to ensure the generalizability of
our Sparse-Tuning for efficient ViT adaptation. Extensive experimental results
demonstrate that our Sparse-Tuning can enhance performance as well as
significantly improve fine-tuning and inference efficiency.In this paper, as
we mainly focus on classification tasks in our experiments, extending our
Sparse-Tuning to other vision tasks, such as segmentation and detection, will
be our future direction.

## References

  * [1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [2] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101–mining discriminative components with random forests.  In Proceedings of the European Conference on Computer Vision, pages 446–461. Springer, 2014.
  * [3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650–9660, 2021.
  * [4] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017.
  * [5] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo.  Adaptformer: Adapting vision transformers for scalable visual recognition.  In Proceedings of the Advances in Neural Information Processing Systems, volume 35, pages 16664–16678, 2022.
  * [6] Xuanyao Chen, Zhijian Liu, Haotian Tang, Li Yi, Hang Zhao, and Song Han.  Sparsevit: Revisiting activation sparsity for efficient high-resolution vision transformer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2061–2070, 2023.
  * [7] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al.  Scaling vision transformers to 22 billion parameters.  In Proceedings of the International Conference on Machine Learning, pages 7480–7512. PMLR, 2023.
  * [8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 248–255. Ieee, 2009.
  * [9] Haiwen Diao, Bo Wan, Ying Zhang, Xu Jia, Huchuan Lu, and Long Chen.  UniPT: Universal parallel tuning for transfer learning with efficient parameter and memory.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
  * [10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In Proceedings of the International Conference on Learning Representations, 2020.
  * [11] Chin-Lun Fu, Zih-Ching Chen, Yun-Ru Lee, and Hung-Yi Lee.  Adapterbias: Parameter-efficient token-dependent representation shift for adapters in nlp tasks.  In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2608–2621, 2022.
  * [12] Minghao Fu, Ke Zhu, and Jianxin Wu.  Dtl: Disentangled transfer learning for visual recognition.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 12082–12090, 2024.
  * [13] Ian J Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Arnoud, and Vinay Shet.  Multi-digit number recognition from street view imagery using deep convolutional neural networks.  arXiv preprint arXiv:1312.6082, 2013.
  * [14] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al.  The" something something" video database for learning and evaluating visual common sense.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5842–5850, 2017.
  * [15] Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang.  Dynamic neural networks: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):7436–7456, 2021.
  * [16] Joakim Bruslund Haurum, Sergio Escalera, Graham W Taylor, and Thomas B Moeslund.  Which tokens to use? investigating token reduction in vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 773–783, 2023.
  * [17] Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang.  Sensitivity-aware visual parameter-efficient fine-tuning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11825–11835, 2023.
  * [18] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022.
  * [19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
  * [20] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  arXiv preprint arXiv:1606.08415, 2016.
  * [21] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In Proceedings of the International Conference on Machine Learning, pages 2790–2799. PMLR, 2019.
  * [22] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.  LoRA: Low-rank adaptation of large language models.  In Proceedings of the International Conference on Learning Representations, 2021.
  * [23] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4700–4708, 2017.
  * [24] Siteng Huang, Biao Gong, Yulin Pan, Jianwen Jiang, Yiliang Lv, Yuyuan Li, and Donglin Wang.  Vop: Text-video co-operative prompt tuning for cross-modal retrieval.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6565–6574, 2023.
  * [25] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In Proceedings of the European Conference on Computer Vision, pages 709–727. Springer, 2022.
  * [26] Zeyinzi Jiang, Chaojie Mao, Ziyuan Huang, Ao Ma, Yiliang Lv, Yujun Shen, Deli Zhao, and Jingren Zhou.  Res-tuning: A flexible and efficient tuning paradigm via unbinding tuner from backbone.  In Proceedings of the Advances in Neural Information Processing Systems, volume 36, 2024.
  * [27] Shibo Jie and Zhi-Hong Deng.  Convolutional bypasses are better vision transformer adapters.  arXiv preprint arXiv:2207.07039, 2022.
  * [28] Shibo Jie and Zhi-Hong Deng.  Convolutional bypasses are better vision transformer adapters.  arXiv preprint arXiv:2207.07039, 2022.
  * [29] Shibo Jie and Zhi-Hong Deng.  Fact: Factor-tuning for lightweight adaptation on vision transformer.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1060–1068, 2023.
  * [30] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023.
  * [31] Simon Kornblith, Jonathon Shlens, and Quoc V Le.  Do better imagenet models transfer better?  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2661–2671, 2019.
  * [32] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International Journal of Computer Vision, 123:32–73, 2017.
  * [33] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [34] Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, 2021.
  * [35] Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang.  Scaling & shifting your features: A new baseline for efficient model tuning.  In Proceedings of the Advances in Neural Information Processing Systems, volume 35, pages 109–123, 2022.
  * [36] Y Liang, C Ge, Z Tong, Y Song, P Xie, et al.  Not all patches are what you need: Expediting vision transformers via token reorganizations.  In Proceedings of the International Conference on Learning Representations, 2022.
  * [37] Mingbao Lin, Mengzhao Chen, Yuxin Zhang, Chunhua Shen, Rongrong Ji, and Liujuan Cao.  Super vision transformer.  International Journal of Computer Vision, 131(12):3136–3151, 2023.
  * [38] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Proceedings of the European Conference on Computer Vision, pages 740–755. Springer, 2014.
  * [39] Ting Liu, Xuyang Liu, Siteng Huang, Honggang Chen, Quanjun Yin, Long Qin, Donglin Wang, and Yue Hu.  DARA: Domain- and relation-aware adapters make parameter-efficient tuning for visual grounding.  arXiv preprint arXiv:2405.06217, 2024.
  * [40] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al.  Swin transformer v2: Scaling up capacity and resolution.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12009–12019, 2022.
  * [41] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021.
  * [42] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  arXiv preprint arXiv:1608.03983, 2016.
  * [43] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [44] Otniel-Bogdan Mercea, Alexey Gritsenko, Cordelia Schmid, and Anurag Arnab.  Time-, memory-and parameter-efficient visual adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
  * [45] Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li.  St-adapter: Parameter-efficient image-to-video transfer learning.  In Proceedings of the Advances in Neural Information Processing Systems, pages 26462–26477, 2022.
  * [46] Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li.  St-adapter: Parameter-efficient image-to-video transfer learning.  NeurIPS, 35:26462–26477, 2022.
  * [47] Yao Qiang, Deng Pan, Chengyin Li, Xin Li, Rhongho Jang, and Dongxiao Zhu.  Attcat: Explaining transformers via attentive class activation tokens.  In Proceedings of the Advances in Neural Information Processing Systems, volume 35, pages 5052–5064, 2022.
  * [48] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In Proceedings of the International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [49] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh.  Dynamicvit: Efficient vision transformers with dynamic token sparsification.  In Proceedings of the Advances in Neural Information Processing Systems, volume 34, pages 13937–13949, 2021.
  * [50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Proceedings of the Advances in Neural Information Processing Systems, volume 30, 2017.
  * [51] Yulin Wang, Rui Huang, Shiji Song, Zeyi Huang, and Gao Huang.  Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition.  In Proceedings of the Advances in Neural Information Processing Systems, volume 34, pages 11960–11973, 2021.
  * [52] Yi Xin, Junlong Du, Qiang Wang, Zhiwen Lin, and Ke Yan.  VMT-Adapter: Parameter-efficient transfer learning for multi-task dense scene understanding.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 16085–16093, 2024.
  * [53] Yi Xin, Junlong Du, Qiang Wang, Ke Yan, and Shouhong Ding.  MmAP: Multi-modal alignment prompt for cross-domain multi-task learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 16076–16084, 2024.
  * [54] Yi Xin, Siqi Luo, Haodi Zhou, Junlong Du, Xiaohong Liu, Yue Fan, Qing Li, and Yuntao Du.  Parameter-efficient fine-tuning for pre-trained vision models: A survey.  arXiv preprint arXiv:2402.02242, 2024.
  * [55] Zunnan Xu, Zhihong Chen, Yong Zhang, et al.  Bridging vision and language encoders: Parameter-efficient tuning for referring image segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
  * [56] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel.  Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1–9, 2022.
  * [57] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer.  Scaling vision transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12104–12113, 2022.
  * [58] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al.  A large-scale study of representation learning with the visual task adaptation benchmark.  arXiv preprint arXiv:1910.04867, 2019.
  * [59] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu.  Neural prompt search.  arXiv preprint arXiv:2206.04673, 2022.
  * [60] Wangbo Zhao, Jiasheng Tang, Yizeng Han, Yibing Song, Kai Wang, Gao Huang, Fan Wang, and Yang You.  Dynamic tuning towards parameter and inference efficiency for vit adaptation.  arXiv preprint arXiv:2403.11808, 2024.

##  Appendix A Additional Experiments

###  A.1 Performance and Efficiency on CIFAR-100

Table 8: Comparison with Mainstream PEFT Methods on CIFAR-100. This table
replicates the data exactly as shown in Figure 1.

Method |  Params. (M)$\downarrow$ | Memory Usage (GB) | Time (sec/epoch) |  GFLOPs $\downarrow$ |  Acc.
---|---|---|---|---|---
Train $\downarrow$ | Inference $\downarrow$ | Train $\downarrow$ | Inference $\downarrow$
Full fine-tuning | 85.8 | 18.82 | 3.43 | 90 | 8 | 17.58 | 90.91
Adapter [21] | 1.19 | 13.74 | 2.61 | 70 | 8 | 17.81 | 91.76
LoRA [22] | 1.19 | 14.07 | 2.89 | 75 | 8 | 17.58 | 91.42
AdaptFormer [5] | 1.19 | 13.69 | 2.61 | 68 | 8 | 17.81 | 92.03
VPT [25] | 0.07 | 14.60 | 3.12 | 77 | 8 | 18.32 | 91.64
DyT [60] | 4.80 | 11.32 | 2.35 | 90 | 7 | 12.21 | 91.01
Sparse-Tuning | 1.10 | 8.89 | 1.53 | 40 | 4 | 11.70 | 92.31

We present the numbers of updated parameters during fine-tuning, GPU memory
usage during both fine-tuning and inference, time for fine-tuning and
inference, GFLOPs, and accuracy of our Sparse-Tuning method compared to other
mainstream PEFT methods on the CIFAR-100 dataset [33]. Evidently, our Sparse-
Tuning achieves state-of-the-art performance while significantly enhancing
efficiency during both the fine-tuning and inference stages.

###  A.2 Effects of Different Bottleneck Dimensions of Dense Adapter

Table 9: Comparison of different bottleneck dimensions of Dense Adapter. $d$ represents the bottleneck dimension of the Dense Adapter. # | $d$ |  Params. (M)$\downarrow$ |  GFLOPs$\downarrow$ | CIFAR-100 | SVHN | Food-101 | Avg.
---|---|---|---|---|---|---|---
(a) | 8 | 0.29 | 11.60 | 91.03 | 96.31 | 89.55 | 92.30
(b) | 16 | 0.56 | 11.63 | 91.75 | 97.02 | 89.92 | 92.90
(c) | 32 | 1.10 | 11.70 | 92.31 | 97.47 | 90.72 | 93.50
(d) | 64 | 2.18 | 11.83 | 92.19 | 96.96 | 90.25 | 93.13
(e) | 128 | 4.35 | 12.09 | 92.11 | 96.15 | 90.31 | 92.47

We explore the impact of the bottleneck dimension $d$ of our Dense Adapter in
Sparse-Tuning to achieve the best trade-off between performance, updated
parameters, and computational cost. As reported in Table 9, a higher
bottleneck dimension $d$ introduces more parameters and higher GFLOPs.
However, with a smaller $d$, the down-projection may lose significant
information about the original features, leading to performance degradation.
We observe that performance peaks at a bottleneck dimension of 32 and declines
thereafter. Therefore, considering the trade-off between trainable parameters,
GFLOPs, and performance, we select a bottleneck dimension of 32.

Figure 5: More visualizations of Token Sparsification by Sparse-Tuning.

##  Appendix B More Visualizations of Token Sparsification

We present more visualization results of Token Sparsification in Figure 5. The
results demonstrate that given various images, the Token Sparsification in our
Sparse-Tuning can effectively maintain the tokens from the foreground regions.

##  Appendix C Implementation Details for Each Task

#### Experimental settings on VTAB-1K.

Following previous works [28, 26], we fine-tune the model for 100 epochs on
each dataset in VTAB-1K [58]. We _do not_ use any data augmentation strategy
in these experiments. We adopt the AdamW [43] optimizer. The base learning
rate is set to 0.01 and gradually decays to 0 based on a cosine schedule [42].

#### Experimental settings on complete image datasets.

We use the settings in Table 10 to fine-tune the ViT with the proposed Sparse-
Tuning. Experiments on other parameter-efficient methods such as AdaptFormer
[5], LoRA [22], and VPT [25] also follow the settings [60] in Table 10.

Table 10: Experimental settings for complete image datasets. We present the hyperparameters in Sparse-Tuning. Configuration | CIFAR-100 [33], SVHN [13], Food-101 [2]
---|---
Optimizer | AdamW [43]
Base learning rate | 0.01
Weight decay | 0.01
Batch size | 128
Training crop size | 224
Learning rate schedule | Cosine decay [42]
GPU numbers | 1
Warmup epochs | 20
Training epochs | 100
Augmentation | RandomResizedCrop
|

#### Experimental settings on video datasets.

We use two video datasets, Kinetics-400 (K400) [4] and Something-Something V2
(SSv2) [14], to evaluate performance as the token count scales up. The
experimental settings are shown in Table 11. The number of input frames is set
to 8. During testing, we use multi-view, a common practice in video action
recognition. Experiments on others PEFT methods also follow these experimental
settings.

Table 11: Experimental settings for complete video datasets. We follow most of
settings in [46]. The number of input frames is set to 8 in all experiments.

Configuration | K400 [4] | SSV2 [14]
---|---|---
Optimizer | AdamW [43]
Base learning rate | 1e-3
Weight decay | 0.01
Batch size | 128
Training epochs | 12 | 50
lr Training resize |  | ShortSideJitter
---
RandomResizedCrop
Training crop size | 224
Learning rate schedule | Cosine decay [42]
Num. testing views | 1 spatial $\times$ 3 temporal | 3 spatial $\times$ 1 temporal

##  Appendix D Pseudocode of Sparse-Tuning

We present the PyTorch-like pseudocode of our Sparse-Tuning in algorithm 1 to
help to better understand the whole process.

Algorithm 1 PyTorch-like pseudocode of Sparse-Tuning for a ViT encoder.

⬇

# H: number of attention heads

# N: number of input tokens

# C: the dimension of token vector

# k: the token keeping rate

# x: the input tokens with shape [N, C], with the first being the [CLS] token

# fc_q, fc_k, fc_v: linear transforms for query, key, and value of self-
attention

# mha_output: multi-head attention output of the N-th layer

# prev_adapter_output: adapter output of the previous layer

# adapter_output_N_3_layer: adapter outputs of the N-3 layer

# proj: linear projection in self-attention

# norm: layer normalization

# ffn: feed-forward network

# dense_adapter: layer-specific adapter network

for name, p in model.named_parameters():

if "adapter" in name or "head" in name:

p.requires_grad = True

else:

p.requires_grad = False

avg_cls_attn = zeros(N-1)

x_out = []

x_residual = x

x = norm(x)

# compute self-attention for each attention head

for i in range(0, H):

q, k, v = fc_q[i](x), fc_k[i](x), fc_v[i](x)

attn = (q @ k.transpose()) / sqrt(C/H)

attn = softmax(attn, dim=1)

x_head = attn @ v

x_out.append(x_head)

cls_attn = attn[0, 1:]

avg_cls_attn += cls_attn

x = concat(x_out, dim=1)

x = proj(x) # shape: [N, C]

x = x + x_residual

avg_cls_attn /= H

sorted_cls_attn, idx = sort(avg_cls_attn)

# compute the number of attentive tokens, without counting the [CLS] token

K = ceil(k * (N - 1))

topk_attn, topk_idx = sorted_cls_attn[:K], idx[:K]

non_topk_attn, non_topk_idx = sorted_cls_attn[K:], idx[K:]

cls_token = x[0:1]

x_without_cls = x[1:]

# obtain the attentive and inattentive tokens

attentive_tokens = x_without_cls[topk_idx]

inattentive_tokens = x_without_cls[non_topk_idx]

# compute the weighted combination of inattentive tokens

fused_token = non_topk_attn @ inattentive_tokens

x_new = concat([cls_token, attentive_tokens, fused_token], dim=0)

# dense adapter processing

adapter_output = dense_adapter(mha_output, prev_adapter_output,
adapter_output_N_3_layer)

x_residual = x_new

x_new = norm(x_new)

x_new = ffn(x_new)

x_new = x_new + x_residual+ adapter_output

return x_new


monthyeardate[]

# Using Deep Learning for Visual Decoding and Reconstruction from Brain
Activity: A Review

Madison Van Horn
The University of Edinburgh
Informatics Forum, Edinburgh, UK, EH8 9AB
<EMAIL_ADDRESS>

###### Abstract

This literature review will discuss the use of deep learning methods for image
reconstruction using fMRI data. More specifically, the quality of image
reconstruction will be determined by the choice in decoding and reconstruction
architectures. I will show that these structures can struggle with
adaptability to various input stimuli due to complicated objects in images.
Also, the significance of feature representation will be evaluated. This paper
will conclude the use of deep learning within visual decoding and
reconstruction is highly optimal when using variations of deep neural networks
and will provide details of potential future work.

##  1 Introduction

The study of the human brain is a continuous growing interest especially for
the fields of neuroscience and artificial intelligence. However, there are
still many unknowns to the human brain. Recent advancements of the combination
of these fields allow for better understanding between the brain and visual
perception. This notion of visual perception uses neural data in order to make
predictions of stimuli [1]. The predictions of stimuli will allow researchers
to not only see if we are capable of reconstructing visual thoughts but
consider the accuracy with these predictions. This reconstruction is
achievable due to the ability of using neural data from tools such as
functional magnetic resonance imaging (fMRI) to assist in recording brain
activity to later use for decoding and reconstructing a stimulus. These
advancements enable the opportunity to further investigate how we can recreate
perceived images and experiences by decoding human brain activity with the use
of machine learning.

As expected, issues arise with decoding visual stimuli. The type of input can
have a great effect in the ability to accurately verify perceived images. For
example, the use of natural images as input for neural decoding and visual
reconstruction proves to remain a difficult task. The obvious solution is to
choose wisely on the type of architecture to handle these complex, non-linear
input data [2]. In order to solve this task of recreating images, deep
learning has naturally transitioned into the field due to its success in other
applications. This learning method is an increasingly popular choice in
representing decoding and reconstruction models because of the great
improvement and enhancement of image quality and reconstruction efficiency.

In this literature review, I will analyze recent developments of visual image
decoding and reconstruction. These methods implement state of the art deep
learning techniques such as deep neural networks, convolutional neural
networks, variational autoencoders, and generative adversarial networks. In
some cases, the same architecture is used for decoding and reconstructing
which is known as an end-to-end model [3, 4]. We are interested in seeing how
these architectures can enhance image reconstruction since there are
difficulties with achieving clear, quality images. I will also examine if
there is a relationship between the architectures for the decoder and
reconstructor as well as a relationship of homology and the process of visual
image reconstruction. We want to learn about the impact of feature
representations and homology. More specifically, the following research
questions will be examined and investigated within this paper:

  * •

What is the role of the architecture in the quality of the image?

  * •

How well are these techniques adaptable to different domains?

  * •

What is the significance of representation for features?

The goal of this paper is to introduce and examine the current state of the
art decoding and reconstruction techniques. Comparisons of the models will be
made as well as present any discrepancies found during the analysis. This
review will not cover the specifics of encoding models nor the fMRI analysis.
As for fMRI, a discussion of this choice in recording brain activity will be
presented with no further analysis of how the technique records data. Other
brain activity recording techniques will not be examined including recording
specific subregions of the brain due to being out of scope of the paper. The
motivation behind this paper is to further understand how we can form a
relationship between our brain and visual perception. This understanding of
image reconstruction can lead to new societal impacts such as the ethics
behind recreating seen images, dreams, and videos.

This literature review will be structured as the following. Section 2 will
provide background details to visual decoding and reconstruction with the use
of fMRI data. Section 3 will contain the literature review with details of the
role deep neural networks and generative networks play in the quality of the
image, discuss the brittleness of the techniques applied to different domains
and review the significance of feature representation. Section 4 will
summarize and conclude the findings of the review as well as provide future
work.

##  2 Overview of Perceived Image Reconstruction

The interest in expanding our knowledge with how the brain perceives
information and images is an ongoing research problem within machine learning
and neuroscience. The type and amount of data needed for the reconstruction is
an issue within this area of research. Although there are successful models
capable of identifying and reconstructing faces from fMRI data, different
models are needed to address other images, such as natural images. Natural
images remain a difficult problem within visual reconstruction due to the
nonlinearity of the image. The amount of data is a constant obstacle within
the field of machine learning, especially for accurately reconstructing
images. Due to brain activity containing high dimensional data while having
low number of samples creates an issue of properly predicting the
reconstructed image during training [5]. Usually, only one out of these two
problems are addressed in reviews.

The ability to reconstruct images is feasible because of the use of fMRI. FMRI
is a neuroimage analysis technique used to measure human brain activity by
using fMRI BOLD signals to measure external stimuli [6]. It is a crucial
imaging technique that has advanced the studies of recreating external stimuli
seen from the brain. Although this paper will not provide further details on
how the fMRI records or extracts the significant signals, it is important to
mention fMRI data is the input data used for reconstructing images. This
technique is advantageous by allowing non-invasive recording of the BOLD
signals, recording the blood flow within the brain [6]. The noise produced
remains an issue in this field due to the unknown of whether the noise is
coming from fMRI data or from the model producing the noise itself. This
unknown greatly affects the accuracy of all the phases including encoding,
decoding, and reconstructing visually seen images.

Figure 1: The above figure displays the process of brain decoding and
reconstruction of images. [7]

Visual reconstruction can be split into three phases: encoding, decoding, and
reconstruction. As stated previously, this review will briefly give overviews
of each phase but will only focus on decoding and reconstructing perceived
images from the human brain. This work flow can be seen in Figure 1 where the
process of brain decoding and reconstruction of images is shown. The encoding
phase focuses on describing the individual voxels in a quantitative matter in
order to keep important information about the stimuli. [8]. This quantitative
information provides descriptions of regions of interest in the human brain.
The encoder must have the necessary visual features in order to provide
decoders enough information to create quality images from stimuli. The
decoding is the process of transitioning high dimensional brain activity into
high dimensional stimuli [9]. After retrieving the fMRI data, neural decoding
occurs in order to understand the recorded brain activity data. This step is
crucial in order to reconstruct the visual imagery process to achieve a
quantitative expression from a phenomenological experience [10]. Deep learning
techniques can also be applied to neural decoders to achieve better decoding
accuracy. The reconstruction phase uses this high dimensional stimuli in order
to recreate perceived images from brain activity data.

The goal of visual reconstruction is to map brain activity to decoded features
and to use these decoded features to reconstruct seen stimuli [11]. This is an
important field of computational neuroscience to not only have a better
understanding of how the brain processes information but how it perceives
information or objects. The rest of the review will focus on the methods and
domains applied to decoding and reconstructing images.

##  3 Literature Review

As this review will only focus on the decoder and reconstruction process of
perceived images, we will elaborate on the role of the architectures allowing
the ability to recreate and improve images. The model’s architecture plays a
crucial role in the ability to show that newly created images are visually
similar to the original image. It is evident deep learning has greatly
expanded in the field of neuroscience and machine learning, allowing the
ability to reconstruct perceived images from brain activity [5]. Different
deep architectures greatly contribute to the further development of visual
reconstruction. We see the evolution of the various architectures and their
roles in recreating the images have increased the quality of the reconstructed
images to be similar in color, texture, and appearance to the original
stimuli.

###  3.1 Deep Neural Networks

The emergence of deep learning greatly influences the model’s structure
focusing on visualizing perceptual objects. Naturally, neural networks were
first experimented to achieve this goal of image reconstruction, especially
convolutional neural networks (CNNs). Shen experimented with implementing deep
neural networks to perform deep image reconstruction from human brain activity
[12]. The method pretrains 1,000 natural object categories with sixteen CNN
layers and three fully-connected layers from ImageNet 2011 release.
Effectiveness of the model was measured by pixel-wise spatial correlation and
human judgment. The authors emphasize on correlating hierarchical visual
features to the fMRI activity data due to similarity between hierarchical
feature representation and the brain. The intentions were to optimize the
image by making the deep neural network features similar to the decoded
features. Although this network structure shows the capability of
reconstructing images, it lacks quality and similarity to the original image.
As the authors are advocates of generality, their model was only trained on
natural objects which were centered. This is an issue when other natural
images are used that can potentially have many objects in the image. Training
objects centered in the middle of the image will not be optimal, leading to
only specific types of images centered having better chances of being
reconstructed. However, this initial implementation further increased interest
in using deep learning for visual reconstruction. More recent publishings by
Shen suggest further improvements of the model by creating an end-to-end
structure to visualize perceptual content from brain activity [3]. The goal of
this model is focusing on improving the quality of the image by finding a
direct mapping with a direct approach of fMRI activity to stimulus space. The
belief of using this an end-to-end approach is believed to be due to potential
information loss when performing the decoding of features. This method
consists of three convolutional networks: one with three fully-connected
layers with six upconvolutional layers, one with five convolutional layers
with three fully-connected layers, and one with five convolutional layers as
well as an average pooling layer and two fully-connected layers. The
experiment consists of natural images of 150 object categories with 1,200
images for training and 50 object categories and 50 images for testing. Pair-
wise spatial correlation and human judgment was used to evaluate the
reconstructed images to the original. In comparison with the previous
approach, Shen’s direct mapping of fMRI data increases accuracy and quality in
the reconstruction. This shows that information loss occurring in an non-end-
to-end model can lead to less quality in images. One advantage between the two
methods is Shen’s 2019 research included an ablation study which is not used
in any of the mentioned papers in this review. This ablation study goes
through each layer of the model and removes a loss function to compare the
reconstructions. This approach is useful in determining if loss functions are
crucial for reconstructing an image and having it resemble the original. The
experiment showed removing the loss functions play significant roles in the
quality representation of the image, indicating all the loss functions have a
large effect in the reconstruction from direct brain activity to perception.

As mentioned previously, Shen was capable of creating an end-to-end network
resulting in an increase in quality within the reconstructed images [3].
Another researcher, Han, approaches a different version of an end-to-end model
attempting to use variational autoencoders, VAEs, for decoding and
reconstructing images and movies [4]. Although encoding is not a focus in this
paper, it is important to note the team tried to expand the same
implementation for encoding, decoding and reconstructing. The goal of the
method is to use VAEs to generalize the decoding procedure. Their results show
VAE is capable of regenerating natural images and movies. After training over
2 million natural images from ImageNet ILSVRC2012 dataset, their method
indicates to be successful by reproducing images without losing significant
color, structure, and other image qualities. This shows that a single
structure is more than capable of reproducing competitive results compared to
other models that use multiple structures to decode and reconstruct images. In
comparison to Shen’s model, VAEs can be considered to outperform CNNs in the
sense that VAEs capture color from fMRI data better. However, there is a
tradeoff in the sense that VAE’s reconstructed images were far blurrier than
CNN’s images. Also, it is interesting to note that Han only reported
quantitative results for movies even though images and movies were used for
training and testing of the model. Providing the statistics for the images
would be useful when comparing against other end-to-end methods. Shen reports
62.9% via structural similarity index measure whereas Han shows about 50%.
Although Shen’s model earns a higher value for accuracy, Han’s method is
acceptable especially for videos.

###  3.2 Generative Networks

Recent developments within visual reconstruction have shown image quality
improvements can be achieved with the use of generative networks. The presence
of generative adversarial networks, GANs, have proved to be consistent in
improving image quality especially with fMRI activity [13]. This compatibility
is successful due to GANs conditional distribution. This conditionality allows
the model to focus on specific details within an image in hopes to focus on
the same important details in the original image [13]. More specifically, in
order to achieve the highest quality and accuracy compared to the original
image, techniques have been expected to incorporate 3 different deep learning
methods for reconstruction.

Addressing the quality of image can be achieved by conditional progressively
growing generative adversarial networks. Huang creates this network with three
deep learning techniques for reconstruction (CAE, LSTM, C-PGAN) [14]. This
architecture is trained and tested on a total of 2750 natural images from 10
object categories. Similarly, 50 images were used for the test set with the
remaining 2700 images for training. An advantage to this approach shows it is
not necessary to pretrain the network with images whereas DNN, GAN, and
autoencoder networks pretrains data. The other models need the pretraining in
order to produce high quality images. This methodology plays a significant
role in image quality due to the progressive growth of the generator and
discrimantor in the model which gradually adds new layers [14]. This allows
specific details to appear during training [14]. Therefore, by using
progressive growth and conditional distributions, this model achieves high
accuracy as well as better image quality.

Another use of generative networks to improve quality of reconstructed images
is by using deep generative neural networks. This approach by VanRullen trains
a VAE network using GAN procedure over a large dataset of 202,599 faces [15].
This methodology produces quality results such as high accuracy and similarity
between reconstructed and original image. The advantage of using this
architecture allows the ability to easily decode unseen images while producing
realistic, clear results. However, this approach uses a dataset of faces
whereas previously mentioned models used natural images. This can potentially
be an indication it will not generalize well to natural images.

Seeliger presents a generative adversarial network for reconstructing images
to achieve similarity with the original image [16]. This is achieved by using
a deep convolutional generative adversarial model to learn the latent space of
compressed data. The 1200 images consisting of 150 object categories were
converted to gray scale. The argument is to test the capabilities of using
this architecture to produce images. The goal of this experiment is to verify
the capability of the model even if the quality of the image is compromised.
Experiments consisted of using three different data sets of handwritten
characters, masked natural images, and natural objects. However, an issue
arose from the not generating images from certain samples, suffering from mode
collapse. This causes the model to only learn kinds of images in a set. Mode
collapsing can be considered to be stuck in local optimum and therefore not
learning well from other kinds of images. Overall, the authors show the
potential of creating images from GANs even with the disturbances encountered.

###  3.3 Adaptability to Different Domains

After examining the various deep learning techniques, it is evident there is a
common theme of focusing on specific datasets. Limitations in the availability
of fMRI data is one noticeable concern within this field. This is mainly due
to the lack of availability of resources including limited stimuli to human
participants in order to record data. Although there is a restriction on the
amount of images to show as stimuli, it is important to recognize that a
strong architecture should handle various forms of data in order to
reconstruct images of faces, animals, landscapes, and various objects. We want
to examine how or in some cases why these techniques are adaptable or brittle.
These architectures tend to focus only on specific forms of reconstruction
such as focusing on details or specific image types. In particular, most of
the reviews focus on specific image types. Due to the difficulty specifically
with reconstructing natural images, the reconstruction architecture’s
adaptability appears to be brittle due to the lack of generalizing well [17].

The most common image types for visual reconstruction are facial and natural
images. Although studies have shown facial reconstruction is capable of
successfully reproducing quality images, more emphasis has been placed on
expanding the quality [17]. As can be seen in VanRullen’s work, a variational
auto-encoder and generative adversarial network are used to reconstruct
celebrity faces. This specific network successfully reconstructs facial images
due to mapping the latent space and fMRI data instead of learning the pixels
[15]. Although facial reconstruction tends to be easier than reconstructing
natural images, it is important to note this specific model provides
adaptability for another domain such as gender recognition. This makes the
model useful for different purposes other than just reconstruction. This
method appears to be similar to Ren’s version of using dual variational auto-
encoder and generative adversarial networks except for testing on natural
images including handwritten numbers, characters, and colored photos [18]. We
will consider these architectures similar enough to compare with different
objects due to using VAEs for decoding and GANs for reconstruction, as well as
using the same metric for measuring similarity between original and newly
reconstructed image, structural similarity index (SSIM). In this case, Ren’s
reconstruction method is useful because not only does the researcher use
various datasets but also makes comparisons to another VAE-GAN architecture,
similar to VanRullen’s model. We can see that VanRullen produced 50.5%
accuracy between the original and reconstructed image on celebrity faces.
However, Ren’s research shows the dual VAE-GAN model outperforms with 58.7%
accuracy on handwritten characters. It also shows the accuracy is 27.7% for
the VAE-GAN model on the same dataset mentioned in Ren’s experiments. This
indicates that the same model, VAE-GAN, does not generalize well to various
datasets. Therefore, this is an example of similar architectures that are
brittle for different domains.

This can also be seen between Güçlütürk’s and Huang’s models. Both
architectures are very similar in structure yet use different input data as
well as generate reconstructed images. Güçlütürk’s approach uses deep
adversarial neural decoding (GANs) for faces [19]. Huang also uses a GAN model
in order to reconstruct natural images [14]. While there are slight
differences in the models, such as Huang’s having a synchronized and
progressively growing GAN and Güçlütürk’s iteratively minimizing the GAN’s
lost function, we can still compare their results with different input images.
Güçlütürk’s achieved 0.6512 with uncertainty of $\pm 0.0493$ with Pearson
Correlation Coefficient [19]. Huang earned slightly above 0.2 with uncertainty
of about $\pm 0.2$ [14]. Although we notice Huang’s model received a smaller
number for representing the correlation of the original and reconstructed
number, it is important to consider a few things. One thing to notice is Huang
used natural images which is a more difficult task. Another thing to consider
is there is a larger number in uncertainty. If we take uncertainty into
account, Huang’s model could achieve a higher number, one closer to
Güçlütürk’s. Altogether, these models show similar architectures with the use
of GANs to reconstruct images. However, there are significant differences in
performance showing the models are brittle to certain input image types.

The lack of adaptability continues with natural movies. Natural movies are
another difficult dataset to reproduce due to the difficulty of splitting and
reconstructing the movie into natural scenes [20]. The goal of presenting
movies as stimuli is to recreate what a person is seeing since it will not
always be in image form [20]. Wen describes how convolutional networks are
used to encode and decode fMRI responses as well as reconstruct the data. In
this case, 972 natural movie clips were shown to participants in order for the
fMRI to record the responses. A CNN model then uses the feature extracted data
to reconstruct the natural movie frames. This paper is significant due to
decoding and reconstructing natural vision within the same structure. This
success in reconstructing natural movies is because of the model focusing on
foreground and suppressed backgrounds [20]. Although this research uses still
images for training, testing of reconstruction of images was not prioritized.
Also, it is mentioned the model needed a reduced number of categories in order
to be applicable to movies [20]. There has not been much research into
producing quality natural movies from brain activity as well as images. Recent
research does not include movie stimuli with the latest architectures.

###  3.4 Feature Representation

As the review will not cover the specific details of encoding, it is important
to recognize the significance of feature representation. Feature
representation is critical in reconstructing images due to containing the
important information that represents the visual stimuli. The representation
not only allows researchers to better understand the importance of feature
vectors, but it also shows evidence of homology between the feature vector and
the human brain architecture.

Hierarchical architecture is ideal especially for feature representation. The
reasoning for this is due to neural networks and the brain structure having
the similar hierarchical representation. Convolutional networks have this
architecture and tend to be common representations for feature extractions. As
can be seen in Wen’s research, the use of this type of network allows for
decoding and categorical semantic representation [20]. While the model is
specifically intended for decoding, the categorical features can be embedded
within the layers of the CNN. Thus, the embedding allows for generalizable
semantic space. Another neural network architecture used for feature
extraction can be seen within Lin’s research of using a CNN architecture [17].
The output of the first fully connected layer proved to be the feature vector.
This step is performed in hopes of capturing more identifiable features within
the image since other representations focus more on capturing the structure of
the image. Lin’s method of feature representation as well as Wen’s shows
feature representation from layers is successful due to being similar to
decoded fMRI. This similarity greatly improves the accuracy in the quality of
the image as well as the decoding.

Deep neural networks achieve this hierarchical representation because of the
similarity in the architecture and brain representation as well. Since the
goal is to have similarities between the network and brain, features are
extracted into deep neural network feature values. Horikawa approaches this
concept by using two different models, AlexNet and VGG19, to decode the fMRI
data into deep neural network feature values [21]. This study shows using deep
neural networks as feature representations shows evidence of homology between
the brain and the decoding architecture. However, Horikawa also concludes this
form of structure indicates unequal decoding representations. This is
important to note because this is one of the main ideas behind using neural
networks for feature representation. These explicit differences of the human
brain and the network architecture goes against the idea of a successful
reconstruction model having similar features of the brain and model.

This form of neural networks is comparable to the previous research by Lin due
to both using deep neural networks, VGG19, for feature representation. Lin
notes that the use of neural networks to represent the features is wanted but
need to avoid large feature size [17]. The difference between the two feature
extractor methods is that Lin uses the first layer as the feature vector
whereas the output of each hidden layer in Horikawa’s research was used. This
greatly reduces the size of features and dimensionality while still producing
effective results. While both methods are similar in feature representation,
Lin’s method proves to preserve the important visual information while
reducing the number of features [17]. Other research by Zhang suggests
hierarchical feature representations are also effective in feature
representation. Similarly to Lin’s, a convolutional neural network’s layers
are used as feature representation from human brain activity [22]. This method
uses eight layers where the output of each layer represents the image’s
features. Interestingly, the lower layers output produced better accuracy
amongst the features by Pearson Correlation Coefficient. This result appears
to agree amongst the other variations in hierarchical feature representation
indicating when using neural networks as feature extractors, all models use
either the first few layer’s output as representation of the features. This
indicates the more important features of the images are preserved in these
lower layers. The previously mentioned methods of using neural networks for
feature representation agree there is correlation between the brain and
hierarchical features structure.

##  4 Summary & Conclusion

We have discussed the advancement in techniques pertaining to visual
reconstruction from brain activity data. It is evident that deep learning
greatly enhances the ability to perform brain decoding and visual
reconstruction because of architectural similarities between the brain and
deep learning [5]. The various architectures play a significant role in the
quality of the reconstructed images. As it can be seen, deep learning greatly
enhances this process. The use of deep learning, such as CNN, VAE, GAN, allows
the ability to create models to extract the necessary information from fMRI
data to use as features for the model. This process proves to be a difficult
challenge especially for the use of natural images. We also examined the
hardships of adjusting models to handle various types of input. This proves to
be a common issue to reconstruct various types of data with the same model.
Even if the data is in the same category, images and movies can play a
significant difference. Lastly, we examine the significance homology plays
within the human brain and the feature representation.

Further research can be improved by testing models on other datasets along
with the natural images. As the natural images are necessary to include due
their difficulty in reconstructing, it is important to also include datasets
such as facial images, to verify the model is generalizing well to various
datasets. Many of the research papers focus only on one particular task
usually referring to one specific type of dataset. Also, future research can
be done to find metrics to measure the accuracy of a reconstructed image.
Currently, there are no set metrics to use in evaluating a reconstructed
image. Most papers will use Pearson’s Correlation Coefficient, SSIM, mean
squared, and other metrics but there is no set guideline to measuring the
accuracy. Most studies need human evaluation to determine the quality of the
image. Further research into measuring the quality of a reconstructed image
should be investigated.

Another potential work can be pursued by making adjustments to the GANs
procedure. As shown in this review, the use of GANs has greatly increased the
quality in reconstructed images. Future work can consider improving this
method by working on the precision and naturalness of the visual reconstructed
product [23]. Research should be investigated to improve the decoding
framework for better extraction of features. More emphasis can be made on the
decoding architecture to include temporal and categorical information from
fMRI activity. A recent paper investigated simultaneously extracting both
types of information from fMRI activity and shows higher accuracy from the
decoding framework [11]. Improving the decoding and reconstruction to
accurately use vital features to reconstruct images will be the future for
brain decoding. Many of the papers in this field focus on the overall accuracy
of the reconstructed image versus the original. Improving the decoding
accuracy could potentially improve the final accuracy of the reconstructed
image. Altogether, the decoding step should be investigated to further see if
it is in its highest optimal state to improve the reconstruction process.

## References

  * [1] Adrian Nestor, Andy C.H. Lee, David C. Plaut, and Marlene Behrmann.  The face of image reconstruction: Progress, pitfalls, prospects.  Trends in Cognitive Sciences, 24(9):747 – 759, 2020.
  * [2] Jesse A Livezey and Joshua I Glaser.  Deep learning approaches for neural decoding across architectures and recording modalities.  Briefings in Bioinformatics, 12 2020.  bbaa355.
  * [3] Guohua Shen, Kshitij Dwivedi, Kei Majima, Tomoyasu Horikawa, and Yukiyasu Kamitani.  End-to-end deep image reconstruction from human brain activity.  Frontiers in Computational Neuroscience, 13:21, 2019.
  * [4] Kuan Han, Haiguang Wen, Junxing Shi, Kun-Han Lu, Yizhen Zhang, and Zhongming Liu.  Variational autoencoder: An unsupervised model for modeling and decoding fmri activity in visual cortex.  bioRxiv, 2018.
  * [5] O. Firat, L. Oztekin, and F. T. Y. Vural.  Deep learning for brain decoding.  In 2014 IEEE International Conference on Image Processing (ICIP), pages 2784–2788, 2014.
  * [6] Mo Chen, Junwei Han, Xintao Hu, Xi Jiang, Lei Guo, and Tianming Liu.  Survey of encoding and decoding of visual stimulus via fmri: an image analysis perspective, Mar 2014.
  * [7] Shuo Huang, Wei Shao, Mei-Ling Wang, and Dao-Qiang Zhang.  fmri-based decoding of visual information from human brain activity: A brief review, Jan 2021.
  * [8] Thomas Naselaris, Kendrick N. Kay, Shinji Nishimoto, and Jack L. Gallant.  Encoding and decoding in fmri.  NeuroImage, 56(2):400 – 410, 2011.  Multivariate Decoding and Brain Reading.
  * [9] Marcel A. J. van Gerven, Katja Seeliger, Umut Güçlü, and Yağmur Güçlütürk.  Current Advances in Neural Decoding, pages 379–394.  Springer International Publishing, Cham, 2019.
  * [10] Yağmur Güçlütürk, Umut Güçlü, Katja Seeliger, Sander Bosch, Rob van Lier, and Marcel A. J. van Gerven.  Reconstructing perceived faces from brain activations with deep adversarial neural decoding.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, pages 4246–4257. Curran Associates, Inc., 2017\.
  * [11] Chong Wang, Hongmei Yan, Wei Huang, Jiyi Li, Jiale Yang, Rong Li, Leiyao Zhang, Liang Li, Jiang Zhang, Zhentao Zuo, and Huafu Chen.  ‘when’ and ‘what’ did you see? a novel fMRI-based visual decoding framework.  Journal of Neural Engineering, 17(5):056013, oct 2020.
  * [12] Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani.  Deep image reconstruction from human brain activity.  bioRxiv, 2017.
  * [13] G. St-Yves and T. Naselaris.  Generative adversarial networks conditioned on brain activity reconstruct seen images.  In 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 1054–1061, 2018.
  * [14] Wei Huang, Hongmei Yan, Chong Wang, Xiaoqing Yang, Jiyi Li, Zhentao Zuo, Jiang Zhang, and Huafu Chen.  Deep natural image reconstruction from human brain activity based on conditional progressively growing generative adversarial networks, Nov 2020.
  * [15] Rufin VanRullen and Leila Reddy.  Reconstructing faces from fmri patterns using deep generative neural networks.  CoRR, abs/1810.03856, 2018.
  * [16] K. Seeliger, U. Güçlü, L. Ambrogioni, Y. Güçlütürk, and M.A.J. van Gerven.  Generative adversarial networks for reconstructing natural images from brain activity.  NeuroImage, 181:775 – 785, 2018.
  * [17] Y. Lin, J. Li, and H. Wang.  Dcnn-gan: Reconstructing realistic image from fmri.  In 2019 16th International Conference on Machine Vision Applications (MVA), pages 1–6, 2019.
  * [18] Ziqi Ren, Jie Li, Xuetong Xue, Xin Li, Fan Yang, Zhicheng Jiao, and Xinbo Gao.  Reconstructing perceived images from brain activity by visually-guided cognitive representation and adversarial learning, 2019.
  * [19] Y. Güçlütürk, U. Güçlü, K. Seeliger, S.E. Bosch, R.J. van Lier, and M.A.J. van Gerven, 2017.
  * [20] Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, and Zhongming Liu.  Neural encoding and decoding with deep learning for dynamic natural vision.  Cerebral Cortex, 28(12):4136–4160, 10 2017.
  * [21] Tomoyasu Horikawa, Shuntaro C. Aoki, Mitsuaki Tsukamoto, and Yukiyasu Kamitani.  Characterization of deep neural network features by decodability from human brain activity, Feb 2019.
  * [22] Chi Zhang, Kai Qiao, Linyuan Wang, Li Tong, Ying Zeng, and Bin Yan.  Constraint-free natural image reconstruction from fmri signals based on convolutional neural network.  Frontiers in Human Neuroscience, 12:242, 2018.
  * [23] Kai Qiao, Jian Chen, Linyuan Wang, Chi Zhang, Li Tong, and Bin Yan.  Biggan-based bayesian reconstruction of natural images from human brain activity.  Neuroscience, 444:92 – 105, 2020.