# Transferability Estimation Based On Principal Gradient Expectation

Huiyan Qi1, Lechao Cheng2, Jingjing Chen1, Yue Yu1, Xue Song1, Zunlei Fengg3,
Yu-Gang Jiang1
1School of Computer Science & Shanghai Collaborative Innovation Center of
Intelligent Visual Computing,
Fudan University
2Zhejiang Lab
3Zhejiang University

###### Abstract

Transfer learning aims to improve the performance of target tasks by
transferring knowledge acquired in source tasks. The standard approach is pre-
training followed by fine-tuning or linear probing. Especially, selecting a
proper source domain for a specific target domain under pre-defined tasks is
crucial for improving efficiency and effectiveness. It is conventional to
solve this problem via estimating transferability. However, existing methods
can not reach a trade-off between performance and cost. To comprehensively
evaluate estimation methods, we summarize three properties: stability,
reliability and efficiency. Building upon them, we propose Principal Gradient
Expectation (PGE), a simple yet effective method for assessing
transferability. Specifically, we calculate the gradient over each weight unit
multiple times with a restart scheme, and then we compute the expectation of
all gradients. Finally, the transferability between the source and target is
estimated by computing the gap of normalized principal gradients. Extensive
experiments show that the proposed metric is superior to state-of-the-art
methods on all properties.

##  1 Introduction

Traditional machine learning work for predicting unseen testing instances with
the knowledge learned from training data [7, 53]. Since there is always a
distribution gap between training data and testing data, the inference
performance is lower than expected [58, 47]. However, as obtaining appropriate
training data is challenging, a surge of interest emerges in transferring
knowledge from the source task into the target task based on the pre-training
mechanism. In the realm of computer vision, the prevalent approaches are
linear probing [5] and fine-tuning. Linear probing adapts the model outputs by
learning a task-related head layer while fine-tuning adapts the parameters of
the entire model. Both linear probing and fine-tuning leverage models pre-
trained on large-scale datasets for downstream tasks.

Figure 1: The figure on the left denotes that the model’s parameters are
updated by first-order gradients in the optimization process. Intuitively, the
gap between the optimal points of the source and target domains can reflect
transferability. However, the point of the target is invisible. To solve this
problem, our PGE collects first-order gradients and computes the expectation
of all gradients as on the right.

Prominent models, including BERT [10] and GPT-3 [3] for natural language
processing, the ResNet [20] and vision transformer [12, 32, 31] variants pre-
trained on ImageNet [41] for vision tasks, and the CLIP [39, 40, 42] variants
for multimodal learning, have significantly outperformed previous approaches
in their fields of expertise. However, it is intuitive that a proper source
domain is more crucial than the scale of pre-training data. Take CIFAR100 [25]
for example, CIFAR10 [25] may be a better source than ImageNet [41].
Therefore, researchers are dedicated to designing transferability estimating
algorithms to determine the best source for the target domain efficiently.

To estimate transferability, researchers have proposed several transferability
metrics, including LEEP [35], LogMe [55], H-score [1], NCE [49], and GBC [36].
These metrics utilize the distribution in label and feature spaces of the
target domain. However, we found that sampling from the target domain with
certain approaches as previous works [35, 36] could change the data
distribution, which may lead to inferior results lacking in stability and
accuracy. Furthermore, the calculations of mentioned metrics demand the
distributions in feature spaces of pre-trained models. But the pre-training
process is time-consuming, which is harmful to efficiency.

As mentioned above, existing metrics can not reach a trade-off between
performance and cost, leading to low usability. To comprehensively evaluate
these estimation methods, we summarize three properties that a good
transferability estimation should possess for the first time. We consider that
a qualified measurement should be stable, reliable, and efficient. Stability
implies that the transferability metric value would not change violently with
a slight oscillation of data distribution. Reliability reflects the accuracy
of selecting a proper source domain. Efficiency requires concise calculations
and low time costs.

In this paper, we suggest a straightforward yet effective technique for
transferability estimation that satisfies the three properties, dubbed
Principal Gradient Expectation (PGE). Similarly to previous metric methods
[35, 55, 1, 49, 36], our approach applies to the scenario where both the
source and target tasks are a single task. Specifically, we consider the
optimization procedure to be an approximation between the initial and the
optimal state in the parameter space. Moreover, the optimal state gap between
the source and target domain indicates transferability. However, as shown in
Figure LABEL:idea, the optimal point of the target is invisible. To quantify
the gap, we collect first-order gradients of the model’s backbone multiple
times and compute the expectation of all gradients (on the right of Figure
LABEL:idea). The expectation is named principal gradient expectation, which
guarantees stability. Then, we design a scheme to compute the gap between
principal gradients. Specifically, we deflate the gap of the principal
gradient expectations between the source and target by Schwarz inequality [46]
to achieve a lower bound as the quantitative transfer gap. In the calculation
process of the proposed metric described above, the three properties are
guaranteed. As we do not utilize the distribution of the target domain and
collect gradients multiple times to compute the expectation of gradient, our
PGE could produce stable results. Considering that supervised pre-training is
more sensitive than unsupervised pre-training [57, 22], we obtain the first-
order gradient in an unsupervised mode. This operation facilitates the
promotion of accuracy in selecting a proper source domain. In addition, the
process of estimating transferability with principal gradient expectation does
not depend on pre-training parameters, which helpfully reduces the time cost.
In the experiments, we show that our proposed method outperforms state-of-the-
art methods in terms of stability, reliability and efficiency. To conclude,
the main contributions of this work can be summarized as follows:

  * •

For the first time, we have summarized the properties that a qualified
transferability metric should satisfy and proposed corresponding evaluation
methods. The properties are stability, reliability and efficiency.

  * •

We propose a novel approach (PGE) based on the principal gradient expectation
for assessing transferability. Moreover, our method satisfied the above
properties.

  * •

We show through experiments that the proposed PGE is strongly related to the
transferred performance. Our approach is simple yet effective and shows its
superiority over other methods.

##  2 Related Work

Our research is closely aligned with the techniques of network adaptation and
transferability estimation. Consequently, we will now delve into the body of
literature surrounding these fields.

###  2.1 Network Adaptation

Network adaptation is prevalent in the computer vision field nowadays. This
technique involves adapting a pre-trained neural network to suit a new task in
situations where data is scarce. Linear probing (LP) and Fine-tuning (FT) are
typical methods for network adaptation. In the training process of LP, the
pre-trained parameters are frozen, while training the head layers associated
with the task [2, 37]. As for FT, the parameters of the entire network are
fine-tuned for downstream tasks to improve transfer accuracy [21]. Researchers
have also explored various methods to improve network adaptation. To alleviate
overfitting, [56]combined a lightweight “side” network with pre-trained
models. [26]integrated each layer of a model into the fine-tuned model. This
method can balance the performance of the source and target without increasing
the model size. [4]improved the ability of a model to extract features by
learning residual feature maps and significantly lessened memory usage.
Upstream bias mitigation(UBM) [23] has been found to benefit fine-tuning
language models. Sparse pruning [9]reduced the predictive error via allocating
pruning weights according to the zero-value weights in each layer.
[18]evaluated three pruning methods and found simple method may yield better
results. Meta-learning has been applied to network adaptation and training
dynamic networks [29]. Some studies [50, 43] have combined meta-learning with
reinforcement learning to improve the performance on new tasks. Recent work
[17] generalized given tasks to new tasks.

In addition, Neural Architecture Search(NAS) has also received significant
attention. Previous NAS works [59, 48, 30] are based on reinforcement learning
algorithms and their cost was high. ENAS [38] simplified the search process by
sharing parameters but it led to lower accuracy. Different NAS works [30, 11,
15] have attempted to reduce the search cost and improve accuracy in recent
years. [16] achieved good results by automatically adjusting the network
architecture to adapt to new tasks. More recently, [27, 28]utilized prompts
which consist of task-specific vectors and optimized prompts via gradients.
Different studies [24, 19, 54] applied prompts to multimodal tasks.
[45]trained vision transformers with prompts and generative knowledge. In this
work, we adopt LP and FT as the adaptation approaches to evaluate our
algorithm by measuring the correlation between the adaptation results and the
transferability computed with the principal gradient.

###  2.2 Transferability Estimation

To properly evaluate the transferability of pre-trained models, Negative
Conditional Entropy (NCE) [49] utilizes a metric derived from information
theory to measure the transferability and difficulty of classification tasks.
NCE assumes that the images in the source domain and the target domain are the
same, but their labels are distinct. They then calculate the transferability
score with the negative conditional entropy between the target and the source
labels. Contemporary work H-score [1] leverages statistics and information
theory to quantify the transferability of feature representations. However,
H-score can only be employed in classification tasks. While LEEP [35] focuses
on computing the joint probability over pseudo labels and the target labels to
yield the log expectation of the empirical predictor. But LEEP will obtain
different transferability scores when models with identical feature extractors
and distinct classification heads. LogMe [55] solve this issue by directly
estimating the maximum value of label evidence given features extracted by
pre-trained models. It models each target label as a linear model with
Gaussian noise, and then optimizes the prior distribution parameters to obtain
the average maximum (log) evidence of labels given the target instances’
embeddings. Recently, GBC [36] desires to measure the amount of overlap
between target classes in the feature space of the pre-trained model. While in
this work, the gradients of the model’s backbone are employed in our
estimation instead of distribution.

##  3 Method

###  3.1 Preliminaries

Considering a deep model $\boldsymbol{\mathit{M}}(w,h)$, we denote the feature
extractor as $w$ and the task-related layer as $h$. For $N$ alternative
sources $\\{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})\\}_{i=1}^{N}$,
$\mathcal{D}_{s(i)}$ and $\mathcal{T}_{s(i)}$ represent the source domain and
the source task, respectively. The target is denoted as
$(\mathcal{D}_{t},\mathcal{T}_{t})$, where $\mathcal{D}_{t}$ and
$\mathcal{T}_{t}$ separately indicate the target domain and the target task.
The goal of this work is to determine the appropriate source via quantify the
transferability between each source $(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})$
and the target $(\mathcal{D}_{t},\mathcal{T}_{t})$, as selecting a proper
source is critical to improving efficiency and effectiveness. Moreover, we use
$\mathcal{G}[(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)});(\mathcal{D}_{t},\mathcal{T}_{t})]$
to denote the transfer gap from an available source to the target.

After analyzing the previous methods [49, 1, 35, 55, 36], we recapitulate
three characteristics of a good transferability metric: stability,
reliability, and efficiency. Next, we provide a comprehensive explanation of
these propositions.

Proposition 1 (Stability). Let I denote a random sampling function (further
details about the sampling methods are described in Sec. 3.4).
$\emph{I}(\mathcal{D}_{t})$ represents a subset drawn according to
$\mathcal{D}_{t}$. The stability indicating the difference between
$\mathcal{G}[(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)});(\boldsymbol{\mathcal{D}_{t}},\mathcal{T}_{t})]$
and
$\mathcal{G}[(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)});(\boldsymbol{\emph{I}(\mathcal{D}_{t})},\mathcal{T}_{t})]$
is bounded, which can be described as:

$|\mathcal{G}[(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)});(\boldsymbol{\mathcal{D}_{t}},\mathcal{T}_{t})]-\mathcal{G}[(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)});(\boldsymbol{\emph{I}(\mathcal{D}_{t})},\mathcal{T}_{t})]|\leq\epsilon.$
(1)

Stability is essential since a finite dataset is a sub-sample in the manifold
space of the data distribution.

Proposition 2 (Reliability).
$\mathcal{A}[(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)});(\mathcal{D}_{t},\mathcal{T}_{t})]$
is defined as the performance of the target domain transferred from an
available source. Moreover, a lower transfer gap indicates better transfer
performance. Thus,
$\mathcal{G}[(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)});(\mathcal{D}_{t},\mathcal{T}_{t})]<\mathcal{G}[(\mathcal{D}_{s(j)},\mathcal{T}_{s(j)});(\mathcal{D}_{t},\mathcal{T}_{t})]$
always enables
$\mathcal{A}[(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)});(\mathcal{D}_{t},\mathcal{T}_{t})]>\mathcal{A}[(\mathcal{D}_{s(j)},\mathcal{T}_{s(j)});(\mathcal{D}_{t},\mathcal{T}_{t})]$.

Proposition 3 (Efficiency). The estimation of the transferability from source
to target is supposed to have low computation and complexity.

Previous studies such as [57, 22], have demonstrated that the transferability
is mainly related to the model’s backbone. However, existing methods employ
task-specific layers $h$ besides the model’s backbone to calculate the
transferability, which may harm reliability and lead to high costs. Our method
only utilizes the gradient of the backbone to estimate the transferability.
Furthermore, to reach a trade-off between computing efficiency and efficacy,
we use the first-order approximation of the loss.

###  3.2 Principal Gradient Expectation (PGE)

We consider the optimization procedure to be a distance approximation between
the initial and the optimal points in the parameter space. To balance
efficiency and effectiveness, we adopt the first-order gradient expectation.
Given a model $\boldsymbol{\mathit{M}}$ and a target
$(\mathcal{D}_{t},\mathcal{T}_{t})$, $\mathcal{L}$ denotes the loss function.
The model is initialized with random weights $\theta_{0}$ where
$\theta_{0}\sim\mathscr{N}(\mathbf{0},I)$. We first compute the gradient of
backbone parameters, denoted as $\nabla\mathcal{L}(\theta_{0})$. Instead of
updating model parameters $\theta$ with $\nabla\mathcal{L}(\theta_{0})$, we
then re-initialize the weights of the model and collect
$\nabla\mathcal{L}(\theta_{0})$ multiple times to compute the expectation of
gradients. We believe gradients reflect the inherent characteristics of the
source task (target task). Furthermore, re-initializing multiple times and
computing the expectation reduces the impact of abnormal gradients. This
expectation value is defined as Principal Gradient Expectation (PGE), which
can be formulated as follows:

$PGE=\mathbb{E}_{\theta_{0}}[\nabla\mathcal{L}(\theta_{0})].$ (2)

By making use of the definition of PGE, we compute the PGE for source
$(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})$ and target
$(\mathcal{D}_{t},\mathcal{T}_{t})$ with their own loss, respectively:

$PGE_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}=\mathbb{E}_{\theta_{0}}[\nabla\mathcal{L}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}(\theta_{0})],$
(3)
$PGE_{(\mathcal{D}_{t},\mathcal{T}_{t})}=\mathbb{E}_{\theta_{0}}[\nabla\mathcal{L}_{(\mathcal{D}_{t},\mathcal{T}_{t})}(\theta_{0})].$
(4)

###  3.3 Transferability Metric based on PGE

In our transfer setting, the restriction for the source and target tasks is
that the backbone of models must be the same. Moreover, we require the source
task $\mathcal{T}_{s}$ and target task $\mathcal{T}_{t}$ to be a single task
as in most existing works. And $\mathcal{T}_{t}$ is not necessary to be the
same as $\mathcal{T}_{s}$. So computing transferability with PGE is task-
irrelevant.

We consider a situation in which the model $\boldsymbol{\mathit{M}}$ only has
one single parameter $\theta$. The initialized value of $\theta$ is denoted as
$\theta_{0}$ and the optimal value is denoted $\theta^{*}$.
$\mathcal{L}(\theta_{0})$ is the initial loss value at $\theta_{0}$ and
$\mathcal{L}(\theta^{*})$ is the loss value at $\theta^{*}$.
$\mathcal{L}^{{}^{\prime}}(\theta_{0})$ represents the derivative at
$\theta_{0}$. According to Taylor’s formula, the loss value at $\theta^{*}$
could be expressed in the following form:

$\mathcal{L}(\theta^{*})=\mathcal{L}(\theta_{0})+\mathcal{L}^{{}^{\prime}}(\theta_{0})(\theta^{*}-\theta_{0})+R(\theta_{0}).$
(5)

$R(\theta_{0})$ is the remainder beyond the first-order approximation. Then we
rewrite the Eq. 5.

$(\theta^{*}-\theta_{0})+\frac{R(\theta_{0})}{\mathcal{L}^{{}^{\prime}}(\theta_{0})}=\frac{\mathcal{L}(\theta^{*})-\mathcal{L}(\theta_{0})}{\mathcal{L}^{{}^{\prime}}(\theta_{0})}.$
(6)

In Eq. 6, $(\theta^{*}-\theta_{0})$ can be regarded as the optimization
distance of $\theta$ from the initial state to the optimal state. However,
since the optimal parameter $\theta^{*}$ is always invisible in practice, we
require a more simplified definition for the gap. The term
$\frac{R(\theta_{0})}{\mathcal{L}^{{}^{\prime}}(\theta_{0})}$ contains high-
order derivatives, which implies the complexity of the optimization surface.
We propose a hypothesis that the optimization distance and the optimization
surface’s complexity can reflect the optimization difficulty of $\theta$, and
they are proportional to $1/\mathcal{L}^{{}^{\prime}}(\theta_{0})$. Therefore,
we define $1/\mathcal{L}^{{}^{\prime}}(\theta_{0})$ as a factor of the
optimization difficulty from $\theta_{0}$ to $\theta^{*}$ as follows:

$\mathcal{f}(\theta^{*},\theta_{0})=\frac{1}{\mathcal{L}^{{}^{\prime}}(\theta_{0})}.$
(7)

Next, we utilize the factors on both source
$(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})$ and target
$(\mathcal{D}_{t},\mathcal{T}_{t})$.

$\mathcal{f}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}(\theta^{*},\theta_{0})=\frac{1}{\mathcal{L}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}^{{}^{\prime}}(\theta_{0})}.$
(8)

$\mathcal{f}_{(\mathcal{D}_{t},\mathcal{T}_{t})}(\theta^{*},\theta_{0})=\frac{1}{\mathcal{L}_{(\mathcal{D}_{t},\mathcal{T}_{t})}^{{}^{\prime}}(\theta_{0})}.$
(9)

We define the gap between
$\theta_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}^{*}$ and
$\theta_{(\mathcal{D}_{t},\mathcal{T}_{t})}^{*}$ as
$\mathcal{g}(\theta_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}^{*},\theta_{(\mathcal{D}_{t},\mathcal{T}_{t})}^{*})$,
where $\theta_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}^{*}$ and
$\theta_{(\mathcal{D}_{t},\mathcal{T}_{t})}^{*}$ represents the optimal model
parameters on the source and target, respectively. Since the value of the
factor could be positive or negative, the absolute value of the two factors’
subtraction is used to represent the gap between
$\mathcal{f}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}(\theta^{*},\theta_{0})$
and $\mathcal{f}_{(\mathcal{D}_{t},\mathcal{T}_{t})}(\theta^{*},\theta_{0})$.

$\displaystyle\mathcal{g}(\theta_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}^{*},\theta_{(\mathcal{D}_{t},\mathcal{T}_{t})}^{*})$
$\displaystyle=\quad$
$\displaystyle\lvert\mathcal{f}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}(\theta^{*},\theta_{0})-\mathcal{f}_{(\mathcal{D}_{t},\mathcal{T}_{t})}(\theta^{*},\theta_{0})\rvert$
$\displaystyle=\quad$
$\displaystyle\lvert\frac{1}{\mathcal{L}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}^{{}^{\prime}}(\theta_{0})}-\frac{1}{\mathcal{L}_{(\mathcal{D}_{t},\mathcal{T}_{t})}^{{}^{\prime}}(\theta_{0})}\rvert$
$\displaystyle=\quad$
$\displaystyle\lvert\frac{\mathcal{L}_{(\mathcal{D}_{t},\mathcal{T}_{t})}^{{}^{\prime}}(\theta_{0})-\mathcal{L}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}^{{}^{\prime}}(\theta_{0})}{\mathcal{L}_{(\mathcal{D}_{t},\mathcal{T}_{t})}^{{}^{\prime}}(\theta_{0})\mathcal{L}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}^{{}^{\prime}}(\theta_{0})}\rvert$
$\displaystyle=\quad$
$\displaystyle\frac{\lvert\mathcal{L}_{(\mathcal{D}_{t},\mathcal{T}_{t})}^{{}^{\prime}}(\theta_{0})-\mathcal{L}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}^{{}^{\prime}}(\theta_{0})\rvert}{\lvert\mathcal{L}_{(\mathcal{D}_{t},\mathcal{T}_{t})}^{{}^{\prime}}(\theta_{0})\mathcal{L}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}^{{}^{\prime}}(\theta_{0})\rvert}.$
(10)

In practice, a model has numerous parameters, thus we expand the above
derivation to a high dimension version
$\mathcal{G}^{\prime}[(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)});(\mathcal{D}_{t},\mathcal{T}_{t})]$.

$\displaystyle\mathcal{G}^{\prime}[(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)});(\mathcal{D}_{t},\mathcal{T}_{t})]$
$\displaystyle=\quad$
$\displaystyle\parallel\mathcal{F}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}(\theta^{*},\theta_{0})-\mathcal{F}_{(\mathcal{D}_{t},\mathcal{T}_{t})}(\theta^{*},\theta_{0})\parallel_{2}$
$\displaystyle=\quad$
$\displaystyle\frac{\parallel\nabla\mathcal{L}_{(\mathcal{D}_{t},\mathcal{T}_{t})}(\theta_{0})-\nabla\mathcal{L}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}(\theta_{0})\parallel_{2}}{\parallel\nabla\mathcal{L}_{(\mathcal{D}_{t},\mathcal{T}_{t})}(\theta_{0})\enspace\nabla\mathcal{L}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}(\theta_{0})\parallel_{2}}.$
(11)

Similar with Eq. 7,
$\mathcal{F}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}(\theta^{*},\theta_{0})$
and $\mathcal{F}_{(\mathcal{D}_{t},\mathcal{T}_{t})}(\theta^{*},\theta_{0})$
are two vectors, measuring the optimization difficulty from $\theta_{0}$ to
$\theta_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}^{*}$ and
$\theta_{(\mathcal{D}_{t},\mathcal{T}_{t})}^{*}$, respectively. We then
deflate
$\frac{\parallel\nabla\mathcal{L}_{(\mathcal{D}_{t},\mathcal{T}_{t})}(\theta_{0})-\nabla\mathcal{L}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}(\theta_{0})\parallel_{2}}{\parallel\nabla\mathcal{L}_{(\mathcal{D}_{t},\mathcal{T}_{t})}(\theta_{0})\enspace\nabla\mathcal{L}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}(\theta_{0})\parallel_{2}}$
with Schwarz inequality [46].

$\frac{\parallel\nabla\mathcal{L}_{(\mathcal{D}_{t},\mathcal{T}_{t})}(\theta_{0})-\nabla\mathcal{L}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}(\theta_{0})\parallel_{2}}{\parallel\nabla\mathcal{L}_{(\mathcal{D}_{t},\mathcal{T}_{t})}(\theta_{0})\enspace\nabla\mathcal{L}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}(\theta_{0})\parallel_{2}}\quad\geq$
$\\\
\quad\quad\quad\quad\quad\quad\frac{\parallel\nabla\mathcal{L}_{(\mathcal{D}_{t},\mathcal{T}_{t})}(\theta_{0})-\nabla\mathcal{L}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}(\theta_{0})\parallel_{2}}{\parallel\nabla\mathcal{L}_{(\mathcal{D}_{t},\mathcal{T}_{t})}(\theta_{0})\parallel_{2}\enspace\parallel\nabla\mathcal{L}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}(\theta_{0})\parallel_{2}}.$
(12)

Considering calculating the gradient once might obtain abnormal gradient, we
employ PGE (Sec. 3.2) to calculate the transferability gap which is defined as
$\mathcal{G}[(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)});(\mathcal{D}_{t},\mathcal{T}_{t})]$.

$\displaystyle\mathcal{G}[(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)});(\mathcal{D}_{t},\mathcal{T}_{t})]=\frac{\parallel
PGE_{(\mathcal{D}_{t},\mathcal{T}_{t})}-PGE_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}\parallel_{2}}{\parallel
PGE_{(\mathcal{D}_{t},\mathcal{T}_{t})}\parallel_{2}\enspace\parallel
PGE_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}\parallel_{2}}$
$\displaystyle=\frac{\parallel\mathbb{E}_{\theta_{0}}\left[\nabla\mathcal{L}_{(\mathcal{D}_{t},\mathcal{T}_{t})}(\theta_{0})\right]-\mathbb{E}_{\theta_{0}}\left[\nabla\mathcal{L}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}(\theta_{0})\right]\parallel_{2}}{\parallel\mathbb{E}_{\theta_{0}}\left[\nabla\mathcal{L}_{(\mathcal{D}_{t},\mathcal{T}_{t})}(\theta_{0})\right]\parallel_{2}\enspace\parallel\mathbb{E}_{\theta_{0}}\left[\nabla\mathcal{L}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}(\theta_{0})\right]\parallel_{2}}.$
(13)

It should be noted that only the gradient of the model’s backbone $w$ is
applied when calculating
$\mathcal{G}[(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)});(\mathcal{D}_{t},\mathcal{T}_{t})]$.
Essentially, this metric measures the disparity between
$(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})$ and
$(\mathcal{D}_{t},\mathcal{T}_{t})$. Finally, we calculate the transferability
scores between each $(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})$ in
$\\{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})\\}_{i=1}^{N}$ and
$(\mathcal{D}_{t},\mathcal{T}_{t})$ with Eq. 3.3. The algorithm for acquiring
the transferability score with PGE is outlined in Algorithm 1.

Algorithm 1 Principal Gradient Expectation

A model $\boldsymbol{\mathit{M}}$ with random initialization $\theta_{0}$,
several different source
$\\{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})\\}_{i=1}^{N}$, and a target
$(\mathcal{D}_{t},\mathcal{T}_{t})$.

Ranking of the results of transfer of all sources to the target with
$\boldsymbol{\mathit{M}}$.

for $n=1\to N+1$ do //$N+1$ denotes N sources and a target.

for $i=1\to I$ do //$I$ denotes the number of iterations.

Input some instances into the model $\boldsymbol{\mathit{M}}$.

Computing $\nabla\mathcal{L}(\theta_{0})$

$\mathbb{E}_{\theta_{0}}[\nabla\mathcal{L}(\theta_{0})]\leftarrow((i-1)\ast\mathbb{E}_{\theta_{0}}[\nabla\mathcal{L}(\theta_{0})]+\nabla\mathcal{L}(\theta_{0}))/i$
//The gradients obtained this time are added to the previously collected
gradients and averaged.

end for

end for

for $i=1\to N$ do //$N$ denotes the number of sources

Calculate
$\frac{\parallel\mathbb{E}_{\theta_{0}}\left[\nabla\mathcal{L}_{(\mathcal{D}_{t},\mathcal{T}_{t})}(\theta_{0})\right]-\mathbb{E}_{\theta_{0}}\left[\nabla\mathcal{L}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}(\theta_{0})\right]\parallel_{2}}{\parallel\mathbb{E}_{\theta_{0}}\left[\nabla\mathcal{L}_{(\mathcal{D}_{t},\mathcal{T}_{t})}(\theta_{0})\right]\parallel_{2}\enspace\parallel\mathbb{E}_{\theta_{0}}\left[\nabla\mathcal{L}_{(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)})}(\theta_{0})\right]\parallel_{2}}$

end for

Sort

return Ranking

###  3.4 Robust Evaluation with Multiple Subsampling

In this section, we introduce the standardized evaluation for the
transferability between different domains.

We define $\mathcal{A}$ as the performance on the target transferred from the
pre-trained model. If
$\mathcal{A}((\mathcal{D}_{s(i)},\mathcal{T}_{s(i)});(\mathcal{D}_{t},\mathcal{T}_{t}))>\mathcal{A}((\mathcal{D}_{s(j)},\mathcal{T}_{s(j)});(\mathcal{D}_{t},\mathcal{T}_{t})))$,
we expect
$\mathcal{G}[(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)});(\mathcal{D}_{t},\mathcal{T}_{t})]<\mathcal{G}[(\mathcal{D}_{s(j)},\mathcal{T}_{s(j)});(\mathcal{D}_{t},\mathcal{T}_{t})]$.
The reliability of our method is validated by computing the correlations
between the ranking of
$\\{\mathcal{G}[(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)});(\mathcal{D}_{t},\mathcal{T}_{t})]\\}_{i=1}^{N}$
and the ranking of
$\\{\mathcal{A}((\mathcal{D}_{s(i)},\mathcal{T}_{s(i)});(\mathcal{D}_{t},\mathcal{T}_{t}))\\}_{i=1}^{N}$.
Kendall’s $\tau$ [14] coefficient, a measure of rank correlation, quantifies
the similarity between two rankings. Kendall’s $\tau$ coefficient is defined
as:

${\displaystyle\tau={\frac{2}{n(n-1)}}\sum_{i<j}\operatorname{sgn}(x_{i}-x_{j})\operatorname{sgn}(y_{i}-y_{j})}.$
(14)

$x_{i}\in\mathcal{X}$ and $y_{i}\in\mathcal{Y}$ ($\mathcal{X}$ and
$\mathcal{Y}$ represent two rankings respectively). And n indicates the number
of items in the ranking. The $sgn(\cdot)$ function is a symbolic function.
Specifically, the range of $\tau$ is $\left[-1,1\right]$, a higher $\tau$
indicates a stronger correlation between
$\\{\mathcal{G}[(\mathcal{D}_{s(i)},\mathcal{T}_{s(i)});(\mathcal{D}_{t},\mathcal{T}_{t})]\\}_{i=1}^{N}$
and
$\\{\mathcal{A}((\mathcal{D}_{s(i)},\mathcal{T}_{s(i)});(\mathcal{D}_{t},\mathcal{T}_{t}))\\}_{i=1}^{N}$.
$\tau=0$ shows no correlation between them.

Following [35, 36], we generate multiple subsets of the target domain
$\mathcal{D}_{t}$ with two approaches: $\emph{I}_{1}$ randomly selects
$\eta_{1}\%$ categories in $\mathcal{D}_{t}$ and all samples in the selected
categories are used. $\emph{I}_{2}$ randomly selects $\eta_{2}\%$ samples from
each category in $\mathcal{D}_{t}$. Concretely, we construct one hundred
different $(\mathcal{D}_{t},\mathcal{T}_{t})$ from $\mathcal{D}_{t}$ with the
two approaches mentioned above and evaluate the performance of the pre-trained
model with the selected target test set. A curve is drawn in which the
horizontal axis represents the sizes of the subsets and the vertical axis
shows the performance of the different test sets. We use the area under the
curve as the measurement of the
$\mathcal{A}((\mathcal{D}_{s(i)},\mathcal{T}_{s(i)});(\mathcal{D}_{t},\mathcal{T}_{t}))$.

(a) The subsets are constructed by SI

(b) The subsets are constructed by SII

Figure 2:  The stability comparison of different methods. $\uparrow$ denotes
that a higher value represents a better source, and $\downarrow$ denotes that
a lower value indicates a better source. We show the transferability scores’
variation of different measurement techniques along with increasing sampling
ratios of subsets. The top row of the figure illustrates that we build target
subsets with strategy SI on CIFAR100. We compare the results with five
existing transferability metrics. In each plot, the horizontal axis means the
sampling ratio of each subset while the vertical axis is the transferability
score. The bottom row of the figure indicates that we construct subsets with
strategy SII on CIFAR100.

##  4 Experiment

In this section, we scrutinize and affirm the proposed transferability metric
with diverse image data. Initially, we introduce the experimental settings,
followed by an evaluation of the proposed PGE from four key perspectives:
stability, reliability, efficiency, and generalizability.

###  4.1 Experimental Settings

Datasets. Our experiments are conducted on the following eight datasets:
CIFAR10 [25], CIFAR100 [25], STL10 [6], Mini-ImageNet [51], CUB [52], MNIST
[8], FGVC-Aircraft (Aircraft) [34], and PASCAL VOC 2012 [13]. Table 1 provides
a summary of the characteristics of these datasets, including resolution size
(Resolution), number of images (Img), and number of classes (Class).

Dataset | Resolution | Img | Class
---|---|---|---
CIFAR10 [25] | $32\times 32$ | 60k | 10
CIFAR100 [25] | $32\times 32$ | 60k | 100
STL10 [6] | $96\times 96$ | 13k | 10
Mini-ImageNet [51] | $84\times 84$ | 60k | 200
CUB [52] | $512\times 512$ | 11.7k | 200
MNIST [8] | $28\times 28$ | 70k | 10
Aircraft [34] | $512\times 512$ | 10k | 100
PASCAL VOC 2012 [13] | $320\times 480$ | 1.7k | 21
Table 1: The summary of datasets.

Transfer Methods. While previous studies have concentrated more on linear
probing, we examine both fine-tuning and linear probing.

(1) _Linear Probing_. This technique involves freezing the feature extractor
of the model and training a task-specific layer from scratch with the target
dataset.

(2) _Fine-Tuning_. Here, we replace the model’s task-specific layer with a new
one and fine-tune the entire model, including the feature extractor and the
task-specific layer, on the target dataset.

Evaluation. We construct the subsets with two distinct methods as follows:

SI: The first approach randomly samples 5% to 100% of the target categories
and uses all images within these categories.

SII: The second approach randomly selects a percentage between 10% to 100%
images within each category.

Notably, we do not sample all target datasets with either of the two sampling
approaches. And we generate subsets with SI when the number of samples in each
category is relatively small.

Implementation Details. We run 600 epochs of linear probing and fine-tuning
for each subset of the target domain (using SGD without Momentum and Cosine
annealing) [33]. And we adjust the learning rate with an initial one as
$0.1\ast batchsize/256$. Three classic backbones are adopted in the
experiments, i.e., Resnet18 [20], Resnet50 [20], and VGG16 [44]. Additional
experiments with more datasets and large-scale models are shown in Appendix.

###  4.2 Stability Comparison

In this section, we compare the proposed PGE with existing techniques,
including LEEP [35], LogMe [55], H-score [1], NCE [49], and GBC [36].
Moreover, we use CIFAR10, STL10, Mini-ImageNet, CUB, and MNIST as source
datasets and CIFAR100 as the target dataset.

Figure 2 shows the variation of transfer scores among different measurement
techniques when increasing the sampling ratio of each subset. It is
acknowledged that even with different subsets of a target domain, the
measurement results should be stable. In Figure 2(a), the subsets are
constructed by SI. We observed that as the sampling ratio arises, H-score [1]
and LogMe [55] increase while LEEP, NCE, and GBC decrease. By contrast, our
proposed PGE is superior to all compared methods for stability, as the
sampling ratio has no effect on its outputs. So we argue that the distribution
of the target domain is an important factor to compute the transferability for
the compared methods, as the scores vary with different distributions. Since
our calculation does not involve the distribution, it is more stable than
others. Moreover, the curves of LEEP [35], LogMe [55], and NCE [49] almost
overlap when randomly sampling categories over CIFAR100 as SI, which
demonstrate that it’s hard to discriminate which is the best source for the
target because they tend to yield similar transferability scores. In addition,
although H-score and GBC gain the same correct source (i.e., CIFAR 10) as our
PGE, PGE produces more stable and distinguishable results. And we can get the
correct result with only a portion of categories from the target dataset. In
Figure 2(b), the subsets are constructed by SII. A similar conclusion could be
drawn as in Figure 2(a) that PGE shows remarkable advantages over other
methods. We think these approaches may fail when the number of images is small
since the data distribution of a small target dataset usually cannot represent
the true distribution of real-world data. In comparison, PGE uses the
expectation of the principle gradient to estimate the transferability gap, as
the expectation can effectively reduce the impact of abnormal gradients.
Therefore, the proposed PGE can still obtain the correct result. In summary,
the proposed PGE yields more consistent and distinguishable results for both
strategies.

| CIFAR10 [25] | STL10 [6] | CIFAR100 [25] | CUB [52] | Aircraft [34] | Average Kendall’s $\tau$
---|---|---|---|---|---|---
| LP | FT | LP | FT | LP | FT | LP | FT | LP | FT
LEEP) [35] | 0.19 | 0 | 0 | 0 | 0 | -0.6 | 0.19 | 0.2 | -0.19 | -0.19 | -0.04
H-score) [1] | 1 | 0.79 | 1 | 1 | 1 | 0 | -0.19 | 0.19 | 0.19 | 0.19 | 0.52
NCE) [49] | 0.39 | 0.19 | -0.2 | -0.2 | 0 | -0.6 | -0.19 | -0.39 | -0.79 | -0.79 | -0.26
LogMe) [55] | 1 | 0.79 | 1 | 1 | 1 | 0 | -0.39 | -0.4 | -0.79 | -0.4 | 0.281
GBC) [36] | 1 | 0.79 | 0.4 | 0.4 | 1 | 0 | 0.39 | 0.39 | 0.19 | -0.19 | 0.44
PGE (Ours) | 1 | 0.79 | 1 | 1 | 1 | 0 | 0.39 | 0.39 | 0.79 | 0.39 | 0.68
|  |  |  |  |  |  |  |  |  |  |
Table 2: The reliability comparison of different methods. The value (higher is better) indicates the correlation between the ranking calculated by the transferability estimation technique and the ranking of transfer performances. Both the source and target tasks are image classification. And we describe the results for five target domains with two transfer methods (Linear Probing (LP) and Fine-Tuning (FT)). It could be found that the proposed method obtains the highest Kendall’s $\tau$) [14] coefficient in most experiments and the highest average Kendall’s $\tau$ [14] coefficient for the different domains. |  PGE Gap $\downarrow$ ($10^{-2}$) | LP(%) | FT(%)
---|---|---|---
CIFAR100 | 0.284 | 50.57 | 66.69
STL10 | 0.773 | 39.86 | 65.39
Mini-ImageNet | 1.118 | 32.04 | 65.28
CUB | 1.125 | 26.20 | 60.67
MNIST | 4.842 | 17.32 | 62.66
|  |  $\tau$ : 1 |  $\tau$ : 0.79
|  |  |
Table 3:  The reliability of PGE scores. $\downarrow$ denotes that a lower gap which represents a better source. Transfer performances were obtained with two transfer methods, i.e., Linear Probing (LP) and Fine-Tuning (FT).  |  PGE Gap $\downarrow$ ($10^{-2}$) | MIoU(%)
---|---|---
CIFAR100 | 3.86 | 65.09
CIFAR10 | 4.04 | 65.03
Mini-ImageNet | 1.17 | 67.85
STL10 | 1.23 | 66.07
CUB | 1.18 | 65.53
|  $\tau$ : 0.79
|  |
Table 4: The generalizability of PGE with PASCAL VOC 2012 dataset.
$\downarrow$ denotes that a lower gap which represents a better source.

###  4.3 Reliability Comparison

In this section, we adopt the consistency between the estimated results and
the real transfer performances to compare the reliability of different
methods. As mentioned in Section 3, the transferability is supposed to
correlate well with the final performance of a model after fine-tuning/linear
probing on the target task. To validate the reliability of the proposed PGE,
we conduct experiments with both fine-tuning and linear probing on CIFAR10,
CIFAR100, STL10, CUB, and Aircraft. We choose five different sources for each
target. As shown in Table 2, the relevance between the transfer performances
and transferability is computed with Kendall’s $\tau$ [14] coefficient. In the
linear probing process, PGE identifies the optimal source for all five
targets. In the fine-tuning process, PGE determines the optimal source for
four targets. Moreover, we achieve the highest correlation for CIFAR10, CUB,
and Aircraft with both fine-tuning and linear probing. Overall, the proposed
PGE obtains the highest average correlation of $0.68$ among all existing
approaches.

The results in Table 2 reveal that H-score and LogMe can produce competitive
results on simple datasets (CIFAR10, STL10, and CIFAR100). However, most of
the existing approaches show poor performance on more challenging datasets
(CUB and Aircraft). We consider that existing methods are sensitive to the
resolution of the images while the proposed PGE is robust enough to alleviate
this issue.

Specifically, Table 3 presents the results of the estimated transferability
and the transfer performance on CIFAR10. When linear probing is adopted, the
transfer performances completely match ($\tau=1$) the ranking of the PGE
results. For fine-tuning, we also obtain Kendall’s $\tau=0.79$, illustrating
the transfer performances and PGE results are well-aligned.

Pre-trained task | Transfer Target | H-score [1] | LEEP [35] | NCE [49] | GBC [36] | LogMe [55] | PGE (ours)
---|---|---|---|---|---|---|---
Classification | Classification | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Classification | Regression | ✗ | ✗ | ✗ | ✗ | ✓ | ✓
Unsupervised | Classification | ✗ | ✗ | ✗ | ✗ | ✓ | ✓
Unsupervised | Regression | ✗ | ✗ | ✗ | ✗ | ✓ | ✓
|  |  |  |  |  |  |

Table 5: Generalizability of different techniques for different transfer
settings. “✓ / ✗” indicates whether the technique is applicable/unapplicable
for this transfer setting.

| PGE Gap $\downarrow$ (Sup.) | PGE Gap $\downarrow$ ( Uns. ($10^{-2}$)) | LP (%) | FT (%)
---|---|---|---|---
Mini-ImageNet | 4.35 | 5.00 | 48.21 | 61.83
CIFAR10 | 1.91 | 5.80 | 45.43 | 61.36
CIFAR100 | 2.56 | 7.79 | 45.34 | 59.47
CUB | 4.75 | 7.81 | 37.93 | 55.82
|  |  |  |

Table 6: The ablation study with supervised (Sup.) and unsupervised (Uns.)
strategies of obtaining gradients. $\downarrow$ denotes that a lower gap which
represents a better source.

###  4.4 Efficiency Comparison

Regarding a new source, the existing methods need the pre-trained parameters
trained on the new domain to measure the transferability, which is time-
consuming. In contrast, our proposed method is more computationally efficient
for the following reasons. First, by employing the gradient of the first-order
optimization, our method does not require any pre-training process for the
source. Second, by analyzing the stability of the proposed method above, we
can find that the proposed PGE can significantly reduce the computational cost
by reducing the amount of data, which is not applicable to all other methods.
Table 7 compares the efficiency. We speed up the estimation process by up to
$7\times$ faster than existing works while obtaining more accurate
transferability.

Method | H-score | LEEP | NCE | GBC | LogMe | PGE (ours)
---|---|---|---|---|---|---
Time(s) | 327159 | 327135 | 327131 | 327212 | 327299 | 45488
|  |  |  |  |  |

Table 7: The time burden of transferability estimation from CIFAR10 to
CIFAR100 with ResNet18.

###  4.5 Generalizability Comparison

We further extend our proposed PGE to compute transferability between source
and target tasks when they are different. The source domains are CIFAR10,
CIFAR100, STL10, Mini-ImageNet, and CUB while the target domain is PASCAL VOC
2012. The estimation process aims to find the most suitable source domain for
the segmentation task based on those classification tasks. The backbone here
is VGG16 and the epochs for pre-training is 100. The results are shown in
Table 4. The PGE values in Table 4 suggest that Mini-ImageNet could be a
suitable source, which is verified by the segmentation accuracy.

We list four common transfer settings, as shown in Table 5. It could be
concluded that all of the approaches can be easily adapted to classification
tasks. However, when the source task and target task are different, H-score,
LEEP, NCE and GBC can not estimate transferability. By contrast, our proposed
PGE and LogMe are more practical to adapt to this setting. For classification
and regression tasks, LogMe designs different modules while our proposed PGE
computes the transferability uniformly without pre-training.

###  4.6 Ablation Study

Recent researches [57, 22] has shown that unsupervised techniques acquire more
low-level and mid-level information. The information is more readily adaptable
to a new domain than supervised techniques. In supervised learning processes,
models tend to learn high-level semantics. In this subsection, we examine
transferability with principle gradient expectation which is obtained by
supervised and unsupervised techniques. The results are shown in Table 6. It
has been experimentally found that the supervised method is more susceptible
than the unsupervised method. A reasonable explanation for this is that
category information is embedded into the model to improve discrimination but
is harmful to calculating transferability.

##  5 Conclusion

Determining which source is the best for a particular target task is
challenging. Moreover, it is computationally costly to determine the source by
fine-tuning/linear probing all possible combinations of the sources and target
task. In this work, we summarize the properties that a good transferability
metric should possess. Building upon them, we propose a simple yet effective
transferability estimation approach termed PGE based on principal gradient
expectation. To properly evaluate the method’s validity, we applied two sub-
sampling techniques to the target domain. The experimental results on both
fine-tuning and linear probing demonstrate that PGE is superior to existing
metrics. Furthermore, since PGE computes the gradient of the backbone under
the unsupervised mode, it is more flexible and can be extended to different
tasks. In the future, it is worth exploring choosing a proper source task or
even a rational composition of source tasks for a given target task.

## References

  * [1] Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, and Leonidas Guibas.  An information-theoretic approach to transferability in task transfer learning.  In 2019 IEEE International Conference on Image Processing (ICIP), pages 2309–2313. IEEE, 2019.
  * [2] Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass.  What do neural machine translation models learn about morphology?  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 861–872, Vancouver, Canada, July 2017. Association for Computational Linguistics.
  * [3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [4] Han Cai, Chuang Gan, Ligeng Zhu, and Song Han.  Tinytl: Reduce memory, not parameters for efficient on-device learning.  Advances in Neural Information Processing Systems, 33:11285–11297, 2020.
  * [5] Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640–9649, 2021.
  * [6] Adam Coates, Andrew Ng, and Honglak Lee.  An analysis of single-layer networks in unsupervised feature learning.  In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215–223. JMLR Workshop and Conference Proceedings, 2011.
  * [7] Oscar Day and Taghi M Khoshgoftaar.  A survey on heterogeneous transfer learning.  Journal of Big Data, 4:1–42, 2017.
  * [8] Li Deng.  The mnist database of handwritten digit images for machine learning research [best of the web].  IEEE signal processing magazine, 29(6):141–142, 2012.
  * [9] Tim Dettmers and Luke Zettlemoyer.  Sparse networks from scratch: Faster training without losing performance.  arXiv preprint arXiv:1907.04840, 2019.
  * [10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [11] Xuanyi Dong and Yi Yang.  Searching for a robust neural architecture in four gpu hours.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1761–1770, 2019.
  * [12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale, 2020.
  * [13] Mark Everingham, SM Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes challenge: A retrospective.  International journal of computer vision, 111(1):98–136, 2015.
  * [14] R Fagin, R Kumar, and D Sivakumar.  Comparing top k lists. in proceedings of the fourteenth annual acm-siam symposium on discrete algorithms, soda’03, 2003.
  * [15] Jiemin Fang, Yuzhu Sun, Qian Zhang, Yuan Li, Wenyu Liu, and Xinggang Wang.  Densely connected search space for more flexible neural architecture search.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10628–10637, 2020.
  * [16] Jiemin Fang, Yuzhu Sun, Qian Zhang, Kangjian Peng, Yuan Li, Wenyu Liu, and Xinggang Wang.  Fna++: Fast network adaptation via parameter remapping and architecture search.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9):2990–3004, 2020.
  * [17] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In International conference on machine learning, pages 1126–1135. PMLR, 2017.
  * [18] Trevor Gale, Erich Elsen, and Sara Hooker.  The state of sparsity in deep neural networks.  arXiv preprint arXiv:1902.09574, 2019.
  * [19] Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, and Gao Huang.  Domain adaptation via prompt learning.  arXiv preprint arXiv:2202.06687, 2022.
  * [20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [21] John Hewitt and Christopher D. Manning.  A structural probe for finding syntax in word representations.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [22] Ashraful Islam, Chun-Fu Richard Chen, Rameswar Panda, Leonid Karlinsky, Richard Radke, and Rogerio Feris.  A broad study on the transferability of visual representations with contrastive learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8845–8855, 2021.
  * [23] Xisen Jin, Francesco Barbieri, Brendan Kennedy, Aida Mostafazadeh Davani, Leonardo Neves, and Xiang Ren.  On transferability of bias mitigation effects in language model fine-tuning.  arXiv preprint arXiv:2010.12864, 2020.
  * [24] Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie.  Prompting visual-language models for efficient video understanding.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXV, pages 105–124. Springer, 2022.
  * [25] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [26] Janghyeon Lee, Donggyu Joo, Hyeong Gwon Hong, and Junmo Kim.  Residual continual learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4553–4560, 2020.
  * [27] Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  arXiv preprint arXiv:2104.08691, 2021.
  * [28] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  arXiv preprint arXiv:2101.00190, 2021.
  * [29] Yan Li, Ethan X Fang, Huan Xu, and Tuo Zhao.  International conference on learning representations 2020.  In International Conference on Learning Representations 2020, 2020\.
  * [30] Hanxiao Liu, Karen Simonyan, and Yiming Yang.  Darts: Differentiable architecture search.  arXiv preprint arXiv:1806.09055, 2018.
  * [31] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo.  Swin transformer v2: Scaling up capacity and resolution.  In International Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [32] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  * [33] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  arXiv preprint arXiv:1608.03983, 2016.
  * [34] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  arXiv preprint arXiv:1306.5151, 2013.
  * [35] Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau.  Leep: A new measure to evaluate transferability of learned representations.  In International Conference on Machine Learning, pages 7294–7305. PMLR, 2020.
  * [36] Michal Pándy, Andrea Agostinelli, Jasper Uijlings, Vittorio Ferrari, and Thomas Mensink.  Transferability estimation using bhattacharyya class separability.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9172–9182, 2022.
  * [37] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  Deep contextualized word representations.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
  * [38] Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean.  Efficient neural architecture search via parameters sharing.  In International conference on machine learning, pages 4095–4104. PMLR, 2018.
  * [39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [40] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [41] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115(3):211–252, 2015\.
  * [42] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv preprint arXiv:2205.11487, 2022.
  * [43] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap.  Meta-learning with memory-augmented neural networks.  In International conference on machine learning, pages 1842–1850. PMLR, 2016.
  * [44] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [45] Kihyuk Sohn, Yuan Hao, José Lezama, Luisa Polania, Huiwen Chang, Han Zhang, Irfan Essa, and Lu Jiang.  Visual prompt tuning for generative transfer learning.  arXiv preprint arXiv:2210.00990, 2022.
  * [46] J Michael Steele.  The Cauchy-Schwarz master class: an introduction to the art of mathematical inequalities.  Cambridge University Press, 2004.
  * [47] Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu.  A survey on deep transfer learning.  In Artificial Neural Networks and Machine Learning–ICANN 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part III 27, pages 270–279. Springer, 2018.
  * [48] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le.  Mnasnet: Platform-aware neural architecture search for mobile.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2820–2828, 2019.
  * [49] Anh T Tran, Cuong V Nguyen, and Tal Hassner.  Transferability and hardness of supervised classification tasks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1395–1405, 2019.
  * [50] Rakshit Trivedi, Mehrdad Farajtabar, Prasenjeet Biswal, and Hongyuan Zha.  Dyrep: Learning representations over dynamic graphs.  In International conference on learning representations, 2019.
  * [51] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.  Matching networks for one shot learning.  Advances in neural information processing systems, 29, 2016.
  * [52] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * [53] Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang.  A survey of transfer learning.  Journal of Big data, 3(1):1–40, 2016.
  * [54] Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun.  Cpt: Colorful prompt tuning for pre-trained vision-language models.  arXiv preprint arXiv:2109.11797, 2021.
  * [55] Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long.  Logme: Practical assessment of pre-trained models for transfer learning.  In International Conference on Machine Learning, pages 12133–12143. PMLR, 2021.
  * [56] Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik.  Side-tuning: a baseline for network adaptation via additive side networks.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16, pages 698–714. Springer, 2020.
  * [57] Nanxuan Zhao, Zhirong Wu, Rynson WH Lau, and Stephen Lin.  What makes instance discrimination good for transfer learning?  arXiv preprint arXiv:2006.06606, 2020.
  * [58] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He.  A comprehensive survey on transfer learning.  Proceedings of the IEEE, 109(1):43–76, 2020.
  * [59] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le.  Learning transferable architectures for scalable image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697–8710, 2018.