# Robust and Efficient Transfer Learning via Supernet Transfer in Warm-started
Neural Architecture Search

Prabhant Singh1, Joaquin Vanschoren1

###### Abstract

Hand-designing Neural Networks is a tedious process that requires significant
expertise. Neural Architecture Search (NAS) frameworks offer a very useful and
popular solution that helps to democratize AI. However, these NAS frameworks
are often computationally expensive to run, which limits their applicability
and accessibility. In this paper, we propose a novel transfer learning
approach, capable of effectively transferring pretrained supernets based on
Optimal Transport or multi-dataset pretaining. This method can be generally
applied to NAS methods based on Differentiable Architecture Search (DARTS).
Through extensive experiments across dozens of image classification tasks, we
demonstrate that transferring pretrained supernets in this way can not only
drastically speed up the supernet training which then finds optimal models (3
to 5 times faster on average), but even yield that outperform those found when
running DARTS methods from scratch. We also observe positive transfer to
almost all target datasets, making it very robust. Besides drastically
improving the applicability of NAS methods, this also opens up new
applications for continual learning and related fields.

## Introduction

Neural architecture search (NAS) (Elsken, Metzen, and Hutter 2019; Zoph and Le
2017) has repeatedly demonstrated the ability to automatically design state-
of-the-art neural architectures, and numerous NAS techniques have been
proposed in research. NAS is extremely useful in reducing the time and effort
that goes into designing neural networks from scratch. Using NAS allows one to
find optimal networks without the manual tuning of architecture and training
parameters. However, this search for architectures can be computationally very
expensive. To deal with this there have been a lot of improvements like smart
search space design, Zero-cost proxies (Abdelfattah et al. 2021), and faster
optimization algorithms (Luo et al. 2018). One of the most researched and
efficient frameworks in NAS is Differentiable architecture search (DARTS (Liu,
Simonyan, and Yang 2019) with 4400+ citations) which trains an over-
parameterized architecture, which we will refer to as the ”supernet”, of which
the architectures being searched for are subgraphs. Using bilevel optimization
and parameter sharing, this method can optimize the model parameters and the
architecture parameters simultaneously, which is drastically more efficient.
Various further improvements have been made to make this approach more robust
and efficient, such as Smooth DARTS (SDARTS (Chen and Hsieh 2021)). Still,
when given a new dataset, DARTS usually has to be trained from scratch, which
is still quite computationally expensive.

Figure 1: Transfer Learning of SDARTS trained supernets for a target task
according to appropriate source task. The source datasets
($\mathcal{D}_{meta}$) shown at the top are the datasets for which we have
available pre-trained supernets($\theta$), the colored graphs. The target
dataset is the dataset we want to model. The heatmap is the Optimal Transport
distance matrix. We transfer the supernet corresponding to the most similar
dataset to the target task, and then continue to train the pretrained supernet
(fine-tuning) for a number of iterations before discretizing it into the final
model.

In this work, we explore how we can effectively transfer DARTS-like supernets
that were pretrained on older tasks, and fine-tune them to new (target) tasks.
As illustrated in Figure 1, we keep a memory (or zoo) of pretrained supernets,
each pretrained on different image classification source tasks. The supernets
are shown here as graphs where each color symbolizes a certain layer (e.g. a
3x3 convolutional layer or a maxpooling layer), and the thickness of the edges
relate to the weight of each layer during supernet training. In DARTS, after
training, only the edges with the highest weights will be kept in the final
model. However, we will store the pretrained models to be able to fine-tune
them to a different task later. Building on SDARTS, we apply Transfer learning
techniques (Renggli et al. 2020) to transfer supernets to new tasks. In
particular, we propose to employ Optimal Transport measures to select source
tasks that are very similar to our target tasks (and hence may transfer
better), as well as multi-dataset pretaining techniques to pretrain a supernet
on a wide range of source tasks at once.

Supernet training is by far the most expensive part of DARTS-like frameworks.
Hence, being able to effectively transfer supernets (from which the neural
networks are sampled) we can significantly speed up NAS. Note that this is
very different from transferring only the final optimal model parameters, as
is done in traditional transfer learning. By transferring the supernet, we
effectively transfer the state of the architecture search process, before the
supernet is collapsed into a discrete neural architecture, and then continue
the architecture search towards a new target task.

When one has multiple tasks and pre-trained networks available to transfer
from then the selection of the best source dataset or model becomes a
difficult problem, since this selection can result in positive transfer as
well as negative transfer (in which the fine-tuned model performs worse than
the one trained from scratch). To achieve robust task selection for transfer
learning in pre-trained supernets we use optimal transport-based metrics in
this paper. Our study shows the existence of negative and positive transfer in
DARTS, and that we can achieve robust positive transfer using Optimal
Transport-based dataset selection for supernet finetuning.

Through this work, we aim to contribute to the understudied area of transfer
learning in Neural Architecture Search. In summary, we make the following
contributions:

  1. 1.

We present an effective transfer learning approach for DARTS-like frameworks
with optimal transport-based distances.

  2. 2.

We present a comprehensive study to evaluate our method and show the
effectiveness of supernet transfer learning in DARTS, as well as the
effectiveness of optimal transfer to select relevant prior dataset with
empirical evidence.

  3. 3.

We introduce baselines and run extensive experiments to show the robustness of
our approach.

## Related Work

### NASTransfer

NAS Transferability has been a relatively underexplored area. Panda et al.
(2021) (AAAI 21)presented the first analysis in this area with several NAS
competitors like DARTS, ENAS, and NAO. This study analyzed the transferability
of these different methods on finally obtained architecture to each other but
never tried to transfer the supernet itself.

### METANAS

METANAS (Elsken et al. 2019) proposed a gradient-based meta-learning with
neural architecture search. METANAS allows meta-learning of a neural
architecture along the weights and adapting it to task-dependent architectures
based on a few labeled data points with gradient descent. METANAS combines
DARTS (Liu, Simonyan, and Yang 2019) with REPTILE (Nichol, Achiam, and
Schulman 2018) to make NAS possible for few-shot learning scenarios. The
METANAS approach is very expensive to compute but can be useful when one is
learning in a few-shot learning scenario.

### MetaD2A and TransferNAS

MetaD2A (Lee, Hyung, and Hwang 2021) and TNAS (Shala et al. 2023) are somewhat
related to our work but not directly comparable. In MetaD2A the authors
propose to generate candidate architectures and rank them conditioned directly
on a task, utilizing a meta-feature extractor. TNAS uses Bayesian Optimization
(BO) with deep-kernel Gaussian Processes, graph neural networks for obtaining
architecture embeddings and a transformer-based dataset encoder. Both methods
utilize transfer learning in similar ways and utilize knowledge from previous
tasks. In this work we do not aim to be state of the art and compare with
these methods, we are aiming to use a well-known transfer learning technique
and observe the effect on a DARTS-like framework which is very widely adopted.
Therefore we do not see it fit to compare our approach with these two methods
as we do not propose a new search method but a new transfer learning-based
approach for DARTS supernets.

## Background

### Differentiable architecture search (DARTS)

DARTS is one of the most widely researched NAS frameworks. DARTS searches for
the architecture of the cells (Micro search) that are stacked together to
compose a full neural network. In a cell, $N$ nodes are arranged as a Directed
Acyclic Graph (DAG), as shown in Figure 2. Each node $x^{(i)}$ represents a
latent feature, and each edge $(i,j)$ is linked with a specific operation
$o(i,j)$. Conducting an efficient search within this structure is challenging
due to the discrete nature of operation choices on each edge. To address this,
DARTS introduces a composite operation $\bar{o}^{(i,j)}$ on each edge:

$\bar{o}^{(i,j)}(x)=\sum_{o\in\mathcal{O}}\frac{\exp(\alpha_{o}^{(i,j)})}{\sum_{o^{\prime}\in\mathcal{O}}\exp(\alpha_{o^{\prime}}^{(i,j)})}o(x),$
(1)

In Equation 1, $\mathcal{O}$ is the candidate operation corpus and
$\alpha_{o}^{(i,j)}$ is the corresponding architecture weight for operation
$o$ on the edge $(i,j)$. The architecture search is relaxed to learning a
continuous architecture weight $A=[\alpha^{(i,j)}]$. This relaxation results
in a bilevel optimization objective:

$\min_{A}L_{\text{val}}(w^{*}(A),A),\quad\text{s.t.}\quad
w^{*}=\underset{w}{\mathrm{arg\,min}}\ L_{\text{train}}(w,A).$ (2)

where $w$ is the network weight and $A$ is the current architecture weight.

### SmoothDARTS

SmoothDARTS, an advancement in Differentiable Architecture Search (DARTS),
addresses its instability issues by employing perturbation-based
regularization. This approach, comprising random smoothing and adversarial
training, smooths the validation loss landscape, enhancing model stability and
generalizability. This regularization implicitly regulates the Hessian norm of
the validation loss, leading to more robust architectures. In this experiment
we choose to use SmoothDARTS as it fits our time and stability requirements it
does not require too much time to complete a search and does not use any zero
coast hacks to speed up the search, this makes it an ideal baseline to
evaluate transferability. SmoothDARTS also takes stabilization methods from
RobustDARTS by Zela et al. (2020) which addresses the problems of performance
drop while sampling and generalization.

### Defining Transferability

For transferability, we have a source dataset $D_{s}$ and a target dataset
$D_{t}$ respectively. In this work, we only consider the scenario where source
and target datasets have the same dimensionality but reside in different
domains. For Neural networks, we have a trained source model $\theta$ with
weights $w_{s}$. In our work we focus on transfer learning usually referred to
as fine-tuning where the entire pre-trained model is trained for a new task.
For a neural network, we define transferability as:

$\text{Trf}(S\rightarrow T)=\mathbb{E}[\text{acc}(y,x;\theta_{S},\theta_{T})]$
(3)

### Preliminary of optimal transport

Optimal transport (OT) or transportation theory, also known as
Kantorovich–Rubinstein duality, is a problem that deals with the
transportation of masses from source to target. This problem is also called
the Monge–Kantorovich transportation problem (Villani 2008). In recent years,
OT has gained significant attention from the machine learning community, as it
provides a powerful framework for designing algorithms that can learn to match
two probability distributions, which is a common task in image and natural
language processing. In this section, we give an introduction to OT and
distance measures related to our work.

In OT, the objective is to minimize the cost of transportation between two
probability distributions. For a cost function between pairs of points, we
calculate the cost matrix $C$ with dimensionality $n\times m$. The OT problem
minimizes the loss function $L_{c}(P):=\langle C,P\rangle$ concerning a
coupling matrix $P$. A practical and computationally more efficient approach
is based on regularization and minimizes $L_{c}^{\epsilon}(P):=\langle
C,P\rangle+\epsilon\cdot r(P)$ where $r$ is the negative entropy, computed by
the Sinkhorn algorithm (Cuturi 2013), and $\epsilon$ is a hyperparameter
controlling the amount of regularization. A discrete OT problem can be defined
with two finite point clouds, $\\{x^{(i)}\\}^{n}_{i=1}$
,$\\{y^{(j)}\\}^{m}_{j=1},x^{(i)},y^{(j)}\in\mathbb{R}^{d}$, which can be
described as two empirical distributions:
$\mu:=\sum^{n}_{i=1}a_{i}\delta_{x^{(i)}},\nu:=\sum^{m}_{j=1}b_{j}\delta_{y^{(j)}}$.
Here, $a$ and $b$ are probability vectors of size $n$ and $m$, respectively,
and the $\delta$ is the Dirac delta.

OT can be efficiently solved by the Sinkhorn algorithm (Cuturi 2013) and
various low-rank approximations (Scetbon, Peyr’e, and Cuturi 2021) as well as
ICNN (Amos et al. 2022). Entropic regularized OT has been used in Domain
adaptation (Courty et al. 2014; Damodaran et al. 2018). Optimal transport-
based algorithms and transfer learning have been an upcoming area of research
with recent works like OTDD (Alvarez-Melis and Fusi 2020) and OTCE (Tan, Li,
and Huang 2021) which focus solely on finding transfer learning metrics
between image datasets.

## Transfer Learning in SDARTS Supernet

In this work, we want to find similarity between two datasets, this similarity
will act as a transferability metric for us. We hypothesize that when two
datasets share a high degree of similarity, the performance of a neural
network pre-trained on one dataset can be effectively transferred to the
other. However, quantifying similarity is a challenging task, especially when
dealing with complex data distributions (like image datasets) rather than
individual data points.

Typically used to calculate the cost involved in moving one probability
distribution to align with another, optimal transport can also be repurposed
to measure the distance between these distributions. We can assume that the
smaller the distance means greater similarity between datasets.

First, we build a collection of labeled datasets $\mathcal{D}_{meta}$ which
will serve as a collection of source datasets to transfer from and we also
build a collection of pre-trained supernets on these datasets $\Theta_{meta}$
which contains supernets $\\{\theta_{1}...\theta_{n}\\}$. We want to find a
dataset similar to the target dataset $D_{Taraget}$. For our transfer learning
setup, we make sure that every time we look for a pre-trained network to
transfer from in that scenario $D_{Target}\notin\mathcal{D}_{meta}$. Now we
want to find the optimal transport distance between these datasets but first,
we need to preprocess these datasets with an embedding function $\phi$. $\phi$
can be any neural network we have available that can be used to make
embeddings of images or data points. We can then use any optimal transport
distance metric available which can return us the distance between these
datasets $d_{ot}$. We build a collection to find distances between each
dataset in $\mathcal{D}_{meta}$ to the target dataset in an array. We select
the dataset with minimal distance as the source dataset. This will be the
dataset we transfer the supernet from. We transfer the supernet with all
corresponding edge weights$(w,\alpha)$ from the corresponding dataset and use
it for our target task and use these weights of supernet to warm start our
supernet in SDARTS. We refer to Algorithm 1 for a generic template of our
proposed framework for transfer learning in Neural Architecture Search.

Algorithm 1 Algorithm for Finding Most Similar Dataset and Performing Transfer

Inputs:

The target dataset we are performing SDARTS on: $D_{\text{target}}$

Meta-data dataset collection: $\mathcal{D}_{\text{meta}}$

pre-trained SDARTS supernets: $\Theta_{meta}=\Sigma(w_{meta},\alpha_{meta})$

Embedding function for our datasets: $\phi$

1:Initialize $\Sigma d_{\text{ot}}$ to $[]$

2:for $D_{i}$ in $\mathcal{D}_{\text{meta}}$ do

3: $d_{\text{ot}_{i}}\leftarrow\text{OT}(\phi(D_{\text{target}}),\phi(D_{i}))$
$\triangleright$ Distance calculation

4: Append $d_{\text{ot}_{i}}$ to $\Sigma d_{\text{ot}}$

5:end for

6:$s\leftarrow\text{argmin}(\Sigma d_{\text{ot}})$ $\triangleright$ Retrieval
of most similar dataset

7:$w_{s},\alpha_{s}\leftarrow\Sigma(w_{meta},\alpha_{meta})[s]$
$\triangleright$ Retrieve $w_{s},\alpha_{s}$ from
$\Sigma(w_{meta},\alpha_{meta})$

8:$w_{t},\alpha_{t}\leftarrow w_{s},\alpha_{s}$ $\triangleright$ Transferring
weights of supernet from source to target task

9:Finding $A$ for $D_{\text{target}}$

10:Generate mixed operation $\bar{o}^{(i,j)}$ for every edge $(i,j)$

11:while not converged do

12: Update architecture $A$ based on equation 2

13:end while

14:$A$

The transfer we do here follows these steps:

  1. 1.

Construct a comprehensive set of labeled datasets, denoted as
$\mathcal{D}_{\text{meta}}=\\{D_{1},D_{2},\ldots,D_{n}\\}$, which serves as a
repository of source datasets for transfer.

  2. 2.

Assemble a corresponding set of pre-trained supernets,
$\Theta_{\text{meta}}=\\{\theta_{1},\theta_{2},\ldots,\theta_{n}\\}$, where
each $\theta_{i}$ is a supernet trained on dataset
$D_{i}\in\mathcal{D}_{\text{meta}}$.

  3. 3.

Implement an embedding function, $\phi:D\rightarrow\mathbb{R}^{k}$, to
preprocess datasets and generate embeddings, facilitating the computation of
distances in a transformed feature space.

  4. 4.

Employ an optimal transport distance metric to compute the dissimilarity,
$d_{\text{ot}}(D_{i},D_{\text{Target}})$, between the embedded representations
of $D_{\text{Target}}$ and each $D_{i}\in\mathcal{D}_{\text{meta}}$.

  5. 5.

Aggregate these distances into a collection,
$\mathcal{D}_{\text{ot}}=\\{d_{\text{ot}}(D_{1},D_{\text{Target}}),\ldots,d_{\text{ot}}(D_{n},D_{\text{Target}})\\}$.

  6. 6.

Identify the dataset $D_{s}\in\mathcal{D}_{\text{meta}}$ with the minimum
distance to $D_{\text{Target}}$, determined by
$s=\arg\min_{i}d_{\text{ot}}(D_{i},D_{\text{Target}})$.

  7. 7.

Execute the transfer of the corresponding supernet,
$\theta_{s}\in\Theta_{\text{meta}}$, and its edge weights
$(w_{s},\alpha_{s})$, to initialize the architecture search for
$D_{\text{Target}}$.

  8. 8.

Utilize these initial weights
$(w_{\text{target}},\alpha_{\text{target}})=(w_{s},\alpha_{s})$ to facilitate
a warm start in the SDARTS supernet procedure for the target task.

## Experimental analysis

In our experimental setup we aim to answer these 4 questions:

  1. 1.

Is there any advantage of transfer learning in DARTS-like supernets? Do both
negative transfers (decrease in performance after transfer) and positive
transfers (increase in performance after transfer) occur when transferring
DARTS Supernets? If so, this underscores the importance of appropriate source
task selection.

  2. 2.

Can optimal transport-based dataset distance be used effectively to find an
optimal source dataset for transfer learning?

  3. 3.

Is the size of the proxy dataset relevant to the performance of transfer
learning?

  4. 4.

What is the impact of transfer learning on the convergence of the DARTS
Supernet $\theta$?

We elaborate on our experimental setup in detail. This section describes the
exact objective we are optimizing for in our experiments, what search space we
are using in DARTS to sample architectures from, which datasets we are using
in $\mathcal{D}_{meta}$, and which optimal transport method and preprocessing
method we are using.

Objective:  Our goal is to perform robust and effective transfer on
SmoothDARTS (SDARTS) Supernets, and we analyze the resulting performances when
using supernet transfer on SmoothDARTS. We aim to learn from these findings
how to design and train NAS algorithms more efficiently and robustly.

Datasets:  We use datasets from the Meta-Album dataset in Mini size (40
examples per class). The details of selected datasets are in Appendix A. Meta-
Album (Ullah et al. 2022) is an image classification meta-dataset designed to
facilitate few-shot learning, transfer learning, and meta-learning, among
other tasks. It includes 40 open datasets, each having at least 20 classes
with 40 examples per class, with verified licenses. They stem from diverse
domains, such as ecology fauna and flora), manufacturing (textures, vehicles),
human actions, and optical character recognition, featuring various image
scales (microscopic, human scales, remote sensing). All datasets are
preprocessed, annotated, and formatted uniformly, and come in 3 versions
(Micro $\rightarrow$ Mini $\rightarrow$ Extended) the detailed list of all
meta album datasets used is present in Appendix B

Search Space:  We use a modified search space on the CNN search space of
SDARTS to incorporate the Meta-album dataset. We use S1 search space from
SDARTS (Chen and Hsieh 2021)

Metrics:  Following Panda et al. (2021) we compute top-1 classification
accuracy of runs on baselines as a metric of performance for our experiments.
We also compute relative improvement for all baselines. Relative improvement
can be described as $RI=\frac{ACC_{a}-ACC_{b}}{ACC_{b}}$ where $ACC_{a}$ is
the accuracy of the method, and $ACC_{b}$ is the accuracy of the baseline.

We use Resnet18 (He et al. 2016) as an embedding function$\phi$, We use
entropic regularisation of 1e-1 with 1000 samples to calculate the OT-based
distance $d_{ot}$. We use OTDD (Alvarez-Melis and Fusi 2020) for our
experiments as it incorporates label cost as well. Every SDARTS run took 3-4
hours on average on an NVIDIA A8000 GPU when trained for 50 epochs. We did not
do early stopping to have more consistent results.

Why did we choose SmoothDARTS?

We use Stabilizing Differentiable Architecture Search via Perturbation-based
Regularization (Zela et al. 2020) in our experiments because it is one of the
stable versions of DARTS as compared to the first implementation (Liu,
Simonyan, and Yang 2019) and it does not use any zero-cost proxies or hacks,
which allows us to assess the effect of transfer learning. SmoothDARTS also
builds on multiple other DARTS-like frameworks that are widely accepted by the
community. DARTS performance also drops steeply when discretization happens
that is why we use SDARTS which already proposes solutions for this problem.
Therefore SDARTS supernet training will result in less performance drop after
discretization.

### Is there any advantage of transfer learning in DARTS? Do negative and
positive transfers exist in DARTS supernets?

To evaluate this question we transfer all supernets $\Theta$ from
$\mathcal{D}_{meta}$ for every target dataset sampled from
$\mathcal{D}_{meta}$ Such that we have a supernet dictionary
$\Theta_{all}=\\{\Theta_{1}...\Theta_{n}\\}$. This is a simple grid search for
finding the optimal transfer dataset. We call the datasets with the highest
transfer accuracy oracle. The output of these experiments is shown in Figure
2. This shows that positive and negative transfers do exist with DARTS
Supernets and do affect the training and validation accuracy of supernets.

Figure 2: Experimental results from experiment 1, We report accuracies
obtained by transferring all datasets in $\mathcal{D}_{meta}$ to each other
and report the performance of the lowest accuracy(red) and highest accuracy
(green) obtained (oracle). We also report the accuracy when training for these
datasets without any transfer (orange) Figure 3: Heatmap of dataset
similarities calculated via $d_{ot}$

### Are optimal transport-based dataset distances effective in finding an
optimal source dataset for transfer learning?

To answer this question we use Algorithm 1 to find the most similar dataset to
the target dataset and then transfer the supernet to the target dataset and
fine-tune it. Our OT-based framework generates the heatmap in Figure 3 We
compare the performance of this experiment in Figure 4 where we compare the
performance of our method with training from scratch, the highest performance
from grid searching $\Theta_{oracle}$ and lowest performance from supernet
transfer. We report the relative improvement of our method and relative
improvement of Oracle over baseline such that
$RI_{OT}=\frac{ACC_{OT}-ACC_{scratch}}{ACC_{scratch}}$ and
$RI_{Oracle}=\frac{ACC_{Oracle}-ACC_{scratch}}{ACC_{scratch}}$ in Table 1.

Figure 4: Performance of transferring from Optimal transport based similarity
$d_{ot}$ vs training DARTS from scratch accuracy.

We also observe that the difference between the accuracies of OT-based dataset
selection and the highest single transfer dataset is greater than 5 percent
for only 7 out of 26 instances.

Dataset | Topic | RI_OT | RI_Oracle
---|---|---|---
PLK | Plankton | 0.0059 | 0.0558
FLW | Flowers | 0.0046 | 0.1233
SPT | 73 Sports | 0.0214 | 0.1299
BRD | Birds | 0.0050 | 0.0210
PLT_VIL | Plant Diseases | 0.0628 | 0.0628
TEX | Textures | 0.0143 | 0.0185
CRS | Cars | 0.2653 | 0.2653
RESISC | Remote Sensing | -0.0090 | 0.0966
ACT_40 | Human Actions | 0.2893 | 0.2893
INS_2 | Insects | 0.1806 | 0.1806
PLT_NET | Plants | 0.3264 | 0.3264
TEX_DTD | Textures (DTD) | 0.0854 | 0.3920
APL | Airplanes | -0.0219 | 0.1360
PNU | Human tissues | 0.0598 | 0.4017
DOG | Dogs | 0.3564 | 0.3564
MED_LF | Medicinal Leaf | -0.0022 | 0.0414
RSICB | RSICB | 0.0185 | 0.0357
ACT_410 | Human Poses | 0.0643 | 0.0936
FNG | Fungi | 0.2000 | 0.4125
PLT_DOC | Plants | 0.3826 | 0.3826
TEX_ALOT | Textures (ALOT) | 0.0002 | 0.0245
AWA | Animals | 0.3458 | 0.3458
INS | Insects | -0.1028 | 0.3575
RSD | Remote Sensing | 0.0679 | 0.1821
PRT | Human Proteins | -0.0159 | 0.0635
BTS | Boats | 0.0649 | 0.5195
Table 1: Relative Improvement (over training DARTS from scratch) of our
Optimal Transport-based method (RI_OT) as well as the Oracle (magically
knowing the best possible single dataset to transfer from). We observe that
supernet transfer using Optimal Transport is effective, often outperforming
training DARTS from scratch, and often (but not always) close to the Oracle
selection.

(a) ACT_40

(b) BRD

(c) APL

(d) FLW

(e) PLT_VIL

(f) INS_2

Figure 5: Training curves of 6 different meta-album datasets. Leave-one-
out(Red) converges fastest, OT-Transfer(Blue) converges much faster than
training SDARTS from scratch(Green), here y-axis is the training accuracy of
$\theta$ and the axis is the number of steps(1-100).

### Is the size of the proxy dataset relevant to the performance on transfer
learning?

To answer this question we train a new supernet $\theta_{meta}$ which is a
single supernet trained on all datasets in $\mathcal{D}_{meta}$ such that
$D_{Target}\notin$ $\mathcal{D}_{meta}$. We then transfer $\theta_{meta}$ to
$\theta_{Target}$ and report the performance. The results are shown in Table 2
where we compare this method with the optimal transport-based method and
Oracle Table 2. In the table, we can see that leave-one-out(LOO) accuracy
beats the OT-Transfer and Oracle almost every time. ‘ This evidence implies
that more data is always better for SDARTS supernet training.

Dataset | RI_OT | RI_LOO | RI_Oracle
---|---|---|---
PLK | 0.0059 | 0.0701 | 0.0558
FLW | 0.0046 | 0.1880 | 0.1233
SPT | 0.0214 | 0.2253 | 0.1299
BRD | 0.0050 | 0.0875 | 0.0210
PLT_VIL | 0.0628 | 0.1154 | 0.0628
TEX | 0.0143 | 0.0285 | 0.0185
CRS | 0.2653 | 0.2207 | 0.2653
RESISC | -0.0090 | 0.1483 | 0.0966
ACT_40 | 0.2893 | 0.4959 | 0.2893
INS_2 | 0.1806 | 0.3889 | 0.1806
PLT_NET | 0.3264 | 0.4653 | 0.3264
TEX_DTD | 0.0854 | 0.6080 | 0.3920
APL | -0.0219 | 0.1623 | 0.1360
PNU | 0.0598 | 0.3932 | 0.4017
DOG | 0.3564 | 0.5927 | 0.3564
MED_LF | -0.0022 | 0.0588 | 0.0414
RSICB | 0.0185 | 0.0542 | 0.0357
ACT_410 | 0.0643 | 0.0965 | 0.0936
FNG | 0.2000 | 0.8250 | 0.4125
PLT_DOC | 0.3826 | 0.4087 | 0.3826
TEX_ALOT | 0.0002 | 0.0370 | 0.0245
AWA | 0.3458 | 0.4292 | 0.3458
INS | -0.1028 | 0.3364 | 0.3575
RSD | 0.0679 | 0.3086 | 0.1821
PRT | -0.0159 | -0.0317 | 0.0635
BTS | 0.0649 | 0.6494 | 0.5195
Table 2: Relative improvement of OT-transfer vs Leave one out vs oracle, we
can see that Leave one out almost always works better than OT transfer and
even oracle which establishes that more data does improve the performance.

### What is the impact of transfer learning on the convergence of DARTS
supernet?

To answer this question we analyze the training curves of supernets. For ease
of display, we show results from 6 datasets and compare the training
accuracy(convergence) of SmoothDARTS with training from scratch, OT-transfer,
and Leave-one-out in Figure 5. We see the same observation from all meta-album
datasets. We observe that the Leave-one-out supernet converges first, then the
OT-Transfer supernet and training from scratch takes the longest. The speedup
in convergence is 3x-5x.

## Conclusion and Discussion

In this work, we demonstrated that transfer learning of supernets + Fine
tuning is a go-to strategy for better-performing and efficient DARTS like NAS
methods, especially better than training from scratch only. Considering
supernet training is the most expensive step in DARTS-like frameworks we
believe that this work provides essential guidelines for future work on
transfer learning in NAS and how can future DARTS-like methods utilize
previously trained knowledge. We like to emphasize that negative and positive
transfers do exist in SmoothDARTS and there should be a transferability metric
that can point to the right dataset. We also show with new data that large
datasets do outperform despite of smart selection of datasets for transfer
learning. As an additional result, we also observed that training accuracy and
convergence were impacted heavily by transfer learning.

We also find that our results differ from earlier studies Panda et al. (2021).
We provide several reasons for this difference between the results of these
two studies. First, Panda et al. (2021) focuses on ENAS (Pham et al. 2018) and
DARTS(original implementation) (Liu, Simonyan, and Yang 2019) as their
baselines. These frameworks are proven to be very unstable. Singh et al.
(2019) have shown that ENAS performance results from search space design, not
search algorithms and the controller does not learn anything. Several studies
have also pointed out the instability of the first DARTS implementation as
described by Zela et al. (2020). Second, SmoothDARTS (Wang et al. 2021) shows
different results as it uses multiple works that provide stability for DARTS
training and hence provide different and more reliable results. One more
reason for the disparity is Meta-album instead of IMAGENET22K, Meta album is a
much more complex dataset that is designed to asses transfer learning and
meta-learning in neural networks. Hence this also makes a difference in
observations.

In our work, we make the following conclusions which can be seen as learnings
and best practices when one is performing transfer learning on DARTS:

  1. 1.

Using a transferability metric like optimal transport provides more robust and
efficient transfer learning for DARTS-like frameworks.

  2. 2.

Having more datasets to transfer from is always better than a single dataset
transfer.

  3. 3.

Transfer learning on $\theta$ speeds up the convergence of DARTS $\theta$ by
3x-5x.

## Limitations

Our method can be hard to scale if the dataset pool is very large as the
dataset computation increases linearly. We also think that the size of the
sampled dataset can adversely affect the performance of data and our method is
not safe if there are adversarial examples present in the dataset. This method
is also only applicable when there is a supernet-like structure available in
the search process to transfer from, if the NAS method does not include that
then learning from previous data is much harder to achieve.

## Future Work

The method presented can be used with any OT-distance and DARTS-like
framework, we see two future research directions that can help to speed up
this framework and provide new insights. In particular, we aim to employ
faster OT approximations instead of OTDD for finding similar datasets like
Wasserstein task embeddings (Courty, Flamary, and Ducoffe 2018). Moreover, we
will apply the same methodology to other upcoming NAS methods like OFA (Cai,
Gan, and Han 2019) and AutoFormer (Wu et al. 2021).

## References

  * Abdelfattah et al. (2021) Abdelfattah, M. S.; Mehrotra, A.; Dudziak, Ł.; and Lane, N. D. 2021.  Zero-Cost Proxies for Lightweight {NAS}.  In _International Conference on Learning Representations_.
  * Alvarez-Melis and Fusi (2020) Alvarez-Melis, D.; and Fusi, N. 2020.  Geometric Dataset Distances via Optimal Transport.  In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds., _Advances in Neural Information Processing Systems_ , volume 33, 21428–21439. Curran Associates, Inc.
  * Amos et al. (2022) Amos, B.; Cohen, S.; Luise, G.; and Redko, I. 2022.  Meta Optimal Transport.  In _International Conference on Machine Learning_.
  * Cai, Gan, and Han (2019) Cai, H.; Gan, C.; and Han, S. 2019.  Once for All: Train One Network and Specialize it for Efficient Deployment.  _ArXiv_ , abs/1908.09791.
  * Chen and Hsieh (2021) Chen, X.; and Hsieh, C.-J. 2021.  Stabilizing Differentiable Architecture Search via Perturbation-based Regularization.  arXiv:2002.05283.
  * Courty, Flamary, and Ducoffe (2018) Courty, N.; Flamary, R.; and Ducoffe, M. 2018.  Learning Wasserstein Embeddings.  In _International Conference on Learning Representations_.
  * Courty et al. (2014) Courty, N.; Flamary, R.; Tuia, D.; and Rakotomamonjy, A. 2014.  Optimal Transport for Domain Adaptation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 39: 1853–1865.
  * Cuturi (2013) Cuturi, M. 2013.  Sinkhorn Distances: Lightspeed Computation of Optimal Transport.  In _Neural Information Processing Systems_.
  * Damodaran et al. (2018) Damodaran, B. B.; Kellenberger, B.; Flamary, R.; Tuia, D.; and Courty, N. 2018.  DeepJDOT: Deep Joint distribution optimal transport for unsupervised domain adaptation.  _ArXiv_ , abs/1803.10081.
  * Elsken, Metzen, and Hutter (2019) Elsken, T.; Metzen, J. H.; and Hutter, F. 2019.  Neural architecture search: A survey.  _The Journal of Machine Learning Research_ , 20(1): 1997–2017.
  * Elsken et al. (2019) Elsken, T.; Staffler, B. S.; Metzen, J. H.; and Hutter, F. 2019.  Meta-Learning of Neural Architectures for Few-Shot Learning.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 12362–12372.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Lee, Hyung, and Hwang (2021) Lee, H.; Hyung, E.; and Hwang, S. J. 2021.  Rapid neural architecture search by learning to generate graphs from datasets.  _arXiv preprint arXiv:2107.00860_.
  * Liu, Simonyan, and Yang (2019) Liu, H.; Simonyan, K.; and Yang, Y. 2019.  DARTS: Differentiable Architecture Search.  In _International Conference on Learning Representations_.
  * Luo et al. (2018) Luo, R.; Tian, F.; Qin, T.; Chen, E.; and Liu, T.-Y. 2018.  Neural Architecture Optimization.  In Bengio, S.; Wallach, H.; Larochelle, H.; Grauman, K.; Cesa-Bianchi, N.; and Garnett, R., eds., _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc.
  * Nichol, Achiam, and Schulman (2018) Nichol, A.; Achiam, J.; and Schulman, J. 2018.  On First-Order Meta-Learning Algorithms.  _ArXiv_ , abs/1803.02999.
  * Panda et al. (2021) Panda, R.; Merler, M.; Jaiswal, M. S.; Wu, H.; Ramakrishnan, K.; Finkler, U.; Chen, C.-F. R.; Cho, M.; Feris, R.; Kung, D.; and Bhattacharjee, B. 2021.  NASTransfer: Analyzing Architecture Transferability in Large Scale Neural Architecture Search.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(10): 9294–9302.
  * Pham et al. (2018) Pham, H.; Guan, M. Y.; Zoph, B.; Le, Q. V.; and Dean, J. 2018.  Efficient Neural Architecture Search via Parameter Sharing.  _ArXiv_ , abs/1802.03268.
  * Renggli et al. (2020) Renggli, C.; Pinto, A. S.; Rimanic, L.; Puigcerver, J.; Riquelme, C.; Zhang, C.; and Lucic, M. 2020.  Which Model to Transfer? Finding the Needle in the Growing Haystack.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 9195–9204.
  * Scetbon, Peyr’e, and Cuturi (2021) Scetbon, M.; Peyr’e, G.; and Cuturi, M. 2021.  Linear-Time Gromov Wasserstein Distances using Low Rank Couplings and Costs.  _ArXiv_ , abs/2106.01128.
  * Shala et al. (2023) Shala, G.; Elsken, T.; Hutter, F.; and Grabocka, J. 2023.  Transfer NAS with Meta-learned Bayesian Surrogates.  In _The Eleventh International Conference on Learning Representations_.
  * Singh et al. (2019) Singh, P.; Jacobs, T.; Nicolas, S.; and Schmidt, M. 2019.  A Study of the Learning Progress in Neural Architecture Search Techniques.  _ArXiv_ , abs/1906.07590.
  * Tan, Li, and Huang (2021) Tan, Y.; Li, Y.; and Huang, S.-L. 2021.  OTCE: A Transferability Metric for Cross-Domain Cross-Task Representations.  _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 15774–15783.
  * Ullah et al. (2022) Ullah, I.; Carrion, D.; Escalera, S.; Guyon, I. M.; Huisman, M.; Mohr, F.; van Rijn, J. N.; Sun, H.; Vanschoren, J.; and Vu, P. A. 2022.  Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classification.  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
  * Villani (2008) Villani, C. 2008.  Optimal Transport: Old and New.
  * Wang et al. (2021) Wang, R.; Cheng, M.; Chen, X.; Tang, X.; and Hsieh, C.-J. 2021.  Rethinking Architecture Selection in Differentiable NAS.  In _International Conference on Learning Representations_.
  * Wu et al. (2021) Wu, H.; Xu, J.; Wang, J.; and Long, M. 2021.  Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting.  In _Neural Information Processing Systems_.
  * Zela et al. (2020) Zela, A.; Elsken, T.; Saikia, T.; Marrakchi, Y.; Brox, T.; and Hutter, F. 2020.  Understanding and Robustifying Differentiable Architecture Search.  In _International Conference on Learning Representations_.
  * Zoph and Le (2017) Zoph, B.; and Le, Q. 2017.  Neural Architecture Search with Reinforcement Learning.  In _International Conference on Learning Representations_.