Revisiting Meta-Learning as Supervised Learning

Wei-Lun Chao∗ Han-Jia Ye∗ De-Chuan Zhan The Ohio State University, USA Nanjing
University, China Nanjing University, China

Mark Campbell Kilian Q. Weinberger Cornell University, USA Cornell University,
USA

###### Abstract

Recent years have witnessed an abundance of new publications and approaches on
meta-learning. This community-wide enthusiasm has sparked great insights but
has also created a plethora of seemingly different frameworks, which can be
hard to compare and evaluate. In this paper, we aim to provide a principled,
unifying framework by _revisiting and strengthening the connection between
meta-learning and traditional supervised learning._ By treating pairs of task-
specific data sets and target models as (feature, label) samples, we can
reduce many meta-learning algorithms to instances of supervised learning. This
view not only unifies meta-learning into an intuitive and practical framework
but also allows us to transfer insights from supervised learning directly to
improve meta-learning. For example, we obtain a better understanding of
generalization properties, and we can readily transfer well-understood
techniques, such as model ensemble, pre-training, joint training, data
augmentation, and even nearest neighbor based methods. We provide an intuitive
analogy of these methods in the context of meta-learning and show that they
give rise to significant improvements in model performance on few-shot
learning.

##  1 Introduction

Meta-learning, or learning to learn, is the sub-field of machine learning
occupied with the search for the best learning strategy as the number of tasks
and learning experiences increases (Vilalta and Drissi, 2002) and has drawn
significant attention recently (Finn et al., 2017a; Andrychowicz et al., 2016;
Vinyals et al., 2016). Meta-learning has been developed in various areas to
advanced algorithm design, including few-shot learning (Ravi and Larochelle,
2017; Snell et al., 2017; Wang and Hebert, 2016), optimization (Li and Malik,
2017; Wichrowska et al., 2017), active learning (Bachman et al., 2017),
transfer learning (Balaji et al., 2018; Ying et al., 2018), unsupervised
learning (Metz et al., 2018; Edwards and Storkey, 2017), etc. Specifically,
meta-learning has demonstrated the capability to generalize learned knowledge
to novel tasks, which greatly reduces the need for training data and time to
optimize.

Figure 1: Meta-learning as supervised learning. By treating a pair of a
training set (left) and the target model (right) as a meta labeled example,
meta-learning learns a shared meta mapping (from left to right) that
generalizes from meta-training examples to novel meta-test examples. Take one-
shot learning for instance. A training set contains one image per class, while
a target model is a classifier trained with many more images from those
classes.

Given its wide applicability and diverse approaches, there is an increasing
need for a principled, unifying meta-learning framework to facilitate future
studies and development. Finn (2018); Metz et al. (2018) have made a notable
step to provide a broad introduction and compare representative algorithms. In
this paper, we aim to push the direction forward by providing a unifying view
of meta-learning through _revisiting and strengthening the connection to
supervised learning_ (Thrun and Pratt, 1998; Baxter, 2000; Maurer, 2005; Wang,
2018; Larochelle, 2018; Finn, 2018).

Generally, meta-learning can be viewed as learning a mapping $g$ that given a
training set $D_{\text{tr}}$ returns a model $h$. The model $h$ is then
applied to data sampled from the same distribution from which $D_{\text{tr}}$
is sampled. For instance, in one-shot learning $D_{\text{tr}}$ corresponds to
a set of $C$ labeled data samples, and $h$ corresponds to a $C$-way
classifier. Learning the mapping can thus be conducted by collecting and
training with $({D_{\text{tr}}},{D_{\text{val}}})$ pairs, in which the
validation set $D_{\text{val}}$ evaluates $h=g(D_{\text{tr}})$ and provides
“supervised” signal to optimize $g$, essentially connecting meta-learning with
supervised learning111Suppose $(x,y)\in D_{\text{val}}$, where $x$ is the
feature and $y$ is the label, Finn (2018) equivalently defines
$h(x)=g(D_{\text{tr}},x)$ and uses $y$ to provide the supervised signal.. This
connection has multiple advantages: it allows transferring ideas and
experiences of supervised learning to meta-learning. For example, Baxter
(2000); Maurer (2005) extended the generalization error bounds from supervised
learning to meta-learning; Vinyals et al. (2016) designed the meta-training
procedure so that meta-test and meta-train conditions match.

In this paper, we aim to generalize, expand, and strengthen the connection
between meta-learning and supervised learning so that on one hand, it can hold
across a wider range of meta-learning approaches and applications; on the
other hand, a variety of understandings and techniques of supervised learning
can be easily extended to meta-learning.

Traditional supervised learning learns a model that given a feature (vector)
returns a label, and we train the model on a set of labeled examples in the
form of (feature, target label) pairs. Inspired by this fact and (Wang and
Hebert, 2016; Garg and Kalai, 2018), we provide a unifying view of meta-
learning by drawing a closer analogy. We _pretend_ that the target model
$h^{*}$ of a training set $D_{\text{tr}}$ can be directly obtained in
collecting meta-training data. One can view the target model $h^{*}$ as the
model that we hope to learn from $D_{\text{tr}}$, similar to the annotated
label that we hope to predict given a feature. In this notion, we can train a
meta model $g$ on a set of $({D_{\text{tr}}},h^{*})$ pairs, which we call the
meta labeled examples (or tasks222There are various definitions of the term
“task.” For example, in (Pan et al., 2010; Finn, 2018), a task contains a data
distribution where the training set $D_{\text{tr}}$ is sampled, and a task can
be sampled from a meta-distribution of tasks. Our definition is equivalent to
combining the two sampling steps and can simplify the notations in this
paper.), and evaluate it on a test task $D_{\text{tr}}^{\text{test}}$. See
Figure 1 for an illustration.

This unifying view allows us to apply meta-learning to various areas while
general understandings of supervised learning still hold. For example, in
meta-learning for domain generalization (Balaji et al., 2018) where
$h=g(D_{\text{tr}})$ is applied to a data distribution different from where
$D_{\text{tr}}$ is sampled, the theoretical analysis in (Baxter, 2000; Maurer,
2005) is no longer applicable. Nevertheless, well understood pitfalls of
supervised learning such as overfitting due to insufficient (meta-) training
examples and distribution drift can still be used to explain why the learned
meta model $g$ may not generalize to novel tasks. We discuss more in
subsection 2.3 and empirically verify this in subsection 5.3.

This unifying view also identifies the essential components to design a meta-
learning algorithm, providing a principled way to apply it. While $h^{*}$ may
not be directly provided in practice, it indicates what kind of data to
collect. For example, in one-shot learning one may expect $h^{*}$ to be a
classifier learned with ample labeled data (Wang and Hebert, 2016; Wang et
al., 2017). To this end, one can first collect ample labeled data to learn
$h^{*}$, followed by learning a meta-learning model $g$ to predict $h^{*}$
from $D_{\text{tr}}$. Alternatively, one can collect a validation set
$D_{\text{val}}$ as a proxy to estimate how $h^{*}$ will perform. See section
3 for more discussions.

Finally, with this unifying view, we broaden the scope of algorithm design for
meta-learning by extending concepts from supervised learning. We conduct
extensive experiments on few-shot learning, a representative area where meta-
learning is applied. We empirically show that well-known supervised learning
techniques such as data augmentation, bagging (Breiman, 1996), joint training
(Argyriou et al., 2007), pre-training (Yosinski et al., 2014), and non-
parametric approaches (Zhang et al., 2006; Weinberger and Saul, 2009) can be
applied at the task level to significantly facilitate meta-learning.

##  2 A Unifying View of Meta-Learning

###  2.1 Background: supervised learning

In supervised learning we collect a training set
$D_{\text{tr}}=\\{(x_{n}\in\mathcal{X},y_{n}\in\mathcal{Y})\\}_{n=1}^{N}$,
composed of $N$ i.i.d. samples from an unknown distribution $\mathcal{D}$ on
$\mathcal{X}\times\mathcal{Y}$. We call $x$ an input, $y$ an output (or
label), and $(x,y)$ a labeled example. We call
$h:\mathcal{X}\mapsto\mathcal{Y}$ a model, which outputs a label for an input.
For instance, in image classification, $x$ is an image, $y$ is a class name
(e.g., “dog”), $\mathcal{D}$ is the distribution of real images, and $h$ is an
image classifier.

Supervised learning searches for a model $h$ given $D_{\text{tr}}$, so that
$h$ will work well on $(x,y)$ sampled from $\mathcal{D}$. That is, $h$ should
have a small generalization error $L_{\mathcal{D}}(h)$ according to a loss
$l:\mathcal{Y}\times\mathcal{Y}\mapsto\mathbb{R}$

$\displaystyle\vspace{-5pt}L_{\mathcal{D}}(h)=E_{(x,y)\sim\mathcal{D}}[l(h(x),y)].$
(1)

To this end, we construct a hypothesis set $\mathcal{H}=\\{h\\}$ of candidate
models and design an algorithm $A_{\mathcal{H}}$ to search $\hat{h}$ from
$\mathcal{H}$ by learning from $D_{\text{tr}}$. We denote
$\hat{h}=A_{\mathcal{H}}(D_{\text{tr}})$. One example of $\mathcal{H}$ is a
neural network with a fixed architecture but undetermined weights.

A popular framework to design supervised learning algorithms is empirical risk
minimization (ERM)

$\displaystyle\hat{h}=A_{\mathcal{H}}(D_{\text{tr}})$
$\displaystyle=\operatorname{arg\,min}_{h\in\mathcal{H}}\cfrac{1}{N}\sum_{n=1}^{N}l(h(x_{n}),y_{n})$
$\displaystyle=\operatorname{arg\,min}_{h\in\mathcal{H}}L_{S}(h),$ (2)

where $L_{S}(h)$ is the training loss. A particular $A_{\mathcal{H}}$ is
characterized by how it performs ERM (e.g., optimization methods). In
practice, we usually design a set of $(\mathcal{H}$, $A_{\mathcal{H}})$ pairs,
denoted as $G$, and select the best pair using a held-out validation set
$D_{\text{val}}=\\{(x_{m},y_{m})\\}_{m=1}^{M}$ sampled i.i.d. from
$\mathcal{D}$

$\displaystyle(\mathcal{H},A_{\mathcal{H}})^{*}$
$\displaystyle=\operatorname{arg\,min}_{(\mathcal{H},A_{\mathcal{H}})\in
G}\cfrac{1}{M}\sum_{m=1}^{M}l(\hat{h}(x_{m}),y_{m})$
$\displaystyle=\operatorname{arg\,min}_{(\mathcal{H},A_{\mathcal{H}})\in
G}L_{V}(\hat{h}),$ (3)

where $\hat{h}=A_{\mathcal{H}}(D_{\text{tr}})$. This is called model selection
and $L_{V}$ is the validation error.

###  2.2 Meta-learning as supervised learning

We provide a framework of meta-learning by drawing analogy to supervised
learning. We use “meta (labeled) example” and “task” interchangeably. To
prevent confusion, we call the model in supervised learning a “base” model
when needed.

#### Definition.

In meta-learning, we collect a meta-training set
$D_{\text{meta}\text{-}\text{tr}}=\\{({D_{\text{tr}}}_{j}\in\mathcal{I},h^{*}_{j}\in\mathcal{O})\\}_{j=1}^{N_{\text{meta}}}$,
composed of $N_{\text{meta}}$ i.i.d. samples from an unknown meta distribution
$\mathcal{D}_{\text{meta}}$ on $\mathcal{I}\times\mathcal{O}$. We call
$D_{\text{tr}}$ a training set (meta input), $h^{*}$ a “target” base model
(meta output), and $(D_{\text{tr}},h^{*})$ a meta labeled example (task). We
call $g:\mathcal{I}\mapsto\mathcal{O}$ a meta model (meta mapping), which
outputs a base model for a training set. For instance, in one-shot $C$-way
learning for image classification,
$D_{\text{tr}}=\\{(x_{n},y_{n})\\}_{n=1}^{N}$ contains $N$ labeled images, one
for each of the $C$ classes (i.e., $N=C$). $h^{*}$ is a strong classifier
trained using ample labeled images.

Meta-learning searches for a meta model $g$ given
$D_{\text{meta}\text{-}\text{tr}}$, so that $g$ will work well on
$(D_{\text{tr}},h^{*})$ sampled from $\mathcal{D}_{\text{meta}}$. That is, $g$
should have a small meta generalization error
$L_{\mathcal{D}_{\text{meta}}}(g)$ according to a meta loss
$l_{\text{meta}}:\mathcal{O}\times\mathcal{O}\mapsto\mathbb{R}$

$\displaystyle
L_{\mathcal{D}_{\text{meta}}}(g)=E_{(D_{\text{tr}},h^{*})\sim\mathcal{D}_{\text{meta}}}[l_{\text{meta}}(g(D_{\text{tr}}),h^{*})].$
(4)

In one-shot learning, we can view $g$ as a predictor of (strong) classifiers
given small training sets.

#### Algorithm.

To this end, we can follow supervised learning to construct a meta hypothesis
set $\mathcal{G}=\\{g\\}$ of candidate meta models and design an algorithm
$B_{\mathcal{G}}$ to search $\hat{g}$ from $\mathcal{G}$ by learning from
$D_{\text{meta}\text{-}\text{tr}}$. We denote
$\hat{g}=B_{\mathcal{G}}(D_{\text{meta}\text{-}\text{tr}})$. We can apply ERM

$\displaystyle B_{\mathcal{G}}(D_{\text{meta}\text{-}\text{tr}})$
$\displaystyle=\arg\min_{g\in\mathcal{G}}\cfrac{1}{N_{\text{meta}}}\sum_{j=1}^{N_{\text{meta}}}l_{\text{meta}}(g({D_{\text{tr}}}_{j}),h^{*}_{j})$
$\displaystyle=\operatorname{arg\,min}_{g\in\mathcal{G}}L_{S_{\text{meta}}}(g),$
(5)

where $L_{S_{\text{meta}}}$ is the meta training error. In practice, we design
a set of $(\mathcal{G}$, $B_{\mathcal{G}})$ pairs and select the best pair
using a held-out meta-validation set
$D_{\text{meta}\text{-}\text{val}}=\\{({D_{\text{tr}}}_{m},h^{*}_{m})\\}_{m=1}^{M_{\text{meta}}}$
sampled i.i.d. from $\mathcal{D}_{\text{meta}}$. This is called meta model
selection (meta-validation).

#### Discussion.

The meta model $g$ and the supervised learning algorithm $A_{\mathcal{H}}$
(cf. Equation 2) have the same forms of inputs and outputs. Thus, $g$ can be
seen as a (supervised) learning algorithm which may involve an optimization
process333Indeed, Metz et al. (2018) point out that many meta-learning
algorithms consist of two levels of learning, in which $g$ is applied at the
inner loop while our $B_{\mathcal{G}}$ is applied at the outer loop..
Nevertheless, the generalizability of $g$ and $A_{\mathcal{H}}$ are
drastically different. $A_{\mathcal{H}}$ is designed or selected by model
selection specifically for $\mathcal{D}$, while $g$ is learned from
$D_{\text{meta}}$ for the purpose of generalizing to tasks sampled from
$\mathcal{D}_{\text{meta}}$.

Moreover, viewing $(D_{\text{tr}},h^{*})$ as a meta labeled example enables
$h^{*}$ to be disentangled from $D_{\text{tr}}$; i.e., $h^{*}$ can be flexibly
defined to provide supervision for various applications, and is not
necessarily a base model that performs on data sampled exactly from where
$D_{\text{tr}}$ is sampled. This notion broadens the applicability of meta-
learning — e.g., to domain generalization where $h^{*}$ is for a different
domain, or to unsupervised (feature) learning where $D_{\text{tr}}$ contains
only unlabeled examples $\\{x_{n}\\}_{n=1}^{N}$. See subsection 2.4 for more
details.

#### Comparisons to previous work.

Viewing meta-learning as learning a mapping $g$ from $D_{\text{tr}}$ to $h$
has been studied in (Thrun and Pratt, 1998; Baxter, 2000; Maurer, 2005).
However, they did not treat $(D_{\text{tr}},h^{*})$ pairs as meta-training
data (cf. Equation 4 and Equation 5) to embody the flexibility in designing
$h^{*}$. In contrast, they mainly analyzed the case where $g(D_{\text{tr}})$
is applied to data from the same distribution where $D_{\text{tr}}$ is
sampled: i.e., each task is a supervised learning problem and the meta-
training objective is to minimize the training loss on $D_{\text{tr}}$ across
multiple tasks with generalization guarantees. In other words, they treated
$(D_{\text{tr}},D_{\text{tr}})$ pairs as meta-training data. Their studies may
not be applied to recent meta-learning, where validation data is explicitly
used to optimize the meta model (cf. Equation 6), and to applications beyond
supervised learning, such as domain generalization and continual learning
(Kaiser et al., 2017), where $h^{*}$ is designed for another data
distribution. Finn (2018); Larochelle (2018) considered
$(D_{\text{tr}},D_{\text{val}})$ pairs but do not explicitly allow
$D_{\text{val}}$ to be sampled from a different distribution. Our work is
inspired by (Wang and Hebert, 2016; Wang et al., 2017; Wang, 2018) which
explicitly defined $h^{*}$ for few-shot learning.

###  2.3 Generalizability of learned meta models

This unifying view enables transferring experiences of learning a base model
to learning a meta model. For example, increasing the size of meta-training
set or minimizing the domain shift between $\mathcal{D}_{\text{meta}}$ and
where the novel tasks will be sampled (Ben-David et al., 2010; Gong et al.,
2012) should improve the generalization ability of the learned meta model
$\hat{g}$. We empirically verify that these experiences are applicable in
section 5.

We can further apply theoretical analysis of supervised learning. Suppose
$l_{\text{meta}}$ is bounded and $|\mathcal{G}|$ is finite, the Chernoff bound
implies that $\mathcal{G}$ is agnostic PAC learnable using ERM (Shalev-Shwartz
and Ben-David, 2014). This has indeed been exploited in (Garg and Kalai, 2018)
to derive a meta-learning bound, but only for meta unsupervised learning.

###  2.4 A principled way to apply meta-learning

Figure 2: Meta labeled examples. A labeled image is marked by a colored box
that indicate its label. The target model in unsupervised learning is a
feature extractor that facilitates clustering or classification. The target
model for domain generalization is a classifier for a different domain.

This unifying view clearly defines the components of meta-learning: (a) meta
labeled example $(D_{\text{tr}},h^{*})$ and meta loss $l_{\text{meta}}$; (b)
meta hypothesis set $\mathcal{G}=\\{g\\}$ and meta algorithm
$B_{\mathcal{G}}$. Applying meta-learning to an area thus requires defining or
designing them accordingly. We discuss two examples below, and Figure 2 gives
an illustration. Please see suppl. for additional examples.

#### Unsupervised learning.

Let us consider learning a feature extractor from unlabeled data to benefit
downstream applications (Boney and Ilin, 2018; Metz et al., 2018; Yu et al.,
2018). Here, $D_{\text{tr}}=\\{x_{n}\\}_{n=1}^{N}$, $h^{*}$ is the target
feature extractor, and $l_{\text{meta}}$ measures the performance gap on the
downstream application. $\mathcal{G}$ is a set of objective functions for
unsupervised learning; $B_{\mathcal{G}}$ can be ERM by stochastic gradient
descent (SGD).

#### Domain generalization.

The goal is to train a base model on source domains (SD) and apply it to
target domains (TD) without fine-tuning. Here,
$D_{\text{tr}}=\\{(x_{n},y_{n})\\}_{n=1}^{N}$ is the labeled data from SD,
while $h^{*}$ is the target base model that works well on TD.
$l_{\text{meta}}$ is the performance gap on TD. $g$ can be a model predictor
or a learning process aware of the domain shifts (Li et al., 2018a; Balaji et
al., 2018; Li et al., 2019). $B_{\mathcal{G}}$ can be ERM by SGD.

Required: $D_{\text{meta}\text{-}\text{tr}}$: a meta-training set

$\alpha$, $\beta$, $K$: hyper-parameters

$B_{\mathcal{G}}$: an iterative meta-learning Algo.,

with $\hat{g}=B_{\mathcal{G}}(D_{\text{meta}\text{-}\text{tr}})$ converged.

Meta input: A novel test task $D_{\text{tr}}^{\text{test}}$

1\. Search $K$NN($D_{\text{tr}}^{\text{test}}$): $K$NN tasks in
$D_{\text{meta}\text{-}\text{tr}}$

2\. Copy $\tilde{g}$ from $\hat{g}$. Fine-tune $\tilde{g}$ on
$K$NN($D_{\text{tr}}^{\text{test}}$), by applying $B_{\mathcal{G}}$ for
$\beta$ extra epochs with step size $\alpha$

Meta output: $\tilde{g}(D_{\text{tr}}^{\text{test}})$

Algorithm 1 meta-KNN (cf. Equation 5)

###  2.5 Supervised learning techniques

Our unifying view enables transferring techniques of supervised learning to
meta-learning with minimum adjustment. In our experiment, we adapt widely-used
techniques for improving generalization abilities or facilitating
optimization, including data augmentation, ensemble methods (Breiman, 1996;
Zhou, 2012; Dietterich, 2000), joint training (e.g., multi-task learning
(Argyriou et al., 2007)), and pre-training (Yosinski et al., 2014) to meta-
learning.

Besides, it is well-known that non-parametric models (Weinberger and Saul,
2009) are able to capture local and heterogeneous structures in data. Contrast
to the fact that most existing meta models are parametric, we present a
procedure called meta-KNN (algorithm 1), inspired by SVM-KNN (Zhang et al.,
2006; Chao et al., 2013), to build a non-parametric meta model. Meta-KNN
begins with training a conventional meta model $\hat{g}$ on the meta-training
set $D_{\text{meta}\text{-}\text{tr}}$. During meta-testing, given a test task
$D_{\text{tr}}^{\text{test}}$, meta-KNN then searches for its $K$ nearest
neighbor (KNN) tasks from $D_{\text{meta}\text{-}\text{tr}}$, and fine-tune
$\hat{g}$ using $B_{G}$ to minimize the meta-training loss computed only on
those KNN tasks. The fine-tuned $\hat{g}$ (i.e., $\tilde{g}$) is then applied
to the test task $D_{\text{tr}}^{\text{test}}$. We empirically show its
superior performance on meta-learning from heterogeneous domains.

##  3 Case Study: Few-Shot Learning

We present a case study on how to apply our meta-learning framework to few-
shot learning. The goal of few-shot learning is to quickly build a model for a
novel task; i.e., with minimum training time and training data. Specifically,
we focus on one-shot $C$-way learning for image classification.

###  3.1 Meta labeled examples and meta losses

The training set $D_{\text{tr}}$ contains one labeled image for each $C$
classes; i.e., $D_{\text{tr}}=\\{(x_{n},y_{n})\\}_{n=1}^{N}$ ($C=N$ in this
case). The target model $h^{*}$ is a $C$-way classifier and the loss
$l_{\text{meta}}(h,h^{*})$ measures how different $h$ compared to $h^{*}$. We
present two examples of $l_{\text{meta}}(h,h^{*})$ as follows.

#### Meta losses in the model space.

$h^{*}$ is the target classifier trained on a larger training set. Let $h$ be
parameterized by $\theta$,
$l_{\text{meta}}(h,h^{\prime})=\|\theta-\theta^{\prime}\|^{2}_{2}$ (Wang and
Hebert, 2016; Wang et al., 2017; Gui et al., 2018).

#### Meta losses from example losses.

Given a validation set $D_{\text{val}}=\\{(x_{m},y_{m})\\}_{m=1}^{M}$ sampled
in the same way as $D_{\text{tr}}$, a reasonable choice of
$l_{\text{meta}}(h,h^{*})$ is $|L_{V}(h)-L_{V}(h^{*})|$, where $L_{V}$ is the
validation error defined in Equation 3. Suppose $h^{*}$ minimizes $L_{V}$,
then $l_{\text{meta}}(h,h^{*})$ is equivalent to $L_{V}(h)$. In other words,
we replace $h^{*}$ and $l_{\text{meta}}$ by $D_{\text{val}}$ and $l$. The
meta-training set $D_{\text{meta}\text{-}\text{tr}}$ thus becomes
$\\{({D_{\text{tr}}}_{j},{D_{\text{val}}}_{j})\\}_{j=1}^{N_{\text{meta}}}$ and
ERM in Equation 5 can be re-written accordingly as (with constants ignored)

$\displaystyle\vspace{-5pt}\hat{g}$
$\displaystyle=\operatorname{arg\,min}_{g\in\mathcal{G}}\sum_{j=1}^{N_{\text{meta}}}{L_{V}}_{j}(g({D_{\text{tr}}}_{j}))$
$\displaystyle=\operatorname{arg\,min}_{g\in\mathcal{G}}\sum_{j=1}^{N_{\text{meta}}}\sum_{m=1}^{M}l(g({D_{\text{tr}}}_{j})(x_{jm}),y_{jm}).\vspace{-5pt}$
(6)

${L_{V}}_{j}$ is the validation loss on
${D_{\text{val}}}_{j}=\\{(x_{jm},y_{jm})\\}_{m=1}^{M}$. Equation 6 has been
applied in many few-shot learning algorithms (Hariharan and Girshick, 2017;
Ravi and Larochelle, 2017; Gidaris and Komodakis, 2018; Qiao et al., 2018;
Snell et al., 2017; Sung et al., 2018; Bertinetto et al., 2016; Vinyals et
al., 2016) and we will focus on it in the next subsections.

###  3.2 Meta models and meta algorithms

We discuss two exemplar designs of $g$: one views $g$ as a supervised learning
algorithm (Finn et al., 2017a) and the other views $g$ as a feed-forward model
predictor (Snell et al., 2017). Both algorithms are used in our experiments.

#### Meta models as learning algorithms.

Since $g$ outputs a classifier $h$ given $D_{\text{tr}}$, it can be seen as a
learning algorithm. Suppose $h$ is parameterized by $\theta$, let us apply ERM

$\displaystyle\vspace{-5pt}g(D_{\text{tr}})=\arg\min_{\theta}L_{S}(\theta)=\arg\min_{\theta}\cfrac{1}{N}\sum_{n=1}^{N}l(h_{\theta}(x_{n}),y_{n}).\vspace{-5pt}$

Since $D_{\text{tr}}$ is small, ERM may suffer over-fitting. One solution is
to apply an iterative optimizer (e.g., gradient descent (GD)) with early
stopping and a carefully chose initialization $\psi$ (Finn et al., 2017a):
early stopping limits the hypothesis set while $\psi$ prevents under-fitting.
Suppose $L_{S}(\theta)$ is differentiable w.r.t. $\theta$, $g$ with one-step
GD of a step size $\alpha$, initialized at $\psi$, is

$\displaystyle
g_{\psi}(D_{\text{tr}})=\hat{\theta}=\psi-\alpha\times\nabla_{\theta}L_{S}(\psi).$
(7)

The meta hypothesis set $\mathcal{G}$ thus becomes $\\{g_{\psi}\\}$ with
different initializations. To search $g_{\hat{\psi}}$, we apply ERM but on the
meta-training set $D_{\text{meta}\text{-}\text{tr}}$ (cf. Equation 6)

$\displaystyle\hat{\psi}=$
$\displaystyle\operatorname{arg\,min}_{\psi}\sum_{j=1}^{N_{\text{meta}}}\sum_{m=1}^{M}l(g_{\psi}({D_{\text{tr}}}_{j})(x_{jm}),y_{jm}).$
(8)

If we apply SGD for optimization then this is the one-step MAML (Finn et al.,
2017a). In other words, MAML and its variants (Lee and Choi, 2018; Finn and
Levine, 2018; Li et al., 2017; Rusu et al., 2019) fit in our framework.

#### Meta models as model predictors.

Alternatively, we can view $g(D_{\text{tr}})$ as a model predictor. For
example, in Prototypical Network (ProtoNet) (Snell et al., 2017)

$\displaystyle
g_{\psi}(D_{\text{tr}})(x)=\operatorname{arg\,max}_{c}\exp(-\|\psi(x)-\psi(x_{c})\|_{2}^{2}),$
(9)

where $x_{c}$ is the image of class $c$, and $\psi$ is a feature extractor.
Snell et al. (2017) learns $\psi$ via ERM (Equation 6) and fits in our
framework. Similar approaches are (Qiao et al., 2018; Sung et al., 2018; Wang
and Hebert, 2016; Vinyals et al., 2016; Wang et al., 2017).

###  3.3 Collecting meta-training sets

After identifying the form of meta labeled example, which is
$(D_{\text{tr}},D_{\text{val}})$ for few-shot learning, the next step is to
collect the meta-training set. While collecting
$(D_{\text{tr}},D_{\text{val}})$ one-by-one seems standard, it might be
inefficient. For image classification it is easier to collect images class-by-
class to first create a pool (named meta-train-pool), which contains many
classes and each has many examples. We then synthesize
$(D_{\text{tr}},D_{\text{val}})$ from the pool. This is the general setting
for few-shot image classification. There will be three pools of disjoint
classes: meta-train pool, meta-val-pool, meta-test-pool.

##  4 Related Work

There are excellent overviews and surveys of meta-learning (Thrun and Pratt,
1998; Vilalta and Drissi, 2002; Lemke et al., 2015; Vanschoren, 2018; Finn,
2018; Metz et al., 2018). There are also theoretical analysis (Baxter, 2000;
Maurer et al., 2016; Maurer, 2005). However, the flexibility to design $h^{*}$
is not considered. Franceschi et al. (2017) connected hyperparameter tuning
and meta-learning, which aligns with the comparison of $A_{\mathcal{H}}$ and
$g$ in subsection 2.2. Garg and Kalai (2018) also related meta-learning to
supervised learning but only for meta-unsupervised learning. Meta-learning has
also been applied to reinforcement learning and imitation learning (Stadie et
al., 2018; Frans et al., 2018; Wang et al., 2016; Duan et al., 2016, 2017;
Finn et al., 2017b; Yu et al., 2018), optimization (Andrychowicz et al., 2016;
Wichrowska et al., 2017; Li and Malik, 2017; Bello et al., 2017),
recommendation systems (Vartak et al., 2017), data augmentation (Ratner et
al., 2017), natural language processing (Huang et al., 2018), architecture
search (Elsken et al., 2018), continual learning (Riemer et al., 2019; Kaiser
et al., 2017; Al-Shedivat et al., 2018; Clavera et al., 2019), transfer and
multi-task learning (Ying et al., 2018; Zhang et al., 2018b), active learning
(Ravi and Larochelle, 2018; Sharma et al., 2018; Bachman et al., 2017; Ravi
and Larochelle, 2018; Sharma et al., 2018; Pang et al., 2018), and teaching
(Fan et al., 2018). Some algorithms aim for versatile purposes (Mishra et al.,
2018; Munkhdalai and Yu, 2017; Ritter et al., 2018; Finn et al., 2017a;
Santoro et al., 2016; Nichol et al., 2018).

##  5 Experiments

We validate the advantages of our unifying view by investigating three aspects
of few-shot learning: (a) whether the number of meta-examples influences the
generalization; (b) whether supervised learning techniques are applicable to
meta-learning; (c) whether our unifying view can facilitate new applications
and algorithm design. We first describe the setups.

#### Datasets.

The MiniImageNet dataset (Vinyals et al., 2016) is a subset of ImageNet
(Russakovsky et al., 2015) and is widely-used in few-shot learning. There are
100 classes and 600 examples per class. We split the datasets following (Ravi
and Larochelle, 2017): there are 64, 16, 20 classes for meta-train-pool, meta-
val-pool, and meta-test-pool. We also consider a challenging case where the
splits are 30, 30, 40 classes, in which the diversity of meta-training
examples is limited. We call the former standard split (SS) and the later
challenging split (CS). We further consider two set of datasets. To enlarge
the heterogeneity among tasks, we synthesize a “Heterogeneous” dataset from
five fine-grained datasets, namely AirCraft (Maji et al., 2013), Car-196
(Krause et al., 2013), Caltech-UCSD Birds (CUB) 200-2011 (Wah et al., 2011),
Stanford Dog (Khosla et al., 2011), and Indoor (Quattoni and Torralba, 2009).
We sample 60 classes from each dataset, and equally split them into meta-
train-pool, meta-val-pool, and meta-test-pool. To investigate the
applicability of meta-learning, we use the Office-Home datasets (Venkateswara
et al., 2017) in a domain generalization problem. There are 65 classes and 4
domains of images per class. We test two domains, “Clipart” and “Product.” See
suppl. for more details.

#### Meta examples (tasks) and evaluation protocols.

We focus on the 1-shot 5-way tasks, unless stated otherwise. We follow (Rusu
et al., 2019) to draw 10,000 tasks $(D_{\text{tr}},D_{\text{val}})$ from meta-
test-pool and there are 15 validation images per class in a task, except for
Table 4. We report the mean accuracy. We found the 95% confidence interval to
be consistently within $[0.001,0.004]$ and thus omit it for brevity.

#### Baselines.

We investigate Prototypical Network (ProtoNet) (Snell et al., 2017), Matching
Network (MatchNet) (Vinyals et al., 2016), and Model Agnostic Meta-Learning
(MAML) (Finn et al., 2017a). We implement the algorithms and use the same
$C$-way setting in meta-training and meta-test for consistency: we disregard
the trick (Snell et al., 2017) that trains with $30$-way tasks but tests with
$5$-way tasks. We use the standard 4-layer ConvNets as the backbone (Vinyals
et al., 2016; Snell et al., 2017) and resize images to $84\times 84$ following
(Vinyals et al., 2016). More details, together with the results using ResNet
(He et al., 2016), are in suppl.

###  5.1 Generalization analysis

(a) MAML

(b) ProtoNet

(c) MatchNet

Figure 3: 1-shot 5-way accuracy on MiniImageNet (SS). We show meta-training
(red) and meta-test (blue) accuracy. We vary the numbers of instances per
class in meta-train-pool to control the number of meta-training examples. All
algorithms suffer over-fitting when the number gets small.

Following our supervised view of meta-learning, the generalization ability of
the learned meta model should be affected by the “effective” number of meta
training examples; i.e., the number of different tasks. We validate the
influence with various meta-train-pool configurations on MiniImageNet (SS). We
keep all the 64 classes in meta-train-pool to keep the meta distribution
intact. Instead, we change the number of instances in each class from 600 to
50 to construct different meta-train-pools. When there are limited instances
per class in the meta-train-pool, the number of unique tasks is constrained as
well. We also evaluate the case of fewer classes in suppl. Figure 3 shows the
change of few-shot accuracy with different meta-train-pools. The trend follows
the supervised view: the more meta-training examples (few-shot tasks) are
provided, the better generalization is achieved. Specifically, learning with
50 instances per class significantly over-fits.

###  5.2 Supervised learning techniques

We investigate five techniques: joint (multi-objective) learning, pre-
training, bagging, data augmentation, and non-parametric methods. We
experiment on MiniImageNet (SS), unless stated otherwise.

#### Multi-objective learning.

In addition to the original objective (i.e., meta-training loss), learning
jointly with related objectives has shown promising results in supervised
learning: it serves as a data-dependent regularization to improve
generalization. Here we add another objective: a 64-way classification with
cross-entropy loss over all classes in meta-train-pool. This classifier shares
weights with the meta model, except for the last layer. In other words, the
shared sub-network should jointly master two tasks: predicting good $C$-way
classifiers and extracting discriminative features for 64-way classification.
Oreshkin et al. (2018) explored this idea, yet we provide more insights in
Table 1.

#### Pre-training.

It is well-known that a good model initialization significantly facilitates
the model optimization for down-stream task (Erhan et al., 2010; Bengio et
al., 2007). Specifically, supervised pre-training (Yosinski et al., 2014) on a
large labeled dataset has been prevalent in many applications. While the
standard setup in few-shot learning trains the meta model from scratch on the
sampled meta labeled examples, we investigate training a 64-way classifier
with cross-entropy loss at first, and use it to initialize the meta model.
Such a strategy is explored in (Chen et al., 2019; Qiao et al., 2018; Rusu et
al., 2019; Li et al., 2018b), yet we provide more insights as follows444We
note that Triantafillou et al. (2019) also provided detailed studies on pre-
training in their latest version..

The results by applying the two techniques are shown in Table 1. We list both
the meta-training and meta-test accuracy. Both techniques consistently improve
the test accuracy for all three algorithms. _However, their underlying
influences are different._ Pre-training achieves the highest meta-training
accuracy, justifying its effectiveness to facilitate optimization. Multi-
objective learning does not increase the meta-training accuracy much but
improves the meta-test accuracy, verifying its ability to improve
generalization. In summary, we show that well-known supervised learning
techniques are applicable to meta-learning in the same manner they benefit
supervised learning.

Table 1: Multi-objective learning and pre-training for 1-shot 5-way classification on MiniImageNet (SS).  | MAML | ProtoNet | MatchNet
---|---|---|---
| Scratch | Pre-Train | Multi-Obj. | Scratch | Pre-Train | Multi-Obj. | Scratch | Pre-Train | Multi-Obj.
Train | 0.540 | 0.576 | 0.554 | 0.539 | 0.602 | 0.537 | 0.547 | 0.662 | 0.598
Test | 0.459 | 0.478 | 0.470 | 0.461 | 0.500 | 0.484 | 0.463 | 0.500 | 0.485

#### Ensemble methods.

Ensemble methods leverage the diversity among a set of basic models to
construct a robust summarized model. It has been comprehensively verified in
supervised learning. We apply a simple ensemble method bagging (Breiman, 1996)
to meta-learning, which reduces the model variances by majority voting over
many classifiers. We generate diverse meta models by learning from different
meta-training sets. Concretely, we sample 10 different sub meta-train-pools
from the original one: each sub-pool contain 48 classes. Then we train 10
basic meta models over those sub meta-train-pools and ensemble the results for
final tests.

Table 2: Bagging for 1/5-shot 5-way classification on MiniImageNet (SS). Single: no bagging. Average: average accuracy of basic meta models. The best result of each shot-algorithm pair is in bold font. | MAML | ProtoNet | MatchNet
---|---|---|---
| Single | Average | Bagging | Single | Average | Bagging | Single | Average | Bagging
1-shot | w/o Pre-Train | 0.459 | 0.448 | 0.482 | 0.461 | 0.463 | 0.491 | 0.463 | 0.457 | 0.482
w/ Pre-Train | 0.478 | 0.439 | 0.498 | 0.500 | 0.492 | 0.526 | 0.500 | 0.495 | 0.525
5-shot | w/o Pre-Train | 0.633 | 0.617 | 0.666 | 0.658 | 0.649 | 0.694 | 0.639 | 0.625 | 0.664
w/ Pre-Train | 0.660 | 0.646 | 0.692 | 0.671 | 0.657 | 0.701 | 0.642 | 0.630 | 0.673

The results are in Table 2. “Single” is the model trained on the original
meta-train-pool, the same as in Table 1. The average performance of all 10
basic meta models are in the “Average” column. (Note that this is not ensemble
yet.) We investigate “Bagging” by combining the prediction probabilities of
the 10 models. The average performance of basic meta models is mostly worse
than that of the meta model trained on the full 64-class meta-train-pool. But
after we ensemble the basic models the performance improves notably,
validating the effectiveness of ensemble for meta-learning. Moreover, we see
that pre-training and bagging are compatible and lead to impressive
0.526/0.701 accuracy for 1/5-shot learning, which is on par with one state-of-
the-art (Qiao et al., 2018) but with a cleaner meta-training procedure. In
suppl. we also investigate this compatibility on CUB (Wah et al., 2011) and
tieredImageNet (Ren et al., 2018) and see similar trends of improvement. We
also experiment with more advanced meta-learning algorithms (Sung et al.,
2018; Ye et al., 2018) and backbones (He et al., 2016) in suppl.

_Given that the above three techniques appear to be meta-model-agnostic by
improving all three meta-learning models consistently, we focus on ProtoNet in
the following experiments._

Table 3: Test accuracy of 1-shot 30-way classification on MiniImageNet (CS). $K$: cluster number to augment meta-train-pool, meta-val-pool classes. $K=1$: no augmentation. $K$ = 1 |  $K$ = 2 |  $K$ = 4 |  $K$ = 8
---|---|---|---
0.146 | 0.146 | 0.155 | 0.161

#### Data augmentation.

Increasing the number of training examples by data augmentation is a popular
technique in supervised learning to improve generalization, especially when
the training examples are not diverse enough. Here we adapt the idea to meta-
learning. We consider a challenging yet more realistic case on MiniImageNet
(CS). Specifically, we re-split the dataset into 30, 30, 40 classes for meta-
train-pool, meta-val-pool, and meta-test-pool, and focus on $1$-shot $30$-way
classification. Note that, this setting might break the applicability of meta-
learning to few-shot learning: every time we will sample the same 30 classes
from meta-train-pool, greatly limiting the diversity of meta labeled examples.
However, this setting is indeed more realistic in practice according to the
well-known long-tailed distribution (Sudderth and Jordan, 2008; Salakhutdinov
et al., 2011; Zhu et al., 2014; Wang et al., 2017): there are more few-shot
classes than many-shot classes.

We present a data augmentation strategy at the meta example (task) level,
inspired by (Hsu et al., 2018). Specifically, we perform K-means within each
class to split a class into $K$ subcategories, resulting in $30\times K$
augmented classes in meta-train-pool (and meta-val-pool). The results are in
Table 3. We cluster each class in the meta-train-pool and meta-val-pool with
$K={1,2,4,8}$ subcategories; $1$ means no K-means. We observe a clear trend:
the more classes we augment to increase task diversity, the higher accuracy we
can achieve. More details and results are in suppl.

Table 4: Parametric (ProtoNet) vs. non-parametric models (ProtoNet-KNN). We show the test accuracy of 1-shot 5-way fine-grained classification on the Heterogeneous dataset. ProtoNet | ProtoNet-KNN
---|---
0.372 | 0.386

#### Non-parametric methods.

Non-parametric approaches are known to capture local and heterogeneous
structures in data. Here we extend the methods to meta-learning by applying
meta-KNN (cf. algorithm 1) to ProtoNet, named ProtoNet-KNN. ProtoNet-KNN first
learns a normal ProtoNet $\hat{g}$ on the meta-training set
$D_{\text{meta}\text{-}\text{tr}}$. $B_{G}$ is therefore SGD that minimizes
the meta-training loss of ProtoNet (i.e., Equation 5 and Equation 6). In meta-
test, given a test few-shot task $D_{\text{tr}}^{\text{test}}$, we search for
its $K$ nearest neighbor (KNN) few-shot tasks from the meta-training set
$D_{\text{meta}\text{-}\text{tr}}$. We characterize each task by its average
features and use the Euclidean distance to measure task similarity. We then
fine-tune $\hat{g}$ using $B_{G}$ (i.e., SGD), but to minimize the meta-
training loss computed only on those KNN tasks555We note that since at first
$\hat{g}$ has been trained to minimize the meta-training loss on
$D_{\text{meta}\text{-}\text{tr}}$, fine-tuning $\hat{g}$ on the “entire”
meta-training set $D_{\text{meta}\text{-}\text{tr}}$ will not lead to further
improvement.. The fine-tuned $\hat{g}$ (i.e., $\tilde{g}$) is then applied to
the test task $D_{\text{tr}}^{\text{test}}$. Here we set $K=100$ and fine-tune
$\hat{g}$ for only $1$ epoch, which takes $<400$ ms. See suppl. for details
and further discussions.

We investigate ProtoNet-KNN on the Heterogeneous dataset described earlier in
setups. We focus on few-shot fine-grained classification: a test task contains
5 classes from one specific dataset. In meta-training, we sample a fine-
grained task randomly from a dataset to train a single ProtoNet. We expect the
learned ProtoNet to be equipped with dataset-agnostic fine-grained
discriminative knowledge. In meta-test, we then apply ProtoNet-KNN to adapt
the ProtoNet to the neighbor tasks of a test task. To facilitate neighbor
searching, we pre-sample 2,000 training tasks from
$D_{\text{meta}\text{-}\text{tr}}$ (cf. algorithm 1). Table 4 shows the meta-
test results (on 2,000 tasks). Non-parametric methods greatly improve the
performance on heterogeneous tasks. See suppl. for additional details
including the hyper-parameters.

###  5.3 Domain generalization & domain shifts

Table 5: Few-shot 5-way domain generalization on Office-Home with two domains: Clipart (C), Product (P). We denote the source/target domain by SD/TD.  | Meta-train | Meta-test | Test Acc
---|---|---|---
Case | SD | TD | SD | TD | 1-Shot | 5-Shot
I-1 | C | C | C | C | 0.341 | 0.477
I-2 | P | C | P | C | 0.296 | 0.350
I-3 | C | C | P | C | 0.275 | 0.342
I-4 | P | P | P | C | 0.264 | 0.283

Existing few-shot learning assumes that the training and validation examples
within a task are from the same domain. In practice it is sometime desirable
to remove the assumption; e.g., we may want to learn a robot in a constraint
environment, where examples are not diverse and are essentially few-shot, but
apply it in the wild. Recall that our supervised view allows the training set
and target model $h^{*}$ of a task to be defined according to the problem at
hand. By defining the target model $h^{*}$ to be the one that will work well
in a different domain, we can remove the same-domain assumption
systematically. We study this idea and apply the trick in section 3 to replace
the target model $h^{*}$ by a validation set collected in a different domain.

We experiment on the Off-Home dataset described earlier in setups, which is
designed for domain adaptation. We focus on two domains: Clipart (C) and
Product (P). There are 65 classes and we split them into 25 for meta-train-
pool, 15 for meta-val-pool, and 25 for meta-test-pool. We work on _few-shot
5-way_ tasks in which a task contains a training set $D_{\text{tr}}$ from one
(source) domain and a validation set $D_{\text{val}}$ from the other (target)
domain. We design several scenarios to illustrate the difference between
domain generalization and meta-domain shift. In domain generalization, a task
$(D_{\text{tr}},D_{\text{val}})$ has its $D_{\text{tr}}$ and $D_{\text{val}}$
sampled from two different domains. In meta-domain shift, the tasks in the
meta-training set and meta-test set are sampled from different meta
distributions.

Table 5 shows the results. Comparing Case I-1 and I-2, we see the challenge of
domain generalization when the source and target domains within a task are
different. We note that in each of these cases the distribution of the meta-
training and meta-test examples are the same666That is, I-1 and I-2 follow the
assumption in supervised learning that the meta-training tasks and meta-test
tasks are sampled from the same meta distribution.. Therefore, there is no
meta-domain shift and the gap simply indicates the difficulty of tasks.

We further investigate meta-domain shifts by constructing meta-training/-test
examples in a different way (Case I-3 and I-4). The results are outperformed
by Case I-2. (Note that all three cases consider the same meta-test examples.)
We attribute this gap to meta-domain shifts (cf. subsection 2.3), which can
potentially be resolved via domain adaptation by casting meta-learning as
supervised learning.

##  6 Conclusion

We revisit and strengthen the connection between meta-learning and supervised
learning, upon which applying meta-learning becomes systematic; theoretical
and empirical understandings of supervised learning can be applied to meta-
learning. We further demonstrate that various supervised learning techniques
can benefit meta-learning in the same ways they benefit supervised learning.
We hope our studies to re-inform the community of such a valuable connection
and inspire new algorithms, theories, and understandings of meta-learning.

## Acknowledgments

This research is supported in part by grants from the National Science
Foundation (III-1618134, III-1526012, IIS-1149882, IIS-1724282, and
TRIPODS-1740822), the Office of Naval Research DOD (N00014-17-1-2175), the
Bill and Melinda Gates Foundation, and the Cornell Center for Materials
Research with funding from the NSF MRSEC program (DMR-1719875). We are
thankful for generous support by SAP America Inc., and Facebook. We also thank
Jie Fu (MILA), Ming-Yu Liu (NVIDIA), Po-Hsuan Chen (Google Brain), and Yu-
Chiang Frank Wang (Natl. Taiwan Univ.) for discussions.

## References

  * Achille et al. (2019a) Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless Fowlkes, Stefano Soatto, and Pietro Perona.  Task2vec: Task embedding for meta-learning.  _CoRR_ , abs/1902.03545, 2019a.
  * Achille et al. (2019b) Alessandro Achille, Giovanni Paolini, Glen Mbeng, and Stefano Soatto.  The information complexity of learning tasks, their structure and their distance.  _CoRR_ , abs/1904.03292, 2019b.
  * Al-Shedivat et al. (2018) Maruan Al-Shedivat, Trapit Bansal, Yuri Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel.  Continuous adaptation via meta-learning in nonstationary and competitive environments.  In _ICLR_ , 2018.
  * Andrychowicz et al. (2016) Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas.  Learning to learn by gradient descent by gradient descent.  In _NIPS_ , 2016.
  * Argyriou et al. (2007) Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil.  Multi-task feature learning.  In _NIPS_ , 2007.
  * Bachman et al. (2017) Philip Bachman, Alessandro Sordoni, and Adam Trischler.  Learning algorithms for active learning.  In _ICML_ , 2017.
  * Balaji et al. (2018) Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa.  Metareg: Towards domain generalization using meta-regularization.  In _NeurIPS_ , 2018.
  * Baxter (2000) Jonathan Baxter.  A model of inductive bias learning.  _Journal of Artificial Intelligence Research_ , 12:149–198, 2000.
  * Bello et al. (2017) Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoc V Le.  Neural optimizer search with reinforcement learning.  In _ICML_ , 2017.
  * Ben-David et al. (2010) Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan.  A theory of learning from different domains.  _Machine learning_ , 79(1-2):151–175, 2010.
  * Bengio et al. (2007) Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle.  Greedy layer-wise training of deep networks.  In _Advances in neural information processing systems_ , pages 153–160, 2007.
  * Bertinetto et al. (2016) Luca Bertinetto, João F Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi.  Learning feed-forward one-shot learners.  In _NIPS_ , 2016.
  * Boney and Ilin (2018) Rinu Boney and Alexander Ilin.  Semi-supervised few-shot learning with maml.  2018\.
  * Breiman (1996) Leo Breiman.  Bagging predictors.  _Machine learning_ , 24(2):123–140, 1996.
  * Chao et al. (2013) Wei-Lun Chao, Jun-Zuo Liu, and Jian-Jiun Ding.  Facial age estimation based on label-sensitive learning and age-oriented regression.  _Pattern Recognition_ , 46(3):628–641, 2013.
  * Chen et al. (2019) Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang.  A closer look at few-shot classification.  In _ICLR_ , 2019.
  * Clavera et al. (2019) Ignasi Clavera, Anusha Nagabandi, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn.  Learning to adapt in dynamic, real-world environments through meta-reinforcement learning.  In _ICLR_ , 2019.
  * Dietterich (2000) Thomas G Dietterich.  Ensemble methods in machine learning.  In _International workshop on multiple classifier systems_ , pages 1–15, 2000.
  * Duan et al. (2016) Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel.  Rl 2: Fast reinforcement learning via slow reinforcement learning.  _arXiv preprint arXiv:1611.02779_ , 2016.
  * Duan et al. (2017) Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAI Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba.  One-shot imitation learning.  In _NIPS_ , 2017.
  * Edwards and Storkey (2017) Harrison Edwards and Amos Storkey.  Towards a neural statistician.  In _ICLR_ , 2017.
  * Elsken et al. (2018) Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter.  Neural architecture search: A survey.  _arXiv preprint arXiv:1808.05377_ , 2018.
  * Erhan et al. (2010) Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio.  Why does unsupervised pre-training help deep learning?  _JMLR_ , 11(Feb):625–660, 2010.
  * Fan et al. (2018) Yang Fan, Fei Tian, Tao Qin, Xiang-Yang Li, and Tie-Yan Liu.  Learning to teach.  In _ICLR_ , 2018.
  * Finn (2018) Chelsea Finn.  _Learning to Learn with Gradients_.  PhD thesis, UC Berkeley, 2018.
  * Finn and Levine (2018) Chelsea Finn and Sergey Levine.  Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm.  In _ICLR_ , 2018.
  * Finn et al. (2017a) Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _ICML_ , 2017a.
  * Finn et al. (2017b) Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine.  One-shot visual imitation learning via meta-learning.  In _CoRL_ , 2017b.
  * Franceschi et al. (2017) Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil.  A bridge between hyperparameter optimization and larning-to-learn.  _arXiv preprint arXiv:1712.06283_ , 2017.
  * Frans et al. (2018) Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman.  Meta learning shared hierarchies.  In _ICLR_ , 2018.
  * Garg and Kalai (2018) Vikas Garg and Adam Kalai.  Supervising unsupervised learning.  In _NeurIPS_ , 2018.
  * Gidaris and Komodakis (2018) Spyros Gidaris and Nikos Komodakis.  Dynamic few-shot visual learning without forgetting.  In _CVPR_ , 2018.
  * Gong et al. (2012) Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman.  Geodesic flow kernel for unsupervised domain adaptation.  In _CVPR_ , 2012.
  * Gui et al. (2018) Liang-Yan Gui, Yu-Xiong Wang, Deva Ramanan, and José MF Moura.  Few-shot human motion prediction via meta-learning.  In _ECCV_ , 2018.
  * Hariharan and Girshick (2017) Bharath Hariharan and Ross Girshick.  Low-shot visual recognition by shrinking and hallucinating features.  In _ICCV_ , 2017.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * Hsu et al. (2018) Kyle Hsu, Sergey Levine, and Chelsea Finn.  Unsupervised learning via meta-learning.  _arXiv preprint arXiv:1810.02334_ , 2018.
  * Huang et al. (2018) Po-Sen Huang, Chenglong Wang, Rishabh Singh, Wen-tau Yih, and Xiaodong He.  Natural language to structured query generation via meta-learning.  In _NAACL_ , 2018.
  * Kaiser et al. (2017) Łukasz Kaiser, Ofir Nachum, Aurko Roy, and Samy Bengio.  Learning to remember rare events.  In _ICLR_ , 2017.
  * Khosla et al. (2011) Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei.  Novel dataset for fine-grained image categorization.  In _First Workshop on Fine-Grained Visual Categorization, IEEE Conference on CVPR_ , 2011.
  * Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _CoRR_ , abs/1412.6980, 2014.
  * Krause et al. (2013) Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In _4th International IEEE Workshop on 3D Representation and Recognition_ , 2013.
  * Larochelle (2018) Hugo Larochelle.  Few-shot learning with meta-learning: Progress made and challenges ahead.  2018\.
  * Lee and Choi (2018) Yoonho Lee and Seungjin Choi.  Gradient-based meta-learning with learned layerwise metric and subspace.  In _ICML_ , 2018.
  * Lemke et al. (2015) Christiane Lemke, Marcin Budka, and Bogdan Gabrys.  Metalearning: a survey of trends and technologies.  _Artificial intelligence review_ , 44(1):117–130, 2015.
  * Li et al. (2018a) Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales.  Learning to generalize: Meta-learning for domain generalization.  In _AAAI_ , 2018a.
  * Li et al. (2018b) Kai Li, Martin Renqiang Min, Bing Bai, Yun Fu, and Hans Peter Graf.  Network reparameterization for unseen class categorization.  2018b.
  * Li and Malik (2017) Ke Li and Jitendra Malik.  Learning to optimize.  In _ICLR_ , 2017.
  * Li et al. (2019) Yiying Li, Yongxin Yang, Wei Zhou, and Timothy M Hospedales.  Feature-critic networks for heterogeneous domain generalization.  In _ICML_ , 2019.
  * Li et al. (2017) Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li.  Meta-sgd: Learning to learn quickly for few shot learning.  _arXiv preprint arXiv:1707.09835_ , 2017.
  * Maji et al. (2013) S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi.  Fine-grained visual classification of aircraft.  Technical report, 2013.
  * Maurer (2005) Andreas Maurer.  Algorithmic stability and meta-learning.  _JMLR_ , 6(Jun):967–994, 2005.
  * Maurer et al. (2016) Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes.  The benefit of multitask representation learning.  _JMLR_ , 17(1):2853–2884, 2016.
  * Metz et al. (2018) Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein.  Learning unsupervised learning rules.  _arXiv preprint arXiv:1804.00222_ , 2018.
  * Mishra et al. (2018) Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel.  A simple neural attentive meta-learner.  In _ICLR_ , 2018.
  * Munkhdalai and Yu (2017) Tsendsuren Munkhdalai and Hong Yu.  Meta networks.  In _ICML_ , 2017.
  * Nichol et al. (2018) Alex Nichol, Joshua Achiam, and John Schulman.  On first-order meta-learning algorithms.  _CoRR, abs/1803.02999_ , 2018.
  * Oreshkin et al. (2018) Boris N Oreshkin, Alexandre Lacoste, and Pau Rodriguez.  Tadam: Task dependent adaptive metric for improved few-shot learning.  In _NeurIPS_ , 2018.
  * Pan et al. (2010) Sinno Jialin Pan, Qiang Yang, et al.  A survey on transfer learning.  _IEEE Transactions on knowledge and data engineering_ , 22(10):1345–1359, 2010.
  * Pang et al. (2018) Kunkun Pang, Mingzhi Dong, Yang Wu, and Timothy Hospedales.  Meta-learning transferable active learning policies by deep reinforcement learning.  _arXiv preprint arXiv:1806.04798_ , 2018.
  * Qiao et al. (2018) Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L Yuille.  Few-shot image recognition by predicting parameters from activations.  In _CVPR_ , 2018.
  * Quattoni and Torralba (2009) Ariadna Quattoni and Antonio Torralba.  Recognizing indoor scenes.  In _CVPR_ , pages 413–420, 2009.
  * Ratner et al. (2017) Alexander J Ratner, Henry Ehrenberg, Zeshan Hussain, Jared Dunnmon, and Christopher Ré.  Learning to compose domain-specific transformations for data augmentation.  In _NIPS_ , 2017.
  * Ravi and Larochelle (2017) Sachin Ravi and Hugo Larochelle.  Optimization as a model for few-shot learning.  In _ICLR_ , 2017.
  * Ravi and Larochelle (2018) Sachin Ravi and Hugo Larochelle.  Meta-learning for batch mode active learning.  In _ICLR Workshop_ , 2018.
  * Ren et al. (2018) Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel.  Meta-learning for semi-supervised few-shot classification.  In _ICLR_ , 2018.
  * Riemer et al. (2019) Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro.  Learning to learn without forgetting by maximizing transfer and minimizing interference.  In _ICLR_ , 2019.
  * Ritter et al. (2018) Samuel Ritter, Jane X Wang, Zeb Kurth-Nelson, Siddhant M Jayakumar, Charles Blundell, Razvan Pascanu, and Matthew Botvinick.  Been there, done that: Meta-learning with episodic recall.  In _ICML_ , 2018.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li.  Imagenet large scale visual recognition challenge.  _IJCV_ , 115(3):211–252, 2015.
  * Rusu et al. (2019) Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell.  Meta-learning with latent embedding optimization.  In _ICLR_ , 2019.
  * Salakhutdinov et al. (2011) Ruslan Salakhutdinov, Antonio Torralba, and Josh Tenenbaum.  Learning to share visual appearance for multiclass object detection.  In _CVPR_ , 2011.
  * Santoro et al. (2016) Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap.  Meta-learning with memory-augmented neural networks.  In _ICML_ , 2016.
  * Shalev-Shwartz and Ben-David (2014) Shai Shalev-Shwartz and Shai Ben-David.  _Understanding machine learning: From theory to algorithms_.  Cambridge university press, 2014.
  * Sharma et al. (2018) Sahil Sharma, Ashutosh Jha, Parikshit Hegde, and Balaraman Ravindran.  Learning to multi-task by active sampling.  In _ICLR_ , 2018.
  * Snell et al. (2017) Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  In _NeurIPS_ , 2017.
  * Stadie et al. (2018) Bradly Stadie, Ge Yang, Rein Houthooft, Peter Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever.  The importance of sampling inmeta-reinforcement learning.  In _NeurIPS_ , 2018.
  * Sudderth and Jordan (2008) Erik B. Sudderth and Michael I. Jordan.  Shared segmentation of natural scenes using dependent pitman-yor processes.  In _NIPS_ , 2008.
  * Sung et al. (2018) Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales.  Learning to compare: Relation network for few-shot learning.  In _CVPR_ , 2018.
  * Thrun and Pratt (1998) Sebastian Thrun and Lorien Pratt.  _Learning to learn_.  Springer Science & Business Media, 1998.
  * Triantafillou et al. (2017) Eleni Triantafillou, Richard S. Zemel, and Raquel Urtasun.  Few-shot learning through an information retrieval lens.  In _Advances in Neural Information Processing Systems 30_ , pages 2252–2262. 2017.
  * Triantafillou et al. (2019) Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle.  Meta-dataset: A dataset of datasets for learning to learn from few examples.  _arXiv preprint arXiv:1903.03096_ , 2019.
  * Vanschoren (2018) Joaquin Vanschoren.  Meta-learning: A survey.  _arXiv preprint arXiv:1810.03548_ , 2018.
  * Vartak et al. (2017) Manasi Vartak, Arvind Thiagarajan, Conrado Miranda, Jeshua Bratman, and Hugo Larochelle.  A meta-learning perspective on cold-start recommendations for items.  In _NIPS_ , 2017.
  * Venkateswara et al. (2017) Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In _CVPR_ , 2017.
  * Vilalta and Drissi (2002) Ricardo Vilalta and Youssef Drissi.  A perspective view and survey of meta-learning.  _Artificial Intelligence Review_ , 18(2):77–95, 2002.
  * Vinyals et al. (2016) Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.  Matching networks for one shot learning.  In _NIPS_ , 2016.
  * Wah et al. (2011) C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The Caltech-UCSD Birds-200-2011 Dataset.  Technical Report CNS-TR-2011-001, California Institute of Technology, 2011\.
  * Wang et al. (2016) Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick.  Learning to reinforcement learn.  _arXiv preprint arXiv:1611.05763_ , 2016.
  * Wang (2018) Yu-Xiong Wang.  _Learning to Learn for Small Sample Visual Recognition_.  PhD thesis, Carnegie Mellon University, 2018.
  * Wang and Hebert (2016) Yu-Xiong Wang and Martial Hebert.  Learning to learn: Model regression networks for easy small sample learning.  In _ECCV_ , 2016.
  * Wang et al. (2017) Yu-Xiong Wang, Deva Ramanan, and Martial Hebert.  Learning to model the tail.  In _NIPS_ , 2017.
  * Wang et al. (2018) Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan.  Low-shot learning from imaginary data.  In _CVPR_ , 2018.
  * Weinberger and Saul (2009) Kilian Q Weinberger and Lawrence K Saul.  Distance metric learning for large margin nearest neighbor classification.  _JMLR_ , 10(Feb):207–244, 2009.
  * Wichrowska et al. (2017) Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein.  Learned optimizers that scale and generalize.  In _ICML_ , 2017.
  * Ye et al. (2018) Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha.  Learning embedding adaptation for few-shot learning.  _arXiv preprint arXiv:1812.03664_ , 2018.
  * Ying et al. (2018) Wei Ying, Yu Zhang, Junzhou Huang, and Qiang Yang.  Transfer learning via learning to transfer.  In _ICML_ , 2018.
  * Yosinski et al. (2014) Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson.  How transferable are features in deep neural networks?  In _NIPS_ , 2014.
  * Yu et al. (2018) Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine.  One-shot imitation from observing humans via domain-adaptive meta-learning.  In _RSS_ , 2018.
  * Zhang et al. (2006) Hao Zhang, Alexander C Berg, Michael Maire, and Jitendra Malik.  Svm-knn: Discriminative nearest neighbor classification for visual category recognition.  In _CVPR_ , 2006.
  * Zhang et al. (2018a) Ruixiang Zhang, Tong Che, Zoubin Ghahramani, Yoshua Bengio, and Yangqiu Song.  Metagan: An adversarial approach to few-shot learning.  In _NeurIPS_ , 2018a.
  * Zhang et al. (2018b) Yu Zhang, Ying Wei, and Qiang Yang.  Learning to multitask.  In _NeurIPS_ , 2018b.
  * Zhou (2012) Zhi-Hua Zhou.  _Ensemble methods: foundations and algorithms_.  Chapman and Hall/CRC, 2012.
  * Zhu et al. (2014) Xiangxin Zhu, Dragomir Anguelov, and Deva Ramanan.  Capturing long-tail distributions of object subcategories.  In _CVPR_ , 2014.

Supplementary Material

We provide details omitted in the main text.

  * •

Appendix A: More examples on applying meta-learning (subsection 2.4 of the
main paper).

  * •

Appendix B: Details of experimental setups (section 5 of the main paper).

  * •

Appendix C: Additional experimental results (subsection 5.1, subsection 5.2
and subsection 5.3 of the main paper).

##  Appendix A More Examples on Applying Meta-Learning

Our unifying view clearly indicates the components of meta-learning.

  * •

Meta labeled example $(D_{\text{tr}},h^{*})$ and meta loss $l_{\text{meta}}$

  * •

Meta hypothesis set $\mathcal{G}=\\{g\\}$ and meta algorithm $B_{\mathcal{G}}$

Applying meta-learning to an area requires defining or designing them
accordingly. We discuss more examples in the following subsections.

###  A.1 Active learning

The goal is to minimize the data labeling effort by querying $N^{\prime}$
informative examples. For classification,
$D_{\text{tr}}=\\{x_{n}\\}_{n=1}^{N}$ is an unlabeled set, $h^{*}$ is the
model learned with the best $N^{\prime}$ queried examples from
$D_{\text{tr}}$, and $l_{\text{meta}}$ measures the performance gap on
classification. $g$ is a learning process constrained to query labels for
$N^{\prime}$ examples by investigating $x$ and the learning progress (Ravi and
Larochelle, 2018; Sharma et al., 2018; Bachman et al., 2017).
$B_{\mathcal{G}}$ can be ERM by SGD or by reinforcement learning algorithms
(Ravi and Larochelle, 2018; Sharma et al., 2018). An illustration of this
procedure can be found in Figure 4.

###  A.2 Optimization

One focus is to search a better updating rule in iterative optimization. In
this case, $D_{\text{tr}}$ is an objective function $L_{S}(\theta)$
parameterized by $\theta$; e.g., the training loss. $h^{*}$ is the optimal
solution of $L_{S}$ or of a related objective $L_{V}$; e.g., the validation
error. $l_{\text{meta}}$ is the gap of objective values. $g$ is an iterative
optimization algorithm with a (meta) learnable updating rule.
$B_{\mathcal{G}}$ can be EMR by SGD (Andrychowicz et al., 2016; Wichrowska et
al., 2017) or reinforcement learning algorithms (Li and Malik, 2017; Bello et
al., 2017).

###  A.3 Imitation learning and reinforcement learning

Imitation learning (Stadie et al., 2018; Frans et al., 2018; Wang et al.,
2016; Duan et al., 2016, 2017; Finn et al., 2017b; Yu et al., 2018) easily
fits into our meta-learning framework, given its similarity to the supervised
learning. For few-shot reinforcement learning (Duan et al., 2017; Wang et al.,
2016), the $D_{\text{tr}}$ is a trial of a few episodes from an MDP and
$h^{*}$ is the target policy, which can be realized by maximizing the reward.

###  A.4 Few-shot learning

The line of learning generative models for few-shot learning (Zhang et al.,
2018a; Wang et al., 2018) can conceptually be thought of as learning a meta
mapping from a (few-shot) training set to a target data generator (or
distribution) that well describes many-shot examples.

Figure 4: Illustration of active learning. A labeled image is surrounded by a
colored box that indicate its label. The target model in active learning is
the classifier trained on the optimally queried images.

##  Appendix B Details of Experimental Setups

#### Datasets.

We synthesize the “Heterogeneous” dataset from five fine-grained
classification datasets, namely AirCraft (Maji et al., 2013), Car-196 (Krause
et al., 2013), Caltech-UCSD Birds (CUB) 200-2011 (Wah et al., 2011), Stanford
Dog (Khosla et al., 2011), and Indoor (Quattoni and Torralba, 2009). We
randomly sampled 60 classes with 50 images each from each of the 5 datasets,
and equally split classes into meta-train-pool, meta-val-pool, and meta-test-
pool. That is, there are 100 classes in meta-train-pool (same for the others),
which includes 20 classes from each dataset.

We also use CUB (Wah et al., 2011) alone to evaluate the few-shot learning
performance. CUB contains a total of 11,788 images of birds over 200 species.
The similarity between classes makes it a difficult classification task when
given limited training examples. Following the setups of (Triantafillou et
al., 2017; Chen et al., 2019) we use the provided bounding box of CUB to crop
the center object of each image, and split the 200 classes into 100/50/50 for
meta-train/val/test-pool.

(a) MAML

(b) ProtoNet

(c) MatchNet

Figure 5: 1-shot 5-way classification accuracy on MiniImageNet (SS). We show
meta-training (red) and meta-test (blue) accuracy. We keep all instances per
class but vary the numbers of meta-training classes. MAML, ProtoNet, and
MatchNet suffer over-fitting when the number of classes gets small.

Besides MiniImageNet and CUB, we further evaluate on the large-scale
tierdImageNet data set (Ren et al., 2018), which has 779,165 images and 608
classes. We use the standard split: 351/97/160 for meta-train/val/test-pool.

Following the conventional way of image pre-processing (Vinyals et al., 2016;
Snell et al., 2017; Finn et al., 2017a), we resize all images to $84\times
84$.

#### Meta examples (tasks) and evaluation protocols.

We follow (Rusu et al., 2019; Ye et al., 2018) to evaluate the few-shot
classification by drawing 10,000 tasks from meta-test-pool and there are 15
validation images per class in a task. That is, a meta labeled example
$(D_{\text{tr}},D_{\text{val}})$ has 15 images per class in $D_{\text{val}}$.
We found the 95% confidence interval to be consistently within $[0.001,0.004]$
and thus omit it for brevity.

#### Baseline methods.

We investigate three popular baselines, namely Model Agnostic Meta-Learning
(MAML) (Finn et al., 2017a), Prototypical Network (ProtoNet) (Snell et al.,
2017), and Matching Network (MatchNet) (Vinyals et al., 2016). MAML implements
an inner optimizer to update the meta-learned classifier initializer, and
ProtoNet/MatchNet learn discriminative embedding for few-shot classification.
ProtoNet uses the distance-based nearest class mean rule for prediction, while
MatchNet utilizes the similarity-based nearest neighbor rule.

We re-implement all three algorithms and use the same $C$-way setting in meta-
training and meta-test for consistency. That is, we disregard the trick (Snell
et al., 2017) that trains with $30$-way tasks but tests with $5$-way tasks. We
apply the first-order MAML and tune the number of updates by meta-validation.
We apply the standard 4-layer ConvNet as the backbone (Vinyals et al., 2016;
Snell et al., 2017). In each layer, convolution, batch normalization, ReLU,
and max-pooling are concatenated sequentially. During meta-training SGD with
Adam (Kingma and Ba, 2014) is employed, with a initial learning rate 2e-3. Our
baseline results are close to those reported in the recent overview of few-
shot learning (Chen et al., 2019).

Figure 6: 1-shot 5-way classification performance of ProtoNet on
tierdImageNet: (Left) varying the instances per meta-train class; (Right)
varying the number of meta-train classes. By showing meta-training (red) and
meta-test (blue) accuracy, we find ProtoNet suffers over-fitting when the
number of classes or the number of instances in meta-train classes get small.

##  Appendix C Additional Experimental Results

In the following we include more experimental results for the meta-learning
analyses.

###  C.1 Empirical generalization analysis

Following our supervised view of meta-learning, the generalization ability of
the learned meta model should be affected by the “effective” number of meta
training examples. Here we investigate such effective number from another
perspective, i.e., the number of classes in the in the meta-train-pool. In
detail, we keep all 600 instances per class in MiniImageNet (SS), but change
the number of available classes for meta-training in the meta-train-pool, from
8 to 64. When there are limited types of classes in the meta-train-pool, the
number of diversified tasks to be sampled is constrained. Figure 5 shows the
change of few-shot accuracy with different meta-train-pools. The trend follows
the supervised learning intuition as well: where there are more available
meta-training classes (types of few-shot tasks), the better generalization
results are achieved. In particular, when there are 8 classes, all models,
namely MAML, ProtoNet, and MatchNet overfit. The 1-shot classification
performance becomes stable when the number of meta-training classes is larger
than 48.

We also conduct the generalization analysis of ProtoNet on the tierdImageNet
data set, via varying both the number of instances per meta-train class and
the number of meta-train classes. Figure 6 depicts the results, whose trends
are similar to Figure 5.

Table 6: Bagging for 1/5-shot 5-way classification on MiniImageNet (SS). “w/o Pre-Train” and “w/ Pre-Train” denote the vanilla training of the meta-learning model on all classes in the meta-train-pool without or with the pre-training strategy. Single: vanilla training of the meta-learning model on all classes in the meta-train-pool. “Bag1” and “Bag2” ensemble the basic models by logits and probabilities respectively. The best result of each shot-algorithm pair is in bold font. | MAML | ProtoNet | MatchNet
---|---|---|---
| Single | Bag1 | Bag2 | Single | Bag1 | Bag2 | Single | Bag1 | Bag2
1-shot | w/o Pre-Train | 0.459 | 0.448 | 0.482 | 0.461 | 0.488 | 0.491 | 0.463 | 0.481 | 0.482
w/ Pre-Train | 0.478 | 0.498 | 0.498 | 0.500 | 0.522 | 0.526 | 0.500 | 0.524 | 0.525
5-shot | w/o Pre-Train | 0.633 | 0.665 | 0.666 | 0.658 | 0.692 | 0.694 | 0.639 | 0.663 | 0.664
w/ Pre-Train | 0.660 | 0.694 | 0.692 | 0.671 | 0.701 | 0.701 | 0.642 | 0.674 | 0.673
Table 7: The 1-shot and 5-shot classification performance of ProtoNet on the CUB data set. “w/o Pre-Train” and “w/ Pre-Train” denote the vanilla training of the meta-learning model on all classes in the meta-train-pool without or with the pre-training strategy. Average: the averaged few-shot classification results of all basic models. “Bag1” and “Bag2” ensemble the basic models with pre-training by logits and probabilities respectively. | w/o Pre-Train | w/ Pre-Train | Average | Bag1 | Bag2
---|---|---|---|---|---
1-Shot | 0.592 | 0.601 | 0.586 | 0.621 | 0.623
5-Shot | 0.762 | 0.777 | 0.758 | 0.793 | 0.793
Table 8: Bagging for 1-shot 5-way classification on MiniImageNet (SS) using RN (Sung et al., 2018) and FEAT (Ye et al., 2018). Single: no bagging. Average: average accuracy of basic meta models. Since FEAT already applies pre-training, we do not show its results without pre-training. The best result of each shot-algorithm pair is in bold font. | RN | FEAT
---|---|---
| Single | Average | Bagging | Single | Average | Bagging
w/o Pre-Train | 0.483 | 0.477 | 0.510 | - | - | -
w/ Pre-Train | 0.504 | 0.498 | 0.529 | 0.552 | 0.545 | 0.559
Table 9: 1-shot 5-way classification of ProtoNet on tierdImageNet. “w/o Pre-Train” and “w/ Pre-Train” denote the vanilla training of the meta-learning model on all classes in the meta-train-pool without or with the pre-training strategy. “Bagging” ensembles the basic models with pre-training by probabilities. | Single | Bagging
---|---|---
w/o Pre-Train | 0.487 | 0.504
w/ Pre-Train | 0.542 | 0.552
Table 10: Bagging for 1-shot 5-way classification on MiniImageNet (SS) using ProtoNet with ResNet-12 as the backbone. Single: no bagging. Average: average accuracy of basic meta models. | Single | Average | Bagging
---|---|---|---
w/ Pre-Train | 0.614 | 0.603 | 0.626

###  C.2 Supervised learning techniques for meta-learning

In this subsection, we first show different strategies to ensemble the basic
meta-learning learner through bagging, and then demonstrate the applicability
of bagging to the CUB and tieredImageNet data set. Finally, We experiment with
more meta-learning algorithms (Sung et al., 2018; Ye et al., 2018) and
backbones (He et al., 2016).

We construct bagging models by sub-sampling classes, and build a meta-learning
model on the subsets. The differences in various subsets result in the
divergence among basic models, which is the key to the success of the ensemble
methods. To aggregate more than one meta-learning models, we consider
averaging their prediction logits (the distance values for ProtoNet and
MatchNet) and the normalized probabilities. We denote the two strategies as
“Bag1” and “Bag2”, respectively. The results are listed in Table 6, where
bagging show consistent improvements among the vanilla meta-learning models.

To demonstrate its applicability, we also evaluate bagging on another meta-
learning benchmark CUB. The results of ProtoNet is summarized in Table 7:
there exists a consistent trend with the results on the MiniImageNet (SS),
i.e., models with pre-training work better, and bagging can further improve
the performance.

We further evaluate the 1-shot 5-way classification performance of ProtoNet w/
and w/o pre-training and bagging on tieredImageNet. Table 9 shows the results,
which indicates that both techniques are compatible to boost the accuracy.

Finally, we experiment with FEAT (Ye et al., 2018) and Relational Network (RN)
(Sung et al., 2018) for 1-shot 5-way classification on MiniImageNet (SS). We
use and adapt their officially released code. We use the standard 4-layer
ConvNet as the backbone for a fair comparison. Table 8 shows the results: both
pre-training and bagging improve accuracy. We note that we cannot fully
reproduce RN’s reported accuracy of training from scratch (i.e., $0.504$, vs.
ours at $0.483$) without detailed hyper-parameter configurations, but with
bagging, we can achieve higher accuracy.

We also experiment with ProtoNet using ResNet-12 (He et al., 2016; Oreshkin et
al., 2018) as the backbone. We perform pre-training. Table 10 shows the
results: bagging again leads to improvement. Both results also outperform
those reported in (Oreshkin et al., 2018) using ResNet-12.

###  C.3 Data augmentation

Table 11: Meta-test accuracy of 1-shot 20/30/40-way tasks on MiniImageNet (CS). We compare no augmentation (i.e., 1) and augmentation with 8 subcategories by K-means. Since there are only 30 classes in the original meta-train-pool, we cannot perform meta-training with 40-way tasks. | Meta-Training Tasks
---|---
| 20 | 30 | 40
| 1 / 8 | 1 / 8 | 1 / 8
Meta- | 20 | 0.192 / 0.207 | 0.189 / 0.208 |  \- / 0.204
Test | 30 | 0.148 / 0.161 | 0.146 / 0.161 |  \- / 0.159
Tasks | 40 | 0.122 / 0.134 | 0.120 / 0.134 |  \- / 0.132
Table 12: Meta-test accuracy of 5-shot 20/30/40-way tasks on MiniImageNet (CS). We compare no augmentation (i.e., 1) and augmentation with 8 subcategories by K-means. Since there are only 30 classes in the original meta-train-pool, we cannot perform meta-training with 40-way tasks. | Meta-Training Tasks
---|---
| 20 | 30 | 40
| 1 / 8 | 1 / 8 | 1 / 8
Meta- | 20 | 0.318 / 0.346 | 0.323 / 0.347 |  \- / 0.357
Test | 30 | 0.257 / 0.283 | 0.262 / 0.285 |  \- / 0.294
Tasks | 40 | 0.220 / 0.244 | 0.224 / 0.245 |  \- / 0.254

We perform K-means within each class to split a class into $K$ subcategories,
resulting in $30\times K$ augmented classes in meta-train-pool (and meta-val-
pool). To ensure the quality of K-means, we pre-train a 30-way classifier on
the whole meta-train-pool and use the features for K-means. We run 30 trials
of K-means to compensate its randomness. We construct a meta-training example
(task) by first picking a trial and sampling 30 classes from the corresponding
augmented meta-train-pool.

Here we provide additional results on MiniImageNet (CS). We further evaluate
on 20-way and 40-way tasks during meta-test and also report 5-shot
classification results. We note that since ProtoNet (Snell et al., 2017)
learns a feature extractor $\psi$ (cf. Equation 9 of the main text) during
meta-training, it can be applied to a task of different ways (e.g., meta-
training with 20-way tasks but meta-test with 40-way tasks).

Table 11 and Table 12 summarize the results. We consider no augmentation or
augmentation with 8 subcategories by K-means and report different combinations
of meta-training and meta-test tasks. We can clearly see the trend (within
each column): data augmentation consistently improves the performance by
generating more diverse meta-training examples.

Moreover, by comparing within each row where the meta-training tasks are
different, we see no clear performance drop when the meta-training and meta-
test tasks are different. Interestingly, for 5-shot tasks (cf. Table 12) meta-
training with more ways seem to be always beneficial, which aligns with the
observation in (Snell et al., 2017). A more detailed investigation is thus
desirable in the future work.

###  C.4 Non-parametric methods

We introduce meta-KNN as a widely-applicable framework to apply non-parametric
methods to meta-learning. Here we provide the hyper-parameters we use. We set
$K=100$, $\beta=1$, and $\alpha=0.0002$ (cf. algorithm 1 of the main text).
$\alpha=0.0002$ is the learning rate when the initial ProtoNet $\hat{g}$
converges.

For a test task, meta-KNN takes 5 ms for searching KNN tasks and 360 ms for
fine-tuning (using one RTX 2080 GPU), which is quite efficient. We note that
meta-KNN (algorithm 1) can easily incorporate various kinds of metrics for KNN
search, and we applied the simple average features with Euclidean distance as
an illustrative example. We expect that applying a more sophisticated and
application-dependent metric (e.g., using higher-order statistics (Achille et
al., 2019b) or Task2Vec (Achille et al., 2019a)) to characterize task
similarity would further improve the performance.

Table 13: Few-shot 5-way domain generalization on Office-Home with two
domains: Clipart (C), Product (P).

| Meta-train | Meta-test | Test Acc
---|---|---|---
Case | Source | Target | Source | Target | 1-Shot | 5-Shot
I-1 | C | C | C | C | 0.341 | 0.477
I-2 | P | C | P | C | 0.296 | 0.350
I-3 | C | C | P | C | 0.275 | 0.342
I-4 | P | P | P | C | 0.264 | 0.283

| Meta-train | Meta-test | Test Acc
---|---|---|---
Case | Source | Target | Source | Target | 1-Shot | 5-Shot
II-1 | P | P | P | P | 0.448 | 0.609
II-2 | C | P | C | P | 0.291 | 0.381
II-3 | C | C | C | P | 0.284 | 0.338
II-4 | P | P | C | P | 0.275 | 0.329

###  C.5 Domain generalization and meta Domain shifts

This section shows more experimental results for domain generalization and
meta domain shifts. We experiment on the Off-Home dataset, which is designed
for domain adaptation. We focus on two domains: Clipart (C) and Product (P).
There are 65 classes and we split them into 25 for meta-train-pool, 15 for
meta-val-pool, and 25 for meta-test-pool. We work on _few-shot 5-way_ tasks in
which a task contains a training set $D_{\text{tr}}$ from one (source) domain
and a validation set $D_{\text{val}}$ from the other (target) domain.

Table 13 summarizes the results. We clearly see the difficulty when the source
and target domains within a task are different by comparing Case I-1 and I-2
(similarly, II-1 and II-2). We note that in each of these cases the
distribution of the meta-training and meta-test examples are the same.
Therefore, there is no meta-domain shift and the gap simply indicates the
difficulty of tasks.

We further investigate meta-domain shifts by constructing meta-training and
meta-test examples in a different way (Case I-3, I-4 and II-3, II-4). We note
that some meta-learning algorithms, by default, cannot learn to handle the
case where $D_{\text{tr}}$ and $D_{\text{val}}$ are sampled from different
domains (e.g., (Nichol et al., 2018)). These additional experiments are meant
to simulate the results by those algorithms. The performance in these cases is
outperformed by Case I-2 and II-2 respectively where each group considers the
same meta-test examples. We argue that this gap indeed results from meta-
domain shifts (cf. subsection 2.3 in the main paper) and might be resolved via
domain adaptation by casting meta-learning as supervised learning.

Table 14: 20-shot 5-way domain generalization on Office-Home with two domains: Clipart (C) and Product (P) domains. | Meta-train | Meta-test | Test Acc
---|---|---|---
Case | Source | Target | Source | Target | 20-Shot
I-1 | C | C | C | C | 0.554
I-2 | P | C | P | C | 0.358
I-3 | C | C | P | C | 0.349
I-4 | P | P | P | C | 0.312
Table 15: 20-shot 5-way domain generalization (the I-2 case) with pre-training and bagging. | Single | Bagging
---|---|---
w/o Pre-Train | 0.358 | 0.383
w/ Pre-Train | 0.377 | 0.392

We further extend our studies of domain generalization beyond the few-shot
learning setting. For a domain generalization task, a base model could be
trained on labeled instances from one (source) domain, not necessarily few-
shot, and tested on instances from another (target) domain. Thus, besides the
1-shot and 5-shot settings as shown before, we examine a many-shot setting
(e.g., 20-shot) with ProtoNet on a domain generalization task.

Table 14 shows the results: the gap between I-2 and I-3 (I-4) reflects the
meta-domain difference between meta-train and meta-test tasks (irrespective of
1- or 20-shot), similar to the problem of domain adaptation in supervised
learning. For I-2 (the source and target domains are Product and Clipart
respectively in both meta-train and meta-test) we further apply pre-training
and bagging. As shown in Table 15, both techniques are applicable to the
domain generalization tasks.