# MAML is a Noisy Contrastive Learner

Chia-Hsiang Kao† Wei-Chen Chiu† Pin-Yu Chen‡
†National Yang Ming Chiao Tung University, Taiwan ‡IBM Research
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Model-agnostic meta-learning (MAML) is one of the most popular and widely-
adopted meta-learning algorithms nowadays, which achieves remarkable success
in various learning problems. Yet, with the unique design of nested inner-loop
and outer-loop updates which respectively govern the task-specific and meta-
model-centric learning, the underlying learning objective of MAML still
remains implicit and thus impedes a more straightforward understanding of it.
In this paper, we provide a new perspective to the working mechanism of MAML
and discover that: MAML is analogous to a meta-learner using a supervised
contrastive objective function, where the query features are pulled towards
the support features of the same class and against those of different classes,
in which such contrastiveness is experimentally verified via an analysis based
on the cosine similarity. Moreover, our analysis reveals that the vanilla MAML
algorithm has an undesirable interference term originating from the random
initialization and the cross-task interaction. We therefore propose a simple
but effective technique, zeroing trick, to alleviate such interference, where
the extensive experiments are then conducted on both miniImagenet and Omniglot
datasets to demonstrate the consistent improvement brought by our proposed
technique thus well validating its effectiveness.

##  1 Introduction

Human can learn from very few samples. They can readily establish their
cognition and understanding to novel tasks, environments, or domains even with
very limited experience in the corresponding circumstances. Meta-learning, a
subfield of machine learning, aims at equipping machines with such capacity
for being able to effectively accommodate to the new scenarios [1, 2]. The
training scheme of meta-learning differs from conventional learning in that:
meta-learning explores effective ways to search for the appropriate inductive
bias [1]. In other words, task-agnostic information is extracted from a
distribution of tasks and is used to improve the learning of unseen tasks from
the same distribution [3].

One successful and highly influential meta-learning algorithm is the Model
Agnostic Meta-Learning (MAML) algorithm [4], which has inspired numerous
follow-up works and extensions [5, 6, 7, 8, 9, 10]. The MAML algorithm
estimates an initial set of model parameters such that a fast adaption of the
model to a new task would only require several runs of update on the
parameters. We review the algorithmic procedure of MAML firstly here with
taking the few-shot classification as an example. A typical few-shot
classification problem refers to classifying samples from some classes (i.e.
query data) after seeing a few examples per class (i.e. support data). When it
comes to the meta-learning scenario, we consider that there exists various
tasks, where each task itself is a few-shot classification problem and
different tasks would have different target classes. MAML aims to meta-learn
the base-model based on various training tasks (i.e. the meta-training
dataset) and evaluate the performance of beta-model on the tasks sampled from
a held-out unseen dataset (i.e. the meta-testing dataset), where the meta-
training procedure of MAML is summarized in Algorithm 1 (with detailed
definition of variables provided later in Section 2). Basically, the meta-
training of MAML follows a bi-level optimization scheme composed of the inner
loop and the outer loop. In the inner loop (also known as fast adaption), the
base-model $\theta$ is updated to ${\theta}^{\prime}$ by using the support set
$S$ for several steps. In the outer loop, a loss is evaluated on the updated
base-model ${\theta}^{\prime}$ using the query set $Q$ and its gradient is
computed with respect to $\theta$ for updating the base-model $\theta$ prior
to the inner loop. Since the outer loop requires the computation of gradient
of gradient (as the update in the inner loop is included in the entire
computation graph), it is also called the second-order MAML (SOMAML)
algorithm. It is also worth noting that, in order to prevent the costly
computation of the Hessian matrix, [4] also proposes a first-order MAML
algorithm (FOMAML) that uses the gradient computed with respect to the inner-
loop-updated parameters ${\theta}^{\prime}$.

Algorithm 1 Second-order MAML algorithm

Require: Distribution over tasks $D$
Require: $\eta,\rho$: inner loop and outer loop learning rates
Require: Randomly initialized base-model parameters $\theta$

1:while not done do

2: Sample tasks $\\{T_{1},\dots T_{N_{batch}}\\}$ from $D$

3: for i$=1,2,\ldots,N_{batch}$ do

4: $S_{i}\leftarrow$ sample $N_{shot}$ data from $T_{i}$

5: $Q_{i}\leftarrow$ sample $N_{query}$ data from $T_{i}$

6: Update the base-model with gradient descent:
$\theta^{\prime}_{i}=\theta-\eta\nabla_{\theta}L_{\theta,Si}$

7: end for

8: Update
$\theta\leftarrow\theta-\rho\sum^{N_{batch}}_{i=1}\nabla_{\theta}L_{\theta^{\prime}_{i},Qi}$

9:end while

The widely accepted intuition behind MAML is that the models are encouraged to
learn a general-purpose representations which are broadly applicable not only
to the seen tasks but also to novel tasks [4, 11, 12]. [11] confirms this
perspective by showing that during fast adaptation, the majority of changes
made are in the final linear layers, while the previous convolution layers (as
the feature extractor) remain almost static. This implies that the models
trained with MAML learn a good feature representation and that they only have
to change the linear mapping from features to outputs during the fast
adaption. Similar ideas as to freezing the feature extractor during the inner
loop have also been explored in [13, 14, 15]. On the other hand, this
intuition has been held as an assumption in theoretical works such as [16, 17,
18] that use representation theory to provide the risk bounds, yet the
validity of the assumption is not provided [19].

While this intuition sounds satisfactory, we take a step further and ask the
following fundamental questions to demystify MAML: (1) In what sense does MAML
guide any model to learn general-purpose representations? And how do the inner
loop and outer loop in the training mechanism of MAML collaboratively prompt
to achieve so? (2) What is the role of the support and query data and how
they, if any, interact with each other? In this paper, we answer these
questions and give new insights on the working mechanism of MAML, which turns
out to be closely connected to the supervised contrastiveness111We use the
term supervised contrastiveness to refer to the setting of using groundtruth
label information to differentiate positive samples and negative samples [20].
This setting is different from (unsupervised/self-supervised) contrastive
learning..

Figure 1: A step-by-step illustration showing the supervised contrastivenes
objective underlying the MAML algorithm. Consider a model trained with MAML
with one step inner-loop update under the 5-way 1-shot few-shot learning
setting. Assume that the final linear layer $\mathbf{w^{0}}$ to be zero, then
the inner loop gradients stamps the $i^{th}$ column of $\mathbf{w^{0}}$ with
the support features of class $i$. In the outer loop, the output of a query
sample is the inner product of $\phi(x_{i})$ and $\mathbf{w^{1}}$, which is
essentially the inner product of the query features and all the supports
features. Thus, MAML displays the characteristic of supervised
contrastiveness, where the support set acts as the positive and negative
samples.

Here we provide a sketch of our analysis in Figure 1 under a 5-way 1-shot
paradigm of few-shot learning, where we take the mean square error (MSE) with
respect to the one-hot encoding of groundtruth label as the objective
function, and the MAML is simplified to have a single inner-loop update. When
the final linear layer $\mathbf{w^{0}}$ is set to be zero at the beginning of
an inner loop, the weight of the linear layer is inscribed with the support
features in the inner loop update. In the outer loop, the output of a query
sample $q_{i}$ is actually the inner product between the query feature
$\phi(q_{i})$ and all the support features (the learning rate is omitted for
now). As the groundtruth is the one-hot vector, the encoder is trained to
either minimize the inner product between the query features and the support
features when they are from different classes (as shown in the green box), or
to pull the inner product between the query features and the support features
towards $1$ when they are of the same class (as shown in the red box). Under
this derivation, the inner loop and the outer loop together manifest a
supervised contrastive objective where the support samples work as positive
and negative samples. Particularly, as the vanilla implementation of MAML uses
non-zero (random) initialization for the linear layer, we will show later that
such initialization leads to a noisy contrastive learning objective which
would impede the training.

In this paper, we present a more general case of MAML with cross entropy loss
and unveil the underlying learning protocol of the vanilla MAML algorithm as
an interfered supervised contrastive learning (cf. Section 2). We then
experimentally verify the supervised contrastiveness of MAML, and proposed to
mitigate the interference with our simple but effective technique of the zero-
initialization and zeroing trick (cf. Section 3). In summary, our main
contributions are three-fold:

  * •

We show that MAML is implicitly a noisy supervised contrastive learning
algorithm where the noise comes from the random initialization of the final
linear layer and the cross-task interaction, in which such noise can
compromise the capability of the feature extractor.

  * •

We provide an analysis based on the cosine similarity to verify the inherent
contrastiveness of the MAML algorithm.

  * •

We experimentally validate our analysis and show that applying the zeroing
trick induces a notable improvement in testing accuracy during training. We
also show that during meta-testing, a pronounced increase in accuracy occurs
when the zeroing trick is applied.

##  2 Why MAML is Implicitly a Noisy Supervised Contrastive Algorithm?

###  2.1 Problem Setup

We provide here the detailed derivation for showing that the MAML is implicit
a noisy supervised contrastive algorithm, where we again adopt the few-shot
classification as the example application. Consider a task distribution $D$
from which a batch of tasks $\\{T_{1},\dots,T_{N_{batch}}\\}$ is drawn at
every iteration. For each task $T_{n}$, we have a support set $S_{n}$ and a
query set $Q_{n}$ where $S_{n}=\\{(s_{m},t_{m})\\}_{m=1}^{N_{way}*N_{shot}}$,
$Q_{n}=\\{(q_{m},u_{m})\\}_{m=1}^{N_{way}*N_{query}}$,
$s_{m},q_{m}\in\mathbf{R}^{N_{in}}$ are data samples with
$t_{m},u_{m}\in\mathbf{R}$ their corresponding labels. We denote $N_{way}$ the
number of classes in each task, and $\\{N_{shot},N_{query}\\}$ respectively
the number of support and query samples per class. To build up the
architecture of our base-model, we consider a (convolution) neural network
$\phi:\mathbf{R}^{N_{in}}\rightarrow\mathbf{R}^{N_{f}}$ (as the feature
extractor/encoder) followed by a fully connected linear head
$\mathbf{w}\in\mathbf{R}^{N_{f}\times N_{way}}$, where $N_{f}$ is the
dimension of the feature space. We denote the $k^{th}$ column of $\mathbf{w}$
as $\mathbf{w_{k}}$. Note that the base-model parameters $\theta$ consist of
$\varphi$ and $\mathbf{w}$, where $\varphi$ denotes the parameters of encoder
$\phi$.

As shown in Algorithm 1, the MAML algorithm adopts a training strategy
comprising the inner loop and the outer loop. At the beginning of each
training iteration, we sample $N_{batch}$ tasks. In the inner loop, the
initial parameters $\\{\phi^{0},\mathbf{w^{0}}\\}$ are updated to
$\\{\phi^{i},\mathbf{w^{i}}\\}$ for $i$ steps to minimize the multi-class
cross entropy loss evaluated on the support dataset $S_{n}$ as

$\begin{split}L_{\\{\phi^{0},\mathbf{w^{0}}\\},S_{n}}&=\mathop{\mathbf{E}}_{(s,t)\sim
S_{n}}\sum_{j=1}^{N_{way}}1_{j=t}[-\log\frac{\exp(\phi^{0}(s)^{\top}\mathbf{w_{j}^{0}})}{\sum_{k=1}^{N_{way}}\exp(\phi^{0}(s)^{\top}\mathbf{w_{k}^{0}})}]\\\
&=\mathop{\mathbf{E}}_{(s,t)\sim
S_{n}}[-\phi^{0}(s)^{\top}\mathbf{w_{t}^{0}}]+\mathop{\mathbf{E}}_{(s,t)\sim
S_{n}}[\log\sum_{k=1}^{N_{way}}\exp(\phi^{0}(s)^{\top}\mathbf{w_{k}^{0}})]\end{split}$
(1)

At the end of $i$ inner loop update, the cross-entropy loss with respect to
the query data $Q_{n}$ becomes

$\begin{split}L_{\\{\phi^{i},\mathbf{w^{i}}\\},Q_{n}}&=\mathop{\mathbf{E}}_{(q,u)\sim
Q_{n}}[-\log\frac{\exp(\phi^{i}(q)^{\top}\mathbf{w_{u}^{i}})}{\sum_{k=1}^{N_{way}}\exp(\phi^{i}(q)^{\top}\mathbf{w_{k}^{i}})}]\\\
&=\mathop{\mathbf{E}}_{(q,u)\sim
Q_{n}}[-\phi^{i}(q)^{\top}\mathbf{w_{u}^{i}}]+\mathop{\mathbf{E}}_{(q,u)\sim
Q_{n}}[\log\sum_{k=1}^{N_{way}}\exp(\phi^{i}(q)^{\top}\mathbf{w_{k}^{i}})]\end{split}$
(2)

The inner loop is run for all sampled tasks. In the outer loop, the gradient
of the loss (cf. Eq. 2 collected over all the tasks) is used to update the
model’s initial parameters $\\{\phi^{0},\mathbf{w^{0}}\\}$. For simplicity,
the predicted probability (softmax) output
$\frac{\exp(\phi(s)^{\top}\mathbf{w_{k}})}{\sum_{j=1}^{N_{way}}\exp(\phi(s)^{\top}\mathbf{w_{j}})}$
of data $s$ with respect to the $k^{th}$ channel of the linear layer
$\mathbf{w}$ is denoted as $g_{s,k,\mathbf{w}}$.

In the following, we aim to show the supervised contrastiveness entailed in
MAM L’s objective function. We assume that the encoder $\phi$ is frozen during
the inner loop. Without loss of generality, we consider a base-model trained
with the original MAML algorithm with $N_{batch}=1$ and one update step in the
inner loop. We start with the inner loop update in Section 2.2 and visit the
outer loop update of the linear layer in Section 2.3. We then present the
outer loop update of encoder in Section 2.4. Finally, Section 2.5 shows the
analysis of the general setting.

###  2.2 Inner Loop of Linear Layer

In the inner loop, the final linear layer $\mathbf{w^{0}}$ is updated to
$\mathbf{w^{1}}$ with a learning rate $\eta$ as in Eq. 3, where we have the
following observations: As the columns of the linear layer are inscribed with
the weighted sum of the features extracted from support samples (i.e. support
features), the second term on the right-hand size of the last equality
indicates that $\mathbf{w^{1}_{k}}$ is pushed towards the direction derived
from the support features of the same class (i.e., class $k$), while being
pulled away from the direction built upon the support features of different
classes. The strength of pushing or pulling is related to the prediction value
$g_{s,k,\mathbf{w^{0}}}$. Therefore, such update of the final linear layer is
analogous to the supervised contrastiveness where the supervision comes from
the class information. More interestingly, if the encoder $\phi$ is frozen
throughout the inner loop update (i.e., almost no inner loop for updating
convolution layers [11]), then the weights of the linear layer are
continuously accumulating the information of support features.

$\begin{split}\mathbf{w^{1}_{k}}&=\mathbf{w^{0}_{k}}-\eta\frac{\partial
L_{\\{\phi,\mathbf{w^{0}}\\},S}}{\partial\mathbf{w^{0}_{k}}}=\mathbf{w^{0}_{k}}+\eta\mathop{\mathbf{E}}_{(s,t)\sim
S}(1_{k=t}-g_{s,k,\mathbf{w^{0}}})\phi(s)\\\
&=\mathbf{w^{0}_{k}}+\eta\mathop{\mathbf{E}}_{\begin{subarray}{c}(s,t)\sim
S\\\
t=k\end{subarray}}(1-g_{s,k,\mathbf{w^{0}}})\phi(s)+\eta\mathop{\mathbf{E}}_{\begin{subarray}{c}(s,t)\sim
S\\\ t\neq k\end{subarray}}(-g_{s,k,\mathbf{w^{0}}})\phi(s)\end{split}$ (3)

###  2.3 Outer Loop of Linear Layer

In the outer loop, $\mathbf{w^{0}}$ is updated using the query data with a
learning rate $\rho$. For SOMAML, the final linear layer is updated as follows
(where the complete derivation is provided in Appendix B.4).

$\displaystyle\mathbf{w^{\prime 0}_{k}}$
$\displaystyle=\mathbf{w^{0}_{k}}+\rho[I-\eta\mathop{\mathbf{E}}_{(s,t)\sim
S}(g_{s,k,\mathbf{w^{0}}})\phi(s)\phi(s)^{\top}]\mathop{\mathbf{E}}_{(q,u)\sim
Q}(1_{k=u}-g_{q,k,\mathbf{w^{1}}})\phi(q)$ (4a) $\displaystyle\ \ \ \ \ \ \ \
\ \ \ +\rho[\eta\sum^{N_{way}}_{m=1}\mathop{\mathbf{E}}_{(s,t)\sim
S}(g_{s,m,\mathbf{w^{0}}}\cdot
g_{s,k,\mathbf{w^{0}}})\phi(s)\phi(s)^{\top}]\mathop{\mathbf{E}}_{(q,u)\sim
Q}(1_{m=u}-g_{q,m,\mathbf{w^{1}}})\phi(q)$ (4b)

For FOMAML, we have

$\begin{split}\mathbf{w^{\prime 0}_{k}}&=\mathbf{w^{0}_{k}}-\rho\frac{\partial
L_{\\{\phi,\mathbf{w^{1}}\\},Q}}{\partial\mathbf{w^{1}_{k}}}=\mathbf{w^{0}_{k}}+\rho\mathop{\mathbf{E}}_{(q,u)\sim
Q}(1_{k=u}-g_{q,k,\mathbf{w^{1}}})\phi(q)\end{split}$ (5)

Both SOMAML and FOMAML have the term $\mathop{\mathbf{E}}_{(q,u)\sim
Q}(1_{k=u}-g_{q,k,\mathbf{w^{1}}})\phi(q)$ which highly resembles Eq. 3 but
with the query features. Therefore, it is obvious that during the outer loop,
the final linear layer is updated by inscribing the weighted query features
where the weights relate to the output prediction $g_{q,k,\mathbf{w^{1}}}$.
For simplicity, we denote the matrix $\mathop{\mathbf{E}}_{(s,t)\sim
S}(g_{s,k,\mathbf{w^{0}}})\phi(s)\phi(s)^{\top}$ as $H$.

Now we turn to the difference in using SOMAML and FOMAML to update the linear
layer. There are plenty of works dedicated to approximating or estimating the
second-order derivatives in the MAML algorithm in a more computational-
efficient or accurate manner [21, 22, 7]. However, most of them do not
interpret the effect of the Hessian matrix. Here, we provide a new perspective
that, for the final linear layer, the Hessian matrix is to inform the
crowdedness of the feature space.

Figure 2: Illustration of the difference between FOMAML and SOMAML in updating
the linear layer. In Eq. 5, the gradient of $\mathbf{w^{0}_{k}}$ is a weighted
sum of the support features (denoted as the red arrow). Regarding SOMAML, if
$\mathbf{w^{0}}$ is zeroed, then the gradient is preconditioned by $I-\eta H$
matrix. In this figure, the support features predominantly occupy the first
and the third quadrant, thus the eigenvector with the largest eigenvalue of
the Hessian matrix points at the $[1,1]$ direction. Then, $I-\eta H$ matrix
biases the gradient towards regions of lower variance (denoted as the blue
arrow).

To this end, we conduct an analysis here. We consider the condition that the
linear layer $\mathbf{w^{0}}$ is a zero matrix at the beginning of the inner
loop, consequently, the term 4b becomes zero. (The proof is provided in
Appendix B.4) Now we focus on the term 4a. Without loss of generality, assume
that $\mathop{\mathbf{E}}_{(s,t)\sim S}\phi(s)$ is a zero vector, then $H$ is
essentially a covariance matrix weighted by the predicted probability
(softmax) output. Since a covariance matrix is positive semi-definite, we can
decompose $H$ into $R\Lambda R^{\top}$ where columns of $R$ are the
eigenvectors of $H$ and the diagonal entries of $\Lambda$ are the eigenvalues
of $H$. Recall that in the Pricipal Component Analysis (PCA) algorithm, larger
eigenvalues of the covariance matrix indicates larger variance of the features
projected onto the corresponding eigenvector. Consequently, the term $\rho
R(I-\eta\Lambda)R^{\top}$ generally preconditions the gradient (i.e.
$\mathop{\mathbf{E}}_{(q,u)\sim Q}(1_{k=u}-g_{q,k,\mathbf{w^{1}}})\phi(q)$) to
the eigen-directions whose variances are relatively small. This preconditioner
helps the features to avoid the crowdedness and to explore the subspace of
smaller variance. We illustrate the concept of such preconditioning in Figure
2.

###  2.4 Outer Loop of Encoder

As the encoder $\phi$ is parameterized by $\varphi$, the outer loop gradient
with respect to $\varphi$ is given by $\frac{\partial
L_{\\{\phi,\mathbf{w^{1}}\\},Q}}{\partial\varphi}$
$=\mathop{\mathbf{E}}_{(q,u)\sim Q}\frac{\partial
L_{\\{\phi,\mathbf{w^{1}}\\},Q}}{\partial\phi(q)}\cdot\frac{\partial\phi(q)}{\partial\varphi}$.
Now we take a deeper look at the backpropagated error $\frac{\partial
L_{\\{\phi,\mathbf{w^{1}}\\},Q}}{\partial\phi(q)}$ of one query data
$(q,u)\sim Q$, based on the following form:

$\begin{split}-\frac{\partial
L_{\\{\phi,\mathbf{w^{1}}\\},Q}}{\partial\phi(q)}&=\mathbf{w_{u}^{1}}-\sum_{j=1}^{N_{way}}(g_{q,j,\mathbf{w^{1}}}\cdot\mathbf{w_{j}^{1}})=\sum_{j=1}^{N_{way}}(1_{j=u}-g_{q,j,\mathbf{w^{1}}})\mathbf{w_{j}^{1}}\\\
&=\underbrace{\sum_{j=1}^{N_{way}}(1_{j=u}-g_{q,j,\mathbf{w^{1}}})\mathbf{w^{0}_{j}}}_{\text{interference
term}}+\underbrace{\eta\sum_{j=1}^{N_{way}}[1_{j=u}-g_{q,j,\mathbf{w^{1}}}][\mathop{\mathbf{E}}_{(s,t)\sim
S}(1_{j=u}-g_{s,j,\mathbf{w^{0}}})\phi(s)]}_{\text{contrastive
term}}\end{split}$ (6)

The second term is similar to Eq. 3, prompting the encoder to be updated such
that $\phi(q)$ is pushed towards the direction derived from the support
features of the same class while being pulled away from the direction built
upon the support features of different classes. Note that this contrastive
term is supervised since the class labels are used to indicate the positive
and negative samples.

Now let us inspect the first term in Eq. 6. In the case of $j=u$, the softmax-
cross entropy loss implicitly forces the model to maximize
$(1-g_{q,u,\mathbf{w^{1}}})\phi(q)^{\top}\mathbf{w^{0}_{u}}$. This can be
problematic because (1) at the beginning of training, $\mathbf{w^{0}}$ is
assigned with random values and (2) $\mathbf{w^{0}}$ is imprinted with query
features of previous tasks as shown in Section 2.3. Consequently, when one-
step inner-loop update is considered, $\phi(q)$ is pushed in the direction of
previous query features or towards a random direction, introducing a cross-
task interference or an initialization interference that slow down the
training of the encoder. Noting that the cross-task interference also occurs
at the testing period.

Intuitively, we can remove the interference by simply setting the
$\mathbf{w^{0}}$ to be zero after the outer loop update, as shown in Algorithm
2. Then $\mathbf{w^{1}_{k}}$ in Eq. 3 becomes
$\eta\mathop{\mathbf{E}}_{(s,t)\sim
S}(1_{k=t}-g_{s,k,\mathbf{w^{0}}})\phi(s)$, meaning that $\mathbf{w^{1}_{k}}$
is exactly the weighted sum of the support features. This draw a close
relationship with the metric-based few-shot learning where the prototypes are
now imprinted in $\mathbf{w^{0}}$. Our experiments (provided in the next
section) agree with this intuition and show that the zero-initialization of
$\mathbf{w^{0}}$, reducing the initial norm of $\mathbf{w^{0}}$, or even
zeroing $\mathbf{w^{0}}$ per iteration can speed up the learning profile. This
is applicable to both SOMAML and FOMAML.

###  2.5 Generalization of our Analysis

The analysis of the outer loop update of the encoder is derived under the
condition of $N_{batch}=1$ and one update step in the inner loop. We include
the full analysis in Appendix B while keeping the assumption of freezing
encoder during the inner loop. This assumption can hardly be dropped because
the behavior of the updated encoder is intractable. Moreover, the assumption
of the frozen encoder is empirically reasonable since previous works of [11,
13, 14, 15] yield comparable performance with keeping the encoder fixed during
the inner loop.

##  3 Experimental Results

###  3.1 Setup

We conduct our experiments on the miniImagenet dataset [23, 24] and the
Omniglot dataset [25]. For the results on Omniglot dataset, please refer to
Appendix D. For the miniImagenet, it contains $84\times 84$ RGB images of 100
classes from the ImageNet dataset with 600 samples per class. We split the
dataset into 64, 16 and 20 classes for training, validation, and testing as
proposed in [24]. In our experiment, we do not perform hyperparameter search
and thus are not using the validation data. For all our experiments of
applying MAML into few-shot classification problem, where we adopt two
experimental settings: 5-way 1-shot and 5-way 5-shot, with the batch size
$N_{batch}$ for these two settings are 4 and 2 respectively[4]. The few-shot
classification accuracy is calculated by averaging the results over 400 tasks
in the test phase. The inner loop step size $\eta$ is set to 0.01. The models
are trained for 30000 iterations [11]. The results are averaged over four
random seeds, where in all the figures of results we use the shaded region to
indicate the standard deviation. For model architecture, optimizer and other
experimental details, please refer to Appendix C.

###  3.2 Cosine similarity analysis during training verifies the implicit
contrastiveness in MAML

In Section 2, we show that the encoder is updated so that the query features
are pushed towards the direction derived from the support features of the same
class and pulled away from the direction built upon the support features of
different classes. Here we verify this supervised contrastiveness
experimentally. Consider a relatively overfitting scenario where there are
five classes of images and for each class there are 20 support images and 20
query images. We fix the support and query set (i.e. data is not resampled
every iteration) to verify the concept that the support features work as
positive and negative samples. Channel shuffling is performed to avoid the
undesirable channel memorization effect. We train the model using FOMAML and
examine how well the encoder can discriminate the data of different classes in
the feature space by measuring the averaged pair-wise cosine similarities
between the features of each class. Note that we treat the support set and the
query set of the same class independently in such analysis. The results are
averaged over 10 random seeds.

As shown in the top row of Figure 3, the model trained with the MAML algorithm
indeed learns to discriminate the features of different classes. Moreover, the
contrast between the diagonal and the off-diagonal entries of the heatmap
increases as we remove the initialization interference (by zero-initializing
the linear layer, shown in the middle row) and remove the cross-task
interference (by applying the zeroing trick every outer loop, shown in the
bottom row), where such property is consistent with our analysis that the MAML
algorithm implicitly contains the interference term which can impede the
encoder from learning a good feature representation. With removing the
interference by zero initialization or the zeroing trick, MAML better
manifests its supervised contrastiveness.

Figure 3: The supervised contrastiveness entailed in the MAML algorithm
manifests when zero initialization or the zeroing trick is applied. The value
in the heatmap is calculated by averaging the pairwise cosine similarities
between query features or between query features and support features. We
consider the setting of having randomly initialized linear layer (top), zero-
initialized linear layer (middle), and the zeroing trick (bottom), and
experiment with various numbers of outer loop updates.

###  3.3 Applying the zeroing trick at testing time increases the testing
accuracy

Before we start to verify our analysis on benchmark dataset, it is worth
noting that the cross-task interference can also occur at meta-testing stage.
During meta-testing, some tasks are drawn from the meta-testing set while each
task contains both the support and query data. The model is updated using the
support data (as fast adaptation) for several steps and then the updated model
is evaluated on the query data. As mentioned in Section 2.4, the testing
accuracy can be highly influenced by the interference term since the linear
layer has been inscribed with the query features of the training sets. To
validate this phenomenon, we show that applying the zeroing trick (zeroing the
last linear layer) before the testing time increases the testing accuracy of
the model trained with FOMAML.

As illustrated in Figure 4, compared to directly entering meta-testing (i.e.
the two subplots at the left), applying the zeroing trick before the meta-
testing stage (i.e. the two subplots at the right) increases the testing
accuracy for the models that are trained by having their linear layer randomly
initialized or zero-initialized (denoted by the red and orange curves
respectively). Besides, the difference in testing performance sustains across
the whole training session. On the other hand, in the two leftmost subplots,
the testing accuracy for red and orange curves increases when more inner loop
update steps are taken (e.g. comparing the red curves across the first and the
second subplots), yet this trend is not observable to the case when the
zeroing trick is used (i.e. green curves). Such phenomenon actually accords
with Equation (3), since having more update steps indicates that more support
features are imprinted onto the final linear layer, which can outweigh the
effect brought by the interference term. While in the case of training with
the zeroing trick, there is no interference term to compete with.

In the following experiments, we evaluate the testing performance only with
applying the zeroing trick before the meta-testing stage. By zeroing the last
linear layer, the potential interference brought by the prior (of the linear
layer) is ignored. We then focus on the capacity of the encoder in learning a
good feature representation.

Figure 4: Using the zeroing trick at testing stage improves the testing
accuracy. The two subplots on the left: original testing setting. The two
subplots at the right: the final linear layer is zeroed before testing time.
The curves in red: the models whose linear layer is randomly initialized. The
curves in yellow: the models whose linear layer is zeroed at initialization.
The curves in green: the models whose linear layer is zeroed after each outer
loop at training stage.

###  3.4 Effect of initialization and the zeroing trick

In Equation 6, we observe an interference term derived from the historical
task features or random initialization. We validate our formula by examining
the effects of (1) reducing the norm of the linear layer at initialization and
check if there are improvements in the testing profile during training; (2)
applying the zeroing trick (i.e. zeroing the linear layer after each outer
loop update) which forces the model to forget the information of previous
query features. We experiment with 5-way 1-shot/5-shot settings using FOMAML
on the miniImagenet dataset.

From Figure 5, the testing accuracy is higher when the norm of the linear
layer at initialization is lower. Compared to random initialization, reducing
the norm via down-scaling the weight of the linear layer by $0.7$ yields
visible differences in testing performance.

As for the case of zeroing the linear layer per outer loop (the purple curve),
the resultant testing accuracy outperforms that of the models with randomly
initialized linear layer. The results with SOMAML are shown in Figure 6.

Figure 5: Effect of initialization and the zeroing trick in testing accuracy.
We show that both reducing the norm of the weight of linear layer and zeroing
the linear layer after each outer loop increase the testing performance. The
curves in red: models with the linear layer randomly initialized. The curves
in orange/green: reducing the norm of the linear layer at initialization by a
factor of $0.7$/ $0.5$. The curve in blue: the linear layer of the model is
zero-initialized. The curve in blue: models trained with the zeroing trick.
Figure 6: Both FOMAML and SOMAML benefit from the zeroing trick. We examine if
reducing or removing the interference can increase the testing performance in
models trained with FOMAML and SOMAML. The results suggest that SOMAML suffers
from the interference term. Note that the second subplots from the right shows
lower testing performance of models trained with the zeroing trick as compared
to the zero-initialized model. This may result from the overfitting problem.

###  3.5 The zeroing trick mitigates the channel memorization problem

The channel memorization problem [9, 26] is a known issue occurring in a non-
mutually-exclusive task setting, e.g., the task-specific class-to-label is not
randomly assigned, and thus the label can be inferred from the query data
alone [27]. Consider a 5-way K-shot experiment where the number of training
classes is 5 $\times$ L. Now we construct tasks by assigning the label $t$ to
a class sampled from class $t$L to $(t+1)$L. It is conceivable that the model
will learn to directly map the query data to the label without using the
information of the support data and thus fails to generalize to unseen tasks.
This phenomenon can be explained from the perspective that the $t^{th}$ column
of the final linear layer already accumulates the query features from $t$Lth
to $(t+1)$Lth classes. Zeroing the final linear layer implicitly forces the
model to use the imprinted information from the support features for inferring
the label and thus mitigates this problem. We use the miniImagenet dataset and
consider the case of $\text{L}=12$. As shown in Figure 7, the zeroing trick
prevents the model from the channel memorization problem whereas zero-
initialization of linear layer only works out at the beginning. Besides, the
performance of models trained with the zeroing trick under this non-mutually-
exclusive task setting equals the ones under the conventional few-shot setting
as shown in Figure 5. As the zeroing trick clears out the final linear layer
and equalizes the value of logits, our result essentially accords with [9]
that proposes a regularizer to maximize the entropy of prediction of the meta-
initialized model.

Figure 7: The performance of the models trained on non-mutually exclusive
tasks. The models are trained under a non-mutually exclusive tasks setting
where there is a one-to-one assignment between class and channel. Under this
circumstance, the zeroing trick tackles with the channel memorization problem
and yields a performance similar to conventional few-shot setting.

##  4 Conclusion

In this paper, we perform an extensive study to demystify how the seminal MAML
algorithm guides the encoder to learn a general-purpose feature representation
and what is the interaction between the support and query set. Our analysis
shows that MAML is implicitly a supervised contrastive learner using the
support features as positive and negative samples to direct the update of the
encoder. Yet, beside the supervised contrastive objective, we unveil an
interference term hidden in the MAML algorithm originated from the random
initialization and the cross-task interaction, which can impede the
representation learning. Driven by our analysis, removing the interference
term by a simple zeroing trick renders the model unbiased to seen or unseen
tasks. With this zeroing trick, we demonstrate constant improvements in both
the training and testing profiles, with experiments conducted on the
miniImagenet and Omniglot datasets.

## References

  * [1] R. Vilalta and Y. Drissi, “A perspective view and survey of meta-learning,” _Artificial Intelligence Review_ , 2002.
  * [2] E. Grant, C. Finn, S. Levine, T. Darrell, and T. Griffiths, “Recasting gradient-based meta-learning as hierarchical bayes,” in _International Conference on Learning Representations (ICLR)_ , 2018.
  * [3] T. Hospedales, A. Antoniou, P. Micaelli, and A. Storkey, “Meta-learning in neural networks: A survey,” _arXiv preprint arXiv:2004.05439_ , 2020.
  * [4] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _International Conference on Learning Representations (ICLR)_ , 2017.
  * [5] A. Nichol, J. Achiam, and J. Schulman, “On first-order meta-learning algorithms,” _arXiv preprint arXiv:1803.02999_ , 2018.
  * [6] A. Rajeswaran, C. Finn, S. Kakade, and S. Levine, “Meta-learning with implicit gradients,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * [7] H. Liu, R. Socher, and C. Xiong, “Taming maml: Efficient unbiased meta-reinforcement learning,” in _International Conference on Machine Learning (ICML)_ , 2019.
  * [8] C. Finn, A. Rajeswaran, S. Kakade, and S. Levine, “Online meta-learning,” in _International Conference on Machine Learning (ICML)_ , 2019.
  * [9] M. A. Jamal and G.-J. Qi, “Task agnostic meta-learning for few-shot learning,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [10] K. Javed and M. White, “Meta-learning representations for continual learning,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * [11] A. Raghu, M. Raghu, S. Bengio, and O. Vinyals, “Rapid learning or feature reuse? towards understanding the effectiveness of maml,” in _International Conference on Learning Representations (ICLR)_ , 2020.
  * [12] M. Goldblum, S. Reich, L. Fowl, R. Ni, V. Cherepanova, and T. Goldstein, “Unraveling meta-learning: Understanding feature representations for few-shot tasks,” in _International Conference on Machine Learning (ICML)_ , 2020.
  * [13] K. Lee, S. Maji, A. Ravichandran, and S. Soatto, “Meta-learning with differentiable convex optimization,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [14] L. Bertinetto, J. F. Henriques, P. H. Torr, and A. Vedaldi, “Meta-learning with differentiable closed-form solvers,” in _International Conference on Learning Representations (ICLR)_ , 2019.
  * [15] C. Liu, C. Xu, Y. Wang, L. Zhang, and Y. Fu, “An embarrassingly simple baseline to one-shot learning,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [16] S. S. Du, W. Hu, S. M. Kakade, J. D. Lee, and Q. Lei, “Few-shot learning via learning the representation, provably,” in _International Conference on Learning Representations (ICLR)_ , 2021.
  * [17] N. Tripuraneni, C. Jin, and M. I. Jordan, “Provable meta-learning of linear representations,” _arXiv preprint arXiv:2002.11684_ , 2020.
  * [18] K. Chua, Q. Lei, and J. D. Lee, “How fine-tuning allows for effective meta-learning,” _arXiv preprint arXiv:2105.02221_ , 2021.
  * [19] Q. Bouniot, I. Redko, R. Audigier, A. Loesch, Y. Zotkin, and A. Habrard, “Towards better understanding meta-learning methods through multi-task representation learning theory,” _arXiv preprint arXiv:2010.01992_ , 2020\.
  * [20] P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, “Supervised contrastive learning,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * [21] X. Song, W. Gao, Y. Yang, K. Choromanski, A. Pacchiano, and Y. Tang, “Es-maml: Simple hessian-free meta learning,” in _International Conference on Learning Representations (ICLR)_ , 2020.
  * [22] J. Rothfuss, D. Lee, I. Clavera, T. Asfour, and P. Abbeel, “Promp: Proximal meta-policy search,” in _International Conference on Learning Representations (ICLR)_ , 2019.
  * [23] O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, and D. Wierstra, “Matching networks for one shot learning,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2016.
  * [24] S. Ravi and H. Larochelle, “Optimization as a model for few-shot learning,” in _International Conference on Learning Representations (ICLR)_ , 2017.
  * [25] B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum, “Human-level concept learning through probabilistic program induction,” _Science_ , 2015.
  * [26] J. Rajendran, A. Irpan, and E. Jang, “Meta-learning requires meta-augmentation,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * [27] M. Yin, G. Tucker, M. Zhou, S. Levine, and C. Finn, “Meta-learning without memorization,” in _International Conference on Learning Representations (ICLR)_ , 2020.
  * [28] A. Antoniou, H. Edwards, and A. Storkey, “How to train your maml,” in _International Conference on Learning Representations (ICLR)_ , 2019.
  * [29] L. Long, “Maml-pytorch implementation,” https://github.com/dragen1860/MAML-Pytorch, 2018.
  * [30] T. Deleu, “Model-agnostic meta-learning,” https://github.com/tristandeleu/pytorch-maml, 2020.

## Appendix

##  Appendix A The MAML Algorithm with the Zeroing Trick

Here we include the second-order MAML algorithm with the zeroing trick below.

Algorithm 2 Second-order MAML algorithm with the zeroing trick

Require: Distribution over tasks $D$
Require: $\eta,\rho$: inner loop and outer loop learning rates
Require: Randomly initialized base-model parameters
$\theta=\\{\phi,\mathbf{w}\\}$

1:Set $\mathbf{w}\leftarrow 0$ (the zeroing trick)

2:while not done do

3: Sample tasks $\\{T_{1},\dots T_{N_{batch}}\\}$ from $D$

4: for n$=1,2,\ldots,N_{batch}$ do

5: $S_{n}\leftarrow$ sample $N_{shot}$ data from $T_{n}$

6: $Q_{n}\leftarrow$ sample $N_{query}$ data from $T_{n}$

7: Update the base-model with gradient descent:
$\theta^{\prime}_{n}=\theta-\eta\nabla_{\theta}L_{\theta,Sn}$

8: end for

9: Update
$\theta\leftarrow\theta-\rho\sum^{N_{batch}}_{n=1}\nabla_{\theta}L_{\theta^{\prime}_{n},Qn}$

10: Set $\mathbf{w}\leftarrow 0$ (the zeroing trick)

11:end while

##  Appendix B A Generalization of our Analysis

In this section, we derive a more general case of the encoder update in the
outer loop. We consider drawing $N_{batch}$ tasks from the task distribution
$D$ and having $N_{step}$ update steps in the inner loop while keeping the
assumption of frozen encoder in the inner loop.

###  B.1 Inner Loop Update for the Linear Layer

To derive a more general case, we use $\mathbf{w^{i,n}_{k}}$ to denote the
$k^{th}$ column of $\mathbf{w^{i,n}}$, where $\mathbf{w^{i,n}}$ is updated
from $\mathbf{w^{0}}$ using support data $S_{n}$ for $i$ inner-loop steps.
Then, we yield the inner loop update for the final linear layer in Equation 7
and Equation 8. Note that in Equation 8, $g_{s,k,\mathbf{w^{i-1,n}}}$ is the
predicted probability (softmax) output of the $k^{th}$ channel which
implicitly requires $\mathbf{w^{i-1,n}}$ for computation.

$\begin{split}\mathbf{w^{i,n}_{k}}&=\mathbf{w^{i-1,n}_{k}}-\eta\frac{\partial
L_{\\{\phi,\mathbf{w^{i-1,n}}\\},S_{n}}}{\partial\mathbf{w^{i-1,n}_{k}}}=\mathbf{w^{i-1,n}_{k}}+\eta\mathop{\mathbf{E}}_{(s,t)\sim
S_{n}}(1_{k=t}-g_{s,k,\mathbf{w^{i-1,n}}})\phi(s)\end{split}$ (7)
$\begin{split}\mathbf{w^{N_{step},n}_{k}}&=\mathbf{w^{0}_{k}}-\eta\sum^{N_{step}}_{i=1}\mathop{\mathbf{E}}_{(s,t)\sim
S_{n}}(1_{k=t}-g_{s,k,\mathbf{w^{i-1,n}}})\phi(s)\end{split}$ (8)

###  B.2 Outer Loop Update for the Linear Layer

We derive the outer loop update for the linear layer in SOMAML, with denoting
$I=\\{1,2,...,N_{way}\\}$:

$\begin{split}\mathbf{w^{\prime
0}_{k}}&=\mathbf{w^{0}_{k}}-\rho\sum^{N_{batch}}_{n=1}\frac{\partial
L_{\\{\phi,\mathbf{w^{N_{step},n}_{k}}\\},Q_{n}}}{\partial\mathbf{w^{0}_{k}}}\\\
&=\mathbf{w^{0}_{k}}-\rho\sum^{N_{batch}}_{n=1}\sum_{\begin{subarray}{c}p_{0}=k,p_{1}\in
I,...,p_{N_{way}}\in
I\end{subarray}}[(\prod^{N_{step}-1}_{i=0}\frac{\partial\mathbf{w^{i+1,n}_{p_{i+1}}}}{\partial\mathbf{w^{i,n}_{p_{i}}}})\frac{\partial
L_{\\{\phi,\mathbf{w^{N_{step},n}}\\},Q_{n}}}{\partial\mathbf{w^{N_{step},n}_{p_{N_{step}}}}}]\end{split}$
(9)

When it comes to FOMAML, we have

$\begin{split}\mathbf{w^{\prime
0}_{k}}&=\mathbf{w^{0}_{k}}-\rho\sum^{N_{batch}}_{n=1}\frac{\partial
L_{\\{\phi,\mathbf{w^{N_{step},n}_{k}}\\},Q_{n}}}{\partial\mathbf{w^{N_{step},n}_{k}}}=\mathbf{w^{0}_{k}}+\rho\sum^{N_{batch}}_{n=1}\mathop{\mathbf{E}}_{(q,u)\sim
Q_{n}}(1_{k=u}-g_{q,k,\mathbf{w^{N_{step},n}}})\phi(q)\end{split}$ (10)

###  B.3 Outer Loop Update for the Encoder

We derive the outer loop update of the encoder as below. Similar to Section
2.4 in the main manuscript, we consider the back-propagated error of one query
data $(q,u)\sim Q_{n}$. Note that the third equality below holds by leveraging
Equation 7.

$\begin{split}-\frac{\partial
L_{\\{\phi,\mathbf{w^{N_{step},n}}\\},Q_{n}}}{\partial\phi(q)}&=\mathbf{w_{u}^{N_{step},n}}-\sum_{i=1}^{N_{way}}(g_{q,i,\mathbf{w^{N_{step},n}}}\cdot\mathbf{w_{i}^{N_{step},n}})\\\
&=\sum_{i=1}^{N_{way}}(1_{i=u}-g_{q,i,\mathbf{w^{N_{step},n}}})\mathbf{w_{i}^{N_{step},n}}\\\
&=\sum_{i=1}^{N_{way}}(1_{i=u}-g_{q,i,\mathbf{w^{N_{step},n}}})[\mathbf{w^{0}_{i}}+\eta\sum^{N_{step}}_{p=1}\mathop{\mathbf{E}}_{(s,t)\sim
S_{n}}(1_{i=t}-g_{s,i,\mathbf{w^{p-1,n}}})\phi(s)]\\\
&=\sum_{i=1}^{N_{way}}(1_{i=u}-g_{q,i,\mathbf{w^{N_{step},n}}})\mathbf{w^{0}_{i}}\\\
&\ \ \ \ \ \ \
+\eta\sum_{i=1}^{N_{way}}(1_{i=u}-g_{q,i,\mathbf{w^{N_{step},n}}})\sum^{N_{step}}_{p=1}\mathop{\mathbf{E}}_{(s,t)\sim
S_{n}}(1_{i=u}-g_{s,i,\mathbf{w^{p-1,n}}})\phi(s)\end{split}$ (11)

###  B.4 The derivation of outer loop update for the linear layer using
second-order MAML

Here, we provide the complete derivation of the outer loop update for the
linear layer. Using SOMAML with support set $S$ and query set $Q$, the update
of the linear layer follows

$\begin{split}\mathbf{w^{\prime 0}_{k}}&=\mathbf{w^{0}_{k}}-\rho\frac{\partial
L_{\\{\phi,\mathbf{w^{1}}\\},Q}}{\partial\mathbf{w^{0}_{k}}}=\mathbf{w^{0}_{k}}-\rho\sum^{N_{way}}_{m=1}\frac{\partial\mathbf{w^{1}_{m}}}{\partial\mathbf{w^{0}_{k}}}\cdot\frac{\partial
L_{\\{\phi,\mathbf{w^{1}}\\},Q}}{\partial\mathbf{w^{1}_{m}}}\\\
&=\mathbf{w^{0}_{k}}-\rho\frac{\partial\mathbf{w^{1}_{k}}}{\partial\mathbf{w^{0}_{k}}}\cdot\frac{\partial
L_{\\{\phi,\mathbf{w^{1}}\\},Q}}{\partial\mathbf{w^{1}_{k}}}-\rho\sum^{N_{way}}_{m\neq
k}\frac{\partial\mathbf{w^{1}_{m}}}{\partial\mathbf{w^{0}_{k}}}\cdot\frac{\partial
L_{\\{\phi,\mathbf{w^{1}}\\},Q}}{\partial\mathbf{w^{1}_{m}}}\\\
&=\mathbf{w^{0}_{k}}+\rho[I-\eta\mathop{\mathbf{E}}_{(s,t)\sim
S}(g_{s,k,\mathbf{w^{0}}}-g^{2}_{s,k,\mathbf{w^{0}}})\phi(s)\phi(s)^{T}]\mathop{\mathbf{E}}_{(q,u)\sim
Q}(1_{k=u}-g_{q,k,\mathbf{w^{1}}})\phi(q)\\\ &\ \ \ \ \ \ \ \ \ \ \
+\rho[\eta\sum_{m\neq k}\mathop{\mathbf{E}}_{(s,t)\sim
S}(g_{s,m,\mathbf{w^{0}}}g_{s,k,\mathbf{w^{0}}})\phi(s)\phi(s)^{T}]\mathop{\mathbf{E}}_{(q,u)\sim
Q}(1_{m=u}-g_{q,m,\mathbf{w^{1}}})\phi(q)\\\
&=\mathbf{w^{0}_{k}}+\rho[I-\eta\mathop{\mathbf{E}}_{(s,t)\sim
S}(g_{s,k,\mathbf{w^{0}}})\phi(s)\phi(s)^{T}]\mathop{\mathbf{E}}_{(q,u)\sim
Q}(1_{k=u}-g_{q,k,\mathbf{w^{1}}})\phi(q)\\\ &\ \ \ \ \ \ \ \ \ \ \
+\rho[\eta\sum^{N_{way}}_{m=1}\mathop{\mathbf{E}}_{(s,t)\sim
S}(g_{s,m,\mathbf{w^{0}}}g_{s,k,\mathbf{w^{0}}})\phi(s)\phi(s)^{T}]\mathop{\mathbf{E}}_{(q,u)\sim
Q}(1_{m=u}-g_{q,m,\mathbf{w^{1}}})\phi(q)\end{split}$ (12)

When we consider the case that $\mathbf{w^{0}}$ is a zero matrix, then
essentially the predicted probability (softmax) output
$g_{s,m,\mathbf{w^{0}}}$ is $\frac{1}{N_{way}}$ for all channel $m$. In this
case, we show in Equation 13 that the third term in Equation 12 turns out to
be zero. The last inequality below holds as the summation of the (softmax)
outputs is one.

$\begin{split}&\frac{\rho\eta}{N^{2}_{way}}[\sum^{N_{way}}_{m=1}\mathop{\mathbf{E}}_{(s,t)\sim
S}\phi(s)\phi(s)^{T}]\mathop{\mathbf{E}}_{(q,u)\sim
Q}(1_{m=u}-g_{q,m,\mathbf{w^{1}}})\phi(q)\\\
&=\frac{\rho\eta}{N^{2}_{way}}[\mathop{\mathbf{E}}_{(s,t)\sim
S}\phi(s)\phi(s)^{T}]\mathop{\mathbf{E}}_{(q,u)\sim
Q}\phi(q)\sum^{N_{way}}_{m=1}(1_{m=u}-g_{q,m,\mathbf{w^{1}}})=0\end{split}$
(13)

##  Appendix C Experimental details in miniImagenet dataset

The model architecture contains four basic blocks and one fully connected
linear layer, where each block comprises a convolution layer with a kernel
size of $3\times 3$ and filter size of $64$, batch normalization, ReLU
nonlineartity and $2\times 2$ max-poling. The models are trained with the
softmax cross entropy loss function using the Adam optimizer with an outer
loop learning rate of 0.001 [28]. Each experiment is run on either a single
NVIDIA 1080-Ti or V100 GPU. The detailed implementation is based on [29] (MIT
License).

##  Appendix D Experiments on Omniglot dataset

Omniglot is a hand-written character dataset containing 1623 character
classes, each with 20 drawn samples from different people [25]. The dataset
set is splitted into training (1028 classes), validation (172 classes) and
testing (423 classes) sets [23]. Since we follow [4] for setting
hyperparamters, we do not use the the validation data. The character images
are resized to 28 $\times$ 28\. For all our experiments, we adopt two
experimental settings: 5-way 1-shot and 5-way 5-shot where the batch size
$N_{batch}$ is $32$ and $N_{query}$ is $15$ for both cases [4]. The inner loop
learning rate $\eta$ is 0.4. The models are trained for 30000 or 20000
iterations using FOMAML or SOMAML, respectively. For channel memorization
problem, the models are trained for 10000 iterations using FOMAML. The few-
shot classification accuracy is calculated by averaging the results over 500
tasks in the test stage. The model architecture follows the architecture used
to train on miniImagenet, but we substitute the convolution with max-pooling
with strided convolution operation as in [4]. The loss function, optimizer,
and outer loop learning rate are the same as the ones used in the experiments
on miniImagenet. Each experiment is run on either a single NVIDIA 1080-Ti or
V100 GPU. The results are averaged over four random seeds, and the standard
deviation is illustrated with the shaded region. The models are trained using
FOMAML unless stated otherwise. The detailed implementation is based on [30]
(MIT License).

We revisit the application of the zeroing trick at the testing stage on
Omniglot in Figure 8 and observe the increasing testing accuracy, in which
such results are compatible with the ones on mini-Imagenet (cf. Figure 4 in
the main manuscript). In the following experiments, we evaluate the testing
performance only after applying the zeroing trick.

In Figure 9, the distinction between the performance of models trained with
the zeroing trick and zero-initialized models is less prominent as compared to
miniImagenet (cf. Figure 5 in the main manuscript) in 5-way 1-shot setting.
This result can be explained by the larger inner loop learning rate $\eta=0.4$
used in Omniglot, since larger learning rate can alleviate the influence of
the interference term in Equation 6 of the main manuscript. We also show the
testing performance of models trained using SOMAML in Figure 10, where there
is little distinction in performance (in comparison to the results on
miniImagenet, cf. Figure 6 in the main manuscript) between the models trained
with the zeroing trick and the ones trained with random initialization.

For channel memorization task, we construct non-mutually-exclusive training
tasks by assigning the label $t$ (where $1\leq t\leq 5$ in 5-way setting) to a
class sampled from class $t$L to $(t+1)$L where L is $205$ on Omniglot. The
class-to-channel assignment is not applied to the testing tasks. The result is
shown in Figure 11. For a detailed discussion, please refer to Section 3.5 in
the main manuscript.

Figure 8: Zeroing the final linear layer before testing time improves the
testing accuracy on Omniglot. The two subplots on the left: original testing
setting. The two subplots at the right: the final linear layer is zeroed
before testing time. The curves in red: the models whose linear layer is
randomly initialized. The curves in yellow: the models whose linear layer is
zeroed at initialization. The curves in green: the models whose linear layer
is zeroed after each outer loop at training stage. Figure 9: Effect of
initialization and the zeroing trick in testing accuracy on Omniglot. The test
performance of the models with reduced initial norm of the weights of final
linear layer is similar to that with the final linear layer being zero-
initialized. The distinction in performance between models trained using the
zeroing trick and zero-initialized model is more prominent in 5-way 5-shot
setting. Figure 10: The effect of the zeroing trick on models trained using
FOMAML and SOMAML on Omniglot. The results suggest that the effect of the
zeroing trick is more prominent in models trained with FOMAML and less visible
when models are trained with SOMAML on Omniglot.  Figure 11: The performance
of the models trained on non-mutually exclusive tasks on Omniglot. The results
are compatible to those on miniImagenet (cf. Figure 11 in the main
manuscript), suggesting that the zeroing trick alleviates the channel
memorization problem. Besides, the testing accuracy of the models trained on
non-mutually exclusive task setting with the zeroing trick (i.e. the green
curves in Figure 11) is lower than that of the models trained on conventional
few-shot setting with the zeroing trick (i.e. the green curves in Figure 7),
whereas in miniImagenet the distinction in performance between the models
trained on these two settings is relatively small.