# PatchMix Augmentation to Identify Causal Features in Few-shot Learning

Chengming Xu∗, Chen Liu∗, Xinwei Sun, Siqian Yang, Yabiao Wang, Chengjie Wang,
Yanwei Fu ∗ indicates equal contributions. Xinwei Sun and Yanwei Fu are the
corresponding authors. This work was supported in part by the National Natural
Science Foundation of China Grant (62076067), Shanghai Municipal Science and
Technology Major Project (2021SHZDZX0103), and ZJ Lab. Chengming Xu, Xinwei
Sun and Yanwei Fu are with the School of Data Science and MOE Frontiers Center
for Brain Science, Fudan University. Yanwei Fu is also with Fudan ISTBI—ZJNU
Algorithm Centre for Brain-inspired Intelligence, Zhejiang Normal University,
Jinhua, China. E-mail: {cmxu18, sunxinwei<EMAIL_ADDRESS>Chen Liu is
with the Department of Mathematics at the Hong Kong University of Science and
Technology. E-mail<EMAIL_ADDRESS>Siqian Yang, Yabiao Wang and
Chengjie Wang are with Youtu Lab, Tencent. Chengjie Wang is also with Shanghai
Jiao Tong Universtity. E-mail: {seasonsyang, caseywang,
<EMAIL_ADDRESS>
###### Abstract

The task of Few-shot learning (FSL) aims to transfer the knowledge learned
from base categories with sufficient labelled data to novel categories with
scarce known information. It is currently an important research question and
has great practical values in the real-world applications. Despite extensive
previous efforts are made on few-shot learning tasks, we emphasize that most
existing methods did not take into account the distributional shift caused by
sample selection bias in the FSL scenario. Such a selection bias can induce
spurious correlation between the semantic _causal_ features, that are causally
and semantically related to the class label, and the other _non-causal_
features. Critically, the former ones should be invariant across changes in
distributions, highly related to the classes of interest, and thus well
generalizable to novel classes, while the latter ones are not stable to
changes in the distribution. To resolve this problem, we propose a novel data
augmentation strategy dubbed as _PatchMix_ that can break this spurious
dependency by replacing the patch-level information and supervision of the
query images with random gallery images from different classes from the query
ones. We theoretically show that such an augmentation mechanism, different
from existing ones, is able to identify the causal features. To further make
these features to be discriminative enough for classification, we propose
_Correlation-guided Reconstruction_ (CGR) and _Hardness-Aware_ module for
instance discrimination and easier discrimination between similar classes.
Moreover, such a framework can be adapted to the unsupervised FSL scenario.
The utility of our method is demonstrated on the state-of-the-art results
consistently achieved on several benchmarks including miniImageNet,
tieredImageNet, CIFAR-FS , CUB, Cars, Places and Plantae, in all settings of
single-domain, cross-domain and unsupervised FSL. By studying the intra-
variance property of learned features and visualizing the learned features, we
further quantitatively and qualitatively show that such a promising result is
due to the effectiveness in learning causal features.

###### Index Terms:

Few-Shot Learning, spurious correlation, causal features, intra-variance
regularization

##  1 Introduction

Among many factors to the successful deep learning applications of modern
society, it is necessary and decisive to collect a large amount of labeled
training data. Typically, prevailing computer vision models such as ResNet [1]
and Faster R-CNN [2] are trained by millions of labeled examples and thus
achieve decent generalization ability. Unfortunately, for some cases such as
the rare species, it is not feasible to collect large amount of labeled and
diverse data which can be used for training.

Motivated by humans’ ability of learning to learn new objects/concepts with
few references, the task of Few-Shot Learning (FSL) is recently studied in the
computer vision and machine learning communities. Generally, the FSL aims at
learning a model, which can generalize to novel/target dataset with few
labelled data (_i.e._ , support sample) available, on base/source dataset of
vast annotated data.

The existing FSL methods [3, 4, 5] exploit the knowledge transferring from the
base to novel categories via meta-learning paradigm. One of the most commonly
used supervision signals by these meta-learning methods is the image class
labels from base dataset [6]. This is the same as many-shot learning. And
particularly in classical many-shot learning, the guidance from class labels
is common and effective under the independent and identically distributed
(_i.i.d_) assumption. Unfortunately, the FSL is suffered from the problem of
distribution shift (Chap.20 in [7]); and the testing distribution of novel
target classes is quite different from the training distribution of those
source/base classes. This is caused by the sample selection bias existed in
data collection and support/query set splitting.

Figure 1: The existing FSL methods typically train the models directly with
image-level annotations. In this way both the causal (_e.g._ the dogs) and
non-causal features (_e.g._ the people, trees and grass) are learned to build
up correlation with the class label, which can lose prediction ability and be
less effective on novel data due to the domain gap. Compared with those
methods, our proposed PatchMix can help the model learn disentangled causal
features by breaking up the dependency between causal and non-causal features,
thus making better generalization to novel categories.

Such a selection bias can induce spurious correlation between causal and non-
causal features of the classes of interest. These two kinds of features
causally and not causally influence the class label respectively. For example,
if the label is ”dog”, its texture, shape are causal features while those
features that are related to the ”people” are non-causal features. Critically,
the causal features should be reliably predictive of the classes of interest,
invariant across changes in distributions, and thus well generalizable to
novel classes. As deep models are typically optimized to well fit the training
data, such ”shortcut non-causal features” would be easier to be learned by
inheriting the spurious correlation [8, 9]. While the non-causal features may
be still good to the supervised learning on the i.i.d data [10], it may hurt
the performance of generalizing to novel data, which has different
distribution from the base data.

We give a detailed example in Fig. 1. As shown in the feature map, the model
has learned to highly associate the grass and persons with the dogs to help
recognize the ‘dog’ class. This makes sense, as obviously people would like to
play with dog on the grass quite often. This leads to the biased combination
of these elements in the sampled training data. Nevertheless, the learned non-
causal features, _i.e._ , the grass and the people, are no longer correlated
with other categories like horse, lion anymore. Compared with IFSL [11] which
also considers spurious correlation but only partly studies the bias inherited
from the pre-trained backbones, our paper will understand and address this
challenge in the general distribution shift in FSL. Even when there is no pre-
training stage, the spurious correlation can still commonly exists via such a
shift.

To this end, we propose enforcing a novel data-augmentation mechanism dubbed
as PatchMix that are effective in learning causal features and hence better
generalization to novel categories. Specifically, our PatchMix replaces some
patches from each query image with class label $A$ with a set of gallery
images from a random different class $B$. Meanwhile, the replaced patches are
labelled as $B$ (_i.e._ , gallery category), rather than the query category,
_i.e._ , $A$. In this regard, the spurious correlation between causal and non-
causal factors that often come up from different patches (_e.g._ , the dog and
the people) is largely weaken, endowing the model with the ability to
disentangle the causal features from others. In particular, we provide a
theoretical analysis from _Structural Causal Model_ (SCM). We show that our
PatchMix operation has the ability of reducing correlation between
spurious/non-causal and causal features; thus it is also able to defunction
the non-causal features during testing on novel classes. Such a
disentanglement can make our PatchMix prominent among other data augmentation
methods like CutMix [12] (see Sec. 3.3 for detailed analysis). After dropping
the non-causal features, our learned features can correlate better with class
labels, and thus have smaller variances among instances from the same novel
category. As the result, models trained with PatchMix can enjoy an easy
classification between different classes.

To make our causal features more discriminative for classification, we further
propose two modules to enhance the PatchMix, namely _hardness-aware_ and
_Correlation-Guided Reconstruction_ (CGR) modules. Specifically, in CGR,
original query image are reconstructed by selecting informative patches from
both query and gallery image based on similarity between each patch and the
query patches. This module can help our model to tell apart features from
different images, thus fulfilling instance discrimination equipped with better
learned features. To further learn discriminative features especially between
similar classes, we in the _hardness-aware_ module propose selecting the query
and gallery images from two classes that are globally most similar to each
other. Specifically, we formulate this problem into a Travelling Salesman
Problem (TSP) on the distance graph, in which categories and negative
similarity among them are formulated as nodes and edges of the graph. Equipped
with such a mixture strategy, we can control the hardness of training
episodes, thus leading to more robustness and better representations.

Moreover, based on CACTUs[13], our PatchMix can be well adapted to
unsupervised learning FSL (_i.e._ , only unlabeled base data is provided),
which is meaningful when the labeling cost is high. Concretely, the
unsupervised pre-training stage is replaced by our novel PatchMoCo which
constrains the patch-level contrast among different images. Then we involve
our PatchMix into the pseudo-label training stage.

In order to validate the efficacy of our method, we conduct experiments on
single-domain, cross-domain and unsupervised FSL with a wide range of
benchmark datasets including miniImageNet, tieredImageNet, CIFAR-FS and CUB.
Extensive results show that PatchMix can achieve the state-of-the-art
performance on all settings comparing with previous methods, implying the
generalization ability. Besides, we show that such an improvement can be
contributed to the ability of learning causal features, indicated by the
concentration of class of interest in visualized feature map and smaller
intra-variance of learned features among instances from the same category
(_a.k.a_ , neural collapse in [14]).

The contributions of this paper can be listed as follows: (1) We propose a
novel data augmentation method tailored for few-shot learning dubbed PatchMix,
that can remove the spurious correlation, and identify the causal features.
Critically, we leverage the well-established causal framework– _Structural
Causal Model_ to help explain and understand our PatchMix. To the best of our
knowledge, it is the first work that presents a unified causal learning
framework for data augmentation to identify causal features in the few-shot
learning tasks. (2) We enhance our PatchMix model with the newly proposed
_correlation-guided reconstruction_ and _hardness-aware_ modules. These new
modules can facilitate better feature learning. (3) We propose a novel
unsupervised FSL method including a new unsupervised pre-training strategy
named PatchMoCo, which is built upon the PatchMix, and a pseudo-label training
stage. (4) We conduct a vast amount of experiments on different settings and
various datasets. The results verify the effective of our proposed method.

Extensions. This paper is an extension of [15]. We have added the following
aspects based on the conference version: (1) We provide a way to explain how
PatchMix can help few-shot learning in the perspective of learning
disentangled causal features by providing a theoretical interpretation. (2) We
further enhance our PatchMix with two novel modules, _i.e_. correlation-guided
reconstruction and hardness-aware PatchMix. These two modules further improve
the efficacy of our PatchMix model. (3) We empirically show that the proposed
CGR and hardness-aware PatchMix can significantly improve the model.
Meanwhile, our proposed method can receive state-of-the-art results on several
settings. The codes&models will be released on the project pages.

##  2 Related Work

Few-shot recognition. Few-shot learning (FSL) aims to recognize the target
classes by adapting the prior ‘knowledge’ learned from base categories. Such
knowledge usually resides in a deep embedding model for the general-purpose
matching of the support and query image pairs. The embedding is normally
learned with enough training instances on base categories and updated by a few
training instances on novel categories. Recent efforts for FSL are made on
optimization, metric-learning and augmentation.

Optimization based methods [16, 5, 17, 18, 19, 20, 21, 22, 23] learn on the
base dataset a good initialization that can be quickly adapted to novel
dataset. Metric-learning based methods [3, 6, 24, 25, 4, 26, 27, 28, 29, 30,
31, 32, 33, 34, 35, 36, 37] learn a good embedding and an appropriate
comparison metric. Specifically, these methods provide a simple way to
recognize novel data by calculating distance metrics between the image
features and prototypes of each class. Augmentation based methods [38, 39, 40,
41] directly alleviate the dataset gap problems in few-shot learning by the
way of producing various kinds of data. Typically, these augmentation methods
utilize the generative model to create new training samples based on available
ones and random noise or using the augmented data in the testing phase, and
trying to learn a more robust new classifier for each task. Essentially, our
PatchMix can still be categorized as an augmentation method. However, our
method is different from those, as we are the _first_ to leverage augmentation
in learning causal features by removing spurious correlation caused by sample
selection bias, in the few-shot learning scenario. As we will show later, such
a disentanglement of causal features can decrease the variance of inner-class
instances’ features, leading to easier classification among different classes.

Note that some of the former FSL methods like MTL [19] utilize the strategy of
hard sample mining, by taking the classes with low accuracy as the hard class.
In contrast, our hardness-aware PatchMix encourages to mix each class with the
most similar class, such that the generated images can have more distracting
information to help improve the model robustness. Besides, by generating
images from similar classes, the learned causal features can be more
discriminative for classifying between these classes that are too similar (in
terms of their causal features, _e.g._ , to classify the dog and the wolf,
although the learned causal features can concentrate on their shape, they are
similar to each other) to be classified.

Data Augmentation. In the vanilla supervised learning, data augmentation is a
widely used technique to facilitate training the deep models. The naïve
augmentation strategies such as random flip and random crop serves as a data
regularization, and have been applied to many practical topics and learning
tasks [1, 42]. Recently many works [43, 12, 44, 45, 46, 47, 48] are proposed
by mixing or substituting content between images. Specifically, Mixup [43]
mixes two alternative images and the corresponding label. CutMix [12] proposes
to directly replace a randomly selected area. Empirically, their efficacy has
been evaluated on both fully-supervised and semi-supervised classification. In
FSL, some works [49] attempt to utilize manifold Mixup [50] to enhance the
pretraining process. However, these works only directly apply existing data
augmentation methods to FSL. As we will discussion in the following context,
our PatchMix, instead of data regularization, is exclusively effective on
removing spurious correlation for FSL, which cannot be realized by other data
augmentation methods, as shown in the comparison between CutMix and our
PatchMix in Sec. 3.3 in terms of the identification of causal features.

Causal Learning. Due to the invariance property of causal relation, there is
an increasing attention paid at the intersection between causal inference and
machine learning, see [51, 52, 53, 15, 54, 55] for out-of-domain
generalization by removing the confounding bias. These works target on
learning causal semantic features for better generalization, in the framework
of _Structural Causal Model_ (SCM) pioneered by Judea Pearl [56, 57]. Although
it is common for FSL to have distributional shifts between training and test
data due to sampling bias in data collection, few attempts have been made to
address this issue. The work that is most related to us is IFSL [11] that
proposed to remove the bias from pre-trained knowledge by considering an
intervened predictor, in the scenario when the pre-training step is adopted.
Our departure here, is leveraging SCM to explicitly model the spurious
correlation between causal and non-causal features, which can happen due to
sample selection bias even without the pre-training stage. With such a
modeling, we theoretically show that our PatchMix is guaranteed to identify
only the causal semantic features during learning. The empirical comparison of
ours with IFSL are in Sec. 4.3.

##  3 Methodology

Problem Formulation. We formulate few-shot learning in the meta-learning
paradigm. Particularly, the FSL model is learned via the episodes. The episode
should imitate the few-shot learning task: few support and query instances are
sampled from several categories to train/evaluate the embedding model; the
sampled support set is fed to the learner to produce a classifier, and then
the loss and accuracy computed on the sampled query set is used respectively
in training and testing phase. In general, we have two sets of data, namely
meta-train set
$\mathcal{D}_{s}=\left\\{\left(\mathbf{I}_{i},y_{i}\right),y_{i}\in\mathcal{C}_{s}\right\\}$
and meta-test set
$\mathcal{D}_{t}=\left\\{\left(\mathbf{I}_{i},y_{i}\right),y_{i}\in\mathcal{C}_{t}\right\\}$
corresponding to the base and novel dataset, individually. $\mathcal{C}_{s}$
and $\mathcal{C}_{t}$ ($\mathcal{C}_{s}\cap\mathcal{C}_{t}=\emptyset$)
represent base and novel category sets respectively. The goal of FSL is to
train a model on $\mathcal{D}_{s}$ which is well generalized to
$\mathcal{D}_{t}$. As the definition of FSL task, the model can learn from few
(_e.g._ , one or five) labelled data from each category of $\mathcal{C}_{t}$.

We follow the former methods [3, 4] to adopt an $N$-way $K$-shot meta-learning
strategy. Here $N$ denotes the number of categories in one episode and $K$
stands for the number of samples for each category in support set.
Specifically, for each episode $\mathcal{T}$, $N$ categories are randomly
sampled from $\mathcal{C}_{s}$ for training and $\mathcal{C}_{t}$ for testing,
$K$ instances each for these selected categories to construct a support set
$\mathcal{S}=\left\\{\left(\mathbf{I}_{i}^{\mathrm{supp}},y_{i}^{\mathrm{supp}}\right)\right\\}$.
Similarly we sample $M$ query samples per category, and thus construct the
query set
$\mathcal{Q}=\left\\{\left(\mathbf{I}_{i}^{\mathrm{q}},y_{i}^{\mathrm{q}}\right)\right\\}$,
and $\mathcal{S}\cap\mathcal{Q}=\emptyset$. Then the episode can be
represented as $\mathcal{T}=\left\\{\mathcal{S},\mathcal{Q}\right\\}$. In
total each episode has $NK$ support images and $NM$ query images. Note that
while some methods, _e.g._ [3] take different shot number during training and
testing, we keep $K$ the same when training and evaluating our model.

This section is organized as follows. We first introduce a base model called
DProto in Sec. 3.1 to help define our PatchMix. Then we introduce the whole
pipeline of our model which is overviewed in Fig. 2. In particular, two stages
of training are involved. For each stage, query image patches from training
episodes are exchanged via our proposed PatchMix (Sec. 3.2) to identify causal
features (Sec. 3.3). The features of mixed images are then used in few-shot
classification and reconstruction using the correlation-guided reconstruction
(CGR) module (Sec. 3.4.1). In the second stage, each training episode is
further enhanced with hardness-aware PatchMix (Sec. 3.4.2), in which images
from similar categories are mixed to induce more hardness for learning more
discriminative features.

###  3.1 A Base Model by Prototypes

We introduce a simple model derived from ProtoNet [3]. Specifically, given a
few-shot episode $\mathcal{T}$, we first utilize a feature extractor network
$\phi$ to obtain the feature maps of all images in the episode as
$X=\phi(\mathbf{I}),\mathbf{I}\in\mathcal{T}$, where $X\in\mathbb{R}^{c\times
h\times w}$. Then the prototype of the $i$-th class is calculated by taking
the spatial-wise average of all support features belonging to this category,
followed by the sample-wise average:

$p_{i}=\frac{1}{K}\sum_{j=1}^{NK}\bar{X}_{j}^{\mathrm{supp}}\cdot\mathbbm{1}(y_{j}^{\mathrm{supp}}=i),i=1,\cdots,N$
(1)

where $\bar{X}_{j}^{\mathrm{supp}}\in\mathbb{R}^{c}$ is the spatial mean of
$X_{j}^{\mathrm{supp}}$. For each query feature map $X^{\mathrm{q}}$, we build
its prediction confidence map $\hat{X}^{\mathrm{q}}\in\mathbb{R}^{N\times
h\times w}$ in the following way. For $i$-th class and spatial position
indexed by $s,t$,
$\hat{X}^{\mathrm{q}}_{i,s,t}=\frac{<X^{\mathrm{q}}_{:,s,t},p_{i}>}{\|X^{\mathrm{q}}_{:,s,t}\|\|p_{i}\|}$,
which is the normalized cosine distance between the prototype of $i$-th class
and the query feature vector in this position. Meanwhile, a global classifier
$f_{gc}$ consisting of a 1D convolutional layer is applied to $X^{\mathrm{q}}$
to get a prediction map.

This baseline model is a ProtoNet modified in two aspects: (1) We guide the
model with patch-level labels instead of image-level ones to bring the
stronger supervision. (2) We follow the former works [6] to add a global
classifier, which is used in training phase to enhance the supervision. Note
that some existing works propose other kinds of modifications on ProtoNet such
as learnable normalization [58], attention modules [24] and more sophisticated
distance metrics [25]. We do not use these terms so that our model is simple
enough to highlight the effect of our proposed PatchMix. We refer to this
baseline model as DProto in the rest of this paper.

Figure 2: The training framework of our proposed model on a 5-way 1-shot task.
In the first stage of training, for each query image we sample a gallery image
whose random patch is then inserted into the corresponding position of query
image. After feature extraction the classification result is achieved by
comparing the feature vector of each position of query feature map to the
averaged support feature vector, from which the classification loss is
calculated together with the mixed label map. Meanwhile the mixed query and
gallery feature map are processed with the correlation-guided module to
reorganize them into one feature map that can restore the original query
image. In the second stage, the PatchMix is further enhance with the hardness-
aware module which controls the difficulty of mixture based on the distance
among classes.

Training. For each input few-shot pair with inputs and outputs
$\\{X^{q},y^{q},\hat{X}^{q}\\}$ ($y_{q}$ is a one-hot encoded vector for
$|\mathcal{C}_{s}|$ classes), the objective function can be written as follow,

$\displaystyle\mathcal{L}$ $\displaystyle=\ell_{f}+\frac{1}{2}\ell_{g}$ (2)
$\displaystyle\ell_{g}$
$\displaystyle=-(\mathrm{log}\textrm{softmax}(f_{gc}(X^{\mathrm{q}})))^{T}y^{q}$
(3) $\displaystyle\ell_{f}$
$\displaystyle=\frac{1}{hw}\sum_{s,t}\mathrm{log}\frac{e^{-\hat{X}^{\mathrm{q}}_{y^{\mathrm{q}},s,t}}}{\sum_{i=1}^{N}e^{-\hat{X}^{\mathrm{q}}_{i,s,t}}}$
(4)

where $\ell_{f}$ is for the few-shot $N$-way classification, and $\ell_{g}$ is
for the global many-shot $|\mathcal{C}_{s}|$-way (e.g., 64 for miniImageNet)
classification. The feature extractor network $\phi$ is optimized based on
$\mathcal{L}$ across randomly sampled episodes from base data.

Testing. In the testing phase, only the feature extractor $\phi$ is used and
the global classifier $f_{gc}$ is not engaged in testing. For each query image
in an testing episode, we first get the confidence map $\hat{X}^{q}$ following
the above method, then the image-level prediction is achieved by spatially
averaging the confidence map.

###  3.2 Image Augmentation by PatchMix

In this part we formally present PatchMix as an augmentation method tailored
for few-shot learning. Concretely, for each query image
$\\{\mathbf{I}^{\mathrm{q}},y^{\mathrm{q}}\\}$, we randomly sample another
query image as the gallery image $\\{\mathbf{I}^{g},y^{g}\\}$ from which we
collect the information to switch. Next we follow [12] to randomly select a
box with width and height $(\hat{w},\hat{h})$ sampled from

$\displaystyle\lambda$ $\displaystyle\sim\mathrm{Unif}(0,1)$ (5)
$\displaystyle\hat{w}$
$\displaystyle=W\sqrt{1-\lambda},\hat{h}=H\sqrt{1-\lambda},$ (6)

where $W$, and $H$ are the width and height of the images. The center
coordinate $(c_{w},c_{h})$ sampled from
$c_{w}\sim\mathrm{Unif}(\lceil{\hat{w}/2}\rceil,W-\lceil{\hat{w}/2}\rceil)$,
$c_{h}\sim\mathrm{Unif}(\lceil{\hat{h}/2}\rceil,H-\lceil{\hat{h}/2}\rceil)$.
Denote $w_{1},w_{2},h_{1},h_{2}$ as the left, right, lower and upper boundary
of the box $(c_{w},c_{h},\hat{w},\hat{h})$. Then we can generate the

the mask $M$ and the mixed image $\tilde{\mathbf{I}}^{\mathrm{q}}$ as:

$\displaystyle M_{i,j}$
$\displaystyle=\left\\{\begin{array}[]{rcl}1&&{w_{1}\leq i\leq w_{2},h_{1}\leq
j\leq h_{2}}\\\ 0&&{o.w.}\\\ \end{array}\right.$ (9)
$\displaystyle\tilde{\mathbf{I}}^{\mathrm{q}}$
$\displaystyle=M\odot\mathbf{I}^{g}+(1-M)\odot\mathbf{I}^{\mathrm{q}}$ (10)

In comparison with CutMix [12] which adopts an image-level soft label as the
ground truth, we still keep the patch-level hard labels to be the supervision
information. We first interpolate the selected box to size of $w\times h$ as

$\displaystyle w_{1}^{{}^{\prime}},w_{2}^{{}^{\prime}}$
$\displaystyle=\frac{w}{W}w_{1},\frac{w}{W}w_{2}$ (11) $\displaystyle
h_{1}^{{}^{\prime}},h_{2}^{{}^{\prime}}$
$\displaystyle=\frac{h}{H}h_{1},\frac{h}{H}h_{2}$ (12)

Then the new label map is set as

$Y_{i,j}=\left\\{\begin{array}[]{rcl}y^{g}&&{w_{1}^{{}^{\prime}}\leq i\leq
w_{2}^{{}^{\prime}},h_{1}^{{}^{\prime}}\leq j\leq h_{2}^{{}^{\prime}}}\\\
y_{q}&&{o.w.}\\\ \end{array}\right.$ (13)

Finally the mixed image $\tilde{\mathbf{I}}^{\mathrm{q}}$ is fed into the
network, classified by method mentioned in Sec. 3.1 and guided by the label
map $Y$. In the subsequent section, we will introduce why such a simple
operation can help learn causal features for better generalization.

###  3.3 Causal Explanation of PatchMix

In this section, we explain the effectiveness of PatchMix in removing the
spurious correlation from the learned representation, in the framework of
_Structural Causal Model_ (SCM) [56]. To describe the causal relations over
all the variables, the SCM incorporates a directed acyclic graph (DAG)
$G=(\mathcal{V},\mathcal{E})$, such that $X\to Y\in\mathcal{E}$ means $X$ has
a direct causal effect to $Y$. Due to its ability to encode priories beyond
data, it has been increasingly leveraged to disentangle the _causal semantic
features_ [53, 15] from other features for out-of-distribution generalization.
Inspired by such a spirit, we provide an ad-hoc analysis to explain how the
PatchMix operator can remove the spurious correlation and only keep causal
features for prediction.

Figure 3: Causal graph of models (a) with sample selection bias, (b) using
PatchMix for patches consisting of single image, (c) using PatchMix for
patches consisting of two images, (d) using Patchmix for patches whose causal
features are completely removed and (e) using CutMix. The $S,Z$ respectively
denote the causal and non-causal features. $Y$ indicates the class label. $C$
is an indicator for train/test set. Red arrows mean the resource of spurious
correlation between class label and non-causal features. The patch image with
PatchMix can generated either from single image or two images, as respectively
marked by the dot blue boxes. For both types, the PatchMix can remove the
dependency from sampling bias. The CutMix replaces the original label $Y$ with
an interpolated one $Y_{c}$; hence it is determined by features from both
images.

The data collection procedure can often incur sampling bias, inducing the
dependency among some originally independent concepts, _e.g._ , the dog is
associated with the grass more often than with the river in the collected
dataset. This can cause the _spurious_ correlation between causal features and
non-causal ones, leading to non-causal features learned during training, which
can hurt the performance.

We formulate this spurious correlation in Fig. 3 (a), in which $S,Z$
respectively denote the causal (_i.e._ , the texture, shape of the dog) and
non-causal features (_i.e._ , features related to other objects such as
people) in terms of its relation with the outcome $Y$. The label $Y$ and non-
causal features $Z$ altogether generates the sampling procedure denoted as
$C$, _e.g._ , the sampler collected the dog (_i.e._ , outcome $Y$) on the
grass (_i.e._ , $Z$) in some communities when people walked with their pets.
Here, the $C$ with $C=1$ (or $C=0$) means that the corresponding sample is in
the training (or test) set. As $C$ is the collider on the path $S\leftarrow
Y\to C\leftarrow Z$, it induces the correlation between $S$ and $Z$, _i.e._ ,
$S\not\perp Z|C$. Moreover, for any sample $I$, the $Y\not\perp Z|S,I,C=1$,
making the non-causal features learned during training process. However, it
may fail to generalize as $P(S,Z|C=1)$ may not equal to $P(S,Z|C=0)$.

On the other hand, as shown in Fig. 3 (b),(c),(d), we may have three cases of
patch images after PatchMix operation: i) Figure 3 (b) contains only causal
features; ii) in Fig. 3 (c), part of both causal and non-causal features are
replaced by others; iii) in Fig. 3 (d), all causal information is removed.

For those patches in $\mathbf{I}^{q}$ that contain causal information, the
case described in Fig. 3 (b) can naturally decrease the influence of non-
causal features. Thus such a case is less correlated with the spurious
features, as it directly removes features from other patches that may be
spurious. For the case described in Fig. 3 (c), the generated patch integrates
$S$ and $Z$ randomly from two different images, breaking the original sampling
procedure and thus the selection bias it induces. In this regard, the causal
features $S$ no longer depends on $Z$. Formally speaking, we can have $Y\perp
Z|S,I$ for these patches in the training set, making it possible to eliminate
the non-causal features if the model is trained well.

Note that our Patchmix can also handle the extreme cases that some patches of
the ”causal features” are completely removed, which corresponds to the graph
in Fig. 3 (d). Theoretically our PatchMix can still reduce the correlation
between spurious features and causal features. Specifically, equipped with
PatchMix, these patches also randomly mix information from other patches. In
this regard, the patterns of original non-causal features are broken, and are
randomly replaced by others, making correlations between these non-causal
features and $Y$ no longer exist. In this way, compared to vanilla patch-level
training methods without any mixing strategies (_i.e._ , base model) or with
other mixing strategies, our PatchMix can enforce the model to pay more
attention to causal features, and remove spurious correlation of other non-
causal features. This result can be verified by a noticeable improvement of
our method over the base models in Tab. I in Tab. V, additionally with
interpretable visualization results in Fig. 7.

######  Theorem 3.1 (Disentangling Causal Features).

Suppose the neural network parameter is composed of two parts:
$(\psi,\theta)$, where $\psi:\mathcal{I}\to\mathbb{R}^{\mathrm{dim}(S)}$
extracts a representation $\psi(I)$ from the image, followed by $\theta$ for
predicting the label. We denote $p^{o}(\psi,\theta)(y|I)$ and
$p^{\mathrm{patch}}(\psi,\theta)(y|I)$ as the trained model on the original
data (with Fig. 3 (a)) and the one on the data after PatchMix (with Fig. 3
(b),(c),(d)). Assume that the structural equation for the image $I$ is
injective, _i.e._ , $I\leftarrow f_{I}(S,Z)$, then if both
$p^{o}(\psi,\theta)(y|I)$ and $p^{\mathrm{patch}}(\psi,\theta)(y|I)$ are
trained to perfectly fit the ground-truth distribution $p^{*}(y|I)$, we have:

  * •

In $p^{o}(\psi,\theta)(y|I)$, the $\psi^{o}(I)$ is dependent on the non-causal
features $Z$.

  * •

In $p^{\mathrm{patch}}(\psi,\theta)(y|I)$, it is with Lebesgue measure 0 for
$\psi^{\mathrm{patch}}(I)$ to depend on $Z$.

######  Remark 3.2.

The injective assumption on $f_{I}$ is widely assumed in the literature, such
as (variational) auto-encoder [59] since it has been empirically verified that
the image can be recovered perfectly from its latent embedding; causal
inference [60, 61] for identifiability consideration and few-shot learning
[62].

###### Proof.

For both $p^{o}(\psi,\theta)(y|I)$ and $p^{\mathrm{patch}}(\psi,\theta)(y|I)$,
we have $Y\perp I|\psi(I)$ since $\psi(I)$ blocks all paths in the neural
network from the input $I$ to the output $Y$, as the predicted label is the
same to the ground-truth label $Y$ since both $p^{o}(\psi,\theta)(y|I)$ and
$p^{\mathrm{patch}}(\psi,\theta)(y|I)$ equal to $p^{*}(y|I)$. According to
Fig. 3 (a), we have $Y\not\perp I|S$, therefore if $\psi^{o}(I)$ does not
depend on $Z$, _i.e._ , $\psi^{o}(I)=h(S)$ for some $h$, it fails to make $Y$
and $I$ conditionally independent. On the other hand, it is sufficient for
$\psi^{\mathrm{patch}}(I)$ to depend only on $S$ to make $Y\perp
I|\psi^{\mathrm{patch}}(I)$. Suppose $\psi^{\mathrm{patch}}(I)=h(S,Z)$ for
some $h$. For the patch from a single whole image (marked by dot blue box in
Fig. 3 (b)), the representation can only contain the causal features $S$ as it
removes other spurious correlated features. To show $h$ is independent to $Z$
for the patch as the mixture of two images (marked by Fig. 3 (c)), recall that
$p^{\mathrm{patch}}_{\theta}(y|\psi^{\mathrm{patch}}(I))=p^{*}(y|[f_{I}^{-1}]_{\mathcal{S}}(I))=p^{*}(y|s)$
with $I=f_{I}(s,z)$ and the injective assumption for $f_{I}$, we have that for
each $s$ and two different $z_{1},z_{2}$, we have
$p^{\mathrm{patch}}_{\theta}(y|h(s,z_{1}))=p^{\mathrm{patch}}_{\theta}(y|h(s,z_{2}))=p^{*}(y|s)$,
which means that the $p^{\mathrm{patch}}_{\theta}(y|h(s,z_{1}))$ does not
depend on $z$. Since it is with Lebesgue measure 0 for the parameter $\theta$
to be 0 on $h(S,Z)$, which means an unblocked path from $Z$ to $Y$ if $h$
depends on $Z$, which comes up a contradiction. Therefore, it is with Lebesgue
measure 0 to let $h$ depend on the non-causal feature $Z$. For Fig. 3 (d), as
original non-causal features $Z$ are randomly mixed with other features, the
correlation between $Z$ and $Y$ is thus broken. ∎

This conclusion means it generically holds for the learned representation to
eliminate the information of non-causal features. This can explain the benefit
of PatchMix in removing the non-causal features during learning, by breaking
the dependency between the causal features and the non-causal ones.

Comparison with CutMix. The difference of our method with CutMix is that the
latter adopts a soft label (namely $Y_{c}$) to integrated images (as shown in
Fig. 3 (e)), which is determined by the label from both images. Therefore,
such a soft label $Y_{C}$ is related to both features from two different
images. In this regard, $Y\not\perp Z|S,I$ for each $I$, leading to the non-
disentanglement between $S$ and $Z$. Therefore, this learning mechanism is not
endowed with disentanglement ability that can be helpful for transferring to
novel categories, as summarized in the following:

######  Theorem 3.3.

Denote $p^{\mathrm{cut}}(\psi,\theta)(y|I)$ as the trained model on the data
after CutMix (with Fig. 3 (d)). Under the same injective assumption on
$f_{I}$, we have that the $\psi^{\mathrm{cut}}(I)$ is dependent on features
$Z$, if $p^{\mathrm{cut}}(\psi,\theta)(y|I)$ is trained to equal to the
ground-truth distribution $p^{*}(y|I)$.

###### Proof.

Similar to the proof of Thm. 3.1, we have $Y_{c}\perp
I|\psi^{\mathrm{cut}}(I)$, since in Fig. 3 (d), we have $Y\not\perp Z|S,I$.
Thus, it is necessary for $\psi^{\mathrm{cut}}(I)$ to depend on both $S$ and
$Z$. Further, if the spurious correlation is strong enough, it is incapable to
disentangle $S$ from $Z$ as they play nearly symmetric roles in affecting $Y$
and generating the input $I$. ∎

Relationship with neural collapse. As introduced in Sec. 1, a recent work [14]
studies the neural collapse that the variance of features from the novel
category is small enough to separate different categories; and it claims that
the property of neural collapse can be well transferred to novel data given
sufficient base training data. Intuitively, the learned disentangled causal
features can better correlate with novel class labels compared with the non-
causal features. Therefore, the model trained with PatchMix is expected to
receive better neural collapse on novel data since PatchMix can help break up
the dependency between causal and non-causal features during training.

Figure 4: The module for correlation-guided reconstruction. Each position of
the mixed query and gallery feature map is compared with the known area from
the original image. Then the similarity is further normalized with gumbel-
softmax to sample an action map that is used to select the patches.

###  3.4 Discriminative Feature Enhancement

Although the non-causal features can be eliminated, it is not necessarily for
these learned causal features to be discriminative enough for classification
between similar classes. For example, the shape feature is causally related to
both the dog and the wolf; however, the learned feature from two classes may
be too similar to be classified. To make our causal features be more
discriminative, we further propose two modules, i.e., _correlation-guided
reconstruction_ module in Sec. 3.4.1 and _hardness-aware PatchMix_ module in
Sec. 3.4.2, respectively for instance discrimination and classifying between
similar classes.

####  3.4.1 Correlation-guided Reconstruction Module

Inspired by recent works on self-supervised learning [63], we propose a novel
module while takes advantage of PatchMix procedure to contrast image features
from different categories. Specifically, given query image
$\mathbf{I}^{\mathrm{q}}$, gallery image $\mathbf{I}^{\mathrm{g}}$ and the
corresponding mixture mask $M$, we can have the resulting mixed image
$\tilde{\mathbf{I}}^{\mathrm{q}}$ and its feature map
$\tilde{X}^{\mathrm{q}}$. Meanwhile, by exchanging the role of
$\mathbf{I}^{\mathrm{q}}$ and $\mathbf{I}^{\mathrm{g}}$ and using $1-M$ as
mask, we can have image $\tilde{\mathbf{I}}^{g}$ and feature map
$\tilde{X}^{g}$ which contain the counterpart of
$\tilde{\mathbf{I}}^{\mathrm{q}}$. In this way, $\mathbf{I}^{\mathrm{q}}$ and
$\mathbf{I}^{\mathrm{g}}$ can be properly reconstructed by selecting correct
patches from these two feature maps. The selection process can be based on
successfully telling apart patch-level features from different images.
Therefore, we make the reconstruction of both of query images and gallery
images as the goal of this module.

Take the $(i,j)-$th patch in $\tilde{\mathbf{I}}^{\mathrm{q}}$, we evaluate
the confidence of it belonging to $\mathbf{I}^{\mathrm{q}}$ and
$\mathbf{I}^{\mathrm{g}}$ as

$\displaystyle\alpha^{\mathrm{q}}_{i,j}$
$\displaystyle=\sum_{m,n=1,1}^{h,w}\sum_{m,n\neq
i,j}M_{m,n}\cdot<\tilde{f}_{m,n}^{\mathrm{q}},\tilde{f}_{i,j}^{\mathrm{q}}>$
(14) $\displaystyle\alpha^{g}_{i,j}$
$\displaystyle=\sum_{m,n=1,1}^{h,w}\sum_{m,n\neq
i,j}M_{m,n}\cdot<\tilde{f}_{m,n}^{\mathrm{g}},\tilde{f}_{i,j}^{q}>$ (15)

In other words, we compare each patch to the known patches from
$\mathbf{I}^{\mathrm{q}}$ except itself. The confidence for belonging to class
$B$ can be calculated in the same way by replacing $M$ and
$\tilde{f}_{m,n}^{\mathrm{q}}$ with $1-M$ and $\tilde{f}_{m,n}^{g}$,
respectively. Then we can select these patches according to the confidence,

$\displaystyle\bar{X}^{\mathrm{q}}_{i,j}$
$\displaystyle=\hat{\alpha}^{g}_{i,j}\tilde{X}_{i,j}^{\mathrm{g}}+\hat{\alpha}^{\mathrm{q}}_{i,j}\tilde{X}_{i,j}^{q}$
(16) $\displaystyle\hat{\alpha}^{g}_{i,j}$
$\displaystyle=\sigma(\alpha^{\mathrm{q}}_{i,j}/T)$ (17)
$\displaystyle\hat{\alpha}^{\mathrm{q}}_{i,j}$
$\displaystyle=\sigma(\alpha^{\mathrm{q}}_{i,j}/T)$ (18)

where $T$ is temperature and $\sigma$ is a normalization function implemented
as gumbel-softmax [64]. After obtaining the merged feature
$\bar{X}^{\mathrm{q}},\bar{X}^{g}$, we process them with a decoder, whose
structure is a reversal of the feature extractor, to generate the
reconstructed images $\bar{\mathbf{I}}^{\mathrm{q}},\bar{\mathbf{I}}^{g}$.

The objective function of this module can be formulated as

$\displaystyle L_{CR}$ $\displaystyle=L_{sel}+\lambda L_{rec}$ (19)
$\displaystyle L_{sel}$
$\displaystyle=\mathrm{CE}(\hat{\alpha}^{\mathrm{q}},M)+\mathrm{CE}(\hat{\alpha}^{g},1-M)$
(20) $\displaystyle L_{rec}$
$\displaystyle=\|\hat{A}-A\|_{1}+\|\hat{B}-B\|_{1}$ (21)

It is noteworthy that unlike the common way in conditional image generation,
our CGR does not directly concatenate $\bar{X}^{\mathrm{q}}$ and $\bar{X}^{g}$
as input. Also, we do not adopt techniques like adaptive normalization [65].
Compared with reconstruction-based method, which was empirically shown [66] to
be not helpful for classification, we additionally separate the features from
different classes by successfully distinguish query images and their gallery
counterpart which have different labels. Without the correlation guidance, the
task can easily degenerate to the task of repairing images _e.g._ , image
inpainting, which unfortunately may not be necessarily helpful to few-shot
classification.

####  3.4.2 Hardness-aware PatchMix Module

Since all query images are mixed with random sampled images from the sample
episode, the vanilla PatchMix lacks control of difficulty, thus it may be even
difficult for learned causal features to be discriminative enough, especially
between similar classes. To resolve this problem, we propose mixing images
from similar classes, in which the similarity among base categories can be
estimated by a coarse prediction model with PatchMix in the first stage.

Specifically, given an episode with support set $\mathcal{S}$ and a trained
feature extractor $\phi$, we can first get the prototype of each category
$p_{i},i=1,\cdots,N$ in the same way as Eq. 1. Then the similarity between
$i$-th class and $j$-th class can be represented by the normalized cosine
similarity between their prototypes:

$\mathrm{sim}(i,j)=\frac{<p_{i},p_{j}>}{\|p_{i}\|\|p_{j}\|}$ (22)

In this way, we can treat the relationship among the $N$ categories as a
complete $N$-node graph whose edges are the negative similarity of the
corresponding prototypes. Then we solve a Travelling Salesman Problem (TSP) by
setting a random class as the starting point, which results in a path with the
lowest cost, thus can generate images from classes that are easy to be
obfuscated owing to high similarity. Each node is assigned as the gallery
class to its parent node, followed by the PatchMix training scheme as
introduced before. The objective function of hardness-aware PatchMix can be
written as

$\displaystyle\mathcal{L}_{ha}$ $\displaystyle=\mathcal{L}+\ell_{kd}$ (23)
$\displaystyle\ell_{kd}$
$\displaystyle=\frac{1}{NQ}\sum_{i=1}^{NQ}\psi(F(\mathbf{I}_{i}^{\mathrm{q}}),\hat{F}(\mathbf{I}_{i}^{\mathrm{q}}))$
(24)

where $\mathcal{L}$ is defined in Eq. 2 and $\psi$ is a knowledge distillation
loss [67], which can be implemented as MSE or KL divergence.

In all, the workflow of our algorithm can be separated into two stage, as
stated in Alg. 1, including training the baseline model in Sec. 3.1 together
with PatchMix and CGR in the first stage and hardness-aware PatchMix in the
second stage.

Algorithm 1 Training FSL Model with PatchMix

0: $\mathcal{D}_{s}$: meta-train set

1: # coarse PatchMix

2: while not converge do

3: Sample batch of episodes $B$ from $\mathcal{D}_{s}$

4: for all episode do

5: for all query sample $\mathbf{I}_{i}^{\mathrm{q}}$ do

6: Apply PatchMix to $\mathbf{I}_{i}^{\mathrm{q}}$ according to Eq. 9

7: end for

8: end for

9: Calculate objective function $\mathcal{L}$ as in Eq. 2

10: Update model parameters according to $\mathcal{L}$

11: end while

12:

13: # hardness-aware PatchMix

14: while not converge do

15: Sample batch of episodes $B$ from $\mathcal{D}_{s}$

16: for all episode do

17: calculate class-wise similarity according to Eq. 22

18: assign gallery class

19: for all query sample $\mathbf{I}_{i}^{\mathrm{q}}$ do

20: Apply PatchMix to $\mathbf{I}_{i}^{\mathrm{q}}$ according to Eq. 9

21: end for

22: end for

23: Calculate objective function $\mathcal{L}$ as in Eq. 23

24: Update model parameters according to $\mathcal{L}$

25: end while

###  3.5 Adaptation to Unsupervised FSL

In few-shot learning, it is sometimes hard to get any labeled samples,
especially when the labeling cost is high. To resolve this problem, some
recent works [13, 68] attempt to emphasize unsupervised few-shot learning.
These works firstly adopt unsupervised learning methods to train the networks,
and then the trained networks are used for testing. For instance, CACTUs [13]
selects several unsupervised learning methods such as DeepCluster [69] to get
a representation vector for each sample. These representation vectors are then
utilized to cluster the samples so that each sample can be assigned by a
pseudo label. After obtaining the pseudo label for each sample, we can apply
some few-shot learning methods to train the networks.

Following the standard practice above, this section further extends PatchMix
to unsupervised FSL. Specifically, we adopt CACTUs-ProtoNet as our baseline.
There are two main components for CACTUs, _unsupervised pretraining_ and
_pseudo label training_. Our adaptation considers using PatchMix for both
parts.

(1) Unsupervised pretraining. We take into account the current progress in
unsupervised learning and employ a modified version of MoCo [63] based on our
PatchMix. For the original version of MoCo, a batch of sampled images are
augmented to generate two sets of images called the key images and the query
images. For each query image $\mathbf{I}^{\mathrm{q}}_{i}$, the key image
$\mathbf{I}^{key}_{i}$ that shares the same image before augmentation is its
positive sample and other key images are negative samples. These images are
fed into networks to get spatially averaged feature vectors, in which vectors
of the $i-$th key image are denoted as $\bar{X}_{i}^{key}\in\mathbb{R}^{N}$
and vectors of the $i-$th query image are denoted as
$\bar{X}_{i}^{\mathrm{q}}\in\mathbb{R}^{N}$. Here we use
$\hat{X}^{\mathrm{q}}_{ij}=\frac{<\bar{X}_{i}^{q},\bar{X}_{j}^{key}>}{\|\bar{X}_{i}^{q}\|\|\bar{X}_{j}^{key}\|}$
to denote the cosine similarity between two features. For each batch of query
images, we calculate its loss with the form

$\mathcal{L}_{MoCo}=\sum_{i}-\mathrm{log}\frac{e^{\hat{X}^{\mathrm{q}}_{ii}/T}}{\sum_{j}e^{\hat{X}^{\mathrm{q}}_{ij}/T}}$
(25)

where $T$ is the temperature.

To implement our modified version which we call PatchMoCo, we first consider a
dense variant of MoCo. We remove the global average pooling layer for query
features and get the spatial feature $X_{i}^{\mathrm{q}}\in\mathbb{R}^{N\times
h\times w}$. Each spatial position of $X_{i}^{\mathrm{q}}$ is denoted as
$X_{i,s,t}^{\mathrm{q}}\in\mathbb{R}^{N}$. Similarly the cosine similarity for
each position $s,t$ is
$\hat{X}^{\mathrm{q}}_{istj}=\frac{<{X}_{ist}^{q},\bar{X}_{j}^{key}>}{\|{X}_{ist}^{q}\|\|\bar{X}_{j}^{key}\|}$.
The altered loss has the following form,

$\mathcal{L}_{d}=\sum_{i}\sum_{st}-\mathrm{log}\frac{e^{\hat{X}^{\mathrm{q}}_{isti}/T}}{\sum_{j}e^{\hat{X}^{\mathrm{q}}_{istj}/T}}$
(26)

Afterwards, we add our PatchMix into the unsupervised pretraining. The process
of mixing patches is as the way in Sec. 3.2. We modify the form of loss
function. If some of its patches are switched for each image, these new
patches are negative samples. For these new patches, we cannot find their
positive samples, so we do not include them into the calculation of final
loss. For convenience, we get a mask $K^{\mathrm{q}}_{i}\in\\{0,1\\}^{h\times
w}$, where 1 means patch without switch and 0 means switched patch. The new
loss function is,

$\mathcal{L}_{PMC}=\sum_{i}\sum_{st}-K^{\mathrm{q}}_{ist}\mathrm{log}\frac{e^{\hat{X}^{\mathrm{q}}_{isti}/T}}{\sum_{j}e^{\hat{X}^{\mathrm{q}}_{istj}/T}}$
(27)

(2) Pseudo-label training: After we get pseudo-label by clustering the feature
of unsupervised pretraining, each training sample can be used as the same in
supervised FSL. Then PatchMix is applied to the training phase as in Sec. 3.2.

##  4 Experiments

###  4.1 Datasets and setting

Datasets. We mainly adopt four datasets for our experiments. i) miniImageNet
dataset [70], containing 600 images in each of the 100 categories, is a small
subset of ImageNet. We follow the split in [16], where 64, 16, 20 categories
are used for train, validation and test set, respectively. ii) tieredImageNet
dataset [71] is a larger subset of ILSVRC-12 dataset. It consists of 34
categories with 779,165 images in total. These categories are further broken
into 608 categories, where 351 categories are used for training, 97 for
validation and 160 for testing. iii) CIFAR-FS [72] divides CIFAR-100 into 64
meta-train categories, 16 meta-val categories and 20 meta-test categories. iv)
CUB [73], a bird dataset with 200 total categories and 6033 total images. In
few-shot learning, 100, 50, 50 species are respectively used for training,
validation and test sets. Besides, Cars [74], Places [75] and Plantae [76] are
also utilized in cross-domain setting following [77]. Images in all datasets
are resized to $84\times 84$ before training and testing.

Experimental setup. Stochastic Gradient Descent (SGD) [78] with $5e-4$ weight
decay and cosine learning rate decay [79] is used to optimize our model. For
miniImageNet and CIFAR-FS, the initial learning rate is set as $0.15$ and
$0.05$ for tieredImageNet. Random cropping, horizontal flipping and color
jittering are adopted for data augmentation during training, which is the same
as in CAN [6]. We test 2000 episodes sampled from meta-test set for all
experiments. For correlation-guided reconstruction, $\lambda_{1}$ is set as
$0.5$ for all datasets, $\lambda_{2}=0.1$ for tieredImageNet and $0.25$ for
other datasets. As for the hardness-aware PatchMix, we empirically set the
distillation function $\psi$ as MSE for 1-shot tasks and KL divergence for
5-shot tasks. Models and codes will be released.

###  4.2 Comparison with state-of-the-art methods

To extensively show the effectiveness of our method, we test our PatchMix
across three commonly-used settings, _i.e._ , single domain, cross domain and
unsupervised few-shot learning. For each setting, we compare our model with
the recent state-of-the-art competitors, where the average accuracy is
reported. For supervised learning settings, we additionally report the 95%
_confidence interval_ (CI).

Model | Backbone |  miniImageNet
---|---|---
1-shot | 5-shot
ProtoNet [3] | Conv4 | 49.42$\pm$0.78 | 68.20$\pm$0.72
MatchingNet [70] | 43.56$\pm$0.84 | 55.31$\pm$0.73
RelationNet [4] | 50.44$\pm$0.82 | 65.32$\pm$0.70
MAML [5] | 48.70$\pm$1.75 | 63.11$\pm$0.92
Dynamic Few-shot [80] | 56.20$\pm$0.86 | 72.81$\pm$0.62
LEO [20] | WRN-28 | 61.76$\pm$0.08 | 77.59$\pm$0.12
PPA [81] | 59.60$\pm$0.41 | 73.74$\pm$0.19
Robust dist++ [82] | 63.28$\pm$0.62 | 81.17$\pm$0.43
wDAE [83] | 61.07$\pm$0.15 | 76.75$\pm$0.11
CC+rot [84] | 62.93$\pm$0.45 | 79.87$\pm$0.33
DC [41]∗ | 67.96$\pm$0.45 | 83.45$\pm$0.31
FEAT [24] | 65.10$\pm$0.20 | 81.11$\pm$0.14
TapNet [85] | Res-12 | 61.65$\pm$0.15 | 76.36$\pm$0.10
MetaOptNet [86] | 62.64$\pm$0.61 | 78.63$\pm$0.46
CAN [6] | 63.85$\pm$0.48 | 79.44$\pm$0.34
FEAT [24] | 66.78$\pm$0.20 | 82.05$\pm$0.14
E3BM [87] | 63.80$\pm$0.40 | 80.10$\pm$0.30
DSN-MR [88] | 64.60$\pm$0.72 | 79.51$\pm$0.50
Net-Cosine [89] | 63.85$\pm$0.81 | 81.57$\pm$0.56
FRN [30] | 66.45$\pm$0.19 | 82.83$\pm$0.13
Tian et.al. [90] | 64.82$\pm$0.60 | 82.14$\pm$0.43
ConstNet [91] | 64.89$\pm$0.23 | 79.95$\pm$0.37
IEPT [29] | 67.05$\pm$0.44 | 82.90$\pm$0.30
MELR [28] | 67.40$\pm$0.43 | 83.40$\pm$0.28
DMF [92] | 67.76$\pm$0.46 | 82.71$\pm$0.31
DeepEMD [25] | 68.77$\pm$0.29 | 84.13$\pm$0.53
Base Model | Res-12 | 64.96$\pm$0.51 | 80.51$\pm$0.33
Ours | 69.38$\pm$0.46 | 84.14$\pm$0.30
Ours+DC | 69.75$\pm$0.44 | 84.88$\pm$0.30
TABLE I:  5-way few-shot accuracies with $95\%$ confidence interval on miniImageNet, * denotes results reproduced by us. Model |  tieredImageNet | CIFAR-FS
---|---|---
1-shot | 5-shot | 1-shot | 5-shot
CC+rot [84] | 70.53$\pm$0.51 | 84.98$\pm$0.36 | 75.38$\pm$0.31 | 87.25$\pm$0.21
DC [41]∗ | 74.05$\pm$0.48 | 88.30$\pm$0.32 | — | —
MetaOptNet [86] | 65.99$\pm$0.72 | 81.56$\pm$0.53 | 72.11$\pm$0.96 | 84.32$\pm$0.65
CAN [6] | 69.89$\pm$0.51 | 84.23$\pm$0.37 | — | —
FEAT [24] | 70.80$\pm$0.23 | 84.79$\pm$0.16 | — | —
E3BM [87] | 71.20$\pm$0.40 | 85.30$\pm$0.30 | — | —
DSN-MR [88] | 67.39$\pm$0.82 | 82.85$\pm$0.56 | — | —
FRN [30] | 72.06$\pm$0.25 | 86.89$\pm$0.14 | — | —
Tian et.al. [90] | 71.52$\pm$0.69 | 86.03$\pm$0.49 | 73.90$\pm$0.80 | 86.90$\pm$0.50
ConstNet [91] | — | — | 75.40$\pm$0.20 | 86.80$\pm$0.20
IEPT [29] | 72.24$\pm$0.50 | 86.73$\pm$0.34 | — | —
MELR [28] | 72.14$\pm$0.51 | 87.01$\pm$0.35 | — | —
DMF [92] | 71.89$\pm$0.52 | 85.96$\pm$0.35 | — | —
DeepEMD [25] | 74.29$\pm$0.32 | 87.08$\pm$0.60 | — | —
Ours | 73.48$\pm$0.51 | 87.35$\pm$0.32 | 77.87$\pm$0.49 | 88.94$\pm$0.32
Ours+DC | 75.06$\pm$0.48 | 88.92$\pm$0.32 | — | —
TABLE II:  5-way few-shot accuracies with $95\%$ confidence interval on
tieredImageNet and CIFAR-FS., * denotes results reproduced by us.

Single-domain results. We adopt three datasets including miniImageNet,
tieredImageNet and CIFAR-FS in the single domain setting, where the model is
trained on the meta-train set of each dataset and tested on the corresponding
meta-test set. The results are shown in Tab. I and Tab. II.

In both settings on miniImageNet, our model surpasses all competitors that
share the same backbone with us. Particularly, we improve over DeepEMD v2 [25]
(_i.e._ , the current state-of-the-arts method) by 0.61% in 1-shot setting.
Besides, our methods also performs better (_e.g._ , outperforms FEAT by 4.28%
on 1-shot and 3.03% on 5-shot), even compared with those with WRN-28 that is
much larger and hence has more capacity than Res-12. We observe that this
improvement is smaller on 5-shot setting, which may due to extra parameters
(_e.g._ , attention module in FEAT and meta-filter in DMF) or specifically-
designed but computational expensive algorithm (_e.g._ , the Earth Mover
Distance in DeepEMD) that can perform better with more support data. In
contrast, our Patchmix, which has the similar procedure of testing with the
basic ProtoNet, is more effective and efficient, in terms of prediction power
and implementation. This can be contributed to the better representation (more
specifically, causally semantic features as will be shown in Sec. 4.3) for
metric-based classification. Meanwhile, our model also enjoys a suitable
confidence interval, which means the robustness against episodes with
different categories and difficulty.

On tieredImageNet and CIFAR-FS the results are nearly consistent with those on
miniImageNet (_e.g._ , our model leads by 2.47% and 1.69% in 1-shot and 5-shot
on CIFAR-FS). Besides, our method is flexible to be further improved when
combined with other orthogonal methods, as implied by the improvement when
combined with DC (_i.e._ , Ours+DC) which enables the data augmentation on
both training and testing phases. Particularly, such a combined method
achieves the best performance on tieredImageNet111Note that as the pre-trained
weight for tieredImageNet is not released in DC, we reproduce it by the
(official DC codes. We also adopt the weight provided in S2M2 [49] (link)
strictly following [41]..

Cross-domain results. In this setting, we follow the previous methods to train
our model on the meta-train set of miniImageNet and test it on the meta-test
set of CUB, Cars, Places and Plantae. Since these datasets are fine-grained
ones, successful classification mainly requires the model to be able to
concentrate on some details which may not be useful in miniImageNet. Therefore
the collapsing base IV can have a more serious damage in such a setting. As
shown in Tab. III, our model outperforms the best competitor by at most 5.82%
on 1-shot and 3.14% on 5-shot among these datasets. This means that when
trained on miniImageNet, our model can learn the domain specific information,
but also additional knowledge that is useful in other domains.

Unsupervised few-shot learning results. In this setting, we use miniImageNet
as the target dataset. The available information is the same as that in
supervised single domain FSL except that labels are not assigned to the base
category images. We compare our method in Tab. IV with CACTUs [13] and UMTRA
[68]. As mentioned in Sec. 3.5, our model is built based on CACTUs-ProtoNet
with a changed cluster method, which results in a 1.80% and 2.56% improvement
on both 1-shot and 5-shot tasks. The superiority further reflects the efficacy
of our proposed PatchMix.

Model | CUB | Cars | Places | Plantae
---|---|---|---|---
1-shot | 5-shot | 1-shot | 5-shot | 1-shot | 5-shot | 1-shot | 5-shot
RelationNet [4] | 42.44$\pm$0.77 | 57.77$\pm$0.69 | 29.11$\pm$0.60 | 37.33$\pm$0.68 | 48.64$\pm$0.85 | 63.32$\pm$0.76 | 33.17$\pm$0.64 | 44.00$\pm$0.60
GNN [93] | 45.69$\pm$0.68 | 62.25$\pm$0.65 | 31.79$\pm$0.51 | 44.28$\pm$0.63 | 53.10$\pm$0.80 | 70.84$\pm$0.65 | 35.60$\pm$0.56 | 52.53$\pm$0.59
LFT [77] | 47.47$\pm$0.75 | 66.98$\pm$0.68 | 31.61$\pm$0.53 | 44.90$\pm$0.64 | 55.77$\pm$0.79 | 73.94$\pm$0.67 | 35.95$\pm$0.58 | 53.85$\pm$0.62
LRP [94] | 48.29$\pm$0.51 | 64.44$\pm$0.48 | 32.78$\pm$0.39 | 46.20$\pm$0.46 | 54.83$\pm$0.56 | 74.45$\pm$0.47 | 37.49$\pm$0.43 | 54.46$\pm$0.46
Ours | 49.47$\pm$0.45 | 68.90$\pm$0.40 | 33.78$\pm$0.37 | 46.78$\pm$0.43 | 60.65$\pm$0.48 | 77.59$\pm$0.38 | 40.22$\pm$0.39 | 56.01$\pm$0.37
TABLE III: 5-way cross-domain few-shot accuracies with $95\%$ confidence interval on CUB, Cars, Places and Plantae. Model | Clustering |  miniImageNet
---|---|---
1-shot | 5-shot
kNN | BiGAN | 25.56 | 31.10
linear | 27.08 | 33.91
MLP | 22.91 | 29.06
CACTUs-MAML [13] | 36.24 | 51.28
CACTUs-ProtoNets [13] | 36.62 | 50.16
kNN | DeepCluster | 28.90 | 42.25
linear | 29.44 | 39.79
MLP | 29.09 | 39.67
CACTUs-MAML [13] | 39.90 | 53.97
CACTUs-ProtoNets [13] | 39.18 | 53.36
CACTUs-ProtoNets [13]∗ | MoCo | 39.18 | 53.36
UMTRA [68] | N/A | 39.93 | 50.73
Ours | PatchMoCo | 41.73 | 55.92
TABLE IV: 5-way few-shot accuracies on miniImageNet in unsupervised setting.*
denotes results produced by us.

Augment | K=1 | K=5
---|---|---
CutMix | 65.91 | 79.10
Mixup | 64.69 | 79.08
Man. Mixup | 64.53 | 77.95
PatchMix | 68.34 | 83.16

CAN | K=1 | K=5
---|---|---
base | 65.05 | 81.42
+CutMix | 65.27 | 79.53
+PatchMix | 67.77 | 82.54
+Imp. PM | 67.79 | 82.76

Recons | K=1 | K=5
---|---|---
w/o | 68.34 | 83.16
vanilla | 68.59 | 83.25
Softmax | 68.81 | 83.78
CGR | 69.38 | 84.14

Distill | K=1 | K=5
---|---|---
vanilla | 68.61 | 83.46
w/o H | 68.82 | 83.62
local | 69.07 | 83.69
global | 69.38 | 84.14

(a)

(b)

(c)

(d)

Unsup. | K=1 | K=5
---|---|---
DC | 40.69 | 54.41
Ours | 41.73 | 55.92

Method | K=1 | K=5
---|---|---
IFSL | 64.78 | 80.08
PatchMix | 68.34 | 83.16

Strategy | K=1 | K=5
---|---|---
mix+ori | 68.06 | 82.95
all mix | 68.61 | 83.46

Grid size | K=1 | K=5
---|---|---
$6\times 6$ | 67.95 | 82.69
$11\times 11$ | 68.61 | 83.46

(e)

(f)

(g)

(h)

TABLE V:  Ablation Studies on miniImageNet 5-way tasks. We show 1-shot(K=1) and 5-shot(K=5) results. (a) PatchMix on our baseline. We compare PatchMix with other commonly-used data augmentation methods based on our baseline model. (b) Plug-in: We apply our proposed method to CAN [6]. (c) CGR: we compare different instantiations of reconstruction as an auxiliary task for our CGR module. (d) Hardness: we try different implementations of the second stage training, including vanilla distillation, using PatchMix and PatchMix with two kinds of hardness. (e) Unsupervise: we test the proposed substitution for DeepCluster in unsupervised FSL. (f) Comparison with IFSL: we compare our model with IFSL, which also explores causal inference in FSL. (g) Mixture strategy: we try different strategies of using mixed images. (h) Pooling: we compare model trained with and without the last pooling layer in the Res-12 backbone Model |  tieredImageNet | CIFAR-FS
---|---|---
1-shot | 5-shot | 1-shot | 5-shot
w/o | 72.28 | 86.24 | 76.57 | 88.15
vanilla | 72.13 | 86.71 | 76.42 | 88.21
Softmax | 72.54 | 86.78 | 77.06 | 87.27
CGR | 73.48 | 87.35 | 77.87 | 88.94
vanilla | 72.56 | 86.39 | 76.97 | 87.83
w/o H | 72.78 | 86.67 | 77.02 | 88.10
local | 72.81 | 86.81 | 77.30 | 88.52
global | 73.48 | 87.35 | 77.87 | 88.94
TABLE VI:  Ablation study of CGR and hardness-aware PatchMix on tieredImageNet
and CIFAR-FS.

###  4.3 Ablation Study

To comprehensively validate the effectiveness of our method, we conduct a
series of ablation studies on the design of each sub-module. The accuracies in
both 1-shot and 5-shot settings on miniImageNet are reported in Tab. V.

####  4.3.1 Substitution experiments for PatchMix

Comparison of PatchMix with other augmentation methods. Firstly and the most
importantly, we testify whether the data augmentation approach in the proposed
PatchMix can benefit the few-shot learning. To this end, we compare PatchMix
the baseline model that introduced in Sec. 3.1, together with three commonly
used data augmentation techniques including Mixup, Manifold Mixup and CutMix.
For simplicity we omit the performance of baseline and only report the
improvement or degeneration of each model compared with baseline. The results
in Tab. V(a) indicates that the CutMix can only improve the baseline by 0.49%
on 1-shot tasks; while in other settings these methods have no improvements
(especially, for manifold Mixup the 5-shot accuracy decreases by 1.39%). In
contrast, our proposed PatchMix can respectively increase by 2.92% and 3.82%
on 1-shot and 5-shot tasks. Such a noticeable improvement, which can be
contributed to the ability of disentangling causal features from others,
betokens PatchMix as an effective data augmentation method.

Moreover, our PatchMix can enjoy a better ”neural collapse” property in FSL.
Specifically, this property, as observed in [14] on FSL, means that the intra-
variance defined as the variance of features from each novel category, can
collapse to 0 for a properly trained neural network on sufficient base data.
Besides, such novel features form a simplex equiangular tight frame. However,
as shown in the experiments in [14], the intra-variance of novel categories
can be generally larger than that of the base categories on miniImageNet,
which limits the transferability of vanilla training strategy from base
categories to the novel ones. To show that our proposed PatchMix can help the
FSL models in terms of better neural collapse on novel categories, we
visualize the intra-variance of different training methods, including the
baseline model where no data augmentation is used as in [14] and commonly-used
augmentation methods like CutMix and Mixup. The results are presented in Fig.
5(a) and (b).

We observe that the model trained with PatchMix continuously enjoys a larger
intra-variance than the baseline model on base categories. The gap is about
17% in the first 40 epochs and 5% in the last 10 epochs. As for the novel
data, we imitate the testing process to randomly sample 5 novel classes each
time for evaluation and calculate the intra-variance with all samples of these
classes. Then we repeat this procedure for 20 times and visualize the averaged
intra-variance. The result is visualized in Fig. 5(c). While the intra-
variance with PatchMix is larger than that of baseline in the first 7 epochs,
it decreases much faster and reaches much smaller value in the end of
training. This result, together with the comparison between baseline and
PatchMix on CIFAR-FS in Fig. 6, empirically elucidates that our PatchMix can
indeed improve the behavior with regard to neural collapse in FSL in terms of
generating more collapsed novel features, due to the ability of ours in
learning causal features. Specifically, as the novel features are not
influenced by the non-causal features which are less correlated to the novel
categories, these novel features have decreased variance that is beneficial to
separate different categories for classification. Beyond the quantitative
results, we further visualize learned features of several images from
different novel categories that are extracted by models trained with and
without the proposed PatchMix, as shown in Fig. 7. We can find that while the
model without PatchMix can hardly focus on the target objects, our proposed
method can fix this problem, leading to better feature maps.

Figure 5: (a) Intra-variance of all 64 base classes on miniImageNet along all
epochs when training with 4 different methods. (b) A zoomed version of 80 to
95 epochs of the left image. (c) Comparison of intra-variance of the selected
5 novel classes between baseline and PatchMix. Our PatchMix can not only
control a larger IV during training, but also produce better novel class
representations with higher accuracy. Figure 6: Intra-variance of left: all 64
base classes right: repeatedly sampled 5 novel classes on CIFAR-FS along all
epochs when training with and without PatchMix. Similar to the results on
miniImageNet, PatchMix leads to both higher base IV and lower novel IV. Figure
7: Visualization of features from model trained Left: with or without PatchMix
and Right: with IFSL or PatchMix, using images in the novel set of
miniImageNet. Objects of interest are highlighted with yellow boxes. The
visualization further illustrates that our PatchMix can disentangle the
features to some extent.

Applying PatchMix to existing FSL models. In fact, our proposed PatchMix can
be directly applied to any few-shot learning method whose output is composed
of a confidence map. To demonstrate the utility of such an application, we
employ CAN [6] as a base model and compare the two stages of PatchMix along
with CAN against CutMix. Results in Tab. V(b) reveal that by applying CutMix
to CAN, the 1-shot accuracy is raised but the 5-shot performance is not
improved which is consistent with the results on our baseline. In the
opposite, utilizing PatchMix and improved PatchMix can boost the accuracy.
Concretely, the first stage is better than the basic CAN by 2.72% and 1.12% on
1-shot and 5-shot tasks, and the second stage results in a further improvement
on both 1-shot and 5-shot tasks. Such results reflect the potential of our
method as a plug-in method when solving few-shot learning problems.

Figure 8: The results of using different methods in our correlation-guided
R=reconstruction. We can find that while vanilla reconstruction and the model
using softmax produce a lot of artifacts, the images generated by the model
using gumbel-softmax as normalization function are more smooth, almost
restituting all basic information in the original images.

Effectiveness of PatchMix in unsupervised representation learning. As
introduced in Sec. 3.5, one of our main adaptation of PatchMix to unsupervised
FSL is the substitution of PatchMix to DeepCluster. To show the effectiveness,
we train our model with DeepCluster feature instead, whose results are shown
in Tab. V(e). We find that using DeepCluster the as clustering method with our
model is still better than the previous methods, and replacing DeepCluster
with PatchMix further improves by 1.04% and 1.51% on 1-shot and 5-shot tasks,
which reflects the efficacy of two modules in unsupervised setting.

Comparison with IFSL. In [11] the authors proposed an intervened predictor in
the testing pipeline from the perspective of causal inference. To compare the
effectiveness, we re-implement this method with Res12 as the backbone and
report the comparison result in Tab. V(f). We follow the original paper to
perform IFSL based on MTL [19]. As shown, our method can significantly
outperform the IFSL on both settings (specifically, respectively improve by
3.10% and 2.86% on 1-shot and 5-shot tasks). Besides, this phenomena also
holds even without CGR and hardness-aware modules. These results imply the
benefit of removing spurious correlation, when the pre-training stage is
missing.

####  4.3.2 Ablation study for model variants among different design choices

Variants for hardness-aware PatchMix. To illustrate the role of our proposed
hardness-aware PatchMix, we compare several variants including vanilla
knowledge distillation, using PatchMix without hardness, using local hardness
and using global hardness. The results are shown in Tab. V(d) and Tab. VI. We
can find that while using vanilla knowledge distillation can help boost the
performance, the improvement is relatively small. Moreover, PatchMix without
hardness cannot bring enhancement on 5-shot tasks. This may attribute to that
the knowledge imposed by PatchMix has been learned in the teacher model, which
is transferred to the student model via distillation. However with more hard
examples in the training stage by mixing images from similar classes, we can
further promote the distillation process. We suspect that the gap is
attributed to the learned discriminative features for classification.

Variants for correlation-guided reconstruction. We test the model with and
without the proposed CGR. Specifically, the variants include model without
reconstruction, model with vanilla reconstruction that query feature map and
gallery feature map are directly concatenated and used as the input of the
decoder, model with correlation-guided reconstruction where softmax is used as
normalization function and our final CGR model where gumbel-softmax is used.
As can be seen in Tab. V(c) and Tab. VI, (1) utilizing vanilla model cannot
improve the base model. This means that a simple recovery without explicit
modelling of the patch selection cannot help the model learn better
representations, which is consistent with the discussion in Sec. 3.4.1. (2)
Using softmax as normalization function can bring smaller improvement and
sometimes even degeneration. In fact, the reconstruction results of this
method is obviously worse than the other two variants, as shown in Fig. 8. We
note that this method is the only one that cannot restore the basic shape and
colors from the input knowledge. The possible reason is that softmax function
makes weight $\alpha_{i,j}\in(0,1)$, which means that it will introduce
information from both patches in each position even if one of them is
uncorrelated to the target image, which may confuse the decoder. Thus even if
the model can learn correct similarity between patches from different images,
a bad supervision on the reconstruction hinders the training. (3) Our final
choice of gumbel-softmax benefits the model with 0.53% and 0.52% higher
accuracies on 1-shot and 5-shot tasks on miniImageNet, and consistent
improvement on CIFAR-FS and tieredImageNet. This method can provide both
better restoration and more precise classification compared to the former two
variants, which justifies the efficacy of CGR and the necessity of using
gumbel-softmax. The discrete distribution generated by gumbel-softmax does not
change the value of original feature, but reorganizing them between two
feature maps instead. Consequently, a good reconstruction is solely
conditioned on a correct selection of patches.

Mixture strategy. In Tab. V(g) we compare two different strategies of using
PatchMix, _i.e._ , using only mixed images or using them together with
original images. The results demonstrates that abandoning the images before
mixture is better by 0.55% and 0.51% on 1-shot and 5-shot tasks, and mixing
two types of images leads to similar performance of that of the baseline
model. One reason may be that such a mixing fails to disentangle causal and
non-causal features, in the same way as CutMix as analyzed in Sec. 3.3.

Does size of feature map affect the performance? One would ask if it is
necessary to modify the ResNet-12. We thus compare the model with and without
the last max pooling layer in Tab. V(h). The results show that deleting the
pooling can bring an improvement of 0.74% on 1-shot and 0.77% on 5-shot. We
argue that the reason is two fold. First, without the max pooling, we can
avoid the information loss to some extent, thus comparing the support and
query feature in a more detailed way. Second, while larger feature maps may
lead to some confusing supervision that some patches do not contain the target
object but is guided by the corresponding labels, our PatchMix can alleviate
this problem by imposing random information from other categories into these
patches, thus making the supervision more robust.

##  5 Conclusion

In this paper we analyze the necessity of learning disentangled causal
features, in order to remove the sample selection bias that is commonly met in
FSL. To solve these problems, we propose PatchMix to switch the patches and
corresponding supervision, which has been theoretically shown to learn causal
features by removing the spurious dependency between causal and non causal
features across patches. Additionally, we propose two extra modules to enhance
PatchMix with more discriminative features. Besides, we present an adaptation
of our model for unsupervised FSL. Experiment results among three different
settings reveal the efficacy of our proposed method.

## References

  * [1] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [2] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” _arXiv preprint arXiv:1506.01497_ , 2015.
  * [3] J. Snell, K. Swersky, and R. Zemel, “Prototypical networks for few-shot learning,” in _Adv. Neural Inform. Process. Syst._ , 2017.
  * [4] F. Sung, Y. Yang, L. Zhang, T. Xiang, P. H. Torr, and T. M. Hospedales, “Learning to compare: Relation network for few-shot learning,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2018.
  * [5] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _ICML_ , 2017.
  * [6] R. Hou, H. Chang, M. Bingpeng, S. Shan, and X. Chen, “Cross attention network for few-shot classification,” in _Adv. Neural Inform. Process. Syst._ , 2019\.
  * [7] K. P. Murphy, _Probabilistic Machine Learning: Advanced Topics_. Cambridge: MIT Press, 2022.
  * [8] K. Xiao, L. Engstrom, A. Ilyas, and A. Madry, “Noise or signal: The role of image backgrounds in object recognition,” in _ICLR_ , 2021.
  * [9] H. Shah, K. Tamuly, A. Raghunathan, P. Jain, and P. Netrapalli, “The pitfalls of simplicity bias in neural networks,” in _NeurPIS_ , 2020.
  * [10] F. Khani and P. Liang, “Removing spurious features can hurt accuracy and affect groups disproportionately,” in _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , 2021, pp. 196–205.
  * [11] Z. Yue, H. Zhang, Q. Sun, and X.-S. Hua, “Interventional few-shot learning,” _Advances in neural information processing systems_ , vol. 33, pp. 2734–2746, 2020.
  * [12] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, “Cutmix: Regularization strategy to train strong classifiers with localizable features,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 6023–6032.
  * [13] K. Hsu, S. Levine, and C. Finn, “Unsupervised learning via meta-learning,” _arXiv preprint arXiv:1810.02334_ , 2018.
  * [14] T. Galanti, A. György, and M. Hutter, “On the role of neural collapse in transfer learning,” _arXiv preprint arXiv:2112.15121_ , 2021.
  * [15] C. Liu, Y. Fu, C. Xu, S. Yang, J. Li, C. Wang, and L. Zhang, “Learning a few-shot embedding model with contrastive learning,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 10, 2021, pp. 8635–8643.
  * [16] S. Ravi and H. Larochelle, “Optimization as a model for few-shot learning,” in _Int. Conf. Learn. Represent._ , 2017.
  * [17] A. Nichol, J. Achiam, and J. Schulman, “On first-order meta-learning algorithms,” _arXiv preprint arXiv:1803.02999_ , 2018.
  * [18] Z. Li, F. Zhou, F. Chen, and H. Li, “Meta-sgd: Learning to learn quickly for few-shot learning,” _arXiv preprint arXiv:1707.09835_ , 2017.
  * [19] Q. Sun, Y. Liu, T.-S. Chua, and B. Schiele, “Meta-transfer learning for few-shot learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 403–412.
  * [20] A. A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell, “Meta-learning with latent embedding optimization,” _arXiv preprint arXiv:1807.05960_ , 2018.
  * [21] X. Li, Q. Sun, Y. Liu, Q. Zhou, S. Zheng, T.-S. Chua, and B. Schiele, “Learning to self-train for semi-supervised few-shot classification,” _Advances in Neural Information Processing Systems_ , vol. 32, pp. 10 276–10 286, 2019.
  * [22] Z. Peng, Z. Li, J. Zhang, Y. Li, G.-J. Qi, and J. Tang, “Few-shot image recognition with knowledge transfer,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 441–449.
  * [23] C. Xing, N. Rostamzadeh, B. Oreshkin, and P. O. O Pinheiro, “Adaptive cross-modal few-shot learning,” _Advances in Neural Information Processing Systems_ , vol. 32, pp. 4847–4857, 2019.
  * [24] H.-J. Ye, H. Hu, D.-C. Zhan, and F. Sha, “Few-shot learning via embedding adaptation with set-to-set functions,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 8808–8817.
  * [25] C. Zhang, Y. Cai, G. Lin, and C. Shen, “Deepemd: Few-shot image classification with differentiable earth mover’s distance and structured classifiers,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 12 203–12 213.
  * [26] Y. Hu, V. Gripon, and S. Pateux, “Leveraging the feature distribution in transfer-based few-shot learning,” _arXiv preprint arXiv:2006.03806_ , 2020\.
  * [27] J. Snell and R. Zemel, “Bayesian few-shot classification with one-vs-each p$\backslash$’olya-gamma augmented gaussian processes,” _arXiv preprint arXiv:2007.10417_ , 2020.
  * [28] N. Fei, Z. Lu, T. Xiang, and S. Huang, “Melr: Meta-learning via modeling episode-level relationships for few-shot learning,” in _International Conference on Learning Representations_ , 2020.
  * [29] M. Zhang, J. Zhang, Z. Lu, T. Xiang, M. Ding, and S. Huang, “Iept: Instance-level and episode-level pretext tasks for few-shot learning,” in _International Conference on Learning Representations_ , 2020.
  * [30] D. Wertheimer, L. Tang, and B. Hariharan, “Few-shot classification with feature map reconstruction networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 8012–8021.
  * [31] J. Oh, H. Yoo, C. Kim, and S.-Y. Yun, “Boil: Towards representation change for few-shot learning,” _arXiv preprint arXiv:2008.08882_ , 2020.
  * [32] J.-C. Su, S. Maji, and B. Hariharan, “When does self-supervision improve few-shot learning?” in _European Conference on Computer Vision_. Springer, 2020, pp. 645–666.
  * [33] F. Wu, J. S. Smith, W. Lu, C. Pang, and B. Zhang, “Attentive prototype few-shot learning with capsule network-based embedding,” in _European Conference on Computer Vision_. Springer, 2020, pp. 237–253.
  * [34] G. S. Dhillon, P. Chaudhari, A. Ravichandran, and S. Soatto, “A baseline for few-shot image classification,” _arXiv preprint arXiv:1909.02729_ , 2019\.
  * [35] A. Afrasiyabi, J.-F. Lalonde, and C. Gagné, “Associative alignment for few-shot image classification,” in _European Conference on Computer Vision_. Springer, 2020, pp. 18–35.
  * [36] K. Li, Y. Zhang, K. Li, and Y. Fu, “Adversarial feature hallucination networks for few-shot learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 13 470–13 479.
  * [37] J. Zhang, C. Zhao, B. Ni, M. Xu, and X. Yang, “Variational few-shot learning,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 1685–1694.
  * [38] Y.-X. Wang, R. Girshick, M. Hebert, and B. Hariharan, “Low-shot learning from imaginary data,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7278–7286.
  * [39] E. Schwartz, L. Karlinsky, J. Shtok, S. Harary, M. Marder, R. Feris, A. Kumar, R. Giryes, and A. M. Bronstein, “Delta-encoder: an effective sample synthesis method for few-shot object recognition,” _arXiv preprint arXiv:1806.04734_ , 2018.
  * [40] J. Kim, H. Kim, and G. Kim, “Model-agnostic boundary-adversarial sampling for test-time generalization in few-shot learning.” in _ECCV (1)_ , 2020, pp. 599–617.
  * [41] S. Yang, L. Liu, and M. Xu, “Free lunch for few-shot learning: Distribution calibration,” _arXiv preprint arXiv:2101.06395_ , 2021.
  * [42] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2017\.
  * [43] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” _arXiv preprint arXiv:1710.09412_ , 2017.
  * [44] C. Michaelis, B. Mitzkus, R. Geirhos, E. Rusak, O. Bringmann, A. S. Ecker, M. Bethge, and W. Brendel, “Benchmarking robustness in object detection: Autonomous driving when winter is coming,” _arXiv preprint arXiv:1907.07484_ , 2019.
  * [45] Y. Gong, Z. Zeng, L. Chen, Y. Luo, B. Weng, and F. Ye, “A person re-identification data augmentation method with adversarial defense effect,” _arXiv preprint arXiv:2101.08783_ , 2021.
  * [46] P. Chen, S. Liu, H. Zhao, and J. Jia, “Gridmask data augmentation,” _arXiv preprint arXiv:2001.04086_ , 2020.
  * [47] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le, “Randaugment: Practical automated data augmentation with a reduced search space,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020, pp. 702–703.
  * [48] G. Ghiasi, Y. Cui, A. Srinivas, R. Qian, T.-Y. Lin, E. D. Cubuk, Q. V. Le, and B. Zoph, “Simple copy-paste is a strong data augmentation method for instance segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 2918–2928.
  * [49] P. Mangla, N. Kumari, A. Sinha, M. Singh, B. Krishnamurthy, and V. N. Balasubramanian, “Charting the right manifold: Manifold mixup for few-shot learning,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2020, pp. 2218–2227.
  * [50] V. Verma, A. Lamb, C. Beckham, A. Najafi, I. Mitliagkas, D. Lopez-Paz, and Y. Bengio, “Manifold mixup: Better representations by interpolating hidden states,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 6438–6447.
  * [51] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz, “Invariant risk minimization,” _arXiv preprint arXiv:1907.02893_ , 2019.
  * [52] B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio, “Toward causal representation learning,” _Proceedings of the IEEE_ , vol. 109, no. 5, pp. 612–634, 2021.
  * [53] X. Sun, B. Wu, X. Zheng, C. Liu, W. Chen, T. Qin, and T.-Y. Liu, “Recovering latent causal factor for generalization to distributional shifts,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [54] J. Peters, P. Bühlmann, and N. Meinshausen, “Causal inference by using invariant prediction: identification and confidence intervals,” _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , vol. 78, no. 5, pp. 947–1012, 2016.
  * [55] D. Rothenhäusler, P. Bühlmann, and N. Meinshausen, “Causal dantzig: fast inference in linear structural equation models with hidden variables under additive interventions,” _The Annals of Statistics_ , vol. 47, no. 3, pp. 1688–1722, 2019.
  * [56] J. Pearl, _Causality_. Cambridge University Press, 2009.
  * [57] J. Pearl _et al._ , “Models, reasoning and inference,” _Cambridge, UK: CambridgeUniversityPress_ , vol. 19, p. 2, 2000.
  * [58] B. Oreshkin, P. R. López, and A. Lacoste, “Tadam: Task dependent adaptive metric for improved few-shot learning,” in _Adv. Neural Inform. Process. Syst._ , 2018.
  * [59] I. Khemakhem, D. Kingma, R. Monti, and A. Hyvarinen, “Variational autoencoders and nonlinear ica: A unifying framework,” in _International Conference on Artificial Intelligence and Statistics_. PMLR, 2020, pp. 2207–2217.
  * [60] D. Janzing, J. Peters, J. Mooij, and B. Schölkopf, “Identifying confounders using additive noise models,” in _Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence_ , 2009, pp. 249–257.
  * [61] J. Peters, J. M. Mooij, D. Janzing, and B. Schölkopf, “Causal discovery with continuous additive noise models,” 2014.
  * [62] T. Teshima, I. Sato, and M. Sugiyama, “Few-shot domain adaptation by causal mechanism transfer,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 9458–9469.
  * [63] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 9729–9738.
  * [64] E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with gumbel-softmax,” _arXiv preprint arXiv:1611.01144_ , 2016.
  * [65] X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1501–1510.
  * [66] T. Robert, N. Thome, and M. Cord, “Hybridnet: Classification and reconstruction cooperation for semi-supervised learning,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 153–169.
  * [67] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” _arXiv preprint arXiv:1503.02531_ , 2015.
  * [68] S. Khodadadeh, L. Bölöni, and M. Shah, “Unsupervised meta-learning for few-shot image classification,” _arXiv preprint arXiv:1811.11819_ , 2018\.
  * [69] M. Caron, P. Bojanowski, A. Joulin, and M. Douze, “Deep clustering for unsupervised learning of visual features,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 132–149.
  * [70] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra _et al._ , “Matching networks for one shot learning,” in _Adv. Neural Inform. Process. Syst._ , 2016.
  * [71] M. Ren, E. Triantafillou, S. Ravi, J. Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, and R. S. Zemel, “Meta-learning for semi-supervised few-shot classification,” 2018.
  * [72] L. Bertinetto, J. F. Henriques, P. H. Torr, and A. Vedaldi, “Meta-learning with differentiable closed-form solvers,” _arXiv preprint arXiv:1805.08136_ , 2018.
  * [73] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona, “Caltech-ucsd birds 200,” 2010.
  * [74] J. Krause, M. Stark, J. Deng, and L. Fei-Fei, “3d object representations for fine-grained categorization,” in _Proceedings of the IEEE international conference on computer vision workshops_ , 2013, pp. 554–561.
  * [75] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba, “Places: A 10 million image database for scene recognition,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 6, pp. 1452–1464, 2017\.
  * [76] G. Van Horn, O. Mac Aodha, Y. Song, Y. Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and S. Belongie, “The inaturalist species classification and detection dataset,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8769–8778.
  * [77] H.-Y. Tseng, H.-Y. Lee, J.-B. Huang, and M.-H. Yang, “Cross-domain few-shot classification via learned feature-wise transformation,” _arXiv preprint arXiv:2001.08735_ , 2020.
  * [78] L. Bottou, “Large-scale machine learning with stochastic gradient descent,” in _Proceedings of COMPSTAT’2010_ , 2010.
  * [79] I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” _arXiv preprint arXiv:1608.03983_ , 2016.
  * [80] S. Gidaris and N. Komodakis, “Dynamic few-shot visual learning without forgetting,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2018.
  * [81] S. Qiao, C. Liu, W. Shen, and A. L. Yuille, “Few-shot image recognition by predicting parameters from activations,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 7229–7238.
  * [82] N. Dvornik, C. Schmid, and J. Mairal, “Diversity with cooperation: Ensemble methods for few-shot classification,” in _Int. Conf. Comput. Vis._ , 2019\.
  * [83] S. Gidaris and N. Komodakis, “Generating classification weights with gnn denoising autoencoders for few-shot learning,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2019.
  * [84] S. Gidaris, A. Bursuc, N. Komodakis, P. Pérez, and M. Cord, “Boosting few-shot visual learning with self-supervision,” in _Int. Conf. Comput. Vis._ , 2019.
  * [85] S. W. Yoon, J. Seo, and J. Moon, “Tapnet: Neural network augmented with task-adaptive projection for few-shot learning,” 2019.
  * [86] K. Lee, S. Maji, A. Ravichandran, and S. Soatto, “Meta-learning with differentiable convex optimization,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2019.
  * [87] Y. Liu, B. Schiele, and Q. Sun, “An ensemble of epoch-wise empirical bayes for few-shot learning,” in _European Conference on Computer Vision_. Springer, 2020, pp. 404–421.
  * [88] C. Simon, P. Koniusz, R. Nock, and M. Harandi, “Adaptive subspaces for few-shot learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 4136–4145.
  * [89] B. Liu, Y. Cao, Y. Lin, Q. Li, Z. Zhang, M. Long, and H. Hu, “Negative margin matters: Understanding margin in few-shot classification,” _arXiv preprint arXiv:2003.12060_ , 2020.
  * [90] Y. Tian, Y. Wang, D. Krishnan, J. B. Tenenbaum, and P. Isola, “Rethinking few-shot image classification: a good embedding is all you need?” _arXiv preprint arXiv:2003.11539_ , 2020.
  * [91] W. Xu, Y. Xu, H. Wang, and Z. Tu, “Attentional constellation nets for few-shot learning,” 2021.
  * [92] C. Xu, Y. Fu, C. Liu, C. Wang, J. Li, F. Huang, L. Zhang, and X. Xue, “Learning dynamic alignment via meta-filter for few-shot learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 5182–5191.
  * [93] V. Garcia and J. Bruna, “Few-shot learning with graph neural networks,” _arXiv preprint arXiv:1711.04043_ , 2017.
  * [94] J. Sun, S. Lapuschkin, W. Samek, Y. Zhao, N.-M. Cheung, and A. Binder, “Explanation-guided training for cross-domain few-shot classification,” in _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 7609–7616.

|  Chengming Xu received his Bachelor’s degree in computer science at Fudan
University in 2018 and is now a fourth year PhD student majoring in statistics
advised by Prof. Yanwei Fu. His research interests include action analysis and
few-shot learning.
---|---
|  Chen Liu is a PhD student at the Department of Mathematics at the Hong Kong
University of Science and Technology under the supervision of Prof. Yuan Yao.
He received the Bachelor degree of Engineering from the School of Mechanical
Engineering, Shanghai Jiaotong University, in 2018 and the Master degree of
Statistics from the School of Data Science, Fudan University, in 2021. His
current research interests include machine learning and its application to
computer vision.
---|---
|  Xinwei Sun is currently an assistant professor with School of Data Science,
Fudan University. He received his Ph.D in school of mathematical sciences,
Peking University in 2018. His research interests mainly focus on statistical
machine learning, causal inference, with their applications on medical
imaging, computer vision and few-shot learning.
---|---
|  Siqian Yang recevied his Ph.D. on Computer Science and Technology from
Tongji University in 2018. He is currently a researcher at Tencent YouTu Lab,
China. His research interests include image processing, computer vision, and
vehicular networks.
---|---
|  Yabiao Wang is a Senior Researcher at Tencent Youtu lab,China. He received
his master degree from Zhejiang University in 2016. He published more than 30
conference papers including CVPR, ICCV,ECCV, and AAAI. He won more than 20
challenge titles. His research interests are object detection/segmentation,
few-shot learning and domain adaptation/generalization.
---|---
|  Chengjie Wang received the B.S degree in computer science from Shanghai
Jiao Tong University, China, in 2011, and double M.S. degrees in computer
science from Shanghai Jiao Tong University, China and Waseda Univesity, Japan,
in 2014. He is currently the Research Director of Tencent YouTu Lab. His
research interests include computer vison and machine learning. He has
published more than 70 refereed papers on major Computer Vision and Artificial
Intelligence Conference and holds over 120 patents in these areas.
---|---
|  Yanwei Fu received hist PhD degree from the Queen Mary University of
London, in 2014. He worked as post-doctoral research at Disney Research,
Pittsburgh, PA, from 2015 to 2016. He is currently a professor with Fudan
University. He was appointed as the Professor of Special Appointment (Eastern
Scholar) at Shanghai Institutions of Higher Learning in 2017, and awarded the
1000 Young talent scholar in 2018. He published more than 100
journal/conference papers including IEEE TPAMI, TMM, ECCV, and CVPR. His
research interests are one-shot learning for images and videos, learning based
3D reconstruction for modelling objects/bodies, robotic grasping, and image
generation/inpainting/editing.
---|---