# Few-shot Classification via Ensemble Learning with Multi-Order Statistics

Sai Yang1 Fan Liu2∗ Delong Chen2&Jun Zhou3 1School of Electrical Engineering,
Nantong University, Nantong, China
2College of Computer and Information, Hohai University, Nanjing, China
3School of Information and Communication Technology, Griffith University,
Queensland, Australia
∗Email<EMAIL_ADDRESS>
###### Abstract

Transfer learning has been widely adopted for few-shot classification. Recent
studies reveal that obtaining good generalization representation of images on
novel classes is the key to improving the few-shot classification accuracy. To
address this need, we prove theoretically that leveraging ensemble learning on
the base classes can correspondingly reduce the true error in the novel
classes. Following this principle, a novel method named Ensemble Learning with
Multi-Order Statistics (ELMOS) is proposed in this paper. In this method,
after the backbone network, we use multiple branches to create the individual
learners in the ensemble learning, with the goal to reduce the storage cost.
We then introduce different order statistics pooling in each branch to
increase the diversity of the individual learners. The learners are optimized
with supervised losses during the pre-training phase. After pre-training,
features from different branches are concatenated for classifier evaluation.
Extensive experiments demonstrate that each branch can complement the others
and our method can produce a state-of-the-art performance on multiple few-shot
classification benchmark datasets.

##  1 Introduction

Figure 1: (a) The traditional methods often use different backbone networks as
individuals, which significantly increases the computation and storage costs.
(b) Our method takes the same backbone and equips different branches with
multi-order statistics as learning individuals. They are parameter-free and
trained jointly, and do not require extra model size and computation time.

Few-shot Classification (FSC) is a promising direction in alleviating the
labeling cost and bridging the gap between human intelligence and machine
models. It aims to accurately differentiate novel classes with only a few
labeled training samples. Due to limited supervision from novel classes, an
extra base set with abundant labeled samples is often used to improve the
classification performance. According to the adopted training paradigms, FSC
methods can be roughly divided into meta-learning-based Finn et al. (2017);
Snell et al. (2017) and transfer-learning-based Chen et al. (2019); Liu et al.
(2020); Afrasiyabi et al. (2020). The first type takes the form of episodic
training, in which subsets of data are sampled from the base set to imitate
the meta-test setting. Since sampling does not cover all combinations, this
paradigm cannot fully utilize the information provided by the base set. In
contrast, the transfer-learning takes the base set as a whole, so it avoids
the drawback of meta-learning and achieves better performance. Many effective
regularization techniques have been exploited in transfer-learning, for
example, manifold mixup Mangla et al. (2020), self-distillation Tian et al.
(2020), and self-supervised learning Zhang et al. (2020b), which leads to
significant improvement on the generalization of image representations and the
FSC performance.

Ensemble learning combines multiple learners to solve the same problem and
exhibits better generalization performance than any individual learners Yang
et al. (2013). When combining ensemble learning with deep Convolutional Neural
Networks (CNN), the new paradigm usually requires large-scale training data
for classification tasks Horváth et al. (2021); Agarwal et al. (2021), making
it challenging to be adopted for FSC. Recently, two notable studies Dvornik et
al. (2019); Bendou et al. (2022) employed an ensemble of deep neural networks
for FSC tasks under either a meta-learning or a transfer-learning setting.
They demonstrated that ensemble learning is also applicable to FSC. Yet, these
works are still preliminary and lack a theoretical analysis to explain the
underlying reason behind the promising performance. To address this challenge,
we provide an FSC ensemble learning theorem for the transfer-learning regime.
Its core idea is a tighter expected error bound on the novel classes, in which
the expected error on the novel classes can be reduced by implementing
ensemble learning on the base classes, given the base classes-novel classes
domain divergence.

The generalization ability of ensemble learning is strongly dependent on
generating diverse individuals. As shown in Figure 1 (a), traditional methods
often use different backbone networks as individuals, which significantly
increases the computation and storage costs. Our work finds that different-
order statistics of the CNN features are complementary to each other, and
integrating them can better model the whole feature distribution. Based on
this observation, we develop a parameter-free ensemble method, which takes the
same backbone and equips different branches with multi-order statistics as
learning individuals. We name this method Ensemble Learning with Multi-Order
Statistics (ELMOS), as shown in Figure 1 (b). The main contributions of this
paper are summarized as follows:

  * •

To our knowledge, this is the first theoretical analysis to guide ensemble
learning in FSC. The derived theorem proves a tighter expected error bound is
available on novel classes.

  * •

We propose an ensemble learning method by adding multiple branches at the end
of the backbone networks, which can significantly reduce the computation time
of the training stage for FSC.

  * •

This is the first time that multi-order statistics is introduced to generate
different individuals in ensemble learning.

  * •

We conduct extensive experiments to validate the effectiveness of our method
on multiple FSC benchmarks.

##  2 Related Work

In this section, we review the related work to the proposed method.

###  2.1 Few-shot Classification

According to how the base set is used, FSC methods can be roughly categorized
into two groups, meta-learning-based Zhang et al. (2020a) and transfer-
learning-based Chen et al. (2019); Liu et al. (2020). Meta-learning creates a
set of episodes to simulate the real FSC test scenarios and simultaneously
accumulate meta-knowledge for fast adaptation. Typical meta-knowledge includes
optimization factors such as initialization parameters Finn et al. (2017) and
task-agnostic comparing ingredients of feature embedding and metric Snell et
al. (2017); Wertheimer et al. (2021). Recent literature on transfer learning
Tian et al. (2020); Chen et al. (2019) questioned the efficiency of the
episodic training in meta-learning, and alternatively used all base samples to
learn an off-the-shelf feature extractor and rebuilt a classifier for novel
classes. Feature representations play an important role in this regime Tian et
al. (2020). To this end, regularization techniques such as negative-margin
softmax loss and manifold mixup Liu et al. (2020); Mangla et al. (2020) have
been adopted to enhance the generalization ability of cross-entropy loss.
Moreover, self-supervised Zhang et al. (2020b); Rajasegaran et al. (2020) and
self-distillation Ma et al. (2019); Zhou et al. (2021) methods have also shown
promising performance in transfer-learning. To this end, supervised learning
tasks can be assisted by several self-supervised proxy tasks such as rotation
prediction and instance discrimination Zhang et al. (2020b), or by adding an
auxiliary task of generating features during the pre-training Xu et al.
(2021b). When knowledge distillation is adopted, a high-quality backbone
network can be evolved through multiple generations by a born-again strategy
Rajasegaran et al. (2020). All these methods suggest the importance of
obtaining generalization representations, and we will leverage ensemble
learning to achieve this goal.

###  2.2 Ensemble Learning

Ensemble learning builds several different individual learners based on the
same training data and then combines them to improve the generalization
ability of the learning system over any single learner. This learning scheme
has shown promising performance on traditional classification tasks with deep
learning on large-scale labeled datasets. Recently, ensemble learning for FSC
methods has been presented. For example, Dvornik et al. (2019) combined an
ensemble of prototypical networks through deep mutual learning under a meta-
learning setting. Bendou et al. (2022) reduced the capacity of each backbone
in the ensemble and pre-trained them one by one with the same routine.
However, the size of the ensemble learner increased for inference in the
former work, while the latter required extra time to pre-train many learning
individuals. Therefore, it still lacks efficient designs for learning
individuals in FSC ensemble learning. Moreover, these works did not involve
any theoretical analysis of the underlying mechanism of ensemble learning in
FSC. In this paper, we investigate why ensemble learning works well in FSC
under the transfer-learning setting. Based on the analysis, we propose an
efficient learning method using a shared backbone network with multiple
branches to generate learning individuals.

###  2.3 Pooling

Convolutional neural network models progressively learn high-level features
through multiple convolution layers. A pooling layer is often added at the end
of the network to output the final feature representation. To this end, Global
Average Pooling (GAP) is the most popular option, however, it cannot fully
exploit the merits of convolutional features because it only calculates the
$1^{st}$-order feature statistics. Global Covariance Pooling (GCP) such as
Deep$\rm O^{2}$P explores the $2^{nd}$-order statistic by normalizing the
covariance matrix of the convolutional features, which has achieved impressive
performance gains over the classical GAP in various computer vision tasks.
Further research shows that using richer statistics may lead to further
possible improvement. For example, Kernel Pooling Cui et al. (2017) generates
high-order feature representations in a compact form. However, a certain order
statistic can only describe partial characteristics of the feature vector from
the view of the characteristic function of random variables. For example, the
first- and second-order statistics can completely represent their statistical
characteristic only for the Gaussian distribution. Therefore, higher-order
statistics are still needed for the non-Gaussian distributions, which are more
ubiquitous in many real-world applications. This motivates us to calculate
multi-order statistics to retain more information on features.

##  3 The Proposed Method

Here we present the proposed method. We start with a formal definition of FSC,
and then present a theorem on FSC ensemble learning. This theorem leads to the
development of an ensemble learning approach with multi-order statistics.

###  3.1 Theory Foundation

Under the standard setting of few-shot classification, three sets of data with
disjoint labels are available, i.e., the base set $S_{b}$, the validation set
$S_{val}$ and the novel set $S_{n}$. In the context of transfer-learning,
$S_{b}$ is used for pre-training a model to well classify the novel classes in
$S_{n}$, with the hyper-parameters tuned on $S_{val}$. Let
$S_{b}=\\{(x_{i},y_{i})\\}_{i=1}^{N_{b}}$ denotes the source domain with
${N_{b}}$ labelled samples and $S_{n}$ denotes the target domain labelled with
$K$ samples in each episode, where $N_{b}>>K$. Let the label function of
$S_{b}$ and $S_{n}$ be $f_{b}$ and $f_{n}$, respectively. During the pre-
training, a learner $h$ is obtained to approximate the optimal mapping
function $h^{*}$ based on all ${N_{b}}$ training samples in $S_{b}$ from all
possible hypotheses $\mathcal{H}$. When ensemble learning is introduced into
the pre-training, several learners denoted as $\\{h_{o}\\}_{o=1}^{O}$ can be
obtained. With the ensemble technique of weighted averaging, the final learner
$\overline{h}$ is produced as:

$\overline{h}=\sum_{o=1}^{O}\alpha_{o}h_{o},$ (1)

where $\alpha_{o}$ is the weight parameter. There is a domain shift between
the base and novel classes Tseng et al. (2020), and we use the $L_{1}$
distance Kifer et al. (2004) to measure the domain divergence between $S_{b}$
and $S_{n}$:

$\mathcal{D}(S_{b},S_{n})=\int\left|\eta_{b}(x)-\eta_{n}(x)\right|\left|\overline{h}(x)-f_{n}(x)\right|dx,$
(2)

where $\eta_{b}(x)$ and $\eta_{n}(x)$ is the density functions of $S_{b}$ and
$S_{n}$ respectively.

######  Theorem 1 (FSC Ensemble Learning)

Let $\mathcal{H}$ be a hypothesis space, for any
$h\in\\{h_{o}\\}_{o=1}^{O}\in\mathcal{H}$ is learned from $S_{b}$, and
$\overline{h}=\sum_{o=1}^{O}\alpha_{o}h_{o}\in\mathcal{H}$, the expected error
on $S_{n}$ respectively with $\overline{h}$ and $h$ holds the following
relationship:

$\displaystyle e_{n}(\overline{h})\leq
e_{b}(\overline{h})+\underbrace{\mathcal{D}(S_{b},S_{n})}_{\text{($S_{b}$-$S_{n}$)
divergence}}+\lambda$ $\displaystyle\leq
e_{b}(h)+\underbrace{\mathcal{D}(S_{b},S_{n})}_{\text{($S_{b}$-$S_{n}$)
divergence}}+\lambda,$

where $\lambda=E_{X\in{S_{b}}}\left|f_{n}(x)-f_{b}(x)\right|$ is a constant,
$e_{n}(\overline{h})$ is the expected error on $S_{n}$ with $\overline{h}$,
$e_{b}(h)$ is the expected error on $S_{b}$ with $h$, $e_{b}(\overline{h})$ is
the expected error on $S_{b}$ with $\overline{h}$.

The proof is provided in the Supplementary Material.

###### Remark 1

The core idea of Theorem 1 is to define a tighter expected error bound on the
novel classes with the learned mapping function in the form of ensemble
learning during the pre-training. Theorem 1 tells that the true error on the
novel classes can be reduced by implementing ensemble learning on the base
classes, given the domain divergence between the novel class and base class.
This can well explain the effectiveness of ensemble learning in few-shot
classification, in which multiple learners are assembled to enhance the
generalization on the base set, resulting in better performance in novel
classes.

###  3.2 FSC via Ensemble Learning with Multi-order Statistics

####  3.2.1 Overview

Our method employs the transfer-learning paradigm in a two-phase manner. In
the first phase, a good feature extractor is pre-trained on the base set. In
the second phase, FSC evaluation is done on the novel set with the pre-trained
feature extractor. Following Theorem 1, we introduce ensemble learning in the
first phase to improve the FSC performance. The key to this phase is to
effectively train multiple diverse individuals. Different from the previous
works Dvornik et al. (2019); Bendou et al. (2022) that use many different
networks as individuals, we add multiple branches after the backbone network
to create individuals for reducing training costs. Each branch calculates
different-order statistics for pooling to highlight the discrepancy between
the individuals. This step is optimized by supervised losses. After pre-
training, features from different branches are concatenated for FSC
evaluation. We name this method as Ensemble Learning with multi-Order
Statistics (ELMOS) for FSC. An overview of ELMOS is shown in Figure 2, and a
flow description of ELMOS is given in Algorithm 1.

Figure 2: An overview of our framework. The images from $S_{b}$ are augmented
by the image processing module and fed into the backbone for feature
extraction. The CNN features from the backbone are then reshaped into the
matrix, which is used to calculate multi-order statistics to equip different
branches. Ensemble learning is implemented by the linear combination of
multiple branches during the pre-training phase.

####  3.2.2 Pre-training via Multi-order Statistics

The proposed model architecture mainly consists of the following four
components: an image processing module, the backbone network, a multi-order
statistics module, and a supervised classifier module. The image processing
module is denoted as $M\left(\cdot\right)$, which performs transformation of
multi-scale rotation to augment the original base set and their label space.
The backbone network is denoted as $B_{\theta}\left(\cdot\right)$ and
parameterized by $\theta$, which converts each image into a tensor of size
$H\times W\times d$. The multi-order statistics module module is denoted as
$S\left(\cdot\right)$, which maps the tensor from the backbone into multiple
feature representations to generate individual learners for ensemble learning.
The supervised classifier module is composed of softmax classifiers
$L_{W}\left(\cdot\right)$ and the projectors $L_{U}\left(\cdot\right)$ with
parameter matrices $W$ and $U$, respectively, which are used to build the
supervised losses for pre-training.

Given $L$ samples be randomly sampled from $S_{b}$ with $C_{b}$ classes, in
which an image and its corresponding label are denoted as $(x_{i},y_{i})$,
$y_{i}\in\\{1,2,...C_{b}\\}$. $M\left(\cdot\right)$ scales the images with the
aspect-ratio of 2:3 and rotates the images with
$\\{0^{\circ},90^{\circ},180^{\circ},270^{\circ}\\}$ under both the new and
the original scales, resulting in eight times expansion of training samples.
Feed $x_{i}$ into $B_{\theta}$ to produce a tensor feature of
$T_{i}=B_{\theta}(x_{i})\in\mathscr{R}^{H\times W\times d}$. Next, we reshape
the tensor $T_{i}$ into the matrix $T_{i}\in\mathscr{R}^{HW\times d}$, and
view each row vector in the matrix $t_{j}\in\mathscr{R}^{d}$ as an observation
of the random variable of $t\in\mathscr{R}^{d}$. When $d=1$, the first
characteristic function of variable $t$ in the Laplace operator is given by:

$\phi(s)=\int_{-\infty}^{+\infty}f(t)e^{st}dt=\int_{-\infty}^{+\infty}e^{st}dF(t),$
(3)

where $f(t)$ and $F(t)$ are the density function and distribution function of
$t$, respectively. Let $\psi(s)=ln\phi(s)$ be the second characteristic
function of the random variable $t$.

######  Theorem 2 (The Inversion Formula for Distributions)

Let $t$ be a random variable with distribution function $F(t)$ and
characteristic function $\phi(s)$. For $a,b\in C(F)$ and $a<b$,

$\displaystyle
F(b)-F(a)={\lim_{c\to\infty}}\frac{1}{2\pi}\int_{-c}^{c}\frac{e^{-sa}-e^{-sb}}{s}\phi(s)ds.$

######  Corollary 1 (Uniqueness)

If two distributions of $F_{1}(t)$ and $F_{2}(t)$ are identical, then the
corresponding characteristic functions $\psi_{1}(s)$ and $\psi_{2}(s)$ are
identical.

See proof of Theorem 2 and Corollary 1 in Shiryaev (2016). From Theorem 2 and
Corollary 1, we can see that there is a one-to-one correspondence between the
characteristic function and the probability density function such that the
characteristic function can completely describe a random variable.

The $o^{th}$-order cumulant of the random variable $t$ is defined as the
$o^{th}$ derivative of function $\psi(s)$ at the origin, which is:

$c_{o}=\frac{d^{o}\psi(s)}{ds^{o}}\bigg{|}_{s=0}.$ (4)

Then the Taylor series expansion of function $\psi(s)$ at the origin with
respect to $s$ yields:

$\psi(s)=c_{1}s+\frac{1}{2}c_{2}s^{2}+...+\frac{1}{o!}c_{p}s^{o}+R_{s}(s^{o}),$
(5)

where $R_{s}(s^{o})$ is the remainder term. It can be seen from Equation (5)
that the $o^{th}$-order cumulant of $t$ is the coefficient of the term $s^{o}$
in Equation (5).

###### Proposition 1

Consider a Gaussion distribution $f(t)$ with mean $\mu$ and variance
$\Sigma^{2}$ for the random variable $t$, its second characteristic function
is:

$\displaystyle\psi(s)=\mu s+\frac{1}{2}\Sigma^{2}s^{2}.$

Consequently, the cumulant of the random variable $t$ are:

$\displaystyle c_{1}=\mu,c_{2}=\Sigma^{2},c_{o}=0\quad(o=3,4,...).$

The proof is provided in the Supplementary Material.

###### Remark 2

Proposition 1 implies that for Gaussian signals only, the cumulants are
identically zero when the order is greater than 2. Please note this conclusion
can be naturally extended to the scenario of multivariate variables when
$d>1$. For the random variables with Gaussian distribution, the first and
second-order statistics can completely represent their statistical
characteristics. However, the non-Gaussian signals are more common in real-
world applications. In this case, higher-order statistics also contain a lot
of useful information. Therefore, we propose a multi-order statistics module
consisting of multiple branches, each equipped with different order statistics
of the tensor feature $T_{i}$.

In particular, we employ three branches in the multi-order statistics module,
which respectively calculate three orders cumulants of the variable $t$ with
the observations in $T_{i}$. The specific formulation of the $1^{st}$-order,
$2^{nd}$-order and $3^{rd}$-order cumulants of $t$ are expressed as:

$\begin{split}&c_{i1}=\frac{1}{H\times W}\sum_{j=1}^{H\times W}t_{j}\quad
c_{i1}\in\mathscr{R}^{d},\\\ &c_{i2}=\frac{1}{H\times{W}}\sum_{j=1}^{H\times
W}(t_{j}-c_{i1})(t_{j}-c_{i1})^{T}\quad c_{i2}\in\mathscr{R}^{d\times d},\\\
&c_{i3}=\frac{1}{H\times W}\sum_{j=1}^{H\times
W}\frac{(t_{j}-c_{i1})^{2}(t_{j}-c_{i1})^{T}}{c_{i2}^{2}c_{i2}^{T}}\quad
c_{i3}\in\mathscr{R}^{d\times d}.\\\ \end{split}$ (6)

As $c_{i2}$ and $c_{i3}$ are $d\times d$ matrices, we flatten them into
$d^{2}$-dimensional vectors and finally get the feature representations of
$z_{i1}$, $z_{i2}$ and $z_{i3}$. We use these three features as individuals in
ensemble learning, which respectively pass through their corresponding softmax
classifier $L_{W}(\cdot)$ and projectors $L_{U}(\cdot)$. So the $o$-th
($o=1,2,3$) outputs are:

$\begin{split}&P_{ij}^{o}=L_{Wo}\left(z_{io}\right)=\frac{exp({{z_{i0}}^{T}w_{oj}})}{\sum_{j=1}^{8C_{b}}{exp({{z_{i0}}^{T}w_{oj}})}},\\\
&{u_{io}}=\left\|{L_{Uo}}(z_{io})\right\|=\left\|{{z_{io}}^{T}}U_{o}\right\|,\end{split}$
(7)

where $L_{Wo}(\cdot)$ is the $o$-th softmax classifier with the parameter
matrix of $W_{o}$, $w_{oj}$ is the $j$-th component of $W_{o}$.
$L_{Uo}(\cdot)$ is the $o$-th projector with the parameter matrix $U_{o}$.
$P_{ij}^{o}$ is the $j$-th component of the output probability from the $o$-th
softmax classifier. $u_{io}$ is the output vector from the $o$-th projector.
We simultaneously employ Classification-Based (CB) loss of cross-entropy and
Similarity-Based (SB) loss of supervised contrastive in supervised learning
for each individual Scott et al. (2021). These two losses are formulated as:

$\displaystyle
L_{CB}^{o}\left(\theta,W_{o}\right)=-{\sum_{i=1}^{8L}\sum_{j=1}^{8C_{b}}y_{ij}logP_{ij}^{o}},$
(8) $\displaystyle L_{SB}^{o}(\theta,U_{o})=$
$\displaystyle-\sum_{i=1}^{8L}log\sum_{q\in{Q(u_{i0})}}\frac{exp(u_{io}\cdot
u_{qo}/\tau)}{\sum_{a=1}^{8L}exp(u_{ao}\cdot u_{qo}/\tau)},$

where $y_{ij}$ is the $j$-th component of label $y_{i}$, $\tau$ is a scalar
temperature parameter. $Q(u_{io})$ is the positive sample set, in which each
sample has the same label as $u_{io}$. $u_{qo}$ is the $q$-th sample in
$Q(u_{io})$. Then the learning objective function for the $o$-th individual
is:

$L_{o}(\theta,W_{o},U_{o})=L_{CB}^{o}\left(\theta,W_{o}\right)+L_{SB}^{o}(\theta,U_{o}).$
(9)

The overall loss function with ensemble learning is:

$L_{overall}=\sum_{o=1}^{O}\alpha_{o}L_{o}(\theta,W_{o},U_{o}),$ (10)

where $\alpha_{o}$ is a weight controlling the contribution of each individual
in the ensemble learning. The pre-training adopts the gradient descent method
to optimize the above loss function.

Input: Base set $S_{b}$, support set $S_{p}$, query set $S_{q}$; augmentation
module $M\left(\cdot\right)$, backbone network $B_{\theta}\left(\cdot\right)$,
multi-order statistics module $S(\cdot)$, softmax classifier $L_{Wo}$,
projector $L_{Uo}$ and logistic regression $g_{\xi}\left(\cdot\right)$;
temperature parameter $\tau$, weight $\alpha_{o}$ ($o=1,2,3$).

Output: Final prediction of the query samples

Stage 1: Pre-training with ensemble learning

for  _numbers of training epochs_ do

Sample a mini-batch with any image of $\\{x_{i},y_{i}\\}$;

Feed $x_{i}$ into $T\left(\cdot\right)$ and $B_{\theta}\left(\cdot\right)$ to
obtain feature map $T_{i}\in\mathscr{R}^{H\times W\times d}$ ;

Pass $T_{i}$ through $S(\cdot)$ to output features $z_{io}$, ($o=1,2,3$);

Pass $z_{io}$ through $L_{Wo}$ and $L_{Uo}$ to get the output probability and
projection feature ;

Calculate optimization loss for each individual via Equation (9);

Calculate overall loss for pre-training via Equation (10);

Update the parameters of $\theta$, $W_{o}$, $U_{o}$ using SGD;

end for

Stage 2: Few-shot evaluation

for  _all iteration = 1, 2, …, MaxIteration_ do

Feed $x_{s}\in{S_{p}}$ into $B_{\theta}(\cdot)$ and $S(\cdot)$ to output
feature $z_{so}$, ($o=1,2,3$);

Concatenate $z_{so}$ into the feature $z_{s}$ to train the classifier of
$g_{\xi}\left(\cdot\right)$;

end for

Classify the query samples according to Equation (13).

Algorithm 1 Ensemble Learning with multi-Order Statistics (ELMOS) for FSC

####  3.2.3 Few-shot Evaluation

The phase of few-shot evaluation still needs to construct a set of $N$-way
$K$-shot FSC tasks, with a support set and a query set in each task. The
support set randomly selects $K$ samples from each of the $N$ classes that are
sampled from $S_{n}$, which is denoted as
$S_{p}={\\{x_{s},y_{s}}\\}_{s=1}^{NK}$, where $(x_{s},y_{s})$ is the $s$-th
images and its corresponding label. The query set consists of the remaining
images in these $N$ classes, which is denoted as
$S_{q}=\\{{x_{q}}\\}_{q=1}^{Q}$ with any image of $x_{q}$. After pre-training,
we get rid of the softmax classifier $L_{W}(\cdot)$ and projectors
$L_{U}(\cdot)$ and fix the backbone network $B_{\theta}(\cdot)$ and the multi-
order statistics module module $S(\cdot)$. The support set $S_{p}$ is input
into $B_{\theta}(\cdot)$ and $S(\cdot)$ to produce the output features:

$z_{so}=B_{\theta}\circ S(x_{s})\quad(o=1,2,3),$ (11)

where $\circ$ is the stack operator. The features $z_{s1},z_{s2},z_{s3}$ are
concatenated into a final expression of $x_{s}$:

$z_{s}=con(z_{s1},z_{s2},z_{s3}),$ (12)

where $con(\cdot)$ is the concatenated operator. A logistic regression
classifier $g_{\xi}\left(\cdot\right)$ parameterized by $\xi$ is then trained
with $z_{s}$ and its corresponding label $y_{s}$. The query image $x_{q}$ is
finally classified as:

$\hat{y}_{q}=g_{\xi}(z_{q}),$ (13)

where $\hat{y}_{q}$ is the inference label value of $x_{q}$.

Table 1: Test accuracy (%) of each branch and their ensemble under 5-way 1-shot and 5-shot tasks on three datasets. Method | Backbone | miniImageNet | CIFAR-FS | CUB
---|---|---|---|---
1-shot | 5-shot | 1-shot | 5-shot | 1-shot | 5-shot
B_1 | ResNet12 |  69.06$\pm$0.44 |  83.61$\pm$0.29 |  77.09$\pm$0.46 |  88.46$\pm$0.34 |  81.46$\pm$0.39 |  92.55$\pm$0.18
B_2 | ResNet12 |  66.42$\pm$0.42 |  85.76$\pm$0.26 |  71.53$\pm$0.48 |  88.83 $\pm$0.27 |  77.79$\pm$0.39 |  94.44$\pm$0.17
B_3 | ResNet12 |  67.68$\pm$0.43 |  82.81$\pm$0.29 |  72.83$\pm$0.46 |  86.34$\pm$0.34 |  83.89$\pm$0.38 |  91.20$\pm$0.17
ELMOS | ResNet12 |  70.30$\pm$0.45 |  86.17$\pm$0.26 |  78.18$\pm$0.41 |  89.87$\pm$0.31 |  85.21$\pm$0.38 |  95.02$\pm$0.16

(a) 5-way 1-shot

(b) 5-way 5-shot

Figure 3: Test accuracy (%) of the classification-based (CB) loss, similarity-based (SB) loss and their combination (CB&SB) under 5-way 1-shot and 5-way 5-shot tasks on three datasets. Table 2: Comparison of results against state-of-the-art methods on CUB dataset.The top three results are marked in red, blue and green. Method | CUB
---|---
1-shot | 5-shot
Meta-learning |  |
Relational Sung et al. (2018) |  55.00$\pm$1.00 |  69.30$\pm$0.80
DeepEMD Zhang et al. (2020a) |  75.65$\pm$0.83 |  88.69$\pm$0.50
BML Zhou et al. (2021) |  76.21$\pm$0.63 |  90.45$\pm$0.36
RENet Kang et al. (2021) |  79.49$\pm$0.44 |  91.11$\pm$ 0.24
FPNWertheimer et al. (2021) | 83.55$\pm$0.19 | 92.92$\pm$0.10
IEPT Zhang et al. (2020b) |  69.97$\pm$0.49 |  84.33$\pm$0.33
APP2S Ma et al. (2022) |  77.64$\pm$0.19 |  90.43$\pm$0.18
MFS Afrasiyabi et al. (2022) |  79.60$\pm$0.80 |  90.48$\pm$0.44
DeepBDC Xie et al. (2022) | 84.01$\pm$0.42 | 94.02$\pm$0.24
HGNN Yu et al. (2022) |  78.58$\pm$0.20 |  90.02$\pm$0.12
INSTARongkai Ma (2022) | 75.26 ± 0.31 | 88.12 ± 0.54
Transfer-learning |  |
Baseline++ Chen et al. (2019) |  60.53$\pm$0.83 |  79.34$\pm$0.61
Neg-Cosine Liu et al. (2020) |  72.66$\pm$0.85 |  89.40$\pm$0.43
S2M2 Mangla et al. (2020) |  80.68$\pm$0.81 |  90.85$\pm$0.44
DC-LRYang et al. (2021) |  79.56$\pm$0.87 |  90.67$\pm$0.35
CCF Xu et al. (2021b) |  81.85$\pm$0.42 |  91.58$\pm$0.32
ELMOS (ours) | 85.21$\pm$0.38 | 95.02$\pm$0.16

##  4 Experiments

###  4.1 Datasets

miniImageNet contains 600 images over 100 classes, which are divided into 64,
16 and 20 respectively for base, validation and novel sets. tiredImageNet
consists of 779, 165 images belonging to 608 classes, which are further
grouped into 34 higher-level categories with 10 to 30 classes per category.
These categories are partitioned into 20 categories (351 classes), 6
categories (97 classes) and 8 categories (160 classes) respectively for base,
validation and novel sets. CIFAR-FS is derived from CIFAR100 and consists of
100 classes with 600 images per class. The total classes are split into 64, 16
and 20 for base, validation and novel sets. Caltech-UCSD Bird-200-2011(CUB)
has a total number of 11,788 images over 200 bird species. These species are
divided into 100, 50, and 50 for the base, validation and novel sets,
respectively.

###  4.2 Implementation Details

In the experiments, we primarily used ResNet12 architecture with 4 residual
blocks. Each block had 3 convolutional layers with 3×3 kernels. The number of
kernels for the 4 blocks was 64, 160, 320, and 640, respectively. A max-
pooling layer was added at the end of the first three blocks. The last block
was branched with three pooling layers, which respectively modeled different
statistical representations of the images. We opted for the SGD optimizer with
a momentum of 0.9 and a weight decay of 5e-4. The learning rate was
initialized to be 0.025. We trained the network for 130 epochs with a batch
size of 32 in all the experiments. For miniImageNet, tiredImageNet and CIFAR-
FS, the learning rate was reduced by a factor of 0.2 at the 70-$th$ and
100-$th$ epoch. For CUB, the learning rate was reduced by a factor of 0.2 for
every 15 epochs after the 75-$th$ epoch. We randomly sampled 2,000 episodes
from $S_{n}$ with 15 query samples per class for both 5-way 1-shot and 5-shot
evaluations, to produce the mean classification accuracy as well as the 95%
confidence interval.

Table 3: Comparison of results against state-of-the-art methods on miniImageNet, tiredImageNet, and CIFAR-FS dataset. ’-’ means the results were not provided by the authors. The top three results are marked in red, blue and green, respectively. Method | Backbone | Venue | miniImageNet | tiredImageNet | CIFAR-FS
---|---|---|---|---|---
1-shot | 5-shot | 1-shot | 5-shot | 1-shot | 5-shot
Meta-learning |  |  |  |  |  |  |  |
DeepEMD Zhang et al. (2020a) | ResNet12 | CVPR’20 |  65.91$\pm$0.82 |  82.41$\pm$ 0.56 |  71.16$\pm$0.87 |  86.03$\pm$0.58 | - | -
CC+rot Gidaris et al. (2019) | ResNet12 | CVPR’20 |  62.93$\pm$0.45 |  79.87$\pm$0.33 |  70.53$\pm$0.51 |  84.98$\pm 0.36$ |  76.09$\pm$0.30 |  87.83$\pm$0.21
BML Zhou et al. (2021) | ResNet12 | ICCV’21 |  67.04$\pm$0.63 |  83.63$\pm$0.29 |  68.99$\pm$0.50 |  85.49$\pm$0.34 |  73.45$\pm$0.47 |  88.04$\pm$0.33
RENet Kang et al. (2021) | ResNet12 | ICCV’21 |  67.60$\pm$0.44 |  82.58$\pm$0.30 |  71.61$\pm$0.51 |  85.28$\pm$0.35 |  74.51$\pm$0.46 |  86.60$\pm$0.32
MeTALBaik et al. (2021) | ResNet12 | CVPR’21 |  66.61$\pm$0.28 |  81.43$\pm$0.25 |  70.29$\pm$0.40 |  86.17$\pm$0.35 | - | -
DAN Xu et al. (2021a) | ResNet12 | CVPR’21 |  67.76$\pm$0.46 |  82.71$\pm$0.31 |  71.89$\pm$0.52 |  85.96$\pm$0.35 | - | -
IEPT Zhang et al. (2020b) | ResNet12 | ICLR’21 |  67.05$\pm$0.44 |  82.90$\pm$0.30 |  72.24$\pm$0.50 |  86.73$\pm$0.34 | - | -
APP2S Ma et al. (2022) | ResNet12 | AAAI’22 |  66.25$\pm$0.20 |  83.42$\pm$0.15 |  72.00$\pm$0.22 |  86.23$\pm$0.15 |  73.12 $\pm$0.22 |  85.69$\pm$0.16
DeepBDC Xie et al. (2022) | ResNet12 | CVPR’22 |  67.34$\pm$0.43 |  84.46$\pm$0.28 | 72.34$\pm$0.49 | 87.31$\pm$0.32 | - | -
MFS Afrasiyabi et al. (2022) | ResNet12 | CVPR’22 |  68.32$\pm$0.62 |  82.71$\pm$0.46 | 73.63$\pm$0.88 | 87.59$\pm$0.57 | - | -
TPMNWu et al. (2021) | ResNet12 | CVPR’22 |  67.64$\pm$0.63 |  83.44$\pm$0.43 |  72.24$\pm$ 0.70 |  86.55 $\pm$ 0.63 | - | -
HGNN Yu et al. (2022) | ResNet12 | AAAI’22 |  67.02$\pm$0.20 |  83.00$\pm$0.13 |  72.05$\pm$0.23 |  86.49$\pm$0.15 | - | -
DSFNZhang and Huang (2022) | ResNet12 | ECCV’22 |  61.27$\pm$0.71 |  80.13$\pm$0.17 |  65.46$\pm$ 0.70 |  82.41$\pm$0.53 | - | -
MTRBouniot et al. (2022) | ResNet12 | ECCV’22 |  62.69$\pm$ 0.20 |  80.95$\pm$0.14 |  68.44 $\pm$0.23 |  84.20 $\pm$0.16 | - | -
Transfer-learning |  |  |  |  |  |  |  |
Baseline++ Chen et al. (2019) | ResNet12 | ICLR’19 |  48.24$\pm$0.75 |  66.43$\pm$0.63 | - | – | - |
Neg-Cosine Liu et al. (2020) | WRN28 | ECCV’20 |  61.72$\pm$0.81 |  81.79$\pm$0.55 | - | – | - |
RFS Tian et al. (2020) | WRN28 | ECCV’20 |  64.82$\pm$0.60 |  82.14$\pm$0.43 |  71.52$\pm$0.69 |  86.03$\pm$0.49 | - | -
CBM Wang et al. (2020) | ResNet12 | MM’20 |  64.77$\pm$0.46 |  80.50$\pm$0.33 |  71.27$\pm$0.50 |  85.81$\pm$0.34 | - | -
SKD Rajasegaran et al. (2020) | ResNet12 | Arxiv’21 |  67.04$\pm$0.85 |  83.54$\pm$0.54 |  72.03$\pm$0.91 |  86.50$\pm$0.58 |  76.9$\pm$0.9 | 88.9$\pm$0.6
IESung et al. (2021) | ResNet12 | CVPR’21 |  67.28$\pm$0.80 | 84.78$\pm$0.33 |  72.21$\pm$0.90 |  87.08$\pm$0.58 | 77.87$\pm$0.85 | 89.74$\pm$0.57
PAL Ma et al. (2019) | ResNet12 | ICCV’21 | 69.37$\pm$0.64 |  84.40$\pm$0.44 |  72.25$\pm$0.72 |  86.95$\pm$0.47 | 77.1$\pm$0.7 |  88.0$\pm$0.5
CCFXu et al. (2021b) | ResNet12 | CVPR’22 | 68.88$\pm$0.43 | 84.59$\pm$0.30 | - | - | - | -
ELMOS (ours) | ResNet12 | - | 70.30$\pm$0.45 | 86.17$\pm$0.26 | 73.84$\pm$0.49 | 87.98$\pm$0.31 | 78.18$\pm$0.41 | 89.87$\pm$0.31
Table 4: Comparison of results with the most related method under 5-way 1-shot and 5-shot tasks on CIFAR-FS and CUB. Method | CIFAR-FS | CUB
---|---|---
1-shot | 5-shot | 1-shot | 5-shot
EASY |  75.24$\pm$0.20 |  88.38$\pm$0.14 |  77.97$\pm$0.20 |  91.59$\pm$0.10
ELMOS |  78.18$\pm$0.41 |  89.87$\pm$0.31 |  85.21$\pm$0.38 |  95.02$\pm$0.16

###  4.3 Ablation Studies

The effectiveness of our method is attributed to the ensemble of different
branches equipped with multi-order statistics. In this section, we conducted
ablation studies to analyze the effect of the $1^{st}$-order, $2^{nd}$-order
and, $3^{rd}$-order statistical pooling and their combination on the
miniImageNet, CIFAR-FS and CUB datasets. Above methods are respectively
denoted as B_1, B_2, B_3,and ELMOS. Their accuracies under 5-way 1-shot and
5-shot tasks on three datasets are shown in Table 1. From the results, we can
see that: (1) On all three datasets, the test accuracy of $B_{1}$ and $B_{3}$
is higher than $B_{2}$ under the 1-shot task, but the test accuracy of $B_{2}$
is higher than $B_{1}$ and $B_{3}$ under the 5-shot task. The above phenomenon
shows that different order statistics provide different information about the
images. (2) The test accuracy of ELMOS is higher than $B_{1}$, $B_{2}$ and
$B_{3}$ under both 1-shot and 5-shot tasks, which illustrates that different
order statistics complement each other. Combing them can bring more useful
information for classification, resulting in higher classification
performance.

For each individual in the ensemble learning, the optimization is
cooperatively accomplished by the Classification-Based (CB) loss and
Similarity-Based (SB) loss Scott et al. (2021). Hence, we conducted ablation
experiments to analyze the contribution of each loss on three benchmark
datasets: miniImageNet, CIFAR-FS and CUB. Subsequently, we pre-trained the
model respectively with CB and SB loss alone and their combination, resulting
in three methods denoted as CB, SB and CB&SB. The test accuracies under
different methods are shown in Figure 3. The test results show that the
accuracy of CB&SB is higher than CB and SB, which implies that both
classification-based and similarity-based losses play important roles in our
method.

###  4.4 Comparison with the Most Related Method

Our method is most related to EASY Bendou et al. (2022), which is also a FSC
ensemble learning method in context of transfer learning. The comparison of
results between them on CIFAR-FS and CUB datasets is shown in Table 4. From
the results, we can see that our method beats EASY by a very large margin
under both 1-shot and 5-shot tasks. Please note that our method is more
efficient that EASY, because EASY needs to pre-train multiple individual
networks, which spends much more pre-training time than our method.

###  4.5 Comparison with State-of-the-Art Methods

We compare the performance of our method with several state-of-the-art
methods. These methods are either meta-learning based or transfer-learning
based. The comparison of results is shown in Table 2 and Table 3. From Table
2, we can see the performance of our method ranks at the top under both 1-shot
and 5-shot tasks on CUB. Specifically, our method exceeds the second-best
model DeepBDC by 1.2% and 1.0% respectively in 1-shot and 5-shot settings.
From Table 3, we can see that our method beats state-of-the-art methods under
both 5-way 1-shot and 5-way 5-shot tasks on the dataset of miniImageNet,
tiredImagegNet, and CIFAR-FS. Specifically, on miniImageNet, PAL and IE behave
the second best respectively in 1-shot and 5-shot settings. Our method beats
them by 0.93% and 1.39%. On tiredImageNet, our method outperforms the second-
best MFS by 0.21% and 0.39% respectively in 1-shot and 5-shot settings. On
CIFAR-FS, our method achieves 0.31% and 0.13% improvement over IE for 1-shot
and 5-shot respectively. In brief, our method consistently outperforms the
state-of-the-art FSC methods under both 1-shot and 5-shot tasks on multiple
datasets. The promising results are achieved because of the generalization
representation obtained by ensemble learning with multi-order on the base set.

##  5 Conclusion

This paper analyzes the underlying work mechanism of ensemble learning in few-
shot classification. A theorem is provided to illustrate that the true error
on the novel classes can be reduced with ensemble learning on the base set,
given the domain divergence between the base and the novel classes. Multi-
order statistics on image features are further introduced to produce learning
individuals to get an effective ensemble learning design. Comprehensive
experiments on multiple benchmarks have illustrated that different-order
statistics can generate diverse learning individuals due to their
complementarity. The promising FSC performance with ensemble learning on the
base set has validated the proposed theorem.

## References

  * Afrasiyabi et al. [2020] Arman Afrasiyabi, Jean-François Lalonde, and Christian Gagné.  Associative alignment for few-shot image classification.  In Proceedings of European Conference on Computer Vision, pages 18–35, Glasgow, UK, November 2020. Springer.
  * Afrasiyabi et al. [2022] Arman Afrasiyabi, Hugo Larochelle, Jean-François Lalonde, and Christian Gagné.  Matching feature sets for few-shot image classification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9014–9024, New Orleans, USA, June 2022. IEEE.
  * Agarwal et al. [2021] Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, , Rich Caruana, and Geoffrey E Hinton.  Neural additive models: Interpretable machine learning with neural nets.  In Proceedings of 34th Annual Conference on Neural Information Processing Systems, pages 4078–4088, 4699–4711, December 2021. Neural Information Processing Systems Foundation.
  * Baik et al. [2021] Sungyong Baik, Janghoon Choi, Heewon Kim, Dohee Cho, Jaesik Min, and Kyoung Mu Lee.  Meta-learning with task-adaptive loss function for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9465–9474, Nashville, USA, June 2021. IEEE.
  * Bendou et al. [2022] Yassir Bendou, Yuqing Hu, Raphael Lafargue, Giulia Lioi, Stéphane Pateux, and Vincent Gripon.  Easy—ensemble augmented-shot-y-shaped learning: State-of-the-art few-shot classification with simple components.  Journal of Imaging, 8(7):179, 2022.
  * Bertinetto et al. [2019] Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi.  Meta-learning with differentiable closed-form solvers.  In Proceedings of 7th International Conference on Learning Representations, New Orleans, USA, May 2019. International Conference on Learning Representations.
  * Bouniot et al. [2022] Quentin Bouniot, Ievgen Redko, Romaric Audigier, Angélique Loesch, and Amaury Habrard.  Improving few-shot learning through multi-task representation learning theory.  In Proceedings of European Conference on Computer Vision, pages 435–452, Tel Aviv, Israel, October 2022. Springer.
  * Chen et al. [2019] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang.  A closer look at few-shot classification.  In Proceedings of 7th International Conference on Learning Representations, New Orleans, USA, May 2019. International Conference on Learning Representations.
  * Cui et al. [2017] Yin Cui, Feng Zhou, Jiang Wang, Xiao Liu, Yuanqing Lin, and Serge Belongie.  Kernel pooling for convolutional neural networks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2921–2930, Venice, Italy, February 2017. IEEE.
  * Dvornik et al. [2019] Nikita Dvornik, Cordelia Schmid, and Julien Mairal.  Diversity with cooperation: Ensemble methods for few-shot classification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3723–3731, Seoul, Korea, February 2019. IEEE.
  * Finn et al. [2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In Proceedings of International Conference on Machine Learning, pages 1126–1135, Sydney,Australia, July 2017.
  * Gidaris et al. [2019] Spyros Gidaris, Andrei Bursuc, Patrick Pérez, and Matthieu Cord.  Boosting few-shot visual learning with self-supervision.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8059–8068, Seoul, Korea, February 2019. IEEE.
  * Horváth et al. [2021] Miklós Z Horváth, Mark Niklas Müller, Marc Fischer, and Martin Vechev.  Boosting randomized smoothing with variance reduced classifiers.  arXiv preprint arXiv:2106.06946, 2021.
  * Kang et al. [2021] Dahyun Kang, Heeseung Kwon, Juhong Min, and Minsu Cho.  Relational embedding for few-shot classification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8822–8833, Nashville, USA, June 2021. IEEE.
  * Kifer et al. [2004] Daniel Kifer, Shai Ben-David, and Johannes Gehrke.  Detecting change in data streams.  In Proceedings of the 31st International Conference on Very Large Databases, pages 180–191, Toronto, Canada;, September 2004. Morgan Kaufmann.
  * Liu et al. [2020] Bin Liu, Yue Cao, Yutong Lin, , Mingsheng Long, and Han Hu.  Negative margin matters: Understanding margin in few-shot classification.  In Proceedings of European Conference on Computer Vision, pages 438–455, Glasgow, UK, November 2020. Springer.
  * Ma et al. [2019] Jiawei Ma, Hanchen Xie, Guangxing Han, Shih-Fu Chang, Aram Galstyan, and Wael Abd-Almageed.  Partner-assisted learning for few-shot image classification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10573–10582, Seoul, Korea, February 2019. IEEE.
  * Ma et al. [2022] Rongkai Ma, Pengfei Fang, Tom Drummond, and Mehrtash Harandi.  Adaptive poincaré point to set distance for few-shot classification.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1926–1934, Austin, Texas, August 2022. AAAI.
  * Mangla et al. [2020] Puneet Mangla, Nupur Kumari, Abhishek Sinha, Mayank Singh, , and Vineeth N Balasubramanian.  Charting the right manifold: Manifold mixup for few-shot learning.  In Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV), pages 2218–2227, Snowmass, USA, March 2020. IEEE.
  * Rajasegaran et al. [2020] Jathushan Rajasegaran, Salman Khan, , Fahad Shahbaz Khan, and Mubarak Shah.  Self-supervised knowledge distillation for few-shot learning.  arXiv preprint arXiv:2006.09785, 2020.
  * Rongkai Ma [2022] Gil Avraham Yan Zuo Rongkai Ma, Pengfei Fang.  Learning instance and task-aware dynamic kernels for few-shot learning.  arXiv preprint arXiv:2112.03494, 2022.
  * Scott et al. [2021] Tyler R Scott, Andrew C Gallagher, and Michael C Mozer.  von mises-fisher loss: An exploration of embedding geometries for supervised learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10612–10622, Nashville, USA, June 2021. IEEE.
  * Shiryaev [2016] Albert N Shiryaev.  Probability-1, volume 95.  Springer, 2016.
  * Snell et al. [2017] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  In Proceedings of 31st Annual Conference on Neural Information Processing Systems, pages 4078–4088, Long Beach, USA, December 2017. Neural Information Processing Systems Foundation.
  * Sung et al. [2018] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales.  Learning to compare: relation network for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1199–1208, Salt Lake City, USA, June 2018. IEEE.
  * Sung et al. [2021] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales.  Exploring complementary strengths of invariant and equivariant representations for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10836–10846, Nashville, USA, June 2021. IEEE.
  * Tian et al. [2020] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola.  Rethinking few-shot image classification: a good embedding is all you need?  In Proceedings of European Conference on Computer Vision, pages 266–282, Glasgow, UK, November 2020. Springer.
  * Tseng et al. [2020] Hung-Yu Tseng, Hsin-Ying Lee, Jia-Bin Huang, and Ming-Hsuan Yang.  Cross-domain few-shot classification via learned feature-wise transformation.  arXiv preprint arXiv:2001.08735, 2020.
  * Wang et al. [2019] Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens van der Maaten.  Simpleshot: Revisiting nearest-neighbor classification for few-shot learning.  arXiv preprint arXiv:1911.04623, 2019.
  * Wang et al. [2020] Zeyuan Wang, Yifan Zhao, Jia Li, and Yonghong Tian.  Cooperative bi-path metric for few-shot learning.  In Proceedings of the 29th ACM International Conference on Multimedia, pages 1524–1532, Seattle, USA, October 2020. ACM.
  * Wertheimer et al. [2021] Davis Wertheimer, Luming Tang, and Bharath Hariharan.  Few-shot classification with feature map reconstruction networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8012–8021, Nashville, USA, 2021.
  * Wu et al. [2021] Jiamin Wu, Tianzhu Zhang, Yongdong Zhang, and Feng Wu.  Task-aware part mining network for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8433–8442, Nashville, USA, June 2021. IEEE.
  * Xie et al. [2022] Jiangtao Xie, Fei Long, Jiaming Lv, Qilong Wang, and Peihua Li.  Joint distribution matters: deep brownian distance covariance for few-shot classification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7962–7971, New Orleans, USA, June 2022. IEEE.
  * Xu et al. [2021a] Chengming Xu, Yanwei Fu, Chen Liu, Chengjie Wang, Jilin Li, Feiyue Huang, Li Zhang, and Xiangyang Xue.  Learning dynamic alignment via meta-filter for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5182–5191, Nashville, USA, June 2021. IEEE.
  * Xu et al. [2021b] Jing Xu, Xinglin Pan, Xu Luo, Wenjie Pei, and Zenglin Xu.  Exploring category-correlated feature for few-shot image classification.  arXiv preprint arXiv:2112.07224, 2021.
  * Yang et al. [2013] Jing Yang, Xiaoqin Zeng, Shuiming Zhong, and Shengli Wu.  Effective neural network ensemble approach for improving generalization performance.  IEEE transactions on neural networks and learning systems, 24(6):878–887, 2013.
  * Yang et al. [2021] Shuo Yang, Lu Liu, and Min Xu.  Free lunch for few-shot learning: distribution calibration.  In Proceedings of 9th International Conference on Learning Representations, New Orleans, USA, May 2021. International Conference on Learning Representations.
  * Yu et al. [2022] Tianyuan Yu, Sen He, Yi-Zhe Song, and Tao Xiang.  Hybrid graph neural networks for few-shot learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3179–3187, Austin, Texas, 2022.
  * Zhang and Huang [2022] Tao Zhang and Wu Huang.  Kernel relative-prototype spectral filtering for few-shot learning.  In Proceedings of European Conference on Computer Vision, pages 541–557, Tel Aviv, Israel, October 2022.
  * Zhang et al. [2020a] Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen.  Deepemd: few-shot image classification with differentiable earth mover’s distance and structured classifiers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12203–12213, Seattle, USA, June 2020. IEEE.
  * Zhang et al. [2020b] Manli Zhang, Jianhong Zhang, , and Songfang Huang.  Iept: Instance-level and episode-level pretext tasks for few-shot learning.  In Proceedings of 7th International Conference on Learning Representations, Addis Ababa, Ethiopian Empire, May 2020. International Conference on Learning Representations.
  * Zhou et al. [2021] Ziqi Zhou, Xi Qiu, Jiangtao Xie, Jianan Wu, and Chi Zhang.  Binocular mutual learning for improving few-shot classification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8402–8411, Seoul, Korea, February 2021. IEEE.

##  6 Supplementary Material

###  6.1 Proof of Theorem 1

###### Proof 1

The expected error with $\overline{h}$ on $S_{b}$ and $S_{n}$ are:

$\displaystyle e_{b}(\overline{h})=e_{S_{b}}(\overline{h},f_{b})=E_{x\in
S_{b}}[\left|\overline{h}(x)-f_{b}(x)\right|]$ (14) $\displaystyle
e_{n}(\overline{h})=e_{S_{n}}(\overline{h},f_{n})=E_{x\in
S_{n}}[\left|\overline{h}(x)-f_{n}(x)\right|].$

Then, we will get the following formula:

$\displaystyle
e_{n}(\overline{h})=e_{n}(\overline{h})+e_{b}(\overline{h})-e_{b}(\overline{h})+e_{S_{b}}(\overline{h},f_{n})-e_{S_{b}}(\overline{h},f_{n})$
(15) $\displaystyle\leq
e_{b}(\overline{h})+\left|e_{S_{b}}(\overline{h},f_{n})-e_{S_{b}}(\overline{h},f_{b})\right|+$
$\displaystyle\left|e_{S_{n}}(\overline{h},f_{n})-e_{S_{b}}(\overline{h},f_{n})\right|$
$\displaystyle\leq
e_{b}(\overline{h})+E_{X\in{S_{b}}}\left|f_{n}(x)-f_{b}(x)\right|+$
$\displaystyle\left|e_{S_{n}}(\overline{h},f_{n})-e_{S_{b}}(\overline{h},f_{n})\right|$
$\displaystyle\leq
e_{b}(\overline{h})+E_{X\in{S_{b}}}\left|f_{n}(x)-f_{b}(x)\right|+$
$\displaystyle\int\left|\eta_{b}(x)-\eta_{n}(x)\right|\left|\overline{h}(x)-f_{n}(x)\right|dx$
$\displaystyle\leq
e_{b}(\overline{h})+E_{X\in{S_{b}}}\left|f_{n}(x)-f_{b}(x)\right|+\mathcal{L}(S_{b},S_{n}).$

The expected error on $S_{b}$ with any learner $h_{o}$ of ensemble learning is
calculated as:

$\displaystyle e_{b}(h_{o})=\int(h_{o}(x)-h^{*}(x))^{2}\eta_{b}(x)dx,$ (16)

where $\eta_{b}(x)$ is the density functions of $S_{b}$, The average error on
$S_{b}$ with the learners of ensemble learning is:

$\displaystyle
e_{b}(h)=\sum_{o=1}^{O}\alpha_{o}\int(h_{o}(x)-h^{*}(x))^{2}\eta_{b}(x)dx.$
(17)

Recall that $\overline{h}=\alpha_{o}\sum_{o=1}^{O}h_{o}$, then the expected
error on $S_{b}$ with $\overline{h}$ is calculated as:

$\displaystyle
e_{b}(\overline{h})=\int(\overline{h}(x)-h^{*}(x))^{2}\eta_{b}(x)dx$ (18)
$\displaystyle=\int(\alpha_{o}\sum_{o=1}^{O}h_{o}(x)-h^{*}(x))^{2}\eta_{b}(x)dx$
$\displaystyle\leq\sum_{o=1}^{O}\alpha_{o}\int(h_{o}(x)-h^{*}(x))^{2}\eta_{b}(x)dx\leq
e_{b}(h).$

###  6.2 Proof of Proposition 1

###### Proof 2

The Gaussian distribution of the random variable $t$ is expressed as:

$f(t)=\frac{1}{\sqrt{2\pi}\Sigma}e^{\\{-\frac{(t-\mu)^{2}}{2\Sigma^{2}}\\}}.$
(19)

According to the definition in Equation (3), the first characteristic function
of random variable $t$ is calculated as:

$\displaystyle\phi(s)=\int_{-\infty}^{+\infty}\frac{1}{\sqrt{2\pi}\Sigma}e^{\\{-\frac{(t-\mu)^{2}}{2\Sigma^{2}}\\}}e^{st}dt$
(20)
$\displaystyle\xlongequal[]{t^{\prime}=t-\mu}\int_{-\infty}^{+\infty}\frac{1}{\sqrt{2\pi}\Sigma}e^{\\{-\frac{(t^{\prime})^{2}}{2\Sigma^{2}}\\}}e^{s(t^{\prime}+\mu)}dt^{\prime}$
$\displaystyle=e^{\mu
s}\int_{-\infty}^{+\infty}\frac{1}{\sqrt{2\pi}\Sigma}e^{\\{-\frac{(t^{\prime})^{2}}{2\Sigma^{2}}\\}}e^{st^{\prime}}dt^{\prime}$
$\displaystyle=e^{\mu
s}\frac{1}{\sqrt{2\pi}\Sigma}\int_{-\infty}^{+\infty}e^{\\{-\frac{t^{\prime
2}}{2\Sigma^{2}}+st^{\prime}\\}}dt^{\prime}.$

The common Gaussian integral formula is expressed as:

$\displaystyle\int_{-\infty}^{+\infty}e^{(-Ax^{2}\pm
2Bx-C)}dx=\sqrt{\frac{\pi}{A}e^{(-\frac{AC-B^{2}}{A})}}.$ (21)

In the right side of Equation (21), let $A=\frac{1}{2\Sigma^{2}}$, $B=s/2$,
$C=0$, the Euqation (20) becomes:

$\displaystyle\phi(s)=e^{\mu s}e^{\frac{1}{2}\Sigma^{2}s^{2}}.$ (22)

The second characteristic function of $\psi(s)$ is formulated as:

$\displaystyle\psi(s)=ln\phi(s)=ln(e^{\mu
s}e^{\frac{1}{2}\Sigma^{2}s^{2}s})=\mu s+\frac{1}{2}\Sigma^{2}s^{2}.$ (23)

Compare Equation (23) with Equation (5), the following coefficients of the
term $s^{o}$ can be obtained in Equation (5):

$\displaystyle c_{1}=\mu,c_{2}=\Sigma^{2},c_{o}=0\quad(o=3,4,...).$ (24)

###  6.3 More Experiments

(a) 5-way 1-shot

(b) 5-way 5-shot

Figure 4: Test accuracy (%) under different values of the parameter
$\alpha_{2}$ in the setting of 5-way 1-shot and 5-shot on three FSC datasets.

(a) 5-way 1-shot

(b) 5-way 5-shot

Figure 5: Test accuracy (%) under different values of the parameter
$\alpha_{3}$ in the setting of 5-way 1-shot and 5-shot on three FSC datasets.
Figure 6: Image reconstruction of features respectively represented by
$1^{st}$-order, $2^{nd}$-order, $3^{rd}$-order statistics.

(a) Baseline

(b) Our method

Figure 7: T-SNE visualization of features on unseen samples of Baseline and our method.  Table 5: Comparison of different methods under cross-domain scenario. Method |  miniImageNet $\rightarrow$ CUB
---|---
1-shot | 5-shot
Prototypical Snell et al. [2017]† |  36.61$\pm$0.53 |  55.23$\pm$0.83
Relational Sung et al. [2018]† |  44.07$\pm$0.77 |  59.46$\pm$0.71
MetaOptNet Bertinetto et al. [2019]†† |  44.79$\pm$0.75 |  64.98$\pm$0.68
IEPT Zhang et al. [2020b] | 52.68$\pm$0.56 | 72.98 $\pm$0.40
FPN Wertheimer et al. [2021] | 51.60$\pm$0.21 | 72.97$\pm$0.18
BML Zhou et al. [2021] | - |  72.42$\pm$0.54
Baseline++ Chen et al. [2019] | - |  62.04$\pm$0.76
SimpleShot Wang et al. [2019]†† | 48.56 | 65.63
S2M2 Mangla et al. [2020] |  48.24$\pm$0.84 |  70.44$\pm$0.75
Neg-Cosine Liu et al. [2020] | - |  67.03$\pm$0.76
GNN+FT Tseng et al. [2020] |  47.47$\pm$0.75 |  66.98$\pm$0.68
ELMOS(ours) | 53.73$\pm$0.47 | 74.37$\pm$0.37

####  6.3.1 Parameter Analysis

The effect of each branch is controlled by the parameters $\alpha_{1}$,
$\alpha_{2}$ and $\alpha_{3}$ in Equation (10). Since the first branch
modeling the $1^{st}$-order statistic is the main branch, we set its
corresponding parameter to 1. Subsequently, we first fixed the value of
$\alpha_{3}$ to be 1, and varied the value of $\alpha_{2}$ between [0, 1] with
an interval of 0.1. The test accuracy under different values is shown in
Figure 4. When $\alpha_{2}$ is 1, the highest performance on miniImageNet
under both 1-shot and 5-shot tasks can be achieved. When $\alpha_{2}$ is 0.3,
we get the highest performance on CUB and CIFAR-FS under both 1-shot and
5-shot tasks. Next, we fixed the value of $\alpha_{2}$ to be 1 on
miniImageNet, 0.3 on CIFAR-FS and CUB, and varied the value of $\alpha_{3}$
between [0, 1] with an interval of 0.1. The test accuracy under different
values is shown in Figure 5. When $\alpha_{3}$ is 1, we get the highest
performance on all three datasets under both 1-shot and 5-shot tasks.

####  6.3.2 Image Reconstruction of Features

The effectiveness of our method is mainly attributed to the diversity of
$1^{st}$-order, $2^{nd}$-order, and $3^{rd}$-order statistic features. We used
the technique of deep image prior to respectively invert different-order
statistic features after the pre-training into RGB images. The reconstruction
results are shown in Figure 6. From the results, we notice that as the order
of statistic feature becomes higher, the reconstructed images become more
smooth. The above phenomenon illustrates that $2^{nd}$-order and
$3^{rd}$-order statistic features are more robust to the singularity variation
such as the noise point than the $1^{st}$-order statistic feature. By
comparison, the $1^{st}$-order statistic feature has stronger ability of
capturing the details of the images than $2^{nd}$-order and $3^{rd}$-order
statistic features. The above analysis has shown that $1^{st}$-order,
$2^{nd}$-order, and $3^{rd}$-order statistic features are complementary.

####  6.3.3 t-SNE Visualization of Features

To show the performance of our method, we visualize the features of the novel
class samples in comparison with the Baseline. Herein, the Baseline pre-
trained the backbone network only with the global average pooling. We randomly
selected 5 classes and 200 samples per class from CIFAR-FS and visualize the
features of the samples using t-SNE. The visualization results are shown in
Figure 7. From the results, we can see that five classes can well separate
from each other in our feature space compared to with the Baseline, which
illustrates that our method can extract better features for unseen novel
classes compared with the Baseline.

####  6.3.4 Comparison of Cross-domain Performance

As stated in our Theorem 1, there exists a domain shift between the base and
novel classes. In the former test, the base and novel classes are in the same
domain, which has a smaller domain divergence than the ones in the different
domains. Now, we large the domain divergence to evaluate our method by doing
cross-domain FSC. Following the protocol in Chen et al. [2019], the model was
trained on miniImageNet and then evaluated on the novel classes in CUB. The
comparison of results is shown in Table 5. From the results, we can see that
our method is better than all the compared methods under 1-shot and 5-shot
tasks. Specifically, our method outperforms the best method of IEPT with the
improvement of 1.05% and 1.39% respectively. Our method does not concern the
domain divergence, but we can also get good cross-domain performance by
implementing ensemble learning to decrease the generalization error on base
classes, because it is also an important term for the true error on novel
classes.


# Compressive Feature Selection for Remote Visual Multi-Task Inference

###### Abstract

Deep models produce a number of features in each internal layer. A key problem
in applications such as feature compression for remote inference is
determining how important each feature is for the task(s) performed by the
model. The problem is especially challenging in the case of multi-task
inference, where the same feature may carry different importance for different
tasks. In this paper, we examine how effective is mutual information (MI)
between a feature and a model’s task output as a measure of the feature’s
importance for that task. Experiments involving hard selection and soft
selection (unequal compression) based on MI are carried out to compare the MI-
based method with alternative approaches. Multi-objective analysis is provided
to offer further insight.

Index Terms—  Feature selection, feature compression, multi-task inference

##  1 Introduction

The emergence of high-end mobile/edge devices with Artificial Intelligence
(AI) hardware opened new doors for AI applications at the edge [1]. However,
constraints on the battery and processing power limit the complexity of Deep
Neural Networks (DNNs) that such devices can run. One promising solution to
this challenge is to divide the DNN workload between the edge device and a
more powerful server (e.g., in the cloud), by having the edge device compute
features and send them to the cloud for further analysis. This approach is
known in the literature under various names, such as collaborative
intelligence [2], collaborative inference [3], computation offloading [4],
split computing [5], and semantic communications [5]. Since edge-side DNN
features have to be sent to the cloud through a capacity-limited channel, the
key challenge becomes how to compress these features without affecting DNN
accuracy [6, 7]. Increasing interest in these topics has motivated
standardization efforts such as MPEG Video Coding for Machines (MPEG VCM) [8],
MPEG Feature Coding for Machines (MPEG FCM) [9] and JPEG AI [10].

An important challenge in coding for machines is related to multi-task
inference. Multi-task DNNs [11, 12, 13] usually consist of a shared backbone
(foundation model) that produces rich features capable of supporting multiple
tasks, and multiple heads, each of which performs a certain task. If all tasks
are required simultaneously, the solution is to compress all the features. But
if only one or a few tasks are required, compressing all features is an
overkill; it would be better to find which features are relevant for the
particular task(s), and only compress those. This requires the ability to
quantify the importance of features for the particular task(s). In this paper,
we examine the effectiveness of mutual information (MI) [14] between a feature
and a task output as a measure of feature importance, and then describe how it
can be used for feature selection. One approach, called hard selection, simply
retains a set of the most important features and discards others. The other
approach, called soft selection, compresses some features more and others
less, according to their importance.

The paper is organized as follows. In Section 2, we review related work on
feature selection in machine learning. In Section 3, we first present our MI
estimator for visual multi-task models. We then present our approach to hard
and soft feature selection. Experiments are presented in Section 4, followed
by conclusions in Section 5.

##  2 Related work

The topic of feature selection has a long history in machine learning [15]. In
fact, feature selection based on MI is well-established in machine learning
[16, 17, 18]. However, there is little prior work on feature selection in DNNs
based on MI, with notable exceptions being [19, 20], both targeted at filter
pruning. Note that feature selection is complementary to feature pruning:
selecting a group of features is equivalent to pruning other (non-selected)
features. However, filter pruning and feature pruning are not equivalent. For
example, consider removing a feature from the output of an add block in a
ResNet [21] structure. This is not equivalent to the removal of the
corresponding filter in the immediately preceding convolutional layer, because
the corresponding channel in the feature tensor is also influenced by the
feature carried over through the skip connection. Nonetheless, popular
criteria for filter pruning, such as norm-based and proximity-based methods,
can also be used for feature pruning (and hence feature selection), by
applying the corresponding methodology to features rather than filter kernels.

Existing approaches to filter pruning can roughly be divided into norm-based
and proximity-based methods. Norm-based methods [22, 23, 24, 25] assume that a
large norm is an indicator of the importance of a particular filter in a given
network. On the other hand, proximity-based approaches [26, 27, 28] are based
on the notion that similar filters may be redundant and can be removed.
Feature pruning has followed similar logic. Existing criteria for feature
pruning include small variance [29], similarity to other features [30],
influence on reconstructing subsequent features [31], low rank [32], or
attention [33].

A fundamental problem with the existing feature/filter pruning approaches is
that they do not account for the task being performed by the DNN. For example,
norm-based methods only consider the characteristics of the feature itself,
not what it is used for. Proximity-based methods consider the relationships
among features, but again do not account for what they are used for. In a
single-task DNN, where all features are used for one task only, this
shortcoming may not be obvious, but it is clearly revealed in multi-task DNNs
where various features are responsible for different tasks to a varying
degree. This is what makes MI an attractive tool for feature selection in
multi-task DNNs, because it allows us to connect a feature with a specific
task.

Despite these potential advantages of MI, the scarcity of prior work on MI-
based feature selection in DNNs is likely due to the fact that both the
intermediate feature space and the output space of a DNN are often high-
dimensional, and this is known to be a challenging setup for estimating MI
[16, 34, 35, 36]. Practical MI estimators often need to be tailored to the
problem at hand. Our work fills this gap and presents an MI estimator suited
to visual multi-task DNNs.

##  3 Proposed methods

###  3.1 Estimating MI in visual multi-task DNNs

We consider visual multi-task DNNs where the features come in the form of a
feature tensor whose channels are feature maps $X$, and the output is an image
$Y$, for example, a segmentation map. The high dimensionality of the features
and the output makes estimating joint probabilities, and therefore MI as well,
challenging due to the “curse of dimensionality” [37]. To address this
challenge, we use the following strategy. First, we divide the output image
into $M\times M$ patches (denoted $\widetilde{Y}$) and also divide feature
maps into spatially corresponding $N\times N$ patches (denoted
$\widetilde{X}$), as illustrated in Fig. 1. We then cluster the output patches
into $K$ clusters, with cluster index denoted $\widehat{Y}$. This effectively
creates an output patch classifier with $K$ classes. If $K$ is reasonably
small, one can use MI estimators for DNN classifiers111We use the binning
estimator from [38]. to estimate $I(\widetilde{X};\widehat{Y})$. Note that
both splitting into patches and clustering reduce the MI, so our estimate
$I(\widetilde{X};\widehat{Y})$ will be a lower bound on the true $I(X;Y)$.

To assess the proposed MI estimator, we compare it with the true MI in a few
cases where the true MI is known. For jointly Gaussian random variables
$X\in\mathbb{R}^{n}$ and $Y\in\mathbb{R}^{m}$, with the covariance matrix of
$[X,Y]^{\top}$ defined as:

$\boldsymbol{\Sigma}=\begin{bmatrix}\boldsymbol{\Sigma}_{X}&\boldsymbol{\Sigma}_{XY}\\\
\boldsymbol{\Sigma}_{YX}&\boldsymbol{\Sigma}_{Y}\end{bmatrix},$ (1)

where $\boldsymbol{\Sigma}_{X}$ and $\boldsymbol{\Sigma}_{Y}$ are covariance
matrices of $X$ and $Y$, respectively, and $\boldsymbol{\Sigma}_{XY}$ and
$\boldsymbol{\Sigma}_{YX}$ are the corresponding cross-covariances, MI is
given by [39]:

$I(X;Y)=\frac{1}{2}\log\left(\frac{\det(\boldsymbol{\Sigma}_{X})\det(\boldsymbol{\Sigma}_{Y})}{\det(\boldsymbol{\Sigma})}\right).$
(2)

We examine the MI estimates in cases where component-wise correlation between
$X$ and $Y$ is given by $\mathsf{corr}(X_{i},Y_{j})=\delta_{ij}\rho$, where
$\rho\in(-1,1)$, and $\delta_{ij}$ is the Kronecker delta.

Fig. 1: Mutual information between an $N\times N$ feature patch
$\widetilde{X}$ and the corresponding $M\times M$ output patch $\widetilde{Y}$

First we focus on the scalar case ($m=n=1$), where

$\boldsymbol{\Sigma}=\begin{bmatrix}1&\rho\\\ \rho&1\end{bmatrix}$ (3)

and (2) becomes:

$I(X;Y)=\log\sqrt{\frac{1}{1-\rho^{2}}}.$ (4)

We examine the effect on the MI estimate of clustering $Y$ into $K$ clusters
using K-means clustering [40]. The true MI and the MI estimates for several
values of $K$ obtained from $4\times 10^{5}$ samples of the mentioned
distributions are shown at the top of Fig. 2. It can be seen that the MI
estimates are always below the true MI (i.e., they provide lower bounds), but
as the number of clusters increases, the estimate gets closer to the true MI.
We repeated the experiments with random state for the K-means clustering 5
times and noticed that the MI estimates are fairly consistent - their variance
was $\approx 10^{-8}$.

Patching, as shown in Fig. 1, is equivalent to extracting a subspace
corresponding to a few coordinates of the original vector. Hence, we examine
the effect of both patching and clustering on 2D Gaussian vectors.
Specifically, we consider standard isotropic 2D Gaussians
$X=[X_{1},X_{2}]^{\top}$ and $Y=[Y_{1},Y_{2}]^{\top}$
($\boldsymbol{\Sigma}_{X}=\boldsymbol{\Sigma}_{Y}=\mathbf{I}$) correlated as
above, such that

$\boldsymbol{\Sigma}=\begin{bmatrix}1&0&\rho&0\\\ 0&1&0&\rho\\\ \rho&0&1&0\\\
0&\rho&0&1\end{bmatrix}.$ (5)

In this case, the true MI (2) becomes

$I(X;Y)=\log\sqrt{\frac{1}{(1-\rho^{2})^{2}}}.$ (6)

To examine the effect of patching, $X$ and $Y$ are split component-wise, and
$X_{i}$, $Y_{i}$ are considered samples of patched univariate Gaussian random
variables $\widetilde{X}$ and $\widetilde{Y}$. Note that $\widetilde{X}$ and
$\widetilde{Y}$ are jointly Gaussian with covariance given by (3), so their MI
is given by (4). Since the expression in (4) is always less than or equal to
(6), we see that patching by itself reduces the MI. Then $\widetilde{Y}$ is
clustered into $K$ clusters to obtain $\widehat{Y}$, and
$I(\widetilde{X};\widehat{Y})$ is then computed. Fig. 2 bottom illustrates
true $I(X;Y)$, the true $I(\widetilde{X};\widetilde{Y}$), and the estimates
obtained by clustering $10^{6}$ samples of $(X,Y)$ and
$(\widetilde{X},\widetilde{Y})$. As before, increasing the number of clusters
increases the MI estimate, but because patched univariate variables have a
lower true MI than the original 2D variables, the estimates never go above
this value. Again, estimates provide a lower bound on the true MI for 2D
variables, but the bound is now looser.

Fig. 2: True and estimated MI for 1D Gaussian random variables (top) and 2D
Gaussian random vectors (bottom).

In summary, MI estimates obtained by patching and clustering provide lower
bounds on the true MI. The bounds are not tight, but they preserve the order –
the higher the true MI, the higher the estimate. Since our feature selection
in the remainder of the paper is based on sorting MI estimates, these
estimates prove to be useful for our purpose, as will be demonstrated by the
experiments.

###  3.2 Feature selection

Let $X_{i}$ be the $i$-th feature map and $Y_{j}$ be the $j$-th task output of
a multi-task visual DNN. We define the importance of $X_{i}$ for $Y_{j}$ as
$I_{i,j}=I(\widetilde{X}_{i};\widehat{Y}_{j})$, where $\widetilde{X}_{i}$
represents $N\times N$ patches from $X_{i}$ and $\widehat{Y}_{j}$ represents
clustered output patches, as discussed earlier. Note that this definition of
feature importance is a function of both the feature $i$ and the task $j$.
This is in contrast with established norm-based and proximity-based methods,
which are functions only of the feature, but not the task. Further, norm- and
proximity-based measures of importance are not scale invariant, because for
any scale factor $s\not\in\\{0,1\\}$, $\lVert sX_{i}\rVert=|s|\cdot\lVert
X_{i}\rVert\neq\lVert X_{i}\rVert$ and $\lVert
sX_{i}-sX_{k}\rVert=|s|\cdot\lVert X_{i}-X_{k}\rVert\neq\lVert
X_{i}-X_{k}\rVert$. Hence, scaling a feature would seem to change its
importance, according to norm- and proximity-based measures. Meanwhile, since
scaling by a factor $s\neq 0$ is an invertible operation and MI is invariant
to invertible transformations [14], we have $I(sX_{i};Y)=I(X_{i};Y)$, showing
that our measure of importance is scale-invariant.

Hard selection. As noted earlier, by hard selection we mean retaining only a
subset of features. Hence, for a feature tensor with $C$ feature maps,
selecting $C^{\prime}$ features essentially means removing the remaining
$C-C^{\prime}$ features. Suppose we are interested in task $j$. The list of
feature importances for task $j$ is $\\{I_{1,j},I_{2,j},...,I_{C,j}\\}$.
Hence, the order of feature importance for task $j$ is

$\mathcal{O}_{j}=(O_{1,j},O_{2,j},...,O_{C,j})=\mathsf{sort}\\{I_{1,j},I_{2,j},...,I_{C,j}\\}.$
(7)

Hard selection then simply means retaining the $C^{\prime}$ top-ranked
features from $\mathcal{O}_{j}$.

Soft selection. In this work, soft selection refers to the following process.
First, a certain number of features is selected as the most important (or
base) features, and the remaining features are declared less important (or
enhancement) features. Base features are quantized to 8-bits, but not
compressed any further. Meanwhile, stronger compression is applied to
enhancement features, as illustrated in Fig. 3. With the notation defined
above, if we are interested in task $j$, then the $C^{\prime}$ top-ranked
features from $\mathcal{O}_{j}$ will be declared base features and the
remaining $C-C^{\prime}$ features will be declared enhancement features, and
compressed more heavily. This way, enhancement features are not completely
removed as in hard selection, but compressed instead. Moreover, the amount of
compression can be controlled, which allows this approach to operate in the
range between complete feature removal (hard selection) and full feature
preservation. This is a useful property to have when the constraint on the
system is specified as the maximum bitrate, rather than the number of
features.

Fig. 3: Illustration of soft feature selection.

In terms of implementing soft selection, any feature compression method can be
used for the enhancement features. In this work, the enhancement features are
first tiled and then compressed using a High Efficiency Video Coding (HEVC)
codec, as in [6]. In HEVC, quality of the compressed signal is controlled via
the quantization parameter QP, and the experiments will examine the behavior
of the model with various values of QP used for the enhancement features.

Fig. 4: Three-task DNN from [41] used in our experiments. Heads 1-3 are
similar to FC8 networks [42].

Fig. 5: Task accuracy vs. hard selection % for semantic segmentation (left),
disparity map estimation (middle) and input reconstruction (right) Fig. 6: An
example of task outputs with $50\%$ hard selection, left to right: semantic
segmentation map, disparity map, and reconstructed input. Top: $\ell_{2}$
norm-based selection. Bottom: MI-based selection.

##  4 Experiments

Our experiments are based on the multi-task DNN from [41], which is
illustrated in Fig. 4. This model was trained on Cityscapes [43] to perform
three tasks: (1) semantic segmentation, (2) disparity map estimation, and (3)
input image reconstruction on Cityscapes. The backbone is similar to the
backbone of YOLOv3 [44] with 74 layers. The split point is layer 36 of the
backbone, which outputs features of size $32\times 64\times 256$, i.e.,
$C=256$. The metrics used to assess the performance of the model on the three
tasks are as follows: mean Intersection over Union (mIoU)for semantic
segmentation, Root Mean Squared error (RMSE) in pixelsfor disparity maps
estimation, and Peak Signal to Noise Ratio (PSNR) in dB for input
reconstruction.

Hard selection. We compare our MI-based feature selection with selection based
on $\ell_{1}$ norm [23], $\ell_{2}$ norm [22], and geometric median [26]. We
used $N=8$ and $M=64$ (Fig. 1) for feature and output patches, respectively,
and further clustered output patches into $K=8$ clusters using K-means
clustering. Fig. 5 shows the task accuracies at five selection percentages
$\\{100,93.75,87.5,75,50\\}\%$ for the benchmarks and the proposed method. As
seen in Fig. 5 (a)-(b), norm-based selection and the proposed MI-based
selection provide similar performance up to $75\%$ in semantic segmentation
and disparity map estimation (with RMSE, the lower the better), while the GM-
based method is slightly worse. At $50\%$ selection on these two tasks, norm-
based selection achieves better performance than the other two approaches, but
the differences are relatively small in relative terms. However, on the input
reconstruction task (Fig. 5 right), MI-based selection achieves significantly
better performance compared to the other approaches across all selection
percentages. This highlights the fact that the norm- and similarity-based
importance cannot capture the feature importance for all tasks in a multi-task
model; they may work well for some tasks, but not others. Fig. 6 shows the
task outputs with $\ell_{2}$-based and MI-based feature selection. The outputs
for semantic segmentation and disparity estimation are similar, but input
reconstruction with MI-based selection is much better. In the remainder of the
experiments, we use $\ell_{2}$-based selection as a benchmark.

Fig. 7: Task accuracy vs. file size in soft selection for semantic
segmentation (left), disparity map estimation (middle) and input
reconstruction (right). Top: $75\%$ base selection. Bottom: $50\%$ base
selection.

Soft selection. Here, features are sorted according to importance ($\ell_{2}$
or MI-based) and 8-bit quantized. Then a certain percentage of the most
important features is selected as base features. The remaining (enhancement)
features are tiled into an image and compressed using
Kvazaar,222https://github.com/ultravideo/kvazaar an open-source HEVC encoder,
using four QP values: {10, 20, 30, 40}. Lower QP values result in more
accurate feature reconstruction at the cost of higher bitrate. Fig. 7 shows
the results in terms of task accuracy vs. average file size in Kbytes, for two
cases – $75\%$ selected as base (first row) and $50\%$ selected as base
(second row). The two single dots in each plot show the accuracy with the
corresponding hard selection, where the given percentage of features is
selected and 8-bit quantized, while others are discarded. We see that soft
selection can close the gap between hard selection and default model
performance at the cost of slightly increased file size. Also, with $50\%$
base selection, the same task accuracies are achieved with fewer bits compared
to $75\%$ base, due to the ability to exercise compression over more
enhancement features.

Multi-objective analysis. So far, we looked at the cases where features are
selected for one of several tasks. But what if all tasks are needed? This
leads to a multi-objective problem. One way to take task preference into
account when selecting features in such a scenario is to define task
distortion $D_{j}(C^{\prime})$ of task $j$ as a fraction of accuracy reduction
due to selecting $C^{\prime}$ features:

$D_{j}(C^{\prime})=\frac{|A_{j}(C)-A_{j}(C^{\prime})|}{A_{j}(C)},$ (8)

where $A_{j}(C)$ is the task accuracy with all $C$ features and
$A_{j}(C^{\prime})$ is the task accuracy with $C^{\prime}$ most important
features for that task. The total distortion $D(C^{\prime})$ with $C^{\prime}$
retained features for a model with $K$ tasks can be defined as:

$D(C^{\prime})=\sum_{j=1}^{K}w_{j}D_{j}(C^{\prime}),$ (9)

where the weights $w_{j}$ are non-negative, sum up to $1$, and reflect task
preferences. For a given $K$-tuple of task preference weights
$(w_{1},w_{2},...,w_{K})$, one can examine the total distortions (9) produced
by various task-specific importance orderings $\mathcal{O}_{j}$ that leads to
the lowest total distortion (9). In the 3-task model studied here,
$w_{1}+w_{2}+w_{3}=1$ with $0\leq w_{1},w_{2},w_{3}\leq 1$, which defines a
triangle in the $(w_{1},w_{2},w_{3})$ space. In the next set of experiments,
we examine the regions of this triangle where MI-based feature selection
achieves lower total distortion (9) compared to $\ell_{2}$-based selection.

Fig. 8 (top) shows which feature selection method – $\ell_{2}$ or Mi-based –
leads lower total distortion at $75\%$ and $50\%$ hard selection. For each
point in the triangle, the total distortion (9) is computed for the two
selection methods and the point is colored depending on which approach leads
to lower total distortion. As seen in the figure, MI-based selection is the
better choice across the entire weight space at $75\%$ selection. At $50\%$
selection, $\ell_{2}$-based approach is preferred near the corners that
correspond to semantic segmentation and depth estimation, but so long as input
reconstruction holds some importance, the MI-based approach is better.
Overall, the MI-based approach wins across $74\%$ of the weight space at
$50\%$ selection. Fig. 8 (bottom) shows the results for $75\%$ and $50\%$ soft
selection across all tested QP values. MI-based selection is better across
virtually all choices of weights in both cases; the only exception is a small
set of weights at the bottom of the triangle for $50\%$ soft selection.

Fig. 8: The set of task weights $(w_{1},w_{2},w_{3})$ over which a particular
feature selection criterion leads to lower total distortion in equation (9).
Top: hard selection. Bottom: soft selection. Left: $75\%$ selection. Right:
$50\%$ selection.

##  5 Conclusions

We presented hard and soft (compressive) feature selection for multi-task DNNs
based on mutual information (MI). Compared to norm- and proximity-based
approaches, the MI-based approach allows one to take task-specific feature
importance into account. It was demonstrated that such approach is preferred
over a large fraction of the task preference weight space. In addition, we
showed that soft selection is able to close the gap between hard selection and
default model performance by using different amounts of feature compression.

## References

  * [1] W. Su, L. Li, F. Liu, M. He, and X. Liang, “AI on the edge: A comprehensive review,” _Artif. Intell. Rev._ , vol. 55, no. 8, pp. 6125–6183, Dec. 2022.
  * [2] Y. Kang, J. H. C. Gao, A. Rovinski, T. Mudge, J. Mars, and L. Tang, “Neurosurgeon: Collaborative intelligence between the cloud and mobile edge,” _SIGARCH Comput. Archit. News_ , vol. 45, no. 1, pp. 615–629, Apr. 2017.
  * [3] N. Shlezinger and I. V. Bajić, “Collaborative inference for AI-empowered IoT devices,” _IEEE Internet of Things Magazine_ , vol. 5, no. 4, pp. 92–98, 2022.
  * [4] A. E. Eshratifar and M. Pedram, “Energy and performance efficient computation offloading for deep neural networks in a mobile cloud computing environment,” in _Proc. ACM Great Lakes Symp. on VLSI (GLSVLSI’18)_ , 2018, pp. 111–116.
  * [5] Y. Matsubara, M. Levorato, and F. Restuccia, “Split computing and early exiting for deep learning applications: Survey and research challenges,” _ACM Comput. Surv._ , vol. 55, no. 5, pp. 1–30, Dec. 2022.
  * [6] H. Choi and I. V. Bajić, “Deep feature compression for collaborative object detection,” in _Proc. IEEE ICIP_ , 2018.
  * [7] Z. Chen, K. Fan, S. Wang, L. Duan, W. Lin, and A. C. Kot, “Toward intelligent sensing: Intermediate deep feature compression,” _IEEE Trans. Image Processing_ , vol. 29, 2020.
  * [8] ISO/IEC, “Draft call for evidence for video coding for machines,” ISO/IEC JTC1/SC29/WG11/w19508, Jul. 2020.
  * [9] ——, “Common test and training conditions for FCM,” ISO/IEC JTC1/SC29/WG04/N0427, Oct. 2023.
  * [10] J. Ascenso, E. Alshina, and T. Ebrahimi, “The JPEG AI standard: Providing efficient human and machine visual data consumption,” _IEEE MultiMedia_ , vol. 30, no. 1, 2023.
  * [11] X. Liu, P. He, W. Chen, and J. Gao, “Multi-task deep neural networks for natural language understanding,” in _Proc. ACL_ , Jul. 2019, pp. 4487–4496.
  * [12] M. Crawshaw, “Multi-task learning with deep neural networks: A survey,” _arXiv preprint arXiv: arXiv:2009.09796_ , 2020.
  * [13] Y. Fang, W. Wang, B. Xie, Q. Sun, L. Wu, X. Wang, T. Huang, X. Wang, and Y. Cao, “EVA: Exploring the limits of masked visual representation learning at scale,” in _CVPR_ , 2023, pp. 19 358–19 369.
  * [14] T. M. Cover and J. A. Thomas, _Elements of Information Theory_ , 2nd ed. Wiley, 2006.
  * [15] I. Guyon and A. Elisseeff, “An introduction to variable and feature selection,” _JMLR_ , 2003.
  * [16] J. R. Vergara and P. A. Estévez, “A review of feature selection methods based on mutual information,” _Neural Comput. & Appl._, 2013.
  * [17] N. Hoque, D. K. Bhattacharyya, and J. K. Kalita, “MIFS-ND: a mutual information-based feature selection method,” _Expert Syst. Appl._ , vol. 41, no. 14, pp. 6371–6385, 2014.
  * [18] M. Beraha, A. M. Metelli, M. Papini, A. Tirinzoni, and M. Restelli, “Feature selection via mutual information: New theoretical insights,” in _IJCNN_ , 2019.
  * [19] M. R. Ganesh, J. J. Corso, and S. Y. Sekeh, “MINT: deep network compression via mutual information-based neuron trimming,” _arXiv preprint arXiv:2003.08472_ , 2020.
  * [20] M. K. Lee, S. Lee, S. H. Lee, and B. C. Song, “Channel pruning via gradient of mutual information for light-weight convolutional neural networks,” in _Proc. IEEE ICIP_ , 2020.
  * [21] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR’16_ , 2016, pp. 770–778.
  * [22] Y. He, G. Kang, X. Dong, Y. Fu, and Y. Yang, “Soft filter pruning for accelerating deep convolutional neural networks,” in _Proc. IJCAI_ , 2018, pp. 2234–2240.
  * [23] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient convnets,” in _ICLR_ , 2017.
  * [24] Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang, “Learning efficient convolutional networks through network slimming,” in _ICCV_ , 2017, pp. 2736–2744.
  * [25] X. Chen, Y. Wang, Y. Zhang, P. Du, C. Xu, and C. Xu, “Multi-task pruning for semantic segmentation networks,” _arXiv preprint arXiv:2007.08386_ , 2020.
  * [26] Y. He, P. Liu, Z. Wang, Z. Hu, and Y. Yang, “Filter pruning via geometric median for deep convolutional neural networks acceleration,” in _CVPR_ , 2019, pp. 4340–4349.
  * [27] P. Singh, V. K. Verma, P. Rai, and V. P. Namboodiri, “Leveraging filter correlations for deep model compression,” in _Proc. IEEE WACV_ , 2020, pp. 824–833.
  * [28] H. Zhuo, X. Qian, Y. Fu, H. Yang, and X. Xue, “Scsp: Spectral clustering filter pruning with soft self-adaption manners,” _arXiv preprint arXiv:1806.05320_ , 2018.
  * [29] A. Polyak and L. Wolf, “Channel-level acceleration of deep face representations,” _IEEE Access_ , vol. 3, 2015.
  * [30] H. Li, C. Ma, W. Xu, and X. Liu, “Feature statistics guided efficient filter pruning,” _arXiv preprint arXiv:2005.12193_ , 2020.
  * [31] J. Luo, J. Wu, and W. Lin, “Thinet: A filter level pruning method for deep neural network compression,” in _ICCV_ , 2017.
  * [32] M. Lin, R. Ji, Y. Wang, Y. Zhang, B. Zhang, Y. Tian, and L. Shao, “HRank: filter pruning using high-rank feature map,” in _CVPR_ , 2020, pp. 1526–1535.
  * [33] K. Yamamoto and K. Maeno, “Pcas: Pruning channels with attention statistics for deep network compression,” _arXiv preprint arXiv:1806.05382_ , 2018.
  * [34] L. Paninski, “Estimation of entropy and mutual information,” _Neural Computation_ , vol. 15, no. 6, pp. 1191–1253, 2003.
  * [35] M. I. Belghazi, A. Baratin, S. Rajeswar, S. Ozair, Y. Bengio, A. Courville, and R. D. Hjelm, “MINE: Mutual information neural estimation,” _PMLR_ , vol. 80, pp. 531–540, 2018.
  * [36] D. McAllester and K. Stratos, “Formal limitations on the measurement of mutual information,” in _Proc. Int. Conf. Artificial Intelligence and Statistics (AISTATS)_ , Aug. 2020, pp. 875–884.
  * [37] R. Bellman, _Dynamic Programming_. Princeton University Press, 1957.
  * [38] A. M. Saxe, Y. Bansal, J. Dapello, M. Advani, A. Kolchinsky, B. D. Tracey, and D. D. Cox, “On the information bottleneck theory of deep learning,” in _ICLR_ , 2018.
  * [39] J. Boets, K. D. Cock, and B. D. Moor, “A mutual information based distance for multivariate gaussian processes,” in _Modeling, Estimation and Control_. Springer, 2007, pp. 15–33.
  * [40] A. Likas, N. Vlassis, and J. J. Verbeek, “The global k-means clustering algorithm,” _Pattern Recognition_ , vol. 36, no. 2, pp. 451–461, 2003.
  * [41] S. R. Alvar and I. V. Bajić, “Bit allocation for multi-task collaborative intelligence,” in _Proc. IEEE ICASSP_ , 2020.
  * [42] E. Shelhamer, J. Long, and T. Darrell, “Fully convolutional networks for semantic segmentation,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 39, no. 4, pp. 640–651, April 2017.
  * [43] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The Cityscapes dataset for semantic urban scene understanding,” in _CVPR_ , 2016.
  * [44] J. Redmon and A. Farhadi, “YOLOv3: An incremental improvement,” _arXiv preprint arXiv:1804.02767_ , 2018.