# A Closer Look at Prototype Classifier for Few-shot Image Classification

Mingcheng Hou  The University of Tokyo  Issei Sato  The University of Tokyo

###### Abstract

The prototypical network is a prototype classifier based on meta-learning and
is widely used for few-shot learning because it classifies unseen examples by
constructing class-specific prototypes without adjusting hyper-parameters
during meta-testing. Interestingly, recent research has attracted a lot of
attention, showing that a linear classifier with fine-tuning, which does not
use a meta-learning algorithm, performs comparably with the prototypical
network. However, fine-tuning requires additional hyper-parameters when
adapting a model to a new environment. In addition, although the purpose of
few-shot learning is to enable the model to quickly adapt to a new
environment, fine-tuning needs to be applied every time a new class appears,
making fast adaptation difficult. In this paper, we analyze how a prototype
classifier works equally well without fine-tuning and meta-learning. We
experimentally found that directly using the feature vector extracted using
standard pre-trained models to construct a prototype classifier in meta-
testing does not perform as well as the prototypical network and linear
classifiers with fine-tuning and feature vectors of pre-trained models. Thus,
we derive a novel generalization bound for the prototypical network and show
that focusing on the variance of the norm of a feature vector can improve
performance. We experimentally investigated several normalization methods for
minimizing the variance of the norm and found that the same performance can be
obtained by using the L2 normalization and embedding space transformation
without fine-tuning or meta-learning.

##  1 Introduction

Few-shot learning is used to adapt quickly to new classes with low annotation
cost. Meta-learning is a standard training procedure to tackle the few-shot
learning problem and _Prototypical Network_ (Snell et al., 2017) a.k.a
ProtoNet is a widely used meta-learning algorithm for few-shot learning. In
ProtoNet, we use a prototype classifier based on meta-learning to predict the
classes of unobserved objects by constructing class-specific prototypes
without adjusting the hyper-parameters during meta-testing.

ProtoNet has the following advantages. (1) Since the nearest neighbor method
is applied on query data and class prototypes during the meta-test phase, no
hyper-parameters are required in meta-test phase. (2) Since the number of data
for few-shot is small, the inference time is almost negligible. (3) The
classifiers can quickly adapt to new environments because they do not have to
be re-trained for the support set when new classes appear. The generalization
bound of ProtoNet in relation to the number of shots in a support set has been
studied (Cao et al., 2020). The bound suggests that the performance of
ProtoNet depends on the ratio of the between-class variance to the within-
class variance of features of a support set extracted using the meta-trained
model.

There have been studies on fine-tuning a linear classifier on the features
extracted using a pre-trained model without meta-learning, which can perform
comparably with the meta-learned models (Chen et al., 2019; Tian et al.,
2020). In these studies, the models are trained with the standard
classification problem, i.e., models are trained with cross-entropy loss after
linear projection from the embedding space to the class-probability space. The
linear classifier with fine-tuning has the following advantages over meta-
learning. (1) Training converges faster than meta-learning. (2) Implementation
is simpler. (3) Meta-learning decreases in performance if the number of shots
does not match between meta-training and meta-testing (Cao et al., 2020);
however, the linear classifier with fine-tuning do not need to take this into
account. However, the fine-tuning-based approach has two drawbacks, i.e., we
have to tune additional hyper-parameters for fine-tuning and to apply fine-
tuning every time a new class appears, which make fast adaptation difficult in
few-shot learning.

To overcome the disadvantages while inherits the advantages of a linear
classifier with fine tuning, we analyze how to avoid meta-learning during the
training phase and fine-tuning during the testing phase. Since a prototype
classifier can be applied to any trained feature extractor and does not
require any hyperparameter or fine-tuning in the testing phase, we focus on
using a prototype classifier on the testing phase and training models in a
standard classification manner. As we discuss in section 4, we found that when
we directly constructed prototypes from the feature vectors extracted using
pre-trained models and applied the nearest neighbor method as in the testing
phase in ProtoNet , this does not perform as well as the linear classifier
with fine-tuning.

We hypothesize that the reason is the difference between the loss function in
ProtoNet and pre-trained models. As described in section 3, if we consider a
prototype as a pseudo sample average of the features in each class, the loss
function of ProtoNet can be considered having a regularizing effect that makes
it closer to the sample average of the class. Since standard classification
training computes cross-entropy loss with additional linear projection to make
the features linearly separable, the loss function does not have such an
effect and can cause large within-class variance. Figure 1 shows a scatter
plot of the features extracted using a neural network with two dimension
output trained on cifar-10 with ProtoNet(1(a)) and cross-entropy loss with a
linear projection layer(1(b)). This figure implies that the features extracted
using a model trained in a standard classification manner distributes away
from the origin and causes large within-class variance along the direction of
the norm of class mean vectors, while that of ProtoNet is more centered to its
class means. This phenomenon is also observed in face recognition literatures
(Wen et al., 2016; Liu et al., 2017; Wang et al., 2018; Deng et al., 2019).

We now focus on the theoretical analysis on a prototype classifier. A current
study (Cao et al., 2020) analyzed an upper bound of risk by using a prototype
classifier. The bound depends on the number of shots of a support set,
between-class variance, and within-class variance. However, the bound requires
the class-conditioned distribution of features to be Gaussian and to have the
same covariance matrix among classes. In addition, since the bound does not
depend on the norm of the feature vectors, it is not clear from the bound what
feature-transformation method can lead to performance improvement. Thus, we
derive a novel bound for a prototype classifier.

Our contributions are threefold.

  1. 1.

We relax the assumption; specifically, the bound does not require that the
features distribute on any specific distribution, and each covariance matrix
does not have to be the same among classes.

  2. 2.

We clarify the effect of the variance of the norm of the feature vectors on
the performance of a prototype classifier.

  3. 3.

We investigate the effectiveness of reducing the variance of the norm
empirically.

(a) ProtoNet

(b) cross-entropy loss with linear projection layer

Figure 1: Distribution of features extracted using a neural network with two
dimensional final layer trained on CIFAR-10 with (a): ProtoNet loss (b):
cross-entropy loss with linear projection layer. The ProtoNet features
distribute closer to its class center than the features extracted using the
model trained on cross-entropy loss with linear projection layer.

##  2 Related Work

We summarize related work by describing a prototype classifier with meta-
learning, fine-tuning-based approach without meta-learning, and theoretical
analysis related to the few-shot learning problem.

#### A prototype classifier with meta-learning

On the basis of the hypothesis that features well distinguished in the
training phase are also useful for classifying new classes, constructing one
or multiple prototypes for classifying unseen examples is a widely used
approach (Vinyals et al., 2016; Snell et al., 2017; Pahde et al., 2021; Ji et
al., 2021; Sung et al., 2018; Allen et al., 2019; Doersch et al., 2020; Qi et
al., 2018). Certain algorithms compute similarities between multiple
prototypes and unseen examples by using their own modules, such as attention
mechanism (Vinyals et al., 2016; Doersch et al., 2020), relation network (Sung
et al., 2018), reweighting mechanisms by taking into account between-class and
within-class interaction (Ji et al., 2021), and latent clusters (Allen et al.,
2019). Prototypes are also constructed in a multi-modal way (Pahde et al.,
2021). Another line of research is transforming the space of extracted
features to a better distinguishable space (Simon et al., 2020; Yoon et al.,
2019; Das et al., 2020; Das & Lee, 2020). Because of its convenience, the
approach with a prototype classifier is adapted to other domain such as
semantic segmentation (Dong & Xing, 2018), text classification (Sun et al.,
2019), and speech recognition (Wang et al., 2019a).

#### Update-based meta-learning

In contrast to the approach with a prototype classifier and meta-learning , in
update-based meta-learning approaches, model parameters are adjusted in the
test phase so that a model can adapt to new classes. Model-agnostic meta-
Learning (MAML) and its variants (Finn et al., 2017; 2018; Rajeswaran et al.,
2017) search for a good initialization parameters that adapt to new classes
with a few labeled data and a few update steps of the parameters. Another
approach involves learning an effective update rule for the parameters of a
base-learner model through a sequence of training episodes (Bertinetto et al.,
2019; Lee et al., 2019). Both approaches require additional learning of hyper-
parameters and training time; thus, they prevent quick adaptation to new
classes.

#### fine-tuning-based approach without meta-learning

Interestingly, recent studies have shown that fine-tuning a linear classifier
with features extracted using a model trained with cross-entropy loss on base-
dataset performs comparably with meta-learning based methods (Chen et al.,
2019) (Wang et al., 2019b). More effective method for fine-tuning in few-shot
settings has been proposed (Yang et al., 2021; Phoo & Hariharan, 2021), such
as calibrating distribution generated by a support set (Yang et al., 2021) ,
self-supervised learning on query data (Phoo & Hariharan, 2021), and
distilling knowledge to obtain better embeddings (Tian et al., 2020). Liu et
al. (2020) focuses on pretraining phase and found the negative margin in
cross-entropy helps improving the performance in few-shot settings. However,
similar to update-based meta-learning, these studies require additional hyper-
parameters and fine-tuning to applied to new classes; thus they prevent quick
adaptation to new classes.

#### Theoretical analysis of few-shot learning

Even though much improvement has been empirically made in few-shot learning,
theoretical analysis is scarce. In the context of meta-learning, Du et al.
(2021) provided a risk bound on the meta-testing phase that is related to the
number of meta-training data and meta-testing data. Lucas et al. (2021)
derived information-theoretic lower-bounds on mini-max rates of convergence
for algorithms that are trained on data from multiple sources and tested on
novel data. Cao et al. (2020) derived a new bound on a prototype classifier
and theoretically demonstrated that the mismatch regarding the number of shots
in a support set between meta-training and meta-testing degrades the
performance of prototypical networks, which has been only experimentally
observed. However, their bound depends on several assumptions: the class-
conditional distributions of features are Gaussian and have the same
covariance matrix among classes. In contrast, we derive a novel bound that
does not depend on any specific distribution.

##  3 Theoretical Analysis of Prototype Classifier in Terms of Variance of
Norm of Feature Vectors

In this section, we first formulate our problem settings and point out the
drawbacks of the current theoretical analysis on a prototype classifier. Next
we provide our novel bound of a prototype classifier with the bound related to
the variance of the norm of the feature vectors. Finally, we list several
methods that can improve the performance of a prototype classifier based on
our bound.

###  3.1 Problem Setting

Let $\mathcal{Y}$ be a space of a class, $\tau$ a probability distribution
over $\mathcal{Y}$, $\mathcal{X}$ a space of input data, $\mathcal{D}$ a
probability distribution over $\mathcal{X}$, $\mathcal{D}_{y}$ a probability
distribution over $\mathcal{X}$ given a class $y$. We define
$\mathcal{D}^{\otimes nk}$ and $\mathcal{D}_{y}^{\otimes n}$ by
$\mathcal{D}_{y}^{\otimes n}=\Pi_{i=1}^{n}\mathcal{D}_{y}$ and
$\mathcal{D}^{\otimes nk}\equiv\Pi_{i=1}^{n}\mathcal{D}_{y}^{\otimes k}$,
respectively. We sample $N$ classes from $\tau$ to form the $N$-way
classification problem. Denote by $K$ a number of annotated data in each class
and $x\in\mathcal{X},y\in\mathcal{Y}$ as input data and its class
respectively. We define a set of support data of class $c$ sampled from $\tau$
as
$S_{c}=\\{\bm{x}_{i}\mid(\bm{x}_{i},y_{i})\in\mathcal{X}\times\mathcal{Y}\cap
y_{i}=c\\}_{i=1}^{K}$ and a set of support data in the $N$-way $K$-shot
classification problem as $S=\bigcup_{c=1}^{N}S_{c}$. Suppose a feature
extractor computes a function $\phi:\mathcal{X}\rightarrow\mathbb{R}^{D}$,
where $D$ is the number of the embedding dimensions. $\overline{\phi(S_{c})}$
is defined by
$\overline{\phi(S_{c})}=\frac{1}{K}\sum_{\bm{x}\in{S_{c}}}\phi(\bm{x})$. Let
$\Phi$ be a space of the extractor function $\phi$. Denote by
$\mathcal{M}:\Phi\times\mathcal{X}\times\left(\mathcal{X}\times\mathcal{Y}\right)^{NK}\rightarrow\mathbb{R}^{N}$
a prototype classifier function that computes probabilities of input $x$
belonging to class $c$ as follows.

$\displaystyle\mathcal{M}(\phi,\bm{x},S)_{c}=p_{\mathcal{M}}(y=c|\bm{x},S,\phi)=\frac{\exp\left(-\|\phi(\bm{x})-\overline{\phi(S_{c})}\|^{2}\right)}{\sum_{l=1}^{N}\exp\left(-\|\phi(\bm{x})-\overline{\phi(S_{l})}\|^{2}\right)},$
(1)

where $\|v\|^{2}=\sum_{d=1}^{D}\left(v^{(d)}\right)^{2}$, and $v^{(d)}$ is the
$d$-th dimension of vector $v$. The prediction of an input $x$, denoted by
$\hat{y}\in\mathcal{Y}$ , is computed by taking argmax for
$\mathcal{M}(\phi,\bm{x},S)$, i.e.,
$\hat{y}=\mathrm{argmax}\mathcal{M}(\phi,\bm{x},S)$. We denote by
$\mathbb{E}_{z\sim q(z)}[g(z)]$ an operation to take the expectation of $g(z)$
over $z$ distributed as $f(z)$ and we simply denote $\mathbb{E}_{z\sim
q(z)}[g(z)]$ as $\mathbb{E}_{z}[g(z)]$ when $z$ is obviously distributed on
$q(z)$. We define $\mathrm{Var}_{z\sim q(z)}[g(z)]$ as an operation to take
variance of $g(z)$ over $z$ distributed as $q(z)$. With $\mathbb{I}$ denoting
the indicator function, we define expected risk $\mathrm{R}_{\mathcal{M}}$ of
a prototype classifier as

$\displaystyle\mathrm{R}_{\mathcal{M}}(\phi)=\mathbb{E}_{S\sim\mathcal{D}^{\otimes
nk}}\mathbb{E}_{c\sim\tau}\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c}}[\mathbb{I}[\mathrm{argmax}\mathcal{M}(\phi,\bm{x},S)\neq
c].$ (2)

For simplicity, we now discuss the binary classification setting. We show a
case of multi-class classification in Appendix A.5 due to lack of space.

Let $c_{1}$ and $c_{2}$ denote any pair of classes sampled from $\tau$. We
consider that a query data $x$ belongs to class $c_{1}$ and support sets $S$
consist of class $c_{1}$’s support set and $c_{2}$’s support set. Then,
equation 2 is written as follows.

$\displaystyle\mathrm{R}_{\mathcal{M}}(\phi)=\mathbb{E}_{S\sim\mathcal{D}^{\otimes
2k}}\mathbb{E}_{c_{1}\sim\tau}\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c_{1}}}[\mathbb{I}[\mathrm{argmax}\mathcal{M}(\phi,\bm{x},S_{c_{1}}\cup
S_{c_{2}})\neq c_{1}]].$ (3)

###  3.2 What Feature-Transformation Method Expected to be Effective?

The current theoretical analysis for a prototype classifier (Cao et al., 2020)
has the following two drawbacks (see Appendix A.1 for the details). The first
is that the modeling assumption requires a class-conditioned distribution of
the features to follow a Gaussian distribution with the same covariance matrix
among classes. For example, when we use the $\mathrm{ReLU}$ activation
function in last layer, it is not normally distributed and the class-
conditioned distribution does not have the same covariance matrix as shown in
Figure 1. The second drawback is that it is not clear what feature-
transformation method can reduce the upper bound. A feature-transformation
method to maximize the between-class variance and minimize the within-class
variance such as linear discriminant analysis (LDA) (Fukunaga, 1990) and
Embedding Space Transformation (EST) (Cao et al., 2020) can be expected to
improve performance; however, it is not clear how the second term of the
denominator changes.

From Figure 1 , the distribution of each class feature stretches in the
direction of the norm of its class-mean feature vector. This property is also
observed in metric learning literatures (Wen et al., 2016; Liu et al., 2017).
A model trained in the cross-entropy loss after linear projection from the
embedding space to the class-probability space computes a probability of input
$x$ belonging to class $c$ given by

$\displaystyle
p(y=c|\bm{x},W,\phi)=\frac{\exp{\left(\phi(\bm{x})^{\top}W_{c}\right)}}{\sum_{j=1}^{N}\exp{\left(\phi(\bm{x})^{\top}W_{j}\right)}},$
(4)

where $W$ is a weight matrix that transforms features from the embedding space
to the class-probability space.

Comparing equation 4 with equation 1, we found that equation 4 does not have a
regularization term similar to the one appearing in equation 1 that forces the
features to be close to its class-mean feature vector. This implies the
features extracted using a model trained with the cross-entropy loss on
equation 4 are less close to its class-mean feature vector than the ProtoNet
loss on equation 1. Through this observation and the property mentioned above,
we hypothesis that normalizing the norm of the feature vectors can push the
features to each class-mean feature vector and can boost the performance of a
prototype classifier trained with cross-entropy loss on equation 4.

###  3.3 Relating Variance of Norm to Upper Bound of Expected Risk

To understand what effect the variance of the norm of the feature vectors has
on the performance of a prototype classifier, we analyze how variance
contributes to the expected risk when an embedding function $\phi$ is fixed.
The following theorem provides a generalization bound for the expected risk of
a prototype classifier in terms of the variance of the norm of the feature
vectors computed using a feature extractor.

######  Theorem 1.

Let $\mathcal{M}$ be an operation of a prototype classifier on binary
classification defined by equation 1. For
$\mu_{c}=\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c}}[\phi(\bm{x})]$,
$\Sigma_{c}=\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c}}[(\phi(\bm{x})-\mu_{c})^{\top}(\phi(\bm{x})-\mu_{c})]$,
$\mu=\mathbb{E}_{c\sim\tau}[\mu_{c}]$,$\Sigma=\mathbb{E}_{c\sim\tau}[(\mu_{c}-\mu)^{\top}(\mu_{c}-\mu)]$,
and $\mathbb{E}_{c\sim\tau}[\Sigma_{c}]=\Sigma_{\tau}$, if $\phi(\bm{x})$ has
the variance of its norm, then the miss-classification risk of the prototype
classifier on binary classification $\mathrm{R}_{\mathcal{M}}$ satisfies

$\displaystyle\mathrm{R}_{\mathcal{M}}(\phi)\leq$ $\displaystyle
1-\frac{4(\mathrm{Tr}(\Sigma))^{2}}{\mathbb{E}\mathrm{V}[h_{L2}(\phi(\bm{x}))]+\mathrm{V}_{\mathrm{Tr}}(\Sigma_{c_{1}})+\mathrm{V}_{\mathrm{wit}}(\Sigma_{\tau},\Sigma,\bm{\upmu})+\mathbb{E}\operatorname{dist_{L2}^{2}}(\bm{\mu}_{c_{1}},\bm{\mu}_{c_{2}})},$
(5)

where

$\displaystyle\mathbb{E}\mathrm{V}[h_{L2}(\phi(\bm{x}))]=$
$\displaystyle\frac{4}{K}\mathbb{E}_{c\sim\tau}\left[\mathrm{Var}_{\bm{x}\sim\mathcal{D}_{c}}\left[\left\|\phi(\bm{x})\right\|^{2}\right]\right],$
(6) $\displaystyle\mathrm{V}_{\mathrm{Tr}}(\Sigma_{c_{i}})=$
$\displaystyle\left(\frac{4}{K}+\frac{2}{K^{2}}\right)\mathrm{Var}_{c\sim\tau}\left[\mathrm{Tr}\left(\Sigma_{c_{i}}\right)\right],$
(7) $\displaystyle\mathrm{V}_{\mathrm{wit}}(\Sigma_{\tau},\Sigma,\bm{\upmu})=$
$\displaystyle\frac{8}{K}\operatorname{Tr}(\Sigma_{\tau})\left(\operatorname{Tr}(\Sigma_{\tau})+\operatorname{Tr}(\Sigma)+\bm{\upmu}^{\top}\bm{\upmu}\right)+4(\operatorname{Tr}(\Sigma)+\bm{\upmu}^{\top}\bm{\upmu})^{2},$
(8)
$\displaystyle\mathbb{E}\operatorname{dist_{L2}^{2}}(\bm{\mu}_{c_{1}},\bm{\mu}_{c_{2}})=$
$\displaystyle\mathbb{E}_{c_{1},c_{2}}\left[\left(\left(\bm{\mu}_{c_{1}}-\bm{\mu}_{c_{2}}\right)^{\top}\left(\bm{\mu}_{c_{1}}-\bm{\mu}_{c_{2}}\right)\right)^{2}\right].$
(9)

######  Remark.

The term $\mathbb{E}\mathrm{V}[h_{L2}(\phi(S_{y}))]$ is the variance of the
norm of the feature vectors. The term
$\mathrm{V}_{\mathrm{Tr}}(\Sigma_{c_{1}})$ is the variance of the summation
with diagonal element of the covariance matrix from each class embedding; it
can be interpreted as the difference of the distribution shape constructed
from each class embedding. The term
$\mathrm{V}_{\mathrm{wit}}(\Sigma_{\tau},\Sigma,\bm{\upmu})$ is correlated
with the within-class variance because
$\frac{\mathrm{V}_{\mathrm{wit}}(\Sigma_{\tau},\Sigma,\bm{\upmu})}{\mathrm{Tr}(\Sigma)}$
is a secondary expression for the ratio of between-class variance and within-
class variance. The term
$\mathbb{E}\operatorname{dist_{L2}^{2}}(\bm{\mu}_{c_{1}},\bm{\mu}_{c_{2}})$ is
the expectation of the Euclidean distance between the class mean vectors .

This bound has the following properties.

  1. 1.

Its derivation does not require the features to be distributed on any specific
distributions and the class-conditioned covariance matrix does not have to be
the same among classes

  2. 2.

The bound can decrease when any of the following statistics decreases with
fixed between-class variance ($\Sigma$): (i) the variance of the norm of the
feature vectors, as what we discussed in Section 3.2, (ii) the difference in
the distribution shape constructed from each class embedding, (iii) the
within-class variance ($\Sigma_{\tau}$), (iv) the Euclidean distance between
the class-mean vectors.

As a result, our bound loosens the modeling assumption of Theorem2 in appendix
A.1 and has its property in 2-(iii). We show the proof in Appendix A.2.

###  3.4 Feature-Transformation Methods

We hypothesize from Theorem 1 that in addition to a feature-transformation
method related to equation 6, eliminating the affect from the difference in
the distribution shape among classes and lowering the ratio of between-class
variance to within-class variance can improve the performance of a prototype
classifier. Regarding equation 9, the ratio of the Euclidean distance between
the class mean vectors and the between-class variance is supposed to be
constant. Thus, we focus on the transformation methods relating to equation 6,
equation 7 and equation 8. We analyze the following feature-transformation
methods: $L_{2}$-normalization ($L_{2}$-norm), variance-normalization, LDA,
EST, and EST+$L_{2}$-norm. We show the detail of each method in Section A.3
due to lack of space.

##  4 Experimental Evaluation

In this section, we experimentally analyzed the effectiveness of the feature-
transformation methods mentioned in Section3.4. The center loss (Wen et al.,
2016) and affinity loss (Hayat et al., 2019) have been proposed to efficiently
pull the features of the same class to their centers in the training phase;
however, we focus on a widely used pre-trained model in this experiments
following the line of studies Chen et al. (2019) and Tian et al. (2020).

###  4.1 Datasets and Evaluation Protocol

miniImageNet. The miniImageNet dataset (Vinyals et al., 2016) is a standard
bench-mark for few-shot learning algorithms for recent studies. It contains
$100$ classes randomly sampled from ImageNet. Each class contains $600$
images. We follow the widely-used splitting protocol proposed by Ravi &
Larochelle (2017) to split the dataset into $64/16/20$ for
training/validation/testing respectively.

tieredImageNet The tieredImageNet dataset (Ren et al., 2018) is another subset
of ImageNet but has $608$ classes. These classes are grouped into $34$ higher
level categories in accordance with the ImageNet hierarchy and these
categories are split into $20$ training ($351$ classes), $6$ validation ($97$
classes), $8$ testing categories ($160$ classes). This splitting protocol
ensures that the training set is distinctive enough from the testing set and
makes the problem more realistic since we generally cannot assume that test
classes will be similar to those seen in training.

CIFAR-FS The CIFAR-FS dataset (Bertinetto et al., 2019) is a recently proposed
fewshot image classification benchmark, consisting of all $100$ classes from
CIFAR-$100$ (Krizhevsky et al., ). The classes are randomly split into $64$,
$16$, and $20$ for meta-training, meta-validation, and meta-testing
respectively. Each class contains $600$ images of size $32\times 32$.

FC100 The FC100 dataset (Lee et al., 2019) is another dataset derived from
CIFAR-$100$ (Krizhevsky et al., ), containing $100$ classes which are grouped
into $20$ categories. These categories are split into $12$ categories for
training ($60$ classes), from $4$ categories for validation ($20$ classes),
$4$ categories for testing ($20$ classes).

###  4.2 Implementation Details

We compared the feature-transformation methods applied on against ProtoNet
(Snell et al., 2017), Baseline and Baseline++ (Chen et al., 2019). In
Baseline, the linear projection layer is fine-tuned on a support set and in
Baseline++, the norm of the linear projection layer and features are
normalized to be constant. We also compared with the feature-transformation
method proposed in a previous study (Wang et al., 2019b). In that study they
transformed the features so that the mean of all features to be origin before
$L_{2}$-normalization. We call this operation as centering+$L_{2}$-norm. We
re-implemented these methods following the training procedure in previous
study (Chen et al., 2019). In the pre-training stage, where the cross-entropy
loss was used and meta-learning was not used, we trained $400$ epochs with a
batch size of $16$. In the meta-training stage for ProtoNet, we trained
$60,000$ episodes for $1$-shot and $40,000$ episodes for $5$-shot tasks. We
used the validation set to select the training episodes with the best
accuracy. In each episode, we sampled $N$ classes to form $N$-way
classification (In meta-training $N$=20 and meta-testing $N$=5 following the
original study of ProtoNet (Snell et al., 2017)). For each class, we selected
$K$ labeled instances as our support set and 16 instances for the query set
for a $K$-shot task.

In the fine-tuning or meta-testing stage for all methods, we averaged the
results over $600$ trials. In each trial, we randomly sampled $5$ classes from
novel classes, and in each class, we also selected $K$ instances for the
support set and 16 for the query set. For Baseline and Baseline++, we used the
entire support set to train a new classifier for $100$ iterations with a batch
size of $4$. With ProtoNet, we used the models trained in the same shots as
meta-testing stage since the mismatch of number of shots between meta-training
and meta-testing degrades performance (Cao et al., 2020). For LDA we set
$\lambda=0.0001$ for equation 18 in Appendix A.3 and for EST we set the
dimensions of the projected space to $60$ following the settings of the
original study (Cao et al., 2020). All methods were trained from scratch, and
the Adam optimizer with initial learning rate $10^{-3}$ was used. We applied
standard data augmentation including random crop, horizontal-flip, and color
jitter in both the pre-training and meta-training stages. We used early
stopping in accordance with the $5$-shot five-way accuracy on the validation
classes for cross-entropy without meta-training and five-way with the
corresponding shot accuracy on the validation classes for ProtoNet. We used a
ResNet12 network and ResNet18 network following the previous study (Chen et
al., 2019; Yoon et al., 2019).

Table 1: Classification accuracies on miniImageNet and tieredImageNet of ProtoNet , fine-tuning-based methods (Chen et al., 2019), and ours. The best performing methods and any other runs within $95\%$ confidence margin are in bold. Method |  miniImageNet |  tieredImageNet
---|---|---
| ResNet12 | ResNet18 | ResNet12 | ResNet18
|  $1$-shot |  $5$-shot |  $1$-shot |  $5$-shot |  $1$-shot |  $5$-shot |  $1$-shot |  $5$-shot
ProtoNet | $53.48\%$ | $73.56\%$ | $56.26\%$ | $74.02\%$ | $55.40\%$ | $77.67\%$ | $60.50\%$ | $81.40\%$
B@FT | $54.54\%$ | 76.50% | $55.41\%$ | 76.95% | $61.67\%$ | 81.62% | $63.38\%$ | 83.18%
B+@FT | $56.33\%$ | $74.62\%$ | $55.07\%$ | $74.15\%$ | $63.02\%$ | 81.07% | $64.20\%$ | $81.62\%$
B@CL2 | $58.66\%$ | $75.98\%$ | $57.67\%$ | $70.78\%$ | $\textbf{64.88}\%$ | $80.42\%$ | $\textbf{65.26}\%$ | $81.63\%$
B+@CL2 | $57.50\%$ | $74.00\%$ | $57.00\%$ | $74.06\%$ | $63.31\%$ | $79.19\%$ | $65.67\%$ | $\textbf{82.49}\%$
B | $46.36\%$ | $73.97\%$ | $43.86\%$ | $72.36\%$ | $50.60\%$ | $78.10\%$ | $56.16\%$ | $80.33\%$
B@L2-N | $57.18\%$ | 77.12% | $56.57\%$ | 76.44% | 64.04% | 81.73% | 65.19% | 82.93%
B@V-N | $-$ | $63.55\%$ | $-$ | $62.78\%$ | $-$ | $66.08\%$ | $-$ | $75.56\%$
B@LDA | $-$ | $73.75\%$ | $-$ | $73.54\%$ | $-$ | $77.44\%$ | $-$ | $80.85\%$
B@EST | $51.28\%$ | $72.80\%$ | $44.19\%$ | $72.99\%$ | $53.90\%$ | $78.09\%$ | $57.12\%$ | $80.59\%$
B@EST+L2-N | $58.00\%$ | 76.90% | $56.39\%$ | 76.24% | 64.54% | 81.40% | 64.71% | 83.24%
B+ | $41.18\%$ | $73.97\%$ | $36.80\%$ | $63.76\%$ | $46.52\%$ | $73.85\%$ | $48.27\%$ | $75.87\%$
B+@L2-N | $57.96\%$ | $75.38\%$ | $57.21\%$ | $74.89\%$ | 64.96% | 81.08% | 65.40% | $82.49\%$
B+@V-N | $-$ | $56.20\%$ | $-$ | $54.31\%$ | $-$ | $65.19\%$ | $-$ | $64.33\%$
B+@LDA | $-$ | $69.60\%$ | $-$ | $74.38\%$ | $-$ | $75.55\%$ | $-$ | $72.88\%$
B+@EST | $47.11\%$ | $69.01\%$ | $47.21\%$ |  $69.11$% | $52.01\%$ | $74.26\%$ | $53.49\%$ | $75.38\%$
B+@EST+L2-N | 63.00% | 77.49% | 62.44% | 77.13% | $57.63\%$ | 80.99% | $60.87\%$ | $81.80\%$
Table 2: Classification accuracies on FC100 and CIFARFS of ProtoNet, fine-tuning-based methods (Chen et al., 2019), and ours. The best performing methods and any other runs within $95\%$ confidence margin are in bold.  Method | CIFARFS | FC100
---|---|---
| ResNet12 | ResNet18 | ResNet12 | ResNet18
|  $1$-shot |  $5$-shot |  $1$-shot |  $5$-shot |  $1$-shot |  $5$-shot |  $1$-shot |  $5$-shot
ProtoNet | $58.65\%$ | $75.33\%$ | $62.05\%$ | 78.25% | $35.56\%$ | $51.12\%$ | $36.02\%$ | $51.02\%$
B@FT | $55.97\%$ | 76.50% | $56.46\%$ | $77.43\%$ | $39.72\%$ | 56.04% | $40.06\%$ | $57.04\%$
B+@FT | 61.08% | $76.15\%$ | $61.34\%$ | 77.86% | $36.01\%$ | $50.73\%$ | $36.93\%$ | $50.41\%$
B@CL2 | $58.61\%$ | $74.73\%$ | $58.09\%$ | $76.52\%$ | $41.58\%$ | $55.82\%$ | $41.51\%$ | $56.44\%$
B+@CL2 | $\textbf{61.59}\%$ | $76.10\%$ | $\textbf{63.17}\%$ | $\textbf{77.49}\%$ | $37.51\%$ | $50.38\%$ | $38.30\%$ | $51.06\%$
B | $44.12\%$ | $72.47\%$ | $47.86\%$ | $75.66\%$ | $31.60\%$ | $52.50\%$ | $35.90\%$ | $55.20\%$
B@L2-N | $59.43\%$ | 77.45% | $58.51\%$ | $77.43\%$ | $40.34\%$ | 56.61% | $40.49\%$ | $57.71\%$
B@V-N | $-$ | $44.66\%$ | $-$ | $67.72\%$ | $-$ | $41.99\%$ | $-$ | $49.69\%$
B@LDA | $-$ | $72.61\%$ | $-$ | $75.25\%$ | $-$ | $52.90\%$ | $-$ | $55.44\%$
B@EST | $51.28\%$ | $72.80\%$ | $55.57\%$ | $75.10\%$ | $44.45\%$ | $52.97\%$ | $47.49\%$ | $55.68\%$
B@EST+L2-N | 60.70% | 77.04% | $61.14\%$ | 77.48% | 47.57% | 56.96% | 50.13% | 59.94%
B+ | $45.73\%$ | $73.97\%$ | $36.80\%$ | $63.76\%$ | $29.72\%$ | $46.85\%$ | $30.76\%$ | $47.62\%$
B+@L2-N | 61.07% | $76.71\%$ | 63.48% | 77.86% | $37.16\%$ | $51.10\%$ | $38.55\%$ | $50.15\%$
B+@V-N | $-$ | $57.20\%$ | $-$ | $57.16\%$ | $-$ | $36.76\%$ | $-$ | $40.66\%$
B+@LDA | $-$ | $68.58\%$ | $-$ | $69.86\%$ | $-$ | $47.10\%$ | $-$ | $47.63\%$
B+@EST | $50.18\%$ | $70.32\%$ | $49.82\%$ | $71.07\%$ | $37.60\%$ | $46.81\%$ | $39.92\%$ | $47.65\%$
B+@EST+L2-N | $59.83\%$ | $76.32\%$ | 63.00% | 77.99% | $40.88\%$ | $50.45\%$ | $41.65\%$ | $50.74\%$

###  4.3 Results

We present the experimental results on the basis of backbones with ResNet12
and ResNet18 for a comprehensive comparison.

#### Method notations in experimental results

We denote B@FT and B+@FT as the fine-tuning-based methods: Baseline and
Baseline++. We also denote B@CL2, B+@CL2 as centering+$L_{2}$-norm (Wang et
al., 2019b) on features trained with Baseline and Baseilne++. B, B@L2-N,
B@V-N, B@LDA, B@EST, and B@EST+L2-N stands for a prototype classifier with
applying no transformation, $L_{2}$-norm, variance-normalization, LDA, EST,
and EST+L2-N on the features extracted using a model pre-trained in the same
manner as B@FT. B+, B+@L2-N, B+@V-N, B+@LDA, B+@EST, and B+@EST+L2-N stands
for a prototype classifier with applying no transformation, $L_{2}$-norm, and
variance-normalization on the features extracted using the same pre-trained
model as with B+@FT.

Since LDA constructs a within-class covariance matrix from a support set and
variance-normalization computes sample variance, we did not investigate the
LDA score and variance-normalization score of $1$-shot settings. Table 1 shows
that the prototype classifier with $L_{2}$-norm and EST+$L_{2}$-norm performs
comparably with ProtoNet and the fine-tuning-based approach in all settings
and EST+$L_{2}$-norm performs the best among the data transformation methods.
In the $1$-shot setting, although EST falls short of ProtoNet and fine-tuning-
based approach, it also improves the performance of a prototype classifier.
The performance gain of both $L_{2}$-norm, EST and EST+$L_{2}$-norm decrease
when the number of shots increases. This phenomenon can be explained through
Theorem 1. The term relating to the variance of the norm and the ratio of the
between-class variance to the within-class variance depends on $K$.Since the
term diminishes with increasing $K$, the performance gain of the feature-
transformation methods decreases. Table 2 shows the prototype classifier with
$L_{2}$-norm and EST+$L_{2}$-norm performs comparably with ProtoNet and the
fine-tuning-based approach in all settings.

Comparing the feature-transformation methods described in Section 3.4 with
centering+$L_{2}$-norm (Wang et al., 2019b), centering+$L_{2}$-norm can
slightly improve the performance of the prototype classifier in several
$1$-shot settings . However, in $5$-shot settings, the boost decreases and
even performs worse than $L_{2}$-norm, e.g. miniImageNet and tieredImageNet
with ResNet12.

The overall trend of Baseline++ is that the performance of B+@L2-N is better
than other methods in most settings. Since Baseline++ requires the norm of the
features to be constant during training, a mismatch exists between the
training features and the testing features without $L_{2}$-normalization. Thus
we cannnot guarantee the features stable and performance degrades without
$L_{2}$-normalization.

Surprisingly, variance normalization performs worse than applying no feature
transformation. This can be explained by the following two reasons. The first
is that the term in the bound related to the difference among the class
distribution shapes is smaller than other terms such as the variance of the
norm and the ratio of the between-class variance to the within-class variance.
Figure 2 shows the term associated with the bound computed from each dataset
and backbone. The larger the value is, the more it contributes to larger risk.
We found the term associated with equation 7 is small compared with other
values and thus variance-normalization did not worked well in our experiment.
The second reason is that estimating variance with a support set is unstable
and make prediction unreliable in few-shot learning.

LDA and EST in $5$-shot setting does not improve performance so much compared
with $L_{2}$-norm variants while in the $1$-shot settings, EST helps improving
performance. This is because in the $5$-shot settings, the values computed
from equation 8 gets smaller with larger $K$ and the effect of equation 8 on
the risk of a prototype classifier in the $1$-shot settings is larger.
Especially, EST outperforms $L_{2}$-norm in FC100 with the $1$-shot setting.
We found from Figure 2 the features of FC100 shows the largest ratio of the
within-class to between-class variance among all dataset. Thus the method
reducing the ratio works better with FC100 features than any other dataset’s
features.

Figure 2: Left: We plotted the values shown in equations 6,7,8 divided by
$\mathrm{Tr(\Sigma)^{2}}$. The values are computed from test-split of each
dataset with ResNet12 and ResNet18. We scaled the values so that each
dataset’s
$\frac{\mathrm{V}_{\mathrm{wit}}(\Sigma_{\tau},\Sigma,\bm{\upmu})}{\mathrm{Tr}(\Sigma)^{2}}$
to be $1$ for simplicity. Right: We showed the ratio of the within-class
variance to the between-class variance computed from the test-split’s features
extracted using each backbone trained on each corresponding dataset. We scaled
the values so that the maximium value to be $1$ for simplicity.

###  4.4 Effect on Covariance Matrix

We further analyzed how $\mathrm{Tr}(\Sigma)$ and $\mathrm{Tr}(\Sigma_{c})$
fluctuate with $L_{2}$-norm. Since $L_{2}$-norm lowers the variance in the
direction of the normalized vector, if the covariance matrix stretches in the
direction of the class mean vector, $L_{2}$-norm can reduce its trace. Thus we
analyzed the cosine similarities between class-mean feature vectors and the
eigenvectors with the largest eigen value of the covariance matrices. Figure 3
illustrates the distribution of cosine similarities of each class-mean feature
vector and the eigenvectors in test class of miniImageNet. This figure shows
$\Sigma_{c}$ tends to stretches to the direction of class mean vector and
$\Sigma$ stretches in the orthogonal direction to each class-mean feature
vector. From this observation, we found that $L_{2}$-norm can reduce
$\mathrm{Tr}(\Sigma_{c})$ and has less effect on $\mathrm{Tr}(\Sigma)$.
$L_{2}$-norm can reduce the term relating to the variance of the norm and the
ratio of the within-class variance to the between-class variance of Theorem 1.

Figure 3: Histogram of cosine similarities with class mean vectors and largest
eigen vectors of Left:$\Sigma$, Right:$\Sigma_{c}$

##  5 Conclusion

We theoretically and experimentally analyzed how the variance of the norm of
feature vectors affects the performance of a prototype classifier and found
that using EST+$L_{2}$-norm makes the classifier comparable with ProtoNet and
the fine-tuning-based approach. We also found that when the number of shots in
a support set increases, the performance gain from a feature-transformation
method decreases, which is consistent with the results of the theoretical
analysis. We believe that this work is a significant step to understanding the
behavior of a prototype classifier, which is crucial for the few-shot learning
problem.

## References

  * Allen et al. (2019) Kelsey R. Allen, Evan Shelhamer, Hanul Shin, and Joshua B. Tenenbaum.  Infinite mixture prototypes for few-shot learning.  _PMLR_ , pp. 232–241, 2019.
  * Bertinetto et al. (2019) Luca Bertinetto, João Henriques, Philip H.S. Torr, and Andrea Vedaldi.  Meta-learning with differentiable closed-form solvers.  In _International Conference on Learning Representations (ICLR)_ , 2019.
  * Cao et al. (2020) Tiansi Cao, Marc T.Law, and Sanja Fidler.  A theoretical analysis of the number of shots in few-shot learning.  In _International Conference on Learning Representations (ICLR)_ , 2020.
  * Chen et al. (2019) Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Wang, and Jia-Bin Huang.  A closer look at few-shot classification.  In _International Conference on Learning Representations (ICLR)_ , 2019.
  * Das & Lee (2020) Debasmit Das and C.S. George Lee.  A two-stage approach to few-shot learning for image recognition.  In _IEEE Transactions on Image Processing_ , volume 29, 2020.
  * Das et al. (2020) Debasmit Das, J. H. Moon, and C. S. George Lee.  Few-shot image recognition with manifolds.  In _International Symposium on Visual Computing (ISVC)_ , pp. 3–14, 2020.
  * Deng et al. (2019) Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_ , 2019.
  * Doersch et al. (2020) Carl Doersch, Ankush Gupta, and Andrew Zisserman.  Crosstransformers: spatially-aware few-shot transfer.  In _Advances in Neural Information Processing Systems_ , pp. 4003–4014, 2020.
  * Dong & Xing (2018) Nanqing Dong and Eric P. Xing.  Few-shot semantic segmentation with prototype learning.  In _British Machine Vision Conference (BMVC)_ , 2018.
  * Du et al. (2021) Simon Shaolei Du, Wei Hu, Sham M. Kakade, Jason D. Lee, and Qi Lei.  Few-shot learning via learning the representation, provably.  In _International Conference on Learning Representations(ICLR)_ , 2021\.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _Proceedings of the 34th International Conference on Machine Learning (ICML)_ , pp. 1126–1135. PMLR, 2017.
  * Finn et al. (2018) Chelsea Finn, Kelvin Xu, and Sergey Levine.  Probabilistic model-agnostic meta-learning.  In _Advances in Neural Information Processing Systems_ , pp. 9516–9527, 2018.
  * Fukunaga (1990) Keinosuke Fukunaga.  _Introduction to Statistical Pattern Recognition (2Nd Ed.)_.  Academic Press Professional, Inc., 1990.
  * Hayat et al. (2019) Munawar Hayat, Salman Khan, Syed Waqas Zamir, Jianbing Shen, and Ling Shao.  Max-margin class imbalanced learning with gaussian affinity.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 6469–6479, 2019.
  * Ji et al. (2021) Zhong Ji, Xingliang Chai, Yunlong Yu, and Zhongfei Zhang.  Reweighting and information-guidance networks for few-shot learning.  In _Neurocomputing_ , volume 423, pp. 13–23, 2021.
  * (16) Alex Krizhevsky, Vinod Nair, , and Geoffrey Hinton.  Cifar-100.
  * Lee et al. (2019) Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto.  Meta-learning with differentiable convex optimization.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 10657–10665, 2019.
  * Liu et al. (2020) Bin Liu, Yue Cao, Yutong Lin, Qi Li, Zheng Zhang, Mingsheng Long, and Han Hu.  Negative margin matters: Understanding margin in few-shot classification.  In _European Conference on Computer Vision (ECCV)_ , 2020.
  * Liu et al. (2017) Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song.  Sphereface: Deep hypersphere embedding for face recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_ , 2017.
  * Lucas et al. (2021) James Lucas, Mengye Ren, Irene Raissa KAMENI KAMENI, Toniann Pitassi, and Richard Zemel.  Theoretical bounds on estimation error for meta-learning.  In _International Conference on Learning Representations(ICLR)_ , 2021\.
  * Pahde et al. (2021) Frederik Pahde, Mihai Puscas, Tassilo Klein, and Moin Nabi.  Multimodal prototypical networks for few-shot learning.  In _Winter Conference on Applications of Computer Vision(WACV)_ , pp. 2644–2653, 2021.
  * Perlman (1974) Michael D. Perlman.  Jensen’s inequality for a convex vector-valued function on an infininite-dimensional space.  In _J Multivar Anal_ , pp. 52–65, 1974.
  * Phoo & Hariharan (2021) Cheng Perng Phoo and Bharath Hariharan.  Self-training for few-shot transfer across extreme task difference.  In _International Conference on Learning Representations(ICLR)_ , 2021\.
  * Qi et al. (2018) Hang Qi, Matthew Brown, and David G Lowe.  Low-shot learning with imprinted weights.  In _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_ , pp. 5822–5830, 2018.
  * Rajeswaran et al. (2017) Aravind Rajeswaran, Chelsea Finn, Sham M. Kakade, and Sergey Levine.  Meta-learning with implicit gradient.  _PMLR_ , pp. 1126–1135, 2017.
  * Ravi & Larochelle (2017) Sachin Ravi and Hugo Larochelle.  Optimization as a model for few-shot learning.  In _International Conference on Learning Representations (ICLR)_ , 2017.
  * Ren et al. (2018) Mengye Ren, Sachin Ravi, Eleni Triantafillou, Jake Snell, Kevin Swersky, Josh B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel.  Meta-learning for semi-supervised fewshot classification.  In _International Conference on Learning Representations (ICLR)_ , 2018.
  * Simon et al. (2020) Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi.  Adaptive subspaces for few-shot learning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_ , 2020.
  * Snell et al. (2017) Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  In _Advances in Neural Information Processing Systems_ , 2017.
  * Sun et al. (2019) Shengli Sun, Qingfeng Sun, Kevin Zhou, and Tengchao Lv.  Hierarchical attention prototypical networks for few-shot text classification.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 476–485, 2019.
  * Sung et al. (2018) Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales.  Learning to compare: Relation network for few-shot learning.  In _Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Tian et al. (2020) Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B. Tenenbaum, and Phillip Isola.  Rethinking few-shot image classification: a good embedding is all you need?  In _European Conference on Computer Vision (ECCV)_ , 2020.
  * Vinyals et al. (2016) Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra.  Matching networks for one shot learning.  In _Advances in Neural Information Processing Systems_ , 2016.
  * Wang et al. (2018) Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu.  Cosface: Large margin cosine loss for deep face recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_ , 2018.
  * Wang et al. (2019a) Jixuan Wang, Kuan-Chieh Wang, Marc T. Law, Frank Rudzicz, and Michael Brudno.  Centroid-based deep metric learning for speaker recognition.  In _International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 3652–3656. IEEE, 2019a.
  * Wang et al. (2019b) Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens van der Maaten.  Simpleshot: Revisiting nearest-neighbor classification for few-shot learning.  volume abs/:1911.04623, 2019b.
  * Wen et al. (2016) Yandong Wen, Kaipeng Zhang, Zhifeng Li, , and Yu Qiao.  A discriminative feature learning approach for deep face recognition.  In _European Conference on Computer Vision (ECCV)_ , 2016.
  * Yang et al. (2021) Shuo Yang, Lu Liu, and Min Xu.  Free lunch for few-shot learning: Distribution calibration.  In _International Conference on Learning Representations(ICLR)_ , 2021\.
  * Yoon et al. (2019) Sung Whan Yoon, Jun Seo, and Jaekyun Moon.  Tapnet: Neural network augmented with task-adaptive projection for few-shot learning.  In _Proceedings of the 36th International Conference on Machine Learning (ICML)_ , 2019.

##  Appendix A Appendix

###  A.1 Existing Upper Bound on Expected Risk for Prototype Classifier

To analyze the behavior of a prototype classifier, we start from the current
study (Cao et al., 2020). The following theorem is the upper bound of the
expected risk of prototypical networks with the next conditions.

  * •

The probability distribution of an extracted feature $\phi(\bm{x})$ given its
class $y=c$ is Gaussian i.e $\mathcal{D}_{y}=\mathcal{N}(\mu_{c},\Sigma_{c})$,
where $\mu_{c}=\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c}}[\phi(\bm{x})]$ and
$\Sigma_{c}=\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c}}[(\phi(\bm{x})-\mu_{c})^{\top}(\phi(\bm{x})-\mu_{c})]$.

  * •

All class-conditioned distributions have the same covariance matrix, i.e.,
$\forall(c,c^{\prime}),\Sigma_{c}=\Sigma_{c^{\prime}}$.

######  Theorem 2.

Let $\mathcal{M}$ be an operation of a prototype classifier on binary
classification defined by equation 1. Then for
$\mu=\mathbb{E}_{c\sim\tau}[\mu_{c}]$ and
$\Sigma=\mathbb{E}_{c\sim\tau}[(\mu_{c}-\mu)^{\top}(\mu_{c}-\mu)]$, the miss-
classification risk of the prototype classifier on binary classification
$\mathrm{R}_{\mathcal{M}}$ satisfies

$\displaystyle\mathrm{R}_{\mathcal{M}}(\phi)\leq
1-\frac{4\operatorname{Tr}(\Sigma)^{2}}{8(1+1/k)^{2}\operatorname{Tr}\left(\Sigma_{c}^{2}\right)+16(1+1/k)\operatorname{Tr}\left(\Sigma\Sigma_{c}\right)+\mathbb{E}\operatorname{dist_{L2}^{2}}(\bm{\mu}_{c_{1}},\bm{\mu}_{c_{2}})},$
(10)

where
$\mathbb{E}\operatorname{dist_{L2}^{2}}(\bm{\mu}_{c_{1}},\bm{\mu}_{c_{2}})=\mathbb{E}_{c_{1},c_{2}}\left[\left(\left(\bm{\mu}_{c_{1}}-\bm{\mu}_{c_{2}}\right)\top\left(\bm{\mu}_{c_{1}}-\bm{\mu}_{c_{2}}\right)\right)^{2}\right]$.

We show the detail of the derivation in Appendix A.2.

###  A.2 Derivation Details of Theorem 2

we will explain the rough sketch of derivation of Theorem 2. In prototype
classifier, from equation 1 and equation 3, $R_{\mathcal{M}}$ is written with
sigmoid function $\sigma$ as follows:

$\displaystyle R_{\mathcal{M}}(\phi)=$
$\displaystyle\mathrm{Pr}_{c_{1},c_{2}\sim\tau,\bm{x}\sim\mathcal{D}_{c_{1}},S\sim\mathcal{D}\otimes
2K}\left(\sigma(\|\phi(\bm{x})-\overline{\phi(S_{c_{2}})}\|-\|\phi(\bm{x})-\overline{\phi(S_{c_{1}})}\|)\leq\frac{1}{2}\right)$
$\displaystyle=$ $\displaystyle\mathrm{Pr(\alpha<0)},$ (11)

where
$\alpha\triangleq\|\phi(\bm{x})-\overline{\phi(S_{c_{2}})}\|-\|\phi(\bm{x})-\overline{\phi(S_{c_{1}})}\|$.
we bound equation A.2 with expectation and variance of $\alpha$ by following
proposition.

######  Proposition 1.

From the one-sided Chebyshev’s inequality, it immediately follows that:

$\mathrm{R}_{\mathcal{M}}(\phi)=\Pr(\alpha<0)\leq
1-\frac{\mathbb{E}_{S\sim\mathcal{D}^{\otimes
2k}}\mathbb{E}_{c_{1},c_{2}\sim\tau}\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c_{1}}}[\alpha]^{2}}{\mathrm{Var}_{S,c_{1},c_{2},\bm{x}}[\alpha]+\mathbb{E}_{S\sim\mathcal{D}^{\otimes
2k}}\mathbb{E}_{c_{1},c_{2}\sim\tau}\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c_{1}}}[\alpha]^{2}}.$
(12)

equation A.2 can be further write down as follows by Law of Total Expectation

$\displaystyle\mathrm{Var}_{S,c,\bm{x}}(\alpha)$
$\displaystyle=\mathbb{E}_{S\sim\mathcal{D}^{\otimes
2k}}\mathbb{E}_{c_{1},c_{2}\sim\tau}\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c}}[\alpha^{2}]-\left(\mathbb{E}_{S\sim\mathcal{D}^{\otimes
2k}}\mathbb{E}_{c_{1},c_{2}\sim\tau}\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c_{1}}}[\alpha]\right)^{2}$
$\displaystyle=\mathbb{E}_{c_{1},c_{2}}\mathbb{E}_{\bm{x},S}[\alpha^{2}|c_{1},c_{2}]-\left(\mathbb{E}_{S\sim\mathcal{D}^{\otimes
2k}}\mathbb{E}_{c_{1},c_{2}\sim\tau}\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c_{1}}}[\alpha]\right)^{2}$
$\displaystyle=\mathbb{E}_{c_{1},c_{2}}[\mathrm{Var}_{\bm{x},S}(\alpha|c_{1},c_{2})+\mathbb{E}_{\bm{x},S}[\alpha|c_{1},c_{2}]^{2}]-\mathbb{E}_{c_{1},c_{2}\sim\tau}\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c_{1}}}\mathbb{E}_{S\sim\mathcal{D}_{\otimes
2K}}[\alpha]^{2}.$

Therefore,

$\displaystyle\Pr(\alpha<0)\leq 1-\frac{\mathbb{E}_{S\sim\mathcal{D}^{\otimes
2k}}\mathbb{E}_{c_{1},c_{2}\sim\tau}\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c}}[\alpha]^{2}}{\mathbb{E}_{c_{1},c_{2}}[\mathrm{Var}_{\bm{x}\sim
D_{c_{1}},S}[\alpha]+\mathbb{E}_{\bm{x}\sim D_{c_{1}},S}[\alpha]^{2}]}.$ (13)

We write down the expection and variance of $\alpha$ with following Lemmas 3
and 4.

######  Lemma 3.

Under the same notation and assumptions as Theorem 2, then,

$\displaystyle\mathbb{E}_{S\sim\mathcal{D}^{\otimes
2k}}\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c_{1}}}[\alpha]$
$\displaystyle=(\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}})\top(\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}})$
$\displaystyle\mathbb{E}_{c_{1},c_{2}\sim\tau}\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c_{1}}}\mathbb{E}_{S\sim\mathcal{D}^{\otimes
2K}}[\alpha]$ $\displaystyle=2\mathrm{Tr}(\Sigma).$

######  Lemma 4.

Under the same notation and assumptions as Theorem 2, then,

$\mathbb{E}_{c_{1},c_{2}}\left[\mathrm{Var}_{\bm{x},S}\left[\alpha|c_{1},c_{2}\right]\right]\leq
8\left(1+\frac{1}{K}\right)\mathrm{Tr}\left(\Sigma_{\tau}((1+\frac{1}{K})\Sigma_{\tau}+2\Sigma)\right).$
(14)

The proofs of the above lemmas are in the current study (Cao et al., 2020).

With Proposition 1 and Lemma 3, Lemma 4, we obtain Theorem 2.

###  A.3 Detail of Feature-Trasnforming Methods

####  $L_{2}$ normalization ($L_{2}$-norm)

From equation 6, normalizing the norm of the feature vectors can improve the
performance of a prototype classifier. We denote a function normalizing the
norm as $\psi_{L2}$ given by

$\displaystyle\psi_{L2}(\phi(\bm{x}))=\frac{\phi(\bm{x})}{\|\phi(\bm{x})\|}.$
(15)

#### variance-normalization

From equation 7, normalizing the Euclidean distance in equation 1 by the
variance of each class embedding can improve the performance of a prototype
classifier. This can be interpreted as a variation of the Mahalanobis distance
with a diagonal covariance matrix. We denote a function that measures the
distance of two vectors normalized by its variance at each dimension as
follows:

$\displaystyle\psi_{\mathrm{maha}}(\phi(\bm{x})_{1}),\phi(\bm{x})_{2}))=\sum_{d=1}^{D}\frac{\left(\phi(\bm{x})_{1})^{(d)}-\phi(\bm{x})_{2})^{(d)}\right)^{2}}{\overline{\mathrm{Var}[\phi(\bm{x})_{1})^{(d)}]}},$
(16)

where $\overline{\mathrm{Var}[z]}$ is sample variance of $z$. equation 1 is
rewritten with $\psi_{\mathrm{maha}}(\phi(\bm{x}_{1}),\phi(\bm{x}_{2}))$ as
the following equation.

$\displaystyle\mathcal{M}(\phi,\bm{x},S)_{j}=\frac{\exp{\left(-\psi_{\mathrm{maha}}\left(\phi(\bm{x}),\overline{\phi(S_{j})}\right)\right)}}{\sum_{l=1}^{N}\exp{\left(-\psi_{\mathrm{maha}}\left(\phi(\bm{x}),\overline{\phi(S_{l})}\right)\right)}}.$
(17)

#### LDA

From equation 8, increasing the ratio of the between-class variance to the
within-class variance can improve the performance of a prototype classifier.
LDA (Fukunaga, 1990) is widely used to search for a projection space that
maximizes the between-class variance and minimizes the within-class variance.
It computes the eigenvectors of a matrix
$\hat{\Sigma_{\tau}}^{-1}\hat{\Sigma}$, where $\hat{\Sigma}$ is the covariance
matrix of prototypes and $\hat{\Sigma_{\tau}}$ is the class-conditioned
covariance matrix. Since the number of data is small in few-shot settings,
$\hat{\Sigma_{\tau}}^{-1}$ cannot be estimated stably and we add a regularizer
term to $\hat{\Sigma_{\tau}}$ and define it as
$\hat{\Sigma_{\tau}}_{\mathrm{reg}}$.

$\displaystyle\hat{\Sigma_{\tau}}_{\mathrm{reg}}=\hat{\Sigma_{\tau}}+\lambda
I,$ (18)

where $\lambda\in R^{D},I\in R^{D\times D}$ is identity matrix.

#### EST

Since computation of LDA is unstable, we also analyze the effect of EST (Cao
et al., 2020). EST computes eigenvectors of a matrix
$\hat{\Sigma}-\rho\hat{\Sigma_{\tau}}$: the difference between the covariance
matrix of the class mean vectors and the class mean covariance matrix with
weight parameter $\rho$. Similar to LDA, EST also searches for the projection
space that maximizes the $\Sigma$ and minimizes the $\Sigma_{\tau}$. We
compute the eigenvectors of the matrix with two small modification. The first
is that we remove hyper-parameter $rho$ and then simply computes the
eigenvectors of $\hat{\Sigma}-\hat{\Sigma_{\tau}}$. The second is that we
constructs the matrix from a support set in the multi-shot setting while we
follows the original study in the $1$-shot setting.

#### EST+$L_{2}$-norm

We hypothesize that the combination of the transforming methods can improve
the performance of a prototype classifier independently from each other.
Specifically, we focus on reducing equation 6 and equation 8 by combining EST
and $L_{2}$-norm. We first apply EST to reduce equation 8 and after that we
apply $L_{2}$-norm to reduce equation 6. At the end of the operation we want
the variance of the norm to be $0$, thus we apply EST and after that we apply
$L_{2}$-norm.

###  A.4 Derivation Details of Theorem 1

We start the proof from equation 13. We first prove the following Lemma 5
related to the expectation statistics of $\alpha$ in equation 13.

######  Lemma 5.

Under the same notations and assumptions as Theorem 1, then,

$\displaystyle\mathbb{E}_{S\sim\mathcal{D}^{\otimes
2k}}\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c_{1}}}[\alpha]$
$\displaystyle=\frac{1}{K}\left(\operatorname{Tr}(\Sigma_{c_{2}})-\operatorname{Tr}(\Sigma_{c_{1}})\right)+(\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}})\top(\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}})$
$\displaystyle\mathbb{E}_{c_{1},c_{2}\sim\tau}\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c_{1}}}\mathbb{E}_{S\sim\mathcal{D}_{\otimes
2K}}[\alpha]$ $\displaystyle=2\mathrm{Tr}(\Sigma).$

###### Proof.

First, from the definition of $\alpha$, we split
$\mathbb{E}_{S\sim\mathcal{D}^{\otimes
2k}}\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c_{1}}}[\alpha]$ in to two parts and
examine them seperately.

$\displaystyle\mathbb{E}_{S\sim\mathcal{D}^{\otimes
2k}}\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c_{1}}}[\alpha]=\underbrace{\mathbb{E}\left[\left\lVert\phi(\bm{x})-\overline{\phi(S_{c_{2}})}\right\rVert^{2}\right]}_{(i)}-\underbrace{\mathbb{E}\left[\left\lVert\phi(\bm{x})-\overline{\phi(S_{c_{1}})}\right\rVert^{2}\right]}_{(ii)}.$
(19)

In regular conditions, for random vector $X$, the expectation of the norm is

$\displaystyle\mathbb{E}[X^{\top}X]=\operatorname{Tr}(\mathrm{Var}(X))+\mathbb{E}[{X}]^{\top}\mathbb{E}[X],$
(20)

and the variance of the vector is

$\displaystyle\mathrm{Var}(X)=\mathbb{E}[XX^{\top}]-\mathbb{E}[X]E[X]^{\top}$
(21)
$\displaystyle\Sigma_{c_{i}}\overset{\Delta}{=}\mathrm{Var}_{\bm{x}\sim\mathcal{D}_{c_{i}}}(\phi(\bm{x})).$
(22)

Hence,

$\displaystyle(i)$ $\displaystyle=\mathbb{E}_{\bm{x}\sim
D_{c_{1}}}\mathbb{E}_{S}\left[\left\lVert\phi(\bm{x})-\overline{\phi(S_{c_{2}})}\right\rVert^{2}\right]$
$\displaystyle=\mathrm{Tr}\left(\mathrm{Var}_{\bm{x}\sim
D_{c_{1}},{S}}\left[{\phi(\bm{x})-\overline{\phi(S_{c_{2}})}}\right]\right)+\mathbb{E}_{\bm{x}}\mathbb{E}_{S}\left[\phi(\bm{x})-\overline{\phi(S_{c_{2}})}\right]^{\top}\mathbb{E}_{\bm{x}}\mathbb{E}_{S}\left[\phi(\bm{x})-\overline{\phi(S_{c_{2}})}\right],$
(23)

where the first term inside the trace can be expanded as:

$\displaystyle\mathrm{Var}\left[\phi(\bm{x})-\overline{\phi(S_{c_{2}})}\right]$
$\displaystyle=\mathbb{E}\left[\left(\phi(\bm{x})-\overline{\phi(S_{c_{2}})}\right)\left(\phi(\bm{x})-\overline{\phi(S_{c_{2}})}\right)^{\top}\right]-(\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}})(\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}})^{\top}$
$\displaystyle=\mathrm{Var}\left[\phi(\bm{x})\right]+\mathbb{E}\left[\phi(\bm{x})\right]\mathbb{E}\left[\phi(\bm{x})\right]^{\top}+\mathrm{Var}\left[\overline{\phi(S_{c_{2}})}\right]+\mathbb{E}\left[\overline{\phi(S_{c_{2}})}\right]\mathbb{E}\left[\overline{\phi(S_{c_{2}})}\right]^{\top}$
$\displaystyle\hskip
17.07164pt-\bm{\upmu}_{c_{2}}\bm{\upmu}_{c_{1}}^{\top}-\bm{\upmu}_{c_{1}}\bm{\upmu}_{c_{2}}^{\top}-(\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}})(\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}})^{\top}$
$\displaystyle=\Sigma_{c_{1}}+\frac{1}{K}\Sigma_{c_{2}}\quad\text{(Last terms
cancel out)}.$ (24)

The second term in equation A.4 is simply as follows.

$\displaystyle\mathbb{E}_{\bm{x}\sim
D_{c_{1}}}\mathbb{E}_{S}\left[\phi(\bm{x})-\overline{\phi(S_{c_{2}})}\right]=\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}}.$
(25)

From equation A.4 and equation 25 we obtain

$\displaystyle(i)=\operatorname{Tr}(\Sigma_{c_{1}})+\frac{1}{K}\operatorname{Tr}(\Sigma_{c_{2}})+(\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}})^{\top}(\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}}).$
(26)

Similarly for ii,

$\displaystyle(ii)$
$\displaystyle=\operatorname{Tr}\left(\mathrm{Var}_{\bm{x}\sim
D_{c_{1}},{S}}\left[\phi(\bm{x})-\overline{\phi(S_{c_{1}})}\right]\right)+\mathbb{E}_{\bm{x}}\mathbb{E}_{S}\left[\phi(\bm{x})-\overline{\phi(S_{c_{1}})}\right]^{\top}\mathbb{E}_{\bm{x}}\mathbb{E}_{S}[\phi(\bm{x})-\overline{\phi(S_{c_{1}})}]$
$\displaystyle=\operatorname{Tr}(\Sigma_{c_{1}})+\frac{1}{K}\operatorname{Tr}(\Sigma_{c_{1}}).$
(27)

From $(i)$ and $(ii)$, and equation 19

$\displaystyle\mathbb{E}_{S\sim\mathcal{D}^{\otimes
2k}}\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c_{1}}}[\alpha]=\frac{1}{K}\left(\operatorname{Tr}\left(\Sigma_{c_{2}}\right)-\operatorname{Tr}\left(\Sigma_{c_{1}}\right)\right)+\left(\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}}\right)^{\top}\left(\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}}\right).$
(28)

Since
$\mathbb{E}_{c_{1},c_{2},\bm{x},S}[\alpha]=\mathbb{E}_{c_{1},c_{2}\sim\tau}[\mathbb{E}_{S\sim\mathcal{D}^{\otimes
2k}}\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c_{1}}}[\alpha]]$,

$\displaystyle\mathbb{E}_{c_{1},c_{2},\bm{x},S}\left[\alpha\right]$
$\displaystyle=\mathbb{E}_{c_{1},c_{2}\sim\tau}\left[\frac{1}{K}(\operatorname{Tr}(\Sigma_{c_{2}})-\operatorname{Tr}(\Sigma_{c_{1}}))+\left(\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}}\right)^{\top}\left(\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}}\right)\right]$
$\displaystyle=\mathbb{E}_{c_{1},c_{2}\sim\tau}\left[\left(\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}}\right)^{\top}\left(\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}}\right)\right]$
$\displaystyle=\mathbb{E}_{c_{1},c_{2}\sim\tau}\left[\bm{\upmu}_{c_{1}}^{\top}\bm{\upmu}_{c_{1}}+\bm{\upmu}_{c_{2}}^{\top}\bm{\upmu}_{c_{2}}-\bm{\upmu}_{c_{1}}^{\top}\bm{\upmu}_{c_{2}}-\bm{\upmu}_{c_{2}}^{\top}\bm{\upmu}_{c_{1}}\right]$
$\displaystyle=\mathrm{Tr}\left(\Sigma\right)+\bm{\upmu}^{\top}\bm{\upmu}+\mathrm{Tr}\left(\Sigma\right)+\bm{\upmu}^{\top}\bm{\upmu}-2\bm{\upmu}^{\top}\bm{\upmu}$
$\displaystyle=2\mathrm{Tr}\left(\Sigma\right).$ (29)

∎

Next we prove the following Lemma 6 related to the conditioned variance of
$\alpha$.

######  Lemma 6.

Under the same notation and assumptions as Theorem 1,

$\displaystyle\mathbb{E}_{c_{1},c_{2}}\mathrm{Var}_{\bm{x}\sim
D_{c_{1}},S\sim\mathcal{D}^{\otimes 2N}}[\alpha]$
$\displaystyle\leq\frac{4}{K}\mathbb{E}_{c\sim\tau}\mathrm{Var}\left[\left\lVert\phi(\bm{x})\right\rVert^{2}\right]+\frac{4}{K}\mathrm{Var}_{c\sim\tau}\left[\operatorname{Tr}(\Sigma_{c})\right]+\mathrm{V}_{\mathrm{wit}}(\Sigma_{\tau},\Sigma,\bm{\upmu}),$
(30)

where

$\displaystyle\mathrm{V}_{\mathrm{wit}}\left(\Sigma_{\tau},\Sigma,\bm{\upmu}\right)$
$\displaystyle=\frac{8}{K}\left(\operatorname{Tr}(\Sigma_{\tau})\right)\left(\operatorname{Tr}(\Sigma)+\bm{\upmu}^{\top}\bm{\upmu}\right)+4\left(\operatorname{Tr}(\Sigma)+\bm{\upmu}^{\top}\bm{\upmu}\right)^{2}.$
(31)

###### Proof.

We start with the inequality between the variance of $2$ random variables. We
define $\mathrm{Cov}(A,B)$ as covariance of $2$ random variables $A,B$.

$\displaystyle\mathrm{Var}[A+B]$
$\displaystyle=\mathrm{Var}[A]+\mathrm{Var}[B]+2\mathrm{Cov}(A,B)$
$\displaystyle\leq\mathrm{Var}[A]+\mathrm{Var}[B]+2\sqrt{\mathrm{Var}[A]\mathrm{Var}[B]}$
$\displaystyle\leq\mathrm{Var}[A]+\mathrm{Var}[B]+2\cdot\frac{\mathrm{Var}[A]+\mathrm{Var}[B]}{2}$
$\displaystyle=2\mathrm{Var}[A]+2\mathrm{Var}[B].$ (32)

For $\mathrm{Var}_{\bm{x}\sim D_{c_{1}},S}[\alpha]$,

$\displaystyle\mathrm{Var}_{\bm{x}\sim D_{c_{1}},S}[\alpha]$
$\displaystyle=\mathrm{Var}\left[\left\lVert\phi(\bm{x})-\overline{\phi(S_{c_{2}})}\right\rVert^{2}-\left\lVert\phi(\bm{x})-\overline{\phi(S_{c_{1}})}\right\rVert^{2}\right]$
$\displaystyle=\mathrm{Var}\left[\left\lVert\overline{\phi(S_{c_{1}})}\right\rVert^{2}-\left\lVert\overline{\phi(S_{c_{2}})}\right\rVert^{2}-2\phi(\bm{x})^{\top}\left(\overline{\phi(S_{c_{2}})}-\overline{\phi(S_{c_{1}})}\right)\right]$
$\displaystyle\leq
2\mathrm{Var}\left[\left\lVert\overline{\phi(S_{c_{1}})}\right\rVert^{2}-\left\lVert\overline{\phi(S_{c_{2}})}\right\rVert^{2}\right]$
$\displaystyle\hskip
28.45274pt+4\mathrm{Var}\left[\phi(\bm{x})^{\top}\left(\overline{\phi(S_{c_{2}})}-\overline{\phi(S_{c_{1}})}\right)\right]\hskip
5.69054pt(\because equation~{}\ref{eq:variance_decomposition})$
$\displaystyle=2\mathrm{Var}\left[\left\lVert\overline{\phi(S_{c_{1}})}\right\rVert^{2}\right]+2\mathrm{Var}\left[\left\lVert\overline{\phi(S_{c_{2}})}\right\rVert^{2}\right]+4\mathrm{Var}\left[\phi(\bm{x})^{\top}(\overline{\phi(S_{c_{2}})}-\overline{\phi(S_{c_{1}})})\right].$
(33)

From $3$rd line to $4$th line we deompose the variance of
$\left\lVert\overline{\phi(S_{c_{1}})}\right\rVert^{2}-\left\lVert\overline{\phi(S_{c_{2}})}\right\rVert^{2}$
use the independence of $\overline{\phi(S_{c_{1}})}$ and
$\overline{\phi(S_{c_{2}})}$ with their class given. Next we focus on
$\mathrm{Var}\left[\phi(\bm{x})^{\top}(\overline{\phi(S_{c_{2}})}-\overline{\phi(S_{c_{1}})})\right]$.

$\displaystyle\mathrm{Var}\left[\phi(\bm{x})^{\top}(\overline{\phi(S_{c_{2}})}-\overline{\phi(S_{c_{1}})})\right]$
$\displaystyle=\mathbb{E}\left[\left(\phi(\bm{x})^{\top}\left(\overline{\phi(S_{c_{2}})}-\overline{\phi(S_{c_{1}})}\right)\right)^{2}\right]$
$\displaystyle\hskip
28.45274pt-\left(\mathbb{E}\left[\phi(\bm{x})\right]^{\top}\mathbb{E}\left[\left(\overline{\phi(S_{c_{2}})}-\overline{\phi(S_{c_{1}})}\right)\right]\right)^{2}$
$\displaystyle\leq\mathbb{E}\left[\left(\left\lVert\phi(\bm{x})\right\rVert^{2}\left\lVert\overline{\phi(S_{c_{2}})}-\overline{\phi(S_{c_{1}})}\right\rVert\right)^{2}\right]-(\bm{\upmu}_{c_{1}}\top(\bm{\upmu}_{c_{2}}-\bm{\upmu}_{c_{1}}))^{2}$
$\displaystyle=\mathbb{E}\left[\left\lVert\phi(\bm{x})\right\rVert^{2}\right]\mathbb{E}\left[\left\lVert\overline{\phi(S_{c_{2}})}-\overline{\phi(S_{c_{1}})})\right\rVert^{2}\right]-\left(\bm{\upmu}_{c_{1}}^{\top}\left(\bm{\upmu}_{c_{2}}-\bm{\upmu}_{c_{1}}\right)\right)^{2}.$
(34)

From the $2$nd line to the $3$rd line we use Cauchy–Schwarz inequality. For
$\mathbb{E}\left[\left\lVert\overline{\phi(S_{c_{2}})}-\overline{\phi(S_{c_{1}})}\right\rVert^{2}\right]$,
with equation 20

$\displaystyle\mathbb{E}\left[\left\lVert\overline{\phi(S_{c_{2}})}-\overline{\phi(S_{c_{1}})}\right\rVert^{2}\right]$
$\displaystyle=\frac{1}{K}\left(\operatorname{Tr}\left(\Sigma_{c_{1}}\right)+\operatorname{Tr}\left(\Sigma_{c_{2}}\right)\right)+\left(\bm{\upmu}_{c_{2}}-\bm{\upmu}_{c_{1}}\right)^{\top}\left(\bm{\upmu}_{c_{2}}-\bm{\upmu}_{c_{1}}\right).$
(35)

Thus
$\mathbb{E}_{c_{1},c_{2}\sim\tau}\left[\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c_{1}},S}\left[\left\lVert\phi(\bm{x})\right\rVert^{2}\right]\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c_{1}},S}\left[\left\lVert\overline{\phi(S_{c_{2}})}-\overline{\phi(S_{c_{1}})}\right\rVert^{2}\right]\right]$
is calculated as follows.

$\displaystyle\mathbb{E}_{c_{1},c_{2}\sim\tau}\left[\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c_{1}},S}\left[\left\lVert\phi(\bm{x})\right\rVert^{2}\right]\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c_{1}},S}\left[\left\lVert\overline{\phi(S_{c_{2}})}-\overline{\phi(S_{c_{1}})}\right\rVert^{2}\right]\right]$
$\displaystyle=\mathbb{E}_{c_{1},c_{2}\sim\tau}\left[(\operatorname{Tr}(\Sigma_{c_{1}})+\bm{\upmu}_{c_{1}}^{\top}\bm{\upmu}_{c_{1}})\left(\frac{1}{K}\left(\operatorname{Tr}(\Sigma_{c_{1}})+\operatorname{Tr}(\Sigma_{c_{2}})\right)+(\bm{\upmu}_{c_{2}}-\bm{\upmu}_{c_{1}})^{\top}(\bm{\upmu}_{c_{2}}-\bm{\upmu}_{c_{1}})\right)\right]$
$\displaystyle=\mathbb{E}_{c_{1},c_{2}\sim\tau}\left[\frac{1}{K}\left(\operatorname{Tr}\left(\Sigma_{c_{1}}\right)^{2}+\operatorname{Tr}(\Sigma_{c_{1}})\operatorname{Tr}\left(\Sigma_{c_{2}}\right)\right)\right]$
$\displaystyle\hskip
14.22636pt+\mathbb{E}_{c_{1},c_{2}\sim\tau}\left[\frac{2}{K}\left(\operatorname{Tr}\left(\Sigma_{\tau}\right)\right)\bm{\upmu}_{c_{1}}^{\top}\bm{\upmu}_{c_{1}}+\bm{\upmu}_{c_{1}}^{\top}\bm{\upmu}_{c_{1}}\left(\bm{\upmu}_{c_{2}}-\bm{\upmu}_{c_{1}}\right)^{\top}\left(\bm{\upmu}_{c_{2}}-\bm{\upmu}_{c_{1}}\right)\right]$
$\displaystyle=\frac{1}{K}\left(\mathbb{E}_{c_{1},c_{2}\sim\tau}\left[\operatorname{Tr}(\Sigma_{c_{1}})^{2}\right]+\mathbb{E}_{c_{1},c_{2}\sim\tau}\left[\operatorname{Tr}(\Sigma_{c_{1}})\right]^{2}\right)$
$\displaystyle\hskip
14.22636pt+\frac{2}{K}\left(\operatorname{Tr}\left(\Sigma_{\tau}\right)\right)\left(\operatorname{Tr}\left(\Sigma\right)+\bm{\upmu}^{\top}\bm{\upmu}\right)+\mathbb{E}\left[\bm{\upmu}_{c_{1}}^{\top}\bm{\upmu}_{c_{1}}(\bm{\upmu}_{c_{2}}-\bm{\upmu}_{c_{1}})^{\top}(\bm{\upmu}_{c_{2}}-\bm{\upmu}_{c_{1}})\right]$
$\displaystyle=\frac{1}{K}\mathrm{Var}_{c\sim\tau}\left[\operatorname{Tr}\left(\Sigma_{c}\right)\right]$
$\displaystyle\hskip
14.22636pt+\frac{2}{K}\operatorname{Tr}\left(\Sigma_{\tau}\right)^{2}+\frac{2}{K}\left(\operatorname{Tr}\left(\Sigma_{\tau}\right)\right)\left(\operatorname{Tr}\left(\Sigma\right)+\bm{\upmu}^{\top}\bm{\upmu}\right)+\mathbb{E}_{c_{1},c_{2}}\left[\bm{\upmu}_{c_{1}}^{\top}\bm{\upmu}_{c_{1}}\left(\bm{\upmu}_{c_{2}}-\bm{\upmu}_{c_{1}}\right)^{\top}\left(\bm{\upmu}_{c_{2}}-\bm{\upmu}_{c_{1}}\right)\right]$
$\displaystyle=\frac{1}{K}\mathrm{Var}_{c\sim\tau}\left[\operatorname{Tr}(\Sigma_{c})\right]$
$\displaystyle\hskip
14.22636pt+\frac{2}{K}\operatorname{Tr}(\Sigma_{\tau})\left(\operatorname{Tr}(\Sigma_{\tau})+\operatorname{Tr}(\Sigma)+\bm{\upmu}^{\top}\bm{\upmu}\right)+\mathbb{E}_{c_{1},c_{2}}\left[\bm{\upmu}_{c_{1}}^{\top}\bm{\upmu}_{c_{1}}\left(\bm{\upmu}_{c_{2}}-\bm{\upmu}_{c_{1}}\right)^{\top}\left(\bm{\upmu}_{c_{2}}-\bm{\upmu}_{c_{1}}\right)\right].$
(36)

Now we take into account the term
$-(\bm{\upmu}_{c_{1}}\top(\bm{\upmu}_{c_{2}}-\bm{\upmu}_{c_{1}}))^{2}$ in
equation A.4,

$\displaystyle\mathbb{E}_{c_{1},c_{2}}\left[\bm{\upmu}_{c_{1}}^{\top}\bm{\upmu}_{c_{1}}(\bm{\upmu}_{c_{2}}-\bm{\upmu}_{c_{1}})^{\top}(\bm{\upmu}_{c_{2}}-\bm{\upmu}_{c_{1}})-(\bm{\upmu}_{c_{1}}^{\top}(\bm{\upmu}_{c_{2}}-\bm{\upmu}_{c_{1}}))^{2}\right]$
$\displaystyle\hskip
28.45274pt=\mathbb{E}_{c_{1},c_{2}}\left[\bm{\upmu}_{c_{1}}^{\top}\bm{\upmu}_{c_{1}}\bm{\upmu}_{c_{2}}^{\top}\bm{\upmu}_{c_{2}}-(\bm{\upmu}_{c_{1}}^{\top}\bm{\upmu}_{c_{2}})^{2}\right]$
$\displaystyle\hskip
28.45274pt\leq\mathbb{E}_{c_{1},c_{2}}\left[\bm{\upmu}_{c_{1}}^{\top}\bm{\upmu}_{c_{1}}\bm{\upmu}_{c_{2}}^{\top}\bm{\upmu}_{c_{2}}\right]$
$\displaystyle\hskip
28.45274pt=\left(\operatorname{Tr}(\Sigma)+\bm{\upmu}^{\top}\bm{\upmu}\right)^{2}.$
(37)

Thus
$\mathbb{E}_{c_{1},c_{2}}\left[\mathrm{Var}[\phi(\bm{x})^{\top}(\overline{\phi(S_{c_{2}})}-\overline{\phi(S_{c_{1}})})]\right]$
is calculated as follows.

$\displaystyle\mathbb{E}_{c_{1},c_{2}}\left[\mathrm{Var}[\phi(\bm{x})^{\top}(\overline{\phi(S_{c_{2}})}-\overline{\phi(S_{c_{1}})})]\right]$
(38)
$\displaystyle=\frac{1}{K}\mathrm{Var}_{c\sim\tau}\left[\operatorname{Tr}(\Sigma_{c})\right]+\frac{2}{K}\operatorname{Tr}(\Sigma_{\tau})\left(\operatorname{Tr}(\Sigma_{\tau})+\operatorname{Tr}(\Sigma)+\bm{\upmu}^{\top}\bm{\upmu}\right)+(\operatorname{Tr}(\Sigma)+\bm{\upmu}^{\top}\bm{\upmu})^{2}.$
(39)

Regarding $\left\lVert\overline{\phi(S_{c})}\right\rVert^{2}$, since the
function computing square norm is convex, next equation holds with
$D$-dimensional Jensen’s inequlality (Perlman, 1974):

$\displaystyle\left\lVert\overline{\phi(S_{c})}\right\rVert^{2}$
$\displaystyle=\left\lVert\frac{1}{K}\sum_{i=0\atop x\in
S_{c}}\phi(\bm{x})\right\rVert^{2}$
$\displaystyle\leq\frac{1}{K}\left\lVert\sum_{i=0\atop x\in
S_{c}}\phi(\bm{x})\right\rVert.$ (40)

Combining equation A.4, equation 38, and equation A.4 we obtain

$\displaystyle\mathbb{E}_{c_{1},c_{2}}\mathrm{Var}_{\bm{x}\sim
D_{c_{1}},S\sim\mathcal{D}^{\otimes 2N}}[\alpha]$
$\displaystyle=\frac{4}{K}\mathbb{E}_{c\sim\tau}\mathrm{Var}_{\bm{x}\sim\mathcal{D}_{c}}\left[\left\lVert\phi(\bm{x})\right\rVert^{2}\right]+\frac{4}{K}\mathrm{Var}_{c\sim\tau}\left[\operatorname{Tr}(\Sigma_{c})\right]$
$\displaystyle\hskip
8.53581pt+\frac{8}{K}\operatorname{Tr}(\Sigma_{\tau})\left(\operatorname{Tr}(\Sigma_{\tau})+\operatorname{Tr}(\Sigma)+\bm{\upmu}^{\top}\bm{\upmu}\right)+4\left(\operatorname{Tr}(\Sigma)+\bm{\upmu}^{\top}\bm{\upmu}\right)^{2}.$
(41)

∎

To complete the proof of Theorem 1, we calculate
$\mathbb{E}_{c_{1},c_{2}}\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c_{1}},S\sim\mathcal{D}^{\otimes
2K}}[\alpha]^{2}$.

$\displaystyle\mathbb{E}_{c_{1},c_{2}}\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c_{1}},S\sim\mathcal{D}^{\otimes
2K}}[\alpha]^{2}$ (42)
$\displaystyle=E_{c_{1},c_{2}}\left[\left(\frac{1}{K}(\operatorname{Tr}(\Sigma_{c_{1}})-\operatorname{Tr}(\Sigma_{c_{2}}))+(\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}})^{\top}(\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}})\right)^{2}\right]$
$\displaystyle=\mathbb{E}_{c_{1},c_{2}}\left[\left(\frac{1}{K}(\operatorname{Tr}(\Sigma_{c_{1}})-\operatorname{Tr}(\Sigma_{c_{2}}))\right)^{2}\right]+\mathbb{E}_{c_{1},c_{2}}\left[\left((\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}})^{\top}(\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}})\right)^{2}\right]$
$\displaystyle\hskip
8.53581pt+2\mathbb{E}_{c_{1},c_{2}}\left[\left(\frac{1}{K}(\operatorname{Tr}(\Sigma_{c_{1}})-\operatorname{Tr}(\Sigma_{c_{2}}))\right)\left((\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}})^{\top}(\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}})\right)\right]$
$\displaystyle=\mathrm{Var}_{c_{1},c_{2}\sim\tau}\left[\frac{1}{K}(\operatorname{Tr}(\Sigma_{c_{1}})-\operatorname{Tr}(\Sigma_{c_{2}}))\right]+\left(\mathbb{E}_{c_{1},c_{2}\sim\tau}\left[\frac{1}{K}(\operatorname{Tr}(\Sigma_{c_{1}})-\operatorname{Tr}(\Sigma_{c_{2}}))\right]\right)^{2}$
$\displaystyle\hskip
8.53581pt+\mathbb{E}_{c_{1},c_{2}}\left[\left((\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}})^{\top}(\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}})\right)^{2}\right]$
$\displaystyle=\frac{2}{K^{2}}\mathrm{Var}_{c\sim\tau}\left[\operatorname{Tr}(\Sigma_{c})\right]+\mathbb{E}_{c_{1},c_{2}}\left[\left((\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}})^{\top}(\bm{\upmu}_{c_{1}}-\bm{\upmu}_{c_{2}})\right)^{2}\right].$
(43)

From $2$nd line to $3$rd line, we use the symmetry of the last term with
respect to $c_{1}$ and $c_{2}$ and erase the term. Combining equation 13,
Lemma 5, Lemma 6, and equation 42, we obtain the bound.

###  A.5 Theorem 1 with N-way Classification

The upper bound on the risk of $N$-way prototype classifier is as follows.

######  Theorem 7.

Let operation of binary class prototype classifier $\mathcal{M}$ as defined in
equation 1. Then for $\overline{\phi(S_{c})}=\frac{1}{K}\Sigma_{\bm{x}\in
S_{c}}\phi(\bm{x})$,
$\mu_{c}=\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c}}[\phi(\bm{x})]$,
$\Sigma_{c}=\mathbb{E}_{\bm{x}\sim\mathcal{D}_{c}}[(\phi(\bm{x})-\mu_{c})^{\top}(\phi(\bm{x})-\mu_{c})]$,
$\mu=\mathbb{E}_{c\sim\tau}[\mu_{c}]$,$\Sigma=\mathbb{E}_{c\sim\tau}[(\mu_{c}-\mu)^{\top}(\mu_{c}-\mu)]$,
$\mathbb{E}_{c\sim\tau}[\Sigma_{c}]=\Sigma_{\tau}$, if $\phi(\bm{x})$ has its
fourth moment, miss classification risk of binary class prototype classifier
$\mathrm{R}_{\mathcal{M}}$ satisfy

$\displaystyle\mathrm{R}(\mathcal{M}(\phi,\bm{x},\\{S_{i}\\}_{i=1}^{N}),y)$
$\displaystyle\leq N-1-\sum_{c=1\atop c\neq
y}^{N}\frac{4(\mathrm{Tr}(\Sigma))^{2}}{\mathbb{E}\mathrm{V}[h_{L2}(\phi(\bm{x}))]+\mathrm{V}_{\mathrm{Tr}}(\Sigma_{y})+\mathrm{V}_{\mathrm{wit}}(\Sigma_{\tau},\Sigma,\bm{\upmu})+\mathbb{E}\operatorname{dist_{L2}^{2}}(\bm{\mu}_{y},\bm{\mu}_{c})},$
(44)

where

$\displaystyle\mathbb{E}\mathrm{V}[h_{L2}(\phi(\bm{x}))]=$
$\displaystyle\frac{4}{K}\mathbb{E}_{y\sim\tau}\left[\mathrm{Var}_{\bm{x}_{c}\sim\mathcal{D}_{c}}\left[\left\|\phi(\bm{x})\right\|^{2}\right]\right],$
$\displaystyle\mathrm{V}_{\mathrm{Tr}}(\Sigma_{y})=$
$\displaystyle\left(\frac{4}{K}+\frac{2}{K^{2}}\right)\mathrm{Var}_{c\sim\tau}\left[\mathrm{Tr}\left(\Sigma_{c}\right)\right],$
$\displaystyle\mathrm{V}_{\mathrm{wit}}(\Sigma_{\tau},\Sigma,\bm{\upmu})=$
$\displaystyle\frac{8}{K}\operatorname{Tr}(\Sigma_{\tau})\left(\operatorname{Tr}(\Sigma_{\tau})+\operatorname{Tr}(\Sigma)+\bm{\upmu}^{\top}\bm{\upmu}\right)+4(\operatorname{Tr}(\Sigma)+\bm{\upmu}^{\top}\bm{\upmu})^{2},$
$\displaystyle\mathbb{E}\operatorname{dist_{L2}^{2}}(\bm{\mu}_{y},\bm{\mu}_{c})=$
$\displaystyle\mathbb{E}_{y,c}\left[\left(\left(\bm{\mu}_{c_{y}}-\bm{\mu}_{c}\right)^{\top}\left(\bm{\mu}_{y}-\bm{\mu}_{c}\right)\right)^{2}\right].$

###### Proof.

Let $x,y$ be the input and its class of a query data. We define $\alpha_{c}$
by
$\alpha_{c}=\left\lVert\phi(\bm{x})-\overline{\phi(S_{c})}\right\rVert^{2}-\left\lVert\phi(\bm{x})-\overline{\phi(S_{y})}\right\rVert^{2}$.
Then a prototype classifier miss-classify a class of input $x$, $\hat{y}$, if
$\exists c\in[1,N],c\neq y,\alpha_{c}<0$ . Hence:
$\mathrm{R}_{\mathcal{M}}(\phi)=\Pr(\bigcup_{\begin{subarray}{c}c=1\\\ c\neq
y\end{subarray}}^{N}\alpha_{c}<0)$

By Frechet’s inequality, next equation holds:

$\mathrm{R}_{\mathcal{M}}(\phi)\leq\sum_{\begin{subarray}{c}c=1\\\ c\neq
y\end{subarray}}^{N}\Pr(\alpha_{i}<0).$

Noting that Theorem 1 can be applied to each term in the summation and then we
obtain Theorem 7 ∎


# Engineering the spectral bandwidth of quantum cascade laser frequency combs

Maximilian Beiser  These authors contributed equally to this work  Institute
of Solid State Electronics, TU Wien, 1040 Vienna, Austria  Nikola Opačak
These authors contributed equally to this work  Institute of Solid State
Electronics, TU Wien, 1040 Vienna, Austria  Johannes Hillbrand  Institute of
Solid State Electronics, TU Wien, 1040 Vienna, Austria  Gottfried Strasser
Institute of Solid State Electronics, TU Wien, 1040 Vienna, Austria  Benedikt
Schwarz  Institute of Solid State Electronics, TU Wien, 1040 Vienna, Austria
Corresponding author<EMAIL_ADDRESS>
###### Abstract

Quantum cascade lasers (QCLs) facilitate compact optical frequency comb
sources that operate in the mid-infrared and terahertz spectral regions, where
many molecules have their fundamental absorption lines. Enhancing the optical
bandwidth of these chip-sized lasers is of paramount importance to address
their application in broadband high-precision spectroscopy. In this work, we
provide a numerical and experimental investigation of the comb spectral width
and show how it can be optimized to obtain its maximum value defined by the
laser gain bandwidth. The interplay of nonoptimal values of the resonant Kerr
nonlinearity and the cavity dispersion can lead to significant narrowing of
the comb spectrum and reveals the best approach for dispersion compensation.
The implementation of high mirror losses is shown to be favourable and results
in proliferation of the comb sidemodes. Ultimately, injection locking of QCLs
by modulating the laser bias around the roundtrip frequency provides a stable
external knob to control the FM comb state and recover the maximum spectral
width of the unlocked laser state.

††journal: arxiv

##  1 Introduction

The realization of optical frequency combs (OFCs) [1, 2] has lead to multiple
breakthroughs in fundamental science and enabled manifold applications. The
spectrum of frequency combs consists of a set of evenly spaced modes with a
well-defined phase relation, which facilitates achievements like high
precision spectroscopy down to attosecond resolution [3], frequency synthesis
and optical clocks [4] and enhancing the search for exoplanets [5]. For
spectroscopic applications, the mid-infrared spectral region is a molecular
fingerprint region, since fundamental rotation-vibrational lines of many
molecules are located there [6]. Conventional frequency comb sources in this
region depend on a chain of optical components which limits their usability
and potential for on-chip integration. To broaden the spectroscopic
applications of such mid-infrared combs, a more compact and electrically
driven frequency comb generator is preferred. Quantum cascade lasers (QCLs)
are unipolar semiconductor lasers that emit in the mid-infrared and terahertz
range, which became the dominant light sources in these spectral regions [7].
In 2012, the first quantum cascade laser frequency comb was demonstrated by
Hugi et al. [8]. Unlike the conventional OFCs that rely on the emission of
short pulses, QCL combs produce frequency-modulated (FM) output with an almost
constant intensity and a linear frequency chirp. Moreover, the frequency comb
regime in a free-running QCL is self-starting and requires no additional
optical elements, which led to broad attention from researches recently [9,
8]. Due to their on-chip integration, QCLs have vast potential to miniaturize
high-precision spectroscopy. Mid-infrared dual-comb spectroscopy based on QCLs
[10] was demonstrated and the viability of this technique for real-time
monitoring of chemical reactions was proven [11, 12]. Furthermore, it was
shown that injection of a microwave signal at the cavity roundtrip frequency
enables locking of the repetition rate of the comb [13], which allows coherent
control of the FM comb states using established radio-frequency (RF)
electronics [14]. Recent research in this field demonstrated FM frequency comb
formation in other laser types as well, e.g. interband cascade lasers [15],
quantum dot lasers [16] and laser diodes [17]. The missing essential step
towards the broadband high-precision spectroscopic application of QCLs is a
reliable control of the FM combs states and enhancement of their optical
spectral bandwidth.

Figure 1:  (a) Evolution of the intensity spectrum with the increase of the
resonant Kerr nonlinearity $\beta_{2}$ for a dispersion compensated cavity
(GVD = 0). Initial small values of $\beta_{2}$ lead to unlocked states, as is
indicated by the autocorrelation value smaller than unity. FM comb formation
appears between $\beta_{2}=$35 and 165 pm/V2 with the autocorrelation equal to
unity. Increase of the Kerr nonlinearity narrows the spectrum. Unlocked states
are obtained above $\beta_{2}=$165 pm/V2. Two insets on the right depict
spectra taken at $\beta_{2}=$35 pm/V2, with maximum spectral width and at 87.5
pm/V2, with smaller number of comb modes. (b) Evolution of the spectrum for
$\beta_{2}=$87.5 pm/V2 as the GVD is tuned. The maximum optical bandwidth is
recovered for non-zero GVD of -1500 fs2/mm.

The dispersion of QCL waveguides was early on identified to play an important
role for FM comb formation. A prevailing method to tune the group velocity
dispersion (GVD) is the use of a Gires-Tournouis etalon, where a planar mirror
is placed behind the facet of the laser to induce a wavelength dependent group
delay [18, 19]. A different approach relies on the coupling of the dielectric
waveguide to a plasmonic resonance in order to decrease the waveguide
dispersion [20]. The experimental observations of FM combs in QCLs have mostly
been found in lasers with rather small GVD, which led to the opinion that full
dispersion compensation is the optimum. However, recent theoretical work [21,
22] revealed the key role of the interplay of multiple effects in FM combs
formation, such as the GVD, the Kerr nonlinearity, the intracavity light
intensity profile, etc. It was shown, that the full compensation of the
dispersion does not necessarily lead to broader comb spectra covering the
entire available gain bandwidth or a comb operation at all.

##  2 Discussion and Results

In this paper, we show several ways of how FM combs can be influenced and
optimized. One of the main goals of FM comb engineering is the control of the
number of comb modes in order to increase the optical bandwidth and ultimately
facilitate a broad ruler for dual-comb spectroscopy. The presented guideline
explains the roles of the GVD and resonant Kerr nonlinearity with a numerical
study and reveals the impact of the cavity facet reflectivities. All sets of
optimal parameters lead to the same maximum spectral width, which is
predefined by the gainwidth of the laser active medium. We further present the
theoretical and experimental proof that the same maximal width can also be
achieved by RF modulation of the injected current. This constitutes an
appealing alternative to spectral width optimization, which does not require a
redesign or adaptation of the laser. In a GVD compensated cavity the laser
modes are innately equidistant, which seemingly facilitates mode-locking and
frequency comb formation. While this is the case for the traditional mode-
locked lasers that emit short pulses, the situation is very different in case
of FM comb operation. The linear frequency chirp observed for FM comb
operation is the result of a complex synchronization phenomenon among the
beatings of neighboring comb modes, analogous to anti-phase synchronization
among coupled clocks [16]. Due to the ultra-fast gain recovery time of QCLs
the spectral gain asymmetry due to Bloch gain yields a giant Kerr nonlinearity
[21, 23]. This induced resonant nonlinearity alters the optimal comb
conditions, which is the reason why FM combs have mostly been found in GVD
compensated cavities [20]. Figure 1 shows the dependence of the intensity
spectrum on the Kerr non-linearity for a dispersion compensated QCL with
uncoated identical facets. The simulation is based on the master equation,
which considers spatial hole burning, dispersion and the Kerr nonlinearity
[21]. The laser was simulated to operate well above threshold at 60$\%$ of the
maximum current. Three different regimes can be distinguished depending on the
Kerr non-linearity. Starting from zero nonlinearity, the laser is unlocked and
emits a broad spectrum. Above a threshold minimum value of the nonlinearity
(around $\beta_{2}$=35 pm/V2), FM comb operation is obtained. Intriguingly, a
further increase of the Kerr nonlinearity leads to a significant narrowing of
the spectrum, while the laser remains in the locked state up to
$\beta_{2}$=165 pm/V2. In the optimal case (35 pm/V2), an FM comb is obtained
with the broadest possible spectral width, while for a non optimal case (87.5
pm/V2) the laser emits a much narrower spectrum. As the Kerr nonlinearity
originates from the resonant optical transitions and the asymmetry of the
gainshape, it is determined by the band structure design [23]. Consequently,
further optimization would require a complete redesign of the active region in
order to attain the optimal and maximal spectral width. In the following we
will discuss three alternative ways to recover the widest possible spectrum
starting from the non-optimal active region (87.5 pm/V2).

Dispersion engineering can be applied to tune the laser towards the optimal FM
operation. It was shown above, that a laser cavity with complete dispersion
compensation is optimal for a nonlinearity of $\beta_{2}$=35 pm/V2. However,
for different nonlinearities a specific non-zero GVD value will result in the
largest spectral span. In figure 1b we display the numerical simulation
results for the non-optimal Kerr non-linearity of $\beta_{2}$=87.5 pm/V2,
while sweeping the GVD from -6000 fs2/mm to zero. By tuning the GVD from zero
to -1500 fs2/mm the same maximal spectral span can be recovered as in the case
of $\beta_{2}$=35 pm/V2. The interplay of non-zero GVD and the Kerr
nonlinearity mimics the same effect on the FM comb spectral output. A further
increase of the GVD leads to an unlocked state and then again to a locked
state with decreasing spectral span.

Figure 2: (a) Calculated autocorrelation as both the Kerr nonlinearity and the
reflectivity of one cavity facet are tuned. Region of the parameter space
where FM comb formation is obtained is marked with black dashed lines. (b)
Calculated spectral width for the same parameter sweep. Optimal region with
the largest number of comb modes is indicated with the arrow. Relevant data
where an FM comb is obtained lies between the black dashed lines

A novel second approach for comb engineering is the use of coatings on the
facets, which changes their reflectivities. In a Fabry-Perot laser cavity the
intensities of the left and right propagating components of the field inside
the cavity are increasing as they propagate, since the facets are semi-
transparent. It has to be noted that the light intensity profile inside the
cavity plays a crucial role in FM comb operation. The slope of the intensity
has a large impact on the FM comb operation, which can consequently be altered
by simply changing the coatings of the laser facets. In figure 2a a colormap
of the calculated autocorrelation of the laser field is displayed, where the
value of one indicates a locked comb state. Conversely, unlocked states yield
an autocorrelation factor smaller than unity. We sweep the reflectivity for
one facet from zero to one with the other facet reflectivity set to one, while
sweeping the Kerr nonlinearity from $\beta_{2}$=0 to 65 pm/V2.

A large part of the parameter space is a locked FM comb state (indicated by
the dashed lines). There are two areas of unlocked states when both the Kerr
nonlinearity and the facet reflectivity are either low or relatively large. If
the comb does not lock for low values of the nonlinearity, it can be brought
into the locked regime by increasing the facet reflectivity. Conversely, for
too high nonlinearities, the reflectivity needs to be decreased. In figure 2b
we show the corresponding spectral width in order to identify regions, where
the comb is characterized with a broad intensity spectrum. Evidently, an
optimal condition for a locked laser state with the broadest spectral output
is obtained with a small reflectivity (close to the lower dashed line). One
can see, that the overlap for optimal conditions, a broad spectrum and locked
laser state, is small compared to the size of the locking area in 2a.
Furthermore, as the nonlinearity is determined by the design of the laser
active region, only the reflectivity value can be tuned. Note that due to the
interplay of the GVD and Kerr nonlinearity, dispersion engineering affects the
comb state similarly as changing the nonlinearity in 2a. Our results emphasize
that changing the reflectivity of the laser facet can serve as a practical way
to impact the FM comb operation, particularly as coating technology is already
a part of laser fabrication.

Figure 3: Simulated (a) intensity spectrum and (b) intermodal phase
differences with (bottom) and without (top) the modulation of the injection
current. (c) Experimental intensity spectrum with (bottom) and without (top)
the modulation of the injection current. (d) Simulated spectral width
depending on the modulation depth.

As a third approach we present a technique which does not require a precise
adjustment of the cavity for a specific active region – the injection of an RF
signal around the cavity roundtrip frequency. In order to model the influence
of the modulation of the injection current on the spectrum of FM combs, we
again employ the coherent master equation [21, 24]. We furthermore provide an
experimental proof, similar to recent work [25]. The laser has the same non-
optimal Kerr nonlinearity of 87.5 pm/V2 and zero GVD. The current modulation
is applied to a short section which covers 10% of the laser cavity. In figure
3a&b we show the intensity spectrum and the phase differences of neighboring
modes of the FM comb, both analyzed with and without the current modulation.
Without modulation, the comb spectrum spans over 74 modes. The intermodal
phase differences are linearly splayed over the full range of $2\pi$, which is
characteristic for FM combs and results in a linear frequency chirp. The lower
plot of figure 3a shows the optical spectrum, when the laser is subjected to
strong current modulation with an amplitude of 10$\%$ of the DC driving
current. With RF modulation the comb spectrum significantly broadens from 74
to 134 modes. The slope of the intermodal phase differences in figure 3b
broadens accordingly and flattens out, demonstrating the locked coherent
behaviour of the FM comb state. We experimentally investigated the predicted
behaviour with a high performance mid-infrared quantum cascade laser (QCL)
frequency comb emitting at 8$\mu$m. The QCL cavity is split into a 3.15 mm
long gain section and a 350 $\mu$m short modulation section optimized for
efficient RF injection. Besides the optimized RF geometry of the modulation
section, the band structure of the QCL active material was designed as a
single stack bifunctional active region [26, 24]. When operated at zero-bias,
the QCL works as detector at its lasing wavelength and thus induces strong
optical absorption. This allows a more efficient modulation of the laser with
larger modulation depths. In order to study the response of the QCL combs
spectrum, we operate the modulation section at a DC bias of 2.8V to maximize
the modulation efficiency. The modulation depth is increased continuously up
to 35 dBm injection power. The associated optical output spectrum is displayed
in figure 3c. The spectrum of the free running QCL shows an optical bandwidth
of 30 cm-1 without any modulation. When injecting RF power at the cold cavity
roundtrip frequency, the spectrum is broadened to 50 cm-1. This experimental
proof shows that RF injection can not only be used to stabilize the FM comb
[14], but also to optimize the FM comb state and recover its maximum spectral
width, which is defined by the laser gainwidth. All three presented methods
gave a similar spectral width under optimal conditions for the FM comb. To
further explore this upper limit, we provide a more detailed study of the
spectral broadening by RF modulation. We simulated the spectral broadening
depending on the modulation depth ($I_{RF}/I_{DC}=0-0.3$) for fixed values of
the Kerr non-linearity of 70, 87.5 and 105 pm/$V^{2}$ and an injection
frequency 10 MHz below the the cold cavity roundtrip time. The results are
displayed in figure 3d. For all values of the nonlinearity, the simulated
behaviour is strikingly similar. The spectral width increases considerably up
to a modulation depth of 0.15, where it saturates at an approximately constant
bandwidth of 140 modes. The maximal optical bandwidth achievable by RF
modulation is practically equal to the width of the unlocked state. This
result shows that the spectral width of the FM comb does not exceed the width
of the unlocked laser [25, 27].

##  3 Conclusion

In conclusion, we provided a guideline to engineer FM combs for broadband
spectroscopic application. We investigated the influence of multiple cavity
parameters of FM combs by using numerical models as well as experiments. Due
to the inherent resonant Kerr nonlinearity of the QCL active region, strictly
reducing and compensating the GVD does not result in an optimal frequency comb
regime, in contrast to previous expectations. The profile of the intracavity
light intensity has a crucial role in the FM comb formation. It can be
controlled by tuning the laser facet reflectivities in order to broaden the
comb spectra. Moreover, we demonstrate that high mirror losses are preferable.
Employing a sophisticated model using the master equation approach, we
simulate the FM comb response to a strong modulation of the injection current
around the cavity roundtrip frequency. The experiment with an injection locked
high performance bifunctional quantum cascade laser frequency comb displays
excellent agreement with the modelled behaviour. To overcome the restrictions
that are dominantly imposed by the laser design, RF injection is the method of
choice to circumvent the change of the laser active region and achieve a broad
spectral output with reasonable effort. The further use of RF modulation as an
efficient technique to broaden the spectrum of FM comb will enhance the QCL
applications in dual-comb spectroscopy.

Funding This project has received funding from the European ResearchCouncil
(ERC) under the European Union’s Horizon 2020 researchand innovation programme
(Grant agreement No. 853014). Disclosures The authors declare no conflicts of
interest.

## References

  * [1] T. W. Hänsch, Reviews of Modern Physics 78, 1297 (2006).
  * [2] J. L. Hall, Reviews of Modern Physics 78, 1279 (2006).
  * [3] A. Baltuska, T. Udem, M. Uiberacker, M. Hentschel, E. Goulielmakis, C. Gohle, R. Holzwarth, V. S. Yakovlev, A. Scrinzi, T. W. Hänsch, and F. Krausz, Nature Publishing 421, 611 (2003).
  * [4] T. Udem, R. Holzwarth, and T. W. Hänsch, Nature 416, 233 (2002).
  * [5] T. Wilken, G. L. Curto, R. A. Probst, T. Steinmetz, A. Manescau, L. Pasquini, J. I. G. Hernández, R. Rebolo, T. W. Hänsch, T. Udem, and R. Holzwarth, Nature 485, 611 (2012).
  * [6] F. Keilmann, C. Gohle, and R. Holzwarth, Optics Letters 29, 1542 (2004).
  * [7] J. Faist, F. Capasso, D. L. Sivco, C. Sirtori, A. L. Hutchinson, and A. Y. Cho, Science 264, 553 (1994).
  * [8] A. Hugi, G. Villares, S. Blaser, H. C. Liu, and J. Faist, Nature 492, 229 (2012).
  * [9] M. Singleton, P. Jouy, M. Beck, and J. Faist, Optica 5, 948 (2018).
  * [10] G. Villares, A. Hugi, S. Blaser, and J. Faist, Nature Communications 5 (2014).
  * [11] J. L. Klocke and T. Kottke, Physical Chemistry Chemical Physics 22, 26459 (2020).
  * [12] N. H. Pinkowski, Y. Ding, C. L. Strand, R. K. Hanson, R. Horvath, and M. Geiser, Measurement Science and Technology 31, 055501 (2020).
  * [13] M. R. St-Jean, M. I. Amanti, A. Bernard, A. Calvar, A. Bismuto, E. Gini, M. Beck, J. Faist, H. C. Liu, and C. Sirtori, Laser & Photonics Reviews 8, 443 (2014).
  * [14] J. Hillbrand, A. M. Andrews, H. Detz, G. Strasser, and B. Schwarz, Nature Photonics 13, 101 (2018).
  * [15] B. Schwarz, J. Hillbrand, M. Beiser, A. M. Andrews, G. Strasser, H. Detz, A. Schade, R. Weih, and S. Höfling, Optica 6, 890 (2019).
  * [16] J. Hillbrand, D. Auth, M. Piccardo, N. Opačak, E. Gornik, G. Strasser, F. Capasso, S. Breuer, and B. Schwarz, Phys. Rev. Lett. 124, 023901 (2020).
  * [17] L. A. Sterczewski, C. Frez, S. Forouhar, D. Burghoff, and M. Bagheri, APL Photonics 5, 076111 (2020).
  * [18] J. Hillbrand, P. Jouy, M. Beck, and J. Faist, Opt. Lett. 43, 1746 (2018).
  * [19] G. Villares, S. Riedi, J. Wolf, D. Kazakov, M. J. Süess, P. Jouy, M. Beck, and J. Faist, Optica 3, 252 (2016).
  * [20] Y. Bidaux, I. Sergachev, W. Wuester, R. Maulini, T. Gresch, A. Bismuto, S. Blaser, A. Muller, and J. Faist, Opt. Lett. 42, 1604 (2017).
  * [21] N. Opačak and B. Schwarz, Phys. Rev. Lett. 123, 243902 (2019).
  * [22] D. Burghoff, Optica 7, 1781 (2020).
  * [23] N. Opačak, S. D. Cin, J. Hillbrand, and B. Schwarz, “Frequency comb generation by bloch gain induced giant kerr nonlinearity,” (2021).
  * [24] J. Hillbrand, N. Opačak, M. Piccardo, H. Schneider, G. Strasser, F. Capasso, and B. Schwarz, Nature Communications 11 (2020).
  * [25] F. Kapsalidis, B. Schneider, M. Singleton, M. Bertrand, E. Gini, M. Beck, and J. Faist, “Mid-infrared quantum cascade laser frequency combs with a microstrip-like line waveguide geometry,” (2020).
  * [26] B. Schwarz, P. Reininger, H. Detz, T. Zederbauer, A. Maxwell Andrews, S. Kalchmair, W. Schrenk, O. Baumgartner, H. Kosina, and G. Strasser, Applied Physics Letters 101, 191109 (2012).
  * [27] J. B. Khurgin, Applied Physics Letters 117, 161104 (2020).

sample