# Improving Few-shot Learning with Weakly-supervised Object Localization

###### Abstract

Few-shot learning often involves metric learning-based classifiers, which
predict the image label by comparing the distance between the extracted
feature vector and class representations. However, applying global pooling in
the back-end of the feature extractor may not produce an embedding that
correctly focuses on the class object. In this work, we propose a novel
framework that generates class representations by extracting features from
class-relevant regions of the images. Given only a few exemplary images with
image-level labels, our framework first localizes the class objects by
spatially decomposing the similarity between the images and their class
prototypes. Then, enhanced class representations are achieved from the
localization results. We also propose a loss function to enhance distinctions
of the refined features. Our method outperforms the baseline few-shot model in
miniImageNet and tieredImageNet benchmarks.

Index Terms—  Few-shot learning, metric learning, representation learning,
weakly supervised object localization

##  1 Introduction

Deep learning models based on Convolutional Neural Networks (CNNs) have shown
remarkable performance in the image classification task [1, 2, 3]. These
models are usually trained in a supervised manner, relying on a large-scale
dataset that provides sufficient labeled image samples. However, in practice,
collecting and labeling thousands of target class images would be expensive,
sometimes impossible. Few-shot learning addresses the classification problems
in the limited conditions, where the information of each class is given with
only a few exemplary (or support) images.

In a few-shot setting, the scarcity of sample data makes conventional
classifiers with a fully-connected layer vulnerable to overfitting. Hence
metric learning-based classifiers are preferred instead, where a query image
is mapped to feature space and then classified by the distances to the class
representations acquired from support images [4, 5, 6, 7, 8, 9, 10]. The
Siamese Network architecture [4] is the first to validate the importance of
feature embedding in one-shot learning. Vinyals et al. [5] propose a training
scheme repeating “episodes” of evaluation scenarios as a form of meta-
learning. Prototypical Networks [6] introduce a simple classifier using the
Euclidean distance to the class prototypes, defined as the centroids of each
class’s support feature vectors. More recent works explore different class
representations and distance metrics. Oreshkin et al. [7] suggest a task-
dependent adaptive metric (TADAM). Zhao et al. [8] propose a variational
Bayesian framework that represents each class as a distribution. It uses a
probabilistic-based metric, which belongs to weighted Euclidean distances.
Deep subspace networks (DSN) [9] utilize subspace representations and
classifies a query based on the shortest distance from the query to its
projections onto subspaces.

(a)

(b)

Fig. 1: A support image may contain class-irrelevant instances. Our method
localizes the class object and acquires the region of interest (RoI) feature.
(a) An image of a man and a dog labeled ‘dog.’ (b) The t-SNE plot of features
of the whole image and the RoI, colored in blue and green respectively.
(‘$\times$’ denotes the average feature.)  Fig. 2: Proposed framework. Using
the class prototype, we localize the class object and extract a class-relevant
feature from each support image to compute refined class representation.
$\odot$ denotes the similarity mapping operation in Eq. 3.

In metric learning, embedding quality plays a crucial role [11]. An image
representation vector is conventionally achieved with a CNN encoder followed
by a global average pooling (GAP) layer. However, GAP does not take into
account the spatial information of where the feature is originated. As a
result, the embedding may contain features of class-irrelevant instances or
background. For instance, Fig. 1(a) shows an exemplary image labeled as ‘dog,’
but the extracted feature represents the overall scene of a man with a dog.
Figure 1(b) shows that this causes an undesirable bias from the feature that
focuses on the foreground object. A class representation obtained from such
impure support features may not express the class specifically. To tackle this
problem, we propose a framework that localizes the class object and extracts
only relevant features at the region of interest (RoI). Our method does not
require ground-truth bounding box annotations and localizes the class objects
with weak supervision.

A naive approach to address a weakly supervised object localization (WSOL)
problem is to use class activation mapping (CAM) [12]. It uses the linear
(i.e., fully connected) classifier’s weights to visualize activations focusing
on the most discriminative parts in the images, consequently highlighting the
class objects. Unfortunately, CAM cannot be directly employed for WSOL in few-
shot realm due to the overfitting issue of the linear classifier. As a
countermeasure, we utilize similarity mapping [13], which is originally
proposed to visualize the similar regions between two images.

Our framework spatially decomposes the similarity between the image features
and the class prototypes to localize foreground objects. More class-relevant
features are obtained from the predicted bounding boxes via RoIAlign-based
feature propagation, resulting in refined class representations. We also
introduce a novel loss function that regards our class representations to
improve the model’s discriminative power. Our method outperforms prior metric
learning-based classifiers that do not consider class object locality in
miniImageNet [5] and tieredImageNet [14] datasets. The proposed framework also
produces good localization results.

##  2 Method

###  2.1 Preliminaries

An $N$-way $K$-shot problem is a few-shot classification task, where a small
support set $S=\\{(\textrm{{x}}_{i},y_{i})\\}_{i=1}^{K}$ is given each for $N$
classes. Here, each $\textrm{{x}}_{i}\in\mathbb{R}^{H\times W\times 3}$ is an
RGB image with height $H$ and width $W$, and $y_{i}\in\\{1,\cdots,N\\}$ is the
corresponding label. $S_{c}$ denotes the support set labeled with class $c$,
while $\textrm{{x}}_{i}^{c}$ denotes its support images.

Prototypical network (ProtoNet) [6] computes a $D$-dimensional vector $p_{c}$
as the prototype for each class using an embedding function
$f_{\phi}:\mathbb{R}^{H\times W\times 3}\rightarrow\mathbb{R}^{D}$ with
learnable parameters $\phi$. A prototype is the mean vector of the embeddings
belonging to its class:

$p_{c}=\frac{1}{|S_{c}|}\sum_{\textrm{{x}}_{i}^{c}\in
S_{c}}f_{\phi}(\textrm{{x}}_{i}^{c}).$ (1)

ProtoNet classifies a query image $\textrm{{x}}^{q}$ as the class of the
nearest neighboring prototype in the Euclidean distance metric:

$\hat{y}^{q}=\min_{c}\|f_{\phi}(\textrm{{x}}^{q})-p_{c}\|.$ (2)

###  2.2 Proposed framework

Figure 2 shows an overview of our framework. We adopt the ProtoNet
architecture to produce ‘prototype’ class representations. During the
embedding process, a feature map before back-end spatial pooling layers
$\textrm{FM}^{\textrm{{x}}}\in\mathbb{R}^{h\times w\times D}$ is retrieved for
each image x, where $h$ and $w$ denote the channels’ height and width.

Class objects are commonly visible among the support images. Thus, each
support image would have regions highly correlated with the class
representation. After the prototypes are derived, we can compute the
similarity map $\textrm{SM}^{\textrm{{x}},c}\in\mathbb{R}^{h\times w\times 1}$
to highlight the class-relevant regions of each image by projecting its class
prototype to every feature map coordinate:

$\displaystyle\textrm{SM}^{\textrm{{x}},c}_{(i,j)}=\frac{p_{c}}{\|p_{c}\|}\cdot\textrm{FM}^{\textrm{{x}}}_{(i,j)}$
$\displaystyle\textrm{for }(i,j)=(1,1),\cdots,(h,w).$ (3)

$\textrm{SM}^{\textrm{{x}},c}$ is a scaled spatial decomposition of the cosine
similarity between the image embedding and the prototype [13]. Next, we
upsample similarity maps to the original image size using bilinear
interpolation and segment the regions of which the value is above a relative
threshold $\tau$ of the max value of each similarity map. Then we take a
bounding box that covers the largest connected component.

For every support image, we extract the RoI feature vector
$f_{\phi}^{c}(\textrm{{x}}_{i}^{c})$ from the bounding box of its
corresponding class via RoIAlign [15] and pooling operations. We define a
novel class representation $r_{c}$ as the mean vector of the RoI features
belonging to its class. Our framework follows the classifying strategy in Eq.
2, replacing $p_{c}$ with our new representation $r_{c}$.

###  2.3 Loss function

In ProtoNet, the logits are defined with the squared Euclidean distance to the
prototypes and a temperature parameter $T$. The embedding function is
optimized with a cross-entropy loss:

$L_{base}(y^{q}=c|\textrm{{x}}^{q})=-\log\frac{\exp{(-\|f_{\phi}(\textrm{{x}}^{q})-p_{c}\|^{2}/T)}}{\sum_{k=1}^{N}\exp{(-\|f_{\phi}(\textrm{{x}}^{q})-p_{k}\|^{2}/T)}}.$
(4)

Similarly, we propose a loss term $L_{roi}$ that considers our novel class
representations instead of the prototypes:

$L_{roi}(y^{q}=c|\textrm{{x}}^{q})=-\log\frac{\exp{(-\|f_{\phi}(\textrm{{x}}^{q})-r_{c}\|^{2}/T)}}{\sum_{k=1}^{N}\exp{(-\|f_{\phi}(\textrm{{x}}^{q})-r_{k}\|^{2}/T)}}.$
(5)

$L_{roi}$ has an explicit objective of distinguishing RoI features and allows
the model to have more discriminative ability. Our final loss function is as
follows:

$L_{ours}=L_{base}+\lambda_{roi}L_{roi},$ (6)

where $\lambda_{roi}$ is a balancing parameter.

##  3 Experiments

###  3.1 Setups

We evaluated our framework on popular subsets of ImageNet [16] in few-shot
learning studies; miniImageNet [5] and tieredImageNet [14] datasets.
MiniImageNet includes 600 images for all 100 classes (i.e. total 60,000
images). We followed the conventional split by [17] and used 64 classes for
training, 16 classes for validation, and 20 classes for testing.
TieredImageNet is a large-scale dataset which considers the high-level
category separation between training and testing classes. TieredImageNet
consists of 351, 97, and 160 classes for training, validation, and testing,
respectively.

For both datasets, we evaluated our approach on 5-way 1-shot and 5-way 5-shot
scenarios. We report the average classification accuracy and the 95%
confidence interval after performing evaluation on 10,000 sampled tasks. Each
task tests 15 query images per class (i.e. 75 queries in total).

###  3.2 Implementation details

Our framework is built on the premise that the prototypes already have
reliable representative power, so we trained the ProtoNet to its extent. We
selected the ResNet-12 encoder used in [18] as our embedding function and
initialized it following the pre-training strategy suggested in [19, 20]. We
appended a single layer classifier and trained it for the whole train-split
classes using cross-entropy loss to initialize the embedding network’s
weights. Then we further optimized the feature extractor for 200 epochs with
$L_{base}$ using the episodic training scheme [5]. To be specific, we repeated
random batches of few-shot classification tasks (the same scenarios used in
evaluation) and trained parameters using SGD optimizer with the learning rate
of 0.0001. The temperature parameter $T$ was set to 64, except the case in
5-shot scenario on tieredImageNet where we used $T$= 32.

Our framework proposes the class object bounding box given $\tau$ = 0.5. We
further optimized the encoder’s parameters with $L_{ours}$, setting
$\lambda_{roi}$= 1.0, 0.5 for 1-shot and 5-shot scenarios, respectively. We
used the $\tau$ value of 0.7 instead of 0.5 while training 5-shot scenarios.
Additional training takes 100 epochs, with the other hyperparameters
unchanged.

##  4 Results

Table 1: Few-shot classification results and 95% confidence intervals on miniImageNet and tieredImageNet. | miniImageNet | tieredImageNet
---|---|---
| 5-way 1-shot (%) | 5-way 5-shot (%) | 5-way 1-shot (%) | 5-way 5-shot (%)
TADAM* [7] | 58.50 $\pm$ 0.30 | 76.70 $\pm$ 0.30 | – | –
Variational FSL* [8] | 61.23 $\pm$ 0.26 | 77.69 $\pm$ 0.17 | – | –
DSN* [9] | 62.64 $\pm$ 0.66 | 78.63 $\pm$ 0.45 | 66.22 $\pm$ 0.75 | 82.79 $\pm$ 0.48
CAN* [21] | 63.85 $\pm$ 0.48 | 79.44 $\pm$ 0.34 |  69.89 $\bm{\pm}$ 0.51 | 84.23 $\pm$ 0.37
Optimized with $\bm{L_{base}}$ |  |
ProtoNet [6] | 63.02 $\pm$ 0.21 | 80.48 $\pm$ 0.14 | 67.76 $\pm$ 0.23 | 84.77 $\pm$ 0.16
Ours | 63.37 $\pm$ 0.21 | 80.68 $\pm$ 0.14 | 68.06 $\pm$ 0.23 | 84.91 $\pm$ 0.16
Optimized with $\bm{L_{ours}}$ |  |
ProtoNet | 64.35 $\pm$ 0.21 | 81.15 $\pm$ 0.14 | 68.42 $\pm$ 0.23 | 84.86 $\pm$ 0.16
Ours |  64.86 $\bm{\pm}$ 0.21 |  81.25 $\bm{\pm}$ 0.14 | 68.87 $\pm$ 0.23 |  85.07 $\bm{\pm}$ 0.16
* Results reported by the original work.

###  4.1 Classification results comparison

We compared our method with metric learning-based models that use different
class representations, all using the ResNet-12 backbone structure. ProtoNet
[6], our baseline work, shares the same weights with our framework. We also
compared with cross attention network (CAN) [21], which shares a similar
motivation to ours. CAN extracts a feature that attends to the target object
by inspecting the spatial correlation between the query and support images.

Table 1 shows the classification performance comparison of our method on the
miniImageNet and tieredImageNet. Our framework produces more discriminative
class representations than the prototypes. Further optimization using the
proposed loss function also enhanced the classification performances for both
ProtoNet and our framework. This implies that training with the consideration
of distinguishing RoI features improves the overall embedding quality. Our
method outperformed prior approaches, except the case in the 5-way 1-shot task
on tieredImageNet. CAN exploits the query image to identify the class object,
and thus have strength in 1-shot scenarios. Our method produces a query-
independent class representation that explicitly localizes and focuses on the
target object.

###  4.2 Localization results analysis

Fig. 3: Localization the class objects from query images in a miniImageNet
5-way 5-shot scenario.

Apart from the classification performance, we were concerned about our
framework’s localizing ability. Our framework produces WSOL predictions as
intermediate outcomes. Once we predict a query label, we can also locate the
most probable region of the class object using the similarity map between the
query feature map and the representation vector of the predicted class. Since
neither miniImageNet nor tieredImageNet provide ground-truth bounding box
annotations, we were not able to assess the localization quality numerically.
However, qualitative results in Fig. 3 show that our framework can
successfully localize the class objects even when there are multiple salient
objects.

Fig. 4: Localization of foreground objects in support images in a miniImageNet
5-way 1-shot scenario.

We further examined if our framework can localize the class object even when a
single support image is given. The class object can be inferred as the common
object among the support images that share the same label. In 1-shot
scenarios, however, we cannot know which object the label indicates without
additional information. Nevertheless, our model finds the region most highly
correlated to its overall feature and gives plausible predictions when the
foreground objects are large and significant. Figure 4 shows some of the
success cases of the 1-shot foreground object localization.

##  5 Conclusion

In this paper, we investigate the representation learning in the few-shot
realm and point out that average pooling of the whole image features can
result in impure representations. To overcome this limitation, we propose a
framework that localizes the class-relevant regions of the images with weak
supervision using class prototypes and extracts a refined class representation
for each class. We also introduce a loss function that directly addresses the
refined features. Our approach outperforms the baseline work in various
benchmarks. It also shows reasonable localization results without bounding box
annotations.

## References

  * [1] Karen Simonyan and Andrew Zisserman,  “Very deep convolutional networks for large-scale image recognition,” 2015.

  * [2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [3] J. Hu, L. Shen, and G. Sun,  “Squeeze-and-excitation networks,”  in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 7132–7141.
  * [4] Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov,  “Siamese neural networks for one-shot image recognition,”  in ICML deep learning workshop. Lille, 2015, vol. 2.
  * [5] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.,  “Matching networks for one shot learning,”  in Advances in neural information processing systems, 2016, pp. 3630–3638.
  * [6] Jake Snell, Kevin Swersky, and Richard Zemel,  “Prototypical networks for few-shot learning,”  in Advances in neural information processing systems, 2017, pp. 4077–4087.
  * [7] Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste,  “Tadam: Task dependent adaptive metric for improved few-shot learning,”  in Advances in Neural Information Processing Systems, 2018, pp. 721–731.
  * [8] Jian Zhang, Chenglong Zhao, Bingbing Ni, Minghao Xu, and Xiaokang Yang,  “Variational few-shot learning,”  in Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 1685–1694.
  * [9] Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi,  “Adaptive subspaces for few-shot learning,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 4136–4145.
  * [10] Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha,  “Few-shot learning via embedding adaptation with set-to-set functions,”  in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 8808–8817.
  * [11] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola,  “Rethinking few-shot image classification: a good embedding is all you need?,”  arXiv preprint arXiv:2003.11539, 2020.
  * [12] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba,  “Learning deep features for discriminative localization,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2921–2929.
  * [13] Abby Stylianou, Richard Souvenir, and Robert Pless,  “Visualizing deep similarity networks,”  in 2019 IEEE winter conference on applications of computer vision (WACV). IEEE, 2019, pp. 2029–2037.
  * [14] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel,  “Meta-learning for semi-supervised few-shot classification,”  in Proceedings of 6th International Conference on Learning Representations ICLR, 2018.
  * [15] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick,  “Mask r-cnn,”  in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961–2969.
  * [16] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei,  “Imagenet: A large-scale hierarchical image database,”  in 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009, pp. 248–255.
  * [17] Sachin Ravi and Hugo Larochelle,  “Optimization as a model for few-shot learning,”  in In International Conference on Learning Representations (ICLR), 2017.
  * [18] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto,  “Meta-learning with differentiable convex optimization,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 10657–10665.
  * [19] Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L Yuille,  “Few-shot image recognition by predicting parameters from activations,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 7229–7238.
  * [20] A. Andrei Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell,  “Meta-learning with latent embedding optimization,”  international conference on learning representations, 2019.
  * [21] Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen,  “Cross attention network for few-shot classification,”  Advances in Neural Information Processing Systems, vol. 32, pp. 4003–4014, 2019.