A Unified Framework for Attention-Based Few-Shot Object Detection]A Unified Framework for Attention-Based Few-Shot Object Detection

1,2]Pierre Le<EMAIL_ADDRESS>
1]Anissa<EMAIL_ADDRESS>
[1]L2TI, Université Sorbonne Paris Nord

Few-Shot Object Detection (FSOD) is a rapidly growing field in
computer vision. It consists in finding all occurrences of a given set of
classes with only a few annotated examples for each class. Numerous methods have
been proposed to address this challenge and most of them are based on attention
mechanisms. However, the great variety of classic object detection frameworks
and training strategies makes performance comparison between methods difficult.
In particular, for attention-based FSOD methods, it is laborious to compare the
impact of the different attention mechanisms on performance. This paper aims at
filling this shortcoming. To do so, a flexible framework is proposed to allow
the implementation of most of the attention techniques available in the
literature. To properly introduce such a framework, a detailed review of the
existing FSOD methods is firstly provided. Some different attention mechanisms
are then reimplemented within the framework and compared with all other
parameters fixed.

§ INTRODUCTION

Few-Shot Object Detection (FSOD) is a challenging problem that aims to find all
occurrences of a class in an image given only a few examples. Impressive
progress has been made in object detection in the past decade, mostly because of
deep convolutional networks (see e.g. [1, 2]).
Current state-of-the-art performs high-quality detection, but it requires large
annotated datasets and days of training to achieve that quality. Often, such
requirements could not be met, and it is quite hard to achieve good performance
for a specific task. The few-shot learning trend focuses specifically on this
kind of use cases where data is scarce. Most few-shot approaches rely on a
two-steps training strategy. First, the model is trained on a large dataset,
slightly different from the desired task, to acquire general knowledge. Second,
a fine-tuning step refines the model to learn the actual task. Of course, plenty
of techniques have been introduced so that the fine-tuning does not disrupt base
learning (see e.g. [3]). This has been extensively
studied for classification in the past years. However, detection is a more
challenging task than classification and has been only tackled recently from a
few-shot perspective.

Current FSOD state-of-the-art is mainly based on attention mechanisms, which aim
at extracting information about the task (i.e. semantic information about the
classes that should be detected) from support examples. This information allows
the network to condition the detection on the examples. It allows the network to
adapt on the fly, provided with a few annotated images and a short fine-tuning
step. A seminal work in this direction is presented in reference
[4] which reweights features from the query image (i.e. the
images in which the model performs the detection) with the features extracted
from the support images. Plenty of methods based on the same idea have been
introduced (see e.g. [5, 6, 7, 8]).

Although the attention mechanisms, deployed in most FSOD methods, might differ
from the original one proposed in [4], the main principle remains
the same. Indeed, it combines information from the query image and the support
set to detect only the objects annotated in the support.

The FSOD field is rapidly growing, and most new papers propose a novel attention
technique. However, there are a lot of design choices that can be considered to
address the FSOD problem. First, the detection framework (e.g. Faster R-CNN
[1] or YOLO [2]) and its backbone (e.g.
ResNet-50 or 101), then the different loss functions (e.g. L1, IoU, Focal Loss)
to train each part of the network and finally all the hyperparameters that are
tied to these methods (i.e. from the learning rate to the class splits for
evaluation). All this makes the comparison between FSOD methods difficult. In
this paper, the attention mechanism is considered as the heart of FSOD because
it combines general features extracted from the input image and the conditioning
features extracted from the support examples. Of course, it should be noted that
not all works on FSOD are attention-based. Indeed, there exist some papers that
use metric learning to solve FSOD (see e.g.
[9, 10, 11]). Instead of
combining query and support features, a generic embedding function is learned.
The detection is then obtained from the comparison of the query and support
embeddings. There are also approaches based only on fine-tuning. Although quite
straightforward these methods generally do not perform as well as
attention-based ones.

The main goal of this paper is to review a part of the attention-based FSOD
methods and compare them fairly. To do so, a modular attention framework is
proposed. It allows regrouping most attention-based methods under the same
notations. Specifically, this framework is divided into three parts: spatial
alignment, global attention, and fusion layer. This separation helps to easily
implement the different mechanisms and facilitates the comparison. Most
importantly, this makes it possible to completely fix the other parameters and
design choices without reimplementing each method. That way, a fair comparison
of the performance of the different mechanisms can be obtained. The modular
splitting also allows a deep understanding of what component of the attention is
key for FSOD and gives insights to build better approaches. Moreover, to help
the development of new techniques for FSOD and future comparisons, the code of
the proposed framework will be made
available[<https://github.com/pierlj/aaf_framework>]. To our
knowledge, there exist two papers that review FSOD methods
[12, 13]. However, they focus on general
FSOD while this work targets attention-based FSOD.

The rest of this paper is organized as follows. First, a review of existing
methods for FSOD is conducted in Section <ref>. A formal description
of the attention framework is then proposed in Section <ref>.
Section <ref> confirms the flexibility of the framework with a
comparative analysis of some existing methods for FSOD. All these methods have
been reimplemented through the proposed framework, achieving similar results to
those reported in the original papers.  Some insights about the key elements of
the attention mechanisms are then provided. Section <ref>
concludes this paper.

§ REVIEW OF EXISTING WORK ON FSOD

This section reviews existing work on few-shot object detection. It begins with
a summary of general object detection methods. Next, it presents the
main principles of few-shot learning, and finally, it explains how these
principles are applied to object detection in the scientific literature.

§.§ Overview of Object Detection Methods

Object detection, among other computer vision challenges, has made impressive
progress with the rise of Convolutional Neural Networks (CNN). Some previous
works, such as [14, 15], leverage CNNs to
classify Region of Interest (RoI) generated by classical methods (e.g. selective
search). However, these methods are slow and achieve relatively low performance.
YOLO [2] and Faster R-CNN [1] have been the
first fully convolutional approaches for object detection and are still the root
of most recent work in this field. These two architectures are the most
representative examples of the two kinds of detectors that exist: one-stage
detectors with a trade-off on speed and two-stages detectors with a trade-off on
accuracy. Plenty of improvements have been introduced over these methods. On the
one hand, for one-stage detectors, Focal Loss [16] improves
training balance between background and foreground. CenterNet
[17] and CornerNet [18] propose new ways
to predict boxes coordinates. FCOS [19] gets away from the
concept of anchors boxes (a predefined set of boxes introduced both in YOLO and
Faster R-CNN). On the other hand, two-stages detectors have been improved with
pyramidal features extraction by [20], improving detection of
objects of various sizes. Mask R-CNN [21] improves performance
by adding an instance segmentation branch to Faster R-CNN. Recently, various
works exploit self-attention mechanisms to increase detection quality.
Dynamic Head combines three different types of attention, based on scale,
location and current task [22]. Pushing even further, DETR
[23] replaces the convolutional regressor and classifier with

§.§ Basic Concepts on Few-Shot Learning

Deep learning based methods achieve impressive results when provided with
sufficient training data. However, it is often quite challenging to gather such
a dataset for a given use case. Hence, Few-Shot Learning (FSL) aims at learning
tasks from only limited data. The main principle is to learn generic knowledge
from large-scale training on a task similar to the actual problem. The model
is then adapted to perform the final task from the scarce data available.

In FSL literature, a task is defined as $K$-shots, $N$-ways learning when the
training set only contains $K$ examples for each of its $N$ classes. It is
common to introduce the support and query sets for a given task. The support
contains the available examples: $K$ images for each of the $N$ classes. It
allows the network to adapt to the task by extracting relevant information from
the examples. The query set contains images from the same classes and is used
either for inference or training. For clarity, query and support images will
refer to elements of the query and support sets respectively. Moreover, to
assess the performance of FSL the set of classes is divided in base (or seen)
and novel (or unseen) classes. Train and test classes can also be found in the
literature. A large quantity of examples is available for base classes that
serve during base training while for novel classes, only $K$ examples are
available per class. Novel classes are presented to the network during
fine-tuning and evaluation only. This allows assessing the generalization
capabilities of the methods.

As summarized in Figure <ref>,  FSL approaches can be
classified as follows: fine-tuning, meta-learning, metric learning and
attention-based. This classification could be refined as in
[24] which offers a complete review of FSL.

Few-shot learning methods can be divided into 4 main categories:
fine-tuning, meta learning, attention-based learning and metric learning.
Those are not completely separated, in particular, fine-tuning is part of
the training strategy of many FSL methods. Episodic task training is also
a widespread strategy for FSL.

Fine-tuning – This is the simplest way to tackle FSL. It
consists in training the model on a large dataset with base classes examples
only and then fine-tune it with a limited number of examples of the novel
classes. While conceptually simple, these approaches prove to be effective.
Nonetheless, they are often prone to catastrophic forgetting
[3]: performance drops on base classes after
fine-tuning. Plenty of tricks have been introduced to alleviate this issue.
Fine-tuning on its own is not very powerful for FSL, but it is part of most
other methods as their training strategy (see Figure

Meta-learning – It attempts to learn models that can quickly
adapt to a task. This is often performed by training two separate models: a
meta-learner and a student (or learner). The meta-learner's goal is to help
the learner to train on new tasks. This can be achieved in different ways. For
instance, in reference [25], a meta-learner that directly
outputs weights updates for the student is proposed. Similarly, reference
[26] proposes to only output initial weights for the learner.

These techniques rely on a two-level optimization procedure. At the lower level,
a task is selected (e.g. for classification: choosing a random subset of
classes), and then the learner is trained for this task. At the higher level,
the performance of the student is assessed on the current task and serves as a
cost function for the meta-learner. This episodic training strategy improves
adaptability to new tasks. However, these methods do not scale very well as the
meta-learner often needs to be quite larger than the actual task learner.

Metric Learning – The goal of metric learning is to learn a
generic embedding function from the base dataset, such that the embedding space
is structured semantically, making it easier to distinguish between
classes. To do so, a metric function is learned over the data manifold (or
equivalently an embedding function that maps the data points into a Euclidean
space). Input images are then classified according to the distance between their
representations and the representations of the support examples in the embedding
space. This has been introduced in the context of few-shot learning by
Prototypical Networks [27].

Similarly, two separate networks can be trained to embed the support and query
sets separately (e.g. [28]). Following these two methods,
a series of improvements have been developed. For instance, in
[29] a second network is trained to compute the similarity
between prototypes and query embedding.

Attention-based Methods – Finally, there are methods similar
to metric learning but which do not directly compare the query and support
representations. Instead, the support representation is used to change the
parameters of the model on the fly to adapt it to new tasks (i.e. new
classes). The original idea has been proposed by [30].
A classification network is trained at the same time as a learnet whose
purpose is to output some weights for the main network from the support images.
That way, the network has dynamic weights and can adapt to new classes.
Practically, the learnet is trained to output kernel weights adapted to
the task from the support examples. This kernel is then plugged into the main
detection network in a dynamic convolutional layer.

This can be seen as an attention mechanism between a query image and support
images, hence the Attention-based methods. Note that this is referred to as
modulation in [12]. The query features are reweighted
by the support features, with a channel-wise multiplication.

Usually, attention highlights features that are relevant to the task. In the
case of self-attention (i.e. attention on the features themselves), this can be
achieved by multiplying channel-wise a globally pooled feature vector with the
map itself, as proposed by [31]. It can also be with spatially
distant features of the same image as in non-local neural networks
[32] or Visual Transformers (ViT) [33]. But
attention can also be computed with features coming from different images. In
this case, features from external images are highlighted in the original map,
this can be referred to as external or cross attention. This is particularly used
for FSL to adapt query features to support examples. For instance, in
Cross-transformers [34], the authors leverage
transformer-like attention to spatially align the supports to the query. This
makes the model more robust to intra-class variance and support choices.

§.§ Review of Few-Shot Object Detection Methods

Few-Shot Learning has been extensively studied in the light of classification
tasks. For object detection however, the scientific literature is scarcer. This
section aims to present some works proposed for FSOD and to compare them. They
are divided into three different groups: transfer learning, metric learning, and
attention-based learning. To our knowledge, there is no work completely based on
meta-learning that tackles FSOD.

§.§.§ Fine-tuning

Low Shot Transfer Detector (LSTD) is a pioneer work on FSOD [35].
It proposes to first train a detector (Faster R-CNN) on a base dataset and then
fine-tune it on a novel set containing only some examples of the novel classes.
To prevent catastrophic forgetting, the authors introduced two regularization
losses so that the network produces similar outputs for the base classes during
fine-tuning. Closely related, reference [36] leverages
the same idea without any additional loss. Instead, they choose to freeze all
the network weights after base training except for the last classification and
regressions layers. Reference [37] also proposes a basic
fine-tuning strategy, but instead of freezing the network, they leverage a
multiscale refinement branch, used only for training, to better guide the
classification branch of the network. This way, a better balance between
positive and negative samples is achieved, making base training and fine-tuning
more efficient. Another method is proposed in [38] where
two Faster R-CNN are trained: one on base classes and one on all classes (base
and novel), as a fine-tuned version of the first one. Their main contribution is
that both detectors are used together to output predictions. First, the
objectness maps of base and novel detectors are combined so that the Region
Proposal Network (RPN) outputs proposals both for base and novel classes. Next,
the predictions of the base and novel heads are combined to get the bounding
boxes for all classes. The authors also introduce a consistency loss  so that
the novel detector is encouraged to retain knowledge about base classes and
alleviate catastrophic forgetting.

§.§.§ Metric Learning

For few-shot classification, metric-learning based methods are probably the most
widespread. However, for detection, only a few works are based on this
technique. RepMet [9] is one of the first, it consists in
learning class representative vectors while training a modified Faster R-CNN
detector. Closely related, [39] learns prototypes vectors as
well as scale factors inside the CenterNet framework. These vectors are used in
the classification head of the detector as class prototypes. The difference is
that the vectors are learned through training and not simply computed from
examples like in Prototypical Faster R-CNN [10]. The
authors of this method proposed to embed a vanilla prototypical network into
Faster R-CNN not only in the detection head but in the RPN as well. This
prevents the RPN to specialize in base classes and output better proposals.
Nevertheless, authors relate mitigated results with this method. They
hypothesize that computing prototypes from examples may not be the best
detection strategy as the background is often dominant in the image, which can
lead to biased prototypes and shortcut learning.

Reference [11] is halfway between RepMet and Prototypical Faster
R-CNN. It computes the prototypes directly from the example but only in the
second stage of the network. In addition, it leverages a contrastive supervised
loss function to learn a semantically aware embedding space. These methods are
all similar in how the class prototypes influence the classification of RoI. On
the contrary, reference [40] uses the prototypes as
reweighting vectors to enhance class-specific features of the embedded RoI. This
is astride metric learning and attention-based methods, but it is presented from
a metric learning perspective by its authors.

§.§.§ Attention-based

To address the shortcoming of metric learning-based methods, some works proposed
attention-based techniques. The idea is to highlight relevant features for
detection based either on the feature itself (self-attention) or the features of
other instances (external attention). A seminal work in this field is
[4], which trains a reweighting module along with a YOLO
detector. The reweighting module outputs class-specific feature vectors through
Global Pooling (GP) from the support set. These are then channel-wise multiplied
with the query features extracted by the backbone (this operation is denoted
CRW for Class Re-Weighting in Table <ref>). Hence,
class-specific query features are generated, and the detection head computes
predictions for each class separately. Many works have been inspired by this
idea. More generally, generating class-specific features and computing the
detection separately is now standard in FSOD.

This has been used with different detection frameworks, for instance, references
[8] and [41] are built upon Faster R-CNN and FCOS
respectively. Other authors proposed to leverage multi-scale reweighting
vectors, such as in [5]. More sophisticated ways to combine
information from query and support were also proposed. For instance, reference
[42] trains three different heads that link, globally, locally,
and patch-to-patch the features from the query and the support. Graph Neural
Networks (GNN) can also combine the features and learn semantic relations
between classes (see e.g. [43, 44]). Another way to
combine query and support is to simply concatenate the features as in
[22], but it requires that the features have the same spatial

Spatial alignment between query and support feature maps.
Similarity matrix is based on outer product between the maps. For sake of
clarity, maps are reshaped as 2-D matrix where the first dimension controls
the spatial position in the map: $m$ positions for the query and $n$ for
the support. $d$ is the number of channels. Similar colors mean that
features are similar.

Another solution proposes to adjust the size of the features maps: spatial
alignment. This consists in spatially reorganizing a feature map to match the
other. For each location in the query map, the features from one support example
(i.e. feature vectors at each location in the map) are linearly combined to
produce a feature in the aligned support map (see Figure
<ref>). This kind of alignment has been introduced in
[32] for video classification. Feature alignment between
successive frames improved overall performance by capturing long-range spatial
dependencies. This idea has already been tested for FSL as mentioned in Section
<ref>. Many choices can be made for the coefficients of the linear
combination, but often they are dynamically determined by computing the
similarity between query and support features (in Table <ref>,
all are referenced indifferently as QS Alignment standing for
query-support alignment). References [45, 46] are
based on this approach. Thus, the aligned maps can be easily combined and fed to
the detection head. The combination is often performed through a non-learnable
fusion layer composed of several point-wise operations and/or concatenation. In
Table <ref>, these are denoted by their circle operator (e.g.
$\oplus$ for addition), except for concatenation and the identity
($[\cdot,\cdot]$ and Id respectively). In some cases, additional
learnable modules are included to process the combination of query and support
features before concatenation, this is denoted as learnable in Table
<ref>. The alignment mechanism can be associated with the global
attention methods mentioned in the previous paragraph: reweighting features
globally based on the support only, as in [7] or the query and
support similarity as in [47] (Global Similarity
Reweighting). Alignment can also be carried out on the feature itself (i.e.
without information from the support) as in [48, 49].

The DETR framework is well suited for this kind of alignment mechanism as it is
based on Visual Transformers. The few-shot variant of DETR, Meta-DETR described
in [48], combines both self-attention and query-support
alignment, achieving impressive performance. It first applies self-attention
with a multi-head attention module and combines query and support features with
a single-head module where query is taken as queries and support as keys and

Table <ref> summarizes this literature analysis. The table is
designed to compare attention mechanisms as this is the focus of this paper.
Therefore, attention mechanisms are divided into three components that arise
from the previous review: spatial alignment, global attention, and fusion layer.
Methods without attention mechanism are also included in the Table to give a
general overview of the methods available for FSOD.

Comparison of the FSOD methods from an attention perspective.
This table separates the attention mechanisms into three components:
spatial alignment, global attention, and fusion layer. Information about
the original detection framework, the date of publication and the
detection head is also included. All frameworks are working with
multiscale features except for the one with the mention no FPN.

Approach                                                                        Name                             Date      Framework             Alignement                       Attention              Fusion
white                                                                        FRW [4]                    2019               YOLO (no FPN)                  None                                      GP + CRW                      None

white                                                                        RSI   [5]                  2019               YOLO                           None                                      GP + CRW                      None

white                                                                        ARMRD [42]                  2020               Faster R-CNN                   None                                      GP + CRW                      None

white                                                                        VEOW  [41]                  2020               Faster R-CNN                   None                                      GP + CRW                      Pooling + $\text{Cat}\lbrack \odot, \ominus, Id \rbrack$

white                                                                        WSAAN [43]                 2020               Faster R-CNN                   None                                      GP + GNN + CRW                None

white                                                                        CACE  [7]               2020               Faster R-CNN                   QS Alignment                              GP + CRW                      None

white                                                                        KT    [44]                   2020               Faster R-CNN                   None                                      GP + GNN + CRW                None

white                                                                        WOFT  [8]                    2021               FCOS                           None                                      GP + CRW                      Pooling + Cat learnable

white                                                                        FPDI  [50]                  2021               Faster R-CNN                   Iterative Alignment via Optimization      CRW                           None

white                                                                        MFRCN [47]                  2021               Faster R-CNN                   RoI Pooling + QS Alignment                Global Similarity RW          $\text{Cat}\lbrack \ominus, \text{Cat} \rbrack$ learnable

white                                                                        MDETR [48]                2021               DETR (no FPN)                  Transformers-based Alignment              Transformers Self-Attention     None

white                                                                        DRL   [51]               2021               Faster R-CNN                   None                                      None                            Pooling + $\text{Cat}\lbrack \odot, \ominus, Id \rbrack$

white                                                                        DANA  [46]               2021               Faster R-CNN and RetinaNet     QS Alignment                              Background Attenuation          Cat

white                                                                        SP    [49]                    2021               Faster R-CNN                   Self-Alignment (query and support)        None                            Cat

-15*[20mm]whiteAttention                             JCACR [45]                 2021               YOLO                           QS Alignment         (higher order)       None                            Cat
white                                                                        PNPDet[39]              2021               Center Net   (no FPN)       None                                      CRW                           None

-2*[1mm]white1.5cmAttention/ Metric       UPE   [40]              2021               Faster R-CNN                   QS Alignment                              None                            Id $ \oplus$ Cat   learnable
white                                                                        RM    [9]          2018               Faster R-CNN                   None                                      None                            None

white                                                                        FSCE  [11]                  2021               Faster R-CNN                   None                                      None                            None

-3*white1.5cmMetric Learning              PFRCN [10]          2021               Faster R-CNN                   None                                      None                            None
white                                                                        LSTD  [35]                 2018               Faster R-CNN                   None                                      None                            None

white                                                                        WOFG  [38]           2020               Faster R-CNN                   None                                      None                            None

white                                                                        FTS   [36]        2020               Faster R-CNN                   None                                      None                            None

-4*white1.5cmFine-tuning                  MSPSR [37]                  2021               Faster R-CNN                   None                                      None                            None

§ AAF FRAMEWORK FOR ATTENTION IN FSOD

In Section <ref>, three main components of attention
mechanisms for FSOD have been identified. To compare them without other
interfering architecture choices, a unifying framework is proposed. The purpose
of this framework is to provide a flexible environment to represent existing
attention techniques. The three main components are: spatial alignment, global
attention and fusion layer. Most attention-based FSOD methods rely on one or
more of these components as shown in Table  <ref>. Therefore, it
seems convenient to provide a flexible unifying framework that can be used to
implement these methods. The framework should take as input the features from
the query image $\phi_q$ as well as the features extracted from every support
images $\phi_s^c$ for $c \in \mathcal{C}$. It outputs class-specific query
features $\bar{\phi}_q^c$ in which features relative to class $c$ are reinforced
(see Figure <ref>). In order to match the three components of
attention described above, the AAF (Alignment, Attention, Fusion) framework is
also divided into three parts as shown in Figure <ref>. Each component
is described below independently and examples of the possible design choices are
given. Of course, this framework is presented from the perspective of object
detection. However, it could be helpful for other kinds of few-shot tasks such
as classification, instance segmentation, or image generation. As long as a task
requires to condition the results on support examples, this framework could be

The main principle of attention-based FSOD method is to extract
features from the query and support images, and combine query and support
features to create multiple query feature maps, each specialized for the
detection of one class. These maps are then processed by the same detection
head to make the actual predictions.

§.§ Query Support Alignment

The alignment module denoted  $\Lambda$, spatially aligns the features from the
query and the support. It is unlikely that objects of the same class appear at
the same position inside query and support images. Therefore, direct comparison
between their respective feature maps will not produce a high response. This is
commonly avoided by pooling the support map and using it as a class-specific
reweighting vector. This trick loses the spatial information about the support
object, which can be detrimental for the matching. Furthermore, in the case of
detection, it is likely that more than one object is visible in an image. Global
pooling is even more detrimental in this case, as it can mix features from
different objects. Instead, an alignment between query and support maps can be
performed through attention. The idea is to re-organize one feature map in
comparison to the other, so that similar features are spatially close in the
maps. The alignment operator is defined as follows:

\begin{align}
\begin{split}
\tilde{\phi}_q^1, ..., \tilde{\phi}_q^c, \tilde{\phi}_s^1, ..., \tilde{\phi}_s^c = \Lambda(\phi_q^1, \phi_s^1, ..., \phi_s^c)  \ \
\ \text{with } \
\begin{cases}
\tilde{\phi}_q^i &= \lambda_q(\phi_q, \phi_s^i)\\
\tilde{\phi}_s^i &= \lambda_s(\phi_s^i, \phi_q)
\end{cases},
\end{split}
\end{align}

where $\lambda_q$, $\lambda_s$ are linear combinations of their first input:
$\lambda(\phi, \rho) = A(\phi, \rho)^T\phi$ and $A \in \mathbb{R}^{m\times n}$
is an affinity matrix representing the similarity between local features from
$\phi \in \mathbb{R}^{m \times d}$ and $\rho \in \mathbb{R}^{n\times d}$. The
definition of the matrix $A$ determines how features are aligned. This
formulation is quite similar to the non-local blocks described in
[32]. An example of such alignment is described in
[52]. Transformers attention can be understood as an
alignment of the values to match the queries-keys similarity. A major difference
is that the same feature map is used for all inputs (query, key, and value) in
visual transformers so that features are self-aligned.

As an example, Meta Faster R-CNN, described in [47], leverages an
alignment module with an affinity matrix $A(\phi, \rho) = \phi\rho^T$ which
represents the similarity between each pair of spatial locations between query
and support maps. Only the support features are aligned so that they match query
features. This can be implemented in the framework with $A_q(\phi, \rho) = I$
and $A_s(\phi, \rho) = \phi\rho^T$ (see Example A in Figure <ref>).

§.§ Global Attention

The global attention module, denoted  $\Gamma$, combines global information of the supports
and the query. It highlights class-specific features and softens irrelevant
information for the task. This operator is defined as follows:

\begin{align}
\begin{split}
\hat{\phi}_q^1, ..., \hat{\phi}_q^c, \hat{\phi}_s^1, ..., \hat{\phi}_s^c = \Gamma(\tilde{\phi}_q^1, ..., \tilde{\phi}_q^c, \tilde{\phi}_s^1, ..., \tilde{\phi}_s^c)
\ \ \text{with} \
\begin{cases}
\hat{\phi}_q^i = \gamma_q(\tilde{\phi}_q^i, \tilde{\phi}_s^i)\\
\hat{\phi}_s^i = \gamma_s(\tilde{\phi}_s^i, \tilde{\phi}_q^i)
\end{cases} \mkern-18mu ,
\end{split}
\end{align}

where $\gamma_q$, $\gamma_s$ combine the global information from their inputs
and highlight features.

This is generally done through channel-wise multiplication.

In this way, class-specific features are highlighted, while features not
relevant to the class are softened. This module formulation is meant to be
flexible so that a wide variety of global attention mechanisms can fit into it.
For instance, reference [4] pools the support maps with a global
max pooling operation ($\text{GP}$) into a reweighting vector and reweights the
query features channels with it: $\gamma(\phi, \rho) = \phi \circledast
GP(\rho)$ (see Example B in Figure <ref>).

§.§ Fusion Layer

The purpose of the fusion component is to combine query and support maps. This
is only applicable when the maps have the same spatial dimension. It is mostly
used alongside with the alignment module, which does not combine the information
from the support and the query but only reorganize the maps according to the
other. In particular, when support and query maps do not have the same spatial
dimension, aligning support to query can fix the size discrepancy. The fusion
operator is defined as follows:

\begin{align}
\begin{split}
\bar{\phi}_q^1, ..., \bar{\phi}_q^c = \Omega(\hat{\phi}_q^1, ..., \hat{\phi}_q^c, \hat{\phi}_s^1, ..., \hat{\phi}_s^c) \ \
\ \text{with } \
\bar{\phi}_q^i = \omega(\hat{\phi}_q^i, \hat{\phi}_s^i),
\end{split}
\end{align}

where $\omega$ is generally the concatenation of the results of multiple point-wise
operators: $\omega(\phi, \rho) = \omega_1(\omega_2(\phi, \rho), ...,\omega_r(\phi,
\rho))$ with $\omega_i \in \{\oplus, \odot, \ominus, [\cdot,\cdot], ...\}$.

An example of such fusion module is presented in [51] in which
the query and support feature maps are element-wise multiplied, subtracted and
then concatenated together: $\omega(\phi, \rho) = [\phi \odot \rho, \phi \ominus
\rho]$ (see Example C in Figure <ref>). The point-wise operators
can also contain small trainable models such as in [47], where small
CNNs are applied after the point-wise operator, but before the concatenation:
$\omega(\phi, \rho) = [\psi_{dot}(\phi \odot \rho), \psi_{sub}(\phi
\ominus \rho), \psi_{cat}([\phi, \rho])]$.

The overall framework results in the successive application of the three
components presented above:

\begin{equation}
\bar{\phi}_s^1, ..., \bar{\phi}_s^c = \text{AAF}(\phi_q^1, \phi_s^1, ..., \phi_s^c) = (\Omega \circ \Gamma \circ \Lambda)(\phi_q^1, \phi_s^1, ..., \phi_s^c).
\end{equation}

Except for the fusion layer which must be applied last, spatial alignment and
global attention can be applied in any order. In most cases, it will produce
different results as they are probably not commutative. However, some FSOD
methods (e.g. [46]) require applying global attention before
spatial alignment, so the framework must be flexible enough to model this as
well. The whole architecture of the AAF framework is illustrated in Figure
<ref>, in which examples from the previous sections are also depicted.

The Alignment Attention Fusion (AAF) module is composed of
three components: spatial alignment $\Lambda$, global attention $\Gamma$
and a fusion layer $\Omega$. Examples for each module are depicted,
these come from FSOD methods in the literature. Example A is presented
in [47], Example B in [4] and Example C in [51].

§ ATTENTION-BASED FSOD COMPARISON

In order to showcase the flexibility of the proposed AAF module, a comparison
between multiple existing works is conducted. Some methods described in Section
<ref> are selected: FRW [4], WSAAN
[43], DANA [46], MFRCN [47] and
DRL [51] (see Table <ref>). These have been
chosen because they represent well the variety of attention mechanisms available
in the literature. FRW is based on class-specific reweighting vectors, WSAAN has
a more sophisticated global attention and computes reweighting vectors inside a
graph structure. DANA and MFRCN leverage query-support alignment in a slightly
different manner and DRL only uses a sophisticated fusion layer. Each of these
methods has been reimplemented within the AAF framework. Of course, some details
differ from the original implementations, but the purpose of this comparison is
to evaluate only the attention mechanisms. In particular, the backbone and the
training strategy (losses and episode tasks) may differ. The main comparison is
done on Pascal VOC dataset [53] following the literature
on FSOD. Some results are also provided on COCO dataset [54].

§.§ AAF Framework Implementation Details

In order to make the comparison fair, some implementation details are kept fixed
for all experiments. The backbone is a ResNet-50 with a 4-layers Feature Pyramid
Network (FPN) on top [20]. It extracts features at 4 different
levels, which should help the network to detect objects at multiple scales. As
features are extracted at multiple levels, attention mechanisms are also
implemented to work at different scales. This may differ from the original
implementation, but most methods are designed to work at multiscale (see Table
<ref>). The networks are trained in an episodic manner. During
each episode, a subset $\mathcal{C}_{ep} \subset \mathcal{C}$ of the classes is
randomly sampled ($\lvert\mathcal{C}_{ep} \rvert= 5$). Only the annotations of
these classes are shown to the networks. It means that the networks are not
penalized for not detecting other classes. An episode is constituted of 100
images for each class, which constitutes a query set of 500 images. The base
training is made of 1000 (3000 for COCO) of such episodes. With a batch size of
8, this corresponds to 60k (180k for COCO) iterations. The optimization is
performed with SGD and a learning rate of $\num{1e-3}$. The support set is
composed of one example per class (i.e. one image with only one annotation) and
a new support set is sampled at each iteration. Next, a fine-tuning stage occurs
with a smaller learning rate ($\num{1e-4}$). During this phase, test classes are
added to the class selection. For these classes only $k$ examples are available
($k \in \{1,3,5,10\}$). Hence, the query and support sets are both composed of
$5k$ images. Images for base classes change between episodes while for test
classes, they remained fixed. To keep the training comparable between multiple
values of $k$, the number of episodes is adjusted so that the same number of
weight updates is performed with each configuration. The selected methods are
compared on Pascal VOC dataset, following the setup detailed in
[4]. Training images are selected from VOC 2007 and 2012 train
and validation set while the evaluation is performed on VOC 2007 test set. Five
classes among the 20 available have been selected as test classes and have only
been seen by the networks during the fine-tuning. For experiments on COCO, 20
classes are selected as test classes. As in the FSOD literature, these classes
are the ones from Pascal VOC dataset. These details are often different from one
method to another for FSOD. This makes the comparison cumbersome between
different works. The purpose of this framework is to facilitate the
implementation of new attention techniques while providing a fair way to compare

§.§ Results and Analysis on Pascal VOC

In Table <ref>, mean Average Precision (mAP) is reported for each
of the selected methods from Section <ref>. mAP is computed with
a $0.5$ overlap threshold. For each method, the evaluation is split into base
and novel classes. Results on base classes demonstrate the ability of the method
to solve object detection with a large dataset available. This should be
compared to the performance of vanilla FCOS on the same classes ($0.68$). While
it is important to keep performance on base classes close to the baseline, the
most important metric for FSOD is performance on novel classes. Evaluation
on novel classes shows how well a method generalizes to unseen classes with
limited data.

As mentioned in [12], two separate ways to evaluate the
performance on novel classes exist in the FSOD literature. The first one,
introduced by [4] samples only one fixed support set at test time
(the same that is used during fine-tuning). The second one consists in repeating
this process with multiple different support. The former is less reliable and
often overestimates the generalization capabilities. The latter is a lot more
time-consuming as it requires at least 30 different runs (i.e. fine-tuning and
evaluation) to become steady. Results from Table <ref> are
computed using the quicker but less reliable method. However, the goal of this
work is to provide a tool that helps to conduct a fair comparison between
different FSOD techniques. Therefore, it must also adopt fair and reliable
metrics. It is planned to perform more reliable comparisons as future work (i.e.
repeating with different support sets and class splits).

Performance comparison between five selected methods (see Section
<ref>). All are reimplemented with the proposed AAF
framework. Mean average precision is reported for each method on base and
novel classes separately and for various numbers of shots (1, 3, 5 and 10).
Bold values indicate the best performing method for each number of shots and
for base/novel classes separately.

2cFRW [4]     2cWSAAN [43]     2cDANA [46]     2cMFRCN [47]      2cDRL [51]
# Shots     Base                Novel              Base            Novel                    Base                Novel               Base                         Novel        Base                    Novel

1           0.599               0.282              0.617           0.309                    0.626               0.328      0.578                        0.302        0.642          0.270

3           0.633               0.311              0.635           0.422           0.642      0.340               0.587                        0.368        0.617                   0.296

5           0.643               0.463              0.647           0.462           0.652               0.426               0.621                        0.408        0.664          0.373

10          0.632               0.487              0.653           0.517           0.650               0.503               0.634                        0.494        0.670          0.480

From Table <ref>, one can observe, on base classes, a slight
decrease in performance compared to FCOS baseline. This is expected, even if the
network has seen a lot of examples of these classes during training, its
predictions are still conditioned on a few examples, which can sometimes be
misleading. On the other hand, performance on novel classes is significantly
lower than the FCOS baseline (i.e. trained with plenty of examples), especially
for low numbers of shots. The number of shots is crucial for performance on
novel classes. The higher the number of shots, the better the network performs.
In average, with 10 examples per class, the network achieves $0.2$ higher mAP
than with 1 example. More examples provide more precise and robust class
representations, improving detection. The same phenomenon is observed with base
classes with a smaller magnitude ($+ 0.04$ mAP from 1 to 10 shots). Figure
<ref> displays these trends clearly, both for base and novel
classes. More detailed results are given in Figure <ref> where
the evolution of performance per class is plotted. From that, one can notice
that performance varies greatly from a class to another. Classes that are the
most difficult benefit the most from having more examples. This is true both for
base and novel classes. In addition, the novel classes that are easier to detect
are often quite similar to some base classes (e.g. cow is similar to
horse and sheep), thus requiring fewer examples to achieve
good detection.

Evolution of $\text{mAP}_{0.5}$ with the number of shots averaged
on base and novel classes. Each line represents one of the
reimplemented methods. Increasing the number of examples improve
performance of the model. This improvement is larger for novel
classes as the network has not been trained (apart from fine-tuning)
to detect them.

This behavior is expected from any few-shot object detection method. Moreover,
performance values are close to what is reported in the original papers. Of
course, these are not the exact same values as many architectural choices differ
from the proposed methods (e.g. backbone, classes splits, losses, etc.).
Nevertheless, it confirms that the proposed AAF framework is flexible enough to
implement different attention mechanisms. Therefore, it is an appropriate tool
to compare different methods and to find what are the most efficient operations
to combine query and support information for FSOD.

$\text{mAP}_{0.5}$ on Pascal VOC against the number of shots for
each class and each method. Dashed lines represent average
performance on all classes, either base classes(top row) or novel
classes(bottom row).

DRL is arguably the simplest method among the five selected as it leverages
only a fusion layer. It combines query features with the features of each
support image through concatenation and point-wise operations, creating
class-specific query features. It is therefore the closest to the regular FCOS
functioning. This explains the very good performance (compared to the baseline)
on base classes and lower mAP on novel classes. Regarding the other methods, FRW
and WSAAN can be easily compared as both are based on global attention. The only
difference is how the class-specific vectors are computed. In FRW, they are
globally pooled from the support feature map. However, WSAAN combines the same
vector with query features in a GNN. This certainly provides more adaptive
class-specific features yielding better results both on base and novel sets.
The remaining methods, DANA and MFRCN both leverage spatial alignment. While it
seems to bring quite an improvement for DANA over FRW or DRL, the gain is
smaller for MFRCN. In both methods, spatial alignment is not used alone. It is
combined with other attention mechanisms. In DANA, a Background Attenuation
block (i.e. a global self-attention) is applied to the support features to
highlight class-relevant features and soften background ones. In MFRCN, aligned
features are reweighted with global vectors computed from the similarity matrix
between query and support features. This last operation may be redundant as the
similarity information is already embedded into the aligned features, whereas
background attenuation could extract new information. To confirm this
hypothesis, a specific ablation study on both of these methods is planned as
future work.

From this comparison, one can conclude that both global attention and spatial
alignment are beneficial for FSOD. However, these improvements may not always be
compatible. Hence, the design of each component must be done carefully so that
spatial alignment, global attention, and fusion work in unison.

§.§ Results and Analysis on COCO

Another set of experiments is conducted on COCO dataset. Only the two
best-performing methods from Section <ref> are selected and trained
on COCO following the same experimental setup. The results are summarized in
Table <ref>. The mAP value are reported following standards from
Pascal VOC ($\text{mAP}_{0.5}$ with one IoU threshold), and COCO
($\text{mAP}_{0.5:0.95}$ with several thresholds). This is done for 1, 5, 10,
and 30 shots. These results comfort the conclusion obtained on Pascal VOC: the
framework is flexible enough to implement various FSOD techniques that achieve
competitive results with state-of-the-art. As on Pascal VOC the networks achieve
better detection with more shots. While more beneficial for novel classes, base
classes also benefit significantly from a higher number of examples, unlike on
Pascal VOC. WSAAN outperforms DANA on Pascal VOC but performs slightly worse on

Performance comparison between the two best performing methods from
Section <ref> on COCO. $\text{mAP}_{0.5:0.95}$ (COCO
mAP, with IoU thresholds ranging from 0.5 to 0.95) and
$\text{mAP}_{0.5}$ values are reported in the table for base and
novel classes separately and for different numbers of shots.

4c|WSAAN [43]                                                         4cDANA [46]

2c$\text{mAP}_{0.5}$                          2c|$\text{mAP}_{0.5:0.95}$      2c$\text{mAP}_{0.5}$                           2c$\text{mAP}_{0.5:0.95}$
# Shot    Base       1c|Novel     Base        Novel           Base       1c|Novel      Base           Novel

1          0.335                  1c|0.120                 0.201                   0.066                       0.355         1c|0.145         0.213             0.078

5          0.399                  1c|0.199                 0.236                   0.105                       0.428         1c|0.222         0.252             0.119

10         0.409                  1c|0.214                 0.244                   0.115                       0.430         1c|0.237         0.256             0.129

30         0.415                  1c|0.222                 0.247                   0.121                       0.435         1c|0.244         0.260             0.133

§ CONCLUSION AND FUTURE WORK

This paper introduces a modular framework that facilitates the implementation of
different attention mechanisms for FSOD. Thanks to this, a fair comparison
of the various techniques has been conducted. The experiments carried out in
this work demonstrate the flexibility of the proposed framework and prove that
it is convenient for comparing attention techniques. More comparisons are
planned as future work, along with in-depth ablation studies and the design of
new attention techniques for FSOD. To help the development of such methods and
future comparisons, the code of the proposed AAF framework will be made

The authors would like to thank COSE for their close collaboration and the
funding of this project.

#1ISBN #1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1<https://doi.org/#1>et al.#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1<><#>1#1#1#1#1#1#1#1#1#1#1#1#1#1PreBibitemsHook

[1]

Ren, S.,
He, K.,
Girshick, R.,
Sun, J.:
Faster r-cnn: Towards real-time object detection with region proposal
Advances in neural information processing systems

[2]

Redmon, J.,
Divvala, S.,
Girshick, R.,
Farhadi, A.:
You only look once: Unified, real-time object detection.
In: Proceedings of the IEEE Conference on Computer Vision and Pattern
pp. 779–788

[3]

Kirkpatrick, J.,
Pascanu, R.,
Rabinowitz, N.,
Veness, J.,
Desjardins, G.,
Rusu, A.A.,
Milan, K.,
Quan, J.,
Ramalho, T.,
Grabska-Barwinska, A., :
Overcoming catastrophic forgetting in neural networks.
Proceedings of the national academy of sciences

[4]

Kang, B.,
Liu, Z.,
Wang, X.,
Yu, F.,
Feng, J.,
Darrell, T.:
Few-shot object detection via feature reweighting.
In: Proceedings of the IEEE/CVF International Conference on Computer
pp. 8420–8429

[5]

Li, X.,
Deng, J.,
Fang, Y.:
Few-shot object detection on remote sensing images.
IEEE Transactions on Geoscience and Remote Sensing,

[6]

Fan, Q.,
Zhuo, W.,
Tang, C.-K.,
Tai, Y.-W.:
Few-shot object detection with attention-rpn and multi-relation
In: Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition,
pp. 4013–4022

[7]

Hsieh, T.-I.,
Lo, Y.-C.,
Chen, H.-T.,
Liu, T.-L.:
One-shot object detection with co-attention and co-excitation.
In: Advances in Neural Information Processing Systems 32,

[8]

Li, X.,
Zhang, L.,
Chen, Y.P.,
Tai, Y.-W.,
Tang, C.-K.:
One-shot object detection without fine-tuning.
arXiv preprint arXiv:2005.03819

[9]

Karlinsky, L.,
Shtok, J.,
Harary, S.,
Schwartz, E.,
Aides, A.,
Feris, R.,
Giryes, R.,
Bronstein, A.M.:
Repmet: Representative-based metric learning for classification and
few-shot object detection.
In: Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition,
pp. 5197–5206

[10]

Le Jeune, P.,
Lebbah, M.,
Mokraoui, A.,
Azzag, H.:
Experience feedback using representation learning for few-shot object detection
on aerial images.
arXiv preprint arXiv:2109.13027

[11]

Sun, B.,
Li, B.,
Cai, S.,
Yuan, Y.,
Zhang, C.:
Fsce: Few-shot object detection via contrastive proposal encoding.
In: Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition,
pp. 7352–7362

[12]

Huang, G.,
Laradji, I.,
Vazquez, D.,
Lacoste-Julien, S.,
Rodriguez, P.:
A survey of self-supervised and few-shot object detection.
arXiv preprint arXiv:2110.14711

[13]

Jiaxu, L.,
Taiyue, C.,
Xinbo, G.,
Yongtao, Y.,
Ye, W.,
Feng, G.,
Yue, W.:
A Comparative Review of Recent Few-Shot Object Detection Algorithms

[14]

Girshick, R.,
Donahue, J.,
Darrell, T.,
Malik, J.:
Rich feature hierarchies for accurate object detection and semantic
In: Proceedings of the IEEE Conference on Computer Vision and Pattern
pp. 580–587

[15]

Girshick, R.:
Fast r-cnn.
In: Proceedings of the IEEE International Conference on Computer
pp. 1440–1448

[16]

Lin, T.-Y.,
Goyal, P.,
Girshick, R.,
He, K.,
Dollár, P.:
Focal loss for dense object detection.
In: Proceedings of the IEEE International Conference on Computer
pp. 2980–2988

[17]

Duan, K.,
Bai, S.,
Xie, L.,
Qi, H.,
Huang, Q.,
Tian, Q.:
Centernet: Keypoint triplets for object detection.
In: Proceedings of the IEEE/CVF International Conference on Computer
pp. 6569–6578

[18]

Law, H.,
Deng, J.:
Cornernet: Detecting objects as paired keypoints.
In: Proceedings of the European Conference on Computer Vision (ECCV),
pp. 734–750

[19]

Tian, Z.,
Shen, C.,
Chen, H.,
He, T.:
Fcos: Fully convolutional one-stage object detection.
In: Proceedings of the IEEE/CVF International Conference on Computer
pp. 9627–9636

[20]

Lin, T.-Y.,
Dollár, P.,
Girshick, R.,
He, K.,
Hariharan, B.,
Belongie, S.:
Feature pyramid networks for object detection.
In: Proceedings of the IEEE Conference on Computer Vision and Pattern
pp. 2117–2125

[21]

He, K.,
Gkioxari, G.,
Dollár, P.,
Girshick, R.:
Mask r-cnn.
In: Proceedings of the IEEE International Conference on Computer
pp. 2961–2969

[22]

Dai, X.,
Chen, Y.,
Xiao, B.,
Chen, D.,
Liu, M.,
Yuan, L.,
Zhang, L.:
Dynamic head: Unifying object detection heads with attentions.
In: Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition,
pp. 7373–7382

[23]

Carion, N.,
Massa, F.,
Synnaeve, G.,
Usunier, N.,
Kirillov, A.,
Zagoruyko, S.:
End-to-end object detection with transformers.
In: European Conference on Computer Vision,
pp. 213–229

[24]

Wang, Y.,
Yao, Q.,
Kwok, J.T.,
Ni, L.M.:
Generalizing from a few examples: A survey on few-shot learning.
ACM Computing Surveys (CSUR)

[25]

Larochelle, S.R.H.:
Optimization as a model for few-shot learning.
In: 5th International Conference on Learning Representations, ICLR
2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings

[26]

Finn, C.,
Abbeel, P.,
Levine, S.:
Model-agnostic meta-learning for fast adaptation of deep networks.
In: International Conference on Machine Learning,
pp. 1126–1135

[27]

Snell, J.,
Swersky, K.,
Zemel, R.:
Prototypical networks for few-shot learning.
In: Guyon, I.,
Luxburg, U.V.,
Bengio, S.,
Wallach, H.,
Fergus, R.,
Vishwanathan, S.,
Garnett, R. (eds.)
Advances in Neural Information Processing Systems,
vol. 30

[28]

Vinyals, O.,
Blundell, C.,
Lillicrap, T.,
kavukcuoglu, k.,
Wierstra, D.:
Matching networks for one shot learning.
In: Lee, D.,
Sugiyama, M.,
Luxburg, U.,
Guyon, I.,
Garnett, R. (eds.)
Advances in Neural Information Processing Systems,
vol. 29

[29]

Sung, F.,
Yang, Y.,
Zhang, L.,
Xiang, T.,
Torr, P.H.,
Hospedales, T.M.:
Learning to compare: Relation network for few-shot learning.
In: Proceedings of the IEEE Conference on Computer Vision and Pattern
pp. 1199–1208

[30]

Bertinetto, L.,
Henriques, J.F.,
Valmadre, J.,
Torr, P.,
Vedaldi, A.:
Learning feed-forward one-shot learners.
In: Advances in Neural Information Processing Systems,
pp. 523–531

[31]

Hu, J.,
Shen, L.,
Sun, G.:
Squeeze-and-excitation networks.
In: Proceedings of the IEEE Conference on Computer Vision and Pattern
pp. 7132–7141

[32]

Wang, X.,
Girshick, R.,
Gupta, A.,
He, K.:
Non-local neural networks.
In: Proceedings of the IEEE Conference on Computer Vision and Pattern
pp. 7794–7803

[33]

Dosovitskiy, A.,
Beyer, L.,
Kolesnikov, A.,
Weissenborn, D.,
Zhai, X.,
Unterthiner, T.,
Dehghani, M.,
Minderer, M.,
Heigold, G.,
Gelly, S.,
Uszkoreit, J.,
Houlsby, N.:
An image is worth 16x16 words: Transformers for image recognition at
In: International Conference on Learning Representations

[34]

Doersch, C.,
Gupta, A.,
Zisserman, A.:
Crosstransformers: spatially-aware few-shot transfer.
arXiv preprint arXiv:2007.11498

[35]

Chen, H.,
Wang, Y.,
Wang, G.,
Qiao, Y.:
Lstd: A low-shot transfer detector for object detection.
In: Proceedings of the AAAI Conference on Artificial Intelligence,
vol. 32

[36]

Wang, X.,
Huang, T.E.,
Darrell, T.,
Gonzalez, J.E.,
Yu, F.:
Frustratingly simple few-shot object detection.
In: International Conference on Machine Learning (ICML)

[37]

Wu, J.,
Liu, S.,
Huang, D.,
Wang, Y.:
Multi-scale positive sample refinement for few-shot object detection.
In: European Conference on Computer Vision,
pp. 456–472

[38]

Fan, Z.,
Ma, Y.,
Li, Z.,
Sun, J.:
Generalized few-shot object detection without forgetting.
In: Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition,
pp. 4527–4536

[39]

Zhang, G.,
Cui, K.,
Wu, R.,
Lu, S.,
Tian, Y.:
Pnpdet: Efficient few-shot detection without forgetting via
plug-and-play sub-networks.
In: Proceedings of the IEEE/CVF Winter Conference on Applications of
Computer Vision,
pp. 3823–3832

[40]

Wu, A.,
Han, Y.,
Zhu, L.,
Yang, Y.:
Universal-prototype enhancing for few-shot object detection.
In: Proceedings of the IEEE/CVF International Conference on Computer
pp. 9567–9576

[41]

Xiao, Y.,
Marlet, R.:
Few-shot object detection and viewpoint estimation for objects in the
In: European Conference on Computer Vision,
pp. 192–210

[42]

Fan, Q.,
Zhuo, W.,
Tang, C.-K.,
Tai, Y.-W.:
Few-shot object detection with attention-rpn and multi-relation
In: CVPR

[43]

Xiao, Z.,
Qi, J.,
Xue, W.,
Zhong, P.:
Few-shot object detection with self-adaptive attention network for
remote sensing images.
IEEE Journal of Selected Topics in Applied Earth Observations and
Remote Sensing

[44]

Kim, G.,
Jung, H.-G.,
Lee, S.-W.:
Few-shot object detection via knowledge transfer.
In: 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC),
pp. 3564–3569

[45]

Chu, J.,
Feng, J.,
Jing, P.,
Lu, W.:
Joint co-attention and co-reconstruction representation learning for
one-shot object detection.
In: 2021 IEEE International Conference on Image Processing (ICIP),
pp. 2229–2233

[46]

Chen, T.-I.,
Liu, Y.-C.,
Su, H.-T.,
Chang, Y.-C.,
Lin, Y.-H.,
Yeh, J.-F.,
Hsu, W.H.:
Should i look at the head or the tail? dual-awareness attention for few-shot
object detection.
arXiv preprint arXiv:2102.12152

[47]

Han, G.,
Huang, S.,
Ma, J.,
He, Y.,
Chang, S.-F.:
Meta faster r-cnn: Towards accurate few-shot object detection with attentive
feature alignment.
arXiv preprint arXiv:2104.07719

[48]

Zhang, G.,
Luo, Z.,
Cui, K.,
Lu, S.:
Meta-detr: Few-shot object detection via unified image-level meta-learning.
arXiv preprint arXiv:2103.11731

[49]

Xu, H.,
Wang, X.,
Shao, F.,
Duan, B.,
Zhang, P.:
Few-shot object detection via sample processing.
IEEE Access

[50]

Gao, Y.,
Hou, R.,
Gao, Q.,
Hou, Y.:
A fast and accurate few-shot detector for objects with fewer pixels in
drone image.

[51]

Liu, W.,
Li, H.,
Yu, S.,
Chen, S.,
Ye, X.,
Wu, J., et al.:
Dynamic relevance learning for few-shot object detection.
arXiv preprint arXiv:2108.02235

[52]

Vaswani, A.,
Shazeer, N.,
Parmar, N.,
Uszkoreit, J.,
Jones, L.,
Gomez, A.N.,
Kaiser, Ł.,
Polosukhin, I.:
Attention is all you need.
In: Advances in Neural Information Processing Systems,
pp. 5998–6008

[53]

Everingham, M.,
Van Gool, L.,
Williams, C.K.,
Winn, J.,
Zisserman, A.:
The pascal visual object classes (voc) challenge.
International journal of computer vision

[54]

Lin, T.-Y.,
Maire, M.,
Belongie, S.,
Hays, J.,
Perona, P.,
Ramanan, D.,
Dollár, P.,
Zitnick, C.L.:
Microsoft coco: Common objects in context.
In: European Conference on Computer Vision,
pp. 740–755

#1ISBN #1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1<https://doi.org/#1>et al.#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1<><#>1#1#1#1#1#1#1#1#1#1#1#1#1#1PreBibitemsHook

[1]

Ren, S.,
He, K.,
Girshick, R.,
Sun, J.:
Faster r-cnn: Towards real-time object detection with region proposal
Advances in neural information processing systems

[2]

Redmon, J.,
Divvala, S.,
Girshick, R.,
Farhadi, A.:
You only look once: Unified, real-time object detection.
In: Proceedings of the IEEE Conference on Computer Vision and Pattern
pp. 779–788

[3]

Kirkpatrick, J.,
Pascanu, R.,
Rabinowitz, N.,
Veness, J.,
Desjardins, G.,
Rusu, A.A.,
Milan, K.,
Quan, J.,
Ramalho, T.,
Grabska-Barwinska, A., :
Overcoming catastrophic forgetting in neural networks.
Proceedings of the national academy of sciences

[4]

Kang, B.,
Liu, Z.,
Wang, X.,
Yu, F.,
Feng, J.,
Darrell, T.:
Few-shot object detection via feature reweighting.
In: Proceedings of the IEEE/CVF International Conference on Computer
pp. 8420–8429

[5]

Li, X.,
Deng, J.,
Fang, Y.:
Few-shot object detection on remote sensing images.
IEEE Transactions on Geoscience and Remote Sensing,

[6]

Fan, Q.,
Zhuo, W.,
Tang, C.-K.,
Tai, Y.-W.:
Few-shot object detection with attention-rpn and multi-relation
In: Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition,
pp. 4013–4022

[7]

Hsieh, T.-I.,
Lo, Y.-C.,
Chen, H.-T.,
Liu, T.-L.:
One-shot object detection with co-attention and co-excitation.
In: Advances in Neural Information Processing Systems 32,

[8]

Li, X.,
Zhang, L.,
Chen, Y.P.,
Tai, Y.-W.,
Tang, C.-K.:
One-shot object detection without fine-tuning.
arXiv preprint arXiv:2005.03819

[9]

Karlinsky, L.,
Shtok, J.,
Harary, S.,
Schwartz, E.,
Aides, A.,
Feris, R.,
Giryes, R.,
Bronstein, A.M.:
Repmet: Representative-based metric learning for classification and
few-shot object detection.
In: Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition,
pp. 5197–5206

[10]

Le Jeune, P.,
Lebbah, M.,
Mokraoui, A.,
Azzag, H.:
Experience feedback using representation learning for few-shot object detection
on aerial images.
arXiv preprint arXiv:2109.13027

[11]

Sun, B.,
Li, B.,
Cai, S.,
Yuan, Y.,
Zhang, C.:
Fsce: Few-shot object detection via contrastive proposal encoding.
In: Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition,
pp. 7352–7362

[12]

Huang, G.,
Laradji, I.,
Vazquez, D.,
Lacoste-Julien, S.,
Rodriguez, P.:
A survey of self-supervised and few-shot object detection.
arXiv preprint arXiv:2110.14711

[13]

Jiaxu, L.,
Taiyue, C.,
Xinbo, G.,
Yongtao, Y.,
Ye, W.,
Feng, G.,
Yue, W.:
A Comparative Review of Recent Few-Shot Object Detection Algorithms

[14]

Girshick, R.,
Donahue, J.,
Darrell, T.,
Malik, J.:
Rich feature hierarchies for accurate object detection and semantic
In: Proceedings of the IEEE Conference on Computer Vision and Pattern
pp. 580–587

[15]

Girshick, R.:
Fast r-cnn.
In: Proceedings of the IEEE International Conference on Computer
pp. 1440–1448

[16]

Lin, T.-Y.,
Goyal, P.,
Girshick, R.,
He, K.,
Dollár, P.:
Focal loss for dense object detection.
In: Proceedings of the IEEE International Conference on Computer
pp. 2980–2988

[17]

Duan, K.,
Bai, S.,
Xie, L.,
Qi, H.,
Huang, Q.,
Tian, Q.:
Centernet: Keypoint triplets for object detection.
In: Proceedings of the IEEE/CVF International Conference on Computer
pp. 6569–6578

[18]

Law, H.,
Deng, J.:
Cornernet: Detecting objects as paired keypoints.
In: Proceedings of the European Conference on Computer Vision (ECCV),
pp. 734–750

[19]

Tian, Z.,
Shen, C.,
Chen, H.,
He, T.:
Fcos: Fully convolutional one-stage object detection.
In: Proceedings of the IEEE/CVF International Conference on Computer
pp. 9627–9636

[20]

Lin, T.-Y.,
Dollár, P.,
Girshick, R.,
He, K.,
Hariharan, B.,
Belongie, S.:
Feature pyramid networks for object detection.
In: Proceedings of the IEEE Conference on Computer Vision and Pattern
pp. 2117–2125

[21]

He, K.,
Gkioxari, G.,
Dollár, P.,
Girshick, R.:
Mask r-cnn.
In: Proceedings of the IEEE International Conference on Computer
pp. 2961–2969

[22]

Dai, X.,
Chen, Y.,
Xiao, B.,
Chen, D.,
Liu, M.,
Yuan, L.,
Zhang, L.:
Dynamic head: Unifying object detection heads with attentions.
In: Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition,
pp. 7373–7382

[23]

Carion, N.,
Massa, F.,
Synnaeve, G.,
Usunier, N.,
Kirillov, A.,
Zagoruyko, S.:
End-to-end object detection with transformers.
In: European Conference on Computer Vision,
pp. 213–229

[24]

Wang, Y.,
Yao, Q.,
Kwok, J.T.,
Ni, L.M.:
Generalizing from a few examples: A survey on few-shot learning.
ACM Computing Surveys (CSUR)

[25]

Larochelle, S.R.H.:
Optimization as a model for few-shot learning.
In: 5th International Conference on Learning Representations, ICLR
2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings

[26]

Finn, C.,
Abbeel, P.,
Levine, S.:
Model-agnostic meta-learning for fast adaptation of deep networks.
In: International Conference on Machine Learning,
pp. 1126–1135

[27]

Snell, J.,
Swersky, K.,
Zemel, R.:
Prototypical networks for few-shot learning.
In: Guyon, I.,
Luxburg, U.V.,
Bengio, S.,
Wallach, H.,
Fergus, R.,
Vishwanathan, S.,
Garnett, R. (eds.)
Advances in Neural Information Processing Systems,
vol. 30

[28]

Vinyals, O.,
Blundell, C.,
Lillicrap, T.,
kavukcuoglu, k.,
Wierstra, D.:
Matching networks for one shot learning.
In: Lee, D.,
Sugiyama, M.,
Luxburg, U.,
Guyon, I.,
Garnett, R. (eds.)
Advances in Neural Information Processing Systems,
vol. 29

[29]

Sung, F.,
Yang, Y.,
Zhang, L.,
Xiang, T.,
Torr, P.H.,
Hospedales, T.M.:
Learning to compare: Relation network for few-shot learning.
In: Proceedings of the IEEE Conference on Computer Vision and Pattern
pp. 1199–1208

[30]

Bertinetto, L.,
Henriques, J.F.,
Valmadre, J.,
Torr, P.,
Vedaldi, A.:
Learning feed-forward one-shot learners.
In: Advances in Neural Information Processing Systems,
pp. 523–531

[31]

Hu, J.,
Shen, L.,
Sun, G.:
Squeeze-and-excitation networks.
In: Proceedings of the IEEE Conference on Computer Vision and Pattern
pp. 7132–7141

[32]

Wang, X.,
Girshick, R.,
Gupta, A.,
He, K.:
Non-local neural networks.
In: Proceedings of the IEEE Conference on Computer Vision and Pattern
pp. 7794–7803

[33]

Dosovitskiy, A.,
Beyer, L.,
Kolesnikov, A.,
Weissenborn, D.,
Zhai, X.,
Unterthiner, T.,
Dehghani, M.,
Minderer, M.,
Heigold, G.,
Gelly, S.,
Uszkoreit, J.,
Houlsby, N.:
An image is worth 16x16 words: Transformers for image recognition at
In: International Conference on Learning Representations

[34]

Doersch, C.,
Gupta, A.,
Zisserman, A.:
Crosstransformers: spatially-aware few-shot transfer.
arXiv preprint arXiv:2007.11498

[35]

Chen, H.,
Wang, Y.,
Wang, G.,
Qiao, Y.:
Lstd: A low-shot transfer detector for object detection.
In: Proceedings of the AAAI Conference on Artificial Intelligence,
vol. 32

[36]

Wang, X.,
Huang, T.E.,
Darrell, T.,
Gonzalez, J.E.,
Yu, F.:
Frustratingly simple few-shot object detection.
In: International Conference on Machine Learning (ICML)

[37]

Wu, J.,
Liu, S.,
Huang, D.,
Wang, Y.:
Multi-scale positive sample refinement for few-shot object detection.
In: European Conference on Computer Vision,
pp. 456–472

[38]

Fan, Z.,
Ma, Y.,
Li, Z.,
Sun, J.:
Generalized few-shot object detection without forgetting.
In: Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition,
pp. 4527–4536

[39]

Zhang, G.,
Cui, K.,
Wu, R.,
Lu, S.,
Tian, Y.:
Pnpdet: Efficient few-shot detection without forgetting via
plug-and-play sub-networks.
In: Proceedings of the IEEE/CVF Winter Conference on Applications of
Computer Vision,
pp. 3823–3832

[40]

Wu, A.,
Han, Y.,
Zhu, L.,
Yang, Y.:
Universal-prototype enhancing for few-shot object detection.
In: Proceedings of the IEEE/CVF International Conference on Computer
pp. 9567–9576

[41]

Xiao, Y.,
Marlet, R.:
Few-shot object detection and viewpoint estimation for objects in the
In: European Conference on Computer Vision,
pp. 192–210

[42]

Fan, Q.,
Zhuo, W.,
Tang, C.-K.,
Tai, Y.-W.:
Few-shot object detection with attention-rpn and multi-relation
In: CVPR

[43]

Xiao, Z.,
Qi, J.,
Xue, W.,
Zhong, P.:
Few-shot object detection with self-adaptive attention network for
remote sensing images.
IEEE Journal of Selected Topics in Applied Earth Observations and
Remote Sensing

[44]

Kim, G.,
Jung, H.-G.,
Lee, S.-W.:
Few-shot object detection via knowledge transfer.
In: 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC),
pp. 3564–3569

[45]

Chu, J.,
Feng, J.,
Jing, P.,
Lu, W.:
Joint co-attention and co-reconstruction representation learning for
one-shot object detection.
In: 2021 IEEE International Conference on Image Processing (ICIP),
pp. 2229–2233

[46]

Chen, T.-I.,
Liu, Y.-C.,
Su, H.-T.,
Chang, Y.-C.,
Lin, Y.-H.,
Yeh, J.-F.,
Hsu, W.H.:
Should i look at the head or the tail? dual-awareness attention for few-shot
object detection.
arXiv preprint arXiv:2102.12152

[47]

Han, G.,
Huang, S.,
Ma, J.,
He, Y.,
Chang, S.-F.:
Meta faster r-cnn: Towards accurate few-shot object detection with attentive
feature alignment.
arXiv preprint arXiv:2104.07719

[48]

Zhang, G.,
Luo, Z.,
Cui, K.,
Lu, S.:
Meta-detr: Few-shot object detection via unified image-level meta-learning.
arXiv preprint arXiv:2103.11731

[49]

Xu, H.,
Wang, X.,
Shao, F.,
Duan, B.,
Zhang, P.:
Few-shot object detection via sample processing.
IEEE Access

[50]

Gao, Y.,
Hou, R.,
Gao, Q.,
Hou, Y.:
A fast and accurate few-shot detector for objects with fewer pixels in
drone image.

[51]

Liu, W.,
Li, H.,
Yu, S.,
Chen, S.,
Ye, X.,
Wu, J., et al.:
Dynamic relevance learning for few-shot object detection.
arXiv preprint arXiv:2108.02235

[52]

Vaswani, A.,
Shazeer, N.,
Parmar, N.,
Uszkoreit, J.,
Jones, L.,
Gomez, A.N.,
Kaiser, Ł.,
Polosukhin, I.:
Attention is all you need.
In: Advances in Neural Information Processing Systems,
pp. 5998–6008

[53]

Everingham, M.,
Van Gool, L.,
Williams, C.K.,
Winn, J.,
Zisserman, A.:
The pascal visual object classes (voc) challenge.
International journal of computer vision

[54]

Lin, T.-Y.,
Maire, M.,
Belongie, S.,
Hays, J.,
Perona, P.,
Ramanan, D.,
Dollár, P.,
Zitnick, C.L.:
Microsoft coco: Common objects in context.
In: European Conference on Computer Vision,
pp. 740–755