# A Survey of Deep Learning for Low-Shot Object Detection

Qihan Huang, Haofei Zhang, Jie Song, Mingli Song

###### Abstract

Object detection is a fundamental task in computer vision and image
processing. Current deep learning based object detectors have been highly
successful with abundant labeled data. But in real life, it is not guaranteed
that each object category has enough labeled samples for training. These large
object detectors are easy to overfit when the training data is limited.
Therefore, it is necessary to introduce few-shot learning and zero-shot
learning into object detection, which can be named low-shot object detection
together. Low-Shot Object Detection (LSOD) aims to detect objects from a few
or even zero labeled data, which can be categorized into few-shot object
detection (FSOD) and zero-shot object detection (ZSD), respectively. This
paper conducts a comprehensive survey for deep learning based FSOD and ZSD.
First, this survey classifies methods for FSOD and ZSD into different
categories and discusses the pros and cons of them. Second, this survey
reviews dataset settings and evaluation metrics for FSOD and ZSD, then
analyzes the performance of different methods on these benchmarks. Finally,
this survey discusses future challenges and promising directions for FSOD and
ZSD.

###### Index Terms:

Few-shot object detection, Zero-shot object detection, Transfer-learning,
Meta-learning.

##  I Introduction

Object detection is a fundamental and challenging task in computer vision,
which aims to localize objects of certain categories in images. Object
detection has been widely applied on many computer vision tasks like object
tracking[88], image annotation[67], scene graph generation[72] and so on.

The general process of object detection is to predict categories for a set of
bounding boxes (imaginary rectangles for reference in the image). Most of the
traditional methods generate these bounding boxes brutally by sliding a window
through the whole image. Viola-Jones (VJ) detector[74] firstly achieves real-
time detection of human faces with three speed-up techniques: integral image,
feature selection and detection cascades. Later, histogram of oriented
gradients (HOG)[12] is proposed and is adopted for feature description by many
object detectors. Among these methods, deformable part-based model (DPM)[16]
is a representative work. DPM divides an object detection task into several
fine-grained detection tasks, then uses a number of part-filters to detect
object parts and aggregate them for final prediction. Although people have
made many improvements, traditional methods are restricted by their slow speed
and low accuracy.

Compared with traditional methods, deep methods have the capacity to achieve
stronger performance with more effective image features. R-CNN[21] is one of
the most representative work of deep learning based methods. It uses selective
search to obtain region proposals (i.e. image regions which are more likely to
contain objects), then extracts their features with a pre-trained CNN model
for further classification and localization. Fast R-CNN[20] improves R-CNN by
using a region of interest (RoI) pooling layer to generate feature-maps for
region proposals from the whole feature-map of an image. Faster R-CNN[64]
further proposes a region proposal network (RPN) instead of traditional
methods to generate region proposals from the whole image feature-map with
anchors (i.e. predefined bounding boxes of certain height and width). R-CNN,
Fast R-CNN and Faster R-CNN are classified into two-stage models since they
firstly generate region proposals and secondly make predictions. For further
speed-up, single-stage models combine these two processes together. YOLO-style
object detectors[63][3][19] are the representative models of single-stage
detection framework. Given the feature-map extracted from the original image,
YOLO-style detectors directly pre-defined anchors with multiple scales over
all locations of the image and predicts the class probability, location offset
and object confidence of each anchor. Single-stage detectors generally
underperform two-stage detectors, and some methods like focal loss[47] have
been proposed to decrease the performance gap between single-stage and two-
stage detectors. Recently, a transformer based detector named DETR[4] has been
proposed. DETR directly forwards a fixed number of object queries into the
transformer and make predictions on them. DETR achieves end-to-end detection
and it has comparable performance with many classic detectors. To mitigate the
slow convergence problem of DETR, deformable DETR[103] proposes a deformable
attention module which accesses only a small set of key sampling points around
a reference and UP-DETR[11] proposes an unsupervised task named random query
patch detection to pre-train DETR.

However, these object detectors depend on large training data. If the training
data are scarce, current object detectors will easily overfit and thus can not
generalize to test data. In real life, it is hard to collect sufficient
labeled data for some object categories due to the scarcity of these
categories or special labeling cost. Therefore, the ability of detecting
objects from a few or even zero labeled data is desired for modern object
detectors and few-shot & zero-shot learning is introduced into object
detection.

Figure 1: Overview of this survey. This survey gives an overall introduction
to low-shot object detection (LSOD), then categories LSOD into three domains:
one-shot object detection (OSOD), few-shot object detection (FSOD) and zero-
shot object detection (ZSD). The more fine-grained categorization of these
three domains are also demonstrated in the figure with three colors, which
will be discussed in detail in later sections. Then the benchmarks for OSOD,
FS and ZSD are summarized and the performances of different LSOD methods on
these benchmarks are compared and analyzed. Finally the future directions of
LSOD are discussed.

Few-shot learning aims to learn a model generalizing from a few labeled
samples. In the past several years, numerous methods are proposed for few-shot
learning, which can be roughly categorized into data augmentation methods,
meta-learning methods and transfer-learning methods. Data augmentation methods
tackle the data scarcity problem directly by generating new samples using
traditional image transformation methods or deep learning based methods like
GAN. Meta-learning methods adopt a learning-to-learn mechanism, which defines
multiple few-shot tasks on another data abundant dataset to train the model
and in this way the model is enabled to quickly adapt to current few-shot
tasks. Instead of training with multiple episodes in meta-learning, transfer-
learning methods aim to learn good image representations directly on another
dataset and transfer them to current dataset. Work[73] investigates some
baseline transfer-learning methods for few-shot image classification and the
experiments demonstrate that they outperform many classic meta-learning
methods.

Previous few-shot learning methods are mainly explored on the task of image
classification. Few-shot object detection (FSOD) is a more challenging task
than few-shot image classification since object detection requires an extra
task to locate the objects. Nevertheless, FSOD methods inherit main ideas of
FSL methods, which can be also categorized into two mainstream methods: meta-
learning methods and transfer-learning methods. The early FSOD methods follow
the meta-learning methods of few-shot learning, which mainly focus on how to
combine the features of support images and query images. These meta-learning
methods can be categoried into feature vector aggregation methods and feature-
map aggregation methods. The transfer-learning based methods are also brought
into few-shot object detection, which pre-trains the detector on another
dataset and then fine-tunes it on current few-shot dataset. Contrastive
learning, metric learning, regularization methods and so on are introduced
into transfer-learning based methods. Besides the former standard FSOD
setting, other settings like semi-supervised FSOD, weakly-supervised FSOD and
incremental FSOD are also explored by researchers.

There still exists a more challenging setting that each few-shot class has
zero labeled samples, which is called zero-shot learning (ZSD). In zero-shot
learning, each class has its semantic attributes as compensation. Actually
FSOD and ZSD can be considered in a condition, some methods[60][33] propose a
combination model to deal with FSOD and ZSD simultaneously. This survey
therefore denotes FSOD and ZSD as low-shot object detection (LSD). The
hierarchically-structured taxonomy of low-shot object detection is illustrated
in Figure 1.

This survey provides a comprehensive review and analysis of current few-shot &
zero-shot object detection algorithms. This survey summarizes the performance
of current few-shot & zero-shot object detection algorithms. This survey
discusses the main challenges and promising future directions for few-shot &
zero-shot object detection. The remainder of this survey is organized as
follows. The more fine-grained categorization and analysis of algorithms for
few-shot & zero-shot object detection is described in Sect. II. Popular
datasets and evaluation criteria of few-shot & zero-shot object detection are
described in Sect. VIII. The performances of current few-shot & zero-shot
object detection algorithms are summarized in Sect. IX. Finally, Sect. XI
concludes the contents of this survey, then discusses the main challenges of
current methods and promising future directions of few-shot & zero-shot object
detection. The key notations used are summarized in Tab. I.

TABLE I: Key Notations Used in This Article Notation |  Description
---|---
$\phi_{q}$ | Feature-map of integral query image $q$
$\phi_{c}$ | Feature-map of integral support image for class $c$
$\phi_{\rm fused}$ | The aggregated feature-map of $\phi_{q}$ and $\phi_{c}$
$\phi_{q}^{i}$ | The $i$-th RoI feature-map of query image $q$
$v_{q}^{i}$ | The $i$-th RoI feature vector of query image $q$
$s_{q}^{i}$ | The $i$-th RoI semantic embedding of query image $q$
$v_{c}$ | The pooled embedding vector of class $c$
$v_{\rm fused}$ | The aggregated feature vector of $v_{q}^{i}$ and $v_{c}$
$s_{c}$ | The semantic embedding of class $c$
$[\cdot]$ | Concatenation operation
${\rm Pool}$ | Pool operation
$\oplus$ | Element-wise sum
$\otimes$ | Channel-wise multiplication
${\rm Conv2d}$ | 2D convolution operation
${\rm FC}$ | FC layer
${\rm softmax}$ | softmax operation
$\sigma(\cdot)$ | sigmoid function

##  II One-Shot Object Detection

One-shot object detection (OSOD) requires to detect objects from a query image
according to only one support image of the same category. Some previous object
tracking methods like SiamFC[6], SiamRPN[37] are forerunners of OSOD and they
are used for comparison with early OSOD methods. Current OSOD methods adopt
the scheme of meta-learning, which defines a large number of one-shot subtasks
on base dataset to train the model. Each of these one-shot subtasks contains a
query image and a randomly selected support image (The category of the support
image is guaranteed to exist in the query image). It is worth noting that
almost all of current OSOD methods adopt Faster R-CNN as detection framework.
The framework of these methods are demonstrated in Figure 2. They first
extract integral feature-maps of query image and support image using the
siamese convolutional feature extractor of Faster R-CNN, then carry out
integral feature aggregation to generate a fused feature-map by aggregating
query feature-map with support feature-map. This fused feature-map is fed into
RPN and RoI layer to generate category-specific region proposals and RoI
features, respectively. Finally these RoI features are used for final
classification task and localization task. Some methods additionally apply RoI
feature aggregation to aggregate RoI features with support feature before
classification task and localization task.

Figure 2: The overall framework of one-shot object detection methods, which is
based on Faster R-CNN framework. The model takes a query image and a support
image as inputs and uses a siamese convolutional feature extractor to extract
the query feature-map and support feature-map. Then it applies integral
feature aggregation to aggregate these two feature-maps into a fused feature-
map and forwards it into RPN and RoI layer to generate region proposals and
RoI features, respectively. The aggregation way is implemented differently in
different OSOD methods. Finally the RoI features are used for classification
task and localization task. Some methods additionally apply RoI feature
aggregation to aggregate RoI features with support feature.

SiamMask[53] is one of the early deep learning based methods for OSOD. It
adopts a simple concatenation based method for integral feature aggregation of
query feature-map $\phi_{q}\in\mathbb{R}^{C\times H_{q}\times H_{q}}$ and
support feature-map $\phi_{c}\in\mathbb{R}^{C\times H_{s}\times H_{s}}$. It
concatenates $\phi_{q}$ with the absolute difference between $\phi_{q}$ and
the pooled embedding vector $v_{c}$ of $\phi_{c}$ to generate the fused
feature-map $\phi_{\rm fused}$:

$\phi_{\rm fused}=[\phi_{q},|\phi_{q}-{\rm
Pool}(\phi_{c})|]\in\mathbb{R}^{2C\times H_{q}\times H_{q}}\text{.}$ (1)

In this method, $\phi_{\rm fused}$ is directly used for further components
(RPN, RoI layer) in Faster R-CNN without other modifications. SiamMask doesn’t
achieve satisfactory performance since it tackles a segmentation task
simultaneous. Nevertheless, as the first method for deep learning based OSOD,
SiamMask proposes MS COCO benchmark for performance comparison and pioneers
many future works on OSOD and sets up a baseline for future work.

OSCD[17] is another early deep learning based method for OSOD, which is
similar with SiamMask. It concatenates $\phi_{q}\in\mathbb{R}^{C\times
H_{q}\times H_{q}}$ with the pooled embedding vector $v_{c}$ of
$\phi_{c}\in\mathbb{R}^{C\times H_{s}\times H_{s}}$ to generate the fused
feature-map $\phi_{\rm fused}$ and forwards $\phi_{\rm fused}$ into RPN to
generate region proposals:

$\phi_{\rm fused}=[\phi_{q},{\rm Pool}(\phi_{c})]\in\mathbb{R}^{2C\times
H_{q}\times H_{q}}\text{.}$ (2)

Different from SiamMask, OSCD continues to apply RoI feature aggregation by
resizing RoI feature $\phi_{q}^{i}$ and each support feature $\phi_{c}$ to the
same shape and concatenating them in depth. Then the aggregated RoI feature-
map is used for classification and localization. OSCD adopts PASCAL VOC
benchmark for evaluation and it outperforms SiamFC and SiamRPN by a large
margin. OSODwT[42] extends this RoI feature aggregation method by applying a
series of pixel-wise convolution blocks on the concatenated features, which
models the relation between feature pixels and boosts the performance.

FOC OSOD[85] also uses convolution blocks to extend the integral feature
aggregation method in SiamMask:

$\phi_{\rm fused}={\rm Conv2d}([\phi_{q},|\phi_{q}-{\rm
Pool}(\phi_{c})|])\in\mathbb{R}^{\frac{3}{2}C\times H_{q}\times
H_{q}}\text{.}$ (3)

FOC OSOD proposes two modules to further improve the performance, including a
CFDA module and a SIH module. The CFDA module aims to extract more powerful
features through a feature deformation operation and a feature attention
operation. The SIH module splits support features for classification head and
localization head, and iterates the classification head one more time.

The above methods (SiamMask, OSCD, OSODwT, FOC OSOD) are categoried into
concatenation-based aggregation methods and the limitation of them is that
they simply aggregate features without considering relatedness between
different parts of two features. However, only some object parts of the query
image have high response to the support image. Attention mechanism tackles
this problem by modeling the correspondence between different parts of support
features and query features. The OSOD methods utilizing attention mechanism
into feature aggregation are categoried into attention-based methods in this
survey.

CoAE[26] is the first attention-based OSOD method. It proposes two aggregation
operations: co-attention operation and co-excitation operation. The co-
attention operation is implemented using the non-local operation[76], which
aggregates two feature-maps according to their element-wise affinity:

$\phi_{\rm fused}=\phi_{q}\oplus\psi(\phi_{q},\phi_{c})\in\mathbb{R}^{C\times
H_{q}\times H_{q}}\text{,}$ (4)

where $\psi$ denotes the non-local operation. The co-excitation operation
aggregates $\phi_{q}$ with the pooled embedding vector $v_{c}$ of $\phi_{c}$
with a channel-wise multiplication:

$\phi_{\rm fused}=\phi_{q}\otimes{\rm Pool}(\phi_{c})\in\mathbb{R}^{C\times
H_{q}\times H_{q}}\text{.}$ (5)

CoAE uses co-attention and co-excitation only for integral feature
aggregation. Besides, CoAE proposes a proposal ranking loss to supervise RPN
based on RoI feature aggregation. CoAE outperforms SiamMask on MS COCO
benchmark and OSCD on PASCAL VOC benchmark, and the attention mechanism for
feature aggregation is also inherited by later methods.

With the wide usage of Transformers[69] in computer vision, some methods like
AIT[7], CAT[48] introduce multi-head attention into OSOD for feature
aggregation. AIT and CAT flatten query feature-map $\phi_{q}$ and support
feature-map $\phi_{c}$ to be feature sequences
$\phi_{q}^{{}^{\prime}}\in\mathbb{R}^{C\times H_{q}H_{q}}$ and
$\phi_{c}^{{}^{\prime}}\in\mathbb{R}^{C\times H_{q}H_{q}}$, then uses multi-
head attention to capture bidirectional correspondence between grids of them:

$\phi_{\rm fused}={\rm
MultiHead}(\phi_{q}^{{}^{\prime}},\phi_{c}^{{}^{\prime}},\phi_{c}^{{}^{\prime}})\in\mathbb{R}^{C\times
H_{q}\times H_{q}}\text{,}$ (6)

where ${\rm Multihead}$ denotes multi-head attention:

$\displaystyle{\rm Multihead}(Q,K,V)=[{\rm head}_{1},\cdots,{\rm
head}_{M}]W^{O}\text{,}$ (7) $\displaystyle{\rm
head}_{i}=f(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V})\text{,}$ (8) $\displaystyle
f(Q,K,V)={\rm softmax}(\frac{QK^{\top}}{\sqrt{d_{k}}}V)\text{,}$ (9)

where $M$ denotes number of heads, $d_{k}$ denotes dimension of keys,
$W_{i}^{Q},W_{i}^{K},W_{i}^{V}$ denote projection matrices. For RoI feature
aggregation, CAT simply concatenates RoI feature vector $v_{q}^{i}$ with the
pooled embedding vector of support feature for classification task.
Differently, AIT uses a transformer encoder to encode RoI features and
forwards them together with support features into a transformer decoder to
obtain the aggregated features. AIT also proposes selective parallel attention
and selective channel attention to reweight outputs from each head and each
channel of features, respectively. The experiments on both MS COCO benchmark
and PASCAL VOC benchmark demonstrate the effectiveness of attention mechanism
in feature aggregation.

Different from above methods which are based on a common framework, OS2D[55]
proposes a dense correlation matching method for OSOD. Given the query
feature-map $\phi_{q}$ and the support feature-map $\phi_{c}$, OS2D first
computes a 4D correlation matrix of shape $\mathbb{R}^{H_{q}\times H_{q}\times
H_{s}\times H_{s}}$ which models the correspondence between all pairs of
locations from these two feature-maps. Then it uses a pre-trained
TransformNet[65] to generate a transformation matrix which can spatially align
the support image with the query image. The classification score of each
location of the support feature-map is obtained by resampling the correlation
matrix with the transformation matrix. With so many differences from other
OSOD methods, OS2D also proposes another OSOD benchmark for evaluation and it
is hard to directly compare it with other OSOD methods as a result.

##  III Standard Few-Shot Object Detection

The previous one-shot object detection setting has a guarantee that every
query image contains objects with the same category as support image during
test time. In other words, the model knows exactly the object categories
contained in each test image. However, this setting is not realistic in real
world and another more challenging low-shot object detection setting is
adopted by more papers, which is named standard few-shot object detection
(standard FSOD) in this survey. Standard FSOD doesn’t have prior information
about object categories in test images and the shot number of each novel
category can be larger than one. Therefore, methods for standard FSOD needs to
balance the performance between one shot-num situation and multiple shot-num
situation.

Early standard FSOD methods adopt the scheme of meta-learning and concentrate
on feature aggregation as OSOD methods. Further more, the increased shot-num
opens up possibility for FSOD methods to adopt the scheme of transfer-
learning, which pre-trains an object detector on base dataset and fine-tunes
this pre-trained model for novel classes on novel dataset. Early transfer-
learning methods like LSTD[8] are outperformed by the meta-learning methods in
that period until the emergence of TFA[77]. In this survey, the meta-learning
methods are first introduced since they are highly correlated to OSOD methods,
then transfer-learning methods are discussed subsequently.

###  III-A Meta-Learning Methods

Similar with OSOD, the meta-learning methods for standard FSOD defines a large
number of few-shot detection tasks on base dataset to train the model.
Different from OSOD methods, each of these few-shot subtasks contains a query
image and a number of support images (single or multiple samples for each
category) since the object categories in the query image is not pre-known.
Another difference is that most meta-learning methods for standard FSOD has an
additional fine-tuning stage to fine-tune the model on novel dataset, which
adopts the similar meta-learning setting as on base dataset. Meta-learning
methods for standard FSOD also concentrates on RoI feature aggregation and
integral feature aggregation as discussed in OSOD methods.

####  III-A1 RoI Feature Aggregation Methods

RoI feature aggregation methods aggregate the RoI feature with each support
feature to generate class-specific RoI feature for classification task and
regression task. Unlike OSOD methods which almost all adopt Faster R-CNN as
detection framework, early meta-learning methods explore RoI feature
aggregation methods on both single-stage detectors and two-stage detectors.
The summary of feature vector aggregation methods are listed in Table II.

TABLE II: Summary of RoI Feature Aggregation Methods Method | Detection Framework | Aggregation Method | Use Dual-Way Aggregation
---|---|---|---
FSRW[29] | YOLOv2 | $z_{i}\otimes w_{c}$ | ✗
Meta R-CNN[84] | Faster R-CNN | $z_{i}\otimes w_{c}$ | ✗
SiamMask[53] | Mask R-CNN | $[z_{i},|z_{i}-w_{c}|]$ | ✗
FsDet[81] | Faster R-CNN | $[z_{i}\otimes w_{c},z_{i}-w_{c},z_{i}]$ | ✗
AFD-Net[49] | Faster R-CNN | $[{\rm FC}(z_{i}\otimes w_{c}),{\rm FC}(z_{i}-w_{c}),z_{i}]$ | ✓

FSRW[29] is the first meta-learning method for standard FSOD. FSRW adopts
YOLOv2 as detection framework and it uses a light-weight CNN as a reweighting
module. This reweighting module takes support image with their bounding box
annotations for each class $c$ as input and embeds it into a feature vector
$v_{c}\in\mathbb{R}^{M}$. $v_{c}$ is used to reweight each feature vector
$v_{q}^{i}\in\mathbb{R}^{M}$ at each cell of the query feature-map, which aims
to highlight the important features corresponding to class $c$. The
reweighting function is simple channel-wise multiplication:

$v_{\rm fused}=v_{q}^{i}\otimes v_{c}\text{.}\in\mathbb{R}^{M}\text{.}$

The obtained class-specific feature vector $v_{\rm fused}$ is used to predict
the object confidence score, classification probability and location offset as
in original YOLOv2.

Meta R-CNN[84] proposes feature aggregation method similar with FSRW. The
major difference is that this work adopts Faster R-CNN as detection framework
instead of YOLOv2. It designs a predictor-head remodeling network (PRN) to
encode support image with bounding box annotations for each class $c$ into a
feature vector $v_{c}\in\mathbb{R}^{M}$. Then channel-wise multiplication is
also adopted to reweight RoI feature vector $v_{q}^{i}$ of each region
proposal generated from RPN:

$v_{\rm fused}=v_{q}^{i}\otimes v_{c}\text{.}\in\mathbb{R}^{M}\text{.}$

$v_{\rm fused}$ is used for further classification and regression. This work
also introduces a meta-loss, which is a cross-entropy loss for classification
of the class feature vectors. With the similar feature aggregation method and
fine-tuning strategy, Meta R-CNN outperforms FSRW on both MS COCO benchmark
and PASCAL VOC benchmark. This may imply that two-stage detection framework is
more suitable for few-shot object detection than one-stage YOLO-style
framework. Actually, later meta-learning methods for FSOD mostly adopt Faster
R-CNN as framework.

FsDet[81] is an extension of Meta R-CNN[84]. The main difference is that FsDet
uses a more complex feature aggregation method. Given the RoI feature vector
$v_{q}^{i}$ and the class feature vector $v_{c}$, the aggregated feature
vector is the concatenation of their channel-wise multiplication, subtraction
and $z_{i}$:

$v_{\rm fused}=[v_{q}^{i}\otimes
v_{c},v_{q}^{i}-v_{c},v_{q}^{i}]\in\mathbb{R}^{3M}\text{.}$

With this extended aggregation method, FsDet outperforms Meta R-CNN on both MS
COCO benchmark and PASCAL VOC benchmark.

AFD-Net[49] is also built on two-stage detection framework, which is a more
complex extension of above methods. With the assumption that classification
task and localization task of object detection requires different image
features, AFD-Net separates RoI feature extraction into classification branch
and localization branch. The outputs of these branches are two different
weighted concatenation of output features from convolution layer and FC layer.
Given the RoI feature vector $v_{q}^{i}$ and the class feature vector $v_{c}$,
AFD-Net aggregates them with two additional FC layers beyond FsDet:

$v_{\rm fused}=[{\rm FC}(v_{q}^{i}\otimes v_{c}),{\rm
FC}(v_{q}^{i}-v_{c}),v_{q}^{i}]\in\mathbb{R}^{3M}$

Different from above methods which all focus on feature vector aggregation,
Meta Faster R-CNN[24] aggregates the RoI feature-map $\phi_{q}^{i}$ and
support feature-map $\phi_{c}$ based on attention mechanism like later OSOD
methods. Given the RoI feature-map $\phi_{q}^{i}$ and support feature-map
$\phi_{c}$ with the same shape $\mathbb{R}^{C\times H\times W}$, it first
computes the affinity matrix $A\in\mathbb{R}^{HW\times HW}$, where $A_{ij}$
denotes the feature similarity between the $i$-th location of $\phi_{q}^{i}$
and the $j$-th location of $\phi_{c}$. This method then performs softmax
normalization on $A$ and uses it as weight to accumulate support feature-map
into RoI feature-map. Besides, $A$ is also used to compute a foreground
attention mask to highlight foreground parts of the image. The highlighted
feature-maps are finally aggregated similar with FsDet.

DAnA-FasterRCNN[9] proposes a RoI feature-map aggregation method similar with
Meta Faster R-CNN. The major difference is that DAnA-FasterRCNN computes the
pixel-wise similarity between the standardized features with extra importance
values at each location of the feature-map.

####  III-A2 Integral Feature Aggregation Methods

Current integral feature aggregation methods for FSOD almost all adopt Faster
R-CNN as detection framework, which aggregate query feature-map and support
feature-map for subsequent RPN and RoI layer. The summary of integral feature
aggregation methods are listed in Table. III.

TABLE III: Summary of Integral Feature Aggregation Methods Method | Detection Framework | Aggregation Method
---|---|---
CoAE[26] | Faster R-CNN | co-excitation
Attention-RPN[15] | Faster R-CNN | multi-relation combination
OSCD[17] | Faster R-CNN | simple concatenation
CAT[48] | Faster R-CNN | attention based
DCNet[27] | Faster R-CNN | attention based
Meta Faster R-CNN[24] | Faster R-CNN | attention based
DAnA-FasterRCNN[9] | Faster R-CNN | attention based

AttentionRPN[15] aggregates two feature-maps by using the support feature-map
as a kernel and sliding it across the query feature-map to compute
similarities at each location:

$\phi_{\rm fused}=\phi_{q}\ast\phi_{c}\text{,}$

where $\ast$ denotes the aggregation operation. With $X\in\mathbb{R}^{C\times
H_{q}\times H_{q}}$ and $Y\in\mathbb{R}^{C\times H_{s}\times H_{s}}$, the
$\ast$ operation is formulated as below:

$(X\ast Y)_{c,h,w}\;=\;\sum\limits_{i,j}X_{c,h+i-1,w+j-1}\cdot
Y_{c,i,j}\text{,}$ (10)

where $i,j\in\\{1,\cdots,H_{s}\\}$.

PNSD[92] embeds Second-Order Pooling (SOP) and Power Normalization (PN)[35]
into AttentionRPN. SOP extracts second-order statistics of data features but
it introduces a nuisance variability, and PN is added to reduce this nuisance
variability. PNSD uses SOP and PN to remodel support feature-map and query
feature-map.

Similar with early feature aggregation methods in OSOD, the above methods
aggregates feature-maps equally without considering the correspondence
difference between different parts of two feature-maps. To tackle this
problem, subsequent methods also introduce attention mechanism into standard
FSOD as current OSOD methods.

DCNet[27] aggregates two feature-maps using the similar manner as Scaled Dot-
Product Attention[69]. It extracts key map and value map from query image and
support image, then calculates the pixel-wise similarities between these two
key maps and uses them to aggregate two value maps. This paper also proposes a
CFA module, which aggregates RoI feature-maps with different resolutions of a
region proposal to capture both holistic and partial information of it. DAnA-
FasterRCNN also applies its attention based RoI feature-map aggregation method
to integral feature aggregation.

###  III-B Other Methods

There are some other meta-learning methods which focus on other issues but not
the aggregation method of features.

Weight Prediction. MetaDet[78] proposes a meta-learning based method which
learns to predict partial weights of the network. MetaDet also adopts Faster
R-CNN as detection framework and separates its components into two types:
category-agnostic components and category-specific components. This paper
assumes that the feature extractor, RPN and other bottom layer of Faster R-CNN
are category-agnostic components, which can be shared by both base and novel
classes. Then it proposes a weight prediction module to predict the weights of
category-specific components: bounding box classification layer and regression
layer. This method first pre-trains Faster R-CNN on base dataset to obtain the
parameters of category agnostic components. Then it freezes these category-
agnostic parameters and meta-trains the weight prediction module. Finally it
uses this module to predict category-specific parameters for novel classes and
fine-tunes the model on novel dataset.

Meta-RetinaNet[41] proposes another meta-learning method for weight
prediction. This work adopts RetinaNet instead of Faster R-CNN as detection
framework and it pre-trains the model on base dataset as MetaDet. Let
$w_{e_{i}}^{p},b_{e_{i}}^{p}$ denotes the pre-trained parameters of the $i$-th
layer of encoder, then the predicted parameters of this layer $\theta_{e_{i}}$
is as below:

$w_{e_{i}}=w_{e_{i}}^{p}\otimes\bm{\omega}_{i}\text{,}$ (11)

where $\bm{\omega}_{i}$ denotes the coefficient vector of the $i$-th layer.
This method freezes $w_{e_{i}}^{p}$ of each layer and uses meta-learning to
learn the adaptive and stable parameters of $\bm{\omega}_{i}$.

Graph Convolution Network. FSOD-KT[34] introduces GCN into the previous work
Meta R-CNN to incorporate knowledge of semantic information into class feature
vectors. It constructs a meta-graph where each node corresponds to feature
vector of each category and each edge represents the correlation between two
nodes. This work constructs edges of this meta-graph by calculating the cosine
similarity between word embeddings of each pair of categories. With the
constructed meta-graph, this work uses a GCN to generate augmented class
feature vectors for further RoI feature aggregation.

DETR. Meta-DETR[90] is a meta-learning method which adopts DETR as detection
framework. Meta-DETR first uses a query encoding branch (QEB) and a support
encoding branch (SEB) to extract the features of query images and support
images, respectively. These two modules adopt the same structure, which
consists of a feature extractor and a transformer encoder following the design
of DETR[5]. The feature extractor generates the feature-map of input image. It
injects positional information into this feature-map, then collapses its
spatial dimensions into one dimension and then feeds it into a transformer
encoder. The outputs of QEB and SEB are aggregated following FsDet into a set
of class-specific features. These features are fed into a transformer decoder
for further detection in the same way as DETR. This paper also introduces a
semantic alignment mechanism (SAM) to avoid reliance on high-level category
specific features.

Consistency Regularization. TIP[36] extends Meta R-CNN by adding two
consistency regularization to support branch and query branch, respectively.
For support branch, this paper assumes that the class representative features
should be invariant to image transformation and the consistency loss for
support branch is the difference of original support features and the
transformed support features. And for query branch, TIP generates region
proposals from transformed images but extract the RoI features from original
images.

###  III-C Transfer-Learning Methods

Transfer-learning methods regard few-shot object detection as a transfer-
learning problem in which the source domain is the base dataset (i.e.,
abundant images which only consist of objects from base categories) and the
target domain is the novel dataset (i.e., few images which only consist of
objects from novel categories). Current transfer-learning methods mostly adopt
Faster R-CNN as detection framework and they have two stages: base training
and few-shot fine-tuning. The base training stage is to train an object
detector on base dataset. After this stage the object detector will obtain a
relatively effective image feature extractor and have a good performance on
detecting base categories. Then in the few-shot fine-tuning stage this pre-
trained object detector will be fine-tuned on novel dataset to detect novel
categories. In this way the common knowledge for feature extraction, proposal
generation can be transferred from base categories to novel categories.

(a) Base Training Stage

(b) Few-Shot Fine-tuning Stage

Figure 3: Overview of two-stage transfer-learning framework for standard
FSOD[77]. In base training stage, the model is trained on base dataset with
abundant data for base categories while in few-shot fine-tuning stage, the
model is trained on a small dataset containing data for both base categories
and novel categories. Current transfer-learning methods mostly adopt Faster
R-CNN as shown in the figure. The light blue components in these two figures
denote intermediate tensors, and the navy blue components and yellow
components denote trainable and freezed modules in Faster R-CNN, respectively.

LSTD[8] is the first method to use this transfer-learning scheme for FSOD.
LSTD adopts an object detector consists of an object classification head as in
Faster R-CNN[64] and a bounding box regression head as in SSD[50]. This method
proposes two regularization terms to the loss function for few-shot fine-
tuning stage to mitigate the overfitting problem. The first term, named
background-depression term, is a L2 regularization term to suppress background
regions in the feature-maps. And the second term, named transfer-knowledge
term, is a cross entropy loss which promotes the fine-tuned model to generate
similar class predictions with the source model. However, the performance of
LSTD is exceeded by the meta-learning methods in that period. And LSTD is
assumed to be hard to extend to situations that source domain is quite
different from target domain. Therefore, few work follows the transfer-
learning scheme after emergence of LSTD.

TFA[77] (Two-Stage Fine-tuning Approach) significantly improves the
performance of transfer-learning methods, which adopts Faster R-CNN as
detection framework. In the base training stage, TFA pre-trains the model on
base dataset as previous transfer-learning methods. Then in the few-shot fine-
tuning stage, it freezes the main components of Faster R-CNN and only fine-
tunes the last two layers (box classification layer and box regression layer)
of Faster R-CNN. The loss function used in the few-shot fine-tuning stage is
the same as in base training stage but a smaller learning rate is applied. The
dataset used in the few-shot fine-tuning stage is a balanced dataset
containing not only the few training samples in novel classes but also few
selected training samples in base classes. This design is to reserve the
model’s detection ability for base classes and mitigate the problem that some
objects in base classes are misclassified into novel classes. With this simple
but effective training strategy, TFA beats early meta-learning methods like
FSRW, MetaDet and Meta R-CNN on both MS COCO benchmark and PASCAL VOC
benchmark. Therefore, subsequent transfer-learning methods mainly follow this
paradigm and they can be categorized into classification based methods and
data augmentation methods.

####  III-C1 Classification-based Methods

Object detection consists of a classification task and a localization task.
The classification-based transfer-learning methods aim to improve the
classification performance of the model. This survey categorized these
classification-based methods into metric learning methods, contrastive
learning methods and semantic-based methods.

Metric Learning Methods. Metric learning is the task of learning a proper
distance metric function over objects, which is designed to make objects from
the same category closer and objects from different categories further. For
classification, metric learning can measure the distance between a query
sample and different classes, in this way the class probability of this query
sample can be obtained. These methods mainly uses reciprocal of distance and
variants of cosine similarity to measure the similarity between object feature
and class prototype. Some methods[31][86] use multiple prototypes to represent
a class instead of only a single prototype. The summary of metric learning
methods for the classification branch of FSOD are listed in Table IV.

TABLE IV: Summary of Metric Learning Methods Method | Detection Framework | Similarity Metric |  Use Multiple Representatives | Fine-tuning Strategy
---|---|---|---|---
RepMet[31] | Faster R-CNN | $\exp(-\frac{d(x_{i},w_{c})^{2}}{2\sigma^{2}})$ | ✓ | parameter replacement
NP-RepMet[86] | Faster R-CNN | $\exp(-\frac{d(x_{i},w_{c})^{2}}{2\sigma^{2}})$ | ✓ | parameter replacement
PNPDet[86] | CentreNet | $\sigma(\tau\cdot\tau_{c}\cdot\frac{w_{c}^{\top}\cdot x_{i}}{|w_{c}|\cdot|x_{i}|})$ | ✗ | parameter retraining
CRaDR[45] | Faster R-CNN | ${\rm softmax}(\alpha*\frac{w_{c}^{\top}\cdot x_{i}}{|w_{c}|\cdot|x_{i}|})$ | ✗ | parameter retraining

RepMet[31] is another transfer learning method which focuses on the metric
learning of RoI classification. Traditional metric learning methods usually
use feature vector as prototype for each class. Instead, RepMet considers each
class has a fixed number of modes and represents each mode with a feature
vector. For RoI feature vector $v_{q}^{i}$, let $d_{ij}(v_{q}^{i})$ denotes
the distance between $v_{q}^{i}$ and the $j$-th mode of the $i$-th class. Then
RepMet proposes a max-margin loss for classification of $v_{q}^{i}$:

${\rm Loss}={\rm
max}(0,\min\limits_{n}d_{m^{*}n}(v_{q}^{i})-\min\limits_{n,m\neq
m^{*}}d_{mn}(v_{q}^{i})+\alpha)\text{,}$ (12)

where $m^{*}$ denotes the correct class and $\alpha$ denotes the margin.
During testing, each novel sample is used to represent a mode of that novel
category. NP-RepMet[86] is an extension of RepMet, which considers to restore
negative modes for each class. NP-RepMet additionally defines a set of
negative modes in the same form as positive modes for each class and proposes
another loss function for negative modes.

PNPDet[89] adopts a single-stage detection framework CenterNet. This methods
first disentangles the recognition of base classes and novel classes by adding
a parallel heatmap prediction sub-network for novel classes. During the few-
shot fine-tuning stage, only this sub-network is trained. Such architecture
can naturally mitigate the problem of forgetting over base classes. Then this
paper proposes Cosine Similarity Comparison Head (CosHead) into this detection
framework as a distance metric learning method for classification. This method
uses feature vector as the prototype of each class following many metric
learning methods. Given the query feature-map $\phi_{q}$, the similarity
between the feature of location $(x,y)$ and the feature vector $v_{c}$ of
class $c$ is computed as below:

$S_{c,x,y}=\sigma(\tau\cdot\tau_{c}\cdot\frac{v_{c}^{\top}\cdot{\phi_{q}}_{x,y}}{|v_{c}|\cdot|{\phi_{q}}_{x,y}|})\text{,}$
(13)

where $\tau_{c}$ is a learnable scale factor for class $c$ to alleviate the
problem of large intra-class variance. PNPDet is easy to implement and it has
the potential to transfer to other detection framework. However, this paper
directly uses the pretrained bounding box regression branch without fine-
tuning from base classes to novel classes, which is designed to be class-
specific in many other object detection methods. And due to other factors like
the chosen of detection framework, the performance of this method is not quite
satisfactory.

CRaDR[45] proposes a Few-Shot Correction Network (FRCN), which extracts
features of region proposals individually and generates classification scores
for them. These classification scores are used to refine the classification
scores of the main model. Besides, this paper proposes a semi-supervised
distractor utilization loss to select unlabeled distractor proposals for novel
categories and a confidence-guided dataset pruning (CGDP) method for filtering
out training images containing unlabeled novel-class objects.

The second category of classification-based methods is contrastive learning.
Bo Sun et al.[70] propose a few-shot object detection method via contrastive
proposal encoding (FSCE). FSCE first proposes a stronger baseline than TFA
which adapts better to novel data. With unfrozening RPN and ROI, the stronger
baseline makes two modifications, (1) double the maximum number of proposals
kept after NMS during few-shot fine-tuning, which brings more foreground
proposals for novel classes, and (2) halving the number of sampled proposals
in RoI head used for loss computation by eliminating only background
proposals. Then FSCE proposes a contrastive branch parallel to the
classification branch and regression branches. It measures the similarity
between different region proposals and adds a contrastive loss function to
maximize the agreement between region proposals from the same category and
promote the distinctiveness of region proposals from different categories.
This paper is the first attempt to introduce contrastive learning into
transfer-learning based FSOD, which significantly improves the performance of
the baseline TFA.

The third category of classification-based methods is semantic-based methods.
SRR-FSD[100] is a FSOD method based on semantic relation reasoning to improve
the classification performance of the model. This method first constructs a
semantic space and projects the visual features into this semantic space. In
this semantic space, a set of word embeddings $W_{e}\in\mathbb{R}^{N\times
d_{e}}$ are used as semantic prototypes to represent all $N$ classes. This
method generates class probability for a projected visual feature $v_{q}^{i}$
by calculating the similarity between the projected visual features and the
semantic prototypes of $N$ classes: $p={\rm softmax}(W_{e}Pv_{q}^{i}+b)$,
where $P$ is the projection matrix. Besides, this method uses a knowledge
graph to augment the semantic prototypes, which aims to reduce the domain gap
between vision and language. This paper is the first attempt to introduce
semantic information into few-shot object detection and it significantly
improves the performance of the baseline TFA. It is necessary to point that
this method suffers less from the shot variance with the use of auxiliary
semantic information, which means that this method performs much better when
the number of labeled samples is extremely small.

####  III-C2 Data Augmentation Methods

Data Augmentation Methods aim to extract multiple and diverse features from
few training images to tackle the data scarce problem of few-shot setting.
Attention mechanism and sample generation are two widely used techniques in
these methods. Attention mechanism is used to capture long-range dependencies
of different locations in an image, and generates a global representation of
image feature. Sample generation aims to generate new samples to tackle the
data scarcity problem of FSOD. The summary of these methods are listed in
Table V.

TABLE V: Summary of Data Extraction Methods Method | Detection Framework | Use Attention | Generate New Samples
---|---|---|---
CT-FSOD[87] | RFB Net | ✓ | ✗
AttFDNet[10] | RFB Net | ✓ | ✗
MSPR[80] | Faster R-CNN | ✗ | ✓
FSSP[82] | YOLOv3-SPP | ✓ | ✓
Halluc[93] | Faster R-CNN | ✗ | ✓
Uni-Proto[79] | Faster R-CNN | ✗ | ✓

CT-FSOD[87] proposes a context transformer which augments the extracted
feature with attention mechanism. This method consists of two steps: affinity
discovery and context aggregation. In the first step, this method extracts the
feature-map $\phi_{q}$ of the input image. Then it constructs a contextual
feature-map $\psi_{q}$ by directly applying a pooling operation on $\phi_{q}$:
$\psi_{q}={\rm Pool}(\phi_{q})$. The affinity discovery step is to investigate
the relationship between each position of original feature-map and the
contextual feature-map, in which the affinity matrix is computed as the dot-
product of $\phi_{q}$ and $\psi_{q}$. In the second step, the previously
obtained affinity matrix is used to weight the contextual feature-map and
aggregate it into original feature-map to obtain the augmented feature-map
$\widehat{\phi}_{q}$. $\widehat{\phi}_{q}$ is used for further classification
task.

AttFDNet[10] is a FSOD method which augments the extracted features using
bottom-up and top-down attention. Given the feature-map
$\phi_{q}\in\mathbb{R}^{C\times H\times W}$, this method uses a convolution
layer to compute a soft attention map which models the spatial response from
the top-down attention. Then this attention map is multiplied with $\phi_{q}$
to obtain the global representation $\widehat{\phi}_{q}$. Next, AttFDNet uses
two convolutional layers with a ReLU layer and a normalization layer (LN) to
obtain the augmented features. The obtain features can be further augmented
using a bottom-up attention model. Besides, AttFDNet introduces two novel loss
function for few-shot fine-tuning stage: object concentration loss and
background-concentration loss. The former loss aims to maximize the cosine
similarity between positive proposals of the same category and the latter loss
aims to minimize the cosine similarity between the negative proposals and the
weights for background.

MSPR (Multi-Scale Positive Sample Refinement)[80] applies data augmentation by
enhancing object scales. MSPR employs FPN in the backbone network to extract
feature-maps in different scales. Then this method uses an auxiliary
refinement branch to generate multi-scale positive samples as object pyramids
and further refine the prediction. This method extracts each positive object
independently and resizes them to various scales as object pyramids. Then
these objects are fed into the FPN and a set of feature pyramids are
generated. This paper selects only one specific scale level of feature-maps
for each object in the object pyramids. Then these selected feature-maps will
be fed back into RPN and detection head for training.

FSSP[82] is a data augmentation method for few-shot object detection using a
multi-scale mechanism and a self-attention mechanism. This multi-scale
mechanism aims to enrich scale distribution of few training samples as
MPSR[80]. This paper uses three steps to enrich the scales of positive
samples: background sparsity, multi-scale replication and random clipping. In
the background sparsity step, the background of the image is set to be sparse
by setting the pixel to be zero, which is to reduce the interference of
negative samples. In the multi-scale replication step, the obtained image is
copied at different scales to form an image pyramid. Finally in the random
clipping step, each image in the image pyramid is randomly clipped multiple
times. After these three steps, each input image is augmented into multiple
images with different scales.

Halluc[93] is a data augmentation method which learns to generate additional
and useful training samples in the RoI feature space. This method proposes a
simple hallucinator to generate hallucinated RoI features used, which is
implemented as a simple two-layer MLP. The input to the hallucinator are a
class prototype, a seed example and random noise. The output of the
hallucinator is the same size as the feature of the input seed example. In the
base-training stage, this paper first trains a Faster RCNN detector on base
classes in the same way as generalized object detection. Then it freezes all
model components in the model except the hallucinator and selects training
samples as seed samples for the hallucinator to generate new samples. The loss
function of the hallucinator is a cross-entropy loss for these hallucinated
samples. And in the few-shot fine-tuning stage, the training of the
hallucinator and the training of the classifier and the regressor are
alternatively conducted. The training of the hallucinator follows the same way
as in the base-training stage. During the training of the classifier and the
regressor, the hallucinator generates a set of foreground examples to replace
some negative samples in the original training set. It is necessary to point
that this method has good performance when the number of training samples is
extremely small but it doesn’t guarantee to outperform the baseline methods
like TFA when the number of training samples becomes larger.

${\rm FSOD}^{\rm up}$[79] augments feature vectors in the fine-tuning stage
using the universal prototypes learned in the base-training stage. This paper
assumes that base categories and novel categories share some invariant object
characteristics and they are able to be learned during the base-training
stage. This method first randomly initializes the universal prototypes. The
query feature-map of the input image is transformed using these universal
prototypes and the transformed feature-map is used for further RPN and
detection head. These universal prototypes mainly represent the image-level
information. For further object-level augmentation, this method makes an
affine transformation to promote these universal prototypes towards object-
level features. After transformation, these conditional prototypes can be used
to augment the original feature-map based on attention mechanism.

####  III-C3 Few Other Methods

Few other methods propose transfer-learning based methods which focus on other
components of detection models. CoRPN[94] modifies the RPN component in Faster
R-CNN for few-shot object detection. This paper assumes that the RPN
pretrained on base classes will miss some objects of novel classes. To tackle
this problem, this paper uses multiple foreground-background classifier in RPN
instead of original single one. During testing, a given proposal box gets the
score from the most certain RPN. And during training, only the most certain
RPN will get the gradient from the given proposal box. This paper proposed two
losses to train these redundant RPN’s. The first loss is a diversity loss
based on covariance of different RPN’s, which encourages these RPN’s to be
distinct. The second loss is a cooperation loss which prevents any RPN from
firmly rejecting any foreground proposal box. The design of this redundant
RPN’s aims to avoid missing potential proposal boxes with novel objects.

##  IV Semi-Supervised Few-Shot Object Detection

Semi-supervised few-shot object detection aims to incorporate unlabeled data
into few-shot object detection. Currently only few methods delve into this
research area.

WL-FSOD[54] is an early method to perform semi-supervised object detection on
a large video dataset with only a few labeled samples. This method starts with
these few labeled samples $\mathcal{L}_{0}$. At each iteration $i$, it uses
the model trained on $\mathcal{L}_{i-1}$ to label the unlabeled video data and
adds them to current labeled set:
$\mathcal{L}=\mathcal{L}\cup\mathcal{L}_{i}$. This method uses uLSIF
(unconstrained Least Squares Importance Fitting)[28] for outlier removal. It
discovers two failure modes of tracking: drift due to spurious motion and
drift due to appearance change. At each iteration, this method only selects
few of labeled samples for model training, which can mitigate the problem of
drift. This method selects boxes without high confidence predicted from
current model, which can increase the coverage of the model.

MSPLD[14] proposes an alternating multi-modal training strategy for semi-
supervised few-shot object detection. This method aims to train multiple
detection models with a few labeled samples and a large pool of unlabeled
samples. During each iteration of training, this method first fuses the
results from all detection models and generates pseudo-labels for unlabeled
samples, then it selects pseudo-labeled samples for the training of each model
and computes the corresponding loss function. Afterwards, this method updates
parameters of models, parameters of sample selection, pseudo-labels
alternatively. It also introduces a self-paced learning regularization term
for multi-modal learning.

##  V Weakly-Supervised Few-Shot Object Detection

Weakly-supervised few-shot object detection aims to tackle the problem of few-
shot object detection using image-level annotations. The difficulty of this
problem is how to incorporate image-level annotations for model training.

NOTE-RCNN[18] starts weakly-supervised FSOD with a small amount of images with
box annotations and a large amount of images. This method generates high-
confidence boxes from images with image-level annotation and uses these mined
boxes to train the model at each iteration. During training, the background
proposal loss for these mined boxes are filtered because false negative is
extremely hard to eliminate. Besides, this paper also uses knowledge
distillation to prevent the feature extractor from overfitting the noisy
annotations.

Similar with NOTE-RCNN, UniT[32] is another weakly-supervised FSOD method
which uses image-level annotations as auxiliary data besides bbox-level
annotations. Before base training and few-shot fine-tuning stages, UniT first
uses OICR (Online Instance Classifier Refinement)[71] to train a weakly-
supervised object detection model on auxiliary abundant image-level labeled
data. Then the base training and few-shot fine-tuning are conducted upon this
weakly-supervised model. During few-shot fine-tuning, this paper also uses
lingual similarity and visual similarity to capture similarities between base
classes and novel classes.

Different from NOTE-RCNN and UniT which use both box annotations and image-
level annotations, StarNet[30] adopts a more challenging setting which uses
only image-level annotations to train the model. Given the input query
feature-map $\phi_{q}\in\mathbb{R}^{r\times r\times c}$ and support feature-
map $\phi_{c}\in\mathbb{R}^{r\times r\times c}$, this method calculates the
normalized support-query similarity matrix $R\in\mathbb{R}^{r\times r\times
r\times r}$. Then this paper defines the voting target as
$t_{i,j,l,m}=(i,j)+(r/2,r/2)-(l,m)$, where $(i,j)$ is the location of query
feature-map and $(l,m)$ is the location of support feature-map. With this
$2r\times 2r$ hypothesis grid, this method accumulates belief score of each
location of this hypothesis grid using a gaussian kernel $\mathcal{G}$:

$V_{\phi_{q},\phi_{c}}(x,y)=\mathcal{G}\ast\sum\limits_{t_{i,j,l,m}=(x,y)}R_{i,j,l,m}\text{.}$
(14)

Then the score of predicting the category label $c$ for $\phi_{q}$ is as
below:

${\rm
SC}_{1}(c;\phi_{q})=\frac{1}{k}\cdot\sum\limits_{\phi_{c}}\max\limits_{x,y}V_{\phi_{q},\phi_{c}}(x,y)\text{.}$
(15)

This paper also introduces back-projection maps, which uses the similarity
matrix between query feature-map and support feature-map to highlight the
related parts of them. Then it uses the highlighted feature-maps to calculate
class prototypes and obtains another classification score ${\rm SC}_{2}$ with
a distance metric function. The final classification score is the combination
of ${\rm SC}_{1}$ and ${\rm SC}_{2}$.

##  VI Incremental Few-Shot Object Detection

Incremental few-shot object detection is another challenging setting of few-
shot object detection. In this setting, novel classes are registered
incrementally with few labeled samples.

ONCE (open-ended centrenet)[56] is an early method for incremental few-shot
object detection, which adopts meta-learning mechanism and is adapted from
centrenet[99]. Centrenet is an object detection framework which is inspired by
keypoint detection methods. It takes the centre point and the spatial size of
the object bounding box as the regression target, which are represented with
2D heatmaps. Centrenet aims to predict these heatmaps and uses a $L_{1}$ loss
to train the network. For incremental setting, this paper also separates
centrenet into category agnostic components and category specific components.
The former components can be shared by base classes and novel classes while
the latter components are specific for each individual class. This paper first
trains category agnostic components in the similar way with the original
centrenet. To detect objects of the incrementally registered class, this paper
introduces generator to generate class code which will be used to filter the
category agnostic feature-map using a convolution operation. This incremental
network is trained via meta-learning. Given an episode containing few support
data and a set of query data, the support data is used to generate class codes
for the detection of query data.

LEAST[39] is an another method for incremental few-shot object detection which
is based on transfer-learning framework. The training of transfer-learning
framework consists of two stages: base training and few-shot fine-tuning. This
paper focuses on the second stage. First, this paper separates its detection
backbone faster rcnn into category agnostic components and category specific
components. During fine-tuning, only the category specific components will be
optimized. Then it introduced knowledge distillation to mitigate the problem
of forgetting the knowledge of detection base objects, which is implemented
with a KL divergence loss. To better preserve the old learnt knowledge, this
paper also introduces an exemplar selection algorithm to select few
informative samples from the old training set, which will be stored for fine-
tuning. It uses k-means algorithm to cluster image features into $K$ clusters,
then it can obtain $K$ feature centroids for each class.

##  VII Zero-Shot Object Detection

Zero-shot object detection separates object classes into seen classes and
unseen classes, where seen classes contains abundant labeled samples while
unseen classes contain none labeled samples. Zero-shot object detection aims
to detect objects of unseen classes using the semantic relationship between
seen classes and unseen classes. According to whether incorporating unlabeled
test images into model training, this survey categorizes zero-shot object
detection into two types: inductive detection and transductive detection.

###  VII-A Inductive Detection

Inductive detection can also be categorized into two types: semantic attribute
based and textual description based. The former type utilizes the word vector
for each class as the auxiliary semantic information while the latter type
utilizes the textual description like a description sentence as the auxiliary
semantic information.

Ankan Bansal et al.[2] propose one of the earliest work for zero-shot object
detection. This work first proposes a simple baseline. With the extracted
feature for the $i$-th box in the query image as $\phi_{q}^{i}$, the baseline
method projects this visual feature into semantic space via a linear
projection. Then these projected semantic features are used to calculate the
cosine similarities with semantic attributes of different classes, which
represent the class probabilities of these boxes. This paper introduces a max-
margin loss for the training of this projection function:

$\mathcal{L}(b_{i},y_{i},\theta)=\sum\limits_{j\neq
i}\max(0,m-S_{i,i}+S_{i,j})\text{,}$ (16)

where $S_{i,j}$ denotes the similarity between the $i$-th region proposal and
the semantic attributes of class $j$. Then this paper introduces two method to
represent background class. The first method is to assign a fixed vector for
the background class in the semantic space but the background appearances are
diverse so it is hard to find such a fixed vector. And the second method is to
assign multiple classes to the background objects. This paper first pre-trains
a model using only data from seen classes, then adopts an iterative EM-like
procedure to train this pretrained model. At each iteration, this method uses
current model to assign labels to randomly sampled background boxes and add
them to the training set for re-training. For further performance boost, this
paper introduces additional data from external classes (not intersect with
seen classes and unseen classes) into training, which is to learn a densely
sampled embedding space.

####  VII-A1 Semantic Attributes Based

Semantic attributes based methods can be categorized into class semantic
relation based methods, data augmentation based methods, loss function based
methods and mapping function based methods.

Class semantic relation methods aim to detect objects of unseen classes by
utilizing the semantic relation between classes, which can be further
categorized into seen-unseen class relation and super-class relation. For
seen-unseen class relation, semantic similarities between seen classes and
unseen classes are used to transfer knowledge from seen classes to unseen
classes. For super-class relation, some coarse-grained categories (super-
classes) are defined to cluster all classes into several groups, which divides
the original classification problem into two sub-problems (coarse-grained
classification and fine-grained classification). In different papers, these
semantic relation are incorporated into different components of the detection
model. The details of class semantic relation methods are summarized in Table
VI.

TABLE VI: Summary of Class Semantic Relation Methods Method | Detection Framework | Relation Type | Component
---|---|---|---
TOPM-ZSD[68] | YOLOv3 | Seen-Unseen Class Relation | Object Confidence Prediction
CG-ZSD[44] | YOLOv3 | Super-Class Relation | Object Classification
JRLNC-ZSD[62] | Faster R-CNN | Super-Class Relation | Object Classification
LSA-ZSD[75] | FNet & CNet & RNet | Seen-Unseen Class Relation | Attributes Augmentation
ACS-ZSD[52] | RetinaNet | Seen-Unseen Class & Super-Class Relation | Attributes Augmentation & Object Classification
DPIF[43] | Faster R-CNN | Seen-Unseen Class Relation | Object Classification

TOPM-ZSD[68] introduces seen-unseen class relation into object confidence
prediction component of zero-shot detection model. Similar with ZS-YOLO, TOPM-
ZSD adopts YOLOv3 as detection framework and projects visual features into
semantic space using a convolutional layer. The difference is that TOPM-ZSD
uses seen-unseen class relation to predict the object confidence score of each
box in the grid. TOPM-ZSD predicts a confidence vector for each box. The
length of the confidence vector is the number of all classes (including seen
classes and unseen classes). The confidence vector is set to be zero if its
corresponding box contains nothing. And if the box contains an object, the
element corresponding to its label is set to be $1$ and others are set to be
the semantic similarity between them and the true class. The design of this
object confidence prediction component aims to decrease the number of novel
objects which are misclassified to be background during inference time.

CG-ZSD[44] introduces super-class relation into classification branch of zero-
shot detection model. CG-ZSD also adopts YOLOv3 as detection framework and it
uses dilated convolution to extract visual features, which aims to enlarge the
reception fields and harvest context features implicitly. To incorporate
super-class information which is inferred from WordNet into the model, the
classification component is divided into two branches. The first branch uses a
MLP layer to predict the super-class probability of each cell in the grid. And
the second branch projects visual features into semantic embeddings and uses
the similarity between these semantic embeddings with class semantic
attributes to obtain class probabilities. The final class probability is the
product of outputs from these two branches. Apart from classification loss and
localization loss in original YOLOv3 model, CG-ZSD also uses a super-class
classification loss for super-class prediction and a margin loss for semantic
embedding learning.

JRLNC-ZSD[62] introduces super-class relation into a ZSD baseline which also
adopts Faster R-CNN as detection framework. This baseline projects RoI visual
features linearly into semantic embeddings and multiplies them with class
semantic attributes to obtain class probabilities. To train this model, JRLNC-
ZSD replaces the classification loss in original Faster R-CNN with the
summation of a max-margin loss and a meta-class loss which utilizes super-
class relation. The max-margin loss aims to separate the prediction response
of the true class from rest of the classes. Similarly, the meta-class loss
aims to pull classes belonging to different meta-classes further apart and
cluster together classes in the same meta-class.

LSA-ZSD[75] introduces seen-unseen class relation into classification branch
of zero-shot detection model. LSA-ZSD adopts a single-stage model as detection
framework consisting of three components: FNet component for feature
extraction, RNet component for object localization and CNet component for
object classification. CNet component is split into two branches and each
branch projects visual features into semantic embeddings using a mapping
matrix. These two branches obtain the class probability of each anchor box by
calculating the inner product between semantic embeddings and enriched
semantic attributes. To be specific, the first branch generates enriched class
semantic attributes by incorporating some semantic attributes of external
classes into seen classes with a linear projection function. The second branch
learns enriched class semantic attributes of seen classes during training.
Then it computes the relationships between seen classes and unseen classes by
solving a ridge regression problem, which aims to linearly align unseen class
semantic attributes with seen class semantic embeddings. These seen-unseen
class relationships are used to generate unseen class semantic attributes.
Finally the class probability is computed as the combination of outputs from
these two branches. The total loss function of LSA-ZSD also consists of a
focal classification loss and a L1 smooth localization loss.

ACS-ZSD[52] incorporates both inter-class relation and super-class relation
into RetinaNet for zero-shot detection. This paper uses explainable attributes
from aPaY dataset for class semantic. Each element of a class semantic vector
can only be 1 or 0, where 1 represents that this class has this attribute and
0 vice versa. This paper assumes that these manually defined semantic
attributes may not be suitable for a certain dataset, and thus it introduces
an unsupervised algorithm to evaluate them. Firstly it uses K-nearest neighbor
algorithm to cluster all classes (including both seen classes and unseen
classes) into several general categories (super-classes). Second another
predicted general category for each unseen class is assigned by selecting the
most frequent general category of its nearest base classes. If these two
predicted general categories for an unseen class are not consistent, the
semantic attributes will be adjusted by adding some common attributes of the
same general category. After establishing the suitable semantic attributes,
ACS-ZSD calculates the similarity between each seen class and unseen class
with their attributes and obtains a class similarity matrix. Finally the class
probability for an unseen class is computed to be the multiplication of this
class similarity matrix and class probabilities for seen classes.

DPIF[43] introduces seen-unseen class relation into a simple baseline which
adopts . The baseline first encodes visual features and class semantic
attributes into a common latent space and compares the projected vectors with
a similarity metric to compute the class probability. After building the
baseline, DPIF establishes the ground-truth semantic association between seen
classes and unseen classes according to the distance between their semantic
attributes. Next DPIF learns a convolutional layer which predicts another
semantic associations for each RoI from RoI visual features. The final class
probability prediction is the output from original baseline model multiplied
with ground-truth semantic association and learnt semantic association.

Data augmentation methods aim to generate synthesized data for unseen classes.
The generated data can be used to retrain the classifier of the detection
model, which is to mitigate data scarcity problem. Currently data augmentation
methods for ZSD are rare and they mainly use GAN or other construction losses
to generate augmented data. The details of these methods are summarized in
Table VII.

TABLE VII: Summary of Data Augmentation Methods Method | Detection Framework | GAN Based
---|---|---
DELO[102] | YOLOv2 | ✗
GTNet[95] | Faster R-CNN | ✓
SYN-ZSD[25] | Faster R-CNN | ✓

DELO[102] proposes a conditional visual feature generator based on YOLOv2
framework to generate synthesized features for unseen classes. DELO first pre-
trains a simple baseline for ZSD and samples a dataset $D_{\rm res}$
containing foreground/background RoIs for the training of this visual feature
generator. This visual feature generator consists of a encoder and a decoder,
which are both simple two fc-layer networks. The encoder takes the input of a
seen feature $v_{q}^{i}$ and a semantic embedding $s_{c}$, then outputs the
distribution of the latent variable $z$. The decoder generates exemplar
feature $\hat{v}_{q}^{i}$ with $z$ and the semantic attribute $s_{c}$ as
inputs. This paper proposes a conditional VAE loss to train this module, which
consists of a KL divergence loss and a reconstruction loss. Besides, this
paper introduces three additional losses to encourage the consistency between
the reconstructed feature $\hat{v}_{q}^{i}$ and the original feature
$v_{q}^{i}$. The first loss is a MSE loss between the confidence score of
$\hat{v}_{q}^{i}$ and $v_{q}^{i}$. The second loss is a cross-entropy loss
which encourages $\hat{v}_{q}^{i}$ to be recognized as the original category.
The third loss is a MSE loss between the predicted semantic attribute of
$\hat{f}$ and the true semantic attribute $s_{c}$. After this conditional VAE
is trained, it can generate synthesized data features $D_{\rm syn}$ for both
seen classes and unseen classes. $D_{\rm syn}$ combined with original data
features $D_{\rm res}$ will be used to retrain the confidence predictor of the
model.

GTNet (Generative Transfer Network)[95] synthesizes visual features of unseen
classes using GAN (generative adversarial network) based on Faster R-CNN
framework. GTNet consists of three modules: CFU (class feature generating
unit), FFU (foreground feature generating unit) and BFU (background feature
generating unit), each of them adopts the design of WGAN[1] and contains a
generator and a discriminator. The generator of CFU takes random Gaussian
noise and the class embedding as inputs, and outputs a synthesized visual
feature of this class. The discriminator of CFU determines whether the input
visual feature is synthesized or not. FFU is used to align the generated
features of CFU towards features extracted from a ground-truth box. The
generator takes random Gaussian noise and the generated feature of CFU, and
outputs the aligned visual feature. BFU is used to generate class-specific
background features. The generator and discriminator adopt the similar
structure as FFU. This paper also uses a classification loss to classify the
synthesized visual features using a pre-trained classification layer and a
embedding loss to encourage synthesized features to be close to the real
features of its class and far from the features of other classes. After this
GTNet is trained, the generated synthesized features will be used to train an
new classifier.

SYN-ZSD[25] is another GAN based method to synthesize visual features which
also adopts Faster R-CNN as detection framework. This paper assumes that the
previous similar method GTNet trains multiple GAN models in a sequence manner,
which makes it hard to fix errors in early stage. To solve this problem, SYN-
ZSD only trains one GAN to synthesize visual features instead. The generator
of this GAN generates synthesized visual features with gaussian-like random
noise and the class embedding as inputs and the discriminator determines
whether the input feature is synthesized. SYN-ZSD adds two classification
losses and a divergence loss for the GAN. For the classification losses, this
paper first pre-trains a classifier for seen classes, which is implemented by
computing the similarity between RoI semantic embeddings and class semantic
embeddings. The classifier for unseen classes is modified from this pre-
trained classifier. The synthesized features are then fed into these two
classifiers and the classification losses can be obtained. The divergence loss
is to encourage the GAN to diversify the synthesized feature space and enhance
chances of generating features from minor modes. The trained GAN generates
synthesized visual features and their corresponding class-labels, which will
be used to update the original classifier.

Loss function based methods attempt to improve the performance of original ZSD
model by designing new loss functions. Current loss function based methods
mainly modify the loss function for classification branch.

VSA-ZSD[59] proposes a polarity loss based on focal loss for the
classification branch of single-stage detectors. VSA-ZSD considers that the
original focal loss can push visual features close to their ground-truth class
vectors but can not push them away from negative class vectors. The proposed
polarity loss can satisfy these two conditions by adding a penalty loss to
focal loss. The form of polarity loss is as below:

$\mathcal{L}_{\rm PL}=\sum\limits_{i}\frac{1}{1+\exp(-\beta(p^{i}-p^{l}))}{\rm
FL}(p^{i},y^{i})$ (17)

where $p^{i}$ denotes the prediction score for class $i$, $l$ denotes the
ground-truth class, $\beta$ denotes a hyperparameter and ${\rm FL}$ denotes
original focal loss. The added penalty loss enforces a large negative margin
to push predictions $p^{i}$ and $p^{l}$ further apart. It assigns a low
penalty to well-classified/easy examples and a high penalty to poorly-
performed/hard cases. Besides this polarity loss, VSA-ZSD introduces word-
vectors of a pre-defined vocabulary to learn a more expressive and flexible
semantic space.

Mapping function based methods aim to find a proper mapping function to map
visual features into semantic space or conversely map semantic features into
visual space. The details of current mapping function based methods are listed
in Table VIII.

TABLE VIII: Summary of Mapping Function Based Methods Method | Detection Framework | Mapping Direction
---|---|---
HRE-ZSD[13] | YOLOv2 | visual $\rightarrow$ semantic
MS-ZSD[23] | Faster R-CNN | visual $\rightarrow$ semantic & semantic $\rightarrow$ visual
BLC[97] | Faster R-CNN | visual $\rightarrow$ semantic
SPGP[83] | Faster R-CNN | visual $\rightarrow$ semantic

HRE-ZSD[13] proposes two methods to project visual features into semantic
space. Then the first method computes the semantic embedding of the $i$-th RoI
in the query image $q$ as the weighted combination of different class semantic
embeddings:

$s_{q}^{i}=\frac{1}{\sum\limits_{c\in C_{B}}p^{c}}\sum\limits_{c\in
C_{B}}p^{c}s_{c}\text{,}$ (18)

where $p^{c}$ denotes the probability that this region proposal is predicted
to be class $y$, $C_{B}$ denotes the set of seen classes and $s_{c}$ denotes
the semantic attributes of class $c$. The second mapping method is basically a
deep convolutional neural network that directly maps the visual features into
semantic space. After obtaining these two embedding representations of each
RoI, HRE-ZSD uses cosine similarity to predict two corresponding class
probabilities.

MS-ZSD[23] proposes a multi-space approach for visual-semantic mapping, which
simultaneously projects visual features into semantic space and projects
semantic embeddings into visual space conversely. For the visual-semantic
mapping, MS-ZSD linearly projects the predicted class probability of RoI
visual features into semantic space with a learnable matrix. For the semantic-
visual mapping, MS-ZSD uses also a linear layer to project class semantic
attributes into visual space. The final predictions is the ensemble of outputs
from these two mapping modules. And MS-ZSD introduces a cross-modal
consistency loss to maintain consistency across the representations from both
modalities.

BLC[97] projects visual features into semantic space with an external
vocabulary, which aims to enhance the richness of semantic information. Let
$D$ denotes the external vocabulary and the class semantic embeddings,
respectively. The semantic embedding of $i$-th RoI visual feature $v_{q}^{i}$
of image $q$ is computed as below:

$s_{q}^{i}=v_{q}^{i}A_{0}DA_{1}\text{,}$ (19)

where $A_{0}$ and $A_{1}$ are two learnable matrices to adapt channel
dimension while $D$ is fixed during training. BLC also proposes two modules:
cascade semantic R-CNN and background learnable RPN. The cascade semantic
R-CNN generates region proposals iteratively with a semantic information flow
structure. At each iteration, it predicts the class probabilities of RoIs
generated from the former semantic branch and generates new RoIs for the next
iteration through a bounding box regression branch. The background learnable
RPN learns background semantic vector from various background visual data.
These modules can improve the performance of ZSD but they bring complicated
iterative computation into ZSD, which is time-consuming.

SPGP[83] proposes a graph based method to project visual features into
semantic space based on Faster R-CNN framework. It constructs three kinds of
graph: category-to-category graph, region-to-category graph, region-to-region
graph. Each node of the category-to-category graph represents the semantic
embedding of a category. The evolved class semantic attributes are obtained
via a GCN layer built on this graph. The region-to-category graph represents
the relationship between region proposals and categories. The weight of edge
between a region proposal and a category is computed using the scaled dot-
product attention mechanism. After this region-to-category graph is obtained,
the semantic feature of each region proposal is obtained via also a GCN layer.
The region-to-region graph represents the relationship between each pair of
region proposals. The weight of each edge of this graph is computed as the
difference of two semantic features using a MLP layer. This can be updated
using a MSE loss between predicted and ground-truth edge weights. Then it uses
another GCN layer to obtain the evolved semantic feature of region proposals.

SMFL-ZSD[40] proposes a mapping function which projects semantic attributes
into visual space conversely. It applies multiple convolutional filters on
semantic attributes to produce multiple latent features. Then these latent
features are concatenated and fed into a MLP layer to generate visual
features, which will be compared with RoI visual features to predict class
probabilities.

####  VII-A2 Textual Description Based

Previous ZSD methods uses semantic attributes as identification information
for each unseen class. Instead, textual description based methods uses textual
description as semantic information. Methods on textual description based ZSD
are rare, and they use different types of textual description. The first type
is class textual description, which is the identification information for each
class. The second type is image textual description, which describe the
content of each image. Current textual description based ZSD methods are
listed in Table IX.

TABLE IX: Summary of Textual Description Based Methods Method | Detection Framework | Textual Type | Text Feature Extractor
---|---|---|---
Class Textual Description | Faster R-CNN | text for each class | LSTM
Image Textual Description | Faster R-CNN | text for each image | CNN

Zhihui Li et al.[46] propose a method for zero-shot object detection with
textual descriptions instead of class semantic embeddings. This model consists
of three modules: a visual branch, a language branch and a bounding box
regression branch. The visual branch takes the fixed feature of each region
proposal as input, and then outputs visual unit activations of them. Each
element of these visual unit activations determines whether certain object
patterns exist in the region proposal. The language branch takes the textual
descriptions with features of region proposals as input, then outputs hidden
features of textual descriptions via a LSTM network. The hidden features are
fed into a FC layer with a softmax function, resulting in an attention vector
on region proposals. The affinity between the each word and the region
proposals is computed as:

$\hat{a}_{t}=g_{t}\sum\limits_{n=1}^{N}A_{t}(n)v_{n}\text{,}$ (20)

where $A_{t}(n)$ denotes the relationship between $t$-th word and the $n$-th
region proposal, $v_{n}$ denotes the visual unit activation on the $n$-th
region proposal and $g_{t}$ is a FC layer with a sigmoid function to calculate
the word-level attention on the $t$-th word. $\hat{a}_{t}$ will then be used
to calculate the class probabilities of region proposals. The bounding box
regression branch is used for bounding box regression, adopting the similar
design as in Faster RCNN.

Licheng Zhang et al.[91] propose another method for textual descriptions based
zero-shot object detection. This method adopts Faster R-CNN as detection
framework and a text CNN for text feature extraction. The RoI features and
text features are concatenated together for further location regression task
and classification task. Besides, this paper uses OHEM technique to select
hard samples for model training.

###  VII-B Transductive Detection

Shafin Rahman et al.[58] propose a transductive method for zero-shot object
detection, which incorporates unlabeled test data into model training. This
method first trains the zero-shot detector based on RetinaNet architecture in
an inductive way. It uses a matrix to linearly project visual features into
semantic space for classification with a sigmoid function. The loss of this
inductive training is a standard focal loss. Then this paper performs
transductive learning with fixed pseudo-labeling and dynamic pseudo-labeling.
The fixed pseudo-labeling generates fixed pseudo-labels for seen classes using
the pre-trained inductive model. The dynamic pseudo-labeling generates pseudo-
labels for both seen classes and unseen classes iteratively. At each
iteration, it uses both fixed and dynamic pseudo-labels to train the model and
generates new dynamic pseudo-labels for next iteration. The loss for both
fixed and dynamic pseudo-labels are standard focal loss. Besides, this paper
introduces a regularization term for dynamic pseudo-labels to maximize the
unseen predictions, which can mitigate the problem that prediction scores
become biased towards seen classes.

###  VII-C Some Other Methods

ZS-YOLO[101] is another one of the earliest work for zero-shot object
detection which adopts YOLOv2 as the detection framework. The model extracts
the feature-map $\phi_{q}\in\mathbb{R}^{h\times w\times c}$ of the input
image. Each cell of the feature-map has $5$ bounding boxes with different
scales and ZS-YOLO predicts their corresponding location coordinates
$\sigma_{q}\in\mathbb{R}^{h\times w\times 5*4}$ and semantic attributes
$\omega_{q}\in\mathbb{R}^{h\times w\times 5*h}$ by forwarding $\phi_{q}$ into
two convolutional layers individually. For object confidence prediction,
$\phi_{q}$, $\sigma_{q}$ and $\omega_{q}$ are concatenated to predict and fed
into another convolutional layer to generate the confidence map with shape
$\mathbb{R}^{h\times w\times 5}$. ZS-YOLO proposes three corresponding loss
functions: object localization loss for location prediction, semantic loss for
semantic attributes prediction and confidence loss for object confidence
prediction.

ZSI[98] is a zero-shot model which tackles detection task and segmentation
task simultaneous. ZSI consists of three components: zero-shot detector,
semantic mask head and synthesized background prediction head. The zero-shot
detector is modified from Faster R-CNN by adding a encoder-decoder structure.
The encoder-decoder structure encodes RoI visual feature-vectors into semantic
vectors and decodes these semantic vectors back to reconstructed visual
feature-vectors. And it predicts class probabilities of these RoIs by
multiplying their semantic embeddings with class semantic attributes. The
semantic mask head adopts the similar encoder-decoder structure as the zero-
shot detector. The encoder and decoder of it are two individual convolutional
layers. It predicts semantic masks of these RoIs using a convolutional layer
with class semantic attributes as convolutional weights. The synthesized
background prediction head introduces visual-semantic projection function into
RPN module to predict foreground/background of RoIs and learns a more
reasonable background class embedding during this process. Apart from original
loss functions of detection task and segmentation task, this work introduces
two reconstruction losses for the zero-shot detector and the semantic mask
head, respectively.

ViLD[22] introduces knowledge distillation into zero-shot detection by
distilling the knowledge from a pre-trained zero-shot image classifier to a
two-stage detector (e.g., Mask R-CNN in this method). ViLD consists of two
components: ViLD-text and ViLD-image. Similar with other ZSD methods, ViLD-
text projects RoI visual features into semantic embeddings and compares them
with class semantic attributes using cosine similarity. The difference is that
ViLD-text allows the background class to learn its own semantic attributes.
ViLD-image selects region proposals with low background probabilities and
applies knowledge distillation by minimizing the distance between RoI semantic
embeddings generated from ViLD-text and a pre-trained zero-shot image
classifier. The final prediction of ViLD is the ensemble of ViLD-text detector
and the pre-trained zero-shot detector. And the total loss function of ViLD is
the original loss of Mask R-CNN with this distillation loss.

Ruotian Luo et al.[51] introduce conditional random field (CRF) into the
classification branch of Faster R-CNN for zero-shot object detection. This
conditional random field takes RoIs generated from RPN as nodes. The unary
potential of each node in the CRF is the class probability predicted by a FC
layer. And the pairwise potential of each edge in the CRF is obtained by a
relationship inference model with an external auxiliary knowledge graph. With
this CRF, the final class probability of each RoI (node) is computed as the
sum of its unary potential and other nodes’ potential multiplied by
corresponding pairwise potential. This paper is the first work to incorporate
class relationship auxiliary information into ZSD.

Sixiao Zheng et al.[96] propose a incremental setting for zero-shot object
detection. In this setting, all classes are also split into seen classes and
unseen classes and each class has its semantic attributes. At each incremental
step, some unseen classes are transferred into seen classes. The model will be
updated using new training samples on these transferred seen classes and needs
to detect objects from both seen classes and unseen classes. At the first
incremental step, the model only needs to learn from newly transferred seen
classes since no existing old classes. As traditional ZSD methods, it adopts
Faster R-CNN as detection framework and linearly projects RoI visual features
into semantic space and uses cosine similarity to compute the class
probabilities of RoIs. This method uses a background-foreground MSE loss, a
reconstruction loss and a triplet loss to train this model. At the subsequent
steps, the model needs to simultaneously learn from newly added seen classes
and maintain the knowledge of old classes. To tackle this challenge, knowledge
distillation is introduced to alleviate the catastrophic forgetting problem in
incremental learning. The losses of knowledge distillation encourage current
model to generate similar projected semantic embeddings and class
probabilities as old model. Besides, this paper proposes an extreme value
analyzer based on Pickands–Balkema–de Haan theorem to differentiate unseen
classes from seen classes.

##  VIII Popular Datasets and Evaluation Criteria

The datasets for few-shot & zero-shot object detection are modified from
generalized object detection datasets like PASCAL VOC dataset, MS COCO dataset
and so on.

###  VIII-A Few-Shot Object Detection

In few-shot object detection, the classes of the dataset are split into two
categories: base classes with abundant labeled samples and novel classes with
few labeled samples. Normally the base classes and novel classes come from the
same generalized object detection dataset, yet they can come from different
datasets. In general, the dataset for few-shot object detection includes
PASCAL VOC dataset split, MS COCO dataset splits, MS COCO & PASCAL VOC dataset
split, FSOD dataset split:

  * •

PASCAL VOC dataset split. Normally, the train & val set of PASCAL VOC 2007
dataset and PASCAL VOC 2012 dataset are used as the training set and the test
set of PASCAL VOC 2007 dataset is used as the test set. PASCAL VOC dataset has
totally $20$ classes. Among them, $15$ classes are selected as base classes
and the remaining $5$ classes as novel classes. PASCAL VOC dataset has $3$
different class splits.

  * •

MS COCO dataset split. MS COCO dataset has totally $80$ classes. Among them,
$60$ classes disjoint with PASCAL VOC dataset are selected as base classes and
the remaining $20$ classes as novel classes. MS COCO dataset has only $1$
class split.

  * •

MS COCO & PASCAL VOC dataset split. $60$ classes of MS COCO dataset disjoint
with PASCAL VOC dataset are selected as base classes and the data of them are
used for base training. The total $20$ classes of PASCAL VOC dataset are
selected as novel classes and data of them are used for novel testing.

The summation of these dataset splits is in Table X.

TABLE X: Summation Of Few-Shot Object Detection Dataset Dataset Split | Base Classes | Novel Classes | Shots Per Class
---|---|---|---
PASCAL VOC split | 15 | 5 | 1, 2, 3, 5, 10
MS COCO split | 60 | 20 | 10, 30
MS COCO & PASCAL VOC split | 60 | 20 | 10

###  VIII-B Zero-Shot Object Detection

Zero-Shot Object Detection Datasets are modified from commonly used general
object detection dataset like PASCAL VOC dataset, MS COCO dataset, ILSVRC-2017
dataset and VisualGenome (VG) dataset. The seen classes and unseen classes
split of these datasets are listed in Table XI. There are different classes
split methods on the same dataset adopted by different papers.

TABLE XI: Summation Of Zero-Shot Object Detection Dataset Dataset | Base Classes | Novel Classes
---|---|---
PASCAL VOC dataset | 5 | 15
10 | 10
15 | 5
16 | 4
MS COCO dataset | 20 | 20
40 | 20
60 | 20
48 | 17
65 | 15
ILSVRC-2017 dataset | 178 | 23
VG dataset | 478 | 130

The class semantic attributes for zero-shot object detection are mainly
borrowed from pre-trained word vectors or manually designed attributes: GloVe,
BERT, aPaY, word2vec and fastText. The details of these semantic attributes
are listed in Table XII.

There are two evaluation settings of ZSD methods. The first setting evaluates
the model on a subset of test data which contains only objects of unseen
classes. The second setting named generalized ZSD (GZSD) evaluates the model
on complete test data, which requires the model to detect both seen classes
and unseen classes. mAP and recall are two mainly evaluation metrics for these
two settings. Generalized ZSD separately calculates the mAP and recall of seen
classes and unseen classes, and then uses harmonic average to calculate the
average performance of seen classes and unseen classes.

TABLE XII: Summation Of Class Attributes for ZSD Semantic Attributes | type | Dimension
---|---|---
GloVe | pre-trained word vectors | 300
BERT | pre-trained word vectors | 768
aPaY | manually designed attributes | 64
word2vec | pre-trained word vectors | 300
fastText | pre-trained word vectors | $-$

##  IX Performances

This section demonstrates and analyzes the performance of different few-shot &
zero-shot object detection methods.

###  IX-A One-Shot Object Detection

Table XIII listed the performance of current OSOD methods on PASCAL VOC
dataset and MS COCO dataset. SiamFC and SiamRPN are two methods originally
proposed for video object tracking, which form the baseline for OSOD.
SiamMask, OSCD, OSODwT and FOC OSOD uses simple concatenation based methods
for feature aggregation with different details and FOC OSOD achieves the
highest performance on PASCAL VOC benchmark among them. Differently, CoAE, AIT
and CAT uses attention based methods for feature aggregation and CAT is the
best method among them. And CAT surpasses FOC OSOD by $4.5$ points on PASCAL
VOC benchmark, which indicates that attention based methods are more promising
for future one-shot object detection.

TABLE XIII: Performances of OSOD methods Type | Method/Shot | PASCAL VOC | MS COCO
---|---|---|---
Tracking | SiamFC(2018)[6] | 13.3 | $-$
SiamRPN(2018)[37] | 14.2 | $-$
Concatenation | SiamMask(2019)[53] | $-$ | 16.8
OSCD(2020)[17] | 52.1 | $-$
OSODwT(2020)[42] | 69.1 | $-$
FOC OSOD(2021)[85] | 71.0 | $-$
Attention | CoAE(2019)[26] | 68.2 | 22.0
AIT(2021)[7] | 73.1 | 24.3
CAT(2021)[48] | 75.5 | 24.4

###  IX-B Standard Few-Shot Object Detection

This subsection demonstrates the performance of standard few-shot object
detection methods on two most commonly used benchmarks: PASCAL VOC dataset and
COCO dataset. For fair comparison, this survey only lists the performance of
methods with released codes.

Table XIV and table XV presents the performances on novel categories of PASCAL
VOC benchmark and MS COCO benchmark, respectively. Some observations can be
summarized from these two tables. First, the newly proposed meta-learning
methods (DCNet, CME, Meta-DETR, DAnA-FasterRCNN, DeFRCN) outperform almost all
transfer-learning methods on both PASCAL VOC benchmark and MS COCO benchmark.
Especially on MS COCO benchmark, the best meta-learning method Meta-DETR
exceeds the best transfer-learning method LEAST by $6.2$ points in $10$-shot
condition. Second, for meta-learning methods, feature-map aggregation methods
(DCNet, CME, DAnA-FasterRCNN, DeFRCN) outperform feature-vector aggregation
methods (FSRW, Meta-RCNN, FsDet) on two benchmarks. This indicates that model
extracts more useful information from feature-map than feature-vector. Third,
for transfer-learning methods, data augmentation methods (Hallucination, MSPR,
${\rm FSOD}^{\rm up}$) have strong performance in extremely few-shot condition
(${\rm shot}=1,2,3$) and they surpasses some newly proposed meta-learning
methods in this condition. Fourth, the performance rank of a method varies on
these two benchmarks. It is not guaranteed that a method with better
performance on one benchmark still outperforms other methods on another
benchmark.

TABLE XIV: Performances of FSOD methods on PASCAL VOC dataset Framework | Method/Shot | Novel Set 1 | Novel Set 2 | Novel Set 3
---|---|---|---|---
| 1 | 2 | 3 | 5 | 10 | 1 | 2 | 3 | 5 | 10 | 1 | 2 | 3 | 5 | 10
Meta-Learning | FSRW(2018)[29] | 14.8 | 15.5 | 26.7 | 33.9 | 47.2 | 15.7 | 15.3 | 22.7 | 30.1 | 40.5 | 21.3 | 25.6 | 28.4 | 42.8 | 45.9
| Meta-RCNN(2019)[84] | 19.9 | 25.5 | 35.0 | 45.7 | 51.5 | 10.4 | 19.4 | 29.6 | 34.8 | 45.4 | 14.3 | 18.2 | 27.5 | 41.2 | 48.1
| FsDet(2020)⋆[81] | 24.2 | 35.3 | 42.2 | 49.1 | 57.4 | 21.6 | 24.6 | 31.9 | 37.0 | 45.7 | 21.2 | 30.0 | 37.2 | 43.8 | 49.6
| DCNet(2021)⋆[27] | 33.9 | 37.4 | 43.7 | 51.1 | 59.6 | 23.2 | 24.8 | 30.6 | 36.7 | 46.6 | 32.3 | 34.9 | 39.7 | 42.6 | 50.7
| CME(2021)[38] | 41.5 | 47.5 | 50.4 | 58.2 | 60.9 | 27.2 | 30.2 | 41.4 | 42.5 | 46.8 | 34.3 | 39.6 | 45.1 | 48.3 | 51.5
| Meta-DETR(2021)⋆[90] | 35.1 | 49.0 | 53.2 | 57.4 | 62.0 | 27.9 | 32.3 | 38.4 | 43.2 | 51.8 | 34.9 | 41.8 | 47.1 | 54.1 | 58.2
| DeFRCN(2021)⋆[57] | 40.2 | 53.6 | 58.2 | 63.6 | 66.5 | 29.5 | 39.7 | 43.4 | 48.1 | 52.8 | 35.0 | 38.3 | 52.9 | 57.7 | 60.8
Transfer-Learning | TFA w/cos(2020)[77] | 39.8 | 36.1 | 44.7 | 55.7 | 56.0 | 23.5 | 26.9 | 34.1 | 35.1 | 39.1 | 30.8 | 34.8 | 42.8 | 49.5 | 49.8
| Np-RepMet(2020)[86] | 37.8 | 40.3 | 41.7 | 47.3 | 49.4 | 41.6 | 43.0 | 43.4 | 47.4 | 49.1 | 33.3 | 38.0 | 39.8 | 41.5 | 44.8
| Hallucination(2021)[93] | 47.0 | 44.9 | 46.5 | 54.7 | 54.7 | 26.3 | 31.8 | 37.4 | 37.4 | 41.2 | 40.4 | 42.1 | 43.3 | 51.4 | 51.4
| MSPR(2020)[80] | 41.7 | $-$ | 51.4 | 55.2 | 61.8 | 24.4 | $-$ | 39.2 | 39.9 | 47.8 | 35.6 | $-$ | 42.3 | 48.0 | 49.7
|  ${\rm FSOD}^{\rm up}(2021)$[79] | 43.8 | 47.8 | 50.3 | 55.4 | 61.7 | 31.2 | 30.5 | 41.2 | 42.2 | 48.3 | 35.5 | 39.7 | 43.9 | 50.6 | 53.5
| FSCE(2021)⋆[70] | 32.9 | 44.0 | 46.8 | 52.9 | 59.7 | 23.7 | 30.6 | 38.4 | 46.0 | 48.5 | 22.6 | 33.4 | 39.5 | 47.3 | 54.0
TABLE XV: Performances of FSOD methods on COCO dataset split Framework | Method/Shot | 10 | 30
---|---|---|---
Meta-Learning | FSRW(2018)[29] | 5.6 | 9.1
Meta-RCNN(2019)[84] | 5.6 | $-$
FsDet(2020)⋆[81] | 12.5 | 14.7
Attention-RPN(2020)[15] | 11.1 | $-$
DCNet(2021)⋆[27] | 12.8 | 18.6
CME(2021)[38] | 15.1 | 16.9
Meta-DETR(2021)⋆[90] | 19.0 | 22.2
DAnA-FasterRCNN(2021)[9] | 18.6 | 21.6
DeFRCN(2021)⋆[57] | 16.8 | 21.2
Transfer-Learning | TFA w/cos(2020)[77] | 10.0 | 13.7
MSPR(2020)[80] | 9.8 | 14.1
LEAST(2021)⋆[39] | 12.8 | $-$
${\rm FSOD}^{\rm up}(2021)$[79] | 11.0 | 15.6
FSCE(2021)⋆[70] | 11.9 | 16.4

###  IX-C Standard Zero-Shot Object Detection

Table XVI lists the performance of standard ZSD methods under two evaluation
settings (ZSD, GZSD) on the most commonly used benchmark: MS COCO benchmark.
Some trends can be found from this table. First, early ZSD methods are not
consistent in the choices of semantic attributes and few of them are evaluated
in the GZSD setting. But newly proposed ZSD methods mostly adopt word2vec as
their semantic attributes and are evaluated in both ZSD setting and GZSD
setting, which is more convenient for performance comparison. Second,
generally the model performance of $48/17$ seen-unseen split is worse than
$65/15$ seen-unseen split, which may be due to less categories and samples in
seen data. Third, current data augmentation based ZSD methods cannot achieve
satisfactory performance compared to newly proposed methods. Since data
augmentation methods can outperform other methods when the shot number is
small, data augmentation method can potentially be explored in the future.

TABLE XVI: Performances of ZSD methods on MS COCO Benchmark Method | Semantic Attributes | Seen/Unseen | ZSD | Seen | Unseen | HM
---|---|---|---|---|---|---
mAP | Recall | mAP | Recall | mAP | Recall | mAP | Recall
SB(2018)[2] | GloVe, fastText | 48/17 | 0.70 | 24.39 | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
DSES(2018)[2] | GloVe, fastText | 48/17 | 0.54 | 27.19 | $-$ | 15.02 | $-$ | 15.32 | $-$ | 15.17
TOPM(2019)[68] | GloVe | 48/17 | 15.43 | 39.2 | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
CG-ZSD(2020)[44] | BERT | 48/17 | 7.2 | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
GTNet(2020)[95] | fastText | 48/17 | $-$ | 44.6 | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
JRLNC-ZSD(2020)[62] | word2vec | 48/17 | 5.05 | 12.27 | 13.93 | 20.42 | 2.55 | 12.42 | 4.31 | 15.45
SPGP(2020)[83] | word2vec | 48/17 | $-$ | 35.4 | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
VSA-ZSD(2020)[59] | word2vec | 48/17 | 10.01 | 43.56 | 35.92 | 38.24 | 4.12 | 26.32 | 7.39 | 31.18
MS-Zero++(2020)[23] | word2vec | 48/17 | $-$ | $-$ | 35.0 | $-$ | 13.8 | 35.0 | 19.80 | $-$
BLC(2020)[97] | word2vec | 48/17 | 10.60 | 48.87 | 42.10 | 57.56 | 4.50 | 46.39 | 8.20 | 51.37
ZSI(2021)[98] | word2vec | 48/17 | 11.4 | 53.9 | 46.51 | 70.76 | 4.83 | 53.85 | 8.75 | 61.16
SMFL-ZSD(2021)[40] | word2vec | 48/17 | 19.42 | 45.32 | 38.07 | 40.04 | 19.81 | 45.28 | 26.06 | 42.50
Transductive(2019)[58] | word2vec | 65/15 | 14.57 | 48.15 | 28.78 | 54.14 | 14.05 | 37.16 | 18.89 | 44.07
CG-ZSD(2020)[44] | BERT | 65/15 | 10.9 | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
LSA-ZSD(2020)[75] | $-$ | 65/15 | 13.55 | 37.78 | 34.18 | 40.32 | 13.42 | 38.73 | 19.27 | 39.51
ACS-ZSD(2020)[52] | aPaY | 65/15 | 15.34 | 47.83 | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
SYN-ZSD(2020)[25] | fastText | 65/15 | 19.0 | 54.0 | 36.90 | 57.70 | 19.0 | 53.90 | 25.08 | 55.74
VSA-ZSD(2020)[59] | word2vec | 65/15 | 12.40 | 37.72 | 34.07 | 36.38 | 12.40 | 37.16 | 18.18 | 36.76
BLC(2020)[97] | word2vec | 65/15 | 14.70 | 54.68 | 36.00 | 56.39 | 13.10 | 51.65 | 19.20 | 53.92
ZSI(2021)[98] | word2vec | 65/15 | 13.6 | 58.9 | 38.68 | 67.11 | 13.60 | 58.93 | 20.13 | 62.76
DPIF(2021)[43] | word2vec | 65/15 | 19.82 | 55.73 | 29.82 | 56.68 | 19.46 | 38.70 | 23.55 | 46.00
SMFL-ZSD(2021)[40] | word2vec | 65/15 | 27.39 | 45.83 | 37.41 | 39.43 | 27.44 | 46.13 | 31.66 | 42.52

##  X Promising Directions

###  X-A Promising Directions for FSOD

####  X-A1 Lightweight FSOD

Current FSOD models are generally modified from representative object
detectors like Faster R-CNN, yolo-style detectors. These large detectors need
to first pre-train on the data abundant base dataset, then fine-tune on the
few-shot novel dataset. The pre-training on base dataset requires large device
cost and time cost as general object detection, which prevents FSOD from real-
life application. Therefore, the lightweight and efficient FSOD is required to
quickly adapt external knowledge into current few-shot dataset.

####  X-A2 Sample Generation Methods for FSOD

The performance comparison result in Table XIV and table XV indicates that
data augmentation methods generally outperform other methods when the shot
number is extremely small. However, current methods rarely delve into how to
generate multiple effective samples for data-scarce categories. Sample
generation methods for FSOD still exist big room for exploration.

####  X-A3 Semi-supervised FSOD

The performance of current state-of-the-art methods for FSOD is still far from
real-life application. In few-shot learning, external unlabeled images
contains enormous useful information, which can be utilized for performance
boost. EPNet[66] uses embedding propagation to generate pseudo-labels for
unlabeled images, then conversely uses these pseudo-labels to update current
few-shot model, which can improve the accuracy by a large margin.

The semi-supervised learning for FSOD is much harder than that for few-shot
image classification. This is because the annotations for FSOD data is more
complex than few-shot image classification with additional location
annotations. The pseudo-labels for location on unlabeled data can be easily
incorrect, which will decrease the performance of the model. Current semi-
supervised methods for FSOD are rare and their performance also falls behind.
Is is hard but necessary to develop more efficient semi-supervised methods for
FSOD.

####  X-A4 New Model Framework for FSOD

Most of current FSOD methods adopt Faster R-CNN as the detection framework.
Actually, Faster-RCNN based FSOD methods have achieved excellent performance
but there may exists more powerful detection framework for FSOD in the future.
For example, vision transformer is a promising direction for new detection
framework. Vision transformer divides images into patches and concatenates
them into a token sequence, then adopts an attention-based encoder-decoder
architecture to extract features of this token sequence. Vision transformers
focus more on integral information of the image than local information, which
has the ability to capture features missed by traditional CNN models.
Currently vision transformer has been widely applied in many other computer
vision areas. In FSOD, the recently proposed Meta-DETR has achieve SOTA
performance on MS COCO benchmark, which exceeds previous Faster R-CNN based
FSOD method by several points. Therefore, the potential of vision transformer
on FSOD still requires exploration.

###  X-B Promising Directions for ZSD

####  X-B1 Auxiliary Data for ZSD

It is shown that auxiliary unlabeled data can boost performance of many few-
shot learning tasks. In ZSD, a few methods attempt to use information of
external classes (not intersect with seen classes and unseen classes) to
augment semantic attributes of seen classes and unseen classes. But generally
speaking, there are rare methods on how to incorporate auxiliary data into
ZSD.

####  X-B2 Active Learning for ZSD

Current ZSD methods pre-train the model on all the data for seen categories
(seen data). However, some images in seen data are not suitable for detecting
objects of unseen categories. For example, some images contain objects of
unseen categories without annotations and these training images will cause the
model to generate more false negative proposals. Active learning aims to
select effective data for model training, which may help to mitigate such
problem.

####  X-B3 ZSD combined with FSOD

There may exist a more complex situation in real world that only some novel
categories have labeled samples, which requires the combination of ZSD and
FSOD. Some methods have been proposed to simultaneously tackle the problem of
FSOD and ZSD. Method[61] introduces a low-shot problem setting that all
classes have their semantic embeddings but only some novel classes have
labeled samples. This method is the first attempt to combine FSOD and ZSD into
one common framework. It is necessary to delve into combination of FSOD and
ZSD in the future.

##  XI Conclusion

This paper conducts a thorough survey on low-shot object detection, which can
be categoried into few-shot object detection (FSOD) and zero-shot object
detection (ZSD), according to whether few labeled samples or semantic
attributes are used for each novel class. Firstly a generalized introduction
to low-shot object detection is given, then the methods for FSOD and ZSD are
categorized and analyzed. The main benchmarks for FSOD and ZSD are summarized
and the performances of different methods are also listed for comparison in
this survey. Finally the promising directions of FSOD and ZSD are discussed.

## References

  * [1] Martín Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein generative adversarial networks.  In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, volume 70 of Proceedings of Machine Learning Research, pages 214–223. PMLR, 2017.
  * [2] Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran.  Zero-shot object detection.  In Computer Vision - ECCV 2018 - 15th European Conference, volume 11205 of Lecture Notes in Computer Science, pages 397–414. Springer, 2018.
  * [3] Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao.  Yolov4: Optimal speed and accuracy of object detection.  CoRR, abs/2004.10934, 2020.
  * [4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European Conference on Computer Vision, pages 213–229, 2020\.
  * [5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In Computer Vision - ECCV 2020 - 16th European Conference, volume 12346 of Lecture Notes in Computer Science, pages 213–229. Springer, 2020.
  * [6] Miaobin Cen and Cheolkon Jung.  Fully convolutional siamese fusion networks for object tracking.  In 2018 IEEE International Conference on Image Processing, ICIP 2018, pages 3718–3722. IEEE, 2018.
  * [7] Ding-Jie Chen, He-Yen Hsieh, and Tyng-Luh Liu.  Adaptive image transformer for one-shot object detection.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, pages 12247–12256. Computer Vision Foundation / IEEE, 2021.
  * [8] Hao Chen, Yali Wang, Guoyou Wang, and Yu Qiao.  LSTD: A low-shot transfer detector for object detection.  In Sheila A. McIlraith and Kilian Q. Weinberger, editors, Proceedings of the AAAI conference on artificial intelligence, pages 2836–2843. AAAI Press, 2018.
  * [9] Tung-I Chen, Yueh-Cheng Liu, Hung-Ting Su, Yu-Cheng Chang, Yu-Hsiang Lin, Jia-Fong Yeh, and Winston H. Hsu.  Should I look at the head or the tail? dual-awareness attention for few-shot object detection.  CoRR, abs/2102.12152, 2021.
  * [10] Xianyu Chen, Ming Jiang, and Qi Zhao.  Leveraging bottom-up and top-down attention for few-shot object detection.  CoRR, abs/2007.12104, 2020.
  * [11] Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen.  UP-DETR: unsupervised pre-training for object detection with transformers.  CoRR, abs/2011.09094, 2020.
  * [12] Navneet Dalal and Bill Triggs.  Histograms of oriented gradients for human detection.  In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), pages 886–893. IEEE Computer Society, 2005.
  * [13] Berkan Demirel, Ramazan Gokberk Cinbis, and Nazli Ikizler-Cinbis.  Zero-shot object detection by hybrid region embedding.  In British Machine Vision Conference 2018, BMVC 2018, page 56. BMVA Press, 2018.
  * [14] Xuanyi Dong, Liang Zheng, Fan Ma, Yi Yang, and Deyu Meng.  Few-example object detection with model communication.  IEEE Trans. Pattern Anal. Mach. Intell., 41(7):1641–1654, 2019\.
  * [15] Qi Fan, Wei Zhuo, Chi-Keung Tang, and Yu-Wing Tai.  Few-shot object detection with attention-rpn and multi-relation detector.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, pages 4012–4021. Computer Vision Foundation / IEEE, 2020.
  * [16] Pedro F. Felzenszwalb, David A. McAllester, and Deva Ramanan.  A discriminatively trained, multiscale, deformable part model.  In 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2008). IEEE Computer Society, 2008.
  * [17] Kun Fu, Tengfei Zhang, Yue Zhang, and Xian Sun.  OSCD: A one-shot conditional object detection framework.  Neurocomputing, 425:243–255, 2021.
  * [18] Jiyang Gao, Jiang Wang, Shengyang Dai, Li-Jia Li, and Ram Nevatia.  NOTE-RCNN: noise tolerant ensemble RCNN for semi-supervised object detection.  In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, pages 9507–9516. IEEE, 2019.
  * [19] Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun.  YOLOX: exceeding YOLO series in 2021.  CoRR, abs/2107.08430, 2021.
  * [20] Ross Girshick.  Fast r-cnn.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
  * [21] Ross B. Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, pages 580–587. IEEE Computer Society, 2014.
  * [22] Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui.  Zero-shot detection via vision and language knowledge distillation.  CoRR, abs/2104.13921, 2021.
  * [23] Dikshant Gupta, Aditya Anantharaman, Nehal Mamgain, Sowmya Kamath S., Vineeth N. Balasubramanian, and C. V. Jawahar.  A multi-space approach to zero-shot object detection.  In IEEE Winter Conference on Applications of Computer Vision, WACV 2020, pages 1198–1206. IEEE, 2020.
  * [24] Guangxing Han, Shiyuan Huang, Jiawei Ma, Yicheng He, and Shih-Fu Chang.  Meta faster R-CNN: towards accurate few-shot object detection with attentive feature alignment.  CoRR, abs/2104.07719, 2021.
  * [25] Nasir Hayat, Munawar Hayat, Shafin Rahman, Salman H. Khan, Syed Waqas Zamir, and Fahad Shahbaz Khan.  Synthesizing the unseen for zero-shot object detection.  In Computer Vision - ACCV 2020 - 15th Asian Conference on Computer Vision, volume 12624 of Lecture Notes in Computer Science, pages 155–170. Springer, 2020.
  * [26] Ting-I Hsieh, Yi-Chen Lo, Hwann-Tzong Chen, and Tyng-Luh Liu.  One-shot object detection with co-attention and co-excitation.  In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, pages 2721–2730, 2019.
  * [27] Hanzhe Hu, Shuai Bai, Aoxue Li, Jinshi Cui, and Liwei Wang.  Dense relation distillation with context-aware aggregation for few-shot object detection.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, pages 10185–10194. Computer Vision Foundation / IEEE, 2021.
  * [28] Takafumi Kanamori, Shohei Hido, and Masashi Sugiyama.  A least-squares approach to direct importance estimation.  J. Mach. Learn. Res., 10:1391–1445, 2009.
  * [29] Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell.  Few-shot object detection via feature reweighting.  In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, pages 8419–8428. IEEE, 2019.
  * [30] Leonid Karlinsky, Joseph Shtok, Amit Alfassy, Moshe Lichtenstein, Sivan Harary, Eli Schwartz, Sivan Doveh, Prasanna Sattigeri, Rogério Feris, Alex M. Bronstein, and Raja Giryes.  Starnet: towards weakly supervised few-shot object detection.  In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, pages 1743–1753. AAAI Press, 2021.
  * [31] Leonid Karlinsky, Joseph Shtok, Sivan Harary, Eli Schwartz, Amit Aides, Rogério Schmidt Feris, Raja Giryes, and Alexander M. Bronstein.  Repmet: Representative-based metric learning for classification and few-shot object detection.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, pages 5197–5206. Computer Vision Foundation / IEEE, 2019.
  * [32] Siddhesh Khandelwal, Raghav Goyal, and Leonid Sigal.  Weakly-supervised any-shot object detection.  CoRR, abs/2006.07502, 2020.
  * [33] Siddhesh Khandelwal, Raghav Goyal, and Leonid Sigal.  Unit: Unified knowledge transfer for any-shot object detection and segmentation.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, pages 5951–5961. Computer Vision Foundation / IEEE, 2021.
  * [34] Geonuk Kim, Honggyu Jung, and Seong-Whan Lee.  Few-shot object detection via knowledge transfer.  In 2020 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2020, pages 3564–3569. IEEE, 2020.
  * [35] Piotr Koniusz, Hongguang Zhang, and Fatih Porikli.  A deeper look at power normalizations.  In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, pages 5774–5783. Computer Vision Foundation / IEEE Computer Society, 2018.
  * [36] Aoxue Li and Zhenguo Li.  Transformation invariant few-shot object detection.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, pages 3094–3102. Computer Vision Foundation / IEEE, 2021.
  * [37] Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu.  High performance visual tracking with siamese region proposal network.  In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, pages 8971–8980. Computer Vision Foundation / IEEE Computer Society, 2018.
  * [38] Bohao Li, Boyu Yang, Chang Liu, Feng Liu, Rongrong Ji, and Qixiang Ye.  Beyond max-margin: Class margin equilibrium for few-shot object detection.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, pages 7363–7372. Computer Vision Foundation / IEEE, 2021.
  * [39] Pengyang Li, Yanan Li, and Donghui Wang.  Class-incremental few-shot object detection.  CoRR, abs/2105.07637, 2021.
  * [40] Qianzhong Li, Yujia Zhang, Shiying Sun, Xiaoguang Zhao, Kang Li, and Min Tan.  Rethinking semantic-visual alignment in zero-shot object detection via a softplus margin focal loss.  Neurocomputing, 449:117–135, 2021.
  * [41] Shaoqi Li, Wenfeng Song, Shuai Li, Aimin Hao, and Hong Qin.  Meta-retinanet for few-shot object detection.  In 31st British Machine Vision Conference 2020, BMVC 2020. BMVA Press, 2020.
  * [42] Xiang Li, Lin Zhang, Yau Pun Chen, Yu-Wing Tai, and Chi-Keung Tang.  One-shot object detection without fine-tuning.  CoRR, abs/2005.03819, 2020.
  * [43] Yanan Li, Pengyang Li, Han Cui, and Donghui Wang.  Inference fusion with associative semantics for unseen object detection.  In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, pages 1993–2001. AAAI Press, 2021.
  * [44] Yanan Li, Yilan Shao, and Donghui Wang.  Context-guided super-class inference for zero-shot detection.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR Workshops 2020, Seattle, WA, USA, June 14-19, 2020, pages 4064–4068. Computer Vision Foundation / IEEE, 2020.
  * [45] Yiting Li, Haiyue Zhu, Yu Cheng, Wenxin Wang, Chek Sing Teo, Cheng Xiang, Prahlad Vadakkepat, and Tong Heng Lee.  Few-shot object detection via classification refinement and distractor retreatment.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, pages 15395–15403. Computer Vision Foundation / IEEE, 2021.
  * [46] Zhihui Li, Lina Yao, Xiaoqin Zhang, Xianzhi Wang, Salil S. Kanhere, and Huaxiang Zhang.  Zero-shot object detection with textual descriptions.  In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, pages 8690–8697. AAAI Press, 2019.
  * [47] Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In IEEE International Conference on Computer Vision, ICCV 2017, pages 2999–3007. IEEE Computer Society, 2017.
  * [48] Weidong Lin, Yuyan Deng, Yang Gao, Ning Wang, Jinghao Zhou, Lingqiao Liu, Lei Zhang, and Peng Wang.  CAT: cross-attention transformer for one-shot object detection.  CoRR, abs/2104.14984, 2021.
  * [49] Longyao Liu, Bo Ma, Yulin Zhang, Xin Yi, and Haozhi Li.  Afd-net: Adaptive fully-dual network for few-shot object detection.  In MM ’21: ACM Multimedia Conference, pages 2549–2557. ACM, 2021.
  * [50] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  Ssd: Single shot multibox detector.  In European conference on computer vision, pages 21–37. Springer, 2016.
  * [51] Ruotian Luo, Ning Zhang, Bohyung Han, and Linjie Yang.  Context-aware zero-shot recognition.  In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, pages 11709–11716. AAAI Press, 2020.
  * [52] Qiao-mei Ma, Chong Wang, Shenghao Yu, Ye Zheng, and Yuqi Li.  Zero-shot object detection with attributes-based category similarity.  IEEE Trans. Circuits Syst. II Express Briefs, 67-II(5):921–925, 2020.
  * [53] Claudio Michaelis, Ivan Ustyuzhaninov, Matthias Bethge, and Alexander S. Ecker.  One-shot instance segmentation.  CoRR, abs/1811.11507, 2018.
  * [54] Ishan Misra, Abhinav Shrivastava, and Martial Hebert.  Watch and learn: Semi-supervised learning of object detectors from videos.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, pages 3593–3602. IEEE Computer Society, 2015.
  * [55] Anton Osokin, Denis Sumin, and Vasily Lomakin.  OS2D: one-stage one-shot object detection by matching anchor features.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 - 16th European Conference, volume 12360 of Lecture Notes in Computer Science, pages 635–652, 2020.
  * [56] Juan-Manuel Pérez-Rúa, Xiatian Zhu, Timothy M. Hospedales, and Tao Xiang.  Incremental few-shot object detection.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 13843–13852. Computer Vision Foundation / IEEE, 2020.
  * [57] Limeng Qiao, Yuxuan Zhao, Zhiyuan Li, Xi Qiu, Jianan Wu, and Chi Zhang.  Defrcn: Decoupled faster R-CNN for few-shot object detection.  CoRR, abs/2108.09017, 2021.
  * [58] Shafin Rahman, Salman H. Khan, and Nick Barnes.  Transductive learning for zero-shot object detection.  In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, pages 6081–6090. IEEE, 2019.
  * [59] Shafin Rahman, Salman H. Khan, and Nick Barnes.  Improved visual-semantic alignment for zero-shot object detection.  In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, pages 11932–11939. AAAI Press, 2020.
  * [60] Shafin Rahman, Salman H. Khan, Nick Barnes, and Fahad Shahbaz Khan.  Any-shot object detection.  In Computer Vision - ACCV 2020 - 15th Asian Conference on Computer Vision, volume 12624 of Lecture Notes in Computer Science, pages 89–106. Springer, 2020.
  * [61] Shafin Rahman, Salman H. Khan, Nick Barnes, and Fahad Shahbaz Khan.  Any-shot object detection.  In Computer Vision - ACCV 2020 - 15th Asian Conference on Computer Vision, volume 12624 of Lecture Notes in Computer Science, pages 89–106. Springer, 2020.
  * [62] Shafin Rahman, Salman H. Khan, and Fatih Porikli.  Zero-shot object detection: Joint recognition and localization of novel concepts.  Int. J. Comput. Vis., 128(12):2979–2999, 2020.
  * [63] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  CoRR, abs/1804.02767, 2018.
  * [64] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
  * [65] Ignacio Rocco, Relja Arandjelovic, and Josef Sivic.  End-to-end weakly-supervised semantic alignment.  In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, pages 6917–6925. Computer Vision Foundation / IEEE Computer Society, 2018.
  * [66] Pau Rodríguez, Issam H. Laradji, Alexandre Drouin, and Alexandre Lacoste.  Embedding propagation: Smoother manifold for few-shot classification.  In Computer Vision - ECCV 2020 - 16th European Conference, volume 12371 of Lecture Notes in Computer Science, pages 121–138. Springer, 2020.
  * [67] Bryan C. Russell, Antonio Torralba, Kevin P. Murphy, and William T. Freeman.  Labelme: A database and web-based tool for image annotation.  Int. J. Comput. Vis., 77(1-3):157–173, 2008.
  * [68] Yilan Shao, Yanan Li, and Donghui Wang.  Zero-shot detection with transferable object proposal mechanism.  In 2019 IEEE International Conference on Image Processing, ICIP 2019, Taipei, Taiwan, September 22-25, 2019, pages 3666–3670. IEEE, 2019.
  * [69] Cem Subakan, Mirco Ravanelli, Samuele Cornell, Mirko Bronzi, and Jianyuan Zhong.  Attention is all you need in speech separation.  In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021, pages 21–25. IEEE, 2021.
  * [70] Bo Sun, Banghuai Li, Shengcai Cai, Ye Yuan, and Chi Zhang.  FSCE: few-shot object detection via contrastive proposal encoding.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, pages 7352–7362. Computer Vision Foundation / IEEE, 2021.
  * [71] Peng Tang, Xinggang Wang, Xiang Bai, and Wenyu Liu.  Multiple instance detection network with online instance classifier refinement.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, pages 3059–3067. IEEE Computer Society, 2017.
  * [72] Yao Teng and Limin Wang.  Structured sparse R-CNN for direct scene graph generation.  CoRR, abs/2106.10815, 2021.
  * [73] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola.  Rethinking few-shot image classification: a good embedding is all you need?  In Computer Vision–ECCV 2020, pages 266–282, 2020.
  * [74] Paul A. Viola and Michael J. Jones.  Rapid object detection using a boosted cascade of simple features.  In 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2001), pages 511–518. IEEE Computer Society, 2001.
  * [75] Kang Wang, Lu Zhang, Yifan Tan, Jiajia Zhao, and Shuigeng Zhou.  Learning latent semantic attributes for zero-shot object detection.  In 32nd IEEE International Conference on Tools with Artificial Intelligence, ICTAI 2020, pages 230–237. IEEE, 2020.
  * [76] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794–7803, 2018.
  * [77] Xin Wang, Thomas E. Huang, Joseph Gonzalez, Trevor Darrell, and Fisher Yu.  Frustratingly simple few-shot object detection.  In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, volume 119 of Proceedings of Machine Learning Research, pages 9919–9928. PMLR, 2020.
  * [78] Yu-Xiong Wang, Deva Ramanan, and Martial Hebert.  Meta-learning to detect rare objects.  In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, pages 9924–9933. IEEE, 2019.
  * [79] Aming Wu, Yahong Han, Linchao Zhu, Yi Yang, and Cheng Deng.  Universal-prototype augmentation for few-shot object detection.  CoRR, abs/2103.01077, 2021.