# Explore the Power of Synthetic Data on Few-shot Object Detection

Shaobo Lin, Kun Wang, Xingyu Zeng, Rui Zhao
Sensetime Research
<EMAIL_ADDRESS>
###### Abstract

Few-shot object detection (FSOD) aims to expand an object detector for novel
categories given only a few instances for training. The few training samples
restrict the performance of FSOD model. Recent text-to-image generation models
have shown promising results in generating high-quality images. How applicable
these synthetic images are for FSOD tasks remains under-explored. This work
extensively studies how synthetic images generated from state-of-the-art text-
to-image generators benefit FSOD tasks. We focus on two perspectives: (1) How
to use synthetic data for FSOD? (2) How to find representative samples from
the large-scale synthetic dataset? We design a copy-paste-based pipeline for
using synthetic data. Specifically, saliency object detection is applied to
the original generated image, and the minimum enclosing box is used for
cropping the main object based on the saliency map. After that, the cropped
object is randomly pasted on the image, which comes from the base dataset. We
also study the influence of the input text of text-to-image generator and the
number of synthetic images used. To construct a representative synthetic
training dataset, we maximize the diversity of the selected images via a
sample-based and cluster-based method. However, the severe problem of high
false positives (FP) ratio of novel categories in FSOD can not be solved by
using synthetic data. We propose integrating CLIP, a zero-shot recognition
model, into the FSOD pipeline, which can filter 90$\%$ of FP by defining a
threshold for the similarity score between the detected object and the text of
the predicted category. Extensive experiments on PASCAL VOC and MS COCO
validate the effectiveness of our method, in which performance gain is up to
21.9$\%$ compared to the few-shot baseline.

##  1 Introduction

In recent years, we have tremendous progress in object detection [34, 33, 24,
2, 4]. However, the impressive performance of these models relies on a large
amount of annotated data. Therefore, the detectors cannot generalize well to
novel categories, especially when the annotated data are scarce. In contrast,
humans can learn to recognize or detect a novel object with only a few labeled
examples. Few-shot object detection (FSOD), which simulates this way, has
attracted increasing attention. In FSOD, an object detector that is trained
using base categories with sufﬁcient data (base images) can learn to detect
novel categories using only a few annotated images (novel images).

Figure 1: How to utilize synthetic data for FSOD? (a) The few-shot novel
categories, including bird, bus, cow, motorbike, and sofa, are from the
dataset of PASCAL VOC split 1. (b) The synthetic data from Stable Diffusion.
(c) The saliency detection map of the synthetic data. (d) The cropped novel
instances are based on the minimum enclosing box of the saliency map. (e)
Combining synthetic instances and the randomly selected background from the
base dataset.

The few training samples restrict the performance of FSOD model. In FSOD, the
number of objects for each novel category is $K$ for $K$-shot detection.
Existing literature always controls the number of novel data ($K$-shot), to
carefully compare the performance of different methods. While this is helpful
to drive research on sophisticated algorithms, it is unhelpful for the
industry to enhance its few-shot based applications by incorporating external
synthetic data. With the recent development of generative models, the text-to-
image generator has made a great process. For example, DALL.E [32], Imagen
[36], and Stable Diffusion [35] can generate high-quality images by simply
using the input text description. These generators can produce diverse
outcomes, implying a brighter future for industrial applications such as
tackling many existing few-shot or long-tailed issues. This encourages us to
explore the impact of synthetic data from text-to-image generators on FSOD
tasks. Our study is carried out on the open-sourced Stable Diffusion. By using
the generated images, we define a new setting of $(K+G)$-shot for the problem
of using synthetic data for few-shot learning, consisting of $K$ real novel
instances and $G$ generated novel instances. Two key questions must be
answered: (1) How to use synthetic data for FSOD? (2) How to find
representative samples from the large-scale synthetic dataset?

How to use synthetic data for FSOD? In Figure 1, the original FSOD dataset
from PASCAL VOC split1 is annotated with the bounding boxes and categories.
Since the synthetic images from Stable Diffusion lack annotations, we cannot
utilize these data directly. Copy-Paste [28, 9, 23] is a simple and effective
data augmentation strategy. Randomly pasting generated novel instances onto
background images can generate a combinatorial number of training data for
free. However, the original generated images can’t be pasted onto the
background image. The reason is that using the generated images, including the
non-negligible background regions and the inaccurate bounding boxes, leads to
performance degradation shown in Table 4. To solve this problem, saliency
object detection is applied to the original generated images, and the minimum
enclosing box is used for cropping the main object. After that, the cropped
object is randomly down-sized and pasted on the image randomly selected from
the base dataset. Besides, we design and compare several input text formats,
finding that the performance of different text formats is related to the few-
shot settings, shown in Figure 3. Intuitively, FSOD can benefit from more
training data by increasing $G$ in the $(K+G)$-shot setting. However, using
more synthetic data can not continuously improve the FSOD model in our work.
We find that using 20 synthetic images works best in most few-shot cases.

Since we can obtain a large amount of generated images from a text-to-image
generator, finding the representative samples is necessary. Authenticity and
diversity are two factors for evaluating the quality of a generated dataset.
We focus on maximizing the diversity of training dataset for selecting the
representative synthetic data because the authenticity of the images from
Stable Diffusion is remarkable. We introduce the sample-based and cluster-
based (k-means [13] / spectral clustering [26]) methods to build a high-
diversity dataset. Specifically, the sample-based method applies uniform
sampling to the data feature of each category. The cluster-based method uses a
clustering algorithm to separate the data feature of each category into
several sub-spaces and select one sample in each sub-space to maximize the
diversity of training dataset. We select the sample nearest to the cluster
centroid for each sub-space. CLIP [31] is used as the feature extractor for
getting the data feature. However, the FP of novel categories is prominent, in
which the base categories are often recognized as novel ones. The problem of
high FP ratio of novel categories can not be solved by synthetic data, shown
in Figure 4. CLIP is a zero-shot model that uses natural language to retrieve
related images. We propose to integrate CLIP into the FSOD model for filtering
the FP. By computing the similarity score between the detected object from the
FSOD model and the name list of categories, we can remove the proposals whose
similarity score with the predicted category is lower than the pre-defined
threshold.

Our key contributions can be summarized as: (1) To the best of our knowledge,
we are the first to leverage the external synthetic novel data from a text-to-
image generator for FSOD, showing significant application potential. (2) We
explore how to utilize the synthetic novel data, including the copy-paste
pipeline, the design of input text, the number of used synthetic images, and
the data selection strategy for discovering the representative synthetic
samples. (3) Our method significantly improves multiple baselines and achieves
state-of-the-art performance on PASCAL VOC and MS COCO.

Figure 2: The pipeline of using synthetic data for FSOD. (a) During the data
generation, the novel dataset is generated by Stable Diffusion. (b) In the
data selection step, the generated data is processed by the selection module
to find representative samples for training. We use the cluster-based method
as an example. After that, a saliency detection is applied to the selected
samples for cropping the novel instances. The synthetic images, added to the
few-shot set for training the few-shot model, are obtained by randomly pasting
novel instances onto the randomly selected background images. The Few-shot
model can be any current few-shot object detector. (c) For FP retrieval, the
similarity of the object feature from the image encoder of CLIP and the text
feature from the text encoder of CLIP is computed. The score thresh filters
the FP based on the similarity score.

##  2 Related Works

###  2.1 Few-shot Object Detection

FSOD is an important yet unsolved task in computer vision. Some works use
meta-learning [8, 18, 44, 40, 21], where a meta-learner is introduced to
acquire class-agnostic meta-knowledge which is transferred to novel classes.
With the help of a meta learner that takes the support images as well as the
bounding box annotations as inputs, the feature re-weighting modules are
applied to a single-stage object detector (YOLOv2) [18] and a two-stage object
detector (Faster R-CNN) [44]. [21] uses a class margin equilibrium (CME)
approach, with the aim to optimize both feature space partition and novel
class reconstruction in a systematic way. Transformation invariant principle
[20] is proposed for various meta-learning models by introducing consistency
regularization on predictions from the transformed images. TFA [39] is a
simple two-stage fine-tuning approach, which significantly outperforms the
earlier meta-learning methods. MPSR [42] adopts multi-scale positive sample
refinement to handle scale variance problem. FSCE [38] aims to learn
contrastive-aware object proposal encodings that facilitate the classification
of detected objects. FADI [1] uses a two-step fine-tuning framework via
association and discrimination, which builds up a discriminative feature space
for each novel class with two integral steps. DeFRCN [29] extends Faster R-CNN
by using gradient decoupled layer for multi-stage decoupling and prototypical
calibration block for multi-task decoupling. Pseudo-Labelling [19] is proposed
to find previously unlabelled instances to obtain high-quality pseudo-
annotations for novel categories from the training dataset. MFDC [43] proposes
to learn three types of class-agnostic commonalities between base and novel
classes: recognition-related, localization-related and distribution
commonalities, which are able to be integrated into most of existing fine-
tuning based methods by performing distillation via a unified distillation
framework.

###  2.2 Text-to-Image Generators

Recently, diffusion models [37, 16] have become a promising generative
modeling framework, achieving state-of-the-art performance on image generation
tasks [5, 17]. DALL.E [32] proposes a two-stage model: a prior that generates
a CLIP [31] image embedding given a text caption and a decoder that generates
an image based on the image embedding. Imagen [36] is a text-to-image
diffusion model based on the power of language models and diffusion models.
Using cross-attention layers, Stable Diffusion [35] uses diffusion models to
synthesize high-resolution images for general conditioning inputs. GLIDE [27]
studies diffusion models for the text-conditional image synthesis by comparing
two guidance strategies: CLIP and classifier-free guidance. The synthetic
images generated from GLIDE are used for zero-shot, few-shot, and regular
image recognition tasks [15], showing the power of synthetic data on image
recognition. However, [15] does not consider the influence of the number of
synthetic data defined by $G$ in the $(K+G)$-shot setting. By comparing Stable
Diffusion to GLIDE in Table 9, we notice that the quality of synthetic data
from Stable Diffusion is superior to that of GLIDE. Therefore, we use Stable
Diffusion in our experiments.

##  3 Method

###  3.1 Preliminary

In FSOD, given a labeled base dataset $D_{B}$, there are $C_{B}$ base classes
with sufficient images in each class. Novel dataset $D_{N}$ with novel classes
$C_{N}$ consists of a few samples in each class. $C_{B}$ and $C_{N}$ do not
have overlapping categories. The number of instances for each class in $C_{N}$
is $K$ for $K$-shot detection. Using the generated images, we define a new
setting of $(K+G)$-shot for the problem of using synthetic data for FSOD,
which consists of $K$ real novel instances and $G$ generated novel instances.
There are two stages in FSOD methods [39, 38, 21]. In the pre-training stage,
the model is trained on base classes to obtain a robust feature
representation. In the fine-tuning stage, the pre-trained model is then fine-
tuned on a balanced few-shot set (or adding the synthetic novel data in our
setting), including both base and novel classes ($C_{B}\cup C_{N}$).

###  3.2 Our Pipeline

In Figure 2, there are three steps in our overall pipeline of using synthetic
data for FSOD: data generation, data selection, and FP retrieval.

Data Generation To study the influence of the input text of text-to-image
generator, several representative text descriptions for Stable Diffusion are
designed by following the format of prefix with the name of a category, shown
in Table 1. Specifically, We can simply use the class name as the input text
whose prefix is ”None”, or ”a / one” with the class name. Besides, we can
combine ”a / one” and ”photo / picture” or use some additional representative
adjectives as the prefix of input text. The comparison among these text
formats will be discussed in section 4.4.2. Using the input text, we can get
many synthetic images according to the category name. Because the entity of
the input text of Stable Diffusion is the name of a category, most of the
generated images only have one target.

Table 1: The design of prefixes in the input text, including seven different settings: None, a, one, a5, one5, real, and adj. Methods | Prefixs | Methods | Prefixs
---|---|---|---
None |  |  |
a | ”a” | one | ”one”
a5 | ”a” | one5 | ’one’
’a photo of ’ | ’a photo of one ’
’a photo of a ’ | ’a picture of one ’
’a picture of ’ | ’one photo of ’
’a picture of a ’ | ’one picture of ’
real | ’real ’ | adj | ’a photo of a good ’
’a real ’ | ’a photo of a large ’
’one real ’ | ’a photo of a nice ’
’a photo of a real ’ | ’a photo of a cool ’
’a photo of one real ’ | ’a photo of a clean ’

Data Selection The generated novel data from Stable Diffusion is processed by
the selection module to find the representative synthetic samples, which
should be high-quality and high-diversity. We focus on maximizing the
diversity of the selected data because the quality of the synthetic images is
high enough. The selection module is based on the CLIP similarity score or
image feature from CLIP. We propose two types of selection methods, including
sample-based and cluster-based methods.

$G_{n}=\\{G_{n}^{i}\\}_{i=1}^{N},\ R_{n}=\\{R_{n}^{i}\\}_{i=1}^{M}$ (1)

where $G_{n}$ includes the generated novel instances of category $n$.
$G_{n}^{i}$ is the $i$-th generated sample of category $n$. $R_{n}$ is the
real novel data of category $n$ from the few-shot dataset. $R_{n}^{i}$ is the
$i$-th real sample of category $n$. $N$ and $M$ are the numbers of generated
and real novel images, respectively.

The sample-based method is directly based on the CLIP score or image feature
of CLIP. As illustrated in Equ 2, the diversity of the generated dataset can
be measured by the CLIP score of the generated instances or the distance
between the CLIP features of generated and real novel instances. High-
diversity means using uniform sampling to maximize the difference among the
selected samples.

$\begin{split}R_{s}=\left\\{\begin{array}[]{l}Se(CLIP(G_{n},T))\\\
Se(cos(CLIP(G_{n}),CLIP(R_{n})))\end{array}\right.\end{split}$ (2)

where $T$ is the input text. By inputting the text and image, $CLIP$ outputs
the similarity score. If $T$ is not used, $CLIP$ means the image encoder of
CLIP, which processes the generated and real novel images for obtaining the
image features. $Se$ is the selection module for finding the representative
samples using uniform sampling. The distance between the generated images and
real images is computed by the cosine similarity. $R_{s}$ is the selected
samples based on the sample-based methods.

Using a clustering algorithm is the second way to find the representative
samples.

$C_{n}=Clu(CLIP(I_{n}),k_{n})$ (3)

where $C_{n}$ is the class center of category $n$. $Clu$ is the clustering
method. $k_{n}$ is the number of centers of category $n$. We use k-means or
spectral clustering to compute the class center. We find that using the
spectral clustering algorithm achieves the best performance.

$R_{c}=\arg\underset{i}{min}\ cos(C_{n},G_{n}^{i})_{i=1}^{N}$ (4)

We select the sample nearest to the $k_{n}$ cluster centroids $C_{n}$ as the
representative samples for category $n$. The distance is computed by the
cosine similarity. $R_{c}$ is the samples selected by the cluster-based
methods.

After finding the representative samples, we apply a saliency detection method
to these generated images. Then, the minimum enclosing box is used for
cropping the main object based on the saliency map. Finally, the cropped
object is down-sized and pasted on the randomly selected background from the
base dataset. These synthetic novel images are introduced into the original
few-shot set to obtain a new few-shot training dataset. The number of the
selected synthetic images, $G$ in the setting of $(K+G)$-shot, is a crucial
hyper-parameter for FSOD, which will be discussed in section 4.4.2.

FP Retrieval In our work, the FP of novel categories is prominent. However,
the problem of high FP ratio can not be solved by introducing synthetic data,
which is shown in Figure 4 in section 4.4.4. With the increase of generated
images, the FP ratio of the original few-shot model ($w/o$ CLIP) does not
decrease. We propose integrating the CLIP into the FSOD model for filtering
the FP. Formally, given $I_{n}$ which is the few-shot training dataset of the
novel category $n$, we use the FSOD model $Fs$ to get the bounding box
$bb_{n}$ and the corresponding confidence score $conf_{n}$.

$R_{n}=I_{n}[bb_{n}];conf_{n},bb_{n}=Fs(I_{n})$ (5)

where $R_{n}$ is the cropped region of $I_{n}$ based on $bb_{n}$, which is
predicted to category $n$ by $Fs$.

$score=\frac{e^{(cos(ImgE(R_{n}),TextE(t_{n})))}}{\sum_{i=1}^{L}e^{(cos(ImgE(R_{n}),TextE(t_{i})))}}$
(6)

where $ImgE$ and $TextE$ are the image encoder and text encoder of CLIP.
$ImgE$ gets the image feature of the detected object $R_{n}$. $TextE$ gets the
text feature of the input text $t_{n}$ of the novel category $n$. We compute
the cosine similarity of the image and text features. The input text $t_{i}$
$\in$ $(B\cup N)$ or $N$ means the base and novel categories or only novel
categories. $L$ is the length of the category list. In Table 8, we find that
the selection of the category list is related to the scale of the dataset,
i.e., PASCAL VOC and MS COCO. Finally, we use the softmax operation to get the
normalized similarity $score$. We remove the detection results of $R_{n}$ (FP)
whose $score$ is lower than the pre-defined CLIP threshold, which is 0.1 in
our experiment. The filtered results are used for final evaluation. With CLIP,
we can reduce the FP ratio up to 90 $\%$ shown in Figure 4.

##  4 Experiments

###  4.1 Datasets and Evaluation Protocols

We evaluate our methods on PASCAL VOC [7, 6] and MS COCO [25]. In PASCAL VOC,
we adopt the common strategy [34, 33] that using VOC 2007 test set for
evaluating while VOC 2007 and 2012 train/val sets are used for training.
Following [44], 5 out of its 20 object categories are selected as the novel
classes, while the remaining as the base classes. We evaluate with three
different novel/base splits from [44], named as split 1, split 2 and split 3.
Each split contains 15 base categories with abundant data and 5 novel
categories with K annotated instances for K = 1, 3, 5, 10. Following [44, 39,
38], we use the mean average precision (mAP) of novel categories at 0.5 IoU
threshold as the evaluation metric and report the results on the official test
set of VOC 2007. When using MS COCO, 20 out of 80 categories are reserved as
novel classes, the rest 60 categories are used as base classes. The detection
performance with COCO-style AP and AP75 for K = 1, 3, 5, 10, 30 shots of novel
categories are reported.

Table 2: Comparison with state-of-the-art few-shot object detection methods on VOC2007 test set for novel classes of the three splits. Black indicates state-of-the-art. Red is the improvement compared to the baseline. * means using our methods: the synthetic images, the selection strategy, and the CLIP model for filtering the FP. |  | split 1 | split 2 | split 3
---|---|---|---|---
Methods / Shots | 1 | 3 | 5 | 10 | 1 | 3 | 5 | 10 | 1 | 3 | 5 | 10
FRCN+ft [44] | ICCV2019 | 11.9 | 29 | 36.9 | 36.9 | 5.9 | 23.4 | 29.1 | 28.8 | 5.0 | 18.1 | 30.8 | 43.4
FRCN+ft-full [44] | ICCV2019 | 13.8 | 32.8 | 41.5 | 45.6 | 7.9 | 26.2 | 31.6 | 39.1 | 9.8 | 19.1 | 35 | 45.1
FR [18] | ICCV2019 | 14.8 | 26.7 | 33.9 | 47.2 | 15.7 | 22.7 | 30.1 | 40.5 | 21.3 | 28.4 | 42.8 | 45.9
MetaDet [40] | ICCV2019 | 18.9 | 30.2 | 36.8 | 49.6 | 21.8 | 27.8 | 31.7 | 43 | 20.6 | 29.4 | 43.9 | 44.1
Meta R-CNN [44] | ICCV2019 | 19.9 | 35 | 45.7 | 51.5 | 10.4 | 29.6 | 34.8 | 45.4 | 14.3 | 27.5 | 41.2 | 48.1
TFA [39] | ICML2020 | 39.8 | 44.7 | 55.7 | 56.0 | 23.5 | 34.1 | 35.1 | 39.1 | 30.8 | 42.8 | 49.5 | 49.8
MPSR [42] | ECCV2020 | 41.7 | 51.4 | 55.2 | 61.8 | 24.4 | 39.2 | 39.9 | 47.8 | 35.6 | 42.3 | 48 | 49.7
CME [21] | CVPR2021 | 41.5 | 50.4 | 58.2 | 60.9 | 27.2 | 41.4 | 42.5 | 46.8 | 34.3 | 45.1 | 48.3 | 51.5
FSCN [22] | CVPR2021 | 40.7 | 46.5 | 57.4 | 62.4 | 27.3 | 40.8 | 42.7 | 46.3 | 31.2 | 43.7 | 50.1 | 55.6
HallucFsDet [46] | CVPR2021 | 47 | 46.5 | 54.7 | 54.7 | 26.3 | 37.4 | 37.4 | 41.2 | 40.4 | 43.3 | 51.4 | 49.6
FSCE [38] | CVPR2021 | 44.2 | 51.4 | 61.9 | 63.4 | 27.3 | 43.5 | 44.2 | 50.2 | 37.2 | 47.5 | 54.6 | 58.5
UPE [41] | ICCV2021 | 43.8 | 50.3 | 55.4 | 61.7 | 31.2 | 41.2 | 42.2 | 48.3 | 35.5 | 43.9 | 50.6 | 53.5
QA-FewDet [10] | ICCV2021 | 42.4 | 55.7 | 62.6 | 63.4 | 25.9 | 46.6 | 48.9 | 51.1 | 35.2 | 47.8 | 54.8 | 53.5
Meta faster-rcnn [11] | AAAI2021 | 43 | 60.6 | 66.1 | 65.4 | 27.7 | 46.1 | 47.8 | 51.4 | 40.6 | 53.4 | 59.9 | 58.6
FADI [1] | NIPS2021 | 50.3 | 54.2 | 59.3 | 63.2 | 30.6 | 40.3 | 42.8 | 48 | 45.7 | 49.1 | 55 | 59.6
DeFRCN [29] | ICCV2021 | 53.6 | 61.5 | 64.1 | 60.8 | 30.1 | 47.0 | 53.3 | 47.9 | 48.4 | 52.3 | 54.9 | 57.4
FCT [12] | CVPR2022 | 49.9 | 57.9 | 63.2 | 67.1 | 27.6 | 43.7 | 49.2 | 51.2 | 39.5 | 52.3 | 57 | 58.7
KFSOD [45] | CVPR2022 | 44.6 | 54.4 | 60.9 | 65.8 | 37.8 | 43.1 | 48.1 | 50.4 | 34.8 | 44.1 | 52.7 | 53.9
Pseudo-Labelling [19] | CVPR2022 | 54.5 | 58.8 | 63.2 | 65.7 | 32.8 | 50.7 | 49.8 | 50.6 | 48.4 | 55.0 | 59.6 | 59.6
MFDC [43] | ECCV2022 | 63.4 | 67.7 | 69.4 | 68.1 | 42.1 | 53.4 | 55.3 | 53.8 | 56.1 | 59.0 | 62.2 | 63.7
DeFRCN* | Ours | 67.5 | 69.8 | 71.1 | 71.5 | 52 | 54.3 | 57.5 | 57.4 | 55.9 | 58.6 | 59.6 | 63.9
Improve | +13.9 | +8.3 | +7 | +10.7 | +21.9 | +7.3 | +4.2 | +9.5 | +7.5 | +6.3 | +4.7 | +6.5
MFDC* | Ours | 68.9 | 69.5 | 70.9 | 73.3 | 50.5 | 56.8 | 58.9 | 55.6 | 59.9 | 62.8 | 65.3 | 66.9
Improve | +5.4 | +1.8 | +1.5 | +5.2 | +8.4 | +3.4 | +3.6 | +1.8 | +3.8 | +3.8 | +3.1 | +3.2
Table 3: Few-shot object detection performance on MS COCO. Black indicates the state-of-the-art. Red is the improvement compared to the baseline. * means using our methods: the synthetic images, the selection strategy, and the CLIP model for filtering the FP. Methods / Shots | 1 | 3 | 5 | 10 | 30
---|---|---|---|---|---
| nAP |  nAP75 | nAP |  nAP75 | nAP |  nAP75 | nAP |  nAP75 | nAP |  nAP75
FR [18] | - | - | - | - | - | - | 5.6 | 4.6 | 9.1 | 7.6
Meta R-CNN [44] | - | - | - | - | - | - | 8.7 | 6.6 | 12.4 | 10.8
CME [21] | - | - | - | - | - | - | 15.1 | 16.4 | 16.9 | 17.8
FSCE [38] | - | - | - | - | - | - | 11.1 | 9.8 | 15.3 | 14.2
UPE [41] | - | - | - | - | - | - | 11 | 10.7 | 15.6 | 15.7
TFA [39] | 3.4 | 3.8 | 6.6 | 6.5 | 8.3 | 8.0 | 10 | 9.3 | 13.7 | 13.4
MSPR [42] | 2.3 | 2.3 | 5.2 | 5.1 | 6.7 | 6.4 | 9.8 | 9.7 | 14.1 | 14.2
QA-FewDet [10] | 4.9 | 4.4 | 8.4 | 7.3 | 9.7 | 8.6 | 10.2 | 9.0 | 11.5 | 10.3
Meta faster-rcnn [11] | 5.0 | 4.6 |  |  | 12.7 | 16.6 | 25.7 | 31.8 | 10.8 | 15.8
FADI [1] | 5.7 | 6.0 | 8.6 | 8.3 | 10.1 | 9.7 | 12.2 | 11.9 | 16.1 | 15.8
DeFRCN [29] | 6.5 | 6.9 | 13.4 | 13.6 | 15.3 | 14.6 | 18.5 | 17.6 | 22.6 | 22.3
FCT [12] | 5.6 | - | 11.1 | - | 14.0 | - | 17.1 | 17 | 21.4 | 22.1
KFSOD [45] | - | - | - | - | - | - | 18.5 | 18.7 | - | -
Pseudo-Labelling [19] | - | - | - | - | - | - | 17.8 | 17.8 | 24.5 | 25.0
MFDC [43] | 10.8 | 11.6 | 15 | 15.5 | 16.4 | 17.3 | 19.4 | 20.2 | 22.7 | 23.2
DeFRCN* | 18.1 | 19.5 | 19.2 | 20.2 | 19.8 | 20.4 | 20.7 | 21.3 | 23.1 | 23.9
Improve | +11.6 | +12.6 | +5.8 | +6.6 | +4.5 | +5.8 | +2.2 | +3.7 | +0.5 | +1.6
MFDC* | 18.1 | 19 | 18.6 | 19.4 | 19.5 | 20.1 | 20.3 | 20.9 | 22.5 | 23
Improve | +7.3 | +7.4 | +3.6 | +3.9 | +3.1 | +2.8 | +0.9 | +0.7 | -0.2 | -0.2
Table 4: The components of our methods based on DeFRCN. The AP50 of DeFRCN on 1 to 10-shot of PASCAL VOC split1 and MS-COCO is reported. DeFRCN (Our impl.) [29] | Add | Crop | Select | CLIP | 1-shot | 3-shot | 5-shot | 10-shot
---|---|---|---|---|---|---|---|---
on PASCAL VOC |  |  |  |  | 52.5 | 60.8 | 62.7 | 61.9
✓ |  |  |  | 51.9 | 60 | 61.9 | 63.6
✓ | ✓ |  |  | 62.2 | 65.5 | 66.5 | 67.8
|  |  | ✓ | 61.8 | 66.6 | 66.3 | 67.4
✓ | ✓ | ✓ |  | 63.5 | 66.3 | 67.5 | 69
✓ | ✓ | ✓ | ✓ | 67.5 | 69.8 | 71.1 | 71.5
on MS-COCO |  |  |  |  | 16.2 | 25.4 | 28.9 | 33.6
✓ | ✓ |  |  | 25.4 | 28 | 30.1 | 33.8
✓ | ✓ | ✓ |  | 27 | 30 | 32.3 | 34.7
| ✓ | ✓ | ✓ | ✓ | 29.2 | 32.2 | 33.1 | 35.1

###  4.2 Implementation Details

Our baselines include DeFRCN [29] and MFDC [43], which are representative
methods in FSOD. These methods use Faster R-CNN [34] with ResNet-101 [14],
which are the same as almost all FSOD methods. The training strategies of our
methods follow the selected baselines. The difference is that we add the
generated novel images to the few-shot set for training. The $G$ in the
setting of $(K+G)$-shot is 20. We adopt the ViT-B/32 [31] as the encoder of
CLIP.

###  4.3 Comparison with State-of-the-art Methods

We compare our approaches to several competitive FSOD methods. The results are
shown in Table 2 and Table 3. Following [11, 21, 42, 1, 19], we use a single
run with the same training images to get the results of different shots.

####  4.3.1 Results on PASCAL VOC and MS COCO.

Following [44, 39, 38], we provide the AP50 of the novel classes on PASCAL VOC
with three splits in Table 2. By using synthetic data, our methods can
outperform the baselines in all few-shot settings and achieve state-of-the-art
performance. The most obvious improvement is up to 21.9$\%$.

We report the COCO-style AP and AP75 of the 20 novel classes on MS COCO in
Table 3. By using synthetic data, our methods can achieve state-of-the-art
performance in almost all settings. The performance gain in the setting of
30-shot is low, since higher shots decline the influence of the synthetic
images from another source which is different from the data from the current
domain.

###  4.4 Ablation Study

Following [44, 38], we do ablation studies on PASCAL VOC split 1. We use
DeFRCN as the baseline.

####  4.4.1 The Components in Our Pipeline

There are four essential steps for applying synthetic data to FSOD: Add, Crop,
Select, and CLIP. In Table 4, Add copies the original synthetic images and
pasts them onto the background. Crop means using saliency detection for the
main target before the pasting operation. Select is finding the top 20
representative samples from the generated dataset. CLIP uses the encoders of
CLIP after the few-shot model to filter the FP. We discover that cropping is
crucial, which can achieve 4$\%$ to 10$\%$ AP50 improvement in PASCAL VOC,
since directly using the original synthetic data hurt the performance due to
the inaccurate bounding boxes. Our selection strategy can improve the
precision on most settings (up to 2$\%$ in PASCAL VOC). CLIP helps the model
to filter most of FP and improve the performance up to 4$\%$. The best
performance on MS COCO is also achieved by using all components.

Figure 3: The influence of different input text for Stable Diffusion and the number of synthetic data on the average AP50 of PASCAL VOC split1 1 to 10-shot. There are seven types of text descriptions: None, a, a5, one, one5, adj, and real. The X-axis is the number of generated images ($G$) for each novel category. Table 5: The type of several pasting methods. The results are averaged by using 10 to 200 generated images for training on 1 to 10-shot of PASCAL VOC split1. The saliency map is the result of saliency object detection. The segmentation map is the result of a supervised image segmentation method. Box uses the minimum enclosing rectangle of the saliency map. Methods | 1-shot | 3-shot | 5-shot | 10-shot
---|---|---|---|---
Box | 62.7 | 65.2 | 65.4 | 66.8
Saliency map | 60.4 | 63.1 | 63.9 | 65.5
Segmentation map | 61.5 | 63.9 | 64.2 | 66

####  4.4.2 The Factors in Data Generation and Selection

There are three factors: the format of input text description, the number of
used synthetic images (G in the setting of $(K+G)$-shot), and the type of
pasting generated novel instances.

In Figure 3, we analyze the influence of the number of used synthetic images
and the input text description for Stable Diffusion. As a result, we can get
the following conclusions. First, almost all input text formats perform best
with 20 synthetic images. Second, except for the format of ”one”, other
formats have relatively high performance on average. In detail, ”a5” and
”one5” achieve the highest performance on average. ”a5” has the most stable
curve. Therefore, in other experiments, we use ”a5” format with 20 synthetic
images per class. In Table 5, we compare three different pasting methods: box,
saliency map, and segmentation map, in which these types of novel instances
are pasted onto the background. The saliency map uses the results of BASNet
[30]. The segmentation map is from a supervised image segmentation method:
deeplabv3+ [3]. Box uses the minimum enclosing rectangle of the saliency map.
The results are averaged by using 10 to 200 generated images for training. In
conclusion, the box type achieves the best performance in all settings.

####  4.4.3 The Selection Module

Table 6 compares different synthetic data selection methods. All selection
methods choose 20 samples out of 200 generated images per class. The features
of samples (generated or original few-shot samples) are based on the image
encoder of CLIP. We find that only considering the quality of synthetic data
performs worse, such as CLIP$\\_$max, since this operation reduces the
diversity of data. Focusing on the high diversity of synthetic data can
improve the few-shot model, such as using a clustering or uniform sampling
method based on the CLIP score. The performance of Instance$\\_$uniform is not
satisfactory because there is a large deviation between the mean feature of
few-shot samples and the actual prototype of a category. The best performance
is achieved by spectral clustering.

Table 6: The comparison of different data selection methods. All selection methods choose 20 samples out of 200 generated images per class. The features of samples are obtained from the image encoder of CLIP. The results of the random selection are averaged by five different selections. Syn$\\_$max selects the top 20 samples which have the minimum distance to the mean features of all generated images. CLIP$\\_$max selects the samples with the top-20 CLIP score. Instance$\\_$max selects the nearest neighbor samples according to the mean feature of few-shot instances. CLIP$\\_$uniform is uniform sampling based on the CLIP score. Instance$\\_$uniform means uniform sampling based on the distance between the mean features of the few-shot instances and the features of generated images. Kmeans$\\_$cluster and Spectral$\\_$cluster get 20 clusters for each category to select the nearest neighbor sample for each cluster. Black is the best, and Red is the second best. Types | Methods | 1-shot | 3-shot | 5-shot | 10-shot
---|---|---|---|---|---
Others | Random | 62.6 | 65.9 | 66.8 | 68.2
Syn$\\_$max | 61.6 | 64.6 | 66.6 | 68.5
Quality-based |  CLIP$\\_$max | 59.9 | 64.2 | 65.9 | 67.2
Instance$\\_$max | 61.6 | 65.2 | 67.2 | 68.6
Sample-based |  CLIP$\\_$uniform | 63.4 | 66.1 | 66.7 | 68.6
Instance$\\_$uniform | 60.6 | 64.7 | 67.4 | 68.3
Cluster-based |  Kmeans$\\_$cluster | 62.6 | 66.6 | 66.6 | 67.9
Spectral$\\_$cluster | 63.5 | 66.3 | 67.5 | 69

Figure 4: The power of CLIP on reducing the number of FP. The generated images are added to the 1-shot setting of PASCAL VOC split1. w/o CLIP means the original few-shot model. w CLIP uses CLIP to filter the FP. The increase of the generated images used does not reduce the FP ratio. The FP ratio of using CLIP is much lower than that without CLIP. Table 7: The filtering thresh for reducing the FP by using CLIP. The CLIP thresh is applied to CLIP score, which is lower than the CLIP thresh is defined as FP. DeFRCN (Our impl.) [29] | Thresh | 1-shot | 3-shot | 5-shot | 10-shot
---|---|---|---|---|---
CLIP Thresh | 0.5 | 65.2 | 68.4 | 68.1 | 69.6
0.3 | 65.8 | 68.6 | 67.5 | 69.2
0.2 | 66.3 | 70.2 | 70 | 71.2
0.1 | 67.5 | 69.8 | 71.1 | 71.5
0 | 63.5 | 66.3 | 67.5 | 69
Table 8: The category list for CLIP. The AP50 of DeFRCN on 1 to 10-shot of PASCAL VOC split1 and MS-COCO is reported. The input of the text encoder of CLIP can be both the base and novel classes or the novel classes only. The experiments are conducted on PASCAL VOC and MS-COCO. DeFRCN (Our impl.) [29] | Type | 1-shot | 3-shot | 5-shot | 10-shot
---|---|---|---|---|---
on PASCAL VOC | On Novel | 62.2 | 66.3 | 65.7 | 68.1
On Base+Novel | 66.3 | 70.2 | 70 | 71.2
on MS-COCO | On Novel | 29.2 | 32.2 | 33.1 | 35.1
On Base+Novel | 27.4 | 29.1 | 30.8 | 31.5

####  4.4.4 How to Use CLIP

In Figure 4, increasing the generated images can not reduce the FP ratio. The
reduction of FP ratio by using CLIP is up to 90 $\%$. In Table 8, we compare
two category lists used as the input of the text-encoder of CLIP: novel and
base categories or only the novel categories. Using novel and base categories
means that CLIP calculates the similarity between the input image and the
input text of all categories(20 in PASCAL VOC and 80 in MS COCO). When using
novel categories, CLIP calculates the similarity between the input image and
the input text of novel categories(5 in PASCAL VOC and 20 in MS COCO). The
results show that the optimal length of the name list is 20, which is all
categories of PASCAL VOC and the novel categories of MS COCO. Table 7 shows
the effect of CLIP thresh on performance. The detected objects with the CLIP
score of the predicted categories, which are lower than the CLIP threshold,
are defined as false positives. These predictions are removed for final
evaluation. The optimal CLIP threshold is 0.1.

Figure 5: The influence of different types of additional images per category:
the images with the pasted synthetic novel instance (copy-paste S), the images
with the pasted real novel instance (copy-paste R), and the real novel images
(R). The results are obtained by adding these images to the original 1-shot
set of PASCAL VOC split1 for training the DeFRCN.

####  4.4.5 Other Analysis

In Figure 5, we study the upper bound of the performance of using synthetic
data. With the increase of synthetic data, copy-paste-based methods, such as
pasting real novel instances or synthetic instances, have a lower upper bound
than using the original novel images. We can notice that the gap between
pasting real and synthetic instances is marginal, which means the synthetic
images are ready for the FSOD tasks. Using more synthetic data can boost
accuracy when the number of data is lower than 50, but the gain does not
continue with the increase of generated data. The performance of FSOD is
mainly limited by the copy-paste itself, not the synthetic data.

We compare another text-to-image generator (GLIDE) to Stable Diffusion. The
results in Table 9 show that the quality of the text-to-image generator
affects the performance of FSOD.

Table 9: The comparison between different text-to-image generators: Stable Diffusion and GLIDE. The results of both methods are averaged by training five times. Twenty samples are randomly selected for each training. Methods | 1-shot | 3-shot | 5-shot | 10-shot
---|---|---|---|---
Stable Diffusion [35] | 62.6 | 65.9 | 66.8 | 68.2
GLIDE [27] | 60.3 | 64.7 | 64.1 | 66.1

##  5 Conclusion

In this work, we leverage the external synthetic novel data from a text-to-
image generator for FSOD. We focus on utilizing the synthetic novel data,
including the copy-paste pipeline, the design of input text for Stable
Diffusion, the number of used synthetic images, and the data selection
strategy for discovering the representative synthetic data. Furthermore, we
propose integrating CLIP into the FSOD model to solve the problem of high FP
ratio in FSOD. Extensive experiments on the few-shot object detection
datasets, i.e., PASCAL VOC and MS COCO, validate the effectiveness of our
method.

## Acknowledgement

This work is sponsored Hetao Shenzhen-Hong Kong Science and Technology
Innovation Cooperation Zone (HZQB-KCZYZ-2021045).

## References

  * [1] Yuhang Cao, Jiaqi Wang, Ying Jin, Tong Wu, Kai Chen, Ziwei Liu, and Dahua Lin.  Few-shot object detection via association and discrimination.  Advances in Neural Information Processing Systems, 34, 2021.
  * [2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * [3] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint arXiv:1706.05587, 2017.
  * [4] Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang.  Dynamic head: Unifying object detection heads with attentions.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7373–7382, 2021.
  * [5] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  * [6] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes challenge: A retrospective.  International journal of computer vision, 111(1):98–136, 2015.
  * [7] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International journal of computer vision, 88(2):303–338, 2010.
  * [8] Qi Fan, Wei Zhuo, Chi-Keung Tang, and Yu-Wing Tai.  Few-shot object detection with attention-rpn and multi-relation detector.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4013–4022, 2020.
  * [9] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph.  Simple copy-paste is a strong data augmentation method for instance segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2918–2928, 2021.
  * [10] Guangxing Han, Yicheng He, Shiyuan Huang, Jiawei Ma, and Shih-Fu Chang.  Query adaptive few-shot object detection with heterogeneous graph convolutional networks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3263–3272, 2021.
  * [11] Guangxing Han, Shiyuan Huang, Jiawei Ma, Yicheng He, and Shih-Fu Chang.  Meta faster r-cnn: Towards accurate few-shot object detection with attentive feature alignment.  arXiv preprint arXiv:2104.07719, 2021.
  * [12] Guangxing Han, Jiawei Ma, Shiyuan Huang, Long Chen, and Shih-Fu Chang.  Few-shot object detection with fully cross-transformer.  arXiv preprint arXiv:2203.15021, 2022.
  * [13] John A Hartigan and Manchek A Wong.  Algorithm as 136: A k-means clustering algorithm.  Journal of the royal statistical society. series c (applied statistics), 28(1):100–108, 1979.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [15] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi.  Is synthetic data from generative models ready for image recognition?  arXiv preprint arXiv:2210.07574, 2022.
  * [16] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  * [17] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans.  Cascaded diffusion models for high fidelity image generation.  J. Mach. Learn. Res., 23:47–1, 2022.
  * [18] Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell.  Few-shot object detection via feature reweighting.  In Proceedings of the IEEE International Conference on Computer Vision, pages 8420–8429, 2019.
  * [19] Prannay Kaul, Weidi Xie, and Andrew Zisserman.  Label, verify, correct: A simple few shot object detection method.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14237–14247, 2022.
  * [20] Aoxue Li and Zhenguo Li.  Transformation invariant few-shot object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3094–3102, 2021.
  * [21] Bohao Li, Boyu Yang, Chang Liu, Feng Liu, Rongrong Ji, and Qixiang Ye.  Beyond max-margin: Class margin equilibrium for few-shot object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7363–7372, 2021.
  * [22] Yiting Li, Haiyue Zhu, Yu Cheng, Wenxin Wang, Chek Sing Teo, Cheng Xiang, Prahlad Vadakkepat, and Tong Heng Lee.  Few-shot object detection via classification refinement and distractor retreatment.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15395–15403, 2021.
  * [23] Shaobo Lin, Kun Wang, Xingyu Zeng, and Rui Zhao.  An effective crop-paste pipeline for few-shot object detection.  arXiv preprint arXiv:2302.14452, 2023.
  * [24] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [25] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [26] Andrew Ng, Michael Jordan, and Yair Weiss.  On spectral clustering: Analysis and an algorithm.  Advances in neural information processing systems, 14, 2001.
  * [27] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2112.10741, 2021.
  * [28] Seulki Park, Youngkyu Hong, Byeongho Heo, Sangdoo Yun, and Jin Young Choi.  The majority can help the minority: Context-rich minority oversampling for long-tailed classification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6887–6896, 2022.
  * [29] Limeng Qiao, Yuxuan Zhao, Zhiyuan Li, Xi Qiu, Jianan Wu, and Chi Zhang.  Defrcn: Decoupled faster r-cnn for few-shot object detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8681–8690, 2021.
  * [30] Xuebin Qin, Zichen Zhang, Chenyang Huang, Chao Gao, Masood Dehghan, and Martin Jagersand.  Basnet: Boundary-aware salient object detection.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7479–7489, 2019.
  * [31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [32] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [33] Joseph Redmon and Ali Farhadi.  Yolo9000: better, faster, stronger.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7263–7271, 2017.
  * [34] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  IEEE transactions on pattern analysis and machine intelligence, 39(6):1137–1149, 2016.
  * [35] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  * [36] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv preprint arXiv:2205.11487, 2022.
  * [37] Yang Song and Stefano Ermon.  Improved techniques for training score-based generative models.  Advances in neural information processing systems, 33:12438–12448, 2020.
  * [38] Bo Sun, Banghuai Li, Shengcai Cai, Ye Yuan, and Chi Zhang.  Fsce: Few-shot object detection via contrastive proposal encoding.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7352–7362, 2021.
  * [39] Xin Wang, Thomas E Huang, Trevor Darrell, Joseph E Gonzalez, and Fisher Yu.  Frustratingly simple few-shot object detection.  arXiv preprint arXiv:2003.06957, 2020.
  * [40] Yu-Xiong Wang, Deva Ramanan, and Martial Hebert.  Meta-learning to detect rare objects.  In Proceedings of the IEEE International Conference on Computer Vision, pages 9925–9934, 2019.
  * [41] Aming Wu, Yahong Han, Linchao Zhu, and Yi Yang.  Universal-prototype enhancing for few-shot object detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9567–9576, 2021.
  * [42] Jiaxi Wu, Songtao Liu, Di Huang, and Yunhong Wang.  Multi-scale positive sample refinement for few-shot object detection.  In European Conference on Computer Vision, pages 456–472. Springer, 2020.
  * [43] Shuang Wu, Wenjie Pei, Dianwen Mei, Fanglin Chen, Jiandong Tian, and Guangming Lu.  Multi-faceted distillation of base-novel commonality for few-shot object detection.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part IX, pages 578–594. Springer, 2022.
  * [44] Xiaopeng Yan, Ziliang Chen, Anni Xu, Xiaoxi Wang, Xiaodan Liang, and Liang Lin.  Meta r-cnn: Towards general solver for instance-level low-shot learning.  In Proceedings of the IEEE International Conference on Computer Vision, pages 9577–9586, 2019.
  * [45] Shan Zhang, Lei Wang, Naila Murray, and Piotr Koniusz.  Kernelized few-shot object detection with efficient integral aggregation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19207–19216, 2022.
  * [46] Weilin Zhang and Yu-Xiong Wang.  Hallucination improves few-shot object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13008–13017, 2021.


# Mitigation Strategies for 42Ar/42K Background Reduction using Encapsulation
with Ultra-Pure Plastic for the LEGEND Experiment

M. Ibrahim Mirza for the LEGEND Collaboration1<EMAIL_ADDRESS>1Department of Physics and Astronomy, The University of Tennessee, Knoxville,
TN 37996, United States

###### Abstract

Neutrinoless double-beta (0$\nu\beta\beta$) decay is the most compelling
approach to determine the Majorana nature of neutrino and measure effective
Majorana neutrino mass. The LEGEND collaboration is aiming to look for
0$\nu\beta\beta$ decay of 76Ge with unprecedented sensitivity. If underground-
sourced argon is not available, the cosmogenically-induced isotope 42Ar and
its decay progeny 42K in the liquid argon active veto could create a
challenging background for the 0$\nu\beta\beta$ signal. We are studying
methodologies to mitigate the 42K background. In order to achieve this,
encapsulation of germanium detectors with 3D-printed technologies using low
background material are currently under investigation. Simulation results of
Poly(ethylene 2,6- naphthalate) (PEN) encapsulation of germanium detectors and
plans to study other potential materials are presented.

##  I Introduction

The Large Enriched Germanium Experiment for Neutrinoless $\beta\beta$ Decay
(LEGEND) is a ultra low radioactive background experiment designed to detect
0$\nu\beta\beta$ decay signal in a 76Ge (Qββ = 2039 keV) [1]. Performance
parameters of the 200 kg and 1000 kg phases of the LEGEND are listed in table
1. Backgrounds in the region of interest (ROI) can be produced by detector
components such as the detector mounts, front end electronics, cables, optical
fibers, re-entrant tube, liquid argon, cryostat, among other sources.

Experiment | LEGEND-200 | LEGEND-1000
---|---|---
Status | In commissioning | Planned
Location | LNGS | LNGS / SNOLAB
Isotope |  76Ge |  76Ge
Total mass | 200 kg | 1000 kg
Exposure | 1 t yr | 10 t yr
Background Index | 2$\times$10-4 cts/(keV kg yr) | < 1$\times$10-5 cts/(keV kg yr)
Discovery Sensitivity | 1027 yr | 1.3 $\times$ 1028 yr
Live time | 5 yr | 10 yr
Majorana neutrino mass | < 34 - 78 meV | < 9 - 21 meV
(discovery sensitivity) |  |
Table 1: The specifications for the two experimental phases of the LEGEND
program.

The work presented here is focused on the concepts and techniques to mitigate
the background caused by argon, i.e. the small concentration of 42Ar and its
daughter isotope 42K in atmospheric argon.

###  I.1 42Ar / 42K BACKGROUND IN LEGEND

Ge detectors are immersed in liquid argon in the LEGEND cryostat. For LEGEND,
liquid argon serves as (i) coolant: because Ge detectors cannot be fully
depleted at room temperature, therefore, they must operate at cryogenic
temperature, (ii) veto for background events: liquid argon is a scintillator,
therefore, by detecting scintillation light near Ge detectors, many background
signals can be effectively suppressed. However, natural argon contains a small
amount of 42Ar, which in turn produces 42K, a source of background. GERDA
Phase II [2] and DEAP-3600 [3] reported the presence of 42K in liquid argon.
nEXO [4] also reported the study of 42K in liquid xenon.
Origin of 42Ar & 42K in a atmospheric argon: Gaseous argon in the upper
atmosphere is constantly bombarded with cosmic rays. When a cosmogenically
produced $\alpha$ particle interacts with atmospheric argon, it produces a
small concentration of 42Ar via the reaction, 40Ar + $\alpha$ $\rightarrow$
42Ar + p + p. Liquid argon is normally extracted from the atmosphere, so 42Ar
is present. 42Ar is a long lived radioactive isotope, with a half-life of 32.9
y. It undergoes $\beta^{-}$ decay via, 42Ar $\rightarrow$ 42K+ \+ $\beta^{-}$
\+ $\overline{\nu_{e}}$, with Qβ = 0.599 MeV. The $\beta$ decay spectrum is
shown in figure 1 (left). Since Qβ $<$ Qββ, 42Ar does not cause a background
at the ROI for the LEGEND experiment.
Additionally, the daughter isotope of 42Ar is 42K, which is also radioactive,
with a short half-life of 12.36 h. It undergoes $\beta^{-}$ decay via, 42K+
$\rightarrow$ 42Ca++ \+ $\beta^{-}$ \+ $\overline{\nu_{e}}$, with Qβ = 3.525
MeV. The $\beta$ decay spectrum is shown in figure 1 (right). Since Qβ $>$ Qββ
the $\beta$ emitted in the 42K decay could cause a background at the ROI.
42K+ can be attracted by the electric field of the detectors and as a result,
accumulate on the Ge surface. Thermally driven convection in liquid argon also
affects the dynamics of both neutral (42Ar) and charged (42K+, $\beta^{-}$)
particles. Backgrounds from 42K decay are expected to comprise 30-40$\%$ of
LEGEND backgrounds in both phases of the experiment. Therefore, it is
essential to reduce the concentration of 42Ar in the argon veto, or otherwise
mitigate this background.

Figure 1: Left: The energy distribution of 42Ar $\beta^{-}$ decay. Right: The
energy distribution of 42K $\beta^{-}$ decay. The red band depicts the ROI.
Data is taken from Ref. [5].

Underground argon: The most promising approach to dramatically reduce 42Ar
backgrounds is the use of underground argon (UGLAr) i.e. argon extracted from
deep underground sources. UGLAr is exposed to a much lower flux of cosmic
particles, with no mechanism for 42Ar production by high-energy $\alpha$
particles. We plan to coordinate with the DarkSide-20k effort [6] to use the
URANIA plant to produce UGLAr extracted from CO2 wells in Colorado, which will
then be transferred to the ARIA plant or other facilities for additional
chemical purification. The LEGEND-1000 project is in negotiations to obtain
approximately 18 tonnes of UGLAr. However, given the high risk associated with
this background, the collaboration is investigating additional strategies to
suppress the 42Ar background.
Alternative methods to mitigate 42Ar background: Several techniques to
suppress the 42Ar background from argon are being studied by the
collaboration: (i) encapsulation of Ge detectors, (ii) slightly depleting
atmospheric sourced argon of 42Ar, (iii) enclosing the detector strings with
ultra-pure nylon shroud with reduced argon volume as in LEGEND-200, (iv)
fabricating Ge detectors with thicker n+ dead layers, (v) doping liquid argon
with non-quenching impurities to neutralize 42K ions, (vi) better pulse-shape
discrimination of $\beta$-induced events.
Many of these techniques can be used to reduce backgrounds whether or not
UGLAr is available. Among the above options in this article we focus on the
encapsulation of the Ge detectors with low radioactivity scintillating
material.

##  II Encapsulation of the Germanium Detector

GERDA phase II data shows the viability of encapsulation for 42K background
suppression [2, 7]. GERDA used the transparent nylon mini shrouds to limit the
volume of argon that could drift into the vicinity of the Ge detectors.
However, to completely eliminate the possibility for ions to drift toward the
Ge detectors, we need to provide a complete encapsulation. 392 Ge detectors
with a total mass of 1000 kg of 76Ge will be deployed in the LEGEND-1000
cryostat. All of the Ge detectors will have different sizes and shapes, so
capsules would need to be custom built for every detector. 3D printing is a
viable solution for encapsulation to meet these demands and Oak Ridge National
Laboratory (ORNL) facilities are equipped with state of the art 3D printers
well suited for R&D studies. Custom-produced PEN is another material
considered for encapsulation.

###  II.1 Simulation Results of Encapsulation

Poly(ethylene 2,6- naphthalate) (PEN) is a transparent plastic scintillator
that has proven its capability as a wavelength shifter from vacuum ultra
violet (VUV) into the blue region of the visible spectrum [8, 9]. PEN have a
same mechanical stability as copper at cryogenic temperature and LEGEND-200
employs PEN as a baseplate to support each Ge detector.
Due to it’s favorable characteristics, we consider a encapsulation design made
entirely of PEN. Using GEANT4 [10], we simulate the geometry of PEN
encapsulated Ge. A layer of PEN is deposited over the top cross-section of the
Ge detector and surrounded with liquid argon, as shown in figure 2 (left). At
the center of the PEN disk, a source of electrons with energy of 3.525 MeV,
equal to Qβ of 42K is simulated. We select the direction of the particle
momenta to be towards the PEN. We calculated the energy deposition by the
electrons in Ge when varying the PEN thickness, as shown in figure 2 (middle).
The simulation shows that 8 mm of PEN is required to reduce the energy of
3.525 MeV electrons to below the ROI. This is a lot of passive material next
to Ge detector and can create another source of radioactive background. This
simulation, however, does not take into account the dead layer present on the
surface of the Ge detectors (1-2 mm thick) or the role of pulse-shape
rejection of near-surface events. Figure 2 (right) shows the energy deposited
in a 9 cm thick Ge detector and 2 mm thick PEN plate, from the incident
electron. From this simulation we can conclude that 2 mm of scintillation
material next to the Ge detector can efficiently tag products of 42K decay. We
show this simulation as an illustrative example and not necessarily a viable
encapsulation design.

Figure 2: Left: simulation of the detector geometry, with red representing the
trajectories of the electrons in PEN and green representing the resulting
gammas trajectories in Ge. Middle: simulation results of energy deposition in
Ge, where the blue band depicts the ROI. Right: simulated energy deposition in
Ge (red) and PEN (blue).

##  III R&D for Encapsulation

R&D for encapsulation of Ge detectors is currently ongoing at ORNL. The
selection of material for encapsulation is under investigation. An ideal
material for encapsulation should be: (i) radiopure, (ii) 3D printable, (iii)
transparent, (iv) scintillating, (v) mechanically stable at cryogenic
temperature, (vi) does not contaminate liquid argon by outgassing. In an
attempt to choose the appropriate material for encapsulation, we have compiled
a database of resins based on their physical properties. We are also
considering tests of resins used for medical and dental purposes, as such
resins have relatively high tensile strength compared to the resins used for
engineering designs.

###  III.1 Ongoing and upcoming activities for encapsulation

As part of our R&D plan, we will perform the outgassing, radiopurity, optical,
mechanical and cryogenic tests for the encapsulation material, which are
described as follows:
Outgassing measurements: The outgassing measurement identifies the molecular
species present inside the encapsulation sample. UTK-ORNL jointly commissioned
a setup for outgassing measurements at ORNL. We have assembled a residual gas
analyzer (RGA) connected to a vacuum chamber, as shown in figure 3. The
encapsulation materials are placed in the vacuum chamber. Prior to the
outgassing measurements, a chamber pressure of around 5 $\times$ 10-7 Torr is
maintained by the combination of roughing and turbo pumps. Gaseous molecules
present in the encapsulation sample will then move out and enter into the RGA.

Figure 3: Recently assembled residual gas analyzer mounted on the outgassing
vacuum chamber in the Physics Division at ORNL

Radiopurity measurements: In an attempt to meet LEGEND background goal, any
supplementary materials used in the detector must have an ultra-low
concentration of the U-238 and Th-232 decay chains and must not introduce
impurities into the liquid argon, which can quench scintillation light. Hence
the next step after the outgassing measurements would be an evaluation of
radioactive impurities in the sample. We will perform ICP-MS and gamma ray
spectrometry. The material having highest level of radiopurity will be
selected as a potential candidate for encapsulation tests of Ge detectors.
Optical measurements: The encapsulation would leverage the liquid argon veto
system for the LEGEND-1000 experiment. Therefore, it is essential to determine
the optical properties of the encapsulation material. Every material under
consideration will undergo optical testing. We intend to perform several
measurements of the optical response of the encapsulation material, including
photon yield, refractive index, attenuation length, peak emission wavelength
and reflectivity. Existing equipment for optical reflectivity measurements at
ORNL will be used, and we will establish an extensive optical test setup.
Mechanical measurements: The encapsulation material will remain in the
cryogenic environment of LEGEND-1000 for at least 10 years. Therefore, the
structural integrity of the material is crucial and every material under
considerations will also undergo extensive mechanical testing. We will test
mechanical performance after the optical testing. Measurement of the tensile
strength of each sample will be determined. For this purpose we intend to
establish a mechanical testing setup.
Cryogenic measurements: Ge detectors will immersed in a liquid argon in the
LEGEND-1000 cryostat. Therefore, the encapsulation samples will undergo
cryogenic testing in liquid argon. We are currently planning to custom-build a
cryostat with a VUV optical readout at ORNL. We will test the encapsulated Ge
detector in an atmospheric argon, enriched with 42Ar. We are considering
artificially producing 42Ar via, 40Ar(7Li3+, $\alpha$p)42Ar reaction [11].

###### Acknowledgements.

We would like to acknowledge support by LEGEND Collaboration, including the
ORNL LEGEND group and Physics Division. This work is supported by the U.S.
DOE, and the NSF, the LANL, ORNL and LBNL LDRD programs; the European ERC and
Horizon programs; the German DFG, BMBF, and MPG; the Italian INFN; the Polish
NCN and MNiSW; the Czech MEYS; the Slovak SRDA; the Swiss SNF; the UK STFC;
the Russian RFBR; the Canadian NSERC and CFI; the LNGS and SURF facilities.

## References

  * Abgrall  _et al._ [2021] N. Abgrall _et al._ (LEGEND), arXiv:2107.11462 (2021).
  * Agostini  _et al._ [2020] M. Agostini _et al._ (GERDA), Phys. Rev. Lett. 125, 252502 (2020).
  * Ajaj  _et al._ [2019] R. Ajaj _et al._ (DEAP), Phys. Rev. D 100, 072009 (2019).
  * Adhikari  _et al._ [2022] G. Adhikari _et al._ (nEXO), J. Phys. G: Nucl. Part. Phys. 49 015104 (2022).
  * [5] _Nuclear Data Services, International Atomic Energy Agency, https://www-nds.iaea.org/_.
  * Aalseth  _et al._ [2018] C. E. Aalseth _et al._ (DarkSide-20k), Eur. Phys. J. Plus 133, 131 (2018), arXiv:1707.08145 [physics.ins-det] .
  * Lubashevskiy  _et al._ [2018] A. Lubashevskiy _et al._ , Eur. Phys. J. C 78: 15; (2018).
  * Manzanillas  _et al._ [2022] L. Manzanillas _et al._ , arXiv:2204.13747 (2022).
  * Kuźniak  _et al._ [2019] M. Kuźniak _et al._ , Eur. Phys. J. C 79, 291 (2019).
  * Agostinelli  _et al._ [2003] S. Agostinelli _et al._ , Nucl. Instrum. Meth. A 506, 250-303 (2003).
  * Pelczar [2016] K. Pelczar, Ph.D. thesis, Jagiellonian University. in Krakow (2016).


# Edge states and the $\eta$ invariant

Rodrigo Fresneda, Lucas de Souza and Dmitri Vassilevich
Center of Mathematics, Computation and Cognition, Universidade Federal do ABC
09210-580, Santo André, SP, Brazil Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We propose a relation between the $\eta$ invariant on a manifold with
boundary, the $\eta$ invariants of edge states, and the $\eta$ invariant in an
infinite volume limit. With the example of planar fermions with bag and chiral
bag boundary conditions we show that this relation holds whenever edge states
are sufficiently well-localized near the boundary. As a by-product we show
that the spectrum of edge modes for chiral bag boundary conditions is linear
but bounded.

##  1 Introduction

The $\eta$ invariant plays an important role in Quantum Field Theory. It
describes the parity anomaly [1, 2, 3] which leads to a Chern–Simons term in
the effective action for gauge fields and to a Hall type conductivity in
condensed matter applications. The Jackiw–Rebbi fermion number fractionization
[4] can be also described through the $\eta$ invariant, see [2, 5].

If there are boundaries, the $\eta$ invariant has specific boundary
contributions. For example, on four-dimensional manifolds these contributions
take a form of Chern–Simons actions on the three-dimensional boundary manifold
[6, 7]. Much attention has been paid to the relations between $\eta$
invariants in the bulk and $\eta$ invariants in effective boundary theories.
Important results in this direction were obtained [8] in the context of
anomaly inflow, where an expression for $\eta$ in terms of boundary states was
derived. However, many problems still remain open. The boundary conditions
used in [8] (and earlier in [9]) are non-hermitian and thus cannot be used for
the Dirac Hamiltonians and for the problem of fermion number fractionization.
In [10], the $\eta$ invariant for local boundary conditions leading to a
hermitian Dirac operator were expressed through the anomalies in effective
boundary theories. However, only massless Dirac operators do not admitting any
boundary states were considered in the work [10]. (For completeness, we also
mention a work [11] which considered domain walls rather than boundaries.)

The purpose of this work is to derive the relations between the $\eta$
invariant on a manifold with boundary and the boundary (edge) modes in
theories with hermitian Dirac operators.

Let us give some main definitions. For more details we refer the reader to
[12]. Let $H$ be a hermitian Dirac type operator on a smooth manifold
$\mathcal{M}$, $\dim\mathcal{M}=n$, with a smooth boundary
$\partial\mathcal{M}$ with eigenvalues $\lambda$. The $\eta$ function of this
operator is defined as

$\eta(s,H)=\sum_{\lambda>0}\lambda^{-s}-\sum_{\lambda<0}(-\lambda)^{-s},$ (1)

where $s$ is a complex parameter. The sum in (1) is convergent if $\Re s$ is
large enough and defines a meromorphic function in the whole complex plane.
The $\eta(0,H)$ measures the spectral asymmetry of $H$. Note that zero
eigenvalues of $H$ are not included in (1). The Atiyah–Patodi–Singer $\eta$
invariant is defined as
$\eta_{H}=\tfrac{1}{2}(\eta(0,H)+\dim\,\mathrm{Ker}(H))$. This invariant jumps
by $\pm 1$ whenever an eigenvalue crosses the origin. The exponentiated $\eta$
invariant $\mathcal{E}(H)=\exp(-2\pi\mathrm{i}\eta_{H})$ is smooth.

Let $t$ be a positive real parameter. Then, there is an asymptotic expansion
at $t\to+0$ of the heat kernel

$\mathrm{Tr}\,\left(Q\exp(-tH^{2})\right)=\sum_{k=0}^{\infty}t^{\frac{k-n}{2}}a_{k}(Q,H^{2})$
(2)

for any smooth matrix-valued function $Q$. The heat kernel coefficients are
very well studied in the literature, see e.g. [13]. Let $\delta H$ be a
variation of $H$. We assume that $\delta H$ is a matrix-valued function rather
than a differential operator. If due to this variation no eigenvalue passes
through $0$, the corresponding variation of the $\eta(0,H)$ reads [3, 14, 15]

$\delta\eta(0,H)=-\frac{2}{\sqrt{\pi}}a_{n-1}(\delta H,H^{2}).$ (3)

This formula will be our main technical tool. We will study exclusively the
perturbative (smooth) contributions to $\eta(0,H)$. This is equivalent to
saying that we will be interested in the logarithm of the exponentiated $\eta$
invariant.

The heat kernel expansion and the formula (3) are valid provided the boundary
conditions are strongly elliptic. Without discussing this requirement in
detail, we just mention that all boundary conditions used in this work are
strongly elliptic. Let us also note that (3) can be used for arbitrary smooth
variations on compact manifolds while on noncompact manifolds the variations
have to be localized and thus cannot change the asymptotic behaviour of
background fields.

We are going to argue that in some situations $\eta(0)$ can be represented as
a sum of a bulk part and a contribution of boundary states. It is not easy to
separate the bulk and boundary contributions in general, especially since in
many cases $\eta(0)$ is topological and may be presented as a volume integral
of a topological density as well as a surface integral. To make this
separation transparent, we propose the following scheme. Consider a Dirac type
operator on $\mathbb{R}^{n}$ such that all background fields entering this
operator are localized near the origin. (Example: a gauge field with field
strength rapidly decaying away of the origin). We introduce a spherical
boundary $S^{n-1}$ of a radius $r$ and impose some boundary conditions. We
expect that if $r$ is sufficiently large the $\eta$ invariant on the ball
splits into two parts, one being the $\eta$ invariant for $\mathbb{R}^{n}$
without boundaries while the other part is given by contribution of the edge
states. The edge states are the eigenstates of Dirac Hamiltonian which decay
exponentially fast as functions of the distance from the boundary. The decay
rate has to be sufficiently fast to ensure localization of the states in a
small neighbourhood of the boundary.

In this paper, we check the proposed relation between $\eta$ functions with
the simplest yet nontrivial example of planar fermions in an external magnetic
field. In the next Section, we consider bag boundary conditions, we show that
the edge states are localized near the boundary, and we confirm the relation.
In Section 3, we generalize boundary conditions to the chiral bag boundary
conditions. We demonstrate that the edge states have a non-Dirac spectrum
which is linear but bounded. Besides, near the threshold the edge states are
not localized leading to a violations of the relation between $\eta$
functions.

##  2 Bag boundary conditions

Consider a plane $\mathbb{R}^{2}$ pierced by a perpendicular magnetic field.
We assume that this field has a finite flux and is concentrated somewhere near
the origin. The Dirac Hamiltonian for fermions confined to the plane reads

$H=\alpha^{j}(\mathrm{i}\partial_{j}-eA_{j})-\beta m,$ (4)

where $\alpha^{j}$ and $\beta$ are hermitian $2\times 2$ matrices satisfying

$\mathrm{tr}\,\left(\alpha^{k}\alpha^{j}\beta\right)=-2\mathrm{i}\epsilon^{kj},\qquad\alpha^{j}\alpha^{k}+\alpha^{k}\alpha^{j}=2g^{jk},$
(5)

with $\epsilon^{jk}$ and $g^{jk}$ being the Levi-Civita tensor and the metric
tensor, respectively. Besides, $\beta\alpha^{j}+\alpha^{j}\beta=0$ and
$\beta^{2}=1$.

The famous Niemi–Semenoff [2] result for infinite plane is

$\eta(0,H)=-\frac{e}{4\pi}\,\frac{m}{|m|}\int_{\mathbb{R}^{2}}\mathrm{d}^{2}x\,\epsilon^{jk}F_{jk}\,.$
(6)

Let us consider a disk $D_{r}$ of radius $r$ centred at the origin. Let
$\mathbf{n}$ be a unit inward pointing normal to the boundary $\partial
D_{r}=S^{1}_{r}$, $\theta$ be a coordinate on $S^{1}_{r}$, and
$\sqrt{h}\mathrm{d}\theta=r\mathrm{d}\theta$ be the induced integration
measure. We introduce

$\eta_{r}(0,H):=\frac{e}{2\pi}\,\frac{m}{|m|}\int_{S_{r}^{1}}\mathrm{d}\theta\,\sqrt{h}\,A_{j}\epsilon^{\mathbf{n}j}$
(7)

so that

$\eta(0,H)=\lim_{r\to\infty}\eta_{r}(0,H).$ (8)

We stress that $\eta_{r}$ is not an $\eta$ function of any operator.

Let us compute the $\eta$ function for $H_{D_{r}}$ which is $H$ restricted to
$D_{r}$ with boundary conditions

$\Pi_{-}\psi|_{S^{1}_{r}}=0$ (9)

with

$\Pi_{\pm}=\tfrac{1}{2}(1\pm\mathrm{i}\varepsilon\beta\alpha^{\mathbf{n}}),$
(10)

where $\varepsilon=\pm 1$. We also define

$\chi:=\Pi_{+}-\Pi_{-},\qquad\Pi_{\pm}=\frac{1}{2}(1\pm\chi).$ (11)

The boundary conditions (9), (10) are known under the name of bag boundary
conditions [16, 17] in the physics literature and under the name of Clifford
boundary conditions in Mathematics [18]. For these conditions, the normal
component of fermion current, $\psi^{\dagger}\alpha^{\mathbf{n}}\psi$,
vanishes at all point of the boundary, so that $H$ is selfadjoint.

For any Dirac type operator ${H}$ on a two-dimensional manifold $\mathcal{M}$
with bag boundary conditions on $\partial\mathcal{M}$

$a_{1}(Q,H^{2})=\frac{1}{8\sqrt{\pi}}\int_{\partial\mathcal{M}}\mathrm{d}\theta\,\sqrt{h}\,\mathrm{tr}\,(Q\chi).$
(12)

By specifying $H=H_{D_{r}}$, $Q=\delta H_{D_{r}}=-e\alpha^{j}A_{j}$ and using
the formula (3) for $n=2$, we obtain

$\delta\eta(0,H_{D_{r}})=\frac{\epsilon
e}{2\pi}\int_{S^{1}_{r}}\mathrm{d}\theta\,\sqrt{h}\,\delta
A_{j}\epsilon^{\mathbf{n}j},$ (13)

which is integrated to

$\eta(0,H_{D_{r}})=\frac{\epsilon
e}{2\pi}\int_{S^{1}_{r}}\mathrm{d}\theta\,\sqrt{h}\,A_{j}\epsilon^{\mathbf{n}j}.$
(14)

Let us analyse the edge states. We fix the gauge $A_{\mathbf{n}}=0$ near the
boundary and consider an auxiliary eigenvalue problem

$\hat{H}\varphi=\lambda\varphi$ (15)

on $\mathbb{R}_{+}$ where the operator $\hat{H}$ is constructed from $H$ in
the following way. Let $e^{\|}$ be a unit vector tangential to the boundary,
so that $\bigl{(}\alpha^{\|}\bigr{)}^{2}=1$ with
$\alpha^{\|}:=e_{j}^{\|}\alpha^{j}$. The corresponding coordinate on the
circle is $x^{\|}=r\theta$. The function $\varphi$ is assumed to be an
eigenfunction of $\mathrm{i}\partial_{\|}-eA_{\|}$ with an eigenvalue $\xi$.
At large $r$, the extrinsic curvature of the boundary is negligible. Thus, the
eigenvalue problem (15) reads

$\displaystyle\mathrm{i}\alpha^{\mathbf{n}}(\partial_{\mathbf{n}}+q(\lambda,m,\xi))\varphi=0,$
(16) $\displaystyle
q(\lambda,m,\xi))=-\mathrm{i}\alpha^{\mathbf{n}}(\alpha^{\|}\xi-\beta
m-\lambda)\,.$ (17)

$\varphi$ depends on a single coordinate $x^{\mathbf{n}}$ and satisfies the
boundary condition

$\Pi_{-}\varphi(x^{\mathbf{n}}=0)=0\,.$ (18)

Edge states correspond to

$\displaystyle\varphi\to 0\quad\mbox{at}\quad x^{\mathbf{n}}\to\infty,$ (19)

which means that the eigenspace of $q(\lambda,m,\xi)$ corresponding to a
positive eigenvalue should coincide with the kernel of $\Pi_{-}$. To resolve
this condition, let us take a particular representation of $\alpha^{j}$ and
$\beta$,

$\beta=\begin{pmatrix}1&0\\\
0&-1\end{pmatrix},\qquad\alpha^{\mathbf{n}}=\begin{pmatrix}0&1\\\
1&0\end{pmatrix},\qquad\alpha^{\|}=\begin{pmatrix}0&\mathrm{i}\\\
-\mathrm{i}&0\end{pmatrix}.$ (20)

With this choice, $\epsilon^{\mathbf{n}\|}=+1$.

We have,

$q(\lambda,m,\xi)=\begin{pmatrix}-\xi&\mathrm{i}(\lambda-m)\\\
\mathrm{i}(\lambda+m)&\xi\end{pmatrix}.$ (21)

The eigenvalues of this matrix are $\pm\mu$ with
$\mu=\sqrt{\xi^{2}-\lambda^{2}+m^{2}}$ while

$\begin{pmatrix}-\xi+\sqrt{\xi^{2}-\lambda^{2}+m^{2}}\\\
\mathrm{i}(\lambda+m)\end{pmatrix}$ (22)

is an eigenvector corresponding to the positive eigenvalue of
$q(\lambda,m,\xi)$.

The kernel of the projector

$\Pi_{-}=\dfrac{1}{2}\begin{pmatrix}1&-\mathrm{i}\varepsilon\\\
\mathrm{i}\varepsilon&1\end{pmatrix}$ (23)

is spanned by the vector

$\begin{pmatrix}-1\\\ \mathrm{i}\varepsilon\end{pmatrix}.$ (24)

The vectors (22) and (24) define the same linear subspace iff

$\dfrac{m}{|m|}=-\varepsilon\quad\mbox{and}\quad\lambda=\varepsilon\xi\,.$
(25)

The fist equation in (25) defines the boundary condition when edge state
exist, while the second one gives the spectrum of Dirac Hamiltonian $H$ on
these modes. Thus we conclude that the restriction of $H$ to edge states reads

$H_{\mathrm{b}}=\varepsilon(\mathrm{i}\partial_{\|}-eA_{\|}).$ (26)

The dependence of edge states on $x^{\mathbf{n}}$ is defined by the positive
eigenvalue of $q$. Due to the second conditions in (25), this eigenvalue is
$\mu=|m|$, so that whenever edge modes exist they decay as
$e^{-|m|x^{\mathbf{n}}}$ and are well localized relatively to $r$ as long as
$|m|r$ is sufficiently large.

The boundary Hamiltonian $H_{\mathrm{b}}$ is a one-dimensional Dirac type
operator. The variation of $\eta(0,H_{\mathrm{b}})$ can be calculated by using
(3) for $n=1$,

$\displaystyle\delta\eta(0,H_{\mathrm{b}})=-\frac{2}{\sqrt{\pi}}a_{0}(\delta
H_{\mathrm{b}},H_{\mathrm{b}}^{2})=-\frac{2}{\sqrt{\pi}}\,\frac{1}{\sqrt{4\pi}}\int_{S^{1}_{r}}\mathrm{d}x^{\|}\,\delta
H_{\mathrm{b}}$
$\displaystyle\qquad=\frac{e\varepsilon}{\pi}\int_{S^{1}_{r}}\mathrm{d}x^{\|}\delta
A_{\|}=\frac{\varepsilon
e}{\pi}\int_{S^{1}_{r}}\mathrm{d}\theta\sqrt{h}\,\delta
A_{\theta}\epsilon^{\mathbf{n}\theta}.$ (27)

The variation (27) is integrated to

$\eta(0,H_{\mathrm{b}})=\frac{\varepsilon
e}{\pi}\int_{S^{1}_{r}}\mathrm{d}\theta\sqrt{h}\,A_{\theta}\epsilon^{\mathbf{n}\theta}.$
(28)

If $m/|m|=\varepsilon$ there are no boundary states and the expressions
$\eta_{r}(0,H)$ coincides with $\eta(0,H_{D_{r}})$. If $m/|m|=-\varepsilon$
the two $\eta$ functions have opposite signs but their difference is
compensated by the contribution of boundary modes (28). In general,

$\eta(0,H_{D_{r}})=\eta_{r}(0,H)+\eta(0,H_{\mathrm{b}}).$ (29)

The second term on the right-hand side of (29) is zero if there are no
boundary modes. At $r\to\infty$, this is exactly the relation which has been
announced in the Introduction. We stress again that in (29) we relate smooth
parts of the $\eta$ functions. In other words, our claim is that variations of
both parts agree as long as eigenvalues do not cross the origin.

##  3 More general boundary conditions

A natural generalization of the boundary conditions considered in the previous
Section consists in the modification of $\chi$ in (11) as

$\chi=\mathrm{i}\varepsilon\beta e^{\tau\beta}\alpha^{\mathbf{n}},$ (30)

with $\tau$ being a real parameter. These conditions are the chiral bag
boundary conditions [19] (modulo the replacement of a Dirac Hamiltonian by a
Euclidean Dirac operator). For chiral bag boundary conditions, the normal
component of fermion current vanishes and $H$ is selfadjoint. Moreover, these
boundary conditions are strongly elliptic [20] so that the heat kernel
expansion (2) exists and the formula (3) for the variation of $\eta(0,H)$ is
still valid. Although the relation (12) is not hold for $\chi$ given in (30),
it has been demonstrated [21] that the smooth part of $\eta(0,H_{D_{r}})$ does
not depend on $\tau$. Therefore, Eq. (14) can still be used. All what remains
is to analyse the edge states. The calculations are similar to that of the
previous Section, so that we do not give details here.

The edge states exist if

$-\varepsilon(\lambda\sinh(\tau)+m\cosh(\tau))\geq 0$ (31)

while the dispersion relation is given by

$\lambda=\frac{\varepsilon\xi}{\cosh(\tau)}-m\tanh(\tau).$ (32)

We see, that the spectrum of boundary Hamiltonian given by (32) is linear, but
edge states exist for any signs of $\varepsilon$ and $m$, while the spectrum
is bounded either above or below.

For Eq. (29) to hold, $\eta(0,H_{b})$ must vanish when $m$ and $\varepsilon$
are of the same sign. We are not aware of any method of computations of $\eta$
invariants for operators with the spectrum restricted as in (31). If we
neglect this restriction for a while,
$H_{b}=\varepsilon(\cosh(\tau))^{-1}(\mathrm{i}\partial_{\|}-eA_{\|})-m\tanh(\tau)$.
This is an operator of Dirac type which can be treated along the same lines as
the boundary Hamiltonian in the previous Section. One thus obtains a non-zero
value of $\eta(0,H_{b})$ which depends on $A_{\|}$ and $m$. It is hard to
conceive that imposing the restriction (31) (which make the spectrum even more
asymmetric) can kill the spectral asymmetry. In fact, Eq. (29) does not need
to hold for chiral bag boundary conditions. The positive eigenvalue of $q$
which defines the decay rate of edge states reads
$|\lambda\sinh(\tau)+m\cosh(\tau)|$. Near the threshold of the spectrum (31)
the decay rate is very small, so that the edge modes are _not_ well localized.

##  4 Conclusions and discussion

In this paper, we have shown with two examples that $\eta(0,H)$ can be
represented as a sum of a bulk part and a contribution from the edge states
whenever the latter ones are localized in a sufficiently small neighbourhood
of the boundary. This gives a new interpretation of the classical result [2]
and opens an avenue for future research. A rigorous demonstration of this
relation for generic Dirac operators is a complicated problem in Mathematics,
which we are going to address in the near future.

As a by-product we derived the spectrum of edge modes for chiral bag boundary
conditions. This spectrum reminds us of the Fermi arc boundary states in Weyl
semimetals [22] which also occupy a bounded region in the momentum space and
points out to new physical applications.

Our results may be applied to calculations of fermion fractionization in the
presence of solitons. This quantity is usually a function of asymptotic values
of background fields (though there is an exception [23]) so that the formula
(3) is not applicable. Having an expression for charge fractionization through
$\eta$ functions on compact manifolds may simplify the calculations.

Finally, we would like to mention some publications which treated spectral
properties and anomalies of Dirac operator and with bag [24, 25, 26, 27],
chiral bag [28] and even some other boundary conditions [29, 30]. The setup
used in these papers was quite different from ours.

### Acknowledgements.

This work was supported in parts by the São Paulo Research Foundation
(FAPESP), grant 2021/10128-0. Besides, the work of LS was supported by the
grant 2022/09068-6 of FAPESP, and DV was supported by the National Council for
Scientific and Technological Development (CNPq), grant 304758/2022-1.

## References

  * [1] A. N. Redlich, “Parity Violation and Gauge Noninvariance of the Effective Gauge Field Action in Three-Dimensions,” Phys. Rev. D29 (1984) 2366–2374.
  * [2] A. J. Niemi and G. W. Semenoff, “Axial Anomaly Induced Fermion Fractionization and Effective Gauge Theory Actions in Odd Dimensional Space-Times,” Phys. Rev. Lett. 51 (1983) 2077\.
  * [3] L. Alvarez-Gaume, S. Della Pietra, and G. W. Moore, “Anomalies and Odd Dimensions,” Annals Phys. 163 (1985) 288\.
  * [4] R. Jackiw and C. Rebbi, “Solitons with Fermion Number 1/2,” Phys. Rev. D 13 (1976) 3398–3409.
  * [5] A. J. Niemi and G. W. Semenoff, “Fermion Number Fractionization in Quantum Field Theory,” Phys. Rept. 135 (1986) 99.
  * [6] M. Kurkov and D. Vassilevich, “Parity anomaly in four dimensions,” Phys. Rev. D96 (2017), no. 2, 025011, 1704.06736.
  * [7] M. Kurkov and D. Vassilevich, “Gravitational parity anomaly with and without boundaries,” JHEP 03 (2018) 072, 1801.02049.
  * [8] E. Witten and K. Yonekura, “Anomaly Inflow and the $\eta$-Invariant,” in The Shoucheng Zhang Memorial Workshop.  9, 2019.  1909.08775.
  * [9] E. Witten, “Fermion Path Integrals And Topological Phases,” Rev. Mod. Phys. 88 (2016), no. 3, 035001, 1508.04715.
  * [10] A. V. Ivanov and D. V. Vassilevich, “Anomaly inflow for local boundary conditions,” JHEP 09 (2022) 250, 2208.00476.
  * [11] H. Fukaya, M. Furuta, Y. Matsuki, S. Matsuo, T. Onogi, S. Yamaguchi, and M. Yamashita, “Mod-two APS index and domain-wall fermion,” Lett. Math. Phys. 112 (2022) 16, 2012.03543.
  * [12] P. B. Gilkey, Asymptotic formulae in spectral geometry.  CRC Press, Boca Raton, 2004.
  * [13] D. V. Vassilevich, “Heat kernel expansion: User’s manual,” Phys. Rept. 388 (2003) 279–360, hep-th/0306138.
  * [14] M. F. Atiyah, V. K. Patodi, and I. M. Singer, “Spectral asymmetry and Riemannian geometry. III,” Math. Proc. Cambridge Phil. Soc. 79 (1976) 71–99.
  * [15] P. B. Gilkey, Invariance theory, the heat equation, and the Atiyah-Singer index theorem.  Publish or Perish, Wilmington, 1984.
  * [16] A. Chodos, R. L. Jaffe, K. Johnson, C. B. Thorn, and V. F. Weisskopf, “A New Extended Model of Hadrons,” Phys. Rev. D9 (1974) 3471–3495.
  * [17] A. Chodos, R. L. Jaffe, K. Johnson, and C. B. Thorn, “Baryon Structure in the Bag Theory,” Phys. Rev. D10 (1974) 2599\.
  * [18] P. Gilkey and L. Smith, “The eta invariant for a class of elliptic boundary value problems,” Commun. Pure Appl. Math. 36 (1983) 85–131.
  * [19] M. Rho, A. S. Goldhaber, and G. E. Brown, “Topological Soliton Bag Model for Baryons,” Phys. Rev. Lett. 51 (1983) 747–750.
  * [20] C. G. Beneventano, P. B. Gilkey, K. Kirsten, and E. M. Santangelo, “Strong ellipticity and spectral properties of chiral bag boundary conditions,” J. Phys. A36 (2003) 11533, hep-th/0306156.
  * [21] A. V. Ivanov, M. A. Kurkov, and D. V. Vassilevich, “Heat kernel, spectral functions and anomalies in Weyl semimetals,” J. Phys. A 55 (2022), no. 22, 224004, 2111.11493.
  * [22] N. P. Armitage, E. J. Mele, and A. Vishwanath, “Weyl and Dirac Semimetals in Three Dimensional Solids,” Rev. Mod. Phys. 90 (2018), no. 1, 015001, 1705.01111.
  * [23] C. Almeida, A. Alonso-Izquierdo, R. Fresneda, J. Mateos Guilarte, and D. Vassilevich, “Nontopological fractional fermion number in the Jackiw-Rossi model,” Phys. Rev. D 103 (2021), no. 12, 125015, 2103.06826.
  * [24] M. Hortacsu, K. D. Rothe, and B. Schroer, “Zero Energy Eigenstates for the Dirac Boundary Problem,” Nucl. Phys. B171 (1980) 530–542.
  * [25] P. Hrasko and J. Balog, “The Fermion Boundary Condition and the $\theta$ Angle in QED in Two-dimensions,” Nucl. Phys. B 245 (1984) 118–126.
  * [26] A. Wipf and S. Dürr, “Gauge theories in a bag,” Nucl. Phys. B 443 (1995) 201–232, hep-th/9412018.
  * [27] A. Kirchberg, K. Kirsten, E. M. Santangelo, and A. Wipf, “Spectral asymmetry on the ball and asymptotics of the asymmetry kernel,” J. Phys. A39 (2006) 9573–9589, hep-th/0605067.
  * [28] P. Gilkey and K. Kirsten, “Stability theorems for chiral bag boundary conditions,” Lett. Math. Phys. 73 (2005) 147–163, math/0510152.
  * [29] Y. A. Sitenko, “Casimir effect with quantized charged spinor matter in background magnetic field,” Phys. Rev. D 91 (2015), no. 8, 085012, 1411.2460.
  * [30] G. Angelone, M. Asorey, P. Facchi, D. Lonigro, and Y. Martinez, “Boundary conditions for the quantum Hall effect,” J. Phys. A 56 (2023) 025301, 2209.09629.