# Gen2Det: Generate to Detect

Saksham Suri$\text{}^{1}$ Work done during internship at Meta. Fanyi
Xiao$\text{}^{2}$ Animesh Sinha$\text{}^{2}$ Sean Chang Culatana$\text{}^{2}$
Raghuraman Krishnamoorthi$\text{}^{2}$ Chenchen Zhu$\text{}^{2}$ *Equal
Advisory Contribution. Abhinav Shrivastava$\text{}^{1}$22footnotemark: 2
University of Maryland, College Park1 Meta2

###### Abstract

Recently diffusion models have shown improvement in synthetic image quality as
well as better control in generation. We motivate and present Gen2Det, a
simple modular pipeline to create synthetic training data for object detection
for free by leveraging state-of-the-art grounded image generation methods.
Unlike existing works which generate individual object instances, require
identifying foreground followed by pasting on other images, we simplify to
directly generating scene-centric images. In addition to the synthetic data,
Gen2Det also proposes a suite of techniques to best utilize the generated
data, including image-level filtering, instance-level filtering, and better
training recipe to account for imperfections in the generation. Using Gen2Det,
we show healthy improvements on object detection and segmentation tasks under
various settings and agnostic to detection methods. In the long-tailed
detection setting on LVIS, Gen2Det improves the performance on rare categories
by a large margin while also significantly improving the performance on other
categories, e.g. we see an improvement of 2.13 Box AP and 1.84 Mask AP over
just training on real data on LVIS with Mask R-CNN. In the low-data regime
setting on COCO, Gen2Det consistently improves both Box and Mask AP by 2.27
and 1.85 points. In the most general detection setting, Gen2Det still
demonstrates robust performance gains, e.g. it improves the Box and Mask AP on
COCO by 0.45 and 0.32 points.

##  1 Introduction

Figure 1:  Existing approaches which utilize synthetic data for detection
training follow a common methodology of generating object-centric images and
pasting instances on real images (top). Gen2Det (middle) instead utilizes
state-of-art grounded inpainting diffusion model to directly generate scene-
centric images. Further, Gen2Det performs filtering to handle unsuitable
generations. Finally, during the detector training we introduce changes to
handle filtered generated data better. As a result, our method consistently
improves over vanilla training and the AP improvements increase (bottom) as
the class becomes rare (_i.e_., long-tailed classes).

Recent developments in generative modelling using diffusion models has
drastically improved the generation quality. Works like LDM [35], DALL-E [32,
33], Imagen [38], Parti [47] have shown the generation power which diffusion
models possess. In addition to these models which can generate high quality
images given a text input, there have been multiple developments in the
direction of higher control in generation. Along this direction exist works
which use conditional control like ControlNet [49] and GLIGEN [21]. There are
also works like LoRA [18] and Dreambooth [36] which provide means to adapt
these large models in a quick and efficient manner to generate specific kinds
of images. With these developments in both quality and control of
synthetically generated images, it is only natural to come back to the
question of “How to best utilize data generated from these models for
improving recognition performance?”.

Most previous works [15, 39, 42] have explored using synthetic data from
diffusion models for classification and pre-training tasks. The goal of our
work is to look at utilizing more realistic scene configurations by generating
images conditioned on the existing layout of boxes and labels and use them for
object detection and segmentation. A recent work, XPaste [50], explores this
problem and utilizes techniques similar to simple copy-paste [12] to paste
synthetically generated object-centric instances onto real images for object
detector training. However, their method relies on off-the-shelf segmentation
methods built over CLIP [30, 48, 29, 40, 24] to extract masks, and is thus
subject to segmentation errors. In addition to the extra components and
compute, the generated images from XPaste are also not realistic as they do
not respect natural layouts due to random pasting as shown in Figure 1 (top).
In contrast, as shown in Figure 1 (middle), Gen2Det leverages state-of-art
diffusion models for grounded inpainting to generate scene-centric images
which look more realistic. The goal of our approach is to show that utilizing
such generated data from state-of-art diffusion models can lead to improvement
in performance of object detection and segmentation models. By using a
grounded inpainting diffusion model as our base generator, we are able to
generate synthetic versions of common detection datasets like LVIS [13] and
COCO [23] in a layout conditioned manner. We then carefully design a set of
filtering and training strategies, with which we demonstrate that we can
improve detection performance when training on the joint set of real and
synthetic images. More specifically, we sample batches of synthetic and real
images with a sampling probability. During loss computation we also modify the
loss for synthetic data to account for filtered instances. Additionally,
without utilizing any additional models or data (including segmentation masks)
during training, we show improvements in segmentation performance as a
byproduct. The clear improvement over vanilla training shown in Figure 1
(bottom) for classwise Box AP with increasing rarity of classes makes a strong
case for such a pipeline especially in long-tailed or low data regimes.

We emphasize the importance of the proposed filtering and training techniques
to incorporate synthetic data through our ablations, as we show that directly
training on the generated data or mixing it with real data in a naive manner
ends up hurting the performance. It is important to do the proper filtering
and loss modifications to mitigate shortcomings of the generations and let the
model learn from the mix of real and synthetic data. We propose Gen2Det as a
general approach to utilize synthetic data for detector training. Due to its
modular nature, different components can be updated as the field progresses.
This includes utilizing better generators, filtering techniques, model
architectures and also training recipes. We highlight our contributions below:

  * •

Propose to use state-of-art grounded-inpainting models to generate synthetic
data in a manner that respects the realistic scene layouts.

  * •

Propose techniques to perform filtering at image and instance level along with
changes in the detector training in order to utilize such synthetic data
effectively and handle imperfections in generated instances.

  * •

Through our experiments we show consistent improvement across datasets and
architectures especially in long-tailed and low data settings. There is also
considerable gains for rare categories.

  * •

We also provide quantitative and qualitative ablations to showcase the effect
of different hyperparameters and components of our approach which would help
further research in usage of such synthetically generated data.

Figure 2:  Gen2Det: our proposed pipeline for generating and utilizing
synthetic data for object detection and segmentation. Gen2Det starts by
generating grounded inpainted images using state-of-art diffusion model. The
generated images are then filtered at image and instance level to remove
globally bad images as well as individual instances which might be low quality
or have hallucinations. Finally, we train object detection and segmentation
models using the filtered data along with our improved training methodology by
introducing sampling and background ignore.

##  2 Related Works

Diffusion Models for Controllable Image Generation. Diffusion models have been
shown to generate images with unprecedented high quality. Amongst these models
a few of the popular models which can generate images from text prompts
include LDM [35], DALL-E [32, 33], Imagen [38], and Parti [47]. In addition to
just text guided generation there have been further improvements in having
more controllable generation and editing through different inputs [37, 44, 3,
26, 17, 28, 49, 21]. One of the popular works ControlNet [49] provides
multiple ways to enable control including using edge maps, scribbles,
segmentation masks amongst other modalities. Another work GLIGEN [21] brings
in more grounding control where they show image generation and inpainting
conditioned on boxes, keypoints, HED maps, edge maps and semantic maps. We
also utilize a similar pre-trained state-of-art diffusion model for grounded
inpainting.

Synthetic Data for Detection. There has been prior non-diffusion based work
exploring the use of synthetic data for detection. Some works have explored
using image-based blending all the way to depth and semantics informed
positioning [11] for creating synthetic data. There have been other works
which use computer graphics based approaches to render synthetic data by
varying lighting and orientations [43, 31]. Further there have been a line of
augmentations belonging to the copy-paste family [12, 9, 10, 8] where object
instances are cropped and pasted on images at different locations. All these
approaches end up repeating instances already present in the training set.

Leveraging Synthetic Data from Diffusion Models. Due to its high quality
generation and the flexibility in handling multimodal data (_e.g_., vision and
language), there has been a lot of recent work in using pretrained diffusion
models for different tasks. Peekaboo [4] shows the use of diffusion models as
zero shot segmentors while other works [19, 6] shows that diffusion models can
act as zero shot classifiers. StableRep [42] on the other hand uses data
generated by diffusion models to train self-supervised models. A few method
also explore using diffusion model features [20, 27, 41, 25, 46] directly for
downstream tasks. There have also been works exploring the use of synthetic
data from diffusion models to improve image classification [15, 39, 1, 2, 16].
While the classification task has had a lot of exploration with synthetic data
due to the object-centric nature of images which these models can generate
easily, the detection task is less explored as it’s harder to generate data
with grounded annotations using these models. A recent study [22] explored the
use of such data in highly constrained few shot settings for detection where
the gains are expected. Recently, XPaste [50] looked at more general
benchmarks and showed consistent improvements using the Centernet2 [51]
architecture. Specifically, XPaste [50] uses diffusion models to generate
object-centric images and uses multiple CLIP [30] based approaches [48, 29,
40, 24] to extract segmentation maps which makes it slow and error-prone.
Following the extraction of instances and their segmentation maps, they use
synthetic and real instances (retrieved from an additional real dataset) to
perform copy-paste [12] augmentation to generate synthetic data. Gen2Det on
the other hand does not use any additional CLIP based approaches. Rather, once
we generate the data we only train with bounding box labels but show
improvements in both box and mask AP. Also in terms of training speed, we are
$3.4\times$ faster compared to XPaste with the same configuration.

##  3 Approach

###  3.1 Overview

Through Gen2Det, we provide a modular way to generate and utilize synthetic
data for object detection and instance segmentation. As shown in Figure 2, we
start by generating data using state-of-art grounded image inpainting
diffusion model. As the generations may not always be perfect we perform image
level and instance level filtering. The image level filtering is performed
using a pre-trained aesthetic classifier [5] while the instance level
filtering is performed using a detector trained on the corresponding real
data. After performing filtering, the images are suitable for training using
our proposed training strategy. While we do not make any architectural changes
to the detector to keep the pipeline general enough, we perform a probability
based sampling of a batch to be composed of synthetic or real samples. Further
while computing the losses, we modify the negatives corresponding to the
synthetic data to be ignored from loss computation to account for the
filtering we performed. We also do not apply mask loss for synthetic samples
as we do not have the segmentation masks corresponding to them. Following
these steps we are able to show consistent improvements in performance.

| Method | Box | Mask
---|---|---
AP | $\text{AP}_{r}$ | $\text{AP}_{c}$ | $\text{AP}_{f}$ | AP | $\text{AP}_{r}$ | $\text{AP}_{c}$ | $\text{AP}_{f}$
Vanilla (real only) | 33.80 | 20.84 | 32.84 | 40.58 | 29.98 | 18.36 | 29.64 | 35.46
Vanilla (synth only) | 11.27 | 6.54 | 9.35 | 15.51 | - | - | - | -
Vanilla (synth+real) | 31.82 | 20.68 | 31.18 | 37.42 | 27.49 | 18.35 | 27.08 | 31.96
XPaste | 34.34 | 21.05 | 33.86 | 40.71 | 30.18 | 18.77 | 30.11 | 35.28
Ours | 34.70 | 23.78 | 33.71 | 40.61 | 30.82 | 21.24 | 30.32 | 35.59

Table 1: Comparisons on LVIS with baselines using Centernet2 architecture. We
report the overall AP and also AP for rare ($\text{AP}_{r}$), common
($\text{AP}_{c}$) and frequent ($\text{AP}_{f}$) classes for both box and mask
evaluations.

| Method | $\text{AP}_{b}$ | $\text{AP}_{m}$
---|---|---
Vanilla (real only) | 46.00 | 39.8
Vanilla (synth only) | 24.51 | -
Vanilla (synth+real) | 44.35 | 37.03
XPaste | 46.60 | 39.90
Ours | 47.18 | 40.44

Table 2: Comparisons with baselines on COCO dataset using Centernet2
architecture. We report the Box AP ($\text{AP}_{b}$) and Mask AP
($\text{AP}_{m}$).

###  3.2 Image Generation

We start our pipeline by generating synthetic images by utilizing a state-of-
art grounded inpainting diffusion model. This model is trained to support
multiple kinds of input conditions. We utilize the model trained for image
inpainting with the image, boxes, and corresponding labels as input.
Specifically, for each image $\mathcal{I}$ in the dataset with box annotations
$\mathcal{B}$, we provide as input the image and the corresponding annotations
to the diffusion model and ask it to inpaint the annotated regions with new
instances of the same class by providing the box label as input. As inpainting
model requires an image and box level text description, for each box $b_{i}$
we use the class name $\textless c_{i}\textgreater$ as the box level prompt
and a concatenation of strings $\textless\text{a }c_{1},\text{a }c_{2},\text{
... and a }c_{n}\textgreater$ as the image level prompt where $n$ is the
number of instances in the image. We show examples of images generated using
this technique in Figure 3.

###  3.3 Filtering

The images generated using the strategy described above may not always contain
good generations so it is important to filter out low quality generations
before using them to train a detection model. We perform two levels of
filtering. First we filter at image level followed by a more granular instance
level filtering. We describe each of these below.

####  3.3.1 Image Level Filtering

The first level of filtering we perform is an image level filtering using a
pretrained classifier which utilizes a CLIP+MLP architecture as described in
[5]. The model is trained to predict how pleasing the image looks visually by
assigning an aesthetic score to each image. We utilize the publicly available
weights from [5] and pass every generated image through the classifier which
returns an aesthetic score for each image. Based on qualitatively analyzing
images and computing the average aesthetic score for real images (from COCO)
we set a threshold for aesthetic filtering $\tau_{a}$. Any image with an
aesthetic score less than $\tau_{a}$ is discarded and its annotations are
removed. We show the effect of this filtering by visualizing discarded samples
in Figure 4.

####  3.3.2 Detector Filtering

As a more granular level of quality assurance, we also perform an extra
filtering step at the instance level. This filtering is designed to remove
annotations for specific generated instances which do not have good generation
quality. In order to perform this filtering we first train a detector on only
the real data. We then pass all the generated images through the trained
detector and store its predictions. Based on the detectors predictions we
evaluate whether a ground truth annotation corresponding to an inpainted
region should be utilized for training or not. This step is important to
handle poor quality/incorrect generations. To do this, for each generated
image we iterate over all ground truth annotations used to generate it. For
each annotation we go through all predictions and remove the ground truth
annotation if there is no overlapping prediction with a score greater than
$\tau_{s}$ and IoU greater than $\tau_{iou}$. This helps us remove ground
truth annotations corresponding to generation which are either bad quality as
the pre-trained detector is not able to predict it with even a low confidence
threshold, or for which the region where they are generated is not the ground
truth region. We show results of the kind of annotations removed using this
filtering in Figure 5.

Figure 3:  Examples of generations using the inpainting diffusion model on the
COCO dataset. The first and fifth columns correspond to the original COCO
images and the rest of the columns are generations with different seeds. These
generated images are then fed to our filtering pipeline for further processing
before use in training.

###  3.4 Model Training

Once the data is ready for use we describe our training strategy to utilize
the synthetic dataset along with the real dataset. It should be noted that for
synthetically generated images we do not have the segmentation masks for the
instances so we do not apply the mask loss on those images.

####  3.4.1 Batch Sampling

We utilize both the real and synthetic data for training the final model. To
do this, we need to figure out how to mix the two datasets together in a way
which lets us utilize the full potential of the real data but at the same time
extract as much useful information from the additional synthetic data as
possible. Naively using only the synthetic dataset and a naive combination of
synthetic and real datasets during training leads to drop in performance as
shown in the ablations. Therefore, to best use both the real and synthetic
datasets, we end up defining a sampling probability $p$. Specifically, a batch
is chosen to comprise of completely synthetically generated samples with the
sampling probability $p$ during training. This sampling strategy interleaves
batches of synthetic and real data and as we will show leads to effective
learning from both data sources.

####  3.4.2 Background Ignore

While the filtering we perform at both image and instance level can deal with
bad quality generations, they also introduce some noise especially for classes
for which the detector itself has poor quality predictions. Essentially, the
detector filtering removes ground truth annotation corresponding to bad
quality/incorrect generations but that does not remove the bad instance itself
from the image. Additionally, the generative model could have also
hallucinated multiple instance of an object class leading to missing
annotations. To counter the effect of both these scenarios we introduce an
ignore functionality during training. Essentially, for both the region
proposal network (RPN) and detector head we ignore the background regions from
the loss computation, if their fg/non-bg class prediction score is higher than
a threshold $\tau_{i}$. This lets us train even in the presence of some bad
quality regions without incorrectly penalizing the detector for predicting
objects at those locations. This is an important change in training to allow
our model to effectively utilize synthetic images with incorrectly
hallucinated instances, as the instance level filtering only removes the
instance annotation but does not handle hallucinations.

Additionally, as stated above we do not utilize any additional mask
information for the synthetic data as we only inpaint using boxes. So while
training we ignore the mask loss for the synthetic images. Even with this, as
shown in our quantitative results, we see an improvement in both box and mask
performance.

##  4 Results

###  4.1 Experimental Setting

We evaluate our approach on LVIS [13] and COCO [23] datasets. These datasets
provide a standard benchmark for object detection and instance segmentation
approaches. LVIS is a long tailed dataset with 1203 classes utilizing the same
images as COCO which contains only 80 classes. The long tailed nature of LVIS
makes it especially appealing to showcase the use of synthetic data for the
rare categories. We report the Average Precision (AP) for both these datasets
and compare using the box AP ($\text{AP}_{b}$) as well as mask AP
($\text{AP}_{m}$). Additionally for LVIS, following standard practice and the
existing definition of rare, common, and frequent classes, we report the
respective box and mask APs for those subset of classes. Further, we also
report results on artificially created subsets of COCO to simulate the low-
data regime by using $1\%$, $5\%$, $10\%$, $20\%$, $30\%$, $40\%$ and $50\%$
randomly selected images from COCO.

Figure 4:  Examples of samples discarded during the image level filtering. As
can be seen, image level filtering is able to remove images with artifacts
present at a global level.

We utilize the Detectron2 framework [45] for training all the models and use
the XPaste code to run their baselines. We use the Centernet2 clone adapted
for instance segmentation. For LVIS, we rerun XPaste to get the results
corresponding to adding only additional synthetic data. For COCO, we directly
adopt their evaluation numbers due to the lack of corresponding configs and
details for reproduction. Following their code we also utilize the
ImageNet-22k [7] pretrained models for the Centernet2 experiments while using
the ImageNet-1k [7] standard models from Detectron2 for the rest of the
experiments. We utilize the Mask R-CNN [14] architecture and the LVIS dataset
for our ablations. The Mask R-CNN [14] and Faster R-CNN [34] are implemented
in the Detectron2 framework [45].

Figure 5: Examples of ground-truth instance annotations discarded by the
detector filtering highlighted in red.

For data generation, we utilize a state-of-art diffusion model for grounded
inpainting using the images from LVIS and COCO along with their annotations
which contain the boxes and class labels. For aesthetic filtering, we utilize
the open source model available in their repository [5]. For aesthetic
filtering we set $\tau_{a}$ to $4.5$ which is roughly the same as the average
aesthetic score on real COCO images. For detector filtering we set $\tau_{s}$
as $0.2$ and $\tau_{iou}$ as $0.3$ for LVIS. We use the same $\tau_{iou}$ for
COCO and set $\tau_{s}$ as $0.1$. During training, we use a sampling
probability $p=0.2$. We set $\tau_{i}$ to 0 for both datasets thus effectively
ignore all background regions corresponding to the synthetic data from the
loss.

Method | Box | Mask
---|---|---
AP | $\text{AP}_{r}$ | $\text{AP}_{c}$ | $\text{AP}_{f}$ | AP | $\text{AP}_{r}$ | $\text{AP}_{c}$ | $\text{AP}_{f}$
Faster R-CNN | 21.39 | 10.05 | 19.46 | 28.51 | - | - | - | -
Faster R-CNN (Ours) | 22.90 | 12.78 | 21.19 | 29.27 | - | - | - | -
Mask R-CNN | 22.29 | 10.63 | 20.15 | 29.80 | 21.83 | 11.15 | 20.42 | 28.10
Mask R-CNN (Ours) | 24.42 | 15.43 | 22.63 | 30.38 | 23.67 | 15.33 | 22.62 | 28.51
Centernet2 | 33.80 | 20.84 | 32.84 | 40.58 | 29.98 | 18.36 | 29.64 | 35.46
Centernet2 (Ours) | 34.70 | 23.78 | 33.71 | 40.61 | 30.82 | 21.24 | 30.32 | 35.59

Table 3:  Results across different backbones on the LVIS dataset. We can see
consistent improvement across various backbones with especially higher gains
on the rare categories.

Image % | Vanilla | Gen2Det
---|---|---
$\text{AP}_{b}$ | $\text{AP}_{m}$ | $\text{AP}_{b}$ | $\text{AP}_{m}$
1 | 11.76 | 11.34 | 14.84 (+3.08) | 13.89 (+2.55)
2 | 16.07 | 15.36 | 18.42 (+2.35) | 17.27 (+1.91)
5 | 19.52 | 18.48 | 22.91 (+3.38) | 21.17 (+2.69)
10 | 25.29 | 23.35 | 27.62 (+2.33) | 25.34 (+1.99)
20 | 29.32 | 26.97 | 31.76 (+2.44) | 29.03 (+2.06)
30 | 33.02 | 30.37 | 34.70 (+1.68) | 31.65 (+1.28)
40 | 34.39 | 31.36 | 36.09 (+1.69) | 32.67 (+1.30)
50 | 36.52 | 33.20 | 37.74 (+1.22) | 34.24 (+1.04)

Table 4:  Performance on the LD-COCO setting. The LD-COCO data is a randomly selected subset of COCO with different percentages of images from the dataset.  Row | Real Data | Synth. Data | Sampling | Img. Filt. | Det. Filt. | Bg. Ignore | Box AP | $\Delta_{\text{AP}}$ | Box $\text{AP}_{r}$ | $\Delta_{\text{AP}_{r}}$
---|---|---|---|---|---|---|---|---|---|---
1 | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | 22.29 | $-$ | 10.63 | $-$
2 | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | 7.67 | -15.25 | 4.57 | -6.06
3 | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | 20.75 | -1.54 | 10.91 | 0.28
4 | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | 23.49 | 1.2 | 13.21 | 2.58
5 | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | 23.78 | 1.49 | 13.79 | 3.16
6 | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | 23.76 | 1.47 | 13.62 | 2.99
7 | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | 23.32 | 1.03 | 12.77 | 2.14
8 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 24.42 | 2.13 | 15.43 | 4.8
Table 5: Ablation of various components of proposed approach on LVIS using
Mask R-CNN.

Config | $1\times$ | $2\times$ | $3\times$ | $4\times$
---|---|---|---|---
AP | 22.50 | 24.42 | 22.85 | 22.31

(a) Longer training schedule.

Probability | $0.0$ | $0.1$ | $0.2$ | $0.3$ | $0.4$ | $0.5$
---|---|---|---|---|---|---
AP | 22.29 | 23.23 | 24.42 | 23.42 | 23.28 | 22.52

(b) Effect of sampling probability.

Synthetic Data | $0\times$ | $1\times$ | $2\times$ | $3\times$ | $4\times$ | $5\times$
---|---|---|---|---|---|---
AP | 22.29 | 23.55 | 24.42 | 23.67 | 23.82 | 23.82

(c) Effect of generated data sample counts.

Table 6: Ablation on different hyperparameters on the LVIS dataset with Mask
R-CNN backbone.

###  4.2 Quantitative Results

####  4.2.1 Comparison with Existing Works

Comparison on LVIS. We start by comparing our approach to the existing works
in Table 2 on the LVIS dataset with CenterNet2 backbone. Over vanilla training
we show that our method improves the Box and Mask AP over rare categories by
$2.94$ and $2.88$ respectively with a $0.9$ and $0.84$ Box and Mask AP
improvement overall. Despite the fact that XPaste utilizes four different CLIP
based models [48, 29, 40, 24] to obtain segmentation masks and is also
$~{}3.5\times$ slower to train compared to our approach on LVIS, we are able
to outperform XPaste by $2.73$ Box AP and $2.47$ Mask AP on the rare
categories and by $0.36\text{ and }0.64$ Box and Mask AP across all
categories. The huge gains in rare categories and overall improvements
highlight our methods effectiveness in both long tailed as well as general
settings.

Comparison on COCO. Similar to LVIS, we compare on the COCO benchmark in Table
2. On COCO too we show an improvement of $1.18$ and $0.64$ on Box and Mask AP
over vanilla training. Compared to XPaste we improve by $0.58$ Box AP and
$0.54$ Mask AP. We note that compared to LVIS the improvements are slightly
lower here as LVIS is a more long-tailed dataset where adding synthetic data
shines.

Comparison on LD-COCO. In order to simulate a low-data regime on COCO with
fewer images and labels, we create random subsets of different percentages of
COCO images. In Table 4 we show results on the low-data (LD) version of COCO.
As can be seen for each row, we show consistent improvement in both Box and
Mask AP. On average we improve the Box and Mask AP by $2.27$ and $1.85$ points
respectively. This shows that adding synthetic data is especially fruitful in
both long tailed and low data regimes.

Comparison across backbones. We further show comparison across architectures
by comparing to performance with vanilla training using only the real data. As
can be seen in Table 3 we show consistent improvement in overall box and mask
AP, and as mentioned above, our method especially shines for the rare
categories. For the MaskRCNN architecture we show a substantial gain in
performance with an improvement of $2.13$ Box AP and $1.84$ Mask AP over just
training on real data. The improvements on rare categories are even higher
with a gain of $4.8$ Box AP and $4.18$ Mask AP, highlighting the efficacy of
using synthetic data in the Gen2Det pipeline for rare/long-tailed categories.
For the same experimental setting on COCO, please see our Supplementary
Materials.

####  4.2.2 Ablation

Effect of different components. Table 5 summarizes the effect of incorporating
different components in our pipeline. We report the Box AP and Box
$\text{AP}_{r}$ for this analysis along with the respective improvement deltas
introduced by each component over the baseline. Row 1 and 2 correspond to only
training with the real and synthetic data respectively. It can be noted that
naively training on only the synthetic or a simple combination of real and
synthetic data (row 3) leads to a degradation in overall AP. Combining the two
sets (row 3) does lead to a marginal increase in Box $\text{AP}_{r}$ due to
the addition of more instances of the rare categories. Adding synthetic data
along with sampling (row 4) gives us a boost of $1.2$ in overall Box AP and
$2.58$ for the rare categories.

Further, incorporating image level filtering which removes bad quality images
in their entirety gives us a improvement of $1.49$ Box AP and $3.16$ Box
$\text{AP}_{r}$ (row 5). Following our filtering pipeline, performing detector
filtering on its own (row 6) does not work much better than just image level
filtering as it ends up removing annotations corresponding to bad quality
generations but the instances still remain in the images. Additionally we also
try performing background loss ignore for the synthetic data without detector
filtering (row 7), and that again does not lead to an improvement on its own
rather leads to a drop in performance. Finally, in row 8 we see that detector
filtering coupled with background ignore takes care of the discarded
annotations and leads to a net total improvement of $2.13$ AP and $4.8$
$\text{AP}_{r}$. It should be noted that detector filtering alone only removes
certain annotations from training, using it with background ignore unlocks the
full potential of such filtering and is important for training with synthetic
data which might not always be perfect.

Effect of Longer training. We try to see the effect of longer training by
increasing the number of iteration for the detector training. We also adjust
the learning rate steps proportionally for each experiment. As can be seen
from Table 6(a), starting with the $1\times$ config and moving to $2\times$
gives us a boost as it is able to utilize the additional samples better but
further longer training does not work that well and actually reduces
performance as it starts overfitting to the data.

Effect of Sampling Probability. Here we vary the sampling probability $p$ used
during training which affects whether the batch should be made from real or
synthetic data. We show the change in performance for different synthetic
probabilities in Table 6(b). As can be seen in all cases we are better than
the vanilla training. We do see that initially increasing the sampling
probability to 0.2 helps and improves performance after which it decreases as
the probability is further increased. This might be because the real data
which is pristine is seen lesser and lesser after a certain probability value,
and therefore we might get negatively impacted by the noise in synthetic data.

Effect of more synthetic samples. Similar to longer training, adding more
synthetic samples also has a curved performance as shown in Table 6(c). Adding
$1\times$ data is better than training on only real data. Further increasing
the number of synthetic samples to $2\times$ improves the performance. In our
setting though, increasing them more leads to a reduction in performance.
Ideally, the goal would be to keep increasing the performance with more
synthetic data but we leave that for future work as it would require looking
into improving diversity of generations which will happen as the generators
improve. Also adding more data under the current scheme ends up generating
more images with the same layout leading to a drop after a certain point.
Increasing diversity in layouts would be another way to improving performance
further. Again, it should be noted that in all cases, our performance is
sufficiently better than vanilla training.

###  4.3 Qualitative Results

We show the outputs of different parts of pipeline qualitatively on the COCO
dataset. First, we show a few samples which are the synthetic images generated
using the pre-trained grounded inpainting diffusion model. Figure 3 shows
qualitative outputs using COCO images and annotations. The first column
corresponds to the real image from COCO dataset followed by multiple synthetic
generations each with a different seed leading to variability in instances.

We highlight a few samples which are rejected from the image level filtering
step in Figure 4. As can be seen these are images which do not look good at an
image level even though some of the instances might look fine. The aesthetic
filtering steps removes such images. It can be seen for example in the first
image in first row, the bicycle generation in the front seems incorrect.
Similarly, in the first image in the second row, the bigger airplane which
covers a major part of the image does not look realistic.

Finally, in Figure 5 we show some examples of instances discarded by detector
filtering. As can be seen in most cases the detector filtering discards
instances which have either been incorrectly generated or are bad quality
generations. For example in the first example in the first row, the generation
does not correspond to the pizza rather a specific topping. Similarly in the
first example in the second row, there is no airplane generated in the ground
truth region.

##  5 Conclusion

With the huge strides in image generation in terms of both quality and
control, we try to tackle the problem of training detection and segmentation
models with synthetic data. Our proposed pipeline Gen2Det utilizes state-of-
art grounded inpainting diffusion model to generate synthetic images which we
further filter at both image and instance level before using in training. We
also introduce some changes in the detector training to utilize the data
better and take care of shortcoming which the data might pose. Along with
detailed quantitative ablations and qualitative outputs we show the efficacy
of different components of our approach. Finally, we show improvement across
both LVIS and COCO and show higher improvements on rare classes in the long
tailed LVIS setting. Additionally, we show improvement in low-data COCO
setting too. Most interestingly, we show improvement in segmentation
performance without using any additional segmentation masks like existing
works. We hope Gen2Det acts as a general modular framework which can be
benefit from future developments in both generation as well as detector
training due to easily replaceable general purpose blocks which it comprises
of.

## Acknowledgements

We thank Justin Johnson for insightful discussion on this work. We also want
to thank Vikas Chandra for helping with the submission.

## References

  * Azizi et al. [2023] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet.  Synthetic data from diffusion models improves imagenet classification.  _arXiv preprint arXiv:2304.08466_ , 2023.
  * Bansal and Grover [2023] Hritik Bansal and Aditya Grover.  Leaving reality to imagination: Robust classification via generated datasets.  _arXiv preprint arXiv:2302.02503_ , 2023.
  * Brooks et al. [2023] Tim Brooks, Aleksander Holynski, and Alexei A Efros.  Instructpix2pix: Learning to follow image editing instructions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18392–18402, 2023.
  * Burgert et al. [2022] Ryan Burgert, Kanchana Ranasinghe, Xiang Li, and Michael S Ryoo.  Peekaboo: Text to image diffusion models are zero-shot segmentors.  _arXiv preprint arXiv:2211.13224_ , 2022.
  * [5] Christophschuhmann.  Christophschuhmann/improved-aesthetic-predictor: Clip+mlp aesthetic score predictor.
  * Clark and Jaini [2023] Kevin Clark and Priyank Jaini.  Text-to-image diffusion models are zero-shot classifiers.  _arXiv preprint arXiv:2303.15233_ , 2023.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 248–255, 2009.
  * Dvornik et al. [2018] Nikita Dvornik, Julien Mairal, and Cordelia Schmid.  Modeling visual context is key to augmenting object detection datasets.  _ArXiv_ , abs/1807.07428, 2018.
  * Dwibedi et al. [2017] Debidatta Dwibedi, Ishan Misra, and Martial Hebert.  Cut, paste and learn: Surprisingly easy synthesis for instance detection.  _2017 IEEE International Conference on Computer Vision (ICCV)_ , pages 1310–1319, 2017.
  * Fang et al. [2019] Haoshu Fang, Jianhua Sun, Runzhong Wang, Minghao Gou, Yong-Lu Li, and Cewu Lu.  Instaboost: Boosting instance segmentation via probability map guided copy-pasting.  _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 682–691, 2019.
  * Georgakis et al. [2017] Georgios Georgakis, Arsalan Mousavian, Alexander C Berg, and Jana Kosecka.  Synthesizing training data for object detection in indoor scenes.  _Robotics: Science and Systems (RSS)_ , 2017.
  * Ghiasi et al. [2021] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph.  Simple copy-paste is a strong data augmentation method for instance segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 2918–2928, 2021.
  * Gupta et al. [2019] Agrim Gupta, Piotr Dollar, and Ross Girshick.  Lvis: A dataset for large vocabulary instance segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5356–5364, 2019.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick.  Mask r-cnn.  2017\.
  * He et al. [2022] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi.  Is synthetic data from generative models ready for image recognition?  _arXiv preprint arXiv:2210.07574_ , 2022.
  * Hemmat et al. [2023] Reyhane Askari Hemmat, Mohammad Pezeshki, Florian Bordes, Michal Drozdzal, and Adriana Romero-Soriano.  Feedback-guided data synthesis for imbalanced classification.  _arXiv preprint arXiv:2310.00158_ , 2023.
  * Höllein et al. [2023] Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner.  Text2room: Extracting textured 3d meshes from 2d text-to-image models.  _arXiv preprint arXiv:2303.11989_ , 2023.
  * Hu et al. [2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ , 2021.
  * Li et al. [2023a] Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak.  Your diffusion model is secretly a zero-shot classifier.  _arXiv preprint arXiv:2303.16203_ , 2023a.
  * Li et al. [2023b] Daiqing Li, Huan Ling, Amlan Kar, David Acuna, Seung Wook Kim, Karsten Kreis, Antonio Torralba, and Sanja Fidler.  Dreamteacher: Pretraining image backbones with deep generative models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 16698–16708, 2023b.
  * Li et al. [2023c] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee.  Gligen: Open-set grounded text-to-image generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22511–22521, 2023c.
  * Lin et al. [2023] Shaobo Lin, Kun Wang, Xingyu Zeng, and Rui Zhao.  Explore the power of synthetic data on few-shot object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 638–647, 2023.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Lüddecke and Ecker [2022] Timo Lüddecke and Alexander Ecker.  Image segmentation using text and image prompts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 7086–7096, 2022.
  * Luo et al. [2023] Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell.  Diffusion hyperfeatures: Searching through time and space for semantic correspondence.  In _Advances in Neural Information Processing Systems_ , 2023.
  * Meng et al. [2021] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  _arXiv preprint arXiv:2108.01073_ , 2021.
  * Mukhopadhyay et al. [2023] Soumik Mukhopadhyay, Matthew Gwilliam, Vatsal Agarwal, Namitha Padmanabhan, Archana Swaminathan, Srinidhi Hegde, Tianyi Zhou, and Abhinav Shrivastava.  Diffusion models beat gans on image classification.  _arXiv preprint arXiv:2307.08702_ , 2023.
  * Patashnik et al. [2023] Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Elor, and Daniel Cohen-Or.  Localizing object-level shape variations with text-to-image diffusion models.  _arXiv preprint arXiv:2303.11306_ , 2023.
  * Qin et al. [2020] Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar Zaiane, and Martin Jagersand.  U2-net: Going deeper with nested u-structure for salient object detection.  page 107404, 2020.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Rajpura et al. [2017] Param S. Rajpura, Ravi Sadananda Hegde, and Hristo Bojinov.  Object detection using deep cnns trained on synthetic images.  _ArXiv_ , abs/1706.06782, 2017.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pages 8821–8831. PMLR, 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022.
  * Ren et al. [2015] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 39:1137–1149, 2015.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22500–22510, 2023.
  * Saharia et al. [2022a] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi.  Palette: Image-to-image diffusion models.  In _ACM SIGGRAPH 2022 Conference Proceedings_ , pages 1–10, 2022a.
  * Saharia et al. [2022b] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022b.
  * Sariyildiz et al. [2023] Mert Bulent Sariyildiz, Karteek Alahari, Diane Larlus, and Yannis Kalantidis.  Fake it till you make it: Learning transferable representations from synthetic imagenet clones.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Su et al. [2022] Yukun Su, Jingliang Deng, Ruizhou Sun, Guosheng Lin, and Qingyao Wu.  A unified transformer framework for group-based segmentation: Co-segmentation, co-saliency detection and video salient object detection, 2022.
  * Tang et al. [2023] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan.  Emergent correspondence from image diffusion.  _arXiv preprint arXiv:2306.03881_ , 2023.
  * Tian et al. [2023] Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan.  Stablerep: Synthetic images from text-to-image models make strong visual representation learners.  _arXiv preprint arXiv:2306.00984_ , 2023.
  * Tremblay et al. [2018] Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, V. Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, and Stan Birchfield.  Training deep networks with synthetic data: Bridging the reality gap by domain randomization.  _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 1082–10828, 2018.
  * Voynov et al. [2023] Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or.  Sketch-guided text-to-image diffusion models.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , pages 1–11, 2023.
  * Wu et al. [2019] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * Xiang et al. [2023] Weilai Xiang, Hongyu Yang, Di Huang, and Yunhong Wang.  Denoising diffusion autoencoders are unified self-supervised learners.  _arXiv preprint arXiv:2303.09769_ , 2023.
  * Yu et al. [2022] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  _arXiv preprint arXiv:2206.10789_ , 2(3):5, 2022.
  * Yun and Lin [2022] Yi Ke Yun and Weisi Lin.  Selfreformer: Self-refined network with transformer for salient object detection.  _arXiv preprint arXiv:2205.11283_ , 2022.
  * Zhang et al. [2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3836–3847, 2023.
  * Zhao et al. [2023] Hanqing Zhao, Dianmo Sheng, Jianmin Bao, Dongdong Chen, Dong Chen, Fang Wen, Lu Yuan, Ce Liu, Wenbo Zhou, Qi Chu, et al.  X-paste: Revisiting scalable copy-paste for instance segmentation using clip and stablediffusion.  2023\.
  * Zhou et al. [2021] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl.  Probabilistic two-stage detection.  _arXiv preprint arXiv:2103.07461_ , 2021.

Supplementary Material

Method | $\text{AP}_{b}$ | $\text{AP}_{m}$
---|---|---
Faster R-CNN | 40.20 | -
Faster R-CNN (Ours) | 40.56 | -
Mask R-CNN | 41.08 | 37.14
Mask R-CNN (Ours) | 41.53 | 37.46
CenterNet2 | 46.00 | 39.8
CenterNet2 (Ours) | 47.18 | 40.44

Table 7: Comparisons across architectures on COCO dataset. We report the Box
AP ($\text{AP}_{b}$) and Mask AP ($\text{AP}_{m}$).

##  6 Experimental Details

In addition to the details mentioned in the main paper we would like to
highlight that we were not able to reproduce the exact results of XPaste from
the code and configs provided and it is an open issue in their Github
repository. Additionally, for learning rate for Faster and Mask R-CNN
experiments we use the default LR of 0.08 which is scaled from 0.02 as we use
$4\times$ the batch size as the default configs. For LVIS, this LR was not
stable so we use an LR of 0.04 accross all Faster and Mask R-CNN experiments.
For LD-COCO we do not perform filtering or background ignore due to already
low number of images and instances.

##  7 Quantitative Results on COCO

In Table 7 we show improvements in both Box ($\text{AP}_{b}$) and Mask
($\text{AP}_{m}$) AP across Faster R-CNN, Mask R-CNN and Centernet2
architectures on the COCO dataset.

##  8 Pasting Object Centric Instances

We use state-of-art text to image diffusion model to generate object centric
images for each instance annotation and paste them on corresponding LVIS
images in a layout satisfying manner using the box coordinates. We use the
prompt $\text{a photo of }<c>$ where $c$ is the class name for that instance.
We then train the detector with vanilla training and sampling synthetic and
real data with equal probabilities. This gives us a Box AP of $22.50$ and Mask
AP of $21.45$ which is lower than the performance obtained through Gen2Det
where we get $24.42$ Box AP and $23.67$ Mask AP.

##  9 Ablations

###  9.1 Ablation on $\tau_{s}$

We vary $\tau_{s}$ which is the score threshold we use for detector filtering.
For this experiment we keep $\tau_{iou}$ fixed at 0.3. We show the results for
this in Table 8. It can be seen that initially increasing the score threshold
helps but after 0.2 increasing it further leads to a reduction in performance.
This might be because we filter out too many annotations.

###  9.2 Ablation on $\tau_{iou}$

We vary $\tau_{iou}$ which is the score threshold we use for detector
filtering. For this experiment we keep $\tau_{s}$ fixed at 0.2. We show the
results for this in Table 9. It can be seen that initially increasing the IoU
threshold helps but after 0.3 increasing it further leads to a reduction in
performance. This might be because we filter out too many annotations.

###  9.3 Ablation on $\tau_{i}$

We vary $\tau_{i}$ which is the score threshold we use for background ignore.
We show the results for this in Table 10. It can be seen that initially
setting the score threshold to 0 which ends up ignoring all background
instances from the synthetic data works the best. After that increasing it
reduces it with sporadic increases at certain thresholds.

##  10 Qualitative Results

###  10.1 Qualitative Inpainted Generation

We show qualitative generation on the LVIS dataset in Figures 6 and 7. These
generations utilize the LVIS annotations and images along with state-of-the-
art grounded inpainting model to generate the instances conditioned on the
class labels, box labels, and input images.

We also show qualitative generation on the COCO dataset in Figures 8 and 9
utilizing COCO annotations and images to generate them.

###  10.2 Qualitative Results for Image Filtering

In Figure 10 we show results of the image level filtering by visualizing
images discarded during this step on the LVIS dataset. It can be seen there
are high level issues in the generations. In the third image in row 1 the sign
seems to have some weird patterns. In row 3 the second image has low quality
aircraft generations. Similarly in the second image in the last row the cat
generation is not great.

We also show more image filtering results on the COCO dataset and visualize
the discarded images in Figure 11.

$\tau_{s}$ | Box | Mask
---|---|---
AP | $\text{AP}_{r}$ | $\text{AP}_{c}$ | $\text{AP}_{f}$ | AP | $\text{AP}_{r}$ | $\text{AP}_{c}$ | $\text{AP}_{f}$
0.1 | 23.85 | 14.04 | 21.89 | 30.36 | 23.16 | 14.21 | 21.92 | 28.49
0.2 | 24.42 | 15.43 | 22.63 | 30.38 | 23.67 | 15.33 | 22.62 | 28.51
0.3 | 23.59 | 13.21 | 21.74 | 30.21 | 22.89 | 13.50 | 21.66 | 28.39
0.4 | 23.75 | 13.48 | 21.97 | 30.25 | 23.03 | 13.81 | 21.86 | 28.38
0.5 | 23.62 | 13.69 | 21.56 | 30.28 | 23.05 | 14.24 | 21.75 | 28.36

Table 8:  Comparisons across different $\tau_{s}$ keeping $\tau_{iou}$ fixed
at 0.3 on the LVIS dataset. We report the overall AP and also AP for rare
($\text{AP}_{r}$), common ($\text{AP}_{c}$) and frequent ($\text{AP}_{f}$)
classes for both box and mask evaluations.

$\tau_{iou}$ | Box | Mask
---|---|---
AP | $\text{AP}_{r}$ | $\text{AP}_{c}$ | $\text{AP}_{f}$ | AP | $\text{AP}_{r}$ | $\text{AP}_{c}$ | $\text{AP}_{f}$
0.1 | 23.49 | 12.90 | 21.70 | 30.15 | 22.82 | 12.90 | 21.77 | 28.35
0.2 | 23.56 | 12.32 | 22.05 | 30.18 | 22.88 | 12.58 | 22.08 | 28.31
0.3 | 24.42 | 15.43 | 22.63 | 30.38 | 23.67 | 15.33 | 22.62 | 28.51
0.4 | 23.73 | 13.09 | 22.02 | 30.32 | 23.12 | 13.40 | 22.13 | 28.49
0.5 | 23.77 | 13.22 | 21.89 | 30.51 | 22.91 | 13.33 | 21.58 | 28.60

Table 9:  Comparisons across different $\tau_{iou}$ keeping $\tau_{s}$ fixed
at 0.2 on the LVIS dataset. We report the overall AP and also AP for rare
($\text{AP}_{r}$), common ($\text{AP}_{c}$) and frequent ($\text{AP}_{f}$)
classes for both box and mask evaluations.

###  10.3 Qualitative Results for Detector Filtering

We present some examples for detector filtering on LVIS in Figure 12. We show
the original and generated instances for each of the images and show the
ground truth instance annotation(s) being discarded for each image in red. We
also show examples of detector filtering on COCO in Figure 13.

$\tau_{i}$ | Box | Mask
---|---|---
AP | $\text{AP}_{r}$ | $\text{AP}_{c}$ | $\text{AP}_{f}$ | AP | $\text{AP}_{r}$ | $\text{AP}_{c}$ | $\text{AP}_{f}$
0.0 | 24.42 | 15.43 | 22.63 | 30.38 | 23.67 | 15.33 | 22.62 | 28.51
0.1 | 23.89 | 14.06 | 21.88 | 30.46 | 23.21 | 14.27 | 21.92 | 28.57
0.2 | 23.92 | 13.82 | 22.08 | 30.41 | 23.34 | 14.63 | 22.14 | 28.50
0.3 | 23.70 | 13.27 | 21.86 | 30.34 | 23.02 | 13.32 | 21.95 | 28.49
0.4 | 24.06 | 14.83 | 22.01 | 30.39 | 23.38 | 15.15 | 22.07 | 28.45
0.5 | 23.69 | 12.31 | 22.09 | 30.46 | 22.99 | 12.61 | 22.06 | 28.60

Table 10:  Comparisons across different $\tau_{i}$ on the LVIS dataset. We
report the overall AP and also AP for rare ($\text{AP}_{r}$), common
($\text{AP}_{c}$) and frequent ($\text{AP}_{f}$) classes for both box and mask
evaluations. Figure 6:  Examples of generations using the inpainting diffusion
model on the LVIS dataset. The first column corresponds to the original LVIS
images and the rest of the columns are generations with different seeds. These
generated images are then fed to our filtering pipeline for further processing
before use in training.  Figure 7:  Examples of generations using the
inpainting diffusion model on the LVIS dataset. The first column corresponds
to the original LVIS images and the rest of the columns are generations with
different seeds. These generated images are then fed to our filtering pipeline
for further processing before use in training.  Figure 8:  Examples of
generations using the inpainting diffusion model on the COCO dataset. The
first column corresponds to the original COCO images and the rest of the
columns are generations with different seeds. These generated images are then
fed to our filtering pipeline for further processing before use in training.
Figure 9:  Examples of generations using the inpainting diffusion model on the
COCO dataset. The first column corresponds to the original COCO images and the
rest of the columns are generations with different seeds. These generated
images are then fed to our filtering pipeline for further processing before
use in training.  Figure 10:  Examples of samples discarded during the image
level filtering on LVIS. As can be seen, image level filtering is able to
remove images with artifacts present at a global level.  Figure 11:  Examples
of samples discarded during the image level filtering on COCO. As can be seen,
image level filtering is able to remove images with artifacts present at a
global level.  Figure 12:  Examples of ground-truth instance annotations
discarded by the detector filtering highlighted in red for LVIS dataset.
Figure 13:  Examples of ground-truth instance annotations discarded by the
detector filtering highlighted in red for COCO dataset.