# Online Open-set Semi-supervised Object Detection via Semi-supervised Outlier
Filtering

Zerun Wang1, Ling Xiao1,Liuyu Xiang2, Zhaotian Weng3, Toshihiko Yamasaki1
1The University of Tokyo;
2Beijing University of Posts and Telecommunications; 3Tsinghua University
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

Open-set semi-supervised object detection (OSSOD) methods aim to utilize
practical unlabeled datasets with out-of-distribution (OOD) instances for
object detection. The main challenge in OSSOD is distinguishing and filtering
the OOD instances from the in-distribution (ID) instances during pseudo-
labeling. The previous method uses an offline OOD detection network trained
only with labeled data for solving this problem. However, the scarcity of
available data limits the potential for improvement. Meanwhile, training
separately leads to low efficiency. To alleviate the above issues, this paper
proposes a novel end-to-end online framework that improves performance and
efficiency by mining more valuable instances from unlabeled data.
Specifically, we first propose a semi-supervised OOD detection strategy to
mine valuable ID and OOD instances in unlabeled datasets for training. Then,
we constitute an online end-to-end trainable OSSOD framework by integrating
the OOD detection head into the object detector, making it jointly trainable
with the original detection task. Our experimental results show that our
method works well on several benchmarks, including the partially labeled COCO
dataset with open-set classes and the fully labeled COCO dataset with the
additional large-scale open-set unlabeled dataset, OpenImages. Compared with
previous OSSOD methods, our approach achieves the best performance on COCO
with OpenImages by +0.94 mAP, reaching 44.07 mAP.

##  1 Introduction

Semi-supervised learning (SSL) [25, 1, 26, 27, 31] significantly improves the
performance of various image recognition tasks by using a large amount of
available unlabeled data. Object detection performance has also greatly
benefited from various semi-supervised object detection (SSOD) methods.

However, there is a strong assumption in current SSOD [21, 14, 13, 30, 28, 33]
methods that the unlabeled data and labeled data are from the same label
space. This assumption keeps current SSOD methods a little bit away from
practical situations. Because the unlabeled dataset in real-world usually
contains OOD samples. The samples may be unseen classes or samples within the
same classes but of different domains (like humans in real-world or
portraits).

Some works [31, 23] have been proposed for the open-set semi-supervised image
classification task. However, these methods are difficult to be applied to
object detection tasks directly since image classification is an image-level
task, but object detection is a more challenging instance-level task [20]. And
the OOD detection methods used for open-set image classification are difficult
to be balanced with object detection losses. To solve this problem, Liu et al.
[20] proposed the first open-set semi-supervised object detection (OSSOD)
methods based on current SSOD frameworks. They aim to distinguish and filter
OOD instances during pseudo-labeling. However, even though it has improved the
performance on open-set unlabeled data, there are still many problems.
Firstly, it uses only labeled data to train the OOD detection module. However,
in the SSOD setting, the limited ID samples in the labeled data cannot
represent those in the unlabeled data well. Moreover, the OOD samples in the
labeled data, which are randomly cropped backgrounds, cannot represent the
real foreground OOD samples in unlabeled data. Then, the OOD detection module
in the previous method needs to be trained separately from the object detector
and requires an additional backbone, which is inefficient considering the
training process and network complexity. Thus, both the accuracy and
efficiency need to be improved, and more OSSOD methods need to be developed.

To alleviate the above issues, we propose an online open-set semi-supervised
object detection framework with a novel semi-supervised outlier filtering
method. Our core idea is to mine more valuable ID and OOD instances from
unlabeled open-set data for the OOD head training. Concretely, we use the OOD
detection head to collect ID and OOD instances in unlabeled data with high
confidence scores in a semi-supervised way. Therefore, the OOD detector in our
method can be trained with real foreground OOD instances and more ID instances
in unlabeled data, which leads to better outlier filtering performance.
Moreover, we integrate the OOD detection head into the object detector and
make it trainable with the original object detection task together. This is
achieved by using the proposals from the detector’s region proposal network
(RPN) as ID and OOD samples based on the Intersection over Union (IoU) to the
ground truth rather than randomly cropping.

The experimental results on several benchmarks show that our method can
improve the model’s performance on the labeled datasets with additional
unlabeled open-set datasets. The unlabeled open-set datasets can be the
partially labeled MS-COCO [18] dataset with open-set classes and the fully
labeled MS-COCO dataset with the additional large-scale open-set unlabeled
dataset, OpenImagesv5 [11]. In conclusion, our main contributions can be
summarized as follows:

  * •

We propose an online end-to-end trainable OSSOD framework by integrating the
OOD detection head into the object detector to improve efficiency.

  * •

We propose a semi-supervised outlier filtering strategy for mining more
valuable ID and OOD instances from unlabeled data to improve accuracy.

  * •

Our proposed method achieves state-of-the-art performance on several OSSOD
benchmarks in both accuracy and efficiency. With our method, the performance
of object detectors can be improved with more practical open-set unlabeled
data.

The source code will be made public once accepted.

##  2 Related works

Semi-supervised object detection. Semi-Supervised Object Detection (SSOD)
methods aim to improve object detection performance with unlabeled data. Some
basic SSOD technologies are transferred from semi-supervised image
classification tasks [25] such as data augmentation [1, 2], teacher-student
framework [26], and exponential moving average (EMA) [27] to generate valuable
and high-quality pseudo labels for training the model. Recent SSOD works are
also focusing on the unique problems which only appear in object detection,
such as solving the class-wise imbalance [21] problem, improving localization
reliability in pseudo label [15, 30, 13], applying dynamic pseudo label
threshold [28], and using dense learnable regions rather than hard pseudo
labels [33, 14]. However, these methods have not considered the presence of
OOD instances in unlabeled data. Our experimental results show that just
directly accepting all the pseudo labels with high confidence scores without
filtering the outliers underutilize the open-set data. Moreover, by filtering
OOD instances, the SSOD methods can be further improved [20].

Figure 1: The difference between our method and [20]: Firstly, we jointly
train the object detector and OOD detection head, while [20] need to train
them separately. Secondly, our OOD detection head can also be trained with
unlabeled data by our proposed semi-supervised outlier method, while the OOD
detection head in [20] is trained with labeled data only.

Out-of-distribution detection. Out-of-distribution (OOD) detection methods aim
to distinguish the samples that do not belong to the target domain. Most of
the current OOD detection methods focus on image classification tasks. They
use different scoring methods to distinguish OOD samples, such as the maximum
softmax probability (MSP) [8] and Mahalanobis distance [12], or use additional
network structure to estimate the probability of belonging to the OOD class
[24]. Liang et al. [16] developed a temperature scaling method to separate the
distribution between ID and OOD samples. Liu et al. [19] further improved this
by mapping the output to an energy-based score. Fort et al. [5] relies on
large-scale pre-trained networks to improve performance. There are also OOD
detection methods that are designed for object detection tasks. Du et al. [9]
propose virtual outliers generation to make the detector able to distinguish
OOD samples. But this process is incompatible with the SSOD framework [20].
Moreover, these methods are trained on sufficient labeled data to obtain the
OOD detection ability, while the OSSOD task is more challenging since the
labeled data is limited.

Open-set semi-supervised learning. Most of the current open-set semi-
supervised learning (OSSL) methods focus on image classification tasks. Yu et
al. [31] proposed a multi-task curriculum learning method to select ID samples
from unlabeled data by alternatively estimating the OOD score for unlabeled
images and training the network. Saito et al. [23] relies on the one-vs-all
OOD detection method to filter OOD samples after pseudo-labeling and use a
consistency regularization loss to learn more effective representations for
detecting outliers. However, these methods are incompatible with object
detection tasks: The main difference is that each image contains one object in
the image classification task but contains a variable number of objects in the
object detection task. Moreover, the number of detected objects in each image
is also variable during training. As a result, we cannot maintain a fixed
number of OOD scores as [31] or augment the image several times for each
object to compute the consistency regularization loss such as [23] considering
the complexity. For object detection, there are some tasks like open-world or
open-vocabulary object detection working on open-set classes [32, 10].
However, there is a significant difference between these tasks and OSSOD:
while they aim to improve the accuracy of detecting new classes, OSSOD aims to
filter out new classes as OOD to improve the accuracy of existing classes. Liu
et al. [20] proposed the only work of OSSOD: the outliers are filtered by a
pre-trained OOD detection network. However, the OOD detection network is
trained separately and only with labeled data. We further improved the
accuracy and efficiency. Fig 1 illustrates the difference between their method
and ours.

##  3 Methods

In this section, we first introduce the basic semi-supervised object detection
framework that we use. Then, we introduce the details of our proposed semi-
supervised outlier filtering method and the entire pipeline of the online
OSSOD.

###  3.1 Revisiting SSOD Framework

Our method is based on original SSOD frameworks. The SSOD task assumes that
the object detector is trained on both labeled dataset
$\bm{D}_{l}=\\{\bm{X}_{l},\bm{Y}_{l}\\}$ and unlabeled dataset
$\bm{D}_{u}=\\{\bm{X}_{u}\\}$. A common pipeline is to set two detectors:
student and teacher models. The teacher model generates pseudo labels
$\bm{\hat{Y}}_{u}$ on unlabeled data. The generated pseudo labels are then
selected by a manually set threshold $\tau$ on the classification confidence
score. Then, the student detector is jointly trained with labeled data and
pseudo-labeled unlabeled data. The total loss function can be defined as:

$\displaystyle\mathcal{L}^{ssod}=\mathcal{L}_{sup}^{ssod}(\bm{X}_{l},\bm{Y}_{l})+\lambda\mathcal{L}_{unsup}^{ssod}(\bm{X}_{u},\delta(\bm{\hat{Y}}_{u},\tau)),$
(1)

where $\mathcal{L}_{sup}$ and $\mathcal{L}_{unsup}$ denote the loss function
training with labeled data and unlabeled data respectively. Each consists of
classification and regression losses in object detection tasks. $\lambda$
controls the weight of learning with unlabeled data. $\delta(\cdot)$ denotes
the thresholding process.

After training the student model, the SSOD framework updates the teacher model
by the student’s parameters using the exponential moving average (EMA) method
to get a more stable result:

$\displaystyle\theta_{t}\leftarrow\alpha\theta_{t}+(1-\alpha)\theta_{s},$ (2)

where $\theta_{t}$ denotes the teacher’s parameters, $\theta_{s}$ denotes the
student’s parameters. $\alpha$ denotes the average weight.

Our method can be applied to many SSOD methods based on the framework
mentioned above because we do not change the original structure of the object
detector and the SSOD framework. In this paper, we use the Unbiased Teacher
[21] as the basic SSOD framework for a fair comparison with previous works,
which follows the abovementioned pipeline.

Figure 2: The entire framework. We use the Unbiased Teacher [21] as our basic
SSOD framework. The model is trained simultaneously on both labeled and
unlabeled data. The left image illustrates training with labeled data. The
right image illustrates the outlier filtering process and training with the
unlabeled data. In this case, dog is the ID class, and cat is the OOD class.

###  3.2 Online OSSOD framework

While the previous method relies on an individual pre-trained OOD detection
network, we alternatively add an OOD detection head after the backbone, which
filters the outliers after pseudo-labeling. We make the OOD head jointly
trainable with the original classification head and regression head, which
forms our online OSSOD framework. Fig. 2 illustrates the entire structure of
our framework.

Network structure. We apply an OOD detection method to filter the outliers.
Most of the OOD detection methods work on image classification and take the
feature of the whole image as input. It cannot be directly used in the SSOD
task. Thus, we proposed to integrate them into our framework as an additional
head with the features of instances as their input. Various OOD detection
methods can be used as our OOD detection head. We experimentally find the one-
vs-all (OVA) head [24] can get relatively better performance. The OVA head in
our framework consists of $N+1$ sub-binary classification heads, where $N$
denotes the total foreground class number and the left one is for the
background class. The OOD detection head takes the feature of proposals after
ROI-pooling as input and predicts the probability of each sample belonging to
ID or OOD classes. With this additional OOD head, each output instance will
get an additional prediction indicating whether the instance is ID or OOD
class.

Outlier pseudo label filtering. With the OOD detection head, the pseudo-
labeled instances can be further filtered by the OOD head before being used
for training regarding the OOD detection head’s confidence score.
Specifically, we use the same threshold $\tau^{\prime}$ as the original OVA
method. The pseudo labels can be filtered as:

$\displaystyle\delta(\bm{x},\tau^{\prime})=\delta(p(\hat{y}^{y}|\bm{x}),\tau^{\prime}),$
(3)

where $\bm{x}$ denotes the feature corresponding to the pseudo-label after the
ROI Pooling layer. $y$ denotes the predicted class from the object detector’s
classification head, $p(\hat{y}^{y}|\bm{x_{u}})$ denotes the related
probability of the $y^{\rm th}$ classification head in the OVA head after
softmax, which is considered as the OOD confidence score.

Training on labeled data. Since the labeled data provide reliable supervision,
our OOD detection head also relies on training with the annotations from
labeled data. The previous method trains the OOD detection network offline
with additional random sampling for OOD classes, we found that the RPN has
already generated and labeled the region proposals regarding the IoU score, so
there is no need to do random sampling. Thus, the OOD detection task can be
trained jointly with the original object detection task.

In detail, following the common settings in object detection[29], we label
those proposals as the class of their closest ground truth if the IoU$>0.7$,
while those proposals away from any ground truth with IoU$<0.3$ will be
labeled as background. Other proposals will be treated as ignored regions. We
directly use these labeled proposals to train the OVA head in our framework by
its loss function, which is defined as:

$\displaystyle\mathcal{L}^{ova}(\bm{x},y)=-\log(p(\hat{y}^{y}|\bm{x}))-\min_{j\in(N+1),j\neq
y}\log(1-p(\hat{y}^{j}|\bm{x})),$ (4)

where $\bm{x}$ denotes the feature of one proposal after the ROI pooling layer
labeled as class $y$.

In the object detection task, it is common to sample a large batch of
proposals for each image and many of them are backgrounds. However, we find
that this foreground-background imbalance will influence the OOD detection
head. Thus, we sample fewer background proposals for each image when training
the OOD head. Specifically, the object detector in Unbiased Teacher sampled
512 proposals per image. We further select 64 proposals among them while
prioritizing keeping the foreground instances.

###  3.3 Semi-supervised outlier filtering

Figure 3: Illustration of the situation with limited labeled data and the
method we proposed. The decision boundary learned with labeled data only can
hardly cover more ID instances in unlabeled data. Our method can address this
issue by introducing more instances of unlabeled data.

The previous method only trained the OOD detection module with the labeled
data. However, as illustrated in Fig 3, with limited labeled data, the
distribution of labeled data may not adequately reflect the distinctions
between ID and OOD classes present in the unlabeled data. By introducing high-
confidence unlabeled data, the OOD head can be exposed to a broader range of
distribution characteristics present in the unlabeled data, thereby improving
performance. Therefore, we propose a novel semi-supervised outlier filtering
method to mine more valuable ID and OOD instances in unlabeled data.

By integrating the OOD head into the SSOD pipeline, we can also get the OOD
confidence score for each predicted instance. The instances with high-
confidence ID or OOD scores can also be used for training the OOD detection
head following the semi-supervised learning scheme. Meanwhile, the parameters
of the OOD head are updated using the EMA, so our method also benefits from
the stable ID and OOD predictions from the teacher model’s OOD detection head.

Data: Labeled set: $\bm{D}_{l}=\\{\bm{X}_{l},\bm{Y}_{l}\\}$, Unlabeled set:
$\bm{D}_{u}=\\{\bm{X}_{u}\\}$

1

Result: Learned weights $\theta_{t},\theta_{s}$ of teacher and student

2 Add OOD head to Teacher $\theta_{t}$ and Student $\theta_{s}$;

3 for  _burn-in iterations_ do

4 Compute supervised loss $\mathcal{L}_{sup}^{ssod}(\bm{X}_{l},\bm{Y}_{l})$;

5 Compute supervised OOD loss
$\mathcal{L}_{sup}^{ood}(\bm{X}_{l},\bm{Y}_{l})$;

6 Update student parameters
$\theta_{s}\leftarrow\theta_{s}-\nabla\mathcal{L}$;

7

8 end for

9initiate teacher $\theta_{t}\leftarrow\theta_{s}$ ;

10 for  _semi-supervised training iterations_ do

11 Generate pseudo-label for unlabeled data
$\bm{Y}_{u}=f(\bm{X}_{u};\theta_{t})$;

12 Apply confidence thresholding for detection heads
$\hat{\bm{Y}}_{u}\leftarrow\delta(\bm{Y}_{u};\tau)$;

13 Apply OOD confidence thresholding for detection heads
$\widetilde{\bm{Y}}_{u}\leftarrow\delta(\hat{\bm{Y}}_{u};\tau^{\prime})$;

14 Compute unsupervised loss
$\mathcal{L}_{unsup}^{ssod}(\bm{X}_{u},\bm{\widetilde{Y}}_{u})$;

15 Compute supervised loss $\mathcal{L}_{sup}^{ssod}(\bm{X}_{l},\bm{Y}_{l})$;

16 Apply OOD confidence thresholding for OOD detection head
$\widetilde{{Y}}_{u}^{ood}\leftarrow\delta(\bm{{Y}}_{u},\tau^{ood})$;

17 Compute unsupervised OOD loss
$\mathcal{L}_{unsup}^{ova}(\bm{X}_{u},\widetilde{\bm{Y}}_{u})$;

18 Compute supervised OOD loss
$\mathcal{L}_{sup}^{ova}(\bm{X}_{l},\bm{Y}_{l})$;

19 Jointly training with all the losses
$\mathcal{L}=\mathcal{L}_{sup}^{ssod}+\lambda\mathcal{L}_{unsup}^{ssod}+\mathcal{L}^{ood}$;

20 Update student parameters
$\theta_{s}\leftarrow\theta_{s}-\nabla\mathcal{L}$;

21 Update teacher by EMA
$\theta_{t}\leftarrow\alpha\theta_{t}+(1-\alpha)\theta_{s}$;

22

23 end for

Algorithm 1 Our Online OSSOD method

Specifically, when training on unlabeled data, we first get the pseudo-labeled
instances from the original detection heads. Then we further select ID
instances from the pseudo label regarding the confidence score of our OOD
detection head as eq. 3. Here, we use a higher threshold $\tau^{ood}$ for
selecting more accurate ID instances following the common SSL setting, and
other predicted ID instances lower than this threshold will be ignored.

We follow the same process as eq. 4 to train the OOD head using unlabeled
data: Those proposals from RPN which has high IoU score with selected ID
instances are labeled as foreground classes, while other proposals are labeled
as background. Note that the background proposals here include not only the
real backgrounds in an image but also some foreground instances which are
predicted as OOD class.

With our proposed method, the OOD detection head can be trained with both
labeled data and unlabeled data. The loss function is defined as follows:

$\displaystyle\mathcal{L}^{ood}=\mathcal{L}^{ova}_{sup}+\lambda^{ood}\mathcal{L}^{ova}_{unsup},$
(5)

where $\mathcal{L}^{ova}_{sup}$ and $\mathcal{L}^{ova}_{unsup}$ denotes
training with labeled and unlabeled data following eq. 4, respectively.
$\lambda^{ood}$ controls the weight of learning with unlabeled data.

In experiments, when training on the unlabeled data, we only sample from the
pseudo-labeled instances and no longer sample from the background. Meanwhile,
we use a threshold $\tau^{ood}$ to select those high-confidence instances
predicted by our OOD detection head. Other pseudo-labeled foreground
instances, which are predicted as ID classes with
$0.5<p(\hat{y}^{y}|\bm{x_{u}})$ but with a lower confidence score than our
threshold $p(\hat{y}^{y}|\bm{x_{u}})<\tau^{ood}$, will not be sampled since
the OOD head’s estimation on these instances may not be accurate. And those
proposals predicted as OOD with ($p(\hat{y}^{y}|\bm{x_{u}})<0.5$) by our OVA
head will be relabeled as the background class.

Following the original SSOD framework, the same strong augmentation on
unlabeled data and the burn-in stage training with labeled data only are also
applied. Finally, our entire framework can be described as in Alg. 1.

##  4 Experiments

###  4.1 Experimental setting and datasets

For a fair comparison, our method is evaluated on the following dataset
combinations proposed by [20]. We use mean Average Precision (mAP) as the
evaluation metric.

COCO-Open. We randomly select 20/40/60 classes as ID classes and the remaining
as OOD classes in MS-COCO 2017 [18] dataset with 80 classes and 117k images.
The dataset is divided into ID, MIX, and OOD sets by splitting the classes.
The images in the ID set contain only instances of ID classes. The images in
the OOD set contain only objects of OOD classes. And the images in the MIX set
contain both objects of ID classes and OOD classes. We then randomly sample
images with annotations from the ID set as the labeled dataset. The rest of
the ID set and other sets are combined as the open-set unlabeled dataset.

COCO-OpenImages. Our method is also evaluated on a large-scale dataset
combination with the entire MS-COCO as the labeled ID dataset and OpenImagesv5
[11] as the open-set unlabeled dataset. OpenImagesv5 contains 1.7M images with
601 classes. Those classes which do not appear in MS-COCO are treated as OOD
classes.

###  4.2 Implementation details

For a fair comparison with [20], we choose the Unbiased Teacher [21] as the
basic SSOD framework. The Unbiased Teacher uses Faster R-CNN [22] as the
object detector with ResNet-50 [7] as the backbone and Feature Pyramid Network
(FPN) [17]. For the main hyper-parameters in our methods, the threshold for
filtering outlier pseudo labels is set to $\tau^{\prime}=0.5$. The weight of
training the OOD head with unlabeled data is set to $\lambda^{ood}=0.1$. The
threshold for select ID instances in unlabeled for semi-supervised learning is
set to $\tau^{ood}=0.7$.

When training the model, we use a batch size of 16 labeled and 16 unlabeled
images for COCO-Open and a batch size of 32 labeled and 32 unlabeled images
for COCO-OpenImages. All experiment is conducted with NVIDIA A100 GPU. We use
4 GPUs for COCO-Open and 8 GPUs for COCO-OpenImages. The other SSOD hyper-
parameters, like the weight of learning with unlabeled data $\lambda$ and the
weight of EMA $\alpha$, are kept the same as the original Unbiased Teacher’s
setting and will be reported in detail in the appendix. The whole framework is
based on Detectron2 [29].

###  4.3 Experiments on OSSOD benchmarks

In the experiments, we first show our state-of-the-art performance by
comparing it with the previous OSSOD method. Then, we show our method’s
effectiveness on different supervision degrees and the number of ID classes.
In ablation studies, we show the performance of using different OOD detection
heads and the effectiveness of our semi-supervised outlier filtering method.
We will conduct more ablation studies in the appendix.

Comparison with the previous OSSOD method. We first show the effectiveness of
our method on the large-scale data combination of MS-COCO and OpenImagesv5.
For a fair comparison, the backbone of our detector is also initiated with
DINO [3] pre-trained weights. The result in Table 1 shows that our method
improves the performance of the Unbiased Teacher significantly, from 41.81 mAP
to 44.07 mAP, and gets the state-of-the-art performance.

Table 1: Experimental results on COCO-OpenImages. Method | Labeled | Unlabeled | mAP $\uparrow$
---|---|---|---
Fully-supervised | COCO | OpenImagesv5 | 40.90
Fully-supervised | COCO+OpenImagesv5 | - | 42.91
Unbiased Teacher[21] | COCO | OpenImagesv5 | 41.81
Liu et al.[20] | COCO | OpenImagesv5 | 43.14
Ours | COCO | OpenImagesv5 | 44.07

Different combinations of unlabeled sets. We also report the performances on
different combinations of the COCO-Open dataset. Note that we did not directly
compare the performance of our method with previous methods on COCO-Open since
we don’t know their selected ID classes and labeled data, and the results can
vary greatly depending on this. In the following experiments, we use ImageNet
[4] pre-trained weights instead of DINO for a fair comparison since it is the
original setting of the Unbiased Teacher. And we found that simply using the
DINO pre-trained weights can improve the detection performance.

Table 2: Experimental results on COCO-Open under different data combinations.

Data combinations | ID | ID+MIX | ID+MIX+OOD
---|---|---|---
# of images | 38,461 | 105,759 | 117,266
Unbiased Teacher[21] | 12.46 | 18.04 | 19.45
Ours | 12.71 | 19.00 | 20.15

Table 3: Experimental results on COCO-Open under different numbers of labeled
images.

# of labeled Images | 1000 | 2000 | 4000
---|---|---|---
Unbiased Teacher[21] | 8.94 ($\pm$0.22) | 15.29 ($\pm$0.87) | 18.36 ($\pm$0.33)
Ours |  10.12 ($\pm$0.17) |  15.89 ($\pm$0.26) |  19.86 ($\pm$0.86)

Table 4: Experimental results on COCO-Open under different numbers of ID
classes.

# of ID classes | 20 | 40 | 60
---|---|---|---
Unbiased Teacher[21] | 13.42 ($\pm$1.38) | 18.36 ($\pm$0.33) | 16.40 ($\pm$0.28)
Ours |  14.29 ($\pm$1.43) |  19.86 ($\pm$0.86) |  16.89 ($\pm$0.07)

We evaluated the performance of our method by using different combinations of
labeled and unlabeled data. We sample 4000 labeled data from the ID set and
use unlabeled data from the pure ID set, ID+MIX set, and ID+MIX+OOD set. The
results in Table 4 show that our method not only works better on open-set
unlabeled data but also can slightly improve the performance on pure-ID data.

From the result, we also found that even using only the SSOD method,
increasing the number of OOD data will not always decrease the performance (in
our experiments, the result of ID+MIX+OOD is better than ID and ID+MIX). The
reason may be that there are more valuable ID data in the MIX set than the OOD
data. Additionally, the OOD set may contain some ignored ID instances or
certain OOD classes that have semantic similarities with ID classes, which can
help to improve the training results on limited labeled data.

Different degrees of supervision. We also evaluated our method by varying the
number of labeled images. We ran each experiment two times with different
random seeds and reported the standard deviation. Table 4 shows the
performance on the ID+MIX set of COCO-Open with 40 ID classes and
1000/2000/4000 labeled images. The result shows that our method can improve
the performance on different degrees of labeled data.

Varying numbers of ID classes. In addition, we evaluated our method by varying
the number of ID classes. We ran each experiment two times with different
random seeds and reported the standard deviation. Table 4 shows the
performance on the ID+MIX set of COCO-Open with different randomly selected ID
classes, all with 4000 labeled images. The result shows that our method is
also effective with different ID classes.

Meanwhile, we observed that the improvement over the baseline brought by our
method varies when different ID classes are selected. The reason may be that
certain classes are inherently easier to distinguish from other classes. When
selecting these classes as ID classes, the baseline SSOD method itself is
capable of partially distinguishing between ID and OOD classes. On the
contrary, when selecting the classes which are more easily to be confused, the
baseline method struggles to distinguish the ID instances. In such cases, our
method brings greater improvements.

Different OOD heads. For the OOD detection head in our method, we evaluated
the performance of several OOD detection methods. Here the Unbiased Teacher
itself can be treated as an MSP base method since it only uses the
classification score to filter the pseudo labels. We used an open-set
detection method, OpenDet2, since it is also used to find new classes. But
their task is different from ours. They want to improve the accuracy of
detecting new classes, but our focus is to ensure that these new OOD classes
do not impact the accuracy of the ID classes. And we also tried adding a
simple binary classification head to distinguish between ID and OOD classes,
named Entropy in the table. When conducting experiments, we follow the same
process while only changing the OOD detection head’s structure and the related
loss function for each method. All the experiments are conducted on the same
dataset combination consisting of 4000 labeled data and 40 ID classes. The
results in Table 5 show that all the OOD methods can help improve our task’s
performance, and the OVA head performs better than others.

Table 5: Ablation study on difference OOD detection methods. Methods | mAP
---|---
Unbiased Teacher [21] | 18.69
Energy [19] | 19.80
OE [9] | 18.70
OpenDet2 [6] | 19.02
Entropy | 20.50
Ours (OVA) | 20.72

##  5 Limitations and further works

Firstly, though the inference efficiency of the proposed method is high, we
still require an additional OOD detection head in the training process, which
needs more computational resources. Secondly, the OOD detection head we used
is transferred from the image classification task. Considering the significant
differences between object detection and image classification, designing a new
OOD detection method specifically for this task could potentially further
improve the performance. We will research these two problems in the future and
further improve existing OSSOD methods.

##  6 Conclusions

In this paper, we proposed an online end-to-end trainable open-set semi-
supervised object detection framework and a novel semi-supervised outlier
filtering method to solve the problem of SSOD on open-set unlabeled data.
Experimental results on several benchmarks demonstrate that the proposed
method archives better accuracy and efficiency compared with the state-of-the-
art methods. We also explored the performances of different OOD detection
methods. With our proposed method, when given an object detection task, in
addition to training with precious labeled data, we can leverage more existing
unlabeled data to improve the performance of the model without the need for
any additional manual labeling and filtering. Therefore, our method has a high
practical value, particularly when the target task has a limited amount of
close-set data available.

## References

  * [1] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel.  Mixmatch: A holistic approach to semi-supervised learning.  Advances in neural information processing systems, 32, 2019.
  * [2] Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao.  Yolov4: Optimal speed and accuracy of object detection.  arXiv preprint arXiv:2004.10934, 2020.
  * [3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
  * [4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [5] Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan.  Exploring the limits of out-of-distribution detection.  Advances in Neural Information Processing Systems, 34:7068–7081, 2021.
  * [6] Jiaming Han, Yuqiang Ren, Jian Ding, Xingjia Pan, Ke Yan, and Gui-Song Xia.  Expanding low-density latent regions for open-set object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [8] Dan Hendrycks and Kevin Gimpel.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  arXiv preprint arXiv:1610.02136, 2016.
  * [9] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich.  Deep anomaly detection with outlier exposure.  arXiv preprint arXiv:1812.04606, 2018.
  * [10] KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian.  Towards open world object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5830–5840, 2021.
  * [11] Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Shahab Kamali, Matteo Malloci, Jordi Pont-Tuset, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Murphy.  Openimages: A public dataset for large-scale multi-label and multi-class image classification.  Dataset available from https://storage.googleapis.com/openimages/web/index.html, 2017.
  * [12] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin.  A simple unified framework for detecting out-of-distribution samples and adversarial attacks.  Advances in neural information processing systems, 31, 2018.
  * [13] Gang Li, Xiang Li, Yujie Wang, Yichao Wu, Ding Liang, and Shanshan Zhang.  Pseco: Pseudo labeling and consistency training for semi-supervised object detection.  In European Conference on Computer Vision, pages 457–472. Springer, 2022.
  * [14] Gang Li, Xiang Li, Yujie Wang, Wu Yichao, Ding Liang, and Shanshan Zhang.  Dtg-ssod: Dense teacher guidance for semi-supervised object detection.  Advances in Neural Information Processing Systems, 35:8840–8852, 2022.
  * [15] Hengduo Li, Zuxuan Wu, Abhinav Shrivastava, and Larry S Davis.  Rethinking pseudo labels for semi-supervised object detection.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36(2), pages 1314–1322, 2022.
  * [16] Shiyu Liang and Yixuan Li.  Enhancing the reliability of out-of-distribution image detection in neural networks.  arXiv preprint arXiv:1706.02690, 2017.
  * [17] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [18] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [19] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li.  Energy-based out-of-distribution detection.  Advances in neural information processing systems, 33:21464–21475, 2020.
  * [20] Yen-Cheng Liu, Chih-Yao Ma, Xiaoliang Dai, Junjiao Tian, Peter Vajda, Zijian He, and Zsolt Kira.  Open-set semi-supervised object detection.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXX, pages 143–159. Springer, 2022.
  * [21] Yen-Cheng Liu, Chih-Yao Ma, Zijian He, Chia-Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Peter Vajda.  Unbiased teacher for semi-supervised object detection.  In International Conference on Learning Representations, 2021.
  * [22] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [23] Kuniaki Saito, Donghyun Kim, and Kate Saenko.  Openmatch: Open-set semi-supervised learning with open-set consistency regularization.  Advances in Neural Information Processing Systems, 34:25956–25967, 2021.
  * [24] Kuniaki Saito and Kate Saenko.  Ovanet: One-vs-all network for universal domain adaptation.  In Proceedings of the ieee/cvf international conference on computer vision, pages 9000–9009, 2021.
  * [25] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  Advances in neural information processing systems, 33:596–608, 2020\.
  * [26] Kihyuk Sohn, Zizhao Zhang, Chun-Liang Li, Han Zhang, Chen-Yu Lee, and Tomas Pfister.  A simple semi-supervised learning framework for object detection.  arXiv preprint arXiv:2005.04757, 2020.
  * [27] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  Advances in neural information processing systems, 30, 2017.
  * [28] Xinjiang Wang, Xingyi Yang, Shilong Zhang, Yijiang Li, Litong Feng, Shijie Fang, Chengqi Lyu, Kai Chen, and Wayne Zhang.  Consistent targets provide better supervision in semi-supervised object detection.  arXiv preprint arXiv:2209.01589, 2022.
  * [29] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * [30] Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang, Lijuan Wang, Fangyun Wei, Xiang Bai, and Zicheng Liu.  End-to-end semi-supervised object detection with soft teacher.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3060–3069, 2021.
  * [31] Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa.  Multi-task curriculum framework for open-set semi-supervised learning.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16, pages 438–454. Springer, 2020.
  * [32] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang.  Open-vocabulary object detection using captions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14393–14402, 2021.
  * [33] Hongyu Zhou, Zheng Ge, Songtao Liu, Weixin Mao, Zeming Li, Haiyan Yu, and Jian Sun.  Dense teacher: Dense pseudo-labels for semi-supervised object detection.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part IX, pages 35–50. Springer, 2022.