# Discovery-and-Selection$:$ Towards Optimal Multiple Instance Learning for
Weakly Supervised Object Detection

Shiwei Zhang, Wei Ke, Lin Yang, Qixiang Ye, Xiaopeng Hong, Yihong Gong, and
Tong Zhang

###### Abstract

Weakly supervised object detection (WSOD) is a challenging task that requires
simultaneously learn object classifiers and estimate object locations under
the supervision of image category labels. A major line of WSOD methods roots
in multiple instance learning which regards images as bags of instance and
selects positive instances from each bag to learn the detector. However, a
grand challenge emerges when the detector inclines to converge to
discriminative parts of objects rather than the whole objects. In this paper,
under the hypothesis that optimal solutions are included in local minima, we
propose a discovery-and-selection approach fused with multiple instance
learning (DS-MIL), which finds rich local minima and select optimal solutions
from multiple local minima. To implement DS-MIL, an attention module is
designed so that more context information can be captured by feature maps and
more valuable proposals can be collected during training. With proposal
candidates, a re-rank module is designed to select informative instances for
object detector training. Experimental results on commonly used benchmarks
show that our proposed DS-MIL approach can consistently improve the baselines,
reporting state-of-the-art performance.

###### Index Terms:

Weakly Supervised Object Detection, Training Strategy, Self-supervised
Attention

##  I Introduction

Weakly supervised object detection (WSOD) has been attracted increasing
attention, due to its effortless annotation that only needs indicator vectors
to demonstrate the existence of each class [1, 2, 3, 4, 5, 6, 7]. Compared
with fully supervised object detection which requires labor-intensive
bounding-box annotations, WSOD significantly reduces the workload of data
annotation. With WSOD, people can leverage rich images with tags on the
internet to learn object-level models, and thereby convert human-supervised
object detection to Webly supervised object modeling.

Multiple Instance Learning (MIL) [8] has been the cornerstone of many WSOD
methods, either with hand-crafted features [9, 10] or deep learning pipelines
[1, 3, 4, 6, 11]. With MIL, images are decomposed to bags of proposals
(instances). Each image from the classes of interest has at least one positive
instance and images from negative classes has no positive instance. WSOD is
considered as an instance classification problem, where object detectors are
constructed by alternates training the classifier and selecting positive
candidate proposal.

MIL-based WSOD networks usually focus on classifier learning and feature
learning, which roughly choose the high-scored candidate as positive samples
for the object localization. Consequently, the detectors rely on
classification score outputted by the MIL classifier, resulting in noisy
proposals of poor localization. The noisy proposals are typically
discriminative object parts instead of whole object extent.

To alleviate the impact of noisy proposals, one solution is re-training an
object detector with pseudo ground-truths (top-scoring proposals) generated by
weakly-supervised object detectors [12, 3, 4, 6]. However, because the number
of the noisy proposals are usually greater than the optimal solution, the
noisy proposal introduced in the training phase could seriously deteriorate
the trained detectors.

The other solution is to explore sophisticated optimization strategies. The
C-MIL method [6] recognized this problem by decomposing the complicated
optimization problem to multiple sub-optimization problems which are easy to
be solved. Nevertheless, as shown in Fig. 1(a), C-MIL remains getting stuck to
the local minimum when the continuation parameters are not properly defined.

Figure 1: Comparison of CMIL based approaches and our DS-MIL. (a) shows that
C-MIL introduced the continuation method into WSOD, but still can not solve
the non-convexity problem completely and localize only part of object (Red
box). (b) shows our motivation which introduce a Discovering Module to find
more local-minima and a Selection Module to choose in the found instances.
This alleviates the non-convexity problem and localizes full object extent
(Green box).

In this paper, we introduce a discovery-and-selection training strategy in
Fig. 1(b) to multiple instance learning network and thereby create DS-MIL. DS-
MIL is implemented by introducing an instance discovery module and an instance
selection module to the multiple instance learning network. It aims to
discover multiple local minima and then select the best sample in these
multiple local minima, alleviating the local minimum issue in WSOD in an
simple-yet-effective fashion.

For the discovery term, inspired by non-local network [13], a self-attention
module is designed so that the feature maps of CNN capture context information
of the object proposals generated by Selective Search. In this manner, we can
find rich local minima, which increases the probability to obtain optimal
solutions during multiple instance learning. For the selection term, we take
an Expectation-Maximization algorithm to re-rank the confidence of the object
proposals, in which we explicitly model instance assignment as a hidden
variable and derive the pseudo-label generation scheme to conduct the E and M
steps respectively. The algorithm assign a high score to the proposals which
lays a decisive role to determine an proposal bag whether belongs to positive.

The contributions of this study are summarized as follows:

  1. 1.

We propose the discovery-and-selection training strategy for WSOD, solving the
local minimum issue of multiple instance learning under the hypothesis that
optimal solutions are included in local minima.

  2. 2.

We design a proposal discovery module which leverages localization information
from multiple locations and finds more reliable proposals. We propose a novel
proposal selection module, which utilize self-supervised attention mechanism
to optimize instance proposals.

  3. 3.

Experimental results on commonly used benchmarks show our proposed DS-MIL
approach can consistently improve the baselines, achieving state-of-the-art
performance.

The rest of this paper is organized as follows: In Section II, we review
related research. In Section III we describe the proposed approach in details.
Experimental results are shown and discussed in Section IV, and we made a
conclusion of our work in Section V.

Figure 2: The Architecture of DS-MIL Network. (1) Discovering Module: Generate
comprehensive features and find more valuable proposals. (2) Selection Module:
Make a selection in all of found proposals. (3) Classification branch:
Generate pseudo labels for the first detection branch by MIL network. (4)
Detection branch: Feed the extracted features and generated pseudo labels to
the next regression branch for proposal classification and regression.

##  II Related Work

WSOD is an attractive computer vision task in which a detector is trained only
with image-level annotations. WSOD is usually solved with MIL based approach,
especially significantly boosted with convolutional neural networks.

###  II-A Multiple Instance Learning for WSOD.

MIL is effective to solve weakly supervised problem with coarse labels [8].
Positive and negative bags are used to train a instance-level classifier in
MIL. A positive bag is a set of instances at least one of which is positive
while a negative bag is a set of negative instances. The WSOD is natural to
treat as a MIL problem. Supposing image is a bag with candidate instances
which are generated by object proposal method [14]. The multi-fold MIL is
proposed to solve large-scale training dataset by diving it to several parts
[9]. In [10], full annotation of extra data is used to train a instance
detector, improving the performance of MIL by transferring representation.
However, the performance gap between weakly supervised and fully supervised
task is insurmountable with traditional MIL approaches.

###  II-B Deep Learning for WSOD

Recently, WSOD largely outperforms the previous State-of-the-arts by combining
deep neural networks and MIL. The Weakly Supervised Deep Detection (WSDDN) [1]
is firstly introduced to deal with WSOD, which is composed of a proposal
classifier branch and a proposal selector branch inspired by MIL. WSDDN
selects the positive samples by aggregating the score of the two branches and
its effectiveness attracts lots of works to follow its framework. The WSDDN
brings the WSOD into a new era.

Feature Learning based WSOD. [15] transfered tracked boxes from weakly-labeled
videos to weakly-labeled images as pseudo ground-truth to train the detector
directly on images. [16] proposed to fuse and filter object instances from
different techniques and perform pixel labeling with uncertainty and they used
the resulting pixel-wise labels to generate bounding boxes for object
detection and attention maps for multi-label classification. Others are
attempt to learn feature representation to gain better performance. [2]
proposed an end-to-end cascaded convolutional network to perform weakly
supervised object detection and segmentation in cascaded manner. [17] proposed
to learn a context-aware CNN with contrast-based contextual modeling. [18]
uses mask to hide the most discriminative part of a image to enforce the
feature extractor to capture the integral extent of object. [19] leverage the
complementary effects of WSOD and Weakly Supervised Semantic Segmentation to
build detection-segmentation cyclic collaborative frameworks. Comprehensive
Attention Self-Distillation (CASD) is proposed to balance feature learning
among all object instances[7]. [5] inspired by a classical thermodynamic
principle, proposed a min-entropy latent model (MELM) and recurrent learning
algorithm for weakly supervised object detection.

Proposal Refinement based WSOD. Several approaches focus on the refinement of
proposal localization. [12] introduces domain adaptation into WSOD to fine-
tune the network to collect class specific object proposals. In [3], Online
Instance Classifier Refinement (OICR) alleviates the part domination problem
by knowledge distillation. [4] is based on OICR, coming up with using proposal
clustering to improve proposal generation and using proposal clusters as
supervision. In order to generate more precise proposals for detection. [20]
designed a weakly supervised region proposal network, [21] proposed a tight
box mining method that leverages surrounding segmentation context derived from
weakly supervised segmentation to suppress low quality distracting candidates
and boost the high-quality ones. [10] proposed a multi-fold MIL detector by
re-labeling proposals and retraining the object classifier iteratively to
prevent the detector from being locked into inaccurate object locations. [22]
proposed a pseudo label excavation algorithm and a pseudo label adaptation
algorithm to refine the pseudo labels obtained by [3]. [23, 24, 11] integrate
bounding box regressor into weakly-supervised detector. [25] leverage weakly
supervised semantic segmentation to remove unprecise proposals.

Optimization Strategy for WSOD. [26] observes that the result of MIL based
detector is unstable when use different initialization and utilizes the
instability to improve the performance of the detector by fusing the results
of differently initialized detectors. C-MIL [6] is proposed in order to
alleviate the non-convexity problem by introducing continuation learning to
WSOD to simplify the original MIL loss function. [27] proposed a self-taught
learning approach to progressively harvest high-quality positive instances.
[28] introduces a generative adversarial segmentation module interacts with
the conventional detection module to avoid being trapped in local-minima.

###  II-C Weakly Supervised Video Action Localization

Similar with the setting of WSOD, Weakly Supervised Video Action Localization
aims to localize and classify the activities in an untrimmed video with only
action label to identify the video has what kind of actions. [29, 30] uses
attention in their methods to compute the importance of each clip. In order to
localize complete activities, some adversarial methods[31, 32] mask the most
conspicuous part of videos.[33] uses a prior that motionless video clips are
unlikely to be actions to separate action clips from complex background. [34,
35, 36] try to use other weak labels such as scripts,images from web or action
lists to train their model. [37] adopts Expectation-Maximization to make the
video proposal selection more accuracy. Inspired by [37], we take the same
selection strategy for object proposal selection, which also shows
effectiveness for WSOD.

###  II-D Attention in Object Detection.

Inspired by the process that humans selectively use an important part of the
data to make a decision, attention mechanism was first proposed to solve
natural language processing problems and then introduced to computer vision
areas[38, 39]. For object detection, attention mechanism could be classified
into two categories: features re-weighting [40, 41, 42] and loss regularizing
[43, 44]. Attention is called self-attention when query is set as itself.
Several previous works, $i.e.$, non-local attention [13] and relation
attention [45], indicate that self-attention is effective to learn the
meaningful representation for conducting the given task. We attempt to
optimizes the location and classification in WSOD by using both self-attention
to explore channel-wise feature re-weighting and normal attention for
proposal-wise loss regularization.

It’s worth exploring how to effectively take the complementary of the feature
learning and proposal selection. By incorporating the attention mechanism, we
propose discovering-and-selection strategy, which towards optimal multiple
instance learning for weakly supervised object detection.

Figure 3: Architecture of Selection Module. We alternate to optimize the
E-step branch(parameter $\sigma$) and M-step branch(parameter $\theta$). Both
scores of two branches are combined with image-level labels to generate pseudo
labels for each other.

##  III Method

###  III-A Revisiting MIL-based WSOD

MIL-based WSOD model usually follows a two-phase learning procedure, $i.e.$,
Classification branch and Detection branch for refinement and regression. It
denotes $I=\\{I^{1},I^{2},...,I^{T}\\}$ as an dataset with $T$ images and
$Y=\\{Y^{1},Y^{2},...,Y^{T}\\}$ indicating object presence or not. Different
from fully supervised object annotation with both location and category, the
$Y^{t}=[y^{t}_{1},y^{t}_{2},...,y^{t}_{C}]\in[0,1]^{C}$ is a binary vector
where $y^{t}_{c}=1$ indicates the presence of at least one object of the
$c$-th category, where $C$ indicates the total number of object categories in
the dataset.

Suppose $R^{t}$ is the candidate proposals for the $t$-th image. Each image is
pre-computed by Selective Search[14] to generate $N$ object proposals
$R^{t}=\\{R_{t}^{1},R_{t}^{2},...,R_{t}^{N}\\}$ for initialization. The
selected proposals $r$ is a latent variable and $R^{t}$ can be regarded as the
solution space. Denoting $\delta$ as the network parameters, the MIL model
with proposal selection $r^{\ast}$ and features $\delta^{\ast}$ to be learned,
can be defined as

$\begin{aligned}
\\{r^{\ast},\delta^{\ast}\\}&=\mathop{\arg\min}_{r,\delta}\mathcal{L}_{(I,Y)}(r,\delta)\\\
&=\mathop{\arg\min}_{r,\delta}(\mathcal{L}_{cls}+\mathcal{L}_{det})\end{aligned}\quad,$
(1)

where the image index $t$ are omitted for short and $\mathcal{L}_{cls}$ and
$\mathcal{L}_{det}$ are the loss functions of instance classification and
proposal detection respectively.

Initially, for instance classification term, the loss function is defined as

$\displaystyle\mathcal{L}_{cls}=-\sum_{c=1}^{C}\\{y_{c}\log
p_{c}(r;\delta)+(1-y_{c})\log(1-p_{c}(r;\delta))\\},$ (2)

where $p_{c}(r;\delta)$ is the joint probability of class $c$ and latent
variable $r$, given learned network parameter $\delta$. It is calculated by a
soft-max operation with the prediction score $s(r;\delta)$, as

$\displaystyle
p_{c}(r;\delta)=\dfrac{exp(s(r;\delta))}{\sum_{R}exp(s(r;\delta))}.$ (3)

Pseudo label $\hat{y}$ for each selection branch is selected from the top-
scoring proposals in previous stage.

Since we get pseudo labels, each proposal now has a bounding-box regression
target and classification target. As a consequence, Selection Loss can be
defined as:

$\displaystyle\mathcal{L}_{det}=\mathcal{L}_{refine}+\lambda\mathcal{L}_{regression}\
,$ (4)

where $\mathcal{L}_{refine}$ is the refine classification loss; and
$\mathcal{L}_{regression}$ is bounding box regression loss. $\lambda$ is used
as a weight to balance the two losses. During the learning, a object detector
is learned to generate instance bags by using the refine loss defined as:

$\displaystyle\mathcal{L}_{refine}=-\sum_{r^{\ast}}\log p(r^{\ast},\delta)\ ,$
(5)

where $p(r^{\ast},\delta)$ prediction score of the pseudo object with soft-max
operation. For bounding box regression loss, smooth-L1 loss is adopted:

$\displaystyle\mathcal{L}_{regression}=\dfrac{1}{N}\sum_{r}\mathcal{L}_{smooth_{L1}}(Target(r),Box(r))\
,$ (6)

where $Box(r)$ is the predicted box for proposal r, and $Target(r)$ is the
regression target generated by pseudo label.

###  III-B DS-MIL method

Optimizing the non-convex loss function and performing instance selection
still remain to be elaborated in WSOD approaches. In C-MIL [6], a continuation
strategy is used in MIL to alleviate these two problems. However, C-MIL is
still easy to be stuck into local minima because the parameters are hard to
choose and the optimization strategy is complex. As a consequence, we decide
to propose a novel training strategy to solve these problems. We recognize
WSOD as a Discovering-and-Selection process, and design the Discovering Module
and Selection Module to model this process, as shown in Fig. 2.

####  III-B1 Discovey

Dealing with localization ambiguity under only classical Convolution layers is
difficult, the high responses are focus on the most discriminative part,
therefore only a few instances are mined. As a consequence, we propose to
integrate a Discovering module into the network to capture more context
information and enforce the feature learning to learn complete object feature.
That means this module could help us discover more important instances.
Following [13], the general self-attention mechanism is defined as:

$v_{m}=\dfrac{1}{C(u_{m})}\sum_{\forall n}f(u_{m},u_{n})g(u_{n})+u_{m},$ (7)

where $u$ and $v$ denote input and output feature, $m$ and $n$ are
corresponding spatial position index . The output signal is normalized by
$C(u_{m})=\sum_{\forall n}f(u_{m},u_{n})$. Function $g(u_{n})$ gives a
representation of input signal $u_{n}$ at each position and all of them are
aggregated into position $m$ with the similarity weights given by
$f(u_{m},u_{n})={\varphi(u_{m})^{T}\varphi(u_{n})}$, which calculates the dot-
product pixel affinity in an embedding space. Here, we take the inner-product
to calculate the affinity between channels and integrate the similarity
weights into Eq.13.

$v_{m}=\dfrac{1}{C(u_{m})}\sum_{\forall
n}Softmax({\varphi(u_{m})^{T}\varphi(u_{n})})\hat{v}_{n},$ (8)

where $\hat{v}_{n}$ is the original feature map. And the similarities are
activated by Softmax. The final feature map is the weighted sum of the
original feature map with normalized similarities. For the final feature map,
because each part of it combines with other parts, more areas will be
activated, part-domination could be improved. The self-attention module
structure is illustrated in Fig. 2. Compared to other self-attention methods,
our proposed self-attention method has two differences: Firstly, we implement
self-attention module on instance-level, which can avoid instance level
feature map mixing other information and save a lot of computation capacity.
Secondly, we cancel the residual connection to avoid changing the activation
intensity.

Figure 4: Heatmaps of baseline and our method depict the effectiveness of
Discovery Module. The heatmaps of baselines shows that baseline method only
activate discriminative region for classification. The heatmaps of DS-MIL
verify that DS-MIL could activate the full object region.

####  III-B2 Selection

Inaccurate classification score for proposals easily cause the localization
ambiguity, e.g., Proposals cover only part of object have higher score. We
propose a selection module to find the confident proposal from the proposal
pool produced by Discovery Module, which is inspired by [37].

From the MIL setting, the proposals cover object determine the label of an
image, while the proposals only cover background can not affect the label of
an image. The proposal is regarded as key proposals when it covers the object
in the image. A binary variable $h_{i}\in\\{0,1\\}$ is used to indicate
whether proposal $R_{i}$ cover the object. We use one estimator $\phi$ to
estimate the probability of a proposal to be a key proposal and one predictor
$\theta$ to predict the probability of a proposal belonging to different
categories. The selection module is defined as

$\begin{aligned}
{\theta^{*},h^{*}}=&\mathop{\arg\max}_{\theta,h}p_{\theta}(y_{c}=1|R,h)\\\
&=\mathop{\arg\max}_{\theta,h}p_{\theta}(y_{c,i}=1|R_{i})\cdot[h_{i}=1]\\\
\end{aligned}\quad,$ (9)

where the maximum operator select the most important proposal for the image,
and $p_{\theta}(y_{c,i}=1|R_{i})$ represents the probability that proposal
$R_{i}$ is classified to the $c$-th category.

As $h$ is a latent variable, $p_{\theta}$ in Eq. 9 could be reformulated [37]
as

$\small\begin{aligned} \log p_{\theta}(y_{c}|R)=&KL(q_{\phi}(h|R)\parallel
p_{\theta}(h|R,y_{c})+\\\ &+\int
q_{\phi}(h|R)l\log\dfrac{p_{\theta}(h,y_{c}|R)}{q_{\phi}(h|R)}dh\\\ &\geq\int
q_{\phi}(h|R)\log p_{\theta}(h,y_{c}|R)dh+H(q_{\phi}(h|R)\end{aligned}\quad,$
(10)

We also use the EM algorithm [46] to optimize $q_{\phi}$ following [37].
Minimizing $KL(q_{\phi}(z_{i}|R_{i})||p_{\theta}(z_{i}|R_{i},y_{c}))$ to
tighten the lower bound in E-step and maximize the lower bound to optimize
$p_{\theta}$ in M-step, as shown in Fig. 3.

In E-step, in order to optimize $q_{\phi}$, we assume the posterior
$p_{\theta}(z_{i}|R_{i},y_{c})$ is proportional to the proposal-level
classification score $p_{\theta}(y_{c}|R_{i})$. Therefore, a pseudo label is
constructed by combining the classification score and the ground-truth class
label. The pseudo-label is formulated as

$\small\hat{h}_{t}=\left\\{\begin{array}[]{lr}1,\text{if}\sum_{c=1}^{C}\mathbbm{1}(p_{\theta}(y_{c}|R_{i})>\zeta\text{
and }y_{c}=1)>0&\\\ 0,\text{otherwise}&\end{array}\right.\quad,$ (11)

where $\zeta$ is a threshold. If an proposal has a classification score over
the threshold for any ground-truth class within the image, the proposal is
regarded as a positive proposal. Otherwise, it is regarded as a negative
proposal. With the pseudo labels, $\phi$ is updated by the binary cross
entropy (BCE) loss as

$\small\mathcal{L}(\phi)=-\hat{h}_{i}\log
q_{\phi}(h_{i}|x_{i})-(1-\hat{h}_{i})\log(1-q_{\phi}(h_{i}|x_{i})).$ (12)

In M-step, as with regard to $\theta$, $H(q_{\phi}(h|R)$ is constant. We
maximize $\int q_{\phi}(z|R)\log p_{\theta}(z,y_{c}|R)dz$, which can be
achieved by optimizing the classification score $p_{\theta}(y_{c}|R_{i})$
given proposal importance $q_{\phi}(z|R_{i})$. As a result, we combine the
proposal importance $q_{\phi}$ and the ground truth class labels to generate a
pseudo label, as

$\hat{y}_{t,c}=\left\\{\begin{array}[]{lr}1,\text{if}y_{c}=1\ \text{and}\
q_{\phi}(z_{i}|R_{i})>\xi,&\\\ 0,\text{otherwise}&\end{array}\right.\quad,$
(13)

where $\xi$ is a dynamic threshold, which is the mean of proposal importance.
Proposals whose importance are higher than the threshold are recognized as
positive, and the importance of negative proposals are lower than the
threshold. With the pseudo labels, we also derive a BCE loss to optimize
$\theta$ as

$\small\mathcal{L}(\theta)=-\hat{y}_{i,c}\log
p_{\theta}(y_{c}|R_{i})-(1-\hat{y}_{i,c})\log(1-p_{\theta}(y_{c}|R_{i})).$
(14)

##  IV Experiment

###  IV-A Datasets and Evaluation Metrics

TABLE I: Ablation study of DS-MIL with Discovering Module (D) and Selection Module (S). Method |  Dataset |  $mAP$
---|---|---
MELM [5] |  VOC2007 |  47.3
MELM+S |  VOC2007 |  48.5
MELM+D |  VOC2007 |  47.8
MELM+2D |  VOC2007 |  48.2
MELM+S+2D |  VOC2007 |  49.5
MELM+S+2D+Reg |  VOC2007 |  55.1
TABLE II: Ablation study of DS-MIL with different baselines.  Method |  Dataset |  $mAP$
---|---|---
OICR [3] |  VOC2007 |  41.2
OICR+S+2D |  VOC2007 |  44.9
PCL [4] |  VOC2007 |  43.5
PCL+S+2D |  VOC2007 |  45.7
TABLE III: Ablation study of the Number of detection branches (K). Method |  K |  $mAP$
---|---|---
DS-MIL |  1 |  50.9
DS-MIL |  2 |  53.8
DS-MIL |  3 |  55.1
DS-MIL |  4 |  53.2

In experiment, we evaluate our approach on three popular datasets: PASCAL VOC
2007&2012[47] and MS-COCO[48]. PASCAL VOC 2007&2012 datasets[47] have 9962 and
22531 images of 20 object classes respectively. Only image-level annotations
are used as supervision in all experiments. For PASCALC VOC, we use the
trainval set(5011 images for 2007 and 11540 for 2012) for training and test
set for testing. For evaluation on PASCAL VOC, two metrics are used to
evaluate our model. First, we evaluate detection performance using mean
Average Precision (mAP) on the PASCAL VOC 2007 and 2012 test set. Second, we
evaluate the localization accuracy using Correct Localization (CorLoc) on
PASCAL VOC 2007 and 2012 trainval set. Based on the PASCAL criterion, a
predicted box is considered positive if it has an IoU $>$ 0.5 with a ground-
truth bounding box. MS-COCO[48] contains 80 categories. We train on train2017
split and evaluate on val2017 split, which consists of 118287 and 5000 images,
respectively. $mAP_{0.5}$ (IoU threshold at 0.5) and $mAP$ (averaged over IoU
thresholds in [0.5 : 0.05 : 0.95]) on val2017 are reported.

###  IV-B Implementation Details

VGG16 [49] pre-trained on ImageNet [50] is used as the backbone in experiment.
Selective Search[14] is used to generate about 2,000 proposals per-image for
PASCAL VOC and MCG is used for MS-COCO. The maximum iteration numbers are set
to be 150k, 160k and 300k for VOC 2007, VOC 2012 and MS-COCO respectively. For
Selection Module, we alternate EM step every 3000 iterations in the first
30000 iterations, then we optimize them jointly. The whole WSOD network is by
stochastic gradient descent (SGD) with a momentum of 0.9, an initial learning
rate of 0.001 and a weight decay of 0.0005. The learning rate will decay with
a factor of 10 at the 75kth, 80kth and 150kth iterations for VOC 2007, VOC
2012 and MS-COCO, respectively. The total number of refinement branches is set
to be 3. For data augmentation, we use six image scales
$\\{480,576,688,864,1000,1200\\}$ (resize the shortest side to one of these
scales) and cap the longest image side to less than 2000 with horizontal flips
for both training and testing.

TABLE IV: The object detection performance comparison with state-of-the-art on
PASCAL VOC 2007 TEST SET (VGG16 backbone, $mAP_{0.5}$).

Method | areo | bike | bird | boat | bottle | bus | car | cat | chair | cow | table | dog | horse | motor | person | plant | sheep | sofa | train | tv | $mAP$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
WSDDN[1] | 46.4 | 58.1 | 35.5 | 25.9 | 14.0 | 66.7 | 53.0 | 39.2 | 8.9 | 41.8 | 26.6 | 38.6 | 44.7 | 59.0 | 10.8 | 17.3 | 40.7 | 49.6 | 56.9 | 50.8 | 34.8
OICR[3] | 58.5 | 63.0 | 35.1 | 16.9 | 17.4 | 63.2 | 60.8 | 34.4 | 8.2 | 49.7 | 41.0 | 31.3 | 51.9 | 64.8 | 13.6 | 23.1 | 41.6 | 48.4 | 58.9 | 58.7 | 41.2
DSTL[27] | 52.2 | 47.1 | 35.0 | 26.7 | 15.4 | 61.3 | 66.0 | 54.3 | 3.0 | 53.6 | 24.7 | 43.6 | 48.4 | 65.8 | 6.6 | 18.8 | 51.9 | 43.6 | 53.6 | 62.4 | 41.7
WCCN[2] | 49.5 | 60.6 | 38.6 | 29.2 | 16.2 | 70.8 | 56.9 | 42.5 | 10.9 | 44.1 | 29.9 | 42.2 | 47.9 | 64.1 | 13.8 | 23.5 | 45.9 | 54.1 | 60.8 | 54.5 | 42.8
PCL[4] | 57.1 | 67.1 | 40.9 | 16.9 | 18.8 | 65.1 | 63.7 | 45.3 | 17.0 | 56.7 | 48.9 | 33.2 | 54.4 | 68.3 | 16.8 | 25.7 | 45.8 | 52.2 | 59.1 | 62.0 | 43.5
ZLDN[51] | 55.4 | 68.5 | 50.1 | 16.8 | 20.8 | 62.7 | 66.8 | 56.5 | 2.1 | 57.8 | 47.5 | 40.1 | 69.7 | 68.2 | 21.6 | 27.2 | 53.4 | 56.1 | 52.5 | 58.2 | 47.6
MELM[5] | 55.6 | 66.9 | 34.2 | 29.1 | 16.4 | 68.8 | 68.1 | 43.0 | 25.0 | 65.6 | 45.3 | 53.2 | 49.6 | 68.6 | 2.0 | 25.4 | 52.5 | 56.8 | 62.1 | 57.1 | 47.3
C-WSL[24] | 62.7 | 63.7 | 40.0 | 25.5 | 17.7 | 70.1 | 68.3 | 38.9 | 25.4 | 54.5 | 41.6 | 29.9 | 37.9 | 64.2 | 11.3 | 27.4 | 49.3 | 54.7 | 61.4 | 67.4 | 45.6
WSPRN[20] | 57.9 | 70.5 | 37.8 | 5.7 | 21.0 | 66.1 | 69.2 | 59.4 | 3.4 | 57.1 | 57.3 | 35.2 | 64.2 | 68.6 | 32.8 | 28.6 | 50.8 | 49.5 | 41.1 | 30.0 | 45.3
C-MIL[6] | 62.5 | 58.4 | 49.5 | 32.1 | 19.8 | 70.5 | 66.1 | 63.4 | 20.0 | 60.5 | 52.9 | 53.5 | 57.4 | 68.9 | 8.4 | 24.6 | 51.8 | 58.7 | 66.7 | 63.6 | 50.5
WSOD2[23] | 65.1 | 64.8 | 57.2 | 39.2 | 24.3 | 69.8 | 66.2 | 61.0 | 29.8 | 64.6 | 42.5 | 60.1 | 71.2 | 70.7 | 21.9 | 28.1 | 58.6 | 59.7 | 52.2 | 64.8 | 53.6
C-MIDN[25] | 53.3 | 71.5 | 49.8 | 26.1 | 20.3 | 70.3 | 69.9 | 68.3 | 28.7 | 65.3 | 45.1 | 64.6 | 58.0 | 71.2 | 20.0 | 27.5 | 54.9 | 54.9 | 69.4 | 63.5 | 52.6
DS-MIL(ours) | 66.9 | 76.6 | 59.5 | 30.6 | 29.2 | 68.0 | 75.1 | 63.6 | 32.9 | 73.4 | 47.7 | 46.4 | 54.5 | 74.6 | 27.4 | 28.1 | 61.2 | 54.7 | 68.0 | 63.8 | 55.1

TABLE V: The Object Localization performance comparison with state-of-the-art
on the PASCAL VOC 2007 TRAINVAL SET (VGG16 backbone, $Corloc$).

Method | areo | bike | bird | boat | bottle | bus | car | cat | chair | cow | table | dog | horse | motor | person | plant | sheep | sofa | train | tv | $mAP$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
WSDDN[1] | 65.1 | 58.8 | 58.5 | 33.1 | 39.8 | 68.3 | 60.2 | 59.6 | 34.8 | 64.5 | 30.5 | 43.0 | 56.8 | 82.4 | 25.5 | 41.6 | 61.5 | 55.9 | 65.9 | 63.7 | 53.5
OICR[3] | 81.7 | 80.4 | 48.7 | 49.5 | 32.8 | 81.7 | 85.4 | 40.1 | 40.6 | 79.5 | 35.7 | 33.7 | 60.5 | 88.8 | 21.8 | 57.9 | 76.3 | 59.9 | 75.3 | 81.4 | 60.6
DSTL[27] | 72.7 | 55.3 | 53.0 | 27.8 | 35.2 | 68.6 | 81.9 | 60.7 | 11.6 | 71.6 | 29.7 | 54.3 | 64.3 | 88.2 | 22.2 | 53.7 | 72.2 | 52.6 | 68.9 | 75.5 | 56.1
WCCN[2] | 83.9 | 72.8 | 64.5 | 44.1 | 40.1 | 65.7 | 82.5 | 58.9 | 33.7 | 72.5 | 25.6 | 53.7 | 67.4 | 77.4 | 26.8 | 49.1 | 68.1 | 27.9 | 64.5 | 55.7 | 56.7
PCL[4] | 79.6 | 85.5 | 62.2 | 47.9 | 37.0 | 83.8 | 83.4 | 43.0 | 38.3 | 80.1 | 50.6 | 30.9 | 57.8 | 90.8 | 27.0 | 58.2 | 75.3 | 68.5 | 75.7 | 78.9 | 62.7
ZLDN[51] | 74.0 | 77.8 | 65.2 | 37.0 | 46.7 | 75.8 | 83.7 | 58.8 | 17.5 | 73.1 | 49.0 | 51.3 | 76.7 | 87.4 | 30.6 | 47.8 | 75.0 | 62.5 | 64.8 | 68.8 | 61.2
MELM[5] | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 61.4
C-WSL[24] | 86.3 | 80.4 | 58.3 | 50.0 | 36.6 | 85.8 | 86.2 | 47.1 | 42.7 | 81.5 | 42.2 | 42.6 | 50.7 | 90.0 | 14.3 | 61.9 | 85.6 | 64.2 | 77.2 | 82.4 | 63.3
WSRPN[20] | 77.5 | 81.2 | 55.3 | 19.7 | 44.3 | 80.2 | 86.6 | 69.5 | 10.1 | 87.7 | 68.4 | 52.1 | 84.4 | 91.6 | 57.4 | 63.4 | 77.3 | 58.1 | 57.0 | 53.8 | 63.8
C-MIL[6] | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 65.0
WSOD2[23] | 87.1 | 80.0 | 74.8 | 60.1 | 36.6 | 79.2 | 83.8 | 70.6 | 43.5 | 88.4 | 46.0 | 74.7 | 87.4 | 90.8 | 44.2 | 52.4 | 81.4 | 61.8 | 67.7 | 79.9 | 69.5
C-MIDN[25] | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 68.7
DS-MIL(ours) | 83.2 | 82.0 | 76.9 | 52.9 | 47.1 | 84.0 | 87.8 | 72.3 | 47.5 | 81.7 | 49.2 | 50.5 | 70.8 | 92.7 | 43.5 | 54.9 | 79.6 | 53.7 | 80.5 | 78.5 | 68.5

###  IV-C Ablation Study

We conduct ablation experiments on PASCAL VOC 2007 to prove the effectiveness
of our proposed DS-MIL approach from 4 perspectives.

For discovery module, we adopt MELM [5] as our baseline to verify its
effectiveness. We add a single discovering module on the baselines, as shown
in Table I. MELM+D improves the performance to 47.8%, which indicates one
discovering module is used. The performance is further improved by 0.9% when
two discovery modules are used. To verify the effect of our newly proposed
selection module, we also used MELM [5] as baseline, as shown in Table I. As
Table I depicted, the selection module improves the performance of MELM by
1.2%. As bounding box regressor is plugged into several WSOD approach and
illustrated that it’s effective for performance gain [24]. Following [24], we
also add the regressor to the proposed approach, and we achieves 55.1% on
PASCAL VOC 2007.

As the discovery module and selection module are plug and play, we conduct
experiments with other two baselines besides MELM, $i.e.$, OICR [3] and PCL
[4]. The results verify that our method gain improvements in all of three
baselines. For each baseline, 1 selection module and 2 discovery modules are
added. In Table II, MELM with these modules gain a 2.2% improvement. In Table
II, the $mAP$ performance increases 3.7% for OICR and 2.3% for PCL.

TABLE VI: The Object detection performance on the PASCAL VOC 2012 TEST SET (VGG16 backbone, $mAP_{0.5}$).  Method |  $mAP$
---|---
OICR[3] |  37.9
PCL[4] |  40.6
C-MIL[6] |  46.7
WSOD2[23] |  47.2
C-MIDN[25] |  50.2
DS-MIL(ours) |  50.1
Figure 5: Visualization of DS-MIL results and the baseline(MELM). In upper part, the results of DS-MIL are shown in green boxes and the results of the baseline are shown in red boxes. In bottom part, some of failure cases of DS-MIL are shown.  TABLE VII: The Object Localization performance on the PASCAL VOC 2012 TRAINVAL SET (VGG16 backbone, $Corloc$) .  Method |  $mAP$
---|---
OICR[3] |  62.1
PCL[4] |  63.2
C-MIL[6] |  67.4
WSOD2[23] |  71.9
C-MIDN[25] |  71.2
DS-MIL(ours) |  70.4

In Fig. 4, we provide some comparisons between the heatmaps of baseline and
our approach. Obviously, the baseline activated the discriminated regions but
ignore full object extent. Compared to the baseline, DS-MIL shows great
performance by activating more regions to cover the whole object. The main
reason accounts for this result is our discovery module could capture more
object extent and provide more accurate object localization information for
detectors. On the contrary, baseline method only considers object
classification and hardly optimizes object localization.

The number of detection branches determines how many times we refine the
detection results. We also conduct some experiments on it. The number of
branches is set to be K, and four different Ks: 1,2,3,4 are adopted. While we
change the value of K, the rest of the hyper-parameters are fixed. Table III
shows the influence of K. We can find that when K is set to be 1, the mAP is
only 50.9%. Then, the performance becomes better with the increasing of K.
When K is set to be 3, it achieves the best performance which is 55.1%. And
the result decreases when the K is equal to 4. The reason is those chosen
proposals are too scattered for the 4th branch.

###  IV-D Comparison with Other Methods

VOC dataset In this comparison, we adopt MELM as our baseline, and add two
discovery modules and one selection module to the baseline network. Besides,
we use bounding box regressor to revise the location of the predicted boxes.
In order to verify the effectiveness of DS-MIL, 12 state-of-the-art WSOD
methods are compared with our method and most of the chosen methods are
published in the last two years. To fully compare with these methods, we
report both mAP results and Corloc results on VOC 2007 and VOC 2012 datasets
are shown in Table IV, Table V, Table VI and Table VII. From the Table IV, we
can see that our method outperforms all methods on VOC2007 dataset and
achieves the highest mAP performance on 9 out of 20 categories. From TableV,
the result is little lower than state-of-the-art methods, but our method also
achieves best performance on bird, bottle, bus, car, cat and train. Table VI
and Table VII shows the competitive results achieved by our method on VOC2012,
it is noteworthy that our proposed method outperforms 4 previous methods and
only little lower(0.4%) than the two-stage method C-MIDN+FRCNN[25].

MS-COCO dataset MS-COCO is larger dataset compared to PASCAL VOC, and only few
previous approaches report results on it for the difficulty of obtaining good
results on it. We report our results in Table VIII. We can find that our
proposed approach achieves 12.3% for $mAP$ and 24.5% for $mAP_{0.5}$ which
significantly outperforms previous works.

TABLE VIII: Comparison with SOTA WSOD results on MS-COCO (VGG16 backbone, $AveragePrecision$). Method |  $mAP$ |  $mAP_{0.5}$
---|---|---
WSDDN[1] |  - |  11.5
MELM[5] |  - |  18.8
PCL[4] |  8.5 |  19.4
PG-PS[52] |  - |  20.7
C-MIDN[25] |  9.4 |  21.4
WSOD2[23] |  10.8 |  22.7
DS-MIL(ours) |  12.3 |  24.5

###  IV-E Visualization

In Fig. 5, we visualize some detection results of our proposed method and the
baseline approach(MELM). The green boxes represent DS-MIL results and the red
boxes represent the baseline, respectively. The first two rows of Fig. 5
proves that our proposed approach largely improves the part-dominant problem
and the third row of Fig. 5 shows DS-MIL has the better capability to detect
multiple objects. As a consequence, we can conclude that DS-MIL performs much
better than the baseline. Moreover, the visualization results also shows that
our approach tends to cover more extent of objects and avoid selecting
incomplete proposals. And these are the effects of Selection Module and
Discovering Module. In the last row of Fig. 5, we also show some failure cases
of our method. As we can see, our detector will recognize multiple objects as
single object or miss some objects. These failures are come from two factors:
(1) The occlusion of objects. (2)The Selective Search algorithm[14] may not
generate good proposal. And we believe these problems could be improved by
applying network with stronger representation ability (e.g. transformer based
network) or combining with Class Activation Map.

##  V Conclusion

In this paper, We proposed an effective and novel method, referred to as DS-
MIL, for weakly supervised object detection. DS-MIL targets alleviating the
part-dominant problem of multiple instance learning using a new training
strategy: discovering-and-selection. This strategy is achieved by introducing
a self-supervised Discovering Module and a EM-based Selection Module. DS-MIL
significantly improved performance of weakly supervised object detection on
PASCAL VOC 2007 and MS-COCO datasets, and achieve competative results on
PASCAL VOC 2012 dataset.

## References

  * [1] H. Bilen and A. Vedaldi, “Weakly supervised deep detection networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2016.
  * [2] A. Diba, V. Sharma, A. Pazandeh, H. Pirsiavash, and L. Van Gool, “Weakly supervised cascaded convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 914–922.
  * [3] P. Tang, X. Wang, X. Bai, and W. Liu, “Multiple instance detection network with online instance classifier refinement,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2843–2851.
  * [4] P. Tang, X. Wang, S. Bai, W. Shen, X. Bai, W. Liu, and A. Yuille, “Pcl: Proposal cluster learning for weakly supervised object detection,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 42, no. 1, pp. 176–191, 2018.
  * [5] F. Wan, P. Wei, J. Jiao, Z. Han, and Q. Ye, “Min-entropy latent model for weakly supervised object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1297–1306.
  * [6] F. Wan, C. Liu, W. Ke, X. Ji, J. Jiao, and Q. Ye, “C-mil: Continuation multiple instance learning for weakly supervised object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2199–2208.
  * [7] Z. Huang, Y. Zou, V. Bhagavatula, and D. Huang, “Comprehensive attention self-distillation for weakly-supervised object detection,” _arXiv preprint arXiv:2010.12023_ , 2020.
  * [8] O. Maron and T. Lozano-Pérez, “A framework for multiple-instance learning,” in _NIPS_ , 1997, pp. 570–576.
  * [9] J. Hoffman, D. Pathak, T. Darrell, and K. Saenko, “Detector discovery in the wild: Joint multiple instance and representation learning,” in _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015_ , 2015, pp. 2883–2891.
  * [10] R. G. Cinbis, J. J. Verbeek, and C. Schmid, “Weakly supervised object localization with multi-fold multiple instance learning,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 39, no. 1, pp. 189–203, 2017.
  * [11] Z. Ren, Z. Yu, X. Yang, M.-Y. Liu, Y. J. Lee, A. G. Schwing, and J. Kautz, “Instance-aware, context-focused, and memory-efficient weakly supervised object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 10 598–10 607.
  * [12] D. Li, J.-B. Huang, Y. Li, S. Wang, and M.-H. Yang, “Weakly supervised object localization with progressive domain adaptation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 3512–3520.
  * [13] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7794–7803.
  * [14] J. R. Uijlings, K. E. Van De Sande, T. Gevers, and A. W. Smeulders, “Selective search for object recognition,” _International journal of computer vision_ , vol. 104, no. 2, pp. 154–171, 2013.
  * [15] K. K. Singh, F. Xiao, and Y. J. Lee, “Track and transfer: Watching videos to simulate strong human supervision for weakly-supervised object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 3548–3556.
  * [16] W. Ge, S. Yang, and Y. Yu, “Multi-evidence filtering and fusion for multi-label classification, object detection and semantic segmentation based on weakly supervised learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1277–1286.
  * [17] V. Kantorov, M. Oquab, M. Cho, and I. Laptev, “Contextlocnet: Context-aware deep network models for weakly supervised localization,” in _European Conference on Computer Vision_. Springer, 2016, pp. 350–365.
  * [18] J. Choe and H. Shim, “Attention-based dropout layer for weakly supervised object localization,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2219–2228.
  * [19] Y. Shen, R. Ji, Y. Wang, Y. Wu, and L. Cao, “Cyclic guidance for weakly supervised joint detection and segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 697–707.
  * [20] P. Tang, X. Wang, A. Wang, Y. Yan, W. Liu, J. Huang, and A. Yuille, “Weakly supervised region proposal network and object detection,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 352–368.
  * [21] Y. Wei, Z. Shen, B. Cheng, H. Shi, J. Xiong, J. Feng, and T. Huang, “Ts2c: Tight box mining with surrounding segmentation context for weakly supervised object detection,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 434–450.
  * [22] Y. Zhang, Y. Bai, M. Ding, Y. Li, and B. Ghanem, “W2f: A weakly-supervised to fully-supervised framework for object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 928–936.
  * [23] Z. Zeng, B. Liu, J. Fu, H. Chao, and L. Zhang, “Wsod2: Learning bottom-up and top-down objectness distillation for weakly-supervised object detection,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 8292–8300.
  * [24] M. Gao, A. Li, R. Yu, V. I. Morariu, and L. S. Davis, “C-wsl: Count-guided weakly supervised localization,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 152–168.
  * [25] Y. Gao, B. Liu, N. Guo, X. Ye, F. Wan, H. You, and D. Fan, “C-midn: Coupled multiple instance detection network with segmentation guidance for weakly supervised object detection,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 9834–9843.
  * [26] B. Liu, Y. Gao, N. Guo, X. Ye, F. Wan, H. You, and D. Fan, “Utilizing the instability in weakly supervised object detection.” in _CVPR Workshops_ , 2019.
  * [27] Z. Jie, Y. Wei, X. Jin, J. Feng, and W. Liu, “Deep self-taught learning for weakly supervised object localization,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 1377–1385.
  * [28] X. Li, M. Kan, S. Shan, and X. Chen, “Weakly supervised object detection with segmentation collaboration,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 9735–9744.
  * [29] L. Wang, Y. Xiong, D. Lin, and L. Van Gool, “Untrimmednets for weakly supervised action recognition and detection,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2017, pp. 4325–4334.
  * [30] P. Nguyen, T. Liu, G. Prasad, and B. Han, “Weakly supervised action localization by sparse temporal pooling network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 6752–6761.
  * [31] K. K. Singh and Y. J. Lee, “Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization,” in _2017 IEEE international conference on computer vision (ICCV)_. IEEE, 2017, pp. 3544–3553.
  * [32] R. Zeng, C. Gan, P. Chen, W. Huang, Q. Wu, and M. Tan, “Breaking winner-takes-all: Iterative-winners-out networks for weakly supervised temporal action localization,” _IEEE Transactions on Image Processing_ , vol. 28, no. 12, pp. 5797–5808, 2019.
  * [33] D. Liu, T. Jiang, and Y. Wang, “Completeness modeling and context separation for weakly supervised temporal action localization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * [34] O. Duchenne, I. Laptev, J. Sivic, F. Bach, and J. Ponce, “Automatic annotation of human actions in video,” in _2009 IEEE 12th International Conference on Computer Vision_. IEEE, 2009, pp. 1491–1498.
  * [35] C. Gan, C. Sun, L. Duan, and B. Gong, “Webly-supervised video recognition by mutually voting for relevant web images and web video frames,” in _European Conference on Computer Vision_. Springer, 2016, pp. 849–866.
  * [36] L. Ding and C. Xu, “Weakly-supervised action segmentation with iterative soft boundary assignment,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 6508–6516.
  * [37] Z. Luo, D. Guillory, B. Shi, W. Ke, F. Wan, T. Darrell, and H. Xu, “Weakly-supervised action localization with expectation-maximization multi-instance learning,” in _European conference on computer vision_. Springer, 2020, pp. 729–745.
  * [38] L. Itti and C. Koch, “Computational modelling of visual attention,” _Nature reviews neuroscience_ , vol. 2, no. 3, pp. 194–203, 2001.
  * [39] H. Larochelle and G. E. Hinton, “Learning to combine foveal glimpses with a third-order boltzmann machine,” in _Advances in neural information processing systems_ , 2010, pp. 1243–1251.
  * [40] S. Woo, J. Park, J.-Y. Lee, and I. So Kweon, “Cbam: Convolutional block attention module,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 3–19.
  * [41] J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu, “Dual attention network for scene segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3146–3154.
  * [42] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141.
  * [43] Z. Huang, W. Ke, and D. Huang, “Improving object detection with inverted attention,” in _2020 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2020, pp. 1294–1302.
  * [44] W. Ke, T. Zhang, Z. Huang, Q. Ye, J. Liu, and D. Huang, “Multiple anchor learning for visual object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 10 206–10 215.
  * [45] H. Hu, J. Gu, Z. Zhang, J. Dai, and Y. Wei, “Relation networks for object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 3588–3597.
  * [46] L. Rabiner and B. Juang, “An introduction to hidden markov models,” _ieee assp magazine_ , vol. 3, no. 1, pp. 4–16, 1986.
  * [47] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” _International journal of computer vision_ , vol. 88, no. 2, pp. 303–338, 2010.
  * [48] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [49] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [50] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein _et al._ , “Imagenet large scale visual recognition challenge,” _International journal of computer vision_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [51] X. Zhang, J. Feng, H. Xiong, and Q. Tian, “Zigzag learning for weakly supervised object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4262–4270.
  * [52] G. Cheng, J. Yang, D. Gao, L. Guo, and J. Han, “High-quality proposals for weakly supervised object detection,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 5794–5804, 2020.

|  Shiwei Zhang (Student Member, IEEE)received the B.S. degree in Beijing
Institute of Technology, Beijing, China, in 2018. And received the M.S. degree
from Columbia University, New York, USA in the Department of Electrical
Engineering. Now, he is pursuing his Ph.D degree in Xi’an Jiaotong University,
Xi’an, China. His current research interests include image classification and
object detection.
---|---
|  Wei Ke (Member, IEEE)received the Ph.D. degree in electrical engineering
from University of Chinese Academy of Sciences, Beijing, China, in 2018. He is
currently an associate professor at Xi’an Jiaotong University. He was a
PostDoc Researcher with the Robotics Institute, Carnegie Mellon University
until 2020. In 2016, he visited the Center for Machine Vision and Signal
Analysis at University of Oulu as a joint Ph.D. student, supported by China
Scholarship Council (CSC). His research interests include computer vision and
deep learning. He has published about 20 papers in refereed conferences and
journals including IEEE CVPR and ECCV. He is the winner of the President Award
of Chinese Academy of Sciences in 2017.
---|---
|  Lin Yang received the B.S. degree in the Department of Software Engineering
from Inner Mongolia University, Hohhot, China, in 2020. And now he is pursuing
the M.S. degree from Xi’an Jiaotong University, Xi’an, China, in software
engineering with the School of Software Engineering. His current research
interests include image classification and object detection.
---|---
|  Qixiang Ye (M’10-SM’15)received the B.S. and M.S. degrees in mechanical and
electrical engineering from Harbin Institute of Technology, China, in 1999 and
2001, respectively, and the Ph.D. degree from the Institute of Computing
Technology, Chinese Academy of Sciences in 2006. He has been a professor with
the University of Chinese Academy of Sciences since 2009, and was a visiting
assistant professor with the Institute of Advanced Computer Studies (UMIACS),
University of Maryland, College Park until 2013. His research interests
include image processing, visual object detection and machine learning. He
pioneered the Kernel SVM based pyrolysis output prediction software which was
put into practical application by SINOPEC in 2012. He developed two kinds of
piecewise linear SVM methods which were successfully applied into visual
object detection. He has published more than 100 papers in refereed
conferences and journals including IEEE CVPR, ICCV, and PAMI, and received the
Sony Outstanding Paper Award.
---|---
|  Xiaopeng Hong (Member, IEEE)received the Ph.D. degree in computer
application technology from the Harbin Institute of Technology, Harbin, China,
in 2010. He is a Distinguished Research Fellow with Xi’an Jiaotong University,
Xi’an, China. He was a Docent with the Center for Machine Vision and Signal
Analysis, University of Oulu, Finland, where he was a Senior Researcher from
2011 to 2019. He has been a PI of several projects, such as the National Key
Research and Development Program Projects, China, and the Infotech Oulu
Postdoctoral funding project. He has (co-)authored over 50 articles in top-
tier journals and conferences, such as the IEEE TRANSACTIONS ON PATTERN ANALYS
IS AND MACHINE INTELLIGENCE (T-PAMI), the IEEE TRANSACTIONS ON IMAGE
PROCESSING (T-IP), Computer Vision and Pattern Recognition (CVPR),
International Conference on Computer Vision (ICCV), and Association for the
Advance of Artificial Intelligence (AAAI). His research about micro-expression
analysis has been reported by international media, including MIT Technology
Review and Daily Mail. His current research interests include visual
surveillance, continual learning, and micro-expression analysis. Hong has been
the Area Chair of the Association for Computing Machinery (ACM) Multimedia
2020 (MM20) and served as a reviewer for a few top-tier journals and
conferences. He has been a (co-)organizer of five international workshops. He
also served as a guest editor of special issues in several journals.
---|---
|  Yihong Gong (Fellow, IEEE)received the B.S., M.S., and Ph.D. degrees in
electrical and electronic engineering from The University of Tokyo, Tokyo,
Japan, in 1987, 1989, and 1992, respectively. He was an Assistant Professor
with the School of Electrical and Electronic Engineering, Nanyang
Technological University, Singapore, for four years. From 1996 to 1998, he was
a Project Scientist with the Robotics Institute, Carnegie Mellon University,
Pittsburgh, PA, USA. In 1999, he joined NEC Laboratories America, Princeton,
NJ, USA, and established the Media Analytics Group for the laboratories, where
he became the Site Manager to lead the entire branch at Cupertino, CA, USA. In
2012, he joined Xi’an Jiaotong University, Xi’an, China, as a Distinguished
Professor, the Vice Director of the National Engineering Laboratory for Visual
Information Processing, and the Chief Scientist of the China National Key
Basic Research Project (973 Project). His current research interests include
pattern recognition, machine learning, and multimedia content analysis.
---|---
|  Tong Zhang received the B.S. and M.S degree from Beihang University,
Beijing, China and New York University, New York, United States in 2011 and
2014 respectively, and he received the Ph.D. degree from the Australian
National University, Canberra, Australia in 2020. He is working as a
postdoctoral researcher at Image and Visual Representation Lab (IVRL), EPFL.
He was awarded the ACCV 2016 Best Student Paper Honorable Mention and the CVPR
2020 Paper Award Nominee. His research interests include unsupervised and
self-supervised learning, subspace and general clustering, graph convolution
network, and saliency detection.
---|---