# Object Localization under Single Coarse Point Supervision

Xuehui Yu1, Pengfei Chen1∗, Di Wu1, Najmul Hassan2, Guorong Li1,
Junchi Yan3, Humphrey Shi2,4, Qixiang Ye1, Zhenjun Han1
1University of Chinese Academy of Sciences, 2U of Oregon, 3Shanghai Jiao Tong
University, 4Picsart AI Research (PAIR) Equal contribution. Corresponding
authors<EMAIL_ADDRESS>
###### Abstract

Point-based object localization (POL), which pursues high-performance object
sensing under low-cost data annotation, has attracted increased attention.
However, the point annotation mode inevitably introduces semantic variance for
the inconsistency of annotated points. Existing POL methods heavily reply on
accurate key-point annotations which are difficult to define. In this study,
we propose a POL method using coarse point annotations, relaxing the
supervision signals from accurate key points to freely spotted points. To this
end, we propose a coarse point refinement (CPR) approach, which to our best
knowledge is the first attempt to alleviate semantic variance from the
perspective of algorithm. CPR constructs point bags, selects semantic-
correlated points, and produces semantic center points through multiple
instance learning (MIL). In this way, CPR defines a weakly supervised
evolution procedure, which ensures training high-performance object localizer
under coarse point supervision. Experimental results on COCO, DOTA and our
proposed SeaPerson dataset validate the effectiveness of the CPR approach. The
dataset and code will be available at https://github.com/ucas-
vg/PointTinyBenchmark/.

##  1 Introduction

Humans can recognize and easily achieve a sense of the objects present in
their eye-sight. In computer-vision, this is usually framed as drawing
bounding boxes around objects [21, 25, 40, 41] or dense annotations of the
entire scene [13, 16]. However, one inevitable circumstance for training such
models is that they require high-quality densely annotated data which is
expensive and difficult to obtain. In some applications [27], just the
object’s location is necessary while costly annotation (e.g. bounding box) is
redundant or even undesirable (e.g. a robotic arm aims at a single point to
pick up an object [27]).

---
Figure 1: Examples of coarse point annotation and the problem of semantic
variance. (Best viewed in color.)

Hence point-based object localization (POL) is studied. Due to the simple and
time-efficient annotation, point-based object localization has attracted
increasing attention in recent years [26, 29]. POL based methods require
point-level object annotations and can predict the object’s location as a 2D
coordinate in the original image.

However, while annotating an object as a point, there can be multiple
candidate points. One problem that arises with optional candidate points is
that multiple regions of varying semantic information are labeled positive for
the same class. Conversely, identical regions with similar semantic
information are labeled differently. Take bird category as an example, during
annotation, we label the bird’s different body parts (e.g. neck and tail etc.)
as positive based on the visible regions in the image. Based on the
annotation, for different images in the dataset, we have labeled same body
part (e.g. neck) of the bird as both positive and negative (see Fig. 1).
Therefore, during training, the model has to consider the neck region as
positive for one image and negative in another (the image where tail is
annotated). This phenomenon introduces ambiguity and confuses the model which
results in poor performance.

Previous works [29, 37] addressed this issue by setting strict annotation
rules by annotating only the pre-defined key-point areas of the object. As a
result, they suffer from the following challenges: i) the key points are not
easy to define, especially for some broadly defined categories where they do
not have a specific shape (Fig. 2 (a)); ii) the key point may not exist in the
image due to the different poses of objects and different camera views (Fig. 2
(b)); iii) when objects have large scale variance, it is difficult to decide
the appropriate granularity of the key points ( Fig. 2 (c)). For a person, if
the head is a key point [29] (coarse-grained) then there remains a large
semantic variance for the large-scale instance (whether to annotate the eye or
nose). If the eye is labeled as a key point [37] (fine-grained) then the
position of eyes for a small-scale instance cannot be identified. Thus, the
complicated annotation rules are required to solve the semantic variance
problem from annotation perspective, which considerably increases the
annotation difficulty and human burden. Therefore, the challenges mentioned
above restrict previous POL methods from exploring multi-class and multi-scale
datasets (e.g. COCO or DOTA).

(a) Chairs with different shapes.

(b) Persons with different poses.

(c) Persons of different sizes.

Figure 2: The difficulties of key-point based annotation. (a) Key points are
hard to define due to the large in-class variance of shape. (b) Key point
(e.g. head) does not exist due to multiple poses and views. (c) Key point’s
granularity (eye, forehead, head or body) is hard to determine due to multiple
scales.

In this paper, we formulate the coarse point-based localization (CPL) paradigm
for training a localizer of a general POL, as shown in Fig. 3. We firstly
adopt a coarse point annotation stragegy, which allows to annotate any point
on an object. Then the coarse point refinement (CPR) algorithm is proposed to
refine the initialized annotated coarse point to the semantic center in the
training set. Finally, the refined points instead of the annotated points are
used as supervision to train a localizer. The proposed CPR is the first
attempt to alleviate semantic variance from the perspective of algorithm
rather than annotation. Specifically, CPR finds the semantic points around the
annotated point through multiple instance learning (MIL) [9], then weighted
averages the semantic points to obtain the semantic center, which has a
smaller semantic variance and a higher tolerance for prediction errors. The
contributions are:

1) We dive into point-based object localization (POL) task, and formulate the
coarse point based localization (CPL) paradigm for general object
localization, extending the previous works to a multi-class/multi-scale POL
task;

2) The coarse point refinement (CPR) algorithm is proposed to alleviate the
semantic variance from the perspective of algorithm rather than rigid
annotation rules;

3) The experimental results show the CPR is effective for CPL, which obtains a
comparable performance with the center point (approximate key point) based
object localization, and improves the performance over 10 points compared with
the baseline;

4) A new dataset with more than 600,000 annotations, named SeaPerson, is
introduced in this paper. This dataset can be used for tiny person detection
and localization.

##  2 Related Work

In this section, we review the relevant point-based vision tasks and the
vision tasks with multiple instance learning.

###  2.1 Vision Tasks under Point Supervision

Pose Estimation. Human or animal pose estimation aims to locate the position
of joint points of persons or animals accurately [31, 48, 50]. There are
several benchmarks built for the task, e.g. COCO [22] and the Human3.6M [17]
datasets are the most well-known ones for 2D and 3D pose estimation and AP-10k
[45] for animal pose estimation. In these datasets, annotations are a set of
accurate key points, and the predicted results are human or aninimal poses
rather than the location of person or animal instances.

Crowd Counting. In this task, accurate head annotation is utilized as point
supervision [52, 37, 29]. The crowd density map [19, 18, 15], generated by
head annotation, is chosen as the optimization objective of the network.
Furthermore, crowd counting focuses on the number of people rather than each
person’s position. It depends on precise key points such as the human head,
while the coarse point object localization task only requires the coarse
position annotation on the human body.

Object Localization. Unlike object detection [21, 25, 12], especially for
rotation detection [42, 43], requiring the exact bounding box information,
object localization applications [26] are often agnostic to object’s scale.
The works [26, 29] train a localizer with points instead of bounding boxes.
These tasks are summarized as POL in our paper. However, they heavily rely on
key-point annotations to reduce the semantic variance.

Different from the above mentioned tasks, our CPL relies upon a coarse point
instead of keypoints and deals the semantic variance problem with a novel
approach.

###  2.2 Vision Tasks with Multiple Instance Learning

The paradigm of MIL [9] is that a bag is positively labeled if it contains at
least one positive instance; otherwise, it is labeled as a negative bag.
Inspired by weakly supervised object detection task, the proposed CPR method
follows the MIL paradigm. With the object category and the coarse point
annotation, we consider sampled points around each annotated point as a bag
and utilize MIL for training.

Image-level Tasks. An image is divided into patches, where patches are seemed
as instances and the entire image as a bag. Content-based image retrieval [47,
49] is a conventional MIL task, which just classifys images by their content.
If the image contains at least one object of a class, the whole bag can be
seen as a positive sample for that class. Otherwise, the bag will be regarded
as a negative sample.

Video-level Tasks. Firstly, the video is divided into segments, which will be
classified separately and then the whole video is seemed as a bag. Following
the above pre-processing, MIL is used to identify specific events in videos
[10, 24, 30]. Additionally, some researchers have applied MIL to video object
tracking [1, 2]. [2] also achieves a robust tracker by constructing instance-
level bag from box, but different from our work, it does not handle the
negative samples to suppress background for MIL.

Object-level Tasks. MIL is widely used in weakly supervised object
localization and detection (WSOL [8, 38] and WSOD [4, 36, 34, 32, 7, 35]),
where only the image-level annotation is utilized. Firstly, Select Search [33]
or Edge Box [53] methods are used to produce proposal boxes, which are then
used as a bag and each of them as an instance. Finally, they classified
positive and negative samples by judging whether the image contains at least
one object of a specific class. WSOL/WSOD, only with image-level annotation,
focus on local regions and can not distinguish instances due to the lack of
object-level annotation.

Annotation of CPL is a coarse point position and the category of each object.
CPR views sampled points around the annotated point as a bag and trains
object-level MIL to find a better and stable semantic center.

---
Figure 3: Pipeline of CPL in three steps: 1) Annotating objects as coarse
points $A$. 2) Refining annotated points to semantic centers $\hat{A}$. 3)
Training a localizer (e.g. P2PNet) with $\hat{A}$ as supervision.
---
Figure 4: The framework of CPR. With $F$, there are three steps in CPR: 1)
Points bag (e.g. $B_{1},B_{2}$) and negative samples (e.g. $Neg_{k}$) are
obtained by point sampling according to annotated points $A$ (green), and then
feature vectors of these points are extracted on $F$. 2) CPRNet are trained
with the feature vectors based on MIL loss, annotation loss and negative loss.
3) Semantic points (red points on the birds) are selected by classification
scores of points in bag (e.g. $B_{1},B_{2}$) predicted by the trained CPRNet.
Finally, the refined points (yellow) $\hat{A}$ are obtained by weighted
averaging the semantic points. (Best viewed in color.)

##  3 Coarse Point Refinement

As shown in Fig. 3, CPR can be regarded as a pre-processing that transforms
the annotations on the training set to a more conductive form for the
subsequent tasks. The main purpose of CPR is to find a semantic point, which
has a smaller semantic variance and a higher tolerance for prediction errors,
to replace the initial annotated point.

In Fig 4 and Algorithm 1, there are three key steps in CPR: i) Point Sampling:
points in the neighborhood of the annotated point are sampled; ii) CPRNet
Training: based on the sampled points, a network is trained to classify
whether the points are in the same category with the annotated point or not;
iii) Point Refinement: based on the scores obtained by CPRNet and the
constraints (details in Sec. 3.3), the points, having similar semantic
information with the annotated point, are chosen as the semantic points and
then weighted with their scores to obtain the semantic center as the final
refined point.

Algorithm 1 Coarse Point Refinement

Input: Training set $D_{train}$, CPRNet $E$.
Output: Refined points $\hat{A}^{D_{train}}$.
Note: $A$ and $C$ are 2D coordinates and category label of annotated points in
image $I$ respectively.

1: $L_{CPR}^{D_{train}}\leftarrow 0$;

2: for $(I,A,C)\in D_{train}$ do

3: Extract feature map $\mathbf{F}$ of $I$ with $E$;

4: // Step1: point sampling

5: $B_{j}\leftarrow bag\\_sampling(a_{j})$ for each $a_{j}\in A$, Eq. 1;

6: $Neg_{k}\leftarrow neg\\_sampling(k)$ for each category $k\in\\{1,$
$2,...K\\}$, Eq. 2;

7: // Step2: CPRNet training

8: Calculate $L_{MIL}$ with $B_{j}$ and $\mathbf{F}$, Eq. 7;

9: Calculate $L_{ann}$ with $A$ and $\mathbf{F}$, Eq. 8;

10: Calculate $L_{neg}$ with $Neg_{k}$ and $\mathbf{F}$, Eq. 10;

11: Sum $L_{MIL}$, $L_{ann}$ and $L_{neg}$ to obtain the CPR loss $L_{CPR}$,
Eq. 3;

12: $L_{CPR}^{D_{train}}\leftarrow L_{CPR}^{D_{train}}+L_{CPR}$;

13: end for

14: Train $E$ by minimizing $L_{CPR}^{D_{train}}$ to obtain $\hat{E}$.

15: // Step3: point refinement

16: $\hat{A}^{D_{train}}\leftarrow\\{\\}$;

17: for $(I,A,C)\in D_{train}$ do

18: $\hat{A}\leftarrow Point\\_Refinement(\hat{E},I,A,C)$, Algorithm 2;

19: $\hat{A}^{D_{train}}\leftarrow\hat{A}^{D_{train}}\cup\\{\hat{A}\\}$;

20: end for

###  3.1 Point Sampling

In this paper, $K$ denotes the number of categories, $a_{j}\in R^{2}$ and
$c_{j}\in\\{0,1\\}^{K}$ denote the annotated point’s 2D coordinate and the
category label of $j$-th instance. $p=(p_{x},p_{y})$ denotes a point on a
feature map.

Point Bag Construction. In Fig. 4, to sample points uniformly in the
neighborhood of $a_{j}$, we define $R$ circles with $a_{j}$ as the center,
where the radius of the $r$-th ($1\leq r\leq R$, $r\in N^{+}$) circle is set
as $r$. Then we sample $r*u_{0}$ ($u_{o}$=8 by default) points with equal
intervals around the circumference of the $r$-th circle, and obtain
$Circle(a_{j},r)$. All sampled points of the $R$ circles are defined as
points’ bag of $a_{j}$, denoted as $B_{j}$ in Eq. 1. The points outside the
feature map are excluded.

$\displaystyle Circle$ $\displaystyle(p,r)=\bigg{\\{}\bigg{(}p_{x}+r\cdot
cos\left(2\pi\cdot\frac{i}{u_{0}\cdot r}\right),p_{y}+$ (1) $\displaystyle
r\cdot sin\left(2\pi\cdot\frac{i}{u_{0}\cdot r}\right)\bigg{)}\ |\ 0\leq
i<r\cdot u_{0},i\in N^{+}\bigg{\\}};$ $\displaystyle
B_{j}=\mathop{\cup}\limits_{1\leq r\leq R}\ Circle(a_{j},r).$

$B_{j}$ is used for calculating the MIL loss for CPRNet training and obtaining
the semantic points for point refinement.

Negative Point Sampling. All integer points on the feature map, outside the
circles with radius $R$ of all annotated points of a given category, will be
selected as negative samples. The negative samples of category $k$ can be
defined as:

$\displaystyle Neg_{k}$ $\displaystyle=\\{(p_{x},p_{y})|p_{x}\leq w,p_{y}\leq
h,p_{x}\in N^{+},p_{y}\in N^{+}$ (2) $\displaystyle\forall(a_{j},c_{j})\ s.t.\
c_{jk}=1,||p-a_{j}||>R\\},$

where $||p-a_{j}||$ is the Euclidean distance between $p$ and $a_{j}$. $w$ and
$h$ are the width and height of a given feature map.

###  3.2 CPRNet Training

This section gives the details of the objective function of training CPRNet
based on the sampled points bag $B_{j}$ ($j\in\\{1,2,..M\\}$) and the negative
points $Neg_{k}$ ($k\in\\{1,2,..K\\}$), where $M$ and $K$ are the amount of
instances and categories. $U$ is defined as the amount of points in $B_{j}$.

CPRNet. CPRNet adopts FPN [20] with ResNet [14] as the backbone. Only P2 or P3
is used due to the lack of scale information in point annotation. After four
3$\times$3 conv layers followed by the ReLU [11] activation, the final feature
map $\mathbf{F}\in\mathbb{R}^{h\times w\times d}$ is obtained, where $h\times
w$ is the corresponding spatial size and $d$ is the dimension of channel. For
a given point $p=(p_{x},p_{y})$, $\mathbf{F}_{p}\in\mathbb{R}^{d}$ denotes the
feature vector of $p$ on $\mathbf{F}$. If $p$ is not an integer point, the
bilinear interpolation is used to obtain $\mathbf{F}_{p}$.

CPR Loss. Object-level MIL loss is introduced to endrow CPRNet the ability of
finding semantic points around each annotated point. Then to overcome the
over-fitting problem of MIL when the data is insufficient, we further
introduce the instance-level prior as supervision by designing annotation and
negative loss. The objective function of CPRNet is a weighted summation of the
three losses:

$\displaystyle L_{CPR}=$ $\displaystyle
L_{MIL}+\alpha_{ann}L_{ann}+\alpha_{neg}L_{neg},$ (3)

where $\alpha_{ann}=0.5$ and $\alpha_{neg}=3$ (by default in this paper). And
$L_{MIL}$, $L_{ann}$ and $L_{neg}$ are based on the focal loss [21]:

$\displaystyle FL(S_{p},c_{j})=\sum\limits_{k=1}^{K}c_{j,k}$
$\displaystyle(1-S_{p,k})^{\gamma}\log(S_{p,k})+$ (4)
$\displaystyle(1-c_{j,k})S_{p,k}^{\gamma}\log(1-S_{p,k}),$

where $\gamma$ is set as 2 in Eq. 4 following the standard focal loss, and
$S_{p}\in\mathbb{R}^{K}$ and $c_{j}\in\\{0,1\\}^{K}$ are the predicted scores
on all categories and the category label, respectively.

Object-level MIL Loss. To find the semantic points during refinement, we refer
to WSOD [4] and design a MIL loss to enable the CPRNet justify whether the
points in $B_{j}$ are in the same category with $a_{j}$. Based on $B_{j}$, the
feature vectors $\\{\mathbf{F}_{p}|p\in B_{j}\\}$ are extracted. As Eq. 5
shows, for each $p\in B_{j}$, a classification branch $fc^{cls}$ is applied to
obtain the logits $[\mathbf{O}^{cls}_{B_{j}}]_{p}$, which is then utilized as
an input of an activation function $\sigma_{1}$ to obtain
$[\mathbf{S}^{cls}_{B_{j}}]_{p}$. Besides, an instance selection branch
$fc^{ins}$ is applied to $\mathbf{F}_{p}$ to obtain $[O^{ins}_{B_{j}}]_{p}$,
which is then utilized as an input of an activation function $\sigma_{2}$ to
obtain the selection score $[S^{ins}_{B_{j}}]_{p}$. The score
$[S^{over}_{B_{j}}]_{p}$ is obtained by taking the element-wise product of
$[S^{ins}_{B_{j}}]_{p}$ and $[S^{cls}_{B_{j}}]_{p}$.

$\displaystyle{[O^{cls}_{B_{j}}]}_{p}=fc^{cls}(F_{p})\in\mathbb{R}^{K},\quad[O^{ins}_{B_{j}}]_{p}=fc^{ins}(F_{p})\in\mathbb{R}^{K};$
(5)
$\displaystyle[S^{cls}_{B_{j}}]_{p}=[\sigma_{1}(O^{cls}_{B_{j}})]_{p}=1/(1+e^{-[O^{cls}_{B_{j}}]_{p}})\in\mathbb{R}^{K};$
$\displaystyle[S^{ins}_{B_{j}}]_{p}=[\sigma_{2}(O^{ins}_{B_{j}})]_{p}=e^{O^{ins}_{p}}/\sum_{p^{\prime}\in
B_{j}}[e^{O^{ins}_{B_{j}}}]_{p^{\prime}}\in\mathbb{R}^{K};$
$\displaystyle[S^{over}_{B_{j}}]_{p}=[S^{ins}_{B_{j}}]_{p}\cdot[S^{cls}_{B_{j}}]_{p}\in\mathbb{R}^{K},$

where $\sigma_{2}$ is a $softmax$ function. Different from MIL in WSOD, the
$sigmoid$ activation function is applied for $\sigma_{1}$, due to its
suitability for binary task compared with the $softmax$ function. Furthermore,
the $sigmoid$ activation function allows to perform multi-label classification
(for the overlapping area of multiply objects’ neighborhood) for points and is
more compatible with focal loss.

The bag-level score $S_{B_{j}}$ is obtained by the summation of all points’
scores in $B_{j}$ by Eq. 6. $S_{B_{j}}$ can be seen as the weighted summation
of the classification score $[S^{cls}_{B_{j}}]_{p}$ of $p$ in $B_{j}$ by the
corresponding selection score $[S^{ins}_{B_{j}}]_{p}$.

$\displaystyle S_{B_{j}}$ $\displaystyle=\sum_{p\in
B_{j}}[S^{over}_{B_{j}}]_{p}\in\mathbb{R}^{K}.$ (6)

The MIL loss is finally given by the focal loss on the predicted bag-level
scores $S_{B_{j}}$ and the category label $c_{j}$ of $a_{j}$:

$\displaystyle L_{MIL}$
$\displaystyle=\frac{1}{M}\sum\limits_{j=1}^{M}FL(S_{B_{j}},c_{j}).$ (7)

Annotation Loss. Due to the lack of explicit positive samples for supervision
in MIL, the network sometimes focuses on the points outside the instance
region and mistakenly regards them as the foreground. Therefore, we introduce
the annotation loss $L_{ann}$, that gives the network accurate positive
samples for supervision via annotated points, to guide MIL training. $L_{ann}$
can guarantee a high score of the annotated point and mitigate mis-
classification to some extent. Firstly, the classification score of
$S_{a_{j}}(j\in{1,2,...M})$ of $a_{j}$ is calcluated as:

$\displaystyle S_{{a_{j}}}$
$\displaystyle=\sigma_{1}(fc^{cls}(F_{a_{j}}))\in\mathbb{R}^{K}.$ (8)

$L_{ann}$ is calculated with focal loss as:

$\displaystyle L_{ann}$
$\displaystyle=\frac{1}{M}\sum\limits_{j=1}^{M}FL(S_{a_{j}},c_{j}).$ (9)

Negative Loss. The conventional MIL adopts binary log loss, and it views the
proposals belonging to other categories as negative samples. For lacking of
explicit supervision from samples in background, the negative samples are not
well suppressed during MIL training. Therefore, based on $Neg_{k}$, the
negative loss $L_{neg}$, the negative part of focal loss, is calculated as
follows, where we set $\gamma=2$.

$\small\begin{split}S_{p}&=\sigma_{1}(fc_{cls}(F_{p}))\in\mathbb{R}^{K};\\\
L_{neg}&=\frac{1}{M}\sum\limits_{k=1}^{K}\sum\limits_{p\in
Neg_{k}}c_{j,k}S_{p,k}^{\gamma}\log(1-S_{p,k}).\end{split}$ (10)

Figure 5: Visualization of CPR. Semantic points (red) around the annotated
point (green) are weighted averaged to obtain the semantic center (yellow) as
final refined point (see Sec. 3.3). The two images on the left show the
results of birds with multiple poses; the three images on the right show
results of objects in multiple categories. Images are cut from the raw ones
(in COCO/DOTA) for better view.

###  3.3 Point Refinement

As described in Algorithm 2, the trained CPRNet $\hat{E}$ is used to refine
the annotated point. Based on $B_{j}$, $[S^{cls}_{B_{j}}]_{p}$ predicted by
$\hat{E}$ and the constrains (details given in following), points with the
same category (similar semantic) with the annotated point are selected,
denoted as $B^{+}_{j}$. Then, the semantic center (final refined point), used
to replace the annotated point, is set as the weighted mean of points in
$B^{+}_{j}$.

To obtain $B^{+}_{j}$, three constraints (line in blue in Algorithm 2) are
introduced. Constraint I is to delete points with small classification scores.
We filter out the point $p\in B_{j}$ whose $S_{p,k_{j}}$ is smaller than the
threshold $\delta_{1}$ (set as 0.1 by default) or
$\delta_{2}*S_{a_{j},k_{j}}$, where $\delta_{2}$ is set as 0.5 by default and
$k_{j}$ is category label (not one-hot format) of $j$-th object. Constraint II
is to delete the points that are not classified correctly. Specifically, the
correct classification means the classification score $S_{p,k_{j}}$ of point
$p$ on the given annotated category $k_{j}$ is higher than the scores on other
categories. Constraint III is to delete the points closer to other object in
the same category. Since two adjacent objects of the same category may
interfere with each other. With the three constraints, the remaining points
construct the $B^{+}_{j}$ and are weighted average to obtain the semantic
center point as the final refined point, which is used as the supervision to
train P2PNet [29]. P2PNet is the SOTA baseline for the POL task and will be
specifically described in experiment section.

With the point sampling, CPRNet training and Point refinement mentioned above,
the CPR can effectively mitigate semantic variance, as shown in Fig. 5.

Algorithm 2 Point Refinement

Input: Trained CPRNet $\hat{E}$, input image $I$, annotated points $A$,
category label of annotated points $C$.
Output: Refined points $\hat{A}$.
Note: $\delta_{1},\delta_{2}$ are thresholds. $K$ is the number of categories.
$S_{p,k}$ is the predicted score on $k$-th category of point $p$.
$k_{j}\in\\{1,2,...K\\}$ is the category label (not one-hot format) of the
j-th object.

1: $\hat{A}\leftarrow\\{\\}$;

2: $\mathbf{F}\leftarrow extract\\_feature(I;\hat{E})$ according to Sec. 3.2;

3: for $a_{j}\in A,c_{j}\in C$ do

4: find $k_{j}\in\\{1,2,..K\\}$ s.t. $c_{jk_{j}}=1$;

5: $B^{+}_{j}\leftarrow\\{a_{j}\\}$;

6:
$S_{a_{j}}\leftarrow\sigma_{1}(fc^{cls}(\mathbf{F}_{a_{j}};\hat{E}))\in\mathbb{R}^{K}$;

7: $B_{j}\leftarrow bag\\_sampling(a_{j})$ according to Eq. 1;

8: for $p\in B_{j}$ do

9:
$S_{p}\leftarrow\sigma_{1}(fc^{cls}(\mathbf{F}_{p};\hat{E}))\in\mathbb{R}^{K}$;

10: $s_{p}\leftarrow S_{p,k_{j}}$;

11: if $s_{p}>\delta_{1}$ and $s_{p}>\delta_{2}*S_{a_{j},k_{j}}$ and
$k_{j}=argmax_{1\leq k\leq K}\ S_{p,k}$ and $a_{j}=argmin_{a\in A}\ ||p-a||$
then

12: $B^{+}_{j}\leftarrow B^{+}_{j}\cup\\{p\\}$;

13: end if

14: end for

15: $\hat{a_{j}}\leftarrow(\sum_{p\in B^{+}_{j}}S_{p}*p)/(\sum_{p\in
B^{+}_{j}}S_{p})$;

16: $\hat{A}\leftarrow\hat{A}\cup\\{\hat{a_{j}}\\}$;

17: end for

##  4 Experiment

###  4.1 Experimental Settings

Datasets. For experimental comparisons, three public available datasets are
used for point supervised localization task: COCO [22], DOTA-v1.0 [39] and
SeaPerson. COCO is MSCOCO 2017, and it has 118k training and 5k validation
images with 80 common categories. Since the ground-truth on the test set are
not released, we train our model on the training set and evaluate it on the
validation set. DOTA(v1.0) provides 2,806 images with 15 object categories. We
utilize training set for the training and validation set for evaluation.
SeaPerson111SeaPerson is a low-resolution tiny person dataset and disclose
little personal privacy from the appearance. is a dataset for tiny person
detection collected through a UAV camera at the seaside. The dataset contains
12,032 images and 619,627 annotated persons with low resolution. The images in
the SeaPerson are randomly selected as training, validation, and test sets
with the proportion of 10:1:10. The details are given in the supplemental
materials.

Coarse Point Annotation In practical scenarios, the coarse point can be
obtained through annotating any single point on an object. However, since
datasets in the experiment are already annotated with masks or bounding boxes,
according to the law of large numbers, it is reasonable that the manually
annotated points follow Gaussian distribution. Furthermore, since the
annotated points must be inside the bounding box or mask of the object, then
an improved Gaussian distribution, named as Rectified Gaussian (RG)
Distribution, is utilized for annotation. $RG(p;0,\frac{1}{4})$ is chosen to
generate the point annotations for the experiments.

$\displaystyle\phi(p;\mu,\sigma)=Gauss(p;\mu,\sigma)\cdot Mask(p);$ (11)
$\displaystyle
RG(p;\mu,\sigma)=\frac{\phi(p;\mu,\sigma)}{\int_{p}\phi(p;\mu,\sigma)}.$

where $\mu$ and $\sigma$ are the mean and standard deviation of Gaussian
distribution, respectively. $Mask(p)\in\\{0,1\\}$ denotes whether point $p$
falls inside the mask of an object. If it is generated from the bounding box
annotation, then the box is treated as a mask.

Evaluation. Similar to WSOD, a point-box distance, calculated between the
point and box, is used for evaluation. Specifically, the distance $d$ between
point $a=(x,y)$ and bounding box $b=(x^{c},y^{c},w,h)$ is defined as:

$\small
d(a,b)={\sqrt{\left(\frac{x-x^{c}}{w}\right)^{2}+\left(\frac{y-y^{c}}{h}\right)^{2}}}.$
(12)

where $(x^{c},y^{c})$, $w$, $h$ are the center point, width, and height of the
bounding box, respectively. The distance $d$ is used as the matching criterion
for POL performance. A point and the object’s bounding box are matched if the
distance $d$ is smaller than a predefined threshold $\tau$. ($e.g.$ $\tau=1.0$
means that as long as the point falls within an matched ground-truth box, the
point successfully matches the ground-truth box.) If a bounding box has
multiple matched points, the point with the highest score is chosen. While a
point has multiple matched objects, the object with the smallest point-box
distance is selected. A true positive (TP) is counted if a point matches an
object. Otherwise, a false positive (FP) is counted. Neither TP nor FP will be
counted if a point matches an object that is annotated as ignore, which
follows the evaluation criteria of pedestrian detection [51] and TinyPerson
benchmark [46]. We adopt mean Average Precision with $\tau=1.0$
($mAP^{all}_{1.0}$) as the main metric for experimental comparisons. We do not
consider a small $\tau$ because it makes the task more like center
localization instead of object localization. Here we also report the results
of $\tau=0.5$ and $\tau=2.0$ in Table 3, which can be informative.

Implementation Details for CPRNet. Our codes are based on MMDetection [6].
Same to the default setting of the object detection on COCO, the stochastic
gradient descent (SGD [5]) algorithm is used to optimize in 1x training
schedule. The learning rate is set to 0.0025 and decays by 0.1 at the 8-th and
11-th epoch, respectively.

###  4.2 Experimental Comparisons

The POL task with coarse point annotation is divided into two key parts:
refining the coarse point annotation and training the point localizer with
refined points.

Detector with Pseudo Box. For training a point localizer, an intuitive idea is
to convert the point-to-point (POL) to a box-to-box (object detection)
problem. Firstly, a fixed-size pseudo-box is generated with each annotated
point as the center. Next, the pseudo-box is used to train a detector.
Finally, during inference, the center points of the boxes predicted by the
trained detector are used as the final output. Following [26], we conduct the
pseudo box for localization and give the performance in row 1 of Table 1. The
difference to [26] is that the RepPoint [44] is utilized rather than Faster
RCNN [25], due to its efficiency.

Multi-Class P2PNet. We adopt P2PNet222In our experiments, we re-implement
P2PNet and further endow it the new ability of handling multi-class
prediction, which to our best efforts, aligns the results with those reported
in the raw paper [29]., to train with point annotation and predict point for
each object during inference, as a stronger baseline for POL task. Improvement
can be made, especially when there are multiple categories: i) The backbone of
P2PNet in this paper is Resnet-50 rather than VGG16 [28]. ii) Instead of using
the Cross-Entropy loss, we adopt the focal loss when optimizing classification
to better deal with the problem of imbalance; iii) The Smooth-$\ell_{1}$ loss
instead of $\ell_{2}$ loss is used for regression. iv) In label assignment,
different from a one-to-one matching in the default P2PNet, we assign top-k
positive samples for each ground-truth and regard remaining samples as
background. And then the NMS [23] post-processing for points is performed to
obtain the final point results. The performances of P2PNet, given in row 2 in
Table 1, improves a lot compared with the pseudo box (row 1 in Table 1).
P2PNet is a stronger baseline for POL task.

---
Figure 6: The top-left and top-right figures are the relative position
distribution (Eq. 14) heatmap of annotated points and final refined points in
bounding boxes. The top-middle figure is RSV curve of refined points (Eq.
LABEL:Eq:a_semantic_variance)) in each epoch during CPRNet training. The
bottom four figures give the annotated points (green) and refined points
(yellow) in four epochs, showing the refined points gradually converge to the
semantic center points.

refinement | localizer | COCO | DOTA | SeaPerson
---|---|---|---|---
- | RepPoint∗ | 37.42 | 47.22 | 47.72
- | P2PNet | 38.48 | 48.34 | 76.52
Self-Refinement (ours) | P2PNet | 50.86 | 60.39 | 84.96
CPR (ours) | P2PNet | 55.46 | 63.81 | 85.86

Table 1: The experimental comparisons ($mAP^{all}_{1.0}$) of localizers in
three datasets: COCO, DOTA and SeaPerson. RepPoint∗ means RepPoint with pseudo
box (details in Sec. 4.2).

pos | MIL | ann | neg | $mAP^{all}_{1.0}$ | $mAP^{all}_{0.5}$ | $mAP^{all}_{2.0}$
---|---|---|---|---|---|---
| ✓ |  |  | 39.07 | 28.37 | 46.90
| ✓ | ✓ |  | 39.45 | 28.22 | 47.42
| ✓ |  | ✓ | 54.24 | 46.56 | 58.94
|  | ✓ | ✓ | 51.82 | 46.24 | 55.67
✓ |  | ✓ | ✓ | 42.72 | 33.07 | 48.81
| ✓ | ✓ | ✓ | 55.46 | 50.23 | 59.49

Table 2: The effect of training loss in CPRNet: MIL loss, annotation loss,
negative loss. The pos loss is for comparison.

Self-Refinement. For refining the coarse point annotation, inspired by [3], we
propose a self-refinement technique that works as a strategy based on self-
iterative learning. Firstly, the aforementioned pseudo box strategy is adopted
to train a point localizer. Then, the weighted mean of the points predicted by
the localizer works as the new supervision. Finally, the refined points are
obtained. With these refined points as supervision, the performance of P2PNet
as point localizer is given in row 3 in Table 1, where it alleviates the
semantic variance problem.

detector | COCO | DOTA | SeaPerson
---|---|---|---
RetinaNet | 32.61 | 51.53 | 48.50
FasterRCNN | 35.29 | 51.15 | 47.93
RepPoint | 37.42 | 47.26 | 47.72
RetinaNet w. CPR | 51.35 | 63.69 | 77.90
FasterRCNN w. CPR | 53.21 | 63.00 | 77.80
RepPoint w. CPR | 53.97 | 60.37 | 78.94

Table 3: $mAP^{all}_{1.0}$ of more architectures on the three datasets.

CPR. Compared with self-refinement, CPR (shown in row 5 of Table 1) obtains
more performance gain, indicating it is more efficient for dealing with the
semantic variance. To quantify the semantic variance of point annotation, the
relative semantic variance (RSV), which is calculated based on the relative
distance of the point to the center point:

$\displaystyle x^{\prime}=$ $\displaystyle\frac{x-x^{c}}{w};\quad
y^{\prime}=\frac{y-y^{c}}{h};$ (13) $\displaystyle RSV$
$\displaystyle=[Var(x^{\prime})*Var(y^{\prime})]^{\frac{1}{2}}.$

where $(x,y)$ is an annotated point or refined point and $(x^{c},y^{c})$ is
the corresponding center point in the bounding box of an object.
$Var(x^{\prime})$ and $Var(y^{\prime})$ are the variance of $x^{\prime}$ and
$y^{\prime}$ of all objects in the dataset, respectively. Statistically, the
smaller RSV means the $(x,y)$ holds a more stable relative position to its
corresponding $(x^{c},y^{c})$, as shown in Eq. LABEL:Eq:a_semantic_variance.
Considering the $(x^{c},y^{c})$ as a strict key point, the intuition behind
the RSV is that a small RSV for a category is equivalent to a strict
annotation, which can effectively reduce the semantic variance of annotations.
As shown in Fig. 6, the annotated coarse point holds a larger RSV, while the
refined point via CPR obtains a smaller RSV.

To show the relative position distribution of annotated points in the bounding
boxes, we calculate $Prob(x^{\prime},y^{\prime})$ as:

$\displaystyle
Prob(x^{\prime}=p_{x},y^{\prime}=p_{y})=\frac{\sum\limits_{1\leq j\leq
M}\mathbf{\mathbb{I}}\\{x^{\prime}_{j}=p_{x}^{\prime}\ and\
y^{\prime}_{j}=p_{y}^{\prime}\\}}{M}.$ (14)

where $(x^{\prime}_{j},y^{\prime}_{j})$ is relative position of annotated
point or refined point for object $j$ in dataset, $\mathbf{\mathbb{I}}\\{*\\}$
is 1 if $*$ is true, otherwise 0. $Prob(x^{\prime},y^{\prime})$ is shown as
the heatmap in Fig. 6.

feature | $R$ | $mAP^{all}_{1.0}$
---|---|---
| 5 | 53.32
| 8 | 55.46
P3 | 10 | 55.19
| 15 | 55.38
| 20 | 55.04
| 25 | 53.85

(a) Different $R$ with P3

feature | $R$ | $mAP^{all}_{1.0}$
---|---|---
| 5 | 48.64
| 10 | 53.76
P2 | 15 | 54.26
| 20 | 54.64
| 30 | 54.24
| 40 | 53.11

(b) Different $R$ with P2

I | II | III | $mAP^{all}_{1.0}$
---|---|---|---
$\delta_{1}$ | $\delta_{2}$
0 | 0.5 | ✓ | ✓ | 45.17
0.1 | 0 | ✓ | ✓ | 54.96
0.1 | 0.5 |  | ✓ | 54.25
0.1 | 0.5 | ✓ |  | 52.69
0.1 | 0.5 | ✓ | ✓ | 55.46

(c) Constraints in refinement.

annotation | CPR | $mAP^{all}_{1.0}$
---|---|---
coarse |  | 38.48
coarse | ✓ | 55.46
center |  | 57.47

(d) Different annotation.

CPR | P2PNet | $mAP^{all}_{1.0}$
---|---|---
ResNet-50 | ResNet-50 | 55.46
ResNet-50 | ResNet-101 | 55.80
ResNet-101 | ResNet-101 | 56.43

(e) Different backbones.

Table 4: Ablation studies.

Performance Analysis. Pseudo box based localizer is almost equivalent to train
a detector that treats the points in the neighborhood of the annotated points
as positive samples and others as negative samples. The general detectors
perform label assignment with IoU, which depends heavily on the scale
information of the given bounding box. However, precise bounding box can not
be obtained from point annotation of POL, leading to poor performance of
Pseudo box based localizer. P2PNet adopts hungarian algorithm to achieve a
purely point-to-point assignment, obtaining better performance than pseudo box
based localizer. However, P2PNet is much sensitive to the accuracy of
annotated points and the semantic variance. Therefore, point refinement
strategy, effectively reducing the semantic variance of the annotation,
achieves better performance. CPR that can better capture the semantic
information, outperforms.

###  4.3 Ablation Studies

To further analyze CPR’s effectiveness and robustness, we conduct more
experiments on COCO.

Training Loss in CPRNet. Ablation study of the training loss is given in Table
2. The CPR loss given in row 6 in Table 2 obtains 55.46 mAP. i) MIL loss. If
the MIL loss is removed (row 4), the CPRNet training relies on the annotation
loss and the negative loss, the performance drops 3.64 points (51.82 _vs_
55.46). When we replace the MIL loss with the pos loss, which treats all the
sampled points in the MIL bag as positive samples (line 5), the performance
sharply declines by 12.74 points (42.72 _vs_ 55.46), showing that MIL can
autonomously discern points belonging to the object. ii) Annotation loss.
Lacking of the annotation loss (row 3), the performance of localization
decreases 1.22 points (54.24 _vs_ 55.46). The annotation loss guides the
training through an given accurate positive supervision. iii) Negative loss.
With the negative loss (row 2), the performance improves by 16.01 points
(55.46 _vs_ 39.45), indicating that only MIL loss is not enough to suppress
the background, and the negative loss is inevitable.

Feature Map Level. The CPRNet is established based on single level feature map
of FPN. Table 4(a) and 4(b) show the performance with different feature map
levels. Since the performance on P3 is similar to that of P2, P3 is chosen for
our experiments in COCO if not otherwise specified.

Sampling Scope. Table 4(a) and 4(b) show the performance of different radius
$R$, where $R$ is a sensitive hyper-parameter in CPRNet. On P3, the best
performance 55.46 is obtained when $R$ is set as 8. If the sampling scope
reduces, such as $R=$ 5, the performance significantly declines to 53.32,
since the sampling scope is limited to a small local region, leading to a
worse refinement. While the scope getting larger, the performance becomes
steady but drops slowly until $R$ is over 25 (53.85), since the bag $B_{j}$
for MIL introduces more noise, which degrades the performance.

Point Refinement Policy. For point refinement, there are three constraints
(described in Sec. 3.3). $\delta_{1}$ and $\delta_{2}$ are threshold of
constraint I. In Table 4(c), it shows that the three constraints together
obtain performance gain.

Upper Bound Analysis. To further validate the CPR, a comparison between CPR
and a strict annotation based localizer, which can be seen as the upper bound
for CPR, is conducted on COCO. Since it is hard to annotate the objects in
general dataset (e.g. COCO) with key points. Therefore, we approximately use
the center point of each objects’ bounding box as a kind of strict point
annotation. The experiment results in Tabel 4(d) show that CPR can achieve a
comparable performance to center point annotation based localizer (55.46 _vs_
57.47).

Localizer Architecture. Table 3 shows that CPR can further improve the
performance of different localizers, such as Faster-RCNN, RetinaNet and
RepPoint.

Backbone. As shown in Table 4(e), due to the stronger backbone Resnet-101 for
CPRNet and P2PNet, it obtains a better performance 56.43.

##  5 Conclusion and Outlook

In this paper, we rethink the semantic variance problem in point-based
annotation caused by the non-uniqueness of optional annotated points. The
proposed CPR samples points in neighbourhood, finds the semantic points on the
object by introducing MIL, and then weighted averages these semantic points to
obtain the semantic center of the object as the supervision for the localizer.
CPR alleviates semantic variance and facilitates the extension of POL task to
multi-class and multi-scale. Comprehensive ablations on multiple datasets
further verify the effectiveness of our model. In future, we will study on an
adaptive $R$ and explore the possibilities of extending CPR to other tasks.

Limitation. The performance is sensitive to $R$, which is not an adaptive
value in this paper and may limit CPR to better deal with the multi-scale of
objects to some extent.

Broader Impact. Similar to most of object detection and localization task, the
bias of dataset from the intrinsic artifacts are not considered.

##  6 Acknowledgements

This work was supported in part by the Youth Innovation Promotion Association
CAS, the National Natural Science Foundation of China (NSFC) under Grant No.
61836012 and 61771447, and the Strategic Priority Research Program of the
Chinese Academy of Sciences under Grant No.XDA27000000. The work was partially
done during Xuehui’s internship at JD Explore Academy, China. We would like to
thank Jing Zhang for helpful discussion and suggestions.

## References

  * [1] Boris Babenko, Ming-Hsuan Yang, and Serge J. Belongie.  Visual tracking with online multiple instance learning.  In CVPR, 2009.
  * [2] Boris Babenko, Ming-Hsuan Yang, and Serge J. Belongie.  Robust object tracking with online multiple instance learning.  IEEE TPAMI, 2011.
  * [3] Hessam Bagherinezhad, Maxwell Horton, and Mohammad Rastegari _et al_.  Label refinery: Improving imagenet classification through label progression.  CoRR, 2018.
  * [4] Hakan Bilen and Andrea Vedaldi.  Weakly supervised deep detection networks.  In CVPR, 2016.
  * [5] Léon Bottou.  Stochastic gradient descent tricks.  In Neural Networks: Tricks of the Trade - Second Edition. Springer, 2012.
  * [6] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin.  MMDetection: Open mmlab detection toolbox and benchmark.  arXiv preprint arXiv:1906.07155, 2019.
  * [7] Ze Chen, Zhihang Fu, and Rongxin Jiang _et al_.  SLV: spatial likelihood voting for weakly supervised object detection.  In CVPR, 2020.
  * [8] Ramazan Gokberk Cinbis, Jakob J. Verbeek, and Cordelia Schmid.  Weakly supervised object localization with multi-fold multiple instance learning.  IEEE TPAMI, 2017.
  * [9] Thomas G. Dietterich, Richard H. Lathrop, and Tomás Lozano-Pérez.  Solving the multiple instance problem with axis-parallel rectangles.  Artificial Intelligence, 1997.
  * [10] Jia-Chang Feng, Fa-Ting Hong, and Wei-Shi Zheng.  MIST: multiple instance self-training framework for video anomaly detection.  In CVPR, 2021.
  * [11] Xavier Glorot, Antoine Bordes, and Yoshua Bengio.  Deep sparse rectifier neural networks.  In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011, 2011.
  * [12] Yuqi Gong, Xuehui Yu, Yao Ding, Xiaoke Peng, Jian Zhao, and Zhenjun Han.  Effective fusion factor in FPN for tiny object detection.  In IEEE WACV, 2021.
  * [13] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV, 2017.
  * [14] Kaiming He, Xiangyu Zhang, and Shaoqing Ren _et al_.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [15] Yutao Hu, Xiaolong Jiang, and Xuhui Liu _et al_.  Nas-count: Counting-by-density with neural architecture search.  In ECCV, 2020.
  * [16] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu.  Ccnet: Criss-cross attention for semantic segmentation.  In ICCV, 2019.
  * [17] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu.  Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.  IEEE TPAMI, 2014.
  * [18] Xiaoheng Jiang, Li Zhang, and Mingliang Xu _et al_.  Attention scaling for crowd counting.  In CVPR, 2020.
  * [19] Victor S. Lempitsky and Andrew Zisserman.  Learning to count objects in images.  In NeurIPS, 2010.
  * [20] Tsung-Yi Lin, Piotr Dollár, and Ross B. Girshick _et al_.  Feature pyramid networks for object detection.  In CVPR, 2017.
  * [21] Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In ICCV, 2017.
  * [22] Tsung-Yi Lin, Michael Maire, and Serge  _et al_. Belongie.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [23] Alexander Neubeck and Luc Van Gool.  Efficient non-maximum suppression.  In ICPR, 2006.
  * [24] Phuc Nguyen, Ting Liu, and Gautam Prasad _et al_.  Weakly supervised action localization by sparse temporal pooling network.  In CVPR, 2018.
  * [25] Shaoqing Ren, Kaiming He, and Ross B. Girshick _et al_.  Faster R-CNN: towards real-time object detection with region proposal networks.  In NeurIPS, 2015.
  * [26] Javier Ribera, David Guera, Yuhao Chen, and Edward J. Delp.  Locating objects without bounding boxes.  In CVPR, 2019.
  * [27] Björn Runow.  Deep learning for point detection in images, 2020.
  * [28] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In ICLR, 2015.
  * [29] Qingyu Song, Changan Wang, Zhengkai Jiang, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Yang Wu.  Rethinking counting and localization in crowds: A purely point-based framework.  In ICCV, 2021.
  * [30] Waqas Sultani, Chen Chen, and Mubarak Shah.  Real-world anomaly detection in surveillance videos.  In CVPR, 2018.
  * [31] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang.  Deep high-resolution representation learning for human pose estimation.  In CVPR, pages 5693–5703, 2019.
  * [32] Peng Tang, Xinggang Wang, and Song Bai _et al_.  PCL: proposal cluster learning for weakly supervised object detection.  IEEE TPAMI, 2020.
  * [33] Koen E. A. van de Sande, Jasper R. R. Uijlings, and Theo Gevers _et al_.  Segmentation as selective search for object recognition.  In ICCV, 2011.
  * [34] Fang Wan, Chang Liu, and Wei Ke _et al_.  C-MIL: continuation multiple instance learning for weakly supervised object detection.  In CVPR, 2019.
  * [35] Fang Wan, Pengxu Wei, Zhenjun Han, Jianbin Jiao, and Qixiang Ye.  Min-entropy latent model for weakly supervised object detection.  IEEE PAMI, 2019.
  * [36] Jiajie Wang, Jiangchao Yao, and Ya Zhang _et al_.  Collaborative learning for weakly supervised object detection.  In IJCAI, 2018.
  * [37] Qi Wang, Junyu Gao, and Wei Lin _et al_.  NWPU-crowd: A large-scale benchmark for crowd counting and localization.  IEEE TPAMI, 2021.
  * [38] Pingyu Wu, Wei Zhai, and Yang Cao.  Background activation suppression for weakly supervised object localization.  arXiv preprint arXiv:2112.00580, 2021.
  * [39] Gui-Song Xia, Xiang Bai, and Jian Ding _et al_.  DOTA: A large-scale dataset for object detection in aerial images.  In CVPR, 2018.
  * [40] Xue Yang, Liping Hou, Yue Zhou, Wentao Wang, and Junchi Yan.  Dense label encoding for boundary discontinuity free rotation detection.  In CVPR, 2019.
  * [41] Xue Yang and Junchi Yan.  Arbitrary-oriented object detection with circular smooth label.  In ECCV, 2019.
  * [42] Xue Yang, Junchi Yan, Qi Ming, Wentao Wang, Xiaopeng Zhang, and Qi Tian.  Rethinking rotated object detection with gaussian wasserstein distance loss.  In ICML, 2021.
  * [43] Xue Yang, Xiaojiang Yang, Jirui Yang, Qi Ming, Wentao Wang Qi Tian, and Junchi Yan.  Learning high-precision bounding box for rotated object detection via kullback-leibler divergence.  In NeurIPS, 2021.
  * [44] Ze Yang, Shaohui Liu, and Han Hu _et al_.  Reppoints: Point set representation for object detection.  In ICCV, 2019.
  * [45] Hang Yu, Yufei Xu, Jing Zhang, Wei Zhao, Ziyu Guan, and Dacheng Tao.  Ap-10k: A benchmark for animal pose estimation in the wild.  In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
  * [46] Xuehui Yu, Yuqi Gong, and Nan Jiang _et al_.  Scale match for tiny person detection.  In IEEE WACV, 2020.
  * [47] Dan Zhang, Fei Wang, and Zhenwei Shi _et al_.  Interactive localized content based image retrieval with multiple-instance active learning.  Pattern Recognition, 2010.
  * [48] Jing Zhang, Zhe Chen, and Dacheng Tao.  Towards high performance human keypoint detection.  IJCV, 129(9):2639–2662, 2021.
  * [49] Qi Zhang, Sally A. Goldman, and Wei Yu _et al_.  Content-based image retrieval using multiple-instance learning.  In ICML, 2002.
  * [50] Qiming Zhang, Yufei Xu, Jing Zhang, and Dacheng Tao.  Vitaev2: Vision transformer advanced by exploring inductive bias for image recognition and beyond.  arXiv preprint arXiv:2202.10108, 2022.
  * [51] Shanshan Zhang, Rodrigo Benenson, and Bernt Schiele.  Citypersons: A diverse dataset for pedestrian detection.  In CVPR, 2017.
  * [52] Yingying Zhang, Desen Zhou, and Siqin Chen _et al_.  Single-image crowd counting via multi-column convolutional neural network.  In CVPR, 2016.
  * [53] C. Lawrence Zitnick and Piotr Dollár.  Edge boxes: Locating object proposals from edges.  In ECCV, 2014.

Appendix

##  Appendix A More experiment.

Sample shape and sample density. In Table 5(b), we ablate on sample density
$u_{0}$ and validate that $u_{0}$ has little impact. Theoretically, circles
are orientation equivalent, which makes it suitable to model multi-view
object. Experimentally, the sampling results of circle and rectangle are very
close due to the discrete sampling, leading to comparable performance (Table
5(a), where different aspect ratio $rect._{w:h}$ are studied).

Threshold of refinement. $\delta_{1}$ and $\delta_{2}$ are thresholds, well
defined in detection and localization tasks. $\delta_{1}$, $\delta_{2}$,
$u_{0}$ have little impact to performance (Table 5).

$region$ | $mAP^{1.0}$
---|---
$circle$ | 55.46
$rect._{1:1}$ | 55.39
$rect._{2:1}$ | 54.77
$rect._{1:2}$ | 54.96
(a) $region$

$u_{0}$ | $mAP^{all}_{1.0}$
---|---
12 | 55.25
8 | 55.46
6 | 55.36
4 | 55.42
(b) $u_{0}$

$\delta_{1}$ | $mAP^{all}_{1.0}$
---|---
0.10 | 55.46
0.15 | 55.98
0.20 | 56.04
0.25 | 55.78
(c) $\delta_{1}$

$\delta_{2}$ | $mAP^{all}_{1.0}$
---|---
0.25 | 55.19
0.50 | 55.46
0.75 | 55.22
1.0 | 55.41
(d) $\delta_{2}$

Table 5: Ablation study for hyper-parameters.

##  Appendix B SeaPerson

SeaPerson is building for tiny person localization, which can help maritime
quick rescue, beach safety inspection and so on. The resolution of images are
mainly 1920 $\times$ 1080 and the person size is extremely low (about 22.6
pixels). Therefore, there is no privacy sensitive information.

Dataset Collection. SeaPerson is collected as : i) Videos are recorded in
various seaside scenes by a RGB camera on a Unmanned Aerial Vehicle. ii) We
sample an image of every 50 frames from video and remove images with high
homogeneity. iii) We annotated all persons in all sampled images with bounding
boxes. iv) Following the rules of coarse point annotation in Sec 4.1, coarse
point annotation is obtained on SeaPerson for POL task.

Dataset Splitting. We randomly split dataset into three subsets (training set,
valid set and test set), while images from the same video sequence cannot be
separated into different subsets. As shown in Table 6, the ratio of images’
number in training set, valid set and test set is about 10:1:10.

Dataset Properties. Our proposed SeaPerson is similar with TinyPerson while
the volumn of SeaPerson is about 7 times that of TinyPerson. The absolute size
and relative size of objects are very small as shown in Table 7 and Fig. 7. In
such scenario, we only care about the position of the object rather than the
size of the object, which makes it very suitable for POL task. In addition,
SeaPerson can also be used as a dataset for tiny object detection due to the
bounding box annotation.

##  Appendix C Implementation Details for CPRNet

ResNet-50 is used as the backbone network unless otherwise specified and FPN
is adopted for feature fusion. P2 (stride is 4) is used for SeaPerson and P3
(stride is 8) is used for COCO and DOTA. The mini-batch is 64/4/4 images and
all models are trained with 8/2/4 GPUs for COCO/DOTA/SeaPerson. The training
epoch numbers are set as 12/12/6, the learning rate are set as 0.0025,
0.00025, 0.0125 and decays by 0.1 at the 8-th/8-th/4-th and 11-th/11-th/5-th
epoch for COCO/DOTA/SeaPerson, respectively. In default settings, the backbone
is initialized with the pre-trained weights on ImageNet and other newly added
layers are initialized with Xavier. The sampling radius $R$ is set as 8/7/5
for COCO/DOTA/SeaPerson by defalut.

In dataset pre-processing, for COCO dataset, the short side of the images is
resized to 400, and the ratio of width and height is kept. In dota dataset,
images are split into sub-images (1024 $\times$ 1024 pixels) with overlap (200
$\times$ 200 pixels). And in SeaPerson, images are split into sub-images (640
$\times$ 640 pixels) with overlap (100 $\times$ 100 pixels). For data
augmentation, only random horizontal is utilized in our CPRNet training.

| TinyPerson V2 | TinyPerson
---|---|---
| #images | #annotations | #images | #annotations
train set | 5711 | 262063 | 794 | 42197
valid set | 568 | 42399 | 816 | 30454
test set | 5753 | 315165 | - | -
sum | 12032 | 619627 | 1610 | 72651

Table 6: Statistic information of SeaPerson and TinyPerson.

dataset | absolute size | relative size | aspect ratio
---|---|---|---
COCO |  99.5 $\pm$ 107.5 |  0.190 $\pm$ 0.203 |  1.213 $\pm$ 1.337
TinyPerson |  17.0 $\pm$ 16.9 |  0.011 $\pm$ 0.010 |  0.690 $\pm$ 0.422
TinyPerson V2 |  22.6 $\pm$ 10.8 |  0.016 $\pm$ 0.007 |  0.723 $\pm$ 0.424

Table 7: The mean and standard deviation of absolute size, relative size and
aspect ratio of objects in different datasets. Size is defined as the square
root of the product of width and height and aspect ratio is the value of width
divided by height. Absolute size is the size of object and relative size is
the value of the object’s size divided by the image’s size. These settings are
followed as [46].
---
Figure 7: Examples of SeaPerson.

##  Appendix D Details of Semantic Variance

The $Var(x^{\prime})$ and $Var(y^{\prime})$ in Eq.
LABEL:Eq:a_semantic_variance are calculate as:

$\displaystyle Mean(x^{\prime})=\frac{1}{M}\sum\limits_{1\leq j\leq
M}{x^{\prime}_{j}};$ (15) $\displaystyle
Mean(y^{\prime})=\frac{1}{M}\sum\limits_{1\leq j\leq M}{y^{\prime}_{j}};$
$\displaystyle
Var(x^{\prime})=\frac{1}{M}(x^{\prime}_{j}-Mean(x^{\prime}))^{2};$
$\displaystyle
Var(y^{\prime})=\frac{1}{M}(y^{\prime}_{j}-Mean(y^{\prime}))^{2};$

where $(x^{\prime}_{j},y^{\prime}_{j})$ is relative position of $j$-th object,
$M$ is the number of objects in dataset. For the objects whose annotated
points or refined points are out of the bounding box, they often are regarded
as wild points during the learning procedure. The wild points account for a
small proportion and will not be learned by the network. However, their $RSV$
will be very large since the $RSV$ is relative to the height and width.
Therefore, to better reflect the semantic variance of the points that really
affect network learning, only the object whose annotated point or refined
point is inside bounding box is used for calculating $RSV$.

##  Appendix E Visualization of CPR

The visulization of CPR on COCO, DOTA and SeaPerson are shown as Fig. 9, Fig.
10 and Fig. 8, respectively.

Figure 8: Visualization of CPR on SeaPerson. The images are cut from original
images for better visualization. Semantic points (red) around the annotated
point (green) are weighted averaged to obtain the semantic center (yellow) as
final refined point (see Sec. 3.3).

Figure 9: Visualization of CPR on COCO. The images are cut from original
images for better visualization. Semantic points (red) around the annotated
point (green) are weighted averaged to obtain the semantic center (yellow) as
final refined point (see Sec. 3.3).

Figure 10: Visualization of CPR on DOTA. The images are cut from original
images for better visualization. Semantic points (red) around the annotated
point (green) are weighted averaged to obtain the semantic center (yellow) as
final refined point (details in Sec. 3.3).


2 |  Prepare the filter insert by folding the paper filter in half to create a semi-circle, and in half again to create a quarter-circle. Place the paper filter in the dripper and spread open to create a cone. |  Fold paper filter in half, then half again. Put filter in dripper, form cone shape.
3 |  Rinse the filter. Pour enough hot water into the filter to wet it. The entire paper filter should be moist. Rinsing the filter will remove any papery residue so your coffee doesn’t have a woodsy taste. |  Wet filter with water to rinse away residue.
4 |  Lift up the dripper and pour out the water. Then set the dripper with the wet filter back on the coffee mug. |  Remove dripper, pour out water, and return dripper to coffee mug.
5 |  Get out a digital scale and measure out 3 tablespoons (about 30 g) of coffee beans. Measure out 30 g of whole beans and place them in your grinder. |  Measure 30g coffee beans on a digital scale (right side), place in grinder (right side).
6 |  Grind the beans until the coffee grounds are the consistency of coarse sand, about 20 seconds. |  Grind beans for 20 seconds, until coarse sand consistency.
7 |  Transfer the coffee grounds to the filter cone. Then place the coffee mug with the dripper on a digital scale and set it to zero. |  Move grounds to filter cone. Set coffee mug with dripper on scale, zero it.
8 |  Slowly pour the water over the grounds in a circular motion. Do not overfill beyond the top of the paper filter. Your scale should read 100 g once you’ve poured enough water into the dripper. |  Slowly pour water in circles over grounds, stopping at 100g on scale.
9 |  Let the coffee drain completely into the mug and wait for 30 seconds and you can complete the task; |  Drain coffee into mug and wait for 30 seconds to end.
Table 3. Study 1 Task 1 sample result. The original and simplified versions of the text are listed. The sample is collected from P10’s experiment session and the simplified results for other participants may vary slightly due to spatial context. Step |  Original |  Simplification
---|---|---
1 |  Before arranging the meeting room, take a moment to tidy up the desk and move anything that’s not necessary to other desks; |  Tidy desk, move the unnecessary items to other desks.
2 |  Once the desk is clear, bring the power strip on the desk and connect the Charger to the power strip so the meeting attendants can use. |  Put power strip on desk, connect phone charger to it.
3 |  Connect the camera’s charger to the power strip and position the camera at the opposite end of the desk from the TV. |  Connect camera to strip, facing opposite of TV.
4 |  Arrange the chairs in the meeting room. Make sure that there’s enough space between each chair - roughly 1.5 feet should suffice. Position one chair on the window side, and place five chairs on the other side. |  Arrange chairs on two sides. Leave space of roughly two A4 papers’ length apart. Window side: 1 chair. Other side: 5 chairs.
5 |  Next, place cups of water and papers on each chair. Each person should have one cup of water and paper; |  Place water, paper onto desk in front of chairs.
6 |  Put up the desk nameplates on on each chair. When Alice is on the side of the window, other desk nameplates should be put on the other side. The sequence is Bob, Amy, Andy, Dave and Luis. |  Place nameplates: Window side: Alice (window); sequence (left to right) on other side: Bob, Amy, Andy, Dave, Luis.
7 |  Since Alice is the VIP in the meeting, place make it clearly by putting the remote controller to Alice’s position. |  Place remote controller at Alice’s position on desk.
Table 4. Study 1 Task 2 sample result. The original and simplified versions of
the text are listed. The sample is collected from P10’s experiment session and
the simplified results for other participants may vary slightly due to spatial
context.