# Anchor Distance for 3D Multi-Object Distance Estimation from 2D Single Shot

Hyeonwoo Yu and Jean Oh Hyeonwoo Yu and Jean Oh are with the Robotics
Institute of Carnegie Mellon University, Pittsburgh, PA 15213, USA
<EMAIL_ADDRESS>
###### Abstract

Visual perception of the objects in a 3D environment is a key to successful
performance in autonomous driving and simultaneous localization and mapping
(SLAM). In this paper, we present a real time approach for estimating the
distances to multiple objects in a scene using only a single-shot image. Given
a 2D Bounding Box (BBox) and object parameters, a 3D distance to the object
can be calculated directly using 3D reprojection; however, such methods are
prone to significant errors because an error from the 2D detection can be
amplified in 3D. In addition, it is also challenging to apply such methods to
a real-time system due to the computational burden. In the case of the
traditional multi-object detection methods, existing works have been developed
for specific tasks such as object segmentation or 2D BBox regression. These
methods introduce the concept of anchor BBox for elaborate 2D BBox estimation,
and predictors are specialized and trained for specific 2D BBoxes. In order to
estimate the distances to the 3D objects from a single 2D image, we introduce
the notion of anchor distance based on an object’s location and propose a
method that applies the anchor distance to the multi-object detector
structure. We let the predictors catch the distance prior using anchor
distance and train the network based on the distance. The predictors can be
characterized to the objects located in a specific distance range. By
propagating the distance prior using a distance anchor to the predictors, it
is feasible to perform the precise distance estimation and real-time execution
simultaneously. The proposed method achieves about 30 FPS speed, and shows the
lowest RMSE compared to the existing methods.

##  I Introduction

Real-time visual perception and understanding of an environment is critical to
the successes of robotic applications including autonomous driving. Visual
Simultaneous Localization and Mapping (SLAM), in particular, is essential for
performing navigation and exploration tasks as well as supporting high-level
missions that require reliable mobility. Recently, SLAM focuses on two major
facets: semantic recognition and spatial understanding [1, 2, 3, 4, 5]. With
the advancement of deep neural networks, several approaches have been
developed to achieve highly accurate results on semantic recognition, taking
advantage of rich and sophisticated semantic features and various disentangled
features such as shape, orientation, or dimensions of 2D and 3D Bounding Box
(BBox) [6, 7, 8, 9, 10, 11]. In this paper, we focus on real-time spatial
understanding including accurate estimation of object locations specifically
addressing the challenge in a monocular camera setting.

The majority of existing work on SLAM systems for autonomous driving relies
heavily on an inverse reprojection method [12, 13, 14, 15] where an optimal
distance is computed by reprojecting a 3D shape or its 3D BBox to a
corresponding 2D BBox. The performance of such a method, however, tends to be
extremely sensitive to that of image segmentation and/or 2D BBox estimation.

Figure 1: The overview of the proposed method. The network estimates the
distance of multi-object with multiple predictors from single image. Each
predictor is specialized to the object located in specific distance range by
using anchor distance. With the anchor distance, predictors are provided
priors of the distance so that accurate estimation under simple and fast
network structure can be achieved.

Alternatively, depth estimation methods [16, 17, 18] can be utilized for
object-level 3D estimation, by taking the median depth of all geometric
features or pixels within the detected 2D BBox [4, 13, 19]. In these methods,
however, it is hard to discard the outlier pixels especially if an object is
severely occluded. Adding a segmentation method such as [20] can have minor
improvements in the performance at the cost of significant computational
burden; estimating the depth of object’s center from depth values of partial
observation is still challenging.

To achieve robust object depth estimation, we can exploit existing single-
shot, multi-object detection approaches, which are specialized in category
classification, segmentation, and 2D BBox regression [20, 21, 22, 23]. Based
on such methods, various approaches have been proposed for object disentangled
representation [24, 25, 26, 27]. Some works on multi-object understanding use
multi-object detector to obtain object region followed by a post-processing
step [28, 9, 29]. Others train their baseline network with Region Proposal
Network (RPN) and perform Region of Interest (RoI) pooling to obtain visual
features of objects [30]. These methods deploy additional structure to
estimate various object representations in addition to the baseline for multi-
object detection. Such an increased complexity of the network makes real-time
performance challenging [20, 28]. To simplify the structure of the network,
prior knowledge of an object can be exploited. Since the purpose of the
existing multi-object detector is to perform the 2D BBox regression, anchor
BBoxes are used as prior knowledge of 2D BBox [23, 22, 31]. To utilize such
prior knowledge represented by anchor BBoxes, several methods construct their
networks by deploying multiple predictors according to anchor BBoxes. These
methods, however, have limitations in learning object representations such as
distance estimation as they mainly focus on the 2D BBox on the image plane.

To bridge the gap between existing multi-object detection and distance
estimation, we introduce the notion of anchor distance. We then propose a
multi-object distance estimation method specifically designed for both real-
time performance and accurate estimation. Given a 2D image as an input, we aim
to estimate the distance to objects in a 3D space. Our work makes the
following contributions to the state-of-the-art methods:

  * •

as shown in Fig. 1, we transform the 2D single shot to 3D spatial grids
represented as predictors so that the proposed method achieves real-time
performance without a 2D RoI pooling network;

  * •

compared to existing 2D detectors, the proposed method achieves robust
detection results with overlapped objects since objects are detected in 3D
grid space; and

  * •

we define and utilize the notion of anchor distance, thus predictors in our
proposed network are specialized and robust to the objects in specific
distance range.

When compared to the state-of-the-art method, the proposed method runs about
$4$ times faster at 30 FPS and shows competitive results in Abs Relative (Abs
Rel) difference, and outstanding results in RMSE.

##  II Related work

In the context of mobile robot navigation, 3D object detection and
localization are compulsory to perform collision avoidance, object-oriented
SLAM or safe navigation in autonomous driving. In this section, we discuss
related works on monocular SLAM specifically focusing on the 3D localization.
Various existing works on this end adopt the idea of inverse perspective
projection [13, 14, 3, 15]. The 2D projection of an object is invertible since
its extent parameters resolve depth ambiguity. That is, by mapping between 2D
BBoxes from an object detection module and 3D BBoxes, these approaches can
estimate the distance in 3D given a 2D input image. Also, in a special setting
such as autonomous driving, we can assume that the height of a camera is known
and all of the objects are placed on the ground. This allows the algorithm to
resolve the scale-ambiguity and estimate object location. These approaches are
still based on 2D BBox regression, the quality of distance estimation is also
bounded by the precision of the 2D BBoxes.

To obtain 2D perception such as 2D BBox regression, using an additional object
detection method is inevitable. Several convolutional neural network
(CNN)-based algorithms have been proposed to perform multi-object detection in
real time. These approaches are mainly customized for specific tasks such as
detection using 2D BBox regression, object categorization, and segmentation.
Because the performance of these approaches depends on the quality of 2D
BBoxes inferred from an object detector, results of 2D BBox detection have
been commonly used as an evaluation criterion for the overall performance
[23]. This trend has led to the development of new techniques and network
designs to improve the performance of 2D BBox regression [22, 31].

Existing multi-object perception techniques mainly focus on 2D BBox regression
and category estimation, and their variations estimate disentangled
representations such as orientation, shape, or distance of objects [7, 24,
32]. Most of the proposed methods exploit visual features to learn the complex
representations of objects [6, 11, 33, 34, 35]. Therefore, by using an
additional multi-object detector, 2D BBox is provided for RoI pooling in order
to obtain visual features for the target region. Other methods are proposed by
modifying the existing multi-object detector structure for directly extracting
features and performing object understanding [36, 24, 20, 30]. However, they
still deploy the RPN for RoI pooling. Moreover, it is necessary to construct
the networks in addition to the baseline structure for object understanding
tasks with a large variation such as depth estimation. For these networks they
use structures with several drawbacks such as the increase of memory footprint
to cover the non-linearity of object understanding and perform elaborate
estimation [30, 32]. These methods are effective in representing specific
aspects of objects, but it is challenging to apply to mobile robot systems in
real-time due to their complexity.

In order to relax the network and achieve more robust results, a number of
methods have been introduced to provide prior information about objects to the
networks using anchors. Techniques for 2D detection exploit anchor of 2D BBox
[23, 22, 31], and techniques for 3D include orientation and distance as well
as 2D BBox in anchor [32, 27]. However, in these methods, anchors are still
defined by clustering based on the 2D BBox of object on 2D image plane. Since
various object representations such as locations, shape, or orientation are
not directly proportional to the projected 2D BBox, it is not suitable to use
anchors arranged with 2D BBox for learning the distance of objects. Therefore,
we define anchor distance for object 3D localization and introduce a method of
training networks based on the anchors that achieves real-time performance.

##  III Approach

In general, when a monocular camera sensor is used for recognizing an object,
the location coordinates can be estimated using the mapping between the 2D
object detection and both the dimension and the orientation of a corresponding
3D BBox. Assume that a 2D BBox of a detected object and the dimension and the
orientation of a 3D BBox are given or have been estimated. With the constraint
that the 3D BBox fits tightly into the 2D detection box on the 2D input image,
3D coordinates of the detected object can be calculated; however, estimating
3D coordinates by overlapping the projection of 3D to 2D BBoxes usually causes
inaccurate results due to the sensitivity to the size of the 2D BBoxes. A
small error in the size of BBox can cause a substantial error in the distance
estimation calculation. Furthermore, this approach can add a significant
calculation burden, as each side of the 2D BBox can correspond to any of the
eight corners of the 3D BBox. Even with the strong constraint that most of the
objects are located on a ground plane, at least 64 configurations must be
checked [10]. To address these challenges, we propose an approach that
achieves both high accuracy and real-time performance by directly estimating
the object distance.

###  III-A 3D Coordinate and 2D Bounding Box

Given 3D directions to the center of object and depth (or distance from
origin), 3D coordinates can solely be determined. For estimating the center of
the 3D object, the center of a 2D BBox can be exploited by reprojecting it to
the 3D real-world coordinate. In this way, we can have a normalized ray
direction vector toward the center of the 3D object from the origin. Hence,
the 3D coordinate of an object can be obtained by multiplying this ray
direction vector and estimated distance. The network can also be trained to
directly estimate all 3D coordinates of an object. In case of using multi-
object detector, however, it can be relatively advantageous to estimate the
center of an object by using the predictors distributed in a grid form. Hence,
it is more accurate to compute the coordinate from reprojection than learning
the $x$,$y$, and $z$ components of location directly. We design our network to
train on the distance from the origin and the 2D BBoxes as in [21, 31].

###  III-B Anchor Distance

In order to achieve the real-time multi-object distance estimation, our
proposed method is based on a simple multi-object detection structure, namely,
YOLOv2 [21]. As the existing network only estimates the center and dimensions
of a 2D BBox, we let our network predict the distance from the origin of the
object as well. The following problem, however, still remains: the purpose of
conventional detectors is to estimate the 2D BBoxes rather than the distances
to the objects, and each predictor are dedicated to learn for the
corresponding BBoxes of different sizes. To reduce such a burden of the
network, a 2D anchor BBox is applied to each predictor as the prior knowledge
about the 2D BBox sizes. These approaches reduce the variation of the 2D
BBoxes that each predictor should predict, resulting in more accurate 2D BBox
estimation.

TABLE I: Variance of the distance of 2D BBox groups and distance groups for KITTI dataset |  |  | (unit : $m^{2}$)
---|---|---|---
# of predictors | order | 2D BBox | distance
(groups) | grouping | grouping
2 | 1 | 14.84 | 25.69
2 | 220.12 | 31.76
3 | 1 | 9.07 | 12.26
2 | 33.26 | 8.60
3 | 186.27 | 20.30
5 | 1 | 7.08 | 5.36
2 | 18.68 | 4.27
3 | 49.57 | 3.10
4 | 91.14 | 9.28
5 | 98.21 | 3.55

In order to achieve 3D coordinate estimations of multiple objects, we
introduce the concept of anchor distance that is similar to the anchor BBox.
To obtain the prior knowledge of the distance in a simple manner, the average
distance corresponding to each anchor BBox can be defined by using the average
of distances of objects belonging to each anchor BBox group. Unfortunately,
the size of an anchor BBox is not exactly proportional to the distance of the
object. When an anchor BBox simply includes the average distance, each
predictor can undesirably learn the mapping between the size of a 2D BBox and
a distance. In other words, the network still learns the 2D BBox regression as
its main task, leaving the burden of distance estimation to each individual
predictor where the distances within a large range should still be estimated.

To address this issue, we define the concept of an anchor distance to train
each predictor as follows. With an anchor distance, each predictor is
specialized in estimating the distance of the object in a specific distance
range, instead of being specialized in specific 2D BBoxes. Similar to
obtaining the anchor BBoxes, the distances of objects are grouped through
$k$-means clustering. Each center of the groups (or clusters) is defined as
anchor distance. We compare the variance of the average distance obtained from
2D BBox clustering and that of the anchor distance from distance clustering
for the case of $k=2,3,5$ in Table I. We use the car category in the KITTI
dataset [37] as an example. The predictors are sorted by the corresponding
distance in ascending order. In the case of 2D BBox grouping, the variance of
the group for the long distance is greater than the one for the short
distance. This is because objects that are far away from the origin are
similar in terms of size of the 2D BBox. On the other hand, in the case of
distance clustering, the variance is much smaller than that in the case of 2D
BBox clustering for all predictors. Therefore, predictors can show more
precise estimation results as they infer more consistent distances with anchor
distance. The results of the network prediction using anchor distance are
denoted as follows:

$\displaystyle d_{i}$ $\displaystyle=d^{a}_{i}\times exp(t_{i})\text{ ,
}\text{ }i\in\\{0,1,...,k-1\\}$ (1)

where $t$ is an output of the network, and $d^{a}$ is the anchor distance.
Similar to [21, 31], we use an exponential function as the final activation
function of the output.

In our method, the dimension of 2D BBox has no effect on distance estimation
directly, but the center of 2D BBox is crucial as it is exploited to calculate
the 3D coordinate by finding the ray direction. To achieve the 2D BBox
regression, priors of the BBox can be defined for our distance grouping.
Simply, we can define the average BBox of clusters for anchor distance by
taking the arithmetic mean of dimensions of the BBoxes. However, our clusters
focus on distance so that BBoxes in a group have large variance in terms of
their sizes. For more accurate BBox regression with anchor distance groups, we
take the average BBox approximately where we minimize not the differences of
dimensions but the differences of intersection over union (IoU) as follows:

$\displaystyle\left(h^{m}\right)^{2}=\frac{\sum w_{j}h_{j}^{2}}{\sum
w_{j}},\text{ }\text{ }\left(w^{m}\right)^{2}=\frac{\sum h_{j}w_{j}^{2}}{\sum
h_{j}}$ (2)

where $h^{m}$ and $w^{m}$ are the height and width of the average BBox,
respectively. $h$ and $w$ are height and width of BBox. Using this average
BBox, the network output for BBox dimension is given as:

$\displaystyle h_{i}$ $\displaystyle=h^{m}_{i}\times exp\left(u_{i}\right)$
$\displaystyle w_{i}$ $\displaystyle=w^{m}_{i}\times
exp\left(v_{i}\right)\text{ , }\text{ }i\in\\{0,1,...,k-1\\}$ (3)

where $u$ and $v$ are outputs of the network. For the center of the 2D BBox,
we follow the similar settings of [21].

Figure 2:  Distance estimation error of $x$ and $z$ axises, which denote the
horizontal location and depth of the object. As the distance of the object
increases, the estimation error increases in case of other methods. The
proposed method shows consistent error independent to the distance from origin
as the method exploits the anchor distance.

###  III-C Predictors

With the anchor distance, we can provide the prior knowledge of the distance
to our network. The predictor is specialized and trained for objects near
specific distance as each anchor distance is assigned to each predictor of the
network. As the variance of the distance inferred by a predictor decreases,
the complexity of the network is decreased. The predictors of the network can
construct a 3D environment without additional structures such as 3D RPN or 3D
convolutional layers.

The existing multi-object detectors utilize 2D anchor BBoxes by clustering 2D
BBoxes. In order to cluster 2D BBoxes, we can use the dimensions of the BBox
or IoU. Likewise, anchor distance can be achieved by using various formats
such as normal, squared, or log-scaled. In our work we apply all three cases
to obtain the anchor distance and train the network. Whilst training, the
existing multi-object detector using anchor BBox chooses the predictor that
infers the closest BBox to the target BBox and assigns that target BBox to
learn. Similarly, the proposed method learns the distance of the target object
by selecting and training a predictor which infers the value closest to the
target distance. Note that the same distance format used for obtaining anchor
distance is also used when comparing the the difference between the target
distance and estimated one by the predictor during training.

At the beginning of training, we found that the predictor’s inference value
highly fluctuates in order to learn distances with quite large variations. In
this case, objects are inconsistently assigned to the predictors regardless of
the anchor distance. Therefore, during the training phase, we use the
predictor’s anchor distance rather than the estimated result for predictor
selection.

###  III-D Training Loss

In the proposed method, the distance from the origin of an object is
estimated. Since a ray vector indicating the center direction of an object is
obtained using a 2D BBox, all 3 components of 3D coordinate can be obtained by
using distance from origin or depth. Therefore, it is possible to separately
train $x$,$y$, and $z$. However, we assume that 2D BBox and 3D location are
independent and let 2D BBox and the distance be learned separately. For the 2D
BBox training, we adopt CIoU loss [38, 31]. For depth estimation, $L_{1}$ loss
is generally used, but in our work $L_{2}$ loss is applied.

##  IV Implementation and Training Details

To implement the proposed observation model, we use the darknet19 structure
[21] for the encoder backbone. We construct the encoder by adding 3
convolutional layers with 1,024 filters followed by one convolutional layer on
top of the backbone. The final convolutional layer has
$k\times\left(4+1\right)$ filters; $4$ for 2D BBox dimensions and center, and
$1$ for the distance. Each predictor estimates 2D BBox and distance from the
origin of the object. We pretrain the backbone network on the Imagenet
classification dataset. We use the Adam optimizer with a learning rate of
$10^{-4}$. For all training processes, a multi-resolution augmentation scheme
is adopted. Similar to [21, 8], Gaussian blur, HSV saturation, RGB invertion
and random brightness are applied to 2D images while training. Random scaling
and translation are not used in order to preserve the 3D coordinates of
objects.

##  V Experiments

We evaluate the proposed method in various aspects. Similar to the traditional
multi-object detectors, as the number of predictors increases, the estimation
can be performed more precisely. In our experiments, we train the network with
various numbers of predictors such as $2$, $3$, $5$ and so on. We compare our
methods to existing methods based on Faster R-CNN using RPN. A depth
estimation method is also compared by using ground truth 2D BBox. We use the
median depth within 2D BBox as the depth of a detected object. For the depth
estimation method, we choose [16] since the method proposes ordinal regression
in order to consider depth intervals. Additionally, we also implement several
methods including ours in order to estimate the distance of objects - 1) 3D
proj: we implement the method that calculates distance by projecting 3D BBox
in 3D space to 2D BBox on 2D image. We construct the network which shares the
same backbone with our proposed method, and train the network on dimensions of
2D, 3D BBox and orientations of objects. 2) anchor BBox: similar to the
existing detector, we evaluate the method that uses anchor BBox and average
distance of groups obtained from 2D BBox clustering. In this case the average
distance is calculated by taking arithmetic mean. We also evaluate the case
without any distance prior. 3) proposed method: the proposed method using
anchor distance and its training scheme is evaluated. For obtaining anchor
distance by distance clustering, normal, squared, and log-scale formats are
used for 3 individual models. For each model, the same format used for anchor
distance is also used for training; while choosing the predictor for the
target object, target distance and estimated distances from predictors are
compared by using the same format of distance that is used for clustering.

For all experiments, we use the car category in the KITTI 3D object detection
dataset. Since each grid in our network has multiple predictors, for
evaluation we choose the predictor which estimates the highest 3D IoU to the
ground truth.

###  V-A Relation Between Object Distance and Error

We compare the results of 3D proj, anchor BBox with and without average
distance, and the proposed method in Fig. 2. We plot the distance error of the
object according to its distance from origin. For the method anchor BBox and
the proposed method, we set $k=5$. For the distance format of the proposed
method, we use squared format. Results of $x$ and $z$ axis are shown, which
denote horizontal and depth of the object location respectively. We leave the
result of $y$ axis out, since the height of cars have small variations
compared to the depth of cars so that errors of $y$ axis are significantly
smaller than errors of the other axes. As shown in the graph, the estimation
result of 3D proj is significantly more inaccurate as the objects locate
further. The further the objects are, the more similar the sizes of the 2D
BBoxes are; therefore, it is challenging to infer the distance precisely since
the estimations are highly sensitive to the errors of small 2D BBoxes of
objects in a far distance. Compared to 3D proj, methods using anchor BBox have
more robust inferences of distance. However, as the variations are high for
the distance of objects located far from the camera, it is still challenging
to estimate the far distance without a tremendous error. From these results we
can conclude that the distance and the 2D BBox of the object are not directly
proportional to each other, but there exists a slight correlation. In the case
of using anchor distance, estimations results show constant and uniform error
regardless of the distance of the object.

TABLE II: Variance of the distance of 2D BBox groups and distance groups for KITTI dataset Method | FPS | # of | $\sigma<{1.25}$ | $\sigma<{1.25}^{2}$ |  | Abs Rel | Sqr Rel | RMSE | RMSElog
---|---|---|---|---|---|---|---|---|---
predictors | (higher is better) |  | (lower is better)
SVR[39] | - | - | 0.345 | 0.595 |  | 1.494 | 47.748 | 18.970 | 1.494
IPM[40] | - | - | 0.701 | 0.898 |  | 0.497 | 35.924 | 15.415 | 0.451
Zhu et al.(ResNet50)[28] | $<15$ | - | 0.796 | 0.924 |  | 0.188 | 0.843 | 4.134 | 0.256
Zhu et al.(VggNet16)[28] | - | 0.848 | 0.934 |  | 0.161 | 0.619 | 3.580 | 0.228
Zhang et al.(MaskRCNN[ResNet50])[30] | $<7$ | - | 0.988 | - |  | 0.051 | - | 2.103 | -
Zhang et al.(MaskRCNN[ResNet50] + addons)[30] | - | 0.992 | - |  | 0.049 | - | 1.931 | -
DORN (depth map estimation) [16] | - | - | 0.883 | 0.934 |  | 0.190 | 1.153 | 4.802 | 0.287
2D anchor BBox w/o distance prior | $<\boldsymbol{35}$ | 3 | 0.906 | 0.977 |  | 0.103 | 0.547 | 4.475 | 0.167
5 | 0.911 | 0.981 |  | 0.096 | 0.474 | 4.225 | 0.157
7 | 0.926 | 0.984 |  | 0.085 | 0.410 | 3.727 | 0.145
2D anchor BBox + average distance | 3 | 0.914 | 0.982 |  | 0.099 | 0.491 | 4.196 | 0.159
5 | 0.923 | 0.984 |  | 0.092 | 0.437 | 3.911 | 0.152
Ours (normal anchor distance) | 3 | 0.949 | 0.988 |  | 0.094 | 0.344 | 3.401 | 0.144
5 | 0.968 | 0.990 |  | 0.084 | 0.230 | 2.527 | 0.131
9 | 0.971 | 0.989 |  | 0.076 | 0.155 | 1.770 | 0.124
Ours (log-scale anchor distance) | 3 | 0.931 | 0.987 |  | 0.098 | 0.401 | 3.903 | 0.149
5 | 0.957 | 0.989 |  | 0.084 | 0.281 | 3.182 | 0.133
9 | 0.972 | 0.990 |  | 0.073 | 0.150 | 1.915 | 0.117
Ours (squared anchor distance) | 3 | 0.952 | 0.988 |  | 0.092 | 0.313 | 2.936 | 0.142
5 | 0.962 | 0.989 |  | 0.084 | 0.220 | 2.080 | 0.134
9 | 0.970 | 0.989 |  | 0.079 | 0.165 | 1.719 | 0.127

###  V-B Depth Estimation and Anchor Distance

We represent the metric evaluations of $z$ coordinate distance (depth)
estimation of the object in Table II. We follow the definitions of the metrics
as in [28]. Compared to the methods based on RPN [28, 30], our method achieves
a better performance in RMSE at a substantially higher frame rate. Using depth
estimation directly for object detection showed degraded performance compared
to other methods as it hardly consider the overlapped or occluded states of
objects.

In order to validate the anchor distance, we also evaluate the methods of 2D
anchor BBox with and without average distance. As the number of predictors
increases, the burden of one predictor decreases and the network can achieve
the robust estimations. In other words, the more number of predictors there
are, the more accurate inference is achieved. This trend is more pronounced
when anchor distance is applied, and in this case the network shows
significantly improved performance than when the predictors simply focus on 2D
BBox.

The proposed method has the highest performance when the squared format is
used, and shows the lowest performance when log-scale is used. We found that
the squared distance format covers the largest range as shown in Table III.
With the large range of anchor distances, the network can handle the objects
distributed in various ranges efficiently.

TABLE III: Anchor Distance of Different Distance Format |  |  |  |  |  | (unit : $m$)
---|---|---|---|---|---|---
order of predictors | 1 | 2 | 3 | 4 | 5
anchor BBox(avr dist) | 7.73 | 13.59 | 23.83 | 33.81 | 52.52
anchor distance | normal | 11.20 | 23.18 | 35.30 | 49.50 | 66.21
log-scale | 7.60 | 15.17 | 24.78 | 37.59 | 57.51
squared | 17.49 | 32.86 | 45.27 | 57.41 | 71.52

Meawhile, in the case of 2D BBox with average distance, prior distance is the
most similar to that of log-scale format. Reversely, log-scale distance
grouping shows the most similar average 2D BBox to that of 2D BBox grouping;
we display the 2D BBoxes of each grouping in Table IV.

TABLE IV: Comparison of 2D BBox Dimensions for KITTI dataset |  |  |  |  |  |  |  |  |  |  |  |  | (unit : pixel)
---|---|---|---|---|---|---|---|---|---|---|---|---|---
# of predictors |  | 2 |  | 3 |  | 5
orders |  | 1 | 2 |  | 1 | 2 | 3 |  | 1 | 2 | 3 | 4 | 5
anchor box(IoU) |  | 151/268 | 51/86 |  | 164/296 | 83/144 | 39/64 |  | 175/321 | 108/183 | 57/104 | 37/61 | 24/35
anchor dist(log-scale) |  | 139/226 | 37/67 |  | 156/246 | 59/103 | 29/53 |  | 181/273 | 104/178 | 57/99 | 36/67 | 23/41
anchor dist(normal) |  | 133/219 | 30/55 |  | 141/229 | 42/75 | 24/42 |  | 156/246 | 63/111 | 38/69 | 27/49 | 19/32
anchor dist(squared) |  | 129/215 | 24/43 |  | 134/220 | 33/60 | 21/36 |  | 140/227 | 42/75 | 29/54 | 22/39 | 17/29

Therefore, we can conclude that the distance in the log-scale format is mostly
related to the 2D BBox, but using the log-scale distance or 2D BBox is not
sufficient to achieve the best performance as the 2D BBox is not directly
proportional to the distance. In other words, using log-scale distance or 2D
BBox for grouping in order to obtain prior distance is not effective. For 3D
localization tasks it is better concentrating not on 2D projection, but on
distance itself for defining anchors of 3D location.

###  V-C Estimation Error and Execution Time

The proposed method shows a significantly better performance on the RMSE error
than that of others, but in the Abs Rel metric the method does not outperform
the existing method using complex structures such as ResNet or Mask-RCNN. In
other words, although our proposed method is slightly inferior to the previous
methods for objects that are close to the origin, it is more robust when the
objects are far from the origin. The structure of the proposed network is
simpler than others, yet our approach using anchor distance and training
scheme consistently generally achieves more accurate estimation regardless of
the distance of objects.

We also present the frame rate in terms of FPS for each method. In [28], they
assume that the 2D BBox regression is given in advance, so we approximately
record its FPS by assuming that YOLOv2 is used. In [30], the FPS of their
method is under 7, as the baseline is MaskRCNN [20] that shows 7 FPS and the
method deploys additional network structure for distance estimation. The
proposed network, which is based on YOLOv2 [21], however, only takes about
0.03 seconds and shows under 35 FPS for the entire estimation process. This
execution time is irrelevant to the number of predictors, as adding one
predictor is equivalent to adding $\left(4+1\right)$ convolutional filters
that only have $\left(3\times 3\right)$ parameters per filter. The method with
3D projection takes another 0.04 seconds after estimating 2D BBox, 3d BBox,
and orientation which takes 0.03 seconds.

Figure 3:  The RMSE and FPS of the methods are shown. Since our method is
based on a real-time detector, it demonstrates high FPS.Adopting the anchor
distance as prior information compensates the simple network structure,
achieving faster and better performance compared to others.

We display the relation between RMSE error and FPS of various methods in Fig.
3.

For qualitative analysis, we display the visualization examples in Fig. 4
represented in a bird-eye view. For 3D proj, we exploit the 2D BBox regression
results from the network with $k=5$. The network for 3D proj shares the same
structure of our proposed method. Also, it estimates orientations and
dimensions of 2D and 3D BBox which are used for calculating distance. Here, we
use a squared format for the proposed method. In 3D proj, the error of the
distance estimation increases since the size of the 2D BBox becomes similar as
the objects are located far away. In the proposed method, the higher the
number of predictors $k$ is, the more accurate the estimated results are.

Figure 4:  Examples of the visualizations in a bird-eye view. We compare the
results of using 3D projection and the results of ours with $k=2,3,5$. Since
the method using 3D projection highly depends on the 2D bounding box and
orientation, it shows incorrect results for the objects located far away or
occluded. The proposed method shows better performance with multiple
predictors, as a number of anchor distances allows the network to estimate the
results with low non-linear complexity.

##  VI Conclusion

We propose a multi-object distance estimation method using anchor distance.
Conventional methods based on multi-object detection train their predictors
based on 2D Bounding Box (BBox). Other techniques for multi-object distance
estimation rely heavily on complex structure for sophisticated distance
estimation. The proposed method can achieve robust estimation and real-time
performance as the method selects and trains the predictors of the network
based on the distance of objects. To build a prior, we define the anchor
distance by clustering the objects with their distance in various formats such
as squared or log-scaled. The anchor distance gives predictors a strong prior
knowledge of distance. Predictors are dedicated to learning objects in a
specific distance range according to their anchor distances. Because the
proposed method trains the network based on distance, it is able to achieve
more accurate estimations. Traditional methods of directly calculating the
distance by projecting 3D BBox to 2D BBox require a large amount of
computation, so an increased number of objects tend to decline the speed of
estimation during execution. Using anchor distance as a prior the proposed
approach develops a computationally concise network and performs single-shot,
real-time, multi-object detection even for an arbitrarily large number of
objects.

## Acknowledgement

We would like to thank Jihoon Moon, who gives us intuitive advice. This work
is in part supported by the Air Force Office of Scientific Research under
award number FA2386-17-1-4660.

## References

  * [1] R. F. Salas-Moreno, R. A. Newcombe, H. Strasdat, P. H. Kelly, and A. J. Davison, “Slam++: Simultaneous localisation and mapping at the level of objects,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2013, pp. 1352–1359.
  * [2] P. Parkhiya, R. Khawad, J. K. Murthy, B. Bhowmick, and K. M. Krishna, “Constructing category-specific models for monocular object-slam,” _arXiv preprint arXiv:1802.09292_ , 2018.
  * [3] S. Yang and S. Scherer, “Cubeslam: Monocular 3-d object slam,” _IEEE Transactions on Robotics_ , vol. 35, no. 4, pp. 925–938, 2019.
  * [4] S. L. Bowman, N. Atanasov, K. Daniilidis, and G. J. Pappas, “Probabilistic data association for semantic slam,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2017, pp. 1722–1729.
  * [5] D. F. Kevin Doherty and J. Leonard, “Multimodal semantic slam with probabilistic data association,” in _2019 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2019.
  * [6] J. Wu, Y. Wang, T. Xue, X. Sun, B. Freeman, and J. Tenenbaum, “Marrnet: 3d shape reconstruction via 2.5 d sketches,” in _Advances in neural information processing systems_ , 2017, pp. 540–550.
  * [7] H. Yu and B. H. Lee, “A variational feature encoding method of 3d object for probabilistic semantic slam,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 3605–3612.
  * [8] H. Yu, J. Moon, and B. Lee, “A variational observation model of 3d object for probabilistic semantic slam,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 5866–5872.
  * [9] Y. Xiang, W. Choi, Y. Lin, and S. Savarese, “Data-driven 3d voxel patterns for object category recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 1903–1911.
  * [10] A. Mousavian, D. Anguelov, J. Flynn, and J. Košecká, “3d bounding box estimation using deep learning and geometry,” in _Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on_. IEEE, 2017, pp. 5632–5640.
  * [11] D. Proklova, D. Kaiser, and M. V. Peelen, “Disentangling representations of object shape and object category in human visual cortex: The animate–inanimate distinction,” _Journal of cognitive neuroscience_ , vol. 28, no. 5, pp. 680–692, 2016.
  * [12] A. Mousavian, D. Anguelov, J. Flynn, and J. Kosecka, “3d bounding box estimation using deep learning and geometry,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 7074–7082.
  * [13] D. P. Frost, O. Kähler, and D. W. Murray, “Object-aware bundle adjustment for correcting monocular scale drift,” in _2016 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2016, pp. 4770–4776.
  * [14] D. Frost, V. Prisacariu, and D. Murray, “Recovering stable scale in monocular slam using object-supplemented bundle adjustment,” _IEEE Transactions on Robotics_ , vol. 34, no. 3, pp. 736–747, 2018.
  * [15] E. Sucar and J.-B. Hayet, “Bayesian scale estimation for monocular slam based on generic object detection for correcting scale drift,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 1–7.
  * [16] H. Fu, M. Gong, C. Wang, K. Batmanghelich, and D. Tao, “Deep ordinal regression network for monocular depth estimation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 2002–2011.
  * [17] J. M. Facil, B. Ummenhofer, H. Zhou, L. Montesano, T. Brox, and J. Civera, “Cam-convs: camera-aware multi-scale convolutions for single-view depth,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2019, pp. 11 826–11 835.
  * [18] D. Wofk, F. Ma, T.-J. Yang, S. Karaman, and V. Sze, “Fastdepth: Fast monocular depth estimation on embedded systems,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 6101–6108.
  * [19] F. Zhong, S. Wang, Z. Zhang, and Y. Wang, “Detect-slam: Making object detection and slam mutually beneficial,” in _2018 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2018, pp. 1001–1010.
  * [20] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2961–2969.
  * [21] J. Redmon and A. Farhadi, “Yolo9000: better, faster, stronger,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 7263–7271.
  * [22] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in _European conference on computer vision_. Springer, 2016, pp. 21–37.
  * [23] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in _Advances in neural information processing systems_ , 2015, pp. 91–99.
  * [24] B. Tekin, S. N. Sinha, and P. Fua, “Real-time seamless single shot 6d object pose prediction,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 292–301.
  * [25] Y. Zhou and O. Tuzel, “Voxelnet: End-to-end learning for point cloud based 3d object detection,” _arXiv preprint arXiv:1711.06396_ , 2017.
  * [26] A. Kundu, Y. Li, and J. M. Rehg, “3d-rcnn: Instance-level 3d object reconstruction via render-and-compare,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 3559–3568.
  * [27] Y. Huo, H. Yi, Z. Wang, J. Shi, Z. Lu, P. Luo, _et al._ , “Learning depth-guided convolutions for monocular 3d object detection,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_. IEEE, 2020, pp. 4306–4315.
  * [28] J. Zhu and Y. Fang, “Learning object-specific distance from a monocular image,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 3839–3848.
  * [29] A. Simonelli, S. R. Bulo, L. Porzi, M. López-Antequera, and P. Kontschieder, “Disentangling monocular 3d object detection,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 1991–1999.
  * [30] Y. Zhang, Y. Li, M. Zhao, and X. Yu, “A regional regression network for monocular object distance estimation,” in _2020 IEEE International Conference on Multimedia & Expo Workshops (ICMEW)_. IEEE, 2020, pp. 1–6.
  * [31] A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, “Yolov4: Optimal speed and accuracy of object detection,” _arXiv preprint arXiv:2004.10934_ , 2020.
  * [32] G. Brazil and X. Liu, “M3d-rpn: Monocular 3d region proposal network for object detection,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 9287–9296.
  * [33] A. A. Soltani, H. Huang, J. Wu, T. D. Kulkarni, and J. B. Tenenbaum, “Synthesizing 3d shapes via modeling multi-view depth maps and silhouettes with deep generative networks,” in _The IEEE conference on computer vision and pattern recognition (CVPR)_ , vol. 3, 2017, p. 4.
  * [34] J. K. Pontes, C. Kong, S. Sridharan, S. Lucey, A. Eriksson, and C. Fookes, “Image2mesh: A learning framework for single image 3d reconstruction,” _arXiv preprint arXiv:1711.10669_ , 2017.
  * [35] Z. Liu, Z. Wu, and R. Tóth, “Smoke: Single-stage monocular 3d object detection via keypoint estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020, pp. 996–997.
  * [36] M. Simony, S. Milzy, K. Amendey, and H.-M. Gross, “Complex-yolo: An euler-region-proposal for real-time 3d object detection on point clouds,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 0–0.
  * [37] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in _Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on_. IEEE, 2012, pp. 3354–3361.
  * [38] Z. Zheng, P. Wang, W. Liu, J. Li, R. Ye, and D. Ren, “Distance-iou loss: Faster and better learning for bounding box regression,” 2019.
  * [39] F. Gökçe, G. Üçoluk, E. Şahin, and S. Kalkan, “Vision-based detection and distance estimation of micro unmanned aerial vehicles,” _Sensors_ , vol. 15, no. 9, pp. 23 805–23 846, 2015.
  * [40] S. Tuohy, D. O’Cualain, E. Jones, and M. Glavin, “Distance determination for an automobile environment using inverse perspective mapping in opencv,” 2010\.