# MBDF-Net: Multi-Branch Deep Fusion Network for 3D Object Detection

Xun Tan1, Xingyu Chen1, Guowei Zhang1, Jishiyu Ding2, Xuguang Lan1, ✉
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University1,
the Second Academy of CASIC2China<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
(2018)

###### Abstract.

Point clouds and images could provide complementary information when
representing 3D objects. Fusing the two kinds of data usually helps to improve
the detection results. However, it is challenging to fuse the two data
modalities, due to their different characteristics and the interference from
the non-interest areas. To solve this problem, we propose a Multi-Branch Deep
Fusion Network (MBDF-Net) for 3D object detection. The proposed detector has
two stages. In the first stage, our multi-branch feature extraction network
utilizes Adaptive Attention Fusion (AAF) modules to produce cross-modal fusion
features from single-modal semantic features. In the second stage, we use a
region of interest (RoI) -pooled fusion module to generate enhanced local
features for refinement. A novel attention-based hybrid sampling strategy is
also proposed for selecting key points in the downsampling process. We
evaluate our approach on two widely used benchmark datasets including KITTI
(Geiger et al., 2013) and SUN-RGBD (Song et al., 2015). The experimental
results demonstrate the advantages of our method over state-of-the-art
approaches.

3D Object Detection, Multi-modal Fusion, Point Cloud Downsampling

✉ Corresponding author

††copyright: acmcopyright††journalyear: 2018††ccs: Computing methodologies
Vision for robotics††ccs: Computing methodologies Scene understanding

##  1\. Introduction

Object detection is an important topic in the computer vision community which
has been widely used in many realistic scenarios. With the development of deep
learning technologies, a variety of image-based methods (Ren et al., 2015; Liu
et al., 2016; Redmon et al., 2016; Tian et al., 2019) have been proposed,
which makes great progress in 2D object detection. However, 2D detection
methods can seldom provide accurate results in 3D space due to the lack of
depth information. That is problematic in many tasks such as automatic driving
and intelligent robot. Therefore, 3D object detection (Guo et al., 2020)
attracts more attention in recent years.

Point cloud is naturally suitable for 3D object detection since it directly
provides 3D information and has strong robustness. However, point cloud is
usually sparse, disordered and also lacks texture information. It is easy to
be disturbed by objects with similar shapes, which limits the accuracy of
detection. Images can provide color and texture information but lack depth
information. Directly using images for 3D object detection is also bothered by
many problems such as occlusion and deformation, which makes it extremely
vulnerable to the influence of the environment.

Obviously, combining the two kinds of data and making good use of the
advantages of each one could greatly help to improve the detection results.
Therefore, many camera and LiDAR fusion-based detection methods have been
proposed, which can be roughly grouped into two lines, including result-level
fusion methods and feature-level fusion methods. The result-level fusion
methods (Qi et al., 2018; Wang and Jia, 2019; Pang et al., 2020) handle each
modality independently and then integrate the corresponding results. But these
methods do not fully utilize the complementary information of the two kinds of
data. The feature-level fusion methods(Ku et al., 2018a; Huang et al., 2020;
Xie et al., 2019; Yoo et al., 2020) generally adopt a two-branch structure for
fusing image features and the point cloud features. However, it is difficult
for existing methods to integrate multi-modal data well. First, due to the
differences in the characteristics of different modalities, simply fusing them
usually leads to unsatisfactory detection results. Second, directly fusing the
two kinds of data involves processing the non-interested regions, which may
have a negative impact on detecting objects.

Figure 1. Illustration of the fusion method for our cross-modal fusion
feature. We present the workflow in AAF module.

To solve this problem, we propose a Multi-Branch Deep Fusion Network (MBDF-
Net) for 3D object detection. Different from previous two-branch based
methods, the proposed model consists of three branches, including an image
branch, a LiDAR branch, and a fusion branch. The image branch and the LiDAR
branch are used for extracting semantic information of the corresponding data
modality. Specially, we utilize several Adaptive Attention Fusion (AAF)
modules (see Figure 1) to generate cross-modal fusion features for the fusion
branch. AAF modules integrate the information of the two modalities and then
evaluate each of them. In this way, the network can choose more essential
information from two modality data and suppress the interference from non-
interest areas. Once the 3D proposals are generated based on the cross-modal
fusion feature, we utilize a region of interest (RoI)-pooled fusion module to
produce enhanced local features for them. This module will enlarge the
proposals and aggregate additional information, including the raw point
coordinates, specific single-modal features, and cross-modal fusion features.
The output local features can lead to more accurate refining results.

Figure 2. Overview of the proposed MBDF-net structure. First, we extract
semantic information from each modality and fuse them to generate cross-modal
fusion features by AAF modules. After that, we use the RoI-pooled fusion
module to produce enhanced local features for proposal refinement. In SA
layers we replace FPS with hybrid sampling.

In addition, we notice that most current methods (Qi et al., 2017; Shi et al.,
2019; Huang et al., 2020) need to downsample points and then use the sampled
points as the centers for extracting range features. In these methods,
farthest point sampling (FPS) (Qi et al., 2017) is often used. It considers
the spatial information to ensure the uniformity of sampling. In this case,
all regions have the same importance. However, in the object detection task,
we believe it is necessary to pay more attention to the regional features of
the objects. Moreover, if the selected point is close to the center of the
object, extracting the range information of this point will help us to get
more accurate location results. In order to select points that balance
uniformity and representation in the downsampling procedures, we proposed a
novel hybrid sampling method. First, we use the FPS to sample more points than
the required number. Then, we utilize the point cloud attention map in the AAF
modules to guide us to choose more critical points from them. The proposed
sampling method not only considers the information of the point itself but
also integrates the information of the point and image.

Our contributions can be summarized as follows:

  * •

We propose a MBDF-Net for 3D object detection. In the first stage, we utilize
AAF modules to produce cross-modal fusion features. In the second stage, we
generate enhanced local features by an RoI-pooled fusion module for better
proposal refinement.

  * •

A novel hybrid sampling method is applied in point downsampling, which can not
only guarantee the uniformity of sampling points but also choose more
representative points.

  * •

We evaluate our methods on both the outdoor KITTI (Geiger et al., 2013)
dataset and the indoor SUNRGBD (Song et al., 2015) dataset. We achieve state-
of-the-art performance.

##  2\. Related work

###  2.1. 3D object detection based on LiDAR

Currently, the LiDAR-based detection algorithms can be roughly grouped into
two lines. The first kind of methods (Chen et al., 2017; Yan et al., 2018;
Kuang et al., 2020; Zhu et al., 2019; Shi et al., 2020c; Ku et al., 2018b;
Yang et al., 2018) usually use a convolutional neural network to extract
features of projected of point clouds. For example, MV3D (Chen et al., 2017)
projects point clouds to bird’s eye view (BEV) perspective and generates
bounding boxes with 2D convolution. MV3D (Chen et al., 2017), SECOND (Yan et
al., 2018), voxel-net (Kuang et al., 2020) builds the voxels of the point
clouds and then processes it with 3D convolution. Although convolution has
better perception ability. It needs the pre-processing procedure and takes
more time during calculation. Another kind of methods such as PointRCNN(Shi et
al., 2019), STD(Yang et al., 2019), 3D-SSD(Yang et al., 2020), voxel-FPN(Kuang
et al., 2020) directly deal with the original point clouds. PointRCNN (Shi et
al., 2019) directly uses PointNet++ (Qi et al., 2017) as the feature
extraction network to generate 3D recommendation boxes by dividing the
foreground points of the point clouds. Recently, the two methods are combined
in PV-RCNN (Shi et al., 2020a). The point-based method is used to process the
whole, and convolution is used to further extract the features in the
proposal.

###  2.2. 3D object detection based on multiple sensors

There are also a number of methods focusing on fusing the information of
different sensors for 3D object detection (Qi et al., 2018; Xu et al., 2018;
Liang et al., 2020; Chen et al., 2017; Ku et al., 2018a; Liang et al., 2018;
Huang et al., 2020; Yoo et al., 2020; Pang et al., 2020). Previous methods
such as Frustum PointNet (Qi et al., 2018) first detect the targets in the
image and then detect the targets in the corresponding point clouds region. In
these methods, the detection performance heavily depends on the accuracy of 2D
detection results. CLOCs (Pang et al., 2020) is proposed to use pre-trained 2D
and 3D detectors directly in the form of late-fusion, making the proposals in
different modalities connected, but not integrating the features. Some methods
such as MV3D (Chen et al., 2017), AVOD (Ku et al., 2018a), and CONTFUSE (Liang
et al., 2018) transform the disordered point cloud into a structured point
cloud, then fuse the features of multiple perspectives with the images.
However, when building multiple perspectives, many resize and crop operations
are involved, which has a risk of losing the spatial information. EPNET (Huang
et al., 2020) utilizes images to guide the feature fusion, which does not
require additional annotation information. 3D-CVF (Yoo et al., 2020)
aggregates the features of multi-view images and uses an automatic calibration
module to reduce errors caused by projection. Currently, most of the feature
fusion methods use a structure of a two-branch network. Compared with our
multi-branch structure, we create intermediate features to integrate useful
information.

##  3\. Method

The proposed MBDF-Net aims to improve the effect of multi-modal fusion. It is
a two-stage object detector. In stage-1, the network extracts key information
from each data modality and fuses them through AAF modules for generating 3D
RoI proposals. Then in stage-2, we adopt an RoI-pooled fusion module to
generate enhanced local features for further refining the generated proposals.
The whole framework of MBDF-Net is presented in Figure 2.

###  3.1. Multi-Branch Feature Extraction Network

Our feature extraction network consists of three branches, including an Image
branch, a LiDAR branch, and a fusion branch.

Figure 3. Illustration of the AAF module. We first use projection to build the
point-to-pixel relationship between image and point. So the transformed image
feature can concatenate with the point feature in the same space. Then we
concatenate them and feed them to the respective convolution layer followed by
a sigmoid function. Finally, fused feature is generated by encoding all branch
features. $\bigoplus$ denotes concatenation operation and $\otimes$ denotes
multiply operation.

Image Branch The image branch is used for extracting features from images. It
consists of four convolution blocks. Each convolution block is composed of a
convolution layer with kernel size $3\times 3$, a Batch Normalization layer,
and a ReLU activation layer. The size of intermediate features produced by
each block is reduced by two times to expand the receptive field. To use the
detailed spatial information in low-level features, we use deconvolution to
restore the feature of each block to the original image size and then
concatenate them. We denote the final image feature as $F_{image}$.

LiDAR Branch The LiDAR branch is used for extracting features for point
clouds. It utilizes the PointNet++ (Qi et al., 2017) with multi-scale grouping
as the backbone network to learn sophisticated and compact point-wise
features. It contains four Set Abstraction (SA) layers and four Feature
Propagation (FP) layers. Each SA layer first downsamples the point clouds and
then extracts the local features. The FP layers are used to upsample the
features to recover the original size of the point cloud. We denote the final
point feature as $F_{point}$.

Fusion Branch The fusion branch is used to extracting cross-modal fusion
features and produce the initial 3D bounding box. Its input is generated by
integrating the output features of the first convolution block and first SA
layer from the image branch and the LiDAR branch respectively. We remove the
first SA layer of PointNet++ (Qi et al., 2017) and use the rest as its
backbone network. As shown in Figure 2, before feeding to each SA layer, we
use AAF module to integrate the key information of the point and image
features with the current cross-modal fusion feature. Through this method, the
generated fused features contain rich point-wise semantic information. In the
end, we utilize AAF module again to fuse $F_{image}$, $F_{point}$, and the
output of this branch to get the final fused feature $F_{fused}$ for 3D
proposal generation.

Adaptive Attention Fusion Module In order to capture the key information of
each data modality, we design an Adaptive Attention Fusion (AAF) module. The
structure is shown in Figure 3.

To fuse the two different views of data, we first use a projection method to
establish the relationship between the points and the image pixels.
Specifically, we multiply each point coordinate $p(x,y,z)$ with a projection
matrix $M$ to generate the corresponding image coordinate
$p^{\prime}(x^{\prime},y^{\prime})$. $M$ is a inner parameters of the camera.
Then we calculate the corresponding features of the point on the image by
using bi-linear interpolation, as the projected points could fall between
adjacent pixels. Finally, we produce the transformed image feature by
assigning the features.

It is difficult to fuse these two data because both are disturbed by non-
interest areas. So we utilize attention mechanism to suppress the interference
and extract essential information for fusion. We use the following formulas to
generate the new fused feature,

(1) $F_{attention-I}=\sigma\left(Conv_{I}\left(F_{image}\oplus
F_{point}\right)\right),$ (2)
$F_{attention-P}=\sigma\left(Conv_{P}\left(F_{image}\oplus
F_{point}\right)\right),$ (3) $\displaystyle
F_{fused}=Conv\big{(}\left(F_{image}\times F_{attention-I}\right)$
$\displaystyle\oplus\left(F_{point}\times F_{attention-P}\right)\oplus
F^{\prime}_{fused}\big{)},$

where $\bigoplus$ represents the concatenation operation, $\sigma$ represents
the sigmoid function. $F_{attention-I}$ and $F_{attention-P}$ represent the
attention maps that indicate the relative importance of images and points.

Figure 4. Illustration of the hybrid sample strategy. We first use FPS to
select $\lambda n$ from $N$ points. Then we choose $n$ points by utilizing
attention to guide.

###  3.2. Hybrid Sampling Strategy

In the SA layers, we need to downsample points to expand the receptive field.
The point cloud is disordered in space. To ensure the uniformity of sampling,
the FPS is often used for covering the whole space as much as possible.
However, The FPS only considers the relative distance between points without
utilizing the rich information in the semantic features. Using attribute
features is another way to downsample points. It also has some problems.
First, the selected points usually concentrate on the region of interest,
which results in a serious aggregation phenomenon. Second, paying too much
attention to the foreground point will decrease the ability to distinguish
background points. Therefore, we present a hybrid sampling strategy (see
Figure 4) which takes both distance and attributes into consideration.

Supposing that we need to sample $n$ points from $N$ points. First of all, we
use the FPS to select $\lambda n(1.0\leq\lambda\leq\frac{N}{n})$ points to
ensure the uniformity of our sampling distribution. Then we get the attention
map of the point $F_{attention-P}$ from AAF module. This map is produced under
the guidance of point cloud and image features and represents important
semantic information for each point. We get these $\lambda n$ attention scores
from the $F_{attention-P}$. Next, these scores are sorted from large to small,
and the $n$ points with higher scores are selected as the downsampling points.
In this way, we select the key points with better attributes from the evenly
distributed points and takes multi-modal information into account.

###  3.3. RoI-Pooled Fusion Refinement Network

Figure 5. Illustration of the RoI-pooled fusion structure. First, we enlarge
the size of the RoI. Then we concatenate additional information in this area
to produce the enhanced local feature. So abundant information is provided to
the follow-up network. The green box is the original range size, and the red
box is the expanded range size.

In the first stage, the proposal generation network gets the initial
regression results and corresponding scores. We use the non-maximum
suppression (NMS) procedure to retain the proposals with high scores. In the
second stage, to produce more precise results, firstly we use the RoI-pooled
fusion module (see Figure 5) to enhance each proposal.

We Keep the coordinates unchanged and expand the width, the length, and the
height of the generated proposal, that is, from $(W,l,H)$ to $(W+a,L+a,H+a)$.
$a$ is the parameter for enlarging the size of the proposals. Expanding the
proposals could provide additional information for increasing the robustness.
Then we aggregate the raw point coordinates, the specific single-modal
features, cross-modal fusion features in each enlarged proposal. In this way,
we can encode local spatial and semantic information to produce more rich
local features for proposals. Finally, we feed these features to the
refinement Network for refining the 3D bounding box.

###  3.4.  Training Objective

Our MBDF-Net is trained via a two-stage process. We use RPN loss to train the
first stage and RCNN loss for the second stage. Each stage adopts a similar
optimizing goal, includes a classification loss, and a regression loss. The
overall objective is defined as:

(4) $\displaystyle L_{total}=$ $\displaystyle\left(L_{RPN-cls}+L_{RPN-
reg}\right)+$ $\displaystyle\left(L_{RCNN-cls}+L_{RCNN-reg}\right)$

where $L_{RPN-cls}$, $L_{RPN-reg}$ indicate the RPN classification loss and
regression loss respectively. The RCNN loss is similar.

To solve the problem that the proportion of positive and negative samples is
too large, we use focal loss (Lin et al., 2017) as the loss of classification.
$L_{cls}$ is given by Eq.(5),

(5) $L_{cls}=-\alpha\left(1-c_{t}\right)^{\gamma}\log c_{t}$

where $c_{t}$ means the probability of the point belongs to the ground truth
category, and we set $\alpha$ = 0.25 and $\gamma$ = 2.0.

Table 1. Performance comparison on the KITTI test set. The 3D and BEV results are evaluated by the mean Average Precision with 40 recall positions. Method | Reference | Modality | 3D Detection | Bird’s Eye View
---|---|---|---|---
Easy | Moderate | Hard | 3D mAP | Easy | Moderate | Hard | BEV mAP
LiDAR-only: |  |
PointPillars(Lang et al., 2019) | CVPR 2019 | Voxel | 82.58 | 74.31 | 68.99 | 75.29 | 90.07 | 86.56 | 82.81 | 86.48
PointRCNN(Shi et al., 2019) | CVPR 2019 | Point | 86.96 | 75.64 | 70.70 | 77.77 | 92.13 | 87.39 | 82.72 | 87.41
Point-GNN(Shi et al., 2020b) | CVPR 2020 | Point | 88.33 | 79.47 | 72.29 | 80.03 | 93.11 | 89.17 | 83.90 | 88.73
3D-SSD(Yang et al., 2020) | CVPR 2020 | Point | 88.36 | 79.57 | 74.55 | 80.83 | 92.66 | 89.02 | 85.86 | 89.18
SA-SSD(He et al., 2020) | CVPR 2020 | Voxel+Point | 88.75 | 79.79 | 74.16 | 80.90 | 95.03 | 91.03 | 85.96 | 90.67
Multi-Modality: |  |
F-PointNet(Qi et al., 2018) | CVPR 2018 | RGB+LiDAR | 82.19 | 69.79 | 60.59 | 70.86 | 91.17 | 84.67 | 74.77 | 83.54
UberATG-MMF(Liang et al., 2019) | CVPR 2019 | RGB+LiDAR | 88.40 | 77.43 | 70.22 | 78.68 | 93.67 | 88.21 | 81.99 | 87.96
PointPainting(Vora et al., 2019) | CVPR 2020 | RGB+LiDAR | 82.11 | 71.70 | 67.08 | 73.63 | 92.45 | 88.11 | 83.36 | 87.97
PI-RCNN(Xie et al., 2019) | AAAI 2020 | RGB+LiDAR | 84.37 | 74.82 | 70.03 | 76.41 | 91.44 | 85.81 | 81.00 | 86.08
EPNet(Huang et al., 2020) | ECCV 2020 | RGB+LiDAR | 89.91 | 79.28 | 74.59 | 81.23 | 94.22 | 88.47 | 83.69 | 88.79
3D-CVF(Yoo et al., 2020) | ECCV 2020 | RGB+LiDAR | 89.20 | 80.05 | 73.11 | 80.79 | 93.52 | 89.56 | 82.45 | 88.51
OURS |  | RGB+LiDAR | 90.87 | 80.00 | 75.04 | 81.97 | 95.36 | 89.18 | 84.24 | 89.59

For the regression, we need to regress center point, size, and orientation
$(x,y,z,l,h,w,\theta)$. And for the relatively small range for the $y,h,w,l,$
we use a smooth L1 loss (Ren et al., 2015). For the $x,z,\theta$, we adopt a
bin-based regression loss (Qi et al., 2018; Shi et al., 2019; Huang et al.,
2020). First, it splits several bins for each point area, then it will predict
which bin the center point will fall in. Last it regresses the residual offset
within the bin. Besides, we utilize IoU loss as a regularization term. The
overall regression loss $L_{reg}$ can be formulated as

(6) $\displaystyle L_{reg}=$ $\displaystyle\sum_{u\in
x,z,\theta}E\left(b_{u},\hat{b}_{u}\right)+\sum_{u\in
x,y,z,h,w,l,\theta}S\left(r_{u},\hat{r}_{u}\right)$ $\displaystyle-
ln\left(\frac{Area\left(P\cap G\right)}{Area\left(P\cup G\right)}\right)$

where $E$ denotes the cross-entropy loss and $S$ means the smooth L1 loss.
$\hat{b}_{u}$ and $\hat{r}_{u}$ indicate the ground truth bins and offsets
respectively. $P$ and $G$ are the predicted bounding box and ground truth.

##  4\. Experiments

###  4.1. Implementation Details

The proposed network takes both images and point cloud as input. The original
resolution of images is $1280\times 384$. We use four convolution blocks with
stride 2 to downsample the image. Then, we employ four parallel transposed
convolution with stride 2, 4, 8, 16 to recover the resolution from features in
different scales, which is the same with EPNet (Huang et al., 2020). For each
scene, we limit the input range of point cloud to (0, 70.4), (-40, 40), (-3,
1) in X, Y and Z directions respectively. We drop those points that are
invisible in the image view. Then we subsample 16384 points from the raw point
cloud as input. We follow the similar structure of PointNet++ (Qi et al.,
2017), which contains four SA layers. We replace the sampling method (FPS)
with our hybrid sampling method. We select $\lambda=1.4$ and demonstrate the
reasons in Section 4.4. The number of the point cloud sampled by the SA layers
is $4096$, $1024$, $256$, and $64$ respectively. The original size of the
point cloud is finally recovered by four FP layers. The settings of the fusion
branch are the same as those of the point cloud branch. To remove the
redundant proposals, we select the top 8000 boxes and set the non-maximum
suppression (NMS) threshold to 0.8. Then we obtain 64 positive candidate boxes
to refine.

For the refinement network, we enlarge the size of each proposal by 0.2. Then,
we sample 512 points from each proposal as input. For the proposals with less
than 512 points, we supplement them with zeros. Next, we concatenate the raw
point coordinates, each specific single-modal feature, and the cross-modal
fusion feature for each proposal. The generated features are fed into three SA
layers with group size (128, 32, 1) to generate a feature vector for object
confidence classification and location regression.

Table 2. 3D object detection results on SUN RGB-D V1 val set. We show the mean of average precision (mAP) across 10 classes with a 3D IoU threshold of 0.25. P and I represent the point cloud and the camera image. Method | Modality | bathtub | bed | bookshelf | chair | desk | dresser | nightstand | sofa | table | toilet | 3D mAP
---|---|---|---|---|---|---|---|---|---|---|---|---
DSS(Song and Xiao, 2016) | P+I | 44.2 | 78.8 | 11.9 | 61.2 | 20.5 | 6.4 | 15.4 | 53.5 | 50.3 | 78.9 | 42.1
2d-driven(Lahoud and Ghanem, 2017) | P+I | 43.5 | 64.5 | 31.4 | 48.3 | 27.9 | 25.9 | 41.9 | 50.4 | 37.0 | 80.4 | 45.1
COG(Ren and Sudderth, 2016) | P+I | 58.3 | 63.7 | 31.8 | 62.2 | 45.2 | 15.5 | 27.4 | 51.0 | 51.3 | 70.1 | 47.6
PointFusion(Xu et al., 2018) | P+I | 37.3 | 68.6 | 37.7 | 55.1 | 17.2 | 24.0 | 32.2 | 53.8 | 31.0 | 83.8 | 44.1
F-PointNet(Qi et al., 2018) | P+I | 43.3 | 81.1 | 33.3 | 64.2 | 24.7 | 32.0 | 58.1 | 61.1 | 51.1 | 90.9 | 54.0
VoteNet(Qi et al., 2019) | P | 74.4 | 83.0 | 28.8 | 75.3 | 22.0 | 29.8 | 62.2 | 64.0 | 47.3 | 90.1 | 57.7
EPN(Ahmed, 2020) | P+I | 79.4 | 88.2 | 32.1 | 17.0 | 37.4 | 53.7 | 50.0 | 65.3 | 53.3 | 95.8 | 57.2
Ours | P+I | 81.5 | 84.7 | 33.0 | 77.3 | 31.2 | 29.0 | 57.7 | 65.6 | 49.9 | 85.5 | 59.5
Table 3. Comparison between the ideas in the proposed method on the KITTI val dataset.  Use Fusion | RoI-Pooled Fusion | Hybrid Sample | Easy | Moderate | Hard | mAP
---|---|---|---|---|---|---
no | no | no | 91.20 | 79.47 | 77.27 | 82.65
yes | no | no | 92.01 | 82.23 | 79.90 | 84.71
yes | yes | no | 92.34 | 82.57 | 80.21 | 85.04
yes | yes | yes | 92.88 | 83.09 | 80.56 | 85.51

We train the network by the Adam (Kingma and Ba, 2014) optimizer. The initial
learning rate, weight decay, and momentum factor are set to 0.002, 0.001, and
0.9 respectively. We use rotation around the vertical $Y$ axis between [-10,
10] degrees, random flipping, and scale transformations with a factor within
[0.95, 1.05] on LiDAR point as data augmentation strategies to prevent over-
fitting.

Table 4. Comparison between the two branch network, three branch network, and three branch network with adaptive attention on KITTI val dataset. Method | Easy | Moderate | Hard | mAP
---|---|---|---|---
Two Branch | 91.25 | 81.69 | 79.48 | 84.14
Three Branch | 91.78 | 82.09 | 79.58 | 84.48
Adaptive Attention | 92.01 | 82.23 | 79.90 | 84.71

###  4.2. KITTI

Figure 6. Illustration of the learned features. (a) and (b) are the single-
modal features. (c) and (d) are the learned attention map. (e) is the final
produced fused feature. We show the raw image in (f).

Dataset KITTI is a standard benchmark dataset for autonomous driving. It
contains 7481 training samples and 7518 test samples. According to the size,
occlusion, and truncation, objects are classified into different difficulty
levels, including Easy, Moderate, and Hard. We follow the general split of
3712 training samples and 3769 validation samples. And we provide the results
both on the validation set and the test set.

Figure 7. Illustration of different designs of networks. Above is the widely
used two-branch structure, Below is the three-branch structure we use.

Metrics The detection result is evaluated using the mean Average Precision
(mAP) with the IoU threshold of 0.7. For the official test benchmark, the mAP
with 40 recall positions is reported. For a fair comparison, we also report
the mAP with 11 recall positions on the validation set.

Main Result We present our results on KITTI benchmark in Table 1. First, we
compare our approach with a single-modal method PointRCNN (Shi et al., 2019)
since both methods use a similar backbone. Compared to (Shi et al., 2019), our
approach gains huge improvements on 3D detection results by $3.91\%$,
$4.36\%$, $4.34\%$, which correspond to the Easy, Moderate, and Hard
difficulty respectively. Then, compared with multi-modal methods, our approach
performs better than state-of-the-art methods MMF (Liang et al., 2019), EPNet
(Huang et al., 2020), and 3D-CVF (Yoo et al., 2020) by $3.29\%$, $0.74\%$,
$1.18\%$ on 3D mAP respectively. These methods use a two-branch structure.
Specially, MMF(Liang et al., 2019) improves 3D detection performance by using
multiple auxiliary tasks such as 2D detection, ground estimation, and depth
completion. 3D-CVF (Yoo et al., 2020) applies an auto-calibrated projection
method to eliminate the error from projecting and utilizes a gated fusion
network to aggregate the camera-LiDAR feature. EPNet (Huang et al., 2020)
integrates different features by adding the processed image information to the
point feature. These results show the superiority of our methods both on
single-modality and multi-modality approaches.

###  4.3. SUN-RGBD

Dataset SUN RGB-D is a single-view RGB-D dataset for 3D scene understanding,
which contains 10335 indoor RGB and depth images. We use the same
training/validation split and report performance on the 10 most common
categories as VoteNet (Qi et al., 2019). We use Average Precision(AP) and the
mean of AP under IoU values of 0.25.

Main Result The evaluation results on SUN-RGBD val set are shown in Table 2.
Our MBDF-Net outperforms all previous methods. For 2D-driven (Lahoud and
Ghanem, 2017) and F-PointNet (Qi et al., 2018) which is a 3D detector that
based on 2D detection on images, we outperforms them by $14.4\%$ and $5.5\%$
mAP respectively. PointFusion (Xu et al., 2018) combines image features and
point features at the last high level. It lacks more accurate position
information in low-level and does not consider the noise from the data. We
outperform it by $15.4\%$. EPN (Ahmed, 2020) is a two-branch detector. Our
model improves by $2.3\%$ in mAP. The results of indoor scenes also prove the
effectiveness of our method.

###  4.4. Ablation Study

To give a deep insight into our approach, we conduct a number of experiments
on the KITTI validation set.

Figure 8. Illustration of the average aggregation degree(AAD) and mAP with
different $\lambda$ threshold.

Analysis of Each Module: We present the result in table 3. First of all,
compared with the results of the pure point cloud, The accuracy of
$AP_{easy}$, $AP_{moderate}$ and $AP_{hard}$ are improved by $0.81\%$,
$2.76\%$ and $2.63\%$ respectively after using the multi-modal information.
Next, when we use the RoI-pooled fusion module, our mAP is improved by
$0.33\%$ due to more abundant information in the second stage. Finally, when
we adopt the hybrid sampling method, we choose more excellent key points,
which greatly improves our accuracy. The accuracy of $AP_{easy}$,
$AP_{moderate}$ and $AP_{hard}$ are improved by $0.54\%$, $0.52\%$ and
$0.45\%$ respectively. These results prove the effectiveness of proposed
modules.

Figure 9. Illustration of the distribution of different sampling methods.
Ground-truth boxes are in green.

Analysis of Fusion Strategies: We present the result in Table 4. Firstly, to
fairly compare with the widely used two-branch structure, we use simplified
two-branch structure and our multi-branch structure (see figure 7). In both
structures, we just use concatenation to combine the different features. Our
multi-branch structure is $0.34\%$ higher than the two-branch structure in
$mAP$. Then, we utilize an adaptive attention mechanism to guide fusion, The
accuracy was improved by $0.33\%$ $mAP$. From this results, we can see that
the proposed multi-branch structure is better than widely used two-branch
structure. Our fusion strategies can effective benefit the detection.

We further visualize the features in Fig 6. (a) and (b) are the point features
and image feature. (c) and (d) show that attention maps focus on the object
areas and suppress the non-interested areas. (e) is the fused feature.

Analysis of Hybrid Sampling Strategy:

First, we define the average aggregation distance ($AAD$). For each point
$P_{i}$, $AAD_{i}$ means the average distance from the three closest points to
the $P_{i}$, which can be formularized as follows:

(7)
$AAD_{i}=\frac{1}{3}\sum_{j=1}^{3}\left(\left(x_{j}-x_{i}\right)^{2}+\left(y_{j}-y_{i}\right)^{2}+\left(z_{j}-z_{i}\right)^{2}\right)$

where $(x,y,z)$ is the central coordinates of the point. We calculate the
average $AAD$ while utilizing our hybrid sampling strategy with different
$\lambda$ in KITTI train set. Specially, when $\lambda$ equals $1.0$, Our
method is equivalent to FPS. We show the result in figure 8. We can observe
that with the increase of $\lambda$ value, the $AAD$ becomes smaller. This is
easy to understand because points will cluster near the target. In other
words, points will be unevenly distributed. In addition, the accuracy shows a
trend of first rising and then falling. It rises from $85.04\%$ to $85.51\%$
and then falling to $84.95\%$. Combining the above phenomena, we can find that
we need to choose the right $\lambda$ for optimal results. Because too small
$\lambda$ only ensures that the points are evenly distributed. Too large
$\lambda$ could lead points to concentrate in the region of interest and
decrease the ability to distinguish background points. In the end, we choose
$\lambda=1.4$ , as it has a medium aggregation value of $1.25$, and optimal
mAP $85.51\%$.

Table 5 shows the Comparison between the results of our network with farthest
point sampling(FPS) and with attention-based hybrid sampling.

Table 5. Comparison between the results of our network with farthest point sampling(FPS) and with attention-based hybrid sampling. We show the results on three benchmarks of the KITTI validation set(Cars). 3D, BEV, and AOS mean 3D Detection, Birds Eye View, and Orientation respectively. Method | mAP @0.5
---|---
3D | BEV | AOS
FPS | 85.04 | 90.97 | 94.79
Attention-based Hybrid Sampling | 85.51 | 92.16 | 94.88

##  5\. Conclusion

In this paper, we present a new camera and LiDAR fusion two-stage framework
MBDF-Net for 3D object detection. In the first stage, the point cloud branch,
and image branch are utilized to obtain high-quality single-modal feature
information. We use the AAF module to capture the key information of single-
modal and produce rich semantic fused features. In the second stage, we use
RoI-pooled fusion module to expand the range of the bounding box and generate
enhanced local feature by providing additional information. In addition, we
propose a novel hybrid sampling method, which can not only ensure the
uniformity of spatial distribution of sampled points but also evaluate the
values of the points themselves for subsequent processing.

##  6\. Acknowledgements

This work was supported in part by NSFC under grant No.621 2500145,
No.62088102, No.61973246, No.91748208, Shaanxi Project under grant
No.2018ZDCXLGY0607, and the program of the Ministry of Education.

## References

  * (1)
  * Ahmed (2020) M. Ahmed. 2020.  Density Based Clustering for 3D Object Detection in Point Clouds. In _Conference on Computer Vision and Pattern Recognition_.
  * Chen et al. (2017) Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. 2017.  Multi-View 3D Object Detection Network for Autonomous Driving. In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Geiger et al. (2013) Andreas Geiger, Philip Lenz, C. Stiller, and R. Urtasun. 2013\.  Vision meets robotics: The KITTI dataset.  _The International Journal of Robotics Research_ 32 (2013), 1231 – 1237.
  * Guo et al. (2020) Yulan Guo, H. Wang, Q. Hu, Hao Liu, Li Liu, and M. Bennamoun. 2020.  Deep Learning for 3D Point Clouds: A Survey.  _IEEE transactions on pattern analysis and machine intelligence_ PP (2020).
  * He et al. (2020) Chenhang He, Hui Zeng, Jianqiang Huang, Xiansheng Hua, and Lei Zhang. 2020.  Structure Aware Single-Stage 3D Object Detection From Point Cloud.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ (2020), 11870–11879.
  * Huang et al. (2020) Tengteng Huang, Zhe Liu, Xiwu Chen, and X. Bai. 2020\.  EPNet: Enhancing Point Features with Image Semantics for 3D Object Detection. In _ECCV_.
  * Kingma and Ba (2014) D. Kingma and J. Ba. 2014\.  Adam: A Method for Stochastic Optimization.  _Computer Science_ (2014).
  * Ku et al. (2018a) Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven Waslander. 2018a.  Joint 3D Proposal Generation and Object Detection from View Aggregation.  _IROS_ (2018).
  * Ku et al. (2018b) Jason T Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven L. Waslander. 2018b.  Joint 3D Proposal Generation and Object Detection from View Aggregation.  _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ (2018), 1–8.
  * Kuang et al. (2020) Hongwu Kuang, B. Wang, J. An, M. Zhang, and Z. Zhang. 2020.  Voxel-FPN: Multi-Scale Voxel Feature Aggregation for 3D Object Detection from LIDAR Point Clouds.  _Sensors (Basel, Switzerland)_ 20 (2020).
  * Lahoud and Ghanem (2017) J. Lahoud and B. Ghanem. 2017.  2D-Driven 3D Object Detection in RGB-D Images. In _2017 IEEE International Conference on Computer Vision (ICCV)_.
  * Lang et al. (2019) Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, and Oscar Beijbom. 2019.  PointPillars: Fast Encoders for Object Detection From Point Clouds. In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Liang et al. (2019) M. Liang, B. Yang, Y. Chen, R. Hu, and R. Urtasun. 2019.  Multi-Task Multi-Sensor Fusion for 3D Object Detection.  _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 7337–7345.
  * Liang et al. (2018) M. Liang, B. Yang, Shenlong Wang, and R. Urtasun. 2018\.  Deep Continuous Fusion for Multi-sensor 3D Object Detection. In _ECCV_.
  * Liang et al. (2020) Zhidong Liang, Ming Zhang, Zehan Zhang, Xian Zhao, and S. Pu. 2020.  RangeRCNN: Towards Fast and Accurate 3D Object Detection with Range Image Representation.  _ArXiv_ abs/2009.00206 (2020).
  * Lin et al. (2017) Tsung Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017.  Focal Loss for Dense Object Detection.  _IEEE Transactions on Pattern Analysis & Machine Intelligence_ PP, 99 (2017), 2999–3007.
  * Liu et al. (2016) W. Liu, Dragomir Anguelov, D. Erhan, Christian Szegedy, S. Reed, Cheng-Yang Fu, and A. Berg. 2016.  SSD: Single Shot MultiBox Detector. In _ECCV_.
  * Pang et al. (2020) S. Pang, D. Morris, and H. Radha. 2020.  CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection.  _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ (2020), 10386–10393.
  * Qi et al. (2019) C. R. Qi, O. Litany, K. He, and L. J. Guibas. 2019\.  Deep Hough Voting for 3D Object Detection in Point Clouds.  _IEEE_ (2019).
  * Qi et al. (2018) Charles R. Qi, W. Liu, Chenxia Wu, H. Su, and L. Guibas. 2018.  Frustum PointNets for 3D Object Detection from RGB-D Data.  _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2018), 918–927.
  * Qi et al. (2017) C. R. Qi, L. Yi, H. Su, and L. Guibas. 2017\.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. In _NIPS_.
  * Redmon et al. (2016) Joseph Redmon, S. Divvala, Ross B. Girshick, and Ali Farhadi. 2016.  You Only Look Once: Unified, Real-Time Object Detection.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ (2016), 779–788.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross B. Girshick, and J. Sun. 2015\.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 39 (2015), 1137–1149.
  * Ren and Sudderth (2016) Z. Ren and E. B. Sudderth. 2016.  Three-Dimensional Object Detection and Layout Prediction Using Clouds of Oriented Gradients. In _IEEE Conference on Computer Vision & Pattern Recognition_.
  * Shi et al. (2020a) Shaoshuai Shi, Chaoxu Guo, L. Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. 2020a.  PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ (2020), 10526–10535.
  * Shi et al. (2019) Shaoshuai Shi, X. Wang, and Hongsheng Li. 2019.  PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud.  _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ (2019), 770–779.
  * Shi et al. (2020c) Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. 2020c.  From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network.  (2020).  arXiv:1907.03670 [cs.CV]
  * Shi et al. (2020b) Weijing Shi, Ragunathan, and Rajkumar. 2020b.  Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud.  (2020).
  * Song et al. (2015) S. Song, S. P. Lichtenberg, and J. Xiao. 2015\.  SUN RGB-D: A RGB-D scene understanding benchmark suite. In _IEEE Conference on Computer Vision & Pattern Recognition_. 567–576.
  * Song and Xiao (2016) S. Song and J. Xiao. 2016\.  Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images.  _IEEE_ (2016).
  * Tian et al. (2019) Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. 2019\.  FCOS: Fully Convolutional One-Stage Object Detection.  _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ (2019), 9626–9635.
  * Vora et al. (2019) S. Vora, A. H. Lang, B. Helou, and O. Beijbom. 2019\.  PointPainting: Sequential Fusion for 3D Object Detection.  (2019).
  * Wang and Jia (2019) Z. Wang and K. Jia. 2019\.  Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal. In _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_.
  * Xie et al. (2019) L. Xie, C. Xiang, Z. Yu, G. Xu, Z. Yang, D. Cai, and X. He. 2019\.  PI-RCNN: An Efficient Multi-sensor 3D Object Detector with Point-based Attentive Cont-conv Fusion Module.  (2019).
  * Xu et al. (2018) D. Xu, D. Anguelov, and A. Jain. 2018.  PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation.  _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 244–253.  https://doi.org/10.1109/CVPR.2018.00033
  * Yan et al. (2018) Yan Yan, Yuxing Mao, and B. Li. 2018.  SECOND: Sparsely Embedded Convolutional Detection.  _Sensors (Basel, Switzerland)_ 18 (2018).
  * Yang et al. (2018) B. Yang, W. Luo, and R. Urtasun. 2018.  PIXOR: Real-time 3D Object Detection from Point Clouds.  _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2018), 7652–7660.
  * Yang et al. (2020) Z. Yang, Y. Sun, Shu Liu, and Jiaya Jia. 2020\.  3DSSD: Point-Based 3D Single Stage Object Detector.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ (2020), 11037–11045.
  * Yang et al. (2019) Z. Yang, Y. Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. 2019.  STD: Sparse-to-Dense 3D Object Detector for Point Cloud.  _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ (2019), 1951–1960.
  * Yoo et al. (2020) Jin Hyeok Yoo, Yeocheol Kim, J. Kim, and J. W. Choi. 2020\.  3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detection. In _ECCV_.
  * Zhu et al. (2019) Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and G. Yu. 2019.  Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection.  _ArXiv_ abs/1908.09492 (2019).