# BEV-MAE: Bird’s Eye View Masked Autoencoders for Outdoor Point Cloud Pre-
training

Zhiwei Lin Yongtao Wang
Wangxuan Institute of Computer Technology, Peking University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Current outdoor LiDAR-based 3D object detection methods mainly adopt the
training-from-scratch paradigm. Unfortunately, this paradigm heavily relies on
large-scale labeled data, whose collection can be expensive and time-
consuming. Self-supervised pre-training is an effective and desirable way to
alleviate this dependence on extensive annotated data. Recently, masked
modeling has become a successful self-supervised learning approach for point
clouds. However, current works mainly focus on synthetic or indoor datasets.
When applied to large-scale and sparse outdoor point clouds, they fail to
yield satisfactory results. In this work, we present BEV-MAE, a simple masked
autoencoder pre-training framework for 3D object detection on outdoor point
clouds. Specifically, we first propose a bird’s eye view (BEV) guided masking
strategy to guide the 3D encoder learning feature representation in a BEV
perspective and avoid complex decoder design during pre-training. Besides, we
introduce a learnable point token to maintain a consistent receptive field
size of the 3D encoder with fine-tuning for masked point cloud inputs.
Finally, based on the property of outdoor point clouds, i.e., the point clouds
of distant objects are more sparse, we propose point density prediction to
enable the 3D encoder to learn location information, which is essential for
object detection. Experimental results show that BEV-MAE achieves new state-
of-the-art self-supervised results on both Waymo and nuScenes with diverse 3D
object detectors. Furthermore, with only 20% data and 7% training cost during
pre-training, BEV-MAE achieves comparable performance with the state-of-the-
art method ProposalContrast. The source code and pre-trained models will be
made publicly available.

##  1 Introduction

3D object detection is one of the most basic tasks in autonomous driving. It
aims to localize objects in 3D space and classify them simultaneously. In
recent years, LiDAR-based 3D object detection methods have achieved
significant success due to the increasing amount of labeled training data [32,
3]. However, existing LiDAR-based outdoor 3D object detection methods often
adopt the paradigm of training from scratch, which brings two defects. First,
the training-from-scratch paradigm largely relies on extensive labeled data.
For 3D object detection, annotating precise bounding boxes and classification
labels is costly and time-consuming, e.g., it takes around 114$s$ to annotate
one object [26] on KITTI[10]. Second, in many practical scenarios, the massive
unlabeled point cloud data is generated by self-driving vehicles daily, which
cannot be used in the training-from-scratch paradigm.

Figure 1: Illustration of several masking strategies in the masked modeling.
MAE[12] masks non-overlapping image patches. BERT[7] masks words or sentences.
Point-MAE[27] uses furthest point sampling and k-nearest neighbors to create
overlapping point patches for masking. Our method (right) projects point
clouds into a BEV plane and masks point clouds in non-overlapping BEV grids.

A simple and desirable solution to alleviate the above two problems is self-
supervised pre-training, which is widely used in fields such as computer
vision [13, 5] and natural language processing [2]. It can learn a general and
transferable feature representation on large-scale unlabeled data by solving
pre-designed pretext tasks[15]. One of the mainstream approaches in self-
supervised learning is masked modeling, as illustrated in Figure 1.
Specifically, researchers adopt masked image modeling [12, 38] and masked
language modeling [7] to pre-train networks by reconstructing images, words,
and sentences from masked inputs. Recently, works [42, 27] propose masked
point modeling for small and dense point clouds, achieving promising
performance on shape classification, shape segmentation, and indoor 3D object
detection. However, these works mainly focus on synthetic or indoor datasets,
such as ShapeNet [4], ModelNet40 [36], and ScanNet [6]. When applied to
outdoor point clouds, where the scale of scenes is more extensive and point
density is more sparse, their results are not satisfactory. There are few
works[21, 40] to explore the self-supervised pre-training for outdoor point
clouds. However, since the data augmentation in outdoor point clouds is not as
diverse as in the 2D images, the proposed methods based on contrastive
learning need complicated pre-processing to produce pseudo instances[21] or
region proposals[40] to avoid model collapse[5, 43].

In this paper, we present bird’s eye view masked autoencoders, dubbed BEV-MAE,
specially for pre-training 3D object detectors on outdoor point clouds. In
particular, instead of randomly masking point clouds or voxels, we propose a
BEV guided masking strategy (right part of Figure 1) for two benefits. First,
we force the 3D encoder to learn feature representation in a BEV perspective
by reconstructing masked information on the BEV plane. Therefore, during fine-
tuning, the pre-trained 3D encoder can facilitate the training process of 3D
detectors in the BEV perspective. Second, current 3D encoders of LiDAR-based
3D object detectors often downsample the resolution of points or voxels to
save the computational overhead, e.g., GPU memory and training time. By using
BEV guided masking strategy, we do not need to design complicated decoders
with upsampling operations[31] since the size of masked girds matches the
resolution of BEV features. We find that BEV-MAE can achieve promising results
with a simple one-layer 3$\times$3 convolution as the decoder.

Besides, since the commonly used sparse 3D convolutions only perform
computation near the occupied areas, the receptive field size of the 3D
encoder may become smaller with masked point inputs, resulting in low learning
efficiency and poor transferability of the 3D encoder. To alleviate this
issue, during pre-training, we replace the masked points with a shared
learnable point token to maintain a consistent receptive field size of the 3D
encoder with fine-tuning. The shared learnable point token can help
communication between unmasked areas without providing additional information
to reduce the difficulty of the pre-training task. Moreover, in addition to
using coordinates of masked point clouds as the reconstruction target [27,
44], we introduce point density prediction for masked areas. Since the point
clouds become sparse when they are far from the LiDAR sensor for outdoor
scenes, the density of point clouds can reflect the distance between points
and the central LiDAR sensor. Naturally, density prediction can guide models
to learn location information, which is critical for object detection.

Our main contributions can be summarized as follows:

  * •

We present a simple and effective self-supervised pre-training method, BEV-
MAE, tailored for 3D object detectors on outdoor point clouds. By using the
proposed BEV guided masking strategy, the 3D encoder of 3D object detectors
can directly learn feature representation in a BEV perspective with a simple
and light decoder.

  * •

We introduce a shared learnable point token to alleviate the inconsistency of
the receptive field size for the 3D encoder during pre-training and fine-
tuning, and propose density prediction to better learn location information
for the 3D encoder.

  * •

Compared with existing self-supervised methods, the proposed BEV-MAE achieves
new state-of-the-art performance on two popular outdoor 3D object detection
benchmarks, i.e., Waymo[32] and nuScenes[3], with less than 1/4 pre-training
epochs.

Figure 2: The overall pipeline of BEV-MAE. We first mask point clouds with the
BEV guided masking strategy. Then, the masked points are replaced with a
shared learnable point token. After extracting BEV features by a 3D encoder
from visible points, we send the features to a light decoder to reconstruct
masked point clouds and predict the point density of masked grids.

##  2 Related Work

###  2.1 3D Object Detection

The objective of 3D object detection is to localize objects of interest with
3D bounding boxes and classify the detected objects. Due to the large domain
gap between indoor and outdoor datasets, the corresponding 3D object detection
methods in the two cases have been developing almost independently. Here we
focus on the works about outdoor 3D object detection. Recent outdoor 3D object
detection approaches[39, 47, 19, 30] based on BEV representation attract much
attention for their convenience in fusing different information, including
multi-view[17, 20], multi-modality[22, 24], and temporal inputs[16]. To obtain
BEV features, LiDAR-based methods first extract point features by a 3D
encoder, such as PointPillar[19] and VoxelNet[47], and then project the
features onto a BEV plane according to the point coordinates. For camera-based
approaches, they extract the 2D features from multi-view images by a prevalent
2D backbone, including ResNet[14] and SwinTransformer[23], and utilize
geometry-based view transformation[28, 25] to construct BEV features from
multi-view image features. After that, for both types of methods, a 2D encoder
and a detection head[39, 41] are applied to process the BEV features and
predict the final detection results. In essence, this pipeline reduces the
task of 3D object detection to 2D object detection. Hence, these methods can
take full advantage of the highly developed 2D object detection algorithms.
Similarly, our work pre-trains the 3D encoder of 3D object detectors in the
BEV space and can directly facilitate the 3D detection task in the BEV
perspective.

###  2.2 Self-supervised Learning for Point Clouds

The main idea of self-supervised learning is to train networks on fully
unlabeled data with pretext tasks. Since no prior human knowledge is
introduced, the pre-trained networks can learn a more general feature
representation and have an excellent transfer ability. Recently, 3D
representation learning of point clouds has attempted to adopt the core idea
of self-supervised learning. PointContrast[37] samples different views from
one scene and utilizes contrastive learning to discriminate the positive
views. DepthContrast[45] transforms RGB-D images into point clouds and further
extends PointContrast by using two different types of point encoders. STRL[18]
constructs positive samples by using spatial augmentation from videos. In
addition to contrastive learning methods, masked modeling for 3D point clouds
has been studied in recent years for its simplicity and generality across
different fields. OcCo[35] proposes to drop points from a camera view and
recover the occlusion part of point clouds. Point-BERT[42] introduces BEiT-
style[1] masked modeling for 3D point clouds with a pre-trained dVAE[29] and
achieves competitive performance on shape classification, shape segmentation,
and indoor 3D object detection. Point-MAE[27] removes the training stage of
dVAE in Point-BERT and reconstructs the masked points directly following
MAE[12]. Point-M2AE[44] proposes a U-Net transformer architecture and adopts a
multi-scale masking strategy to learn the multi-scale spatial geometries.
However, these methods are mainly applied to the synthetic or indoor dataset.
They fail to be directly transferred to large-scale outdoor point clouds due
to the large domain gap between datasets.

To address this issue, more recently, GCC-3D[21] proposes geometry-aware
contrast and harmonized cluster to learn both geometry and semantic
information from outdoor point clouds. ProposalConstrast[40] samples region
proposals for each point cloud with farthest point sampling (FPS) and jointly
optimizes inter-proposal discrimination and inter-cluster separation for
outdoor point clouds. However, these works need complex pre-processing on
input point clouds to produce pseudo instances or region proposals.

In this paper, we propose a simple masked modeling framework, BEV-MAE, for
pre-training large-scale outdoor point clouds. Without any pre-processing on
point clouds, BEV-MAE adopts a BEV guided masking strategy to learn BEV
feature representation and achieves new state-of-the-art self-supervised
learning performance.

##  3 Method

In this section, we introduce the proposed BEV-MAE in detail. The overall
pipeline is shown in Figure 2. BEV-MAE first uses a BEV guided masking
strategy to mask point clouds. Then, these masked points are replaced with a
shared learnable point token. After that, we send the processed points into a
3D encoder and a light decoder sequentially. Finally, the light decoder will
reconstruct the masked points and predict the point density of the masked
area.

###  3.1 BEV Guided Masking Strategy

In outdoor LiDAR-based 3D object detection, the point clouds are often divided
into regular voxels. Naturally, a straightforward masking strategy is to mask
the voxels like masking patches in vision[12, 38]. However, this simple voxel
masking strategy makes subsequent decoder design difficult. Besides, it takes
little consideration of the type of feature representation in the mainstream
outdoor 3D object detection methods, i.e., BEV feature representation. To this
end, we propose a BEV guided masking strategy to mask points in the BEV plane.

Specifically, assuming the resolution of the features in the BEV perspective
after encoding and transformation is $X\times Y\times C$, we first pre-define
a grid-shaped BEV plane with the size of $X\times Y$. Then, we project each
LiDAR point $p_{k}$ into a corresponding BEV grid $g_{i,j}$ of the pre-defined
BEV plane according to its point coordinates $(x_{p_{k}},y_{p_{k}})$.
Obviously, each BEV grid will contain a various number of points:

$g_{i,j}=\\{p_{k}~{}|~{}\lfloor x_{p_{k}}/d\rfloor=i,~{}\lfloor
y_{p_{k}}/d\rfloor=j\\},$ (1)

where $d$ is the downsample ratio of the 3D encoder and $\lfloor x\rfloor$
denotes rounding down of $x$. After that, we randomly select a large fraction
of non-empty BEV grids, i.e., $g_{i,j}\neq\varnothing$, as the masked grids
$\\{g^{m}_{i}\\}$ and denote the remaining BEV grids as visible grids
$\\{g^{v}_{i}\\}$. Finally, we obtain the visible and masked point clouds by
merging the points in $\\{g^{m}_{i}\\}$ and $\\{g^{v}_{i}\\}$, formulated as
$\\{p_{k}^{v}\\}=\mathop{\cup}\limits_{\tiny{i}}g^{v}_{i}$ and
$\\{p_{k}^{m}\\}=\mathop{\cup}\limits_{\tiny{i}}g^{m}_{i}$ respectively.

###  3.2 Learnable Point Token

The 3D encoder of recent voxel-based 3D object detectors typically consists of
several sparse convolution operations, which only process the features near
the non-empty voxels. Thus, when only taking visible point clouds
$\\{p_{k}^{v}\\}$ as input, the size of the receptive field of the 3D encoder
becomes smaller. To address this issue, we replace the masked point clouds
$\\{p_{k}^{m}\\}$ with a shared learnable point token. In detail, we use
coordinates of full point clouds as the input indexes[39] of sparse
convolution and replace the feature of masked point clouds with the shared
learnable point token in the first sparse convolution layer. We keep the other
sparse convolutional layers unchanged. The only purpose of the proposed shared
learnable point token is to pass the information from one point or voxel to
another to maintain the size of the receptive field. It does not introduce any
additional information, including the coordinates of masked points.

###  3.3 Decoder Design

Due to the BEV guided masking strategy, the size of masked areas is aligned
with the resolution of the BEV features. Thus, we can directly predict the
reconstruction results of one masked grid from the corresponding BEV features
without any upsampling operations.

The decoder of BEV-MAE is only used during pre-training to solve the masking
task. Naturally, the design of the decoder is a flexible manner that is
independent of the 3D encoder architecture. We test three types of decoders,
i.e., one-layer 3$\times$3 convolution, transformer block[34, 8, 12], and
residual convolution block[14], and find the highly lightweight decoder, a
one-layer 3$\times$3 convolution, can achieve impressive performance while
reducing pre-training time and GPU memory cost. More details can be found in
Section 4.5.

###  3.4 Reconstruction Target

The proposed BEV-MAE is supervised by two tasks, i.e., point cloud
reconstruction and density prediction. For each task, a separate linear layer
is applied as the prediction head to predict results. We describe each task
below.

####  3.4.1 Point Cloud Reconstruction

Like previous works[27, 44], BEV-MAE reconstructs the masked input by
predicting the coordinate of masked points. However, each masked grid in the
pre-defined BEV plane contains a different number of points, leading to the
challenges of designing the prediction head.

To address this issue, we propose to reconstruct the local structure of the
masked point clouds with a set-to-set prediction. Specifically, we apply a
linear layer to predict a set of 3D points with a fixed number, denoted as
$P_{i}=\\{p_{l}~{}|~{}l=1,2,...,L\\}$, for each masked grid $g_{i}^{m}$. Given
the original points $\hat{P}_{i}=\\{\hat{p_{k}}~{}|~{}k=1,2,...,N\\}$ in
$g_{i}^{m}$, where $N$ varies with the grid, we utilize Chamfer Distance[9]
between predictions $P_{i}$ and the ground-truth $\hat{P}_{i}$ as the
reconstruction loss:

$\mathcal{L}_{c}^{i}=\frac{1}{L}\sum_{p_{l}\in
P_{i}}{\min_{\hat{p_{k}}\in\hat{P_{i}}}\lvert\lvert
p_{l}-\hat{p_{k}}\rvert\rvert_{2}^{2}}+\frac{1}{N}\sum_{\hat{p_{k}}\in\hat{P_{i}}}{\min_{p_{l}\in
P_{i}}\lvert\lvert\hat{p_{k}}-p_{l}\rvert\rvert_{2}^{2}}.$ (2)

Then, we average the loss over all the masked grids as the final
reconstruction loss:

$\mathcal{L}_{c}=\frac{1}{n_{m}}\sum_{i=1}^{n_{m}}\mathcal{L}_{c}^{i},$ (3)

where $n_{m}$ is the number of masked grids. The Chamfer distance measures the
distance between two sets with different cardinalities. Thus, it forces the
shape of predicted point clouds to mimic the local structure of masked
inputs[9]. More analysis can be found in Section 4.6.

Moreover, since the coordinate value of each ground-truth point varies largely
in the different grids, directly predicting the absolute coordinates of each
point may cause instability during pre-training. To alleviate this issue, we
propose to predict the normalized coordinate in point cloud reconstruction.
Specifically, we first calculate the coordinate offset of each ground-truth
point to the center of its corresponding BEV grid and then normalize the
offset value by the size of the BEV grid.

Table 1: Comparisons between BEV-MAE and the state-of-the-art self-supervised
learning methods on Waymo. All detectors are fine-tuning with 20% training
samples on Waymo following the OpenPCDet[33] configuration. ∗ denotes the
results are from the paper[21]. † presents pre-training both 3D encoder and 2D
encoder. ‘Epochs’ indicates the effective pre-training epochs accounting for
actual trained samples/views[46]. ‘Dataset fraction’ means the data fraction
of Waymo training set used for pre-training. ‘Time’ refers to the pre-training
time estimated by an 8-GPU P40 machine.

Method |  Epochs |  Time |  Dataset fraction | L2 (mAP/APH)
---|---|---|---|---
Overall | Vehicle | Pedestrian | Cyclist
SECOND | - | - | - | 60.62 / 56.86 | 64.26 / 63.73 | 59.72 / 50.38 | 57.87 / 56.48
ProposalContrast (SECOND) | 2$\times$50 | 64h | 100% | 60.74+0.12 / 57.01+0.15 | 64.40 / 63.83 | 60.17 / 50.82 | 57.87 / 56.45
ProposalContrast (SECOND)† | 2$\times$50 | 72h | 100% | 60.90+0.28 / 57.17+0.31 | 64.50 / 63.90 | 60.33 / 51.00 | 57.90 / 56.60
BEV-MAE (SECOND) | 20 | 5h | 20% | 60.86+0.24 / 57.15+0.29 | 64.38 / 63.85 | 60.10 / 50.84 | 58.11 / 56.76
BEV-MAE (SECOND) | 20 | 25h | 100% |  61.03+0.43 / 57.30+0.44 | 64.42 / 63.87 | 59.97 / 50.65 | 58.69 / 57.39
CenterPoint∗ | - | - | - | 63.46 / 60.95 | 61.81 / 61.30 | 63.62 / 57.79 | 64.96 / 63.77
GCC-3D (CenterPoint)∗ | 2$\times$40 | - | 100% | 65.29+1.83 / 62.79+1.84 | 63.97 / 63.47 | 64.23 / 58.47 | 67.68 / 66.44
CenterPoint | - | - | - | 65.60 / 63.21 | 64.18 / 63.69 | 65.22 / 59.68 | 67.41 / 66.25
ProposalContrast (CenterPoint) | 2$\times$50 | 64h | 100% | 66.41+0.81 / 63.84+0.63 | 65.03 / 64.53 | 65.93 / 59.95 | 68.26 / 67.04
ProposalContrast (CenterPoint)† | 2$\times$50 | 72h | 100% | 66.67+1.07 / 64.20+0.99 | 65.22 / 64.80 | 66.40 / 60.49 | 68.48 / 67.38
BEV-MAE (CenterPoint) | 20 | 5h | 20% | 66.70+1.10 / 64.25+1.04 | 64.71 / 64.22 | 66.21 / 60.59 | 69.11 / 67.93
BEV-MAE (CenterPoint) | 20 | 25h | 100% |  66.92+1.32 / 64.45+1.24 | 64.78 / 64.29 | 66.25 / 60.53 | 69.73 / 68.52
PV-RCNN++ | - | - | - | 69.97 / 67.58 | 69.18 / 68.75 | 70.88 / 65.21 | 69.84 / 68.77
ProposalContrast (PV-RCNN++) | 2$\times$50 | 64h | 100% | 70.30+0.33 / 67.78+0.20 | 69.45 / 69.00 | 71.42 / 65.68 | 70.04 / 69.05
ProposalContrast (PV-RCNN++)† | 2$\times$50 | 72h | 100% | 70.49+0.52 / 67.98+0.40 | 69.47 / 68.95 | 71.28 / 65.31 | 70.73 / 69.59
BEV-MAE (PV-RCNN++) | 20 | 5h | 20% | 70.45+0.48 / 67.96+0.38 | 69.44 / 69.02 | 71.14 / 65.21 | 70.77 / 69.65
BEV-MAE (PV-RCNN++) | 20 | 25h | 100% |  70.54+0.57 / 68.11+0.53 | 69.53 / 69.07 | 71.50 / 65.69 | 70.60 / 69.56

Table 2: Comparisons between BEV-MAE and GCC-3D on nuScenes. ‘Epochs’
indicates the effective pre-training epochs[46].

Method | Epochs | mAP | NDS | Car | Truck | CV. | Bus | Trailer | Barrier | Motor. | Bicycle | Ped. | TC.
---|---|---|---|---|---|---|---|---|---|---|---|---|---
CenterPoint | - | 56.2 | 64.5 | 84.8 | 53.9 | 16.8 | 67.0 | 35.9 | 64.8 | 55.8 | 36.4 | 83.1 | 63.4
GCC-3D (CenterPoint) | 2$\times$40 | 57.3+1.1 | 65.0+0.5 | 85.0 | 54.7 | 17.6 | 67.2 | 35.7 | 65.0 | 56.2 | 36.0 | 82.9 | 63.7
BEV-MAE (CenterPoint) | 20 | 57.2+1.0 | 65.1+0.6 | 84.9 | 54.9 | 16.5 | 67.2 | 35.9 | 65.2 | 56.0 | 36.2 | 83.2 | 63.5

####  3.4.2 Point Density Prediction

Different from image, language, and indoor point clouds, outdoor point clouds
have the property that the density of the point clouds becomes small when they
are far from the LiDAR sensor. Consequently, the density can reflect the
location of each point or object. Besides, for object detection, the
localization ability of detectors is essential. Based on the above analysis,
we propose another task for BEV-MAE, i.e., point density prediction, to guide
the 3D encoder to achieve a better localization ability.

Concretely, for each masked grid $g_{i}^{m}$, we count the number of point
clouds in this grid and calculate the density $\hat{\rho_{i}}$ by dividing the
number of point clouds with the occupied volume in 3D space as the ground-
truth for density prediction. Then, we use a linear layer as the prediction
head to obtain the density prediction $\rho_{i}$. We supervise this task with
the Smooth-$\ell_{1}$[11] loss:

$\mathcal{L}_{d}^{i}=Smooth\text{-}{\ell_{1}}(\rho_{i}-\hat{\rho_{i}}).$ (4)

Similarly, we average the loss over all the masked grids as the final density
prediction loss:

$\mathcal{L}_{d}=\frac{1}{n_{m}}\sum_{i=1}^{n_{m}}\mathcal{L}_{d}^{i}.$ (5)

####  3.4.3 Loss Function

The overall loss function for BEV-MAE is:

$\mathcal{L}=\alpha_{c}\mathcal{L}_{c}+\alpha_{d}\mathcal{L}_{d},$ (6)

where $\alpha_{c}$ and $\alpha_{d}$ are balance weights for each loss. In our
experiments, both $\alpha_{c}$ and $\alpha_{d}$ are set to 1.

Table 3: Results about data efficiency on Waymo. We first pre-train the 3D encoder on the full Waymo dataset and then fine-tune it on various fractions of Waymo dataset with CenterPoint as the 3D object detector. Consistent improvements are obtained under each setting.  Method |  Dataset fraction for fine-tuning | LEVEL_1 | LEVEL_2
---|---|---|---
mAP | APH | mAP | APH
From scratch | 1% | 29.80 | 25.58 | 26.27 | 22.62
BEV-MAE | 35.27+5.47 | 31.63+6.05 | 31.26+4.99 | 28.08+5.46
From scratch | 5% | 58.02 | 54.94 | 52.33 | 49.57
BEV-MAE | 60.74+2.71 | 57.55+2.61 | 54.92+2.59 | 52.06+2.49
From scratch | 20% | 68.42 | 65.56 | 62.27 | 59.65
BEV-MAE | 70.00+1.58 | 67.19+1.63 | 63.85+1.58 | 61.26+1.61
From scratch | 50% | 71.65 | 69.02 | 65.45 | 63.02
BEV-MAE | 72.20+0.55 | 69.50+0.48 | 66.00+0.55 | 63.50+0.48
From scratch | 100% | 73.28 | 70.67 | 67.11 | 64.69
BEV-MAE | 73.50+0.22 | 70.90+0.23 | 67.36+0.25 | 64.95+0.26

##  4 Experimental Results

###  4.1 Implementation Details

We evaluate the proposed BEV-MAE on two popular large-scale autonomous driving
datasets, i.e., nuScenes and Waymo Open Dataset. We mainly focus on the
evaluation metrics of mAP and NDS for nuScenes and the more difficult LEVEL_2
metric (L2 mAP and L2 APH) for Waymo. We adopt VoxelNet as the 3D encoder. We
fine-tune the pre-trained 3D encoder with three prevalent 3D detectors, i.e.,
SECOND[39], CenterPoint[41], and PV-RCNN++[30].

During pre-training, We train BEV-MAE for 20 epochs and use the Adam optimizer
with the one-cycle schedule. The maximum learning rate is set to 0.0003 with a
batch size of 4. The masking ratio is 0.7 and the number of predicted points
$L$ in point cloud reconstruction is 20 for each masked grid. For fine-tuning,
we adopt the default training schedule as training-from-scratch baselines in
OpenPCDet. The maximum learning rate is set to 0.006 with a batch size of 4.
For Waymo, all detectors are fine-tuning with 20% training samples.

###  4.2 Main Results

We compare the proposed BEV-MAE with the state-of-the-art self-supervised
learning methods for LiDAR-based 3D Object Detection in Table 1 and Table 2.

On the Waymo validation set, Table 1 shows that BEV-MAE substantially improves
the performance of training-from-scratch baselines and outperforms previous
state-of-the-art self-supervised learning methods with three types of 3D
detectors. Besides, we observe that, with only 20% training data and 7%
computation cost, BEV-MAE achieves similar performance compared to
ProposalContrast with 100% data for pre-training, further demonstrating the
efficiency of BEV-MAE pre-training. Specifically, based on SECOND, BEV-MAE
brings 0.43 mAP and 0.44 APH improvement for the training-from-scratch
baseline. For CenterPoint, BEV-MAE improves the training-from-scratch baseline
by 1.32 mAP and 1.24 APH on average, outperforming the state-of-the-art
ProposalContrast by 0.25 mAP and 0.25 APH. Moreover, our method reaches 68.11
APH with PV-RCNN++, which surpasses the training-from-scratch baseline and
ProposalContrast by 0.53 APH and 0.13 APH, respectively.

For nuScenes, in Table 2, BEV-MAE achieves a higher NDS with only 20 pre-
training epochs compared to the state-of-the-art method GCC-3D.

Overall, the results demonstrate BEV-MAE is more effective and efficient on
outdoor point clouds compared to the state-of-the-art self-supervised learning
methods based on contrastive learning. In addition, the pre-trained encoder
shows excellent cross-model generalizability over several 3D detectors.

###  4.3 Data Efficiency

To assess the data efficiency of BEV-MAE, we train the 3D detectors with
different amounts of labeled data. Concretely, we first pre-train the 3D
encoder of CenterPoint with BEV-MAE on the full Waymo dataset and then fine-
tune it on various fractions of the Waymo training data, i.e., 1%, 5%, 20%,
50%, and 100%. To save computation, the 3D detectors are trained with 12
epochs[41]. As shown in Table 3, pre-training with BEV-MAE can consistently
improve the detection results on different fractions of the training data.
Especially, our method can obtain more significant gains when the labeled data
is less. For example, BEV-MAE surpasses the training-from-scratch baseline by
6.05 APH on LEVEL_1 and 5.46 APH on LEVEL_2 under 1% labeled data setting.
This suggests the potential of BEV-MAE in using large amounts of unlabeled
data.

###  4.4 Transfer Learning

We evaluate the transferability of the pre-trained encoder, which indicates
the cross-dataset generalization of the feature representation learned by BEV-
MAE, by fine-tuning the encoder on a different dataset. We only use the
coordinates as the input feature of point clouds for compatibility across
different datasets. The results are present in Table 4. We can find that BEV-
MAE consistently improves the detection performance across different datasets.
Besides, we find that CenterPoint achieves better results when pre-training
and fine-tuning on the same dataset. We suspect that the domain gaps between
two datasets may hurt the transfer performance, e.g., the density of point
clouds on Waymo is five times higher than it of nuScenes. Moreover, when pre-
trained on the combined dataset of Waymo and nuScenes, the detection
performance on both datasets can be further improved, which indicates that
BEV-MAE benefits from a larger amount of data.

Table 4: Results of transfer learning. The contents in column and row show the
datasets for pre-training and fine-tuning respectively.

Pre-train Fine-tune | nuScenes | Waymo
---|---|---
mAP | NDS | L2 mAP | L2 APH
Random init. | 48.6 | 58.4 | 63.97 | 61.53
nuScenes | 49.7+1.1 | 58.9+0.5 | 64.79+0.82 | 62.28+0.75
Waymo | 49.4+0.8 | 58.8+0.4 | 65.13+1.16 | 62.63+1.10
nuScenes + Waymo | 50.1+1.5 | 59.1+0.7 | 65.36+1.39 | 62.89+1.36

Table 5: Ablation on main components of BEV-MAE. ‘Learnable token’ denotes the
shared learnable point token. Each component brings performance improvement
for BEV-MAE.

Pre-train | Reconstruction target | Learnable token | L2 mAP | L2 APH
---|---|---|---|---
None | - | - | 65.60 | 63.21
BEV-MAE | Coord. (w/o norm) | $\checkmark$ | 65.66 | 63.09
Coord. (w norm) | $\checkmark$ | 66.20 | 63.71
Density | $\checkmark$ | 65.80 | 63.27
Coord. (w norm) + Density | $\times$ | 66.49 | 63.99
Coord. (w norm) + Density | $\checkmark$ | 66.70 | 64.25

###  4.5 Ablation Study

In this section, we conduct ablation experiments to analyze the effectiveness
of each setting of BEV-MAE, including the main components and hyper-
parameters. In the following experiments, we use CenterPoint as the 3D
detector. We first pre-train the 3D encoder on the 20% training data of Waymo
with BEV-MAE, and then evaluate its performance by fine-tuning with
CenterPoint on the same 20% training data of Waymo.

####  4.5.1 Main Component

We evaluate the effectiveness of each component of the proposed BEV-MAE, as
shown in Table 5. We can see that only pre-training with normalized coordinate
or density prediction can significantly improve 3D detection performance. For
coordinate prediction, the normalization operation to process the coordinates
of points is essential. Besides, the detection results can be further improved
by taking both coordinate and density prediction as the pre-training task,
achieving 66.49 mAP and 63.99 APH. Moreover, pre-training with the shared
learnable token brings additional performance improvement of 0.43 mAP and 0.46
APH.

####  4.5.2 Decoder Design

In addition to one-layer 3$\times$3 convolution, we also test two other
decoders with more complex design, i.e., Transformer block[34, 8, 12] and
Residual Conv block[14], as shown in Table 6. From the table, we can find that
the detection performance drops significantly with more complex decoders. The
performance decreases more when using the Transformer block as the decoder,
which we assign to the different architecture between the encoder and decoder
(Sparse Convolution vs. Transformer). Besides, we also observe that the
complex decoders bring additional training cost, e.g., Residual Conv block
brings additional 0.2$\times$ training cost compared with one-layer 3$\times$3
Conv. These results indicate that the exploration of decoder design may not be
essential for BEV-MAE. A simple one-layer 3$\times$3 convolution is adequate
for practice. Similar conclusions can be found in SimMiM[38].

Table 6: Ablation on different deocders. One-layer 3$\times$3 Conv achieves
best results with least training cost.

Decoder | L2 mAP | L2 APH | Training cost
---|---|---|---
One-layer 3$\times$3 Conv | 66.70 | 64.25 | 1$\times$
Residual Conv block | 66.61 | 64.09 | 1.2$\times$
Transformer block | 65.80 | 63.26 | 1.4$\times$

Table 7: Ablation on the masking strategy. Pre-training with the BEV guided
masking strategy performs better on the downstream 3D detection task with less
GPU memory consumption and pre-training cost.

Masking strategy | L2 mAP | L2 APH | Memory | Training cost
---|---|---|---|---
Voxel masking | 66.63 | 64.16 | 12.6G | 1.4$\times$
BEV guided masking | 66.70 | 64.25 | 4.1G | 1$\bm{\times}$

Figure 3:  Results of point cloud reconstruction on Waymo validation set. For
each triplet, we show the masked point cloud input (left), our BEV-MAE
reconstruction (middle), and the ground-truth (right). BEV-MAE can predict the
reasonable shape of most objects and capture the height of the points. Best
viewed by zooming in.

####  4.5.3 Masking Strategy

We study how different masking strategies, including a simple voxel masking
strategy and the proposed BEV guided masking strategy, affect the
effectiveness of representation learning. For the voxel masking strategy, we
randomly mask non-empty voxels and use sparse deconvolution layers[31] as the
decoder to recover the points in masked voxels. In Table 7, we observe that
the proposed BEV guided masking strategy achieves better transfer performance
on downstream 3D object detection tasks. In addition, compared with the voxel
masking strategy, the BEV guided masking strategy significantly reduces the
GPU memory (4.1G vs. 12.6G) and training costs (1$\times$ vs. 1.4$\times$)
during pre-training.

Table 8: Ablation on masking ratio. The fine-tuning results are less sensitive to the masking ratio.  Masking ratio | L2 mAP | L2 APH
---|---|---
50% | 66.45 | 64.00
60% | 66.62 | 64.13
70% | 66.70 | 64.25
80% | 66.52 | 64.06

####  4.5.4 Analysis of Hyper-parameters

Masking ratio. Table 8 ablates the influence of the masking ratio. It shows
that the proposed BEV-MAE works well within a wide range of masking ratios
(50%-80%). The best results are achieved when the masking ratio is 70%.

Number of predicted points. Table 9 ablates the influence of a varying number
of predicted points in point cloud reconstruction. We notice that the
performance is optimal with 20 predicted points and drops significantly when
the number of predicted points is too small or large. We attribute to the fact
that, when the number of predicted points is small, the reconstructed points
are insufficient to represent the local structure of masked points. Thus, the
optimization objective does not converge well. By contrast, for a large number
of predicted points, the reconstructed points are redundant, leading to the
over-fitting of masked points.

Table 9: Ablation on the number of predicted points in point cloud reconstruction. The fine-tuning performance drops when the number of predicted points is too large or small.  Number of predicted points | L2 mAP | L2 APH
---|---|---
10 | 66.25 | 63.71
20 | 66.70 | 64.25
30 | 66.43 | 63.92
40 | 66.06 | 63.54

###  4.6 Visualization

We visualize the reconstruction results of BEV-MAE in Figure 3. The pre-
trained BEV-MAE can predict the reasonable shape of most objects and the
plausible boundaries of surrounding scenes. Moreover, the model can capture
the height of the points, which indicates that the 3D encoder has learned
feature representation with much 3D location information in the BEV
perspective.

##  5 Conclusions

In this work, we address the problem of self-supervised pre-training on
outdoor point clouds. We present BEV-MAE to pre-train the 3D encoder of LiDAR-
based 3D object detectors. Instead of simply masking points or voxels, we
propose a BEV guided masking strategy for better BEV representation learning
and to avoid complex decoder design. Besides, we introduce a shared learnable
point token to maintain the receptive field size of the encoder during pre-
training and fine-tuning. By leveraging the properties of outdoor point
clouds, we propose point density prediction to guide the encoder to learn
location information. Experimental results show that BEV-MAE achieves new
state-of-the-art self-supervised results on both Waymo and nuScenes with
diverse prevalent 3D object detectors. Moreover, with only 20% pre-training
data and 7% cost, BEV-MAE achieves comparable performance with
ProposalContrast, which shows that the proposed BEV-MAE is more efficient than
previous self-supervised learning methods based on contrastive learning.

## Supplementary

##  A Implementation Details

###  A.1 Waymo Setup

Waymo Open Dataset is a large-scale autonomous-driving dataset for 3D
detection, consisting of 798 training, 202 validation, and 150 testing
sequences. Each sequence contains about 200 frames of LiDAR points and camera
images. In our experiments, we keep the point clouds in a range of [-75.2m,
75.2m] for the X and Y axis, and [-2m, 4m] for the Z axis as the inputs. Voxel
size is set to (0.1m, 0.1m, 0.15m) for VoxelNet. We use the coordinates of
XYZ, intensity, and elongation as the initial point feature for both pre-
training and fine-tuning.

###  A.2 nuScenes Setup

The nuScenes dataset contains 28,130 training samples and 6,019 validation
samples under 850 scenes. For experiments on nuScenes, we set the detection
range to [-51.2m, 51.2m] for the X and Y axis, and [-5m, 3m] for the Z axis.
Voxel size is set as (0.1m, 0.1m, 0.2m). During pre-training and fine-tuning,
we concatenate consecutive 10 sweep point clouds as the input and use the
coordinates of XYZ, intensity, and timestamp as the initial point feature.

###  A.3 Decoder Architecture

####  A.3.1 One-layer 3$\times$3 Conv

We use one 3$\times$3 convolutional layer followed by a batch normalization
layer and a ReLU as the default decoder. The number of output channels of the
3$\times$3 convolutional layer is the same as the number of input channels.

####  A.3.2 Residual Conv block

We adopt the bottleneck block in ResNet[14] as the Residual Conv block.
Specifically, the initial BEV feature will be passed through three
convolutional layers. Then, the new feature will be added to the initial
feature and sent to a one-layer MLP. Batch normalization and ReLU are adopted
right after each convolution.

####  A.3.3 Transformer block

We adopt a similar strategy to divide the BEV feature into several patches as
ViT[8]. Each patch is flattened to a $C$-dimension vector and mapped into a
256-dimension feature vector with a linear projection layer. Then, these
vectors are sent to a standard transformer block to update their features.
Finally, another linear projection layer will project feature vectors into the
$C$-dimension vectors, which will be reshaped into the 2D BEV feature.

##  B More Results

###  B.1 Pre-training Epochs

Self-supervised learning methods often benefit from longer pre-training
time[5, 12]. We show the results of BEV-MAE with different pre-training epochs
in Table A. We use CenterPoint as the 3D detector for fine-tuning. The 3D
encoder is pre-trained on the full Waymo dataset. We can find that the
downstream detection performance can be further improved with a longer pre-
training epoch. Besides, we observe the saturation of BEV-MAE at 50 epochs. We
attribute to the fact that the amount of data in Waymo may be insufficient for
large-scale self-supervised learning.

Table A: Results of BEV-MAE with different pre-training epochs. All detectors
are fine-tuning with 20% training samples on Waymo.

Method | Epochs | L2 mAP | L2 APH
---|---|---|---
CenterPoint | - | 65.60 | 63.21
BEV-MAE (CenterPoint) | 10 | 66.66 | 64.16
BEV-MAE (CenterPoint) | 20 | 66.92 | 64.45
BEV-MAE (CenterPoint) | 50 | 66.95 | 64.47

###  B.2 More Diverse 3D Detector

To further evaluate the cross-model generalization of BEV-MAE, we try to pre-
train more diverse 3D detectors with BEV-MAE, i.e., PointPillar[19] (point-
based method) and PointAugmenting[DBLP:conf/cvpr/pointaug] (multi-modality
method). The results are present in Table B. We can find that BEV-MAE brings
consistent improvements for each detector.

Table B: Results of diverse 3D Detectors pre-training with BEV-MAE. All detectors are fine-tuning with 20% training samples on Waymo.  Method | Pre-training | L2 mAP | L2 APH
---|---|---|---
PointPillar | None | 63.40 | 59.38
PointPillar | BEV-MAE | 63.76 | 59.69
PointAugmenting | None | 66.72 | 64.23
PointAugmenting | BEV-MAE | 67.22 | 64.70

## References

  * [1] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei.  Beit: BERT pre-training of image transformers.  In ICLR, 2022.
  * [2] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In NeurIPS, 2020.
  * [3] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In CVPR, 2020.
  * [4] Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  Shapenet: An information-rich 3d model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In ICML, 2020.
  * [6] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In CVPR, 2017.
  * [7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In NAACL, 2019.
  * [8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2020.
  * [9] Haoqiang Fan, Hao Su, and Leonidas J. Guibas.  A point set generation network for 3d object reconstruction from a single image.  In CVPR, 2017.
  * [10] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In CVPR, 2012.
  * [11] Ross B. Girshick.  Fast R-CNN.  In ICCV, 2015.
  * [12] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick.  Masked autoencoders are scalable vision learners.  In CVPR, 2022.
  * [13] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In CVPR, 2020.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [15] Gabriel Huang, Issam Laradji, David Vazquez, Simon Lacoste-Julien, and Pau Rodriguez.  A survey of self-supervised and few-shot object detection.  IEEE TPAMI, 2022.
  * [16] Junjie Huang and Guan Huang.  Bevdet4d: Exploit temporal cues in multi-camera 3d object detection.  arXiv preprint arXiv: 2203.17054, 2022.
  * [17] Junjie Huang, Guan Huang, Zheng Zhu, and Dalong Du.  Bevdet: High-performance multi-camera 3d object detection in bird-eye-view.  arXiv preprint arXiv: 2112.11790, 2021.
  * [18] Siyuan Huang, Yichen Xie, Song-Chun Zhu, and Yixin Zhu.  Spatio-temporal self-supervised representation learning for 3d point clouds.  In ICCV, 2021.
  * [19] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  In CVPR, 2019.
  * [20] Yiming Li, Shunli Ren, Pengxiang Wu, Siheng Chen, Chen Feng, and Wenjun Zhang.  Learning distilled collaboration graph for multi-agent perception.  In NeurIPS, 2021.
  * [21] Hanxue Liang, Chenhan Jiang, Dapeng Feng, Xin Chen, Hang Xu, Xiaodan Liang, Wei Zhang, Zhenguo Li, and Luc Van Gool.  Exploring geometry-aware contrast and clustering harmonization for self-supervised 3d object detection.  In ICCV, 2021.
  * [22] Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, and Zhi Tang.  Bevfusion: A simple and robust lidar-camera fusion framework.  arXiv preprint arXiv: 2205.13790, 2022.
  * [23] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In ICCV, 2021.
  * [24] Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela Rus, and Song Han.  Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation.  arXiv preprint arXiv: 2205.13542, 2022.
  * [25] Yuexin Ma, Tai Wang, Xuyang Bai, Huitong Yang, Yuenan Hou, Yaming Wang, Yu Qiao, Ruigang Yang, Dinesh Manocha, and Xinge Zhu.  Vision-centric BEV perception: A survey.  arXiv preprint arXiv: 2208.02797, 2022.
  * [26] Qinghao Meng, Wenguan Wang, Tianfei Zhou, Jianbing Shen, Luc Van Gool, and Dengxin Dai.  Weakly supervised 3d object detection from lidar point cloud.  In ECCV, 2020.
  * [27] Yatian Pang, Wenxiao Wang, Francis E. H. Tay, Wei Liu, Yonghong Tian, and Li Yuan.  Masked autoencoders for point cloud self-supervised learning.  In ECCV, 2022.
  * [28] Jonah Philion and Sanja Fidler.  Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d.  In ECCV, 2020.
  * [29] Jason Tyler Rolfe.  Discrete variational autoencoders.  In ICLR, 2017.
  * [30] Shaoshuai Shi, Li Jiang, Jiajun Deng, Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hongsheng Li.  PV-RCNN++: point-voxel feature set abstraction with local vector representation for 3d object detection.  arXiv preprint arXiv: 2102.00463, 2021.
  * [31] Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li.  From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network.  IEEE TPAMI, 2020.
  * [32] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov.  Scalability in perception for autonomous driving: Waymo open dataset.  In CVPR, 2020.
  * [33] OpenPCDet Development Team.  Openpcdet: An open-source toolbox for 3d object detection from point clouds.  https://github.com/open-mmlab/OpenPCDet, 2020.
  * [34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [35] Hanchen Wang, Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt J. Kusner.  Unsupervised point cloud pre-training via occlusion completion.  In ICCV, 2021.
  * [36] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao.  3d shapenets: A deep representation for volumetric shapes.  In CVPR, 2015.
  * [37] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas J Guibas, and Or Litany.  Pointcontrast: Unsupervised pre-training for 3d point cloud understanding.  In ECCV, 2020.
  * [38] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu.  Simmim: a simple framework for masked image modeling.  In CVPR, 2022.
  * [39] Yan Yan, Yuxing Mao, and Bo Li.  Second: Sparsely embedded convolutional detection.  Sensors, 2018.
  * [40] Junbo Yin, Dingfu Zhou, Liangjun Zhang, Jin Fang, Cheng-Zhong Xu, Jianbing Shen, and Wenguan Wang.  Proposalcontrast: Unsupervised pre-training for lidar-based 3d object detection.  In ECCV, 2022.
  * [41] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl.  Center-based 3d object detection and tracking.  In CVPR, 2021.
  * [42] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu.  Point-bert: Pre-training 3d point cloud transformers with masked point modeling.  In CVPR, 2022.
  * [43] Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Trung X. Pham, Chang D. Yoo, and In So Kweon.  How does simsiam avoid collapse without negative samples? A unified understanding with self-supervised contrastive learning.  In ICLR, 2022.
  * [44] Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li.  Point-m2ae: Multi-scale masked autoencoders for hierarchical point cloud pre-training.  arXiv preprint arXiv: 2205.14401, 2022.
  * [45] Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra.  Self-supervised pretraining of 3d features on any point-cloud.  In ICCV, 2021.
  * [46] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong.  ibot: Image bert pre-training with online tokenizer.  In ICLR, 2022.
  * [47] Yin Zhou and Oncel Tuzel.  Voxelnet: End-to-end learning for point cloud based 3d object detection.  In CVPR, 2018.