# Cohere3D: Exploiting Temporal Coherence for Unsupervised Representation
Learning of Vision-based Autonomous Driving

Yichen Xie1 , Hongge Chen2, Gregory P. Meyer2, Yong Jae Lee2, Eric M. Wolff2,
Masayoshi Tomizuka1, Wei Zhan1, Yuning Chai2, Xin Huang2
1University of California, Berkeley 2Cruise LLC
This work was done when Yichen was an intern at Cruise LLC.

###### Abstract

Due to the lack of depth cues in images, multi-frame inputs are important for
the success of vision-based perception, prediction, and planning in autonomous
driving. Observations from different angles enable the recovery of 3D object
states from 2D image inputs if we can identify the same instance in different
input frames. However, the dynamic nature of autonomous driving scenes leads
to significant changes in the appearance and shape of each instance captured
by the camera at different time steps. To this end, we propose a novel
contrastive learning algorithm, Cohere3D, to learn coherent instance
representations in a long-term input sequence robust to the change in distance
and perspective. The learned representation aids in instance-level
correspondence across multiple input frames in downstream tasks. In the
pretraining stage, the raw point clouds from LiDAR sensors are utilized to
construct the long-term temporal correspondence for each instance, which
serves as guidance for the extraction of instance-level representation from
the vision-based bird’s eye-view (BEV) feature map. Cohere3D encourages a
consistent representation for the same instance at different frames but
distinguishes between representations of different instances. We evaluate our
algorithm by finetuning the pretrained model on various downstream perception,
prediction, and planning tasks. Results show a notable improvement in both
data efficiency and task performance.

##  1 Introduction

Recent work in vision-based autonomous driving has demonstrated promising
performance in 3D detection [19, 27], map reconstruction [42, 30], prediction
[10, 22], and planning [17, 23]. Despite the rich semantics provided by
images, the absence of depth information poses a critical challenge in
recovering 3D geometries from a single-frame camera input. Multi-frame inputs
aggregate the observation of the same instances (or regions) from different
perspectives at different time steps through correspondence matching to
determine their 3D states. However, the dynamic nature of autonomous driving
scenes complicates the extraction of reliable instance-level information
across temporal frames. The varying relative positions of objects to the
cameras lead to significant alternations in their appearance within the
images, presenting a big hurdle in maintaining consistent instance
identification over time. This inconsistency complicates finding an instance-
level correspondence across multiple temporal frames, which serves as a
fundamental barrier to effective fusion of multi-frame inputs. Thus, coherent
instance-wise representations that are robust to the change of observation
viewpoint and distance are critical for effective temporal fusion.

Figure 1: Temporal coherence of instance representations. Our algorithm
encourages instance representations in the BEV space to be coherent across
time and viewpoints.

Unfortunately, fine-grained annotations are needed to learn this robust
representation, since all the objects and their temporal correspondences have
to be identified. Given the immense human effort required for labeling, only a
small portion of the sensor data collected by self-driving cars can be
annotated. To this end, unsupervised representation learning provides a
practical solution for exploiting all of the unlabeled data.

For this reason, we propose a novel contrastive learning method  _Cohere3D_
designed to produce long-term coherent instance representations (Fig. 1) for
vision-based perception, prediction, and planning tasks. Our approach
leverages the richness of visual data while circumventing the constraints
posed by the lack of depth information. Utilizing commonly available raw LiDAR
point clouds in the 3D space, our method constructs a long-term temporal
correspondence for each instance. This correspondence serves as a foundation
for extracting instance-level representations from vision-based bird’s eye-
view (BEV) feature maps. It thus enhances the model’s ability to maintain
instance representation consistency across frames.

Our method focuses on the extraction and learning of long-term temporal
correspondences, a critical aspect, but often overlooked in previous
representation learning methods for autonomous driving tasks. By aligning
representations of the same instance across multiple frames and
differentiating between distinct instances, our framework achieves a
significant improvement in both data efficiency and overall performance in
downstream vision-based tasks such as 3D object detection, map reconstruction,
motion prediction, and end-to-end planning.

The main contributions of our paper are:

  1. 1.

We identify long-term instance representation coherence as a key component for
successful vision-based 3D representation learning, which was not well-studied
by prior works.

  2. 2.

We introduce a simple yet effective contrastive learning framework, Cohere3D,
tailored for vision-based autonomous driving tasks, emphasizing the robust
learning of long-term temporal correspondences.

  3. 3.

We demonstrate superior performance and data efficiency of our pretraining
algorithm across vision-based 3D perception, prediction, and planning tasks.

##  2 Related Works

#### Camera-based Autonomous Driving

Cameras are cheap and ubiquitous sensors that are often used in autonomous
driving. Due to the limited field-of-view of a single camera, the surrounding
environment is often captured by multiple cameras from different perspective
views. To combine multi-view information in 3D perception tasks, prior works
[19, 27, 34] apply an LSS-style [40] projection strategy to transform image
features from different perspective views into a unified bird’s eye-view
space, while others [51, 33, 53] utilize a set of 3D queries to learn the
inter-view correspondence with transformer-based networks [5]. However,
without overlapping camera fields-of-views in autonomous driving scenarios,
single-frame camera inputs lack reliable depth cues. To this end, a few
camera-based perception algorithms [18, 29] exploit several neighboring frames
to recover the 3D geometry. SOLOFusion [39] takes a step further by
considering long-term temporal information, which allows camera-based methods
[48, 31] to achieve performance competitive to LiDAR-based methods [26].
Beyond perception tasks, recent work also develops end-to-end pipelines based
on camera inputs. ViP3D [10] and PIP [22] directly predict motion trajectories
using visual information. UniAD [17] and VAD [23] apply a joint training
strategy to assist end-to-end planning.

High-quality 3D annotations are essential for all the above algorithms.
Despite recent efforts on auto-labeling [41, 56] or sim-to-real transfer [7]
to provide meaningful labels, the offboard perception system is expensive to
build and the domain transfer gap still remains an open challenge. Orthogonal
to these works, we propose a novel unsupervised representation learning
algorithm that makes full use of the abundant unlabeled sensor data.

Figure 2: Overview of Cohere3D. We construct long-term instance-level temporal
correspondence via unlabeled LiDAR point clouds, and design a contrastive
learning framework to encourage the long-term coherence of instance
representations. After pretraining, the online network is transferred to
downstream task-specific models.

#### Pretraining for Perception and Prediction

Unsupervised learning algorithms [14, 9, 15, 1, 54] learn expressive
representation without any human annotations. Some pioneering research adapts
contrastive learning [52, 20] or masked autoencoder [38, 58] frameworks to 3D
point clouds data, but most of them are constrained to a single-object or
indoor data. To handle large outdoor scenes in autonomous driving scenarios,
ProposalContrast [57] devises a region-level contrastive learning algorithm.
BEV-MAE [32] and GeoMAE [44] follow MAE [15] to predict the missed information
in the BEV space. Despite the increasing popularity of cameras in autonomous
driving, most methods only consider point clouds as input. Occ-BEV [36]
predicts the occupancy of each voxel in the 3D space with image inputs.
However, due to its lack of semantics and temporal cues, this method provides
limited benefits to complex perception tasks. For prediction tasks, previous
works [28, 55] propose pretraining methods for the traditional two-stage
motion prediction pipeline, which requires manually annotated historical
trajectories and semantic maps as inputs.

In contrast to these works, we emphasize the long-term temporal coherence of
instance-level representations and propose a novel unsupervised learning
algorithm targeted for complex vision-based autonomous driving tasks.

#### Representation Learning from Temporal Cues

Previous research [49, 12, 21, 45, 46, 11] uses the temporal information
inside videos for unsupervised representation learning. In particular, several
papers share common insights with us to develop contrastive learning
algorithms to utilize the fine-grained temporal correspondence. They construct
the temporal correspondence using visual features [49], optical flow [12], or
cycle-consistency [21]. However, they are limited to 2D tasks for image or
video processing. TARL [37] utilizes the temporal cues for the pretraining of
LiDAR-based 3D perception models, but their clustering algorithm can only
retrieve short-term temporal correspondences. In contrast, our algorithm
targets the long-term temporal correspondence to learn a temporally coherent
instance representation for vision-based 3D tasks.

##  3 Methodology

In this section, we introduce our novel unsupervised representation learning
algorithm Cohere3D, tailored for vision-based autonomous driving tasks. Our
approach utilizes unlabeled multi-view images as the primary input.
Additionally, during the pretraining phase, we incorporate raw LiDAR point
clouds as supplementary guidance. It is important to note that these point
clouds are not required during the inference stage of the downstream tasks.

The methodology is structured into several key components: First, we lay out a
straightforward formulation of our method in Sec. 3.1. Next, Sec. 3.2 details
how we construct temporal correspondences, which play a critical role in
guiding the extraction of the bird’s eye-view (BEV) features, as elaborated in
Sec. 3.3. Our approach is underpinned by a contrastive learning framework,
which we describe in Sec. 3.4. This framework employs an online network
alongside a momentum-updated target network, aimed at learning robust
representations that are temporally coherent. An overview of Cohere3D is shown
in Fig. 2.

###  3.1 Formulation

Vision-based perception algorithms often resort to several past frames of
input images to detect the object in 3D space. In essence, this process can be
formulated as a form of structure from motion (SfM) that recovers the 3D state
of an instance based on its observation from multiple perspective views at
different time steps:

$\mathbf{O}^{3D,t}_{m}=g\left(\left\\{h\left(\mathbf{O}^{2D,t-k}_{m}\right)\right\\}_{k=0}^{K}\right),$
(1)

where $\mathbf{O}^{3D,t}_{m}$ is the 3D state of instance $m$ at time step $t$
including its 3D size and pose, $\mathbf{O}^{2D,t}_{m}$ is the 2D visual
pattern of the instance $m$ appearing on the image inputs $\mathcal{I}^{t}$ at
frame $t$, $h(\cdot)$ encodes the 2D visual patterns at each time step
separately, and $g(\cdot)$ recovers the 3D object states by combining multi-
frame 2D information.

In this case, it is necessary to find the instance-wise correspondence
$\left\\{\mathbf{O}^{2D,t-k}_{m}\right\\}_{k=0}^{K}$, i.e. we need to identify
the same instance $m$ across different time steps. Due to the motion of the
ego-vehicle and surrounding scenes, the shape and appearance of each instance
in a 2D image can vary significantly at different time steps. Given a set of
instances $\left\\{\mathbf{O}^{2D,t}_{m}\right\\}_{m=1}^{M}$ at each frame
$t$, it is difficult to correlate each of them with the instances at other
frames across a long-term temporal sequence.

To ensure the effective fusion of multi-frame inputs, we should encourage the
model $h(\cdot)$ to learn coherent instance-wise features robust to temporal
changes, facilitating the instance-wise temporal correspondence in downstream
tasks. At each frame $t$, the image features are projected into the BEV space
as $\mathbf{F}_{BEV}^{t}$ [40]. We extract the instance representation
$\mathbf{f}^{t}_{m}$ corresponding to each instance $\mathbf{O}^{2D,t}_{m}$
from the BEV feature map. The goal of our approach is to make the same
instance representation $\\{\mathbf{f}^{t-k}_{m}\\}_{k=0}^{K}$ consistent in a
long sequence and make the representations of different objects
$\\{\mathbf{f}^{t}_{m}\\}_{m=1}^{M}$ distinct from each other. Concretely, we
compute the temporal average representation $\bar{\mathbf{f}}_{m}$ for each
instance $m$ over multiple historical frames as follows:

$\bar{\mathbf{f}}_{m}=\frac{1}{K+1}\sum_{k=0}^{K}\mathbf{f}^{t-k}_{m}.$ (2)

Our representation encourages the instance representation $\mathbf{f}^{t}_{m}$
to be near the temporal average representation $\bar{\mathbf{f}}_{m}$ and
distinct from other instance representations $\bar{\mathbf{f}}_{n},n\neq m$.

Figure 3: Construction of long-term temporal correspondence. The first and
last scans corresponding to each frame are correlated by clustering. The
Hungarian algorithm connects every two neighboring frames to generate long-
term correspondence.

###  3.2 LiDAR-guided Temporal Correspondence

Given the varying appearance of each instance at different input frames, it is
difficult to construct the instance-level temporal correspondence from pure
vision inputs without any explicit human annotations. Alternatively, we build
the long-term temporal correspondence using raw LiDAR point clouds
$\mathcal{P}^{t}$. The LiDAR sensor can have a higher frequency than the
cameras, so there may be multiple LiDAR scans within the small interval
between two camera frames. For the camera image captured at time step $t$,
which we refer to as frame $t$, we collect all LiDAR scans captured between
$t-1$ and $t$, to create the point clouds for frame $t$, denoted as
$\mathcal{P}^{t}$. We apply a two-stage strategy to retrieve the long-term
temporal correspondence (Fig. 3). In the first stage, we identify different
instances from the LiDAR scans corresponding to frame $t$. In the second
stage, the inter-frame instance-wise correspondences are generated by matching
the identified instances in the first stage.

#### Instance Identification

We apply the method in [16] to remove the ground points (black points in Fig.
3) from $\mathcal{P}^{t}$ to get the non-ground point clouds
$\mathcal{\hat{P}}^{t}$. The non-ground point clouds $\mathcal{\hat{P}}^{t}$
are clustered with HDBSCAN algorithm [4] into several clusters
$\mathcal{\hat{P}}^{t}=\bigcup_{m=1}^{M}\mathcal{C}^{t}_{m}$, where
$\mathcal{C}^{t}_{m}$ includes points from multiple LiDAR scans between time
step $t-1$ and $t$. We consider the points from the LiDAR scans immediately
after time step $t-1$ and immediately before $t$, denoted as
$\mathcal{C}^{t,s}_{m}$ and $\mathcal{C}^{t,e}_{m}$. We discard invalid
clusters $m$, whose number of points is fewer than a threshold $\tau_{n}$ in
either the first scan $\mathcal{C}^{t,s}_{m}$ or the last scan
$\mathcal{C}^{t,e}_{m}$. Each cluster $\mathcal{C}^{t}_{m}$ is considered as a
single instance. It is likely to have different center locations in the first
and last LiDAR scans, denoted as $\mathbf{c}^{t,s}_{m}$ and
$\mathbf{c}^{t,e}_{m}$, which indicates the intra-frame motion of this
instance in the short term between the first and last LiDAR scans.

#### Long-term Matching

Given the instance-wise motions from $\\{\mathbf{c}^{t,s}_{m}\\}_{m=1}^{M}$ to
$\\{\mathbf{c}^{t,e}_{m}\\}_{m=1}^{M}$ in a single frame $t$, we connect them
to the previous frame $t-1$ by finding the correspondence between
$\\{\mathbf{c}^{t,s}_{m}\\}_{m=1}^{M}$ at the beginning of frame $t$ and the
instance centers $\\{\mathbf{c}^{t-1,e}_{m}\\}_{m=1}^{M}$ at the end of the
frame $t-1$. Note that point clouds corresponding to different frames are in
different coordinate systems due to the motion of the ego-vehicle. Given the
rotation $\mathbf{R}$ and translation $\mathbf{p}$ of the ego-car between
frame $t-1$ and $t$, the instance centers
$\\{\mathbf{c}^{t-1,e}_{m}\\}_{m=1}^{M}$ should be transformed into the
coordinate of frame $t$ as follows:

$\mathbf{\hat{c}}^{t-1,e}_{m}=\mathbf{R}^{T}\mathbf{c}^{t-1,e}_{m}-\mathbf{R}^{T}\mathbf{p}.$
(3)

The Hungarian algorithm [25] is then used to find the correspondence between
$\\{\mathbf{c}^{t,s}_{m}\\}_{m=1}^{M}$ and
$\\{\mathbf{\hat{c}}^{t-1,e}_{m}\\}_{m=1}^{M}$. To handle the disappearance or
appearance of instances, we pad empty instances with maximal distance to all
the other instances. We set a threshold of $\tau_{d}=0.5m$ for the distance
between the matched centers, where the valid matching should satisfy
$||\mathbf{\hat{c}}^{t-1,e}_{m}-\mathbf{c}^{t,s}_{m}||_{2}\leq\tau_{d}$. This
threshold can tolerate the possible object motion over a small interval
($<0.05s$) between the two scans, while avoiding the mismatch between
different instances. The instance centers
$\\{\mathbf{\hat{c}}^{t,e}_{m}\\}_{m}$ from the last LiDAR scan of frame $t$
will then be used to connect frame $t$ and $t+1$. In this way, we combine
several intra-frame short-term instance trajectories to achieve long-term
trajectories $\\{\mathcal{C}_{m}^{t-k}\\}_{k\leq K}$ for each instance $m$ by
connecting $\mathbf{\hat{c}}^{t-k-1,e}_{m}$ and $\mathbf{c}^{t-k,s}_{m}$ frame
by frame. Since different instances may have different life cycles, we set $K$
to be the maximal number of historical frames.

###  3.3 Temporal Instance Feature

For each frame $t$, we sample $N_{F}$ points uniformly on the BEV space from
all the clusters $\left\\{\mathcal{C}_{m}^{t}\right\\}_{m=1}^{M}$ as
$\left\\{\mathcal{S}_{m}^{t}\right\\}_{m=1}^{M}$ where
$\mathcal{S}_{m}^{t}\subseteq\mathcal{C}_{m}^{t}$ and
$\sum_{m=1}^{M}|\mathcal{S}_{m}^{t}|=N_{F}$. Given the vision-based BEV
feature map $\mathbf{F}_{BEV}^{t}$, we apply bilinear sampling to extract the
features $\mathbf{f}_{m,j}^{t}$ for each sampled point
$\mathbf{s}_{m,j}^{t}\in\mathcal{S}_{m}^{t}$. At each frame $t$, we compute
the instance-level representation by averaging the point-level features
belonging to the same instance, i.e.,

$\mathbf{f}_{m}^{t}=\frac{1}{N_{m}}\sum_{j=1}^{N_{m}}\mathbf{f}_{m,j}^{t},$
(4)

where $N_{m}$ is the number of sampled points belonging to the instance $m$.
For each instance, we maintain a memory bank
$\mathcal{B}_{m}=[\mathbf{f}_{m}^{t-k}]_{k\leq K}$ for its instance
representation at different time steps. The memory bank for each instance is
created at the time step of its first appearance and will be updated at each
following time step as long as this instance is matched at frame $t$. Next,
the temporal average representation $\bar{\mathbf{f}}_{m}$ of instance $m$ is
calculated following Eq. 2 based on the instance representation in the memory
bank $\mathcal{B}_{m}$.

However, the foreground objects only account for a small part of the input
images. To exploit the semantics from the whole image, we also sample $N_{B}$
background points uniformly on the BEV space not occupied by any valid
clusters. Their corresponding features are extracted as
$\\{\mathbf{f}_{b,l}^{t}\\}_{l=1,2,\dots,N_{B}}$. Due to their static nature,
the background representations are more stable. Therefore, we skip the
temporal averaging operation.

###  3.4 Depth-aware Contrastive Learning

After obtaining instance-level features through LiDAR-guided temporal
correspondence, we apply constrastive learning to encourage a consistent
representation for the same instance. On top of the standard Siamese network
framework, we leverage a few techniques to ensure the stability and robustness
of the unsupervised pretraining, as detailed in this section.

#### Architecture

We apply a Siamese network framework [9] with an online network $f(\cdot)$ and
a target network $\hat{f}(\cdot)$. Both networks include an image backbone to
encode the input images, a view transformation module to project the image
features into the BEV space, and a BEV encoder to encode the BEV features. The
online network is updated by gradient descent, while the target network uses
the exponential moving average (EMA) of the online network. For the input
image $\mathcal{I}^{t}$ for frame $t$, we generate two different augmented
versions $a^{o}(\mathcal{I}^{t})$ and $a^{t}(\mathcal{I}^{t})$ with a set of
spatial augmentations $\mathcal{A}$, $a^{o},a^{t}\in\mathcal{A}$. The
augmented versions are fed into the online and target networks separately. The
online and target networks output the BEV features $\mathbf{F}^{t}_{BEV}$ and
$\mathbf{\hat{F}}^{t}_{BEV}$, where

$\mathbf{F}^{t}_{BEV}=f(a^{o}(\mathcal{I}^{t})),\quad\mathbf{\hat{F}}^{t}_{BEV}=\hat{f}(a^{t}(\mathcal{I}^{t})).$
(5)

#### Depth-aware Representation Learning

Stable representations in the target network play an important role in
contrastive learning [50]. However, due to the depth ambiguity of 2D inputs,
it is quite unreliable to transform the image features from the perspective
view to the BEV space. Many camera-based 3D perception models [40, 19, 39]
project the image features with the estimated depth distribution
$\mathbf{d}_{hw}=\left[p_{hw}^{1},p_{hw}^{2},\dots,p_{hw}^{D}\right]$ for each
pixel $(h,w)$ w.r.t. a set of discrete depths defined by
$[d_{0}+\Delta,\dots,d_{0}+D\Delta]$ and $\sum_{i=1}^{D}p_{hw}^{i}=1$. To get
more reliable supervision in the pretraining stage, we combine the ground-
truth depth distribution $\hat{\mathbf{d}}_{hw}$ from the LiDAR sensor with
the estimated depth distribution $\mathbf{d}_{hw}$, where
$\hat{\mathbf{d}}_{hw}=\left[\hat{p}_{hw}^{1},\hat{p}_{hw}^{2},\dots,\hat{p}_{hw}^{D}\right]$
is a one-hot distribution with only $\hat{p}_{hw}^{k}=1$ if
$d_{0}+k\cdot\Delta$ is the ground-truth depth. We name this index to $k_{GT}$
and merge the ground-truth depth and estimated depths by setting
$p_{hw}^{k_{GT}}=1$ in $\mathbf{d}_{hw}$ and then re-normalizing the merged
depth distribution to ensure $\sum_{i=1}^{D}p_{hw}^{i}=1$. We only merge the
estimated and ground-truth depth for the target network while the online
network only uses the estimated depth distribution. As a result, the target
network can achieve much better positional precision in the BEV space,
enabling a more reliable self-supervision in the contrastive learning
framework.

#### View Transformation Mask

We add a dropout mask in the view transformation process by setting a
percentage, $r$, of the estimated depth distribution
$\left[p_{hw}^{1},p_{hw}^{2},\dots,p_{hw}^{D}\right]$ to zero. This dropout
strategy is only adopted to the online network, since the target network
prefers a more stable representation for self-supervision. In this way, we
encourage the network to learn stronger BEV representation to compensate for
the missing information.

#### Contrastive Loss Function

The basic goal of our unsupervised learning is to encourage the instance-wise
representation to be invariant to temporal changes. We extract the point-level
features at the current time step $t$ from the BEV features of online network
and target network respectively as in Sec. 3.3. The temporal average feature
$\hat{\bar{\mathbf{f}}}_{m}$ of each instance $m$ is derived from the target
network point-level features $\hat{\mathbf{f}}_{m,j}^{t}$ for each sampled
point $\mathbf{s}^{t}_{m,j}$ following Eq. 4 and Eq. 2. We get inspiration
from [43] to apply a point-to-instance contrastive loss as follows:

$\mathcal{L}=\frac{1}{N_{F}}\underset{\mathbf{s}_{m,j}^{t}\in\mathcal{S}_{m}^{t}}{\sum}\frac{\exp(\mathbf{f}_{m,j}^{tT}\hat{\bar{\mathbf{f}}}_{m}/\tau)}{\sum_{n=1}^{M}\exp(\mathbf{f}_{m,j}^{tT}\hat{\bar{\mathbf{f}}}_{n}/\tau)+\sum_{l=1}^{N_{B}}\exp(\mathbf{f}_{m,j}^{tT}\hat{\mathbf{f}}_{b,l}^{t}/\tau)},$

(6)

where $\tau$ is the temperature set as $0.1$ and all the features are
normalized. This loss will make the local features belonging to the same
instances close to each other across the long-term temporal sequence and keep
them distant from other instance features as well as background features.

##  4 Experiments

This section presents a comprehensive evaluation of our proposed method. We
begin by detailing the experimental setup, including dataset, and
pretraining/finetuning setups in Sec. 4.1. Our approach is then compared
against established baselines to demonstrate its effectiveness in vision-based
autonomous driving tasks. Specifically, we evaluate our method on 3D object
detection (Sec. 4.2), HD map construction (Sec. 4.3), motion prediction (Sec.
4.4), and planning (Sec. 4.5). Additionally, we provide an analysis of our
algorithm in Sec. 4.6 and ablation studies in Sec. 4.7.

###  4.1 Implementation Details

#### Dataset

We use the nuScenes dataset [3] for both pretraining and finetuning. It is the
most common benchmark for vision-based autonomous driving tasks, including
700/150/150 scenes for training/validation/test. It provides point clouds
collected by a 32-beam LiDAR and multi-view images from six cameras. For
pretraining, we only use the raw point clouds and images from the training set
without any annotations.

#### Unsupervised Pretraining

In the pretraining stage, we follow the network architecture of short-term
SOLOFusion [39] (ResNet50 backbone [13]) without detection heads. Different
from 2D representation learning [14, 9], strong data augmentation has
potential negative effects in our setting, bringing ambiguity in 3D geometry
recovery from images [2]. Thus, we only apply weak data augmentations same as
[39] to the inputs, including random crops, scaling, and rotations of the
images, as well as random rotation and scaling for the BEV representation. We
sample $N_{F}=N_{B}=1000$ foreground and background points from the BEV space
at each frame. The momentum of the target network is set as $0.99$. To learn
the long-term representation coherence, we use at most $K=16$ (8 seconds)
historical frames. A dropout percentage of $r=0.3$ is adopted in the view
transformation of the online network. We pretrain the model for 24 epochs
using Adam optimizer [24] with a batch size of 32 and a learning rate of 1e-4
on four NVIDIA A100 40GB GPUs.

#### Supervised Finetuning

We initialize the network for different tasks with our pretrained weight in
the supervised finetuning stage. We follow the same training strategies as the
from-scratch training for each task model.

Table 1: 3D object detection using SOLOFusion [39] on nuScenes

(a) Short-term model variant Finetune Percent | Pretraining Strategy | NDS | mAP
---|---|---|---
$10\%$ | ImageNet [39] | 21.3 | 15.6
Cohere3D (ours) | 24.3 | 19.4
$20\%$ | ImageNet [39] | 24.3 | 20.0
Cohere3D (ours) | 25.8 | 22.3
$50\%$ | ImageNet [39] | 30.8 | 28.1
Cohere3D (ours) | 32.4 | 29.1
$100\%$ | ImageNet [39] | 39.1 | 34.4
BYOL [9] | 37.2 | 33.2
MonoDepth2 [8] | 36.4 | 32.1
Occ-BEV [36] | 39.0 | 34.3
Cohere3D (ours) | 41.3 | 35.1

(b) Long-term model variant Finetune Percent | Pretraining Strategy | NDS | mAP
---|---|---|---
$10\%$ | ImageNet [39] | 20.6 | 13.2
Cohere3D (ours) | 22.9 | 16.4
$20\%$ | ImageNet [39] | 28.5 | 23.1
Cohere3D (ours) | 32.3 | 25.0
$50\%$ | ImageNet [39] | 40.9 | 32.7
Cohere3D (ours) | 43.1 | 33.6
$100\%$ | ImageNet [39] | 49.7 | 40.6
BYOL [9] | 48.9 | 39.7
MonoDepth2 [8] | 47.3 | 37.4
Occ-BEV [36] | 49.2 | 40.4
Cohere3D (ours) | 50.2 | 40.8

Table 2: 3D object detection using BEVDet [19] on nuScenes Finetune Percent | Pretraining Strategy | NDS | mAP
---|---|---|---
$10\%$ | ImageNet [19] | 11.7 | 8.0
Cohere3D (ours) | 16.2 | 10.8
$100\%$ | ImageNet [19] | 35.0 | 28.3
Cohere3D (ours) | 36.1 | 28.6

###  4.2 Object Detection

To verify the effectiveness of our method on 3D perception tasks, we finetune
a vision-based detector, SOLOFusion [39], for camera-based 3D object
detection, using our pretrained weight. We consider both short-term and long-
term variants of SOLOFusion model, where the former only considers one
neighbor frame but the latter resorts to the past 16 frames. More
specifically, we finetune the model using $10\%,20\%,50\%,100\%$ of the
training set separately, and report the mAP and NDS metrics [3] on the
validation set. The results are reported in Tab. 1.

The most important advantage of unsupervised learning is to improve data
efficiency with limited annotation by utilizing other unlabeled data. Not
surprisingly, for both short-term and long-term variants, Cohere3D
significantly outperforms the from-scratch training baseline using ImageNet
weights, especially when only a subset of the training data is used for
finetuning. In addition, the results show that the pretraining can provide
performance gains even when finetuning the models on the entire training set.
In this case, the pretraining does not include extra information but utilizes
the inherent temporal cues inside the training data. Our pretraining also
outperforms several other unsupervised learning baselines, including BYOL [9],
a 2D contrastive learning algorithm that does not take the 3D geometry into
account, and MonoDepth2 [8] and Occ-BEV [36] that consider geometric
information but ignore semantics.

The representation learned in the pretraining stage is agnostic to different
network architectures. For example, we transfer our pretrained backbone to
another 3D object detection model, BEVDet [19]. Tab. 2 shows significant
performance improvement brought by our pretraining when we finetune the model
on both a subset and the full training set.

###  4.3 HD Map Construction

Apart from detecting foreground objects, we examine our learned representation
on background detection through an HD map construction task. We apply MapTR
[30] as the task model, and finetune it on the full training set using our
pretrained weights. Our pretraining improves the mAP metric from $50.3$ to
$51.2$ in comparison to ImageNet pretraining. This improvement indicates that
Cohere3D can also help the model to better understand the background scene
such as map lanes.

Table 3: Motion prediction using VAD-tiny [23] on nuScenes Pretraining Strategy | Perception | Prediction
---|---|---
NDS | mAP |  EPA (car) |  EPA (pedestrian)
ImageNet [23] | 39.0 | 27.0 | 59.8 | 29.0
Cohere3D (ours) | 41.4 | 27.9 | 63.1 | 33.1

###  4.4 Motion Prediction

Learning a coherent instance representation over time not only helps to
determine the current 3D states of each instance, but also leads to stronger
temporal reasoning, such as predicting the future motion of these instances.
To verify this hypothesis, we apply a state-of-the-art vision-based prediction
and planning model, VAD [23], as the task model for camera-based motion
prediction task. We transfer our pretrained backbone weights as its
initialization. We finetune the VAD-tiny model on nuScenes using the full
train set. This model outputs both perception and end-to-end motion prediction
results. The perception performance is measured by mAP and NDS, while the
prediction performance is evaluated using the End-to-end Prediction Accuracy
(EPA) [10] metric. The results are shown in Tab. 3. On the perception metrics,
our pretraining leads to performance gain compared to the baseline, which is
consistent with our observations in Sec. 4.2. Also, the end-to-end motion
prediction benefits from our pretraining method, which encourages instance-
level coherence over long-term historical frames. As a result, without using
any extra data, Cohere3D greatly lifts the performance of the vision-based
end-to-end motion prediction task.

Table 4: End-to-end Planning using VAD-tiny [23] on nuScenes

Pretraining Strategy |  $L_{2}$ (m) | Collision (%)
---|---|---
$1s$ | $2s$ | $3s$ | $1s$ | $2s$ | $3s$
ImageNet [23] | 0.46 | 0.76 | 1.12 | 0.21 | 0.35 | 0.58
Cohere3D (ours) | 0.28 | 0.52 | 0.84 | 0.11 | 0.26 | 0.56

###  4.5 End-to-end Planning

Our pretraining benefits the downstream task models in predicting the current
(Sec. 4.2) and future (Sec. 4.4) 3D states of surrounding environments. This
improved scene understanding also facilitates the planning of the ego-vehicle.
We apply VAD-tiny [23] as the baseline for the end-to-end planning task. The
performance is measured by the $L_{2}$ error and collision rates of the future
trajectory. Results in Tab. 4 demonstrate that Cohere3D significantly boosts
the end-to-end planning performance, through better estimates of the current
and future states of surrounding agents.

Figure 4: Qualitative comparison of SOLOFusion model trained from scratch
(left) vs with pretraining (right). Our pretraining is effective at improving
recall rates, reducing false positives, and localizing objects in the 3D
space.

###  4.6 Analysis

Qualitative Results In Fig. 4, we show some qualitative 3D object detection
results using SOLOFusion [39] model. Our pretraining leads to some notable
improvements. First, our pretraining provides instance representation robust
to the observation perspective views. This helps the detector to recall more
objects. Second, our contrastive learning framework samples both foreground
and background features to help learn to distinguish between the objects and
background clutter. This reduces the false positive results in the downstream
detection task. Finally, our temporally coherent representation benefits the
multi-frame temporal fusion to achieve better 3D object pose estimation.

Convergence Speed Fig. 5 shows the performance of short-term SOLOFusion [39]
model at different training epochs during finetuning. Our pretraining leads to
superior performance in the early finetuning stage than the from-scratch
training. Even though we do not use any labels during pretraining, our
unsupervised learning algorithm indeed learns meaningful feature
representation directly transferable to the downstream task.

Figure 5: Performance at different finetuning epochs. Our pretraining leads to
faster convergence.

LiDAR-Guided Long-term Correspondence Our pretraining algorithm emphasizes the
long-term coherence of instance representation. This representation quality
heavily depends on the accuracy of our long-term correspondence constructed in
the pretraining stage. Fig. 6 provides some examples of long-term
correspondence guided by the LiDAR point clouds. We find that our method can
retrieve reliable instance trajectories without any explicit human annotations
over multiple historical frames for those dynamic instances. This provides a
solid foundation for our contrastive learning over temporal cues.

Figure 6: LiDAR-guided long-term correspondence. It can reliably track each
instance over long-term historical frames even if it is captured by different
cameras from varying angles. Readers may zoom in to see the projected point
clouds clearly.

###  4.7 Ablation Studies

In this part, we justify our design by examining the role of each module in
our pretraining framework. We first highlight the contribution of our key
component, long-term temporal coherence over instance representations. Then,
we discuss the role of each design decision in the contrastive learning
framework. All the experiments in this section use the short-term version
SOLOFusion [39] on 3D object detection task. The model is finetuned on the
full nuScenes training set.

#### Long-term Temporal Coherence

Table 5: Ablation study of long-term temporal coherence Historical Frame | NDS | mAP
---|---|---
0 | 39.9 | 34.8
1 | 40.1 | 34.6
4 | 41.1 | 34.9
16 | 41.3 | 35.1

Our algorithm concentrates on the extraction and learning of long-term
instance-wise temporal correspondence. In the pretraining stage, we construct
a memory bank for the historical instance features to encourage the temporal
coherence of instance representations. We tried different numbers of
historical frames in Tab. 5. Without any temporal information, a significant
performance drop is observed. This verifies that long-term temporal coherence
is crucial for the downstream task. Results also show that more historical
frames can lead to stronger representation and bring better performance to the
downstream task.

#### Contrastive Learning Modules

We also examine the role of other modules in our contrastive learning
framework, as summarized in Tab. 6. The results show that it is important to
merge the ground-truth depth with the estimated depth in the view
transformation module of the target network (Sec. 3.4). Since the depth
estimation from image inputs is fragile, the ground-truth depth from the LiDAR
point clouds can notably reduce the positional ambiguity in the target
network’s BEV features. As a result, this leads to a more stable target in the
contrastive learning framework. Secondly, the extraction of background point-
level features (Sec. 3.3) makes full use of semantics in the input images
outside of the foreground objects. It also helps to distinguish the foreground
from the background. Finally, extra dropout in the LSS module [40] for view
transformation (Sec. 3.4) results in stronger representation. Therefore, all
the above design decisions contribute meaningfully to the final performance of
our algorithm.

Table 6: Ablation study of contrastive learning modules Background Points |  Merged Depth |  LSS Mask | NDS | mAP
---|---|---|---|---
| ✓ | ✓ | 39.8 | 35.0
✓ |  | ✓ | 38.3 | 34.8
✓ | ✓ |  | 40.2 | 34.7
✓ | ✓ | ✓ | 41.3 | 35.1

##  5 Conclusions

We propose a novel contrastive learning algorithm, Cohere3D, for unsupervised
representation learning of vision-based 3D autonomous driving tasks. The
algorithm encourages the long-term coherence of instance features across
multiple frames, effectively and robustly handling changes in observation
viewpoints. By utilizing unlabeled LiDAR point clouds to establish long-term
instance-level correspondences, Cohere3D enhances the feature sampling from
the BEV space. Through comprehensive experiments, we show that our algorithm
not only improves data efficiency, but also significantly boosts the final
performance of downstream 3D perception, prediction, and planning tasks.

## References

  * Bai et al. [2022] Yutong Bai, Xinlei Chen, Alexander Kirillov, Alan Yuille, and Alexander C Berg.  Point-level region contrast for object detection pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16061–16070, 2022.
  * Brazil et al. [2023] Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari.  Omni3d: A large benchmark and model for 3d object detection in the wild.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13154–13164, 2023.
  * Caesar et al. [2020] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 11621–11631, 2020.
  * Campello et al. [2013] Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander.  Density-based clustering based on hierarchical density estimates.  In _Pacific-Asia conference on knowledge discovery and data mining_ , pages 160–172. Springer, 2013.
  * Carion et al. [2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _European conference on computer vision_ , pages 213–229. Springer, 2020.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2021.
  * Chen et al. [2020] Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl.  Learning by cheating.  In _Conference on Robot Learning_ , pages 66–75. PMLR, 2020.
  * Godard et al. [2019] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow.  Digging into self-supervised monocular depth estimation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 3828–3838, 2019.
  * Grill et al. [2020] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  _Advances in neural information processing systems_ , 33:21271–21284, 2020.
  * Gu et al. [2023] Junru Gu, Chenxu Hu, Tianyuan Zhang, Xuanyao Chen, Yilun Wang, Yue Wang, and Hang Zhao.  Vip3d: End-to-end visual trajectory prediction via 3d agent queries.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5496–5506, 2023.
  * Gupta et al. [2023] Agrim Gupta, Jiajun Wu, Jia Deng, and Li Fei-Fei.  Siamese masked autoencoders.  _arXiv preprint arXiv:2305.14344_ , 2023.
  * Han et al. [2020] Tengda Han, Weidi Xie, and Andrew Zisserman.  Self-supervised co-training for video representation learning.  _Advances in Neural Information Processing Systems_ , 33:5679–5690, 2020.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * He et al. [2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9729–9738, 2020.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 16000–16009, 2022.
  * Himmelsbach et al. [2010] Michael Himmelsbach, Felix V Hundelshausen, and H-J Wuensche.  Fast segmentation of 3d point clouds for ground vehicles.  In _2010 IEEE Intelligent Vehicles Symposium_ , pages 560–565. IEEE, 2010.
  * Hu et al. [2023] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al.  Planning-oriented autonomous driving.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 17853–17862, 2023.
  * Huang and Huang [2022] Junjie Huang and Guan Huang.  Bevdet4d: Exploit temporal cues in multi-camera 3d object detection.  _arXiv preprint arXiv:2203.17054_ , 2022.
  * Huang et al. [2021a] Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du.  BEVDet: High-performance multi-camera 3d object detection in bird-eye-view.  _arXiv preprint arXiv:2112.11790_ , 2021a.
  * Huang et al. [2021b] Siyuan Huang, Yichen Xie, Song-Chun Zhu, and Yixin Zhu.  Spatio-temporal self-supervised representation learning for 3d point clouds.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 6535–6545, 2021b.
  * Jabri et al. [2020] Allan Jabri, Andrew Owens, and Alexei Efros.  Space-time correspondence as a contrastive random walk.  _Advances in neural information processing systems_ , 33:19545–19560, 2020.
  * Jiang et al. [2022] Bo Jiang, Shaoyu Chen, Xinggang Wang, Bencheng Liao, Tianheng Cheng, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, and Chang Huang.  Perceive, interact, predict: Learning dynamic and static clues for end-to-end motion prediction.  _arXiv preprint arXiv:2212.02181_ , 2022.
  * Jiang et al. [2023] Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang.  Vad: Vectorized scene representation for efficient autonomous driving.  In _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2023.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kuhn [1955] Harold W Kuhn.  The hungarian method for the assignment problem.  _Naval research logistics quarterly_ , 2(1-2):83–97, 1955.
  * Lang et al. [2019] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 12697–12705, 2019.
  * Li et al. [2023a] Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li.  Bevdepth: Acquisition of reliable depth for multi-view 3d object detection.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 1477–1485, 2023a.
  * Li et al. [2023b] Yiheng Li, Seth Z Zhao, Chenfeng Xu, Chen Tang, Chenran Li, Mingyu Ding, Masayoshi Tomizuka, and Wei Zhan.  Pre-training on synthetic driving data for trajectory prediction.  _arXiv preprint arXiv:2309.10121_ , 2023b.
  * Li et al. [2022] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai.  Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers.  In _European conference on computer vision_ , pages 1–18. Springer, 2022.
  * Liao et al. [2022] Bencheng Liao, Shaoyu Chen, Xinggang Wang, Tianheng Cheng, Qian Zhang, Wenyu Liu, and Chang Huang.  MapTR: Structured modeling and learning for online vectorized hd map construction.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Lin et al. [2023] Xuewu Lin, Tianwei Lin, Zixiang Pei, Lichao Huang, and Zhizhong Su.  Sparse4d v2: Recurrent temporal fusion with sparse model.  _arXiv preprint arXiv:2305.14018_ , 2023.
  * Lin and Wang [2022] Zhiwei Lin and Yongtao Wang.  Bev-mae: Bird’s eye view masked autoencoders for outdoor point cloud pre-training.  _arXiv preprint arXiv:2212.05758_ , 2022.
  * Liu et al. [2022] Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun.  Petr: Position embedding transformation for multi-view 3d object detection.  In _European Conference on Computer Vision_ , pages 531–548. Springer, 2022.
  * Liu et al. [2023] Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela L Rus, and Song Han.  Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 2774–2781. IEEE, 2023.
  * Loshchilov and Hutter [2018] Ilya Loshchilov and Frank Hutter.  Fixing weight decay regularization in adam.  2018\.
  * Min et al. [2023] Chen Min, Xinli Xu, Dawei Zhao, Liang Xiao, Yiming Nie, and Bin Dai.  Occ-bev: Multi-camera unified pre-training via 3d scene reconstruction.  _arXiv preprint arXiv:2305.18829_ , 2023.
  * Nunes et al. [2023] Lucas Nunes, Louis Wiesmann, Rodrigo Marcuzzi, Xieyuanli Chen, Jens Behley, and Cyrill Stachniss.  Temporal Consistent 3D LiDAR Representation Learning for Semantic Perception in Autonomous Driving.  In _Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Pang et al. [2022] Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan.  Masked autoencoders for point cloud self-supervised learning.  In _European conference on computer vision_ , pages 604–621. Springer, 2022.
  * Park et al. [2022] Jinhyung Park, Chenfeng Xu, Shijia Yang, Kurt Keutzer, Kris M Kitani, Masayoshi Tomizuka, and Wei Zhan.  Time will tell: New outlooks and a baseline for temporal multi-view 3d object detection.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Philion and Fidler [2020] Jonah Philion and Sanja Fidler.  Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16_ , pages 194–210. Springer, 2020.
  * Qi et al. [2021] Charles R Qi, Yin Zhou, Mahyar Najibi, Pei Sun, Khoa Vo, Boyang Deng, and Dragomir Anguelov.  Offboard 3d object detection from point cloud sequences.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6134–6144, 2021.
  * Saha et al. [2022] Avishkar Saha, Oscar Mendez, Chris Russell, and Richard Bowden.  Translating images into maps.  In _2022 International conference on robotics and automation (ICRA)_ , pages 9200–9206. IEEE, 2022.
  * Sun et al. [2023] Jiachen Sun, Haizhong Zheng, Qingzhao Zhang, Atul Prakash, Z Morley Mao, and Chaowei Xiao.  Calico: Self-supervised camera-lidar contrastive pre-training for bev perception.  _arXiv preprint arXiv:2306.00349_ , 2023.
  * Tian et al. [2023] Xiaoyu Tian, Haoxi Ran, Yue Wang, and Hang Zhao.  Geomae: Masked geometric target prediction for self-supervised point cloud pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13570–13580, 2023.
  * Tong et al. [2022] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang.  Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.  _Advances in neural information processing systems_ , 35:10078–10093, 2022.
  * Wang et al. [2023a] Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao.  Videomae v2: Scaling video masked autoencoders with dual masking.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14549–14560, 2023a.
  * Wang et al. [2019] Ning Wang, Yibing Song, Chao Ma, Wengang Zhou, Wei Liu, and Houqiang Li.  Unsupervised deep tracking.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 1308–1317, 2019.
  * Wang et al. [2023b] Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang.  Exploring object-centric temporal modeling for efficient multi-view 3d object detection.  _arXiv preprint arXiv:2303.11926_ , 2023b.
  * Wang and Gupta [2015] Xiaolong Wang and Abhinav Gupta.  Unsupervised learning of visual representations using videos.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2794–2802, 2015.
  * Wang et al. [2022a] Xiao Wang, Haoqi Fan, Yuandong Tian, Daisuke Kihara, and Xinlei Chen.  On the importance of asymmetry for siamese representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16570–16579, 2022a.
  * Wang et al. [2022b] Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon.  DETR3D: 3d object detection from multi-view images via 3d-to-2d queries.  In _Conference on Robot Learning_ , pages 180–191. PMLR, 2022b.
  * Xie et al. [2020] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany.  Pointcontrast: Unsupervised pre-training for 3d point cloud understanding.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16_ , pages 574–591. Springer, 2020.
  * Xie et al. [2023] Yichen Xie, Chenfeng Xu, Marie-Julie Rakotosaona, Patrick Rim, Federico Tombari, Kurt Keutzer, Masayoshi Tomizuka, and Wei Zhan.  Sparsefusion: Fusing multi-modal sparse representations for multi-sensor 3d object detection.  _arXiv preprint arXiv:2304.14340_ , 2023.
  * Xie et al. [2021] Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu.  Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16684–16693, 2021.
  * Xu et al. [2022] Chenfeng Xu, Tian Li, Chen Tang, Lingfeng Sun, Kurt Keutzer, Masayoshi Tomizuka, Alireza Fathi, and Wei Zhan.  Pretram: Self-supervised pre-training via connecting trajectory and map.  In _European Conference on Computer Vision_ , pages 34–50. Springer, 2022.
  * Yang et al. [2021] Bin Yang, Min Bai, Ming Liang, Wenyuan Zeng, and Raquel Urtasun.  Auto4d: Learning to label 4d objects from sequential point clouds.  _arXiv preprint arXiv:2101.06586_ , 2021.
  * Yin et al. [2022] Junbo Yin, Dingfu Zhou, Liangjun Zhang, Jin Fang, Cheng-Zhong Xu, Jianbing Shen, and Wenguan Wang.  Proposalcontrast: Unsupervised pre-training for lidar-based 3d object detection.  In _European Conference on Computer Vision_ , pages 17–33. Springer, 2022.
  * Zhang et al. [2022] Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li.  Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training.  _Advances in neural information processing systems_ , 35:27061–27074, 2022.

Supplementary Material

In this supplementary material, we elaborate on the details of our pretraining
and finetuning in Sec. 6. We then visualize additional qualitative results for
end-to-end planning in Sec. 7. Finally, the limitations and future work are
discussed in Sec. 8.

##  6 Implementation Details

In this part, we present extra implementation details in addition to Sec. 4.1.

###  6.1 Pretraining Architecture

In the pretraining stage, we follow the network architecture of SOLOFusion
[39] without the detection heads and the long-term BEV feature concatenation
component. The multi-view camera inputs are encoded by an image backbone
separately. We then perform a stereo matching of the encoded features with the
image features from the previous frame, to construct a cost volume to improve
the per-frame depth estimation. Using the image features and the depth
estimation results, we generate a BEV feature map through LSS-style [40]
projection. The output is encoded by a lightweight convolutional neural
network to derive the BEV features $\mathbf{F}^{t}_{BEV}$. We follow prior
contrastive learning works [9, 6] to design a projection head and a prediction
head for the point-wise features sampled from the BEV features. Both heads are
two fully-connected layers with a LayerNorm layer and a ReLU layer in between.
The projection head is attached to both online and target networks, while the
prediction head is only included in the online network, following the
projection head.

###  6.2 Pretraining Data Augmentation

Same as [19, 39], we perform weak data augmentations on both image inputs and
the projected BEV features. For the image inputs, we apply random resizing
$(0.94,1.11)$, random rotation $(-5.4^{\circ},5.4^{\circ})$, and random
horizontal flipping. For the BEV features, we apply random resizing
$(0.95,1.05)$, random rotation $(-22.5^{\circ},22.5^{\circ})$, and random
horizontal/vertical flipping.

###  6.3 Finetuning Details

In this part, we present the training strategies used during the finetuning
stage of each task, which are the same as their default from-scratch training
settings.

#### Object Detection

For both SOLOFusion [39] and BEVDet [19] models, they are finetuned for 24
epochs using a learning rate of 2e-4, a batch size of 64, and the AdamW
optimizer [35].

#### Map Reconstruction

The MapTR [30] model is finetuned for 24 epochs using a batch size of 32 and
the AdamW optimizer [35]. The learning rate is 6e-4 for the entire model
except 6e-5 for the image backbone, and cosine annealing is applied to the
learning rate.

#### Prediction and Planning

We use the VAD-tiny [23] model for both motion prediction and end-to-end
planning. The model is finetuned for 60 epochs using a batch size of 8 and the
AdamW optimizer [35]. The learning rate is 2e-4 for the entire model except
2e-5 for the image backbone, and cosine annealing is applied to the learning
rate.

##  7 E2E Planning Qualitative Results

In this part, we provide two qualitative comparisons of the VAD-tiny [23]
model for end-to-end planning with and without our Cohere3D pretraining in
Fig. 7. These end-to-end planning examples reflect the effectiveness of our
Cohere3D pretraining for all tasks, including detection, mapping, prediction,
and planning. Results show that our Cohere3D pretraining improves the
performance of the end-to-end planning model in different aspects, including
(1) reconstructing accurate HD maps from the camera inputs, (2) detecting
surrounding vehicles and predicting their future motions reliably, and (3)
producing map-compliant and safe planning results in different scenarios.

(a) Cohere3D pretraining identifies the boundary of the road to provide a map-
compliant plan for turning.

(b) Cohere3D pretraining detects the car from the opposite direction to avoid
potential conflicts during planning.

Figure 7: Visualizations of end-to-end planning results without (top) and with
(bottom) Cohere3D pretraining. Green denotes the ego-vehicle and planning
results. Red refers to the surrounding vehicles and prediction results. Black
lines and blue lines are the predicted road boundaries and predicted lane
dividers, respectively.

##  8 Limitation and Future Work

Although Cohere3D pretraining demonstrates promising results in multiple
tasks, there are still several limitations. Firstly, our pretraining relies on
pairwise raw LiDAR and multi-view camera sensors, since we require point
clouds to construct the long-term fine-grained correspondence. Secondly, our
experiments focus primarily on nuScenes dataset [3] due to availability of
high-quality LiDAR-image pairwise data in open-source driving datasets.

Our future work should focus on solving these limitations. For instance, we
can potentially leverage unsupervised tracking models [47, 21] to replace the
LiDAR point clouds for the construction of long-term correspondence.
Additionally, we will explore the transferability of Cohere3D on more driving
data in the future.


of the homogeneous Grothendieck polynomial is Lorentzian. Motivated by this,
we wonder about the Lorentzian property of the following polynomial: for any
polymatroid $\mathscr{P}$, we consider the polynomial

$g_{\mathscr{P}}(w_{0},w_{1},\ldots,w_{p})\;:=\;\sum_{\mathbf{n}\in\mathscr{P}_{\leq}}-\mu_{\Gamma}(\mathbf{n},\hat{1})(-w_{0})^{r(\mathscr{P})-|\mathbf{n}|}{\mathbf{w}}^{\mathbf{n}}\in\mathbb{Z}[w_{0},\ldots,w_{p}].$

The sign in $-w_{0}$ is there to ensure that the coefficients are always
nonnegative, since $\mu_{\Gamma}$ alternates in sign. Via a similar
construction, the proof of Mason’s conjecture in [HUH_BRANDEN, Theorem 4.14]
proceeds by associating to each matroid $\mathscr{M}$ the polynomial
$f_{\mathscr{M}}(w_{0},w_{1},\ldots,w_{p})=\sum_{\mathbf{n}\in\mathscr{M}_{\leq}}w_{0}^{r(\mathscr{M})-|\mathbf{n}|}{\mathbf{w}}^{\mathbf{n}}\in\mathbb{Z}[w_{0},\ldots,w_{p}]$
and proving that $f_{\mathscr{M}}$ is Lorentzian. We consider the
normalization operator
$N:\mathbb{Z}[w_{0},\ldots,w_{p}]\rightarrow\mathbb{Z}[w_{0},\ldots,w_{p}]$
given by $N(w_{0}^{a_{0}}\cdots w_{p}^{a_{p}})=\frac{w_{0}^{a_{0}}\cdots
w_{p}^{a_{p}}}{a_{0}!\cdots a_{p}!}$. Since the support of a Lorentzian
polynomial is a polymatroid, in light of §missing 7.4, it is natural to ask
the following:

######  Question .

Is $N(g_{\mathscr{P}})$ Lorentzian?

## Acknowledgments

Most of the research included in this article was developed in the
Mathematisches Forschungsinstitut Oberwolfach (MFO) while the authors were in
residence at the institute in November-December 2022. The authors thank MFO
for their hospitality and excellent conditions for conducting research. F.C.
was partially funded by FONDECYT Grant 1221133. Y.C.R. was partially funded by
an FWO Postdoctoral Fellowship (1220122N). F.M. was partially supported by the
grants G0F5921N (Odysseus programme) and G023721N from the Research Foundation
- Flanders (FWO), and the grant iBOF/23/064 from KU Leuven. J.M. was partially
funded by NSF Grant DMS #2001645/2303605. The authors thank Alejandro Morales
and José Samper for helpful conversations.

## References