# CRKD: Enhanced Camera-Radar Object Detection with Cross-modality Knowledge
Distillation

Lingjun Zhao*, Jingyu Song*†, Katherine A. Skinner
University of Michigan, Ann Arbor, MI USA
<EMAIL_ADDRESS>
(October 2023)

###### Abstract

In the field of 3D object detection for autonomous driving, LiDAR-Camera (LC)
fusion is the top-performing sensor configuration. Still, LiDAR is relatively
high cost, which hinders adoption of this technology for consumer automobiles.
Alternatively, camera and radar are commonly deployed on vehicles already on
the road today, but performance of Camera-Radar (CR) fusion falls behind LC
fusion. In this work, we propose Camera-Radar Knowledge Distillation (CRKD) to
bridge the performance gap between LC and CR detectors with a novel cross-
modality KD framework. We use the Bird’s-Eye-View (BEV) representation as the
shared feature space to enable effective knowledge distillation. To
accommodate the unique cross-modality KD path, we propose four distillation
losses to help the student learn crucial features from the teacher model. We
present extensive evaluations on the nuScenes dataset to demonstrate the
effectiveness of the proposed CRKD framework. The project page for CRKD is
https://song-jingyu.github.io/CRKD.

††footnotetext: *Equal contribution.††footnotetext: †Corresponding
author.††footnotetext: This work was supported by a grant from Ford Motor
Company via the Ford-UM Alliance under award N028603.

##  1 Introduction

Perception is an important module for achieving safe and effective autonomous
driving [3, 15, 58]. 3D object detection is an essential task in perception as
it is of great significance for subsequent tasks [53, 50, 57, 47]. Among
various perceptual sensors used by autonomous driving researchers, LiDAR,
camera and radar are the most common choices to enable autonomy on the road
[53]. Sensor fusion techniques are usually used to improve the detector’s
performance and robustness. LiDAR-Camera (LC) fusion has been widely
demonstrated as the top-performing sensor configuration for 3D object
detection [2, 7, 39, 1, 65, 14]. However, the high cost of LiDAR constrains
the wide application of this configuration. Though Camera-Only (CO) detectors
have demonstrated impressive performance in recent Bird’s-Eye-View (BEV) based
frameworks [35, 34, 17, 16], the camera’s vulnerability to lighting conditions
and lack of accurate depth measurements motivates researchers to turn to other
sensors such as radar. Radar is robust to varying weather and lighting
conditions and features automotive-grade design and low cost. Radars are
already highly accessible on most cars equipped with driver assistance
features. Compared with LiDAR, the radar measurements are sparse and noisy,
which makes designing a Camera-Radar (CR) detector challenging. Recent CR
detectors have leveraged the advancement brought by BEV-based Camera-Only (CO)
detectors [35, 17, 16, 34] to achieve further improvement in accuracy and
robustness to weather and lighting changes [28, 74].

Figure 1: We propose CRKD to conduct a novel cross-modality knowledge
distillation path from a LiDAR-camera teacher to a camera-radar student. We
present a radar chart to illustrate the complementary nature of these sensing
configurations and the improvement that CRKD can enable.

Despite the advancement in architecture design, there is still a distinct
performance gap when comparing LiDAR-Only (LO) and LC detectors against CO and
CR detectors. Recent research has focused on applying the Knowledge
Distillation (KD) technique to alleviate this gap [12, 72, 26, 55, 13].
Generally, KD features a teacher-student framework that aims to propagate the
informative knowledge from a well-performing teacher model to facilitate the
learning process of the student model. This usually leads to improved
performance compared to simply training the student model on the same task.
The KD technique has been employed in either intra-modal [70, 56, 68, 29] or
cross-modal [5, 6, 13, 18, 72, 26] configurations for 3D object detection.
Though many cross-modal methods use a single-modality detector as the teacher
model to leverage the privileged LiDAR data that is widely available in open-
source datasets, they mainly focus on distilling knowledge to a LiDAR-based or
camera-based student detector. We argue the importance of designing a
distillation path from an LC teacher detector to a CR student detector, which
could benefit from the existing superior design of LC detectors and the shared
point cloud representation between measurements of LiDAR and radars [63].

Inspired by the above observations, we propose CRKD: an enhanced Camera-Radar
3D object detector with cross-modality Knowledge Distillation (Fig. 1) that
distills knowledge from an LC teacher detector to a CR student detector. To
our best knowledge, CRKD is the first KD framework that supports a fusion-to-
fusion distillation path. As the LiDAR sensor is used only during training, we
emphasize the value of CRKD as it could facilitate the practical application
of perceptual autonomy with a low-cost and robust CR sensor configuration.

To summarize, our main contributions are as follows:

  * •

We propose a novel cross-modality KD framework to enable LC-to-CR distillation
in the BEV feature space. With the transferred knowledge from an LC teacher
detector, the CR student detector can outperform existing baselines without
additional cost during inference.

  * •

We design four KD modules to address the notable discrepancies between
different sensors to realize effective cross-modality KD. As we operate KD in
the BEV space, the proposed loss designs can be applied to other KD
configurations. Our improvement also includes adding a gated network to the
baseline model for adaptive fusion.

  * •

We conduct extensive evaluation on nuScenes [2] to demonstrate the
effectiveness of CRKD. CRKD can improve the mAP and NDS of student detectors
by $3.5\%$ and $3.2\%$ respectively. As our method focuses on a novel KD path
with a large modality gap, we provide thorough study and analysis to support
our design choices.

##  2 Related Works

Figure 2: An overview of the proposed cross-modality (LC-to-CR) distillation
framework CRKD. We narrow down the modality gap by unifying both the teacher
and the student into the BEV space with similar 3D object detector structure.
We refine the model design to enable adaptive fusion and design four novel
distillation losses for effective cross-modality KD. During inference, only CR
input is needed.

###  2.1 Multi-modality 3D Object Detection

The multi-modality 3D object detectors generally outperform single-modality
detectors in accuracy and robustness as the perceptual sensors (e.g., LiDAR,
camera and radar) can complement each other [53]. Among the common sensor
combinations, LC is the best-performing modality configuration on most
existing datasets [2, 9, 43, 7]. In general, LiDAR and camera are fused in
different ways. One trend is to augment LiDAR points with features from
cameras [51, 52, 49, 19, 60], which is usually referred to as early fusion.
Other solutions apply deep feature fusion in a shared representation space
[67, 31, 33]. One trending choice is to leverage the BEV space to deliver
impressive improvement [39, 37]. There are also methods that fuse information
at a later stage. In [1, 4, 61], features are independently extracted and
aggregated via proposals or queries in the detection head, while some methods
[45, 46] combine the output candidates from single-modality detectors.

Nevertheless, the LC configuration is less accessible to consumer cars due to
the high cost of LiDAR. Thus, the potential for CR detectors stands out due to
the robustness brought by radars and the potential of large-scale deployment.
One of the key challenges facing CR detectors is how to handle the discrepancy
in sensor views and data returns. CenterFusion [44] applies feature-level
fusion by associating radar points with image features via a frustum-based
method. Following this, more feature-level fusion methods are proposed in [8,
23, 73, 59]. Recently, as many camera-based methods [35, 34] have started to
leverage the unified BEV space by transforming camera features from
perspective view (PV), many CR detectors also explore fusion of camera and
radar features in the BEV space [25, 22]. Though LiDAR data is not available
in real inference and deployment, its wide appearance in open-source dataset
[2] has motivated researchers to leverage LiDAR data to guide the feature
transformation process [28]. Motivated by the aforementioned works, CRKD also
unifies features in the BEV space and leverages LiDAR data in a KD-based
framework. To our best knowledge, CRKD is the first framework that improves CR
detectors with cross-modality KD from a top-performing LC teacher detector.

###  2.2 Cross-modality Knowledge Distillation

The idea of KD is initially proposed in [12] for model compression in an image
classification task. It is then extended to the field of object detection for
model compression and performance improvement [36]. Specifically, in the field
of 3D object detection, a group of KD methods requires that the teacher and
student models use the same modality such as LO-to-LO (L2L) [70, 56, 54, 64,
21] and CO-to-CO (C2C) [68, 69, 29]. In contrast, cross-modality KD focuses on
KD with different modality configurations. Typical paths include LiDAR-to-
Camera (L2C) [10, 40, 6, 5, 13, 31, 18] and Camera-to-LiDAR (C2L) [72, 48].
Recently, new cross-modality KD paths including a fusion-based modality have
been explored. In UniDistill [72], a universal framework that supports
multiple KD paths is proposed. By unifying the features from different
modalities to the shared BEV space, it supports L2C, C2L, LC-to-LO (LC2L) and
LC-to-Camera (LC2C). DistillBEV [55] also supports L2C and LC2C by leveraging
the shared BEV space. X3KD [26] proposes a cross-modal and cross-task KD
framework for L2C KD. It is evaluated on an LO-to-CR (L2CR) KD path as a
supplementary task. However, it lacks specific consideration for the large
domain discrepancies with radars and further experiments and analysis with the
L2CR KD path. Among the existing prior works, we observe the lack of KD
methods that can support a fusion-to-fusion distillation path and handle
domain differences with radars. We argue the importance of conducting
distillation from an LC teacher to a CR student to leverage the shared BEV
feature space of camera-based detectors and the shared point cloud
representation of LiDAR and radar measurements. According to our best
knowledge, we are the first to investigate a KD framework with a fusion-to-
fusion path. We demonstrate that our novel framework improves the detection
performance of CR detectors comprehensively.

##  3 Method

We show an overview of CRKD in Fig. 2. We set up the teacher and student
models with a similar BEV-based encoder-decoder head architecture. Taking
advantage of the shared BEV feature space, we build CRKD based on the highly
optimized BEVFusion [39] codebase. We use BEVFusion-LC as the teacher model
and BEVFusion-CR as the baseline student model. The detector head in both
models is set as CenterHead [66] for response KD.

To account for the challenging cross-modality fusion-to-fusion KD, we design
several KD modules. We propose cross-stage radar distillation with a learning-
based calibration module to enable the radar encoder to learn a more accurate
scene-level object distribution. A mask-scaling feature KD is designed for
feature imitation on foreground regions while accounting for inaccurate view
transformation to BEV features for objects that are far from the sensor and
dynamic. We apply a relation KD to maintain relation consistency in scene-
level geometry. In addition, we improve the response KD design with class-
specific loss weights to better leverage the CR model’s ability to capture
dynamic objects. Details of the proposed KD modules will be discussed in the
following sections.

###  3.1 Model Architecture Refinement

We add a gated network [20, 67, 50] to BEVFusion [39] to enable the model to
learn to generate attention weights on the single-modality feature maps to
fuse the complementary modalities adaptively. Specifically, the gated network
learns the gated features as follows:

$\tilde{F}_{M_{1}}=F_{M_{1}}\times\sigma(\mbox{Conv}_{M_{1}}(\mbox{Concat}(F_{M_{1}},F_{M_{2}}))),$
(1)
$\tilde{F}_{M_{2}}=F_{M_{2}}\times\sigma(\mbox{Conv}_{M_{2}}(\mbox{Concat}(F_{M_{1}},F_{M_{2}}))),$
(2)

where $\tilde{F}_{M_{1}}$ and $\tilde{F}_{M_{2}}$ are the gated features for
modality $M_{1}$ and $M_{2}$, $F_{M_{1}}$ and $F_{M_{2}}$ are the input
feature maps to the gated network from the backbone and view transform module
of modality $M_{1}$ and $M_{2}$, respectively, $\sigma$ denotes the sigmoid
function, and $\mbox{Conv}_{M_{1}}$ and $\mbox{Conv}_{M_{2}}$ are two separate
convolution layers for $M_{1}$ and $M_{2}$ that could learn channel-wise
attention weights for the input features. The output of the gated network is
further fused by a convolutional fusion module in BEVFusion [39]. We apply the
adaptive gated network to our teacher and student models to learn the relative
importance between input modalities. This modification improves the detection
performance of the teacher and student models, and also makes feature-based
distillation more effective since the gated feature maps encode informative
scene geometry from both input modalities.

In the LC teacher model, we denote the camera feature map as
$F^{T}_{c}\in\mathbb{R}^{C^{T}_{c}\times H\times W}$ and the LiDAR feature map
as $F^{T}_{l}\in\mathbb{R}^{C^{T}_{l}\times H\times W}$. Similarly, the camera
and radar features in the student model are denoted as
$F^{S}_{c}\in\mathbb{R}^{C^{S}_{c}\times H\times W}$ and
$F^{S}_{r}\in\mathbb{R}^{C^{S}_{r}\times H\times W}$, respectively. We denote
the fused feature map in the teacher model and the student model as
$F^{T}_{f}$ and $F^{S}_{f}$. We keep the spatial dimension $H\times W$
consistent for all feature maps, and also set $C^{T}_{c}=C^{S}_{c}$ and
$C^{T}_{l}=C^{S}_{l}$ for feature mimicking across the feature dimension.

###  3.2 Cross-Stage Radar Distillation (CSRD)

Though the measurements of radar and LiDAR are both represented as point
clouds, the physical meaning behind them are slightly different. Compared with
LiDAR, radar points are much sparser and can be interpreted as a list of
object-level points with velocity measurements [53, 63], while LiDAR is denser
and captures geometry-level information. Observing this gap, we argue the
common method of direct feature imitation may not work well in this scenario.
Instead, as radar measurements are sparse and represent scene-level object
distribution, we propose a novel Cross-Stage Radar Distillation (CSRD) method.
Specifically, we design a distillation path between the radar feature map and
the scene-level objectness heatmap predicted by the LC teacher model, which is
denoted as $Y^{T}\in\mathbb{R}^{K\times H\times W}$, where $K$ is the number
of classes. Since radars are generally believed to be noisy in the range and
azimuth angle measurements, we design a calibration module to learn to
compensate the noise. Specifically, we pass $F^{S}_{r}$ to three blocks of
convolutional layer, batch normalization and ReLU activation with kernel size
$3\times 3$. We add another $1\times 1$ convolution layer to project the
calibrated feature map to $\hat{F}^{S}_{r}\in\mathbb{R}^{H\times W}$. The CSRD
loss $\mathcal{L}_{csrd}$ is formed as follows:

$\mathcal{L}_{csrd}=\frac{1}{H\times
W}\sum_{i}^{H}\sum_{j}^{W}\|\hat{Y}^{T}_{i,j}-\mbox{$\hat{F}^{S}_{r}$}_{i,j}\|_{1},$
(3)

where $\hat{Y}^{T}\in\mathbb{R}^{H\times W}$ is obtained by taking the mean
along the $K$ dimension of $\sigma(Y^{T})$.

###  3.3 Mask-Scaling Feature Distillation (MSFD)

We propose feature distillation for aligning the camera feature maps and the
fused feature maps. It has been acknowledged in many works [72, 6, 5, 71] that
direct feature imitation between teacher and student models may not work
effectively in 3D object detection tasks due to the notable imbalance between
the foreground and background. Therefore, a common fix to this issue is to
generate a mask $M\in\mathbb{R}^{H\times W}$ to only distill information from
the foreground region. Meanwhile, more works have demonstrated that the
boundary region of the foreground can also contribute to effective KD [5, 71].
We follow this finding and propose Mask-Scaling Feature Distillation (MSFD)
that is aware of object range and movement. For the student CR model, the
detection performance is mainly dependent on the depth prediction for images
and the geometric accuracies of radar points. Since the range and object
movement can cause extra challenges for view transformation to BEV space, we
scale up the area of the foreground region to account for the potential
misalignment. We increase the width and length of the mask by $\alpha$ and
$\beta$ if the objects are in the range groups $[r_{1},r_{2}]$ and
$[r_{2},\infty]$. We also increase the width and length by $\alpha$ and
$\beta$ if the velocities along that axis are within $[v_{1},v_{2}]$ and
$[v_{2},\infty]$. In practice, We clip the increase of object size within a
pre-defined range to balance between different sizes of objects. We form the
MSFD loss as follows:

$\mathcal{L}_{msfd}=\frac{1}{H\times
W}\sum_{i}^{H}\sum_{j}^{W}M_{i,j}\|{F_{i,j}^{T}}-F_{i,j}^{S}\|_{2},$ (4)

where $F^{T}$ and $F^{S}$ represent the feature maps in the teacher and
student model, respectively. We compute MSFD loss for the gated camera feature
maps ($\tilde{F}^{T}_{c}$ and $\tilde{F}^{S}_{c}$) and the fused feature maps
($F^{T}_{f}$ and $F^{S}_{f}$).

###  3.4 Relation Distillation (RelD)

While the aforementioned CSRD and MSFD can handle feature-level distillation
effectively, we follow MonoDistill [6] to highlight the importance of
maintaining similar geometric relations in the scene level between the teacher
and student models. We compute the affinity matrix describing cosine
similarity of the fused feature map. We propose the RelD loss as follows:

$C_{i,j}=\frac{F_{i}^{\top}F_{j}}{\|F_{i}\|_{2}\cdot\|F_{j}\|_{2}},$ (5)

where $C_{i,j}$ denotes the cosine similarity value at $(i,j)$ in the affinity
matrix, and $F_{i}$ and $F_{j}$ represent the $i^{th}$ and $j^{th}$ feature
map separately. The scene-level information gap between the student and
teacher models can be computed as the $\mathcal{L}_{1}$ norm between their
respective affinity matrices. We refer to the RelD loss as
$\mathcal{L}_{reld}$, as shown below:

$\mathcal{L}_{reld}=\frac{1}{H\times
W}\sum\limits_{i=1}^{H}\sum\limits_{j=1}^{W}\|C^{T}_{i,j}-C^{S}_{i,j}\|_{1},$
(6)

where $H$ and $W$ represent the BEV spatial size, and $C^{T}$ and $C^{S}$
denote the affinity matrix of the student and teacher model, respectively. In
CRKD, we compute $\mathcal{L}_{reld}$ between the fused feature maps of the
teacher and student models since they are the input to the decoder and
detector heads. The refined feature maps with distilled relation information
could improve the detection performance. Moreover, in order to distill the
scene-level relation information of different scales, we apply a downsampling
operation and a convolutional block. Then we use these multi-level feature
maps to calculate the multi-scale RelD losses and take the average value as
the final loss term.

###  3.5 Response Distillation (RespD)

Response Distillation has been proven effective in image classification [12]
and 3D object detection [13, 72, 55]. The predictions inferred by the teacher
are served as the soft labels for the student. The soft labels and the hard
labels are combined to supervise the learning of the student model. We refer
to the RespD design in CMKD [13] and improve it to be aware of modality
strength. Since radar has the unique advantage of direct velocity measurements
due to the Doppler effect [74, 63], we set larger weights for dynamic classes
in RespD to allow higher priority for dynamic objects to leverage the student
CR model’s strength. The loss for dynamic response distillation is denoted as
$\mathcal{L}_{resp}$, consisting of the classification loss
$\mathcal{L}_{cls}$ and the regression loss $\mathcal{L}_{reg}$.
$\mathcal{L}_{cls}$ is for object categories and is computed using the Quality
Focal Loss (QFL) [30]. $\mathcal{L}_{reg}$ is for 3D bounding box regression
and can be obtained by calculating the $SmoothL1$ loss. We compute these two
losses as:

$\mathcal{L}_{cls}=\sum\limits_{i=1}^{K}QFL(P^{T}_{C_{i}},P^{S}_{C_{i}})\times
w_{i},$ (7)
$\mathcal{L}_{reg}=\sum\limits_{i=1}^{K}Smooth\mathcal{L}1(P^{T}_{B_{i}},P^{S}_{B_{i}})\times
w_{i},$ (8)

where $P^{T}_{C_{i}}$ and $P^{S}_{C_{i}}$ denote the classification
predictions of the $i^{th}$ task generated by the teacher and student model,
and $P^{T}_{B_{i}}$ and $P^{S}_{B_{i}}$ denote the regression predictions in
the teacher and student models. $K$ is the number of tasks in the Centerhead
[66], and $w_{i}$ represents the class-specific weights.

###  3.6 Overall Loss Function

We combine the proposed KD loss and standard 3D object detection loss
$\mathcal{L}_{det}$. The overall loss function we apply in the training stage
is:

$\mathcal{L}=\lambda_{1}\cdot\mathcal{L}_{csrd}+\lambda_{2}\cdot\mathcal{L}_{msfd}+\lambda_{3}\cdot\mathcal{L}_{reld}+\lambda_{4}\cdot\mathcal{L}_{respd}+\lambda_{5}\cdot\mathcal{L}_{det},$
(9)

where $\lambda_{1}$ through $\lambda_{5}$ are hyperparameters we set for
weighting different loss components.

##  4 Experiments

Set | Method | Modality | Backbone | Resolution | mAP$\uparrow$ | NDS$\uparrow$ | mATE$\downarrow$ | mASE$\downarrow$ | mAOE$\downarrow$ | mAVE$\downarrow$ | mAAE$\downarrow$
---|---|---|---|---|---|---|---|---|---|---|---
| BEVFormer-S [35] | C | R101 | $900\times 1600$ | 37.5 | 44.8 | 0.725 | 0.272 | 0.391 | 0.802 | 0.200
| BEVDet [17] | C | R50 | $256\times 704$ | 29.8 | 37.9 | 0.725 | 0.279 | 0.589 | 0.860 | 0.245
| RCM-Fusion [22] | C+R | R101 | $900\times 1600$ | 44.3 | 52.9 | - | - | - | - | -
| CenterFusion [44] | C+R | DLA34 | $450\times 800$ | 33.2 | 45.3 | 0.649 | 0.263 | 0.535 | 0.540 | 0.142
| CRAFT [24] | C+R | DLA34 | $448\times 800$ | 41.1 | 51.7 | 0.494 | 0.276 | 0.454 | 0.486 | 0.176
| RCBEV [73] | C+R | SwinT | $256\times 704$ | 37.7 | 48.2 | 0.534 | 0.271 | 0.558 | 0.493 | 0.209
| BEVFusion [39] | C+R | SwinT | $256\times 704$ | 43.2 | 54.1 | 0.489 | 0.269 | 0.512 | 0.313 | 0.171
| UVTR (L2C) [31] | C$\Diamond$ | R101 | $900\times 1600$ | 37.2 | 45.0 | 0.735 | 0.269 | 0.397 | 0.761 | 0.193
| BEVDistill (BEVFormer-S) [5] | C$\Diamond$ | R50 | $640\times 1600$ | 38.6 | 45.7 | 0.693 | 0.264 | 0.399 | 0.802 | 0.199
| UniDistill (LC2C) [72] | C$\Diamond$ | R50 | $256\times 704$ | 26.5 | 37.8 | - | - | - | - | -
| BEVSimDet [71] | C$\Diamond$ | SwinT | $256\times 704$ | 40.4 | 45.3 | 0.526 | 0.275 | 0.607 | 0.805 | 0.273
| X3KD (LC2C) [26] | C$\Diamond$ | R50 | $256\times 704$ | 39.0 | 50.5 | 0.615 | 0.269 | 0.471 | 0.345 | 0.203
| DistillBEV (BEVDet) [55] | C$\Diamond$ | R50 | $256\times 704$ | 34.0 | 41.6 | 0.704 | 0.266 | 0.556 | 0.815 | 0.201
| X3KD (L2CR) [26] | C+R$\Diamond$ | R50 | $256\times 704$ | 42.3 | 53.8 | - | - | - | - | -
| CRKD | C+R$\Diamond$ | R50 | $256\times 704$ | 43.2 | 54.9 | 0.450 | 0.267 | 0.442 | 0.339 | 0.176
_val_ | CRKD | C+R$\Diamond$ | SwinT | $256\times 704$ | 46.7 | 57.3 | 0.446 | 0.263 | 0.408 | 0.331 | 0.162
| BEVFormer-S [35] | C | R101 | $900\times 1600$ | 40.9 | 46.2 | 0.650 | 0.261 | 0.439 | 0.925 | 0.147
| BEVDet$\dagger$ [17] | C | SwinT | $640\times 1600$ | 42.4 | 48.2 | 0.528 | 0.236 | 0.395 | 0.979 | 0.152
| RCM-Fusion [22] | C+R | R101 | $900\times 1600$ | 49.3 | 58.0 | 0.485 | 0.255 | 0.386 | 0.421 | 0.115
| CenterFusion$\dagger$ [44] | C+R | DLA34 | $450\times 800$ | 32.6 | 44.9 | 0.631 | 0.261 | 0.516 | 0.614 | 0.115
| CRAFT$\dagger$ [24] | C+R | DLA34 | $448\times 800$ | 41.1 | 52.3 | 0.467 | 0.268 | 0.456 | 0.519 | 0.114
| RCBEV [73] | C+R | SwinT | $256\times 704$ | 40.6 | 48.6 | 0.484 | 0.257 | 0.587 | 0.702 | 0.140
| UVTR (L2C) [31] | C$\Diamond$ | V2-99 | $900\times 1600$ | 45.2 | 52.2 | 0.612 | 0.256 | 0.385 | 0.664 | 0.125
| X3KD (LC2C) [26] | C$\Diamond$ | R101 | $640\times 1600$ | 45.6 | 56.1 | 0.506 | 0.253 | 0.414 | 0.366 | 0.131
| UniDistill (LC2C) [72] | C$\Diamond$ | R50 | $256\times 704$ | 29.6 | 39.3 | 0.637 | 0.257 | 0.492 | 1.084 | 0.167
| X3KD (L2CR) [26] | C+R$\Diamond$ | R50 | $256\times 704$ | 44.1 | 55.3 | - | - | - | - | -
_test_ | CRKD | C+R$\Diamond$ | SwinT | $256\times 704$ | 48.7 | 58.7 | 0.404 | 0.253 | 0.425 | 0.376 | 0.111

Table 1: Comparison on the nuScenes dataset. We group methods based on
modality and whether KD is used. We include existing SOTA works that use
single-frame image input for fair comparison. Methods [5, 55, 71] missed in
the _test_ split group do not report their results with single-frame image
input. The proposed CRKD outperforms the baseline methods in most metrics.
$\Diamond$ denotes the distillation-based methods. $\dagger$ denotes using
test time augmentation. The best is bolded and the second best is underlined.

###  4.1 Experimental Setup

We evaluate our method on the nuScenes dataset [2] as the three modalities
(i.e., LiDAR, camera and radar) are all available. We follow the official
split that has 700 scenes for training and 150 scenes for validation. We set
$[-54m,-54m,-5m]\times[54m,54m,3m]$ as the region to conduct object detection.
We use the mean-Average-Precision (mAP) and NuScenes Detection Score (NDS) [2]
as the main evaluation metrics. We also report the True Positive (TP) metrics
[2] for comprehensive evaluation.

Our implementation is based on the MMDetection3D codebase [7]. All of our
experiments are conducted using 4 NVIDIA A100 GPUs. We set the default
BEVFusion-LC with Centerhead [66] as the teacher model. As mentioned in Sec.
3.1, we add the adaptive gated network to the baseline BEVFusion-CR model and
denote it as BEVFusion-CR*. We set BEVFusion-CR* as the student model. We set
the camera backbone as SwinT [38] and image resolution as $256\times 704$ for
both the teacher and student models. We also test CRKD with a ResNet R50
backbone [11] for comprehensive evaluation. The BEV spatial size is set to
$180\times 180$. We use PointPillars [27] as the backbone of the radar branch
in BEVFusion-CR*. We set the same Centerhead [66] as the detector head for the
student model. During distillation, we freeze the teacher model and train the
student model for 20 epochs. We set the batch size as $8$ and learning rate as
1e-4. We include more implementation details and results in the supplementary
material.

###  4.2 Quantitative Results

We give an overall comparison of CRKD with existing CO and CR detectors with
single-frame image input on nuScenes [2]. We follow common practices [5, 72,
26, 55] to show a complete comparison on both the _val_ and _test_ splits. As
shown in Tab. 1, CRKD is the top-performing model in most metrics on the _val_
set of nuScenes. We also present the performance of CRKD on the _test_ split.
The results show that CRKD has the best or second best performance on most
metrics without using any test-time optimization techniques (e.g., test-time
augmentation, larger image resolution). Overall, CRKD is the most consistent
in achieving high performance across all of the baselines.

We also show a complete comparison of per-class AP in Tab. 2 to break down the
improvement brought by CRKD. The results show that CRKD achieves consistent
improvement in AP of all the classes. There is an interesting finding that
larger gains of CRKD come from dynamic classes, which indicates that CRKD
successfully helps the student model leverage its strength on dynamic object
detection more effectively due to the availability of direct velocity
measurements from radar.

Model | Modality | Car | Truck | Bus | Trailer | CV | Ped | Motor | Bicycle | TC | Barrier | mAP$\uparrow$
---|---|---|---|---|---|---|---|---|---|---|---|---
Teacher | L+C | 88.4 | 62.4 | 73.8 | 40.6 | 29.2 | 78.7 | 75.3 | 65.8 | 74.9 | 72.3 | 66.1
Baseline | C+R | 72.1 | 37.8 | 48.9 | 18.3 | 12.6 | 48.4 | 42.0 | 33.8 | 58.8 | 59.6 | 43.2
Student | C+R | 72.2 | 41.3 | 51.0 | 19.2 | 15.2 | 49.0 | 46.2 | 35.5 | 59.1 | 60.1 | 44.9
CRKD | C+R | 74.8(+2.7) | 44.1(+6.3) | 53.6(+4.7) | 20.6(+2.3) | 16.9(+4.3) | 50.6(+2.2) | 46.8(+4.8) | 38.2(+4.4) | 61.5(+2.7) | 60.1(+0.5) | 46.7(+3.5)

Table 2: Comparison of the per-class AP results of the BEVFusion-LC (teacher),
BEVFusion-CR (baseline), BEVFusion-CR* (student) and CRKD models on the
nuScenes _val_ split. We quantitatively show the improvement made by CRKD over
the baseline.

Model | Gated | RespD | CSRD | MSFD | RelD | mAP$\uparrow$ | NDS$\uparrow$ | mATE$\downarrow$ | mASE$\downarrow$ | mAOE$\downarrow$ | mAVE$\downarrow$ | mAAE$\downarrow$
---|---|---|---|---|---|---|---|---|---|---|---|---
Baseline |  |  |  |  |  | 43.2 | 54.1 | 0.489 | 0.269 | 0.512 | 0.313 | 0.171
| ✓ |  |  |  |  | 44.9 | 55.9 | 0.464 | 0.267 | 0.458 | 0.304 | 0.165
| ✓ | ✓ |  |  |  | 45.7 | 56.7 | 0.448 | 0.262 | 0.409 | 0.330 | 0.166
| ✓ | ✓ | ✓ |  |  | 46.0 | 57.0 | 0.445 | 0.261 | 0.407 | 0.326 | 0.163
| ✓ | ✓ | ✓ | ✓ |  | 46.2 | 57.2 | 0.439 | 0.260 | 0.394 | 0.332 | 0.166
CRKD | ✓ | ✓ | ✓ | ✓ | ✓ | 46.7 | 57.3 | 0.446 | 0.263 | 0.408 | 0.331 | 0.162

Table 3: Ablation study of the proposed modules in CRKD evaluated on the
nuScenes _val_ split. The baseline is the BEVFusion-CR model.

Module | LiDAR | Heatmap | mAP$\uparrow$ | NDS$\uparrow$
---|---|---|---|---
| ✓ |  | 44.9 | 56.3
CSRD |  | ✓ | 46.0 | 57.0
(a) Ablation study of Cross-stage Radar Distillation (CSRD).

Module | Mask | Mask-scaling | mAP$\uparrow$ | NDS$\uparrow$
---|---|---|---|---
| ✓ |  | 46.0 | 56.7
MSFD |  | ✓ | 46.2 | 56.9
(b) Ablation study of Mask-scaling Feature Distillation (MSFD).

Module | Vanilla | Adapt | mAP$\uparrow$ | NDS$\uparrow$
---|---|---|---|---
| ✓ |  | 45.9 | 56.9
RelD |  | ✓ | 46.2 | 57.0
(c) Ablation study of Relation Distillation (RelD).

Module | Vanilla | Dynamic | mAP$\uparrow$ | NDS$\uparrow$
---|---|---|---|---
| ✓ |  | 45.3 | 56.7
RespD |  | ✓ | 45.7 | 56.7
(d) Ablation study of Response Distillation (RespD).

Table 4: Ablation Study of single distillation modules on the nuScenes val split. Fuser | In Channels | Out Channels | mAP$\uparrow$ | NDS$\uparrow$
---|---|---|---|---
Conv | 80 + 256 | 256 | 43.2 | 54.1
| 64 + 64 | 64 | 44.2 | 54.3
| 128 + 128 | 256 | 44.4 | 54.7
Gated | 80 + 256 | 256 | 44.9 | 55.9
Table 5: Ablation study of fusion module design and number of channels of the
student model on the nuScenes _val_ split. Figure 3: Qualitative results on
nuScenes. We show zoomed-in views in panel b and c for the highlighted regions
in panel a, with the border dash as the correspondence. We show the ground
truth annotation in red, teacher prediction in green, student prediction in
yellow, CRKD prediction in blue, and radar points in magenta. In (1a) to (1c),
we show an example frame where CRKD has more accurate predictions and fewer
false predictions than the student model. In (2a) to (2c) we show another
example frame where CRKD even outperforms the LC teacher by detecting a missed
car and rejecting several false predictions. Best viewed on screen and in
color.

###  4.3 Ablation Studies

To further break down the improvement brought from each module we design,
extensive experiments are conducted to discuss and validate our design
choices. We firstly present the main ablation study in Tab. 3. It can be
observed that all the proposed modules have contributed to the superior
performance of CRKD. Among the four proposed KD losses, we see the most
improvement comes from RespD, which indicates the significance of RespD in
cross-modality KD. The other three losses contribute more to the improvement
of mAP. This finding validates our design objectives in improving object
localization as CSRD and MSFD supervise on the feature maps, and RelD tries to
align the scene-level geometric relation.

We next demonstrate the experiments we conduct to validate the design choices
of each module. As mentioned before, response distillation (RespD) brings the
most improvement among all the proposed KD modules. Our empirical finding
indicates that for other KD modules, the best-performing design alone may not
be the best choice that works with RespD and other modules. We conjecture that
this inconsistency in performance gain comes from the considerably large
domain discrepancies between LC and CR. We would like to highlight the
importance of RespD in the cross-modality KD work and use the experiments of
combining different modules with RespD to guide the overall design of CRKD.
Therefore, for the following ablation study in Tab. 4, unless otherwise
mentioned, we show results of experiments with using RespD together. All the
ablation study instances are trained using the same setting as the full model.

####  4.3.1 Effect of CSRD

We conduct an ablation study to validate the proposed CSRD module. As
mentioned in Sec. 3.2, the radar points represent object-level information.
Therefore, the common practice to distill information at the same stage of the
network may not work well for radar feature maps. We propose CSRD to add
cross-stage supervision on the object-level information. We demonstrate the
model with CSRD outperforms that using LiDAR feature maps as the distillation
source in Tab. 4(a). The significant improvement from CSRD validates that the
higher-level objectness heatmap provides more suitable guidance to distill
radar features.

####  4.3.2 Effect of MSFD

We conduct a comparison between the proposed mask-scaling strategy accounting
for object range and velocity and the common foreground mask of ground truth
bounding boxes. In Tab. 4(b), the improvement achieved with the proposed mask-
scaling strategy verifies that our mask-scaling strategy in MSFD helps to
achieve more effective KD.

####  4.3.3 Effect of RelD

We study the effect of applying a convolutional block after the downsampling
operation for the feature maps used in RelD. We name the instance with the
convolution layer as Adapt and the baseline instance as Vanilla in Tab. 4(c).
The results verify our design choice.

####  4.3.4 Effect of RespD

Though RespD has been a widely used loss term in several KD works, we are the
first to weight the RespD loss differently based on different classes. In
practice, we set $w_{i}$ as 2 for the dynamic classes and $1$ for static
classes. This design allows the training supervision to prioritize dynamic
classes, which radars are more capable of detecting. In the vanilla setting we
set $w_{i}$ as $1$ for all classes. As shown in Tab. 4(d), applying the class-
specific weights helps to improve the overall performance of the student
detector.

####  4.3.5 Effect of Model Architecture Refinement

We study the effect of adding the adaptive gated network to the original
fusion module in BEVFusion [39] and tuning the number of channel dimensions.
As shown in Tab. 5, the addition of the gated network brings notable
improvement over the default fusion module in the baseline BEVFusion-CR model.
We also notice that matching the input and the output channel dimension of the
fusion module to the teacher model (Camera: 80, LiDAR: 256) brings additional
improvement. The following KD operation also gets benefit from the same
channel dimension setting between the teacher and student models since no
additional channel-wise projection is needed for feature-level KD.

###  4.4 Qualitative Results

We show the visualization of the 3D object detection results to highlight the
effectiveness of CRKD in Fig. 3. With CRKD, the detector is able to predict
fewer false positive predictions and localize the objects better. We also show
an impressive comparison between the teacher LC model and CRKD to demonstrate
that CRKD can even outperform the teacher model with the help of radar
measurements. This qualitative example validates the effectiveness of the CRKD
framework and the value of radars for modern perception for autonomous
driving. We will include more qualitative examples and discussion in the
supplementary material.

##  5 Conclusion

We have proposed CRKD, a novel KD framework that supports a cross-modality
fusion-to-fusion KD path for 3D object detection. We leverage the BEV space to
design a novel LC-to-CR KD framework. We design four distillation losses to
address the significant domain gap and facilitate the distillation process in
this cross-modality setting. We also introduce the adaptive gated network to
learn the relative importance between two expert feature maps. Extensive
experiments show the effectiveness of CRKD in improving the detection
performance of CR detectors. We hope CRKD will inspire future research to
leverage our proposed KD framework to further explore the potential of CR
detectors to improve the reliability of this widely accessible sensor suite.
In future work, we plan to extend the proposed CRKD framework to other
perception tasks such as occupancy mapping.

## References

  * Bai et al. [2022] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai.  Transfusion: Robust lidar-camera fusion for 3d object detection with transformers.  In _CVPR_ , 2022.
  * Caesar et al. [2020] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In _CVPR_ , 2020.
  * Carmichael et al. [2024] Spencer Carmichael, Austin Buchan, Mani Ramanagopal, Radhika Ravi, Ram Vasudevan, and Katherine A Skinner.  Dataset and benchmark: Novel sensors for autonomous vehicle perception.  _arXiv preprint arXiv:2401.13853_ , 2024.
  * Chen et al. [2023a] Xuanyao Chen, Tianyuan Zhang, Yue Wang, Yilun Wang, and Hang Zhao.  Futr3d: A unified sensor fusion framework for 3d detection.  In _CVPR Workshop_ , 2023a.
  * Chen et al. [2023b] Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinhong Jiang, and Feng Zhao.  Bevdistill: Cross-modal bev distillation for multi-view 3d object detection.  In _ICLR_ , 2023b.
  * Chong et al. [2022] Zhiyu Chong, Xinzhu Ma, Hong Zhang, Yuxin Yue, Haojie Li, Zhihui Wang, and Wanli Ouyang.  Monodistill: Learning spatial features for monocular 3d object detection.  In _ICLR_ , 2022.
  * Contributors [2020] MMDetection3D Contributors.  MMDetection3D: OpenMMLab next-generation platform for general 3D object detection.  https://github.com/open-mmlab/mmdetection3d, 2020.
  * Drews et al. [2022] Florian Drews, Di Feng, Florian Faion, Lars Rosenbaum, Michael Ulrich, and Claudius Gläser.  Deepfusion: A robust and modular 3d object detector for lidars, cameras and radars.  In _IROS_ , 2022.
  * Geiger et al. [2012] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In _CVPR_ , 2012.
  * Guo et al. [2021] Xiaoyang Guo, Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li.  Liga-stereo: Learning lidar geometry aware representations for stereo-based 3d detector.  In _ICCV_ , 2021.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2015.
  * Hong et al. [2022] Yu Hong, Hang Dai, and Yong Ding.  Cross-modality knowledge distillation network for monocular 3d object detection.  In _ECCV_ , 2022.
  * Hu et al. [2023a] Haotian Hu, Fanyi Wang, Jingwen Su, Yaonong Wang, Laifeng Hu, Weiye Fang, Jingwei Xu, and Zhiwang Zhang.  Ea-lss: Edge-aware lift-splat-shot framework for 3d bev object detection.  _arXiv preprint arXiv:2303.17895_ , 2023a.
  * Hu et al. [2023b] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al.  Planning-oriented autonomous driving.  In _CVPR_ , 2023b.
  * Huang and Huang [2022] Junjie Huang and Guan Huang.  Bevdet4d: Exploit temporal cues in multi-camera 3d object detection.  _arXiv preprint arXiv:2203.17054_ , 2022.
  * Huang et al. [2021] Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du.  Bevdet: High-performance multi-camera 3d object detection in bird-eye-view.  _arXiv preprint arXiv:2112.11790_ , 2021.
  * Huang et al. [2022] Peixiang Huang, Li Liu, Renrui Zhang, Song Zhang, Xinli Xu, Baichao Wang, and Guoyi Liu.  Tig-bev: Multi-view bev 3d object detection via target inner-geometry learning.  _arXiv preprint arXiv:2212.13979_ , 2022.
  * Huang et al. [2020] Tengteng Huang, Zhe Liu, Xiwu Chen, and Xiang Bai.  Epnet: Enhancing point features with image semantics for 3d object detection.  In _ECCV_ , 2020.
  * Jacobs et al. [1991] Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton.  Adaptive mixtures of local experts.  In _Neural computation_ , 1991.
  * Ju et al. [2022] Bo Ju, Zhikang Zou, Xiaoqing Ye, Minyue Jiang, Xiao Tan, Errui Ding, and Jingdong Wang.  Paint and distill: Boosting 3d object detection with semantic passing network.  In _ACM MM_ , 2022.
  * Kim et al. [2023a] Jisong Kim, Minjae Seong, Geonho Bang, Dongsuk Kum, and Jun Won Choi.  Rcm-fusion: Radar-camera multi-level fusion for 3d object detection.  _arXiv preprint arXiv:2307.10249_ , 2023a.
  * Kim et al. [2020] Youngseok Kim, Jun Won Choi, and Dongsuk Kum.  Grif net: Gated region of interest fusion network for robust 3d object detection from radar point cloud and monocular image.  In _IROS_ , 2020.
  * Kim et al. [2023b] Youngseok Kim, Sanmin Kim, Jun Won Choi, and Dongsuk Kum.  Craft: Camera-radar 3d object detection with spatio-contextual fusion transformer.  In _AAAI_ , 2023b.
  * Kim et al. [2023c] Youngseok Kim, Juyeb Shin, Sanmin Kim, In-Jae Lee, Jun Won Choi, and Dongsuk Kum.  Crn: Camera radar net for accurate, robust, efficient 3d perception.  In _ICCV_ , 2023c.
  * Klingner et al. [2023] Marvin Klingner, Shubhankar Borse, Varun Ravi Kumar, Behnaz Rezaei, Venkatraman Narayanan, Senthil Yogamani, and Fatih Porikli.  X3kd: Knowledge distillation across modalities, tasks and stages for multi-camera 3d object detection.  In _CVPR_ , 2023.
  * Lang et al. [2019] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  In _CVPR_ , 2019.
  * Lei et al. [2023] Kai Lei, Zhan Chen, Shuman Jia, and Xiaoteng Zhang.  Hvdetfusion: A simple and robust camera-radar fusion framework.  _arXiv preprint arXiv:2307.11323_ , 2023.
  * Li et al. [2023a] Jianing Li, Ming Lu, Jiaming Liu, Yandong Guo, Yuan Du, Li Du, and Shanghang Zhang.  Bev-lgkd: A unified lidar-guided knowledge distillation framework for multi-view bev 3d object detection.  In _IEEE IV_ , 2023a.
  * Li et al. [2020] Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang.  Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection.  In _NeurIPS_ , 2020.
  * Li et al. [2022a] Yanwei Li, Yilun Chen, Xiaojuan Qi, Zeming Li, Jian Sun, and Jiaya Jia.  Unifying voxel-based representation with transformer for 3d object detection.  In _NeurIPS_ , 2022a.
  * Li et al. [2022b] Yao Li, Jiajun Deng, Yu Zhang, Jianmin Ji, Houqiang Li, and Yanyong Zhang.  ${\mathsf{ezfusion}}$: A close look at the integration of lidar, millimeter-wave radar, and camera for accurate 3d object detection and tracking.  In _IEEE RAL_ , 2022b.
  * Li et al. [2022c] Yanwei Li, Xiaojuan Qi, Yukang Chen, Liwei Wang, Zeming Li, Jian Sun, and Jiaya Jia.  Voxel field fusion for 3d object detection.  In _CVPR_ , 2022c.
  * Li et al. [2023b] Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li.  Bevdepth: Acquisition of reliable depth for multi-view 3d object detection.  In _AAAI_ , 2023b.
  * Li et al. [2022d] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai.  Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers.  In _ECCV_ , 2022d.
  * Li et al. [2023c] Zhihui Li, Pengfei Xu, Xiaojun Chang, Luyao Yang, Yuanyuan Zhang, Lina Yao, and Xiaojiang Chen.  When object detection meets knowledge distillation: A survey.  In _IEEE TPAMI_ , 2023c.
  * Liang et al. [2022] Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, and Zhi Tang.  Bevfusion: A simple and robust lidar-camera fusion framework.  In _NeurIPS_ , 2022.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _ICCV_ , 2021.
  * Liu et al. [2023a] Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela L Rus, and Song Han.  Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation.  In _ICRA_ , 2023a.
  * Liu et al. [2023b] Zhe Liu, Xiaoqing Ye, Xiao Tan, Errui Ding, and Xiang Bai.  Stereodistill: Pick the cream from lidar for distilling stereo-based 3d object detection.  In _AAAI_ , 2023b.
  * Loshchilov and Hutter [2016] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  _arXiv preprint arXiv:1608.03983_ , 2016.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Mei et al. [2022] Jieru Mei, Alex Zihao Zhu, Xinchen Yan, Hang Yan, Siyuan Qiao, Liang-Chieh Chen, and Henrik Kretzschmar.  Waymo open dataset: Panoramic video panoptic segmentation.  In _ECCV_ , 2022.
  * Nabati and Qi [2021] Ramin Nabati and Hairong Qi.  Centerfusion: Center-based radar and camera fusion for 3d object detection.  In _WACV_ , 2021.
  * Pang et al. [2020] Su Pang, Daniel Morris, and Hayder Radha.  Clocs: Camera-lidar object candidates fusion for 3d object detection.  In _IROS_ , 2020.
  * Pang et al. [2022] Su Pang, Daniel Morris, and Hayder Radha.  Fast-clocs: Fast camera-lidar object candidates fusion for 3d object detection.  In _WACV_ , 2022.
  * Pang et al. [2023] Ziqi Pang, Jie Li, Pavel Tokmakov, Dian Chen, Sergey Zagoruyko, and Yu-Xiong Wang.  Standing between past and future: Spatio-temporal modeling for multi-camera 3d multi-object tracking.  In _CVPR_ , 2023.
  * Sautier et al. [2022] Corentin Sautier, Gilles Puy, Spyros Gidaris, Alexandre Boulch, Andrei Bursuc, and Renaud Marlet.  Image-to-lidar self-supervised distillation for autonomous driving data.  In _CVPR_ , 2022.
  * Sindagi et al. [2019] Vishwanath A Sindagi, Yin Zhou, and Oncel Tuzel.  Mvx-net: Multimodal voxelnet for 3d object detection.  In _ICRA_ , 2019.
  * Song et al. [2024] Jingyu Song, Lingjun Zhao, and Katherine A Skinner.  Lirafusion: Deep adaptive lidar-radar fusion for 3d object detection.  _arXiv preprint arXiv:2402.11735_ , 2024.
  * Vora et al. [2020] Sourabh Vora, Alex H Lang, Bassam Helou, and Oscar Beijbom.  Pointpainting: Sequential fusion for 3d object detection.  In _CVPR_ , 2020.
  * Wang et al. [2021] Chunwei Wang, Chao Ma, Ming Zhu, and Xiaokang Yang.  Pointaugmenting: Cross-modal augmentation for 3d object detection.  In _CVPR_ , 2021.
  * Wang et al. [2023a] Li Wang, Xinyu Zhang, Ziying Song, Jiangfeng Bi, Guoxin Zhang, Haiyue Wei, Liyao Tang, Lei Yang, Jun Li, Caiyan Jia, and Lijun Zhao.  Multi-modal 3d object detection in autonomous driving: A survey and taxonomy.  In _IEEE IV_ , 2023a.
  * Wang and Solomon [2021] Yue Wang and Justin M Solomon.  Object dgcnn: 3d object detection using dynamic graphs.  In _NeurIPS_ , 2021.
  * Wang et al. [2023b] Zeyu Wang, Dingwen Li, Chenxu Luo, Cihang Xie, and Xiaodong Yang.  Distillbev: Boosting multi-camera 3d object detection with cross-modal knowledge distillation.  In _ICCV_ , 2023b.
  * Wei et al. [2022] Yi Wei, Zibu Wei, Yongming Rao, Jiaxin Li, Jie Zhou, and Jiwen Lu.  Lidar distillation: Bridging the beam-induced domain gap for 3d object detection.  In _ECCV_ , 2022.
  * Wilson et al. [2022] Joey Wilson, Jingyu Song, Yuewei Fu, Arthur Zhang, Andrew Capodieci, Paramsothy Jayakumar, Kira Barton, and Maani Ghaffari.  Motionsc: Data set and network for real-time semantic mapping in dynamic environments.  In _IEEE RAL_ , 2022.
  * Wilson et al. [2023] Joey Wilson, Yuewei Fu, Arthur Zhang, Jingyu Song, Andrew Capodieci, Paramsothy Jayakumar, Kira Barton, and Maani Ghaffari.  Convolutional bayesian kernel inference for 3d semantic mapping.  In _ICRA_ , 2023.
  * Wu et al. [2023] Zizhang Wu, Guilian Chen, Yuanzhu Gan, Lei Wang, and Jian Pu.  Mvfusion: Multi-view 3d object detection with semantic-aligned radar and camera fusion.  In _ICRA_ , 2023.
  * Xu et al. [2021] Shaoqing Xu, Dingfu Zhou, Jin Fang, Junbo Yin, Zhou Bin, and Liangjun Zhang.  Fusionpainting: Multimodal fusion with adaptive attention for 3d object detection.  In _IEEE ITSC_ , 2021.
  * Yan et al. [2023] Junjie Yan, Yingfei Liu, Jianjian Sun, Fan Jia, Shuailin Li, Tiancai Wang, and Xiangyu Zhang.  Cross modal transformer: Towards fast and robust 3d object detection.  In _ICCV_ , 2023.
  * Yan et al. [2018] Yan Yan, Yuxing Mao, and Bo Li.  Second: Sparsely embedded convolutional detection.  In _MDPI Sensors_ , 2018.
  * Yang et al. [2020] Bin Yang, Runsheng Guo, Ming Liang, Sergio Casas, and Raquel Urtasun.  Radarnet: Exploiting radar for robust perception of dynamic objects.  In _ECCV_ , 2020.
  * Yang et al. [2022a] Jihan Yang, Shaoshuai Shi, Runyu Ding, Zhe Wang, and Xiaojuan Qi.  Towards efficient 3d object detection with knowledge distillation.  In _NeurIPS_ , 2022a.
  * Yang et al. [2022b] Zeyu Yang, Jiaqi Chen, Zhenwei Miao, Wei Li, Xiatian Zhu, and Li Zhang.  Deepinteraction: 3d object detection via modality interaction.  In _NeurIPS_ , 2022b.
  * Yin et al. [2021] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl.  Center-based 3d object detection and tracking.  In _CVPR_ , 2021.
  * Yoo et al. [2020] Jin Hyeok Yoo, Yecheol Kim, Jisong Kim, and Jun Won Choi.  3d-cvf: Generating joint camera and lidar features using cross-view spatial feature fusion for 3d object detection.  In _ECCV_ , 2020.
  * Zeng et al. [2023] Jia Zeng, Li Chen, Hanming Deng, Lewei Lu, Junchi Yan, Yu Qiao, and Hongyang Li.  Distilling focal knowledge from imperfect expert for 3d object detection.  In _CVPR_ , 2023.
  * Zhang et al. [2022] Linfeng Zhang, Yukang Shi, Hung-Shuo Tai, Zhipeng Zhang, Yuan He, Ke Wang, and Kaisheng Ma.  Structured knowledge distillation towards efficient and compact multi-view 3d detection.  _arXiv preprint arXiv:2211.08398_ , 2022.
  * Zhang et al. [2023] Linfeng Zhang, Runpei Dong, Hung-Shuo Tai, and Kaisheng Ma.  Pointdistiller: Structured knowledge distillation towards efficient and compact 3d detection.  In _CVPR_ , 2023.
  * Zhao et al. [2023] Haimei Zhao, Qiming Zhang, Shanshan Zhao, Jing Zhang, and Dacheng Tao.  Bevsimdet: Simulated multi-modal distillation in bird’s-eye view for multi-view 3d object detection.  _arXiv preprint arXiv:2303.16818_ , 2023.
  * Zhou et al. [2023a] Shengchao Zhou, Weizhou Liu, Chen Hu, Shuchang Zhou, and Chao Ma.  Unidistill: A universal cross-modality knowledge distillation framework for 3d object detection in bird’s-eye view.  In _CVPR_ , 2023a.
  * Zhou et al. [2023b] Taohua Zhou, Junjie Chen, Yining Shi, Kun Jiang, Mengmeng Yang, and Diange Yang.  Bridging the view disparity between radar and camera features for multi-modal fusion 3d object detection.  In _IEEE IV_ , 2023b.
  * Zhou et al. [2022] Yi Zhou, Lulu Liu, Haocheng Zhao, Miguel López-Benítez, Limin Yu, and Yutao Yue.  Towards deep radar perception for autonomous driving: Datasets, methods, and challenges.  In _MDPI Sensors_ , 2022.
  * Zhu et al. [2019] Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu.  Class-balanced grouping and sampling for point cloud 3d object detection.  _arXiv preprint arXiv:1908.09492_ , 2019.

Supplementary Material

We provide this supplementary material with additional details to support the
main paper.

##  Appendix A Implementation Details

In this section, we provide implementation details of the CRKD framework to
enable cross-modality Knowledge Distillation (KD) from a LiDAR-Camera (LC)
teacher detector to a Camera-Radar (CR) student detector.

###  A.1 Data Augmentation

Our data processing pipeline is mainly adopted from the open-source
implementation of BEVFusion [39]. The pipeline of processing the camera image
is the same in the teacher and student models. The images of the cameras from
6 Perspective View (PV) are loaded and resized to $256\times 704$. During the
training process, data augmentation is applied to the images. We resize the
image with scaling factors in the range of $[0.38,0.55]$. We also set the
random rotation in the range of $[-5.4\degree,5.4\degree]$. The images are
normalized following the default practice in [7]. For the LiDAR input, the
keyframe point cloud is loaded along with $9$ previous sweeps. During
training, random resizing is applied with the scaling factor in $[-0.9,1.1]$.
Translation augmentation is also applied with a limit of $0.5\text{m}$. For
the radar input, the keyframe is loaded with $6$ previous sweeps. We follow
BEVFusion [39] to select radar data dimensions. The training time augmentation
of radar points is the same as LiDAR data for consistency. We also apply the
class-balanced grouping and sampling (CBGS) strategy during training [75]. We
do not apply any test-time augmentation for any of our models.

###  A.2 Teacher Model

As mentioned in the main paper, we add a gated network to the BEVFusion-LC
[39] and denote it as BEVFusion-LC*. We use the CenterHead [66] as the
detector head in the BEVFusion-LC*. There are two streams in the teacher model
for LiDAR and cameras. The LiDAR point cloud is encoded as the Bird’s Eye View
(BEV) feature map through the LiDAR encoder and BEV reduction module (flatten
over $z$ dimension). For the camera stream, the images are loaded and pre-
processed to a resolution of $256\times 704$. We use the SwinT [38] backbone
to process the images from $6$ cameras separately. The PV features are
transformed to BEV by taking advantage of the efficient PV-to-BEV
transformation module in BEVFusion [39]. The BEV feature maps from the LiDAR
stream and camera stream are passed into the gated network to obtain gated
feature maps with attentional relative importance between the input features.
These gated feature maps are further fused by the original convolutional
fusion module in BEVFusion [39]. The fused feature map is then fed into a
decoder and the CenterHead [66] to generate object predictions. We train the
teacher model for $20$ epochs using an AdamW optimizer [42]. The initial
learning rate is set as 2e-4 with a cosine annealing schedule [41, 7]. The
batch size is set as $16$. The object sampling strategy [62] is applied for
the first $15$ epochs. During distillation, the pre-trained teacher weight is
loaded and frozen.

###  A.3 Student Model

Similar to the LC teacher model, the gated network is also applied to the CR
student model, which is denoted as BEVFusion-CR*. The stream to process the
camera images is the same as the teacher model. The input radar data is
processed by a PointPillar-based backbone [27, 7] to obtain the BEV feature
map for the radar stream. The feature maps from these two streams are fused
via the gated network and the convolutional fusion module in BEVFusion [39].
To maintain the consistency between the teacher and student models, the
student model also uses the CenterHead [66] as the detector head. We train the
student model following the same setting as the teacher model. During
distillation, we load the pre-trained BEVFusion-CR* model and operate cross-
modality distillation with the proposed CRKD framework.

###  A.4 Base Loss Choice

In general, $\mathcal{L}_{2}$ is more common for feature KD. However, for
CSRD, we have to consider the sensor properties. Due to radar’s sparse
measurements, some objects may be missed, causing radar features at
corresponding locations to be outliers when computing loss with the objectness
heatmap. We use $\mathcal{L}_{1}$ to downplay this effect as it penalizes
large errors less heavily than $\mathcal{L}_{2}$, which leads to 0.4%
improvement in mAP than using $\mathcal{L}_{2}$. For MSFD, we follow common
practice ($\mathcal{L}_{2}$) as the domain gap is relatively small (shared
camera modality). For RelD, we agree with reviewer vB2P that applying
$\mathcal{L}_{1}$ between similarity matrices is appropriate. For RespD, we
mainly follow existing works (e.g., CMKD [13], BEVSimDet [71]) to choose the
base loss. Our method is fairly robust to base loss choice, while the final
design aligns with our design consideration and brings the best performance.

|  |  NDS$\uparrow$ |  mAP$\uparrow$
---|---|---|---
Method | Modality | $[0\text{m},20\text{m}]$ | $[20\text{m},30\text{m}]$ | $[30\text{m},50\text{m}]$ | $[0\text{m},20\text{m}]$ | $[20\text{m},30\text{m}]$ | $[30\text{m},50\text{m}]$
Teacher | L+C | 76.71 | 68.63 | 50.57 | 77.11 | 62.37 | 38.25
Student | C+R | 63.03 | 52.87 | 38.86 | 58.54 | 38.64 | 19.50
CRKD | C+R | 65.53(+2.50) | 53.52(+0.65) | 39.21(+0.35) | 61.59(+3.05) | 39.04(+0.40) | 20.53(+1.03)
Table 6: Performance breakdown by range evaluated on the nuScenes _val_ split.
We quantitatively show the improvement made by CRKD over the student model.

##  Appendix B Supplementary Experiments

|  |  NDS$\uparrow$ |  mAP$\uparrow$
---|---|---|---
Method | Modality | Sunny | Rainy | Day | Night | Sunny | Rainy | Day | Night
Teacher | L+C | 70.22 | 71.01 | 70.54 | 44.92 | 66.02 | 65.48 | 66.25 | 41.12
Student | C+R | 55.60 | 57.56 | 56.37 | 33.40 | 44.73 | 47.27 | 45.78 | 23.94
CRKD | C+R | 56.56(+0.96) | 59.97(+2.41) | 57.59(+1.22) | 34.22(+0.82) | 45.95(+1.22) | 49.59(+2.32) | 47.16(+1.38) | 24.14(+0.20)

Table 7: Performance breakdown by weather and lighting evaluated on the
nuScenes _val_ split. We quantitatively show the improvement made by CRKD over
the student model.

###  B.1 CRKD

After loading the pre-trained weights for the teacher and student models, we
add the four proposed KD loss terms to the normal detection loss and start the
KD training process. We disable the object sampling strategy [62] during
distillation. We set the learning rate as 1e-4 with the cosine annealing
strategy and train the model for 20 epochs. The batch size is set as 8. For
the loss weights, we set the hyperparameters
$\lambda_{1},\lambda_{2},\lambda_{3},\lambda_{4},\lambda_{5}$ as
$100,10,0.25,1,1$, respectively. More specifically, we set 1 as the weight for
$\mathcal{L}_{respd}$ and $\mathcal{L}_{det}$ since they both compute loss for
box regression and classification. The weight of $\mathcal{L}_{reld}$ is
$0.25$ as it sums up the loss of $4$ downsampled affinity map pairs. We
empirically select the weights of $\mathcal{L}_{csrd}$ and
$\mathcal{L}_{msfd}$ (i.e., 100 and 10) to balance with other loss modules.
For the Mask-Scaling Feature Distillation (MSFD), we set $r_{1}$ and $r_{2}$
as $20\text{m}$ and $30\text{m}$. The mask-scaling factors $\alpha$ and
$\beta$ are set as $0.25$ and $0.5$. For the velocity threshold, we set
$v_{1}$ and $v_{2}$ as $0.3\text{m/s}$ and $0.8\text{m/s}$. We also clip the
object size expanding value within $[0.5\text{m},4\text{m}]$ to balance
between different sizes of objects.

###  B.2 CRKD Improvement Analysis

Since CRKD is performing a novel KD path (LC to CR), we conduct more
experiments to break down the improvement brought by CRKD to provide further
insight. As the camera sensor is shared in both the teacher and student
models, we narrow down our focus to the difference between LiDAR and radar
integration. Radars have better long-range detection capability and weather
robustness than LiDAR [63, 74, 32]. In practice, we group objects by their
range to the ego-vehicle and the weather of the scene they belong to. We show
mAP and NDS of the teacher model, the student model and CRKD. We highlight the
quantitative improvement KD brings over the student model. As shown in Tab. 6,
we can see that the most improvement comes from the short-range group. This
finding demonstrates that CRKD helps the CR student detector to refine its
detections in the short-range group, which can be considered as one of LiDAR’s
strengths as LiDAR has satisfying density for objects that are near to the
LiDAR. We are also surprised to see that for mAP, the improvement in long-
range group is more than the medium-range group. This finding can provide
evidence that cross-modality KD can also enhance the strength of the student
detector. In addition, we group different scenes according to the weather and
lighting conditions. Table 7 demonstrates increased performance from CRKD
across all weather conditions, compared to the baseline student model.
Notably, we see a more significant increase in improvement from CRKD in rainy
weather. This finding supports that cross-modality KD can help the student to
learn and leverage radar’s robustness to the varying weather for better
results.

###  B.3 Radar Distillation Design

Module | w/o calib | w/ calib |  mAP$\uparrow$ |  NDS$\uparrow$
---|---|---|---|---
| ✓ |  | 45.9 | 56.9
CSRD |  | ✓ | 46.0 | 57.0
Table 8: Ablation study of CSRD with the radar calibration module. Module | GT | Teacher Heatmap |  mAP$\uparrow$ |  NDS$\uparrow$
---|---|---|---|---
| ✓ |  | 45.9 | 56.8
CSRD |  | ✓ | 46.0 | 57.0
Table 9: Ablation study of CSRD with the different distillation sources. Module | max | mean |  mAP$\uparrow$ |  NDS$\uparrow$
---|---|---|---|---
| ✓ |  | 45.6 | 56.8
CSRD |  | ✓ | 46.0 | 57.0
Table 10: Ablation study of CSRD with different channel-wise pooling methods
on the heatmap predicted by the teacher model. Figure 4: Visualization of the
ungated and gated camera feature maps in the teacher detector. The scene
geometry can be more easily interpreted from the gated feature map, as it has
encoded information from the LiDAR point cloud. Best viewed in color.

CRKD presents a novel distillation path to a CR detector. We specifically
design the KD module for radars, which has not been previously studied. We
present more ablation studies to justify our design choice. We hope our work
can bring more insights for future KD frameworks that leverage the radar
sensor. In the proposed Cross-Stage Radar Distillation (CSRD) module, we
design a calibration module to account for the noisy radar measurements. We
conduct an ablation study to understand the effect of the calibration module.
Table 8 demonstrates that the calibration module helps to further improve the
performance of the student detector.

In addition to the ablation study in the main paper, we show another ablation
study of the best distillation source for the CSRD module. Specifically, we
compare between using the ground truth heatmap or the heatmap predicted by the
teacher model. The results in Tab. 9 show that the objectness heatmap
predicted by the teacher detector is a better distillation source for radar
distillation.

We additionally compare taking the max or mean pooling along the class
dimension of the objectness heatmap $Y^{T}$ predicted by the teacher detector.
Table 10 shows that taking the mean value along different classes of the
source heatmap brings more improvement.

Module | Ungated | Gated |  mAP$\uparrow$ |  NDS$\uparrow$
---|---|---|---|---
| ✓ |  | 45.5 | 56.8
MSFD |  | ✓ | 45.7 | 56.9
Table 11: Ablation study of MSFD with the gated camera feature. Module | Cam | Fused | Cam&Fused |  mAP$\uparrow$ |  NDS$\uparrow$
---|---|---|---|---|---
| ✓ |  |  | 45.7 | 56.9
MSFD |  | ✓ |  | 45.8 | 56.7
|  |  | ✓ | 46.2 | 56.9
Table 12: Ablation study of MSFD with different distillation locations. Module | Dense | Gaussian | Ours |  mAP$\uparrow$ |  NDS$\uparrow$
---|---|---|---|---|---
| ✓ |  |  | 45.7 | 56.8
MSFD |  | ✓ |  | 45.5 | 56.7
|  |  | ✓ | 46.0 | 57.0
Table 13: Ablation study of MSFD with different feature masking algorithms.

###  B.4 Feature Distillation Location

We also experiment with introducing feature distillation at different
locations. Since we introduce the gated network to the original BEVFusion [39]
model, we design an ablation experiment justifying the introduction of the
gated feature map to improve the feature distillation. Specifically, we
compare using the gated camera feature map or the ungated camera feature map
as the feature distillation source. The results shown in Tab. 11 demonstrate
that the gated feature map serves as a more effective distillation source. We
additionally show a qualitative example in Fig. 4 to demonstrate the benefits
of using the gated feature map. The gated feature map has more informative
scene-level geometry thanks to the gated network and learned relative
importance weight.

Since the teacher and student models are both fusion-based, we have multiple
options of feature distillation locations (e.g., camera feature, fused
feature). For the proposed Mask-Scaling Feature Distillation (MSFD) module, we
experiment between different locations. As shown in Tab. 12, the most
effective design of MSFD is to perform the distillation of the gated camera
feature map and fused feature map together. Moreover, we conduct an experiment
testing alternative foreground mask generation methods. We experiment with not
including any foreground mask compared to methods that include a foreground
mask [39, 4, 74]. To complement the ablation study in the main paper, we
compare the proposed CRKD module against the same instance without any
foreground mask (denoted as dense). In addition, we try CRKD with a Gaussian-
style heatmap [5, 71]. The results are shown in Tab. 13. This table
demonstrates that though there are certain papers reporting having a Gaussian
heatmap is helpful [5, 71], the most effective masking strategy in our
scenario is still to apply the proposed mask-scaling strategy.

###  B.5 Response Distillation: Strength Amplification or Weakness Mitigation?

To better study the most suitable choice for the Response Distillation (RespD)
module, we design an experiment trying to answer an insightful question: is
the cross-modality distillation most helpful in amplifying the strength of the
student or mitigating the weakness of the student? It is widely recognized
that radars are more capable of perceiving dynamic objects [44, 32, 74].
Therefore, the CR student may benefit from the radar’s strength. As we have
the flexibility of varying the loss weight $w_{i}$ for different classes in
RespD, we experiment with different loss weight settings. In addition to the
ablation study of RespD reported in the main paper, we conduct an experiment
with static setting where the loss weights for static classes are set to $2$
while the weights for the other classes are set to $1$. In the static setting,
priority is given to the static classes, which radars are less capable of
detecting. As shown in Tab. 14, the RespD module works better when we
prioritize the learning of dynamic objects, which indicates that the RespD
module is more effective when designed to be amplifying the strength of the
student detector. The results complement the ablation study we show in the
main manuscript, demonstrating the effectiveness of the proposed dynamic RespD
module. We hope this interesting finding could provide some guidance to future
study about designing cross-modality distillation to leverage the strength of
different modalities effectively.

Module | Vanilla | Static | Dynamic |  mAP$\uparrow$ |  NDS$\uparrow$
---|---|---|---|---|---
| ✓ |  |  | 45.3 | 56.7
RespD |  | ✓ |  | 45.4 | 56.6
|  |  | ✓ | 45.7 | 56.7
Table 14: Ablation study of Response Distillation (RespD) with different
weight settings. Figure 5: More Qualitative results on nuScenes. We show
zoomed-in views in panel b and c for the highlighted regions in panel a, with
the border dash as the correspondence. The highlighted regions are enclosed
with border dash in ellipse. We show the ground truth annotation in red,
teacher prediction in green, student prediction in yellow, CRKD prediction in
blue, and radar points in magenta. In (1a) to (1c), we show an example frame
where CRKD can capture the object missed by the student with the guidance of
the teacher. In (2a) to (2c) we show an example frame where CRKD can reject
false predictions by the student model. In (3a) to (3b) and (4a) to (4b), we
show two examples where CRKD rejects false predictions by the teacher model
and generates more accurate predictions. In (5a) to (5c), we show an example
where CRKD outperforms both the teacher and student models by capturing missed
objects and generating less false predictions. Best viewed on screen and in
color.

###  B.6 Additional Qualitative Results

We show additional qualitative results of CRKD in Fig. 5. In the first two
samples (sample 1 and 2), we firstly show that CRKD is able to outperform the
student model since its predictions are more aligned with the ground truth. We
credit this improvement to the effective design of CRKD. We also show
additional examples (sample 3 and 4) where CRKD can even outperform the
teacher detector thanks to the long-range detection capability of radars. In
the last sample frame (sample 5), we show that CRKD is capable of capturing
the object that is missed by the student model. In addition, it is also
demonstrated that CRKD is able to maintain accurate predictions where the
teacher and student models generate false predictions.