# RCDN: Towards Robust Camera-Insensitivity Collaborative Perception via
Dynamic Feature-based 3D Neural Modeling

Tianhang Wang
Tongji University
<EMAIL_ADDRESS>
&Fan Lu
Tongji University
<EMAIL_ADDRESS>
&Zehan Zheng
Tongji University
<EMAIL_ADDRESS>&Guang Chen
Tongji University
<EMAIL_ADDRESS>
&Changjun Jiang
Tongji University
<EMAIL_ADDRESS>
Corresponding Author

###### Abstract

Collaborative perception is dedicated to tackling the constraints of single-
agent perception, such as occlusions, based on the multiple agents’ multi-view
sensor inputs. However, most existing works assume an ideal condition that all
agents’ multi-view cameras are continuously available. In reality, cameras may
be highly noisy, obscured or even failed during the collaboration. In this
work, we introduce a new robust camera-insensitivity problem: how to overcome
the issues caused by the failed camera perspectives, while stabilizing high
collaborative performance with low calibration cost? To address above
problems, we propose RCDN, a Robust Camera-insensitivity collaborative
perception with a novel Dynamic feature-based 3D Neural modeling mechanism.
The key intuition of RCDN is to construct collaborative neural rendering field
representations to recover failed perceptual messages sent by multiple agents.
To better model collaborative neural rendering field, RCDN first establishes a
geometry BEV feature based time-invariant static field with other agents via
fast hash grid modeling. Based on the static background field, the proposed
time-varying dynamic field can model corresponding motion vectors for
foregrounds with appropriate positions. To validate RCDN, we create OPV2V-N, a
new large-scale dataset with manual labelling under different camera failed
scenarios. Extensive experiments conducted on OPV2V-N show that RCDN can be
ported to other baselines and improve their robustness in extreme camera-
insensitivity settings. Our code and datasets will be available soon.

##  1 Introduction

Multi-agent collaborative perception[1, 2, 3, 4, 5] obtains better and more
holistic perception by allowing multiple agents to exchange complementary
perceptual information. This field has the potential to effectively address
various persistent challenges in single-perception, such as occlusion[6, 7].
The associated techniques and systems also process significant promise in
various domains, such as the utilization of multiple unmanned aerial aircraft
for search and rescue operations[8, 9, 10], the automation and mapping of
multiple robots[11, 12, 13]. As an emerging field, the research of
collaborative perception faces several issues that need to be addressed. These
challenges include the need for high-quality datasets[14, 15, 16, 17], the
formulation of models that are agnostic to specific tasks and models[18, 19],
and the ability to handle pose error and adversarial attacks[20, 21].

Figure 1: Illustration of noisy camera situations (blurred, occluded and even
failed) during collaboration and the perception result w.o./w. RCDN. orange
for drivable areas segmentation, blue for lanes and teal for dynamic vehicles.

However, a vast majority of existing works do not seriously account for the
harsh realities[22, 23] of real-world sensors in the collaboration, such as
blurred, high noise, interruption and even failure. These factors directly
undermine the basic collaboration premise[24, 25] of reconstructing the
holistic view based on the multi-view sensors that severely impact the
reliability and quality of collaborative perception process. This raises a
critical inquiry: how to overcome the issues caused by the failed cameras’
perspectives while stabilizing high collaborative performance with low
calibration cost? The designation camera insensitivity overcomes the
unpredictable essence of the specific failure camera numbers and time; see
Figure 1 for an illustration. To address this issue, one viable solution is
adversarial defense[26]. By robust defense strategy, adversarial defense
bypasses camera insensitivity among blurred and noise. However, its
performance is suboptimal[27] and has been shown to be particularly vulnerable
to noise ratios[20] and failed camera numbers.

To address this robust camera insensitivity collaborative perception problem,
we propose RCDN, a Robust Camera-insensitivity collaborative perception with a
Dynamic feature-based 3D Neural modeling mechanism. The core idea is to
recover noisy camera perceptual information from other agents’ views by
modeling the collaborative neural rendering field representations.
Specifically, RCDN has two collaborative field phases: a time-invariant static
background field and time-varying dynamic foreground field. In the static
phases, RCDN sets other baselines’ backbone as the collaboration base and
undertakes end-to-end training to create a robust unified geometry BEV feature
space for all agents. Then, the geometry BEV feature combines the hash grid
modeling, an explicit and multi-resolution network, to generate static
background views through $\alpha$-composed accumulation of RGB values along a
ray at a fast speed. In the dynamic phase, RCDN utilizes 4D spatiotemporal
position features to model the dynamic motion of 3D points, which learns an
accurate motion field under optical priors and spatiotemporal regularization.
The proposed RCDN has two major advantages: i) RCDN can handle camera
insensitivity collaboration under unknown noisy timestamps and numbers; ii)
RCDN does not put any extra communication burden into inference stage and
costs little computation burden.

In our efforts to validate the effectiveness of RCDN, we identified a gap: the
lack of a comprehensive collaborative perception dataset that accounts for
different camera noise scenarios. To address this, we create the OPV2V-N, an
expansive new dataset derived from OPV2V, featuring meticulously labeled
timestamps and camera IDs. This advancement aims to support and enhance
research in camera-insensitive collaborative perception. Extensive experiments
on OPV2V-N show RCDN’s remarkable performance when other baselines equipped
with RCDN under extreme camera-insensitivity setting, improving w.o. RCDN
baseline methods by about $157.91\%$.

##  2 Related Works

#### Robust Single Perception.

Single-agent perceptions[28, 29, 27, 30, 31, 32] have tackled the robust
camera setting with other sensor modals. [27] reveals that camera-based
methods [32] can be easily effected by camera working conditions. Some
works[30, 29] introduce LiDAR into perception system and design a soft-
association mechanism between the LiDAR and the inferior camera-side, to
relieve the negative impacts caused by cameras. MVX-Net[31] improves the
combination pipeline of LiDAR and cameras by leveraging the VoxelNet[33]
architecture. CRN[28] introduces the low-cost Radar to replace the LiDAR,
which can provide precise long-range measurement and operates reliably in all
environments. However, as for the camera-only situation, few work seeks to
solve this because recovering just from the single-view is highly ill-posed
(with infinitely many solutions that match the input image). With the recent
rapid development of V2X[34], we now can introduce the multi-agent and multi-
view based collaborative perception setting to explore this extreme situation.

#### Collaborative Perception.

Perception tasks for single agents can be adversely affected by factors such
as limited sensor fields of view and physical ambient occlusions. To address
the aforementioned challenges, collaborative perception[35, 36, 37] can attain
more comprehensive perceptual output by exchanging perception data. Early
techniques involved the transmission of either unprocessed sensory input
(referred to as early fusion) or the results of perception (referred to as
late fusion). Nevertheless, recent research has been examining the transfer of
intermediate features to achieve a balance between performance and bandwidth.
Some works[38, 39, 40, 41] devote selecting the most informative messages to
communicate. DiscoNet[42] utilizes knowledge distillation to achieve a better
trade-off between performance and bandwidth. V2X-ViT[43] presents a unified
V2X framework based on Transformer that takes into account the heterogeneity
of V2X system. Meanwhile, some learnable or mathematical based methods[44, 45,
46, 47] have also been proposed to correct the pose errors and latency.
Moreover, some works[48, 49] reveal that the holistic character of
collaborative perception can improve the effect of driving planning and
control tasks. However, most existing papers do not take the harsh realities
of real-world sensors into account, such as blurred, high noise, occlusion and
even failure, which directly undermine the basic collaboration premise of
multi-view based modeling, negatively impacting performance. This work
formulates camera-insensitivity collaborative perception, which considers
real-world camera sensor conditions.

#### Neural Rendering.

Neural radiance fields[50] aim to utilize implicit neural representations to
encode densities and colors of the scene. This approach takes advantage of
volumetric rendering to synthesize views, and it can be effectively optimized
from 2D multi-view images. Hence, numerous works have enhanced NeRF in terms
of rendering quality[51, 52, 53], efficiency[54, 55, 56, 57], etc. For
example, Mip-NeRF[58] utilizes cone tracing instead of ray tracing in standard
NeRF volume rendering by introducing integrated positional encoding, which
greatly improves the render quality. To improve the efficiency of training and
inference processes, Instant-NGP[59] proposes a learned parametric multi-
resolution hash for efficient encoding, which also leads to high compactness.
Some works have also extended NeRF to large-scale urban autonomous scenes[60,
61, 62]. In this work, we first introduce neural rendering to collaborative
perception. The proposed collaborative neural rendering field representations
will address the problem of recovering highly noisy perceptual messages.

##  3 Problem Formulation

Consider $N$ agents in a scene, where each agent can send and receive
collaboration messages from other agents. For the $n$-th agent, let
$\mathcal{X}_{n}^{t_{i}}=\\{{\mathcal{I}}_{c}^{t_{i}}\\}_{c=1}^{c_{n}}$ and
$\mathcal{Y}_{n}^{t_{i}}$ be the raw observation and the perception ground-
truth at time current $t_{i}$, respectively, where $\mathcal{I}_{c}^{t_{i}}$
is the $c$-th camera images recorded at $i$-th timestamp, and
$\mathcal{P}_{m\to n}^{t_{i}}$ is the collaboration message sent from the
agent $m$ at time $t_{i}$. The key of the camera insensitivity is that the
specific noisy camera number and corresponding timestamp are unpredictable.
Therefore, each agent has to encounter invalid view information, which
contains both local observation and collaboration messages sent from other
agents. Then, the task of camera insensitivity collaborative perception is
formulated as:

$\displaystyle\max_{\theta_{1},\theta_{2},\mathcal{P}}~{}\sum_{n=1}^{N}g\left(\widehat{\mathbf{Y}}_{n}^{t_{i}},{\mathbf{Y}}_{n}^{t_{i}}\right)$
(1) $\displaystyle{\rm
subject~{}to~{}~{}}\widehat{\mathbf{Y}}_{n}^{t_{i}}=\bm{c}_{\theta_{2}}(\bm{\pi}_{\theta_{1}}({\mathcal{\psi}}(\mathcal{X}_{n}^{t_{i}},\\{\mathcal{P}^{t_{i}}_{m\rightarrow
n}\\}_{m=1}^{N-1}))),$

where $g(\cdot,\cdot)$ is the perception evaluation metrics,
$\widehat{\mathbf{Y}}_{n}^{t_{i}}$ is the perception result of the $n$-th
agent at time $t_{i}$, $\psi(\cdot,\cdot)$ is the camera noise function to
simulate the harsh realities of the real-world situation,
$\bm{\pi}_{\theta_{1}}(\cdot)$ is the proposed collaborative neural rendering
field network RCDN with trainable parameters $\theta_{1}$, and
$\bm{c}_{\theta_{2}}$ is the existing collaborative perception network with
trainable parameters $\theta_{2}$. Note that the proposed RCDN is to recover
the noisy camera views caused by the $\psi$ function, making collaborative
perception system more robust to the unpredictable situation of noisy camera
data.

Given such high noisy camera view, the performances of collaborative
perception system would be significantly degraded since the mainstream
collaborative perception utilizes the multi-view camera-based BEV features for
communication and downstream tasks, and using such damaged features would
contain erroneous information during the perception process. In the next
section, we will introduce RCDN to address this issue.

##  4 RCDN

Figure 2: System overview. The geometry BEV generation module provides feature
sampling for later processes. The collaborative static and dynamic fields are
performed in parallel to model the background and foreground, respectively.
Note that MCP is short for the multi-agents collaborative perception process.

This section proposes a robust camera-insensitivity collaborative perception
system, RCDN. Figure 2 overviews the framework of the RCDN module in Sec.4.1.
The details of three key modules of RCDN can be found in Sec.4.2-4.4.

###  4.1 Overall Architecture

The problem of noisy camera view results in the sub-optimization of the
holistic multi-view based BEV features generation in the collaboration
messages. That is, the collaboration messages from both self and other agents
would be noisy or damaged for the fusion process. The proposed RCDN addresses
this issue with two key notions: i) we construct novel collaborative neural
rendering field representations, enabling collaborative perception to recover
from the noisy camera view; and ii) we establish time-invariant and time-
varying fields for background and foreground, respectively, making the
collaborative neural rendering field more accurate.

Mathematically, let the $n$-th agent be the ego agent and
$\mathcal{X}_{n}^{t_{i}}$ be its raw observation at the $t_{i}$ timestamp of
agent $n$. The proposed camera-insensitivity collaborative perception system
RCDN is formulated as follows:

$\displaystyle\mathbf{F}_{n}^{t_{i}}$
$\displaystyle=f_{\rm{enc}}(\psi(\mathcal{X}_{n}^{t_{i}},\\{\mathcal{X}_{j}^{t_{i}}\\}_{j=1}^{N-1})),$
(2a) $\displaystyle\mathbf{V}_{icv}^{t_{i}}$
$\displaystyle=f_{\rm{geo\\_bev}}(\mathbf{F}_{n}^{t_{i}}),$ (2b)
$\displaystyle(\sigma^{s},\mathbf{c}^{s})$
$\displaystyle=f_{\rm{static}}(\mathbf{r}(u_{k}),\mathbf{V}_{icv}^{t_{i}}(\mathbf{r}(u_{k}))),$
(2c)
$\displaystyle(\mathbf{s}_{fw},\mathbf{s}_{bw},\sigma^{d}_{t_{i}},\mathbf{c}^{d}_{t_{i}},\mathbf{b})$
$\displaystyle=f_{\rm{dynamic}}(\mathbf{r}(u_{k}),\mathbf{V}_{icv}^{t_{i}}(\mathbf{r}(u_{k})),t_{i}),$
(2d)
$\displaystyle\widetilde{\mathcal{X}}_{n}^{t_{i}},\\{\widetilde{\mathcal{X}}_{j}^{t_{i}}\\}_{j=1}^{N-1}$
$\displaystyle=f_{\rm{render}}(\sigma^{s},\mathbf{c}^{s},\sigma^{d}_{t_{i}},\mathbf{c}^{d}_{t_{i}},b),$
(2e) $\displaystyle\widehat{\mathbf{Y}}_{n}^{t_{i}}$
$\displaystyle=f_{\rm{mcp}}(\widetilde{\mathcal{X}}_{n}^{t_{i}},\\{\widetilde{\mathcal{X}}_{j}^{t_{i}}\\}_{j=1}^{N-1}),$
(2f)

where $\mathbf{F}_{n}^{t_{i}}\in\mathbb{R}^{C\times H\times W}$ is the BEV
feature maps of the $n$-th agent at timestamp $t_{i}$ with $H,W$ the size of
BEV map and $C$ the number of channels;
$\mathbf{V}_{icv}^{t_{i}}\in\mathbb{R}^{C\times Z\times H\times W}$ is the
implicit collaborative geometry volume feature of the scenarios; which is
lifted from BEV plane with the $Z$ height; $\mathbf{r}(u(k))$ is the ray from
the failed camera center $\mathbf{o}\in\mathbb{R}^{2}$ through a given pixel
on the image plane as $\mathbf{r}(u(k))=\mathbf{o}+u(k)\mathbf{d}$, where
$\mathbf{d}\in\mathbb{R}^{3}$ is the normalized viewing direction;
$f_{\rm{static}}$ is a explicit hash grid based representation to model the
collaborative static scenarios volume density $\sigma^{s}\in\mathbb{R}^{1}$
and corresponding color $\mathbf{c}^{s}\in\mathbb{R}^{3}$; $f_{\rm{dynamic}}$
is the dynamic collaborative neural network takes the interpolated 4D-tuple
$(\mathbf{r}(u(k)),t_{i})$ and sampled $\mathbf{V}_{icv}^{t_{i}}$ feature as
input and predict 3D collaborative scene flow vectors
$\mathbf{s}_{fw},\mathbf{s}_{bw}\in\mathbb{R}^{3}$, dynamic volume density
$\sigma_{t_{i}}^{d}$, color $\mathbf{c}_{t_{i}}^{d}$ and blending weight
$\mathbf{b}\in\mathbb{R}^{2}$; and
$\widetilde{\mathcal{X}}_{n}^{t_{i}},\\{\widetilde{\mathcal{X}}_{j}^{t_{i}}\\}_{j=1}^{N-1}$
is the recovered noisy camera images at timestamp $t_{i}$ after collaborative
rendering; and $\widehat{\mathbf{Y}}_{n}^{t_{i}}$ is the final output of the
system. In summary, Step 2a extracts BEV perceptual features from observation
data. Step 2b generates the collaborative geometry BEV volume feature map for
each timestamp, enabling feature sampling in Step 2c and 2d. Step 2d models
the static background field of collaboration scenarios. Step 2d models the
dynamic foreground field of collaboration objects. Step 2e gets the global
volume density and color information by combining both static and dynamic
field models to recover the failed camera perspective images. Finally, Step 2f
outputs the final perceptual results with repaired images.

Note that i) Step 2a is done locally, Step 2b-2f are performed after receiving
the messages from others. The proposed RCDN does not require any extra
transmission during the inference process, which is bandwidth friendly; and
ii) Step 2c and 2d are performed in parallel to save inference time; and iii)
Same as [42, 47], RCDN adopts the feature representations in bird’s eye view
(BEV), where the feature maps of all agents are projected to the same global
coordinate system. We now elaborate on the details of Steps 2b-2e in the
following subsections.

###  4.2 Collaborative Geometry BEV Volume Feature

Given the BEV feature map of each agent, Step 2b aims to construct a unified
collaborative geometry BEV volume feature for each timestamp of the scenario.
The intuition is that [63] points out that combing with generic feature
representations can avoid the per-scene "network memorization" phenomenon[50],
which will improve the efficiency of the optimization process. Therefore,
using the geometry BEV feature can enable the subsequent Step 2c, 2d to learn
more generic networks for both static and dynamic collaborative neural fields,
respectively.

To implement, we use a geometry-aware decoder $D_{geo}$ to transform the BEV
feature $\mathbf{F}_{n}^{t_{i}}$ into the intermediate feature
$\mathbf{F}_{n}^{{}^{\prime}t_{i}}\in\mathbb{R}^{C\times 1\times X\times Y}$
and $\mathbf{F}_{height,n}^{t_{i}}\in\mathbb{R}^{1\times Z\times X\times Y}$,
and this feature is lifted from BEV plane to an implicit collaborative volume
feature $\mathbf{V}_{icv}^{t_{i}}\in\mathbb{R}^{C\times Z\times X\times Y}$:

$\mathbf{V}_{icv}^{t_{i}}=\mathrm{sigmoid}(\mathbf{F}_{height,n}^{t_{i}})\cdot\mathbf{F}_{n}^{{}^{\prime}t_{i}},$
(3)

where $\cdot$ represents dot production along the channel. Eq. 3 lifts the
items on the BEV plane into 3D collaborative volume with the estimated height
position $\mathrm{sigmoid}(\mathbf{F}_{height,n}^{t_{i}})$.
$\mathrm{sigmoid}(\mathbf{F}_{height,n}^{t_{i}})$ represents whether there is
an item at the corresponding height. Ideally, the collaborative volume feature
$\mathbf{V}_{icv}^{t_{i}}$ contains all the scene items information in the
corresponding position.

###  4.3 Static Collaborative Neural Field

After getting the collaborative volume feature $\mathbf{V}_{icv}^{t_{i}}$,
Step 2c aims to construct the background of camera views with the static
collaborative neural field. Given an arbitrary 3D scenario position
$\mathbf{x}\in\mathbb{R}^{3}$ and a 2D viewing direction
$\mathbf{d}\in\mathbb{R}^{2}$, we aims to estimate static scenarios volume
density $\sigma^{s}$ and emitted RGB color $\mathbf{c}^{s}$ using the fast
hash grid-based [59] neural network:

$(\mathbf{c}^{s},\sigma^{s})=\mathrm{MLP}(\mathbf{G}_{\theta}^{s}(\mathrm{contract}(\mathbf{x}),\mathbf{d});f),~{}~{}f=\mathbf{V}_{icv}^{t_{i}}(\mathbf{x}),$
(4)

where $f=\mathbf{V}_{icv}^{t_{i}}(\mathbf{x})$ is the neural feature
trilinearly interpolated from the collaborative geometry BEV volume
$\mathbf{V}_{icv}^{t_{i}}$ at the location $\mathbf{x}$,
$\mathbf{G}_{\theta}^{s}(\cdot,\cdot)$ is explicit multi-level hash grid
representation with the generic $f$ features for fast static collaborative
neural field training. Meanwhile, owing to the collaborative scenarios are
unbounded, we utilize $\mathrm{contract}(\cdot)$[51] to map 3D scenario
position into a bounded ball of radius 2 with regularization, making the
estimation optimization process faster and better. Hence, we can compute the
color of the pixel (corresponding to the ray $\mathbf{r}(u_{k})$ using
numerical quadrature for approximating the collaborative volume rendering
interval[64]:

$\displaystyle\mathbf{C}^{s}(\mathbf{r})$
$\displaystyle=\sum_{k=1}^{K}T^{s}(u_{k})\alpha^{s}(\sigma^{s}(u_{k})\delta_{k})\mathbf{c}^{s}(u_{k}),$
(5a) $\displaystyle{T}^{s}(u_{k})$
$\displaystyle=\mathrm{exp}\left(-\sum_{k^{{}^{\prime}}=1}^{k-1}\sigma^{s}(u_{k})\delta_{k}\right),$
(5b)

where $\alpha^{s}(x)=1-\mathrm{exp}(-x)$ and $\delta_{k}=u_{k+1}-u_{k}$ is the
distance between two quadrature points. The $K$ quadrature points
$\\{u_{k}\\}_{k=1}^{K}$ are drawn uniformly between $u_{n}$ and $u_{f}$, which
denotes the near and far of the bounded collaborative scenarios.
$T^{s}(u_{k})$ indicates the accumulated transmittance from $u_{n}$ to
$u_{k}$. Here, we denote $\mathbf{r}_{i}$ as the rays passing through the
pixel $i$. Then, the collaborative static neural loss $\mathcal{L}_{static}$
is defined to minimize the $l_{2}$-loss between the estimated colors
$\mathbf{C}^{s}(\mathbf{r}_{i})$ and the ground truth colors
$\mathbf{C}^{gt}(\mathbf{r}_{i})$ in the static regions (where
$\mathbf{M}({\mathbf{r}_{i}})=0$):

$\mathcal{L}_{static}=\sum_{i}\Arrowvert{\mathbf{C}^{s}(\mathbf{r}_{i})-\mathbf{C}^{gt}(\mathbf{r}_{i})\cdot(1-\mathbf{M}(\mathbf{r}_{i}))}\Arrowvert_{2}^{2}$
(6)

###  4.4 Dynamic Collaborative Neural Field

While the static collaborative neural field is being modeled, Step 2d is
building the dynamic collaborative neural field to construct the foreground of
camera views. Our dynamic collaborative neural field takes 4D spatiotemporal
position features as input to model dynamic motion of 3D scene flow
$\mathbf{s}_{fw},\mathbf{s}_{bw}$, volume density $\sigma_{t_{i}}^{d}$, color
$\mathbf{c}_{t_{i}}^{d}$ and blending weight $\mathbf{b}$ (Note that blending
weights learns how to blend the results from both the static and dynamic
collaborative neural fields in an unsupervised manner, avoiding background’s
structure and appearance conflict the moving objects.):

$(\mathbf{s}_{fw},\mathbf{s}_{bw},\mathbf{c}_{t_{i}}^{d},\sigma_{t_{i}}^{d},\mathbf{b})=\mathrm{MLP}(\Delta(\mathbf{G}_{\theta}^{d}(\mathrm{contract}(\mathbf{x}),\mathbf{d}),t_{i});f),~{}~{}f=\mathbf{V}_{icv}^{t_{i}}(\mathbf{x}),$
(7)

where $G_{\theta}^{d}$ shares the same hash grid representations, but for the
dynamic collaborative neural field optimization; $\Delta(\cdot,\cdot)$ is the
temporal interpolation functions, which makes the $\mathrm{MLP}$ can
efficiently learn the features between keyframes in a scalable manner.
Meanwhile, to improve the temporal consistency of the proposed field, we
compute the collaborative scene flow neighbors
$\mathbf{r}(u_{k})+\mathbf{s}_{fw}$ and $\mathbf{r}(u_{k})-\mathbf{s}_{bw}$
with the predicted collaborative scene flow $\mathbf{s}_{fw},\mathbf{s}_{bw}$
to warp the collaborative neural field from the neighboring time instance to
the current time. Hence, we can obtain the corresponding density and color of
adjacent time by querying the same $\mathrm{MLPs}$ model at
$\mathbf{r}(u_{k})+\mathbf{s}$:

$\displaystyle(\mathbf{c}_{t_{i}+1}^{d},\sigma_{t_{i}+1}^{d})$
$\displaystyle=\mathrm{MLP}(\Delta(\mathbf{G}_{\theta}^{d}(\mathrm{contract}(\mathbf{x}+\mathbf{s}_{fw}),\mathbf{d}),t_{i}+1))$
(8a) $\displaystyle(\mathbf{c}_{t_{i}-1}^{d},\sigma_{t_{i}-1}^{d})$
$\displaystyle=\mathrm{MLP}(\Delta(\mathbf{G}_{\theta}^{d}(\mathrm{contract}(\mathbf{x}-\mathbf{s}_{bw}),\mathbf{d}),t_{i}-1))$
(8b)

We can compute the color of a dynamic pixel of collaborative view at time
$t_{i}$. Hence, with both the static and dynamic collaborative neural fields
model, we can easily compose them into a complete model using the predicted
blending weight $\mathbf{b}$ and render full color
$\mathbf{C}^{full}(\mathbf{r})$ frames at noisy views and time. We utilize the
following approximate of collaborative volume rendering integral:

$\mathbf{C}_{t_{i}}^{full}(\mathbf{r})=\sum_{k=1}^{K}T_{t_{i}}^{full}\left(\alpha^{d}(\sigma_{t_{i}}^{d}\delta_{k})(1-\mathbf{b})\mathbf{c}_{t_{i}}^{d}+\alpha^{s}(\sigma^{s}\delta_{k})\mathbf{b}\mathbf{c}^{s}\right)$
(9)

Similar to the static collaborative rendering loss, we train the dynamic
collaborative neural model by minimizing the $l_{2}$ reconstruction loss under
time unit $\tau=\\{t_{i},t_{i}-1,t_{i}+1\\}$:

$\mathcal{L}_{dyn}=\sum_{t\in\tau}\sum_{i}\Arrowvert(\mathbf{C}_{t}^{full}(\mathbf{r}_{i})-\mathbf{C}^{gt}(\mathbf{r}_{i}))\Arrowvert_{2}^{2}$
(10)

To reduce the amount of ambiguity caused by the sparse views during
collaborative perception process, we construct motion matching loss to
constrain the proposed dynamic collaborative neural field. As we do not have
direct 3D supervision for predicted collaborative scene flow from the motion
$\mathrm{MLP}$ model, we utilize 2D optical flow $\mathit{\boldsymbol{f}}$ as
indirect supervision. Specifically, we first use the estimated collaborative
scene flow to obtain the corresponding 3D point. Then, we project these 3D
points onto the 2D reference frame with $\varphi(\cdot)$ function. Hence, we
can compute the projected collaborative scene optical flow and enforce it to
match the estimated optical flow as follows:

$\mathcal{L}_{opt}=\sum_{i}\left(\varphi(\mathit{\boldsymbol{s}}_{\\{bw,fw\\}}(\mathbf{r}_{i}))-\mathit{\boldsymbol{f}}_{\\{bw,fw\\}}^{gt}(\mathbf{r}_{i})\right)$
(11)

Meanwhile, we also regularize the consistency of the collaborative scene flow
by minimizing the cycle consistency loss $\mathcal{L}_{cyc}$. See more details
in the Appendix B.6.

###  4.5 Training Details and Optimization

To train the overall system, we supervise two tasks: static and dynamic
collaborative neural fields, respectively. Meanwhile, during the training
process, the static collaborative field and dynamic collaborative field are
trained separately. The total reconstruction loss is a weighted combination as
the following:

$\mathcal{L}_{total}=\lambda_{1}\mathcal{L}_{static}+\lambda_{2}\mathcal{L}_{dyn}+\lambda_{3}\mathcal{L}_{opt}+\lambda_{4}\mathcal{L}_{cyc}$
(12)

##  5 Experimental Results

We create the first camera-insensitivity collaborative perception dataset and
conduct extensive experiments on OPV2V-N. To ensure the consistency of the
input noisy camera data and verify the effectiveness of RCDN, we set the noisy
camera data to be in the failed situation[27]. Meanwhile, the task of the
experiments is map segmentation, including the performance of the vehicle,
drivable area (Dr. area) and lane, totaling three classes. We utilize the
Intersection over Union (IoU) between map prediction and ground truth map-view
labels as the performance metric.

###  5.1 Datasets

OPV2V-N. To facilitate research on camera-insensitivity for collaborative
perception, we propose a simulation dataset dubbed OPV2V-N. In OPV2V dataset,
there is a lack of mask labels for distinguishing between foreground and
background views, as well as optical flow labels for supervising the scene
flow. For this purpose, we collect more data to bridge the gap between neural
field and collaborative perception, leading to the new OPV2V-N datasets.
Specifically, we utilize the OneFormer[65] detector to extract the foreground
mask labels and mainstream RAFT[66] detector to compute the optical flow
between image pairs. Meanwhile, we manually annotate which part of the
performance degradation is triggered by camera failure in different scenarios.
See more details in the Appendix A.

###  5.2 Quantitative Evaluation

Table 1: Map-view segmentation of different baseline methods w.o/w the
proposed RCDN on the OPV2V-N camera-track with one random noisy camera failure
in the testing phase. We report IoU for all classes.

Model / Metric | Static Part (Perf. Comparison) | Dynamic Part Vehicle
---|---|---
Drivable Area | Lane
Normal | Failure | Normal | Failure | Normal | Failure
w.o/w. RCDN |  w.o/w. RCDN |  w.o/w. RCDN
F-Cooper[1] | 45.44 | 28.87/44.89($\uparrow$55.49%) | 33.17 | 15.95/32.23($\uparrow$102.07%) | 63.33 | 29.70/61.76($\uparrow$107.95%)
AttFuse[16] | 45.59 | 27.99/44.38($\uparrow$58.56%) | 33.76 | 18.77/31.50($\uparrow$67.82%) | 54.14 | 24.76/52.15($\uparrow$110.62%)
DiscoNet[42] | 42.30 | 24.31/38.54($\uparrow$58.54%) | 24.24 | 12.29/22.97($\uparrow$86.90%) | 46.56 | 9.25/43.03($\uparrow$365.19%)
V2VNet[35] | 41.70 | 27.99/39.72($\uparrow$41.91%) | 27.14 | 10.52/25.24($\uparrow$139.92%) | 42.57 | 11.28/42.76($\uparrow$279.08%)
CoBEVT[6] | 51.96 | 32.08/47.19($\uparrow$47.10%) | 34.19 | 14.45/29.55($\uparrow$104.50%) | 56.61 | 32.41/55.10($\uparrow$70.01%)

Benchmark comparison. The baseline methods include F-Cooper[1], AttFuse[16],
DiscoNet[42], V2VNet[35] and CoBEVT[6]. All methods use the same BEV feature
encoder based on CVT[67]. To validate the portability of the RCDN, we compare
different baseline methods w.o/w. RCDN under unpredictable camera failure
settings. Table 1 shows that the map-view segmentation performance of
different baseline methods w.o/w. the proposed RCDN with only one number
random noisy camera failure in the testing phase on the OPV2V-N dataset. We
see that i) for static part, each baseline method with one camera failure
drops about $37.73\%/42.54\%/32.87\%$ (Avg/Max/Min) and
$52.93\%/61.25\%/44.40\%$ for drivable area and lane, respectively. However,
each baseline method w. RCDN under the same camera failure situation only
decreases about $5.34\%/9.17\%/1.22\%$ and $7.08\%/13.59\%/2.85\%$,
respectively. Compared to the w.o RCDN baseline methods, RCDN can improve the
performance of drivable area and lane for $52.32\%/58.54\%/47.10\%$ and
$100.37\%/139.92\%/67.82\%$, respectively; ii) compared to the static part, as
we all know, the fusion stage in collaborative perception process needs more
effort on the multi-view based BEV feature map to highlight the corresponding
dynamic part. Hence, the baseline methods’ dynamic performance suffers more
from camera failure than the static part, causing about a
$60.75\%/42.72\%/80.14\%$ performance drop. Nevertheless, RCDN also
demonstrates robustness to the dynamic foreground object modeling, with only a
$3.31\%/7.58\%/0.47\%$ performance decrease for the dynamic part, improving
the w.o RCDN baseline methods’ performance by $186.57\%/365.19\%/70.01\%$.
Meanwhile, as for the communication cost, similar to [42], we only utilize the
$\mathbf{C}^{gt}$ labels during the training stage, meaning we leave the
communication burden to the training stage and do not introduce extra
information during the inference.

Robust to extremely noisy camera data. We conduct experiments to validate the
performance under the impact of random noisy camera numbers. Figure 3 shows
the map-view segmentation performance of the different baselines methods
w.o/w. the proposed RCDN under varying levels of camera failures situation on
OPV2V-N, where the $x$-axis is the expectation of the number of random failed
cameras during the inference stage and $y$-axis the segmentation performance.
Note that, when the $x$-axis is at 0, it represents standard collaborative
perception without any camera failures. We see that i) the proposed RCDN can
stabilize all the baseline methods in both static and dynamic part of map-view
segmentation performance at all camera failure settings; ii) as for the static
part, with the RCDN can maintain the $87.84\%/88.72\%/86.64\%$ Dr. area
performance of the standard setting even under three random failed views
during the collaboration process, compared with the w.o. RCDN only about
$47.68\%/57.48\%/37.15\%$. Note that the V2VNet baseline method’s performance
degrades sharply as the failed camera number increases, however, with RCDN,
the V2VNet can settle in a considerable performance even with the failed
camera number increases; iii) as for the dynamic part, some baseline methods
are crashed even with only one random camera failure situation, e.g. DiscoNet
only maintains about $19.87\%$ performance of the standard collaborative
perception setting, and almost every baseline method is unusable when there
are three random camera failures, only about $20.73\%/28.11\%/13.09\%$ of the
standard situation. Nevertheless, with the RCDN, we see that all baseline
methods still perform well even when three random failed camera situation
appear, maintaining the $84.95\%/90.81\%/75.93\%$ dynamic performance of the
standard situation.

Figure 3: Comparison of the performance of other baseline methods w.o/w the
proposed RCDN under the random noisy (failed situation) camera numbers from 0
to 3. RCDN can be ported to other baseline methods and stabilize the
performance under different level camera failure situations on OPV2V-N
dataset.

Table 2: Ablation Study on OPV2V-N dataset.

Modules | Dr. Area | Lanes | Dynamic Veh.
---|---|---|---
Neural Field | Time Model
✗ | ✗ | 24.55 | 10.07 | 30.67
✔ | ✗ | 24.47 | 11.71 | 41.55
✔ | ✔ | 27.37 | 10.63 | 46.65

Figure 4: Effectiveness of dynamic neural field.

###  5.3 Qualitative Evaluation

Visualization of segmentation. We illustrate the map-view segmentation of
other baseline methods w.o/w. RCDN and the corresponding repaired camera view
in Figure 5. The random camera failure number is one. The orange represents
the drivable area, the blue represents the lanes and the teal represents the
vehicles. We can see that i) other baseline methods show significant
improvement in w. RCDN under noisy camera data; ii) V2VNet that collapses with
noise camera data can also achieve the same level of performance as the origin
data with the help of RCDN.

Figure 5: Visualization of different baseline methods w. RCDN with one random
camera failure.

###  5.4 Ablation Study

Figure 6: Comparison between existing dynamic field modeling and the proposed
RCDN.

We conduct ablation studies on OPV2V-N with the CoBEVT baseline method. Table
4 assesses the effectiveness of the proposed two field phases. We see that i)
only one neural field can recover most static part performance from the noisy
camera data; ii) the proposed time model in collaborative dynamic fields can
handle the motion blurry caused by the vehicles, shown in Figure 4. Meanwhile,
we compare the training efficiency of the proposed RCDN with existing dynamic
fields modeling methods[68], as illustrated in Figure 6. Our approach, which
leverages explicit grid and geometry feature-based representations,
accelerates the training process by approximately $24\times$ compared to the
existing implicit MLP-based modeling, while also achieving superior PSNR
quality. See more discussions in the Appendix B.2.

##  6 Conclusion and Limitation

We formulate the camera-insensitivity collaborative perception task, which
considers harsh realities of real-world sensors that may cause unpredictable
random camera failures during collaborative communication. We further propose
RCDN, a robust camera-insensitivity collaborative perception with a novel
dynamic feature-based 3D neural modeling. The core idea of RCDN is to
construct collaborative neural rendering field representations to recover
failed perceptual messages sent by multiple agents. Comprehensive experiments
show that RCDN can be portable to other baseline methods and stabilize the
performance with a considerable level under all settings and far superior
robustness with random camera failures.

#### Limitation and future work.

The current work focuses on addressing the camera-insensitivity problem in
collaborative perception. It is evident that accurate reconstruction can
compensate for the negative impact of noisy camera features on collaborative
perception. In the future, we expect more works on exploring real-time
collaborative neural field modeling with 3D Gaussian splatting.

## References

  * [1] Qi Chen, Xu Ma, Sihai Tang, Jingda Guo, Qing Yang, and Song Fu.  F-cooper: Feature based cooperative perception for autonomous vehicle edge computing system using 3d point clouds.  In Proceedings of the 4th ACM/IEEE Symposium on Edge Computing, pages 88–100, 2019.
  * [2] Runsheng Xu, Yi Guo, Xu Han, Xin Xia, Hao Xiang, and Jiaqi Ma.  Opencda: an open cooperative driving automation framework integrated with co-simulation.  In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 1155–1162. IEEE, 2021.
  * [3] Yiming Li, Dekun Ma, Ziyan An, Zixun Wang, Yiqi Zhong, Siheng Chen, and Chen Feng.  V2x-sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving.  IEEE Robotics and Automation Letters, 7(4):10914–10921, 2022.
  * [4] James Tu, Tsunhsuan Wang, Jingkang Wang, Sivabalan Manivasagam, Mengye Ren, and Raquel Urtasun.  Adversarial attacks on multi-agent communication.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7768–7777, 2021.
  * [5] Jiaxun Cui, Hang Qiu, Dian Chen, Peter Stone, and Yuke Zhu.  Coopernaut: End-to-end driving with cooperative perception for networked vehicles.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17252–17262, 2022.
  * [6] Runsheng Xu, Zhengzhong Tu, Hao Xiang, Wei Shao, Bolei Zhou, and Jiaqi Ma.  Cobevt: Cooperative bird’s eye view semantic segmentation with sparse transformers.  2022\.
  * [7] Hao Xiang, Runsheng Xu, and Jiaqi Ma.  Hm-vit: Hetero-modal vehicle-to-vehicle cooperative perception with vision transformer.  arXiv preprint arXiv:2304.10628, 2023.
  * [8] Ebtehal Turki Alotaibi, Shahad Saleh Alqefari, and Anis Koubaa.  Lsar: Multi-uav collaboration for search and rescue missions.  IEEE Access, 7:55817–55832, 2019.
  * [9] Jürgen Scherer, Saeed Yahyanejad, Samira Hayat, Evsen Yanmaz, Torsten Andre, Asif Khan, Vladimir Vukadinovic, Christian Bettstetter, Hermann Hellwagner, and Bernhard Rinner.  An autonomous multi-uav system for search and rescue.  In Proceedings of the First Workshop on Micro Aerial Vehicle Networks, Systems, and Applications for Civilian Use, page 33–38, New York, NY, USA, 2015.
  * [10] Yue Hu, Shaoheng Fang, Weidi Xie, and Siheng Chen.  Aerial monocular 3d object detection.  IEEE Robotics and Automation Letters, 8(4):1959–1966, 2023.
  * [11] Lukas Bernreiter, Shehryar Khattak, Lionel Ott, Roland Siegwart, Marco Hutter, and Cesar Cadena.  A framework for collaborative multi-robot mapping using spectral graph wavelets.  The International Journal of Robotics Research, 0(0):02783649241246847, 0.
  * [12] Luiz Eugênio Santos Araújo Filho and Cairo Lúcio Nascimento Júnior.  Multi-robot autonomous exploration and map merging in unknown environments.  In 2022 IEEE International Systems Conference (SysCon), pages 1–8, 2022.
  * [13] Yiming Li, Juexiao Zhang, Dekun Ma, Yue Wang, and Chen Feng.  Multi-robot scene completion: Towards task-agnostic collaborative perception.  In 6th Annual Conference on Robot Learning, 2022.
  * [14] Runsheng Xu, Xin Xia, Jinlong Li, Hanzhao Li, Shuo Zhang, Zhengzhong Tu, Zonglin Meng, Hao Xiang, Xiaoyu Dong, Rui Song, et al.  V2v4real: A real-world large-scale dataset for vehicle-to-vehicle cooperative perception.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13712–13722, 2023.
  * [15] Haibao Yu, Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang, Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, et al.  Dair-v2x: A large-scale dataset for vehicle-infrastructure cooperative 3d object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21361–21370, 2022.
  * [16] Runsheng Xu, Hao Xiang, Xin Xia, Xu Han, Jinlong Li, and Jiaqi Ma.  Opv2v: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication.  In 2022 International Conference on Robotics and Automation (ICRA), pages 2583–2589. IEEE, 2022.
  * [17] Walter Zimmer, Gerhard Arya Wardana, Suren Sritharan, Xingcheng Zhou, Rui Song, and Alois C. Knoll.  Tumtraf v2x cooperative perception dataset.  In 2024 IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR). IEEE/CVF, 2024.
  * [18] Runsheng Xu, Jinlong Li, Xiaoyu Dong, Hongkai Yu, and Jiaqi Ma.  Bridging the domain gap for multi-agent perception.  arXiv preprint arXiv:2210.08451, 2022.
  * [19] Yunsheng Ma, Juanwu Lu, Can Cui, Sicheng Zhao, Xu Cao, Wenqian Ye, and Ziran Wang.  Macp: Efficient model adaptation for cooperative perception.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3373–3382, 2024.
  * [20] Yiming Li, Qi Fang, Jiamu Bai, Siheng Chen, Felix Juefei-Xu, and Chen Feng.  Among us: Adversarially robust collaborative perception by consensus.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 186–195, October 2023.
  * [21] James Tu, Tsunhsuan Wang, Jingkang Wang, Sivabalan Manivasagam, Mengye Ren, and Raquel Urtasun.  Adversarial attacks on multi-agent communication.  In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7748–7757, 2021.
  * [22] Francesco Secci and Andrea Ceccarelli.  On failures of rgb cameras and their effects in autonomous driving applications.  In 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE), pages 13–24, 2020.
  * [23] Kui Ren, Qian Wang, Cong Wang, Zhan Qin, and Xiaodong Lin.  The security of autonomous driving: Threats, defenses, and future directions.  Proceedings of the IEEE, 108(2):357–372, 2020.
  * [24] Rui Song, Chenwei Liang, Hu Cao, Zhiran Yan, Walter Zimmer, Markus Gross, Andreas Festag, and Alois Knoll.  Collaborative semantic occupancy prediction with hybrid feature fusion in connected automated vehicles.  2024\.
  * [25] Xiang Li, Junbo Yin, Wei Li, Chengzhong Xu, Ruigang Yang, and Jianbing Shen.  Di-v2x: Learning domain-invariant representation for vehicle-infrastructure collaborative 3d object detection.  Proceedings of the AAAI Conference on Artificial Intelligence, 38(4):3208–3215, Mar. 2024.
  * [26] Hao Xiang, Runsheng Xu, Xin Xia, Zhaoliang Zheng, Bolei Zhou, and Jiaqi Ma.  V2xp-asg: Generating adversarial scenes for vehicle-to-everything perception.  In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3584–3591, 2023.
  * [27] Kaicheng Yu, Tang Tao, Hongwei Xie, Zhiwei Lin, Tingting Liang, Bing Wang, Peng Chen, Dayang Hao, Yongtao Wang, and Xiaodan Liang.  Benchmarking the robustness of lidar-camera fusion for 3d object detection.  In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 3188–3198, 2023.
  * [28] Youngseok Kim, Juyeb Shin, Sanmin Kim, In-Jae Lee, Jun Won Choi, and Dongsuk Kum.  Crn: Camera radar net for accurate, robust, efficient 3d perception.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 17615–17626, October 2023.
  * [29] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai.  Transfusion: Robust lidar-camera fusion for 3d object detection with transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1090–1099, June 2022.
  * [30] Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, and Zhi Tang.  Bevfusion: A simple and robust lidar-camera fusion framework.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 10421–10434. Curran Associates, Inc., 2022.
  * [31] Vishwanath A. Sindagi, Yin Zhou, and Oncel Tuzel.  Mvx-net: Multimodal voxelnet for 3d object detection.  In 2019 International Conference on Robotics and Automation (ICRA), pages 7276–7282, 2019.
  * [32] Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon.  Detr3d: 3d object detection from multi-view images via 3d-to-2d queries.  In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors, Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 180–191. PMLR, 08–11 Nov 2022.
  * [33] Yin Zhou and Oncel Tuzel.  Voxelnet: End-to-end learning for point cloud based 3d object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4490–4499, 2018.
  * [34] Maria Christopoulou, Sokratis Barmpounakis, Harilaos Koumaras, and Alexandros Kaloxylos.  Artificial intelligence and machine learning as key enablers for v2x communications: A comprehensive survey.  Vehicular Communications, 39:100569, 2023.
  * [35] Tsun-Hsuan Wang, Sivabalan Manivasagam, Ming Liang, Bin Yang, Wenyuan Zeng, and Raquel Urtasun.  V2vnet: Vehicle-to-vehicle communication for joint perception and prediction.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 605–621, 2020.
  * [36] Yifan Lu, Yue Hu, Yiqi Zhong, Dequan Wang, Siheng Chen, and Yanfeng Wang.  An extensible framework for open heterogeneous collaborative perception.  In The Twelfth International Conference on Learning Representations, 2024.
  * [37] Yue Hu, Yifan Lu, Runsheng Xu, Weidi Xie, Siheng Chen, and Yanfeng Wang.  Collaboration helps camera overtake lidar in 3d detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9243–9252, 2023.
  * [38] Yen-Cheng Liu, Junjiao Tian, Chih-Yao Ma, Nathan Glaser, Chia-Wen Kuo, and Zsolt Kira.  Who2com: Collaborative perception via learnable handshake communication.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 6876–6883, 2020.
  * [39] Yen-Cheng Liu, Junjiao Tian, Nathaniel Glaser, and Zsolt Kira.  When2com: multi-agent perception via communication graph grouping.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4106–4115, 2020.
  * [40] Yue Hu, Shaoheng Fang, Zixing Lei, Yiqi Zhong, and Siheng Chen.  Where2comm: Communication-efficient collaborative perception via spatial confidence maps.  Advances in neural information processing systems, 35:4874–4886, 2022.
  * [41] Tianhang Wang, Guang Chen, Kai Chen, Zhengfa Liu, Bo Zhang, Alois Knoll, and Changjun Jiang.  Umc: A unified bandwidth-efficient and multi-resolution based collaborative perception framework.  In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8153–8162, 2023.
  * [42] Yiming Li, Shunli Ren, Pengxiang Wu, Siheng Chen, Chen Feng, and Wenjun Zhang.  Learning distilled collaboration graph for multi-agent perception.  Advances in Neural Information Processing Systems, 34:29541–29552, 2021.
  * [43] Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Ming-Hsuan Yang, and Jiaqi Ma.  V2x-vit: Vehicle-to-everything cooperative perception with vision transformer.  ArXiv, abs/2203.10638, 2022.
  * [44] Nicholas Vadivelu, Mengye Ren, James Tu, Jingkang Wang, and Raquel Urtasun.  Learning to communicate and correct pose errors.  In 4th Conference on Robot Learning (CoRL), 2020.
  * [45] Yifan Lu, Quanhao Li, Baoan Liu, Mehrdad Dianati, Chen Feng, Siheng Chen, and Yanfeng Wang.  Robust collaborative 3d object detection in presence of pose errors.  In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 4812–4818. IEEE, 2023.
  * [46] Zixing Lei, Shunli Ren, Yue Hu, Wenjun Zhang, and Siheng Chen.  Latency-aware collaborative perception.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII, pages 316–332. Springer, 2022.
  * [47] Sizhe Wei, Yuxi Wei, Yue Hu, Yifan Lu, Yiqi Zhong, Siheng Chen, and Ya Zhang.  Asynchrony-robust collaborative perception via bird’s eye view flow.  In Advances in Neural Information Processing Systems, 2023.
  * [48] Dian Chen and Philipp Krähenbühl.  Learning from all vehicles.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17222–17231, 2022.
  * [49] Ruizhao Zhu, Peng Huang, Eshed Ohn-Bar, and Venkatesh Saligrama.  Learning to drive anywhere.  In 7th Annual Conference on Robot Learning, 2023.
  * [50] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [51] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470–5479, 2022.
  * [52] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.  Zip-nerf: Anti-aliased grid-based neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19697–19705, 2023.
  * [53] Wenbo Hu, Yuling Wang, Lin Ma, Bangbang Yang, Lin Gao, Xiao Liu, and Yuewen Ma.  Tri-miprf: Tri-mip representation for efficient anti-aliasing neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19774–19783, 2023.
  * [54] Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5501–5510, 2022.
  * [55] Cheng Sun, Min Sun, and Hwann-Tzong Chen.  Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5459–5469, 2022.
  * [56] Haian Jin, Isabella Liu, Peijia Xu, Xiaoshuai Zhang, Songfang Han, Sai Bi, Xiaowei Zhou, Zexiang Xu, and Hao Su.  Tensoir: Tensorial inverse rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165–174, 2023.
  * [57] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.  3d gaussian splatting for real-time radiance field rendering.  ACM Transactions on Graphics, 42(4):1–14, 2023.
  * [58] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855–5864, 2021.
  * [59] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  ACM transactions on graphics (TOG), 41(4):1–15, 2022.
  * [60] Konstantinos Rematas, Andrew Liu, Pratul P Srinivasan, Jonathan T Barron, Andrea Tagliasacchi, Thomas Funkhouser, and Vittorio Ferrari.  Urban radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12932–12942, 2022.
  * [61] Fan Lu, Yan Xu, Guang Chen, Hongsheng Li, Kwan-Yee Lin, and Changjun Jiang.  Urban radiance field representation with deformable neural mesh primitives.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 465–476, 2023.
  * [62] Fan Lu, Kwan-Yee Lin, Yan Xu, Hongsheng Li, Guang Chen, and Changjun Jiang.  Urban architect: Steerable 3d urban scene generation with layout prior.  arXiv preprint arXiv:2404.06780, 2024.
  * [63] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su.  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.  Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  * [64] Robert A Drebin, Loren Carpenter, and Pat Hanrahan.  Volume rendering.  ACM Siggraph Computer Graphics, 22(4):65–74, 1988.
  * [65] Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi.  OneFormer: One Transformer to Rule Universal Image Segmentation.  2023\.
  * [66] Zachary Teed and Jia Deng.  Raft: Recurrent all-pairs field transforms for optical flow.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419. Springer, 2020.
  * [67] Brady Zhou and Philipp Krähenbühl.  Cross-view transformers for real-time map-view semantic segmentation.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13750–13759, 2022.
  * [68] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang.  Dynamic view synthesis from dynamic monocular video.  In Proceedings of the IEEE International Conference on Computer Vision, 2021.
  * [69] A. Schmied, T. Fischer, M. Danelljan, M. Pollefeys, and F. Yu.  R3d3: Dense 3d reconstruction of dynamic scenes from multiple cameras.  In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3193–3203, Los Alamitos, CA, USA, oct 2023. IEEE Computer Society.
  * [70] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [71] Sanqing Qu, Tianpei Zou, Florian Röhrbein, Cewu Lu, Guang Chen, Dacheng Tao, and Changjun Jiang.  Upcycling models under domain and category shift.  In CVPR, 2023.
  * [72] Anuroop Gaddam, Tim Wilkin, and Maia Angelova.  Anomaly detection models for detecting sensor faults and outliers in the iot - a survey.  In 2019 13th International Conference on Sensing Technology (ICST), pages 1–6, 2019.
  * [73] Tianhang Wang, Kai Chen, Guang Chen, Bin Li, Zhijun Li, Zhengfa Liu, and Changjun Jiang.  Gsc: A graph and spatio-temporal continuity based framework for accident anticipation.  IEEE Transactions on Intelligent Vehicles, 9(1):2249–2261, 2024.

##  Appendix A OPV2V-N

To facilitate the research on camera-insensitivity for collaborative
perception: i) firstly, as we discussed in related works, multi-view based
collaborative perception heals the ill-posed of recovering noisy camera images
just from single-view. Owing there are no labels of multi-view based overlap
regions in existing collaborative perception, we manually collect the multi-
view based overlap regions for RCDN experiments, shown in Figure 7. In detail,
we will record the corresponding vehicle IDs, camera IDs and duration time
$t_{start},t_{end}$ of the multi-view based overlap regions; ii) secondly, as
we need to distinguish the foreground and background for static and dynamic
collaborative neural fields, respectively. We extend the OPV2V[16] with more
data format, such as the optical flow (supervise the $\mathbf{s}_{fw,bw}$),
mask labels, to bridge the gap between neural field and collaborative
perception, as shown in Figure 8.

Figure 7: Visualization of manually labeling mechanisms. Note that the red
circles represent the multi-view based overlap regions that are suitable for
the random noisy situation. We will record the corresponding vehicle IDs,
camera IDs and duration $t_{start},t_{end}$ of the overlap regions. Figure 8:
Visualization of extra data format.

Data analysis. We manually annotate about 65 scenes, which consists of a total
of 6138 collaborative samples. Figure 10 presents some statistical analysis
results regarding the OPV2V-N dataset. The OPV2V-N covers situations about
61.86%, 33.47%, and 4.66% for two, three, and four V2X collaborative agents,
respectively. Meanwhile, before we conduct the corresponding RCDN experiments,
we validate whether the random noisy camera data will affect the collaborative
perception system. Table 10 shows that i) the noise actually degrades the
system performance; ii) compared to static scenes, dynamic vehicles are more
susceptible to the influence of noisy data. With this prior knowledge, we
decided to explore the RCDN algorithms and need to pay more attention to
optimizing the design for dynamic vehicle perception. We also visualize the
specific degradation caused by the noisy camera data, shown in Figure 9. Note
that Figure 9 (a) degradation with missed vehicle inspections; Figure 9 (b)
degradation with missed Dr. area and lane inspections; Figure 9 (c)
degradation with both missed vehicle and Dr. area, lane inspections; Figure 9
(d) no degradation. Also, to make sure that the random noisy camera data is
always inputted the same way and that performance does not change because of
the different types of noise, like blurred or occluded, we replace the
manually annotated camera IDs under the camera failure situation[27].

Figure 9: Visualization of different performance degradation with random noisy
camera data.

Table 3: The validation experiments on whether random noise will affect
collaborative perception systems. Note that we utilize the current SOTA map-
segmentation method, CoBEVT.

OPV2V-N | w. random noisy | w.o. random noisy
---|---|---
Dr. Area | 49.37 | 52.64
Lanes | 34.80 | 37.96
Dynamic Veh. | 39.81 | 47.49

Figure 10: The distributions of V2X collaborative agents.

##  Appendix B Detailed Information about Experiments

###  B.1 Implementation Details.

For collaborative perception part, we assume all the AVs have a 70m
communication range following[43], and all the vehicles out of this
broadcasting radius of ego vehicle will not have any collaboration. We compare
with the state-of-the-art multi-agent perception algorithms: F-Cooper,
AttFuse, V2VNet, DiscoNet and CoBEVT w.o/w. the proposed RCDN.

Table 4: Inference time for each chunk.

Modules | Time Cost
---|---
$f_{static}$ | 4.47$\pm$0.11ms
$f_{dynamic}$ | 3.94$\pm$0.21ms
$f_{render}$ | 20.98$\pm$0.22ms

Meanwhile, to make a fair comparison, we first employ CVT to extract the BEV
feature from camera rigs for all methods. The transmitted BEV intermediate
representation has a resolution of 32$\times$32$\times$128; For collaborative
neural fields part, we pretrain the BEV decoder with the mcp encoder for
better performance, and the geometry collaborative volume feature has a
resolution of 128$\times$128$\times$128\. Same as [69], we select
$(t-1,t,t+1)$ as the mini training unit and train the whole model with the
Adam[70] optimizer and cosine annealing learning rate scheduler with initial
learning rate of 5e-4 on a single RTX 3090 24G GPU with AMD Ryzen Threadripper
3960X. As for the inference time, we record the corresponding time in Table 4.
Note that chunk is the smallest unit of parallel processing of the image,
e.g., if the image size is $(400,400)$, the chunk size is $4096$ pixels, the
number of each image’s parallel chunks is about 40.

###  B.2 Discussion on RCDN.

Figure 11: Visualization of domain gap between normal view and RCDN repaired
view.

Theoretically, the RCDN reconstructs the entire collaborative scenario field,
according to radial field theory[50], so whichever camera has the noise
problem can actually be recovered. In this regard, we experimentally validate
the RCDN using CoBEVT and V2VNet, and the corresponding results are in Table
5. We can see that i) if all the RCDN reconstructed cameras are used, the
performance is much better compared to using all the noisy camera data, e.g.
as for CoBEVT, about $62.38\%/123.29\%/262.70\%$ increment for the Dr. area,
lanes and dynamic vehicles, respectively. ii) compared to using all normal
cameras, using all reconstructed RCDN cameras will degrade the performance,
e.g., as for V2VNet, about $20.94\%/21.70\%/35.73\%$ decrement for the Dr.
area, lanes and dynamic vehicles, respectively. To address this phenomenon, we
visualize the perspective of the reconstructed camera views, shown in Figure
11, and it is not difficult to find that there is a domain gap[18, 71] between
the reconstructed and the normal cameras. Meanwhile, the backbone used to
extract the BEV is trained by using the normal camera, so if all reconstructed
cameras are used, it does cause a certain degree of degradation. Thanks to the
development of abnormal detection algorithms[72, 73], it is easy to find noisy
camera data. Hence, we only replace the corresponding noisy data without
replacing all data for better performance.

Table 5: Performance comparison (Dr. Area/Lanes/Dynamic Veh.)

methods/setting | all Nor. data | all RCDN data | all noisy data
---|---|---|---
CoBEVT | 51.96/34.19/56.61 | 36.78/20.90/44.54 | 22.65/9.36/12.28
V2VNet | 41.70/27.14/42.57 | 32.97/21.25/27.36 | 18.12/8.76/6.78

###  B.3 Benchmarks

We conduct extensive experiments on current collaborative perception
methodologies with the proposed RCDN. Table 6 presents the segmentation
performance under the expectation of random noisy camera numbers from 0 to 3
on OPV2V-N, which corresponds to the numerical results shown in Figure 3 in
the main text. We see that RCDN can be portable to other baseline methods and
stabilize the performance even under the extreme camera-insensitivity setting.
We also visualize some training scene samples, shown in Figure 12.

Table 6: Performance of RCDN with other baseline methods. Note that $-$
represents the failed results.

| Dr. Area/Lanes/Dynamic Veh.
---|---
| w.o. RCDN | w. RCDN
Method Performance Number | n=0 | n=1 | n=2 | n=3 | n=0 | n=1 | n=2 | n=3
| scene1 | 34.03/17.55/55.56 | 19.37/6.67/28.96 |  20.41/7.77/24.39 | 22.68/9.27/17.56 | 34.03/17.55/55.56 | 35.46/16.82/52.69 | 36.85/16.23/44.45 | 35.31/16.88/43.49
| scene2 | 48.14/24.45/55.32 | 44.55/16.81/20.89 | 35.79/17.10/24.98 | 35.83/16.34/22.39 | 48.14/24.45/55.32 | 48.50/24.75/52.66 | 47.54/22.84/52.86 | 46.90/19.58/54.19
| scene3 | 35.00/29.86/66.79 | 25.43/18.01/38.00 | 26.11/14.99/26.28 | 21.94/11.56/24.74 | 35.00/29.86/66.79 | 35.93/30.10/67.70 | 34.40/29.51/61.14 |  36.77/28.86/67.09
| scene4 | 65.36/63.79/77.24 | 30.95/17.74/31.34 | 27.02/18.48/16.31 | -/-/7.03 | 65.36/63.79/77.24 | 64.98/62.73/77.77 | 65.93/62.15/67.66 | -/-/70.18
| scene5 | 44.67/30.20/61.74 | 24.03/20.50/29.32 | 25.57/15.51/15.49 | 24.03/17.55/17.32 | 44.67/30.20/61.74 | 39.56/26.73/58.00 | 38.20/23.83/52.49 | 38.51/19.31/52.02
F-Cooper | Avg | 45.44/33.17/63.33 | 28.87/15.95/29.70 |  26.98/14.77/21.49 | 26.12/13.68/17.808 | 45.44/33.17/63.33 | 44.89/32.23/61.76 | 44.58/30.91/55.72 | 39.37/21.16/57.39
| scene1 | 32.29/20.38/43.32 | 19.07/14.04/26.07 | 17.58/10.94/14.55 | 17.28/12.77/11.68 | 32.29/20.38/43.32 | 29.32/17.16/42.11 | 28.25/15.05/37.25 | 27.05/16.39/35.11
| scene2 | 49.38/20.84/52.16 | 35.85/16.16/20.36 | 30.96/18.91/10.93 | 27.28/17.40/12.14 | 49.38/20.84/52.16 | 51.08/22.75/47.93 | 53.02/26.69/41.57 |  53.15/26.38/40.16
| scene3 | 38.79/32.05/57.73 | 32.64/26.01/36.76 | 30.67/24.36/22.91 | 25.48/17.04/15.75 | 38.79/32.05/57.73 | 38.19/29.78/56.18 | 36.75/28.60/46.56 | 38.86/31.12/42.80
| scene4 | 58.33/57.97/63.95 | 24.26/14.31/20.98 | 19.26/12.38/14.84 | -/-/9.18 | 58.33/57.97/63.95 | 58.65/55.44/63.11 | 54.56/48.79/60.59 | -/-/62.64
| scene5 | 49.18/37.55/53.56 | 28.15/23.32/19.65 |  23.22/18.34/18.09 | 23.15/18.47/19.30 | 49.18/37.55/53.56 | 44.65/32.35/51.41 | 41.89/25.55/47.17 | 39.74/21.54/46.21
AttFuse | Avg | 45.59/33.76/54.14 | 27.99/18.77/24.76 | 24.34/16.99/16.26 | 23.30/16.42/13.61 | 45.59/33.76/54.14 | 44.38/31.50/52.15 | 42.89/28.94/46.63 | 39.70/23.86/45.38
| scene1 | 45.31/23.68/43.26 | 15.73/6.68/7.47 | 11.85/7.38/3.71 | 13.15/9.24/5.35 | 45.31/23.68/43.26 | 34.20/19.25/38.88 | 30.34/13.73/38.84 |  23.42/7.85/22.83
| scene2 | 36.58/8.20/37.10 | 31.87/8.96/11.59 | 24.74/10.80/11.43 | 22.70/11.36/7.62 | 36.58/8.20/37.10 | 33.60/8.37/34.90 | 35.72/11.28/32.53 | 30.72/11.83/29.87
| scene3 | 28.61/17.05/35.23 | 19.51/10.48/2.50 | 14.87/25.98/3.56 | 14.42/8.73/- | 28.61/17.05/35.23 | 28.89/18.44/28.63 | 25.98/16.46/22.40 | 27.33/18.86/21.04
| scene4 | 50.28/37.95/62.15 | 32.42/20.69/11.13 |  24.30/12.44/7.84 | -/-/3.46 | 50.28/37.95/62.15 | 49.50/38.23/59.25 | 51.25/40.25/55.49 | -/-/54.21
| scene5 | 50.74/34.33/55.07 | 22.03/14.64/13.54 | 22.59/13.58/16.91 | 27.55/14.70/14.06 | 50.74/34.33/55.07 | 46.50/30.58/53.51 | 41.86/28.56/48.83 | 44.22/25.07/48.82
DiscoNet | Avg | 42.30/24.24/46.56 | 24.31/12.29/9.25 | 19.67/14.04/8.69 | 21.25/12.32/6.10 | 42.30/24.24/46.56 | 38.54/22.97/43.03 | 37.03/22.06/39.62 |  37.33/23.10/35.35
| scene1 | 39.83/19.49/28.06 | 20.52/7.15/8.21 | 11.41/7.68/5.68 | 9.54/7.33/3.94 | 39.83/19.49/28.06 | 39.28/19.95/28.72 | 36.13/15.55/28.25 | 30.56/15.21/19.67
| scene2 | 40.21/14.45/39.60 | 37.38/7.19/14.73 | 29.05/6.84/6.77 | 19.64/6.66/8.58 | 40.21/14.45/39.60 | 38.45/14.96/39.45 | 34.61/8.07/39.17 | 33.40/12.29/33.17
| scene3 | 33.24/26.83/31.89 | 20.11/6.11/2.93 |  12.39/7.28/8.37 | 9.48/7.01/1.49 | 33.24/26.83/31.89 | 29.53/20.95/35.15 | 24.75/16.82/26.92 | 27.94/16.59/20.52
| scene4 | 49.76/36.24/57.83 | 28.50/17.23/11.52 | 26.74/18.85/9.37 | 14.54/12.00/8.00 | 49.76/36.24/57.83 | 48.34/36.74/58.46 | 47.00/36.37/57.31 | 53.40/40.01/56.45
| scene5 | 45.47/38.71/55.45 | 33.46/14.91/18.99 | 29.66/17.04/16.38 | 24.23/14.99/14.10 | 45.47/38.71/55.45 | 42.98/33.59/52.06 | 39.04/28.94/52.93 |  39.21/28.99/47.79
V2VNet | Avg | 41.70/27.14/42.57 | 27.99/10.52/11.28 | 21.85/11.54/9.31 | 15.49/9.60/7.22 | 41.70/27.14/42.57 | 39.72/25.24/42.77 | 36.31/21.15/40.92 | 36.90/22.62/35.52
| scene1 | 30.59/12.43/47.62 | 24.55/10.07/30.67 | 23.23/8.19/18.83 | 22.99/11.63/14.59 | 30.59/12.43/47.62 | 27.37/10.63/46.65 | 27.22/11.31/41.38 | 26.96/10.59/36.99
| scene2 | 39.28/10.72/54.25 | 37.47/14.26/28.43 |  32.32/12.94/16.57 | 23.43/8.41/11.93 | 39.28/10.72/54.25 | 37.47/9.64/50.48 | 44.97/15.85/49.88 | 39.51/9.77/46.46
| scene3 | 47.80/37.85/60.40 | 24.91/11.90/33.73 | 18.34/11.15/18.29 | 15.78/11.56/63.49 | 47.80/37.85/60.40 | 49.45/37.51/60.02 | 50.96/40.09/57.53 | 49.85/37.13/63.49
| scene4 | 73.62/61.44/61.70 | 46.69/20.88/38.91 | 34.26/12.43/12.10 | -/-/8.67 | 73.62/61.44/61.70 | 70.27/59.07/62.01 | 68.48/56.13/53.58 |  -/-/56.71
| scene5 | 68.50/48.53/59.06 | 26.80/15.13/30.32 | 28.38/8.42/14.19 | 23.51/11.26/10.67 | 68.50/48.53/59.06 | 51.39/30.88/56.35 | 52.02/30.53/51.86 | 46.52/25.31/53.37
CoBEVT | Avg | 51.96/34.19/56.61 | 32.08/14.45/32.41 | 27.31/10.63/15.99 | 22.05/10.53/11.52 | 51.96/34.19/56.61 | 47.19/29.55/55.10 | 48.73/30.78/50.85 | 46.10/27.78/51.40

Figure 12: Visualization of some training scenes samples. Note that we cover
the classical scenes, including the four-way Intersection, T Intersection,
Midblock, Entrance Ramp and Curvy Segment.

###  B.4 PSNR Results

Table 7: The PSNR results.

| PSNR
---|---
F-Cooper | 26.11
AttFuse | 25.79
DiscoNet | 25.86
V2VNet | 26.14
CoBEVT | 25.89

We also record the corresponding PSNR results of different baseline methods w.
RCDN’s reconstruction’s image view, as shown in Table 7. Note that the term
peak signal-to-noise ratio (PSNR) is an expression for the ratio between the
maximum possible value (power) of a signal and the power of distorting noise
that affects the quality of its representation. Hence, the higher PSNR, the
better image quality.

###  B.5 Geometry BEV Volume Feature

Figure 13: Visualization of w.o/w. geometry BEV volume feature modeling.

We utilize the geometry BEV volume feature to speed up the training process
and improve the generality of the collaborative neural fields. We observe that
with $f_{geo\\_bev}$ the RCDN can obtain higher PSNR initial values and a
shorter training process, as shown in Figure 13.

###  B.6 Loss Functions

Similar to [68], we regularize the collaborative scene flow to be spatially
smooth by minimizing the difference between neighboring 3D points’ scene flow.
To regularize the consistency of the collaborative scene flow, we have the
scene flow cycle consistency regularization as follows:

$\displaystyle\mathcal{L}_{cyc}$
$\displaystyle=\sum\Arrowvert{\mathbf{s}_{fw}(\mathbf{r},t)+\mathbf{s}_{bw}(\mathbf{r}+\mathbf{s}_{fw}(\mathbf{r},t),t+1)}\Arrowvert_{2}^{2}$
(13a)
$\displaystyle+\sum\Arrowvert{\mathbf{s}_{bw}(\mathbf{r},t)+\mathbf{s}_{fw}(\mathbf{r}+\mathbf{s}_{bw}(\mathbf{r},t),t+1)}\Arrowvert_{2}^{2}$
(13b)

As for the weights of different losses in Eq 12, we set
$\lambda_{1},\lambda_{2},\lambda_{4}=1.0$ and $\lambda_{3}=0.1$ for training.
Meanwhile, we train the whole network for about 1000 epochs, which takes about
20 to 30 minutes.

##  Appendix C Visualization

We visualize some segmentation results of different baseline methods w.o/w
RCND under the different scenes, shown in Figure 14-18.

Figure 14: Visualization of baseline method of F-Cooper w.o/w. RCDN with one
random camera failure. Figure 15: Visualization of baseline method of AttFuse
w.o/w. RCDN with one random camera failure. Figure 16: Visualization of
baseline method of DiscoNet w.o/w. RCDN with one random camera failure. Figure
17: Visualization of baseline method of V2VNet w.o/w. RCDN with one random
camera failure. Figure 18: Visualization of baseline method of CoBEVT w.o/w.
RCDN with one random camera failure.


# On eliminating blocking interference of RFID unauthorized reader detection
system

1st Degang Sun  School of Cyber Security (of UCAS)
Beijing, China
<EMAIL_ADDRESS>2nd Yue Cui  Institute of Information Engineering (of
CAS)
School of Cyber Security (of UCAS)
Beijing, China
<EMAIL_ADDRESS>3rd Siye Wang  Institute of Information Engineering (of CAS)
School of Cyber Security (of UCAS)
Beijing, China
<EMAIL_ADDRESS>4th Yanfang Zhang  Institute of Information Engineering
(of CAS)
School of Cyber Security (of UCAS)
Beijing, China
zhangyanfang.ac.cn

###### Abstract

RFID as an important component technology of IoT faces important security
risks while being rapidly applied, among which the discovery of unauthorized
readers in space is crucial. There are some researches proposed the
unauthorized reader detection algorithm based on commercial off the
shell(COTS) devices, but these detection algorithms are often easily affected
by moving objects blocking interference in space, causing false alarms. We
propose a new method of eliminating moving object interference, which can
reduce the system false alarm rate to less than 7.9% by experimental testing.

###### Index Terms:

RFID security, Human-centered computing, Privacy protection

##  I Introduction

Due to the reader is the main way to obtain the UHF RFID air interface data,
and in the ISO 18000-6C protocol[1] for RFID system used by the COTS devices
lacks of security certification between the tags and the reader, so that any
unauthorized reader in the space can communicate with the tags and get access
to the tags data, resulting in data leakage, data tampering and other security
threats[2].

And for the security of commercial RFID system, there are many researchers’
related work, such as: Pramod present physical unclonable function based
unilateral authentication protocol for RFID system[3]. The algorithm has been
implemented on an 8-bit open-loop resonator based chipless RFID tag based
system and is validated using BASYS 2 FPGA board based platform [4]. Security
analysis shows that SLAP guarantees the functionalities of mutual
authentication as well as resistance to various attacks such as de-
synchronization attack, replay attack and traceability attack, etc [5]. But
taking the cost, scalability into consideration, using the commercial reader
itself as monitoring equipment, and does not impose too much encryption
calculation, through its signal when normal communication with the tags to
discern whether there is an unauthorized reader is the best way. Commercial
reader can obtain the data like, RSSI, Phase, time stamp, throughput rate.
Using these indicators such as [6] and [7]. When using these methods in the
office environment, we found that the surrounding pedestrian walking or
blocking will affect the signal, causing false alarm, so this paper introduces
how to eliminate the blocking interference of these moving objects.

##  II unauthorized reader detection system analysis

In this section, we will analyze the RFID unauthorized reader(UR) detection
system based on ASTI(adjacent signals time interval) which is a optimal
parameter comparing to throughput rate because lower time delay[6]. In
addition, according to the needs of actual use, we chose a common scenario to
arrange the experiment - placing the reader at the entrance(as 1).And when
human or other object passes by, how the performance of UR detection system
changes.

Figure 1: Experiment setup

###  II-A UR detection system

We setup the pre-experiment as 1,the normal reader is set at the entrance and
start up to communicate with the five tags equally spaced at the board in
front of the reader. Then the UR randomly enters and start. Figure2 shows one
of the results,the black line indicates the time UR starts. ASTI is calculated
by:

$ASTI=Timestamp(i)-Timestamp(i-1)$ (1)

Figure 2: ASTI & RSSI under unauthorized reader

It’s obvious that after UR starting, the variation of ASTI is significant but
the RSSI varies as small fluctuations. So the change rate of ASTI can be
adopted as a parameter for UR detection. We adopt the detection algorithm
in[7].

In the next part, we start the normal reader and UR, then a man walk through
the entrance to simulator the true environment for daily use. Figure 3 shows
one of the experiment results. The light blue line indicates when the man
start to block the tag and normal reader.

Figure 3: ASTI & RSSI under unauthorized reader and passing-by

From the result, we can find that ASTI also changes noticeable as human
passing so does the RSSI. But because the RSSI is not sensitive to whether UR
is at present, we can use RSSI as the parameter for human-centered computing.

Human passing-by and UR cause the same result for the ASTI UR detection so the
false-alarm rate(FPR) will be high in the real environment. It’s vital to find
the way to eliminate the interference of human.

###  II-B Model of moving human

First we model and analyze the action of a moving object passing a doorway
using pedestrian data as an example. Suppose someone with top view length and
width $l_{h}$ and $d_{h}$ respectively, passes the doorway from the middle
with speed $V$m/s, the tag is equally spaced $d_{set}$ placed, the effective
length of the signal reception of the tag is $l_{tagi}$, the red line part is
the angle of the human body completely blocking the sight distance path, the
corresponding impact distance of $d_{in}$ and impact time $T_{in}$ can be
found as follows:

$d_{in}=\frac{\tfrac{1}{2}(d_{door}+l_{h})}{tan(\frac{\pi}{2}-\theta_{tag1B})}-\frac{\tfrac{1}{2}(d_{door}-l_{h})}{tan(\frac{\pi}{2}-\theta_{tag1A})}+d_{h}$
(2)

$T_{in}=\frac{d_{in}}{V}$ (3)

Figure 4: Moving object passing doorway model

In RFID system, the signal received by the reader antenna can be divided into
line-of-sight (LOS) signal and multipath effect signal superposition caused by
other reflection paths.

$H(f)=\sum_{n=0}^{N-1}\rho_{n}e^{j\theta_{n}}e^{-j2\pi f\tau_{n}}$ (4)

Equation (4) represents the superposition of the arrival signals of N paths.
where: $\rho_{n}$ denotes the intensity of the signal; $\theta_{n}$ denotes
the phase bit; $\tau_{n}$ denotes the arrival delay. In general, the signal
energy of the line-of-sight path (the straight-line path between the sender
and the receiver, denoted as n=0 part) ($\rho_{n}$ in Eq.) is much larger than
the signal energy of the other paths, and thus dominates in the superposition.
dominant position. In an ideal environment, when multipath effects do not
exist, it can be considered that $T_{in}\doteq\Delta t_{tagi}$. In practice,
the tag acquires multivariate data due to environmental multipath and the
passage of objects generating new multipath effects

##  III Eliminating blocking interference method

From the previous experiments and modeling analysis, we figure out the key of
getting the time by passing-by is the velocity estimation of the object. So in
this section, we will introduce the method for velocity estimation and
furthermore eliminating the blocking interference.

###  III-A Data pre-processing

We use a Moving Average Filter to smooth the acquired data for further
processing. Due to its polling mechanism, the RFID reader can only communicate
with one tag at a time and the sampling frequency of commercial readers may
only reach 30Hz in the environment, however, since the object is always in
motion, we want to capture more readings at the same time. Therefore, we use
the Hermite Interpolating Polynomial (PCHIP) method to interpolate the data
samples at the desired point in time, which allows for more accurate
interpolation at a considerably higher efficiency.

###  III-B Velocity estimation based on Pearson correlation coefficient(PCCs)

At the beginning, a database of moving object speed should be building as the
reference for matching. To achieve this, we arranged a dummy placing at a
vehicle fixed on the slide rail, which would moving at the set direction with
a set velocity. We used the machine to simulate real human or other object
passing through the door like 1. The velocity is raised in steps of 0.02 from
0.1m/s to 2m/s and at each velocity tested for 200 times then taking the
average number as the result.

When the back end computer newly receives RSSI from the reader, Pearson
correlation coefficient[8] will be used for matching its velocity to the
database. Then select the velocity which PCCs is the closest to 1 as the
estimation of the velocity of the moving human.

The Pearson correlation coefficient between two variables is defined as the
product of the covariance of the two variables divided by their standard
deviations:

${\displaystyle\rho_{X,Y}={\mathrm{cov}(X,Y)\over\sigma_{X}\sigma_{Y}}={E[(X-\mu_{X})(Y-\mu_{Y})]\over\sigma_{X}\sigma_{Y}}}$
(5)

The closer the correlation coefficient of two variables is to 1, the more
significant the linear relationship between the two variables, i.e., the more
relevant. One important mathematical property of Pearson’s correlation
coefficient is that because a change in the position and scale of the two
variables does not cause a change in this coefficient, i.e. it the invariance
of the change (determined by the sign). That is, if we move $X$ to $a+bX$ and
move $Y$ to $c+dY$, where $a,b,c$ and $d$ are constants, it does not change
the correlation coefficient of the two variables.

###  III-C Eliminating blocking interference

After moving speed database building for the velocity estimation goal, when we
receive data from the reader, we have to decide whether there is someone
passing through causing alarm or real UR existing.

From the pre-experiment, it can be calculated that the change rate of RSSI
with UR is less than 5%. So it can be used as a threshold to judge object
passing-by.

The moment UR detection system send alarm, RSSI change rate is calculated and
if the rate is larger than 5%, velocity estimation part work to match the
object speed. Then $T_{in}$ the time of UR detection influenced add the set
window time will be thought as the new window time back to UR detection
system:$window=window+T_{in}$. If the system still alarms, We think UR is
shown up.

##  IV Experiment

In this section, we arranged some experiment to evaluate our method. The
experiment environment was a normal office, devices setup like Figure 1,
tested 500 times each group. We choose two Impinj R420 as readers and tags are
of H47.

###  IV-A Accuracy of UR detection system

We tested the accuracy of UR detection system in the three conditions:

  * •

without human passing-by:free

  * •

with human passing-by:interference

  * •

with human passing-by and eliminating method:eliminated

The result shows that our method can obviously reduce the interference of
human passing-by.

Figure 5: Accuracy of UR detection system

###  IV-B Evaluation of FPR

In this experiment, we tested for evaluation of FPR which is the most concern
of our method. It’s can be seen from the Figure6 , our method significantly
lower the FPR comparing to the unused one.

Figure 6: Evaluation of FPR in three conditions

###  IV-C Accuracy of velocity estimation

In this experiment, we arranged an experiment for testing velocity estimation
part. High accuracy has occupied in all three groups of experiment.

Figure 7: Accuracy of velocity estimation

##  V conclusion

This paper proposes a new method of eliminating the human or other object
moving in the area causing false alarm for the unauthorized reader detection
system. At first, analyze the UR detection system and modeling moving human,
then finding key of velocity estimation. Next, by building speed database and
using PCCs, put forward the method to eliminate the interference. At last,
test our method through experiment and the results show the excellent
performance.

## Acknowledgment

This subject comes from

## References

## References

  * [1] Iso [EB/OL]. https://www.iso.org/home.html
  * [2] Lawson N. Side-channel attacks on cryptographic software [J]. IEEE Security & Privacy, 2009, 7(6): 65-68.
  * [3] Pramod Kumar Maurya; Satya Bagchi; A Secure PUF-Based Unilateral Authentication Scheme for RFID System, WIRELESS PERSONAL COMMUNICATIONS, 2018.
  * [4] Vijay Sharma; Amogh Vithalkar; Mohammad S. Hashmi; Lightweight Security Protocol for Chipless RFID in Internet of Things (IoT) Applications, 2018 10TH INTERNATIONAL CONFERENCE ON COMMUNICATION SYSTEMS & NETWORKS (COMSNETS), 2018.
  * [5] Hanguang Luo; Guangjun Wen; Jian Su; Zhong Huang; SLAP: Succinct and Lightweight Authentication Protocol for Low-cost RFID System, WIRELESS NETWORKS, 2018.
  * [6] Huang Weiqing, Ding Chang, Cui Yue, et al. RFID Air Port intrusion detection technology based on malicious reader discovery [J]. Acta Sinica Sinica, 2018.7
  * [7] Sun D, Cui Y, Feng Y, et al. URTracker: Unauthorized Reader Detection and Localization Using COTS RFID[C]//International Conference on Wireless Algorithms, Systems, and Applications. Springer, Cham, 2021: 339-350.
  * [8] Zeinali M, Shahmorad S, Mirnia K. Hermite and piecewise cubic hermite interpolation of fuzzy data [J]. Journal of Intelligent & Fuzzy Systems, 2014, 26(6): 2889-2898.