# XLD: A Cross-Lane Dataset for Benchmarking Novel Driving View Synthesis

Hao Li1,2, Ming Yuan2, Yan Zhang2, Chenming Wu2, Chen Zhao2,
Chunyu Song2, Haocheng Feng2, Errui Ding2, Dingwen Zhang1, Jingdong Wang2
1Northwestern Polytechnical University , 2 Baidu VIS

###### Abstract

Thoroughly testing autonomy systems is crucial in the pursuit of safe
autonomous driving vehicles. It necessitates creating safety-critical
scenarios that go beyond what can be safely collected from real-world data, as
many of these scenarios occur infrequently on public roads. However, the
evaluation of most existing NVS methods relies on sporadic sampling of image
frames from the training data, comparing the rendered images with ground truth
images using metrics. Unfortunately, this evaluation protocol falls short of
meeting the actual requirements in closed-loop simulations. Specifically, the
true application demands the capability to render novel views that extend
beyond the original trajectory (such as cross-lane views), which are
challenging to capture in the real world. To address this, this paper presents
a novel driving view synthesis dataset and benchmark specifically designed for
autonomous driving simulations. This dataset is unique as it includes testing
images captured by deviating from the training trajectory by $1-4$ meters. It
comprises six sequences encompassing various time and weather conditions. Each
sequence contains $450$ training images, $120$ testing images, and their
corresponding camera poses and intrinsic parameters. Leveraging this novel
dataset, we establish the first realistic benchmark for evaluating existing
NVS approaches under front-only and multi-camera settings. The experimental
findings underscore the significant gap that exists in current approaches,
revealing their inadequate ability to fulfill the demanding prerequisites of
cross-lane or closed-loop simulation. Our dataset is released publicly at the
project page: https://3d-aigc.github.io/XLD/.

##  1 Introduction

Autonomous driving (AD) simulation, which bridges the gap between the real
world and the virtual world, is essential for the testing and development of
on-vehicle autonomous driving software [7]. Research indicates that employing
effective simulation methods can significantly expedite the evaluation of
safety tests for autonomous driving, achieving a speedup of approximately
$10^{3}$ to $10^{5}$ times faster compared to real-world testing [20]. This
compelling evidence underscores the importance of leveraging simulation to
enhance the efficiency and effectiveness of autonomous driving development.
However, the self-driving industry primarily conducts system testing using two
approaches: log replay, which involves testing on pre-recorded real-world
sensor data, and real-world driving, where new miles are driven to gather
additional data for testing purposes [58].

In closed-loop simulation, it is essential for the vehicle to have the freedom
to respond to control commands within the simulation environment, rather than
being confined to strictly following the original trajectory from logs. To
corporate with the recently developed end-to-end autonomous driving [61, 23],
designing a neural simulator for AD simulation [58, 54, 52], which can render
photo-realistic images on novel views for closed-loop simulation and algorithm
training, is in high demands. The main scientific problems boil down to the 3D
reconstruction [33] and novel view synthesis (NVS) [56, 54], which are also
long-standing problems in computer vision and computer graphics. Traditional
methods such as [40, 41] have dominated the major deployment of 3D scene
reconstruction for a long time. However, those reconstructed scenes cannot be
directly used to produce photo-realistic novel views, thus imposing large
restrictions on sensor simulations. As a result, the industrial bridges the
sim-to-real gap by parametric and procedural modeling technique [47] or human-
involved creations. With the recent rapid development of 3D implicit field,
such as neural radiance field (NeRF [35]) and explicit primitive
representations, i.e., 3D Gaussian Splatting (3DGS [25]), reconstructing a
scene from a collection of images serves as the foundation of end-to-end
autonomous driving simulation [22]. These techniques enable rendering high-
quality and photorealistic images on novel views.

Presently, the majority of approaches evaluate the performance of novel view
synthesis (NVS) results by splitting the dataset into training and testing
sets. However, this strategy of splitting and sampling leads to an
interpolation benchmark, which we argue is insufficient for evaluating whether
the trained models can effectively render simulation-ready (i.e., cross-lane)
and high-fidelity data for closed-loop simulations. On the contrary, our
proposed DBN dataset is a brand-new benchmark that evaluates the synthesis
quality on a cross-lane view with additionally captured GT images. Our dataset
and benchmark focus on assessing the NVS capability specifically for cameras
in cross-lane scenes. The primary objective is to evaluate the performance of
cameras in generating accurate and realistic novel views in scenarios
involving multiple lanes. Specifically, we introduce the XLD dataset, which
encompasses the generation of $150\times 3$ rendered images for each scene.
Additionally, we evaluate cross-lane novel view synthesis by rendering $50$
images with deviations of $1$m, $2$m, and $4$m from the training trajectory.
Using the XLD dataset, we conduct a benchmark of leading methods, which are
based on either NeRF or 3DGS, using well-established NVS metrics. The results
of our benchmark demonstrate that our proposed dataset offers a comprehensive
evaluation benchmark tailored specifically to the requirements of closed-loop
simulation. Furthermore, our benchmarking results reveal intriguing findings,
emphasizing the value of the proposed dataset.

##  2 Related Work

###  2.1 Autonomous Driving Simulation

In the past few years, there has been a surge in the use of autonomous driving
simulations [29]. These simulators are instrumental in validating planning and
control mechanisms, producing educational and evaluative datasets, and
significantly cutting down the time needed to perform these functions. The
current landscape is dominated by two predominant categories of simulation
tools: model-based and data-driven. Model-based simulation platforms, such as
PyBullet [18] MuJoCo [46], AirSim [42] and CARLA [19], utilize advanced
computer graphics to replicate vehicles and their surroundings. However, the
manual effort required to construct these models and to program the dynamics
of the vehicles can be quite demanding and lengthy. Moreover, the visual
output may sometimes fall short of the necessary realism, which can adversely
affect the efficacy of perception systems when they are put into operation.

Previously, NVS heavily relied on conventional image processing techniques.
For instance, Chaurasia et al.[12] propose the use of depth synthesis from
over-segmented graph structures, while AADS[28] employs filtered and completed
dense depth maps for warping novel view images through image stitching. A
data-driven simulation platform VISTA [3, 2] leverages datasets from the real
world to create comprehensively labeled and hyper-realistic simulations. In
more recent times, there has been a wave of innovations that employ the NeRF
method to superficially simulate driving perspectives. These new approaches
excel in the creation of hyper-realistic images and have been shown to surpass
traditional view synthesis algorithms in the realm of autonomous driving
simulation. Recent advances in neural novel view synthesis significantly
accelerate the rapid development of the next-gen driving simulation, which
exhibits superior expressiveness and flexibility compared to traditional
methods. Our dataset and benchmark work is specifically designed for those
methods.

###  2.2 NeRF-based NVS for Driving Simulation

The introduction of neural radiance field (NeRF) revolutionized NVS by
incorporating coordinate-based representation within multilayer perceptron
(MLP) architectures, leading to significant performance improvements. Building
upon NeRF, numerous subsequent works have further adapted these algorithms to
fulfill requirements such as efficient training, anti-aliasing rendering,
large-scale reconstruction, etc.. InstantNGP [36] proposes using a multi-
resolution hash grid with a shallow MLP network to eliminate large MLP
networks. mip-NeRF [4, 5] uses anti-aliased conical frustums instead of rays
to reduce objectionable aliasing artifacts, which enables NeRF to represent
fine details. zip-NeRF [6] borrows the ideas from rendering and signal
processing that combine mip-NeRF with InstantNGP. Nerfacto [45] integrates
many advantages of existing methods to provide an all-in-one solution for NeRF
training. Block-NeRF [44] tackles the reconstruction of large-scale urban
scenes by division. To handle dynamics, NSG [38] decomposes dynamic scenes
into scene graphs and learns a structured representation. SUDS [48] factorizes
a large scene into three hash table data structures, encoding static, dynamic,
and far-field radiance fields. MARS [54] is an instance-aware and modular
simulator based on NeRF, which models dynamic foreground instances and static
background environments separately. UniSim [58] transforms a recorded log into
a realistic closed-loop multi-sensor simulation, which incorporates dynamic
object priors and utilizes a convolutional network to handle and complete
unseen regions. EmerNeRF [57] employs a self-bootstrapping approach to
simultaneously capture scene geometry, appearance, motion, and semantics,
which enables comprehensive and synchronized modeling of these elements by
stratifying scenes into static and dynamic fields. UC-NeRF [16] addresses the
challenge of under-calibrated multi-view novel view synthesis through layer-
based color correction and virtual warping techniques.

###  2.3 3DGS-based NVS for Driving Simulation

Motivated by the NeRF-based methods and point-based differentiable rendering
[59, 1, 39, 30], 3D Gaussian Splatting (3DGS) [25] opens a new era with the
leading advantages in explicit representation and real-time rendering
capability. Within a concise timeframe, numerous methods [14, 56] have emerged
that focus on road scene reconstruction and NVS by leveraging the 3DGS
representation. For instance, PVG [14, 27] introduces periodic vibration-based
temporal dynamics to reconstruct dynamic urban scenes. StreetGaussian [56]
models the dynamic urban street environment as a collection of point clouds
with semantic logits and 3D Gaussians, each associated with either a
foreground vehicle or the background. DrivingGaussian [62] uses incremental
static 3D Gaussians to represent the scene’s static background. It also
employs a composite dynamic Gaussian graph to handle multiple moving objects
with LiDAR data. [31] introduce a hybrid method that combines radiance fields
with 3DGS representation, effectively eliminating the requirement for point
initialization in the context of urban scene NVS. [53] propose a hybrid
Gaussian representation specifically designed for performing online dense
mapping in unbounded large-scale scenes. GaussianPro [15] utilizes priors from
reconstructed scene geometries and patch-matching techniques to generate
precise Gaussians, leveraging the scene’s existing structure. DC-Gaussian [50]
introduces adaptive image decomposition for modeling reflections and
occlusions. It incorporates illumination-aware obstruction modeling to handle
reflections and occlusions under varying lighting conditions in urban scene
novel view synthesis. Our dataset and benchmark specifically focus on
evaluating the performance of neural-based driving simulation in NVS,
particularly in cross-lane scenarios. A portion of the mentioned methods serve
as our baselines, taking into account the code availability. Moreover,
generalizable 3D-GS methods such as PixelSplat [11], Mvsplat [13], and GGRt
[26] also attempts to synthesis novel images within a well-pretrained
generalizable feed-forward Gaussian networks.

Figure 1: Our datasets encompass six distinct scenes, where each scene
involves the vehicle following an on-road trajectory. To generate training
data for the cameras and LiDAR sensor, we sample $150$ waypoints along each
trajectory. The trajectory is visually emphasized using the color red.

###  2.4 Datasets in Autonomous Driving

In autonomous driving training and benchmarking, there exist a number of
datasets available. For example, KITTI [21]. KITTI-360 [32], vKITTI [8],
CityScapes [17], Mapillary [37], ApolloScape [24], Waymo Open Dataset [43].
nuScenes [9], Argoverse [10] and Argoverse 2 [51]. BDD100K [60]. OpenLane-V2
[49], etc.. Those previous works have laid the groundwork for research and
development in autonomous driving algorithms. A comprehensive survey of
datasets related to autonomous driving refers to [34]. However, as far as our
knowledge goes, none of the existing research has specifically addressed the
evaluation of novel view synthesis techniques tailored for autonomous driving
simulation, particularly in terms of their ability to meet the high demands of
cross-lane capability.

##  3 Dataset

Unlike previous datasets (both real-world and synthetic) that only capture
single trajectories on the road, our dataset also captures several
trajectories parallel to the main trajectories to better evaluate the novel-
view render quality. To generate cross-lane data in the created worlds, we
utilize Carla [19], a autonomous driving simulator platform built on Unreal
Engine.

###  3.1 Sensor Setup

To meet the needs of most NeRF and 3D-GS algorithms, the sensors used to
capture data include three color cameras (i.e. ‘left-front’, ‘front’, ‘right-
front’) and one 3D laser scanner. The spatial relationships between sensors
and the vehicle are fixed as shown in Fig. 2.

All three RGB cameras share identical intrinsic, lens parameters.
Specifically, they have a resolution of $1920\times 1280$ pixels, a field of
view (FOV) of $49.5^{\circ}\times 36.7^{\circ}$, a sensitivity of ISO 100, and
a shutter speed of 5ms. The 3D laser scanner features 64 laser beams, a
scanning range of 60 meters, and laser beam angles ranging from $-30^{\circ}$
to $30^{\circ}$ in vertical direction.

###  3.2 Data Generation

Our simulation environment consists of six scenes ("Town01," "Town02,"
"Town03," "Town04," "Town05," and "Town10", all of them are under CC-BY
License provided by [19]) with various weather conditions, such as sunshine
and rain, closely resembling real-world settings. An overview of all the
scenes and the trajectories is shown in Fig. 1. Take one scene for example,
the training set samples 150 times to capture three cameras’ images, LiDAR
points, and the vehicle’s extrinsic information. The vehicle forward distance
between two samples is 2 about meters. The test set includes four groups
captured by the ’front’ camera. These images are aligned parallel to the
training set, with each group exhibiting a progressive deviation of 0m, 1m,
2m, and 4m along the y-axis in vehicle coordinate, as shown in Fig. 2.
Additionally, a few novel-view-synthesis for AD methods like GaussianPro [15],
MARS [54], and UC-NeRF [16] need annotated sky masks to split the scene into
foreground-sky and model the color compensation separately. We employ a pre-
trained SegFormer [55] to effectively infer semantic segmentation masks and
extract sky masks from them.

Figure 2: The composition of our training set and testing set. The training
set consists of three RGB cameras (’front’, ’left-front’, ’right-front’)
mounted on our vehicle lane. For the test set, we sample image sequences along
trajectories that run parallel to the vehicle’s route. This encompasses four
distinct test trajectories, each offset from the vehicle’s trajectory by 0
meters, 1 meter, 2 meters, and 4 meters, respectively. The sampling interval
for the test set is five times the sampling interval used for the training
set.

##  4 Benchmark

###  4.1 Benchmarking Environment

In order to thoroughly assess the performance and computational efficiency of
the evaluating methods, we carried out a series of experiments utilizing an
NVIDIA Tesla V100 16GB GPU. Our method was extensively validated across five
distinct datasets featuring driving scenarios. We have presented our findings
through both qualitative and quantitative assessments, which underscore the
excellent performance and efficiency when juxtaposed with alternative
methodologies.

###  4.2 Benchmarking Methods

InstantNGP [36]: We employ the Adam optimizer and maintain similar parameter
settings as the original Instant-NGP implementation: the learning rate is
$1\times 10^{-4}$, the number of feature dimensions per entry is $F=8$, the
number of levels is $L=10$, and the hash tables is $2^{4}$. We train the model
with 30,000 steps.

Nerfacto [45]: We use the implementation in [45] without pose refinement to
test our benchmark. We employ the Adam optimizer with $1\times 10^{-3}$
learning rate. We train the model with 30,000 steps.

MARS [54]: We inherit most of the parameter settings as the original MARS
implementation. We employ the RAdam optimizer with $1\times 10^{-3}$ learning
rate. Due to the fact that our scenes are static without moving objects, we
disable $\mathcal{L}_{sem}$, the rest of loss functions remain the same. We
train the model with 50,000 steps.

UC-NeRF [16]: We inherit most of the parameter settings as the original UC-
NeRF implementation. We employ the AdamW optimizer with $2.5\times 10^{-3}$
learning rate. The weight of sky loss is set to $2\times 10^{-3}$. We train
the model with 40,000 steps.

3DGS [25]: We use the implementation of NerfStudio to evaluate our dataset. We
employ the AdamW optimizer with $1\times 10^{-3}$ learning rate. For
stability, we “warm-up” the computation in lower resolution. Specifically, we
start the optimization using 4 times smaller image resolution and we upsample
twice after 500 and 1000 iterations. We train the model with 30,000 steps.

PVG [14]: We employ the Adam optimizer and maintain a similar learning rate
for most parameters as the original PVG implementation while we adjust the
learning rate of the amplitude $\mathbf{A}$, opacity decaying $\mathbf{\beta}$
and opacity $\mathbf{O}$ to $3\times 10^{-5}$, $0.02$ and $0.005$
respectively. We train the model with 30,000 steps.

GaussianPro [15]: In alignment with the approach described in GaussianPro, our
models are trained for 30,000 iterations across all scenes following
GaussianPro’s training schedule and hyperparameters. The interval step of the
progressive propagation strategy is set to 50 where propagation is performed 3
times. The threshold $\sigma$ of the absolute relative difference is set to
0.8. For the planar loss, we set $\beta=0.001$ and $\gamma=0.001$.

DC-Gaussian [50]: To align the performance described in DC-Gaussian, we set
loss coefficient 0.001 for both photometric and sky losses. Sky loss is the
same as UC-NeRF.

###  4.3 Used Metrics

We adopt the evaluation criteria employed by the aforementioned methods, which
comprise Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and
Learned Perceptual Image Patch Similarity (LPIPS) as our evaluation metrics.
Furthermore, we ensure transparency and clarity by providing a detailed
description of the experimental framework employed for comparing the methods.

Table 1: Results on our proposed dataset with the different offsets using _front-only_ camera. Method |  w/o Offset | Offset-1m | Offset-2m | Offset-4m
---|---|---|---|---
|  $\uparrow$PSNR |  $\uparrow$SSIM |  $\downarrow$LPIPS |  $\uparrow$PSNR |  $\uparrow$SSIM |  $\downarrow$LPIPS |  $\uparrow$PSNR |  $\uparrow$SSIM |  $\downarrow$LPIPS |  $\uparrow$PSNR |  $\uparrow$SSIM |  $\downarrow$LPIPS
\- NeRF-based |  |  |  |  |  |  |  |  |
Instant-NGP [36] | 29.76 | 0.894 | 0.253 | 23.44 | 0.814 | 0.346 | 22.37 | 0.790 | 0.386 | 20.95 | 0.768 | 0.443
UC-NeRF [16] | 35.95 | 0.936 | 0.311 | 30.07 | 0.896 | 0.355 | 25.17 | 0.863 | 0.367 | 22.89 | 0.797 | 0.420
MARS [54] | 30.21 | 0.873 | 0.146 | 27.40 | 0.851 | 0.169 | 24.95 | 0.847 | 0.194 | 23.29 | 0.818 | 0.235
NeRFacto [45] | 27.39 | 0.888 | 0.252 | 23.49 | 0.824 | 0.314 | 21.64 | 0.786 | 0.379 | 20.82 | 0.769 | 0.412
EmerNeRF [57] | 31.76 | 0.907 | 0.126 | 28.66 | 0.878 | 0.150 | 26.05 | 0.852 | 0.182 | 24.80 | 0.837 | 0.203
\- Gaussian-based |  |  |  |  |  |  |  |  |
3DGS [25] | 30.87 | 0.916 | 0.274 | 23.26 | 0.873 | 0.334 | 22.01 | 0.829 | 0.396 | 19.17 | 0.768 | 0.460
PVG [14] | 37.78 | 0.960 | 0.189 | 26.84 | 0.882 | 0.296 | 24.42 | 0.854 | 0.335 | 23.17 | 0.841 | 0.353
GaussianPro [15] | 31.62 | 0.919 | 0.263 | 22.61 | 0.856 | 0.338 | 21.26 | 0.819 | 0.383 | 18.75 | 0.772 | 0.445
DC-Gaussian [50] | 31.29 | 0.919 | 0.264 | 26.82 | 0.884 | 0.298 | 25.24 | 0.871 | 0.319 | 22.90 | 0.844 | 0.360

###  4.4 Experimental Results

We evaluate multiple novel-view-synthesis methods using our datasets. These
methods are trained in two modes: front-only mode, utilizing images captured
solely by the front camera, and multi-camera mode, incorporating images from
all three cameras. We conduct separate evaluations on trajectories with 0m,
1m, 2m, and 4m offsets. Our evaluation encompasses both qualitative and
quantitative experiments, ensuring a comprehensive analysis of the methods’
performance. Furthermore, it is worth noting that EmerNeRF [57] demonstrates
the highest performance in the cross-lane dataset, with an average PSNR of
26.50 dB. We attribute this superior performance to its inherent capability of
self-supervised scene decomposition and positional embedding decomposition. We
place detailed experimental results in Tab. 1 and Tab. 2. More detailed
results please refer to the supplementary material and our webpage.

Table 2: Results on our proposed dataset with the different offsets using _left-front, front, right-front_ cameras. Method |  w/o Offset | Offset-1m | Offset-2m | Offset-4m
---|---|---|---|---
|  $\uparrow$PSNR |  $\uparrow$SSIM |  $\downarrow$LPIPS |  $\uparrow$PSNR |  $\uparrow$SSIM |  $\downarrow$LPIPS |  $\uparrow$PSNR |  $\uparrow$SSIM |  $\downarrow$LPIPS |  $\uparrow$PSNR |  $\uparrow$SSIM |  $\downarrow$LPIPS
\- NeRF-based |  |  |  |  |  |  |  |  |
Instant-NGP [36] | 29.24 | 0.888 | 0.262 | 23.52 | 0.847 | 0.344 | 22.39 | 0.815 | 0.382 | 21.05 | 0.783 | 0.428
UC-NeRF [16] | 33.12 | 0.912 | 0.360 | 30.07 | 0.896 | 0.355 | 28.81 | 0.881 | 0.373 | 26.87 | 0.870 | 0.421
MARS [54] | 31.37 | 0.887 | 0.151 | 29.28 | 0.874 | 0.157 | 28.54 | 0.869 | 0.165 | 26.49 | 0.847 | 0.193
NeRFacto [45] | 29.39 | 0.890 | 0.246 | 22.91 | 0.850 | 0.307 | 22.26 | 0.809 | 0.348 | 21.03 | 0.779 | 0.393
EmerNeRF [57] | 31.51 | 0.894 | 0.146 | 29.41 | 0.878 | 0.152 | 28.66 | 0.873 | 0.160 | 26.62 | 0.851 | 0.188
\- Gaussian-based |  |  |  |  |  |  |  |  |
3DGS [25] | 29.74 | 0.914 | 0.312 | 22.08 | 0.842 | 0.359 | 21.34 | 0.824 | 0.402 | 19.47 | 0.796 | 0.441
PVG [14] | 33.33 | 0.933 | 0.256 | 26.62 | 0.878 | 0.318 | 25.50 | 0.870 | 0.332 | 23.35 | 0.849 | 0.360
GaussianPro [15] | 28.13 | 0.889 | 0.321 | 21.90 | 0.839 | 0.379 | 20.85 | 0.822 | 0.404 | 19.36 | 0.795 | 0.443
DC-Gaussian [50] | 30.23 | 0.912 | 0.271 | 26.74 | 0.883 | 0.315 | 25.53 | 0.8715 | 0.329 | 23.53 | 0.860 | 0.357

###  4.5 Findings

####  4.5.1 NeRFs show better performance than 3D-GS averagely

Lately, there has been a shift in research interest within the community,
transitioning from NeRF towards 3D-GS. Notably, 3D-GS has achieved state-of-
the-art results on datasets such as KITTI and Waymo. However, it is worth
noting that these datasets primarily focus on evaluating interpolation images
without considering significant offsets (i.e. cross-lane NVS). Consequently, a
crucial aspect of our dataset is to assess and compare the novel-view
synthesis performance between NeRF and 3D-GS methods under challenging
conditions involving substantial offsets.

Figure 3: Visualization of the rendered images using DC-Gaussian and EmerNeRF
with different offsets (i.e. 0m, 2m, and 4m) in two scenes. The discriminated
areas are highlighted, and the areas with better results are marked as
$\square$, while wrose results are marked as $\square$.

As shown in Tab. 2, in the scenario with no offset, compared with EmerNeRF
[57], PVG [14] achieves better performance by 0.2dB in PSNR metrics with
3-cameras setting and 1.83dB in PSNR metrics with 1-camera setting. In the
scenario with 4m offset, EmerNeRF [57] and UC-NeRF [16] show a much more
powerful ability to synthesize images in cross-lane novel view. compared with
the SOTA Gaussian method (i.e. DC-Gaussian [50]), they achieve leading
performance by 1.90dB and 3.43dB in PSNR separately. The visualization results
are shown in Fig. 3.

####  4.5.2 3D-GS methods with self-decomposition can handle background better

In this comparison, we examine the self-decomposition method (PVG [14])
alongside traditional approaches (3D-GS [25] and GaussianPro [15]), the
quantitative results are shown in Table 3. The traditional method initializes
3D-GS primitives, denoted as
$\mathcal{G}=\left\\{\left(\sigma_{i},\boldsymbol{\mu}_{i},\Sigma_{i},c_{i}\right)\right\\}_{i=1}^{G}$,
using LiDAR point clouds and extends them to areas lacking geometric features
through cloning and splitting operations. However, these techniques lack a
geometry prior and heavily rely on photometric loss, resulting in an issue of
over-fitting. This problem becomes evident in our benchmark, particularly in
the context of cross-lane novel view synthesis, as illustrated in Figure 4.

Table 3: Results on three 3D-GS methods with the different offsets with multi-camera setting. Offset | 3D-GS | GaussianPro | PVG
---|---|---|---
PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
0m | 27.23 | 0.892 | 0.278 | 26.97 | 0.890 | 0.283 | 34.08 | 0.948 | 0.219
1m | 22.33 | 0.846 | 0.331 | 21.89 | 0.840 | 0.335 | 28.11 | 0.907 | 0.270
2m | 21.15 | 0.825 | 0.355 | 20.76 | 0.819 | 0.362 | 26.63 | 0.894 | 0.289
4m | 19.24 | 0.797 | 0.393 | 19.01 | 0.792 | 0.399 | 23.96 | 0.873 | 0.317

When not initialized by point clouds, the Gaussian points representing the
background tend to overfit the training data in order to minimize rendering
loss, often at the expense of positional accuracy. While they may appear
satisfactory within the training trajectory, their performance degrades
significantly when rendering images with offsets from the training trajectory.
In Figure 4, it can be observed that the street lamps are accurately modeled
in the rendered images without any offset. However, when the offset is
increased to 2m and 4m, these lamps become obscured by the background Gaussian
points, resulting in a substantial decline in performance. Specifically, there
is a decrease in performance of -7.99dB and -7.69dB for 3D-GS [25] and
GaussianPro [15], respectively, as we move from 0 meters to 4 meters offset.

Figure 4: Visualization of the rendered images and depth maps using 3D-GS
[25], GaussianPro [15], and PVG [14] with different offsets (i.e. 0m, 2m, and
4m) in two scenes. The discriminated areas are highlighted, and the areas with
better results are marked as ’$\square$’.

In contrast, the self-decomposition method, such as PVG [14], incorporates
frequency information into 3D-GS primitives to create a unified representation
of dynamic objects and the background. These primitives are denoted as
$\mathcal{G}^{k}_{t}=\\{\tilde{\mu}^{k}_{t},\Sigma^{k},\widetilde{\alpha}^{k}_{t},S^{k}\\}_{k=1}^{G}$.
Additionally, PVG employs a high-resolution environment cube map, denoted as
$f_{sky}(d)=c_{sky}$, to effectively handle high-frequency details in the sky,
where $d$ represents the ray direction. These techniques enable the network to
accurately model the sky and avoid local minima. As depicted in Figure 4, PVG
produces depth maps that better represent the background, leading to more
realistic images compared to other Gaussian methods. It achieves an impressive
improvement of 4.72dB in PSNR compared to the baseline 3D-GS [25], resulting
in higher fidelity rendered images.

####  4.5.3 Multi-camera benefits the cross-lane novel view synthesis

Using EmerNeRF [57] as an example, we compare the performance differences
between mulitple-camera and front-camera settings shown in Fig. 5.
Particularly, NeRF with only front camera training performs better than the
network using three cameras by 0.25db in PSNR metrics when rendering images
without offsets. However, when it comes to an offset, we observe a trend that
NeRF using multiple-camera training performs better than using only front-
camera training, up to 1.82db PSNR improvement in the scenario of 4m offset.
These results indicate that using more cameras can significantly boost the
performance of corss-lane novel view synthesis ability.

Figure 5: In the left column: we visualize the novel-view-synthesis results
with different offsets and camera numbers, the discriminate areas are
highlighted. In the right column, we show the performance improvement between
3-cameras and 1-camera settings using PSNR, SSIM, and LPIPS metrics.

####  4.5.4 The quality of geometry reconstruction is key to cross-lane NVS

Both NeRF and 3D-GS methods have demonstrated excellent performance on
established autonomous driving datasets such as KITTI and Waymo. In these
datasets, the primary function of the NVS methods is often to replay or
interpolate existing views within the training trajectory, rather than
generating images in truly novel views. However, our cross-lane novel view
synthesis benchmark offers a distinct evaluation scenario that differs
significantly from previous efforts validated on Waymo or KITTI datasets. This
benchmark introduces novel challenges and evaluation criteria specifically
designed to assess the performance of NVS methods when synthesizing images
from viewpoints across different lanes. As a result, the evaluation of NeRF
and 3D-GS methods in this benchmark provides unique insights beyond their
performance on traditional datasets.

As shown in the right column of Fig. 6, all the selected methods perform well
in the scenarios without any offset, which is identical to the previous
validation efforts, and PVG even achieves leading performance up to 37.78dB in
PSNR. When increasing the offset to 4 meters, its performance drops
significantly while UC-NeRF [16] and EmerNeRF [57] surpass PVG by 3.16dB and
1.63dB in PSNR metric.

To this end, we visualize both the rendered RGB images and depth maps with
different methods, as shown in the left column of Fig. 6. UC-NeRF and EmerNeRF
perform reasonable depth maps, with proper distribution of background
buildings and sky. PVG successfully detaches the sky but performs poorly on
the background buildings. As for GaussianPro, it overfits the training images
and fails in estimating depth. These seriously affect the render performance
in novel views with large offsets, where UC-NeRF, EmerNeRF, and PVG can render
the rough structure of the scene in novel views but GaussianPro fails. These
phenomena insight that the NeRF / Gaussian methods need to achieve precise
geometry reconstruction to obtain better performance in cross-lane NVS.

Figure 6: In the left column: we visualize the novel-view-synthesis results of
four NeRF as well as Gaussian methods with different offsets. In the right
column, we demonstrate the quantitative comparison between these four methods
with different offsets using PSNR metrics.

###  4.6 Discussion

Based on the results, it is evident that current approaches exhibit a
substantial gap, indicating their limited capability to meet the rigorous
requirements of cross-lane or closed-loop simulation. Future research
endeavors can leverage our proposed dataset and benchmark as a means to gauge
the extent to which novel methods can advance towards achieving closed-loop
simulation.

Limitation. Currently, our dataset has certain limitations as it was generated
using the Unreal-based Carla simulator. We see our work represents an initial
stride towards the accurate evaluation of novel driving view synthesis. As a
future endeavor, we plan to curate a real-world dataset encompassing cross-
lane ground truth data. This expansion will enhance the authenticity and
applicability of our evaluation, further advancing the field of novel driving
view synthesis.

Potential negative societal impacts arise from the widespread adoption of
autonomous driving vehicles. These include job displacement for professional
drivers, such as taxi and truck drivers, as well as potential economic
disruption in industries reliant on transportation. Additionally, concerns
regarding privacy and data security may arise due to increased collection and
utilization of personal information by autonomous vehicles. Ethical
considerations, such as decision-making during accidents, also pose challenges
that need to be addressed to ensure public trust and safety.

##  5 Conclusion

In conclusion, this paper addresses the challenge that existing evaluation
methods for NVS fall short in meeting the requirements of closed-loop
simulations, which demand the capability to render views beyond the original
trajectory. We introduce a unique driving view synthesis dataset and benchmark
for autonomous driving simulations. This dataset includes testing images
captured by deviating from the training trajectory, enabling the evaluation of
NVS approaches in a realistic manner. Our dataset establishes a much-needed
benchmark for advancing NVS techniques in closed-loop autonomous driving
simulation.

## Acknowledgments

The authors would like to thank Yuanyuan Gao and Jingfeng Li for preparing the
experiments.

## References

  * [1] Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos, Dmitry Ulyanov, and Victor Lempitsky.  Neural point-based graphics.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16, pages 696–712. Springer, 2020.
  * [2] Alexander Amini, Igor Gilitschenski, Jacob Phillips, Julia Moseyko, Rohan Banerjee, Sertac Karaman, and Daniela Rus.  Learning robust control policies for end-to-end autonomous driving from data-driven simulation.  IEEE Robot. Automat. Lett. (RA-L), 2020.
  * [3] Alexander Amini, Tsun-Hsuan Wang, Igor Gilitschenski, Wilko Schwarting, Zhijian Liu, Song Han, Sertac Karaman, and Daniela Rus.  Vista 2.0: An open, data-driven simulator for multimodal sensing and policy learning for autonomous vehicles.  In Proc. IEEE Int. Conf. Robot. Automat. (ICRA), 2022.
  * [4] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855–5864, 2021.
  * [5] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470–5479, 2022.
  * [6] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.  Zip-nerf: Anti-aliased grid-based neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19697–19705, 2023.
  * [7] Christian Berger and Bernhard Rumpe.  Engineering autonomous driving software.  arXiv preprint arXiv:1409.6579, 2014.
  * [8] Yohann Cabon, Naila Murray, and Martin Humenberger.  Virtual kitti 2.  arXiv preprint arXiv:2001.10773, 2020.
  * [9] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020.
  * [10] Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, et al.  Argoverse: 3d tracking and forecasting with rich maps.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8748–8757, 2019.
  * [11] David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann.  pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19457–19467, 2024.
  * [12] Gaurav Chaurasia, Sylvain Duchene, Olga Sorkine-Hornung, and George Drettakis.  Depth synthesis and local warps for plausible image-based navigation.  ACM transactions on graphics (TOG), 32(3):1–12, 2013.
  * [13] Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai.  Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images.  arXiv preprint arXiv:2403.14627, 2024.
  * [14] Yurui Chen, Chun Gu, Junzhe Jiang, Xiatian Zhu, and Li Zhang.  Periodic vibration gaussian: Dynamic urban scene reconstruction and real-time rendering.  arXiv preprint arXiv:2311.18561, 2023.
  * [15] Kai Cheng, Xiaoxiao Long, Kaizhi Yang, Yao Yao, Wei Yin, Yuexin Ma, Wenping Wang, and Xuejin Chen.  Gaussianpro: 3d gaussian splatting with progressive propagation.  In Proc. of the Int. Conf. on Machine Learning (ICML), 2024.
  * [16] Kai Cheng, Xiaoxiao Long, Wei Yin, Jin Wang, Zhiqiang Wu, Yuexin Ma, Kaixuan Wang, Xiaozhi Chen, and Xuejin Chen.  Uc-nerf: Neural radiance field for under-calibrated multi-view cameras.  In The Twelfth International Conference on Learning Representations, 2023.
  * [17] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016.
  * [18] Erwin Coumans and Yunfei Bai.  Pybullet, a python module for physics simulation for games, robotics and machine learning.  2016\.
  * [19] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun.  Carla: An open urban driving simulator.  In Proc. of the Conf. on Robot Learning (CoRL), 2017.
  * [20] Shuo Feng, Haowei Sun, Xintao Yan, Haojie Zhu, Zhengxia Zou, Shengyin Shen, and Henry X Liu.  Dense reinforcement learning for safety validation of autonomous vehicles.  Nature, 615(7953):620–627, 2023.
  * [21] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In 2012 IEEE conference on computer vision and pattern recognition, pages 3354–3361. IEEE, 2012.
  * [22] Lei He, Leheng Li, Wenchao Sun, Zeyu Han, Yichen Liu, Sifa Zheng, Jianqiang Wang, and Keqiang Li.  Neural radiance field in autonomous driving: A survey.  arXiv preprint arXiv:2404.13816, 2024.
  * [23] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al.  Planning-oriented autonomous driving.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17853–17862, 2023.
  * [24] Xinyu Huang, Xinjing Cheng, Qichuan Geng, Binbin Cao, Dingfu Zhou, Peng Wang, Yuanqing Lin, and Ruigang Yang.  The apolloscape dataset for autonomous driving.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 954–960, 2018.
  * [25] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.  3d gaussian splatting for real-time radiance field rendering.  ACM Transactions on Graphics, July 2023.
  * [26] Hao Li, Yuanyuan Gao, Dingwen Zhang, Chenming Wu, Yalun Dai, Chen Zhao, Haocheng Feng, Errui Ding, Jingdong Wang, and Junwei Han.  Ggrt: Towards generalizable 3d gaussians without pose priors in real-time.  arXiv preprint arXiv:2403.10147, 2024.
  * [27] Hao Li, Jingfeng Li, Dingwen Zhang, Chenming Wu, Jieqi Shi, Chen Zhao, Haocheng Feng, Errui Ding, Jingdong Wang, and Junwei Han.  Vdg: Vision-only dynamic gaussian for driving simulation.  arXiv preprint arXiv, 2024.
  * [28] Wei Li, CW Pan, Rong Zhang, JP Ren, YX Ma, Jin Fang, FL Yan, QC Geng, XY Huang, HJ Gong, et al.  Aads: Augmented autonomous driving simulation using data-driven algorithms.  Science robotics, 2019.
  * [29] Yueyuan Li, Wei Yuan, Songan Zhang, Weihao Yan, Qiyuan Shen, Chunxiang Wang, and Ming Yang.  Choose your simulator wisely: A review on open-source simulators for autonomous driving.  IEEE Transactions on Intelligent Vehicles, 2024.
  * [30] Zhuopeng Li, Chenming Wu, Liangjun Zhang, and Jianke Zhu.  Dgnr: Density-guided neural point rendering of large driving scenes.  IEEE Transactions on Automation Science and Engineering, 2024.
  * [31] Zhuopeng Li, Yilin Zhang, Chenming Wu, Jianke Zhu, and Liangjun Zhang.  Ho-gaussian: Hybrid optimization of 3d gaussian splatting for urban scenes.  arXiv preprint arXiv:2403.20032, 2024.
  * [32] Yiyi Liao, Jun Xie, and Andreas Geiger.  Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3292–3310, 2022.
  * [33] Ligang Liu, Xi Xia, Han Sun, Qi Shen, Juzhan Xu, Bin Chen, Hui Huang, and Kai Xu.  Object-aware guidance for autonomous scene reconstruction.  ACM Transactions on Graphics (TOG), 37(4):1–12, 2018.
  * [34] Mingyu Liu, Ekim Yurtsever, Jonathan Fossaert, Xingcheng Zhou, Walter Zimmer, Yuning Cui, Bare Luka Zagar, and Alois C Knoll.  A survey on autonomous driving datasets: Statistics, annotation quality, and a future outlook.  IEEE Transactions on Intelligent Vehicles, 2024.
  * [35] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  Communications of the ACM, 65(1):99–106, 2021.
  * [36] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  ACM transactions on graphics (TOG), 41(4):1–15, 2022.
  * [37] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder.  The mapillary vistas dataset for semantic understanding of street scenes.  In Proceedings of the IEEE international conference on computer vision, pages 4990–4999, 2017.
  * [38] Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide.  Neural scene graphs for dynamic scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2856–2865, 2021.
  * [39] Darius Rückert, Linus Franke, and Marc Stamminger.  Adop: Approximate differentiable one-pixel point rendering.  ACM Transactions on Graphics (ToG), 41(4):1–14, 2022.
  * [40] Johannes Lutz Schönberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [41] Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm.  Pixelwise view selection for unstructured multi-view stereo.  In European Conference on Computer Vision (ECCV), 2016.
  * [42] Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor.  Airsim: High-fidelity visual and physical simulation for autonomous vehicles.  In Field and Service Robotics: Results of the 11th International Conference, pages 621–635. Springer, 2018.
  * [43] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al.  Scalability in perception for autonomous driving: Waymo open dataset.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2446–2454, 2020.
  * [44] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar.  Block-nerf: Scalable large scene neural view synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8248–8258, 2022.
  * [45] Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, et al.  Nerfstudio: A modular framework for neural radiance field development.  In ACM SIGGRAPH 2023 Conference Proceedings, pages 1–12, 2023.
  * [46] Emanuel Todorov, Tom Erez, and Yuval Tassa.  Mujoco: A physics engine for model-based control.  In Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2012.
  * [47] Apostolia Tsirikoglou, Joel Kronander, Magnus Wrenninge, and Jonas Unger.  Procedural modeling and physically based rendering for synthetic data generation in automotive applications.  arXiv preprint arXiv:1710.06270, 2017.
  * [48] Haithem Turki, Jason Y Zhang, Francesco Ferroni, and Deva Ramanan.  Suds: Scalable urban dynamic scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12375–12385, 2023.
  * [49] Huijie Wang, Tianyu Li, Yang Li, Li Chen, Chonghao Sima, Zhenbo Liu, Bangjun Wang, Peijin Jia, Yuting Wang, Shengyin Jiang, et al.  Openlane-v2: A topology reasoning benchmark for unified 3d hd mapping.  Advances in Neural Information Processing Systems, 36, 2024.
  * [50] Linhan Wang, Kai Cheng, Shuo Lei, Shengkun Wang, Wei Yin, Chenyang Lei, Xiaoxiao Long, and Chang-Tien Lu.  Dc-gaussian: Improving 3d gaussian splatting for reflective dash cam videos.  arXiv preprint arXiv:2405.17705, 2024.
  * [51] Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al.  Argoverse 2: Next generation datasets for self-driving perception and forecasting.  arXiv preprint arXiv:2301.00493, 2023.
  * [52] Chenming Wu, Jiadai Sun, Zhelun Shen, and Liangjun Zhang.  Mapnerf: Incorporating map priors into neural radiance fields for driving view simulation.  In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7082–7088. IEEE, 2023.
  * [53] Ke Wu, Kaizhao Zhang, Zhiwei Zhang, Shanshuai Yuan, Muer Tie, Julong Wei, Zijun Xu, Jieru Zhao, Zhongxue Gan, and Wenchao Ding.  Hgs-mapping: Online dense mapping using hybrid gaussian representation in urban scenes.  arXiv preprint arXiv:2403.20159, 2024.
  * [54] Zirui Wu, Tianyu Liu, Liyi Luo, Zhide Zhong, Jianteng Chen, Hongmin Xiao, Chao Hou, Haozhe Lou, Yuantao Chen, Runyi Yang, et al.  Mars: An instance-aware, modular and realistic simulator for autonomous driving.  In CAAI International Conference on Artificial Intelligence, pages 3–15. Springer, 2023.
  * [55] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo.  Segformer: Simple and efficient design for semantic segmentation with transformers.  Advances in neural information processing systems, 34:12077–12090, 2021.
  * [56] Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, and Sida Peng.  Street gaussians for modeling dynamic urban scenes.  arXiv preprint arXiv:2401.01339, 2024.
  * [57] Jiawei Yang, Boris Ivanovic, Or Litany, Xinshuo Weng, Seung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler, Marco Pavone, et al.  Emernerf: Emergent spatial-temporal scene decomposition via self-supervision.  In Proc. of the Int. Conf. on Learning Representations (ICLR), 2024.
  * [58] Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun.  Unisim: A neural closed-loop sensor simulator.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1389–1399, 2023.
  * [59] Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine-Hornung.  Differentiable surface splatting for point-based geometry processing.  ACM Transactions on Graphics (TOG), 38(6):1–14, 2019.
  * [60] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell.  Bdd100k: A diverse driving dataset for heterogeneous multitask learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2636–2645, 2020.
  * [61] Wenzhao Zheng, Ruiqi Song, Xianda Guo, and Long Chen.  Genad: Generative end-to-end autonomous driving.  arXiv preprint arXiv:2402.11502, 2024.
  * [62] Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang, Deqing Sun, and Ming-Hsuan Yang.  Drivinggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes.  In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024.


Also at ]Department of Physics, Faculty of Science, Kyoto University,
Kitashirakawa Oiwake-cho, Sakyo-ku, Kyoto 606-8502, Japan

Also at ]Tomonaga Center for the History of the Universe (TCHoU), University
of Tsukuba, Japan

DOSUE-RR Collaboration

# Search for Dark Photon Dark Matter in the Mass Range
$\mathbf{41\text{--}74}\,\bm{\mu}\mathbf{eV}$
using Millimeter-Wave Receiver and Radioshielding Box

S. Adachi<EMAIL_ADDRESS>Hakubi Center for Advanced
Research, Kyoto University, Kyoto 606-8501, Japan  [  R. Fujinaka  Department
of Physics, Faculty of Science, Kyoto University, Kyoto 606-8502, Japan  S.
Honda  Division of Physics, Faculty of Pure and Applied Sciences, University
of Tsukuba, Ibaraki, 305-8571, Japan  [  Y. Muto  Department of Physics,
Faculty of Science, Kyoto University, Kyoto 606-8502, Japan  H. Nakata
Department of Physics, Faculty of Science, Kyoto University, Kyoto 606-8502,
Japan  Y. Sueno  Department of Physics, Faculty of Science, Kyoto University,
Kyoto 606-8502, Japan  T. Sumida  Department of Physics, Faculty of Science,
Kyoto University, Kyoto 606-8502, Japan  J. Suzuki  Department of Physics,
Faculty of Science, Kyoto University, Kyoto 606-8502, Japan  O. Tajima
Department of Physics, Faculty of Science, Kyoto University, Kyoto 606-8502,
Japan  H. Takeuchi  Department of Physics, Faculty of Science, Kyoto
University, Kyoto 606-8502, Japan

###### Abstract

Dark photons have been considered potential candidates for dark matter. The
dark photon dark matter (DPDM) has a mass and interacts with electromagnetic
fields via kinetic mixing with a coupling constant of $\chi$. Thus, DPDMs are
converted into ordinary photons at metal surfaces. Using a millimeter-wave
receiver set in a radioshielding box, we performed experiments to detect the
conversion photons from the DPDM in the frequency range 10–18 GHz, which
corresponds to a mass range $\text{41--74}\,\mu\text{eV}$. We found no
conversion photon signal in this range and set the upper limits to
$\chi<(0.5\text{--}3.9)\times 10^{-10}$ at a 95% confidence level.

##  I Introduction

The study on dark matter is a prominent topic in the fields of particle
physics and cosmology. Dark matter is localized in most galactic halos.
However, its interactions with ordinary particles, standard-model particles,
remain unexplained except via gravity. Dark photons are considered potential
candidates for dark matter. The mass ($m_{\rm DP}$) is nonzero and the dark
photon interacts with the electromagnetic fields via kinetic mixing with a
coupling constant of $\chi$ [1]. Dark photon dark matter (DPDM) in the
$\mu\text{eV}$ or meV mass ranges have been predicted to exist in the context
of high-scale inflation models and a part of string theories [2, 1].

DPDMs convert to standard-model photons at the boundary of the mediums via the
kinetic mixing [3]. Metal surfaces are used for such conversions [4, 5, 6, 7,
8]. Because the speed of DPDM ($v_{\rm DP}$) is considerably low compared to
the speed of light ($v_{\rm DP}/c\approx 10^{-3}$), the direction of the
conversion photons is almost perpendicular to the surface of the plate, within
$0.1^{\circ}$ [9]. The frequency of the conversion photons approximately
$\nu_{0}$ roughly corresponds to the mass of the DPDM because of energy
conservation, that is, $h\nu_{0}\simeq m_{\rm DP}c^{2}$, where $h$ is the
Planck constant. Owing to the low speed of the DPDM, the frequency width of
the conversion photons ($\Delta\nu_{\text{DP}}$) is considerably narrow
compared with the peak frequency, $\Delta\nu_{\text{DP}}/\nu_{0}\approx
10^{-6}$ [10]. Thus, the conversion photons should be observed as a peak in
the frequency spectrum.

The signal power of the conversion photons $P_{\text{DP}}$ is denoted by [4],

$\displaystyle P_{\text{DP}}$ $\displaystyle=$ $\displaystyle(6.4\times
10^{-2}\,\mathrm{aW})\times\left(\frac{\chi}{10^{-10}}\right)^{2}\left(\frac{A_{\text{eff}}}{10\,\text{cm}^{2}}\right)$
(1)
$\displaystyle\quad\times\left(\frac{\rho}{0.39\,\text{GeV}/\text{cm}^{3}}\right)\left(\frac{\alpha}{\sqrt{2/3}}\right)^{2},$

where $A_{\text{eff}}$ is the effective antenna aperture, $\rho=0.39\pm
0.03\,\text{GeV/cm}^{3}$ is the energy density of the dark matter in the
galactic halo [11], and $\alpha$ is a factor determined by the angular
distribution of the DPDM field relative to the sensitive polarization axis. We
selected $\alpha=\sqrt{1/3}$ for the case of random distribution using a
single-polarization detector [4].

The DOSUE-RR (Dark-photon dark-matter observing system for Un-Explored Radio-
Range) aims to detect the conversion photons of DPDM using millimeter-wave
receivers. We achieved stringent constraints, $\chi<(\text{0.3--2.0})\times
10^{-10}$ in the mass range $74\text{--}110\,\mu\text{eV}$ by using a
cryogenic millimeter-wave receiver [6]. Searches using millimeter-wave
receivers have the advantage of broad coverage of the mass range. In this
study, we switched the explored mass range $\text{41--74}\,\mu\text{eV}$,
corresponding to the frequency range 10–18 GHz.

The remainder of this paper is organized as follows. In Sec. II, we describe
the experimental setup. The calibration procedures are described in Sec. III.
Section IV details the measurement procedure and analysis of the DPDM search.
Section V describes the systematic errors. In Sec. VI, we present the
constraints on the coupling constant $\chi$. Finally, Sec. VII presents the
conclusions of this study.

##  II Experimental Setup

Figure 1 shows the setup to search for DPDM. A radioshielding box was
essential for this measurement because manmade millimeter-wave signals and
their harmonics can enter the antenna, for example, Wi-Fi and
telecommunication signals. This box comprises 1 mm-thick aluminum plates and
sheets of the radio-wave absorber, Eccosorb CV-3, with a reflectance of
$-40\,\text{dB}$ above 8 GHz [12]. An aluminum plate with a thickness of 4 mm
and an area of $690\,\text{mm}\times 690\,\text{mm}$ was placed at the top of
the box. The conversion photons from the plate surface entered the horn
antenna at the bottom of the box. The horn antenna was Pasternack
PE9856/SF-60, with an aperture size of $123.8\,\text{mm}\times
91.9\,\text{mm}$. Using a tiltmeter, we confirmed that the antenna direction
was perpendicular to the plate with $0.1^{\circ}$ precision.

The signals collected by the antenna were transmitted to a coaxial cable
through a waveguide-to-coaxial adapter. A low-noise amplifier (Low-Noise
Factory LNF-LNR6_20A) and another amplifier (Mini-Circuits ZVA-183G-S+) were
used to amplify the signals. An isolator (Pasternack PE8304) and two
attenuators were inserted to suppress standing waves over the signal path. The
amplified signals were measured using a signal analyzer (ANRITSU MS2840A). The
signal analyzer performed a Fast Fourier Transform to measure the frequency
spectra. The resolution bandwidth was set to 300 Hz
($\ll\Delta\nu_{\text{DP}}\approx 10\,\text{kHz}$). The signal analyzer can
simultaneously obtain spectral data for a limited frequency range of 2.5 MHz
using this resolution bandwidth. There are 32,769 data points in each dataset
for the 2.5 MHz range, that is, the frequency interval was 76.3 Hz.

Figure 1:  Our setup for the DPDM search. Optical components were assembled in
a radioshielding box. We closed the front panel of the box when we collected
data. The box comprises aluminum panels and radio-wave absorbers. We set an
aluminum plate and a horn antenna at the top and bottom of the box,
respectively. The conversion photons were collected by the antenna. The
amplified signals were measured by a signal analyzer.

##  III Calibration

###  III.1 Effective antenna aperture and beam pattern

The effective antenna aperture $A_{\text{eff}}$ is a visible area of the
antenna and is smaller than its physical aperture area. This can be estimated
from the beam pattern $B$ using the antenna theorem [13]:

$\displaystyle A_{\text{eff}}\Omega_{A}$ $\displaystyle=$
$\displaystyle\lambda^{2},$ $\displaystyle\Omega_{A}$ $\displaystyle=$
$\displaystyle\int B(\theta,\phi)d\Omega,$

where $\lambda$ is the wavelength, $\theta$ is the polar angle, and $\phi$ is
the azimuthal angle. $B$ is defined as the relative antenna gain in each
direction, whose peak value is normalized to 1. Using a finite element method
simulation software, ANSYS-HFSS (2022R2), we simulated the beam patterns from
10.0 GHz to 18.0 GHz with 0.5 GHz intervals. Thereafter, we calculated the
$A_{\text{eff}}$ at each frequency, as shown in Figure 2.

Figure 2:  Simulated $A_{\mathrm{eff}}$ as a function of the frequency. A gray
band indicates the size of the systematic error.

To validate the simulation results, we measured the beam patterns at each
frequency using a near-field measurement system, as shown in Fig. 3. The
vector network analyzer outputs a monochromatic millimeter-wave signal from
the transmitter antenna, which is the horn antenna used in the DPDM search.
The output signal was collected using another antenna (the receiver antenna)
set on an X-Y movable stage. Further, the power was measured using the vector
network analyzer. The measurements were repeated by moving the receiver
antenna. The maximum angular coverage of these measurements was $20^{\circ}$.
The measured frequencies were the same as those in the simulation, that is,
from 10.0 GHz to 18.0 GHz with 0.5 GHz intervals. We used two types of
receiver antenna with the waveguides WR-90 and WR-62 for 10.0–12.0 GHz and
12.5–18.0 GHz measurements, respectively. The distances between the two
antennas were $192\,\text{mm}$ and $238\,\text{mm}$ for measurements with
WR-90 and WR-64, respectively.

Figure 3:  Conceptual illustration for the setup of the beam pattern
measurement. A vector network analyzer outputs a monochromatic millimeter-wave
signal from a transmitter antenna, which is the horn antenna used in the DPDM
search. The output signal was collected by a receiver antenna set on the X-Y
movable stage. The beam pattern was measured by moving the X-Y stage. The beam
patterns of the receiver antennas were known in advance and their effects were
considered in calculating the beam pattern of the transmitter antenna.

The beam pattern measured at 10.0 GHz is shown in Fig. 4. The simulated beam
pattern was superimposed for comparison. The beam widths at each frequency are
shown in Fig. 5. The difference between the measurement and simulation was
less than 6.6%, which is sufficiently small for this study.

Figure 6 shows the measured beam center at each frequency. We observed shifts
in the beam center from $0\,^{\circ}$, which did not appear in the simulation.
This was evident above 17 GHz in the E-plane. The mechanical asymmetry in the
E-plane of the waveguide-to-coaxial adapter causes the shifts. Therefore, a
systematic error was assigned. The gray band in the E-plane of Fig. 6
indicates the size of the systematic error used in the analysis.

As regards $A_{\text{eff}}$, we evaluated the difference between the
simulation and measurement by using calculations with limited angular coverage
within $\text{17.5}\,^{\circ}$. The maximum difference between them is
$6.2\%$, which is assigned as a systematic error, as shown in Fig. 2.

Figure 4:  Beam pattern $B$ of the horn antenna at 10 GHz in E-plane (left)
and H-plane (right). Points and solid lines represent the measurement and
simulation, respectively.  Figure 5:  Beam width, full-width-half-maximum
(FWHM) of the beam pattern, as a function of frequency in E-plane (left) and
H-plane (right). Points and solid lines indicate the measurement and
simulation, respectively. The bottom panels show their ratios. The scan step
of the X-Y stage dominates the error in the measurement.  Figure 6:  Measured
beam centers at each frequency in E-plane (left) and H-plane (right). The open
markers and filled markers indicate the type of receiver antenna used in the
measurements. Shifts from $0^{\circ}$ were assigned to the systematic error. A
gray band indicates the size of the systematic error.

###  III.2 Receiver gain

The power measured by the signal analyzer $P_{\text{out}}$ is an amplified
total power of the receiver input power $P_{\text{in}}$ and receiver noise,
which is represented by the receiver temperature $T_{\text{rx}}$:

$P_{\text{out}}=G\left(P_{\text{in}}+k_{\text{B}}T_{\text{rx}}\Delta\nu\right),$
(2)

where $G$ is the receiver gain, $k_{\text{B}}$ is the Boltzmann constant, and
$\Delta\nu$ is the frequency bandwidth. To obtain $G$ and $T_{\text{rx}}$, we
performed a calibration using hot (room temperature) and cold (77K) blackbody
radiation sources. The millimeter-wave absorber Eccosrb CV-3 was used as the
blackbody radiation source. The temperature under room conditions
($T^{\text{hot}}\sim 290\,\text{K}$) was measured using a radiation
thermometer for each calibration. Cold radiation
($T^{\text{cold}}=77\,\text{K}$) was emitted by an absorber immersed in a
liquid-nitrogen bath made of a foamed polystyrene box, transparent to radio
waves [14, 15]. Each source was set in front of the antenna aperture to fully
cover the antenna. Each $P_{\text{in}}$ was calculated as
$P_{\text{in}}^{\text{hot}}=k_{\text{B}}T^{\text{hot}}\Delta\nu$ or
$P_{\text{in}}^{\text{cold}}=k_{\text{B}}T^{\text{cold}}\Delta\nu$. On this
basis, we obtained $G$ and $T_{\text{rx}}$ according to Eq. (2). In this
calibration, we set the resolution bandwidth to be the same as that used in
the DPDM search. Figure 7 shows the $G$ and $T_{\text{rx}}$ as a function of
frequency. The typical gain was $60\,\text{dB}$ and $T_{\text{rx}}$ was
$\approx 200\,\text{K}$. The system temperature is the sum of $T_{\text{rx}}$
and input radiation ($T_{\text{in}}$) during the DPDM search; that is,
$T_{\text{sys}}=T_{\text{in}}+T_{\text{rx}}$. $T_{\text{in}}$ is dominated by
the thermal loading at room temperature.

Figure 7:  Receiver gain ($G$), receiver temperature ($T_{\text{rx}}$), and
system temperature ($T_{\text{sys}}$) as a function of frequency.
$T_{\text{in}}\equiv T_{\text{sys}}-T_{\text{rx}}$ corresponds to loading into
the antenna, which was dominated by the room temperature radiation ($\sim
290\,\text{K}$). A loss of the waveguide-to-coaxial adapter increases the
$T_{\text{rx}}$ at 15.7 GHz.

##  IV Measurement and analysis

The DPDM search was performed from March 6 to 17, 2023. We collected data for
frequencies ranging from 10.0 GHz to 18.0 GHz, corresponding to a DPDM mass
range of 41 $\mu\text{eV}$ to 74 $\mu\text{eV}$. The time to accumulate data
was set to 2 s for each data chunk. The center frequency was shifted by 2.0
MHz after obtaining 12 chunks of data for each frequency region. We collected
48,000 data chunks in 4,000 frequency regions. Two consecutive frequency
ranges were overlapped and used to estimate the statistical error in the
analysis. Further, we performed gain calibration before and after each 100 MHz
data acquisition. The typical time interval between the calibrations was 40
min. The maximum difference in $G$ was $1.5\%$.

The analysis method is the same as that used in our previous study [6]. The 12
data chunks at each frequency were averaged and rebinned to
$\Delta\nu=2\,\text{kHz}$. The averaged spectra were converted to
$P_{\text{in}}$ by Eq. (2). To extract the signal power of the conversion
photons $P_{\text{DP}}$, we fitted a function to the $P_{\text{in}}$ spectra
at each frequency for the signal peak $\nu_{0}\equiv m_{\text{DP}}c^{2}/h$.
The fitting function at $\nu_{0}$ comprises a signal model, $f_{\text{sig}}$
and the baseline background noise $f_{\text{bg}}$, which is a one-dimensional
polynomial $f_{\text{bg}}(\nu;a,b)=a(\nu-\nu_{0})+b$,

$f(\nu;P_{\text{DP}},a,b)=P_{\text{DP}}\times
f_{\text{sig}}(\nu;\nu_{0})+f_{\text{bg}}(\nu;\nu_{0},a,b)~{}.$ (3)

Here, $a$ and $b$ are the parameters used to model the baseline noise, and
$f_{\text{sig}}$ is the difference in the cumulative functions introduced to
account for the effect of the finite bin width

$f_{\text{sig}}(\nu;\nu_{0})=F(\nu+\Delta\nu/2;\nu_{0})-F(\nu-\Delta\nu/2;\nu_{0}),$
(4)

where,

$\displaystyle F(v)$ $\displaystyle=$
$\displaystyle\int_{0}^{v}{\text{d}v^{\prime}}\int^{4\pi}{\text{d}\Omega~{}g(\mathbf{v^{\prime}};v_{\text{c}},\mathbf{v_{\text{E}}})~{}v^{\prime
2}}~{},$ (5) $\displaystyle g(\mathbf{v};v_{\text{c}},\mathbf{v_{\text{E}}})$
$\displaystyle=$
$\displaystyle\frac{1}{\left(\sqrt{\pi}v_{\text{c}}\right)^{3}}\exp{-\frac{|\mathbf{v}+\mathbf{v_{\text{E}}}|^{2}}{v_{\text{c}}^{2}}}.$
(6)

Here, $F$ is a function of the DPDM speed;
$v\equiv\absolutevalue{\mathbf{v}}=c\sqrt{1-(\nu_{0}/\nu)^{2}}$;
$g(\mathbf{v})$ denotes the velocity distribution,; $v_{\text{c}}$ denotes the
circular rotational speed of the galaxy, and $\mathbf{v_{\text{E}}}$ is the
velocity of the Earth within the frame of the Galaxy. We assume a Maxwell-
Boltzmann distribution for $g(\mathbf{v})$ [16]. Furthermore, we assumed
$\absolutevalue{\mathbf{v}_{\text{E}}}=v_{\text{c}}=220\,\text{km}/\text{s}$
in the fittings, as in many dark matter searches [17, 18, 19, 20, 21]. The
speed of the Earth $v_{\text{E}}$ varied from $220\,\text{km}/\text{s}$ to
$242\,\text{km}/\text{s}$ during the search measurement owing to the
revolution and rotation of the Earth [17]. This caused a small change in the
peak frequency. This effect is considered as a systematic error in Sec. V.

We varied the peak frequency $\nu_{0}$ from 10.0 to 18.0 GHz at small
intervals of $\Delta\nu=2\,\text{kHz}$ and performed a fit by floating
$P_{\text{DP}}$, $a$, and $b$ in the frequency range from
$\nu_{0}-50\,\text{kHz}$ to $\nu_{0}+200\,\text{kHz}$. The error on each data
point in the fit was estimated from the standard deviations of the data for
the 0.25 MHz regions below and above the fit range.

Figure 8 shows the fit results for the data. Over the entire frequency range,
$P_{\text{DP}}$ was less than a few aW. We calculated the statistical
significance at each frequency in the same manner as in our previous study
[6]. The minimum local $p$ value was $1.1\times 10^{-6}\,\%$ at 13.913524 GHz.
The spectrum at this frequency is shown in the top panel of Fig. 9.
Furthermore, we calculated the global $p$ value (i.e., the probability of
exceeding the minimum local $p$ value at any frequency) to be $2.2\,\%$, which
is not a significant excess. The calculation methods for the local and global
$p$ values are described in Appendix A.

Figure 8:  Extracted signal powers (top) and their local $p$ values (bottom)
at each frequency in the initial data taken from March 6 to 17, 2023.

To further verify the results at 13.913524 GHz, we collected additional data
focusing on the frequency range from 13.912 to 13.914 GHz on April 6, 2023. We
collected 600 data chunks, having 50 times more statistics than the initial
data. The spectrum and fit results at 13.913524 GHz are shown in the bottom
panel of Fig. 9. The local $p$ value is 5.7%. The $P_{\text{DP}}$ and local
$p$ values in this frequency range are shown in Fig. 10. We confirmed that
there was no significant excess. Even after combining initial and additional
data, the local $p$ value was 1.2%. Therefore, we concluded that no
significant signal exists.

Figure 9:  Measured powers in the frequency region of the minimum $p$ value in
the initial data. Fit curves are overlaid. The top panel shows the results of
the initial data. The bottom panel shows the results of the additional data
whose statistics are 50 times higher than the initial data. No significant
signal was found in the data with increased statistics.  Figure 10:  Extracted
signal powers (top) and their local $p$ values (bottom) as a function of
frequency for the additional data. The arrows indicate the frequency of the
minimum $p$ value in the initial data.

##  V Systematics Uncertainties

Table 1:  Systematic uncertainties associated with the coupling constant $\chi$. The uncertainty of the antenna pointing direction depends on the frequency.  Source | ($\%$)
---|---
Antenna-pointing direction | 0.3–7.8
Effective aperture area ($A_{\text{eff}}$) | 3.1
Earth speed variation ($v_{\text{E}}$) | 2.3
Frequency binning | 1.3
Gain | 1.1
Alignment of instruments |  $<$ 0.1
Direction of conversion photons |  $<$ 0.1
Dark matter density ($\rho$) | 3.9
Total | 5.8–9.7

The systematic uncertainties associated with the coupling constant $\chi$ are
summarized in Table 1. Uncertainty from antenna-pointing direction was
estimated from the beam center shift from $0\,^{\circ}$ as mentioned in Sec.
III.1. The beam center shifts decrease the measured power of the conversion
photons. These effects were estimated assuming a Gaussian beam pattern. Larger
errors were assigned above $17\,\text{GHz}$, as shown in the Fig. 6. The
uncertainty in $A_{\text{eff}}$ was estimated from the difference between the
calibration and simulation results, as shown in Fig. 2. The variation in Earth
speed $v_{\text{E}}$ used in Eq. (5) during the search measurements creates a
small bias in $P_{\text{DP}}$. This was estimated by changing $v_{\text{E}}$
used in the fit for the DPDM signal simulation. Frequency binning causes
another bias in the fit. This was estimated from the variation in the
$P_{\text{DP}}$ when the signal frequency was varied by $\pm\Delta\nu/2=\pm
1\,\text{kHz}$ in the simulation. Some of the uncertainties in the gain were
conservatively considered by the maximum difference between the two
calibrations, as mentioned in Sec. IV (1.5% on $P_{\text{DP}}$ corresponds to
0.8% on $\chi$). Furthermore, gain uncertainties from the source temperature
and emissivity were considered (0.7% on $\chi$). The square root of the sum of
their squares was assigned to the total systematic error of the gain. For
instrumental alignment, the tilts of the plate and antenna were at most
$0.1^{\circ}$ and $0.05^{\circ}$, respectively. This effect was estimated
using the same method as that used for the uncertainty from the antenna-
pointing direction. Similarly, the uncertainty related to the direction of the
conversion photons was estimated to be less than $0.1\%$. For the dark matter
density, we used the uncertainty described in [11]. The total systematic error
was $\text{5.7--9.7}\%$.

##  VI Constraints on the Coupling Constant

The upper bounds of $P_{\text{DP}}$ at a 95% confidence level for each
frequency were calculated in the same manner as in [6]. The details are
provided in Appendix A. The upper limits on $P_{\text{DP}}$ combining the
initial and additional data, were converted into the upper limits on $\chi$
using Eq. (1). Systematic uncertainties were considered during this process.
As shown in Fig. 11, we obtained the limits for $\chi<(\text{0.5--3.9})\times
10^{-10}$ at a 95% confidence level in the mass range $\text{41--
74}\,\mu\text{eV}$. This is tighter than that provided by the cosmological
observations [1]. In a different mass range, our previous study which used a
cryogenic system already achieved tighter constraints than this study [6].

Figure 11:  Upper limits to the coupling constant $\chi$ at 95% confidence
level. [6, 7, 8, 22, 23, 5, 24, 1, 25, 26, 27, 3]

##  VII Conclusions

We searched for DPDM using a room-temperature millimeter-wave receiver and
radioshielding box. The explored frequency range 10–18 GHz corresponds to the
mass range $\text{41--74}\,\mu\text{eV}$. We found no significant signal and
set the upper limits of $\chi<(0.5\text{--}3.9)\times 10^{-10}$ at the 95%
confidence level. The achieved constraint was tighter than that obtained from
cosmological observations.

## Acknowledgements

Part of this work was supported by Microwave Energy Transmission Laboratory
(METLAB), Research Institute for Sustainable Humanosphere, Kyoto University.
The authors thank Naoki Shinohara and Hideki Ujihara. This research was
supported by JSPS KAKENHI under grant numbers 20K14486, 20K20427, 21H01093,
21H05460, and 23H00111. This research was also supported by grants from the
Murata and Sumitomo Foundations. SA and TS acknowledge the Hakubi Project and
SPIRITS Program of Kyoto University, respectively.

##  Appendix A Statistical Calculations

###  A.1 Local $p$ value

To understand the statistical significance of the fit results for
$P_{\text{DP}}$, we employed the null-sample method. This method was used in
our previous study [6]. The null samples are the spectra generated by
subtracting six data chunks from the other six data chunks in the measured 12
data chunks, which model the noise-only spectra. There were 462 combinations
(i.e., $\binom{12}{6}\times 1/2$) generating null samples at each frequency
range. For each null sample at each frequency, we obtained $P_{\text{DP}}$,
error $\sigma$, and their ratio $x\equiv P_{\text{DP}}/\sigma$. Figure 12
shows the distribution of the $x$ obtained from the fit to the data and null
samples. The null-sample distribution $\mathcal{P}(x)$ is consistent with the
data. We calculated a local $p$ value under the zero-signal hypothesis for the
fitted result with $x=x_{0}$ as follows:

$p=\frac{\int_{x_{0}}^{\infty}{\mathcal{P}(x)}dx}{\int_{-\infty}^{\infty}{\mathcal{P}(x)}dx}~{}.$
(7)

The fraction of the shaded area in Fig. 12 corresponds to this value.

Figure 12:  Distributions of $x\equiv P_{\mathrm{DP}}/\sigma$ for data and
null samples($\mathcal{P}(x)$). Here, the plot for the null samples was
normalized to be the same area as the data. The local $p$ value for the fit
result with $x=x_{0}$ was obtained from the fraction of the shaded area.

###  A.2 Global $p$ value

There are many fitting results in the frequency range 10–18 GHz ($4\times
10^{6}$). The global $p$ value, which considers the look-elsewhere effect, is
important for determining whether a signal signature exists. Using the minimum
$p$ local $p^{\text{min}}$, the global $p$ value for $N$ fittings was
calculated using the following formula [5, 28]:

$p_{\text{global}}(p^{\text{min}};N)=1-(1-p^{\text{min}})^{\sum_{i=0}^{N}\mu_{i}},$
(8)

where $\mu_{i}$ ($0<\mu_{i}\leq 1$) is a scale factor for $i$-th fitting that
accounts for the independence between the fittings in the neighbor bins.
Because the peak width of the DPDM signal depends on the frequency, $\mu_{i}$
depends on the frequency. To estimate $\mu$ for this experiment, we separated
the null samples into 2 MHz frequency intervals, each of which contained 1000
fit results, and checked the distribution of the minimum local $p$ values
($p^{\text{min}}$) in each of the 2 MHz intervals. The $p^{\text{min}}$
distribution for the frequency range of 10.0–10.1 GHz is shown in the top
panel of Fig. 13. A global $p$ value with $N=1000$ is calculated from the
$p^{\text{min}}$ distribution $\mathcal{Q}(p^{\text{min}})$ as follows:

$p_{\text{global}}(p^{\text{min}}_{0};N)=\frac{\int_{0}^{p^{\text{min}}_{0}}{\mathcal{Q}(p^{\text{min}};N)}dp^{\text{min}}}{\int_{0}^{\infty}{\mathcal{Q}(p^{\text{min}};N)}dp^{\text{min}}}~{}.$
(9)

By using the plot of the calculated $p_{\text{global}}$ at each
$p^{\text{min}}$, we estimated $\mu$ within this frequency range, as shown in
the bottom panel of Fig. 13.

Figure 13:  $p^{\text{min}}$ distribution (top) when a number of fittings $N$
is $1000$ and a fitting of global $p$ value (bottom) with a free parameter of
$\mu$ in the frequency range 10.0–10.1 GHz. The fraction of the shaded area in
the top panel is an estimated global $p$ value of the probability that the
minimum local $p$ value is $p_{0}$ when $N=1000$.

We extracted $\mu$ for each frequency range of 10.0–10.1, 11.0–11.1 GHz,…, and
17.0–17.1 GHz as shown in Fig. 14. By fitting with a linear function, $\mu$ at
$\nu_{i}$ can be obtained as

$\mu(\nu_{i})=(-0.025\pm 0.001)\nu_{i}\text{[GHz]}+(0.86\pm 0.02),$ (10)

where $\nu_{i}$ is the signal peak frequency at the $i$-th fit. We calculated
the global $p$ value using Eq. (8) and Eq. (10).

Figure 14:  Estimated $\mu$ for each frequency and the linear fit.

###  A.3 Upper Limits on $P_{\text{DP}}$

Using Eq. (7), we obtained $p=0.05$ at $x_{0}=1.79$, which is slightly larger
than that of a normal Gaussian distribution (1.65 [29]). This is because the
distribution tail of $\mathcal{P}(x)$ is slightly wider than the normal
Gaussian. The upper bounds on $P_{\text{DP}}$ at the 95% confidence level for
each frequency were calculated as

$\text{max}(0,P_{\text{DP}})+1.79\sigma~{}.$ (11)

## References

  * Arias  _et al._ [2012] P. Arias, D. Cadamuro, M. Goodsell, J. Jaeckel, J. Redondo, and A. Ringwald, J. Cosmol. Astropart. Phys. 2012 (06), 013.
  * Graham  _et al._ [2016] P. W. Graham, J. Mardon, and S. Rajendran, Phys. Rev. D 93, 103520 (2016).
  * Caputo  _et al._ [2021] A. Caputo, A. J. Millar, C. A. J. O’Hare, and E. Vitagliano, Phys. Rev. D 104, 095029 (2021).
  * Horns  _et al._ [2013] D. Horns, J. Jaeckel, A. Lindner, A. Lobanov, J. Redondo, and A. Ringwald, J. Cosmol. Astropart. Phys. 2013 (04), 016.
  * Tomita  _et al._ [2020] N. Tomita, S. Oguri, Y. Inoue, M. Minowa, T. Nagasaki, J. Suzuki, and O. Tajima, J. Cosmol. Astropart. Phys. 2020 (09), 012.
  * Kotaka  _et al._ [2023] S. Kotaka _et al._ (DOSUE-RR Collaboration), Phys. Rev. Lett. 130, 071805 (2023).
  * Ramanathan  _et al._ [2023] K. Ramanathan, N. Klimovich, R. Basu Thakur, B. H. Eom, H. G. Leduc, S. Shu, A. D. Beyer, and P. K. Day, Phys. Rev. Lett. 130, 231001 (2023).
  * Brun  _et al._ [2019] P. Brun, L. Chevalier, and C. Flouzat, Phys. Rev. Lett. 122, 201801 (2019).
  * Jaeckel and Knirck [2016] J. Jaeckel and S. Knirck, J. Cosmol. Astropart. Phys. 2016 (01), 005.
  * Sikivie [1983] P. Sikivie, Phys. Rev. Lett. 51, 1415 (1983).
  * Catena and Ullio [2010] R. Catena and P. Ullio, J. Cosmol. Astropart. Phys. 2010 (08), 004.
  * Laird Technologies, Inc. [2021] Laird Technologies, Inc., Eccosorb CV Datasheet (2021).
  * Kraus [1966] J. D. Kraus,  _Radio astronomy_ (McGraw-Hill, 1966).
  * Goldsmith [1998] P. Goldsmith,  _Quasioptical Systems: Gaussian Beam Quasioptical Propogation and Applications_, IEEE Press Series on RF and Microwave Technology (Wiley, 1998).
  * Choi  _et al._ [2013] J. Choi, H. Ishitsuka, S. Mima, S. Oguri, K. Takahashi, and O. Tajima, Rev. Sci. Instrum. 84, 114502 (2013) .
  * Drukier  _et al._ [1986] A. K. Drukier, K. Freese, and D. N. Spergel, Phys. Rev. D 33, 3495 (1986).
  * Mayet  _et al._ [2016] F. Mayet, A. Green, J. Battat, J. Billard, N. Bozorgnia, G. Gelmini, P. Gondolo, B. Kavanagh, S. Lee, D. Loomba _et al._ , Phys. Rep. 627, 1 (2016).
  * Akerib  _et al._ [2017] D. S. Akerib _et al._ (LUX Collaboration), Phys. Rev. Lett. 118, 021303 (2017).
  * Aprile  _et al._ [2018] E. Aprile _et al._ (XENON Collaboration 7), Phys. Rev. Lett. 121, 111302 (2018).
  * Fu  _et al._ [2017] C. Fu, X. Cui, X. Zhou, X. Chen, Y. Chen, D. Fang, K. Giboni, F. Giuliani, K. Han, X. Huang, _et al._ (PandaX-II Collaboration), Phys. Rev. Lett. 118, 071301 (2017).
  * Agnes  _et al._ [2018] P. Agnes _et al._ (The DarkSide Collaboration), Phys. Rev. Lett. 121, 111303 (2018).
  * Dixit  _et al._ [2021] A. V. Dixit, S. Chakram, K. He, A. Agrawal, R. K. Naik, D. I. Schuster, and A. Chou, Phys. Rev. Lett. 126, 141302 (2021).
  * Cervantes  _et al._ [2022] R. Cervantes, G. Carosi, C. Hanretty, S. Kimes, B. H. LaRoque, G. Leum, P. Mohapatra, N. S. Oblath, R. Ottens, Y. Park, G. Rybka, J. Sinnis, and J. Yang, Phys. Rev. Lett. 129, 201301 (2022).
  * Fan  _et al._ [2022] X. Fan, G. Gabrielse, P. W. Graham, R. Harnik, T. G. Myers, H. Ramani, B. A. D. Sukra, S. S. Y. Wong, and Y. Xiao, Phys. Rev. Lett. 129, 261801 (2022).
  * Vinyoles  _et al._ [2015] N. Vinyoles, A. Serenelli, F. Villante, S. Basu, J. Redondo, and J. Isern, J. Cosmol. Astropart. Phys. 2015 (10), 015.
  * Aprile  _et al._ [2022] E. Aprile _et al._ (XENON Collaboration), Phys. Rev. D 106, 022001 (2022).
  * [27] B. T. McAllister, A. Quiskamp, C. A. J. O’Hare, P. Altin, E. N. Ivanov, M. Goryachev, and M. E. Tobar, Ann. der Physik , 2200622 (2023).
  * Foster  _et al._ [2018] J. W. Foster, N. L. Rodd, and B. R. Safdi, Phys. Rev. D 97, 123006 (2018).
  * Workman and Others [2022] R. L. Workman  _et al._ (Particle Data Group), PTEP 2022, 083C01 (2022).