# Are NeRFs ready for autonomous driving?
Towards closing the real-to-simulation gap

Carl Lindström†,1,2 Georg Hess†,1,2 Adam Lilja1,2
Maryam Fatemi1 Lars Hammarstrand2 Christoffer Petersson1,2 Lennart Svensson2
1Zenseact 2Chalmers University of Technology
<EMAIL_ADDRESS>chalmers.se}

###### Abstract

Neural Radiance Fields (NeRFs) have emerged as promising tools for advancing
autonomous driving (AD) research, offering scalable closed-loop simulation and
data augmentation capabilities. However, to trust the results achieved in
simulation, one needs to ensure that AD systems perceive real and rendered
data in the same way. Although the performance of rendering methods is
increasing, many scenarios will remain inherently challenging to reconstruct
faithfully. To this end, we propose a novel perspective for addressing the
real-to-simulated data gap. Rather than solely focusing on improving rendering
fidelity, we explore simple yet effective methods to enhance perception model
robustness to NeRF artifacts without compromising performance on real data.
Moreover, we conduct the first large-scale investigation into the real-to-
simulated data gap in an AD setting using a state-of-the-art neural rendering
technique. Specifically, we evaluate object detectors and an online mapping
model on real and simulated data, and study the effects of different pre-
training strategies. Our results show notable improvements in model robustness
to simulated data, even improving real-world performance in some cases. Last,
we delve into the correlation between the real-to-simulated gap and image
reconstruction metrics, identifying FID and LPIPS as strong indicators.

$\dagger$$\dagger$footnotetext: These authors contributed equally to this
work.

##  1 Introduction

The development of autonomous vehicles (AVs) requires substantial and accurate
testing to ensure safe behavior when deployed in the real world. In general,
this has required practitioners to collect vast amounts of real-world data.
Unfortunately, such collection is time-consuming and limits which safety-
critical scenarios can be explored as to not risking the safety of other road
users.

Figure 1: Using NeRFs for autonomous driving testing requires perception
models to treat rendered and real images similarly. A BEVFormer model trained
on real data detects objects (blue) in high-quality renderings (top). However,
when quality decreases (bottom), e.g., scenes challenging for the NeRF, the
same model fails to detect even close-by cars. Instead of emphasizing
rendering fidelity, we propose to make models robust to these distortions.
Fine-tuning the same model on NeRF-like images (red) reduces the real-to-sim
gap without harming real-world performance.

Neural rendering techniques, such as Neural Radiance Fields (NeRFs) [26] or
Gaussian Splatting [15], provide an attractive alternative, as they can be
used to simulate new scenarios in already collected data. Consequently, they
enable practitioners to explore system behavior, from pixel to torque, for
safety-critical scenarios that would be difficult to collect in the real
world. Multiple works have recently explored how to apply NeRFs to autonomous
driving (AD) data [44, 35, 43, 41, 48]. With constantly increasing rendering
quality, and decreasing computational demands, these methods can be expected
to provide scalable and cost-effective options for offloading real-world
testing.

Nevertheless, employing neural rendering techniques for AV testing gives rise
to a fundamental concern: How reliably can conclusions drawn from simulated
data be transferred to real data? To address this concern, it is essential to
assess whether the system, trained on real data, interprets real and simulated
data similarly, as highlighted in Fig. 1. This divergence, termed the
_real2sim_ gap [44], has received far less attention than its counterpart
_sim2real_ gap, which pertains to transferring knowledge acquired in
simulation to the real world [11]. Traditionally, the real2sim problem has
been addressed by improving the realism of rendered images. However, it is
unknown how well common novel view synthesis (NVS) performance measures, such
as PSNR, LPIPS [46], SSIM [38], or FID [10], correlate with a small real2sim
gap, making it difficult to state what quality a given NeRF must reach to be
useful for AV testing. Yet another aspect is that we are typically interested
in the rendering quality when deviating from the original trajectory, _i.e_.,
in a setting where such metrics cannot be calculated due to the lack of ground
truth data to compare with.

In this paper, we propose a novel perspective on reducing the gap between real
and simulated data for different perception modules of an autonomous system.
Rather than improving upon the rendering quality, we aim to make the
perception models more robust to NeRF artifacts without degrading performance
on real data. We believe this direction to be complementary to increasing NeRF
performance, and a potential key for making scalable, virtual AV testing a
reality. As a first step in this direction, we show that even simple data
augmentation techniques can have a large effect on model robustness against
NeRF artifacts.

Further, we perform the first extensive real2sim gap study on a large-scale AD
dataset [3] and evaluate the performance of multiple object detectors [36, 24,
19] alongside an online mapping model [21] on both real data and data from a
state-of-the-art (SOTA) neural rendering method [35]. Our investigation
encompasses the impact of diverse data augmentation techniques during
training, as well as the fidelity of NeRF renderings during inference. We find
that integrating such data during model fine-tuning notably enhances their
robustness to simulated data, and in some cases, even elevates performance on
real data. Lastly, we investigate the correlation between the real2sim-gap and
common image reconstruction metrics to provide insights into what matters for
applying NeRFs as simulators for AD data. We find LPIPS and FID to be strong
indicators of the real2sim-gap, and further confirm that our proposed
augmentations reduce the sensitivity to poor view synthesis.

##  2 Related work

NeRFs for autonomous driving: NeRFs have emerged as a promising approach for
simulating AD data. In contrast to game engine-based methods, NeRFs remove the
need for manual asset creation and are optimized to create sensor-realistic
renderings by design. However, a key challenge for NeRFs is handling the scale
and dynamics of automotive scenes. Neural Scene Graphs [30], Panoptic Neural
Fields [17] and Panoptic NeRF [7] separate the background from moving actors
by modeling each component with a separate, rigid, multi-layer perceptron
(MLP). Still, these methods struggle with scaling to large scenes due to the
limited expressiveness of the MLP. S-NeRF [43] addresses this by building upon
Mip-NeRF 360 [2] to better handle unbounded scenes. However, its long training
time makes it impractical to simulate many scenes. MARS [41] and UniSim [44]
utilize the hash-grid representation from iNGP [27] and achieve efficient
models, although with limitations on sensor configuration. NeuRAD [35]
introduces efficient ways of modeling the important aspects of AD data, and
achieves state-of-the-art performance across five AD datasets [3, 39, 8, 42,
1]. Furthermore, both UniSim and NeuRAD make efforts to tailor their
evaluation to closed-loop AD simulation. Nonetheless, their testing sets are
small in the context of AD perception, and their applicability to down-stream
tasks on a larger scale is unexplored.

Perception for autonomous driving: Perception in 3D is a critical component of
many autonomous driving solutions. Due to cameras’ low cost and high
availability, camera-based methods have been the subject of extensive research
in recent years. For 3D object detection, FCOS3D [36] is a one-stage monocular
object detector, building upon the 2D object detector FCOS [34], but
regressing targets in 3D rather than 2D. PETR [24] further supports the multi-
view setting and instead builds upon the decoder architecture from DETR [4],
adding encoding points in the camera frustum into the image features.
BEVFormer [19] also adopts a query-based architecture for multi-view input but
encodes features into a bird’s eye view (BEV) representation instead.

In addition to detecting objects in 3D, many autonomous vehicles also estimate
the road elements of their surroundings, also known as online mapping [18, 20,
25, 21, 45]. MapTRv2 [21] is a current SOTA method which uses a vector-based
representation for detected road elements such as lane dividers and road
boundaries. Similarly to BEVFormer, MapTRv2 encodes image features and lifts
them into a BEV representation. The objects’ class and geometry are estimated
through a DETR-like [4] transformer decoder.

Domain adaptation and multi-task learning: Domain adaptation (DA) is a field
aiming to cope with issues arising when the training and evaluation data come
from different distributions [6, 23]. Although different DA definitions exist,
unsupervised DA is the most commonly studied version and assumes no access to
labels in the target domain [40]. When labels exist in both the source and
target domain, the problem can be categorized both as supervised domain
adaptation [40] and multi-task learning [31, 47]. Multiple works study how to
learn different tasks, such as semantic segmentation and depth estimation,
simultaneously [29, 16, 13]. Our setting, however, differs from both these
aspects, as the labels for the source and target domains are identical. Before
neural rendering techniques, these situations rarely arose, and therefore have
only been scarcely studied [44].

Figure 2: Overview of our data pipeline for fine-tuning (top) and evaluating
(bottom) perception models. We explore three different augmentation methods
for fine-tuning.

(a) Original

(b) Image Augmentation

(c) NeRF

(d) Img2Img

Figure 3: Examples of our different data augmentation strategies to make
perception models more robust.

##  3 Method

Testing and validation of AD functionality can be greatly accelerated by NeRF-
fueled simulation engines, as they can enable the exploration of new virtual
scenarios using already collected data. For such simulation results to be
reliable, however, the AD systems must behave the same way when exposed to
rendered and real data. Previously, this has been addressed by improving the
simulation by rendering more realistic sensor data. In this work, we propose
an alternative and complementary approach, where we instead adjust the AD
system to be less sensitive to the differences between real and simulated
data.

As a first step in this direction, we explore how different fine-tuning
strategies can make perception models more robust to artifacts in the rendered
data. Specifically, given already trained models, we fine-tune the perception
models using images designed to improve performance on rendered images while
maintaining performance on real data, see Fig. 2. Besides reducing the
real2sim gap, this can potentially lower requirements on sensor-realism,
opening for a wider applicability of neural rendering methods, and lessening
computational needs for training and evaluating of the said methods. Note
that, while we focus on perception models, our methodology can easily be
extended to end-to-end models as well [12, 14].

Last, we acknowledge that one can imagine multiple ways to achieve the goal of
making models more robust, for instance by drawing inspiration from domain
adaptation [23, 6] and multi-task learning [47] literature. However, fine-
tuning requires minimal model-specific adjustments, allowing us to study a
range of models easily.

###  3.1 Image augmentations

A classic strategy to obtain increased robustness to artifacts is to use image
augmentations [32, 28]. Here, we select augmentations to represent various
distortions present in rendered images. More specifically, we add random
Gaussian noise, convolve the image with a Gaussian blur kernel, apply
photometric distortions similar to the ones found in SimCLR [5], and, finally,
downsample and upsample the image. The augmentations are applied sequentially,
each with some probability. Details on hyperparameters can be found in the
supplementary material, Sec. A.1.

###  3.2 Fine-tuning with mixed-in rendered images

Another natural way to adapt perception models to NeRF-rendered data is to
include such data during fine-tuning. This involves training a NeRF method on
the same dataset used to supervise perception models
$\mathcal{D}_{\text{train}}^{\text{real}}$. However, training NeRFs on all of
$\mathcal{D}_{\text{train}}^{\text{real}}$ can be prohibitively expensive for
large datasets. Instead, we train NeRFs on a subset
$\mathcal{D}^{\text{nerf}}\subset\mathcal{D}_{\text{train}}^{\text{real}}$.
Note that in addition to annotations for the given perception task, NeRFs for
AD typically add the requirement of data in $\mathcal{D}^{\text{nerf}}$ to be
sequential, where some additionally require labels for tasks such as 3D object
detection [44, 35], semantic segmentation [17], or multiple types of labels
[41].

Next, we divide the images for the selected sequences in
$\mathcal{D}^{\text{nerf}}$ into NeRF training
$\mathcal{D}^{\text{nerf}}_{\text{train}}$ and holdout
$\mathcal{D}^{\text{nerf}}_{\text{fine-tune}}$ sets. Fine-tuning of the
perception models is done on their entire training dataset
$\mathcal{D}_{\text{train}}^{\text{real}}$, and for images that have a
rendered correspondence in $\mathcal{D}^{\text{nerf}}_{\text{fine-tune}}$, we
use the rendered images with probability $p$. This implies that the images
utilized for fine-tuning have not been seen by the NeRF model.

###  3.3 Image-to-image translation

As mentioned previously, rendering NeRF data is an expensive data augmentation
technique. Furthermore, it requires sequential data and potentially additional
labeling beyond what is needed for the perception task. That is, to obtain a
scalable method we would ideally like an efficient strategy to obtain NeRF
data for single images. To this end, we propose to learn to generate NeRF-like
images using an image-to-image method. Given a real image, the model
translates the image to the NeRF domain, effectively introducing artifacts
typical for NeRFs. This enables us to vastly increase the amount of NeRF-like
images during fine-tuning at a limited computational cost. We train the image-
to-image model [37] using rendered images
$\mathcal{D}^{\text{nerf}}_{\text{fine-tune}}$ and their corresponding real
images. See Fig. 3 for visual examples of the different augmentation
strategies.

##  4 Results

In this section, we first describe our experimental setting. We then show how
the real2sim gap differs for different perception models, tasks, and
augmentation strategies. Next, we study how the models with the smallest gap
perceive renderings with large changes in viewpoint where no corresponding
real images have been collected. Finally, we show the correlation between
common metrics for novel view synthesis and the real2sim gap.

###  4.1 Experimental setup

We assess the real2sim gap by comparing the performance of multiple perception
models when applied to real data versus their NeRF-rendered counterparts. We
validate our approach to improving the robustness of perception models by
implementing various augmentation strategies during fine-tuning of the models,
as detailed in Sec. 3. Striving for a general approach adaptable to diverse
models and tasks, we employ the same augmentation techniques across all models
and tasks. Given the critical need to not sacrifice real-world performance, we
also assess our augmentation methods on real data. Specifically, we test the
three augmentation techniques described in Secs. 3.1, 3.2 and 3.3; traditional
image augmentations, rendered data, and image-to-image translation. See Sec.
A.2 respectively Sec. A.4 for details regarding the datasets used during fine-
tuning, and Sec. A.3 for details on training the image-to-image model.

Neural rendering: We opt to employ NeuRAD [35], the current SOTA method for
NVS on AD data, as our NeRF-method. NeuRAD relies on a hashgrid representation
[27] and upsampling techniques for efficient learning and inference of high-
resolution data. To handle dynamic scenes, it models all actors to be rigid
entities, which limits expressiveness for deformable actors such as
pedestrians. Nonetheless, NeuRAD is currently the most performant NVS
technique across multiple AD datasets. For the perception validation
sequences, we train NeuRAD on all images except those held out for evaluating
the perception models. For reference, statistics for the NeuRAD trainings are
shown in Tab. 8, where we observe similar performance on images used for data
augmentation as the ones used for evaluating the real2sim gap.

Dataset and tasks: Our evaluations are conducted on nuScenes, a widely
recognized research ground for perception models. Compared to other large AD
datasets such as Waymo Open Dataset [33], Argoverse2 [39] or Zenseact Open
Dataset [1], nuScenes uses lower resolution cameras and lidars, posing an
interesting challenge for neural rendering methods. We evaluate all perception
models on a subset of the official validation split, namely scenes collected
at daytime and without heavy rain. This is because lens flares and water spray
cannot currently be handled by any neural rendering method for AD data. This
filtering results in 111 scenes used for evaluation, see Sec. B.1 for details.

Further, we evaluate two tasks: 3D object detection and online mapping. These
tasks consider complementary aspects of the real2sim gap, as the former
focuses on foreground objects, while the latter targets the static background.
Both tasks and corresponding models are described in more detail below.

3D object detection: We evaluate the gap across three camera-only 3D object
detection (3DOD) models, chosen to represent different aspects of prevalent
model architectures. Namely, we apply FCOS3D [36], a fully-convolutional
monocular detector, PETR [24], a multi-view and 3D adaptation of DETR [4], and
BEVFormer [19] a multi-view detector centered around the bird’s-eye-view
representation. We follow the evaluation protocol established by the nuScenes
object detection task and report mean Average Precision (mAP) and nuScenes
Detection Score (NDS) on both real and rendered data. Additionally, we analyze
the consistency between detections made on real and rendered data. This way,
we do not only consider performance in absolute terms, but also measure if the
model makes the same mistakes on both types of data. To this end, we compute
NDS twice, with a distance threshold of $2$$\mathrm{m}$, each time treating
the other set of detections as ground truth. We average the results from both
evaluations to get our detection agreement (DA). All models are initialized
from weights pre-trained on nuScenes and fine-tuned for a fixed number of
gradient steps. See Sec. B.2 for more details on model weights and
hyperparameters used for fine-tuning.

Online mapping: To extend our evaluations beyond the 3D object detection task,
we also evaluate MapTRv2 [21] on the task of online mapping. Following the
evaluation framework outlined in [22], we compute mAP for the classes
“divider”, “boundary”, and “crossing”. We also compute detection agreement in
the same fashion as for 3DOD, alternating which set of detections is used as
ground truth.

For completeness, we use both the original validation split (same as for 3DOD)
and the geographically disjoint split recently proposed in [22]. In short, the
original split suffers from data leakage, as there is significant geographical
overlap between training and validation/testing samples. As an effect,
generalization performance is largely overestimated when using the original
split. Note that for the geographically disjoint split, we again remove scenes
at night or with rain, resulting in 154 scenes used for evaluation. See Sec.
B.1 for details.

###  4.2 Real2sim gap on interpolated views

We begin with studying the gap on interpolated views. These viewpoints lie in
between images used to supervised NeuRAD and have corresponding real images.
While these views arguably are easier to render than, for instance, shifts in
the ego-vehicle position, they allow a direct comparison between real and
simulated data. The real2sim gaps for 3DOD and online mapping models are
reported in Tab. 1, and discussed in detail below. For all metrics, the gap is
expressed as the relative performance drop compared to the real-world
performance of the model without augmentations.

Table 1: Real2sim results for the 3D object detection models and the online mapping method MapTRv2, fine-tuned with different strategies. ”Sim“ indicates that the model was evaluated on rendered data from NeuRAD. For online mapping, Original and Geographically Disjoint (Geogr.) refers to the splits used for training and evaluation.  Fine-tuning method |  Evaluation data | 3D object detection | Online mapping
---|---|---|---
FCOS3D | PETR | BEVFormer | Original | Geogr.
mAP | NDS | DA | mAP | NDS | DA | mAP | NDS | DA | mAP | DA | mAP | DA
Real data only | Real | $32.2$ | $39.8$ | $100.0$ | $38.6$ | $43.1$ | $100.0$ | $38.4$ | $48.5$ | $100.0$ | $64.5$ | $100.0$ | $26.6$ | $100.0$
Sim | $13.5$ | $28.8$ | $46.3$ | $20.2$ | $31.6$ | $55.4$ | $29.1$ | $42.7$ | $76.6$ | $54.0$ | $71.8$ | $23.2$ | $67.1$
Gap (%) | $58.1$ | $27.6$ | $53.7$ | $47.7$ | $26.7$ | $44.6$ | $24.2$ | $12.0$ | $23.4$ | $16.3$ | $28.2$ | $12.8$ | $32.9$
Image augmentations | Real | $32.5$ | $40.0$ | $100.0$ | $34.0$ | $38.9$ | $100.0$ | $38.9$ | $48.6$ | $100.0$ | $65.1$ | $100.0$ | $26.5$ | $100.0$
Sim | $13.5$ | $28.9$ | $46.5$ | $20.4$ | $30.0$ | $57.6$ | $31.0$ | $44.0$ | $77.6$ | $55.2$ | $72.3$ | $23.8$ | $70.1$
Gap (%) | $58.1$ | $27.4$ | $53.5$ | $47.2$ | $30.4$ | $42.4$ | $19.3$ | $9.3$ | $22.4$ | $14.4$ | $27.7$ | $10.5$ | $29.9$
NeRF | Real | $31.2$ | $38.6$ | $100.0$ | $35.1$ | $40.0$ | $100.0$ | $38.5$ | $48.3$ | $100.0$ | $64.9$ | $100.0$ | $26.9$ | $100.0$
Sim | $23.5$ | $33.6$ | $58.7$ | $29.3$ | $37.3$ | $70.7$ | $31.7$ | $44.5$ | $78.9$ | $56.0$ | $74.5$ | $24.6$ | $70.5$
Gap (%) | $27.0$ | $15.6$ | $41.3$ | $24.1$ | $13.5$ | $29.3$ | $17.4$ | $8.2$ | $21.1$ | $13.2$ | $25.5$ | $7.5$ | $29.5$
Image-to-image | Real | $32.5$ | $39.8$ | $100.0$ | $31.4$ | $37.2$ | $100.0$ | $37.5$ | $48.1$ | $100.0$ | $62.5$ | $100.0$ | $25.2$ | $100.0$
Sim | $24.5$ | $34.3$ | $57.3$ | $26.1$ | $35.1$ | $67.9$ | $33.0$ | $44.9$ | $80.7$ | $56.8$ | $74.8$ | $24.3$ | $73.9$
Gap (%) | $23.9$ | $13.8$ | $42.7$ | $32.4$ | $18.6$ | $32.1$ | $14.1$ | $7.4$ | $19.3$ | $11.9$ | $25.2$ | $8.3$ | $26.1$

Gap for models without fine-tuning: Despite leveraging the current SOTA in NVS
for AD data, Tab. 1 shows a significant real2sim gap across all models and
tasks. Notably, there is a considerable variation in the gap among different
3DOD models, with BEVFormer exhibiting the smallest gap, whereas the mAP
performance of FCOS3D is more than halved. Further, we observe a greater gap
for mAP than for NDS. NDS is a weighted score where half of it consists of
mAP, while the other half measures errors in terms of translation, scale,
orientation, velocity, and attribute for true positives only. Considering
this, the gap mainly stems from spurious or missing detection, while the
quality of the true positives in terms of scale, orientation, etc., is less
affected. For the online mapping task, we can see a smaller gap than for the
3DOD models, which is natural since NeuRAD often renders the static parts of a
scene more accurately than the dynamic parts [35].

Image augmentations: The efficacy of basic image augmentations varies among
the different models. Both BEVFormer and MapTRv2 demonstrate enhancements on
simulated data while maintaining or improving performance on real data.
However, FCOS3D exhibits minimal to no improvement on simulated data, despite
enhancing performance on real data. In contrast, PETR displays a larger gap in
terms of NDS, along with a significant decline in performance on real data.
Analyzing the detection agreement shows improved consistency across the board,
albeit with relatively modest improvements for FCOS3D.

Rendered data: Incorporating rendered images during fine-tuning decreases the
gap across all models. FCOS3D and PETR demonstrate substantial improvements of
$74.1$% respectively $45.5$% in mAP on simulated data, with slight
degradations on real data. Additionally, the models fine-tuned on rendered
data show significant improvements in terms of detection agreement, indicating
a higher consistency to the real-world detections.

Image-to-image translation: Finally, fine-tuning with image-to-image
translated images leads to a significant increase in performance on simulated
data across all models. It is notable that this artificial extension with
NeRF-like data performs better than using the actual NeRF-data for multiple
methods. However, for most methods, this also comes with some penalty in mAP
performance on real data.

Detection agreement across different distances: To further gain insights into
the consistency of detections relative to the detection distance, we assess
the detection agreement across various fractions of the evaluation range used
in the nuScenes protocol. Specifically, we examine the detection agreement for
our 3D object detection models, with our different augmentations, across
evaluation ranges ranging from $10$% to $100$% of the official protocol’s
evaluation range, which is 30–50m depending on the class. The results,
illustrated in Fig. 4, reveal a decrease in detection agreement as evaluation
distances increase, which aligns with the anticipated degradation in detection
performance over longer distances. Interestingly, the fine-tunings with NeRF
and image-to-image translated data notably reduce this effect for FCOS3D and
PETR, as evidenced by an increasing disparity relative to the other
augmentations.

(a) FCOS3D.

(b) PETR.

(c) BEVFormer.

Figure 4: Detection agreement vs. fraction of the evaluation range, evaluated
for the 3DOD models with different fine-tuning methods.

###  4.3 Real2sim gap on extrapolated views

To apply a NeRF in, for instance, closed-loop simulation or data generation,
it needs to produce meaningful images not only for interpolated views but even
more so when extrapolating to views away from the original trajectory. To
address this, we render images when laterally shifting or rotating the ego-
vehicle, effectively simulating scenarios such as the vehicle performing a
lane change or having departed from its original lane. We evaluate the
detection agreement and perception performance by adjusting the 3D labels and
detections to accommodate the specified shift or rotation. For clarity, we
only evaluate models pre-trained with the image-to-image augmentation, as
these displayed the best robustness in Tab. 1.

Compared to the interpolation setting, the extrapolation comes with unique
challenges for evaluating the real2sim gap. First, the absence of real images
in these novel scenarios prevents a comprehensive analysis of the true
disparity. Instead, the field has traditionally relied on FID [9] as a
performance measure on extrapolated camera views. Second, there are no
assurances that the perception model would produce identical detections from
the altered viewpoint, even with real data. By shifting the ego-vehicle, a
scenario can become more challenging, _e.g_., by introducing partial
occlusions, or the scenario can simply be less common in the collected data,
_e.g_., images during a lane-shift. Thus, it is hard to completely disentangle
these effects from the rendering quality.

Following previous work [35, 44] we render views when the ego vehicle has been
moved $\pm 1$ and $\pm 2$ meters laterally. To ensure that the shifted views
remain reasonable, _e.g_. not inside other road users or structures, we select
a smaller subset of scenes and manually validate the shifts’ feasibility. See
Sec. B.1 for further details. The FID score and perception performance on
lateral shifts can be seen in Tab. 2 and Tab. 3 for 3DOD and online mapping,
respectively. While the performance of all object detection methods drops as
the shift increases, the ranking among the methods persists. BEVFormer is the
most robust, with NDS-score dropping only $5$ points from shifting the input
data $2$m from the original position. For the online mapping method MapTRv2,
the drops are relatively large. Even shifting the ego position $1$m yields a
$13$% and $18$% drop on original and geographically disjoint splits,
respectively. This discrepancy is surprising since the input data is the same
as for BEVFormer and the architectures are fairly similar. By inspecting
evaluation samples, we see that the model struggles with predicting rare
training events, _e.g_., the vehicle traveling slightly outside the road as
Fig. 5 exemplifies.

For the rotation, the cameras are rotated around the ego-vehicle reference
frame at discrete angles $\pm 5$°, $\pm 15$°, $\pm 30$°, $\pm 90$° and $180$°.
As the resulting camera positions are expected to remain within, or close to,
the ego vehicle’s original extent, we here use the same validation sets as in
Sec. 4. Upon inspecting the renderings, they do not deteriorate noticeably
with increasing rotation angle. In Tab. 4 we see similar behavior as for the
lateral shift. The online mapping performance under these rotations
deteriorates for each angle as we rotate further from the original pose for
both the original and geographically disjoint splits.

Although the image quality degrades for larger rotations as indicated by the
FID scores, we find the mapping performance to be overly sensitive to these
viewpoint changes, as shown in Fig. 6. We theorize that the drop in
performance also stems from these scenarios being very rare in the training
data. For instance, lane changes with harsh attack angles towards a lane
marker are scarce compared to in-lane driving.

To this end, we fine-tune MapTRv2 with simulated rotated views of all angles
on the training data inserted into the full training set. As Tab. 4 depicts,
the performance on rotated views can be improved substantially by
incorporating simulated such scenarios also during training. For instance, the
performance using the geographically disjoint split improves from $5.2$ to
$17.3$ mAP on novel views perpendicular to the original poses. This is also
reflected in the predictions visualized in Fig. 6. Further, it is notable that
the performance on real data is improved from $26.6$ to $27.5$ mAP. This
indicates that utilizing NeRF-rendered data also for training could be
beneficial and that all performance gap is not attributed to the quality of
the renderings. However, it is important to stress that disentangling how much
of the performance gap stems from image quality or scenarios being outside the
training distribution remains challenging.

Figure 5: Online mapping predictions and ground truth for images shifted to
the left (top), real images (middle), and rendered images without shift
(bottom). When input data is shifted 2m to the left, the left road boundary,
highlighted in red, should be straddled by the ego vehicle. However, the
predictions maintain the ego vehicle within the boundary despite the image
shift.  Table 2: Results for 3D object detection models fine-tuned with image-
to-image translated data on laterally shifted views.

Data |  Shift (m) | FID$\downarrow$ | FCOS3D | PETR | BEVFormer
---|---|---|---|---|---
mAP | NDS | DA | mAP | NDS | DA | mAP | NDS | DA
Real | $0$ | - | $30.9$ | $38.4$ | $100.0$ | $35.5$ | $40.1$ | $100.0$ | $35.6$ | $45.6$ | $100.0$
Sim | $0$ | $52.0$ | $23.3$ | $33.1$ | $61.6$ | $27.5$ | $35.0$ | $71.3$ | $32.3$ | $43.7$ | $77.4$
Sim | $\pm 1$ | $77.0$ | $19.9$ | $30.5$ | $42.6$ | $25.4$ | $33.3$ | $63.5$ | $29.6$ | $41.6$ | $68.3$
Sim | $\pm 2$ | $95.7$ | $16.5$ | $28.8$ | $39.7$ | $22.2$ | $31.1$ | $53.8$ | $26.9$ | $39.5$ | $59.5$

Table 3: mAP results for MapTRv2 fine-tuned with image-to-image translated data evaluated on laterally shifted camera views. The performance deteriorates as the lateral shift increases for both the original and geographically disjoint (Geogr. splits). Data | Shift (m) |  FID$\downarrow$ | Original | Geogr.
---|---|---|---|---
Real | $0$ | - | $60.6$ | $26.9$
Sim | $0$ | $52.0$ | $59.2$ | $25.7$
Sim | $\pm 1$ | $77.0$ | $51.5$ | $21.2$
Sim | $\pm 2$ | $95.7$ | $39.6$ | $19.2$
Table 4: mAP performances for the different training methods on MapTRv2. Performance of rotated novel views can be improved substantially by injecting training data with simulated rotations. | Finetuning | Real | Sim
---|---|---|---
| $0^{\circ}$ | $0^{\circ}$ | $\pm 5^{\circ}$ | $\pm 15^{\circ}$ | $\pm 30^{\circ}$ | $\pm 90^{\circ}$ | $180^{\circ}$
Original |  FID$\downarrow$ | - | $58.4$ | $67.9$ | $83.8$ | $99.7$ | $126.3$ | $112.3$
Img2Img | $62.5$ | $56.8$ | $53.9$ | $35.4$ | $19.3$ | $6.4$ | $19.5$
| NeRF | $64.9$ | $56.0$ | $53.1$ | $35.3$ | $19.3$ | $6.8$ | $19.1$
| NeRF+Rot | $64.8$ | $56.1$ | $57.3$ | $51.5$ | $43.3$ | $35.6$ | $36.0$
Geogr. |  FID$\downarrow$ | - | $54.1$ | $68.6$ | $83.5$ | $99.7$ | $126.8$ | $113.3$
Img2Img | $25.2$ | $24.3$ | $22.2$ | $17.4$ | $11.9$ | $5.8$ | $12.8$
| NeRF | $26.9$ | $24.6$ | $22.6$ | $17.3$ | $11.4$ | $5.2$ | $12.1$
| NeRF+Rot | $27.5$ | $25.5$ | $24.1$ | $21.7$ | $18.4$ | $17.3$ | $16.4$
Figure 6: Online mapping predictions and ground truth for rotated input
images, ccw denotes counter-clockwise. The predictions are greatly improved by
injecting rotated scenarios during fine-tuning of the model.

###  4.4 Real2sim gap correlation to image metrics

The real2sim gap has traditionally been addressed by improving the quality of
rendered images, commonly assessed using PSNR, SSIM, LPIPS and FID under the
assumption that improving these metrics reduces the gap. To test this
assumption, we examine the correlation between common NVS metrics and our
detection agreement. Specifically, we compute PSNR, SSIM, LPIPS and FID scores
for the rendered images, and detection agreement for different augmentations
applied during fine-tuning of BEVFormer. Subsequently, we aggregate the
results per scene and analyze the correlation between the aggregated data
points. For the model fine-tuned with our most promising method, image-to-
image, we also include FID and corresponding detection agreement for the
shifted scenes from Sec. 4.3, enabling us to measure the correlation for the
3DOD model and NeRF in a more practical and useful setting. Our findings,
illustrated in Fig. 7 for each NVS metric and divided by augmentation, and in
Fig. 9 in the supplementary material for the FID score on shifted scenes,
reveal a clear correlation to detection agreement across all metrics. Notably,
LPIPS and FID exhibit the strongest correlation and fewest outliers,
indicating that perceptual similarity matters more to the perception model
than mere reconstruction quality. Moreover, our results indicate that in the
absence of our proposed augmentations, the model becomes considerably more
sensitive to low-quality images. Consequently, our results show that FID can
be a useful indicator in the extrapolated setting where the other metrics are
not applicable due to the lack of ground truth, _e.g_., to understand how
large an extrapolation can be performed for a given requirement on detection
agreement.

(a) PSNR$\uparrow$

(b) SSIM$\uparrow$

(c) LPIPS$\downarrow$

(d) FID$\downarrow$

Figure 7: Detection agreement vs. novel view synthesis metrics for BEVFormer
fine-tuned with different augmentations.

##  5 Conclusion

Neural Radiance Fields (NeRFs) have emerged as a promising avenue for
simulating autonomous driving (AD) data. However, to be practically useful,
one must understand how the behavior of an AD system performed on simulated
data transfers to real data. Our large-scale investigations reveal a
performance gap between perception models exposed to simulated and real
images.

In contrast to earlier approaches, which concentrate on improving the
rendering quality, this paper investigates how perception models can be
altered to be more robust to NeRF simulated data. We show that fine-tuning
with NeRF, or NeRF-like, data substantially reduces the real2sim gap for both
object detection and online mapping methods without sacrificing performance on
real data. Furthermore, we show that generating new scenarios outside the
existing training distribution, _e.g_. simulating lane departures, can improve
the performance on real data. Studies on commonly used image metrics within
the NeRF community reveal that LPIPS and FID scores exhibits the strongest
correlation with perception performance. This suggests that perceptual
similarity holds greater significance for the perception model compared to
mere reconstruction quality.

In summary, we argue that NeRF-simulated data is valuable for AD, especially
when using our proposed approach to enhancing the perception models’
robustness. Further, NeRF-data is not only useful for testing AD systems on
simulated data, but also for improving the performance of perception models on
real data.

### Acknowledegments

We thank Adam Tonderski and William Ljungbergh for valuable discussions. This
work was partially supported by the Wallenberg AI, Autonomous Systems and
Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.
Computational resources were provided by the National Academic Infrastructure
for Supercomputing in Sweden (NAISS) at NSC Berzelius, partially funded by the
Swedish Research Council, grant agreement no. 2022-06725.

## References

  * Alibeigi et al. [2023] Mina Alibeigi, William Ljungbergh, Adam Tonderski, Georg Hess, Adam Lilja, Carl Lindström, Daria Motorniuk, Junsheng Fu, Jenny Widahl, and Christoffer Petersson.  Zenseact open dataset: A large-scale and diverse multimodal dataset for autonomous driving.  In _ICCV_ , pages 20178–20188, 2023.
  * Barron et al. [2022] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In _CVPR_ , pages 5470–5479, 2022.
  * Caesar et al. [2020] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In _CVPR_ , pages 11621–11631, 2020.
  * Carion et al. [2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _European conference on computer vision_ , pages 213–229. Springer, 2020.
  * Chen et al. [2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR, 2020.
  * Farahani et al. [2021] Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R Arabnia.  A brief review of domain adaptation.  _Advances in data science and information engineering: proceedings from ICDATA 2020 and IKE 2020_ , pages 877–894, 2021.
  * Fu et al. [2022] Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou, Andreas Geiger, and Yiyi Liao.  Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmentation.  In _2022 International Conference on 3D Vision (3DV)_ , pages 1–11. IEEE, 2022.
  * Geiger et al. [2013] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  _The International Journal of Robotics Research_ , 32(11):1231–1237, 2013.
  * Heusel et al. [2017a] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2017a.
  * Heusel et al. [2017b] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017b.
  * Hu et al. [2023a] Xuemin Hu, Shen Li, Tingyu Huang, Bo Tang, Rouxing Huai, and Long Chen.  How simulation helps autonomous driving: A survey of sim2real, digital twins, and parallel intelligence.  _IEEE Transactions on Intelligent Vehicles_ , 2023a.
  * Hu et al. [2023b] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al.  Planning-oriented autonomous driving.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 17853–17862, 2023b.
  * Ishihara et al. [2021] Keishi Ishihara, Anssi Kanervisto, Jun Miura, and Ville Hautamaki.  Multi-task learning with attention for end-to-end autonomous driving.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , pages 2902–2911, 2021.
  * Jiang et al. [2023] Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang.  Vad: Vectorized scene representation for efficient autonomous driving.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 8340–8350, 2023.
  * Kerbl et al. [2023] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.  3d gaussian splatting for real-time radiance field rendering.  _ACM Transactions on Graphics_ , 42(4), 2023.
  * Khattar et al. [2021] Apoorv Khattar, Srinidhi Hegde, and Ramya Hebbalaguppe.  Cross-domain multi-task learning for object detection and saliency estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3639–3648, 2021.
  * Kundu et al. [2022] Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas J Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser.  Panoptic neural fields: A semantic object-aware neural scene representation.  In _CVPR_ , pages 12871–12881, 2022.
  * Li et al. [2022a] Qi Li, Yue Wang, Yilun Wang, and Hang Zhao.  Hdmapnet: An online hd map construction and evaluation framework.  In _2022 International Conference on Robotics and Automation (ICRA)_ , pages 4628–4634. IEEE, 2022a.
  * Li et al. [2022b] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai.  Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers.  In _European conference on computer vision_ , pages 1–18. Springer, 2022b.
  * Liao et al. [2023a] Bencheng Liao, Shaoyu Chen, Xinggang Wang, Tianheng Cheng, Qian Zhang, Wenyu Liu, and Chang Huang.  MapTR: Structured modeling and learning for online vectorized HD map construction.  In _The Eleventh International Conference on Learning Representations_ , 2023a.
  * Liao et al. [2023b] Bencheng Liao, Shaoyu Chen, Yunchi Zhang, Bo Jiang, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang.  Maptrv2: An end-to-end framework for online vectorized hd map construction.  _arXiv preprint arXiv:2308.05736_ , 2023b.
  * Lilja et al. [2023] Adam Lilja, Junsheng Fu, Erik Stenborg, and Lars Hammarstrand.  Localization is all you evaluate: Data leakage in online mapping datasets and how to fix it.  _arXiv preprint arXiv:2312.06420_ , 2023.
  * Liu et al. [2022a] Xiaofeng Liu, Chaehwa Yoo, Fangxu Xing, Hyejin Oh, Georges El Fakhri, Je-Won Kang, Jonghye Woo, et al.  Deep unsupervised domain adaptation: A review of recent advances and perspectives.  _APSIPA Transactions on Signal and Information Processing_ , 11(1), 2022a.
  * Liu et al. [2022b] Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun.  Petr: Position embedding transformation for multi-view 3d object detection.  In _European Conference on Computer Vision_ , pages 531–548. Springer, 2022b.
  * Liu et al. [2023] Yicheng Liu, Tianyuan Yuan, Yue Wang, Yilun Wang, and Hang Zhao.  Vectormapnet: End-to-end vectorized hd map learning.  In _International Conference on Machine Learning_ , pages 22352–22369. PMLR, 2023.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _ECCV_ , 2020.
  * Müller et al. [2022] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM TOG_ , 41(4):1–15, 2022.
  * Murphy [2022] Kevin P Murphy.  _Probabilistic machine learning: an introduction_.  MIT press, 2022.
  * Navon et al. [2022] Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya.  Multi-task learning as a bargaining game.  _arXiv preprint arXiv:2202.01017_ , 2022.
  * Ost et al. [2021] Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide.  Neural scene graphs for dynamic scenes.  In _CVPR_ , pages 2856–2865, 2021.
  * Pan and Yang [2009] Sinno Jialin Pan and Qiang Yang.  A survey on transfer learning.  _IEEE Transactions on knowledge and data engineering_ , 22(10):1345–1359, 2009.
  * Simard et al. [2003] Patrice Y Simard, David Steinkraus, John C Platt, et al.  Best practices for convolutional neural networks applied to visual document analysis.  In _Icdar_. Edinburgh, 2003.
  * Sun et al. [2020] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al.  Scalability in perception for autonomous driving: Waymo open dataset.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 2446–2454, 2020.
  * Tian et al. [2020] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  Fcos: A simple and strong anchor-free object detector.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(4):1922–1933, 2020.
  * Tonderski et al. [2023] Adam Tonderski, Carl Lindström, Georg Hess, William Ljungbergh, Lennart Svensson, and Christoffer Petersson.  Neurad: Neural rendering for autonomous driving.  _arXiv preprint arXiv:2311.15260_ , 2023.
  * Wang et al. [2021] Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin.  Fcos3d: Fully convolutional one-stage monocular 3d object detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 913–922, 2021.
  * Wang et al. [2018] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8798–8807, 2018.
  * Wang et al. [2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ , 13(4):600–612, 2004.
  * Wilson et al. [2021] Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays.  Argoverse 2: Next generation datasets for self-driving perception and forecasting.  In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks 2021)_ , 2021.
  * Wilson and Cook [2020] Garrett Wilson and Diane J Cook.  A survey of unsupervised deep domain adaptation.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ , 11(5):1–46, 2020.
  * Wu et al. [2023] Zirui Wu, Tianyu Liu, Liyi Luo, Zhide Zhong, Jianteng Chen, Hongmin Xiao, Chao Hou, Haozhe Lou, Yuantao Chen, Runyi Yang, Yuxin Huang, Xiaoyu Ye, Zike Yan, Yongliang Shi, Yiyi Liao, and Hao Zhao.  Mars: An instance-aware, modular and realistic simulator for autonomous driving.  _CICAI_ , 2023.
  * Xiao et al. [2021] Pengchuan Xiao, Zhenlei Shao, Steven Hao, Zishuo Zhang, Xiaolin Chai, Judy Jiao, Zesong Li, Jian Wu, Kai Sun, Kun Jiang, Yunlong Wang, and Diange Yang.  Pandaset: Advanced sensor suite dataset for autonomous driving.  In _2021 IEEE International Intelligent Transportation Systems Conference (ITSC)_ , pages 3095–3101, 2021.
  * Xie et al. [2023] Ziyang Xie, Junge Zhang, Wenye Li, Feihu Zhang, and Li Zhang.  S-neRF: Neural radiance fields for street views.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Yang et al. [2023] Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun.  Unisim: A neural closed-loop sensor simulator.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1389–1399, 2023.
  * Yuan et al. [2024] Tianyuan Yuan, Yicheng Liu, Yue Wang, Yilun Wang, and Hang Zhao.  Streammapnet: Streaming mapping network for vectorized online hd map construction.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 7356–7365, 2024.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 586–595, 2018.
  * Zhang and Yang [2021] Yu Zhang and Qiang Yang.  A survey on multi-task learning.  _IEEE Transactions on Knowledge and Data Engineering_ , 34(12):5586–5609, 2021.
  * Zhou et al. [2023] Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang, Deqing Sun, and Ming-Hsuan Yang.  Drivinggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes.  _arXiv preprint arXiv:2312.07920_ , 2023.

Supplementary Material

##  Appendix A Training details

###  A.1 Image augmentation hyperparameters

Tab. 5 shows hyperparameter selection for the image augmentation method.

Table 5: Hyperparameters for image augmentation during fine-tuning. $p$’s are
referring to the probability of an augmentation being applied.

Parameter | Value
---|---
Gaussian noise $p$ | 0.5
Additive Gaussian noise $\delta$ | $\sim\mathcal{N}(0,10)$
Gaussian blur $p$ | 0.5
Gaussian blur kernel size | $5\times 5$
Down- and upsampling factor | $10$
Down- and upsampling method | Bilinear
Photometric, additive brightness $\delta$ | $\sim\mathcal{U}(-32,32)$
Photometric, multiplicative contrast $\delta$ | $\sim\mathcal{U}(0.5,1.5)$
Photometric, multiplicative saturation $\delta$ | $\sim\mathcal{U}(0.5,1.5)$
Photometric, additive hue $\delta$ | $\sim\mathcal{U}(-18,18)$

###  A.2 NeRF augmentation

To generate NeRF-rendered training images for the perception models, we train
NeuRAD [35] on a subset of the nuScenes [3] train set. We select all scenes
that were collected at daytime and do not have any rain, resulting in 491 out
of 750 scenes. Then we overlap this set with the geographical train split
proposed in [22], leaving us with 316 scenes. This way, we can use the NeRF
rendering both for the 3D object detection and online mapping task. Among the
316 scenes, we randomly select 110 scenes, namely 0402, 0323, 0252, 0048,
0419, 0856, 0949, 0769, 0435, 0812, 0284, 0394, 0673, 0250, 0288, 0006, 0400,
0736, 0264, 0527, 0359, 0290, 0990, 0256, 0234, 0731, 0300, 0439, 0244, 0698,
0525, 0122, 0075, 0254, 0055, 0163, 0740, 0978, 0712, 0544, 0976, 0021, 0292,
0848, 0792, 0066, 0405, 0200, 0675, 0260, 0375, 0542, 0710, 0988, 0242, 0294,
0381, 0165, 0685, 0157, 0053, 0388, 0286, 0304, 0507, 0298, 0706, 0665, 0790,
0218, 0190, 0034, 0687, 0421, 0671, 0032, 0236, 0505, 0854, 0726, 0044, 0351,
0384, 0805, 0539, 0203, 0407, 0373, 0246, 0361, 0767, 0139, 0194, 0701, 0058,
0230, 0228, 0716, 0392, 0437, 0302, 0060, 0192, 0655, 0240, 0128, 0296, 0787,
0206, 0679. The selected sequences result in a total of 26478 images,
constituting $15.7$% of the original training set.

###  A.3 Image-to-image training

We use the NeRF renderings outlined in Sec. A.2 to train a pix2pixHD model
[37]. The 110 scenes with six cameras result in 26478 training samples for the
pix2pixHD model. We use the official
implementation111https://github.com/NVIDIA/pix2pixHD and train the base model
for 80 epochs, followed by tuning at a higher resolution for 45 epochs.

###  A.4 Image-to-image augmentation

Using our image-to-image model, trained as outlined in Sec. A.3, we generate
images for all 750 scenes in the nuScenes training set.

##  Appendix B Experiment details

###  B.1 Evaluation scenes

To validate the perception models, we train NeuRAD [35] on multiple nuScenes
validation scenes and generate images for annotated frames. Note that these
frames are held out from the NeuRAD training. From the original 150 nuScenes
validation scenes, we select all scenes collected at day-time without rainy
weather, yielding 111 scenes, namely 0003, 0012, 0013, 0014, 0015, 0016, 0017,
0018, 0035, 0036, 0038, 0039, 0092, 0093, 0094, 0095, 0096, 0097, 0098, 0099,
0100, 0101, 0102, 0103, 0104, 0105, 0106, 0107, 0108, 0109, 0110, 0221, 0268,
0269, 0270, 0271, 0272, 0273, 0274, 0275, 0276, 0277, 0278, 0329, 0330, 0331,
0332, 0344, 0345, 0346, 0519, 0520, 0521, 0522, 0523, 0524, 0552, 0553, 0554,
0555, 0556, 0557, 0558, 0559, 0560, 0561, 0562, 0563, 0564, 0565, 0770, 0771,
0775, 0777, 0778, 0780, 0781, 0782, 0783, 0784, 0794, 0795, 0796, 0797, 0798,
0799, 0800, 0802, 0916, 0917, 0919, 0920, 0921, 0922, 0923, 0924, 0925, 0926,
0927, 0928, 0929, 0930, 0931, 0962, 0963, 0966, 0967, 0968, 0969, 0971, 0972.

For the geographical split [22], we use all scenes in the geographical
validation split that are collected at day-time without rain. Further, as
NeuRAD requires annotations for training, we remove all scenes without
annotations, resulting in 67 scenes. For more rigorous evaluation, we also
include all scenes from the geographical test split that have annotations and
were collected at day-time without rain. This adds another 87 scenes, totaling
in 154 scenes, namely 0002, 0019, 0043, 0046, 0061, 0151, 0158, 0159, 0348,
0355, 0356, 0357, 0358, 0377, 0385, 0945, 0947, 0981, 0982, 0983, 0018, 0036,
0268, 0275, 0276, 0344, 0345, 0411, 0182, 0183, 0315, 0423, 0424, 0425, 0860,
0861, 0862, 0863, 0864, 0925, 0926, 0927, 0928, 0071, 0170, 0171, 0172, 0173,
0174, 0175, 0209, 0210, 0211, 0212, 0500, 0501, 0518, 0660, 0661, 0662, 0663,
0664, 0738, 0821, 0109, 0331, 0523, 0007, 0008, 0009, 0024, 0025, 0026, 0027,
0028, 0029, 0030, 0042, 0050, 0057, 0123, 0124, 0154, 0155, 0364, 0365, 0370,
0379, 0380, 0383, 0952, 0953, 0955, 0956, 0957, 0958, 0959, 0960, 0016, 0966,
0413, 0414, 0415, 0416, 0417, 0184, 0185, 0187, 0188, 0316, 0427, 0428, 0429,
0430, 0858, 0919, 0920, 0921, 0924, 0069, 0073, 0176, 0207, 0208, 0213, 0263,
0396, 0397, 0398, 0509, 0528, 0529, 0530, 0531, 0532, 0533, 0534, 0535, 0536,
0658, 0744, 0746, 0747, 0749, 0750, 0751, 0752, 0757, 0758, 0759, 0760, 0817,
0110, 0330.

For the evaluations on laterally shifted views, we use a smaller subset of
scenes from our previously chosen 111 scenes. We select scenes on the criteria
that our lateral shifts do not result in the camera ending up inside other
road users or structures. This results in 14 scenes, namely 0523, 0924, 0921,
0928, 0268, 0919, 0109, 0926, 0018, 0344, 0345, 0016, 0276, 0925.

###  B.2 Fine-tuning of 3D object detection models

We start all fine-tunings from model weights pre-trained on nuScenes. For
FCOS3D and PETR, we utilize the implementations from the
mmdetection3d-framework222https://github.com/open-mmlab/mmdetection3d and use
the model weights and corresponding training configurations reported there.
For BEVFormer we use the official
implementation333https://github.com/fundamentalvision/BEVFormer and model
weights corresponding to the small version. See Tab. 6 for details on the
hyperparameters used for each fine-tuning.

Table 6: Hyperparameters used to fine-tune the 3D object detection models. Model | Augmentation | Learning rate | Epochs
---|---|---|---
FCOS3D | None | $2e-6$ | 6
Image aug. | $2e-5$ | 6
NeRF | $1e-4$ | 6
Img2Img | $1e-4$ | 6
PETR | None | $1e-8$ | 12
Image aug. | $2e-5$ | 12
NeRF | $2e-5$ | 12
Img2Img | $2e-5$ | 12
BEVFormer | None | $2e-5$ | 4
Image aug. | $4e-5$ | 4
NeRF | $4e-5$ | 4
Img2Img | $4e-5$ | 4

###  B.3 Fine-tuning of online mapping

Also for the online mapping method MapTRv2 we start from the pre-trained
weights for both the original and geographically disjoint splits. Tab. 7
reports the hyperparameters used for the different fine-tunings.

Table 7: Hyperparameters used to fine-tune the online mapping method MapTRv2. Model | Augmentation | Learning rate | Epochs
---|---|---|---
Original | None | $1e-5$ | 5
Image aug. | $1e-4$ | 5
NeRF | $1e-4$ | 5
Img2Img | $1e-4$ | 5
Geogr. | None | $1e-4$ | 5
Image aug. | $1e-4$ | 5
NeRF | $1e-4$ | 5
Img2Img | $1e-3$ | 5

###  B.4 NeuRAD results

In Tab. 8, we report standard novel view synthesis metrics for the different
data splits. We observe NeuRAD to perform similar for all data subsets, hence
expecting the artifacts in the images used for augmentation having similar
style as the ones used for evaluation.

Table 8: Novel view synthesis performance for NeuRAD on held-out images for the different splits. Split |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
---|---|---|---
Orig. val | 26.50 | 0.7893 | 0.2566
Orig. train | 26.52 | 0.7975 | 0.2456
Geo. val+test | 26.88 | 0.8009 | 0.2558

##  Appendix C Additional results

###  C.1 Real2sim 3DOD

We visualize the real2sim results on 3DOD models, reported in Tab. 1, as bar
plots in Fig. 8.

Figure 8: Bar plots of the real2sim gap for the 3DOD-models.

###  C.2 Correlation to FID for shifted scenes

We illustrate the correlation between detection agreement and FID, isolated
for only shifted sequences and divided by the shift amount and direction, in
Fig. 9. The detection agreement is computed for BEVFormer fine-tuned with
image-to-image translated images.

Figure 9: Detection agreement vs. FID scores for BEVFormer fine-tuned with
image-to-image translated images, evaluated on the lane shift evaluation set
with different shifts.


Holographic insulator/superconductor phase transition in Born-Infeld
electrodynamics

Nan Baia, Yi-Hong Gaoa, Bu-Guan Qia and Xiao-Bao Xua

a State Key Laboratory of Theoretical Physics,
Institute of Theoretical Physics,
Chinese Academy of Sciences, P.O. Box 2735, Beijing 100190, China

<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>and<EMAIL_ADDRESS>
We studied holographic insulator/superconductor phase transition in the
framework of Born-Infeld electrodynamics both numerically and analytically.
First we numerically study the effects of the Born-Infeld electrodynamics on
the phase transition, find that the critical chemical potential is not changed
by the Born-Infeld parameter. Then we employ the variational method for the
Sturm-Liouville eigenvalue problem to analytically study the phase transition.
The analytical results obtained are found to be consistent with the numerical
results.

##  1 Introduction

The AdS/CFT correspondence [1, 2, 3] which relates string theory on
asymptotically AdS spacetime to a conformal field theory on the boundary, has
been extensively applied in nuclear physics and condensed matter physics. In
particular the related developments of dual gravitational models which
describe insulator/superconductor phase transitions in condensed matter
physics have been proposed in [4, 5].

The Einstein-Maxwell theory coupled to a charged scalar field would be the
simplest model for holographic superconductor. In this model, there exists a
critical temperature $T_{c}$ [4], below which charged AdS black holes may have
non-zero scalar hair. In the context of AdS/CFT connection, a charged scalar
field is dual to an operator which carries the global $U(1)$ charge. Having
non-zero scalar hair indicates that the expectation value of dual operator is
non-zero which leads to the break of global $U(1)$ symmetry, thus can be
understood as a second order conductor/superconductor phase transition. The
behavior of AC conductivity in these phases [5] make the interpretation above
reliable.

There have been many discussions concerning holographic superconductor models
among which general relativity coupled to Maxwell field plus a charged scalar
field attracts most attention [6]-[18] and the related Lagrangian is

$\mathcal{L}=-\frac{1}{4}F^{2}-|\nabla_{\mu}\psi-
iqA_{\mu}\psi|^{2}-m^{2}|\psi|^{2},\qquad F=F_{\mu\nu}F^{\mu\nu}$ (1)

However,the non-linear extension of the original Maxwell electrodynamics to
Born-Infeld (BI) model [28] also caused intensive investigations [19]-[27],
since BI electrodynamics possesses many interesting physical properties as
finite total energy and the invariance under electromagnetic duality. In
string theory, BI action can also describe gauge fields which arise from
D-brane attached with open strings [29, 30]. The effects of BI electrodynamics
on the holographic superconductors in the background of Ads-Schwarzschild
black hole spacetime has been analyzed numerically in [22] and analytically in
[25].The results shows it is much more difficult to have scalar condensation
in BI electrodynamics which is similar to the one of holographic
superconductors in Einstein-Gauss-Bonnet gravity [8, 15], where the higher
curvature corrections make condensation harder.

The authors of [10] have constructed a model describing an
insulator/superconductor phase transition at zero temperature, using a five
dimensional AdS soliton in an Einstein-Maxwell-charged scalar field. The
generalization of holographic insulator/superconductor has been carried out
[31, 32, 33].The aim of the present article is to extend holographic
insulator/superconductor in the framework of Born-Infeld electrodynamics. The
main purpose here is to see what effect of Born-Infeld scale parameter on the
holographic insulator/superconductor. With both numerical and analytic method,
we find a bit unexpected result that the effect of Born-Infeld electrodynamics
is too small to be seen in our study, but it is consistent with [27], which
state that “the critical chemical potential is not changed by the coupling
parameter $b$”.

The paper is organized as follows. In section 2 we study holographic
insulator/super-conductor in the framework of Born-Infeld electrodynamics in
the probe limit. We numerically calculate the critical chemical potential
$\mu_{c}$, the condensations of the scalar operators $\langle\cal{O}\rangle$
and charge density $\rho$. We also study the conductivity of the theory. In
section 3 we reproduce the same result as the numerical one for the critical
chemical potential $\mu_{c}$ by the Sturm-Liouville analytical method. The
last section is devoted to conclusions.

##  2 Numerical analysis of the critical chemical potential $\mu_{c}$

In order to construct the holographic insulator/superconductor phase
transition, we consider the fixed background of five dimensional AdS soliton,
e.g. in the probe limit. The Ads soliton metric reads

$\displaystyle ds^{2}$
$\displaystyle=L^{2}\frac{dr^{2}}{f(r)}+r^{2}(-dt^{2}+dx^{2}+dy^{2})+f(r)d\chi^{2}\
,$ (2) $\displaystyle f(r)$ $\displaystyle=r^{2}-\frac{r_{0}^{4}}{r^{2}}$ (3)

where $L$ is the AdS radius and $r_{0}$ the tip of the soliton which is a
conical singularity in this solution. Essentially this solution can be
obtained from a five dimensional AdS black hole solution by making use of
double Wick rotations [10]. We can also set $L=1$ and $r_{0}$ = 1 without the
loss of generality.

We now consider a charged complex scalar field and an electric field in the
above background, we investigate Born-Infeld electrodynamics instead of
Maxwell’s therefore the corresponding Lagrangian density can be

$\displaystyle\mathcal{L}=\mathcal{L}_{BI}-|\nabla_{\mu}\psi-
iqA_{\mu}\psi|^{2}-m^{2}|\psi|^{2}$ (4)

where $\psi$ is a charged complex scalar field, $\mathcal{L}_{BI}$ is the
Lagrangian density of the Born-Infeld electrodynamics

$\displaystyle\mathcal{L}_{BI}=\frac{1}{b}\bigg{(}1-\sqrt{1+\frac{bF}{2}}\bigg{)}.\quad
F\equiv F_{\mu\nu}F^{\mu\nu}$ (5)

The BI coupling parameter $b$ describes the nonlinearity of electrodynamics.
The above BI lagrangian could be reduced to Maxwell lagrangian
$-\frac{1}{4}F_{\mu\nu}F^{\mu\nu}$ naturally in the limit $b\rightarrow 0$ .
We further assume that the BI coupling parameter $b$ is small,since the Ads
Soliton of background spacetimes is the solution of the Einstein equation in
Einstein-Maxwell-scalar field theory.

To study the insulator/superconductor phase transition we will consider the
following ansatz [10]

$\displaystyle A_{\mu}=(\phi(r),0,0,0),\;\;\;\;\psi=\psi(r)$ (6)

The equations of motion are

$\displaystyle\psi^{\prime\prime}+\left(\frac{f^{\prime}}{f}+\frac{3}{r}\right)\psi^{\prime}+\left(\frac{q^{2}\phi^{2}}{r^{2}f}-\frac{m^{2}}{f}\right)\psi=0$
(7)

and

$\displaystyle\phi^{\prime\prime}+\left(\frac{1}{r}+\frac{f^{\prime}}{f}\right)\phi^{\prime}-\left(\frac{bf^{\prime}}{2r^{2}}+\frac{2bf}{r^{3}}\right)\phi^{\prime
3}-\frac{2q^{2}\psi^{2}}{f}\bigg{(}1-\frac{bf}{r^{2}}\phi^{\prime
2}\bigg{)}^{3/2}\phi=0$ (8)

To seek the solutions of the equations (7) and (8), we impose the boundary
condition at the tip $r=r_{0}$ and the boundary $r=\infty$. Near the boundary,
the scalar field $\psi(r)$ and the scalar potential $\phi(r)$ behave as

$\displaystyle\psi$
$\displaystyle=\frac{\psi_{-}}{r^{\lambda_{-}}}+\frac{\psi_{+}}{r^{\lambda_{+}}},$
(9) $\displaystyle\phi$ $\displaystyle=\mu-\frac{\rho}{r^{2}}$ (10)

where

$\displaystyle\lambda_{\pm}=2\pm\sqrt{4+m^{2}}$ (11)

is the conformal dimension of the condensation operator
$\langle\cal{O}\rangle$ belongs to the boundary field theory,and $\psi_{-}$
and $\psi_{+}$ are the vacuum expectation values for the boundary operator
$\langle\cal{O}\rangle$. $\mu$ and $\rho$ are the chemical potential and the
charge density,respectively.

For simplicity we will take $m^{2}=-\frac{15}{4}$, which is above the
Breitenlohner-Freedman bound [34], hence
$\psi=\frac{\psi_{-}}{r^{\frac{3}{2}}}+\frac{\psi_{+}}{r^{\frac{5}{2}}}$. In
order to have a stable boundary theory, we impose the boundary condition as
$\psi_{-}=0$ or $\psi_{+}=0$ [5]. In this paper, we shall set $\psi_{-}=0$ and
$\langle\cal{O}\rangle=\psi_{+}$.

At the tip, we have

$\displaystyle\psi$
$\displaystyle=\alpha_{0}+\alpha_{1}\log(r-r_{0})+\alpha_{2}(r-r_{0})+\dots,$
(12) $\displaystyle\phi$
$\displaystyle=\beta_{0}+\beta_{1}\log(r-r_{0})+\beta_{2}(r-r_{0})+\dots$ (13)

We can impose the Neumann boundary condition $\alpha_{1}=\beta_{1}=0$ to make
the physical quantities finite. Note that the equations of motion (7) and (8)
have the scaling symmetry,
$(\psi,\phi,\mu,q,b)\to(\lambda\psi,\lambda\phi,\lambda\mu,q/\lambda,b\lambda^{2})$.
Thus we will set $q=1$ in the following discussions. Introducing a new
variable $z=1/r$, we rewrite the equation of motion (7) and (8) into

$\displaystyle\psi^{\prime\prime}+\left(\frac{f^{\prime}}{f}-\frac{1}{z}\right)\psi^{\prime}+\left(\frac{\phi^{2}}{z^{2}f}-\frac{m^{2}}{z^{4}f}\right)\psi=0$
(14)

and

$\displaystyle\phi^{\prime\prime}+\left(\frac{f^{\prime}}{f}+\frac{1}{z}+\left(2f-\frac{zf^{\prime}}{2}\right)bz^{5}\phi^{\prime
2}\right)\phi^{\prime}-\frac{2\psi^{2}}{z^{4}f}\left(1-bfz^{6}\phi^{\prime
2}\right)^{3/2}\phi=0$ (15)

We use shooting method numerically calculate the critical potential $\mu_{c}$,
the condensations of the scalar operators $\langle\cal{O}\rangle$ and charge
density $\rho$ for $b=0.01$. Then we change Born-Infeld parameter $b$ of the
theory, choosing $b=0,0.1,0.2,0.3$, find that the result varied so small that
we can neglect the effect of Born-Infeld parameter $b$. It is a somewhat
surprising result to us. However, our analytic calculation support this.

Figure 1:  Here we plot the behaviors of $\langle\cal{O}\rangle$ and charge
density $\rho$ for various values of chemical potential $\mu$. Here
$\mu_{c}=1.888$.

Now we holographically calculate the conductivity $\sigma(\omega)$ by
perturbing the gauge field [5]. To do so, we consider the following ansatz for
the gauge field along $x$ direction

$A_{x}=A_{x}(r)e^{-i\omega t}$ (16)

Neglecting the backreaction of the perturbational field on background metric,
one get the equation of motion of $A_{x}$ :

$\displaystyle\begin{split}A_{x}^{\prime\prime}+\left(\frac{f^{\prime}}{f}+\frac{1}{r}\right)A_{x}^{\prime}+\left(\frac{\omega^{2}}{r^{2}f}-\frac{2\psi^{2}}{f}\big{[}1+b(-\frac{f\phi^{\prime
2}}{r^{2}}-\frac{\omega^{2}A_{x}^{2}}{r^{4}}+\frac{fA_{x}^{\prime
2}}{r^{2}})\big{]}^{1/2}\right)A_{x}\\\
-\frac{b}{2f}\frac{1}{1+b(-\frac{f\phi^{\prime
2}}{r^{2}}-\frac{\omega^{2}A_{x}^{2}}{r^{4}}+\frac{fA_{x}^{\prime
2}}{r^{2}})}\big{[}\frac{2f\phi^{\prime
2}}{r^{3}}-\frac{f^{\prime}\phi^{\prime
2}}{f}-\frac{2f\phi^{\prime}\phi^{\prime\prime}}{r^{2}}+\frac{4\omega^{2}A_{x}^{2}}{r^{5}}-\\\
\frac{2\omega^{2}A_{x}A_{x}^{\prime}}{r^{4}}-\frac{2fA_{x}^{\prime
2}}{r^{3}}+\frac{f^{\prime}A_{x}^{\prime
2}}{r^{2}}+\frac{2fA_{x}^{\prime}A_{x}^{\prime\prime}}{r^{2}}\big{]}=0\end{split}$
(17)

Near the boundary($r\to\infty$), the behavior of the gauge field is

$A_{x}=A_{x}^{(0)}+\frac{A_{x}^{(1)}}{r^{2}}+\frac{A_{x}^{(0)}\omega^{2}}{2}\frac{\log\Lambda
r}{r^{2}}+\cdots$ (18)

the holographic conductivity is given as follows [11]

$\sigma(\omega)=\frac{-2iA_{x}^{(1)}}{\omega A_{x}^{(0)}}+\frac{i\omega}{2}$
(19)

Figure 2: The conductivity for operator $\langle\mathcal{O}\rangle$ for
different values of $b$.

In the figure 2 we have plotted the behavior of the conductivity for different
Born-Infield parameter $b$ when the condensation is non-zero, we find a pole
at $\omega\to 0$ suggesting that we have an infinite conductivity according to
the Kramers-Kronig relation as expected for the superconductor phase. We
observed the effect of Born-Infield parameter $b$ on the conductivity, the gap
frequency becomes larger as Born-Infield parameter $b$ increases. This is
different from those of Born-Infeld electrodynamics in Schwarzschild AdS black
hole spacetime[22].

##  3 Analytical calculation of the critical chemical potential $\mu_{c}$

From the results of the numerical calculations above, we found there exists a
critical chemical potential $\mu_{c}$. When chemical potential $\mu$ exceeds
this critical value,the solution is unstable and a scalar hair will emerge
which means the condensations take place.This is the superconductor phase.For
$\mu<\mu_{c}$,the scalar field is zero,which can be viewed as an insulator
phase.The critical chemical potential $\mu_{c}$ can be obtained by using
Sturm-Liouville (SL) eigenvalue method.

When $\mu\leq\mu_{c}$,the scalar field $\psi=0$,Eq.(15) reduces to

$\displaystyle\phi^{\prime\prime}+\left(\frac{f^{\prime}}{f}+\frac{1}{z}+\left(2f-\frac{zf^{\prime}}{2}\right)bz^{5}\phi^{\prime
2}\right)\phi^{\prime}=0$ (20)

By substituting $\phi^{\prime}$ to a new variable ,the equation will be
transformed into a first order ODE of Bernoulli type, see appendix A , and the
general solution is

$\displaystyle\begin{split}\phi\left(z\right)=&\frac{b^{2}\left(117z^{8}-306z^{4}+221\right)z^{9}}{1989}-\frac{2bC_{1}\left(77z^{12}-315z^{8}+495z^{4}-385\right)z^{3}}{1155}+\\\
&C_{1}^{2}\left(\frac{z^{13}}{13}-\frac{4z^{9}}{9}+\frac{6z^{5}}{5}-\frac{1}{3z^{3}}-4z\right)+C_{2}\\\
\end{split}$ (21)

where $C_{1}$ and $C_{2}$ are the integration constants. The boundary
condition (10) near the infinity $z=0$ imposes $C_{1}=0$ which keeps $\phi$ a
finite value and $C_{2}=\mu$. Thus,

$\displaystyle\phi\left(z\right)=\frac{b^{2}\left(117z^{8}-306z^{4}+221\right)z^{9}}{1989}+\mu$
(22)

When $\mu\rightarrow\mu_{c}$ ,we take $\phi\left(z\right)$ above into the
equation (14),thus the equation of motion for $\psi$ becomes

$\displaystyle\psi(z)\left(\frac{\left(\frac{b^{2}\left(117z^{8}-306z^{4}+221\right)z^{9}}{1989}+\mu\right)^{2}}{z^{2}f(z)}-\frac{m^{2}}{z^{4}f(z)}\right)+\left(\frac{f^{\prime}(z)}{f(z)}-\frac{1}{z}\right)\psi^{\prime}(z)+\psi^{\prime\prime}(z)=0$
(23)

As in [9],we introduce a trial function $F\left(z\right)$ near the boundary
$z=0$ which satisfies

$\displaystyle\psi(z)\sim\langle{\cal O}_{i}\rangle z^{\lambda_{i}}F(z),$ (24)

We impose the boundary condition for $F(z)$ as $F(0)=1$ and
$F^{\prime}(0)=0$.Thus the equation of motion for $F(z)$ is

$\displaystyle\begin{split}&F^{\prime\prime}(z)+\left(\frac{f^{\prime}(z)}{f(z)}+\frac{2}{z}\right)F^{\prime}(z)+\\\
&\left(\left(B^{2}\left(\frac{z^{17}}{17}-\frac{2z^{13}}{13}+\frac{z^{9}}{9}\right)+\mu\right)^{2}+\frac{3}{2}\left(zf^{\prime}(z)-f(z)\right)+\frac{3f(z)}{4}-\frac{m^{2}}{z^{2}}\right)\frac{F(z)}{z^{2}f(z)}=0\\\
\end{split}$ (25)

Following the standard procedure of Sturm-Liouville eigenvalue problem [35],we
introduce a new function $T(z)$

$\displaystyle T(z)=\left(1-z^{4}\right)z^{2\lambda_{i}-3}$ (26)

Multiplying $T(z)$ to both sides of the equation of $F(z)$ ,we have

$\displaystyle\begin{split}&\left(T(z)F^{\prime}(z)\right)^{\prime}+\frac{F(z)T(z)}{1-z^{4}}\left(\left(b^{2}\left(\frac{z^{17}}{17}-\frac{2z^{13}}{13}+\frac{z^{9}}{9}\right)+\mu\right)^{2}-\frac{m^{2}}{z^{2}}+\right.\\\
&\left.(\lambda_{i}-1)\lambda_{i}\left(\frac{1}{z^{2}}-z^{2}\right)+\lambda_{i}\left(\left(-\frac{2}{z^{3}}-2z\right)z+z^{2}-\frac{1}{z^{2}}\right)\right)=0\end{split}$
(27)

Furthermore we define another two functions $U$ and $V$ as

$\displaystyle\begin{split}U=&\frac{1}{1-z^{4}}\left(b^{4}\left(\frac{z^{17}}{17}-\frac{2z^{13}}{13}+\frac{z^{9}}{9}\right)^{2}+2b^{2}\mu\left(\frac{z^{17}}{17}-\frac{2z^{13}}{13}+\frac{z^{9}}{9}\right)-\frac{m^{2}}{z^{2}}+\right.\\\
&\left.(\lambda_{i}-1)\lambda_{i}\left(\frac{1}{z^{2}}-z^{2}\right)+\lambda_{i}\left(\left(-\frac{2}{z^{3}}-2z\right)z+z^{2}-\frac{1}{z^{2}}\right)\right)\end{split}$
(28) $\displaystyle V=T(z)/(1-z^{4})$ (29)

Then we can obtain the eigenvalue of $\mu^{2}$ by substituting $U$ and $V$
into the expression

$\displaystyle\mu^{2}=\frac{\int^{1}_{0}T\left(F^{\prime
2}-UF^{2}\right)dz}{\int^{1}_{0}VF^{2}dz},$ (30)

In order to estimate the minimum value of $\mu^{2}$,we assume the trial
function to be $F(z)=1-az^{2}$,where $a$ is a constant.In the normal case ,the
r.h.s of $\mu^{2}$ is an expression of $a$ and other parameters already given
as constants, e.g Born-Infeld parameter b in our case. But the problem we are
dealing with is a bit different for the expression of $\mu^{2}$ contains $\mu$
as well,

$\displaystyle\mu^{2}=\mu^{2}\left(a,b,\mu\right)$ (31)

To avoid this difficulty ,we suppose $\mu$ on the r.h.s of the above
expression to be $\tilde{\mu}$ at first, and consider $\tilde{\mu}$ as a new
constant parameter. Then we can compute the extreme value of $\mu^{2}$ as we
have done in the normal situation. But at this time the values of $a$
corresponding to the extremal eigenvalues of $\mu^{2}$ are related to the
parameter $\tilde{\mu}$, therefore,

$\displaystyle\mu^{2}=\mu^{2}\left(a(\tilde{\mu}),b,\tilde{\mu}\right)=\mu^{2}(\tilde{\mu},b)$
(32)

At last ,for consistency, $\tilde{\mu}$ is required to be $\mu$,and we obtain
an equation for $\mu$

$\displaystyle\mu_{m}^{2}=\mu_{m}^{2}(\mu_{m},b)$ (33)

where a subscript “m” is added to indicate this is an equation of extreme
value $\mu$. For the calculation details, see appendix B. The analytical
calculation shows the minimum eigenvalues of $\mu^{2}$ and the corresponding
values of $a$ for different Born-Infeld parameter factor $b$ are almost the
same,which is $a=0.330$ and $\mu_{min}=1.890$. We see the analytical results
fit perfectly with the numerical ones.

##  4 Conclusion

In this paper ,we considered the Born-Infeld electrodynamics in the AdS
soliton background.By numerical calculations , we found there exists a
critical chemical potential $\mu_{c}$.When the chemical potential $\mu$ is
lower then the critical chemical potential $\mu_{c}$,the AdS soliton
background is stable and dual field theory can be viewed as an insulator.When
$\mu>\mu_{c}$ ,the background begins to be unstable and a scalar hair will be
developed as the condensation, thus can be interpreted as the superconductor
phase.So the insulator and superconductor phase transition appeared in the
Maxwell electrodynamics also exists in its non-linear extension.We compute the
critical chemical potential $\mu_{c}$ both analytically and numerically and
the results of these two methods fit perfectly good.We also found the effect
of different Born-Infeld parameters are very small that even can be neglected.

## Acknowledgments

We are very grateful of M. Alishahiha and L. Li for their useful helps.

##  Appendix A Bernoulli type first order ODE

The standard Bernoulli type ordinary equation takes the form

$\displaystyle y^{\prime}=f(t,y(t))=g(t)y(t)+h(t){y(t)}^{n}$ $\displaystyle
n\neq 1$ (34)

We make the substitution $\xi=\phi^{\prime}$ in the equation (20) which
reduces to

$\displaystyle\frac{d\xi}{dz}=-\left(\frac{f^{\prime}}{f}+\frac{1}{z}\right)\xi-\left(2f-\frac{zf^{\prime}}{2}\right)bz^{5}\xi^{3}$
(35)

It is a Bernoulli type ordinary equation and the idea to solve the equation
(34) is to convert it into a linear ODE. Make the substitution

$\displaystyle v(t)={y(t)}^{1-n}$ (36)

thus

$\displaystyle v(t)^{\prime}=(1-n){y(t)}^{-n}y(t)^{\prime}$ (37)
$\displaystyle y(t)^{\prime}=\frac{y(t)^{n}}{1-n}v(t)^{\prime}$ (38)

so

$\displaystyle v(t)^{\prime}=(1-n)g(t)v(t)+(1-n)h(t)$ (39)

which is a standard linear ODE,and can be solved by the standard procedure.

##  Appendix B Computation of $\mu_{min}$

Substituting $U$,$V$ into expression (30) ,we get

$\displaystyle\mu^{2}=\frac{A(a)\tilde{\mu}+B(a)}{C(a)}$ $\displaystyle A(a)$
$\displaystyle=$
$\displaystyle-254592b^{2}\lambda_{i}\left(\lambda_{i}^{14}+141\lambda_{i}^{13}+8953\lambda_{i}^{12}+337449\lambda_{i}^{11}+8370439\lambda_{i}^{10}+\right.$
$\displaystyle\left.142959663\lambda_{i}^{9}+1704340979\lambda_{i}^{8}+13997457507\lambda_{i}^{7}+75168485084\lambda_{i}^{6}+\right.$
$\displaystyle\left.226063117896\lambda_{i}^{5}+135629702768\lambda_{i}^{4}-1262172448656\lambda_{i}^{3}-\right.$
$\displaystyle\left.2753135394624\lambda_{i}^{2}+1021968576000\lambda_{i}+2540624486400\right)\left(a^{2}\left(64\lambda_{i}^{6}+\right.\right.$
$\displaystyle\left.\left.2848\lambda_{i}^{5}+53040\lambda_{i}^{4}+517088\lambda_{i}^{3}+2740052\lambda_{i}^{2}+7415574\lambda_{i}+7977879\right)-\right.$
$\displaystyle\left.2a\left(64\lambda_{i}^{6}+2912\lambda_{i}^{5}+55888\lambda_{i}^{4}+570128\lambda_{i}^{3}+3209404\lambda_{i}^{2}+9320246\lambda_{i}+\right.\right.$
$\displaystyle\left.\left.10774995\right)+64\lambda_{i}^{6}+2976\lambda_{i}^{5}+58864\lambda_{i}^{4}+628992\lambda_{i}^{3}+3790660\lambda_{i}^{2}+\right.$
$\displaystyle\left.12084582\lambda_{i}+15758847\right)$ $\displaystyle C(a)$
$\displaystyle=$ $\displaystyle
3956121(\lambda_{i}-2)(\lambda_{i}-1)\lambda_{i}(\lambda_{i}+1)(\lambda_{i}+8)(\lambda_{i}+9)(\lambda_{i}+10)(\lambda_{i}+11)(\lambda_{i}+12)$
$\displaystyle(\lambda_{i}+13)(\lambda_{i}+14)(\lambda_{i}+15)(\lambda_{i}+16)(\lambda_{i}+17)(\lambda_{i}+18)(2\lambda_{i}+7)(2\lambda_{i}+9)$
$\displaystyle(2\lambda_{i}+11)(2\lambda_{i}+13)(2\lambda_{i}+15)(2\lambda_{i}+17)(2\lambda_{i}+19)\left(\frac{a^{2}}{\lambda_{i}+1}-\frac{2a}{\lambda_{i}}+\frac{1}{\lambda_{i}-1}\right)$

$\displaystyle B(a)$ $\displaystyle=$ $\displaystyle
a^{2}\left(8\lambda_{i}^{3}+180\lambda_{i}^{2}+1318\lambda_{i}+3135\right)\left(16\lambda_{i}^{13}+1504\lambda_{i}^{12}+61872\lambda_{i}^{11}+\right.$
$\displaystyle\left.1461236\lambda_{i}^{10}+21801935\lambda_{i}^{9}+212273937\lambda_{i}^{8}+1333101101\lambda_{i}^{7}+\right.$
$\displaystyle\left.4996108013\lambda_{i}^{6}+8010950487\lambda_{i}^{5}-11997217421\lambda_{i}^{4}-68501188673\lambda_{i}^{3}-\right.$
$\displaystyle\left.66321743349\lambda_{i}^{2}+59135273262\lambda_{i}+73109116080\right)\left(2\left(-8\left(64b^{4}-\right.\right.\right.$
$\displaystyle\left.\left.\left.35605089\right)\lambda_{i}^{5}-96\left(368b^{4}-85715955\right)\lambda_{i}^{4}-16\left(59872b^{4}-\right.\right.\right.$
$\displaystyle\left.\left.\left.7536410505\right)\lambda_{i}^{3}-48\left(258472b^{4}-19251803493\right)\lambda_{i}^{2}-\right.\right.$
$\displaystyle\left.\left.1728\left(39269b^{4}-1933664031\right)\lambda_{i}+3956121\lambda_{i}^{6}+3828259169280\right)+\right.$
$\displaystyle\left.3956121\left(\lambda_{i}^{5}+70\lambda_{i}^{4}+1940\lambda_{i}^{3}+26600\lambda_{i}^{2}+180384\lambda_{i}+483840\right)m^{2}\right)-$
$\displaystyle
2a\lambda_{i}\left(16\lambda_{i}^{11}+1632\lambda_{i}^{10}+73016\lambda_{i}^{9}+1880616\lambda_{i}^{8}+30716889\lambda_{i}^{7}+\right.$
$\displaystyle\left.329379372\lambda_{i}^{6}+2305033064\lambda_{i}^{5}+9928719744\lambda_{i}^{4}+21299033680\lambda_{i}^{3}-\right.$
$\displaystyle\left.4869591744\lambda_{i}^{2}-119590571520\lambda_{i}-169885900800\right)\left(3956121\left(8\lambda_{i}^{8}+\right.\right.$
$\displaystyle\left.\left.676\lambda_{i}^{7}+24482\lambda_{i}^{6}+495859\lambda_{i}^{5}+6136697\lambda_{i}^{4}+47461414\lambda_{i}^{3}+\right.\right.$
$\displaystyle\left.\left.223683648\lambda_{i}^{2}+586275651\lambda_{i}+652759965\right)\left(2\lambda_{i}(\lambda_{i}+2)+(\lambda_{i}+1)m^{2}\right)-\right.$
$\displaystyle\left.128b^{4}\left(64\lambda_{i}^{9}+5408\lambda_{i}^{8}+195792\lambda_{i}^{7}+3929640\lambda_{i}^{6}+46829160\lambda_{i}^{5}+\right.\right.$
$\displaystyle\left.\left.322610184\lambda_{i}^{4}+1147216028\lambda_{i}^{3}+1425431605\lambda_{i}^{2}-1194241044\lambda_{i}-\right.\right.$
$\displaystyle\left.\left.1751976837\right)\right)+\lambda_{i}\left(16\lambda_{i}^{12}+1792\lambda_{i}^{11}+88864\lambda_{i}^{10}+2567628\lambda_{i}^{9}+\right.$
$\displaystyle\left.47829603\lambda_{i}^{8}+599455521\lambda_{i}^{7}+5108576957\lambda_{i}^{6}+29104930589\lambda_{i}^{5}+\right.$
$\displaystyle\left.104893992911\lambda_{i}^{4}+206701115187\lambda_{i}^{3}+113193419679\lambda_{i}^{2}-\right.$
$\displaystyle\left.236408070717\lambda_{i}-223243908030\right)\left(3956121\left(8\lambda_{i}^{8}+612\lambda_{i}^{7}+19974\lambda_{i}^{6}+\right.\right.$
$\displaystyle\left.\left.362715\lambda_{i}^{5}+4001532\lambda_{i}^{4}+27406788\lambda_{i}^{3}+113514256\lambda_{i}^{2}+\right.\right.$
$\displaystyle\left.\left.259082880\lambda_{i}+248371200\right)\left(2\lambda_{i}+m^{2}\right)-128b^{4}\left(64\lambda_{i}^{8}+4832\lambda_{i}^{7}+\right.\right.$
$\displaystyle\left.\left.154832\lambda_{i}^{6}+2705144\lambda_{i}^{5}+27068168\lambda_{i}^{4}+140926760\lambda_{i}^{3}+\right.\right.$
$\displaystyle\left.\left.253029340\lambda_{i}^{2}-493864887\lambda_{i}-1682001090\right)\right)$

Taking $\tilde{\mu}=\mu$, $\lambda_{i}=\frac{5}{2}$ , b=0.01 and solving the
above equation, we get

$\displaystyle\mu_{min}=1.89037$

where we have discarded the negative value of $\mu_{min}$.

## References

  * [1] J. M. Maldacena, Adv. Theor. Math. Phys. 2, 231 (1998). [arXiv:hep-th/9711200]
  * [2] S. S. Gubser, I. R. Klebanov, A. M. Polyakov, Phys. Lett. B 428, 105 (1998). [arXiv:hep-th/9802109]
  * [3] E. Witten, Adv. Theor. Math. Phys. 2, 253 (1998). [arXiv:hep-th/9802150]
  * [4] S. S. Gubser, Phys. Rev. D 78, 065034 (2008). [arXiv:0801.2977]
  * [5] S. A. Hartnoll, C. P. Herzog, G. T. Horowitz, Phys. Rev. Lett. 101, 031601 (2008). [arXiv:0803.3295]
  * [6] S. A. Hartnoll, C. P. Herzog, G. T. Horowitz, JHEP 0812, 015 (2008). [arXiv:0810.1563]
  * [7] S. A. Hartnoll, Class. Quantum Grav. 26, 224002 (2009). [arXiv:0903.3246]
  * [8] R. Gregory, S. Kanno, J. Soda, JHEP 0910, 010 (2009). [arXiv:0907.3203]
  * [9] G. Siopsis, J. Therrien, JHEP 1005, 013 (2010). [arXiv:1003.4275]
  * [10] T. Nishioka, S. Ryu, T. Takayanagi, JHEP 1003, 131 (2010). [arXiv:0911.0962]
  * [11] G. T. Horowitz, M. M. Roberts, Phys. Rev. D 78, 126008 (2008). [arXiv:0810.1077]
  * [12] G. T. Horowitz, M. M. Roberts, JHEP 0911, 015 (2009). [arXiv:0908.3677]
  * [13] R.G. Cai, H. Q. Zhang, Phys. Rev. D 81, 066003 (2010). [arXiv:0911.4867]
  * [14] H. B. Zeng, X. Gao, Y. Jiang, H. S. Zong, JHEP 1105, 002 (2011). [arXiv:1012.5564]
  * [15] Q. Pan, B. Wang, E. Papantonopoulos, J. Oliveira, A. Pavan, Phys. Rev. D 81, 106007 (2010). [arXiv:0912.2475]
  * [16] Q. Pan, J. Jing, B. Wang, JHEP 1111, 088 (2011). [arXiv:1105.6153]
  * [17] R. G. Cai, H. F. Li, H. Q. Zhang, Phys. Rev. D 83, 126007 (2011). [arXiv:1103.5568]
  * [18] R. A. Konoplya, A. Zhidenko,Phys. Lett. B686:199-206, 2010.[arXiv:0909.2138]
  * [19] D.Roychowdhury, Phys Rev D 86, 106009 (2012).[arXiv:1211.0904]
  * [20] D.Roychowdhury, Physics Letters B 718 (2013) pp. 1089-1094.[arXiv:1211.1612]
  * [21] R.Banerjee, S.Gangopadhyay, D.Roychowdhury and A.Lala, [arxiv:1208.5902]
  * [22] J. Jing, S. Chen, Phys. Lett. B 686, 68 (2010). [arXiv:1001.4227]
  * [23] J. Jing, Q. Pan, S. Chen, JHEP 1111, 045 (2011). [arXiv:1106.5181]
  * [24] J. Jing, L. Wang, Q. Pan, S. Chen, Phys. Rev. D 83, 066010 (2011). [arXiv:1012.0644]
  * [25] S. Gangopadhyay, D. Roychowdhury, JHEP 1205, 002 (2012). [arXiv:1201.6520]
  * [26] S. Gangopadhyay, D. Roychowdhury, JHEP 1205, 156 (2012). [arXiv:1204.0673]
  * [27] J. Jing, Q. Pan, S. Chen, Phys. Lett. B 716, 385 (2012). [arXiv:1209.0893]
  * [28] M. Born, L. Infeld, Proc. Roy. Soc. A 144, 425 (1934).
  * [29] G. W. Gibbons, Rev. Mex. Fis. 49S1, 19 (2003). [arXiv:hep-th/0106059]
  * [30] A. A. Tseytlin. [arXiv:hep-th/9908105]
  * [31] G. T. Horowitz, B. Way, JHEP 1011, 011 (2010). [arXiv:1007.3714]
  * [32] A. Akhavan, M. Alishahiha, Phys. Rev. D 83, 086003 (2011). [arXiv:1011.6158]
  * [33] M. Montull, O. Pujol s, A. Salvio, P. J. Silva, JHEP 1204, 135 (2012). [arXiv:1202.0006]
  * [34] P. Breitenlohner, D. Freedman, Ann. Phys. 144, 197 (1982).
  * [35] I. M. Gelfand and S. V. Fomin, “Calculaus of Variations,” (1963) Revised English Edition, Translated and Edited by R. A. Silverman,Prentice-Hall, Inc. Englewood Cliff, New Jersey.