# Large-Scale 3D Semantic Reconstruction for Automated Driving Vehicles with
Adaptive Truncated Signed Distance Function

Haohao Hu∗,1, Hexing Yang†,1, Jian Wu†,1, Xiao Lei†,1, Frank Bieder2, Jan-
Hendrik Pauls1 and Christoph Stiller1,2 ∗Corresponding author. †Authors
contributed equally.1 Authors are with the  Institute of Measurement and
Control Systems, Karlsruhe Institute of Technology, Karlsruhe, Germany.
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Authors are with the
FZI Research Center for Information Technology, Karlsruhe, Germany.
<EMAIL_ADDRESS>
###### Abstract

The Large-scale 3D reconstruction, texturing and semantic mapping are nowadays
widely used for automated driving vehicles, virtual reality and automatic data
generation. However, most approaches are developed for RGB-D cameras with
colored dense point clouds and not suitable for large-scale outdoor
environments using sparse LiDAR point clouds. Since a 3D surface can be
usually observed from multiple camera images with different view poses, an
optimal image patch selection for the texturing and an optimal semantic class
estimation for the semantic mapping are still challenging.

To address these problems, we propose a novel 3D reconstruction, texturing and
semantic mapping system using LiDAR and camera sensors. An Adaptive Truncated
Signed Distance Function is introduced to describe surfaces implicitly, which
can deal with different LiDAR point sparsities and improve model quality. The
from this implicit function extracted triangle mesh map is then textured from
a series of registered camera images by applying an optimal image patch
selection strategy. Besides that, a Markov Random Field-based data fusion
approach is proposed to estimate the optimal semantic class for each triangle
mesh. Our approach is evaluated on a synthetic dataset, the KITTI dataset and
a dataset recorded with our experimental vehicle. The results show that the 3D
models generated using our approach are more accurate in comparison to using
other state-of-the-art approaches. The texturing and semantic mapping achieve
also very promising results.

##  I INTRODUCTION

In recent years, the accurate 3D reconstruction, texturing and semantic
mapping of surrounding environments has attracted more and more attention. On
the one hand, an accurately generated 3D geometric model of the real world
extends functionalities of applications like virtual reality (VR) and
augmented reality (AR). On the other hand, for automated driving vehicles, an
accurate 3D environment representation is essential for various tasks, such as
mapping, localization, decision making and trajectory planning. A realistic
environment model also allows us to simulate different driving scenarios for
the algorithm developing and testing in the field of autonomous driving, which
can speed up the development sufficiently. An accurately reconstructed and
semantic mapped 3D model can accelerate the training data generation for deep
learning without too much extra effort. Besides that, the semantic mapped 3D
models can also be used to extract semantic HD maps for automated driving
vehicles automatically. However, the large-scale 3D reconstruction, texturing
and semantic mapping based on LiDAR and camera sensors remains challenging.

Figure 1:  A result example of our approach including the 3D reconstructed
triangle mesh map (in blue), the textured 3D model (in violet) and the
semantic mapped 3D model (in red) based on LiDAR and camera measurements.
Figure 2:  Our large-scale 3D reconstruction, texturing and semantic mapping
pipeline for automated driving vehicles using LiDAR and camera sensors. An
accurate 3D geometric environment model is obtained from accumulated LiDAR
point clouds using Adaptive TSDF. The reconstructed 3D model is accurately
textured from a series of registered camera images with an optimal image patch
selection strategy and color optimization techniques. With a MRF-based data
fusion approach, the 3D geometric model is then semantic mapped using multiple
semantic segmented camera images.

In the past decade, research on dense 3D reconstruction has produced many
compelling works using depth sensors based on the time-of-flight (ToF) or
structured light technique [1, 2, 3, 4, 5]. Noteworthy, these for indoor
scenes designed systems are not suitable for large-scale outdoor environments,
since the measuring range of those depth sensors is small and the data quality
is strongly influenced by outdoor light and weather conditions. The recent
advanced LiDARs enable another feasible solution for the 3D reconstruction of
large-scale outdoor environments. Compared with RGB-D cameras, the depth
measurements of LiDARs are more accurate, reliable and reach a larger
measuring range.

Nevertheless, the sparse and unevenly distributed LiDAR measurement points in
the 3D space makes the 3D reconstruction very challenging. Moreover, the
reconstructed 3D geometric models from LiDAR point clouds do not contain any
textures directly, which is however very useful. Texturing geometric models
from images is challenging because of the illumination and exposure time
changes over time. Besides that, an accurate semantic mapping of 3D geometric
models is also challenging because of multiple observations of one triangle
mesh. In addition, moving objects in the input images make the texturing
complex and uncertain. The work presented in [6] performs a large-scale 3D
reconstruction and texturing algorithm using LiDAR and camera sensors with
Truncated Signed Distance Function (TSDF). Our approach builds on this
previous work and make several improvements. Additionally, a semantic mapping
pipeline of the reconstructed 3D models using semantic segmentation is also
introduced, which is a perfect input for further applications like the
automatic generation of semantic HD maps.

In this work, we propose a large-scale 3D reconstruction, texturing and
semantic mapping approach based on LiDAR and camera measurements. To handle
the sparsity and heterogeneous density of the LiDAR point clouds, we propose
an Adaptive Truncated Signed Distance Function (Adaptive TSDF)-based
volumetric data fusion algorithm based on the well established work InfiniTAM
[4]. Im comparison to the original Truncated Signed Distance Function (TSDF),
our truncation distance is adaptively adjusted according to the density of
LiDAR point measurements and the flatness of the reconstructed surfaces. By
benefiting from works [7, 8], the reconstructed 3D geometric models can be
textured with a very high resolution. Through simplifying the optimization
problem and parallelizing the part of the algorithm on a commercial GPU
device, we can speed up the texturing process efficiently. Secondly, applying
an optimal image patch selection strategy and color optimization techniques
improves the texturing quality of 3D models. In addition, we use a Markov
Ransom Filed (MRF)-based data fusion approach to realize an accurate semantic
mapping of 3D models with multiple semantic segmented images.

We evaluate our approach on a synthetic dataset generated by LGSVL [9], the
real world KITTI dataset [10] and also a real world dataset recorded with our
experimental vehicle Bertha One [11]. Results show that our large-scale 3D
reconstruction approach generates much smoother and more accurate 3D geometric
models than our previous work. The texturing and semantic mapping system
provides also very accurate and promising results for further applications
such as automatic semantic HD map generation. The main contributions of this
work are listed as follow:

  1. 1.

The Adaptive TSDF-based volumetric data fusion algorithm can deal with the
heterogeneous density of LiDAR point clouds and improve the quality and
continuousness of the reconstructed surfaces.

  2. 2.

The optimal image patch selection, consistency check and color optimization
strategy improve the texturing quality and make our textured model seamless.

  3. 3.

The MRF-based data fusion approach improves the semantic class estimation for
each reconstructed triangle face based on multiple semantic segmented images.

##  II RELATED WORK

###  II-A 3D Geometric Reconstruction

A suitable 3D environment representation of sensor measurements is a key to a
high-quality 3D reconstruction. In comparison to unstructured point-based [12,
13, 14] and surfel-based representations [15, 16], volumetric methods with
implicit surfaces have the most potential and are widely used on 3D
reconstruction algorithms [1, 2, 3, 4, 5]. However, those approaches are
developed for indoor environments using depth sensors like RGB-D cameras but
not suitable for outdoor LiDAR scans because of the sparsity and heterogeneous
density. The work proposed in [6] extends KinectFusion [1] to outdoor scenes
using TSDF [17] with a large truncation distance at the cost of losing model
details. In [18], an adaptive neighborhood kernel based on Gaussian confidence
evaluation is introduced. Nevertheless, it’s difficult to store statistics for
all voxels due to memory limitation.

###  II-B Texture Mapping

For indoor scenarios, frameworks based on RGB-D cameras [1, 4, 5] obtain
impressive textured models. However, textures are encoded as either per-vertex
color or per-face color and thus coupled to the mesh resolution. For outdoor
scenarios using LiDARs, the textured map can not be obtained directly. The
reconstructed 3D models are usually textured from a series of registered
camera images. Several approaches like [19, 20] select and blend multiple
images per face to obtain a global-consistent texture. To solve this problem,
the methods proposed in [7, 8] select only one image per face to texture 3D
models. If images are registered with small errors or the reconstructed models
are slightly inaccurate, textures from different views might be misaligned,
producing ghost and strong visible seams. In this work, we apply an optimal
image patch selection strategy to solve the single face multi-view image
issue. Besides that, several color optimization techniques are introduced to
reduce the texture seams and improve the texture continuousness.

###  II-C Semantic Mapping

The 3D semantic reconstruction are divided into indoor and outdoor scenes. For
indoor scenes, individual pre-defined object models [21], volumetric models
[22] [23], meshes [24] or quadrics [25] are used. However, outdoor scenes can
not be represented by distinct components or simple geometric primitives.
Continuous structures like road markers, barriers, and buildings are
represented in different ways. The authors in [26] assess the scene in terms
of its primitives, such as ground, vegetation and barriers with the Dempster
Shafer theory on a 2D image plane. The approach presented in [27] extends the
work [28] to a 3D semantic reconstruction, which deals with indoor
environments and works with a limited sensing range, resolution and quality.
Instead of using 3D mesh maps, some approaches like [29] [30] use outer
contours of semantic edges to reconstruct 3D edges directly, which needs less
resources. However, the reduced mesh resolution also limits the reconstruction
quality.

##  III 3D Geometric Reconstruction

Figure 3:  Adaptive TSDF-based implicit surface representation. Each yellow
dot represents a measurement point, the red solid line corresponds to the
reconstructed surface and the numbers in grids are TSDF values. To ensure the
completeness of the reconstructed surface and retain as much detail as
possible, the truncation distance $\varepsilon$ is dynamically adapted to the
point measurements as shown in the red dotted line.

Similar as KinectFusion [1], we update the implicit surface representation of
measurements at each frame parallel. The voxel block hashing contributed by
[3, 4] allows for a volumetric fusion at scale. The basic technique used here
for the 3D reconstruction is TSDF and one of the most important parameter is
called truncation distance $\varepsilon$, which denotes the influence range of
a measurement point on the updating of the 3D voxel array. By using a small
$\varepsilon$, the completeness and continuousness of the reconstructed
surfaces can be worsened because of a low point cloud density. A big
$\varepsilon$ makes the reconstructed surfaces smoother but the details, where
the structure is complex and narrowly, won’t be reconstructed accurately.
Choosing a suitable $\varepsilon$ for different environments is always
challenging, because of the various of the point cloud density and environment
complexity. To solve this problem, as illustrated in Fig. 3, we introduce a
dynamically computed truncation distance $\varepsilon$ based on the point
cloud density and structure complexity adaptively, which allows to cope with
the sparsity and heterogeneous density of LiDAR point clouds. Most
importantly, to tackle the heterogeneous density of the input data, we
calculate the 3D statistical distribution of points for each voxel block
consisting of $8\times 8\times 8$ voxels, rather than for each voxel as
proposed in [18]. This concession allows us to keep a good trade-off between
rich information and memory consumption.

###  III-A Planar Surface Estimation

As we are limited by the physical memory, we calculate the statistical
distribution upon insertion of new measurement incrementally. We consider a
point cloud consists of $n$ points
$\mathcal{P}\\!\to\\!\\{p_{\mathrm{0}},p_{\mathrm{1}},...,p_{\mathrm{n}}\\}$,
the 3D statistical distribution includes the number of points $n$, the mean
$E_{\mathrm{\mu}}$ and the covariance matrix $C_{\mathrm{\sigma}}$. Suppose we
have two point clouds
$\mathcal{P}_{\mathrm{1}}\\!\to\\!\\{p_{\mathrm{0}},p_{\mathrm{1}},...,p_{\mathrm{n_{1}}}\\}$,
$\mathcal{P}_{\mathrm{2}}\\!\to\\!\\{p_{\mathrm{0}},p_{\mathrm{1}},...,p_{\mathrm{n_{2}}}\\}$,
where $n_{\mathrm{1}}$, $E^{\mathrm{1}}_{\mathrm{\mu}}$,
$C^{\mathrm{1}}_{\mathrm{\sigma}}$ and $n_{\mathrm{2}}$,
$E^{\mathrm{2}}_{\mathrm{\mu}}$, $C^{\mathrm{2}}_{\mathrm{\sigma}}$ represents
its distribution information respectively. For the sum of the two point
clouds, the new distribution $E_{\mathrm{\mu}}^{\prime}$,
$C_{\mathrm{\sigma}}^{\prime}$ can be computed by:

$\begin{split}E_{\mathrm{\mu}}^{\prime}&=\frac{n_{\mathrm{1}}E_{\mathrm{\mu}}^{\mathrm{1}}+n_{\mathrm{2}}E_{\mathrm{\mu}}^{\mathrm{2}}}{n_{\mathrm{1}}+n_{\mathrm{2}}}\\\
C_{\mathrm{\sigma}}^{\prime}&=\frac{1}{n_{\mathrm{1}}+n_{\mathrm{2}}-1}\\{(n_{\mathrm{1}}-1)C_{\mathrm{\sigma}}^{\mathrm{1}}+(n_{\mathrm{2}}-1)C_{\mathrm{\sigma}}^{\mathrm{2}}\\\
&\quad+n_{\mathrm{1}}(E_{\mathrm{\mu}}^{\prime}-E_{\mathrm{\mu}}^{\mathrm{1}})(E_{\mathrm{\mu}}^{\prime}-E_{\mathrm{\mu}}^{\mathrm{1}})^{\intercal}\\\
&\quad+n_{\mathrm{2}}(E_{\mathrm{\mu}}^{\prime}-E_{\mathrm{\mu}}^{\mathrm{2}})(E_{\mathrm{\mu}}^{\prime}-E_{\mathrm{\mu}}^{\mathrm{2}})^{\intercal}\\}\text{.}\end{split}$
(1)

Especially for a point cloud just with exactly one point
$\mathcal{P}\to\\{p_{\mathrm{0}}\\}$ hold $E_{\mathrm{\mu}}=p_{\mathrm{0}}$
and $C_{\mathrm{\sigma}}=[0]_{\mathrm{3\times 3}}$. After having obtained the
local statistics, the covariance matrix $C_{\mathrm{\sigma}}$ can be used to
estimate the local surface through a Principle Component Analysis (PCA).
Suppose the eigenvectors
$(\vec{v}_{\mathrm{1}},\vec{v}_{\mathrm{2}},\vec{v}_{\mathrm{3}})$ and the
eigenvalues $(\lambda_{\mathrm{1}},\lambda_{\mathrm{2}},\lambda_{\mathrm{3}})$
with $\lambda_{\mathrm{1}}>\lambda_{\mathrm{2}}>\lambda_{\mathrm{3}}$, the
least dominant axis $\vec{v}_{\mathrm{3}}$ is used to represent the unoriented
normal direction of the estimated surface. Since the normal direction should
be consistent and oriented towards the sensor, the normal of the planar
surface can be written as:

$\begin{split}\vec{n}=\left\\{\begin{array}[]{lr}\vec{v}_{\mathrm{3}}\ \
\quad\text{if}\
\vec{v}_{\mathrm{3}}\cdot(\vec{p}_{\mathrm{s}}-E_{\mathrm{\mu}})>0\\\
-\vec{v}_{\mathrm{3}}\quad\text{otherwise,}\end{array}\right.\end{split}$ (2)

where $\vec{p}_{\mathrm{s}}$ is the 3D position of the sensor. Furthermore,
the flatness of the surface $P_{\mathrm{flat}}$ can be represented as:

$P_{\mathrm{flat}}=1-\frac{\lambda_{\mathrm{3}}}{\lambda_{\mathrm{2}}}\text{.}$
(3)

###  III-B Implicit Surface Representation

As described before, large truncation distance $\varepsilon$ ensures the
surface’s completeness and a small $\varepsilon$ preserves much model details.
Therefore, we adaptively adjust the $\varepsilon$ according to the point cloud
density and surface flatness using:

$\varepsilon=\max(\varepsilon_{\mathrm{min}},\frac{kP_{\mathrm{flat}}}{n}\varepsilon_{\mathrm{max}})\text{,}$
(4)

where $\varepsilon_{\mathrm{min}}$ and $\varepsilon_{\mathrm{max}}$ are the
lower and upper bound, $n$ is the number of measurement points inside the
voxel block and $k$ is the weight to describe the importance of the flatness.
Additionally, to reduce the effects of outliers and systematic distortions of
range sensors, a weight $w$ for each measurement is calculated using
$w=\vec{n}\cdot(\vec{p}_{\mathrm{s}}-\vec{p}_{\mathrm{i}})$, where $\vec{n}$
is the surface normal, $\vec{p}_{\mathrm{s}}$ is the sensor position and
$\vec{p}_{\mathrm{i}}$ represents a measurement point. The basic idea is that
a lower angle between the incident light and the normal of the surface usually
corresponds to better measurement samples.

##  IV Texture Mapping

###  IV-A Image Selection

To obtain texture maps with a high resolution, we texture each face with the
best image patch rather than select and blend multiple images. To check the
face visibility for all combinations of images and faces in parallel, we
implement a BVH tree [31] in CUDA [32]. For each face, we select the best
image patch using a MRF-based energy function [7]:

$\sum_{F_{\mathrm{i}}\in\text{faces}}E_{\mathrm{Q}}(F_{\mathrm{i}},\mathbf{I}_{\mathrm{i}})+\lambda\sum_{F_{\mathrm{i}},F_{\mathrm{j}}\in\text{edges}}E_{\mathrm{S}}(F_{\mathrm{i}},F_{\mathrm{j}},\mathbf{I}_{\mathrm{i}},\mathbf{I}_{\mathrm{j}})\text{,}$
(5)

where $E_{\mathrm{Q}}$ represents the quality of the image
$\mathbf{I}_{\mathrm{i}}$ to be used for face $F_{\mathrm{i}}$ and
$E_{\mathrm{s}}$ corresponds to the smoothness between two adjacent faces
textured from different images. The work in [7] uses the angle between the
local viewing direction and the face normal as the cost. However, that is
insufficient for us since the vehicle motion makes the angle difference not
obvious. In [33, 34], a face is projected into image and the projection area
size is used as the quality term or the blending weight. Nevertheless, this
does not account for extreme cases such as underexposure or overexposure of
the recorded images. The authors of [8] use both the gradient magnitude of the
image and the face’s projection area size to handle out-of-focus blur. We
choose a similar term since it accounts for similar effects and thus the first
term can be written as the sum of the gradient magnitude
$||\Delta(\mathbf{I}_{\mathrm{i}}(p))||_{\mathrm{2}}$ of all pixels within the
projection area:

$E_{\mathrm{Q}}(F_{\mathrm{i}},\mathbf{I}_{\mathrm{i}})=\int_{\phi(F_{\mathrm{i}},\mathbf{I}_{\mathrm{i}})}||\Delta(\mathbf{I}_{\mathrm{i}}(p))||_{\mathrm{2}}\mathrm{d}p\text{.}$
(6)

Considering two adjacent faces $F_{\mathrm{i}}$ and $F_{\mathrm{j}}$ share an
edge $E_{\mathrm{ij}}$, the color seam between them can be evaluated by
integrating all pixel differences along this edge [7]. However, this is a
significant bottleneck of this algorithm and taking account of by selecting
the same image for adjacent faces can achieve the same goal with the
simplified term:

$E_{\mathrm{S}}(F_{\mathrm{i}},F_{\mathrm{j}},\mathbf{I}_{\mathrm{i}},\mathbf{I}_{\mathrm{j}})=\left\\{\begin{array}[]{lr}1\quad\text{if}\
\mathbf{I}_{\mathrm{i}}=\mathbf{I}_{\mathrm{j}}\\\
0\quad\text{otherwise.}\end{array}\right.$ (7)

###  IV-B Color Adjustment

Due to color discontinuities or wrong textures caused by occlusions like
moving objects, more complex processing steps are necessary to optimize the
textures.

####  IV-B1 Vignetting Correction

The color discontinuities are mainly caused by the vignetting of images, where
the brightness or saturation of pixels around the image drop significantly
compared to the image center. To reduce this effect, we use a vignetting
correction algorithm [35]:

$g(r)=1+ar^{2}+br^{4}+cr^{6}$ (8)

where $g(r)$ is the gain and $r$ corresponds to the distance from the image
center $(\bar{u},\bar{v})$ to the pixel being treated $(u,v)$:

$r=\frac{\sqrt{(u-\bar{u})^{2}+(v-\bar{v})^{2}}}{\sqrt{\bar{u}^{2}+\bar{v}^{2}}}$
(9)

After choosing a set of suitable parameters $a$, $b$ and $c$, the adjusted
image without vignetting can be obtained by:

$\mathbf{I}^{\prime}(u,v)=g(r)\times\mathbf{I}(u,v)$ (10)

####  IV-B2 Photo Consistency Check

To remove color outliers, the photo consistency check algorithm proposed in
[8] is applied in the HSV color space. This algorithm assumes that the
majority of images have the correct color.

####  IV-B3 Global Color Adjustment

To make our textured maps as seamless as possible, we use a continuous
formulation of global color adjustment to further improve the appearance of
the texture using the approach proposed in [7]. To minimize color
discontinuities, we compute an additive correction $g_{\mathrm{v}}$ as in [7]
for each vertex by minimizing:

$\sum_{v_{\mathrm{i}}}(f_{v_{\mathrm{i}}^{\mathrm{l}}}+g_{v_{\mathrm{i}}^{\mathrm{l}}}-(f_{v_{\mathrm{i}}^{\mathrm{r}}}+g_{v_{\mathrm{i}}^{\mathrm{r}}}))^{2}+\lambda\sum_{v_{\mathrm{i}},v_{\mathrm{j}}\in\text{edges}}(g_{v_{\mathrm{i}}^{\mathrm{l}}}-g_{v_{\mathrm{j}}^{\mathrm{r}}})^{2}$
(11)

where $f_{v_{\mathrm{i}}^{\mathrm{l}}}$ is calculated by averaging color
samples along the edge $E_{v_{\mathrm{i}}v_{\mathrm{j}}}$. The first term in
Eq. 11 ensures that the colors on both sides of an edge are as similar as
possible, while the second term aims at finding the smallest absolute additive
correction. After finding optimal $g_{\mathrm{v}}$ for all vertices, the
correction for each pixel inside a face is interpolated using barycentric
coordinates.

##  V Semantic Mapping

Figure 4: Elimination of invisible faces.

In addition to a textured geometric model, semantic information is also
critical to interpret surrounding environments. If the input source in the
texturing procedure are changed to semantic segmented images, we can get a
colorized mesh map, in which the color of each face represents a semantic
class. The only difference to the previous step presented in Sec. IV-A is the
label here corresponds to semantic classes. The face visibility is checked
with a BVH tree [31]. We use a MRF-based energy function [7] to select the
face class:

$\sum_{F_{\mathrm{i}}\in\mathrm{faces}}\\!\\!E_{\mathrm{D}}(F_{\mathrm{i}},C_{\mathrm{m}})+\lambda\\!\\!\sum_{F_{\mathrm{i}},F_{\mathrm{j}}\in\mathrm{edges}}\\!\\!E_{\mathrm{S}}(F_{\mathrm{i}},F_{\mathrm{j}},C_{\mathrm{m}},C_{\mathrm{n}})\text{,}$
(12)

where $E_{\mathrm{D}}$ denotes the data term and represents the probability of
the face $F_{\mathrm{i}}$ to be class $C_{\mathrm{m}}$ concerning the semantic
image data. $E_{\mathrm{s}}$ corresponds to the smoothness between two
adjacent faces. To calculate the probability of each face, we project the face
into image and use the size of the projected face as the data term like in
[33]. Since a face is visible in multiple frames, the accumulated size after
normalization is used as the final probability. To deal with faces on the
edge, a part of which belongs to $C_{\mathrm{m}}$ and another part belongs to
$C_{\mathrm{n}}$, we only consider the center pixel of a triangle. The area of
the triangle is calculated as the number of votes:

$E_{\mathrm{D}}(F_{\mathrm{i}},C_{\mathrm{m}})=\frac{\sum\limits_{I_{\mathrm{k}}\in\mathrm{visible}}S_{I_{\mathrm{k}}}(F_{\mathrm{i}},C_{\mathrm{m}})}{\sum\limits_{m=0}^{N}\sum\limits_{I_{\mathrm{k}}\in\mathrm{visible}}S_{I_{\mathrm{k}}}(F_{\mathrm{i}},C_{\mathrm{m}})}\text{,}$
(13)

where $N$ is the total number of classes and $I_{\mathrm{k}}$ is one of the
visible frames after occlusion detection with the BVH tree.
$S_{I_{\mathrm{k}}}(F_{\mathrm{i}},C_{\mathrm{m}})$ is the size of face
$F_{\mathrm{i}}$ in frame $I_{\mathrm{k}}$, which is labelled as class
$C_{\mathrm{m}}$. Another issue is to deal with invisible faces as illustrated
in Fig. 4. To address this problem, we adjust the smoothing term
$E_{\mathrm{s}}$ in our MRF-based energy function Eq. 13. The class
probability of each face is initialized with an uniform distribution to all
classes, which means
$E_{\mathrm{D}}\left(C_{\mathrm{1}}\right)=E_{\mathrm{D}}\left(C_{2}\right)=\ldots=E_{\mathrm{D}}\left(C_{\mathrm{N}}\right)=\frac{1}{N}$.
Then let the smoothing term $E_{\mathrm{S}}$ judge, which class the face gets.

##  VI EXPERIMENTAL EVALUATION

We evaluate our 3D reconstruction, texturing and semantic mapping system on: a
synthetic dataset from LGSVL[9], the real world KITTI dataset [10] and a
dataset recorded with our experimental vehicle Bertha One [11] equipped with a
Velodyne VLS-128 LiDAR and cameras.

###  VI-A 3D Geometric Reconstruction

In LGSVL [9], we simulate an experiment vehicle moving around a SUV at a
distance of $10$ m and a velocity of $5$ m/s to evaluate the reconstruction
accuracy of generated models. The experiment vehicle is equipped with a top-
mounted 128-beam LiDAR $2.3$ m above ground. The rays are specialized
according to the Velodyne VLS-128 calibration with a frequency of $10$ Hz.
Additionally, we add white noise to the simulated ranges with a standard
deviation of $1$ cm. In this experiment, we use a voxel size of $2.5$ cm and a
truncation distance of $\varepsilon=10$ cm, $\varepsilon=30$ cm or an adaptive
truncation distance range $\varepsilon_{\mathrm{apt}}:[10,30]$ cm. The results
in Fig. 5 shows that a large $\varepsilon$ ensures the reconstructed surface’s
completeness, while a small $\varepsilon$ retains much model details. Fig. 6
illustrates the quantitative accuracy evaluation of the reconstructed models.
It shows that the reconstructed surface using adaptive TSDF is smoother and
more accurate than the surface generated by standard TSDF. In Fig. 6(b), the
maximum distance error between the reconstructed surfaces and ground truth is
$11$ cm, while in Fig. 6(c) only $6$ cm.

(a) $\varepsilon=10$ cm

(b) $\varepsilon=30$ cm

(c) $\varepsilon_{\mathrm{apt}}:[10,30]$ cm

Figure 5:  3D model of a SUV with a voxel size of $2.5$ cm.

Besides that, we also evaluate our adaptive TSDF-based 3D reconstruction on
the real world KITTI dataset. As shown in Fig. 7, by benefiting from our
adaptive data fusion strategy the reconstructed model in Fig. 7(b) has a
higher resolution and contains more details. Furthermore, to evaluate the
effect of voxel size on the reconstruction’s quality, we design four
experiments with a voxel size of $5$ cm, $10$ cm, $15$ cm, and $20$ cm using
our adaptive TSDF-based algorithm on the dataset recorded with our experiment
vehicle. The truncation distance $\varepsilon_{\mathrm{apt}}$ is dynamically
computed in the range of $[40,80]$ cm. The results in Fig. 8 show that a
smaller voxel size can not significantly improve the quality of the
reconstructed models because of the measurement errors of the LiDAR sensor and
the slightly inaccurate sensor poses. However, a too large voxel size is not
friendly to smaller objects, as illustrated in Fig. 8(c) and Fig. 8(d). Since
a small voxel size means a large amount of faces, it is necessary to choose a
suitable voxel size according to different tasks and sensor specifics. In the
rest of this chapter, if without special declarations, the voxel size is set
as $10$ cm and $\varepsilon_{\mathrm{apt}}$ is dynamically computed in the
range of $[40,80]$ cm.

(a) Error [cm]

(b) $\varepsilon=30$ cm

(c) $\varepsilon_{\mathrm{apt}}:[10,30]$ cm

Figure 6:  Accuracy evaluation of reconstructed models.

(a) $\varepsilon=80$ cm

(b) $\varepsilon_{\mathrm{apt}}:[40,80]$ cm

Figure 7:  3D reconstruction on the KITTI dataset with a voxel size of $10$ cm
and different truncation distance $\varepsilon$.

(a) Voxel size: $5$ cm

(b) Voxel size: $10$ cm

(c) Voxel size: $15$ cm

(d) Voxel size: $20$ cm

Figure 8:  3D reconstruction on our dataset with different voxel size and an
adaptive truncation distance $\varepsilon_{\mathrm{apt}}:[40,80]$ cm.

(a) Mesh texturing

(b) Without color adjustment

(c) Vignetting coorection

(d) With color adjustment

Figure 9:  Model texturing and color optimization.

(a)

(b)

(c)

(d)

Figure 10:  The situations presented in (a) and (b) illustrate the wrong
textures caused by occlusions. (c) and (d) are the corrected textures after
the photo consistency check.

###  VI-B Texture Mapping

To evaluate the texture mapping process, we use the dataset recorded with our
experimental vehicle. As argued previously, textures obtained from the view
selection contain strong color discontinuities and wrong texture patches
caused by occlusions, as shown in Fig. 9(b). To handle the vignetting issue of
image pixels, we perform vignetting correction for images as a preprocessing
step. Fig. 9(c) shows that the vignetting correction algorithm can
significantly improve the texture’s quality. Further, the effect of the photo
consistency check is shown in Fig. 10. In Fig. 9(d) we show the texturing
result with the global color adjustment technique.

###  VI-C Semantic Mapping

Basically, in order to create accurate semantic mapped models, the voxel size
for the 3D geometric reconstruction should be as fine as possible. Road
markings such as dashed lines and arrows need extremely fine voxel size to be
reconstructed. As shown in the Fig. 11, a voxel size of $5$ cm is a good
choice although it results a large number of faces. Using bigger voxel sizes,
it is worse to recognize the arrow markings with a reasonable accuracy.
Secondly, we evaluate the MRF-based fusion approach compared to a naive
method, which only consider the data term and chose the label with the highest
probability. As illustrated in Fig. 12, the edge of the road markings are
reconstructed smoother using the MRF-based method with less outliers. Another
advantage of the MRF-based approach is to estimate the semantic class for the
invisible faces. As shown in Fig. 13(b), all the semantic class of the
invisible faces are greatly estimated with the help of the smoothing term
$E_{\mathrm{S}}$ presented in Eq. 13.

(a) Voxel size:$5$cm.

(b) Voxel size:$10$cm.

(c) Voxel size:$15$cm.

Figure 11: Road marking with different voxel sizes.

(a) Data term only.

(b) Data term + smoothing term.

Figure 12: MRF in comparison to naive approach.

(a) Original mesh map with white faces.

(b) Mesh map processed with MRF.

Figure 13: Elimination of invisible white faces with MRF.

##  VII CONCLUSION AND FUTURE WORK

In this work, we present a large-scale 3D reconstruction, texturing and
semantic mapping system. For the 3D reconstruction, an adaptive TSDF is
proposed to tackle the heterogeneous density of LiDAR point clouds. The
texture mapping procedure using an optimal image patch selection and color
optimization techniques has a high efficiency and robustness. Using a MRF-
based method, geometric models can be accurate and seamless semantic textured.

By applying our approach, several applications for autonomous driving are
enabled. The accurately textured 3D models can be used as a realistic
simulation environment. Compared to synthetic simulators, our models are more
similar to the real world and can accelerate the algorithm developing and
testing sufficiently. Besides that, the semantic mapped models can be used to
generate training data automatically for deep learning algorithms. In
addition, it is also very helpful for the automatic generation of semantic HD
maps for automated driving vehicles.

As future works we focus on: Adapting the voxel size for the reconstruction to
different semantic classes. Extracting semantic HD maps from the semantic
models automatically for autonomous driving. Generating training data for deep
learning approaches using the semantic models.

## Acknowledgment

This research is accomplished within the project “UNICARagil”. We acknowledge
the financial support for the project by the Federal Ministry of Education and
Research of Germany (BMBF).

## References

  * [1] Shahram Izadi et al.  “KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera”  In _Proceedings of the 24th annual ACM symposium on User interface software and technology_ , 2011, pp. 559–568
  * [2] Thomas Whelan, Michael Kaess, Maurice Fallon, Hordur Johannsson, John Leonard and John McDonald  “Kintinuous: Spatially extended kinectfusion”, 2012
  * [3] Matthias Nießner, Michael Zollhöfer, Shahram Izadi and Marc Stamminger  “Real-time 3D reconstruction at scale using voxel hashing”  In _ACM Transactions on Graphics (ToG)_ 32.6 ACM New York, NY, USA, 2013, pp. 1–11
  * [4] Olaf Kähler, Victor Adrian Prisacariu, Julien P.. Valentin and David W. Murray  “Hierarchical Voxel Block Hashing for Efficient Integration of Depth Images”  In _IEEE Robotics and Automation Letters_ 1.1, 2016, pp. 192–197
  * [5] Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi and Christian Theobalt  “Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration”  In _ACM Transactions on Graphics (ToG)_ 36.4 ACM New York, NY, USA, 2017, pp. 1
  * [6] Tilman Kühner and Julius Kümmerle  “Large-Scale Volumetric Scene Reconstruction using LiDAR”  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 6261–6267  IEEE
  * [7] Victor Lempitsky and Denis Ivanov  “Seamless mosaicing of image-based texture maps”  In _2007 IEEE conference on computer vision and pattern recognition_ , 2007, pp. 1–6  IEEE
  * [8] Michael Waechter, Nils Moehrle and Michael Goesele  “Let there be color! Large-scale texturing of 3D reconstructions”  In _European conference on computer vision_ , 2014, pp. 836–850  Springer
  * [9] Guodong Rong et al.  “Lgsvl simulator: A high fidelity simulator for autonomous driving”  In _2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)_ , 2020, pp. 1–6  IEEE
  * [10] Andreas Geiger, Philip Lenz and Raquel Urtasun  “Are we Ready for Autonomous Driving? The KITTI Vision Benchmark Suite”  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2012
  * [11] Ö.. Taş et al.  “Making Bertha Cooperate–Team AnnieWAY’s Entry to the 2016 Grand Cooperative Driving Challenge”  In _IEEE Transactions on Intelligent Transportation Systems_ 19.4, 2018, pp. 1262–1276  DOI: 10.1109/TITS.2017.2749974
  * [12] Jean-Emmanuel Deschaud  “IMLS-SLAM: scan-to-model matching based on 3D data”  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , 2018, pp. 2480–2485
  * [13] Michiel Vlaminck, Hiep Luong, Werner Goeman and Wilfried Philips  “3D scene reconstruction using Omnidirectional vision and LiDAR: A hybrid approach”  In _Sensors_ 16.11 Multidisciplinary Digital Publishing Institute, 2016, pp. 1923
  * [14] Frank Neuhaus, Tilman Koß, Robert Kohnen and Dietrich Paulus  “Mc2slam: Real-time inertial lidar odometry using two-scan motion compensation”  In _German Conference on Pattern Recognition_ , 2018, pp. 60–72
  * [15] Thomas Whelan, Stefan Leutenegger, R Salas-Moreno, Ben Glocker and Andrew Davison  “ElasticFusion: Dense SLAM without a pose graph”  In _Robotics: Science and Systems_ , 2015
  * [16] Hanspeter Pfister, Matthias Zwicker, Jeroen Van Baar and Markus Gross  “Surfels: Surface elements as rendering primitives”  In _Proceedings of the 27th annual conference on Computer graphics and interactive techniques_ , 2000, pp. 335–342
  * [17] Brian Curless and Marc Levoy  “A volumetric method for building complex models from range images”  In _Proceedings of the 23rd annual conference on Computer graphics and interactive techniques_ , 1996, pp. 303–312
  * [18] Luis Roldão, Raoul Charette and Anne Verroust-Blondet  “3D Surface Reconstruction from Voxel-based Lidar Data”  In _2019 IEEE Intelligent Transportation Systems Conference (ITSC)_ , 2019, pp. 2681–2686  IEEE
  * [19] M. Callieri, P. Cignoni, M. Corsini and R. Scopigno  “Masked photo blending: Mapping dense photographic data set on high-resolution sampled 3D models”  In _Computers & Graphics_ 32.4 Elsevier BV, 2008, pp. 464–473  DOI: 10.1016/j.cag.2008.05.004
  * [20] Lazaros Grammatikopoulos, Ilias Kalisperakis, George Karras and Elli Petsa  “Automatic multi-view texture mapping of 3D surface projections”  In _3D Virtual Reconstruction & Visualization of Complex Architectures (3D Arch ’07_, 2007, pp. 12–13
  * [21] Renato F Salas-Moreno, Richard A Newcombe, Hauke Strasdat, Paul HJ Kelly and Andrew J Davison  “Slam++: Simultaneous Localisation and Mapping at the Level of Objects”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2013, pp. 1352–1359
  * [22] John McCormac, Ronald Clark, Michael Bloesch, Andrew Davison and Stefan Leutenegger  “Fusion++: Volumetric Object-level Slam”  In _2018 international conference on 3D vision (3DV)_ , 2018, pp. 32–41  IEEE
  * [23] Margarita Grinvald et al.  “Volumetric Instance-aware Semantic Mapping and 3D Object Discovery”  In _IEEE Robotics and Automation Letters_ 4.3 IEEE, 2019, pp. 3037–3044
  * [24] Qiaojun Feng, Yue Meng, Mo Shan and Nikolay Atanasov  “Localization and Mapping using Instance-specific Mesh Models”  In _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2019, pp. 4985–4991  IEEE
  * [25] Lachlan Nicholson, Michael Milford and Niko Sünderhauf  “Quadricslam: Dual quadrics from object detections as landmarks in object-oriented slam”  In _IEEE Robotics and Automation Letters_ 4.1 IEEE, 2018, pp. 1–8
  * [26] Jean-Baptiste Bordes, Philippe Xu, Franck Davoine, Huijing Zhao and Thierry Denoeux  “Information Fusion and Evidential Grammars for Object Class Segmentation”  In _Fifth IROS Workshop on Planning, Perception and Navigation for Intelligent Vehicles_ , 2013, pp. 165–170
  * [27] Antoni Rosinol, Marcus Abate, Yun Chang and Luca Carlone  “Kimera: an Open-source Library for Real-time Metric-semantic Localization and Mapping”  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 1689–1696  IEEE
  * [28] Helen Oleynikova, Zachary Taylor, Marius Fehr, Roland Siegwart and Juan Nieto  “Voxblox: Incremental 3d Euclidean Signed Distance Fields for On-board MAV Planning”  In _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2017, pp. 1366–1373  IEEE
  * [29] Markus Herb, Tobias Weiherer, Nassir Navab and Federico Tombari  “Crowd-sourced Semantic Edge Mapping for Autonomous Vehicles”  In _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2019, pp. 7047–7053  IEEE
  * [30] Markus Herb, Tobias Weiherer, Nassir Navab and Federico Tombari  “Lightweight Semantic Mesh Mapping for Autonomous Vehicles”  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , 2021, pp. 6732–3738  IEEE
  * [31] Tero Karras  “Maximizing parallelism in the construction of BVHs, octrees, and k-d trees”  In _Proceedings of the Fourth ACM SIGGRAPH/Eurographics conference on High-Performance Graphics_ , 2012, pp. 33–37
  * [32] NVIDIA, Péter Vingelmann and Frank H.P. Fitzek  “CUDA, release: 10.2.89”, 2020  URL: https://developer.nvidia.com/cuda-toolkit
  * [33] Cédric Allène, Jean-Philippe Pons and Renaud Keriven  “Seamless image-based texture atlases using multi-band blending”  In _2008 19th international conference on pattern recognition_ , 2008, pp. 1–4  IEEE
  * [34] Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler and Michael Cohen  “Unstructured lumigraph rendering”  In _Proceedings of the 28th annual conference on Computer graphics and interactive techniques_ , 2001, pp. 425–432
  * [35] Torsten Rohlfing  “Single-Image Vignetting Correction by Constrained Minimization of log-Intensity Entropy”, 2012  DOI: 10.6084/M9.FIGSHARE.96145