# Extrinsic Calibration and Verification of Multiple Non-overlapping Field of
View Lidar Sensors

Sandipan Das1,2, Navid Mahabadi2, Addi Djikic2, Cesar Nassir2, Saikat
Chatterjee1, Maurice Fallon3 1 KTH EECS, Sweden<EMAIL_ADDRESS>
2 Scania CV AB, Sweden. {sandipan.das,navid.mahabadi,addi.
<EMAIL_ADDRESS>
3Oxford Robotics Institute, UK<EMAIL_ADDRESS>
###### Abstract

We demonstrate a multi-lidar calibration framework for large mobile platforms
that jointly calibrate the extrinsic parameters of non-overlapping Field-of-
View (FoV) lidar sensors, without the need for any external calibration aid.
The method starts by estimating the pose of each lidar in its corresponding
sensor frame in between subsequent timestamps. Since the pose estimates from
the lidars are not necessarily synchronous, we first align the poses using a
Dual Quaternion (DQ) based Screw Linear Interpolation. Afterward, a Hand-Eye
based calibration problem is solved using the DQ-based formulation to recover
the extrinsics. Furthermore, we verify the extrinsics by matching chosen lidar
semantic features, obtained by projecting the lidar data into the camera
perspective after time alignment using vehicle kinematics. Experimental
results on the data collected from a Scania vehicle [$\sim$ 1 Km sequence]
demonstrate the ability of our approach to obtain better calibration
parameters than the provided vehicle CAD model calibration parameters. This
setup can also be scaled to any combination of multiple lidars.

##  I Introduction

Multi-sensor calibration is one of the most important prerequisites for
successful mobile autonomy in the real-world. To have complete $360^{\circ}$
sensing coverage, mobile vehicles often have multiple sensors around the
vehicle, and in fact there can be extra sensors for redundancy and safety
reasons (see Fig. 1). This complexity poses a challenge when calibrating the
sensors to an acceptable level before perceiving the environment coherently.

Recent advances in lidar sensing have promoted their usage in the sensing
stack of autonomous vehicles including wider Field-of-View (FoV), longer range
and higher resolution. Since, lidar sensors can provide range information
along with semantics [1] they have become a key component in the perception
pipeline of autonomous platforms.

However, significant calibration degradation can occur due to issues such as
thermal expansion/contraction of the lidar mount relative to the platform;
navigating on rough terrain (as found in a mine) or the general effect of wear
and tear over the long run. Hence, it is imperative to be able to re-
calibrate, adopt and update the lidar extrinsic parameters in situ. Offline
calibration or target-based lidar calibration processes [2, 3, 4, 5, 6, 7] can
be time-intensive and costly for large mobile platforms. Marker-less extrinsic
calibration with state optimization [8] or trajectory optimization [9, 10, 11]
are more desirable because of their time efficiency and ease of usage by an
end-user.

Figure 1: Illustration of the Field of View (FoV) of two lidars and four
cameras positioned around the data collection vehicle. The scale factors of
the sensor ranges and the vehicle dimension are only for illustrative
purposes. The vehicle base frame $\mathtt{{B}}$, is located at the center of
the rear axle. The sensor frames of the 2 lidars are: $\mathtt{L}_{{F}|{R}}$,
where $F$ and $R$ are the front right and rear left lidars.

###  I-A Motivation

To create an efficient extrinsic calibration framework suitable for multiple
lidars we started with data collected using a Scania bus with similar sensor
FoVs to that shown in Fig. 1. In spite of the various available solutions for
multi-lidar calibration there were two major challenges. Problem 1) Although
the sensors were synchronized with PTP synchronization [12], the estimated
poses from the state estimation which were used for motion based calibration
were not necessarily time synchronized. Problem 2) Because of the limited FoV
overlap between the front right and the rear left lidar, we could not verify
the quality of the estimated calibration as it was not possible to inspect if
the point clouds expressed a consistent view of the surroundings.

Prior work has addressed the time alignment issue by correlating the angular
velocity norms of both pose signals [13]. Further calibration refinement has
been performed based on overlapping FoV [14, 15]. For non-overlapping FoV
scenario, calibration verification was handled by matching features in a
global map [16] in the visual domain.

In our work, to address Problem 1) we have used dual quaternion (DQ) [17]
based time alignment of the poses using screw linear interpolation (ScLERP)
[18] as it provides the shortest path between pose estimates on
$\mathrm{SE}(3)$ manifold. We have tackled Problem 2) by evaluating the
correspondences of selected similar features observed from non-overlapping FoV
lidars at different timestamps, analyzed based on vehicle kinematics. We have
additionally used cameras, which were placed below the corresponding lidars as
shown in Fig. 2 to improve the quality of feature selection from point clouds.

###  I-B Contribution

Our work is motivated by the Hand-Eye calibration method of Tsai et al [9].
Our specific contributions are additional procedures which increase the
accuracy and robustness of the extrinsic calibration process as follows:

  * •

A DQ based prior time alignment of the poses based on ScLERP before applying
the DQ based Hand-Eye calibration.

  * •

An additional verification method using semantic matching to evaluate the
extrinsic calibration quality for non-overlapping FoV lidars.

  * •

Experimental results and verification using data collected with a Scania bus
mounted with the sensor setup as shown in Fig. 2, with FoV schematics similar
to Fig. 1.

##  II Related Work

There are several techniques to obtain the extrinsic calibration parameters of
the multi-sensor platform. This section categorizes those methods and briefly
describes the relevant work in each category.

###  II-A SLAM-based methods

Simultaneous Localization and Mapping (SLAM) approaches normally estimate the
extrinsic parameters by considering it as a state variable in the pose graph
optimization or bundle adjustment optimization in visual SLAM. Kummerle et al
[8] extended the SLAM hyper graph formulation to handle calibration and
kinematic parameters without prior knowledge of the environment by including
wheels radii and sensor position as state variables. Jones et al [19],
proposed a visual inertial SLAM system that localized the sensor positions in
the map frame to estimate the calibration parameter between a camera and an
inertial measurement unit (IMU). This approach relied heavily on a robust SLAM
solver along with a rich motion sequence. Similarly, Carrera et al [16]
presented a visual SLAM algorithm for multi-camera extrinsic calibration by
building a global landmark map from each camera and then fusing the maps by
means of feature correspondences using similarity transformation bundle
adjustment. This approach required driving in a circular path as the basic
prerequisite for calibration to ensure the observability of similar features.
To address this, Heng et al [20] extended the prior work by incorporating
wheel odometry and formulating a least-squares problem. Their method was
restricted to ground robots with odometry. In contrast, their recent work [21]
had no odometry requirement and utilized visual SLAM and bundle adjustment
using a multi-camera system where each sensor built its own map. Map merging
was then performed using a hand-eye calibration method and the results were
utilized in pose graph optimization to estimate the extrinsics among sensors.

###  II-B Appearance-based methods

The recent appearance based approaches can be clustered into two categories;
first, those using artificial markers or a known calibration target (e.g. a
checkerboard), and second automatic geometric feature extraction methods which
do not rely on markers. Appearance based methods in general could be seen as
registration problems where artificial markers or geometrical features are
employed. Gao et al [4], for instance, recovered the calibration parameters
between multiple lidars by manually observing reflective landmarks in the
scene. Zhou et al [6], presented a method to calibrate lidar and camera sensor
modalities using a checkerboard by combining 3D line and plane features.
Similarly, in [2], [3] and [7], the authors proposed lidar to camera
calibration by utilizing plane feature registration using a chessboard.
Automatic data extraction from sensors makes the usage of artificial markers
redundant. Choi et al. [5] determined the relative extrinsic parameters
between 2D-lidars by extracting orthogonal planes and utilizing the co-planar
features of the scan points. Geometric features such as points, lines and
planes were extracted in [22] and matched to calibrate a dual lidar system.
Levinson et al [23] presented an online camera to lidar calibration optimizer
by aligning edge points from camera images and corresponding lidar points.
However, for all these methods to work the general assumption was that the
features were visible through an overlapping FoV correspondence.

###  II-C Motion-based methods

Incremental motion based approaches recover the calibration parameters using
the well-known hand-eye calibration formulation, [24]. The calibration was
estimated between a gripper (hand) and a camera (eye) by moving the gripper in
some trajectory, while exploiting the fact that the transformation between the
camera and the gripper was always the same. More recent research has focused
on extending the work to multi-sensor modalities by aligning the motion of
each individual sensor. Brookshire et al [25] utilized the DQ method [26] to
recover the relative transformation of two camera sensors mounted on a rigid
body. The same principle was also applied by Heng et al in camodocal [20]. A
more generic sensor configuration was proposed by Taylor et al. [10] for
multi-modal sensor arrays calibration including 3D-lidar, IMU and camera by
motion combination of sensors using hand-eye formulation. They also provided
the uncertainty of the calibration. The same authors extended the work to
incorporate the sensor’s accuracy reading and estimated the timing offsets
between the sensors as well [11]. Moreover, overlapping fields of view was
applied to further enhance the calibration when applicable. Jiao et al [14]
exploited the hand-eye calibration of a multi-lidar system by motion alignment
to recover an initial estimation of the extrinsic. A refinement process using
point-to-plane features was then used to further enhance the calibration
parameters. In their recent work M-LOAM [15], they used similar techniques
with edge and planar features for the calibration refinements.

Both SLAM based approach and motion based approach can be used to solve the
extrinsic calibration problem for our system. In our work we adapted a motion
based method to solve the extrinsic calibration problem due to the accuracy of
the poses estimated via LOAM [27] based approaches. We performed the
trajectory alignment as an initialization step similar to [14]. With this
initialization, we first did the time alignment based on DQ based ScLERP.
After that we recovered the individual sensor poses and formulated the problem
as a Hand-Eye calibration problem. The Hand-Eye problem was solved using a DQ
based closed form solution [26]. Finally, we developed a verification method
for evaluating the quality of the calibration by semantic matching of the
chosen lidar features.

##  III Problem Statement

###  III-A Reference Frames

Figure 2: Reference frames conventions for our vehicle platform. The world
frame $\mathtt{W}$ is a fixed frame, while the base frame $\mathtt{{B}}$, as
shown in Fig. 1, is located at the rear axle center of the vehicle. Each
sensor unit contains the two optical frames $\mathtt{C}$, an IMU frame,
$\mathtt{I}$, and lidar frame $\mathtt{L}$.

We start by describing the necessary notation and reference frames used in our
system according to the convention by Furgale [28]. The reference frames are
shown in Fig. 2. The vehicle base frame $\mathtt{{B}}$ is located on the
center of the rear-axle of the vehicle. Sensor readings from lidars, cameras
and IMUs are represented in their respective sensor frames as
$\mathtt{L}_{k}$, $\mathtt{C}_{k}$ and $\mathtt{I}_{k}$ respectively, where
$k$ denotes the location of the sensor in the vehicle.

For example, $\mathtt{L}_{F}$ and $\mathtt{L}_{R}$ represent the front right
and rear left lidar frames. In our system we calibrated $\mathtt{L}_{R}$ with
respect to $\mathtt{L}_{F}$ and used an known transformation between
$\mathtt{{B}}$ and $\mathtt{L}_{F}$ from the vehicle CAD files. After
calibration, we converted the extrinsics to vehicle base frame for
verification and visualization purpose discussed in Sec. IV-C.

###  III-B Pose Estimation Frames

For pose estimation of individual lidars we computed the relative
transformation between subsequent lidar readings in its corresponding sensor
frame. Conscutive estimates were available at times $t_{i-1}$ and $t_{i}$,
denoted as ${\mathbf{T}}{{}_{\mathtt{L}_{i-1}\mathtt{L}_{i}}}$, where
$\mathbf{T}=\left[\begin{array}[]{ll}\mathbf{R_{3\times 3}}&\mathbf{t_{3\times
1}}\\\ \mathbf{0}^{\top}&1\end{array}\right]\in\mathrm{SE}(3)$ represents a
homogeneous transformation consisting of rotation,
$\mathbf{R}\in\mathrm{SO}(3)$ and translation $\mathbf{t}\in\mathbb{R}^{3}$.
In our system we denote the iterative lidar pose transforms as
${\mathbf{T}}{{}_{F_{i-1}F_{i}}}$ and ${\mathbf{T}}{{}_{R_{i-1}R_{i}}}$. For
brevity of discussion, we will denote them as ${\mathbf{T}}{{}_{F}}$ and
${\mathbf{T}}{{}_{R}}$ respectively. In future, we would extend this approach
to solve the multi-lidar calibration scenario as the data collection vehicle
is equipped with more than two lidars, as shown in Fig. 2.

###  III-C Problem Formulation

Figure 3: Transformation in between the lidars $\mathtt{L}_{F}$ and
$\mathtt{L}_{R}$ in between times $t_{i-1}$ and $t_{i}$.

Given a set of point clouds, ${\cal P}_{\mathtt{L}_{k:1(1){\cal
K}}}^{t_{i:1(1){\cal N}}}$, from ${\cal K}$ different lidars in a time
sequential order $t_{1}\dots t_{\cal N}$, in their respective sensor frames,
we wish to recover the extrinsics between the lidars as,
${\mathbf{T}}{{}_{\mathtt{L}_{a}\mathtt{L}_{b}}}$, where $a,b:1(1){\cal K}$
and $a\neq b$. We assume the lidars are fixed rigidly to the vehicle and all
the other lidars are calibrated with respect to a reference lidar. The multi-
lidar calibration task is formulated in terms of a Hand-eye [9] calibration
problem, where the extrinsics are recovered by matching the homogeneous
transformations of the multiple lidars. For brevity of discussion we will
consider two non-overlapping FoV lidars, the front right and rear left, and
aim to recover the extrinsics. However, it can be extended to multi-lidar
setup as well. With regards to the Hand-Eye method, we can recover the
extrinsic calibration in between the lidars, as shown in Fig. 3, by using the
following equation,

${\mathbf{T}}{{}_{F}}{\mathbf{T}}{{}_{FR}}={\mathbf{T}}{{}_{FR}}{\mathbf{T}}{{}_{R}}$
(1)

which can be further decomposed into,

$\displaystyle\mathbf{R}_{F}\mathbf{R}{{}_{FR}}$
$\displaystyle=\mathbf{R}{{}_{FR}}\mathbf{R}{{}_{R}}$ (2)
$\displaystyle(\mathbf{R}{{}_{F}}-{\mathbf{I}}_{3\times 3})\mathbf{t}_{FR}$
$\displaystyle=\mathbf{R}{{}_{FR}}\mathbf{t}{{}_{R}}-\mathbf{t}{{}_{F}}.$

Multiple solutions including iterative and closed-form methods along with
their benefits and drawbacks [29] have been discussed to solve the Hand-Eye
calibration eq. 1 and 2. The solution methods can be further categorized as
separable and simultaneous solutions. The separable solutions solve for
$\mathbf{R_{3\times 3}}$ and $\mathbf{t_{3\times 1}}$ separately, whereas the
simultaneous solutions solve them jointly. In our work, we have adapted a
simultaneous approach using DQ [26].

##  IV Methodology

###  IV-A Initialization

Pose estimation of the individual sensors was computed using the VILENS
estimator [30]. However, instead of using a feature-based configuration, our
configuration fused ICP-based point cloud matching [31] with IMU pre-
integration using GTSAM [32, 33] to estimate the motion of the individual
lidars and to correct for motion distortion effects. We used the set of
estimated poses $\mathbf{t_{F}}$ and $\mathbf{t_{R}}$ recovered at ${\cal N}$
time iterations to estimate an initial ${\mathbf{T}}{{}_{FR}}$ by solving the
following least-squares optimization,

$\mathbf{R}^{*},\mathbf{t}^{*}=\operatorname*{arg\,min}_{\mathbf{R},\mathbf{t}}\|\mathbf{R}\mathbf{t}_{F}+\mathbf{t}-\mathbf{t}_{R}\|^{2}.$
(3)

We used the closed form solution proposed by Umeyama [34] to recover the
initial optimal $\mathbf{R}^{*}$ and $\mathbf{t}^{*}$ which gives similar
results as Horn et al. [35] in quaternion space. This initialization was
necessary for the DQ based optimizer to generate better results.

###  IV-B Dual Quaternion Based Hand-Eye Calibration

Homogeneous transformation matrices are orthogonal and due to floating point
errors, operations on them often result in matrices that need to be re-
normalized. Although, this can be achieved using the Gram-Schmidt method, this
is however slower than quaternion normalization. DQs are the natural extension
to quaternions; because, unlike quaternions, they can handle translation as
well in the same compact notion.

####  IV-B1 Preliminaries

We review DQ’s and its operations briefly; more details are available here
[36, p. 53–62]. DQ has the advantages of a quaternion such as gimbal lock
problem and unambiguous representation but with less intensive operations for
transformations. A DQ, $\mathbf{q}$, can be written as,

$\mathbf{q}=q_{r}+\varepsilon q_{d},$ (4)

where the real part $q_{r}$, and the dual part $q_{d}$, are quaternions.
$\varepsilon$ is the dual unit, defined by $\varepsilon^{2}=0$ and
$\varepsilon\neq 0$. The conjugate of a DQ, $\mathbf{q}$ is:

$\mathbf{q^{\star}}=q_{r}^{\star}+\varepsilon q_{d}^{\star},$ (5)

where, $q^{\star}=(q_{w},\text{-}q_{x},\text{-}q_{y},\text{-}q_{z})$, is
quaternion conjugate. Some operations of DQs, $\mathbf{q_{1}}$ and
$\mathbf{q_{2}}$ are as follows:

$\displaystyle\mathbf{q_{1}}+\mathbf{q_{2}}$
$\displaystyle=(q_{r}^{1}+q_{r}^{2})+\varepsilon(q_{d}^{1}+q_{d}^{2})$ (6)
$\displaystyle\mathbf{q_{1}}\otimes\mathbf{q_{2}}$
$\displaystyle=q_{r}^{1}\otimes q_{r}^{2}+\varepsilon(q_{r}^{1}\otimes
q_{d}^{2}+q_{d}^{1}\otimes q_{r}^{2})$ (7) $\displaystyle\|\mathbf{q}\|^{2}$
$\displaystyle=\mathbf{q}\otimes\mathbf{q}^{\star}$ (8)
$\displaystyle\mathbf{q^{-1}}$ $\displaystyle=\mathbf{q}^{\star},\quad\text{if
}\|\mathbf{q}\|^{2}=1,$ (9)

where, $\otimes$ and $\|\mathbf{\cdot}\|^{2}$ represent quaternion product and
norm of a DQ respectively. Given a transformation,
$\mathbf{T}(\mathbf{R},\mathbf{t})$ we can convert it to DQ form as,

$\mathbf{q(\mathbf{T})}=q(\mathbf{R})+\varepsilon\frac{1}{2}q(\mathbf{t})q(\mathbf{R})$
(10)

where, $q(\mathbf{R})$ is obtained from Euler-Rodrigues formula and
$q(\mathbf{t})=(0,t_{x},t_{y},t_{z})$. Conversely, we can convert a DQ,
$\mathbf{q}(q_{r},q_{d})$ to a homogeneous transformation form as,

$\mathbf{T}(\mathbf{q})=\left[\begin{array}[]{cc}\mathbf{R}(q_{r})&2q_{d}\otimes
q_{r}^{*}\\\ 0^{\top}&1\end{array}\right]$ (11)

where, $\mathbf{R}(q_{r})$ is obtained from Rodrigues’ rotation formula.

####  IV-B2 Pose alignment

Since, the estimated poses are not necessarily time synchronized we aligned
them using ScLERP [18], which is a generalization of SLERP [37] before solving
the Hand-Eye problem. The interpolation between two quaternions is given by
SLERP whereas, the interpolation between two DQs is given by ScLERP as,

$\displaystyle\text{ScLERP}(n;\mathbf{q_{1}},\mathbf{q_{2}})$
$\displaystyle=\mathbf{q_{1}}\otimes(\mathbf{q_{1}}^{-1}\otimes\mathbf{q_{2}})^{n}$
(12)

where, $n\in[0,1]$ and
$\mathbf{q}^{n}=\cos(\frac{n\theta}{2})+\mathbf{k}\cdot\sin(\frac{n\theta}{2})$.
Here, $\theta$ is the dual angle of rotation around the unit DQ, $\mathbf{k}$
with zero scalar part. This ensures synchronized lidar poses at all
timestamps, improving Hand-Eye calibration results.

####  IV-B3 Solving Hand-Eye Calibration

To solve the Hand-Eye calibration problem in DQ space, we initially converted
the lidar transformations in between subsequent timestamps in DQ formulation.
$\mathbf{q_{{\textstyle\mathstrut}F}}$ and
$\mathbf{q_{{\textstyle\mathstrut}R}}$ represent the front right and rear left
converted transformations in DQ space respectively, whereas, $\mathbf{q}$
represents the extrinsics in DQ formulation. Similar, to eq. 1, the screw
motion concatenation in DQ space yields,

$\mathbf{q_{{\textstyle\mathstrut}F}}=\mathbf{q}\mathbf{q_{{\textstyle\mathstrut}R}}\mathbf{q^{\star}}.$
(13)

From Screw Congruence Theorem [38] it has been shown only the vector parts in
the quaternion contribute to the derivation of unknown DQ, $\mathbf{q}$, as
the scalar parts are equal. The vector part $\vv{q}$, in a quaternion $q$ is,
$(0,\vv{q})$. With the derivations shown in [26], eq. 13 can be re-written as,

$\left[\begin{array}[]{cccc}\vv{q_{r}^{\text{F}}}-\vv{q_{r}^{\text{R}}}&{[\vv{q_{r}^{\text{F}}}+\vv{q_{r}^{\text{R}}}]_{\times}}&\mathbf{0}_{3\times
1}&\mathbf{0}_{3\times 3}\\\ &&&\\\
\vv{q_{d}^{\text{F}}}-\vv{q_{d}^{\text{R}}}&{[\vv{q_{d}^{\text{F}}}+\vv{q_{d}^{\text{R}}}]_{\times}}&\vv{q_{r}^{\text{F}}}-\vv{q_{r}^{\text{R}}}&{[\vv{q_{r}^{\text{F}}}+\vv{q_{r}^{\text{R}}}]_{\times}}\end{array}\right]\left[\begin{array}[]{l}q_{r}\\\
\\\ q_{d}\end{array}\right]=0,$

(14)

where, $[\>\cdot\>]_{\times}$ represents a skew-symmetric matrix and the first
term, $\mathbf{S}$ is a 6$\times$8 matrix and the second term, is a 8$\times$1
vector. Based on the data collected at ${\cal N}$ different timestamps we can
construct a 6${\cal N}\times$8 matrix,
$\mathbf{M}=(\mathbf{S}^{\text{T}}_{1}\quad\mathbf{S}_{2}^{\text{T}}\quad\dots\quad\mathbf{S}^{\text{T}}_{\cal
N})^{\text{T}}$. However, as this is not a full-rank matrix, the last two
right singular vector obtained from SVD of $\mathbf{M}$ spans the nullspace of
$\mathbf{M}$, which is used to recover $\mathbf{q}$. We also recorded
transformations in non-planar trajectories as mentioned in [26] for recovering
solutions of eq. 14 and solved it within a certain batch size, instead of
applying SVD on the whole 6${\cal N}\times$8 matrix at once.

###  IV-C Calibration Verification

To verify the accuracy of extrinsics, we associated extracted lidar semantic
features between the lidars. The semantic features from lidar were extracted
by associating the projected point clouds into the segmented camera images.
The image segmentation was performed using a trained Efficient-Net model. We
used an unsupervised method to calculate a time offset based on vehicle
kinematics which described the observation delay between the lidars,
corresponding to similar features as represented in Fig. 4.

Figure 4: Semantic feature matching in between $L_{F}$ and $L_{R}$ lidars
viewed at $\delta t=t_{j}-t_{i}$ time apart so that FoVs correspond to similar
region. First, lidar $L_{F}$ detects a set of features at time $t_{i}$ and
$\delta t$ time later $L_{R}$ observes the previously observed set of features
(by $L_{F}$) at time $t_{j}$.

####  IV-C1 Efficient-Net for semantic segmentation

The author of EfficientNet [39] demonstrated that model scaling can be
achieved by carefully balancing network depth, width, and resolution, leading
to a better performance with the fixed computation resources. Based on this
study, we used a modified version of EfficientNet to semantically segment the
camera images. The modified network architecture backbone is shown in Table I,
where each row describes a stage $i$ with $\hat{L}_{i}$ layers, with input
resolution $\langle\hat{H}_{i},\hat{W}_{i}\rangle$ and output channels
$\hat{C}_{i}$. MBConv layers are mobile inverted bottlenecks from the
MobileNetV2 architecture [40] where squeeze-and-excitation optimization have
also been added from [41]. The segmentation head consists of convolutional
layers and bilinear interpolation to upsample the network output to its
original input resolution.

Stage | Operator | Resolution | #Channels | #Layers
---|---|---|---|---
$i$ | $\hat{\mathcal{F}}_{i}$ | $\hat{H}_{i}\times\hat{W}_{i}$ | $\hat{C}_{i}$ | $\hat{L}_{i}$
1 | Conv3x3 | $640\times 1024$ | 32 | 1
2 | MBConv1, k3x3 | $320\times 512$ | 16 | 1
3 | MBConv6, k3x3 | $160\times 256$ | 24 | 2
4 | MBConv6, k5x5 | $80\times 128$ | 40 | 2
5 | MBConv6, k3x3 | $40\times 64$ | 80 | 3
6 | MBConv6, k5x5 | $40\times 64$ | 112 | 3
7 | MBConv6, k5x5 | $20\times 32$ | 192 | 4
8 | MBConv6, k3x3 | $20\times 32$ | 320 | 1
9 | Segmentation Head | $640\times 1024$ | 24 | -

TABLE I:  Modified EfficientNet-B0 baseline network architecture.

####  IV-C2 Association of semantic features

We chose to match curb features between the lidars because the lidar
projection error was stable across frames for the curb semantics. First, we
sampled an arbitrary number of curb semantic points from the $L_{F}$ lidar by
projecting the points into camera perspective, for a specific duration to
extract a segment of the curb. Then based on the calculated time offset
obtained from vehicle kinematics, we sampled curb semantics from the $L_{R}$
lidar. Since the point clouds of both lidars were represented in the base
frame, we expected that the same set of features should be observable from
both lidars. Finally we calculated the root mean square error (RMSE) between
the commonly observed curb features. The association between the feature
points was established using tri-linear interpolation on the sorted points. We
therefore sampled several curb features using the aforementioned technique and
statistically compared RMSE before and after the calibration. Curb extraction
and association results are presented in Sec. V-B2.

##  V Experimental Results

###  V-A Dataset

We collected the data from a Scania autonomous bus platform consisting of 4
corner lidars and 8 cameras. For ground truth pose estimates (GT) we used data
from a Novatel GNSS. All the sensing data was synchronized using PTP
synchronization.

###  V-B Results

####  V-B1 Pose Estimation and DQ based Hand-Eye calibration

We estimate the poses of the $L_{F}$ and $L_{R}$ lidars and convert them to
the vehicle base frame for comparison. The pose estimates are shown in Fig. 5
and Fig. 6 along with the GT for the initial calibration parameters from the
vehicle model.

Figure 5: Results of the odometry estimation from $L_{F}$ (blue) and $L_{R}$
(magenta) lidars plotted in Google Maps with respect to the GT (green) with
initial calibration parameters. The GT was generated using Novatel GNSS.

Figure 6: Pose estimation for $L_{F}$ and $L_{R}$ lidars with respect to GT
with initial calibration estimation.

The relative pose error (RPE) and absolute pose error (APE) metrics are used
to evaluate the local and global consistency respectively of the estimated
trajectory of $L_{R}$ lidar with respect to $L_{F}$ lidar, before and after
calibration with the Hand-Eye method after time alignment. The ScLERP based
time alignment increased the consistency in between the DQ poses for the
scalar part match which was a pre-condition for deriving eq. 14. We re-
estimate the poses of the $L_{R}$ lidar after putting in new calibration
parameters.

RPE compares the difference between the individual relative poses within two
timestamps $t_{i}$ and $t_{j}$ as,
$RPE_{i,j}=({\mathbf{T}}{{}_{\mathtt{L}_{F}(t_{i})}^{-1}}{\mathbf{T}}{{}_{\mathtt{L}_{F}(t_{j})}})^{-1}({\mathbf{T}}{{}_{\mathtt{L}_{R}(t_{i})}^{-1}}{\mathbf{T}}{{}_{\mathtt{L}_{R}(t_{j})}})$.
APE is based on the absolute relative pose between two poses
${\mathbf{T}}{{}_{\mathtt{L}_{F}(t_{i})}},{\mathbf{T}}{{}_{\mathtt{L}_{R}(t_{i})}}\in\mathrm{SE}(3)$
at timestamp $t_{i}$ as,
$APE_{i}={\mathbf{T}}{{}_{\mathtt{L}_{F}(t_{i})}^{-1}}{\mathbf{T}}{{}_{\mathtt{L}_{R}(t_{i})}}$.

Figure 7: RPE between $L_{F}$ and $L_{R}$ lidars before and after calibration.

Figure 8: APE between $L_{F}$ and $L_{R}$ lidars before and after calibration.

We compare the RPE, Fig. 7 and APE, Fig. 8 between $L_{F}$ and $L_{R}$ lidar
trajectories before and after calibration, which shows that we recover better
calibration estimates in between the $L_{F}$ and $L_{R}$ lidar.

####  V-B2 Semantic verification

For semantic verification we train the EfficientNet model on a diversified
scenario set (city driving, snowy conditions, dessert area, suburban area).
The overall training accuracy is 79.10% and the mean IoU (Intersection over
Union) is 0.495. Whereas, for curb class the accuracy is 67.20% and the IoU is
0.557. The model performance is correlated with the amount of data it is
trained on. We extract the curb points by doing a perspective projection of
the lidar points in image space and choosing points closer to the curb pixels
within a bound of $\pm$5 pixels.

Figure 9: Lidar point cloud projected into camera perspective with filtered
curb semantics, shown as cyan colored points. Figure 10: Calibrated lidar
point clouds from $L_{F}$ and $L_{R}$ lidar with their respective image
visuals with corresponding curb features highlighted.

As seen in Fig. 9, we extract the curb points from the front lidar. And
similar extraction was performed on the rear lidar as well. After calculating
the time offset based on vehicle kinematics we could verify for the calibrated
scenario the RMSE was statistically lower. A full representation of the
calibrated lidar point clouds are shown in Fig. 10.

##  VI Conclusion

We have presented a multi-lidar calibration method that helps in automatically
determine and verify the quality of the extrinsic calibration parameters
between the lidars. We have also shown that aligning the poses with ScLERP and
applying the DQ based Hand-Eye calibration improves the quality of the
extrinsic calibration parameters. In future, the extrinsics can be used in
state vector in SLAM solutions and be recovered online by solving the DQ based
Hand-Eye calibration. Furthermore, the verification part of the extrinsics can
be improved by calculating the correlation in between the features of
individual lidars in a common map reference frame. For the verification
purpose we assumed the perspective projection from the lidar to camera did not
introduce any errors into the system. This can be improved by extracting
semantic features directly from the point cloud by training a supervised model
on labeled point cloud data.

##  VII Acknowledgements

This research has been conducted as part of development of autonomous
transport solutions at Scania, Sweden. It was jointly funded by Swedish
Foundation for Strategic Research (SSF) and Scania. The research was also
affiliated with WASP.

## References

  * [1] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “PointNet++: Deep hierarchical feature learning on point sets in a metric space,” in _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 5105–5114.
  * [2] L. Huang and M. Barth, “A novel multi-planar LIDAR and computer vision calibration procedure using 2D patterns for automated navigation,” in _IEEE Intelligent Vehicles Symposium_ , 2009, pp. 117–122.
  * [3] G. Pandey, J. McBride, S. Savarese, and R. Eustice, “Extrinsic calibration of a 3d laser scanner and an omnidirectional camera,” _IFAC Proceedings Volumes_ , vol. 43, no. 16, pp. 336–341, 2010.
  * [4] C. Gao and J. R. Spletzer, “On-line calibration of multiple lidars on a mobile vehicle platform,” in _IEEE International Conference on Robotics and Automation_ , 2010, pp. 279–284.
  * [5] D.-G. Choi, Y. Bok, J.-S. Kim, and I. S. Kweon, “Extrinsic calibration of 2-d lidars using two orthogonal planes,” _IEEE Transactions on Robotics_ , vol. 32, no. 1, pp. 83–98, 2016.
  * [6] L. Zhou, Z. Li, and M. Kaess, “Automatic extrinsic calibration of a camera and a 3D LiDAR using line and plane correspondences,” in _EEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2018, pp. 5562–5569.
  * [7] E.-s. Kim and S.-Y. Park, “Extrinsic calibration between camera and lidar sensors by matching multiple 3d planes,” _Sensors_ , vol. 20, no. 1, 2020\.
  * [8] R. Kümmerle, G. Grisetti, C. Stachniss, and W. Burgard, “Simultaneous parameter calibration, localization, and mapping for robust service robotics,” in _Advanced Robotics and its Social Impacts_ , 2011, pp. 76–79.
  * [9] R. Y. Tsai, R. K. Lenz, _et al._ , “A new technique for fully autonomous and efficient 3 d robotics hand/eye calibration,” _IEEE Transactions on robotics and automation_ , vol. 5, no. 3, pp. 345–358, 1989.
  * [10] Z. Taylor and J. Nieto, “Motion-based calibration of multimodal sensor arrays,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2015, pp. 4843–4850.
  * [11] Z. Taylor and J. Nieto, “Motion-based calibration of multimodal sensor extrinsics and timing offset estimation,” _IEEE Transactions on Robotics_ , vol. 32, no. 5, pp. 1215–1229, 2016.
  * [12] A. Vallat and D. Schneuwly, “Clock synchronization in telecommunications via ptp (ieee 1588),” pp. 334–341, 2007.
  * [13] F. Furrer, M. Fehr, T. Novkovic, H. Sommer, I. Gilitschenski, and R. Siegwart, “Evaluation of combined time-offset estimation and hand-eye calibration on robotic datasets,” in _Field and Service Robotics_. Springer, 2018, pp. 145–159.
  * [14] J. Jiao, Y. Yu, Q. Liao, H. Ye, R. Fan, and M. Liu, “Automatic calibration of multiple 3d lidars in urban environments,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2019, pp. 15–20.
  * [15] J. Jiao, H. Ye, Y. Zhu, and M. Liu, “Robust odometry and mapping for multi-lidar systems with online extrinsic calibration,” _IEEE Transactions on Robotics_ , pp. 1–10, 2021.
  * [16] G. Carrera, A. Angeli, and A. J. Davison, “SLAM-based automatic extrinsic calibration of a multi-camera rig,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2011, pp. 2652–2659.
  * [17] K. Daniilidis and E. Bayro-Corrochano, “The dual quaternion approach to hand-eye calibration,” in _International Conference on Pattern Recognition (ICPR)_ , vol. 1. IEEE, 1996, pp. 318–322.
  * [18] L. Kavan and J. Žára, “Spherical blend skinning: a real-time deformation of articulated models,” in _Symposium on Interactive 3D Graphics and Games_ , 2005, pp. 9–16.
  * [19] E. S. Jones and S. Soatto, “Visual-inertial navigation, mapping and localization: A scalable real-time causal approach,” _The International Journal of Robotics Research_ , vol. 30, no. 4, pp. 407–430, 2011.
  * [20] L. Heng, B. Li, and M. Pollefeys, “Camodocal: Automatic intrinsic and extrinsic calibration of a rig with multiple generic cameras and odometry,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2013, pp. 1793–1800.
  * [21] L. Heng, G. H. Lee, and M. Pollefeys, “Self-calibration and visual slam with a multi-camera system on a micro aerial vehicle,” _Autonomous Robots_ , vol. 39, no. 3, pp. 259–277, Oct. 2015.
  * [22] M. He, H. Zhao, F. Davoine, J. Cui, and H. Zha, “Pairwise lidar calibration using multi-type 3d geometric features in natural scene,” in _2013 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2013, pp. 1828–1835.
  * [23] J. Levinson and S. Thrun, “Automatic online calibration of cameras and lasers.” in _Robotics: Science and Systems_ , vol. 2. Citeseer, 2013, p. 7.
  * [24] R. Y. Tsai, R. K. Lenz, _et al._ , “A new technique for fully autonomous and efficient 3D robotics hand/eye calibration,” _IEEE Transactions on Robotics and Automation_ , vol. 5, no. 3, pp. 345–358, 1989.
  * [25] J. Brookshire and S. Teller, “Extrinsic calibration from per-sensor egomotion,” _Robotics: Science and Systems Conference_ , pp. 504–512, 2013\.
  * [26] K. Daniilidis, “Hand-eye calibration using dual quaternions,” _International Journal of Robotics Research_ , vol. 18, no. 3, pp. 286–298, 1999.
  * [27] J. Zhang and S. Singh, “Loam: Lidar odometry and mapping in real-time.” in _Robotics: Science and Systems_ , vol. 2, no. 9, 2014.
  * [28] P. Furgale, “Representing robot pose: The good, the bad, and the ugly,” in _workshop on Lessons Learned from Building Complex Systems, IEEE International Conference on Robotics and Automation (ICRA). http://paulfurgale. info/news/2_ , vol. 14, no. 6, 2014, p. 9.
  * [29] M. Shah, R. D. Eastman, and T. Hong, “An overview of robot-sensor calibration methods for evaluation of perception systems,” in _Proceedings of the Workshop on Performance Metrics for Intelligent Systems_ , 2012, pp. 15–20.
  * [30] D. Wisth, M. Camurri, S. Das, and M. Fallon, “Unified multi-modal landmark tracking for tightly coupled lidar-visual-inertial odometry,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 1004–1011, 2021.
  * [31] F. Pomerleau, F. Colas, R. Siegwart, and S. Magnenat, “Comparing ICP Variants on Real-World Data Sets,” _Autonomous Robots_ , vol. 34, no. 3, pp. 133–148, Feb. 2013.
  * [32] C. Forster, L. Carlone, F. Dellaert, and D. Scaramuzza, “On-manifold preintegration for real-time visual-inertial odometry,” _IEEE Transactions on Robotics_ , vol. 33, no. 1, pp. 1–21, 2017.
  * [33] F. Dellaert and M. Kaess, “Factor graphs for robot perception,” _Foundations and Trends in Robotics_ , vol. 6, pp. 1–139, Aug. 2017.
  * [34] S. Umeyama, “Least-squares estimation of transformation parameters between two point patterns,” _IEEE Transactions on Pattern Analysis & Machine Intelligence_, vol. 13, no. 04, pp. 376–380, 1991.
  * [35] B. K. Horn, “Closed-form solution of absolute orientation using unit quaternions,” _Journal of the Optical Society of America_ , vol. 4, no. 4, pp. 629–642, 1987.
  * [36] J. M. McCarthy, _Introduction to theoretical kinematics_. MIT press, 1990.
  * [37] K. Shoemake, “Animating rotation with quaternion curves,” in _Annual Conference on Computer graphics and Interactive Techniques_ , 1985, pp. 245–254.
  * [38] H. H. Chen, “A screw motion approach to uniqueness analysis of head-eye geometry,” in _Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_. IEEE Computer Society, 1991, pp. 145–146.
  * [39] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 6105–6114.
  * [40] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 4510–4520.
  * [41] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 7132–7141.