# Multi-Echo Denoising in Adverse Weather

Alvari Seppänen1, Risto Ojala2, Kari Tammi1 1A. Seppänen, and Kari Tammi are
with Aalto University and University of Helsinki, Finland.
<EMAIL_ADDRESS>Ojala is with Aalto University, Finland.

###### Abstract

Adverse weather can cause noise to light detection and ranging (LiDAR) data.
This is a problem since it is used in many outdoor applications, e.g. object
detection and mapping. We propose the task of multi-echo denoising, where the
goal is to pick the echo that represents the objects of interest and discard
other echoes. Thus, the idea is to pick points from alternative echoes that
are not available in standard strongest echo point clouds due to the noise. In
an intuitive sense, we are trying to see through the adverse weather. To
achieve this goal, we propose a novel self-supervised deep learning method and
the characteristics similarity regularization method to boost its performance.
Based on extensive experiments on a semi-synthetic dataset, our method
achieves superior performance compared to the state-of-the-art in self-
supervised adverse weather denoising (23% improvement). Moreover, the
experiments with a real multi-echo adverse weather dataset prove the efficacy
of multi-echo denoising. Our work enables more reliable point cloud
acquisition in adverse weather and thus promises safer autonomous driving and
driving assistance systems in such conditions. The code is available at
https://github.com/alvariseppanen/SMEDNet

###### Index Terms:

Deep Learning for Visual Perception, AI-Based Methods, Intelligent
Transportation Systems, Visual Learning, Computer Vision for Transportation.

##  I Introduction

The impact of adverse weather conditions on light detection and ranging
(LiDAR) sensor data can be enormous. Airborne particles, including rain [1, 2,
3], snowfall [4, 5, 6, 7, 3], and fog [8, 1, 9, 3] cause unwanted absorptions,
refractions, and reflections of the LIDAR signal, which causes cluttered and
missing points. This is a critical issue as point clouds are typically used
for determining the accessible volume of the environment, for instance in
obstacle detection methods. Furthermore, it affects other downstream
perception algorithms, namely object detection [10, 11, 12, 13], which is a
vital component, e.g. in automated driving and driving assistance systems.
Moreover, accident rates for human drivers are notably higher in adverse
weather conditions, as reported by the European Commission [14] and the US
Department of Transportation [15]. Therefore, reliable perception data is
crucial in such conditions.

Figure 1: In the concept of multi-echo denoising, multiple echoes are acquired
for a single emitted pulse and the echo caused by the object of interest is
picked. Generally in single-echo approaches, only the strongest echo (red) is
available. Thus, crucial information about the object of interest can be lost.
I and R denote intensity and range, respectively.

Previous LiDAR adverse weather denoising work has mainly used single-echo
point clouds, where noise caused by adverse weather is removed from the point
cloud. This work proposes to use multi-echo point clouds and pick the echo
that represents the objects of interest and discard the echoes that represent
airborne particles (noise) or irrelevant artifacts caused by refractions, for
example. We use point cloud data format as it is provided by most of-the-shelf
LiDARs, however, using raw photon count histograms has been shown to be
beneficial when denoising LiDAR data in fog [16].

Our multi-echo approach is beneficial in several ways. First, the approach
utilizes information that is unavailable in single-echo approaches. That is,
objects might be occluded by airborne particles in single-echo approaches, but
in multi-echo approaches, they are visible in the form of alternative echoes
and thus provide valuable information. Second, multi-echo provides information
if multiple objects are located at the same observation angle, which can be
useful for determining the noise caused by adverse weather. With these
insights, the promise of our approach is increased and more reliable LiDAR
information in adverse weather conditions and thereby we increase the safety
and performance of downstream tasks.

It is a nontrivial task to determine, which echoes are useful since most do
not represent any physical object, but rather are artifacts from reflections
or refractions. In this work, we present a novel self-supervised approach
SMEDNet (Self-supervised-Multi-Echo-DenoiseNet) to achieve this goal. More
specifically, the task is to pick an echo for each laser pulse that is most
likely representing an object of interest and discard other echoes. The
resulting point cloud has a single echo per discrete observation angle.
Although multiple echoes can represent an object of interest, the output is
defined as a traditional single echo point cloud because many downstream tasks
process point clouds in such a format. Moreover, the single echo format is
generally sufficient in clear weather conditions. The explained task is
illustrated in Fig. 1. Finally, we propose a characteristics similarity
regularization method to boost the performance of our self-supervised
framework. The proposed method improves both convergence and performance at
test time.

The contributions of this paper are summarized as follows:

  * •

We propose and formulate the task of multi-echo denoising for LiDAR in adverse
weather.

  * •

We propose a novel self-supervised approach SMEDNet that learns this task,
motivated by the fact that point-wise labels are laborious to obtain
especially for multi-echo point clouds. The approach is based on neighborhood
correlation and a novel blind spot method, extending the work of Bae et al.
[17].

  * •

We propose the characteristics similarity regularization for LiDAR adverse
weather denoising to boost the performance of our method.

##  II Related Work

###  II-A Classical methods

Several classical methods have been presented in the literature for denoising
adverse weather-corrupted point clouds. They typically exploit the fact that
the expected density of noise caused by adverse weather is low i.e. the noisy
points are more sparse compared to valid points. Rönnbäck et al. proposed an
approach based on median filtering of the LiDAR range measurements corrupted
by snowfall [18]. Median filtering showed a substantial reduction in snowflake
detects. Radius outlier removal (ROR) and statistical outlier removal (SOR),
proposed by Rusu et al. [19], remove points based on their distance to
neighboring points. A constant radius threshold based on statistics defines
whether a point is an inlier or an outlier. However, these methods do not
perform well with LiDAR data as it has drastically varying point density. This
leads to ROR and SOR filters being biased on the measured range. Charron et
al. [20] presented an algorithm called dynamic radius outlier removal (DROR).
It solves the problem of varying density via a dynamic radius threshold. The
threshold is adjusted based on the measured range and thus range induced bias
is eliminated. Park et al. [21] discovered that adverse weather, specifically
snowfall, caused points to have typically lower intensity compared to valid
points. They developed low-intensity outlier removal (LIOR), which removes
points noise points based on the intensity value and ROR output. Later on,
dynamic distance–intensity outlier removal (DDIOR) presented by Wang et al.
[22] fuses LIOR and DROR to improve the performance. More recently, Li et al.
proposed to use spatiotemporal features to identify points caused by adverse
weather [23]. They utilized the phenomenon that adverse weather-induced points
follow unpredictable trajectories.

###  II-B Learned methods

Deep learning has been established as a versatile framework, and it has been
shown to be successful in a multitude of tasks, including adverse weather
denoising. Because adverse weather denoising can be thought of as a semantic
segmentation task, general-purpose segmentation networks can be utilized. Most
popular general-purpose approaches use voxelized [24, 25], bird’s eye view
[26], or spherical projection input [27, 28, 29, 30]. The raw point cloud
approaches exist as well [31, 32, 33, 34, 35, 36, 37, 38]. While these
general-purpose semantic segmentation approaches can be utilized to denoise
point clouds corrupted by adverse weather, they are not designed for the given
task, thus they can lack performance.

Parallels can be drawn between adverse weather denoising and deep image
denoising where camera images are restored as these methods modify the
original pixels and try to acquire a clean image. This can be done supervised
or self-supervised manner. One of the pioneering works is the Noise2Noise
[39], which maps a noisy image to another noisy image with the same underlying
signal. Later on, methods that do not require image pairs have been developed
[40, 41, 42, 43]. Therefore, these methods tend to have more flexible use
cases. Despite the success of deep image denoising, the methods do not
directly suit the purpose of LiDAR adverse weather denoising as the data is
sparse and the task is to remove noise points and keep valid points unaltered.
Some works have developed dense point cloud denoising methods [44, 45], which
have an equivalent goal to deep image denoising. However, these methods are
yet to be implemented on the adverse weather denoising task.

Deep learning methods specifically targeted to the adverse weather denoising
task have been also presented in the literature, and they are typically
projection-based as it provides a good trade-off between accuracy and
computational cost. Heinzler et al. [46] proposed the WeatherNet architecture
for denoising fog and rain. Seppänen et al. introduced 4DenoiseNet [47], which
utilizes spatiotemporal information from consecutive point clouds and proposed
a semi-synthetic dataset for training without human-annotated data. Bae et al.
[17] developed SLiDE, which can be trained in a self-supervised manner via
reconstruction difficulty. Yu et al. [48] used the fast Fourier and the
discrete wavelet transforms to construct a loss function for training a
network for snow segmentation. Our work is inspired by the network
architecture of [47] and the self-supervised pipeline of [17], and builds the
multi-echo denoising framework on them.

Preliminary work has been done on airborne particle classification using
multi-echo point clouds. Stanislas et al. [49] studied supervised deep
learning methods for classifying airborne particles using the multi-echo
measurement as a feature. That is, they predicted the class only for the first
echo and used the alternative echo merely as a feature. This approach has the
limitation of not utilizing alternative echoes to their fuller potential, as
they can be used for viable substitute points to replace points of the
strongest echo point cloud, which is the goal and contribution of our work. We
are the first ones to present a method that picks substitutes from a multi-
echo point cloud to replace invalid points of the strongest echo point cloud.

##  III Methods

Multi-echo input formulation. A traditional single-echo point cloud is defined
as $\mathbf{P}_{s}\in\mathbb{R}^{N_{p}\times N_{c}}$, where $N_{p}$ and
$N_{c}$ denote the number of points and channels, respectively. A multi-echo
point cloud is defined as $\mathbf{P}_{m}\in\mathbb{R}^{N_{p}\cdot N_{e}\times
N_{c}}$, where $N_{e}$ denotes the number of echoes per emitted laser pulse.
In this work, point clouds are processed in an ordered format. We define
multi-echo ordered format as a spherical projection of the point cloud
$\Gamma:\mathbb{R}^{N_{p}\cdot N_{e}\times N_{c}}\to\mathbb{R}^{H\times
W\times N_{e}\times N_{c}}$, where $H$ and $W$ stand for image dimensions of
the projection. As echoes from the same laser pulse have the same azimuth and
elevation angles, they are stacked on dimension $N_{e}$, and a vector along
this dimension is also denoted as an echo group $E_{g}$. Finally, the multi-
echo ordered point cloud is $\mathbf{P}_{meo}\in\mathbb{R}^{H\times W\times
N_{e}\times N_{c}}$.

Multi-echo denoising task formulation. The intuition behind multi-echo
denoising is to see through the noise caused by e.g adverse weather by
recovering points that carry useful information from other echoes.
Additionally, misleading or irrelevant points caused by other echoes are to be
discarded. To formulate this task, let $M(\mathbf{P}_{nm})=\mathbf{P}_{c}$,
where $\mathbf{P}_{nm}\in\mathbb{R}^{N_{p}\cdot N_{e}\times N_{c}}$ is the
noisy multi-echo point cloud and $\mathbf{P}_{c}\in\mathbb{R}^{(N_{p}\cdot
N_{e}-N_{n}-N_{r})\times N_{c}}$ is the obtained clean point cloud. $N_{n}$
denotes the number of removed noise points, and $N_{r}$ denotes other
irrelevant points, for instance, duplicate, and artifact points caused by the
refraction of the laser. The remaining points $\mathbf{P}_{c}$ include
substitutes recovered from alternative echoes. Ultimately the goal is to
obtain $\mathbf{P}_{c}$ such that it is equivalent to a standard strongest
echo point cloud in clear weather, as it is the default singe-echo mode. In
this work, $M(\cdot)$ is defined as a self-supervised neural network, which is
described in the following subsections.

###  III-A SMEDNet

Multi-echo neighbor encoder. The multi-echo ordered point cloud
$\mathbf{P}_{meo}$ is the input to the multi-echo neighbor encoder, which is
one of our contributions and the main difference to the work of Bae et al.
[17]. This module processes $\mathbf{P}_{meo}$ into upper bound KNN sets using
Euclidean distance to the reference point cloud i.e. strongest echo point
cloud and encodes these values into a feature tensor $\mathbf{\Pi}$. The
reference point cloud is the strongest point cloud given that it is the
default in clear weather, and thus most likely to contain points of objects of
interest. This is illustrated in Fig. 2. We formulate the multi-echo neighbor
encoder as a set of convolutions. For simplicity, components of
$\mathbf{P}_{meo}$ are denoted as: $\mathbf{P}_{xyz}$ – Cartesian coordinates,
$\mathbf{P}_{\theta\phi}$ – azimuth and elevation coordinates, and
$\mathbf{P}_{r}$ – range coordinates.

$\mathbf{\Pi}=\mathbf{w}_{t}*\mathbf{P}_{meo}\\\
=\mathbf{w}_{t}(\mathbf{P}_{r}[\mathit{argmink}_{\vec{p}_{2}}(\psi(\vec{p}_{2},\vec{p}_{3}))]\\\
\oplus(\mathbf{P}_{\theta\phi}[\vec{p}_{2}]-\mathbf{P}_{\theta\phi}[\mathit{argmink}_{\vec{p}_{2}}(\psi(\vec{p}_{2},\vec{p}_{3}))]))$
(1)

where $\mathbf{w}_{t}\in\mathbb{R}^{k\cdot N_{e}\times S_{o}}$ indicates
trainable weights, $\vec{p}_{2}=(h,w,0)$, $h\in[\\![0,H]\\!]$, and
$w\in[\\![0,W]\\!]$ indicates a pixel coordinate on the strongest echo point
cloud. $argmink_{\vec{p}_{2}}(\cdot)$ returns indices of $k\times N_{e}$
strongest echo values which minimize Euclidian distance to multi-echo queries
$\vec{p}_{3}=(h,w,\hat{e}),\hat{e}\in[\\![0,N_{e}]\\!]$. $\oplus$ is the
concatenation operation, and

$\psi(\vec{p}_{2},\vec{p}_{3})=||\mathbf{P}_{xyz}[\vec{p}_{2}+\Delta\vec{p}_{2}]-\mathbf{P}_{xyz}[\vec{p}_{3}]||_{2}\\\
\Delta\vec{p}_{2}\in\mathbf{A}_{c}\qquad\forall\psi(\vec{p}_{2},\vec{p}_{3})<C_{r}$
(2)

where $C_{r}$ is a fixed radius cutoff hyper-parameter defining the upper
bound for the neighbor search, which ensures that only local points are
considered. $\mathbf{A}_{c}$ defines the elements considered in the search.
$\mathbf{P}_{\theta\phi}$ are encoded to preserve the 3D information because
the grid positions of the neighbors $\mathbf{P}_{meo}$ are lost due to the
nature of the KNN search. Hereby, the architecture can utilize the original 3D
information for the predictions. The final output of the module is the
activated features. Next, the coordinate and correlation learner models
process these features.

Coordinate learner. The task of the coordinate learner is to predict the
coordinate of a point $p_{i}$ given the neighboring points of $p_{i}$. To
simplify the task and enable faster convergence, the three dimensions
$p_{i}\in\mathbb{R}^{3}$ are reduced to one
$r_{i}=||p_{i}||_{2}\in\mathbb{R}$. The coordinate learner learns point
coordinates by minimizing the absolute distance error to the actual
coordinates.

Correlation learner. The task of the correlation learner is to predict the
predictability of the coordinate of point $p_{i}$. More precisely, the task is
to predict inversely the correlation to the neighboring points, hence the name
correlation learner. This is done because the loss is formulated in such a
manner that the correlation learner predicts a high value for $p_{i}$ if its
coordinate is difficult to predict i.e. to compensate for a high coordinate
prediction error. During training this network learns to identify points that
are caused by objects of interest, assuming that those points have a higher
neighborhood correlation compared to other points. During inference time the
points can be filtered based on the predicted value of the correlation
learner, thus only the correlation learner is used during inference time
(Section III-D).

Exclude self – Include neighbors. The key difference between the coordinate
and correlation learner is that blind spots are added to the input of the
coordinate learner during training. Here we present a novel variant of the
blind spot method. Previously, it has been implemented on deep image denoising
[40, 50, 43], where the center pixel of a traditional convolution filter is
blanked. Here we combine the multi-echo neighbor encoder with the blind-spot
approach. That is, for a $p_{i}$ the input is its KNN set without the query
(Fig. 2). To learn the coordinate of $p_{i}$, only its neighbors are processed
and the coordinate of $p_{i}$ is predicted. However, to learn the correlation
of $p_{i}$, the whole KNN set including the query, is processed. This ensures
that the model can utilize the valuable information of the query point. The
benefit of our method is that physically neighboring data points are more
relevant when estimating the value of the hidden point compared to prior work
where more unrelated 2D grid relations are used. Following [41], we pick a
random subset of points to be excluded in order to avoid over-fitting.

Figure 2: In the proposed multi-echo neighbor encoder, KNN sets are computed
using multi-echo query and KNN reference, we use the strongest echo point
cloud as it is de facto in single-echo approaches. Two example queries
illustrate how KNN sets look. ”Exclude self – Include neighbors” simply
discards KNN queries and keeps the values. From this, the coordinate learner
predicts the queries. The point cloud on the right illustrates the final
output of our method, where substitutes are points from alternative echoes.

###  III-B Characteristics similarity regularization

To boost the performance of our method, we present a novel characteristics
similarity regularization (CSR) method for the adverse weather denoising task.
This method accelerates the convergence of the self-supervised learning regime
and increases the performance of the learned model. Notably, it requires only
one hyper-parameter, the size of the search $k_{CSR}$. The effect of $k_{CSR}$
on performance is small, thus it is easy to tune. The CSR is built on the
following assumptions of the nature of the noise caused by adverse weather. 1)
An expected echo caused by an airborne particle has a typical intensity [7].
2) Airborne particles cause more sparse point clouds compared to other objects
[20]. Based on these assumptions, a process that guides the predictions into
similar distribution as the intensity and sparsity should increase the
convergence. With this insight in mind, we build a regularization process to
guide the predictions to the distribution mentioned above. 111It is important
to note here that we do not assume any specific distribution of the
characteristics of the noise points, but instead encourage the model to learn
the connections between these characteristics and other features relative to
the output.

Let us define the components of the characteristics map $\mathbf{\Theta}$. The
map has the same spatial dimensions as the output of the network, which enable
straightforward implementation of the CSR. As for point-source radiation, the
expected intensity measurement from a LiDAR follows the inverse-square law
$i\propto\frac{1}{r^{2}}$, where $r$ is the distance of the measurement. To
eliminate the contribution of range, the intensity matrix is defined as

$\mathbf{I}=\mathbf{I}_{raw}\odot\mathbf{P}_{r}^{2}$ (3)

where $\mathbf{P}_{r}=||\mathbf{P}_{xyz}||_{2}$. The sparsity is defined as
follows,

$\mathbf{S}=E_{d}(\mathbf{P}_{meo})\odot\frac{1}{\mathbf{P}_{r}}$ (4)

where $E_{d}(\cdot)$ returns Euclidean distance to the nearest neighbor for
each point. Thus, sparsity can be obtained as a free lunch from
$\mathbf{\Pi}$. It is normalized with the range matrix $\mathbf{P}_{r}$, as
sparsity $s\propto r$. Then, we get the characteristics map as a concatenation
of the components $\mathbf{\Theta}=\mathbf{I}\oplus\mathbf{S}$. The similarity
is computed using normal distributions of neighbors of $\mathbf{\Theta}$. That
is, for each element in $\mathbf{\Theta}$, arguments of $k_{CSR}$-nearest-
neighbors are computed using the Euclidean distance. A regression goal for a
corresponding output is the Z-score of the distribution collected with the
neighbor search. This regression goal forms the characteristics similarity
loss term using the absolute error. The described regularization process
forces those correlation learner outputs $\mathbf{O}_{cor}$ that have similar
characteristics to have similar values, which we expect will encourage the
model to converge. Finally, we summarize CSR as the Algorithm (1).

Algorithm 1 Characteristics similarity regularization

for $\text{a batch}\in\text{batches}$ do

$\mathbf{I}\leftarrow\text{Equation (\ref{eq:intesity})}$ $\triangleright$
Intensity

$\mathbf{S}\leftarrow\text{Equation (\ref{eq:sparsity})}$ $\triangleright$
Sparsity

$\mathbf{\Theta}\leftarrow\textsc{concat}(\mathbf{I},\mathbf{S})$
$\triangleright$ Characteristics map

$\mathbf{\Phi}\leftarrow\mathbf{O}_{cor}[\textsc{argNeigbors}(\mathbf{\Theta},k_{CSR})]$

$\mathbf{\Xi}\leftarrow|\textsc{Zscore}(\mathbf{O}_{cor},\mathbf{\Phi})|$
$\triangleright$ Characteristics

similarity loss

###  III-C Architecture and loss function

Architecture. Our proposed architecture is presented in Fig. 3. The
architecture consists of two encoder-decoder neural networks. The networks are
identical except for the input since the coordinate learner processes point
clouds with blind spots. The encoder-decoder is to close resemblance to
4DenoiseNet [47] using three residual blocks. Please refer to the
supplementary paper for more details.

Figure 3: The proposed self-supervised multi-echo denoising architecture
(SMEDNet). White modules are used for training only.

Loss function. Similarly to [17], two networks are trained jointly. As stated
above, the ”exclude self – include neighbors” is performed only for a randomly
selected subset of points, denoted by $\mathbf{P}_{s}$. Therefore, we train
only those points. The loss function in its final form is defined as follows:

$\mathcal{L}=\frac{1}{|\mathbf{P}_{s}|}\sum_{p_{s}\in\mathbf{P}_{s}}\left(\frac{\lambda\cdot|\mathbf{O}^{coo}_{p_{s}}-\mathbf{P}^{r}_{p_{s}}|}{\lceil\mathbf{P}^{r}\rceil_{p_{s}}\odot\textsc{exp}(\mathbf{O}^{cor}_{p_{s}})}+\mathbf{O}^{cor}_{p_{s}}+\mathbf{\Xi}_{p_{s}}\right)$
(5)

where $\mathbf{O}^{coo}$ and $\mathbf{O}^{cor}$ are the outputs of the
coordinate and correlation learner, respectively. The loss function minimizes
the coordinate prediction error at the numerator and the denominator forces
the output $\mathbf{O}^{cor}$ to a high value when the coordinate is difficult
to predict i.e. noisy adverse weather points. Thus, during inference,
$\mathbf{O}^{cor}$ is used as a score to define the noisy points. A fixed
hyper-parameter $\lambda$ scales this term relative to the other terms. A
regulating term $+\mathbf{O}^{cor}$ prevents predictions from exploding. The
division by $\lceil\mathbf{P}^{r}\rceil$ reduces the range-induced bias of the
learning process as the correlation to neighbors is inversely proportional to
the range. To make the learning process more stable,
$\lceil\mathbf{P}^{r}\rceil$ is rounded up to full meters. The model converges
when the coordinate learner has learned meaningful high-level features of
valid points and the correlation learner learns to output a high score, i.e.
low correlation value, for invalid points.

###  III-D Inference mode

During the inference mode, only the correlation learner is used. The output
$\mathbf{O}_{cor}$ is processed into class labels. For this purpose, we
formulate the multi-echo denoising classes in the following manner:

  * •

valid strongest echo, $(S\land T)\to VS$,

  * •

potential substitute, $(\lnot S\land T\land B\land D)\to PS\in E_{g}$,

  * •

discarded, $(\lnot VS\lor\lnot PS)\to DI$,

where $S,T,B,D$ are boolean formulas for ”the strongest echo”, ”satisfies a
threshold”, ”the best score”, and ”a different coordinate to strongest”,
respectively. $E_{g}$ denotes the echo group i.e. the values along the
$N_{e}$-dimension of $\mathbf{O}_{cor}$. Here, we formulate $B$ in general
form, but in our case, it is equal to the predicted correlation value
$\mathbf{O}_{cor}$. During the inference time, the $DI$ labeled points are
removed ($T_{m}$ in Fig. 3), and the remaining point cloud is the final output
of our method.

##  IV Experimental results

###  IV-A Implementation details

Datasets. The experiments are conducted with the STF dataset [3], which
includes multi-echo LiDAR point clouds from snowfall, rain, and fog. This is a
suitable dataset for our experiments because it has a wide variance of adverse
weather conditions in traffic scenarios which are our main interest. STF [3]
is used only for qualitative experiments as it does not have point-wise
labels. Therefore, a semi-synthetic single-echo SnowyKITTI dataset [47] is
used for the quantitative experiments, which is a well-suited benchmark for
comparison and ablation studies.

2.5-echo point cloud. The LiDAR used in this study provides the strongest and
last echo point clouds. If the last echo is the same as the strongest echo, it
is replaced by the second strongest echo. Thus, we call it a 2.5-echo point
cloud. Hypothetically, more than 2.5 echoes are beneficial, as those could
represent objects of interest. Therefore, our method is formulated without a
restriction on the number of echoes i.e. the echo number has a dedicated
tensor dimension and the operations are agnostic to the size of this
dimension. One case where the last echoes are useful is when the strongest
echo is caused by adverse weather and the last echo is caused by the object of
interest in the same echo group.

Training and testing details. The hyper-parameter settings were selected with
a grid search as follows. The learning rate is 0.01, the learning rate decay
is 0.99 to avoid over-fitting, the fixed hyper-parameter in Eq. (5) $\lambda$
= 5, and the $k_{CSR}=9$. We use the stochastic gradient descent optimizer
with a momentum of 0.9 and train for 30 epochs. The train/validate/test-set
split ratio is 55/10/35, where all sets have a wide range of adverse weather
corrupted point clouds but are from different sequences. During inference, a
threshold $T_{n}=0$ is empirically found to yield the best results. The models
are run on an RTX 3090 GPU using Python 3.8.10 and PyTorch 1.12.1 [51].

###  IV-B Quantitative results

Quantitative experiments are conducted with the semi-synthetic SnowyKITTI [47]
LiDAR adverse weather dataset as labeled multi-echo data is not available. To
enable our algorithm to work on single-echo data, only the strongest echo is
used. Single-echo performance on the SnowyKITTI-dataset [47] can be seen in
Table I. Our algorithm is compared to the state-of-the-art self-supervised
SLiDE [17] and with well-proven classical methods DROR [20] and LIOR [21]. The
methods are evaluated with the widely adopted Intersection over Union (IoU)
metric. The results are presented separately for the light, medium, and heavy
levels of snowfall to have a more detailed indication of performance. Based on
the results, the performance varies depending on the noise level. Our method
achieves superior performance to these methods in all levels of noise.

We also report the runtime and parameter count as adverse weather denoising
methods are typically intended for a pre-processing step before other tasks,
thus they add latency and memory usage to the pipeline. Therefore, attention
should be also given to these metrics. As indicated by the results, our method
has approximately the same runtime and uses 35% fewer parameters compared to
SLiDE [17]. This is significant as fewer parameters help with over-fitting,
reduce memory footprint, and enable faster convergence. A slight difference in
the runtime between our method and SLiDE [17] results from the input layer, as
the neighbor encoder is slower than a standard 2D convolution. Overall, our
method achieves superior performance and thus sets the new state-of-the-art in
self-supervised adverse weather denoising.

Table I: Results on the SnowyKITTI [47] dataset. * – no training required. The bold font indicates the best values. Type | Method | IoU | Runtime | Param.
---|---|---|---|---
Light | Medium | Heavy | ms | $\cdot 10^{6}$
Classical | DROR* [20] | 0.451 | 0.450 | 0.439 | 120 | $10^{-5}$
LIOR* [21] | 0.448 | 0.447 | 0.434 | 120 | $10^{-5}$
Self-supervised | SLiDE [17] | 0.775 | 0.622 | 0.741 | 2.0 | 1.73
SMEDNet (Ours) | 0.933 | 0.854 | 0.843 | 2.2 | 1.13
| Improvement | 0.158 | 0.232 | 0.102 |  |

###  IV-C Ablation study

To assess the contributions of the characteristics similarity regularization,
cutoff radius $C_{r}$, the style of the input convolution, and the neural
network, an ablation study was conducted. Table II presents the IoU
measurements when ablating the aforementioned modules. The ablation of CSR
indicates that performance on the test set increases with the inclusion of the
module. Excluding $C_{r}$ has a significant effect on the performance as well.
We suspect that this is because limiting the input neighbors with a certain
threshold excludes points that are irrelevant for the coordinate prediction.
KNN search is replaced by simply including the grid neighbors, which is equal
to a standard 2D convolution. The performance decrease is due to the fact that
grid neighbors can be spatially unrelated to the center point. The neural
network was changed to WeatherNet [46], as 4DenoiseNet [47] has shown better
performance in supervised learning [47]. The network affects also self-
supervised performance based on our experiment.

Table II: Ablations of different modules. CSR is the characteristics similarity regularization, $C_{r}$ is the cutoff radius of the multi-echo neighbor encoder, and the NS indicates the style of the neighbor search, where KNN is k-nearest-neighbors and GridN is the grid neighbors. NN denotes a neural network, where 4DN is 4DenoiseNet [47], and WN is WeatherNet [46]. CSR | $C_{r}$ | NS | NN | IoU
---|---|---|---|---
Light | Medium | Heavy
✗ | ✓ | KNN | 4DN | 0.912 | 0.825 | 0.802
✓ | ✗ | KNN | 4DN | 0.871 | 0.819 | 0.808
✓ | ✓ | GridN | 4DN | 0.854 | 0.795 | 0.806
✓ | ✓ | KNN | WN | 0.885 | 0.819 | 0.792
✓ | ✓ | KNN | 4DN | 0.933 | 0.854 | 0.843

###  IV-D Qualitative results and discussion

Multi-echo experiments are evaluated qualitatively as labeled data is not
available. We analyze the performance on the STF [3] dataset. The results are
presented in Fig. 5, where each row indicates an individual sample. The
corrupted strongest echo is on the left-hand side, where a detail window is
denoted with fuchsia. The strongest echo is visualized to highlight the idea
of recovering substitute points from alternative echoes. With that, we want to
emphasize that the input to the methods is the multi-echo point cloud. The
baseline method is next to the corrupted strongest echo and our SMEDNet is on
the right-hand side. For these, far-away views and detailed close-ups are
visualized.

We compare our method to the baseline Multi-Echo DROR (MEDROR). It is a simple
modification to the classical DROR [20]. In particular, the neighbors are
computed for each echo, where the reference point cloud is the strongest point
cloud. This is similar to the procedure presented in Fig. 2. The searched
neighbors are then thresholded similarly to [20] but instead of just computing
this for the strongest echo it is computed for all echoes in the echo group.
Then, based on the inlier-outlier classification of [20], we classify the
points according to Section III-D. Note that, the score is now binary instead
of a float. Please refer to the supplementary paper for more details.

The substitute points are visualized in red. These are recovered points from
alternative echoes (in our case conditional last and second strongest). As
seen in Fig. 5, SMEDNet successfully finds viable substitute points. The
results prove the concept of multi-echo denoising in adverse weather. They
also demonstrate that a simple classical method such as MEDROR is not
sufficient as there are only a few recovered substitute points and many false
negatives.

Classical MEDROR performs much better with low levels of noise compared to a
high level of noise. Note here that the threshold of MEDROR can be adjusted
but we noticed that it results in a vast amount of false positives.
Contrarily, our SMEDNet performs well in all levels of noise, even in extreme
conditions in Fig. 5. This shows that our method achieves superior
performance.

Limitation. As stated in Section IV-A the number of echoes is limited to the
strongest, conditional second strongest, and last echo due to the LiDAR model.
The proposed task and method should be therefore studied with a LiDAR that
produces more echoes, for instance, an arbitrary number of strongest echoes
and the last echo e.g. 1st strongest, 2nd strongest, 3rd strongest, …, and
last echo. We hypothesize that more echoes are beneficial as the potential
substitutes can be in any echo number. This is a great opportunity for future
research, and we hope that LiDAR manufacturers could provide such equipment.

(a)

Input

Baseline MEDROR

Our SMEDNet

(a)

(b)

(c)

(d)

Figure 5: Multi-echo denoising performance on real data. Each row is an
individual sample. The corrupted strongest echo is on the leftmost column, the
output of the baseline method MEDROR is in the middle, and the output of our
SMEDNet is on the right column. Red indicates potential substitute points.
MEDROR mostly fails, whereas our SMEDNet picks successfully viable substitute
points.

##  V Conclusion

We proposed the task of multi-echo denoising in adverse weather. The idea is
to recover points that carry useful information from alternative echoes. This
can be thought of as seeing through the adverse weather-induced noise. We also
proposed a self-supervised deep learning method for achieving this goal,
namely SMEDNet, and proposed characteristics similarity regularization to
boost its performance. Our method achieved new state-of-the-art self-
supervised performance on a semi-synthetic dataset. Moreover, strong
qualitative results on a real multi-echo adverse weather dataset prove the
concept of multi-echo denoising. Compared to a classical MEDROR we show the
superiority of deep learning-based SMEDNet. Based on the results, our work
enables safer and more reliable LiDAR point cloud data in adverse weather and
therefore should increase the safety of, for instance, autonomous driving and
driving assistance systems.

## References

  * [1] Andrew M Wallace, Abderrahim Halimi, and Gerald S Buller.  Full waveform lidar for adverse weather conditions.  IEEE transactions on vehicular technology, 69(7):7064–7077, 2020\.
  * [2] Christopher Goodin, Daniel Carruth, Matthew Doude, and Christopher Hudson.  Predicting the influence of rain on lidar in adas.  Electronics, 8(1):89, 2019.
  * [3] Mario Bijelic, Tobias Gruber, Fahim Mannan, Florian Kraus, Werner Ritter, Klaus Dietmayer, and Felix Heide.  Seeing through fog without seeing fog: Deep multimodal sensor fusion in unseen adverse weather.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [4] Maria Jokela, Matti Kutila, and Pasi Pyykönen.  Testing and validation of automotive point-cloud sensors in adverse weather conditions.  Applied Sciences, 9(11):2341, 2019.
  * [5] Matti Kutila, Pasi Pyykönen, Maria Jokela, Tobias Gruber, Mario Bijelic, and Werner Ritter.  Benchmarking automotive lidar performance in arctic conditions.  In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pages 1–8. IEEE, 2020.
  * [6] Sébastien Michaud, Jean-François Lalonde, and Philippe Giguere.  Towards characterizing the behavior of lidars in snowy conditions.  In International Conference on Intelligent Robots and Systems (IROS), 2015.
  * [7] Harold W O’Brien.  Visibility and light attenuation in falling snow.  Journal of Applied Meteorology and Climatology, 9(4):671–683, 1970\.
  * [8] Mario Bijelic, Tobias Gruber, and Werner Ritter.  A benchmark for lidar sensors in fog: Is detection breaking down?  In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 760–767. IEEE, 2018.
  * [9] Robin Heinzler, Philipp Schindler, Jürgen Seekircher, Werner Ritter, and Wilhelm Stork.  Weather influence and classification with automotive lidar sensors.  In 2019 IEEE intelligent vehicles symposium (IV), pages 1527–1534. IEEE, 2019.
  * [10] Anh The Do and Myungsik Yoo.  Lossdistillnet: 3d object detection in point cloud under harsh weather conditions.  IEEE Access, 2022.
  * [11] Anh Do The and Myungsik Yoo.  Missvoxelnet: 3d object detection for autonomous vehicle in snow conditions.  In 2022 Thirteenth International Conference on Ubiquitous and Future Networks (ICUFN), pages 479–482. IEEE, 2022.
  * [12] Martin Hahner, Christos Sakaridis, Mario Bijelic, Felix Heide, Fisher Yu, Dengxin Dai, and Luc Van Gool.  Lidar snowfall simulation for robust 3d object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16364–16374, 2022.
  * [13] Martin Hahner, Christos Sakaridis, Dengxin Dai, and Luc Van Gool.  Fog simulation on real lidar point clouds for 3d object detection in adverse weather.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15283–15292, 2021.
  * [14] European Commission.  Road safety in the european union.  https://doi.org/10.2832/169706, 2018.  Accessed: 2022-02-09.
  * [15] U.S. Department of Transportation: Federal Highway Administration.  How do weather events impact roads?  https://ops.fhwa.dot.gov/weather/q1_roadimpact.htm, 2019.  Accessed: 2023-02-09.
  * [16] Tzu-Hsien Sang and Chia-Ming Tsai.  Histogram-based defogging techniques for lidar.  In 2022 IEEE 16th International Conference on Solid-State & Integrated Circuit Technology (ICSICT), pages 1–3. IEEE, 2022.
  * [17] Gwangtak Bae, Byungjun Kim, Seongyong Ahn, Jihong Min, and Inwook Shim.  Slide: Self-supervised lidar de-snowing through reconstruction difficulty.  In European Conference on Computer Vision, pages 283–300. Springer, 2022.
  * [18] Sven Ronnback and Ake Wernersson.  On filtering of laser range data in snowfall.  In 2008 4th International IEEE Conference Intelligent Systems, volume 2, pages 17–33. IEEE, 2008.
  * [19] Radu Bogdan Rusu and Steve Cousins.  3d is here: Point cloud library (pcl).  In 2011 IEEE international conference on robotics and automation, pages 1–4. IEEE, 2011.
  * [20] Nicholas Charron, Stephen Phillips, and Steven L Waslander.  De-noising of lidar point clouds corrupted by snowfall.  In 2018 15th Conference on Computer and Robot Vision (CRV), pages 254–261. IEEE, 2018.
  * [21] Ji-Il Park, Jihyuk Park, and Kyung-Soo Kim.  Fast and accurate desnowing algorithm for lidar point clouds.  IEEE Access, 8:160202–160212, 2020.
  * [22] Weiqi Wang, Xiong You, Lingyu Chen, Jiangpeng Tian, Fen Tang, and Lantian Zhang.  A scalable and accurate de-snowing algorithm for lidar point clouds in winter.  Remote Sensing, 14(6):1468, 2022.
  * [23] Boyang Li, Jieling Li, Gang Chen, Hejun Wu, and Kai Huang.  De-snowing lidar point clouds with intensity and spatial-temporal features.  In 2022 International Conference on Robotics and Automation (ICRA), pages 2359–2365. IEEE, 2022.
  * [24] Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han.  Searching efficient 3d architectures with sparse point-voxel convolution.  In European conference on computer vision, pages 685–702. Springer, 2020.
  * [25] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin.  Cylindrical and asymmetrical 3d convolution networks for lidar segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9939–9948, 2021.
  * [26] Zixiang Zhou, Yang Zhang, and Hassan Foroosh.  Panoptic-polarnet: Proposal-free lidar point cloud panoptic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13194–13203, 2021.
  * [27] Bichen Wu, Alvin Wan, Xiangyu Yue, and Kurt Keutzer.  Squeezeseg: Convolutional neural nets with recurrent crf for real-time road-object segmentation from 3d lidar point cloud.  In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1887–1893. IEEE, 2018.
  * [28] Bichen Wu, Xuanyu Zhou, Sicheng Zhao, Xiangyu Yue, and Kurt Keutzer.  Squeezesegv2: Improved model structure and unsupervised domain adaptation for road-object segmentation from a lidar point cloud.  In 2019 International Conference on Robotics and Automation (ICRA), pages 4376–4382. IEEE, 2019.
  * [29] Chenfeng Xu, Bichen Wu, Zining Wang, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka.  Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation.  In European Conference on Computer Vision, pages 1–19. Springer, 2020.
  * [30] Ryan Razani, Ran Cheng, Ehsan Taghavi, and Liu Bingbing.  Lite-hdseg: Lidar semantic segmentation using lite harmonic dense convolutions.  In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 9550–9556. IEEE, 2021.
  * [31] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.
  * [32] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  Advances in neural information processing systems, 30, 2017.
  * [33] Loic Landrieu and Martin Simonovsky.  Large-scale point cloud semantic segmentation with superpoint graphs.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4558–4567, 2018.
  * [34] Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, and Jan Kautz.  Splatnet: Sparse lattice networks for point cloud processing.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2530–2539, 2018.
  * [35] Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, and Qian-Yi Zhou.  Tangent convolutions for dense prediction in 3d.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3887–3896, 2018.
  * [36] Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham.  Randla-net: Efficient semantic segmentation of large-scale point clouds.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11108–11117, 2020.
  * [37] Radu Alexandru Rosu, Peer Schütt, Jan Quenzel, and Sven Behnke.  Latticenet: fast spatio-temporal point cloud segmentation using permutohedral lattices.  Autonomous Robots, 46(1):45–60, 2022.
  * [38] Xu Yan, Jiantao Gao, Jie Li, Ruimao Zhang, Zhen Li, Rui Huang, and Shuguang Cui.  Sparse single sweep lidar point cloud segmentation via learning contextual shape priors from scene completion.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3101–3109, 2021.
  * [39] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila.  Noise2noise: Learning image restoration without clean data.  arXiv preprint arXiv:1803.04189, 2018.
  * [40] Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila.  High-quality self-supervised deep image denoising.  Advances in Neural Information Processing Systems, 32, 2019.
  * [41] Yuhui Quan, Mingqin Chen, Tongyao Pang, and Hui Ji.  Self2self with dropout: Learning self-supervised denoising from single image.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1890–1898, 2020.
  * [42] Tao Huang, Songjiang Li, Xu Jia, Huchuan Lu, and Jianzhuang Liu.  Neighbor2neighbor: Self-supervised denoising from single noisy images.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14781–14790, 2021.
  * [43] Zejin Wang, Jiazheng Liu, Guoqing Li, and Hua Han.  Blind2unblind: Self-supervised image denoising with visible blind spots.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2027–2036, 2022.
  * [44] Pedro Hermosilla, Tobias Ritschel, and Timo Ropinski.  Total denoising: Unsupervised learning of 3d point cloud cleaning.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 52–60, 2019.
  * [45] Shitong Luo and Wei Hu.  Differentiable manifold reconstruction for point cloud denoising.  In Proceedings of the 28th ACM international conference on multimedia, pages 1330–1338, 2020.
  * [46] Robin Heinzler, Florian Piewak, Philipp Schindler, and Wilhelm Stork.  Cnn-based lidar point cloud de-noising in adverse weather.  IEEE Robotics and Automation Letters, 5(2):2514–2521, 2020.
  * [47] Alvari Seppanen, Risto Ojala, and Kari Tammi.  4denoisenet: Adverse weather denoising from adjacent point clouds.  IEEE Robotics and Automation Letters, 2022.
  * [48] Ming-Yuan Yu, Ram Vasudevan, and Matthew Johnson-Roberson.  Lisnownet: Real-time snow removal for lidar point clouds.  In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6820–6826. IEEE, 2022.
  * [49] Leo Stanislas, Julian Nubert, Daniel Dugas, Julia Nitsch, Niko Sünderhauf, Roland Siegwart, Cesar Cadena, and Thierry Peynot.  Airborne particle classification in lidar point clouds using deep learning.  In Field and Service Robotics: Results of the 12th International Conference, pages 395–410. Springer, 2021.
  * [50] Alexander Krull, Tim-Oliver Buchholz, and Florian Jug.  Noise2void-learning denoising from single noisy images.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2129–2137, 2019.
  * [51] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32, 2019.