# Detecting Line Segments
in Motion-blurred Images with Events

Huai Yu, Hao Li, Wen Yang, Lei Yu, and Gui-Song Xia  H. Yu, H. Li, W. Yang, L.
Yu are with the School of Electronic Information, Wuhan University, Wuhan
430072, China. E-mail: {yuhuai, lihao2015, yangwen<EMAIL_ADDRESS>G. S.
Xia is with the School of Computer Science, Wuhan University, Wuhan 430072,
China<EMAIL_ADDRESS>H. Li is also with Baidu Inc, Beijing,
China. H. Yu and H. Li contribute equally to this work.

###### Abstract

Making line segment detectors more reliable under motion blurs is one of the
most important challenges for practical applications, such as visual SLAM and
3D reconstruction. Existing line segment detection methods face severe
performance degradation for accurately detecting and locating line segments
when motion blur occurs. While event data shows strong complementary
characteristics to images for minimal blur and edge awareness at high-temporal
resolution, potentially beneficial for reliable line segment recognition. To
robustly detect line segments over motion blurs, we propose to leverage the
complementary information of images and events. To achieve this, we first
design a general frame-event feature fusion network to extract and fuse the
detailed image textures and low-latency event edges, which consists of a
channel-attention-based shallow fusion module and a self-attention-based dual
hourglass module. We then utilize the state-of-the-art wireframe parsing
networks to detect line segments on the fused feature map. Moreover, due to
the lack of line segment detection dataset with pairwise motion-blurred images
and events, we contribute two datasets, i.e., synthetic FE-Wireframe and
realistic FE-Blurframe, for network training and evaluation. Extensive
experiments on both datasets demonstrate the effectiveness of the proposed
method. When tested on the real dataset, our method achieves $63.3\%$ mean
structural average precision (msAP) with the model pre-trained on the FE-
Wireframe and fine-tuned on the FE-Blurframe, improved by $32.6$ and $11.3$
points compared with models trained on synthetic only and real only,
respectively. The codes, datasets, and trained models are released at
https://levenberg.github.io/FE-LSD.

###### Index Terms:

Line segment detection; Motion blur; Frame-event fusion; Attention.

##  1 Introduction

Line segment detection in images with the emergence of motion blurs often
suffers from severe performance degradation due to the motion ambiguities and
texture erasures [1], which brings a serious burden to geometric structure
perception especially when encountering high-speed motions of imaging
platforms, e.g., robotics [2], visual SLAM [3], and autonomous driving [4]. In
contrast to those in images with clear and sharp edges, geometric structures
in images with motion blurs are often blurred out, and it poses intense
challenges to accurately detect and localize line segments in the images, see
Fig. 1 (a) for an instance.

Although promising progress has been reported in the past decades on line
segment detection, see e.g., [5, 6, 7, 8], few of them consider the scenarios
with blurry textures and diverse motions, thus often losing their
effectiveness when dealing with motion-blurred images. As illustrated in Fig.
1, due to the overlapping of scene information at different times in
blurriness, line segments no longer appear as distinctive elongated edges,
while as blurred bands mixed with the background. To handle this issue, an
intuitive solution is to deblur the image first and then detect line segments
[9]. The detection performance highly depends on the quality of motion
deblurring, which itself is ill-posed due to motion ambiguities. Moreover,
motion deblurring methods are usually developed for improving the image
quality but not designed for a better structure perception [1, 10].

Figure 1: Illustrative examples of different methods. Line segments are drawn
on clear images at the end of camera exposure for better visualization.
Conventional (d)LSD [5] and (e)FBSD [11] have many false detections while the
original (f)LETR [12], (g)HAWP [6] and (j)ULSD [7] have many missing alarms on
the (a)motion-blurred image. The retrained (h)HAWP and (k)ULSD on the
concatenation of (a)images and (b)events detect more line segments. The
proposed (i)FE-HAWP and (l)FE-ULSD have the best performance.

On the other hand, if directly performing the detection in motion-blurred
images, we can only determine the distribution or the blurred band of each
line segment [13]. To measure the distribution, a thickness parameter is
introduced to describe the line width [14, 11]. Nevertheless, it can neither
model diverse blurred lines (e.g., rotated around a point on lines) nor
geometrically locate the position of a line segment. Even if utilizing
learning-based models [15, 8], training with line annotations is still unable
to model the blur distribution and determine the line position. Therefore, it
is far from satisfactory to detect only on blurred images, regardless of
whether deblurring, motion modeling, or data-driven approaches are used.

Recently, the newly developed event cameras have attracted increasing
attention, outputting pixel-level brightness changes instead of standard
intensity frames [16]. Event data is asynchronously generated at intensity
edges with extremely high temporal resolution (up to $\mu$s), thus it is edge-
aware and free of motion blurs [4]. Motivated by this characteristic, we
propose to introduce event data into the task of line segment detection in
motion-blurred images. On one hand, texture and structure layout can be
maintained by image frames, which also help to suppress the effect of event
noise. On the other hand, event data can recognize the distinctive edges in
blurred images and address diverse camera motions. Consequently, event cameras
show strong complementary characteristics to RGB cameras for their minimal
blur and edge awareness at high-temporal resolution. Therefore, if events can
be used in the data-driven training process, we will have great potential to
improve the robustness of line detection and localization to motion blur.
However, it remains challenging for the effective fusion of the two domain
data to detect line segments. Several key problems need to be investigated:
how to avoid the influence of event noises on line endpoint prediction, how to
make the fused feature map be cognitive to line edges with either fast or slow
camera motions, and how to solve the problem of lacking line segment detection
datasets with both visual and event data.

In this paper, we propose a robust line segment detection method in motion-
blurred images with events, i.e., Frame-Event fused Line Segment Detection
(FE-LSD). As the frontend, a novel feature fusion backbone is designed to
extract and fuse information from images and events, which consists of a
channel-attention-based shallow module and a self-attention-based stacked dual
hourglass module. The shallow fusion module is designed to suppress event
noise and enhance image edge features. The stacked dual hourglass module is
developed to conduct multi-scale fusion and obtain the fused feature map. As
the backend, the state-of-the-art line segment detectors, e.g., HAWP and ULSD,
are used to detect line segments on the fused feature map, namely FE-HAWP and
FE-ULSD respectively. A glance at the performance of different methods is
shown in Fig. 1 (i) and (l), where the proposed FE-HAWP and FE-ULSD have the
best performance. As far as we know, this is the first work of robust line
segment detection in motion-blurred images with events. Our contributions can
be summarized as follows:

  * •

Events are introduced to assist motion-blurred images for line segment
detection, which can robustly address the performance degradation caused by
motion blurs. The idea of fusing images and events fully exploits their
complementary properties of low-latency event edges and detailed image
textures, thus effectively improving the accuracy and robustness of line
detection over diverse camera motions.

  * •

A general frame-event feature fusion network is designed to extract and fuse
the information from images and events. The concatenation of shallow fusion
and multi-scale decoder fusion fully explores the channel-attention and self-
attention mechanism, thus enhancing the feature extraction of events and
frames.

  * •

Two frame-event line segment detection datasets, i.e., synthetic FE-Wireframe
and real-world FE-Blurframe, are constructed for line segment detection in
motion-blurred images. Sufficient qualitative and quantitative comparisons
demonstrate the effectiveness and robustness of the proposed method.

##  2 Related Work

Line segment detection has been a focused fundamental problem in computer
vision over decades, especially with the revolution of deep learning. In this
section, we will review the image-based and event-based line segment detection
methods. Image-based methods recall the track from traditional edge-based
clustering to the latest learning-based algorithms. While for the event-based
methods, we mainly analyze the recent adaptions of frame-based methods to
events, such as Hough transform [17] and LSD [18].

###  2.1 Image-based line segment detection

Traditional line detection methods mostly rely on edge detection and
aggregation of linear geometric properties, such as Hough transform [19, 20],
LSD [5], EDLines[21], and linelet[22]. These methods often follow an
unsupervised manner on the CPU and thus can be efficiently deployed on
embedding systems. However, their performance highly depends on the edge
quality and is sensitive to motion blurs. In recent years, learning-based
techniques have shown great progress to detect line segments. The first
learning-based wireframe parsing algorithm DWP [23] fuses the predicted line
segments and junctions with two CNNs to obtain the line segments.
Subsequently, AFM [24] is proposed to transform this task into an attraction
field regression problem using the line attraction field representation. Those
two methods are the first to introduce deep learning to improve the
performance of line segment detection. However, they both rely on heuristic
post-processing and thus are not end-to-end. To address this issue, a series
of end-to-end line segment detection methods have been proposed [15, 6, 25,
26]. The first one, L-CNN[15] utilizes a proposal sampling module to conduct
the fusion of lines and junctions in the network, eliminating the non-
differentiable heuristic post-processing and ensuring it is fully
differentiable. After that, the HAWP [6] combines the attraction field map
[24] with the L-CNN network, which effectively improves the quality of line
segment proposals. Subsequently, a trainable Hough transform module is added
to the existing line segment detection network [27] to enhance the linear
geometric constraints. Meanwhile, considering that line segments formulate
graphs, PPGNet[25] and LGNN[26] introduce Graph Neural Network (GNN) to fuse
the line segments and junctions for optimization, which eventually yields more
accurate line detection results. Recently, Transformer was introduced to the
line segment detection network. By combining it with CNN, LETR [8] achieves
state-of-the-art performance in an end-to-end manner. However, its main
drawback is that model training is time-consuming, and inference is relatively
slow. Recently, self-supervised [28] and semi-supervised [29] methods have
also been proposed to overcome the problem of limited training samples.

These learning-based line segment detection methods have demonstrated state-
of-the-art performance. However, they are trained and tested on clear images
without motion blurs. For real-world applications such as mobile robotics and
autonomous driving, the fast relative motion between the camera and objects
will inevitably generate motion blurs in RGB images. The blurriness can
significantly compromise or even invalidate the performance of existing
methods.

There are few existing works for line segment detection in motion-blurred
images. Debled-Rennesson first presents the concept of blurred line segments
in images and gives its mathematical definition [30] by introducing a
thickness parameter to describe the line width. Then this blurred line
representation model is further extended to gray-scale images, and a semi-
automatic line segment selection tool is implemented [31]. Later, FBSD [11] is
proposed based on the adaptive directional scans and the control of assigned
thickness to achieve automatic blurred line segment detection. Compared with
the traditional LSD[5] and EDLines[21], FBSD effectively improves the line
location accuracy and recall when handling blurs. However, the blurred line
definition according to Debled-Rennesson can only determine the width bands of
line segments. Thus it is hard to give an accurate line location with large
motion blurs.

###  2.2 Event-based line segment detection

Over the past few years, several event-based line segment detection methods
[32, 33, 2] have been proposed using the newly developed event camera, which
does not suffer motion blur due to its asynchronous characteristics and high
temporal resolution [34]. The Hough transform is the first method [17] for
event-based line segment detection and has recently been used for high-speed
train localization and mapping [33], demonstrating the effectiveness in
handling high-speed motions. After that, the ELiSeD [18] is proposed by
adapting LSD [5] to the event camera. Based on the assumption of a constant
velocity model during a short-time motion period, the fast detection and
persistent tracking of translating line segments are achieved by plane
detection in event data stream [35]. However, it cannot handle rotating and
drastically moving line segments. To improve the robustness over motion speed,
an event-based line segment detection method using iteratively weighted least-
squares fitting was proposed [32]. Recently, DVS event cameras are used for
lane detection using basic CNN and transformer networks[2, 36], which give
some insights for learning-based line detection. In [37], an online event-
based powerline tracker is proposed by using hibernation to cope with the line
direction changes. Most of these methods are validated only on simple data,
thus lacking quantitative evaluation and comparison. Additionally, there is no
publicly available dataset as the testing benchmark, and the line segment
detection performance in real environments is yet to be verified.

Although event cameras do not suffer motion blurs, the line segment detection
performance on pure event stream is also unsatisfactory because of the lack of
texture information and instability to slow motions or static conditions.
Detailed texture can improve the performance in weakly textured areas and
maintain structural integrity. Slow motions or static event cameras cannot
capture informative events. In contrast, RGB cameras can clearly capture
detailed edges and do not have motion blur when moving slowly and being
stationary. Therefore, we propose to fully utilize the strengths of both
cameras to achieve line segment detection in diverse motion conditions by
fusing visual images and event streams. However, this task is challenging,
such as the lack of frame-event fusion strategy for geometric structure
perception, and the lack of line segment detection dataset with visual images
and event stream data.

##  3 Methodology

###  3.1 Problem statement

We first introduce the concept of blurred line segments proposed by Debled-
Rennesson [30]. A discretized straight line is defined as:

$l=\\{(x,y)|c\leq ax-by<c+v\\},$ (1)

where $(x,y)\in\mathbb{Z}^{2}$ is the pixel coordinate,
$\\{a,b,c,v\\}\in\mathbb{Z}^{4}$ are the four parameters to determine the line
segment, in which $v$ describes line width. An additional thickness parameter
$\mu$ is defined as the minimum of the horizontal and vertical distances
between the lines $ax-by$=$c$ and $ax-by$ = $c+v$, i.e., $\mu$ =
$\frac{v}{max(|a|,|b|)}$.

(a) Start exposure

(b) End exposure

(c) During exposure

Figure 2: Blurred line segments in motion-blurred images. From left to right
are the line segments at the beginning, end, and within the camera exposure
time.

In fact, the blurry distribution of line segments is generally complex in
motion-blurred images, as shown in Fig. 2. A simple thickness parameter cannot
precisely model the line distributions. In this paper, line segments in images
are defined as a function of time $l(t),0\leq t\leq T$, which are detected on
the clear image captured at time $t$. $T$ is the camera exposure time. The
imaging process for motion-blurred images can be treated as the average of all
clear images $I(t)$ during $T$, i.e.,
$I_{B}$=$\frac{1}{T}\int_{t=0}^{T}I(t)dt$. Since the temporal resolution is
lost by averaging over time, it is difficult to detect line segments by only
relying on the motion-blurred image. Therefore, we introduce the high-temporal
resolution events $\mathcal{E}$=$\\{(x_{i},y_{i},p_{i},t_{i})\\}_{t_{i}\in T}$
into this task. Given the motion-blurred image $I_{B}$ and the temporally
aligned events $\mathcal{E}$ during $T$, the line segment detection with the
frame and event data can be defined as:

$\mathcal{L}(T)=\mathrm{FE}\text{-}\mathrm{LSD}(I_{B},\mathcal{E}),$ (2)

where $\mathcal{L}(T)$ denotes the set of line segments at time $T$ and
$\mathrm{FE}$-$\mathrm{LSD}$ denotes the line segment detection function.

###  3.2 Event representation

Unlike RGB cameras streaming images at a certain frequency, event cameras do
not output synchronous frames but event points asynchronously. For each pixel
$\boldsymbol{u}$=$\left[x,y\right]^{T}$, if the captured luminance $L$ changes
beyond a contrast threshold $C$ at time $t$, an event point $e$=$(x,y,t,p)$ is
triggered. The increase and decrease of luminance will result in positive
($p$=$+1$) and negative ($p$=$-1$) polarities, respectively.

$\Delta L=p\left(L(x,y,t)-L(x,y,t-\Delta t)\right)\geq C,$ (3)

where $t-\Delta t$ is the time of the last event point generated at that
pixel.

(a) Raw event stream

(b) EST representation

Figure 3: Event data representation using the EST (Left: raw event stream,
red-positive events, blue-negative events; Right: EST representation with
$B=5$). Figure 4: FE-LSD network structure. It mainly consists of two modules:
feature fusion backbone and back-end line detector, in which the feature
fusion backbone has one shallow module and several dual hourglass modules.

Event streams have asynchronous and sparse characteristics. To follow the CNN
architecture, the asynchronous event streams are often transformed into fixed-
size tensors. The most commonly used event representations including Event
Counting (EC) [38], Surface of Active Event (SAE) [39], voxel grid [40] and
Event Spike Tensor (EST) [41]. Among these the EST is a four-dimensional grid
of $H$$\times$$W$$\times$$B$$\times$$2$, where the time dimension is uniformly
divided into $B$ bins. Each bin further encodes the polarity as the positive
and negative dimensions. Therefore, it retains the most temporal and polarized
information. The EST is calculated by:

$\mathbf{EST}_{\pm}(x,y,b)=\sum_{e_{k}\in\mathcal{E}_{\pm}}\delta\left(x-x_{k},y-y_{k}\right)\max\\{0,1-|b-t_{k}^{*}|\\},$
(4)

where $b$$\in$$\\{0,1,\cdots,B-1\\}$,
$t_{k}^{*}\triangleq\frac{B-1}{T}\left(t_{k}-t_{0}\right)$, $t_{0}$ is the
earliest timestamp of the event stream during time $T$,
$e_{k}\in\mathcal{E}_{\pm}$ means the positive and negative EST are computed
respectively with the polarities of events. Fig. 3 shows the event
reconstruction based on the EST representation. The two spatial-temporal
lattices corresponding to the positive and negative polarity of EST is further
stacked into a $H$$\times$$W$$\times$$2B$ tensor to fit the common requirement
of CNN.

###  3.3 Network structure

Given the motion-blurred image and the aligned EST data, an effective fusion
strategy is designed to extract the complementary information for line segment
detection. We constructively fuse the shallow and deep features from images
and events, and then use two state-of-the-art line segment detectors, i.e.,
HAWP [6] and ULSD [7] to get the final detection results (Fig. 4). The feature
fusion backbone network includes two kinds of modules: (i) Shallow module,
which is designed to extract shallow features and suppress the event noises,
and (ii) Dual hourglass module, which follows an encoder-decoder structure to
conduct a multi-scale feature fusion for the feature map generation.

####  3.3.1 Shallow module

With the resized $H$$\times$$W$$\times$$3$ RGB image and the spatially-aligned
$H$$\times$$W$$\times$$2B$ EST data, the shallow module first separately down-
samples the RGB and EST using the Shallow Layer 1 (including 7$\times$7
convolution with step size 2, BatchNorm, and ReLU), which can obtain the
preliminary RGB and EST features with the same number of channels. Then the
image and event features are fused by the Shallow Fusion Block (SFB), and the
fused feature is further added with the original RGB and EST features,
respectively. Next, the Shallow Layer 2 (three residual blocks [42] and one
maximum pooling module after the first residual block) is used to down-sample
and extract features. The refined features are inputted into the second SFB
module. Finally, the fused feature is added with the outputs of Shallow Layer
2 to obtain the same dimension image feature $X_{F}^{(0)}$ and the event
feature $X_{E}^{(0)}$, which will be fed to the dual hourglass module
subsequently.

Figure 5: Shallow Fusion Block (SFB).

The SFB is the core part of the shallow module, and its network structure is
shown in Fig. 5. For the input image feature $X_{F}$ and the EST feature
$X_{E}$, the SFB first concatenates them together by channels and retains the
number of channels with a 1$\times$1 convolution. Then the attention
$Attn_{F}$ and $Attn_{E}$ of the two-channel features are computed separately
using the channel-attention (CA) blocks. The attention is further multiplied
with the original features and then added with the original features of the
other modality to achieve attention-weighted feature fusion. Finally, the
fused features are refined using a residual block to obtain the shallow image
feature $X^{o}_{F}$ and event feature $X^{o}_{E}$. The whole processing
process of the SFB can be described by

$\displaystyle X$ $\displaystyle=\mathrm{Conv_{1\times
1}}\left(\mathrm{Concat}(X_{F},X_{E})\right)$ (5) $\displaystyle Attn_{F}$
$\displaystyle=\mathrm{CA}_{F}(X)$ $\displaystyle Attn_{E}$
$\displaystyle=\mathrm{CA}_{E}(X)$ $\displaystyle X^{o}_{F}$
$\displaystyle=\mathrm{Res}_{F}(X_{F}+X_{E}\odot Attn_{E})$ $\displaystyle
X^{o}_{E}$ $\displaystyle=\mathrm{Res}_{E}(X_{E}+X_{F}\odot Attn_{F}),$

where $\mathrm{Conv_{1\times 1}}$ denotes 1$\times$1 convolution,
$\mathrm{Concat}$ denotes the concatenation by channel, $\mathrm{CA}$ denotes
the channel-attention module, $\mathrm{Res}$ denotes the residual module and
$\odot$ denotes the element-wise multiplication.

####  3.3.2 Dual hourglass module

As in HAWP [6] and ULSD [7], an encoder-decoder network, i.e., stacked
hourglass network [43], is used to obtain the feature map for line segment
detection. With the two-branch shallow features, we design the stacked dual
hourglass module to further fuse and extract features, as shown in Fig. 4. The
image and event features will be first fused into one feature with the
encoder-decoder network, followed by a residual block. Then the fused feature
is added with the input image and event features respectively to recover the
two branch features for the next dual hourglass module. For the last dual
hourglass module, the fused feature after the residual block is directly
outputted as the final feature map for the subsequent line segment detector.
The calculation of the stacked dual hourglass module is as:

$\displaystyle Y^{(i)}$
$\displaystyle=\mathrm{Res}(\mathrm{E\mbox{-}D}(X_{F}^{(i)},X_{E}^{(i)}))$ (6)
$\displaystyle X_{F}^{(i+1)}$ $\displaystyle=X_{F}^{(i)}+Y^{(i)}$
$\displaystyle X_{E}^{(i+1)}$ $\displaystyle=X_{E}^{(i)}+Y^{(i)},$

where $\mathrm{E\mbox{-}D}$ denotes the encoder-decoder network, $X_{F}^{(i)}$
and $X_{E}^{(i)}$ denote the output image and event features of the $i$-th
dual hourglass module, respectively ($X_{F}^{(0)}$ and $X_{E}^{(0)}$ are the
outputs of the shallow module). $Y_{i}$ denotes the fused features of the
$i$-th dual hourglass module output.

Figure 6: The encoder-decoder network in the dual hourglass module.

The core of the dual hourglass module is the encoder-decoder network, as shown
in Fig. 6, which consists of a pairwise image feature encoder, event feature
encoder, decoder fusion block (DFB), and decoder. Encoders and decoders are
implemented by residual blocks. The feature map is down-sampled during feature
encoding using a max-pooling layer with step size 2. During decoding, it is
up-sampled with step size 2 to ensure the same feature size for the encoder,
decoder, and DFB at the same level. The DFB (Fig. 7) consists of a 1$\times$1
convolution and a transformer module using 2 Layer Normalization (LN) modules,
a lightweight Multi-head Self-attention (MHSA) module, and an Inverse Residual
Feed-forward Network (IRFFN) [44]. The DFB module first fuses the input image-
event features to one feature by channel concatenation and 1$\times$1
convolution, and then uses the transformer [45] to further fuse and extract
deep features. The transformer is used mainly for two reasons: (i) Compared
with CNN, the transformer has a stronger capacity to capture global features,
which is beneficial for the long-distributed target detection such as line
segments; (ii) For the fusion of image and event features, the transformer can
provide self-attention as well as cross-attention for global cross-modal
feature interaction and information fusion.

Figure 7: The decoder fusion block in the encoder-decoder network.

The structure of the Lightweight MHSA module is shown in the right-bottom of
Fig. 7. Compared with the original MHSA [45], this lightweight version adds a
$k\times k$ convolution (with step size $k$) to reduce the spatial size of $K$
and $V$, thus reducing the computational complexity and memory consumption.
Additionally, a learnable relative position encoder is used to provide the
relative position information between pixels. For the input feature
$X\in\mathbb{R}^{h\times w\times c}$, the Lightweight MHSA module computes the
output feature $Y\in\mathbb{R}^{h\times w\times c}$ as:

$\displaystyle X^{\prime}$ $\displaystyle=\mathrm{Conv_{k\times k}}(X)$ (7)
$\displaystyle Q$ $\displaystyle=\mathrm{Conv_{1\times 1,Q}}(X)$
$\displaystyle K$ $\displaystyle=\mathrm{Conv_{1\times 1,K}}(X^{\prime})$
$\displaystyle V$ $\displaystyle=\mathrm{Conv_{1\times 1,V}}(X^{\prime})$
$\displaystyle Attn$
$\displaystyle=\mathrm{Softmax}\left(\frac{Q\cdot{(K+\mathrm{RPE})}^{T}}{d_{h}}\right)$
$\displaystyle Y$ $\displaystyle=Attn\cdot V,$

where $\mathrm{RPE}\in\mathbb{R}^{\frac{h}{k}\times\frac{w}{k}\times c}$ is
the relative position encoder, $N_{h}$ is the number of multiple heads, and
$d_{h}$ is the dimension of each head.

After the features go through the Lightweight MHSA module, they are then fed
to the IRFFN [44]. It consists of 1$\times$1 convolutions and 3$\times$3
deepwise (DW) convolution, where the 3$\times$3 deep convolution can reduce
the computational cost while extracting local features. The outputs of the
first two convolutions are activated by GELU and BatchNorm. The whole IRFFN
computation process can be expressed as:

$\displaystyle X^{\prime}$ $\displaystyle=\mathrm{Conv}_{1\times 1}(X)$ (8)
$\displaystyle Y$ $\displaystyle=\mathrm{Conv}_{1\times
1}(\mathrm{DWConv}(X^{\prime})+X^{\prime}).$

####  3.3.3 Line Segment Detector

After obtaining the fused feature map, we use state-of-the-art line segment
detectors to conduct the line segment detection. The process is divided into
two stages, i.e., the line proposal network and the line classification
network. During the first stage, the junction proposal module and the line
proposal module are used to generate junction and line proposals,
respectively. Then the line and junction proposals are matched based on the
connection relationship to generate the final line segment candidates. For the
second stage, each line candidate is associated with a feature vector from the
fused feature map by geometric matching. Then the line classification network
is used to classify the line segments by ground truth binary supervision.
Finally, candidates with confidence scores higher than the set threshold are
selected as the final detection results.

By combining the feature fusion backbone with the line proposal and
classification network, line segment detection in motion-blurred images can be
achieved using synchronized visual images and EST data. The feature fusion
backbone network can be easily applied to existing line segment detectors by
simply replacing their feature extraction backbone. To verify the generality
of the feature fusion network, we apply it to the current state-of-the-art
line segment detectors HAWP [6] and ULSD [7], named FE-HAWP and FE-ULSD
respectively.

##  4 Experiments and Analysis

In this section, we first detail the dataset, the experimental settings, and
evaluation metrics. Then the significant component configurations of FE-LSD
are analyzed. Finally, FE-LSD is evaluated and compared with the state-of-the-
art.

###  4.1 Dataset

LSD datasets such as Wireframe [23] and YorkUrban [46] play a vital role in
LSD methods. However, there is no publicly available LSD dataset with
geometrically and temporally aligned RGB images and events. Additionally,
annotation from scratch on either motion-blurred RGB images or EST frames is
difficult to obtain accurate line segment annotations. Based on these
requirements, we first build a larger-scale synthetic dataset with the
existing Wireframe dataset [23]. Then, considering there may be large gaps
between synthetic and real data, we contribute a real dataset by collecting
real data and manually labeling line segments.

Figure 8: Synthetic LSD dataset generation with the registered motion-blurred
RGB image and events.

FE-Wireframe: The goal of synthetic data generation is to avoid laborious line
annotation over motion blurs. With the labeled Wireframe dataset [23], we use
ESIM [47] to generate the synthetic data, and the construction pipeline is
shown in Fig. 8. Firstly, the RGB frame $I$ is re-projected to 3D space and
then projected to image space to obtain frame $I_{0}$ using the initial camera
pose and camera intrinsics. With the simulated camera trajectory over a short
time window $T$, frame $I_{k}$ can be generated at any interpolation time
$t_{k}$. Next, using the two adjacent frames $I_{k-1}$ and $I_{k}$ at time
$t_{k-1}$ and $t_{k}$, we can obtain the intensity changes and use Eq. 3 to
generate the events $\mathcal{E}_{k}$. The time window $T$=$30$ ms, and the
threshold $C$ for event contrast is randomly sampled from a uniform
distribution $U(0.05,0.5)$ for every image sequence. Then we can obtain the
event data $\mathcal{E}=\bigcup_{k=0}^{N-1}\mathcal{E}_{k}$ and the simulated
frames $\mathcal{I}=\left\\{I_{k}\right\\}_{k=0}^{N}$ for $N$+$1$ interpolated
images. To directly use the original line annotation of the Wireframe dataset,
the camera motion trajectory is designed to ensure that the last image $I_{N}$
is exactly the original image. The motion-blurred image $I_{B}$ is then
obtained by averaging the $N$+$1$ images. Finally, we obtain the synthetic
event data $\mathcal{E}$, the synchronized motion-blurred image $I_{B}$, and
the line segment annotations. The FE-Wireframe dataset consists of 5000 pairs
of frame-event training samples and 462 pairs of testing samples.

FE-Blurframe: For the generation of the real dataset with Frame-Event Line
Segment detection in Motion-blur (FE-Blurframe), we build a dual camera system
similar to [48], which consists of a DAVIS 346 event camera, FLIR RGB camera,
and a beam splitter mounted in front of the two cameras. 52 data sequences are
collected on the campus of Wuhan University, including typical indoor and
outdoor scenes, such as libraries, dining rooms, gymnasiums, office buildings,
and classrooms. Each data sequence includes an event stream and APS images
from the event camera, and synchronized RGB images from the FLIR camera. The
resolution of the event camera and FLIR RGB camera are $346$$\times$$260$ and
$640$$\times$$512$, respectively. The frame rate for FLIR is set as 200 Hz to
ensure that a clear image can be captured for line segment annotation.

We crop 800 fragments from the 52 sequences, each fragment contains events
over 30 ms and the synchronized 7 RGB frames. Then, the Super SloMo [49]
algorithm is used to interpolate the RGB images from 7 to 40 frames for
motion-blurred image generation. For the annotation, we label on the last
clear image of the 30ms interval. We refer to the clear image and event data
to improve and validate the annotation quality. Finally, the real dataset
contains 800 samples, including clear images, synthetic blurred images, event
streams, and line segment annotation. The dataset is randomly divided into a
training set with 600 samples and a test set with 200 samples.

###  4.2 Experimental settings and evaluation metrics

Experimental setup. In the experiment, the temporal dimension of EST $B$ is
set to 5. The sizes of the image and EST are re-scaled to $512$$\times$$512$
before being input into the network. In the feature fusion backbone, the
number of pairwise hourglass modules is set to 2, and the number of feature
encoders in each module is set to 5. After the feature fusion backbone
network, we obtain a fused feature map with the size of
$128$$\times$$128$$\times$$256$. The parameters and the training
configurations of FE-HAWP are also set to be the same as the original HAWP.
The network is trained using the Adam optimizer [50]. The learning rate,
weight decay, and training batch size for network training are set to
$4$$\times$$10^{-4}$, $1$$\times$$10^{-4}$, and 4, respectively. In the line
proposal module, the numbers of junction proposals and line proposals are set
as $K_{\mathrm{junc}}$=300, $K_{\mathrm{line}}$=5000. To train the line
classifier, the numbers of positive and negative samples from LPN are set as
$N_{\mathrm{pos}}$=300, $N_{\mathrm{neg}}$=300, respectively. The threshold of
the binary classification confidence score is set the same as it is in HAWP
[6] and ULSD [7]. Additionally, we add $N_{\mathrm{pos}}^{*}$=300 positive
samples from ground truth line annotations to the training dataset. To obtain
the feature vector for each line, 32 points are uniformly sampled from the
line, and the dimension of the feature vector is set as 1024. Meanwhile, the
training utilizes a step decay learning strategy, the network is trained with
30 epochs. After 25 epochs, the learning rate is divided by 10. The data
argumentation used in this work includes horizontal and vertical flips, and
$180\degree$ rotation. The model training and testing are conducted on a
single NVIDIA RTX 3090 GPU.

Evaluation metrics. We use evaluation metrics commonly used in line segment
detection tasks, including APH, FH, mean structural Average Precision (msAP),
sAP5, sAP10, sAP15, junction mean Average Precision (mAPJ) and Frames Per
Second (FPS).

APH and FH are calculated using precision and recall, which follow the
traditional LSD [5] and L-CNN [23]. sAP is proposed in L-CNN [15] to reflect
the geometric structure. The distance between the predicted line and the GT
line is used to determine whether a predicted line segment is a true positive.
sAP5, sAP10, and sAP15 are the sAP values when the distance is less than 5,
10, and 15 pixels, respectively. While msAP is the average of sAP5, sAP10, and
sAP15. For the distance calculation:

$d(l,\hat{l})=\frac{1}{2}\min\left(\sum_{i=0}^{1}\left\|\boldsymbol{p}_{i}-\boldsymbol{\hat{p}}_{i}\right\|,\sum_{i=0}^{1}\left\|\boldsymbol{p}_{i}-\boldsymbol{\hat{p}}_{1-i}\right\|\right),$
(9)

where $\\{\boldsymbol{p}_{i}\\}_{i=0}^{1}$ and
$\\{\boldsymbol{\hat{p}}_{i}\\}_{i=0}^{1}$ are the start and end points of GT
and predicted line segments.

mAPJ is used to measure the precision of junction predictions, which is
computed in a similar way to the sAP of line segments. A true positive
junction is determined by calculating the Euclidean distance between predicted
and GT junctions. mAPJ is the average accuracy when setting the distance
thresholds as 0.5, 1, and 2 pixels.

###  4.3 Configuration analyses

In this section, we first conduct experiments on the synthetic FE-Wireframe
dataset to analyze the influence of the three main configurations, i.e., input
data fusion, event representation, and fusion strategy. Then, the model
trained on the FE-Wireframe dataset is fine-tuned on the real dataset to show
the transfer learning results.

TABLE I: Impact of the input data fusion on network performance

| F | E | sAP5 | sAP10 | sAP15 | msAP | mAPJ | APH | FH
---|---|---|---|---|---|---|---|---|---
HAWP | ✓ |  | 41.6 | 47.3 | 49.9 | 46.3 | 44.0 | 73.5 | 72.2
| ✓ | 25.3 | 29.7 | 32.0 | 29.0 | 31.2 | 54.4 | 59.1
✓ | ✓ | 45.1 | 50.4 | 52.9 | 49.5 | 46.8 | 75.0 | 73.2
ULSD | ✓ |  | 40.8 | 46.7 | 49.4 | 45.6 | 45.1 | 67.5 | 70.6
| ✓ | 25.9 | 30.6 | 33.1 | 29.8 | 32.6 | 48.4 | 59.2
✓ | ✓ | 47.0 | 52.7 | 55.2 | 51.7 | 48.8 | 72.2 | 73.7

  * 1

F - image frames, E - events

####  4.3.1 Input data fusion

To analyze the effect of input data on network performance, we train on the
blurred image only, the EST only, and the concatenation of both using the
original HAWP [6] and ULSD [7] with the stacked hourglass network backbone. It
should be noticed that the results on blurred images are retrained using
blurred images. The results are shown in Table I. As for single-modal input,
higher accuracy can be obtained using only image frames. This is because the
event data loses a lot of texture information, and event noises have a
negative impact on junction prediction. Most importantly, when using events to
enhance image-based line segment detection, the fusion using naive
concatenation can comprehensively improve the line detection performance. For
example, the msAP of HAWP on frame-event concatenation is 3.2 and 20.5 points
higher than the model trained with images and events, respectively. The ULSD
model shows consistent improvements with 6.1 and 21.9 points higher msAP.
These results demonstrate the benefits of edge-aware events over high-speed
motion blurs. Thus simple concatenation can fuse the complementary information
for line segment detection.

TABLE II: Impact of event representation on network performance | Event | sAP5 | sAP10 | sAP15 | msAP | mAPJ | APH | FH
---|---|---|---|---|---|---|---|---
FE-HAWP | EST | 48.7 | 53.9 | 56.2 | 53.0 | 49.4 | 77.1 | 75.1
Voxel Grid | 48.3 | 53.5 | 55.8 | 52.6 | 48.9 | 76.9 | 75.3
EC+SAE | 48.3 | 53.7 | 55.8 | 52.6 | 49.1 | 77.0 | 75.1
EC+SAE* | 47.3 | 52.5 | 54.8 | 51.5 | 47.8 | 76.0 | 74.1
FE-ULSD | EST | 50.9 | 56.5 | 58.8 | 55.4 | 51.1 | 75.3 | 75.9
Voxel Grid | 50.3 | 55.7 | 58.0 | 54.7 | 50.9 | 74.8 | 75.3
EC+SAE | 50.7 | 56.1 | 58.3 | 55.0 | 50.9 | 74.3 | 75.5
EC+SAE* | 49.4 | 55.0 | 57.3 | 53.9 | 50.4 | 74.0 | 74.7

####  4.3.2 Event representation

Different event representation determines the richness of the event
information, directly affecting the performance of FE-LSD. To investigate the
impact, four different event representations, i.e., Event Spike Tensor (EST)
[41], voxel grid [40], Event Count + Surface of Active Event(EC+SAE) [38, 39],
Event Count + Surface of Active Event without polarity (EC+SAE∗), are tested.
The channels of event frames are 10, 5, 4, and 2 for the four representations,
respectively. Experimental results are shown in Table II. Among the four
representations, when adding polarity information to the Voxel Grid and
EC+SAE∗, i.e., EST and EC+SAE, the comprehensive performance is improved for
all metrics. Besides, EST and Voxel Grid retain the temporal information
compared with EC+SAE and EC+SAE∗, therefore obtaining much better accuracy
than the other two representations, indicating the importance of the temporal
information for the motion-blurred line segment detection task. As for the
pixel-based metrics APH and FH, neither of them is affected by the event
representations, indicating that polarity and temporal information mainly
enhance the accuracy of structured line detection. Most importantly, the EST-
based line detectors have the highest accuracy, which encodes the polarity
information and retains the most temporal information. Therefore, EST is
selected as the final event representation of FE-LSD.

TABLE III: Impact of the fusion module on network performance | SFB | DFB | sAP5 | sAP10 | sAP15 | msAP | mAPJ | APH | FH
---|---|---|---|---|---|---|---|---|---
FE-HAWP |  |  | 43.6 | 49.1 | 51.5 | 48.1 | 45.8 | 74.5 | 73.5
✓ |  | 46.0 | 51.2 | 53.4 | 50.2 | 47.6 | 75.5 | 73.3
| ✓ | 48.5 | 53.7 | 55.9 | 52.7 | 48.9 | 76.8 | 74.6
✓ | ✓ | 48.7 | 53.9 | 56.2 | 53.0 | 49.4 | 77.1 | 75.1
FE-ULSD |  |  | 45.5 | 51.2 | 53.7 | 50.1 | 47.6 | 71.7 | 72.8
✓ |  | 46.5 | 52.2 | 54.5 | 51.1 | 48.9 | 71.1 | 73.2
| ✓ | 50.1 | 55.6 | 57.9 | 54.5 | 50.5 | 74.5 | 75.0
✓ | ✓ | 50.9 | 56.5 | 58.8 | 55.4 | 51.1 | 75.3 | 75.9

####  4.3.3 Fusion module analysis

For the fusion of image and event features, we test the effect of the proposed
SFB and DFB, and the results are shown in Table III. The SFB is directly
removed from the network when unused, and the element-by-element addition is
used when the DFB is unused. From Table III, we can see that the network has
the lowest accuracy when no fusion module is used, with 48.1$\%$ and 50.1$\%$
msAP for FE-HAWP and FE-ULSD, respectively. Then, msAP is improved by 2.1 and
1.0 points when the SFB is added, and it increases by 4.6 and 4.4 points when
only the DFB is used, for FE-HAWP and FE-ULSD, respectively. The gains brought
by DFB are larger than SFB, indicating the importance of the multi-scale
encoder-decoder fusion of deep features. Therefore, the global attention in
Transformer is helpful for line segment detection. When both SFB and DFB are
used, we obtain the highest accuracy with 53.2$\%$ and 55.2$\%$ msAP for FE-
HAWP and FE-ULSD, respectively. The above quantitative results verify the
effectiveness of the proposed SFB and DFB.

Figure 9: Comparison of Class Activation Map (CAM) visualization results on
the synthetic dataset.

Meanwhile, we further qualitatively analyze the effectiveness of the two
fusion blocks by visualizing the feature map using Class Activation Map (CAM)
[11]. The results are shown in Fig. 9. The first row shows some sample CAMs
trained using FE-HAWP without SFB and DFB, while the second row shows the
corresponding samples using FE-HAWP with both modules. The CAM reflects the
network attention distribution to different regions in the image, and the red
indicates greater attention to these parts. By adding the SFB and DFB, it is
clear that the network pays more attention to the edges and has stronger
responses to the endpoints of line segments. These visualizations validate the
importance of the SFB and DFB modules in improving the network’s ability to
recognize and locate line segments.

####  4.3.4 Transfer learning on real data

In our experiment, the synthetic data generated on the Wireframe dataset
relieves the burden of extensive data annotation on real images and events.
However, the model trained on synthetic data has generalization issues to the
real-world dataset. Therefore, we build a real dataset FE-Blurframeto conduct
transfer learning. To verify the effectiveness of transfer learning, the three
models, i.e., trained on synthetic only, trained on real only, and pre-trained
on synthetic and fine-tuned on real, are evaluated on the same real dataset.
The results are shown in Table IV. Compared with real data training (R) and
fused data training (F), it can be observed that the models trained on
synthetic only (S) obtain $30.7\%$ and $34.4\%$ msAP on the real-world testing
set for FE-HAWP and FE-ULSD, respectively. This performance degradation is
similar to the HAWP [6] when it is trained on Wireframe only and tested on the
YorkUrban dataset. The reasons are mainly two-fold: (i) The significant data
differences between synthetic and real datasets; (ii) The synthetic event data
is obtained based on the ideal event generation model, while the quality of
real event data is affected by the lens, noise, contrast threshold and other
factors for event cameras. Then comparing the transfer learning models (F)
with the models trained on synthetic (S) and real only (R), the performance of
the synthetically pre-trained model has improved substantially after being
fine-tuned on the real dataset and also has a significant improvement over the
model trained on real only. The msAPs are finally improved to $63.3\%$ and
$62.9\%$ for the FE-HAWP and FE-ULSD models, respectively.

TABLE IV: Effect of model transfer and fine-tuning on real dataset

| Train | sAP5 | sAP10 | sAP15 | msAP | mAPJ | APH | FH
---|---|---|---|---|---|---|---|---
FE-HAWP | S | 26.8 | 31.4 | 33.7 | 30.7 | 30.1 | 45.9 | 54.9
R | 47.5 | 53.0 | 55.4 | 52.0 | 50.9 | 74.0 | 73.9
F | 59.8 | 64.2 | 65.9 | 63.3 | 60.1 | 82.0 | 79.7
FE-ULSD | S | 30.4 | 35.3 | 37.6 | 34.4 | 31.4 | 48.0 | 62.5
R | 47.3 | 52.9 | 55.2 | 51.8 | 52.2 | 72.9 | 73.7
F | 59.3 | 63.8 | 65.7 | 62.9 | 61.0 | 77.8 | 77.1

  * -

S-training on the synthetic only, R - training on the real only, F - pre-
training on the synthetic and fine-tuning on real.

Figure 10: Fine-tuning impact on the training process

We further plot the curves of loss and $msAP$ in Fig. 10. The red curves are
trained from the pre-trained model, and the green curves are trained from
scratch using random initialization. It is clear that fine-tuning the pre-
trained model can speed up the convergence and improve the final performance.
These results indicate a notable difference between the synthetic and the
real-world dataset, which seriously affects the model’s generalization ability
when evaluating from simulation to the real world. Nevertheless, the large-
scale synthetic dataset is helpful for model pre-training, thus releasing the
data annotation labor and improving the convergence speed and accuracy on the
real dataset.

###  4.4 Main results for comparison

To evaluate the performance of the proposed FE-LSD, sufficient comparisons are
conducted on the synthetic FE-Wireframe and real FE-Blurframe datasets,
respectively. The competitors include traditional LSD [5], FBSD [11] (designed
for blurred images), the learning-based methods L-CNN [15], HAWP [6], LETR
[12]. These learning-based methods are directly evaluated using the officially
trained models on the motion-blurred images. For FE-Wireframe, we additionally
use the official HAWP and ULSD models on the deblurred images with RED-Net
[51], i.e., HAWP† and ULSD†. Finally, since the proposed FE-LSD takes both
image and event data inputs, these methods are retrained on concatenating
images and EST to give fair play (i.e., L-CNN‡, HAWP‡, ULSD‡ and LETR‡).

Figure 11: Result comparisons on FE-Wireframe dataset (Lines are plotted on
clear images but detected on the motion-blurred images and events)

####  4.4.1 Results on the FE-Wireframe dataset

Fig. 11 shows the qualitative comparisons on the FE-Wireframe dataset. The
first row is the input motion-blurred images, and the second is the
corresponding event inputs. For better result visualization, the detected
lines and the manually-labeled lines are plotted on the clear RGB image at the
end moment of exposure. Since LSD and FBSD are gradient-based methods, they
can detect a large number of segments in motion-blurred images. However, the
majority of the line segments are false detection or fragmented segments,
resulting in low quantitative metrics. On the other hand, the learning-based
methods HAWP and LETR have a low detection rate on blurry images when using
the pre-trained models. This is because the officially trained models are
driven by clear images and cannot handle the motion-blur issues. However, when
we retrain them using the concatenation of blurry images and events, the
correct detection is significantly increased. Finally, the proposed FE-HAWP
and FE-ULSD have the best performance with the highest detection rate and the
lowest false alarm compared to the ground truth.

TABLE V: Quantitative comparisons on synthetic FE-Wireframe dataset

Method | sAP5 | sAP10 | sAP15 | msAP | mAPJ | APH | FPS
---|---|---|---|---|---|---|---
LSD[5] | 0.1 | 0.6 | 1.1 | 0.6 | 3.0 | 19.5 | 76.7
FBSD[11] | 0.2 | 0.4 | 0.9 | 0.5 | 2.9 | 24.9 | 21.7
L-CNN[15] | 3.4 | 5.1 | 6.2 | 4.9 | 7.0 | 22.7 | 28.8
HAWP[6] | 3.5 | 5.1 | 6.3 | 5.0 | 6.8 | 21.7 | 36.6
HAWP† | 7.0 | 9.8 | 11.2 | 9.3 | 9.3 | 24.8 | 39.7
ULSD [7] | 3.5 | 5.3 | 6.8 | 5.2 | 7.5 | 20.2 | 39.7
ULSD† | 6.8 | 9.8 | 11.6 | 9.4 | 10.6 | 16.6 | 41.0
LETR[12] | 2.8 | 5.0 | 6.5 | 4.8 | 7.3 | 21.9 | 4.2
L-CNN‡ | 40.6 | 45.8 | 48.2 | 44.8 | 45.6 | 70.5 | 10.6
HAWP‡ | 45.1 | 50.4 | 52.9 | 49.5 | 46.8 | 75.0 | 26.8
ULSD‡ | 47.0 | 52.7 | 55.2 | 51.7 | 48.8 | 72.2 | 32.2
LETR‡ | 24.7 | 34.7 | 39.7 | 33.1 | 25.4 | 66.1 | 3.9
FE-HAWP | 48.7 | 53.9 | 56.2 | 53.0 | 49.4 | 77.1 | 13.1
FE-ULSD | 50.9 | 56.5 | 58.8 | 55.4 | 51.1 | 75.3 | 13.9

  * 1

${{\dagger}}$ deblur first using [51] and detect line segments using the
corresponding official models.

  * 1

${{\ddagger}}$ retrain with the concatenation of image and EST.

Table V summarizes the quantitative comparisons on the synthetic FE-Wireframe
dataset, while Fig. 12 shows the PR curve of sAP10 and APH, respectively. When
only given the motion-blurred images, the traditional LSD, FBSD, and deep
learning-based L-CNN, HAWP, ULSD and HAWP have low accuracy. The highest msAP
is obtained by ULSD for $5.2\%$, and FBSD obtains the highest APH for
$24.9\%$, respectively. Image deblur with events can improve the subsequent
line segment detection performance. However, these results are far from the
standard for SLAM and 3D reconstruction. Then, when taking both image and EST
inputs, the retrained models perform much better than only blurry image input.
The highest msAP is obtained by the retrained ULSD${\ddagger}$ for $51.7\%$.
These improvements demonstrate that the event data can enhance the edge
awareness for motion-blurred images. Finally, we obtain the highest
performance for all metrics using our proposed frame-event fusion module. The
msAPs of FE-HAWP and FE-USLD are higher than the retrained ULSD${\ddagger}$
for 1.3 and 3.7 points, respectively. Except for APH, FE-ULSD obtains the
highest accuracy, while APH is slightly lower than FE-HAWP. This is caused by
the low recall rate of FE-ULSD when the confidence threshold is set to 0.5, as
shown in the APH PR curves.

Regarding to the detection speed, both FE-ULSD and FE-HAWP are slower than the
original methods because they use a feature fusion backbone with one more
encoder branch than the original stacked hourglass network and introduce the
more time-consuming Transformer for feature fusion. Nevertheless, compared
with the transformer-based method LETR [12], the proposed method is nearly
three times faster. These results prove the effectiveness of the proposed
method.

Figure 12: PR curves of sAP10 and APH on the FE-Wireframe dataset

####  4.4.2 Results on FE-Blurframe dataset

Table VI and Fig. 13 give the quantitative comparisons on the collected FE-
Blurframe dataset. The proposed FE-HAWP and FE-ULSD still achieve the best
accuracy compared with other learning-based methods. FE-HAWP obtains $52.0\%$
msAP, $9.4$ points higher than the retrained HAWP${\dagger}$. FE-ULSD obtains
$51.8\%$ msAP, $5.1$ points higher than the retrained ULSD${\dagger}$. Because
the Bezier equipartition representation of line segments in ULSD is more
sensitive to image noises than the attraction field map representation in
HAWP, the gain of FE-ULSD is not as significant as FE-HAWP for real images.
Furthermore, if we use the transfer learning strategy of fine-tuning on the
pre-trained models, i.e., FE-HAWP‡ and FE-ULSD‡, the performances are improved
by $11.3$ and $11.1$ points compared with FE-HAWP and FE-ULSD, respectively.

TABLE VI: Quantitative comparisons on the FE-Blurframe dataset

Method | sAP5 | sAP10 | sAP15 | msAP | mAPJ | APH | FPS
---|---|---|---|---|---|---|---
LSD[5] | 1.1 | 2.8 | 4.1 | 2.7 | 5.1 | 29.4 | 61.0
FBSD[11] | 0.9 | 1.9 | 2.7 | 1.8 | 5.1 | 34.2 | 15.9
L-CNN[15] | 7.5 | 11.5 | 13.7 | 10.9 | 12.4 | 27.9 | 29.7
HAWP[6] | 8.4 | 12.8 | 15.3 | 12.2 | 12.4 | 32.0 | 38.1
ULSD [7] | 6.8 | 10.8 | 13.0 | 10.2 | 11.8 | 26.7 | 40.6
LETR[12] | 7.1 | 13.0 | 16.8 | 12.3 | 12.1 | 30.2 | 3.6
L-CNN† | 34.0 | 40.3 | 43.0 | 39.1 | 40.3 | 66.0 | 17.7
HAWP† | 37.0 | 43.9 | 46.9 | 42.6 | 41.6 | 67.9 | 29.0
ULSD† | 42.0 | 47.8 | 50.4 | 46.7 | 48.5 | 67.0 | 32.2
LETR† | 22.6 | 33.8 | 38.8 | 31.7 | 23.2 | 57.7 | 3.3
FE-HAWP | 47.5 | 53.0 | 55.4 | 52.0 | 50.9 | 74.0 | 12.9
FE-ULSD | 47.3 | 52.9 | 55.2 | 51.8 | 52.2 | 72.9 | 12.9
FE-HAWP‡ | 59.8 | 64.2 | 65.9 | 63.3 | 60.1 | 82.0 | 13.3
FE-ULSD‡ | 59.3 | 63.8 | 65.7 | 62.9 | 61.0 | 77.8 | 13.9

  * 1

${{\dagger}}$ retrain with the concatenation of image and EST.

  * 1

${{\ddagger}}$ pre-train on FE-Wireframe and fine-tune on FE-Blurframe.

Figure 13: PR curves of sAP10 and APH on FE-Blurframe.

Fig. 14 shows the qualitative results on FE-Blurframe dataset. The
conventional LSD and FBSD have consistent performances as on the synthetic FE-
Wireframe dataset. The officially trained models, HAWP and LETR, still have
large missing detection but with higher accuracy than the same model on the
synthetic FE-Wireframe dataset. This is because of the significant data
differences between synthetic and real-world datasets. Then retraining on the
concatenation of images and events greatly improves the performance and yields
more detected line segments. Similar to the results on FE-Wireframe, the
proposed FE-HAWP and FE-ULSD detect the most line segments with the highest
similarity to the ground truth labels, reflecting the effectiveness of the
proposed method.

Figure 14: Result comparisons on real FE-Blurframedataset (Lines are plotted
on clear images but detected on the motion-blurred images and events)

###  4.5 Discussions

By fusing image and event data, we effectively improve the performance of line
segment detection on motion-blurred images. However, it is still hard to
determine the performance of the fusion over diverse camera motions.
Therefore, we analyze the motion blur degree by counting the average
displacement of junctions. The distributions of displacement and the
corresponding line detection performance are plotted in Fig. 15, where FE-HAWP
and HAWP are evaluated. The x-axis represents the blurriness, which is
computed by averaging junction flow from the start image to the end image for
each data. It can be observed that the blurriness is between 0-60 pixels, and
the blur distributions of the synthetic FE-Wireframe and the real FE-
Blurframeare slightly different. The msAP distributions of official HAWP
indicate that larger blurs will result in lower line detection accuracy. Then
when using the proposed FE-HAWP on the fusion of images and events, the line
detection accuracy is much higher and more robust to the different blurriness.

Figure 15: Motion blur distributions and the line detection results upon the
displacements. Top-left: Motion blur distributions on FE-Wireframe; Top-right:
Motion blur distributions on FE-Blurframe; Bottom-left: msAP distributions of
HAWP and FE-HAWP on FE-Wireframe; Bottom-right: msAP distributions of HAWP and
FE-HAWP on FE-Blurframe.

##  5 Conclusion

In this paper, we have presented a line segment detection method using events
to address its performance degradation in motion-blurred images. The proposed
frame-event feature fusion backbone fully exploits the complementary
information between images and event data with shallow and multi-scale decoder
fusion. Therefore, the structural edge information can be well extracted in
the feature map regardless of slow or fast camera motions. The state-of-the-
art line segment detectors, HAWP and ULSD, are then employed to conduct the
end-to-end line segment detection. To train our model and contribute to the
community, two frame-event line segment detection datasets, FE-Wireframe and
FE-Blurframe, are constructed with motion-blurred images and spatially-
temporally-aligned event data. Extensive component configuration analyses
validate the reasonability of our design, and comprehensive comparisons to the
state-of-the-arts demonstrate the effectiveness of the proposed method. In the
future, we will further explore the line endpoints flow in event streams to
facilitate line segment detection and improve efficiency.

## References

  * [1] M. Jin, G. Meishvili, and P. Favaro, “Learning to extract a video sequence from a single motion-blurred image,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 6334–6342.
  * [2] F. Munir, S. Azam, M. Jeon, B.-G. Lee, and W. Pedrycz, “LDNet: End-to-end lane marking detection approach using a dynamic vision sensor,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 7, pp. 9318–9334, 2022.
  * [3] R. Gomez-Ojeda, F.-A. Moreno, D. Zuniga-Noël, D. Scaramuzza, and J. Gonzalez-Jimenez, “PL-SLAM: A stereo SLAM system through the combination of points and line segments,” _IEEE Transactions on Robotics_ , vol. 35, no. 3, pp. 734–746, 2019.
  * [4] Q. Wang, T. Han, Z. Qin, J. Gao, and X. Li, “Multitask attention network for lane detection and fitting,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 33, no. 3, pp. 1066–1078, 2022.
  * [5] R. Grompone von Gioi, J. Jakubowicz, J.-M. Morel, and G. Randall, “LSD: A fast line segment detector with a false detection control,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 32, no. 4, pp. 722–732, 2010.
  * [6] N. Xue, T. Wu, S. Bai, F. Wang, G.-S. Xia, L. Zhang, and P. H. Torr, “Holistically-attracted wireframe parsing,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 2788–2797.
  * [7] H. Li, H. Yu, J. Wang, W. Yang, L. Yu, and S. Scherer, “ULSD: Unified line segment detection across pinhole, fisheye, and spherical cameras,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 178, pp. 187–202, 2021.
  * [8] Y. Xu, W. Xu, D. Cheung, and Z. Tu, “Line segment detection using transformers without edges,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 4255–4264.
  * [9] W. Xu and S. McCloskey, “2d barcode localization and motion deblurring using a flutter shutter camera,” in _Proceedings of the IEEE Workshop on Applications of Computer Vision_ , 2011, pp. 159–165.
  * [10] K. Purohit, A. Shah, and A. Rajagopalan, “Bringing alive blurred moments,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 6830–6839.
  * [11] P. Even, P. Ngo, and B. Kerautret, “Thick line segment detection with fast directional tracking,” in _Proceedings of the International Conference on Image Analysis and Processing_ , 2019, pp. 159–170.
  * [12] Y. Xu, W. Xu, D. Cheung, and Z. Tu, “Line segment detection using transformers without edges,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 4255–4264.
  * [13] I. Debled-Rennesson, F. Feschet, and J. Rouyer-Degli, “Optimal blurred segments decomposition of noisy shapes in linear time,” _Computers & Graphics_, vol. 30, no. 1, pp. 30–36, 2006.
  * [14] L. Buzer, “A simple algorithm for digital line recognition in the general case,” _Pattern Recognition_ , vol. 40, no. 6, pp. 1675–1684, 2007.
  * [15] Y. Zhou, H. Qi, and Y. Ma, “End-to-end wireframe parsing,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 962–971.
  * [16] S. Tulyakov, D. Gehrig, S. Georgoulis, J. Erbach, M. Gehrig, Y. Li, and D. Scaramuzza, “Time lens: Event-based video frame interpolation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 16 155–16 164.
  * [17] S. Seifozzakerini, W.-Y. Yau, B. Zhao, and K. Mao, “Event-based hough transform in a spiking neural network for multiple line detection and tracking using a dynamic vision sensor,” in _Proceedings of the British Machine Vision Conference_ , 2016, pp. 94.1–94.12.
  * [18] C. Brandli, J. Strubel, S. Keller, D. Scaramuzza, and T. Delbruck, “ELiSeD — an event-based line segment detector,” in _Proceedings of the Second International Conference on Event-based Control, Communication, and Signal Processing_ , 2016, pp. 1–7.
  * [19] Z. Xu, B.-S. Shin, and R. Klette, “Accurate and robust line segment extraction using minimum entropy with hough transform,” _IEEE Transactions on Image Processing_ , vol. 24, no. 3, pp. 813–822, 2015.
  * [20] E. J. Almazàn, R. Tal, Y. Qian, and J. H. Elder, “MCMLSD: A dynamic programming approach to line segment detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 5854–5862.
  * [21] C. Akinlar and C. Topal, “EDLines: A real-time line segment detector with a false detection control,” _Pattern Recognition Letters_ , vol. 32, no. 13, pp. 1633–1642, 2011.
  * [22] N.-G. Cho, A. Yuille, and S.-W. Lee, “A novel linelet-based representation for line segment detection,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 40, no. 5, pp. 1195–1208, 2018.
  * [23] K. Huang, Y. Wang, Z. Zhou, T. Ding, S. Gao, and Y. Ma, “Learning to parse wireframes in images of man-made environments,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 626–635.
  * [24] N. Xue, S. Bai, F.-D. Wang, G.-S. Xia, T. Wu, L. Zhang, and P. H. Torr, “Learning regional attraction for line segment detection,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 43, no. 6, pp. 1998–2013, 2021.
  * [25] Z. Zhang, Z. Li, N. Bi, J. Zheng, J. Wang, K. Huang, W. Luo, Y. Xu, and S. Gao, “PPGNet: Learning point-pair graph for line segment detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 7098–7107.
  * [26] Q. Meng, J. Zhang, Q. Hu, X. He, and J. Yu, “LGNN: A context-aware line segment detector,” in _Proceedings of the 28th ACM International Conference on Multimedia_ , 2020, pp. 4364–4372.
  * [27] Y. Lin, S. L. Pintea, and J. C. van Gemert, “Deep hough-transform line priors,” in _Proceedings of the European Conference on Computer Vision_ , 2020, pp. 323–340.
  * [28] R. Pautrat, J.-T. Lin, V. Larsson, M. R. Oswald, and M. Pollefeys, “SOLD2: Self-supervised occlusion-aware line description and detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 11 363–11 373.
  * [29] N. Kong, K. Park, and H. Goka, “Hole-robust wireframe detection,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2022, pp. 2684–2693.
  * [30] I. Debled-Rennesson, R. Jean-Luc, and J. Rouyer-Degli, “Segmentation of discrete curves into fuzzy segments,” _Electronic Notes in Discrete Mathematics_ , vol. 12, pp. 372–383, 2003.
  * [31] B. Kerautret and P. Even, “Blurred segments in gray level images for interactive line extraction,” in _International Workshop on Combinatorial Image Analysis_ , 2009, pp. 176–186.
  * [32] D. Reverter Valeiras, X. Clady, S.-H. Ieng, and R. Benosman, “Event-based line fitting and segment detection using a neuromorphic visual sensor,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 30, no. 4, pp. 1218–1230, 2019.
  * [33] F. Tschopp, C. von Einem, A. Cramariuc, D. Hug, A. W. Palmer, R. Siegwart, M. Chli, and J. Nieto, “Hough2Map – iterative event-based hough transform for high-speed railway mapping,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 2745–2752, 2021.
  * [34] G. Gallego, T. Delbrück, G. Orchard, C. Bartolozzi, B. Taba, A. Censi, S. Leutenegger, A. J. Davison, J. Conradt, K. Daniilidis, and D. Scaramuzza, “Event-based vision: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 1, pp. 154–180, 2022.
  * [35] L. Everding and J. Conradt, “Low-latency line tracking using event-based dynamic vision sensors,” _Frontiers in Neurorobotics_ , vol. 12, 2018.
  * [36] W. Cheng, H. Luo, W. Yang, L. Yu, S. Chen, and W. Li, “Det: A high-resolution dvs dataset for lane extraction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2019, pp. 0–0.
  * [37] A. Dietsche, G. Cioffi, J. Hidalgo-Carrió, and D. Scaramuzza, “Powerline tracking with event cameras,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2021, pp. 6990–6997.
  * [38] A. I. Maqueda, A. Loquercio, G. Gallego, N. García, and D. Scaramuzza, “Event-based vision meets deep learning on steering prediction for self-driving cars,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 5419–5427.
  * [39] R. Benosman, C. Clercq, X. Lagorce, S.-H. Ieng, and C. Bartolozzi, “Event-based visual flow,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 25, no. 2, pp. 407–417, 2013.
  * [40] A. Z. Zhu, L. Yuan, K. Chaney, and K. Daniilidis, “Unsupervised event-based learning of optical flow, depth, and egomotion,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 989–997.
  * [41] D. Gehrig, A. Loquercio, K. Derpanis, and D. Scaramuzza, “End-to-end learning of representations for asynchronous event-based data,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 5632–5642.
  * [42] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.
  * [43] A. Newell, K. Yang, and J. Deng, “Stacked hourglass networks for human pose estimation,” in _Proceedings of the European Conference on Computer Vision_ , vol. 9912, 2016, pp. 483–499.
  * [44] J. Guo, K. Han, H. Wu, C. Xu, Y. Tang, C. Xu, and Y. Wang, “CMT: Convolutional neural networks meet vision transformers,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 12 175–12 185.
  * [45] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in Neural Information Processing Systems_ , vol. 30, 2017.
  * [46] N.-G. Cho, A. Yuille, and S.-W. Lee, “A novel linelet-based representation for line segment detection,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 40, no. 5, pp. 1195–1208, 2017.
  * [47] H. Rebecq, D. Gehrig, and D. Scaramuzza, “ESIM: an open event camera simulator,” in _Proceedings of the 2nd Conference on Robot Learning_ , 2018, pp. 969–982.
  * [48] Z. W. Wang, P. Duan, O. Cossairt, A. Katsaggelos, T. Huang, and B. Shi, “Joint filtering of intensity images and neuromorphic events for high-resolution noise-robust imaging,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 1606–1616.
  * [49] H. Jiang, D. Sun, V. Jampani, M.-H. Yang, E. Learned-Miller, and J. Kautz, “Super slomo: High quality estimation of multiple intermediate frames for video interpolation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 9000–9008.
  * [50] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _Proceedings of the International Conference on Learning Representations_ , 2015.
  * [51] F. Xu, L. Yu, B. Wang, W. Yang, G.-S. Xia, X. Jia, Z. Qiao, and J. Liu, “Motion deblurring with real events,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 2583–2592.