Stereo Visual Odometry with Deep Learning-Based Point and Line Feature
  Matching using an Attention Graph Neural Network
Robust feature matching forms the backbone for most Visual Simultaneous
Localization and Mapping (vSLAM), visual odometry, 3D reconstruction, and
Structure from Motion (SfM) algorithms. However, recovering feature matches
from texture-poor scenes is a major challenge and still remains an open area of
research. In this paper, we present a Stereo Visual Odometry (StereoVO)
technique based on point and line features which uses a novel feature-matching
mechanism based on an Attention Graph Neural Network that is designed to
perform well even under adverse weather conditions such as fog, haze, rain, and
snow, and dynamic lighting conditions such as nighttime illumination and glare
scenarios. We perform experiments on multiple real and synthetic datasets to
validate the ability of our method to perform StereoVO under low visibility
weather and lighting conditions through robust point and line matches. The
results demonstrate that our method achieves more line feature matches than
state-of-the-art line matching algorithms, which when complemented with point
feature matches perform consistently well in adverse weather and dynamic
lighting conditions.
Shenbagaraj Kannapiran, Nalin Bendapudi, Ming-Yuan Yu, Devarth Parikh, Spring Berman, Ankit Vora, Gaurav Pandey
Stereo Visual Odometry with Deep Learning-Based Point and Line Feature Matching using an Attention Graph Neural Network

###### Abstract

Robust feature matching forms the backbone for most Visual Simultaneous Localization and Mapping (vSLAM), visual odometry, 3D reconstruction, and Structure from Motion (SfM) algorithms. However, recovering feature matches from texture-poor scenes is a major challenge and still remains an open area of research. In this paper, we present a Stereo Visual Odometry (StereoVO) technique based on point and line features which uses a novel feature-matching mechanism based on an Attention Graph Neural Network that is designed to perform well even under adverse weather conditions such as fog, haze, rain, and snow, and dynamic lighting conditions such as nighttime illumination and glare scenarios. We perform experiments on multiple real and synthetic datasets to validate our method's ability to perform StereoVO under low-visibility weather and lighting conditions through robust point and line matches. The results demonstrate that our method achieves more line feature matches than state-of-the-art line-matching algorithms, which when complemented with point feature matches perform consistently well in adverse weather and dynamic lighting conditions.

## I Introduction

With the advances in development and deployment of self-driving vehicles and mobile robots, there is a growing need for high-resolution, accurate visual odometry algorithms that can be deployed on low-cost camera sensors. Although existing localization algorithms perform well under ideal conditions, they usually tend to fail or under-perform in adverse weather conditions such as fog, rain, and snow and in dynamic lighting conditions such as glare and nighttime illumination. The development of visual odometry algorithms that perform effectively under such conditions remains an open area of research.

Existing vision-based localization algorithms rely primarily on conventional point features, such as SIFT [3], SURF [4], and ORB [5], or learning-based point features, such as SuperPoint [6] and LIFT [7], to perform temporal feature matching or learn alternate representations using monocular camera(s) [8, 9] and obtain camera pose estimates. However, point features become unreliable in the adverse conditions mentioned above. To overcome this, we develop a visual odometry technique that includes line segment features in addition to point features (see example application in Fig. 1). In foggy scenes where conventional point features like ORB and SIFT are unable to detect enough features, learning-based feature detectors like SuperPoint perform well. Line feature detectors such as SOLD2 [10] and L2D2 [11] also perform well in such scenarios. However, most point feature detectors, including ORB, SIFT, and SuperPoint, tend to demonstrate poor performance in nighttime scenarios. In comparison, the line features detected by state-of-the-art line detectors such as SOLD2 and L2D2 tend to remain consistent in scenes with low illumination, thereby indicating the need to leverage line features for nighttime conditions.

Fig. 1: Grayscale images of (a) a clear-sky scenario from the Ford AV dataset [1]; (b) a nighttime scenario from the Oxford Car dataset [2]. Rows 1 and 3 show point matches between frames \(i\) and \(i+1\); rows 2 and 4 show line matches between frames \(i\) and \(i+1\). Point and line matches were generated by the method presented in this paper (Method 2 in Section IV).
Given the importance of line features, the next step in integrating line features with the visual odometry framework is to perform feature-matching for line features. Existing techniques either utilize line-based descriptors or apply point descriptors to points sampled from detected lines. Both types of techniques rely on visual descriptors, which tend to fail in texture-poor scenes, justifying the need for a line-matching solution that is constrained by the positions of line features and visual cues. Position-constrained line-matching ensures the prevention of line feature mismatches, particularly in scenes where point features are sparse or similar structures appear repeatedly, such as trusses of a bridge and windows in urban high-rises.

Apart from vision-based sensors, inertial sensors such as Inertial Measurement Units (IMUs) can be used to aid the system to perform Visual-Inertial Odometry (VIO), which yields better accuracy. However, we restricted our focus to just vision-based systems to showcase our method's capabilities without the aid of other such sensors. We developed our method with the goal of easily integrating it into a standard self-driving research vehicle for real-time deployment, and hence we assume that our visual-odometry based pose estimate will ultimately be fused with GNSS (Global Navigation Satellite System) and IMU-based pose estimates in an extended Kalman filter-based framework to provide more accurate pose estimates.

Our contributions can be summarized as follows:

* We developed a novel line-matching technique using an Attention Graph Neural Network that is capable of acquiring robust line matches in feature-poor scenarios by sampling and detecting self-supervised learning-based point features along the lines with encoded position constraints.
* We integrated point features and fine-tuned line features in a Stereo Visual Odometry framework to maintain consistent performance in adverse weather and dynamic lighting conditions and compared the performance of our method to that of state-of-the-art point and line feature matching techniques.

We discuss related work in Section II, give a technical description of our approach in Section III, and describe experiments and results in Section IV. Section V concludes the paper and provides an outlook on future work.

## II Related Work

In this section, we give an overview of previous work related to visual odometry, graph matching, point feature matching, and line matching. Given the amount of prior research, particularly in the visual odometry field (e.g., [12, 13, 14]), a compilation of all existing visual odometry algorithms is beyond the scope of this paper. Visual odometry (VO) techniques are usually classified as either direct VO or feature-based VO. Feature-based solutions are primarily used for their reliability, high accuracy, and robustness, and will therefore be the focus of this paper.

### _Point feature detection and matching_

Point feature detection lies at the heart of most vision-based algorithms. The paper [15] presents a comprehensive survey of different classical feature detectors such as SIFT, ORB, and SURF and learning-based detectors such as LF-Net [16] and SuperPoint and compares their performance on three evaluation tasks in terms of robustness, repeatability, and accuracy. Classical feature-matching techniques usually involve finding descriptors, matching them using a nearest neighbor search, and finally removing outliers to obtain robust matches. Over time, researchers moved towards developing more robust and accurate feature detectors and descriptors to improve matching. Then graph neural network (GNN)-based matching systems such as SuperGlue [17] were developed, which outperformed all existing feature matching techniques by using an attentional GNN. The attention mechanism enables the GNN to selectively focus on the most relevant features (i.e., nodes and edges) when comparing two graphs, which improves the accuracy of feature matching and its robustness to noisy or incomplete graphs. For this reason, we used SuperGlue for point feature matching in our Stereo Visual Odometry framework.

### _Line feature detection and matching_

Line feature detection and matching is a well-researched topic. Classical line detector algorithms rely on geometric constraints to extract lines and find correspondences. Similarly, line segment descriptors can be constructed from the appearance of a neighborhood of the detected line, without resorting to any other photometric or geometric constraints such as the mean-standard deviation line descriptor (MLSD) [18], which constructs the line descriptors by computing the mean and variance of the gradients of pixels in the neighboring region of a line segment. The work [19] proposes a Line Band Descriptor (LBD) that computes gradient histograms over bands for improved robustness and efficiency. Recent advancements in learning-based line segment descriptors, e.g., LLD [20] and DLD [21], demonstrate excellent performance with the use of a convolutional neural network (CNN) to learn the line descriptors. In [11], the authors propose a novel line segment detector and descriptor, Learnable Line Detector and Descriptor (L2D2), which enables efficient extraction and matching of 2D lines via the angular distance of 128-dimensional unit descriptor vectors. The paper [22] presents a novel Graph Convolutional Network-based line segment matching technique that learns local line segment descriptors through end-to-end training.

In [10], the authors propose SOLD2, a self-supervised learning-based line detector that is similar to SuperPoint and does not require any annotation, enabling the system to generalize to multiple scenarios. For this reason, we chose SOLD2's line detector module as a baseline for our method. SOLD2 also includes a line-matching algorithm to enable occlusion awareness. However, unlike SuperGlue, SOLD2's matching algorithm does not take advantage of the position information of the features, which is critical in scenes that
contain repetitive structures such as windows in urban high-rises.

### _Visual SLAM / Odometry with point and line features_

As described in [23, 24], visual SLAM methods that incorporate both point and line features have been developed to improve localization accuracy and computational efficiency over conventional point-based approaches in challenging scenarios, making the VO pipeline more comprehensive and robust to real-world conditions. One example is the visual-inertial SLAM method in [25], which includes several enhancements in line detection and an optical flow-based line feature tracker. Another is the line classification methodology for a Visual Inertial Odometry system that is presented in [26], which exploits the distinctive characteristics of structural (parallel) and non-structural (non-parallel) line features to develop a two-parameter line feature representation, leading to more efficient SLAM computations. However, despite the benefits afforded by using both point and line features, these visual SLAM techniques often exhibit poor performance in scenarios with repeated, similar-looking point and line features, such as those found in traffic environments (e.g., building facades, pedestrian crosswalks). The design of our StereoVO technique was motivated in part by this limitation.

## III StereoVO WITH POINTS AND LINES

Our proposed StereoVO framework is developed to perform well in texture-poor scenarios and relies on tracking a set of point and line correspondences. The framework is based on the SuperGlue [17] network, with an additional constraint (constraint (3) in Section III-B) that greatly improves performance. We first provide a overview of the StereoVO framework, followed by the notation and definitions that we use in our Attention Graph Neural Network architecture. This is followed by a description of the Optimal Matching layer for both point and line features, and a brief summary of how to obtain pose estimates from the point and line correspondences.

### _Overview_

An outline of the proposed StereoVO framework is shown in Fig. 2. The stereo images obtained from the camera are initially undistorted. The left camera image is used to obtain point and line matches, and the right camera image is used to obtain 3D points and 3D line estimates from disparity maps generated using the stereo images.

To improve the accuracy of StereoVO, the left camera image is pre-processed using a semantic segmentation algorithm to remove dynamic objects, such as cars and pedestrians, thereby generating a mask that highlights static features in the scene. In StereoVO, focusing on stable features improves the reliability, precision, and robustness of the camera pose estimates. We employ SegFormer [27], a state-of-the-art semantic segmentation algorithm, out of the box to mask the classes of interest.

In the next step, we perform point and line segment feature detection on the masked input images. Since our goal is to implement the framework on a full-size autonomous vehicle and ensure that it is capable of performing well under adverse weather and dynamic lighting conditions, we tested a variety of point feature detectors, including SIFT, ORB, and LIFT, in example scenarios with such conditions and chose SuperPoint since it outperformed the others. SuperPoint is a CNN-based self-supervised framework that is trained on the MS-COCO dataset [28] and performs real-time point feature detection out of the box without any fine-tuning.

We selected SOLD2 for line feature matching, since it has a similar CNN architecture to SuperPoint. To improve the performance of SOLD2 in low-light conditions and other adverse weather conditions, we fine-tuned the network on synthetic data generated using the CARLA driving simulator [29]. Since SOLD2 performs well in ideal daytime conditions, we used the line features detected by SOLD2 as ground truth and changed weather and lighting conditions for the same scenes in CARLA to generate multi-weather and lighting-augmented data. The SOLD2 algorithm also performs line matching by sampling lines and performing feature matching between the samples to aid in occlusion awareness. However, this results in incorrect matches in feature-poor scenarios. To overcome this, we introduced position constraints on the line features by sampling points along the lines, using SuperPoint to detect point features from these sets of sampled points, and encoding the point

Fig. 2: Outline of the proposed Stereo Visual Odometry framework.
features with keypoint position awareness. This is explained further in the next sections. Both the point and line feature detection networks are designed to function effectively across various datasets without being overly reliant on specific training data. Their generalizability enables them to detect point and line features in diverse contexts and domains, making them versatile tools for a wide range of applications. Figure 3 provides additional details of the framework.

### _Notation and definitions_

Consider a pair of images labeled \(A\) and \(B\). Each image \(\alpha\in\{A,B\}\) has \(L_{\alpha}\) line features, indexed by set \(\mathcal{L}_{\alpha}\subset\mathbb{Z}_{+}\), and \(P_{\alpha}\) point features, indexed by set \(\mathcal{P}_{\alpha}\subset\mathbb{Z}_{+}\) and referred to as _P-point features_. The pixels comprising the line features are extracted using SOLD2 line feature extractor, and SuperPoint is used to detect \(Q_{\alpha}\) point features from these pixels, indexed by set \(\mathcal{Q}_{\alpha}\subset\mathbb{Z}_{+}\) and referred to as _L-point features_. Each point feature is associated with a position \(\mathbf{p}\) and a visual descriptor vector \(\mathbf{d}\). The feature position is defined as \(\mathbf{p}=[u\ v\ c]^{T}\), where \(u\) and \(v\) are the pixel coordinates of the point and \(c\) is the descriptor detection confidence. We will use the notation \((\mathbf{p}_{p})_{i}^{\alpha}\) to indicate the position of P-point feature \(i\in\mathcal{P}_{\alpha}\) in image \(\alpha\) and \((\mathbf{p}_{l})_{i}^{\alpha}\) to indicate the position of L-point feature \(i\in\mathcal{Q}_{\alpha}\) in image \(\alpha\). We define the visual descriptors \((\mathbf{d}_{p})_{i}^{\alpha}\), \((\mathbf{d}_{l})_{i}^{\alpha}\) similarly. The subscript \(x\) will refer to either \(p\) or \(l\).

The representation \((\mathbf{y}_{x})_{i}^{\alpha}\), \(x\in\{p,l\}\), for each keypoint \(i\) in image \(\alpha\) is a high-dimensional vector that encodes the keypoint's position and visual descriptor. The keypoint position is embedded into \((\mathbf{y}_{x})_{i}^{\alpha}\) as follows using a multi-layer perceptron (MLP), similar to the SuperGlue architecture:

\[(\mathbf{y}_{x})_{i}^{\alpha}=(\mathbf{d}_{x})_{i}^{\alpha}+MLP_{encoder}( \mathbf{p}_{x})_{i}^{\alpha} \tag{1}\]

Our framework enforces the following constraints. (1) Any P-point feature from one image has exactly one match to a P-point feature in the other image; similarly for L-point features. (2) All P-point features and L-point features that are occluded or undetected will be unmatched. (3) A line feature \(l_{a}\in\mathcal{L}_{A}\) in image \(A\) is matched to a line feature \(l_{b}\in\mathcal{L}_{B}\) in image \(B\) if most of the L-point features on \(l_{a}\) are matched to L-point features on \(l_{b}\).

### _Attention Graph Neural Network_

An Attention Graph Neural Network (GNN) forms the first layer of the architecture. The network encodes both the positions and visual descriptors of the keypoints, which ultimately improves the performance of the network over a conventional graph neural network. The position constraints increase line-matching robustness and ensure that incorrect line matches do not occur in cases where images contain repetitive structures, such as windows in high-rise buildings. We developed separate GNNs for point and line feature matching, one with nodes defined as the P-point features and the other with nodes defined as the L-point features. Each GNN has a different set of losses and weights, since the networks compute different estimates of the geometric and photometric cues.

As in the SuperGlue architecture, aggression is achieved through both self- and cross-attention mechanisms. Given a feature that corresponds to a particular node in one image, self-attention aggregates features that correspond to adjacent nodes in the same image, and cross-attention aggregates similar features that correspond to nodes in another image. The framework attends to individual point features' positions and their positions relative to adjacent point features, as in SuperGlue. Let \((\mathbf{h}_{x})_{i}^{\alpha}\), \(x\in\{p,l\}\), denote the matching descriptor for keypoint \(i\) in image \(\alpha\). The matching descriptors are defined as:

\[(\mathbf{h}_{x})_{i}^{\alpha}=\mathbf{W}_{x}(\mathbf{y}_{x})_{i}^{\alpha}+ \mathbf{b}_{x},\ \ x\in\{p,l\},\ \alpha\in\{A,B\}, \tag{2}\]

where \(\mathbf{W}_{x}\) is a weight matrix and \(\mathbf{b}_{x}\) is a bias vector.

### _Optimal Matching layer_

The Optimal Matching layer forms the second block of the framework, similar to SuperGlue. The input to this layer is the structural affinity between the two GNNs that have been encoded, defined in terms of affinity matrices \(\mathbf{S}_{p}\in\mathbb{R}^{P_{A}\times P_{B}}\) and \(\mathbf{S}_{l}\in\mathbb{R}^{Q_{A}\times Q_{B}}\). The \((i,j)\)-th entry of each matrix represents the affinity score between point feature \(i\) in image \(A\) and point feature \(j\) in image \(B\) and is defined as follows:

\[(\mathbf{S}_{x})_{i,j}=\text{exp}\left(\frac{(\mathbf{h}_{x}^{T})_{i}^{A} \mathbf{E}_{x}(\mathbf{h}_{x})_{j}^{B}}{\delta_{x}}\right),\ \ x\in\{p,l\}, \tag{3}\]

where \(\mathbf{E}_{x}\) is a learnable weight matrix and \(\delta_{x}\) is a tunable hyperparameter. The network is subject to the constraints described in Section III-B. As in SuperGlue, the unmatched and occluded P-point and L-point features are assigned to a dustbin, which augments each affinity matrix with an additional row and column that are both filled with a single learnable parameter.

We formulate the constrained optimization problem (4)-(5) below to solve for the assignment matrices \(\mathbf{P}_{p}\in\mathbb{R}^{P_{A}\times P_{B}}\) and \(\mathbf{P}_{l}\in\mathbb{R}^{Q_{A}\times Q_{B}}\):

\[\text{max}\sum_{i=1}^{N+1}\sum_{j=1}^{M+1}(\mathbf{S}_{x})_{i,j}(\mathbf{P}_{x })_{i,j},\ \ x\in\{p,l\} \tag{4}\]

\[\mathbf{P}_{x}\mathbf{1}_{N+1}=\mathbf{a}\ \ \text{and}\ \ \mathbf{P}_{x}^{T} \mathbf{1}_{M+1}=\mathbf{b}, \tag{5}\]

where \(M=P_{A}\), \(N=P_{B}\) for \(x=p\); \(M=Q_{A}\), \(N=Q_{B}\) for \(x=l\); and \(\mathbf{a}\) and \(\mathbf{b}\) are biases. As in SuperGlue, this constitutes a differentiable optimal transport problem and can be solved using the Sinkhorn algorithm [30], which is GPU-optimized. The algorithm is iterated until convergence.

Since all layers of the network are differentiable, we use the negative log-likelihood loss as the matching prediction loss. We backpropagate from ground truth matches to visual descriptors. Let \(GT_{p}\) be the set of ground truth matches of P-point features, \(\{(i,j)\}\subset\mathcal{P}_{A}\times\mathcal{P}_{B}\), and \(GT_{l}\) be the set of ground truth matches of L-point features, \(\{(i,j)\}\subset\mathcal{Q}_{A}\times\mathcal{Q}_{B}\). The sets \(\mathcal{A}_{p}\subseteq\mathcal{P}_{A}\) and \(\mathcal{B}_{p}\subseteq\mathcal{P}_{B}\) will denote the unmatched P-point features in both images, and similarly, \(\mathcal{A}_{l}\subseteq\mathcal{Q}_{A}\) and \(\mathcal{B}_{l}\subseteq\mathcal{Q}_{B}\) will denote the unmatched L-point
features. We define two losses, one for P-point features and one for L-point features:

\[Loss_{x} = -\sum_{(i,j)\in GT_{x}}\log(\mathbf{P}_{x})_{i,j}-\sum_{i\in\mathcal{ A}_{x}}\log(\mathbf{P}_{x})_{i,M+1}\] \[\qquad\quad-\sum_{j\in\mathcal{B}_{x}}\log(\mathbf{P}_{x})_{N+1,j },\qquad x\in\{p,l\},\]

where \(M=P_{A}\), \(N=P_{B}\) for \(x=p\) and \(M=Q_{A}\), \(N=Q_{B}\) for \(x=l\).

### _Camera pose estimation_

As a final step, we perform camera pose estimation by using the point and line matches from 2D-3D point and line correspondences between successive frames. To obtain 3D points and lines, we perform stereo matching using the disparity map generated from the stereo camera images. Since this is a well-established topic and open source modules are readily available, a detailed description of camera pose estimation is outside the scope of this paper.

## IV Experiments

In this section, we present the results of experiments that compare the performance of our StereoVO framework to that of state-of-the-art algorithms for point and line feature matching. We compared **Method 1**, which combines the SuperGlue point-matching algorithm and SOLD2 line-matching algorithm, to **Method 2**, which combines the SuperGlue point-matching algorithm and our novel line-matching algorithm. We tested both methods on the following datasets:

* Ford AV dataset [1], collected by autonomous vehicles, which consists of stereo camera images with accurate ground-truth trajectories obtained from LiDAR-based ICP. Our test data was drawn from Log 3 (Vegetation with clear sunny sky) and Log 4 (Residential area with clear sky).
* Nighttime stereo camera images from the Oxford car dataset [2], collected by an autonomous vehicle. Our test data consisted of images of residential areas with Visual Odometry (VO)-based ground truth.
* Synthetic stereo camera images with ground-truth trajectories from the urban environment in Town 10 of CARLA [29]. Our test data consisted of images of the same scenes under a variety of weather and lighting conditions, such as fog, nighttime, and glare.

Our StereoVO framework was run in real-time on an NVIDIA RTX 2080Ti GPU at around 7 FPS (142 ms). To ensure real-time operation, the framework requires a minimum of 6 GB of GPU memory (VRAM). Point and line matching results from Method 2 are shown in Fig. 1 and Fig. 4 for scenes from real and synthetic datasets, respectively, under various weather and lighting conditions.

### _Comparison of estimated trajectories and pose error_

We applied each method to estimate the vehicle camera poses from the Ford AV Log 4 dataset and generated vehicle trajectories from these pose estimates. The trajectories are plotted in Fig. 5, along with the ground truth (GT) trajectory. The figure shows that both methods yield trajectories that are close to the GT trajectory at all times. Note that at coordinates (2965, -660) and (3175, -825), indicated by arrows in the figure, the vehicle was at a complete stop, causing drift in the visual odometry. This drift can be reduced by fusing measurements from other sensors, such as GNSS and IMU, with the StereoVO estimates. To quantify the deviation of the trajectory generated using each method from the GT trajectory, we computed the Absolute Pose Error (APE) over time between each trajectory and the GT trajectory. Figure 6 compares the time series of the APE for both trajectories and shows that our method (Method 2) outperforms Method 1, in that it generally produced lower APE values over the sampled 160-s period.

Fig. 3: The proposed point and line feature matching architecture, consisting of point and line feature layers, an Attention Graph Neural Network layer for both points and lines, and an Optimal Matching layer that generates an assignment matrix using the Sinkhorn algorithm.
We also evaluated the performance of Method 2 on synthetic data from CARLA for the following scenarios: (1) daytime with clear sky; (2) daytime with fog; and (3) nighttime with no street lights. Figure 7a plots the trajectory generated over 200 frames for each scenario, along with the ground truth (GT) trajectory, and Figure 7b shows the box plot of the absolute pose error (APE) for each scenario. Both figures indicate that the discrepancy between the estimated and GT trajectories is higher for the fog and nighttime scenarios than for the daytime scenario, as expected, but that all three estimated trajectories are relatively close to the GT trajectory.

Table I reports the root-mean-square error (RMSE) be

Fig. 4: Every pair of rows starting from the top in this figure shows grayscale images of various synthetic scenarios from CARLA: (a) clear sky scenario; (b) nighttime scenario; (c) foggy scenario. The first row shows point matches between frames \(i\) and \(i+1\), and the second row shows line matches between frames \(i\) and \(i+1\). Point and line matches were generated by Method 2.

Fig. 5: Comparison of Methods 1 and 2 on the Ford AV Log 4 dataset: ground truth (GT) trajectory and trajectories that were generated using camera pose estimates from each method.

Fig. 6: Comparison of Methods 1 and 2 on the Ford AV Log 4 dataset: Absolute Pose Error (APE) over a sample period of 160 s. The \(x\)-axis displays the time recorded by the vehicle.

Fig. 7: Evaluation of Method 2 on the CARLA synthetic dataset: (a) Ground truth (GT) trajectory and trajectories that were generated using camera pose estimates from images of the same scenes in CARLA under daytime, fog, and nighttime conditions. (b) Box plot of Absolute Pose Error (APE) over a sample of 200 frames for the same scenarios.
tween the estimated vehicle position and its ground truth position, obtained from simulated noise-free GNSS data, over 2000 frames in each of the three simulated scenarios in CARLA. The estimated positions were computed using Method 2 with only point features (detected by SuperPoint), only line features (obtained by our line-matching algorithm), or both point and line features. For each scenario, the use of both point and line features yields a lower RMSE value than either point features or line features alone. Hence, the inclusion of line features in the StereoVO framework leads to improved performance, particularly in low-visibility and low-light conditions.

### _Comparison of number of feature detections and matches_

We also compared the number of point or line features that different algorithms detected and matched, along with the percentage of detected features that were matched, in 200 frames of the Ford AV, Oxford car, and CARLA datasets. Table II lists these quantities for point features that were detected by SuperPoint and matched by SuperGlue and line features that were detected by the SOLD2 line detector and matched by either the SOLD2 line-matching algorithm or ours. The table shows that our line-matching algorithm recovers more line matches than the SOLD2 line-matching algorithm in each tested dataset.

Figure 8 plots the number of point features detected by SuperPoint and the number of line matches obtained by our algorithm in each frame of the daytime, fog, and nighttime scenarios simulated in CARLA. The figure shows that the number of line matches in each frame is not significantly affected by the visibility conditions (clear or foggy) or light level (daytime or nighttime) in the scene. However, the numbers of point features detected in the fog and nighttime scenarios are consistently lower than the number detected in the daytime scenario, and are substantially lower in some frames. This is also reflected in the first row of Table II, which shows fewer total point detections in the CARLA fog and nighttime scenarios than in the daytime scenario.

These results indicate that our line-matching algorithm exhibits robust performance as a scene becomes more texture-poor due to adverse weather conditions and/or low illumination. In turn, this robustness in feature matching maintains the accuracy of the camera pose estimates under such conditions.

## V Conclusion

We have presented a real-time stereo visual odometry framework aided by Attention Graph Neural Networks. This framework incorporates self-supervised learning-based point and line features and uses a novel line-matching technique that samples line features into point features with encoded position constraints. Using real datasets from autonomous vehicles and synthetic datasets from the CARLA driving simulator, we demonstrated that our framework produces robust line-matching in feature-poor scenes and scenes containing repetitive structures, e.g., Manhattan-world scenarios. In these tests, our framework outperformed state-of-the-art point and line feature matching algorithms in terms of the error between estimated and ground-truth vehicle poses, the percentage of detected line features that were matched, and the variability in number of identified features with respect to visibility and lighting conditions. One direction for future work is to incorporate planar features into the framework to improve its robustness. Moreover, developing a single end-to-end framework that performs temporal and stereo matching for both point and line features would be a promising step toward increasing the method's accuracy, robustness, and computation efficiency.

Fig. 8: Number of point features detected by SuperPoint (top three plots) and number of line features obtained by our line-matching algorithm (bottom three plots) in each frame of three runs in CARLA under daytime, fog, and nighttime conditions.