Geometry-aware Feature Matching for Large-Scale Structure from Motion
Gonglin Chen Jinsen Wu Haiwei Chen Wenbin Teng Zhiyuan Gao Andrew Feng Rongjun Qin Yajie Zhao

Defect Landscape Engineering to Tune Skyrmion-Antiskyrmion Systems in FeGe
    Serena Eley
    September 9, 2024
==========================================================================



§ ABSTRACT 


Establishing consistent and dense correspondences across multiple images is crucial for Structure from Motion (SfM) systems. Significant view changes, such as air-to-ground with very sparse view overlap, pose an even greater challenge to the correspondence solvers. We present a novel optimization-based approach that significantly enhances existing feature matching methods by introducing geometry cues in addition to color cues. This helps fill gaps when there is less overlap in large-scale scenarios. Our method formulates geometric verification as an optimization problem, guiding feature matching within detector-free methods and using sparse correspondences from detector-based methods as anchor points. By enforcing geometric constraints via the Sampson Distance, our approach ensures that the denser correspondences from detector-free methods are geometrically consistent and more accurate. This hybrid strategy significantly improves correspondence density and accuracy, mitigates multi-view inconsistencies, and leads to notable advancements in camera pose accuracy and point cloud density. It outperforms state-of-the-art feature matching methods on benchmark datasets and enables feature matching in challenging extreme large-scale settings.



    


§ INTRODUCTION



Correspondence, as perhaps the most fundamental problem in computer vision, helps us with mapping the world. Solving correspondence is required in most applications in Augmented Reality and Robotics since it derives the camera poses and 3D point locations <cit.>.


In the 3D reconstruction from a set of uncalibrated images, the correspondence problem meets two additional demands: 1) the demand to be able to observe a corresponding point across multiple views, which is known as the track length of a correspondence point, and 2) the demand to find a dense set of such points such that they are sufficient for the recovery of 3D geometry of the scene. 

In this paper, we consider a very difficult correspondence problem—dense, consistent correspondences across a sequence of frames with large camera baselines. To our knowledge, no prior works have achieved satisfactory results in the presence of these challenges: both detector-based and detector-free methods struggle with significant view changes, as illustrated in Figure <ref>, due to their inherent limitations. Detector-based methods often match only a small number of keypoints, making accurate image registration difficult. 



While the 3D reconstruction problem described thereof has been well studied for decades, they are notably tackled by either requiring a pre-calibration of the captured cameras <cit.>, or the requirement that consecutive frames captured are closely located to each other <cit.>. In recent years, however, correspondence methods based on learned artificial neural networks (ANNs) have demonstrated significant improvements over the accuracy and robustness of the matching between a pair of in-the-wild, uncalibrated images <cit.>, which has given rise to the possibility of reconstructing 3D scenes from the abundant, in-the-wild photos captured in daily setting <cit.>. In recent years, correspondence methods using learned artificial neural networks (ANNs) have demonstrated significant improvements in the accuracy and robustness of the matching between a pair of in-the-wild, uncalibrated images. This makes 3D scene reconstruction from the abundant, in-the-wild photos captured in daily settings possible.

Recent state-of-the-art correspondence methods are separated into two major branches: the detector-based methods <cit.> that employ two independent steps to extract feature descriptor for key points first and perform matching later, and the detector-free methods <cit.> that find matches in an end-to-end trained neural networks in one shot. While the detector-based methods excel in performing matching among image sets with rich texture and small view point changes, their reliance on the keypoint detection and description poses significant challenges when images of interest lack textures, or their local features are not sufficient for matching.

On the other hand, the detector-free methods <cit.> naturally find much denser correspondences between a pair of images with a dense search, but they typically lack control of the locations of keypoints across views and are optimized for sub-pixel correspondences that depend on specific image pairs. As shown in Figure <ref>, this pair dependency makes them unsuitable for maintaining long-track correspondence across multiple images. Therefore, it leads to poor performance when these methods are directly applied to Structure-from-Motion (SfM) reconstruction. Though quantization on the found matches can mitigate the problem of short track length, it comes at the price of accuracy and requires additional refinement that makes the entire system more computationally expensive. 


We aim to bridge the gap between the detector-based and detector-free methods in order to obtain dense correspondences that can be tracked across multiple views with large baselines - addressing a very difficult correspondence problem in SfM reconstruction from in-the-wild image collections. Specifically, we propose a novel optimization-based method that refines the pair-wise dense correspondences by iteratively filtering and reassigning correspondence under a geometric constraint formulated by the Sampson Distance <cit.>. This method allows the early rejection of outliers during the matching steps, improving the reliability of correspondences in large-baseline scenarios. 
Interestingly, we have observed that incorporating both priors from the existing detector-based method and enforcing the geometric constraint helps our model achieve better camera prediction and longer track length among the found correspondences, as demonstrated in Section <ref>. 
The designed optimization module thus consistently improves upon any detector-based and detector-free backbones, by providing denser correspondences compared to the detector-based methods, while refining the detector-free results to be more accurate and consistent across multiple views. 3D point cloud reconstructed with our feature matcher thus excels in both density and accuracy compared to previous state-of-the-art approaches.

To validate the effectiveness of our proposed method, we have conducted comprehensive evaluations using several publicly available datasets, including Image Matching Competition Benchmark <cit.> and MegaDepth <cit.>. We compared the SfM reconstruction results using matches 

from our feature matcher against several state-of-the-art baseline methods. The 

results demonstrated better accuracy in estimated camera poses and longer track length compared to those from the baselines.

Additionally, we have extended our evaluation to a challenging large-baseline setting that requires mapping from the UAV to street view perspectives (Figure  <ref>).


§ RELATED WORK







 §.§ Detector-based Feature Matching
 A conventional detector-based matching framework typically detects keypoints in images, describes them using high-dimensional vectors, and further applies a matching algorithm to find the correspondences between these image sets <cit.>. Historically, numerous handcrafted methods like SIFT <cit.>, SURF <cit.>, and ORB <cit.> have been used for the keypoint detection.  Lately, learning-based methods <cit.>, particularly those leveraging convolution neural networks (CNNs), have significantly improved the performance over the handcrafted features and shown robustness under large viewpoint and illumination changes. For instance, SuperPoint <cit.> achieved a good performance by creating a vast dataset of pseudo-ground-truth interest points and using a self-supervised interest point detector. SuperGlue <cit.> learns feature matching using point features from a graph neural network (GNN) and attention layers that integrate global context information. 

Patch2Pix <cit.> has made further improvement with patch-level matches and geometry-guided refinement that has enhanced the correspondence accuracy. Patch2Pix <cit.> utilizes the Sampson distance <cit.> to refine initial patch-level matches to pixel-level accuracy, ensuring geometric consistency and precision. However, Patch2Pix <cit.> is limited to refining existing sparse matches and add new matches or extend beyond the initial patch-level matches.
Despite their success, detector-based methods are limited by their dependency on feature detectors, which struggle to consistently identify distinct interest points in areas that lack clear and distinguishable features.  



 §.§ Detector-free Feature Matching
 Detector-free approaches <cit.> bypass the feature detection stage and directly generate matches, following a concept that originates from SIFT Flow <cit.>.  The introduction of learning-based methods that employ contrastive loss to train pixel-wise feature descriptors <cit.> marked a significant development in this field. These methods - similar to detector-based approaches - employ the nearest neighbor search for post-processing to match the dense descriptors. 

NCNet <cit.> learns dense correspondences in an end-to-end manner, creating 4D cost volumes that account for all possible inter-image matches. 4D convolutions are applied to refine these volumes, promoting consistency across the correspondence field.  Building on this work, SparseNCNet <cit.> improved the efficiency with sparse convolutions. DRCNet <cit.> improved dense matching accuracy with a coarse-to-fine architecture.  While NCNet addresses all potential matches, its 4D convolution’s receptive field is limited to small local neighborhoods.  

LoFTR <cit.>, in contrast, aims for a broader match consensus, leveraging the expansive receptive field of transformers.  ASpanFormer <cit.>, MatchFormer <cit.>, and Efficient LoFTR <cit.> follows the transformer-based design to utilize multilevel cross-attention to mutually update image features, thereby incorporating dual-view contexts for achieving both global and local consensus.  While these methods represent significant improvements to the state-of-the-art, they still struggle to uphold high-level contexts such as epipolar geometries.  Therefore, we propose a geometry-aware approach that enforces epipolar consistency in dense feature matching.

 

 §.§ Structure from Motion
 Structure from Motion (SfM) methods have been extensively studied <cit.> in the past. Some of these efforts focus on the scaling to large-scene reconstruction <cit.>. SfM methods mainly use detector-based matching at the beginning of their pipelines, which limits the overall SfM performance in challenging cases. Some end-to-end SfM methods propose to directly model camera poses <cit.>. These methods avoid the bottleneck of feature detection but have limited scalability to real-world scenarios. Although Recent differentiable rendering methods <cit.> can optimize poses and the scene representation simultaneously, they rely on the traditional SfM pipelines like COLMAP<cit.> to provide the initial camera poses. Lindenberger et al.  <cit.> improves the accuracy of SfM and visual localization by refining keypoints, camera poses, and 3D points post to the traditional SfM pipelines. Unlike previous SfM methods, our proposed feature matcher ensures robustness and local geometry consistency at the first step of an SfM system.


§ METHODS



The core of our methods is an optimization module that operates on existing sparse matches M_s and two feature maps f_c^A and f_c^B from a pair of images. The module optimizes for the relative camera poses between the two images and refines the input coarse correspondences into refined coarse correspondence M_c_final that is geometric consistent under a Sampson distance <cit.> constraint. In Section <ref>, we provide an overview of how the sparse matches and feature maps are obtained, as well as the definition of Sampson distance. In Section <ref>, we discuss the details of our optimization module.



 §.§ Preliminaries





  §.§.§ Detector Based Method
 
Given two images I^A and I^B, our method accepts a detector-based backbone F_db that first identifies the sparse interest points with a keypoint detector. Then, local features are extracted around the identified interest points, encoding the local visual appearance in high-dimensional vectors. These features are then matched between I^A and I^B using either learned neural networks <cit.> or nearest-neighbor search to generate a set of sparse correspondences M_s. In our implementation, we use SuperPoint <cit.> + SuperGlue <cit.> to obtain sparse correspondence between a given image pair as it has been shown that the two methods combined can provide anchor points that introduce robust and good priors to guide dense matching <cit.>. We also evaluate the performance of different detector-based backbones in Section <ref>.



  §.§.§ Detector Free Method

Given two images I^A and I^B, our method accepts a detector-free backbone F_df that processes the entire image to output semi-dense correspondences without relying on explicit keypoint detection. Instead, dense feature extraction is performed using deep neural networks. The output features f_c^A and f_c^B are then matched using similarity metrics, which are generated to quantify the confidence and select the final correspondences. In our implementation, we use AspanFormer <cit.> as our detector-free backbone.



  §.§.§ Sampson Distance

Given a fundamental matrix F_m and the correspondence P^A_i, P^B_j, the Sampson Distance d(i,j) is the distance of P^A_i, P^B_j to their true epipolar line, which is given by:


    d(i,j) = ( 𝐏_j^B^⊤𝐅_𝐦𝐏_i^A )^2/(𝐅_𝐦𝐏_i^A)_1^2 + (𝐅_𝐦𝐏_i^A)_2^2 + (𝐅_𝐦^⊤𝐏_j^B)_1^2 + (𝐅_𝐦^⊤𝐏_j^B)_2^2,


where point 𝐏_i^A = (x_i, y_i, 1)^⊤ and point 𝐏_i^B = (x_j, y_j, 1)^⊤ are two points in I^A and I^B respectively and (𝐅_𝐦𝐏_i^A)_k^2 is the k-th entry of the vector (𝐅_𝐦𝐏_i^A). Therefore, the Sampson distance <cit.> gives the 
first-order approximation of the reprojection error. The error is minimized when two points are a perfect match.



 §.§ Geometry Aware Optimization Module



Our proposed optimization module aims to enforce geometric consistency among correspondences and guide the feature matching with geometric information. We utilizes M_s from f_db to initialize the fundamental matrix. With dense features f_c^A and f_c^B from f_df, our module first computes a confidence map P_c using the similarity metrics.
Then the Sampson Distance <cit.> is used as a geometry constraint of an image pair to iteratively select and reassign the correspondences by updating P_c. Such constraints can reduce mismatches and address missed matches due to insufficient appearance features and repeating texture patterns.

Initial Step
Given the features f_c^A and f_c^B, the initial confidence matrix is obtained with a dual-softmax operator. For the i-th feature in I^A and the j-th feature in I^B, the similarity score is calculated as

    S(i, j) = ⟨𝐟_c^A(i), 𝐟_c^B(j) ⟩ ,

where ⟨ a, b ⟩ 
denotes the dot product. Then the confidence level P_c is calculated as

    P_c(i, j) = softmax(S(i, ·))_j ·softmax(S(·, j))_i ,

Based on the initial confidence matrix, 


    M_c = {(ĩ,j̃)|∀(ĩ,j̃)∈ MNN(P_c), P_c(ĩ,j̃) ≥θ_iter} .


we follow LoFTR <cit.> 
to apply mutual nearest neighbor (MNN) criteria using equation <ref> to obtain the initial matches. The row index i represents the index of the patch from I^A and the column index j represents the index from I^B. (ĩ,j̃) is a match if and only if the confidence level is the maximum of both ith row and jth column. We choose points where their confidences are larger than θ_iter as matches to enter iterative steps.

The initial fundamental matrix F_m is calculated using the normalized 8-points method if there exist enough good anchor points in M_s, which is at least 10 points with confidence larger than 0.5. Otherwise, the initial fundamental matrix will be calculated using the top half matches from equation  <ref>.

Iterative Steps
Our iterative steps begin with initialized coarse matches M_s and fundamental matrix F_m and update them in each iteration. The Sampson distance error is calculated for all initial matches according to Equation <ref>.
Geometric confidence is then estimated from the Sampson distances as 



    P_d(i, j) = sigmoid(relu(τ-d(i, j)),

where τ is a distance threshold, and smaller values of τ will lead to more strict constraints. The geometric confidence for correspondences that have distances larger than the distance threshold will be 0.5. In other words, the confidence of matches that violate the geometry constraint will be penalized.

Finally, the confidence map is updated by the geometric confidence map by a given weight w. Therefore, the final confidence is

    P_c(i, j)_i =  P_c(i, j)_i-1*P_d(i, j)_i-1· w.

At every iteration, the updated confidence map is normalized using min-max normalization to avoid extreme values. When the iteration ends, the module will select the final coarse-level matches 
M_c_final based on the final confidence map from the last iteration with mutual nearest neighbor criteria according to equation <ref>, but with a different threshold θ_c to avoid selecting a large amount of correspondences with very small confidences.  

The refined matches M_c_final are further fed into a correlation-based refinement block, which is the same with LoFTR <cit.>, to obtain the final matching results.





 §.§ Structure from Motion Reconstruction

The final matching results are fed into COLMAP <cit.> for estimating structure from motion. By integrating our robust and dense matches into COLMAP <cit.>, our method is shown to enhance the accuracy and completeness of the camera poses and reconstructed point cloud, resulting in superior 3D models, even in challenging scenarios with significant viewpoint changes, as discussed in Section <ref>. 



 §.§ Implementation Details

The Sampson Distance threshold τ is set to 10 which allows a few pixels of error. The iteration of the optimization module is set to 10 and the weight w is 1.2. The confidence threshold θ_iter is set to a small value of 0.01. The idea is to conduct a comprehensive search across the space. The effect of different hyperparameters will be discussed in Section <ref>.


§ EXPERIMENTS


In this section, we demonstrate the performance of our method in the SfM reconstruction task in terms of the accuracy of camera pose estimation and its application in challenging real-world scenarios. We also conduct a comprehensive ablation study. All experiments were run on an RTX 4090 GPU with 24GB VRAM and a 13th Gen Intel® Core™ i9-13900KF CPU with 32 cores and 126GB RAM.



  §.§.§ Dataset
 

We utilize the Image Matching Challenge (IMC) 2021 <cit.> and ScanNet <cit.> benchmark datasets to evaluate our method's performance in SfM reconstruction for outdoor and indoor settings respectively. As there are no publicly available air-to-ground datasets, we collected two additional air-to-ground datasets to demonstrate our method's ability to find reliable correspondences in this challenging scenario.

The IMC Phototourism benchmark <cit.> encompasses multiple large-scale outdoor scenes. The test set includes 9 sites, and each set has 100 images. The ground truth or pseudo ground truth camera poses are obtained from COLMAP <cit.> on the entire collection of images and then sampled into test sets. ScanNet <cit.> contains 1613 monocular indoor sequences with ground truth poses and depth maps. We use the data split of the test set following SuperGlue <cit.> and LoFTR <cit.>, which includes 15 sequences. The primary challenge lies in accurately recovering the camera poses from in-the-wild images with significant viewpoint and illumination changes. For all test scenes, the images are considered unordered.

The first collected air-to-ground dataset consists of 16 UAV images and 21 ground images, while the second dataset contains 16 UAV images and 24 ground images. The primary challenge for these datasets is to accurately register UAV images with ground view images. For all scenes, the images are considered unordered and are matched exhaustively. When aerial images are matched with ground images, to avoid failure in finding correspondences due to resizing, the aerial images are cropped into quarters plus a center region to generate five pairs. The matches are subsequently merged to obtain the final matches. This strategy is used for all methods.



  §.§.§ Metrics and Comparing Methods


We report the metric area-under-curve (AUC) to evaluate camera pose accuracy for IMC 2021 <cit.> and ScanNet <cit.>. Specifically, for every reconstructed model, we aligned the predicted camera poses with the ground truth camera poses. We then computed the absolute angular translation and rotation errors. The AUC selects the maximum between these two errors and outputs the area under the accuracy threshold curve. As the collected air-to-ground datasets lack ground-truth camera poses, we follow <cit.> to report the number of registered images, the number of 3D points, the average track length, and the average projection error (px) in this setting to evaluate the methods by the completeness of the reconstructed model.





We compared our method against four different methods:
SIFT <cit.>+NN, which is the default feature matching method in COLMAP <cit.>; SuperPoint <cit.>+Superglue <cit.>, a robust method widely used in SfM; ALIKED <cit.>+LightGlue <cit.>, a state-of-the-art detector-based feature matching method; DKM <cit.> and RoMa <cit.>, two state-of-the-art dense feature matching methods. We omit RoMa <cit.> for air-to-ground datasets since DKM <cit.> appears to be superior to RoMa <cit.> in terms of 3D reconstruction for outdoor scenes.

In order to reduce the computational time, the number of matches per pair is limited to 4K for all methods. Bold text indicates the best results, and underlined text indicates the second best.



 §.§ Comparison to State of the Art Methods











  §.§.§ Results


SfM results using our feature matching method on both the IMC 2021 <cit.> and the ScanNet <cit.> datasets have shown significant improvements in performance. As shown in Table <ref> and Table <ref>, our feature matcher contributes to the highest average pose estimation AUC in both datasets. Specifically, Table <ref> indicates an increase of +3.1 in AUC@3 for estimated camera poses compared to ALIKED <cit.> + LG <cit.>, as can be visually shown in Figure <ref>. Similar gains are observed for ScanNet <cit.>, as shown in Table <ref>. These findings underscore the efficacy of our method in achieving superior SfM results. 

As shown in Table <ref> and Figure <ref>, compared to other methods, our method is able to register all images and is the only method that can align the ground images with UAV images for set 2. We have noticed that reconstructed camera poses using DKM <cit.> in set 1 are apparently wrong with ground images pointing down and speculate that it is due to the lack of multi-image matches for feature tracking. In addition, DKM <cit.> also fails in reconstructing a model for UAV images in set 2. These results further validate the practical utility and effectiveness of our feature matcher in real-world scenarios.



 §.§ Ablation Study



We have conducted an ablation study to investigate the impact of different parameters and design choices on both feature matching and 3D reconstruction. 

Ablation study on Homography Estimation. Feature matching is evaluated on the widely adopted HPatches dataset <cit.>. HPatches <cit.> contains a total of 108 sequences with significant illumination changes and large viewpoint changes. We follow the evaluation protocol of LoFTR <cit.> to resize the shorter size of the image to 480 and report AUCs at 3 different thresholds. Table <ref> shows results on different parameters and design choices on HPathes <cit.>. The “Baseline” denotes only using ASpanFormer <cit.>. The results show the effects of different parameters in feature matching. Additionally, the full method outperforms other design variations, confirming the robustness and effectiveness of our approach in enhancing feature matching accuracy. More experiments related to feature matching can be found in supplementary material.
 









Ablation study on 3D reconstruction. In order to comprehensively assess the efficacy of our proposed components within the SfM system, we also measure the impact of the aforementioned design changes on 3D reconstruction. To this end, we randomly sample 20 images from each scene of IMC 2021 phototourism datasets <cit.> and compare the SfM reconstruction results based on pose estimation. We follow IMC <cit.> to report relative pose errors for all possible pairs since we are using subsets for this experiment. All results are averaged across all scenes and all methods are evaluated on the same subset of data. 

“SG” denotes anchor points from SuperGlue <cit.>; if the optimization (“Geo”) module is not enabled, anchor points are naively concatenating to final matches. If there are no anchor points, the optimization (“Geo") module is initialized with the top half most confident matches from only the detector-free backbone. As shown in Tables <ref> and <ref>, the results again demonstrate the effectiveness of our proposed method as it brings improvement over the baseline methods in the accuracy of SfM reconstruction.

Ablation study on detector-based backbones. Following the setting of Section <ref>, Table <ref> compares the performance of using two detector-based methods on the IMC dataset <cit.>. This demonstrates that SP<cit.> + SG<cit.> is more effective in providing priors in guiding dense matching.



§ LIMITATION AND FUTURE WORK




The main limitation of our method is efficiency, due to the computational cost of the transformer-based architecture and the significant number of correspondences produced, as shown in Table <ref>. Utilizing a newly developed backbone model with a focus on efficiency, such as Efficient LoFTR <cit.>, can improve it with slightly degraded performance.

Due to our method's reliance on the backbone model to provide initial matches for fundamental matrix estimation and its heuristic design, our method could fail with poor initial matches from backbone models. Nevertheless, our evaluations against baselines demonstrate a marked improvement in accuracy, as well as the capability to find reliable matches in challenging scenarios. Furthermore, we believe the accuracy of SfM reconstruction can be further improved by adopting multi-view refinement techniques, such as those proposed by pixSfM <cit.>.



§ CONCLUSION

We propose the geometry-aware optimization module, an innovative optimization approach that bridges detector-based matcher and detector-free matcher to improve SfM reconstruction and accuracy of estimated camera poses. This module leverages epipolar constraints, iteratively optimizes for the best relative pose, and applies Sampson distance weighting to refine match precision. The introduction of geometry optimization enhances the system's ability to handle challenging scenarios, including drastic changes in perspective, scale, and illumination. Our method is shown to be also effective in 3D reconstruction for large baselines, a scenario where conventional methods often fail and perform unsatisfied. 


§ ACKNOWLEDGEMENT

Supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/ Interior Business Center (DOI/IBC) contract number 140D0423C0075. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily
representing the official policies or endorsements, either expressed or implied, of IARPA, DOI/IBC, or the U.S. Government.

    ieeenat_fullname
    



