# SCENES: Subpixel Correspondence Estimation With Epipolar Supervision

Dominik A. Kloepfer
Visual Geometry Group
University of Oxford
<EMAIL_ADDRESS>João F. Henriques
Visual Geometry Group
University of Oxford
<EMAIL_ADDRESS>Dylan Campbell
School of Computing
Australian National University
<EMAIL_ADDRESS>
###### Abstract

Extracting point correspondences from two or more views of a scene is a
fundamental computer vision problem with particular importance for relative
camera pose estimation and structure-from-motion. Existing local feature
matching approaches, trained with correspondence supervision on large-scale
datasets, obtain highly-accurate matches on the test sets. However, they do
not generalise well to new datasets with different characteristics to those
they were trained on, unlike classic feature extractors. Instead, they require
finetuning, which assumes that ground-truth correspondences or ground-truth
camera poses and 3D structure are available. We relax this assumption by
removing the requirement of 3D structure, e.g., depth maps or point clouds,
and only require camera pose information, which can be obtained from odometry.
We do so by replacing correspondence losses with epipolar losses, which
encourage putative matches to lie on the associated epipolar line. While
weaker than correspondence supervision, we observe that this cue is sufficient
for finetuning existing models on new data. We then further relax the
assumption of known camera poses by using pose estimates in a novel
bootstrapping approach. We evaluate on highly challenging datasets, including
an indoor drone dataset and an outdoor smartphone camera dataset, and obtain
state-of-the-art results without strong supervision.

##  1 Introduction

(a) Source pixel

(b) Putative match, epipolar line

(c) Matches before SCENES finetuning

(d) Matches after SCENES finetuning

Figure 1: SCENES (Subpixel Correspondence EstimatioN with Epipolar
Supervision) learns to find high-quality local image matches without requiring
correspondence supervision. Instead, pose supervision alone is used,
encouraging matches to lie on epipolar lines. The red pixel in (a) corresponds
to the red epipolar line in (b). The network initially matches the red pixel
to the blue pixel, but our epipolar losses preference matches on the epipolar
line (not necessarily the closest point). The correspondences found by the
state-of-the-art MatchFormer algorithm [45] are displayed before (c) and after
(d) finetuning with epipolar supervision. The colour denotes whether the
squared symmetrical epipolar distance is below (green) or above (red) a
threshold of $5\cdot 10^{-4}$. Images are from the challenging EuRoC-MAV drone
dataset [6].

The task of finding corresponding points in two or more views of a scene—that
is, the subpixel positions that correspond to the same visible 3D surface
location—is a fundamental low-level image processing problem. It underpins
many computer vision tasks, including relative pose or fundamental matrix
estimation [18, 49, 24], visual localisation [39, 31], Structure-from-Motion
(SfM) [36, 33], and Simultaneous Localisation and Mapping (SLAM) [27, 14], and
is related to optical flow estimation [20, 2, 40, 21] when the camera motion
is small.

Classically approached in two stages with keypoint detection followed by local
feature extraction and matching [26, 30, 12], recent approaches [38, 45, 9]
eschew the detection step to ensure that dense matches can be extracted even
in texture-poor regions. However, the supervision requirements of most
approaches are onerous: ground-truth correspondences are assumed to be
available. While these can be obtained from camera poses and 3D structure
(depth maps, point clouds or meshes), these are often not readily accessible.
This makes it challenging to train or finetune these models on new datasets or
in new regimes. For example, a robot preloaded with the latest local feature
matching network would not be able to adapt the model when deployed in a new
environment, unless it were able to reconstruct the environment. Obtaining a
good quality reconstruction would itself require a good matching network,
unless auxiliary sources of information, like a 3D sensor, were available. To
resolve this chicken-and-egg problem, we instead propose an approach that only
requires approximate known camera poses, which can be acquired from odometry
or GPS combined with an accelerometer. This allows, in particular, for model
adaptation and finetuning in domains where ground-truth 3D structure is
difficult to procure.

To address this problem, we propose to fine-tune existing models using
epipolar losses, drop-in replacements for the correspondence losses in state-
of-the-art transformer-based matching networks. These encourage putative
matches to lie on the associated epipolar lines, which are computed using the
relative camera poses. While significantly weaker than correspondence
supervision, we observe that this cue is sufficient for finetuning existing
models on new data. Finally, we propose a bootstrapping approach that removes
the need for camera pose supervision as well, allowing us to use an existing
pre-trained model to improve itself. This expands the types of settings where
these models can be deployed and reduces the annotation and pre-processing
burdens. To summarise, our contributions are:

  1. 1.

Epipolar losses—drop-in replacements for existing matching losses that remove
the need for strong correspondence supervision; and

  2. 2.

Strategies for adapting pre-trained models to new domains with only pose
supervision, or no supervision.

We evaluate on two challenging datasets with significantly different
characteristics from the pre-training data: an indoor drone dataset and an
outdoor phone camera dataset. Compared to the original model, our weakly
supervised and unsupervised finetuning strategies allow us to find keypoint
matches that achieve significantly improved pose estimation results.

##  2 Related work

#### Epipolar geometry in deep learning.

Several studies have leveraged geometric priors based on epipolar geometry to
enhance the performance of deep learning methods across various tasks. For
instance, He et al. [19] conduct direct feature aggregation along epipolar
lines to improve 3D human pose estimation, while Bhalgat et al. [4] guide a
transformer model to prioritise attending to features along the epipolar line,
leading to enhanced 3D object pose estimation. Yifan et al. [48] introduce
geometrical embeddings based on epipolar geometry in place of positional
embeddings for 3D reconstruction tasks using transformer models. Furthermore,
Prasad et al. [28], Li et al. [23], and Yang et al. [47] all employ epipolar
geometry for depth prediction tasks.

We also make use of epipolar geometry in our proposed pixel correspondence
estimator. However, we do not explicitly constrain the model to predict
matches on the epipolar lines, but rather encourage this behaviour through our
epipolar loss functions. In this respect, our work is most similar to CAPS
[44], which also learns keypoint feature descriptors. That approach however is
designed for training from scratch, so cannot benefit from pre-training and
always requires camera pose supervision, unlike our proposed unsupervised
strategy.

#### Keypoint detection and feature matching.

The problem of creating correspondences between points in image pairs has been
a longstanding challenge in computer vision. Early efforts involved hand-
crafted criteria for keypoint detection, description, and matching in well-
established methods such as the Harris corner detector [17], SIFT [26], SURF
[3], and ORB [30]. More recently, deep neural network methods have
outperformed classical techniques for both keypoint detection and matching.
Approaches like SuperPoint [11], D2Net [13], R2D2 [29], and DISK [43] excel at
keypoint detection, while works such as SuperGlue [32], SeededGNN [8],
ClusterGNN [37], and the very recent LightGlue[25] achieve impressive feature
matching results. An alternative class of approaches eschew the detection step
entirely. These detector-free methods, including LoFTR [38], MatchFormer [45]
and ASpanFormer [9], have demonstrated remarkable performance by creating
matches on a dense grid, eliminating the need for separately selected
keypoints, and achieve state-of-the-art accuracy. Due to their success, we
focus on this detector-free paradigm as well, although our approach is not
limited to these models.

#### Unsupervised domain adaptation.

Finetuning a large model pre-trained on large amounts of data (a
computationally expensive process) in order to improve performance or decrease
training time is a well-established practice in the field of deep learning,
and a comprehensive review would exceed the scope of this paper. Of particular
relevance to the present work is the sub-field of unsupervised domain
adaptation (see, e.g., Wilson and Cook [46] and Farahani et al. [15] for
surveys). In unsupervised domain adaptation, a base model trained on labeled
data is fine-tuned to improve performance on a different dataset with limited
or no labels. However, the majority of papers in this area focus on methods to
make the base model more adaptable to domain shifts, with comparatively less
emphasis on the fine-tuning process itself. We instead consider how to fine-
tune a model that was trained with strong supervision in domains where such
ground-truth information is unavailable.

##  3 Epipolar supervision for matching

In this section, we present Subpixel Correspondence EstimatioN with Epipolar
Supervision (SCENES), our method for estimating image point matches without
correspondence supervision. We first define the problem, then review the
relevant background material on epipolar geometry and detector-free feature
matching methods. Next, we present our epipolar loss functions, drop-in
replacements for existing matching losses. Finally, we describe two training
settings for adapting pre-trained models to new domains, one with pose
supervision and the other without supervision.

The problem under consideration is that of finding the corresponding points in
two or more views of a scene: the (subpixel) image positions that correspond
to the same visible 3D point. Without loss of generality, we consider the two
image case. Formally, given two images ${\mathtt{I}}_{1}$ and
${\mathtt{I}}_{2}$ with dimensions $H_{1}\times W_{1}$ and $H_{2}\times W_{2}$
and camera matrices ${\mathtt{P}}_{1}$ and ${\mathtt{P}}_{2}$, we aim to find
pairs of (homogeneous) image coordinates
$({\mathbf{x}}_{1}\in{\mathtt{I}}_{1},{\mathbf{x}}_{2}\in{\mathtt{I}}_{2})$
with high recall and precision such that
${\mathbf{x}}_{1}={\mathtt{P}}_{1}{\mathbf{X}}$ and
${\mathbf{x}}_{2}={\mathtt{P}}_{2}{\mathbf{X}}$, where $X\in\mathbb{R}^{3}$ is
a visible surface point on the scene geometry.

###  3.1 Background

#### Epipolar geometry.

We begin with a brief review of epipolar geometry; for a more detailed
treatment, refer to Hartley and Zisserman [18]. The epipolar geometry of two
views is the geometry of the intersection of the two image planes with the
pencil of planes through the baseline joining the camera centres. For a 3D
point ${\mathbf{X}}$ imaged in two views at 2D points ${\mathbf{x}}_{1}$ and
${\mathbf{x}}_{2}$, the epipolar geometry defines the relation between these
image points via the fundamental matrix ${\mathtt{F}}$. Expressing
${\mathbf{x}}_{1}$ in homogeneous coordinates and interpreting it as a ray
extending from the camera centre through ${\mathbf{x}}_{1}$ in the image
plane, then the projection of this line into a second image is the epipolar
line corresponding to ${\mathbf{x}}_{1}$. The corresponding point
${\mathbf{x}}_{2}$ must lie on this line, a relation that is captured by the
equation

${\mathbf{x}}_{2}^{\mathsf{T}}{\mathtt{F}}{\mathbf{x}}_{1}=0$ (1)

where the epipolar line corresponding to the point ${\mathbf{x}}_{1}$ in image
2 is given by ${\mathbf{l}}_{12}={\mathtt{F}}{\mathbf{x}}_{1}$. This
constitutes a significant constraint on where the corresponding point might be
located, reducing the search space from 2D to 1D.

The fundamental matrix relating two images can be computed directly from the
images, either from (at least seven) point correspondences or from their
relative pose and camera intrinsics, without requiring any information about
the 3D geometry of the scene. That is, given two camera matrices
${\mathtt{P}}_{1}={\mathtt{K}}_{1}[{\mathtt{I}}\mid\textbf{0}]$ and
${\mathtt{P}}_{2}={\mathtt{K}}_{2}[{\mathtt{R}}\mid{\mathbf{t}}]$ with a
rotation ${\mathtt{R}}$, translation ${\mathbf{t}}$ (up to scale) and camera
intrinsic matrices ${\mathtt{K}}_{1}$ and ${\mathtt{K}}_{2}$, the fundamental
matrix ${\mathtt{F}}\in\mathbb{R}^{3\times 3}$ is given by

${\mathtt{F}}={\mathtt{K}}_{2}^{-\mathsf{T}}[{\mathbf{t}}]_{\times}{\mathtt{R}}{\mathtt{K}}_{1}^{-1},$
(2)

with $[{\mathbf{t}}]_{\times}=\left[\begin{smallmatrix}0&-t_{3}&t_{2}\\\
t_{3}&0&-t_{1}\\\ -t_{2}&t_{1}&0\end{smallmatrix}\right]$ is the matrix
representation of the cross product. Since a good estimate of the relative
camera pose is often available from odometry and the intrinsic parameters are
often recorded in the image metadata, then the fundamental matrix and hence
the epipolar lines can be easily obtained.

#### Local feature matching.

State-of-the-art detector-free image matching algorithms, starting with LoFTR
[38] and including MatchFormer [45] and ASpanFormer [9], aim to find as many
accurate matches as possible for $m$ pixels in the first image, arranged in a
grid at locations ${\mathbf{x}}_{1}^{i}\in{\mathtt{I}}_{1}$ with
$i=1,\dots,m$. They involve two stages: coarse and fine.

In the first stage, these methods estimate a set of high-level matches by
extracting coarse features, generally for $8\times 8$ pixel patches. These
coarse features are then used to construct a confidence matrix
${\mathtt{C}}\in\mathbb{R}^{m\times m}$ that for each sampling location
${\mathbf{x}}_{1}^{i}$ in the first image estimates the probability that the
respective pixel patch at location ${\mathbf{x}}_{2}^{j}\in{\mathtt{I}}_{2}$
in the second image is a correct match. This coarse-level confidence matrix is
supervised using a classification-type loss $\mathcal{L}_{c}$, such as cross-
entropy or negative log-likelihood, with respect to a ground-truth binary mask
${\mathtt{M}}\in\mathbb{R}^{m\times m}$ that is $1$ for ground-truth
correspondences (calculated using ground-truth camera poses and depths) and
$0$ otherwise. Letting ${\mathbf{x}}_{g}^{i}\in{\mathtt{I}}_{2}$ be the
location of the ground-truth match for location ${\mathbf{x}}_{1}^{i}$, we
have ${\mathtt{M}}_{ij}=1\textrm{ iff
}{\mathbf{x}}_{2}^{j}={\mathbf{x}}_{g}^{i}.$ The coarse level loss is then a
function of this ground-truth matrix and the confidence matrix:
$\mathcal{L}_{c}=f({\mathtt{C}},{\mathtt{M}}).$

In the second stage, an initial set of $m^{\prime}\leq m$ coarse matches
computed from the coarse-level confidence matrix are refined at a finer level.
From a set of finer features (generally $2\times 2$ or $4\times 4$ pixel
patches), the second stage crops windows surrounding each of the coarse level
matches, and returns a refined set of matched image coordinates
$\hat{{\mathbf{x}}}_{2}^{k}$ with $k=1,\dots,m^{\prime}$. Whereas the coarse
stage is only designed to compute matches on a discrete grid, the fine stage
returns floating-point coordinates. The supervision of this stage therefore
also differs from that of the coarse stage: instead of a classification-type
loss, a regression loss $\mathcal{L}_{f}$ is used, which is a function of the
distance $d$ between the location of the ground-truth match and the refined
matching coordinates:
$\mathcal{L}_{f}=\frac{1}{m^{\prime}}\sum_{k}^{m^{\prime}}g(d(\hat{{\mathbf{x}}}_{2}^{k},{\mathbf{x}}_{g}^{k})).$

###  3.2 Epipolar losses

The aforementioned classification and regression losses used to train state-
of-the-art local feature matching models require the ground-truth
correspondences. However, unless the dataset has ground-truth camera poses and
3D geometry, these can be difficult to obtain. To circumvent this limitation,
we reformulate both loss types such that they only require the fundamental
matrix ${\mathtt{F}}$ for a given image pair ${\mathtt{I}}_{1}$ and
${\mathtt{I}}_{2}$, by replacing the ground-truth correspondence matrix
${\mathtt{M}}$ and the distance to the ground-truth match $d$ with epipolar
variants ${\mathtt{M}}^{\textrm{epi}}$ and $d^{\textrm{epi}}$, as visualised
in Fig. 2. The fundamental matrix ${\mathtt{F}}$ of an image pair can be
computed from the relative camera pose, or estimated using a keypoint matches
between the two images.

(a) Classification: ${\mathtt{C}}$

(b) Classification: ${\mathtt{M}}^{\textrm{epi}}$

$d^{\textrm{epi}}$$\hat{{\mathbf{x}}}_{2}$${\mathbf{l}}_{12}={\mathtt{F}}{\mathbf{x}}_{1}$
(c) Regression: $d^{\textrm{epi}}$

Figure 2:  Visualisation of the epipolar classification (coarse) and
regression (fine) losses. The epipolar classification loss assigns the
‘ground-truth’ match location to the highest probability point on the epipolar
line and compares the resulting binary mask ${\mathtt{M}}^{\textrm{epi}}$ to
the predicted confidence map ${\mathtt{C}}$ via a cross-entropy loss or
similar. The epipolar regression loss computes the perpendicular distance
between the predicted match $\hat{{\mathbf{x}}}_{2}$ and the epipolar line
${\mathbf{l}}_{12}$. Note the thin epipolar line is plotted in (a) and (b) for
visualisation purposes only, it is not part of the confidence map
${\mathtt{C}}$ or binary mask ${\mathtt{M}}$.

#### Epipolar classification loss.

For the coarse classification loss, a simple strategy would be to replace the
binary mask ${\mathtt{M}}$ of the ground-truth point correspondences with a
binary mask of the epipolar lines computed using the fundamental matrix. With
$e_{1}^{i}\subset{\mathtt{I}}_{2}$ the set of coordinates in the second image
that form the epipolar line for point
${\mathbf{x}}_{1}^{i}\in{\mathtt{I}}_{1}$, this would require
${\mathtt{M}}^{\textrm{epi}}_{ij}=1\textrm{ iff }{\mathbf{x}}_{2}^{j}\in
e_{1}^{i}$. However, fine-tuning the model in this way would train it to
predict equally high matching scores for every pixel along the epipolar line.
This means that the probability mass for the coarse matches is diffused along
the entire epipolar line, reducing the reliability of the coarse matches.
Instead, we assign the ground-truth match location at the position on the
epipolar line that has the highest probability, taking inspiration from the
max-epipolar loss of Bhalgat et al. [4]. That is, we define

$\displaystyle{\mathtt{M}}^{\textrm{epi}}_{ij}=\begin{cases}1&\textrm{if
}j=\operatorname*{arg\,max}_{k:\,{\mathbf{x}}_{2}^{k}\in
e_{1}^{i}}({\mathtt{C}}_{ik})\\\ 0&\textrm{otherwise.}\end{cases}$ (3)

#### Epipolar regression loss.

For the fine-level regression loss, we replace the distance between the
estimated and the ground-truth match with the perpendicular distance between
the estimated match $\hat{{\mathbf{x}}}_{2}^{i}$ and the epipolar line
${\mathbf{l}}_{12}^{i}$, as was also done in Wang et al. [44]. In homogeneous
coordinates, with the third coordinate normalised to 1,

$d^{\textrm{epi}}({\mathbf{x}}_{1}^{i},\hat{{\mathbf{x}}}_{2}^{i})=\frac{|{\mathbf{x}}_{1}^{i\mathsf{T}}{\mathtt{F}}^{\mathsf{T}}\hat{{\mathbf{x}}}_{2}^{i}|}{\sqrt{([{\mathtt{F}}^{\mathsf{T}}\hat{{\mathbf{x}}}_{2}^{i}]_{1})^{2}+([{\mathtt{F}}^{\mathsf{T}}\hat{{\mathbf{x}}}_{2}^{i}]_{2})^{2}}}.$
(4)

Here $[{\mathtt{F}}^{\mathsf{T}}\hat{{\mathbf{x}}}_{2}^{i}]_{k}$ refers to the
$k$-th component of the three-dimensional vector
$[{\mathtt{F}}^{\mathsf{T}}\hat{{\mathbf{x}}}_{2}^{i}]$.

#### Total epipolar loss.

The total epipolar loss is then

$\mathcal{L}^{\textrm{epi}}=(1-\lambda)f({\mathtt{C}},{\mathtt{M}}^{\textrm{epi}})+\frac{\lambda}{m^{\prime}}\sum_{k}^{m^{\prime}}g(d^{\textrm{epi}}({\mathbf{x}}_{1}^{k},{\mathbf{x}}_{2}^{k}))$
(5)

where typically the function $f$ is a binary cross entropy function, $g$ is a
simple linear scaling, and $\lambda$ is a hyper-parameter that controls the
relative importance of the classification and regression losses. We note that
it is generally straightforward to adapt most loss functions that use ground-
truth image correspondences to supervise keypoint matches.

###  3.3 Finetuning with weak supervision

Reformulating the losses in terms of the epipolar lines and distances lessens
the supervision requirements. This facilitates the adaptation of pre-trained
models to new domains, where ground-truth correspondences may be difficult to
obtain. Instead, given a fundamental matrix relating two images, which can be
obtained from the relative pose between the cameras (a rotation ${\mathtt{R}}$
and translation ${\mathbf{t}}$ up to scale) and the camera intrinsic matrices
${\mathtt{K}}_{1}$ and ${\mathtt{K}}_{2}$ as shown in eq. 2, both
${\mathtt{M}}^{\textrm{epi}}$ and $d^{\textrm{epi}}$ can be computed. A pre-
trained model can then be finetuned on a new dataset by optimising
$\mathcal{L}^{\textrm{epi}}$.

###  3.4 Finetuning without supervision

In scenarios where accurate ground-truth camera poses are unavailable, the
model can still be refined via the epipolar loss using a bootstrapping
strategy. While the ground-truth fundamental matrix ${\mathtt{F}}$ cannot be
computed without ground-truth camera poses, it can be estimated from (at least
seven) correspondences, such as those found by a pre-trained matching model.
Our strategy, then, is to apply a pre-trained model to a dataset from a new
domain to obtain a set of (poor quality) image correspondences, then use a
robust estimator such as RANSAC to estimate the fundamental matrices, which
are used as supervision for our epipolar loss. The key intuition is that even
if the pre-trained model elicits many incorrect matches for an image pair, the
robust estimator should still be able to estimate the (approximately) correct
fundamental matrix in many cases, which will provide a supervision signal for
the incorrect matches. As we show in Sec. 4, while this bootstrapping method
is naturally less performant than finetuning using ground-truth fundamental
matrices, it can still lead to significant performance improvements on
challenging datasets.

###  3.5 Discussion

While we do not know the ground-truth locations of the keypoint matches in the
absence of ground-truth depth and/or ground-truth pose information, we do know
that they must lie on the respective epipolar lines.

For the coarse stage, we want to refine the network’s predicted confidence
matrix to align with our best estimate of the location of the match. Since we
are starting from a pre-trained network that does not make random predictions,
this best estimate is the location with the highest probability for being a
match that is also on the epipolar line, i.e., the location on the epipolar
line for which the matching confidence ${\mathtt{C}}_{ij}$ is maximised. This
provides a useful training signal so long as the highest probability location
on the epipolar line is near the correct match location sufficiently
frequently.

For the fine stage, replacing the distance to the ground-truth match location
with the distance to the epipolar line in the regression loss also provides an
imperfect-yet-useful training signal. Even though the gradient vector when
taking the derivative of $d^{\textrm{epi}}$ w.r.t. ${\mathbf{x}}_{2}^{k}$ may
not point exactly in the same direction as the gradient vector from taking the
derivative of $d$ w.r.t. ${\mathbf{x}}_{2}^{k}$, the angle between them is
less than ninety degrees (they have a positive inner product
$\langle\frac{\partial d}{\partial{\mathbf{x}}_{2}^{k}},\frac{\partial
d^{\textrm{epi}}}{\partial{\mathbf{x}}_{2}^{k}}\rangle>0$). Following the
gradient vector of the epipolar loss therefore also tends to decrease the
distance of the match to the ground-truth match location.

The bootstrapping approach, using estimated instead of ground-truth
fundamental matrices, ultimately improves the keypoint matches generated by
improving the consistency of the matches. By design, algorithms such as RANSAC
ignore many or most available point-matches and only use a subset to compute
the final hypothesis for the fundamental matrix. While the points that were
used for fundamental matrix estimation will experience minimal loss, the
remaining matches receive a rich gradient signal during training. In an effort
to improve the consistency of all generated matches with the subset of matches
that was used to estimate the fundamental matrices, the network will learn to
adapt to the new environment’s idiosyncrasies.

##  4 Experiments

To demonstrate the utility of the SCENES fine-tuning for downstream tasks, we
compare the performance of a number of keypoint correspondence estimation
methods

In this section, we demonstrate the utility of SCENES fine-tuning by comparing
the performance of fine-tuned models on the downstream task of relative camera
pose estimation with that of a number of existing keypoint correspondence
estimation methods. We evaluate on three datasets: the EuRoC-MAV drone dataset
[6], the San Francisco Landmark dataset [7], and the Aachen day–night dataset
[34, 35] in the supplement. We then conduct an ablation study on the fine-
tuning algorithm to assess the design decisions.

###  4.1 Pose-only supervision: EuRoC-MAV dataset

The EuRoC-MAV drone dataset [6] comprises eleven image sequences captured by a
remotely-operated drone in two distinct environments: a spacious machine room
and a smaller room. Leveraging a laser tracker in the machine room and a
motion capture system in the other room, accurate drone (and camera) poses
were meticulously recorded. The machine room significantly differs in scale
and content from ScanNet [10]. It is much larger than the offices, apartments,
etc. included in ScanNet, and contains industrial machinery that is not
commonly found in such locations. The image sequences vary in difficulty based
on their lighting levels and amount of motion blur. Even the images in the
‘easy’ sequences differ significantly from those in the ScanNet dataset,
especially with respect to the distribution of camera poses, when comparing
drone footage to handheld camera images.

To use the dataset for correspondence matching, we need to generate image
pairs that have enough overlap to make matching keypoints possible, but not so
much that the task becomes trivial (i.e., nearly-identical images). Without
ground-truth depth information, we cannot calculate the true overlap between
the images. Instead, we compute a ‘pseudo-overlap’ score using the known
camera poses, which visual inspection shows yields a good range of overlaps
between image pairs. More details on the dataset construction can be found in
the supplementary material.

#### Implementation details.

We fine-tune the MatchFormer-lite method [45] pre-trained on the indoor
ScanNet dataset [10], and evaluate using the images from two held-out
sequences of the highest difficulty, one from the machine room and one from
the indoor room. We leave all training settings from the original training run
unchanged, with the exception of decreasing the weight decay from $0.1$ to
$0.01$ and decreasing the learning rate from $3\times 10^{-3}$ to $1\times
10^{-4}$. Computational limitations also mean that we reduced the batch size
from $64$ to $12$. We also fine-tune the ASpanFormer method [9], also
pretrained on the indoor ScanNet dataset [10]. Again, we mostly retain the
training settings from the original training run, except that we reduce the
batch size to $8$ due to computational limitations.

#### Baselines.

We compare with a number of state-of-the-art image matchers on the dataset.
Since its loss function only requires camera poses, we train the CAPS [44]
method from scratch on EuRoC-MAV using the authors’ settings. For the other
baselines, we use the implementations and pre-trained weights (not trained on
EuRoC-MAV) provided by the respective authors, choosing the ‘indoor’ versions
where there is a choice. For every method we compute relative camera pose
estimates from the matched keypoints using a five-point essential matrix
estimator with RANSAC, as implemented in the OpenCV [5] library, with a
matching threshold of $0.5$ pixels.

#### Results.

Table 1: Relative pose estimation performance on the indoor EuRoC-MAV dataset [6]. We report the area under the curve (AUC) at thresholds of $5\degree$, $10\degree$ and $20\degree$ and the matching precision (P) at a threshold of $5\cdot 10^{-4}$. Our SCENES method exhibits significant gains in performance across these metrics. Method | Pose Estimation AUC (%) |  P (%)
---|---|---
$@5\degree$ |  $@10\degree$ |  $@20\degree$ |
SP [11] \+ SG [32] |  0.6 |  2.0 |  4.4 |  17.3
SP [11] \+ CAPS [44] |  1.4 |  6.4 |  16.3 |  20.6
SP [11] \+ LG [25] |  7.6 |  22.9 |  40.2 |  38.3
DISK [43] \+ LG [25] |  6.6 |  17.9 |  31.6 |  35.9
MatchFormer-large [45] |  5.1 |  16.0 |  31.8 |  38.4
MatchFormer-lite [45] |  3.0 |  11.6 |  25.1 |  35.0
\+ SCENES fine-tuning |  +6.1 |  +11.9 |  +17.7 |  +28.8
= Ours (MF) |  9.1 |  23.5 |  42.8 |  63.8
ASpanFormer [9] |  4.8 |  17.6 |  34.5 |  38.7
\+ SCENES fine-tuning |  +6.8 |  +8.8 |  +10.2 |  +18.1
= Ours (ASF) |  11.6 |  26.2 |  44.7 |  66.8

Like previous works [32, 38, 45, 9, 25], we report the area under the
cumulative curve (AUC) of the pose error (the maximum of the angular error in
rotation and translation) at thresholds of $5\degree$, $10\degree$ and
$20\degree$. We also report the matching precision (P): the average percentage
of predicted matches with a squared symmetrical epipolar distance smaller than
$5\cdot 10^{-4}$. As can be seen in Tab. 1 and in the qualitative results in
Fig. 3, the fine-tuned methods show a large jump in performance compared to
the original pre-trained methods. The fine-tuned version of the small
MatchFormer-lite network even outperforms the previous state-of-the-art, the
combination of the SuperPoint detector and the LightGlue matcher. The
performance of the baseline methods on this dataset is considerably worse than
on the dataset they were trained on (ScanNet), showing that their ability to
generalise to novel data distributions is limited. Our SCENES fine-tuning
approach goes some way to alleviating this issue. The CAPS algorithm also
struggles, not having benefited from pre-training with ground-truth point
correspondences on a different dataset.

(a) Matches before SCENES fine-tuning

(b) Matches after SCENES fine-tuning

(c) Matches before SCENES fine-tuning

(d) Matches after SCENES fine-tuning

Figure 3: Qualitative matching results on the EuRoC-MAV drone dataset [6]. The
correspondences found by the state-of-the-art MatchFormer algorithm [45] are
displayed before (a, c) and after (b, d) SCENES fine-tuning. The colour
denotes whether the squared symmetrical epipolar distance is below (green) or
above (red) a threshold of $5\cdot 10^{-4}$. Our method improves the quality
and number of correct matches.

###  4.2 Bootstrapped supervision: SF dataset

The San Francisco Landmark dataset [7] is a collection of roughly one million
perspective images obtained from panorama images collected by a car in
downtown San Francisco. The GPS measurements for the camera position and
orientation are also provided, though these measurements are frequently
inaccurate by up to dozens of metres. Test images were collected using camera
phones, and Torii et al. [42] provides accurate reference poses for 598 of
them. While the GPS measurements are too inaccurate to compute fundamental
matrices, we can still use them to generate image pairs using a similar
‘pseudo-overlap’ score as before. The reader is referred to the appendix for
details.

This dataset represents a realistic example of a setting without any pose
supervision available for training or fine-tuning, while still containing a
small test dataset with reference poses that allows us to evaluate the effect
of the fine-tuning. We give results of our bootstrapping method applied to the
EuRoC-MAV dataset (where accurate poses can be collected much more easily) in
the supplementary material.

#### Implementation details.

We again fine-tune the Matchformer-lite [45] and ASpanFormer [9] methods, this
time using the respective ‘outdoor’ weights trained on the MegaDepth dataset
[22] as initialisation. To employ the bootstrapping strategy described in Sec.
3.4 in the absence of ground-truth poses for training images, we use each of
the base models to compute fundamental matrices for one million randomly
selected image pairs using OpenCV’s implementation of RANSAC [16]. To maximise
the reliability of these estimates, we only use those image pairs for training
for which the base model extracted at least $100$ keypoint matches and for
which the estimated fundamental matrices result in at least $20$ inliers. This
leaves $749,581$ and $688,466$ image pairs for fine-tuning the Matchformer-
lite and ASpanFormer methods respectively.

The different dataset collection methods for the training and testing datasets
lead to meaningfully different data distributions. To ensure that the networks
are fine-tuned to adjust to the San Francisco environment without overfitting
on features such as the camera location, we add an equal number of image pairs
from the MegaDepth dataset [22] to each batch. The losses for these MegaDepth
images are the original correspondence losses.

For the Matchformer-lite model, we use the same training settings as during
the initial training, with the exception of again decreasing the weight decay
to $0.01$ and the learning rate to $1\cdot 10^{-4}$. We also decreased the
batch size to $8$ image pairs taken from each of the San Francisco and the
MegaDepth datasets. For the ASpanFormer model, we decreased the batch size
further to two image pairs from each of the two datasets, and also decreased
the fraction of coarse matches that are used for the fine-level supervision
during training from $30\%$ in the original training run to $10\%$. Due to
computational limitations, we also had to change the function $f$ used to
compute the coarse level loss from a cross entropy function to the sparse
formulation used in LoFTR [38].

#### Baselines.

We provide results for the same set of state-of-the-art baselines as for the
EuRoC-MAV dataset, computing the relative camera poses using RANSAC as before.
This time, we choose the ‘outdoor’ versions when multiple sets of weights are
available. To aid comparability between different methods, we pad all images
to $640\times 640$ pixels.

#### Results.

We report the area under the cumulative curve (AUC) of the pose error at
thresholds of $5\degree$, $10\degree$ and $20\degree$. Following Sarlin et al.
[32], the threshold for the matching precision in this outdoor setting is
$1\cdot 10^{-4}$.

This setting, where the images of the training dataset were collected in a
very different way to those in the testing dataset, does not play to the
strengths of a fine-tuning approach, as an important type of regularity that
the network might adapt to is now the source of misleading training signals.
Despite this, and despite the additional noise resulting from the
bootstrapping approach, the results in Tab. 2 show noticeable performance
improvements after fine-tuning. When fine-tuning with bootstrapping on the
EuRoC-MAV dataset (see the supplementary), where training and testing sets
were collected in more similar ways, the performance gains are even more
substantial.

These results show the feasibility of our bootstrapping strategy for fine-
tuning a keypoint matcher on datasets without any ground-truth information.

Table 2: Relative pose estimation performance on the outdoor San Francisco Landmarks dataset [7]. We report the area under the curve (AUC) at thresholds of $5\degree$, $10\degree$ and $20\degree$ and the matching precision (P) at a threshold of $10^{-4}$. Even without any available ground-truth correspondences, poses or 3D structure, our SCENES fine-tuning method is able to improve the performance of matching algorithms on this challenging dataset. Method | Pose Estimation AUC (%) |  P (%)
---|---|---
$@5\degree$ |  $@10\degree$ |  $@20\degree$ |
SP [11] \+ SG [32] |  1.2 |  4.1 |  9.9 |  15.1
SP [11] \+ LG [25] |  19.4 |  37.4 |  54.1 |  34.4
DISK [43] \+ LG [25] |  19.7 |  37.8 |  55.2 |  36.7
MatchFormer-large [45] |  21.2 |  39.6 |  56.8 |  32.4
MatchFormer-lite [45] |  19.8 |  36.9 |  54.0 |  32.1
\+ SCENES fine-tuning |  +0.1 |  +2.0 |  +2.2 |  -1.0
= Ours (MF) |  19.9 |  38.9 |  56.2 |  31.1
ASpanFormer [9] |  20.1 |  38.4 |  55.4 |  33.4
\+ SCENES fine-tuning |  +1.1 |  +0.4 |  0.0 |  -0.4
= Ours (ASF) |  21.2 |  38.8 |  55.4 |  33.0

###  4.3 Ablation study

Table 3: Ablation study comparing variants of our design, evaluated with respect to relative pose estimation performance on the indoor EuRoC-MAV dataset [6]. We report the area under the curve (AUC) at thresholds of $5\degree$, $10\degree$ and $20\degree$ and the matching precision (P) at a threshold of $5\cdot 10^{-4}$. The row “w/o classification argmax” denotes using the naïve epipolar classification loss, where ${\mathtt{M}}^{\textrm{epi}}_{ij}=1$ for all pixels on the epipolar line, not just the highest probability pixel. The rows ‘perturb poses by $x\degree$’ randomly perturb the ground truth translation and rotation by $x$ degrees.  Method | Pose Estimation AUC (%) |  P (%)
---|---|---
$@5\degree$ |  $@10\degree$ |  $@20\degree$ |
MatchFormer-lite [45] |  3.0 |  11.6 |  25.1 |  35.0
Ours (MF) |  9.1 |  23.5 |  42.8 |  63.8
w/o fine-level loss |  5.5 |  15.2 |  29.4 |  61.2
w/o coarse-level loss |  10.0 |  20.7 |  34.4 |  69.9
w/o classification argmax |  0.0 |  0.0 |  0.0 |  0.0
perturb poses by $1\degree$ |  8.4 |  20.2 |  36.5 |  63.1
perturb poses by $2\degree$ |  4.2 |  14.7 |  30.0 |  41.1

In this section, we analyse the impact of design choices on our method. We
evaluate modifications to the fine-tuning of the MatchFormer-lite model on the
EuRoC-MAV dataset [6]. The results of the ablations are in Tab. 3, using the
same metrics as before in Tab. 1. We evaluate the impact of three alterations
to the full method: removing the fine-level loss ($\lambda=0$), removing the
coarse-level loss ($\lambda=1$), and removing the argmax in eq. 3, setting
${\mathtt{M}}^{\textrm{epi}}_{ij}=1\textrm{ iff }{\mathbf{x}}_{2}^{j}\in
e_{1}^{i}$ and zero otherwise. This sets the mask to 1 for all pixels on the
epipolar line, not just the highest probability pixel.

If we remove the loss term for either the coarse or fine matching stage, the
network focuses exclusively on the remaining task and its performance on the
other deteriorates. Removing the fine-level loss leads the network to create a
large number of approximate matches, but these are too noisy to estimate
accurate poses. Removing the coarse-level loss leads the network to create
fewer matches. These may be very accurate, but too few matches make the RANSAC
pose estimator less robust and prone to fail entirely. While this model does
better for metrics where high accuracy is important, it does notably worse at
the metrics with less stringent thresholds ($10\degree$ and $20\degree$). On
balance, a mixture of both losses leads to the overall most reliable
performance.

Finally, as mentioned in Sec. 3.2, removing the argmax in the classification
loss actively discourages the network to select good coarse matches. The
matching probability in the confidence matrix ${\mathtt{C}}$ becomes spread-
out along the entire epipolar line, which reduces the probability that an
approximately correct coarse match will be selected. Fine-tuning with this
loss causes the model to diverge, failing to provide sufficient accurate
matches for pose estimation.

#### Impact of Noisy Ground-truth Poses.

We also evaluate the impact of noise in the ground truth poses on the
effectiveness of the SCENES fine-tuning. We fine-tune the Matchformer-lite
model on the EuRoC-MAV dataset as in Sec. 4.1, but randomly perturb the ground
truth poses (both rotation and translation) by magnitudes of one and two
degrees. These results can also be found in Tab. 3. While seemingly small
levels of pose noise affect the performance considerably, this is not too
problematic. As the results using our bootstrapping approach (cf. Sec. 4.2 and
the supplementary material) show, existing keypoint matching methods already
perform well enough to estimate fundamental matrices to sufficient accuracy
for SCENES fine-tuning.

For further quantitative and qualitative results and ablations, we refer the
interested reader to the appendix.

##  5 Discussion and Conclusion

#### Limitations.

One limitation is that the training signal is weaker than the fully-supervised
approach, since a match predicted to be at the wrong place on the epipolar
line does not incur a penalty. Moreover, the epipolar classification loss in
particular can be misleading, since it encourages the match to move towards
the highest probability position on the epipolar line, which may be suboptimal
and entrenches any biases learned by the pre-trained model.

#### Conclusion.

We have proposed a method for estimating subpixel correspondences between two
images, without requiring ground-truth correspondences for training. To do so,
we reformulate the standard classification and regression matching losses as
epipolar losses, which only require ground-truth fundamental matrices during
training. These can be computed when the relative camera poses and intrinsics
are known, or can be estimated by an adequate pre-trained matching network. We
develop two strategies for adapting pre-trained matching models to new
domains: one where the fundamental matrices are provided, and another where
they must be estimated, a bootstrapping approach. We demonstrate compelling
performance on challenging datasets, improving the pose estimation results of
state-of-the-art methods using our weakly supervised and unsupervised
finetuning strategies. This opens the door to transfer learning or active
learning applications.

#### Future work.

Four directions warrant further investigation. First, the bootstrapping
approach could be made iterative, where the fundamental matrices are re-
estimated every few epochs as the model improves. This may reduce the impact
of incorrect matches from the pre-trained network. Second, the approach could
be coupled with recent methods that infer camera poses given a sparse
collection of images [49, 24]. Third, given uncertainty estimates for the
relative poses, the epipolar losses could be modified to reflect this
uncertainty, transforming an epipolar line into a probability distribution on
the image plane. Finally, note that the true match must lie between the
epipole (projection of the first camera centre into the second image) and the
projection of the point at infinity. This additional constraint could be used
to improve the signal obtained from our epipolar losses.

#### Acknowledgements.

We are grateful for funding from EPSRC AIMS CDT EP/S024050/1 (D.K.),
Continental AG (D.C.), and the Royal Academy of Engineering (RF/201819/18/163,
J.H.). We would also like to thank Yash Bhalgat for valuable discussions.

## References

  * Arandjelovic et al. [2016] Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic.  Netvlad: Cnn architecture for weakly supervised place recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5297–5307, 2016.
  * Baker and Matthews [2004] Simon Baker and Iain Matthews.  Lucas–Kanade 20 years on: A unifying framework.  _IJCV_ , 56:221–255, 2004.
  * Bay et al. [2006] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool.  Surf: Speeded up robust features.  In _Computer Vision–ECCV 2006: 9th European Conference on Computer Vision, Graz, Austria, May 7-13, 2006. Proceedings, Part I 9_ , pages 404–417. Springer, 2006.
  * Bhalgat et al. [2023] Yash Bhalgat, Joao F Henriques, and Andrew Zisserman.  A light touch approach to teaching transformers multi-view geometry.  In _CVPR_ , pages 4958–4969, 2023.
  * Bradski [2000] G. Bradski.  The OpenCV Library.  _Dr. Dobb’s Journal of Software Tools_ , 2000.
  * Burri et al. [2016] Michael Burri, Janosch Nikolic, Pascal Gohl, Thomas Schneider, Joern Rehder, Sammy Omari, Markus W Achtelik, and Roland Siegwart.  The euroc micro aerial vehicle datasets.  _The International Journal of Robotics Research_ , 2016.
  * Chen et al. [2011] David Chen, Sam Tsai, Vijay Chandrasekhar, Gabriel Takacs, Huizhong Chen, Ramakrishna Vedantham, Radek Grzeszczuk, and Bernd Girod.  Residual enhanced visual vectors for on-device image matching.  In _2011 Conference Record of the Forty Fifth Asilomar Conference on Signals, Systems and Computers (ASILOMAR)_ , pages 850–854. IEEE, 2011.
  * Chen et al. [2021] Hongkai Chen, Zixin Luo, Jiahui Zhang, Lei Zhou, Xuyang Bai, Zeyu Hu, Chiew-Lan Tai, and Long Quan.  Learning to match features with seeded graph matching network.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 6301–6310, 2021.
  * Chen et al. [2022] Hongkai Chen, Zixin Luo, Lei Zhou, Yurun Tian, Mingmin Zhen, Tian Fang, David Mckinnon, Yanghai Tsin, and Long Quan.  Aspanformer: Detector-free image matching with adaptive span transformer.  In _ECCV_ , pages 20–36. Springer, 2022.
  * Dai et al. [2017] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5828–5839, 2017.
  * DeTone et al. [2018] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superpoint: Self-supervised interest point detection and description.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , pages 224–236, 2018.
  * Dusmanu et al. [2019a] Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler.  D2-net: A trainable cnn for joint description and detection of local features.  In _CVPR_ , pages 8092–8101, 2019a.
  * Dusmanu et al. [2019b] Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler.  D2-net: A trainable cnn for joint description and detection of local features.  In _Proceedings of the ieee/cvf conference on computer vision and pattern recognition_ , pages 8092–8101, 2019b.
  * Engel et al. [2017] Jakob Engel, Vladlen Koltun, and Daniel Cremers.  Direct sparse odometry.  _IEEE TPAMI_ , 40(3):611–625, 2017.
  * Farahani et al. [2021] Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R Arabnia.  A brief review of domain adaptation.  _Advances in data science and information engineering: proceedings from ICDATA 2020 and IKE 2020_ , pages 877–894, 2021.
  * Fischler and Bolles [1981] Martin A Fischler and Robert C Bolles.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.  _Communications of the ACM_ , 24(6):381–395, 1981\.
  * Harris et al. [1988] Chris Harris, Mike Stephens, et al.  A combined corner and edge detector.  In _Alvey vision conference_ , pages 10–5244. Citeseer, 1988.
  * Hartley and Zisserman [2004] Richard Hartley and Andrew Zisserman.  _Multiple View Geometry in Computer Vision_.  Cambridge University Press, 2 edition, 2004.
  * He et al. [2020] Yihui He, Rui Yan, Katerina Fragkiadaki, and Shoou-I Yu.  Epipolar transformers.  In _Proceedings of the ieee/cvf conference on computer vision and pattern recognition_ , pages 7779–7788, 2020.
  * Horn and Schunck [1981] Berthold KP Horn and Brian G Schunck.  Determining optical flow.  _Artificial intelligence_ , 17(1-3):185–203, 1981\.
  * Jiang et al. [2021] Shihao Jiang, Dylan Campbell, Yao Lu, Hongdong Li, and Richard Hartley.  Learning to estimate hidden motions with global motion aggregation.  In _ICCV_ , pages 9772–9781, 2021.
  * Li and Snavely [2018] Zhengqi Li and Noah Snavely.  Megadepth: Learning single-view depth prediction from internet photos.  In _Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Li et al. [2021] Zhaoshuo Li, Xingtong Liu, Nathan Drenkow, Andy Ding, Francis X Creighton, Russell H Taylor, and Mathias Unberath.  Revisiting stereo depth estimation from a sequence-to-sequence perspective with transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 6197–6206, 2021.
  * Lin et al. [2023] Amy Lin, Jason Y Zhang, Deva Ramanan, and Shubham Tulsiani.  Relpose++: Recovering 6d poses from sparse-view observations.  _arXiv preprint arXiv:2305.04926_ , 2023.
  * Lindenberger et al. [2023] Philipp Lindenberger, Paul-Edouard Sarlin, and Marc Pollefeys.  LightGlue: Local Feature Matching at Light Speed.  In _ICCV_ , 2023.
  * Lowe [1999] David G Lowe.  Object recognition from local scale-invariant features.  In _ICCV_ , pages 1150–1157. IEEE, 1999.
  * Mur-Artal et al. [2015] Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos.  ORB-SLAM: a versatile and accurate monocular SLAM system.  _IEEE Transactions on Robotics_ , 31(5):1147–1163, 2015.
  * Prasad et al. [2018] Vignesh Prasad, Dipanjan Das, and Brojeshwar Bhowmick.  Epipolar geometry based learning of multi-view depth and ego-motion from monocular sequences.  In _Proceedings of the 11th Indian Conference on Computer Vision, Graphics and Image Processing_ , pages 1–9, 2018.
  * Revaud et al. [2019] Jerome Revaud, Philippe Weinzaepfel, César De Souza, Noe Pion, Gabriela Csurka, Yohann Cabon, and Martin Humenberger.  R2d2: repeatable and reliable detector and descriptor.  _arXiv preprint arXiv:1906.06195_ , 2019.
  * Rublee et al. [2011] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski.  Orb: An efficient alternative to sift or surf.  In _2011 International conference on computer vision_ , pages 2564–2571. Ieee, 2011.
  * Sarlin et al. [2019] Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk.  From coarse to fine: Robust hierarchical localization at large scale.  In _CVPR_ , pages 12716–12725, 2019.
  * Sarlin et al. [2020] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superglue: Learning feature matching with graph neural networks.  In _CVPR_ , pages 4938–4947, 2020.
  * Sarlin et al. [2023] Paul-Edouard Sarlin, Philipp Lindenberger, Viktor Larsson, and Marc Pollefeys.  Pixel-perfect structure-from-motion with featuremetric refinement.  _IEEE TPAMI_ , 2023.
  * Sattler et al. [2012] Torsten Sattler, Tobias Weyand, Bastian Leibe, and Leif Kobbelt.  Image retrieval for image-based localization revisited.  In _BMVC_ , page 4, 2012.
  * Sattler et al. [2018] Torsten Sattler, Will Maddern, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Marc Pollefeys, Josef Sivic, et al.  Benchmarking 6dof outdoor visual localization in changing conditions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8601–8610, 2018.
  * Schonberger and Frahm [2016] Johannes L Schonberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In _CVPR_ , pages 4104–4113, 2016.
  * Shi et al. [2022] Yan Shi, Jun-Xiong Cai, Yoli Shavit, Tai-Jiang Mu, Wensen Feng, and Kai Zhang.  Clustergnn: Cluster-based coarse-to-fine graph neural network for efficient feature matching.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12517–12526, 2022.
  * Sun et al. [2021] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou.  LoFTR: detector-free local feature matching with transformers.  In _CVPR_ , pages 8922–8931, 2021.
  * Taira et al. [2018] Hajime Taira, Masatoshi Okutomi, Torsten Sattler, Mircea Cimpoi, Marc Pollefeys, Josef Sivic, Tomas Pajdla, and Akihiko Torii.  InLoc: indoor visual localization with dense matching and view synthesis.  In _CVPR_ , 2018.
  * Teed and Deng [2020] Zachary Teed and Jia Deng.  RAFT: recurrent all-pairs field transforms for optical flow.  In _ECCV_ , pages 402–419. Springer, 2020.
  * Toft et al. [2022] Carl Toft, Will Maddern, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Marc Pollefeys, Josef Sivic, Tomas Pajdla, Fredrik Kahl, and Torsten Sattler.  Long-term visual localization revisited.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(4):2074–2088, 2022.
  * Torii et al. [2019] Akihiko Torii, Hajime Taira, Josef Sivic, Marc Pollefeys, Masatoshi Okutomi, Tomas Pajdla, and Torsten Sattler.  Are large-scale 3d models really necessary for accurate visual localization?  _IEEE transactions on pattern analysis and machine intelligence_ , 43(3):814–829, 2019.
  * Tyszkiewicz et al. [2020] Michał Tyszkiewicz, Pascal Fua, and Eduard Trulls.  Disk: Learning local features with policy gradient.  _Advances in Neural Information Processing Systems_ , 33:14254–14265, 2020.
  * Wang et al. [2020] Qianqian Wang, Xiaowei Zhou, Bharath Hariharan, and Noah Snavely.  Learning feature descriptors using camera pose supervision.  In _Proc. European Conference on Computer Vision (ECCV)_ , 2020.
  * Wang et al. [2022] Qing Wang, Jiaming Zhang, Kailun Yang, Kunyu Peng, and Rainer Stiefelhagen.  Matchformer: Interleaving attention in transformers for feature matching.  In _ACCV_ , pages 2746–2762, 2022.
  * Wilson and Cook [2020] Garrett Wilson and Diane J Cook.  A survey of unsupervised deep domain adaptation.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ , 11(5):1–46, 2020.
  * Yang et al. [2022] Zhenpei Yang, Zhile Ren, Qi Shan, and Qixing Huang.  Mvs2d: Efficient multi-view stereo via attention-driven 2d convolutions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8574–8584, 2022.
  * Yifan et al. [2022] Wang Yifan, Carl Doersch, Relja Arandjelović, Joao Carreira, and Andrew Zisserman.  Input-level inductive biases for 3d reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6176–6186, 2022.
  * Zhang et al. [2022] Jason Y. Zhang, Deva Ramanan, and Shubham Tulsiani.  RelPose: Predicting probabilistic relative rotation for single objects in the wild.  In _ECCV_ , 2022.

Supplementary Material

##  A Heuristic for Creating Image Pairs

In this section, we describe in more detail the heuristics used to create
image pairs for both training and testing, using only camera pose estimates.

If ground-truth depth information is available, the overlap between two images
can be computed and valid image pairs are those with an overlap within some
range. This range is chosen to ensure there is enough overlap between the two
images of a pair to compute accurate keypoints, but not so much overlap to
make this task trivial.

Without ground-truth-depth, we assume a ‘pseudo-ground-truth-depth’, and use
this to compute approximate image overlaps. While this approximation is
extremely rough, visual inspection of the generated image pairs showed that it
nonetheless selects image pairs with the desired characteristic of enough, but
not too much, overlap.

#### EuRoC-MAV.

For both the EuRoC-MAV training and testing datasets, we assume that each
image was taken from inside a hemisphere. The hemisphere consists of a
horizontal plane at a certain height $z_{\textrm{plane}}<z_{\textrm{camera}}$
and a spherical dome with radius $r_{\textrm{sphere}}$ centered at
$(x_{\textrm{camera}},y_{\textrm{camera}},z_{\textrm{plane}})$.

For the machine room environments, we chose $z_{\textrm{plane}}=-2.0$m and
$r_{\textrm{sphere}}=10.0$m, and for the smaller room we chose
$z_{\textrm{plane}}=0.0$m and $r_{\textrm{sphere}}=3.0$m.

We also undistort the images in the EuRoC-MAV dataset and scale and crop them
to a resolution of $640\times 480$ pixels.

#### San Francisco.

The San Francisco training set images are provided together with GPS camera
pose information. These are too inaccurate to use as supervision signal, but
we are able to use them to generate image pairs.

Since most images in the training set are taken from the middle of a road with
multi-storey building on either side, we assume that each image was taken from
inside a rectangular box aligned with the current driving direction. The left
and right sides of the box are at a distance of $10$ meters from the camera,
the bottom plane is $2$ meters below the camera, the front and back planes are
at a distance of $25$ meters from the camera, and the top plane is at
infinity. We estimate the driving direction as the vector between locations of
the preceding and the succeeding images. Pixels whose rays intersect with the
front or back planes are always counted as not overlapping with another image.

Since the test set contained comparatively few images, we selected appropriate
image pairs manually.

We scale the images so that their longest side has length $640$ pixels, then
pad them to size $640\times 640$ pixels.

##  B Further Ablation Study

In this section, we give the results of some additional ablations (Tabs. 1 and
2). Firstly, we investigate the impact of slight variations in the definition
of $e^{i}_{1}\in{\mathtt{I}}_{2}$, the set of coordinates in the second image
that form the epipolar line for the point
${\mathbf{x}}_{1}^{i}\in{\mathtt{I}}_{1}$. Since we are working with pixel
patches of finite size, $e_{1}^{i}$ is the set of pixels that the epipolar
line “passes through”. Formally, if $w$ is the width of a single pixel patch,
this means that

$e^{i}_{1}=\\{{\mathbf{x}}_{2}:{\mathbf{x}}_{2}\in{\mathtt{I}}_{2},d^{\textrm{epi}}({\mathbf{x}}^{i}_{1},{\mathbf{x}}_{2})\leq\theta\frac{w}{2}\\}$
(6)

where $\theta$ is a parameter determining how close the epipolar line has to
pass to the center of the pixel patch to count as “passing through” the patch.
In the experiments in the main paper, we set $\theta=\sqrt{2}$, but other
values are possible. Secondly, we investigate the impact of adding or removing
images from the original training set (ScanNet [10] for indoor models, and
Megadepth [22] for outdoor models) to each batch, using the original
correspondence losses for these samples.

Table 1: Ablation study comparing variants of our design, evaluated on the indoor EuRoC-MAV dataset [6]. We report the area under the curve (AUC) at thresholds of $5\degree$, $10\degree$ and $20\degree$ and the matching precision (P) at a threshold of $5\cdot 10^{-4}$. For a detailed explanation of the parameter $\theta$, please refer to the main body and in particular to eq. 6. The row labelled ‘Updated version’ refers to a finetuned model that was finetuned for longer than the original finetuned version (‘Ours’) but not as long as the ablations (26 epochs compared to 20 and 36 epochs respectively).  Method | Pose Estimation AUC (%) |  P (%)
---|---|---
$@5\degree$ |  $@10\degree$ |  $@20\degree$ |
MatchFormer-lite [45] |  3.0 |  11.6 |  25.1 |  35.0
Ours (MF) |  9.1 |  23.5 |  42.8 |  63.8
Updated version (MF) |  12.9 |  26.7 |  42.1 |  69.9
w/ $\theta=1.0$ |  11.8 |  27.0 |  43.8 |  71.0
w/ $\theta=3\sqrt{2}$ |  14.8 |  30.3 |  46.9 |  56.8
w/ ScanNet samples |  13.9 |  28.5 |  44.2 |  68.2
Table 2: Ablation study comparing variants of our design, evaluated on the outdoor San Francisco dataset [7] . We report the area under the curve (AUC) at thresholds of $5\degree$, $10\degree$ and $20\degree$ and the matching precision (P) at a threshold of $10^{-4}$. For a detailed explanation of the parameter $\theta$, please refer to the main body and in particular to eq. 6.  Method | Pose Estimation AUC (%) |  P (%)
---|---|---
$@5\degree$ |  $@10\degree$ |  $@20\degree$ |
MatchFormer-lite [45] |  19.8 |  36.9 |  54.0 |  32.1
Ours (MF) |  19.9 |  38.9 |  56.2 |  31.1
w/ $\theta=3\sqrt{2}$ |  19.7 |  38.1 |  55.2 |  31.4
w/o Megadepth samples |  18.3 |  35.6 |  52.7 |  30.1

The additional ablations for the EuRoC-MAV dataset in Tab. 1 are consistently
stronger than the original finetuned model that we evaluated in the main
paper. This suggests that the model whose results were reported in the main
paper had not fully converged. This hypothesis that is supported by the
similarly improved results for the ‘Updated version’, which was trained for
longer than the original finetuned version reported in the main paper, but not
as long as the ablations in Tab. 1 (26 epochs compared to 20 and 35 epochs
respectively).

In any case, the consistently strong improvements with respect to the baseline
model show the robustness of our finetuning method to moderate changes in
certain parameters and the training algorithm. We can note that a lower value
for $\theta$ leads to lower pose estimation performance, as the pixel patch
with maximum confidence on the epipolar line is more likely to be further from
the correct match. The lower value for $\theta$ however leads to fewer coarse
matches, so that the matches that are found are more likely to be accurate,
increasing the precision.

The ablations for the San Francisco dataset in Tab. 2 illustrate the
importance of the added Megadepth samples to avoid excessively overfitting
onto the San Francisco training set distribution, which is slightly different
to the test set distribution. While increasing the parameter $\theta$ by a
factor of $3$ slightly decreases the localisation performance, finetuning
still improves performance.

##  C Additional Quantitative Results

In this section we provide some additional quantitative results on the
relative pose estimation task for the EuRoC-MAV and the San Francisco
datasets. We report the fraction of relative pose estimates for which the
angular rotation error and the angular translation error are below thresholds
of $5\degree,10\degree,20\degree$, as well as the median angular rotation and
translation errors in degrees (Tab. 3, Tab. 4).

Table 3: Relative pose estimation performance on the indoor EuRoC-MAV dataset [6]. We report the fraction of angular rotation and translation errors below the thresholds $5\degree$, $10\degree$ and $20\degree$ in percent and the median angular rotation and translation errors in degree. Our SCENES method exhibits significant gains in performance across these metrics. Method | Rotation
---|---
$<5\degree$ (%) |  $<10\degree$ (%) |  $<20\degree$ (%) |  Median (°)
SP [11] \+ SG [32] |  5.20 |  12.0 |  29.3 |  39.3
SP [11] \+ LG [25] |  54.0 |  67.6 |  77.1 |  4.16
DISK [43] \+ LG [25] |  44.1 |  56.5 |  64.7 |  6.86
MatchFormer-large [45] |  43.5 |  61.6 |  74.3 |  6.18
MatchFormer-lite [45] |  37.3 |  54.4 |  64.7 |  7.66
\+ SCENES finetuning |  +17.6 |  +26.0 |  +29.6 |  -3.40
= Ours (MF) |  54.9 |  80.4 |  94.3 |  4.26
ASpanFormer [9] |  48.4 |  65.9 |  74.6 |  5.28
\+ SCENES finetuning |  +10.3 |  +12.3 |  +15.9 |  -1.43
= Ours (ASF) |  58.7 |  78.2 |  90.5 |  3.85
| Translation
|  $<5\degree$ (%) |  $<10\degree$ (%) |  $<20\degree$ (%) |  Median (°)
SP [11] \+ SG [32] |  2.80 |  7.53 |  17.2 |  48.6
SP [11] \+ LG [25] |  26.1 |  50.9 |  65.1 |  9.74
DISK [43] \+ LG [25] |  19.9 |  38.8 |  51.4 |  18.1
MatchFormer-large [45] |  18.8 |  38.8 |  56.2 |  14.0
MatchFormer-lite [45] |  11.8 |  30.3 |  48.5 |  21.5
\+ SCENES finetuning |  +14.9 |  +19.3 |  +24.6 |  -11.2
= Ours (MF) |  26.7 |  49.6 |  73.1 |  10.3
ASpanFormer [9] |  21.1 |  45.3 |  61.7 |  11.7
\+ SCENES finetuning |  +7.9 |  +7.3 |  +10.7 |  -2.49
= Ours (ASF) |  29.0 |  52.6 |  72.4 |  9.21
Table 4: Relative pose estimation performance on the outdoor San Francisco dataset [7]. We report the fraction of angular rotation and translation errors below the thresholds $5\degree$, $10\degree$ and $20\degree$ in percent and the median angular rotation and translation errors in degree. Method | Rotation
---|---
$<5\degree$ (%) |  $<10\degree$ (%) |  $<20\degree$ (%) |  Median (°)
SP [11] \+ SG [32] |  7.95 |  17.4 |  32.7 |  33.0
SP [11] \+ LG [25] |  64.5 |  79.0 |  87.4 |  3.32
DISK [43] \+ LG [25] |  64.0 |  80.8 |  89.0 |  3.38
MatchFormer-large [45] |  65.8 |  82.6 |  88.7 |  3.30
MatchFormer-lite [45] |  63.4 |  79.0 |  87.2 |  3.41
\+ SCENES finetuning |  +0.4 |  +1.6 |  +2.0 |  +0.1
= Ours (MF) |  63.8 |  80.6 |  89.2 |  3.42
ASpanFormer [9] |  63.4 |  79.7 |  87.2 |  3.47
\+ SCENES finetuning |  +1.7 |  +0.2 |  +0.4 |  -0.18
= Ours (ASF) |  65.1 |  79.9 |  87.6 |  3.29
| Translation
|  $<5\degree$ (%) |  $<10\degree$ (%) |  $<20\degree$ (%) |  Median (°)
SP [11] \+ SG [32] |  10.6 |  26.3 |  49.9 |  20.0
SP [11] \+ LG [25] |  51.2 |  68.7 |  78.6 |  4.77
DISK [43] \+ LG [25] |  51.7 |  70.9 |  78.8 |  4.76
MatchFormer-large [45] |  54.1 |  71.5 |  81.0 |  4.30
MatchFormer-lite [45] |  51.7 |  68.9 |  79.0 |  4.69
\+ SCENES finetuning |  +2.8 |  +2.4 |  +1.4 |  -0.35
= Ours (MF) |  54.5 |  71.3 |  80.4 |  4.34
ASpanFormer [9] |  53.6 |  71.5 |  80.6 |  4.42
\+ SCENES finetuning |  -0.4 |  -0.6 |  -1.1 |  +0.12
= Ours (ASF) |  53.2 |  70.9 |  79.5 |  4.54

##  D Additional Qualitative Results

In this section, we provide some additional qualitative results for finetuning
the Matchformer-lite model on the EuRoC-MAV [6] dataset in Fig. 4 and the San
Francisco [7] dataset in Fig. 5. The lines indicating matched keypoints are
coloured based on whether the matches’ symmetric epipolar error is above or
below a threshold of $5\cdot 10^{-4}$ for the indoor EuRoC-MAV dataset and
$10^{-4}$ for the outdoor San Francisco dataset.

The performance improvements on the EuRoc-MAV dataset after finetuning using
ground-truth camera poses are very noticeable, with both more and more
accurate keypoint matches being found. The performance improvements on the San
Francisco dataset after finetuning using our bootstrapping method without
ground-truth poses are less stark but still evident in an increased number of
accurate matches being found. The improvements are weakest for image pairs
that the base model also fails on. This can be explained by the finetuning
mostly increasing performance by improving the consistency of matches.

(a)

(b)

(c)

(d)

Figure 4: Matches for the Matchformer-lite [45] model before (left) and after
(right) SCENES-finetuning on the EuRoC-MAV dataset.

(a)

(b)

(c)

(d)

Figure 5: Matches for the Matchformer-lite [45] model before (left) and after
(right) SCENES-finetuning on the San Francisco dataset.

##  E Localisation on Aachen Day-Night

We also provide results of models fine-tuned with our method as part of a
larger image localisation pipeline. In particular, following previous papers
[32, 38, 9, 25], we use the HLoc pipeline [31] and the Aachen Day-Night-v1.1
[34, 35] dataset.

#### Implementation Details.

To create the image pairs for finetuning in the absence of even approximate
camera pose information, we compute embeddings for each image using NetVLAD
[1] and retain all image pairs for which these embeddings have a cosine
similarity score of at least $0.24$.

We finetune the Matchformer-lite outdoor model using the Aachen Day-Night
training images using the same settings as for the San Francisco dataet, using
our bootstrapped approach with fundamental matrices estimated by the base
model, and adding an equal number of image pairs from the MegaDepth dataset
[22] to each batch. We also again set the weight decay to $0.01$, the learning
rate to $1\cdot 10^{-4}$, and the batch size to $8$ image pairs taken from the
Aachen Day-Night and MegaDepth datasets each.

#### Baselines.

We also provide results for the same set of keypoint localisation and matching
algorithms as in our other experiments. To ensure the results are comparable,
we resize all images so that their longest side has a length of $640$ pixels.
To reduce the computational cost, we follow the pre-implemented algorithms in
the HLoc pipeline and restrict the matches to return a maximum of one match
per $8\times 8$ pixel patch for the dense matching algorithms (Matchformer[45]
and ASpanFormer[9]).

#### Results.

The results as computed by the Long-Term Visual Localization benchmark [41]
server (https://www.visuallocalization.net) can be found in Tab. 5. Our
bootstrapped SCENES method improves the localisation performance on the
daytime images, but decreases the localisation performance on nighttime
images. This is not entirely surprising – the Aachen Day-Night training set
consists only of daytime images, and finetuning processes generally sacrifice
improved performance for slightly decreased generalisation ability.

Table 5: Image Localisation performance on the Aachen Day-Night-v1.1 [34, 35] dataset using the HLoc[31] toolbox. Method |  Day |  Night
---|---|---
$(0.25\textrm{m},2\degree)/(0.5\textrm{m},5\degree)/(1.0\textrm{m},10\degree)$
SP [11] \+ SG [32] |  87.1 / 94.3 / 98.3 |  69.1 / 88.0 / 97.9
SP [11] \+ LG [25] |  88.1 / 94.7 / 98.9 |  72.3 / 89.5 / 99.0
DISK [43] \+ LG [25] |  89.7 / 95.9 / 99.0 |  77.0 / 91.6 / 99.5
ASpanFormer [9] |  88.3 / 95.5 / 98.9 |  76.4 / 90.6 / 99.0
MatchFormer-large [45] |  87.7 / 95.8 / 98.9 |  74.9 / 91.1 / 99.0
MatchFormer-lite [45] |  87.7 / 94.3 / 98.4 |  70.7 / 90.6 / 99.0
\+ SCENES finetuning |  +0.2 / +0.4 / +0.1 |  -1.6 / -0.5 / -0.6
= Ours (MF) |  87.9 / 94.7 / 98.5 |  69.1 / 90.1 / 98.4

##  F Bootstrapping on EuRoC-MAV

In this section we provide the results of fine-tuning the MatchFormer-lite and
ASpanFormer models on the EuRoC-MAV dataset without ground-truth pose
supervision. Instead, we follow our bootstrapping strategy and compute the
epipolar loss using the fundamental matrices estimated by the pre-trained
models.

The dataset and evaluation are the same as in Sec. 4.1 in the main paper, and
the results are available (together with the results from the main paper) in
Tab. 6.

Table 6: Relative pose estimation performance on the indoor EuRoC-MAV dataset [6]. We report the area under the curve (AUC) at thresholds of $5\degree$, $10\degree$ and $20\degree$ and the matching precision (P) at a threshold of $5\cdot 10^{-4}$. Our SCENES method exhibits significant gains in performance across these metrics even when not using any ground truth poses for fine-tuning (‘bootstrapping’). Method | Pose Estimation AUC (%) |  P (%)
---|---|---
$@5\degree$ |  $@10\degree$ |  $@20\degree$ |
SP [11] \+ SG [32] |  0.6 |  2.0 |  4.4 |  17.3
SP [11] \+ LG [25] |  7.6 |  22.9 |  40.2 |  38.3
DISK [43] \+ LG [25] |  6.6 |  17.9 |  31.6 |  35.9
MatchFormer-large [45] |  5.1 |  16.0 |  31.8 |  38.4
MatchFormer-lite [45] |  3.0 |  11.6 |  25.1 |  35.0
\+ SCENES finetuning |  +6.1 |  +11.9 |  +17.7 |  +28.8
= Ours (MF) |  9.1 |  23.5 |  42.8 |  63.8
MatchFormer-lite [45] |  3.0 |  11.6 |  25.1 |  35.0
\+ SCENES bootstrapping |  +5.2 |  +12.3 |  +17.7 |  +6.5
= Ours (MF-BS) |  8.2 |  23.9 |  42.8 |  41.5
ASpanFormer [9] |  4.8 |  17.6 |  34.5 |  38.7
\+ SCENES finetuning |  +6.8 |  +8.8 |  +10.2 |  +18.1
= Ours (ASF) |  11.6 |  26.2 |  44.7 |  66.8
ASpanFormer [9] |  4.8 |  17.6 |  34.5 |  38.7
\+ SCENES bootstrapping |  +3.8 |  +7.1 |  +8.4 |  +3.4
= Ours (ASF-BS) |  8.6 |  24.7 |  42.9 |  42.1

Even with the additional noise in the supervisory signal due to using more
approximate estimated fundamental matrices, the performance gain due to the
SCENES fine-tuning is almost as substantial as when using ground-truth
relative poses and fundamental matrices.

We might explain this by noting that the epipolar loss already only provides
an approximate supervisory signal that is least likely to be misleading when
the detected pixel match is relatively accurate. The estimated fundamental
matrices used int he bootstrapping method are least accurate for image pairs
that the pre-trained model performs worst on, but these are the image pairs
that the epipolar loss is least useful even when using ground-truth
fundamental matrices. The additional noise due to estimating fundamental
matrices therefore does not do a lot of additional harm in those cases. This
hypothesis is supported by the decrease in performance of the bootstrapping
method being largest for the smallest threshold; when the keypoint accuracy
needs to be highest, the additional noise is most deleterious.

These results underscore that our bootstrapping method is a very viable
approach to easily adapt a pre-trained pixel matching model to new domains.