# Hyperbolic embedding of brain networks as a tool for epileptic seizures
forecasting

Martin Guillemaud  These authors contributed equally to this work  Paris Brain
Institute (ICM), CNRS, Inserm, Sorbonne University, Inria-Paris. Pitié
Salpêtrière University Hospital. Paris, France  Louis Cousyn  These authors
contributed equally to this work  Paris Brain Institute (ICM), Sorbonne
Université, CNRS UMR, Inserm.
AP-HP, Department of Neurology, Epilepsy Unit, Center of Reference for Rare
Epilepsies, ERN EPICARE, Pitié Salpêtrière University Hospital, Paris, France
Vincent Navarro  Paris Brain Institute (ICM), Sorbonne Université, CNRS UMR,
Inserm.
AP-HP, Department of Neurology, Epilepsy Unit, Center of Reference for Rare
Epilepsies, ERN EPICARE, Pitié Salpêtrière University Hospital, Paris, France
Mario Chavez  CNRS, Pitié Salpêtrière University Hospital. Paris, France

###### Abstract

The evidence indicates that intracranial EEG connectivity, as estimated from
daily resting state recordings from epileptic patients, may be capable of
identifying preictal states. In this study, we employed hyperbolic embedding
of brain networks to capture non-trivial patterns that discriminate between
connectivity networks from days with (preictal) and without (interictal)
seizure. A statistical model was constructed by combining hyperbolic geometry
and machine learning tools, which allowed for the estimation of the
probability of an upcoming seizure. The results demonstrated that representing
brain networks in a hyperbolic space enabled an accurate discrimination (85%)
between interictal (no-seizure) and preictal (seizure within the next 24
hours) states. The proposed method also demonstrated excellent prediction
performances, with an overall accuracy of 87% and an F1-score of 89% (mean
Brier score and Brier skill score of $0.12$ and $0.37$, respectively). In
conclusion, our findings indicate that representations of brain connectivity
in a latent geometry space can reveal a daily and reliable signature of the
upcoming seizure(s), thus providing a promising biomarker for seizure
forecasting.

††preprint: APS/123-QED

##  I Introduction

In the last decades, complex networks have provided an increasingly
challenging framework for the study of connected systems (from social sciences
to biology and physics), based on the interplay between the wiring
architecture and the dynamical properties of the coupled elements [1]. In
neurosciences, the representation of brain networks as graphs allows to better
describe their non-trivial connectivity properties in a compact and objective
way [2, 3]. In this mapping of brain data (e.g. scalp or intracranial
electroencephalography or magnetoencephalography) to networks, nodes usually
represent brain regions or recording sites (e.g., electrodes, sensors), and
edges or links indicate functional connections between them, based on an
estimated statistical relationship between the recorded signals. Regardless of
the modality of acquisition, the use of graph analysis in neurosciences has
become essential to characterize pathological or physiological states in terms
of connectivity brain networks [2, 3, 4]. The application of different graph
metrics, including node’s degree, centrality, communicability, and efficiency,
has yielded insights into brain function in both healthy and pathological
conditions [4].

Epilepsy is a neurological disorder nowadays conceptualized as a network
disease with functionally and/or structurally aberrant connections on
virtually all spatial scales [5]. According to this concept, a large-scale
epileptic network comprises brain areas that can generate and sustain normal
and physiological dynamics during the seizure-free interval, and are mainly
involved in the generation, maintenance, spread, and termination of
pathophysiological activities such as seizures [5, 6]. Network connectivity
analysis in epilepsy has provided valuable information on seizure onset and
propagation, as well as on the functional organization of the brain during the
seizure-free interval [6]. During seizures, brain networks have been found to
display a more regular structure with less variability [5, 6]. Nevertheless,
current representations of brain networks fail to provide reliable biomarkers
to predict seizure onset or to estimate a risk of seizure occurrence. [7, 5,
6].

Embedding (or vector space) methods identify a lower-dimensional space in
which high-dimensional complex data can be represented. Modern dimensionality
reduction methods learn similarities and proximities between points
distributed over a hidden manifold in a multidimensional feature space. These
methods then preserve, embed, and visualize the data in a low-dimensional
space. Although Euclidean geometry serves as a standard framework for studying
our physical reality, an increasing body of evidence suggests that non-
Euclidean geometries are a more appropriate framework for capturing non-
trivial features (e.g., hierarchical or multiscale structure) often observed
in different biological networks, such as the olfactory space, cell
development data, single-cell sequencing data, and the brain connectivity [8].

The prevailing network embedding methodologies assume a zero-curvature (or
flat) space and evaluate the distance between embedded nodes according to
Euclidean metrics. These approaches, however, are limited when dealing with
complex hierarchical structures, as the resulting pairwise distances in the
embedded spaces are substantially distorted [8]. In network analysis, the
question of whether it exists a hidden (latent) non-Euclidean geometry from
which complex connectivity emerges has recently attracted the attention of the
scientific community and is getting more and more ground [9, 10]. Recently,
spaces with negative curvature (hyperbolic geometries) have attracted a lot of
attention as they enable low-distortion embeddings of hierarchical or multi-
scale connectivity structures [8]. The possibility to find an effective
congruency between brain network characteristics and its representation in
hyperbolic spaces offers thus the possibility to understand its structure and
address the study of brain organization in a novel and promising framework. In
some cases, endowed metrics of such spaces allow the use of machine learning
tools to perform statistical analysis of the embedded points (e.g. clustering,
prediction, …) [11].

Previous studies [12, 13] showed that brain connectivity matrices, estimated
from intracranial electroencephalography (EEG) recordings, allowed the
identification of preictal states. In the present study we used network
embedding methods to predict the risk of seizure occurrence. Our approach
aimed at exploring hyperbolic geometry to represent functional brain networks
of patients with epilepsy, and to work in a more adapted space representation
to unveil patterns that could potentially result in robust biomarkers for
seizure forecasting.

##  II  Material and method

###  II.1 Data acquisition

The EEG dataset used in this study was previously presented in [14]. Briefly,
daily 10-minute resting-state intracranial EEG recordings were obtained in 10
patients (mean age 30.7 years) with drug-resistant focal epilepsy from January
2019 to July 2021 in the Epilepsy Unit of the Pitié-Salpêtrière Hospital
(Paris, France). The study was conducted in accordance with the Helsinki
Declaration and approved by an institutional review board (project C11-16 and
C19-55 of the French National Institute of Health and Medical Research
sponsor).

The mean number of daily recordings per patient was 11. Each daily recorded
period was labeled as “preictal” in case an electro-clinical seizure occurred
in the next 24 hours, or “interictal” otherwise. Connectivity graphs were
derived from matrices of phase locking values (PLV) estimated between pairs of
EEG signals during 20-second non-overlapping epochs (resulting in 30
connectivity matrices per day). Synchrony (PLV) values were obtained in the
typical EEG frequency bands: delta ($\delta$) 1-4 Hz, theta ($\theta$) 4-8 Hz,
alpha ($\alpha$) 8-13 Hz, beta ($\beta$) 13-30 Hz, low gamma (low $\gamma$)
30-49 Hz and high gamma (high $\gamma$) 51-90 Hz. Only contacts in the gray
matter were considered and a bipolar montage between adjacent contacts was
applied.

###  II.2 Hyperbolic embedding of brain networks

In this study, we employed hyperbolic geometrical space for the representation
of brain networks. Compared with graph mappings in Euclidean spaces,
hyperbolic embeddings exhibit low distortion and can unfold network
properties, such as clustering or hierarchical community structure [15, 16,
17], which could reveal key structural principles underlying the organization
of the brain. The main steps required to embed the brain networks into this
geometric space include: i) filtering the connectivity matrix, ii) embedding
the network into the geometric space, and iii) aligning the embedding in the
hyperbolic space.

_Networks filtering_. The initial stage of network embedding consists of
filtering the PLV matrices by applying a threshold to cancel a percentage of
the weakest values. Here, we followed Ref. [18], and the synchrony matrices
were filtered such that the final networks reached a predetermined mean
degree, fixed here to 4. Although a spanning tree filter could be applied to
ensure a unique acyclic subgraph that connects all nodes [19], here we
preferred the method of Ref. [18] as it emphasizes the intrinsic global and
local properties of the network, while preserving its sparsity.

_Networks embedding_. Hyperbolic embedding methods are becoming increasingly
prevalent in the literature on neural circuits [20, 21, 17, 22]. Hyperbolic
geometry studies metric spaces with a constant negative curvature that do not
conform to Euclidean geometry. Such embedding methods typically project the
nodes of a network onto a hyperboloid, which can then be projected onto a two-
dimensional hyperbolic space model, such as the Poincaré disk or the so-called
Klein model disk. Methods for mapping a network into the hyperbolic disk
essentially belong to two families [23, 24]: generative model-based (e.g.,
Mercator [25]) and machine learning-based. Here, we used the coalescent
embedding method [26] to project the brain networks onto the Poincaré disk
$\mathbb{D}$. The method belongs to the second family, and we choose it for
its remarkable versatility and especially its computational speed [27].

In a nutshell, the method first adjusts the network’s edges’ weights, giving
more weight to those connections that play a greater role in information
transmission. Next, the graph is projected onto a 2-dimensional space using a
non-linear dimension reduction method such as Isomap or the Laplacian
eigenmaps [28]. Here we employed the technique of Laplacian Eigenmaps as it is
computationally efficient and it optimally preserves nodes’ local neighborhood
information [29]. At this stage, all nodes have polar coordinates $(r,\theta)$
in the 2-dimensional space. The nodes undergo an equidistant adjustment in
which all radii are set to 1 and the angular coordinates of the nodes are
modified while maintaining their angular order within the circle. Finally,
each vertex is assigned a radius based on its rank in the order of increasing
degree. The radius of a node is thus related to its popularity, while the
angular distance between two nodes is related to their topological similarity.

In the Poincaré disk, the hyperbolic distance $\mathrm{dist}_{hyp}(i,j)$
between each pair of nodes $i$ and $j$, assigned with radii $(r_{i},r_{j})$
and angles $(\theta_{i},\theta_{j})$ with coordinates $(r_{i},\theta_{i})$ and
$(r_{j},\theta_{j})$, is computed according to the hyperbolic law of
cosines[30]:

$\cosh\mathrm{dist}_{hyp}(i,j)=\cosh r_{i}\times\cosh r_{j}\\\ -\sinh
r_{i}\times\sinh r_{j}\times\cos(\pi-|\pi-|\theta_{i}-\theta_{j}||)$ (1)

_Graphs alignment_. Prior to comparing groups of embedded networks, it was
necessary to correct or align the embeddings, as a minor perturbation could
result in a random angular offset to the nodes’ positions in the hyperbolic
disk. A minor connectivity perturbation may thus result in two embeddings with
the same similarities between nodes but with a different structure regarding
the coordinates in the disk. [31]. To address this issue, a small rotation
angle between two embeddings was added such that the following hyperbolic
similarity score was minimized:

$\Gamma_{\mathrm{similarity}}=\sum_{i=0}^{N}\mathrm{dist}_{hyp}\left(\mathrm{Pos_{o}}(i),\mathrm{Rot}_{\theta}\left[\mathrm{Pos_{p}}(i)\right]\right)$
(2)

where $\mathrm{Pos_{o}}(i)$ and $\mathrm{Pos_{p}}(i)$ denote the positions of
the node $i$ in the first (or reference) and second hyperbolic disk, and
$\mathrm{Rot}_{\theta}\left[\cdot\right]$ indicates a rotation in the
Poincare’s disc with an angle $\theta\in\left[-\pi,\pi\right[$. In order to
facilitate comparison in subsequent steps, all networks from both groups were
aligned using the same reference network.

###  II.3  Hyperbolic score disk calculation

Our approach aimed at identifying disparities between preictal and interictal
connectivity patterns in the hyperbolic embedding of the brain networks. To
achieve this, we first defined a Gaussian model in the hyperbolic space for
each embedded node of the interictal networks (reference group). Subsequently,
the proximity of nodes from a network to the interictal connectivity can be
evaluated through the hyperbolic error [32]. Learning from the interictal
networks, the most representative data in the dataset, alleviates the bias
related to class-imbalanced training datasets. For each network in both the
preictal and interictal group, the position of each node in the hyperbolic
disk was examined and the node was assigned the value of the hyperbolic
Gaussian probability density function of this node in the reference interictal
group. The principle is illustrated in Fig. 1.

It should be noted that hyperbolic space is a Riemannian manifold, rather than
a vector space. As a result, the basic operations of vectors and matrices are
either intractable or not explicitly defined in hyperbolic spaces.
Nevertheless, for a point $z\in\mathbb{D}$, the tangent space at $z$, denoted
by $T_{z}\mathbb{D}$, is an inner product space, which contains the tangent
vector with all possible directions at $z$. We can transport points in
$\mathbb{D}$ to the tangent space $T_{z}\mathbb{D}$, and apply most vector
operators (average, gradients, …) within this Euclidean subspace, and then
project back the result to the disk. Transportation between $T_{z}\mathbb{D}$
and $\mathbb{D}$ can be achieved via the logarithmic and exponential maps,
$\mathrm{Log}_{z}$ and $\mathrm{Exp}_{z}$, respectively, defined in Appendix
A.1.

To estimate the hyperbolic error we first projected each node $i$ from all
reference (interictal) networks into a tangent space to estimate their
barycenter $\hat{z}_{i}$ with the algorithm detailed in Appendix A.2. For each
node $i^{k}$ from network $k$ in the second group, with coordinates
$(x_{i}^{k},y_{i}^{k})$ in the hyperbolic disk $\mathbb{D}$, the score value
was calculated using the following formula:

$\displaystyle\mathrm{Score}(i^{k})$
$\displaystyle=\mathrm{P}_{i}^{\mathrm{ref}}\left({W_{i}^{k}}\right)$
$\displaystyle=\frac{1}{\sqrt{\left(2\pi\right)^{2}\|V_{i}\|}}\mathrm{Exp}\left[-\frac{1}{2}{{W_{i}^{k}}}^{T}V_{i}^{-1}{W_{i}^{k}}\right]$
(3)

where

${W_{i}^{k}}=\begin{bmatrix}\mathrm{Real}\left(\mathrm{Log}_{\hat{z}_{i}}\left(x_{i}^{k}+jy_{i}^{k}\right)\right)\\\
\mathrm{Im}\left(\mathrm{Log}_{\hat{z}_{i}}\left(x_{i}^{k}+jy_{i}^{k}\right)\right)\end{bmatrix}$
(4)

denotes the projected coordinates of node $i^{k}$ from network $k$ in the
tangent space centered on $\hat{z}_{i}$ (the hyperbolic barycenter of nodes
$i$ in the reference group). $\|V_{i}\|$ is the determinant of covariance
matrix $V_{i}$ of the nodes $i$ from the reference embeddings projected in the
tangent space centered on $\hat{z}_{i}$, and $j$ denotes the imaginary unit
$j=\sqrt{-1}$. It should be noted that the barycenter of the distribution of
the set of points projected in this tangent space has coordinates $(0,0)$.

$\mathrm{P}_{i}^{\mathrm{ref}}\left({W_{i}^{k}}\right)$ is the hyperbolic
Gaussian model of nodes $i$ in the reference interictal networks, estimated at
the position of the $i^{\mathrm{th}}$ node of a network $k$. Once the scores
of all the nodes have been calculated, the disk is discretized with a grid,
and the errors of the nodes can be interpolated in the whole disk (one for
each network). To perform the interpolation, the scores of the $n=8$ nearest
nodes to each pixel of the grid are averaged, with the weights being inversely
proportional to the distances between the pixel and the nodes’ positions on
the disk. This results in a Hyperbolic Score Disk (HDS) for each network (see
Fig. 1).

Figure 1:  Calculation of the Hyperbolic Score Disks. First, we computed the
hyperbolic Gaussian distributions of the nodes from the reference group
(interictal networks). The networks’ nodes from the second group are then
compared to these distributions, and a score is attributed to each node.
Finally, the scores of the nodes are interpolated across the entire hyperbolic
disk.

###  II.4 Discrimination in the Hyperbolic Score Disk

To ascertain whether there were any significant differences between the
hyperbolic score disks of the preictal and interictal groups, a Student’s
t-test was employed. Firstly, all HSDs from the two groups were obtained. For
a statistical cross-validation of the preictal-interictal networks separation,
the two groups can be a subsample of the total number of disks. With the two
groups defined, all pixels in the HSD are vectorized, and compared with a
t-test to obtain a vector of t-values (one for each pixel of the disk), which
can then be back-projected onto the original hyperbolic disk. Permutation
tests were employed to provide an accurate approximation of the $p_{value}$
for each pixel. These tests entailed randomly permuting individuals between
the two groups and subsequently performing t-tests on the resulting groups to
compare them with the original value. The region of interest (referred here to
as the ROI) is the region of the disk where the $p_{values}$ are the smallest,
indicating the region of the disk where the two groups exhibit the greatest
statistical divergence. An example of ROI is illustrated in Fig. 2.

###  II.5 Prospective forecasting

Figure 2:  Forecasting method for one patient and one frequency band. In this
example, the aim is to forecast the seizure likelihood of networks from day 6
using data from days 1 to 5. To achieve this, Gaussian models are associated
to all nodes using interictal networks from days 1 to 5. The region of the HSD
where the two groups exhibit the largest statistical difference (ROI) is then
defined. The values within this ROI are utilized to train a logistic
regression model, which predicts the category of each network of the following
day(s).

The model was trained using data from all previous days to predict the
probability of a given network being in the preictal group (with a seizure
within the next 24 hours). The training set was required to include at least
one day from each group (preictal and interictal). Gaussian models were
constructed from the embedded interictal networks, and the hyperbolic score
disk was calculated for all networks in the training set. The two groups of
HSD (interictal and preictal networks) are then compared, and a region of
interest (ROI) is identified within the hyperbolic disk, as previously
described. For each embedded network from the learning set, the median value
of the pixels within the ROI is calculated and used as an independent variable
for the prediction. As the connectivity matrices were available in multiple
frequency bands, these steps were repeated for each frequency band. Each disk
(30 per day) was assigned to the class of the day (with or without seizures)
and was associated with a vector containing the median values of the ROIs for
each frequency band. The aforementioned data were subsequently utilized to fit
a logistic regression, thereby enabling a new network to be assigned a
probability, $p$, of belonging to a preictal day. For each new day, the
probability of belonging to the preictal class was calculated by averaging the
probabilities associated with the 30 disks of the day. The combinations of
bands with the highest prediction score, calculated using the training set of
each patient, were then utilized in a logistic regression for the prediction
of successive days. It is important to note that the selected combination of
bands may differ between patients. The methodology for a single frequency band
is illustrated in Fig. 2.

###  II.6 Evaluation of classification and forecasting performances

As connectivity patterns may be highly correlated between short epochs from
the same daily recording, which could result in overestimated discrimination
performances, we considered a leave-one-day-out cross-validation: for each
patient, all 30 networks from a single day were assigned to the testing
dataset, whereas the remaining data was used to train the algorithm. Once the
model had been trained, it was tested on the data from the removed day, and
its preictal probability were calculated. This process was repeated for all
the days.

Table 1:  Discrimination performances. Results indicate the mean values $\pm$ the standard deviation over the patients. They are presented for each frequency band considered independently, and the final row displays those obtained from the best combination of bands in each patient. Band | AUC Value | F1 score | Accuracy
---|---|---|---
$\delta$ | $0.387\pm 0.35$ | $0.33\pm 0.34$ | $0.56\pm 0.28$
$\theta$ | $0.32\pm 0.29$ | $0.29\pm 0.32$ | $0.50\pm 0.22$
$\alpha$ | $0.623\pm 0.26$ | $0.38\pm 0.27$ | $0.62\pm 0.09$
$\beta$ | $0.46\pm 0.40$ | $0.39\pm 0.42$ | $0.53\pm 0.33$
Low $\gamma$ | $0.6675\pm 0.31$ | $0.64\pm 0.16$ | $0.75\pm 0.16$
High $\gamma$ | $0.277\pm 0.35$ | $0.28\pm 0.35$ | $0.55\pm 0.21$
Combined | $\mathbf{0.932\pm 0.07}$ | $\mathbf{0.79\pm 0.18}$ | $\mathbf{0.85\pm 0.13}$

(a)  (b)

Figure 3:  Accuracy and F1 score results based on the minimum number of days
from each class in the training set. Data are presented in scatter plots where
each small circle represents a patient, and the red line indicates the median.
The term “Anteriority-based model” refers to the method that bases its
predictions on the immediate previous day’s state.

The discriminatory potential and forecasting performance of the model were
evaluated based on several metrics, including accuracy, recall, precision,
F1-score, and the area under the receiver operating characteristic curve
(AUC). Accuracy measures the proportion of correct predictions made by the
learning model. Recall or sensitivity quantifies the model’s ability to
identify all relevant positives. Precision, also known as positive predictive
value, assesses the model’s ability to distinguish true positives from false
positives. To take into account the class imbalance of the dataset (i.e., the
number of preictal days is lower compared to interictal epochs) we used here
the F1-score, which is a weighted average of recall and precision. F1-score
strikes a balance between minimizing false negatives and false positives,
thereby ensuring a reliable assessment of the model’s effectiveness. Finally,
the AUC, which ranges from 0.5 for a random classification to 1 for a perfect
classification, provides a comprehensive assessment of the model’s
performance.

To provide a comprehensive evaluation of our forecasting model we computed the
Brier score (i.e., the mean squared error over every forecast; from 0 [perfect
prediction] to 1 [worst prediction]) and the Brier skill score (BSS), which
measures the improvement over a default prediction (given the limited number
of recording sessions, the default probability of seizures was estimated as
the proportion of the total preictal/interictal epochs for each patient), and
take values from 0 (no improvement) to 1 (perfect forecasting), with negative
values indicating worse performances than default predictions.

##  III Results

###  III.1 Discrimination of preictal networks

We first evaluated the ability of the proposed method to discriminate between
interictal and preictal networks. Classification performances were assessed by
AUC, the accuracy, and the F1-score and are presented in Tab. 1. In general,
the connectivity patterns associated with activities of separated frequency
bands cannot distinguish between preictal and interictal days. Only networks
estimated from low $\gamma$ oscillations (30-49 Hz) performed better than a
random classifier. Nevertheless, classification performances considerably
increased when the information from different frequency bands was combined for
each patient: All patients reached AUCs of $\geqslant 0.78$ (mean AUC $=0.93$,
mean F1 score $=0.79$, and mean accuracy $=0.85$). These findings suggest that
the combination of the connectivity in different frequency bands conveys
information that can distinguish preictal from interictal epochs.

###  III.2 Forecasting of upcoming seizure(s)

The forecasting method was first assessed by examining the number of days from
each class in the training set. The results are presented in Fig. 3, which
plots the accuracy and F1 score against the minimum number of days from each
class in the training set. When only one day ($n=1$) from each class was
included in the training set, the average accuracy of prediction was $0.71$
[min. 0.25, max. 1] and the mean F1-score was $0.75$ [min. 0.25, max. 1].
However, these values increased considerably to $0.83$ [min. 0.5, max. 1] for
the accuracy and $0.87$ [min. 0.61, max. 1] for the F1-score when the model
was trained with three days from each class. When every day was predicted with
a model trained with data from all previous days, performances reached the
highest values of accuracy $0.87$ and F1-score $0.89$ (See Fig. 3). The use of
different reference networks (from previous days) in the alignement of the
embedded data yielded similar performances.

Figure 4:  Forecasting results for all patients using data from all previous
days in the training model. Probabilities that give the wrong class to the day
are highlighted in bold.

Fig. 4 presents the forecasting results for all patients when data from all
the previous days were used for training the model. Patient-specific
predictions were performed with the most discriminant combination of networks
in different frequency bands. Of the 28 preictal days (days with seizures), 22
(78.5%) were correctly predicted with a probability of $>50$%. False positive
predictions were observed in only two of 23 (8.6%) interictal periods (in two
patients), and they exclusively occurred after two consecutive days with
seizures.

The efficacy of our method was evaluated in comparison to a basic anteriority-
based model, which is a classifier that bases its predictions on the previous
day’s state. This simple model yielded an average accuracy value of $0.72$,
but this value could be overestimated as accuracy does not take into account
the class imbalance (i.e., the relatively reduced number of preictal days). In
this case, the F1-score, which combines precision and recall, provides a
better assessment of model performance. Here, the anteriority-based model
yielded a moderate F1-score of $0.67$. Further, we noticed that this value was
very close to that obtained with a non-informative (random), which yielded an
averaged F1-score of $0.68$ (for each patient, this non-informative model’s
F1-score can be estimated by $2p/(p+1)$, where $p$ is the rate of preictal
epochs).

Forecasting results were consistent with those of our previous study, in which
we demonstrated that brain connectivity obtained from vigilance-controlled
resting-state recordings could accurately predict the risk of upcoming
seizures. [14]. The results, presented in Table. 2, demonstrate the
forecasting performance obtained for the distinct frequency bands when the
prediction models for each day were trained with data from all preceding days.
Combining information from different frequency bands yielded a good predictor
with an average Brier score and BSS of $0.12$ [min. = 0.003, max. = 0.48] and
$0.36$ [min. = -0.43, max. = 0.7O], respectively. These values confirm that
this prediction model performed better than a non-informative model to
forecast the risk of upcoming seizure(s). We notice, however, that
connectivity networks from the $\beta$ and low $\gamma$ band, also provided
good predictors, in agreement with previous findings [14].

##  IV Discussion

This study provided empirical evidence supporting the hypothesis that
hyperbolic spaces are suitable for representing complex brain connectivity
patterns. This makes it a promising framework for accurately distinguishing
between interictal and preictal states. In contrast to standard seizure
prediction methods that track changes in EEG dynamics by analyzing long-term
recordings [7, 33], our study confirmed that short and daily single resting-
state recordings can reflect a pro-ictal state, at least at the daily level.

Table 2:  Forecasting performances. The given results are the averaged values over the patients $\pm$ the standard deviation. The results are presented for each band considered independently and the final row displays the result for the best combination of bands for each patient. BS : Brier Score, BSS : Brier Skill Score.  Band | BS | BSS | F1 Score | Accuracy
---|---|---|---|---
$\delta$ | $0.10\pm 0.12$ | $0.40\pm 0.44$ | $0.36\pm 0.38$ | $0.49\pm 0.31$
$\theta$ | $0.08\pm 0.05$ | $0.52\pm 0.41$ | $0.53\pm 0.35$ | $0.55\pm 0.27$
$\alpha$ | $0.08\pm 0.04$ | $0.48\pm 0.41$ | $0.44\pm 0.35$ | $0.52\pm 0.20$
$\beta$ | $0.08\pm 0.05$ | $0.44\pm 0.41$ | $0.67\pm 0.34$ | $0.65\pm 0.33$
Low $\gamma$ | $0.08\pm 0.06$ | $0.46\pm 0.45$ | $0.70\pm 0.31$ | $0.74\pm 0.22$
High $\gamma$ | $0.08\pm 0.06$ | $0.50\pm 0.40$ | $0.46\pm 0.35$ | $0.55\pm 0.35$
Combined | $0.12\pm 0.12$ | $0.36\pm 0.35$ | $0.89\pm 0.14$ | $0.87\pm 0.17$

The classification performances demonstrated a high degree of discriminating
power when the information from brain connectivity in different frequency
bands was combined for each patient. The combination of networks from
different frequency bands yielded the most accurate predictions (an accuracy
of 85% and F1-score of 79%), but it is notable that connectivity networks from
the beta and low gamma bands also demonstrated excellent forecasting results.
For all patients, the predictive performance was enhanced by increasing the
quantity of learning data. For each day, the optimal prediction was achieved
when the models were recursively trained with networks from all preceding
days. For these models, false positives were only detected in two patients,
and only occurred immediately after consecutive preictal periods. This may
indicate a pro-ictal condition, in which seizures are more likely to occur
after a day of seizures. However, homeostatic control mechanisms could prevent
their emergence.

It should be noted that other non-Euclidean embeddings (e.g. spherical or
elliptic), and that alternative dimensional reduction methods (e.g., Isomaps,
Locally Linear embeddings [28]) can also be used. Similarly, representations
in different hyperbolic spaces, such as Lorentz-Klein, $d-$dimensional
Poincaré’s ball, or the half-space model, can be used as alternatives to
Poincaré’s disk. Although other hyperbolic mappings (e.g., Mercator [25]
HyperMap [34] or Hydra [35] among others) are alternatives to projecting brain
networks into the 2D Poincaré disk, the coalescent embedding method used here
encompasses various benefits. These include a short computational time and the
absence of stochastic minimization procedures, thus preventing errors
introduced by local minima.

Despite the good classification and prediction results, our study has some
limitations. First, the vigilance-controlled resting-state iEEG recordings
were obtained from hospitalized patients who were candidates for presurgical
evaluation. Although no trivial correlations were found between the occurrence
of seizures and the controlled medication tapering [14], the results cannot be
directly generalized to out-of-hospital real-life conditions. Similarly, given
the heterogeneity of electrode location among the patients, we cannot
generalize a single unique prediction model. While a larger cohort of patients
is necessary to map out the limits of the proposed approach, the results do
suggest that the method has the potential to be a promising tool for the
analysis of brain networks.

To summarize, our findings indicate that non-Euclidean embeddings of brain
networks may offer more discriminative information for distinguishing between
interictal and preictal states, thereby providing a promising tool for
accurately forecasting the daily risk of upcoming seizure(s).

###### Acknowledgements.

M.G. acknowledges doctoral support from the Ecole Normale Supérieure de Lyon,
France.

##  Appendix A Hyperbolic Gaussian distribution

Let us consider an ensemble of points in the Poincaré’s disk. As in the case
of the Euclidean space, the distribution of the points can be characterized by
a barycenter and a covariance matrix. This information is sufficient to
approximate the distribution by a two-dimensional Gaussian distribution.

###  A.1 Hyperbolic logarithmic and exponential maps

The logarithmic map $\mathrm{Log}_{u}(v)$ provides a way to project a point
$v\in\mathbb{D}$ (in complex coordinates) into the tangent space
$T_{u}\mathbb{D}$ at the point $u$, as follows :

$\mathrm{Log}_{u}(v)=(1-|u|^{2})\mathrm{atanh}\left(\left|\frac{v-u}{1-\bar{u}v}\right|\right)e^{j\theta}$
(5)

where $\theta=\mathrm{arg}\left(\frac{v-u}{1-\bar{u}v}\right)$, and
$\mathrm{atanh}(\cdot)$ denotes to the inverse of the hyperbolic tangent
function.

The exponential map $\mathrm{Exp}_{u}(v)$ allows to do the inverse
transformation, i.e. projecting a point $v\in T_{u}\mathbb{D}$ from the
tangent space at $u$ into the disk $\mathbb{D}$ in the following way:

$\mathrm{Exp}_{u}(v)=\frac{\left(u+e^{i\theta}\right)e^{2\|v\|}+\left(u-e^{j\theta}\right)}{\left(1+\bar{u}e^{j\theta}\right)e^{2\|v\|}+\left(1-\bar{u}e^{j\theta}\right)}$
(6)

where $\theta=\mathrm{arg}(v)$ and $\|v\|=\frac{|v|}{1-|u|^{2}}$.

###  A.2  Hyperbolic barycenter computation

To estimate the barycenter of a set of points in the Poincaré disk, we used
the algorithm presented in Ref. [32]. The algorithm converges to the
barycenter of the points by means of recursive projections in the tangent
space. It iterates between the Poincaré disk and the tangent space until
convergence is achieved. The algorithm is presented in Algorithm. 1.

###  A.3 Estimation of covariance matrix in the hyperbolic disk

To estimate the covariance matrix of a set of points $z_{i}$ in the hyperbolic
disk, we first calculated their barycenter $u$ with the algorithm described
above. The points distribution is then projected into the tangent space
centered on the estimated barycenter $\hat{z}$ of the points on
$T_{u}\mathbb{D}$, using the logarithmic map $\mathrm{Log}_{\hat{u}}(\cdot)$.
In this Euclidean subspace, the barycenter is the point with coordinates
$(0,0)$ and the covariance matrix can finally be calculated as follows:
$V_{\hat{z}}=\mathrm{Covariance}\left(\left[\mathrm{Log}_{\hat{z}}(z_{i})\right]^{T}\right)$

Data:

$\quad Z=\\{z_{i},1\leq i\leq n\\}$ : complex coordinates of the points in the
Poincaré’s disk,

$z_{init}$ : Barycentre initialisation (complex coordinates),

$\quad\tau_{b}$ : step size (strictly positive),

$\quad d_{\mathrm{thres}}$ : threshold of convergence for $d$.

Result:

$\quad\hat{z}$ : numerical approximation of the barycenter (in complex
coordinates)

$\hat{z}\leftarrow z_{init}$ ;

$d\leftarrow 1\mathrm{e}6$ ;

$N\leftarrow n$ ;

while  _$d\geq d_{thres}$_ do

$\mu\leftarrow\frac{2}{n}\sum_{i=1}^{n}\mathrm{Log}_{\hat{z}}\left(z_{i}\right)$
;

$\hat{z}\leftarrow\mathrm{Exp}_{\hat{z}}\left(\tau_{b}\mu\right)$ ;

$d\leftarrow\sqrt{\frac{|\mu|^{2}}{\left(1-|\hat{z}|^{2}\right)^{2}}}$ ;

end while

Algorithm 1 Barycentre computation

## References

  * Boccaletti  _et al._ [2006] S. Boccaletti, V. Latora, Y. Moreno, M. Chavez, and D.-U. Hwang, Complex networks: Structure and dynamics, Physics Reports 424, 175 (2006).
  * Reijneveld  _et al._ [2007] J. C. Reijneveld, S. C. Ponten, H. W. Berendse, and C. J. Stam, The application of graph theoretical analysis to complex networks in the brain, Clinical Neurophysiology 118, 2317 (2007).
  * Bullmore and Sporns [2009] E. Bullmore and O. Sporns, Complex brain networks: graph theoretical analysis of structural and functional systems, Nature Reviews Neuroscience 10, 186 (2009).
  * De Vico Fallani  _et al._ [2014] F. De Vico Fallani, J. Richiardi, M. Chavez, and S. Achard, Graph analysis of functional brain networks: practical issues in translational neuroscience, Philosophical Transactions of the Royal Society B: Biological Sciences 369, 20130521 (2014).
  * Kramer and Cash [2012] M. A. Kramer and S. S. Cash, Epilepsy as a disorder of cortical network organization, The Neuroscientist 18, 360 (2012), pMID: 22235060.
  * van Diessen  _et al._ [2013] E. van Diessen, S. J. H. Diederen, K. P. J. Braun, F. E. Jansen, and C. J. Stam, Functional and structural brain networks in epilepsy: What have we learned?, Epilepsia 54, 1855 (2013).
  * Kuhlmann  _et al._ [2018] L. Kuhlmann, K. Lehnertz, M. P. Richardson, B. Schelter, and H. P. Zaveri, Seizure prediction — ready for a new era, Nature Reviews Neurology 14, 618 (2018).
  * Boguñá  _et al._ [2021] M. Boguñá, I. Bonamassa, M. De Domenico, S. Havlin, D. Krioukov, and M. Á. Serrano, Network geometry, Nature Reviews Physics 3, 114 (2021).
  * Allard and Serrano [2020] A. Allard and M. Á. Serrano, Navigable maps of structural brain networks across species, PLoS Computational Biology 16, 1 (2020).
  * Zheng  _et al._ [2020] M. Zheng, A. Allard, P. Hagmann, Y. Alemán-Gómez, and M. Á. Serrano, Geometric renormalization unravels self-similarity of the multiscale human connectome, Proceedings of the National Academy of Sciences 117, 20244 (2020).
  * Hofmann  _et al._ [2008] T. Hofmann, B. Schölkopf, and A. J. Smola, Kernel methods in machine learning, The Annals of Statistics 36, 1171 (2008).
  * Le Van Quyen  _et al._ [2005] M. Le Van Quyen, J. Soss, V. Navarro, R. Robertson, M. Chavez, M. Baulac, and J. Martinerie, Preictal state identification by synchronization changes in long-term intracranial eeg recordings, Clinical Neurophysiology 116, 559 (2005).
  * Van Mierlo  _et al._ [2014] P. Van Mierlo, M. Papadopoulou, E. Carrette, P. Boon, S. Vandenberghe, K. Vonck, and D. Marinazzo, Functional brain connectivity from eeg in epilepsy: Seizure prediction and epileptogenic focus localization, Progress in Neurobiology 121, 19 (2014).
  * Cousyn  _et al._ [2023] L. Cousyn, R. B. Messaoud, K. Lehongre, V. Frazzini, V. Lambrecq, C. Adam, B. Mathon, V. Navarro, and M. Chavez, Daily resting-state intracranial eeg connectivity for seizure risk forecasts, Epilepsia 64, e23 (2023).
  * Nickel and Kiela [2018] M. Nickel and D. Kiela, Learning continuous hierarchies in the Lorentz model of hyperbolic geometry, in  _Proceedings of the 35th International Conference on Machine Learning_, Proceedings of Machine Learning Research, Vol. 80, edited by J. Dy and A. Krause (PMLR, 2018) pp. 3779–3788.
  * Saxena  _et al._ [2020a] C. Saxena, T. Liu, and I. King, A survey of graph curvature and embedding in non-euclidean spaces, in  _Neural Information Processing_ , edited by H. Yang, K. Pasupa, A. C.-S. Leung, J. T. Kwok, J. H. Chan, and I. King (Springer International Publishing, 2020) pp. 127–139.
  * Sharpee [2019] T. O. Sharpee, An argument for hyperbolic geometry in neural circuits, Current Opinion in Neurobiology 58, 101 (2019), computational Neuroscience.
  * De Vico Fallani  _et al._ [2017] F. De Vico Fallani, V. Latora, and M. Chavez, A topological criterion for filtering information in complex brain networks, PLoS Computational Biology 13, 1 (2017).
  * Stam  _et al._ [2014] C. Stam, P. Tewarie, E. Van Dellen, E. Van Straaten, A. Hillebrand, and P. Van Mieghem, The trees and the forest: characterization of complex brain networks with minimum spanning trees, International Journal of Psychophysiology 92, 129 (2014).
  * Cacciola  _et al._ [2017] A. Cacciola, A. Muscoloni, V. Narula, A. Calamuneri, S. Nigro, E. A. Mayer, J. S. Labus, G. Anastasi, A. Quattrone, A. Quartarone, D. Milardi, and C. V. Cannistraci, Coalescent embedding in the hyperbolic space unsupervisedly discloses the hidden geometry of the brain (2017), arXiv:1705.04192 [q-bio.NC] .
  * Gao  _et al._ [2020] S. Gao, G. Mishne, and D. Scheinost, Poincaré embedding reveals edge-based functional networks of the brain, in  _Medical Image Computing and Computer Assisted Intervention – MICCAI 2020_ , edited by A. L. Martel, P. Abolmaesumi, D. Stoyanov, D. Mateus, M. A. Zuluaga, S. K. Zhou, D. Racoceanu, and L. Joskowicz (Springer International Publishing, 2020) pp. 448–457.
  * Whi  _et al._ [2022] W. Whi, S. Ha, H. Kang, and D. S. Lee, Hyperbolic disc embedding of functional human brain connectomes using resting-state fMRI, Network Neuroscience 6, 745 (2022).
  * Saxena  _et al._ [2020b] C. Saxena, T. Liu, and I. King, A survey of graph curvature and embedding in non-euclidean spaces (Springer Science and Business Media Deutschland GmbH, 2020) pp. 127–139.
  * Nickel and Kiela [2017] M. Nickel and D. Kiela, Poincaré embeddings for learning hierarchical representations, in  _Advances in Neural Information Processing Systems (NIPS 2017)_, Vol. 30, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Curran Associates, Inc., Long Beach, CA, USA, 2017) pp. 6341–6350.
  * García-Pérez  _et al._ [2019] G. García-Pérez, A. Allard, M. Á. Serrano, and M. Boguná, Mercator: uncovering faithful hyperbolic embeddings of complex networks, New Journal of Physics 21, 123033 (2019), 118.
  * Muscoloni  _et al._ [2017] A. Muscoloni, J. M. Thomas, S. Ciucci, G. Bianconi, and C. V. Cannistraci, Machine learning meets complex networks via coalescent embedding in the hyperbolic space, Nature Communications 8, 1615 (2017).
  * Longhena  _et al._ [2024] A. Longhena, M. Guillemaud, and M. Chavez, Detecting local perturbations of networks in a latent hyperbolic embedding space, Chaos: An Interdisciplinary Journal of Nonlinear Science 34, 063117 (2024), https://pubs.aip.org/aip/cha/article-pdf/doi/10.1063/5.0199546/19978086/063117_1_5.0199546.pdf .
  * von Luxburg [2007] U. von Luxburg, A tutorial on spectral clustering, Statistics and Computing 17, 395 (2007).
  * Belkin and Niyogi [2003] M. Belkin and P. Niyogi, Laplacian eigenmaps for dimensionality reduction and data representation, Neural computation 15, 1373 (2003).
  * Kitsak  _et al._ [2020] M. Kitsak, I. Voitalov, and D. Krioukov, Link prediction with hyperbolic geometry, Physical Review Research 2, 43113 (2020).
  * Gürsoy  _et al._ [2023] F. Gürsoy, M. Haddad, and C. Bothorel, Alignment and stability of embeddings: measurement and inference improvement, Neurocomputing 553, 126517 (2023).
  * Gerald  _et al._ [2020] T. Gerald, H. Zaatiti, H. Hajri, N. Baskiotis, and O. Schwander, From node embedding to community embedding: A hyperbolic approach (2020), arXiv:1907.01662 [cs.LG] .
  * Beniczky  _et al._ [2021] S. Beniczky, P. Karoly, E. Nurse, P. Ryvlin, and M. Cook, Machine learning and wearable devices of the future, Epilepsia 62, S116 (2021).
  * Papadopoulos  _et al._ [2014] F. Papadopoulos, C. Psomas, and D. Krioukov, Network mapping by replaying hyperbolic growth, IEEE/ACM Transactions on Networking 23, 198 (2014).
  * Keller-Ressel and Nargang [2020] M. Keller-Ressel and S. Nargang, Hydra: a method for strain-minimizing hyperbolic embedding of network-and distance-based data, Journal of Complex Networks 8, cnaa002 (2020).


# Generalized Correspondence Matching via Flexible Hierarchical Refinement and
Patch Descriptor Distillation

Yu Han, Ziwei Long, Yanting Zhang🖂, Jin Wu, Zhijun Fang, Rui Fan🖂 This
research was supported by the National Natural Science Foundation of China
under Grants 62233013, 62206046, and U2033218, the Shanghai Sailing Program
under Grant 21YF1401300, the Science and Technology Commission of Shanghai
Municipal under Grant 22511104500, the Fundamental Research Funds for the
Central Universities, and Xiaomi Young Talents Program. (_Y. Han and Z. Long
contributed equally to this work_) (🖂 _Corresponding authors: Y. Zhang and R.
Fan_). Y. Han, Y. Zhang, and Z. Fang are with the School of Computer Science
and Technology, Donghua University, Shanghai 201620, P. R. China (e-mails:
<EMAIL_ADDRESS>{ytzhang, zjfang}@dhu.edu.cn). J. Wu is with the
Department of Electronics and Computer Engineering, the Hong Kong University
of Science and Technology, Clear Water Bay, Hong Kong SAR, P. R. China
(e-mail: jin_wu_uestc@hotmail.com). Z. Long and R. Fan are with the Machine
Intelligence & Autonomous Systems (MIAS) Group, the College of Electronics &
Information Engineering, Shanghai Research Institute for Intelligent
Autonomous Systems, the State Key Laboratory of Intelligent Autonomous
Systems, and Frontiers Science Center for Intelligent Autonomous Systems,
Tongji University, Shanghai 201804, P. R. China (e-mails: {2053301,
rfan}@tongji.edu.cn).

###### Abstract

Correspondence matching plays a crucial role in numerous robotics
applications. In comparison to conventional hand-crafted methods and recent
data-driven approaches, there is significant interest in plug-and-play
algorithms that make full use of pre-trained backbone networks for multi-scale
feature extraction and leverage hierarchical refinement strategies to generate
matched correspondences. The primary focus of this paper is to address the
limitations of deep feature matching (DFM), a state-of-the-art (SoTA) plug-
and-play correspondence matching approach. First, we eliminate the pre-defined
threshold employed in the hierarchical refinement process of DFM by leveraging
a more flexible nearest neighbor search strategy, thereby preventing the
exclusion of repetitive yet valid matches during the early stages. Our second
technical contribution is the integration of a patch descriptor, which extends
the applicability of DFM to accommodate a wide range of backbone networks pre-
trained across diverse computer vision tasks, including image classification,
semantic segmentation, and stereo matching. Taking into account the practical
applicability of our method in real-world robotics applications, we also
propose a novel patch descriptor distillation strategy to further reduce the
computational complexity of correspondence matching. Extensive experiments
conducted on three public datasets demonstrate the superior performance of our
proposed method. Specifically, it achieves an overall performance in terms of
mean matching accuracy of 0.68, 0.92, and 0.95 with respect to the tolerances
of 1, 3, and 5 pixels, respectively, on the HPatches dataset, outperforming
all other SoTA algorithms. Our source code, demo video, and supplement are
publicly available at mias.group/GCM.

##  I Introduction

Figure 1:  Comparison between DFM and our proposed GCM on the Hpatches
dataset. Our approach produces more correspondences in the regions with low
texture or repetitive patterns.

Correspondence matching between images is crucial for a wide range of computer
vision and robotics applications, e.g., simultaneous localization and mapping
[1, 2, 3], 3D geometry reconstruction [4, 5, 6], and stereo matching [7, 8, 9,
10]. Conventional hand-crafted approaches extract keypoints using human-
designed local feature detectors and descriptors, such as the scale-invariant
feature transform (SIFT) [11] and speeded up robust features (SURF) [12].
Correspondence pairs are then determined using the nearest neighbor search
(NNS) algorithm [13]. With the recent advances in deep learning, data-driven
approaches [14, 15, 16, 17, 18, 19, 20] have demonstrated compelling results.

Conventional hand-crafted approaches generally leverage a sequential pipeline
for keypoint detection, description, and matching [21, 22]. Their overall
performance is often determined by the weakest component within this pipeline,
akin to the “barrel effect” principle. Moreover, errors in the earlier stages
can accumulate and propagate to the later stages, making it tricky to improve
the overall performance [23]. While data-driven approaches have significantly
outperformed hand-crafted methods, detector-based methods [14, 15, 16, 17] may
still struggle in texture-less regions, and detector-free approaches could
face information loss due to manually selected scales [18, 19, 20].
Additionally, most data-driven approaches require a large amount of well-
annotated data for model training, often resulting in unsatisfactory
performance when applied to new scenarios [24].

To address these limitations, deep feature matching (DFM) [25], a plug-and-
play approach built upon a hierarchical matching refinement paradigm is
proposed. DFM utilizes a VGG [26] model pre-trained on the ImageNet [27]
database to extract multi-scale features, with no need for additional training
with well-annotated data. Furthermore, DFM leverages a coarse-to-fine strategy
in conjunction with the NNS algorithm to perform hierarchical feature matching
from the deepest layers to the shallowest ones. DFM significantly improves
accuracy and robustness compared to conventional hand-crafted methods,
outperforming even some approaches trained with correspondences [24].

Nevertheless, DFM has three significant limitations. The most fatal drawback
is its high demand on the backbone network, limiting compatibility to those
capable of providing feature maps with the same size as the input image, such
as VGG. Another drawback is the lack of extensive experimental evaluation of
DFM in various computer vision and robotics tasks, except for image
classification. This raises questions about its performance when using
different backbone networks, especially those trained for dense correspondence
matching, such as [28, 29, 30]. Finally, the hierarchical refinement strategy
employed in DFM has the potential to propagate errors from deeper layers to
shallower ones, resulting in lower density and quality of correspondence
matching, as shown in Fig. 1.

To address the challenges mentioned above, this paper introduces Generalized
Correspondence Matching (GCM) based on flexible hierarchical refinement and
patch descriptor distillation. First, we omit the pre-defined threshold used
in the hierarchical refinement process of DFM by leveraging a more flexible
NNS strategy, thereby preventing the exclusion of repetitive yet valid matches
in early stages. Furthermore, we expand the applicability of DFM to
accommodate various types of backbones, pre-trained across diverse computer
vision tasks, including image classification [26, 31, 32, 33, 34, 35],
semantic segmentation [36, 37, 38, 39], and stereo matching [28, 30, 29]. This
is accomplished by incorporating a patch descriptor to function as the
highest-resolution feature maps (with the same resolution as the input image)
in DFM. Additionally, we propose a novel strategy for patch descriptor
distillation, which further enhances the overall efficiency of correspondence
matching. What surprises us is that several backbones demonstrate improved
performance when the patch descriptor is distilled. Extensive experiments
conducted on the HPatches dataset[40] demonstrate the superior mean matching
accuracy (MMA) achieved by GCM. Moreover, as illustrated in Fig. 1, GCM is
capable of producing denser and more accurate matched correspondences,
particularly in repetitive or low-texture areas when compared to DFM.

Figure 2: The architecture of our proposed GCM.

##  II Related Work

###  II-A Correspondence Matching

Early approaches, as exemplified by SIFT [11], typically employ hand-crafted
local visual features, e.g., gradients, angles, and blobs, to detect
distinctive interest points and generate descriptors. Correspondence pairs are
subsequently determined via NNS [24]. However, these algorithms have limited
adaptability to diverse and complex datasets owing to their reliance on
manually designed features [16]. Moreover, they tend to be highly sensitive to
variations in lighting, scale, and viewpoint, which constrains their
robustness in real-world scenarios [14]. In addition, they often require
significant domain expertise for feature design and may struggle to generalize
effectively across different tasks or domains.

Over the last half-decade, data-driven methods [16, 15, 17] have demonstrated
superior performance compared to conventional hand-crafted approaches. Among
them, detector-based algorithms have remained the dominant choice for
correspondence matching. As an example, SuperPoint [14] leverages homographic
adaptation in conjunction with MagicPoint [41] to enhance detector performance
and generate pseudo ground-truth interest points for unlabeled images in a
self-supervised fashion. Another data-driven approach, repeatable and reliable
detector and descriptor (R2D2) [16], is built upon a trainable convolutional
neural network (CNN). R2D2 enhances the descriptor quality by placing a strong
emphasis on the reliability of feature points. This is achieved in conjunction
with a trainable CNN designed for both feature description and detection,
known as D2-Net [15]. Such improvements greatly advance R2D2’s feature
description capabilities. In contrast, DFM [25], which relies on VGG-19 model,
does not offer such precision in the description. Additionally, SuperGlue [17]
leverages a graph neural network (GNN) equipped with an attention mechanism
and a differentiable Sinkhorn algorithm [42] to compute matches between two
sets of detected features and their corresponding descriptor vectors.
SuperGlue achieves notably superior performance when compared to the NNS
methods.

Other approaches, such as neighborhood consensus networks (NCNet) designed for
image correspondence estimation [18] forgo the feature detection phase and
instead directly match points distributed across a dense grid rather than
sparse locations. NCNet constructs a 4D cost volume with neighborhood
consensus to enumerate all possible matches between images. In contrast,
efficient neighborhood consensus networks via submanifold sparse convolutions
(SparseNCNet) [19] employs a more condensed form of the correlation tensor,
storing a subset and substituting the dense 4D convolution with a sparser
convolution technique to improve correspondence matching efficiency. Epipolar-
guided pixel-level correspondences (Patch2pix) [43] utilizes a pre-trained
backbone to extract patch-level matches and employs a two-stage regressor to
refine these matches to pixel-level precision. On the other hand, detector-
free local feature matching with Transformers (LoFTR) [20] achieves a high
level of robustness but can be slower due to the large number of processes it
involves. Although efficiency can be improved by reducing the resolution of
the input image, this may affect matching accuracy to some extent. In this
paper, our GCM is built upon the foundation of DFM. Our primary focus is to
tackle the limitations of DFM, particularly the high demand for feature maps
and the necessity of a predefined threshold for hierarchical refinement.

###  II-B Knowledge Distillation

Knowledge distillation is a commonly used technique for model compression,
initially introduced for image classification. Unlike pruning and quantization
techniques [44] used in model compression, knowledge distillation techniques
focus on training a smaller, more lightweight model [45]. This is achieved by
exploiting supervised information from larger, high-performance models,
ultimately reducing both time and space complexity. Due to its success across
various tasks, knowledge distillation is regarded as an effective multitasking
approach, applicable to classification [46], semantic segmentation [47], and
object detection [48, 49]. In this paper, we utilize the knowledge
distillation technique to further reduce the complexity of the patch
descriptor.

##  III Methodology

###  III-A Architecture Overview

The architecture of our proposed GCM is shown in Fig. 2. GCM follows a two-
step correspondence matching strategy, similar to DFM [25]. Initially, we
employ a pre-trained backbone network as the deep feature extractor to obtain
the feature maps $\mathcal{F}=\\{\bm{F}_{1},...,\bm{F}_{k}\\}$. Here,
$\boldsymbol{F}_{l}\in\mathbb{R}^{\frac{H}{2^{l}}\times\frac{W}{2^{l}}\times
C_{l}}$ represents the $l$-th layer of feature maps, with $H$ and $W$ denoting
the height and width of the input RGB image $\bm{I}\in\mathbb{R}^{H\times
W\times 3}$, respectively. Next, we perform NNS on the last layers of feature
maps, denoted as $\bm{F}_{k}^{A}$ and $\bm{F}_{k}^{B}$, to obtain coarse
matches $\mathcal{M}_{k}^{A,B}$. Here, the superscripts $A$ and $B$ represent
two images of $\bm{I}^{A}$ and $\bm{I}^{B}$. Based on these matches, we
estimate the homography matrix $\bm{H}$, which is further utilized to generate
a warped image $\bm{I}^{C}$ from $\bm{I}^{B}$.

In the second step, we perform feature extraction followed by a flexible
hierarchical refinement (FHR) module, which will be detailed in Section III-D,
to obtain the matched correspondences. In specific, we construct hierarchical
feature map layers
$\\{\bm{F}_{1}^{A},\bm{F}_{2}^{A},...,\bm{F}_{k}^{A},\bm{F}_{0}^{A}\\}$ for
$\bm{I}^{A}$, and
$\\{\bm{F}_{1}^{B},\bm{F}_{2}^{B},...,\bm{F}_{k}^{B},\bm{F}_{0}^{B}\\}$ for
$\bm{I}^{B}$. Note that, $\bm{F}_{0}\in\mathbb{R}^{H\times W\times C_{0}}$ is
obtained through a patch descriptor described in Section III-C, which provides
feature maps with the same size as the original images. The FHR module
operates in a coarse-to-fine manner, processing from the deepest layer
$\bm{F}_{k}$ to shallow layers down to $\bm{F}_{1}$, and finally $\bm{F}_{0}$,
where we obtain the final matches between $\bm{I}^{A}$ and the warped image
$\bm{I}^{C}$. Later, taking advantage of the estimated $\bm{H}$ in the first
step, we can trace back to the matched correspondences between $\bm{I}^{A}$
and the original image of $\bm{I}^{B}$.

Our method generalizes the baseline DFM approach and is compatible with
various types of backbone networks. These networks can be pre-trained for a
diverse range of computer vision tasks, regardless of whether they can produce
feature maps with the same resolution as the original images.

###  III-B Nearest Neighbor Search

Since the dense NNS utilized in DFM is a local matching approach that
necessitates a manually defined threshold and tends to discard correct matches
in cases of repetitiveness, we adopt a flexible, parameter-free NNS strategy.

Given the feature maps $\bm{F}^{A}$ and $\bm{F}^{B}$ extracted from images
$\bm{I}^{A}$ and $\bm{I}^{B}$, we identify potential matches by determining
the nearest neighbors based on the feature distance between $p^{A}$ and
$p^{B}$ (with $\bm{f}^{A}$ and $\bm{f}^{B}$ denoting the features at the point
$p^{A}$ and $p^{B}$ in $\bm{F}^{A}$ and $\bm{F}^{B}$, respectively):

$d(p^{A},p^{B})=1-\phi(\bm{f}^{A},\bm{f}^{B}),$ (1)

where $\phi(\cdot,\cdot)$ represents the cosine similarity as follows:

$\phi(\bm{f}^{A},\bm{f}^{B})=\frac{\bm{f}^{A}\cdot\bm{f}^{B}}{\left\|\bm{f}^{A}\right\|_{2}\left\|\bm{f}^{B}\right\|_{2}}.$
(2)

For a specific point $p^{A}$ within the feature map $\bm{F}^{A}$, it is
matched to $p^{B}$ if the distance to $p^{B}$ is minimal. A match
$(p^{A},p^{B})$ is confirmed only if it is mutual, meaning that $p^{A}$ and
$p^{B}$ are recognized as a matched pair only if $p^{B}$ is also matched with
$p^{A}$.

###  III-C Patch Descriptor Distillation

To reduce time and space complexity, we develop an additional distilled patch
descriptor as a more lightweight feature description network alternative.
Given that descriptors of R2D2 [16] are sourced from an L2-normalized feature
map, our strategy is to train the final layer of the student model’s backbone
using the final feature layer of the teacher model’s backbone. As shown in
Fig. 4, the student model reduces the number of intermediate layers and
directly assimilates the final feature map from the teacher model.
$\boldsymbol{X}^{T}\in\mathbb{R}^{H\times W\times 128}$ denotes the output of
the last layer of the teacher model backbone network, and
$\boldsymbol{X}^{S}\in\mathbb{R}^{H\times W\times 128}$ denotes the last layer
output of the student model backbone network. The loss function is defined as
follows:

$\mathcal{L}=\frac{1}{|\mathcal{P}|}\sum_{(i,j)\in\mathcal{P}}(1-|\phi(\boldsymbol{X}^{T}_{ij},\boldsymbol{X}^{S}_{ij})|),$
(3)

where the point set $\mathcal{P}$ contains all points in the image,
represented by coordinates $(i,j)$ for each pixel. The function $\phi$ is
defined in Equation (2). This lightweight patch descriptor enables the faster
generation of $\boldsymbol{F}_{0}\in\mathbb{R}^{H\times W\times 128}$ for
facilitating flexible hierarchical refinement.

###  III-D Flexible Hierarchical Refinement

The flexible hierarchical refinement (FHR) module consists of multiple
iterations, with a single iteration illustrated in Fig. 3. Given the matched
features $\mathcal{M}_{l}^{A,C}$ in the $l$-th layer, where every match
$(p_{l}^{A},p_{l}^{C})\in\mathcal{M}_{l}^{A,C}$ is associated with feature
$(\bm{f}_{l}^{A},\bm{f}_{l}^{C})$, this module returns matched correspondences
$\mathcal{M}_{l-1}^{A,C}$ in the $(l-1)$-th layer. During this step, it allows
the matched features in $\bm{F}_{l}^{A}$ and $\bm{F}_{l}^{C}$ to identify the
corresponding sub-region pairs in $\bm{F}_{l-1}^{A}$ and $\bm{F}_{l-1}^{C}$.
In specific, every point in $\bm{F}_{l}$ is mapped to $\bm{F}_{l-1}$ and
corresponds to a $2\times 2$ patch on $\bm{F}_{l-1}$. By applying NNS to these
sub-regions, matched correspondences $\mathcal{M}_{l-1}^{A,C}$ between
$\bm{F}_{l-1}^{A}$ and $\bm{F}_{l-1}^{C}$ can be obtained.

Figure 3: An illustration of one iteration within the hierarchical refinement
module.

There are $(k+1)$ iterations in total, consistent with the number of
hierarchical feature layers from $\bm{F}_{k}$ to $\bm{F}_{0}$. As for the
initial input of FHR, we employ the NNS on the $k$-th layer’s $\bm{F}_{k}$
from the two images to obtain an initial correspondence. Subsequently, we map
these points to $\bm{F}_{k-1}$ and establish paired sub-regions between
$\bm{I}^{A}$ and $\bm{I}^{C}$. Using this correspondence, we employ the NNS to
establish refined correspondences within each group of patches. This process
is iteratively applied until the first layer of $\bm{F}_{1}$ is reached,
resulting in correspondences at half size of the input images. Finally, we
project the matched points onto the descriptor layer $\bm{F}_{0}$ and perform
correspondence matching at this layer using NNS between every paired patch
associated with $\bm{I}^{A}$ and $\bm{I}^{C}$. Due to potential accumulated
errors from the previous stages, we introduce a ratio test as in [15, 11] at
the final iteration to filter out inferior matches.

It is worth noting that this hierarchical refinement paradigm is adaptable to
a wide range of backbone networks.

Figure 4: Our proposed patch descriptor distillation strategy. We employ the
R2D2 backbone as the teacher descriptor to generate a more lightweight student
descriptor. BN refers to batch normalization.

##  IV EXPERIMENTS

###  IV-A Datasets and Evaluation Protocol

Here is a summary of the datasets and evaluation setups used in our
experiments:

① HPatches: We conduct experiments on the HPatches dataset, utilizing mean
matching accuracy (MMA) and homography estimation accuracy as metrics. For
both image matching and homography estimation, we follow the evaluation setup
detailed in [25].

② MegaDepth: We evaluate the outdoor pose estimation accuracy of our proposed
method on the MegaDepth dataset [50]. We quantify the pose error by computing
the area under the receiver operating characteristic (AUC), following the
setup detailed in [51] and [20]. Additionally, we also compute match precision
following the setup in [17].

③ ScanNet: We evaluate the indoor pose estimation accuracy on the ScanNet
dataset [52]. Please refer to [17] for detailed setups.

TABLE I: MMA comparison on the HPatches dataset. “$r$” denotes ratio test threshold. Category |  Method | Overall | Illumination | Viewpoint | Matches
---|---|---|---|---|---
@1px | @3px | @5px | @1px | @3px | @5px | @1px | @3px | @5px
Hand-Crafted |  SIFT+NNS[11] | 0.35 | 0.50 | 0.54 | 0.37 | 0.49 | 0.52 | 0.33 | 0.52 | 0.55 | 0.4K
Fully Supervised |  R2D2+NNS[16] | 0.33 | 0.76 | 0.84 | 0.38 | 0.81 | 0.90 | 0.29 | 0.71 | 0.79 | 1.6k
|  LoFTR[20] | 0.63 | 0.91 | 0.93 | 0.68 | 0.95 | 0.96 | 0.59 | 0.86 | 0.91 | 2.6K
Plug-and-Play |  DFM ($r$=0.90) [25] | 0.51 | 0.85 | 0.93 | 0.63 | 0.91 | 0.97 | 0.42 | 0.81 | 0.89 | 7.0K
|  DFM ($r$=0.60) [25] | 0.61 | 0.88 | 0.94 | 0.77 | 0.93 | 0.98 | 0.47 | 0.84 | 0.90 | 1.0K
|  Ours ($r$=0.95) | 0.53 | 0.85 | 0.92 | 0.56 | 0.87 | 0.94 | 0.50 | 0.84 | 0.90 | 14.4K
|  Ours ($r$=0.60) | 0.68 | 0.92 | 0.95 | 0.74 | 0.93 | 0.97 | 0.63 | 0.91 | 0.94 | 3.1K
TABLE II: MMA comparison on the HPatches dataset among backbone networks pre-
trained for a variety of computer vision tasks. “$r$” denotes ratio test
threshold. “$\ast$” denotes the backbone used in R2D2 [16].

Task |  Method | Overall | Illumination | Viewpoint | Matches
---|---|---|---|---|---
@1px | @3px | @5px | @1px | @3px | @5px | @1px | @3px | @5px
Image Classification |  ResNet18 [31] | 0.53 | 0.85 | 0.92 | 0.56 | 0.87 | 0.94 | 0.50 | 0.84 | 0.90 | 14.4K
|  ResNet18 ($r$=0.6) [31] | 0.68 | 0.92 | 0.95 | 0.74 | 0.93 | 0.97 | 0.63 | 0.91 | 0.94 | 3.1K
|  ResNet18* ($r$=0.6) [16] | 0.69 | 0.91 | 0.95 | 0.74 | 0.91 | 0.96 | 0.64 | 0.92 | 0.94 | 2.0K
|  VGG19 [26] | 0.56 | 0.86 | 0.92 | 0.60 | 0.89 | 0.95 | 0.51 | 0.84 | 0.89 | 13.3K
|  ResNet50 [31] | 0.52 | 0.85 | 0.92 | 0.57 | 0.87 | 0.94 | 0.47 | 0.83 | 0.90 | 13.7K
|  ResNeXt50 [32] | 0.50 | 0.85 | 0.92 | 0.55 | 0.88 | 0.94 | 0.46 | 0.83 | 0.90 | 12.4K
|  MobileV3 [33] | 0.50 | 0.82 | 0.89 | 0.55 | 0.84 | 0.90 | 0.45 | 0.81 | 0.88 | 12.8K
|  EfficientNetV2 [34] | 0.50 | 0.84 | 0.92 | 0.56 | 0.87 | 0.93 | 0.44 | 0.82 | 0.90 | 13.9K
Semantic  Segmentation |  FCN[38] | 0.46 | 0.78 | 0.85 | 0.49 | 0.78 | 0.85 | 0.44 | 0.77 | 0.85 | 10.9K
|  DeepLabV3 [37] | 0.37 | 0.62 | 0.70 | 0.47 | 0.76 | 0.84 | 0.26 | 0.50 | 0.57 | 4.4K
|  Swin Tranformer[39] | 0.53 | 0.80 | 0.88 | 0.60 | 0.81 | 0.89 | 0.46 | 0.79 | 0.87 | 4.7K
Stereo Matching |  CREStereo[29] | 0.51 | 0.73 | 0.75 | 0.57 | 0.78 | 0.81 | 0.46 | 0.68 | 0.70 | 23.3K
|  PSMNet[30] | 0.31 | 0.51 | 0.56 | 0.38 | 0.58 | 0.63 | 0.24 | 0.44 | 0.49 | 4.8K
|  DeepPruner[28] | 0.31 | 0.50 | 0.55 | 0.37 | 0.57 | 0.62 | 0.24 | 0.44 | 0.48 | 5.2K
- |  Average Pooling | 0.60 | 0.85 | 0.88 | 0.66 | 0.88 | 0.92 | 0.53 | 0.81 | 0.84 | 17.4K

TABLE III: Homography estimation results on the HPatches dataset. Method | Overall | Illumination | Viewpoint
---|---|---|---
@1px | @3px | @5px | @1px | @3px | @5px | @1px | @3px | @5px
SIFT[11]+NNS | 0.42 | 0.74 | 0.85 | 0.54 | 0.86 | 0.93 | 0.30 | 0.54 | 0.71
SuperPoint [14]+NNS | 0.46 | 0.78 | 0.85 | 0.57 | 0.92 | 0.97 | 0.35 | 0.65 | 0.74
D2Net [15]+NNS | 0.38 | 0.71 | 0.82 | 0.66 | 0.95 | 0.98 | 0.12 | 0.49 | 0.67
R2D2 [16]+NNS | 0.47 | 0.77 | 0.82 | 0.63 | 0.93 | 0.98 | 0.32 | 0.64 | 0.70
SuperPoint [14]+SuperGlue [17] | 0.51 | 0.82 | 0.89 | 0.60 | 0.92 | 0.98 | 0.42 | 0.71 | 0.81
SuperPoint [14]+CAPS [53] | 0.49 | 0.79 | 0.86 | 0.62 | 0.93 | 0.98 | 0.36 | 0.65 | 0.75
SuperPoint [14]+ClusterGNN [54] | 0.52 | 0.84 | 0.90 | 0.61 | 0.93 | 0.98 | 0.44 | 0.74 | 0.81
SIFT+CAPS [53] | 0.36 | 0.77 | 0.85 | 0.48 | 0.89 | 0.95 | 0.26 | 0.65 | 0.76
Patch2Pix [43] | 0.50 | 0.79 | 0.87 | 0.71 | 0.95 | 0.98 | 0.30 | 0.64 | 0.76
MatchFormer | 0.55 | 0.81 | 0.87 | 0.75 | 0.95 | 0.98 | 0.37 | 0.68 | 0.78
LoFTR | 0.63 | 0.91 | 0.93 | 0.68 | 0.95 | 0.96 | 0.59 | 0.86 | 0.91
DFM [25] | 0.41 | 0.74 | 0.85 | 0.63 | 0.91 | 0.97 | 0.21 | 0.59 | 0.74
Ours | 0.55 | 0.84 | 0.90 | 0.73 | 0.95 | 0.98 | 0.38 | 0.73 | 0.81

###  IV-B Implementation Details

In our experiments, we employ ResNet18, pre-trained on the ImageNet dataset,
as the default deep feature extractor. For the feature description, we use the
patch descriptor from the distilled R2D2. We only incorporate a ratio test at
the descriptor layer to correct errors from previous stages. Similar to the
setup used in [25], two ratio tests (thresholds: 0.95 and 0.60) are applied.
Our model is trained on the same datasets used to train R2D2. During the
training process, we utilize the Adam optimizer with an initial learning rate
of $1\times 10^{-4}$ and a batch size of 2. The training process converges
after 10 epochs on an NVIDIA GTX1650 GPU. The student model contains 292K
parameters, compared to the teacher model, which has 486K parameters. This
reduction in parameters demonstrates the efficiency gains achieved through our
patch descriptor distillation strategy.

We incorporate a patch descriptor that allows the proposed method to be
compatible with the most popular backbone networks. It is worth noting that
our method is not directly compatible with the Swin Transformer architecture
[39], which initially applies a $4\times$ downsampling, resulting in a feature
map with a maximum resolution equivalent to only a quarter of the input image.
To ensure compatibility with Swin Transformer, we up-sample the points
directly to match the required resolution.

###  IV-C Image Matching Performance Evaluation

Table I demonstrates that our proposed GCM outperforms all other existing
algorithms in terms of overall and viewpoint MMA on the HPatches dataset. It
achieves results comparable to LoFTR and better results than R2D2.

Furthermore, the results presented in Table IV-A confirm the compatibility of
our algorithm with various backbone networks that are pre-trained for a wide
range of computer vision tasks, including image classification, semantic
segmentation, and stereo matching. Additionally, we propose an alternative
approach that does not rely on a visual backbone network, by substituting the
feature maps in the hierarchical refinement strategy with descriptors average
pooling to different scales. This method is represented as Average Pooling in
Table. IV-A. The unsatisfactory results achieved with stereo-matching backbone
networks are somewhat unexpected, and it is possible that these networks,
primarily designed for 1D dense search, may not be well-suited for solving 2D
search problems that are more relevant to correspondence matching tasks. This
analysis provides valuable insights into the performance variation observed
with different types of backbone networks and highlights the need for
specialized feature extraction and matching strategies in correspondence
matching applications. Considering the speed advantage of the ResNet18 model,
we choose ResNet18 as the backbone for experiments. It is also worth noting
that the difference in performance between using the original R2D2 backbone
and a lightweight distilled R2D2 backbone is minimal. This demonstrates the
effectiveness of the distillation strategy in reducing computational
complexity without sacrificing performance.

TABLE IV: Pose estimation performance evaluation on the MegaDepth dataset. These results denote the percentages of correctly estimated poses with pose errors below 5/10/20∘, respectively.  Method | Pose Estimation AUC | Precision
---|---|---
@5∘ | @10∘ | @20∘
SIFT+NNS | 16.70 | 28.15 | 41.84 | 35.53
D2-Net | 4.44 | 8.27 | 14.17 | 48.58
SuperPoint+NNS | 29.01 | 44.74 | 58.92 | 57.43
R2D2+NNS | 41.15 | 58.88 | 72.84 | 81.51
SuperPoint+SuperGlue | 46.10 | 63.82 | 77.68 | 99.66
Patch2Pix | 39.70 | 55.06 | 67.77 | 80.17
LoFTR | 52.42 | 69.26 | 81.41 | 96.78
DFM | 25.91 | 41.70 | 56.19 | 91.92
Ours | 34.61 | 52.24 | 67.10 | 87.41

###  IV-D Homography Estimation Performance Evaluation

In Table III, the accuracy metrics reported for overall, illumination, and
viewpoint matching at various thresholds provide a comprehensive performance
evaluation of homography estimation for both our method and the compared
algorithms. Notably, while GCM achieves the second-best performance across all
these categories, it outperforms the majority of data-driven approaches that
are trained using fully supervised correspondence ground truth.

TABLE V: Pose estimation performance evaluation on the ScanNet dataset. “$\dagger$” denotes the models trained on the outdoor datasets and evaluated on the ScanNet dataset (indoor).  Method | Pose Estimation AUC | Precision
---|---|---
@5∘ | @10∘ | @20∘
SIFT+NNS | 5.83 | 13.06 | 22.47 | 40.30
SuperPoint+NNS | 9.43 | 21.53 | 36.40 | 50.40
Patch2Pix† | 9.57 | 21.22 | 34.56 | 50.59
R2D2† | 6.82 | 16.37 | 28.02 | 46.51
SuperPoint+SuperGlue | 16.24 | 34.13 | 52.88 | 84.33
SuperPoint+SuperGlue† | 15.68 | 32.66 | 49.87 | 80.45
LoFTR | 20.27 | 39.63 | 57.47 | 83.92
LoFTR† | 17.57 | 34.46 | 51.88 | 70.16
DFM | 3.73 | 9.76 | 18.94 | 74.77
Ours† | 11.03 | 23.73 | 38.93 | 67.63

###  IV-E Pose Estimation Performance Evaluation

Tables IV and V provide the performance evaluations for pose estimation in
outdoor and indoor environments, respectively. The MegaDepth dataset presents
a significant challenge due to the necessity of matching under extreme
viewpoint changes and addressing issues related to high texture repetition.
Despite utilizing a non-specialized network for deep feature extraction, our
method consistently demonstrates impressive performance on demanding test
data. While it may not reach the state-of-the-art performance level achieved
by fully supervised methods in pose estimation, our approach represents a
notable improvement over the baseline plug-and-play algorithm DFM.
Additionally, the experimental results on the ScanNet dataset demonstrate that
our proposed GCM achieves performance levels comparable to fully supervised
methods and outperforms the baseline algorithm DFM across most threshold
values.

###  IV-F Abaltion Studies

TABLE VI:  Ablation Study of each component on Hpatches dataset.  FHR | Descriptor | Distillation | Homography Estimation Accuracy
---|---|---|---
@1px |  @3px |  @5px
|  |  |  0.37 |  0.68 |  0.80
| ✓ |  |  0.41 |  0.72 |  0.82
| ✓ | ✓ |  0.42 |  0.72 |  0.83
✓ |  |  |  0.38 |  0.71 |  0.83
✓ | ✓ |  |  0.45 |  0.75 |  0.84
✓ | ✓ | ✓ |  0.44 |  0.76 |  0.85

We conduct an ablation study to determine the individual and combined effects
of FHR, patch descriptor, and descriptor distillation on the homography
estimation accuracy across the overall Hpatches dataset, with the setup as
referenced in Sec. IV-C. The results are presented in Table VI. VGG19 is
utilized as the backbone network and the baseline model (the first row) is
designated as DFM, with random sample consensus (RANSAC) for homography matrix
estimation to ensure experimental consistency. Our key insights:

  * •

Patch descriptors play a crucial role in improving accuracy, in the
configurations where it is employed (the second and fifth rows). These results
underscore its essential contribution to baseline performance enhancement and
integration with FHR for further improvements.

  * •

Descriptor distillation significantly refines patch descriptor quality,
leading to more accurate homography estimation. This is evident in the
configurations incorporating descriptor distillation (the third and sixth
rows).

  * •

FHR contributes to a notable increase in accuracy (from the fourth to the
sixth rows). When applied in combination with patch descriptor and
distillation, FHR significantly elevates the model’s overall performance.

##  V Conclusion

In summary, this paper introduced three significant technical contributions to
address the limitations present in DFM: (1) a flexible hierarchical refinement
strategy that eliminates the need for a pre-defined threshold, initially
utilized in the hierarchical refinement process of DFM; (2) the incorporation
of a patch descriptor that extends the applicability of DFM to accommodate a
wide range of backbone networks, pre-trained across diverse robot perception
tasks, such as semantic segmentation and stereo matching; (3) a novel patch
descriptor distillation strategy, which further reduces the computational
complexity of correspondence matching and enhances the practical applicability
of our method in real-world robotics applications. Through performance
evaluations including image matching, homography estimation, and pose
estimation, the effectiveness of our proposed algorithm is validated.
Particularly noteworthy is that our method achieves state-of-the-art overall
mean matching accuracy, outperforming both conventional hand-crafted
approaches and data-driven methods trained via fully supervised learning. We
are confident that our method can be readily integrated into a variety of
downstream real-world robotics applications.

## References

  * [1] M. U. M. Bhutta et al., “Loop-box: Multiagent direct slam triggered by single loop closure for large-scale mapping,” IEEE Transactions on Cybernetics, vol. 52, no. 6, pp. 5088–5097, 2020.
  * [2] R. Mur-Artal et al., “ORB-SLAM: A versatile and accurate monocular slam system,” IEEE Transactions on Robotics, vol. 31, no. 5, pp. 1147–1163, 2015.
  * [3] M. U. M. Bhutta and M. Liu, “Pcr-pro: 3d sparse and different scale point clouds registration and robust estimation of information matrix for pose graph slam,” in 2018 IEEE 8th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER), pp. 354–359, IEEE, 2018.
  * [4] R. Fan et al., “Road surface 3D reconstruction based on dense subpixel disparity map estimation,” IEEE Transactions on Image Processing, vol. 27, no. 6, pp. 3025–3035, 2018.
  * [5] J. L. Schonberger et al., “Structure-from-motion revisited,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [6] R. Fan et al., “Rethinking road surface 3-D reconstruction and pothole detection: From perspective transformation to disparity map segmentation,” IEEE Transactions on Cybernetics, vol. 52, no. 7, pp. 5799–5808, 2022.
  * [7] Z. Wu et al., “S3M-Net: Joint learning of semantic segmentation and stereo matching for autonomous driving,” IEEE Transactions on Intelligent Vehicles, 2024.  DOI: 10.1109/TIV.2024.3357056.
  * [8] H. Zhao et al., “Dive deeper into rectifying homography for stereo camera online self-calibration,” in 2024 International Conference on Robotics and Automation (ICRA), 2024.  in press.
  * [9] H. Wang et al., “PVStereo: Pyramid voting module for end-to-end self-supervised stereo matching,” IEEE Robotics and Automation Letters, vol. 6, no. 3, pp. 4353–4360, 2021.
  * [10] Z. Wu et al., “SG-RoadSeg: End-to-end collision-free space detection sharing encoder representations jointly learned via unsupervised deep stereo,” in 2024 International Conference on Robotics and Automation (ICRA), 2024.  in press.
  * [11] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” International Journal of Computer Vision, vol. 60, pp. 91–110, 2004.
  * [12] H. Bay et al., “SURF: Speeded up robust features,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 404–417, 2006.
  * [13] M. Muja et al., “Scalable nearest neighbor algorithms for high dimensional data,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 36, no. 11, pp. 2227–2240, 2014.
  * [14] D. DeTone et al., “SuperPoint: Self-supervised interest point detection and description,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 224–236, 2018\.
  * [15] M. Dusmanu et al., “D2-Net: A trainable cnn for joint description and detection of local features,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8092–8101, 2019.
  * [16] J. Revaud et al., “R2D2: Reliable and repeatable detector and descriptor,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 32, 2019.
  * [17] P.-E. Sarlin et al., “SuperGlue: Learning feature matching with graph neural networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4938–4947, 2020.
  * [18] I. Rocco et al., “NCNet: Neighbourhood consensus networks for estimating image correspondences,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 2, pp. 1020–1034, 2020.
  * [19] I. Rocco et al., “Efficient neighbourhood consensus networks via submanifold sparse convolutions,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 605–621, 2020.
  * [20] J. Sun et al., “LoFTR: Detector-free local feature matching with transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8922–8931, 2021.
  * [21] M. U. M. Bhutta, Towards a Swift Multiagent Slam System for Large-Scale Robotics Applications.  Hong Kong University of Science and Technology (Hong Kong), 2021.
  * [22] J. Ma et al., “Image matching from handcrafted to deep features: A survey,” International Journal of Computer Vision, vol. 129, pp. 23–79, 2021.
  * [23] U. Efe et al., “Effect of parameter optimization on classical and learning-based image matching methods,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2506–2513, 2021.
  * [24] C. Zhou et al., “E3CM: Epipolar-constrained cascade correspondence matching,” Neurocomputing, vol. 559, p. 126788, 2023.
  * [25] U. Efe et al., “DFM: A performance baseline for deep feature matching,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4284–4293, 2021.
  * [26] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in 3rd International Conference on Learning Representations (ICLR 2015), Computational and Biological Learning Society, 2015.
  * [27] A. Krizhevsky et al., “ImageNet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 25, Curran Associates, Inc., 2012.
  * [28] S. Duggal et al., “DeepPruner: Learning efficient stereo matching via differentiable patchmatch,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4384–4393, 2019.
  * [29] J. Li et al., “Practical stereo matching via cascaded recurrent network with adaptive correlation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16263–16272, 2022.
  * [30] J.-R. Chang and Y.-S. Chen, “Pyramid stereo matching network,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5410–5418, 2018.
  * [31] K. He et al., “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2016.
  * [32] S. Xie et al., “Aggregated residual transformations for deep neural networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1492–1500, 2017.
  * [33] A. Howard et al., “Searching for MobileNetV3,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1314–1324, 2019.
  * [34] M. Tan and Q. Le, “EfficientNetV2: Smaller models and faster training,” in International Conference on Machine Learning (ICML), pp. 10096–10106, PMLR, 2021.
  * [35] N. Ma et al., “ShuffleNet V2: Practical guidelines for efficient cnn architecture design,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 116–131, 2018.
  * [36] O. Ronneberger et al., “U-Net: Convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 234–241, Springer, 2015.
  * [37] L.-C. Chen et al., “Rethinking atrous convolution for semantic image segmentation,” arXiv preprint arXiv:1706.05587, 2017.
  * [38] J. Long et al., “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3431–3440, 2015.
  * [39] Z. Liu et al., “Swin Transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10012–10022, 2021.
  * [40] V. Balntas et al., “Hpatches: A benchmark and evaluation of handcrafted and learned local descriptors,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5173–5182, 2017.
  * [41] D. DeTone et al., “Toward geometric deep SLAM,” arXiv preprint arXiv:1707.07410, 2017.
  * [42] R. Sinkhorn, “A relationship between arbitrary positive matrices and doubly stochastic matrices,” The Annals of Mathematical Statistics, vol. 35, no. 2, pp. 876–879, 1964.
  * [43] Q. Zhou et al., “Patch2Pix: Epipolar-guided pixel-level correspondences,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4669–4678, 2021.
  * [44] J.-H. Park et al., “Quantized sparse training: A unified trainable framework for joint pruning and quantization in dnns,” ACM Transactions on Embedded Computing Systems, vol. 21, no. 5, pp. 1–22, 2022.
  * [45] B. Zhao et al., “Decoupled knowledge distillation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11953–11962, 2022.
  * [46] G. Hinton et al., “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015.
  * [47] T. He et al., “Knowledge adaptation for efficient semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 578–587, 2019.
  * [48] Q. Li et al., “Mimicking very efficient network for object detection,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6356–6364, 2017.
  * [49] G. Chen et al., “Learning efficient object detection models with knowledge distillation,” Advances in Neural Information Processing Systems (NeurIPS), vol. 30, pp. 742–751, 2017.
  * [50] Z. Li and N. Snavely, “MegaDepth: Learning single-view depth prediction from internet photos,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2041–2050, 2018.
  * [51] M. Tyszkiewicz et al., “Disk: Learning local features with policy gradient,” Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 14254–14265, 2020.
  * [52] A. Dai et al., “ScanNet: Richly-annotated 3D reconstructions of indoor scenes,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5828–5839, 2017.
  * [53] Q. Wang et al., “Learning feature descriptors using camera pose supervision,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 757–774, 2020.
  * [54] Y. Shi et al., “ClusterGNN: Cluster-based coarse-to-fine graph neural network for efficient feature matching,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12517–12526, 2022.