# Motion Deblurring with Real Events

Fang Xu1, Lei Yu1111Corresponding author, Bishan Wang1, Wen
Yang1111Corresponding author, Gui-Song Xia2, Xu Jia3111Corresponding author,
Zhendong Qiao4, Jianzhuang Liu4
1School of Electronic Information, Wuhan University
2School of Computer Science, Wuhan University
3School of Artificial Intelligence, Dalian University of Technology
4Noah’s Ark Lab, Huawei Technologies
{xufang, ly.wd, bswang, yangwen<EMAIL_ADDRESS><EMAIL_ADDRESS>{qiaozhendong<EMAIL_ADDRESS>
###### Abstract

In this paper, we propose an end-to-end learning framework for event-based
motion deblurring in a self-supervised manner, where real-world events are
exploited to alleviate the performance degradation caused by data
inconsistency. To achieve this end, optical flows are predicted from events,
with which the blurry consistency and photometric consistency are exploited to
enable self-supervision on the deblurring network with real-world data.
Furthermore, a piece-wise linear motion model is proposed to take into account
motion non-linearities and thus leads to an accurate model for the physical
formation of motion blurs in the real-world scenario. Extensive evaluation on
both synthetic and real motion blur datasets demonstrates that the proposed
algorithm bridges the gap between simulated and real-world motion blurs and
shows remarkable performance for event-based motion deblurring in real-world
scenarios.

##  1 Introduction

††footnotetext: The research was partially supported by the National Natural
Science Foundation of China, No. 61871297 and the Fundamental Research Funds
for the Central University of China, WHU No. 2042020kf0019 and DUT No.
82232026. And the numerical calculations have been done on the supercomputing
system in the Supercomputing Center of Wuhan University.

Due to motion ambiguities as well as the erasure of intensity textures [12],
the task of motion deblurring is severely ill-posed [11, 25]. With the help of
an event camera which can “continuously” emit events asynchronously with
extremely low latency (in the order of $\mu$s) [15, 6], inherently embedding
motions and textures [2], the task of motion deblurring [24, 22, 28] can be
essentially alleviated. Many event-based motion deblurring methods have been
proposed by learning from synthesized dataset composed of simulated events and
blurry images as well as sequences of sharp clear ground-truth images [10,
31]. However, the inconsistency between synthetic and real data degrades the
performance of inference on real-world event cameras [29].

Figure 1: Illustrative example of inconsistency between synthesized and real-
world motion blurs with respect to the event time-surface [14]: (a) a real-
world motion blurred image; (b) the time-surface of real-world events
corresponding to (a); (c) the time-surface of events collected with the same
trajectory as (a) but at a slow motion speed; (d) deblurred results
respectively by eSL-Net [31] (top row), LEDVDI [16] (middle row) and our
proposed RED-Net (bottom row) that trained with real-world events and real-
world motion blurs, sequence predictions of the blue area are shown on the
right. Our proposed RED-Net generates less halo artifacts and achieves the
best visualization performance.

The physical intrinsic noise of event cameras raises the difficulty of
simulating labeled events that highly match the real event data [26]. Even
though the event simulator to some extent reduces the gap by considering the
pixel-to-pixel variation in the event threshold [26], additional noise effects
such as background activity noise and false negatives [1] still exist, leading
to tremendous discrepancy between the virtual events synthesized from event
simulators [26] and the real events emitted by event cameras. An alternative
approach is to build a labeled dataset composed of real-world events
accompanying with synthesized blurry images, and then train networks on it
[10]. Unfortunately, obtaining such pairs is not always easy, which needs to
be captured with a slow motion speed as well as under good lighting conditions
to avoid motion blur. Subsequent blurry image synthesis and alignment on
temporal domain is also tedious but indispensable. Furthermore, inconsistency
still exists between the events associated with synthesized and real-world
motion blur in that limited read-out bandwidth leads to more event timing
variations, as shown in Fig. 1.

In this paper, we propose a novel framework of learning the event-based motion
deblurring network in a self-supervised manner, where real-world events and
real-world motion blurred images are exploited to alleviate the performance
degradation caused by data inconsistency and bridge the gap between
simulations and real-world scenario. The proposed framework consists of two
neural networks, an event-based motion deblurring network (Deblur-Net) and an
event-based optical flow estimation network (OF-Net). The Deblur-Net is fed
with both events and a single motion blurred image, and outputs a sequence of
sharp clear images, while the OF-Net receives events and provides motions
between the reconstructed sharp clear images. We relate motions and sharp
clear images according to the photometric constancy [36]. The real-world
motion blurs with non-linearities are considered by a piece-wise linear motion
(PLM) model which improves the accuracy of optical flow and thus provides a
more precise blurring model between the reconstructed intensity images from
Deblur-Net and the blurry input. The overall network is jointly trained end-
to-end over partially labeled dataset composed of synthetic data using ESIM
[26] with ground-truth sharp clear images and real-world data only containing
real-world events and real-world motion blurred images, which can be captured
simultaneously by the DAVIS [3] or with a dual camera set connecting the event
camera and the RGB camera with a beam splitter [32].

The main contributions of our work are in three folds:

  * •

We propose a framework of learning the event-based motion deblurring network
with real-world events, which remarkably improves the performance of motion
deblurring in the real-world scenario.

  * •

We propose a piece-wise linear motion model to consider the motion
nonlinearity, based on which the OF-Net is ameliorated for real events and
outputs accurate and dense motion flows.

  * •

Extensive experiments show that the proposed method can yield high quality
sharp frames and achieve state-of-the-art results on the real blurry event
dataset.

##  2 Related Work

Motion Deblurring. Even though motion blur degenerates photographs by
averaging over the exposure time intervals, it inherently embeds both motions
and textures of the moving objects and thus enables the possibility of motion
deblurring for visualization of the dynamic scene behind blurred photographs
[12, 11, 25]. Early attempts address this problem by assuming spatially
uniform motion where blurs can be modelled as the convolution of a blur kernel
with a sharp image, leading to the kernel-based motion deblurring approaches
by means of regularized deconvolution [13, 30]. For motion blurs caused by
complex motion behaviors, pixel/patch-wise motion flows are often required to
flexibly depict non-uniform motions. To achieve this end, the optical flow is
often estimated from a single blurry image [7] or consecutive frames [34] and
provides photometric constancy [8] between the recovered latent sharp images,
leading to flow-based motion deblurring approaches. Motion blurs bring the
ambiguity of temporal ordering as well as the erasure of spatial textures
[12], thus several priors have been successfully employed for blur kernels,
_e.g_., sparsity [33], Gaussian scale mixture [5], and dark channel [21], and
for optical flows, _e.g_., linear motion [17], and order invariance [12],
which however may generate artifacts and degrade deblurring performance if the
above assumptions are not fulfilled [35].

An alternative approach is to learn a neural network for motion deblurring
directly from the data, which often achieves prominent performance [12, 11,
25, 7, 20]. Jin _et al_. pioneer to recover a sequence of sharp frames from a
single motion-blurred image [12], where they sequentially train multiple
neural networks with a temporal invariant loss. Purohit _et al_. utilize a
single recurrent neural network to generate the entire sequence [25] which,
however, may suffer from ambiguities in temporal ordering. Take this into
account, Jin _et al_. tackle the problem of the temporal ambiguity by feeding
multiple motion blurred images [11], while Rengarajan _et al_. additionally
use two consecutive images over a short exposure time [27]. On the other hand,
Chen _et al_. propose a Reblur2Deblur network [4] for motion deblurring via
self-supervised learning to leverage the physical model and the data prior,
where multiple blurry images and a linear motion assumption between them are
essentially required.

Event-Based Motion Deblurring. Event camera measures per-pixel brightness
change and outputs an event once the change exceeds a threshold [6]. The
triggered events can be “continuously” emitted asynchronously with extremely
low latency and thus provide missing information during exposure intervals if
the motion blur happens [24]. The temporal ambiguity and texture erasure can
be easily tackled by introducing events into the deblurring algorithms [24,
16]. Event-based motion deblurring methods can be categorized into two groups,
_i.e_., model driven and data driven algorithms. Model-based algorithms relate
events, blurry images and the corresponding latent sharp clear images
according to the physical event generation principle [24, 23, 28]. However,
due to the imperfections of physical implementation including intrinsic noise
and limited read-out bandwidth [9, 6], real events are essentially with noise
both in temporal and spatial domains [15] which inevitably degrades
performance [28].

Data driven algorithms relax the above limitations by utilizing neural
networks and directly learn the relation from a blurry image to a sequence of
sharp clear images with the aid of events [16, 31, 10]. For training purpose,
a synthesized dataset composed of labeled events and blurry images is commonly
simulated from sharp clear video sequences [10, 31] which, however, may have
inconsistency to the real events due to event noise in the spatio-temporal
domain. Even though data variations have been considered by manually adding
noise [31, 29], the generalizability remains limited for real event cameras.

Jiang _et al_. [10] build the Blur-DVS dataset based on real events triggered
at a slow motion speed (to obtain a sequence of sharp clear intensity images
as the ground truth) and train their network on it, where the sim-to-real gap
is reduced to some extent [16]. However, building such a dataset is strenuous
and requires strict conditions to obtain blur-free ground-truths, _e.g_.,
relative slow motion between camera and scenario and good lighting
environment. Besides, due to the limited read-out bandwidth, the inconsistency
between events emitted at a slow motion speed and their counterparts at a fast
motion speed also exists.

Learning with real-world data is more adaptive to the real-world scenario than
synthetic data, and moreover, real-world events and motion blurs can be easily
captured at extremely low cost without sophisticated procedures, which
motivates us to propose a new framework to learn event-based motion deblurring
by leveraging the real-world events and real-world motion blurred images.

##  3 Problem Statement

Event-based motion deblur (ED) aims at reconstructing a sequence of sharp and
clear latent images $\\{{\bf I}_{t}\\}_{t\in T}$ from a single motion blurred
image $\bf B$ captured with exposure time $T$ and corresponding events ${\bf
E}_{T}\triangleq\\{({\bf x}_{i},p_{i},t_{i})\\}_{t_{i}\in T}$ triggered in $T$
where $t_{i}$ and ${\bf x}_{i}$ respectively denote the timestamp and the
pixel location of the $i$-th event, and $p_{i}\in\\{+1,-1\\}$ is the polarity.
Many algorithms are proposed to tackle the problem of ED by learning-based
approaches [16, 10, 31], _i.e_.,

${\bf I}_{t}=\mbox{ED-Net}\left({\bf B},{\bf E}_{T}\right),\quad t\in T$ (1)

which are commonly trained over a synthetic dataset
$\mathcal{D}_{s}\triangleq\\{\hat{\bf B}_{k},\hat{\bf
E}_{T_{k}},\tilde{\mathcal{G}}_{k}\\}_{k}$ where $\hat{\bf B}_{k}$ and
$\hat{\bf E}_{T_{k}}$ are synthesized from the ground-truth sequence
$\tilde{\mathcal{G}}_{k}$, _i.e_., $\hat{\bf
E}_{T_{k}}=\mbox{ESIM}(\tilde{\mathcal{G}}_{k})$ [26] and $\hat{\bf
B}_{k}=\mbox{Avg}(\tilde{\mathcal{G}}_{k})$.

Real-World Events. Real-world events $\tilde{\bf E}_{T_{k}}$ are different
from the simulated events $\hat{\bf E}_{T_{k}}$ in two aspects: (1) event
noises in spatial domain are principally induced by physical intrinsic camera
imperfections (_etc_., variant event threshold), background activities, false
negatives, _etc_., [6]; (2) event noises in temporal domain are produced owing
to the limited read-out bandwidth and bring timing variances [9], which
commonly exist at a fast motion speed when the motion blur happens. Statistics
of noises from both aspects are too complicated to be addressed in current
event simulators [26], leading to inconsistency between synthetic dataset
$\mathcal{D}_{s}$ and its real-world counterparts.

For spatial event noises, [16] has built a real event dataset
$\mathcal{D}_{e}\triangleq\\{\hat{\bf B}_{k},\tilde{\bf
E}_{T_{k}},\tilde{\mathcal{G}}_{k}\\}_{k}$ composed of the real-world events
$\tilde{\bf E}_{T_{k}}$, the real-world sharp clear ground-truth images
$\tilde{\mathcal{G}}_{k}$ captured at an ultra slow speed to avoid motion
blurs, and synthesized blurry images $\hat{\bf
B}_{k}=\mbox{Avg}(\tilde{\mathcal{G}}_{k})$. Theoretically, training on
$\mathcal{D}_{e}$ is able to take into account the spatial noises, while the
temporal noises cannot be considered since they always accompany with real-
world motion blurs where no ground-truth images can be obtained. It raises the
problem of learning on unlabeled real-world dataset
$\mathcal{D}_{r}\triangleq\\{\tilde{\bf B}_{k},\tilde{\bf E}_{T_{k}}\\}_{k}$,
with real-world blurry images $\tilde{\bf B}_{k}$ and real-world events
$\tilde{\bf E}_{T_{k}}$.

Real-World Motions. Motions and intensity textures are coupled together based
on the photometric consistency and thus commonly exploited for the motion
deblur [11, 27]. For simplicity, most existing motion deblur algorithms assume
linear motion during the exposure time $T$ [23, 4], where the motion flow
stays constant. However, it is not always correct for dynamic scenes since
real-world motions might be nonlinear especially at a fast motion speed.

Thus the main obstacles of leveraging the power of events to learn motion
deblurring in real world are in two folds. First, the network should be
trained on real-world data with real events and real motion blurs. Second, the
motion non-linearity should be considered to deal with motion blurs in real-
world complex dynamic scenes.

##  4 Method

Figure 2: Overview of the proposed learning framework for Real-world Event
based motion Deblurring Network (RED-Net), where the blur-consistency and
photometric-consistency provide self-supervised losses for real-world datasets
and the reconstruction error provides the supervised loss for synthetic
datasets.

Existing event-based deblurring methods are usually developed within a
supervised learning framework, of which performance is limited to the specific
training data (simulated events or synthesized blurred image from the dataset
captured with slow motion). Taking into account both deblurring and the
physical image blur formation process, we construct a semi-supervised learning
framework which can generalize well to real-world motion-blurred images caused
by fast motion.

Fig. 2 illustrates the overall architecture of our framework. The Deblur-Net
takes a single motion-blurred image ${\mathbf{B}}$ and its event data
${\mathbf{E}}$ as input and outputs a sequence of sharp frames
$\\{\mathbf{I}_{m}\\}_{m=0}^{M-1}$, where $M$ is the number of estimated sharp
frames. The supervised loss that compares the estimated sharp frames with
ground truth only works for labeled synthesized dataset. To guide the semi-
supervised learning of Deblur-Net, our framework consists of an additional OF-
Net for motion estimation, which takes event data ${\mathbf{E}}$ as input and
outputs optical flow serving as motion information. On the strength of the
physical blur formation process, a motion-blurred image can be re-rendered on
the basis of estimated sharp images and motion information, which provides a
blur-consistency constraint by comparing with the original motion-blurred
image. Besides, estimated sharp images are propagated by the estimated optical
flow, which provides an additional constraint of photometric-consistency by
comparing the propagated sharp images with the directly recovered ones.

Considering the motion non-linearity in real-world complex dynamic scenes, we
take full advantage of high temporal resolution of event camera. Since event
data is able to provide continuous motion information in a period, we do not
need strong assumption on linear constant motion in a large time interval as
other deblurring methods [7, 23, 4, 17]. Event data can be used in modeling
highly nonlinear motion happened in the exposure time.

###  4.1 Blur-consistency Constraint

Providing the latent sharp image $\mathbf{L}_{t}(\mathbf{x})$ during the
exposure time interval $T$, the blurring process can be physically formulated
as the average of them in the duty cycle,

$\bar{\mathbf{B}}({\mathbf{x}})=\frac{1}{|T|}\int_{t\in
T}\mathbf{L}_{t}(\mathbf{x})dt\approx\frac{1}{N}\sum_{n=0}^{N-1}\mathbf{L}_{n}(\mathbf{x}),$
(2)

where $\\{\mathbf{L}_{n}\\}_{n=0}^{N-1}$ is the discrete version of
$\mathbf{L}_{t}$ and a large enough $N\gg M$ is required to achieve small
discretization error. Providing recovered sequences
$\\{\mathbf{I}_{m}\\}_{m=0}^{M-1}$ output from the Deblur-Net, one can resolve
$\mathbf{L}_{n}$ by interpolating with $\\{\mathbf{I}_{m}\\}_{m=0}^{M-1}$ over
the motion field ${\bf u}_{n}({\bf x})$ representing per-pixel flow from
$\mathbf{L}_{0}$ to $\mathbf{L}_{n}$ _i.e_.,

$\mathbf{L}_{n}(\mathbf{x})=\mathbf{L}_{0}(\mathbf{x}+\mathbf{u}_{n}(\mathbf{x})).$
(3)

For simplicity, a linear motion (LM) model is commonly assumed [17, 23, 7],
_i.e_.,

$\mathbf{u}_{n}(\mathbf{x})=n\mathbf{v}(\mathbf{x}),$ (4)

with $\mathbf{v}$ denoting the per-pixel unit motion flow which is assumed to
be constant during the exposure time interval $T$.

To consider the motion non-linearity for real-world scenario, we propose a
piece-wise linear motion (PLM) model by making full use of rich motion
information encoded in events. We divide event stream within the exposure time
$T$ into $M-1$ equal time intervals and the LM model is assumed within each
interval. Finally, we can get the following PLM model, for $n\in[mK,(m+1)K)$,

$\mathbf{u}_{n}(\mathbf{x})=(n-mK)\mathbf{v}_{m}(\mathbf{x})+\mathbf{u}_{mK}(\mathbf{x}),$
(5)

with $\mathbf{v}_{m}$ the per-pixel unit motion flow that associates with the
$m$-th time interval, $m=0,1,...,M-1$ and $K\triangleq N/M$ ($N$ can be chosen
as integer times of $M$). And moreover, $\mathbf{v}_{m}$ can be predicted from
the OF-Net using events in the $m$-th time interval and resolve a nonlinear
motion field $\mathbf{u}_{n}(\mathbf{x})$.

Then providing the sequence of sharp frames $\\{\mathbf{I}_{m}\\}_{m=0}^{M-1}$
output from the Deblur-Net, the $n$-th latent image $\mathbf{L}_{n}$ with
$n\in[mK,(m+1)K)$ can be resolved by warping the $m$-th frame $\mathbf{I}_{m}$
using the motion field defined in (5),

$\begin{split}\mathbf{L}_{n}(\mathbf{x})&=\mbox{Warp}\left(\mathbf{I}_{m}(\mathbf{x}),\mathbf{u}_{n}(\mathbf{x})\right)\\\
&\triangleq\mathbf{I}_{m}(\mathbf{x}+(n-mK)\mathbf{v}_{m}(\mathbf{x})).\end{split}$
(6)

Hence, we can re-render a corresponding reblurred image based on (2) using the
generated latent sharp images $\\{\mathbf{L}_{n}\\}_{n=0}^{N-1}$. A blur-
consistency loss is constructed between the reblurred image $\bar{\mathbf{B}}$
and the original blurry input ${\mathbf{B}}$:

$\mathcal{L}_{blur}=\|\bar{\mathbf{B}}-\mathbf{B}\|_{1}.$ (7)

By imposing this loss in the whole framework, we build a bridge between the
motion-blurred image and its corresponding sharp clear images, which provides
a self-supervision to the network. For real-world motion blurs with motion
non-linearities, PLM is important to ensure the model accuracy and thus
provide faithful supervision even with noise disturbances.

###  4.2 Photometric-consistency Constraint

In addition to the blur-consistency constraint on the sequence level, further
photometric-consistency constraint is imposed, which utilizes the inter-frame
connection between the temporal consistency of estimated sharp images and
motion information encodes in events. We warp $\mathbf{I}_{m+1}$ to
$\mathbf{I}_{m}$ with the predicted optical flow $\mathbf{v}_{m}$ via the OF-
Net by feeding events in the $m$-th time interval and obtain the warped image:

$\bar{\mathbf{I}}_{m}(\mathbf{x})=\mathbf{I}_{m+1}(\mathbf{x}+K\mathbf{v}_{m}(\mathbf{x})).$
(8)

Then photometric-consistency loss is constructed by comparing the propagated
sharp images with the recovered ones directly predicted by Deblur-Net:

$\mathcal{L}_{photo}=\frac{1}{M-1}\sum_{m=0}^{M-2}\|\bar{\mathbf{I}}_{m}-\mathbf{I}_{m}\|_{1}.$
(9)

###  4.3 Optimization

We propose to train the Deblur-Net and OF-Net over a partially labeled
dataset, composed of the synthetic dataset $\mathcal{D}_{s}$ with ground-truth
sharp clear images and the real-world dataset $\mathcal{D}_{r}$ without ground
truth. Apparently, the blur-consistency and the photometric-consistency are
applicable for both $\mathcal{D}_{s}$ and $\mathcal{D}_{r}$, and provide self-
supervised losses respectively defined as
$\mathcal{L}_{blur}^{s},\mathcal{L}_{photo}^{s}$ and
$\mathcal{L}_{blur}^{r},\mathcal{L}_{photo}^{r}$ according to (7) and (9).
Besides, for the synthetic dataset $\mathcal{D}_{s}$ with a sequence of
ground-truth sharp clear images $\tilde{\mathcal{G}}$, we use it to supervise
the network via a reconstruction error loss,

$\mathcal{L}_{error}=\frac{1}{M}\sum_{m=0}^{M-1}||\mathbf{I}_{m}-\mathbf{G}_{m}||_{1},$
(10)

where $\mathbf{G}_{m}\in\tilde{\mathcal{G}}$. Thus the overall function is as
follows:

$\mathcal{L}=\mathcal{L}_{error}^{s}+\alpha\mathcal{L}_{blur}^{s}+\beta\mathcal{L}_{photo}^{s}+\gamma\mathcal{L}_{blur}^{r}+\delta\mathcal{L}_{photo}^{r},$
(11)

with $\alpha$, $\beta$, $\gamma$ and $\delta$ denoting the balancing
parameters.

##  5 Experiments

###  5.1 Experimental Settings

Dataset. The proposed Real-world Event-based motion Deblurring network (RED-
Net) is trained in a self-supervised manner, where one synthetic dataset
(GoPro) is provided for training with ground-truth and two datasets
respectively captured by a DAVIS240C camera (HQF provided in [29] containing
real events but blur-free intensity frames) and a DAVIS346 camera (RBE built
in this paper for the real-world scenario with real events and real motion
blurs).

GoPro: Based on the GoPro dataset [18], we build a synthetic dataset as [31]
composed of simulated events and synthetic blurry images as well as sharp
clear ground-truth images. We first increase the frame rate by interpolating 7
images between consecutive frames [19] and then generate both events and
blurry images based on the interpolated high frame-rate sequences. ESIM [26]
is exploited to simulate events with consideration of per-pixel threshold
variation. And blurry images are simply obtained by averaging over 49
consecutive images.

HQF: We construct a similar dataset as Blur-DVS [16] based on HQF dataset
[29], which contains real-world events and sharp clear ground-truth frames
captured simultaneously from a DAVIS240C that are well-exposed and minimally
motion-blurred. Providing the ground-truth frames, motion blurs can be
synthesized using the same manner as the GoPro dataset. Finally, the HQF
dataset contains real-world events and synthesized blurry images as well as
the ground-truth frames. The ground-truth frames are provided only for
quantitative evaluation. In training stage of our proposed RED-Net over the
HQF dataset, only real events and synthesized blurry images are used.

RBE: A Real-world Blurry images and Events (RBE) dataset is built with a
DAVIS346 camera and only contains real-world events and real-world motion
blurs, which can be collected in a facilitated manner. Thus the RBE dataset
can provide a large number of paired real-world data, which can be fed into
the RED-Net and improve the adaptivity to the real-world scenario.

Implementation Details For the Deblur-Net and OF-Net modules, we take
advantage of existing neural network architectures which have performed well
in the past for the respective learning tasks. Inspired by that a sharp image
can be mapped from a blurry image using a residual term encoded with events
[24], we select a residual network to predict a sequence of sharp frames from
a single motion-blurred image and its event data. In particular, we adopt the
residual network from Jin et al. [11]. For the OF-Net, we adopt the EV-FlowNet
architecture from Zhu et al. [36], which is widely used to predict optical
flow from events.

Our network is implemented using Pytorch on a single NVIDIA Geforce RTX 3090
GPU. During training, we randomly crop the samples into $128\times 128$
patches. Adam optimizer is used and the maximum epoch of training iterations
is set to 30. The learning rate starts at $10^{-4}$, then decays by $25\%$
every five epochs from the $15$-th epoch. The weighting factors
$\alpha,\beta,\gamma$ and $\delta$ are all set to $1$, the number of deblurred
images $\\{\mathbf{I}_{m}\\}_{m=0}^{M-1}$ and latent sharp images
$\mathbf{L}_{n}$ warped from $\mathbf{I}_{m}$ are respectively $M=7$ and
$K=11$. The optical flow module is pretrained on the Multi-Vehicle Stereo
Event Camera dataset (MVSEC) [36], where the reference ground-truth optical
flow is provided.

RED-Nets. To validate the effectiveness of exploiting real-world data, RED-Net
is trained respectively over three different datasets, _i.e_., GoPro,
GoPro+HQF and GoPro+RBE, and final networks are respectively denoted as RED-
GoPro, RED-HQF and RED-RBE. Specifically, RED-GoPro is trained only over
synthesized GoPro dataset with the supervision of ground-truth frames. Both
RED-HQF and RED-RBE are trained in a self-supervised manner, where RED-HQF
uses real-world events but synthetic motion blurs, while RED-RBE uses real-
world events and motion blurs.

To validate the superiority of piece-wise linear motion model (PLM), we
replace it with the linear motion model (LM) as (4), and train RED-Net
respectively on GoPro and GoPro+RBE datasets. Specifically, the resulted
network is denoted as RED-(GoPro/RBE)-LM and its counterpart with PLM is
denoted as RED-(GoPro/RBE)-PLM. By default, RED-(GoPro/RBE) means
RED-(GoPro/RBE)-PLM in the following sections.

###  5.2 Results of Optical Flow

Fig. 3 presents the optical flow (OF) outputs of OF-Net from different RED-
Nets and their corresponding deblur results of a real-world motion blurred
image containing a fast moving magic cube. For comparison, the output of OF-
Net pretrained on MVSEC dataset (_i.e_. EV-FlowNet) is also given. Apparently,
the output of OF-Net from RED-RBE trained on real data achieves the best
performance with consistent motion and sharp clear edge. Thus the deblur image
of RED-RBE has the best qualitative appearance as well.

To reveal the advantage of PLM, we compare the outputs of OF-Net from RED-
GoPro-LM as shown in Fig. 3 and RED-GoPro-PLM as shown in Fig. 3. It is
obvious that RED-GoPro-PLM gives better OF result with clear edges and stable
background while RED-GoPro-LM suffers from blurry effects and background
movement. Furthermore, when violating the linear motion assumption, the result
of OF-Net from RED-GoPro-LM is even worse than its origin, i.e. EvFlowNet as
shown in Fig. 3 and 3.

Besides, we compare the OF results of EvFlowNet and two OF-Nets respectively
from RED-GoPro-PLM and RED-RBE as shown in Fig. 3, 3 and 3. With the
supervision of blurry image and the intensity sequence according to the blur-
consistency loss (7), OF-Nets from both RED-GoPro-PLM and RED-RBE can give
better results than EvFlowNet. It reveals that the motion information embedded
in the blurry images helps the prediction of OF within our framework.
Moreover, OF-Net from RED-RBE trained over the real dataset outperforms that
of RED-GoPro-PLM trained over the synthetic dataset and gives more consistent
motions of the cube surface as shown in Fig. 3 and 3. And it validates the
advantage and necessity of training on real-world dataset.

Figure 3: Optical flow output of OF-Net: (a) pre-trained on MVSEC (equivalent to EV-FlowNet), (b) RED-GoPro-LM, (c) RED-GoPro-PLM and (d) RED-RBE. The deblurred results (f), (g) and (h) of the motion blurred image (e) are respectively corresponding to (b), (c) and (d). Table 1: Quantitative comparisons of proposed RED-Nets trained over different datasets to the state-of-the-arts. RED-GoPro, RED-HQF, and RED-RBE are respectively trained on GoPro, GoPro+HQF and GoPro+RBE. Note that LEDVDI only outputs 6 frames for sequence prediction, while the others output 7 frames.  Method | Single frame prediction comparison |  | Sequence prediction comparison
---|---|---|---
GoPro |  | HQF |  | GoPro |  | HQF
PSNR | SSIM |  | PSNR | SSIM |  | PSNR | SSIM |  | PSNR | SSIM
blur2mflow [7] | 21.818 | 0.6454 |  | 21.539 | 0.6298 |  | / | / |  | / | /
LEVS [12] | 21.950 | 0.6406 |  | 21.900 | 0.6367 |  | 19.893 | 0.5546 |  | 19.068 | 0.5403
EDI [24] | 21.497 | 0.6510 |  | 20.321 | 0.6212 |  | 20.945 | 0.6326 |  | 19.081 | 0.5873
eSL-Net [31] | 24.791 | 0.8009 |  | 20.438 | 0.6017 |  | 23.955 | 0.7578 |  | 19.866 | 0.5851
LEDVDI [16] | 22.856 | 0.7334 |  | 22.221 | 0.7567 |  | 22.673 | 0.7329 |  | 21.558 | 0.7355
RED-GoPro | 28.984 | 0.8499 |  | 24.149 | 0.7332 |  | 28.343 | 0.8359 |  | 23.118 | 0.7116
RED-HQF | 27.137 | 0.8361 |  | 25.650 | 0.7661 |  | 26.640 | 0.8204 |  | 23.872 | 0.7292
RED-RBE | 26.672 | 0.8372 |  | 25.717 | 0.7629 |  | 26.302 | 0.8247 |  | 24.076 | 0.7340

###  5.3 Comparisons with State-of-the-art Methods

Since our method is able to remove motion blur and reconstruct a sequence of
latent sharp clear images, the deblur performance is evaluated on both single
frame and sequence reconstructions. We compare the proposed RED-Nets to state-
of-the-art conventional deblurring methods including blur2mflow [7] and LEVS
[12], and event-based motion deblurring methods including EDI [24], eSL-Net
[31] and LEDVDI [16]. Due to the lack of ground-truth for real-world motion
blurs, we exploit the HQF dataset constructed from blur-free frames for
quantitative evaluations, where the effectiveness of training with real-world
events are validated. The quantitative results are presented in Tab.1 and show
that the proposed RED-Nets bring remarkable improvements comparing to the
state-of-the-arts, on average 3.5 dB gain on single frame prediction and 2.5
dB gain on sequence prediction over the HQF dataset with real-world events.
Detailed analyses are presented in the following.

Inconsistency of Simulated and Real-world Events. We first compare the
deblurring performance of conventional methods, _i.e_., blur2mflow and LEVS to
event-based methods, _i.e_., EDI and eSL-Net, over the synthetic GoPro dataset
and the HQF dataset with real-world events. It is shown that EDI provides
comparable performance to learning based conventional methods with the help of
events. For the learning based approaches, eSL-Net enhanced with simulated
events outperforms blur2mflow and LEVS by a large margin on the synthetic
GoPro dataset. However, the overwhelming performance of eSL-Net is not
retained on the HQF dataset with real-world events, which reflects the
inconsistency of simulated and real-world events.

Effectiveness of Training with Real-world Events. Comparing to RED-GoPro
trained only on synthetic dataset, the other two RED-Nets, _i.e_., RED-HQF and
RED-RBE, trained respectively over the HQF and RBE dataset with real events,
bring a performance drop on the synthetic GoPro dataset but achieve the PSNR
gain on the HQF dataset with real-world events. Specifically, RED-HQF obtains
on average 1.1 dB gain on PSNR, which shows the effectiveness of training with
real-world events and bridges the sim-to-real gap. Furthermore, we validate
this point by training RED-RBE with the real-world data in the RBE dataset
which can be constructed in a facilitated manner.

Event-based Motion Deblur with Real-World Events. As shown in Tab. 1, RED-
HQF/RBE outperforms EDI and eSL-Net over the HQF dataset with real-world
events by a large margin. Furthermore, we exploit the recently proposed
network, _i.e_., LEDVDI trained with real-world events for fair comparisons.
Different from our proposed method, LEDVDI is trained with real-world events
in a supervised manner. Our approach requires neither blurry synthesis nor
strict slow moving speed to collect blur-free ground-truth images, and thus is
much easier to be adopted in real-world scenario. From the quantitative
results in Tab. 1, it is shown that both RED-HQF and RED-RBF achieve better
performance than LEDVDI for single frame prediction and sequence prediction,
which validates the superiority of our proposed framework. Note that LEDVDI
deals with single frame prediction and sequence prediction separately with
different networks, while our proposed RED-HQF/RBE tackle both problems in one
network.

Real-World Motion Deblur. The above quantitative comparisons are all based on
synthesized motion blurs. Due to the lack of ground-truth frames, the
performance with real-world motion blurs are qualitatively investigated, as
shown in Fig. 4. We choose two scenes captured manually with a DAVIS346
camera, _i.e_., labfloor and window. Without the aid of events, blur2mflow and
LEVS fail to recover sharp clear images. For event-based methods, EDI fails to
cope with severe motion blurs as illustrated in window. Among the learning
based approaches, RED-RBE achieves the best visualization performance, while
the others are likely to give over smoothed effects or generate halo artifacts
around black edges, for instance the bottom of garbage can in labfloor and the
top-left corner of TV in window. Compared with RED-HQF and LEDVDI trained with
synthesized motion blurs, RED-RBE is trained with real-world motion blurs
which can alleviate the problem of event noise in temporal domain, _e.g_.,
timing chaos caused by limited read-out bandwidth and thus give less halo
artifacts.

Blurry ImageBlurry Image

blur2mflowblur2mflow

LEVSLEVS

EDIEDI

eSL-NeteSL-Net

LEDVDILEDVDI

RED-HQFRED-HQF

RED-RBFRED-RBF

Blurry ImageBlurry Image

blur2mflowblur2mflow

LEVSLEVS

EDIEDI

eSL-NeteSL-Net

LEDVDILEDVDI

RED-HQFRED-HQF

RED-RBFRED-RBF

Figure 4: Qualitative results of motion deblur for 2 blurry scenes by 7
different methods where the top-two rows correspond to labfloor and the
bottom-two correspond to window. For each scene, from top-left to bottom-right
are respectively the blurry image and its deblurred result by blur2mflow [7],
LEVS [12], EDI [24], eSL-Net [31], LEDVDI [16] and our proposed RED-HQF and
RED-RBE. Table 2: Ablation study of proposed framework w/o synthesized events
(SynEv), blurring with LM (LM) , blurring with PLM (PLM) and real-world events
(ReEv).

Methods | SynEv | LM | PLM | ReEv | GoPro | HQF
---|---|---|---|---|---|---
Deblur-Net [11] |  |  |  |  | 23.563 | 21.373
Deblur-Net-GoPro | $\checkmark$ |  |  |  | 28.451 | 23.553
RED-GoPro-LM | $\checkmark$ | $\checkmark$ |  |  | 28.752 | 23.988
RED-GoPro | $\checkmark$ |  | $\checkmark$ |  | 28.984 | 24.149
RED-RBE-LM | $\checkmark$ | $\checkmark$ |  | $\checkmark$ | 26.451 | 25.266
RED-RBE | $\checkmark$ |  | $\checkmark$ | $\checkmark$ | 26.672 | 25.717

###  5.4 Ablation Study

The proposed RED-Net improves the performance of motion deblurring by self-
supervising with real-word events. To achieve this end, optical flows are
predicted from events with which the piece-wise linear motion model (PLM) is
proposed to accurately model the blurry procedure even with motion non-
linearities. To find out what contributes to the superior performance of the
approach, we compare a few variants with and without use of synthesized events
(SynEv), blurring with LM (LM), blurring with PLM (PLM) and real-world events
(ReEv), as shown in Tab. 2. From the table, we can draw the following
conclusions:

Importance of Events. We validate the importance of events by training the
Deblur-Net without events over the synthetic GoPro dataset, and the
quantitative deblur results over 2 dataset are shown in the first row of Tab.
2. Comparing to RED-Nets with events, there is a large performance gap. It
shows that the rich motion information encoded in events can effectively
improve the deblurring performance.

Linear _vs_. piece-wise Linear Motion Model. With the optical flow output of
the OF-Net, motions provide the inter-frame relationship inside the
reconstructed sequence as well as the blurry consistency which relies on the
physical blurry procedure. Inaccurate motion model will violate the blurry
consistency and thus degrades the deblurring performance. We validate this
point by respectively apply LM and PLM in (2) and their performance can be
referred from Tab. 2 in the 3rd, 4th, 5th and 6th rows. Apparently, RED-Nets
with PLM outperform that with LM and achieve PSNR gains up to 0.5 dB on the
real-world dataset after training with real-world events, while it only gains
0.16 dB without real-world events. It demonstrates that PLM plays an important
role especially for real-world events.

Synthetic _vs_. Real-world Events. To validate this point, we train RED-Net
respectively on the synthetic GoPro dataset, _i.e_., RED-GoPro and on the
real-world RBE dataset, _i.e_., RED-RBE. Their results are shown in the 4th
and 6th rows of Tab. 2. And we also trained RED-GoPro and RED-RBE with LM and
present their results respectively in the 3rd and 5th rows of Tab. 2. By
comparisons, we can easily find that the self-supervision with real-world
events can achieve PSNR gains 1.6 dB with PLM (1.3 dB with LM) on real-world
dataset. It shows that the supervision of real-world events plays a major role
in improving the deblurring performance on real-world dataset.

##  6 Conclusion

A self-supervised learning framework for event-based motion deblurring is
proposed where real-world events and real-world motion blurs are exploited to
alleviate the performance degradation caused by data inconsistency. Motion
non-linearities are also considered through a PLM model to improve the
accuracy of physical blur-consistency. Within the proposed learning framework,
we evaluate RED-Net over different datasets and validate the effectiveness of
the proposed PLM-based blur-consistency and photometric-consistency over the
real-world dataset. With a blurry image and corresponding events, the proposed
RED-Net can produce a sequence of sharp clear intensity images as well as
motion flows between them. Extensive experiments demonstrate that the proposed
method can achieve the state-of-the-art with real-world events.

## References

  * [1] R Baldwin, Mohammed Almatrafi, Vijayan Asari, and Keigo Hirakawa.  Event probability mask (epm) and event denoising convolutional neural network (edncnn) for neuromorphic cameras.  In CVPR, pages 1701–1710, 2020.
  * [2] Ryad Benosman, Charles Clercq, Xavier Lagorce, Sio-Hoi Ieng, and Chiara Bartolozzi.  Event-based visual flow.  IEEE Transactions on Neural Networks and Learning Systems, 25(2):407–417, 2013.
  * [3] Christian Brandli, Raphael Berner, Minhao Yang, Shih-Chii Liu, and Tobi Delbruck.  A 240$\times$180 130 db 3 $\mu$s latency global shutter spatiotemporal vision sensor.  IEEE Journal of Solid-State Circuits, 49(10):2333–2341, 2014.
  * [4] Huaijin Chen, Jinwei Gu, Orazio Gallo, Ming-Yu Liu, Ashok Veeraraghavan, and Jan Kautz.  Reblur2deblur: Deblurring videos via self-supervised learning.  In IEEE International Conference on Computational Photography (ICCP), pages 1–9, 2018.
  * [5] Rob Fergus, Barun Singh, Aaron Hertzmann, Sam T Roweis, and William T Freeman.  Removing camera shake from a single photograph.  In ACM SIGGRAPH, pages 787–794. 2006.
  * [6] Guillermo Gallego, Tobi Delbruck, Garrick Michael Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew Davison, Jorg Conradt, Kostas Daniilidis, and Davide Scaramuzza.  Event-based vision: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020\.
  * [7] Dong Gong, Jie Yang, Lingqiao Liu, Yanning Zhang, Ian Reid, Chunhua Shen, Anton Van Den Hengel, and Qinfeng Shi.  From motion blur to motion flow: A deep learning solution for removing heterogeneous motion blur.  In CVPR, pages 2319–2328, 2017.
  * [8] Tae Hyun Kim and Kyoung Mu Lee.  Generalized video deblurring for dynamic scenes.  In CVPR, pages 5426–5434, 2015.
  * [9] IniVation.  Understanding the Performance of Neuromorphic Event-Based Vision Sensors.  IniVation, https://inivation.com/, 1 edition, 05 2020.
  * [10] Zhe Jiang, Yu Zhang, Dongqing Zou, Jimmy Ren, Jiancheng Lv, and Yebin Liu.  Learning event-based motion deblurring.  In CVPR, pages 3320–3329, 2020.
  * [11] Meiguang Jin, Zhe Hu, and Paolo Favaro.  Learning to extract flawless slow motion from blurry videos.  In CVPR, pages 8112–8121, 2019.
  * [12] Meiguang Jin, Givi Meishvili, and Paolo Favaro.  Learning to extract a video sequence from a single motion-blurred image.  In CVPR, pages 6334–6342, 2018.
  * [13] Dilip Krishnan, Terence Tay, and Rob Fergus.  Blind deconvolution using a normalized sparsity measure.  In CVPR, pages 233–240, 2011.
  * [14] Xavier Lagorce, Garrick Orchard, Francesco Galluppi, Bertram E Shi, and Ryad B Benosman.  Hots: a hierarchy of event-based time-surfaces for pattern recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(7):1346–1359, 2016.
  * [15] Patrick Lichtsteiner, Christoph Posch, and Tobi Delbruck.  A 128 $\times$ 128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor.  IEEE Journal of Solid-State Circuits, 43(2):566–576, 2008.
  * [16] Songnan Lin, Jiawei Zhang, Jinshan Pan, Zhe Jiang, Dongqing Zou, Yongtian Wang, Jing Chen, and Jimmy Ren.  Learning event-driven video deblurring and interpolation.  In ECCV, 2020.
  * [17] Peidong Liu, Joel Janai, Marc Pollefeys, Torsten Sattler, and Andreas Geiger.  Self-supervised linear motion deblurring.  IEEE Robotics and Automation Letters, 5(2):2475–2482, 2020.
  * [18] Seungjun Nah, Sungyong Baik, Seokil Hong, Gyeongsik Moon, Sanghyun Son, Radu Timofte, and Kyoung Mu Lee.  Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study.  In CVPRW, pages 1974–1984, 2019.
  * [19] Simon Niklaus, Long Mai, and Feng Liu.  Video frame interpolation via adaptive separable convolution.  In ICCV, pages 261–270, 2017.
  * [20] Thekke Madam Nimisha, Akash Kumar Singh, and Ambasamudram N Rajagopalan.  Blur-invariant deep learning for blind-deblurring.  In ICCV, pages 4752–4760, 2017.
  * [21] Jinshan Pan, Deqing Sun, Hanspeter Pfister, and Ming-Hsuan Yang.  Blind image deblurring using dark channel prior.  In CVPR, pages 1628–1636, 2016.
  * [22] Liyuan Pan, Richard Hartley, Cedric Scheerlinck, Miaomiao Liu, Xin Yu, and Yuchao Dai.  High frame rate video reconstruction based on an event camera.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020\.
  * [23] Liyuan Pan, Miaomiao Liu, and Richard Hartley.  Single image optical flow estimation with an event camera.  In CVPR, pages 1669–1678, 2020.
  * [24] Liyuan Pan, Cedric Scheerlinck, Xin Yu, Richard Hartley, Miaomiao Liu, and Yuchao Dai.  Bringing a blurry frame alive at high frame-rate with an event camera.  In CVPR, pages 6820–6829, 2019.
  * [25] Kuldeep Purohit, Anshul Shah, and AN Rajagopalan.  Bringing alive blurred moments.  In CVPR, pages 6830–6839, 2019.
  * [26] Henri Rebecq, Daniel Gehrig, and Davide Scaramuzza.  Esim: an open event camera simulator.  In Conference on Robot Learning, pages 969–982, 2018.
  * [27] Vijay Rengarajan, Shuo Zhao, Ruiwen Zhen, John Glotzbach, Hamid Sheikh, and Aswin C Sankaranarayanan.  Photosequencing of motion blur using short and long exposures.  In CVPRW, pages 510–511, 2020.
  * [28] Cedric Scheerlinck, Nick Barnes, and Robert Mahony.  Continuous-time intensity estimation using event cameras.  In ACCV, pages 308–324, 2018.
  * [29] Timo Stoffregen, Cedric Scheerlinck, Davide Scaramuzza, Tom Drummond, Nick Barnes, Lindsay Kleeman, and Robert Mahony.  Reducing the sim-to-real gap for event cameras.  In ECCV, 2020.
  * [30] Libin Sun, Sunghyun Cho, Jue Wang, and James Hays.  Edge-based blur kernel estimation using patch priors.  In IEEE International Conference on Computational Photography (ICCP), pages 1–8, 2013.
  * [31] Bishan Wang, Jingwei He, Lei Yu, Gui-Song Xia, and Wen Yang.  Event enhanced high-quality image recovery.  In ECCV, 2020.
  * [32] Zihao W Wang, Peiqi Duan, Oliver Cossairt, Aggelos Katsaggelos, Tiejun Huang, and Boxin Shi.  Joint filtering of intensity images and neuromorphic events for high-resolution noise-robust imaging.  In CVPR, pages 1609–1619, 2020.
  * [33] Li Xu, Shicheng Zheng, and Jiaya Jia.  Unnatural $\ell_{0}$ sparse representation for natural image deblurring.  In CVPR, pages 1107–1114, 2013.
  * [34] Haichao Zhang and Jianchao Yang.  Intra-frame deblurring by leveraging inter-frame camera motion.  In CVPR, pages 4036–4044, 2015.
  * [35] Shangchen Zhou, Jiawei Zhang, Jinshan Pan, Haozhe Xie, Wangmeng Zuo, and Jimmy Ren.  Spatio-temporal filter adaptive network for video deblurring.  In ICCV, pages 2482–2491, 2019.
  * [36] Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis.  Ev-flownet: Self-supervised optical flow estimation for event-based cameras.  In Robotics: Science and Systems, 2018.


# ClusVPR: Efficient Visual Place Recognition with Clustering-based Weighted
Transformer

Yifan Xu, Pourya Shamsolmoali, and Jie Yang Y. Xu, and J. Yang are with the
Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong
University, Shanghai, China. Emails: {yifanxu<EMAIL_ADDRESS>Shamsolmoali is with the School of Communication and Electrical Engineering,
East China Normal University, Shanghai, China. Email<EMAIL_ADDRESS>
###### Abstract

Visual place recognition (VPR) is a highly challenging task that has a wide
range of applications, including robot navigation and self-driving vehicles.
VPR is particularly difficult due to the presence of duplicate regions and the
lack of attention to small objects in complex scenes, resulting in recognition
deviations. In this paper, we present ClusVPR, a novel approach that tackles
the specific issues of redundant information in duplicate regions and
representations of small objects. Different from existing methods that rely on
Convolutional Neural Networks (CNNs) for feature map generation, ClusVPR
introduces a unique paradigm called Clustering-based Weighted Transformer
Network (CWTNet). CWTNet leverages the power of clustering-based weighted
feature maps and integrates global dependencies to effectively address visual
deviations encountered in large-scale VPR problems. We also introduce the
optimized-VLAD (OptLAD) layer that significantly reduces the number of
parameters and enhances model efficiency. This layer is specifically designed
to aggregate the information obtained from scale-wise image patches.
Additionally, our pyramid self-supervised strategy focuses on extracting
representative and diverse information from scale-wise image patches instead
of entire images, which is crucial for capturing representative and diverse
information in VPR. Extensive experiments on four VPR datasets show our
model’s superior performance compared to existing models while being less
complex.

###### Index Terms:

Visual place recognition, Vision transformer, Self-supervised.

##  I Introduction

Visual Place Recognition (VPR), also known as image Geo-localization, is a
technique that aims to determine the location of query images from unknown
locations by comparing them against a database of reference images from known
locations [1, 2]. VPR plays a significant role in various AI applications,
including robot navigation [3] and self-driving vehicle [4]. Recent
advancements in deep CNNs [5, 6] and Vision Transformers (ViTs) [7, 8] have
significantly improved the performance of VPR methods by extracting more
discriminative and comprehensive features from images [9, 10, 11, 12]. In this
paper, we focus on the VPR problem as an example of an image retrieval task.
Our objective is to determine the location of an image by comparing it with
geo-tagged images of a reference database. The illustration of our VPR
architecture is shown in Fig 1.

Figure 1: An illustration of the visual place recognition task using our
model.

The main challenge in VPR is to generate discriminative representations from
images with varying perspectives and appearances, such as environmental
variations (e.g., seasons, illumination changes, dynamic occlusions). CNNs
with NetVLAD [13] are commonly used in VPR [9, 11, 14] due to their ability to
extract comprehensive and discriminative features from images. However, their
effectiveness is limited in complex scenes with visual distractions, such as
occlusions and dynamic objects, because of the locality assumption of CNNs. In
such cases, the extracted features may not be able to capture the important
information for VPR, leading to inaccurate localization. To tackle this issue,
recent studies [15, 10, 16] have explored the use of ViTs that have shown
promising results in various vision tasks. However, ViTs face specific
challenges when applied to VPR tasks.

First, ViTs simply process the entire image as a sequence of image tokens and
cannot explicitly consider the importance of individual image regions. This
can be problematic in VPR tasks where certain regions, such as duplicate
regions (regions with similar visual content) or small objects, can have a
significant impact on recognition accuracy. Ignoring these regions can lead to
deviations and inaccurate localization. Second, ViTs lack certain inductive
biases that are present in CNNs, such as locality and translation
equivariance. These biases are crucial for effectively learning from image
data and capturing spatial relationships between image features. Third, VPR
datasets often have noisy GPS labels, which can introduce uncertainties and
inconsistencies in the training process. Additionally, relying on weakly-
supervised learning methods [9, 11] limits the availability of precise
annotations, which can negatively impact the performance and accuracy of ViTs
in VPR tasks.

Based on the above observations, we introduce a new model called ClusVPR. Our
ClusVPR consists of CWTNet and OptLAD layers. By leveraging the self-attention
property, the CWTNet enhances the model’s ability to capture global context
information. Additionally, we address the issue of the unequal importance of
image regions in VPR by calculating weights for image tokens using the
k-nearest neighbor (KNN) clustering algorithm. This allows us to assign
different weights to image tokens based on their relevance and significance
and effectively minimize visual deviations that commonly arise in complex VPR
tasks. To improve computational efficiency, we incorporate sparse attention
[17], into the CWTNet. Sparse attention allows us to focus on the most
relevant image tokens and minimize ineffective computations. Furthermore,
compared to the NetVLAD, OptLAD layer in ClusVPR plays a key role in
optimizing the model’s efficiency. It achieves this by splitting high-
dimensional representation vectors into uniform groups and effectively
reducing the dimension of the global descriptor.

Recent works [18, 11] have addressed the challenge of noisy GPS labels in VPR
by adopting approaches that select the most similar image to the query image
in the representation space as the top-ranked positive (ground-truth) image.
These works are trained by using the triplet loss [19] to generate feature
representations for the query image that are similar to the top-ranked
positive image. However, the effectiveness of these methods is often limited
due to the lack of information in the top-ranked positive images and the high
appearance variability in large-scale VPR datasets, leading to instability and
a lack of robustness. One potential solution to address this issue is to use
high-ranked positive images during the training process. These images,
although less similar to the query images, can provide valuable and diverse
information that is often overlooked. To address this, we propose a pyramid
self-supervised strategy that allows us to extract accurate information from
scale-wise image patches, taking advantage of the diverse information provided
by high-ranked positive images. More precisely, we split and merge the input
images into scale-wise patches and generate local representations. We compare
the local representations between the input images to compute scale-wise
similarity scores. These scores are then used to refine the pyramid self-
supervision, enabling the model to learn and capture the variations and
details present in the image patches at different scales. In this strategy,
the training process is divided into multiple generations to gradually improve
the accuracy of the scale-wise similarity scores.

The major innovations and contributions of this paper are as follows:

  * •

We introduce ClusVPR, a novel approach that aggregates clustering-based
weighted feature maps into a compact and discriminative representation. This
approach enhances global context inference and improves the processing of
unequally important image regions. Additionally, our OptLAD layer reduces
model complexity and enhances robustness to variations and noise.

  * •

A pyramid self-supervised strategy is designed to extract more representative
and diverse information from scale-wise image patches. This approach avoids
learning from entire images and instead focuses on extracting information from
each scale level.

The remainder of the paper is organized as follows: we first discuss the
related works in Section II. Then, we detail the proposed ClusVPR and pyramid
selfsupervised strategies in Section III. In Section IV, we provide the
implementation details and experimental results. Finally, in Section V we
conclude the paper.

##  II Related work

###  II-A Recent progress on visual place recognition (VPR)

VPR aims to establish a mapping between an image and its corresponding spatial
location. In VPR, the goal is to find the geographical location of a query
image, and the predicted coordinates are considered correct if they are close
to the ground truth position. Early VPR approaches [20, 21] relied on
classical techniques like SIFT [22] and SURF [23] for detecting keypoints.
However, these handcrafted features are not suitable for dealing with the
large variations in appearance that are encountered in complex scenes. In
recent years, CNN-based methods have demonstrated superior performance in VPR
[13, 11, 16]. Among deep learning representation methods, NetVLAD [1] has
emerged as a highly effective technique for VPR tasks. NetVLAD is a
differentiable implementation of VLAD that is trained end-to-end with a CNN
backbone for direct place recognition. It has been widely adopted in various
works [9, 11, 24, 14]. Task-specific patch-level features have also been
explored for VPR [25, 9].

The Patch-NetVLAD [11] introduces the use of the NetVLAD to extract
descriptors from predefined image patches. Indeed, patch-level descriptors
typically referred to local descriptors, encoding content from local patches.
There are several researches on developing more compact descriptors, either
through dimensionality reduction techniques [26, 25] or by replacing NetVLAD
with lighter pooling layers such as GeM [27] and R-MAC [28]. Recently, Berton
et al. [29] introduced a benchmark for VPR, providing a standardized framework
to evaluate various global-retrieval-based methods. This benchmark allows for
a fair comparison between different approaches in the field. In [12], a
generalized contrastive Loss (GCL) is developed for training networks using
graded similarity labels and [30] considers knowledge distillation to build an
efficient VPR method.

To summarize, the existing models that use the entire image for training often
lead to inaccurate global representation learning. To address this issue, our
proposed method leverages a pyramid self-supervised strategy, which involves
using scale-wise image patches instead of the entire image. This approach
enables the model to capture diverse and representative information during the
training process.

Figure 2: The pipeline of the training architecture. Our model consists of a
CNN backbone and the proposed ClusVPR.

###  II-B Transformer-based Approaches

The Transformer [31] was initially proposed for Natural Language Processing
tasks and later introduced to vision tasks as the ViT [7]. In ViT, each image
patch is treated as a token, allowing the model to process images using a
sequence-based approach. The vanilla ViT requires large-scale training
datasets, such as ImageNet [28], to achieve comparable results to CNNs [5].
DeiT [32], on the other hand, introduces a data-efficient training strategy
for ViT that outperforms CNNs on the average-scale datasets. The vanilla ViT
has been adopted in the recent VPR benchmarks, demonstrating competitive
performance on global retrieval tasks. [15] applied ViT to image retrieval
tasks by utilizing the [class] token from the final layer as a global feature.
[33] takes a different approach by incorporating multiple transformer layers
while still relying on a CNN-based backbone feature extractor. DELG [25] is a
framework that extracts both local and global features. However, to improve
the representation of features, HAF [34] is introduced which consist of a
hierarchical attention network. DASGIL [35] presents a multi-task architecture
with a shared encoder to generate global representation. Building upon this,
[36] focuses on filtering semantic information using an attention mechanism.
These approaches while promising, but they often overlook the specific
challenges faced in VPR tasks, such as the varying importance of image regions
due to duplicate regions and ignored small objects. In our ClusVPR, we tackle
this challenge by introducing a novel approach that computes weights for image
tokens using the k-nearest neighbor (KNN). This enables us to reduce visual
deviations and enhance the accuracy of VPR.

##  III Proposed Model

This section presents a detailed description of our ClusVPR model. The
training architecture is shown in Fig 2.

###  III-A CWTNet of the ClusVPR

To address the issue of deviations in VPR tasks caused by duplicate regions
and overlooked small objects, we introduce CWTNet, which computes clustering-
based weighted feature maps. CWTNet divides channels into two parallel
branches, namely the local and global branches. The local branch uses depth-
wise convolution layers to extract local features and provides positional
information for our Clustering-based Weighted Transformer (CWT), because
$3\times 3$ convolutions can provide sufficient positional information [37],
eliminating the need for positional embedding as used in ViTs. The global
branch uses the proposed CWT to obtain global context. The local and global
features are then fused through feature concatenation to generate the final
output. The structure of the CWTNet is shown in Fig 2.

ViTs [7, 32] typically take a sequence of 1D token embeddings as input. Given
a feature map $\textbf{m}\in{\bf R}^{C\times H\times W}$, we first split it
into a number of individual feature patches $\textbf{m}_{p}\in{\bf R}^{N\times
P^{2}\cdot C}$, followed by their subsequent down-sampling into corresponding
feature vectors $\textbf{x}_{p}\in{\bf R}^{N\times C}$. Here, $(H,W)$
represents the resolution of the feature map, $C$ denotes the number of
channels, $P$ is the down-sampling rate, and $N=HW/P^{2}$ represents the total
number of token embeddings. Subsequently, $\textbf{x}_{p}$ is used for the
input token embeddings. We introduce a clustering-based weighted module that
uses the KNN to calculate the weights for the input token embeddings
$\textbf{x}_{p}$. Given a set of token embeddings $\textbf{x}_{p}$, the
clustering density $\rho_{i}$ of each token embedding $x_{i}$ is computed
through its $k_{n}$-nearest neighbors:

$d_{i}=\sum_{x_{j}\in\text{KNN}(x_{i})}{\|x_{i}-x_{j}\|}^{2}_{2},$ (1)
$\rho_{i}=\text{exp}(-\frac{d_{i}-d_{min}}{k(d_{max}-d_{min})}),$ (2)

where $x_{i}$ and $x_{j}$ are the corresponding token embeddings, and
$\text{KNN}(x_{i})$ represents the $k_{n}$-nearest neighbors of $x_{i}$.
$d_{i}$ is the sum of distances between $x_{i}$ and its $k_{n}$-nearest
neighbors. The weight of each token embedding $x_{i}$ is computed by

$w_{i}=\frac{1-\rho_{i}}{\sum_{j=1}^{N}(1-\rho_{j})},$ (3)
$w_{i}=\frac{w_{i}-w_{min}}{w_{max}-w_{min}},$ (4)

where $w_{i}$ denotes the potential value of each token embedding. We use ViT
[7] as the basis for the CWT, but we remove the Layer Normalization (LN) to
enable our clustering-based weighted module to operate more effectively.

In the clustering-based weighted multi-head self-attention (CMSA), we use the
clustering-based weights to compute the self-attention head. Given the token
embeddings $\textbf{x}_{p}$ as the input sequence of the CWT, a single self-
attention head can be represented by

$\textbf{Q}=\textbf{x}_{p}\textbf{W}_{q},\
\textbf{K}=\textbf{x}_{p}\textbf{W}_{k},\
\textbf{V}=\textbf{x}_{p}\textbf{W}_{v},$ (5)
$\textbf{A}=\text{softmax}(\frac{\textbf{Q}\textbf{K}^{T}}{\sqrt{D_{k}}})((\lambda_{c}\textbf{I}_{v}+\textbf{W}_{c})\textbf{V}).$
(6)

here $\textbf{W}_{q}$, $\textbf{W}_{k}$, $\textbf{W}_{v}$ are linear
projection matrices, and Q, K, V represent the query, key, and value matrices.
$D_{k}$ denotes the channel quantity of the token embeddings, and
$\textbf{I}_{v}$ is an identity matrix with the same rank as V.
$\textbf{W}_{c}$ is a diagonal matrix whose main diagonal elements are the
clustering-based weights $w_{i}$. Finally, $\lambda_{c}$ is a scaling factor,
which we set to $0.5$. Thus, our CWT is formulated as follows:

$\textbf{z}_{p}=\text{CMSA}(\textbf{x}_{p})+\textbf{x}_{p},\
\textbf{x}_{p}=\text{DS}(\textbf{m}_{p}),$ (7)
$\textbf{z}^{\prime}_{p}=\text{MLP}(\textbf{z}_{p})+\textbf{z}_{p},$ (8)
$\textbf{m}_{\text{out}}=\text{US}(\textbf{z}^{\prime}_{p})+\textbf{m}_{p}.$
(9)

where the CWT consists of a down-sampling layer (DS), a CMSA module, a multi-
layer perceptron (MLP), and an up-sampling layer (US). The DS is implemented
using average pooling. The MLP comprises a GELU [38] activation layer and two
linear layers. The US is implemented using depthwise separable transposed
convolution, and the output sequence $\textbf{m}_{\text{out}}$ is reshaped
into a clustering-based weighted feature map $\textbf{m}_{c}\in{\bf
R}^{C\times H\times W}$.

###  III-B OptLAD of the ClusVPR

Considering that the high dimensionality of global descriptors obtained by the
standard NetVLAD is unsuitable for large-scale VPR tasks, a PCA layer with
whitening [39] is adopted to reduce the dimensionality to a suitable value
$N^{\prime}$. However, this approach requires a large number of parameters.
For example, for aggregating a feature map with $1024$ channels through $128$
clusters, using a PCA layer with a $4096$-dimension output vector would
require about $537$M parameters. This makes it difficult to deploy the model
on resource-constrained mobile devices. To address this issue, we reduce the
dimension of global descriptors, thereby reducing the number of parameters
required by the PCA layer. This is achieved by splitting the input vector into
uniform clusters of relatively low-dimensional vectors prior to executing the
VLAD operation. Furthermore, a group weight $\beta_{g}$ calculated by GeM [27]
is introduced to improve the non-linearity ability of the OptLAD.

Figure 3: The OptLAD structure. The primary parameters for each layer are
indicated in brackets.

As Fig 3 illustrates the details, the input feature map $\textbf{m}_{c}\in{\bf
R}^{C\times H\times W}$ has $D$ $C$-dimensional pixel-level descriptors
($D=HW$). The pixel-level descriptors are first expanded by $\lambda$ via a
$1\times 1$ convolution. Then, the input descriptors
$\\{\hat{x}_{i}\\},i=1,\cdots,D$ are split into G uniform groups. The low-
dimensional descriptors are shown by $\\{\hat{x}_{gi}\\},g=1,\cdots,G$. Given
a set of cluster centers for each group $\\{c_{gk}\\},k=1,\cdots,K$, the
output of the global descriptor $V$ is represented as a $\frac{\lambda
CK}{G}$-dimension vector. To better understand the output of $V$, we can also
express it as a $\frac{\lambda C}{G}\times K$ matrix. Therefore, the element
(j, k) of $V$ can be computed by

$V(j,k,g)=\sum_{i=1}^{D}\alpha_{k}(\hat{x}_{gi})(\hat{x}_{gi}(j)-c_{gk}(j)),$
(10) $V(j,k)=\sum_{g=1}^{G}\beta_{g}(\hat{x})V(j,k,g),$ (11)

where $\hat{x}_{gi}(j)$ is the $j$-th dimension of the low-dimensional
descriptor $\hat{x}_{gi}$. $c_{gk}(j)$ is the $j$-th dimension of the cluster
center $c_{gk}$. $\alpha_{k}$ is used to find the proximity between the lower-
dimensional descriptor $\hat{x}_{gi}$ and the cluster center $c_{gk}$. The
$\beta_{g}$ is used to merge the output of each group.

$\alpha_{k}(\hat{x}_{gi})=\frac{\text{exp}(w_{gk}^{T}\hat{x}_{gi}+b_{gk})}{\sum_{k^{\prime}=1}^{K}\text{exp}(w_{gk^{\prime}}^{T}\hat{x}_{gi}+b_{gk^{\prime}})},$
(12)
$\beta_{g}(\hat{x})=\sigma\left(\frac{1}{D}\sum_{i=1}^{D}\left(\frac{1}{D^{\prime}}\sum_{j=1}^{D^{\prime}}\hat{x}_{gi}(j)^{p_{i}}\right)^{\frac{1}{p_{i}}}\right).$
(13)

where $D^{\prime}=\frac{\lambda C}{G}$ is the dimension of $\hat{x}_{gi}$, and
$\sigma(\cdot)$ indicates the sigmoid function. The first weight
$\alpha_{k}(\hat{x}_{gi})$, represents the distance between each pixel-level
descriptor $\hat{x}_{gi}$ and the cluster center $c_{gk}$, while the second
weight $\beta_{g}(\hat{x})$, is a scaling coefficient across groups.

The matrix $V$ is obtained by Intra-Normalization [40] and then re-converted
into a $\frac{\lambda CK}{G}$-dimensional vector, which is L2-normalized to
obtain a compact global descriptor. Finally, we apply a PCA layer to reduce
the dimension of the global descriptor to a proper value $N^{\prime}$.
Compared to the standard NetVLAD, the OptLAD with PCA requires approximately
$\frac{\lambda}{G}$ times fewer parameters.

###  III-C Pyramid self-supervised strategy

The VPR datasets [41, 13] lack fine-grained annotations and only provide GPS
locations. Recent works [18, 11] have adopted a simplified evaluation
protocol, considering only the top-ranked positive image as the ground truth.
During training, these models use the standard triplet loss [19], which
encourages the feature representation of a query image to be closer to its
top-ranked positive image. However, due to the limited information in top-
ranked positive images, this approach lacks robustness. Instead, we use high-
ranked positives to gather comprehensive information from scale-wise image
patches.

In particular, for a query image $q$ and a high-ranked positive image $p^{l}$,
the feature maps
$\\{\textbf{m}_{q}^{\theta_{0}},\textbf{m}_{p^{l}}^{\theta_{0}}\\}$ are
extracted from the CNN backbone, where $\theta_{0}$ represents the network’s
initial parameters. To obtain scale-wise patches, we carry out feature maps
$\\{\textbf{m}_{q}^{\theta_{0}},\textbf{m}_{p^{l}}^{\theta_{0}}\\}$ splitting
and merging operations as evidently depicted in Fig 4. Then, we feed the
resulting feature maps
$\\{\textbf{m}_{q_{1}}^{\theta_{0}},\cdots,\textbf{m}_{q_{4}}^{\theta_{0}},\textbf{m}_{p^{l}_{1}}^{\theta_{0}},\cdots,\textbf{m}_{p^{l}_{22}}^{\theta_{0}}\\}$
separately to the ClusVPR to generate local representations
$\\{f_{q_{1}}^{\theta_{0}},\cdots,f_{q_{4}}^{\theta_{0}},$
$f_{p^{l}_{1}}^{\theta_{0}},\cdots,f_{p^{l}_{22}}^{\theta_{0}}\\}$. To compute
similarity between $q$ and $p^{l}$, we calculate pyramid similarity scores by

$\displaystyle\mathcal{S}_{\theta_{0}}(\tau_{0})=\text{softmax}([\langle
f_{q_{1}}^{\theta_{0}},f_{p^{l}_{1}}^{\theta_{0}}\rangle/\tau_{0},\cdots,\langle
f_{q_{1}}^{\theta_{0}},f_{p^{l}_{22}}^{\theta_{0}}\rangle/\tau_{0},$ (14)
$\displaystyle\cdots,\langle
f_{q_{4}}^{\theta_{0}},f_{p^{l}_{1}}^{\theta_{0}}\rangle/\tau_{0},\cdots,\langle
f_{q_{4}}^{\theta_{0}},f_{p^{l}_{22}}^{\theta_{0}}\rangle/\tau_{0}]),$

where $\tau_{0}$ is a hyperparameter that influences the level of smoothness
in the pyramid similarity scores $\mathcal{S}_{\theta_{0}}$, and
$\langle\cdot,\cdot\rangle$ represents the inner product operation.

Figure 4: The proposed pyramid self-supervised strategy and how scale-wise
patches are split and merged, where the green borders indicate high similarity
with the query patch while the red borders denote low similarity. The pyramid
similarity scores are progressively updated during training.

We enhance pyramid similarity scores through $\omega$ generations, as
illustrated in Fig 4. The $(\upsilon-1)$-th generation’s pyramid similarity
$\mathcal{S}_{\theta{\upsilon-1}}$ supervises the $\upsilon$-th network using
Kullback-Leibler (KL) divergence. The pyramid similarity loss for a high-
ranked positive image $p^{l}$ is written as:

$\mathcal{L}_{s}^{\theta_{\upsilon}}(q,p^{l})=\ell_{kl}(\mathcal{S}_{\theta_{\upsilon}}(1),\mathcal{S}_{\theta_{\upsilon-1}}(\tau_{\upsilon-1})).$
(15)

where $\theta_{\upsilon},\upsilon=1,\cdots$ and $\omega$ are the parameters
associated with the $\upsilon$-th generation network. $\ell_{kl}$ represents
the KL divergence, which can be written as
$\ell_{kl}(y,\hat{y})=\sum_{i}\hat{y}(i)\log(\hat{y}(i)/y(i))$. The target
similarity vector $\mathcal{S}_{\theta_{\upsilon-1}}$ is generated using the
softmax, with a temperature coefficient $\tau_{\upsilon-1}$ to control its
smoothness. A larger temperature coefficient leads to a more uniformly
distributed similarity vector, meaning that the pyramid similarity loss
focuses on more matching pairs. So, in the early generations,
$\tau_{\upsilon-1}$ is set to a larger value.

TABLE I: The precision and complexity comparison of our model on four VPR benchmarks. Method | MSLS val | MSLS chall | Pitts30k | Pitts250k | Tokyo 24/7 | TokyoTM | Model Complexity
---|---|---|---|---|---|---|---
R@1 | R@5 | R@1 | R@5 | R@1 | R@5 | R@1 | R@5 | R@1 | R@5 | R@1 | R@5 | Params(M) | FLOPs(G)
NetVLAD [13] | 70.1 | 80.8 | 41.4 | 53.7 | 85.6 | 92.9 | 86.0 | 93.2 | 73.3 | 82.9 | 93.9 | 96.8 | 148.97 | 94.35
SARE [18] | 76.4 | 81.6 | 44.3 | 55.8 | 87.2 | 93.8 | 89.0 | 95.5 | 79.7 | 86.7 | 94.5 | 96.7 | 148.97 | 94.35
SFRS [9] | 77.2 | 82.5 | 45.8 | 56.1 | 89.4 | 94.7 | 90.7 | 96.4 | 85.4 | 91.1 | 95.8 | 98.2 | 148.97 | 94.35
HAF [34] | 78.5 | 85.9 | 47.3 | 56.3 | 86.6 | 93.7 | 89.4 | 95.8 | 78.2 | 84.4 | 94.8 | 97.8 | 158.87 | 1791.27
Patch-NetVLAD [11] | 79.5 | 86.2 | 48.1 | 57.4 | 88.7 | 94.5 | 89.8 | 95.9 | 86.0 | 88.6 | 95.2 | 97.8 | 148.97 | 94.22
GPM [14] | 80.3 | 86.8 | 48.5 | 58.6 | 89.2 | 94.6 | 91.2 | 96.7 | 85.4 | 89.9 | 96.3 | 98.2 | 151.34 | 95.24
GCL-NetVLAD [12] | 78.0 | 85.1 | 47.6 | 57.4 | 88.9 | 93.8 | 91.5 | 97.1 | 85.7 | 90.2 | 96.2 | 98.0 | 148.97 | 94.35
ClusVPR | 82.7 | 88.5 | 49.4 | 58.2 | 90.8 | 95.2 | 92.4 | 96.9 | 87.5 | 91.5 | 96.8 | 98.7 | 53.12 | 92.14

Our training is supervised using a triplet loss, similar to [34, 11].
Specifically, each triplet consists of a query image $q$, a corresponding top-
ranked positive image $p^{*}$, and a set of high-ranked negative images
($n_{i}|_{i=1}^{M}$), where the rank of positive or negative images is
determined by the Euclidean distance between their global representations.
However, the standard triplet loss [19] lacks robustness and relies on the
proper selection of negative images. Hence, a softmax-based triplet loss is
used to notably increase the distinction among the positive pair and a set of
negative pairs, as given below,

$\mathcal{L}_{t}^{\theta}(q,p^{*},n)=\sum_{i=1}^{M}-\log\frac{\exp\langle
f^{\theta}_{q},f^{\theta}_{p^{*}}\rangle}{\exp\langle
f^{\theta}_{q},f^{\theta}_{p^{*}}\rangle+\exp\langle
f^{\theta}_{q},f^{\theta}_{n_{i}}\rangle},$ (16)

where $\theta$ is the network’s parameter, and $\epsilon$ is the margin
between positive and negative pairs. The $p^{*}$ is chosen as the top-ranked
images of the gallery that located within 10 meters from the $q$, while
$\\{n_{i}|_{i=1}^{M}\\}$ represents a set of randomly chosen gallery images
from the top 500, located at a distance of 25 meters away from $q$.

Therefore, our model is supervised by the pyramid similarity loss in
conjunction with the softmax-based triplet loss. Given $K$ high-ranked
positive images $\\{p^{k}|_{k=1}^{K}\\}$, the total loss can be computed by

$\mathcal{L}_{total}^{\theta_{\upsilon}}=\mathcal{L}_{t}^{\theta_{\upsilon}}(q,p^{*},n)+\lambda_{s}\sum_{k=1}^{K}\mathcal{L}_{s}^{\theta_{\upsilon}}(q,p^{k}).$
(17)

where $\lambda_{s}$ is the loss scaling coefficient.

##  IV Experiments

This section evaluates the efficacy of our model on both VPR and image
retrieval datasets compared to several state-of-the-art models [13, 18, 9, 34,
11, 14, 12]. Specifically, we show the details of datasets, implementations,
and quantitative/qualitative results.

###  IV-A Datasets

Our model is evaluated on four VPR datasets – the MSLS [42],
Pitts30k/250k-test [43], Tokyo 24/7 [41], and TokyoTM-val [13]. These datasets
include GPS labels, and images have varying appearances and perspectives. Our
model’s performance is evaluated based on the recommended train-test settings
of the datasets. To further demonstrate the generalization ability, we also
assessed our model on three image retrieval datasets – the Paris 6K [44],
Oxford 5K [45] and the Holidays [46].

###  IV-B Implementation details

####  IV-B1 Model settings

The base ClusVPR contains four CWTNets for computing clustering-based weighted
feature maps. For each CWTNet, the patch size on the feature map is set as
$2\times 2$, the down-sampling rate $P$ is $2$, and the number of nearest
neighbors $k_{n}$ is set as $10$. For the OptLAD, the number of cluster
centers $K$ is $64$, and we set $\lambda=2,G=8$. The dimension $N^{\prime}$ of
the global representation is $4096$. For a fair comparison, we use the VGG-16
backbone for feature extraction, similar to the other works.

####  IV-B2 Training settings

In our experiments, we train the model on Pitts30k-train and test it on
Pitts30/250k-test, Tokyo 24/7-test, and TokyoTM-val, following standard VPR
procedures. Additionally, we train the model on MSLS training set, and test it
on MSLS validation and Challenge sets. The training consists of five
generations, each comprising eight epochs. We optimize the loss function using
stochastic gradient descent (SGD) with a learning rate of 0.0001, weight decay
of 0.001, and momentum of 0.9. We conducted a grid search to find the optimal
hyperparameters. The loss scaling coefficient $\lambda_{s}$ is set to 0.55,
and the temperature coefficient $\tau_{\upsilon}$ is set to 0.06.

###  IV-C Comparison with the state-of-the-arts

ClusVPR is compared with seven models– NetVLAD [13], SARE [18], SFRS [9], HAF
[34], Patch-NetVLAD [11], GPM [14], and GCL [12] with VGG-NetVLAD. We ensure a
fair comparison by keeping the training set, cluster quantity, and dimension
of global representations consistent with those used in other models.

####  IV-C1 VPR benchmarks

Table I represents the results of our ClusVPR model in comparison with other
baselines on the VPR benchmarks. The table reports precision-recall and model
complexity metrics. The number of parameters (Params) includes parameters of
the entire model. The FLOPs are computed based on an input image size (640,
480). The accuracy is measured using precision-recall. The top-$k$ recall, on
the other hand, represents the percentage of query images that are correctly
retrieved from the top-$k$ ranked gallery images. The results presented in
Table I demonstrate that our model outperforms other baselines on most
benchmarks. Notably, our model surpasses the GPM, improving rank-1 recall by
2.4%, 0.9%, 1.6%, 1.2% 2.1% and 0.5%, respectively. Moreover, our model
generates only around one-third of the parameters produced by other models,
indicating its efficient use of parameters.

####  IV-C2 Image retrieval benchmarks

Table II presents the mean Average Precision (mAP) results of our model
compared to other state-of-the-art models on image retrieval benchmarks. In
this experiment, all models are trained on Pitts30K and tested without any
fine-tuning. For the Oxford 5K and Paris 6K, the models are evaluated on both
full and cropped query images. The ”Full” setting indicates that the entire
image is taken as a query, while for ”Crop”, only the landmark region is used.
For the Holidays dataset, the query images are used directly.

TABLE II: Comparison of Mean Average Precision (mAP) on three image retrieval benchmarks. Method | Oxford 5K | Paris 6K | Holidays
---|---|---|---
full | crop | full | crop
NetVLAD [13] | 69.1 | 71.6 | 78.5 | 79.7 | 83.0
SARE [18] | 71.7 | 75.5 | 82.0 | 81.1 | 80.7
SFRS [9] | 73.9 | 76.7 | 82.5 | 82.4 | 80.5
HAF [34] | 71.9 | 75.8 | 82.3 | 81.5 | 81.4
Patch-NetVLAD [11] | 72.6 | 76.1 | 82.3 | 81.9 | 81.7
GPM [14] | 73.3 | 76.4 | 82.8 | 82.3 | 82.3
GCL-NetVLAD [12] | 73.5 | 76.7 | 82.7 | 82.5 | 82.5
ClusVPR | 74.6 | 77.9 | 84.2 | 83.4 | 83.8

The results demonstrate that ClusVPR is effective in improving the performance
of image retrieval. In particular, our model surpasses the performance of
other models on all datasets, showing improvements of 1.1%, 1.2%, 1.5%, 0.9%
and 1.3% respectively, compared to GCL.

###  IV-D Qualitative Evaluation

The qualitative results of NetVLAD, GPM and our model on challenging cases
with complex environments and varying lighting conditions can be seen in Fig
5. It’s important to note that we used the feature maps before the VLAD
operation for all test models and followed the approach detailed in [47] to
produce the attention maps. In the first four cases, our model attends to
discriminative regions such as signs and buildings, while the other two models
wrongly focus on dynamic objects or obstacles such as pedestrians, lights,
cars, and trees. In particular, in the third and fourth samples, where there
are lighting issues and occlusions, our model successfully conducts matching
based on discriminative landmarks while disregarding dynamic objects or
barriers.

Figure 5: Visualization results of challenging cases. The attention maps
overlaid on the images highlight the regions of interest identified by the
models. Green and red borders around the top-1 retrieved images indicate
successful or unsuccessful retrieval results, respectively. Figure 6: The
visualization of CNN-based feature maps (CNN FMs) and clustering-based
weighted feature maps (CWFMs) generated from the query images and their top-1
retrieved gallery images.

However, the other two models seem to be focusing more on lights and cars,
which are dynamic objects and obstacles that can appear or disappear in any
gallery image. This could result in inaccurate retrieval results due to the
distortions caused by such objects. In the last two challenging cases of Fig
5, where the query image contains complex scenes and lighting conditions, all
models retrieved the wrong top-1 gallery images. However, our model still
focuses on discriminative regions and retrieves a gallery image with a similar
building structure.

To demonstrate the efficacy of the ClusVPR in addressing the unequal
importance of image regions in VPR and improving global context inference, we
conducted a comparison between CNN-based feature maps (the outputs of the CNN
backbone) and clustering-based weighted feature maps (the outputs of the
CWTNet). We selected two daytime and two nighttime query images from the
Pitts30k-test and Tokyo 24/7 and show the results in Fig 6. The comparison
demonstrates that the CWTNet of ClusVPR eliminates redundancy from duplicate
regions and enhances discriminative information from small objects.

###  IV-E Ablation experiments

Several ablation experiments are conducted to evaluate the effectiveness of
our ClusVPR and pyramid self-supervised strategy (PSS). The experiments aimed
to evaluate the effectiveness of different components in our model and compare
their performance.

TABLE III: Ablation experiments of ClusVPR and pyramid self-supervised strategy (PSS). Method | MSLS val | MSLS chall | Pitts250k | Tokyo 24/7
---|---|---|---|---
R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
NetVLAD + V-Triplet | 70.9 | 80.9 | 83.8 | 41.8 | 53.9 | 56.1 | 86.5 | 93.9 | 95.8 | 75.3 | 83.9 | 87.4
V-ClusVPR + V-Triplet | 74.2 | 81.3 | 84.6 | 44.6 | 55.8 | 56.9 | 88.4 | 95.4 | 96.4 | 79.9 | 87.1 | 90.5
ClusVPR + V-Triplet | 75.3 | 81.8 | 85.9 | 44.5 | 56.1 | 57.2 | 88.9 | 95.6 | 96.7 | 80.1 | 87.0 | 90.7
NetVLAD + S-Triplet | 76.4 | 81.7 | 85.1 | 44.6 | 55.9 | 56.7 | 89.1 | 95.6 | 96.8 | 80.4 | 87.3 | 90.8
V-ClusVPR + S-Triplet | 79.0 | 84.9 | 90.4 | 46.3 | 56.6 | 60.2 | 89.9 | 96.0 | 96.9 | 83.7 | 89.3 | 91.9
ClusVPR + S-Triplet | 79.8 | 85.1 | 90.3 | 47.2 | 56.9 | 60.4 | 90.2 | 95.9 | 97.0 | 84.6 | 89.9 | 92.2
NetVLAD + PSS | 77.4 | 82.9 | 85.8 | 46.1 | 56.4 | 58.2 | 90.5 | 96.2 | 97.1 | 85.1 | 90.3 | 92.9
V-ClusVPR + PSS | 81.6 | 87.8 | 91.6 | 48.9 | 58.1 | 62.0 | 91.7 | 96.6 | 97.6 | 86.7 | 91.1 | 93.6
ClusVPR + PSS | 82.7 | 88.5 | 92.4 | 49.4 | 58.2 | 62.3 | 92.4 | 96.9 | 97.6 | 87.5 | 91.5 | 93.9

The results of ablation experiments on the Pitts250k and Tokyo 24/7 are
presented in Table III. It is important to note that V-ClusVPR in our
experiments refers to the ClusVPR with the standard NetVLAD instead of OptLAD.
S-Triplet denotes softmax-based triplet loss, whereas V-Triplet denotes
standard triplet loss. The following observations can be drawn from the
experiments.

Softmax-based triplet loss shows superior efficacy compared to the standard
triplet loss and can accelerate the convergence. Specifically, on the
Pitts250k, the models with S-Triplet outperform the models with V-Triplet,
with improvements on rank-1 recall of 2.6%, 1.5%, and 1.3% respectively.

ClusVPR successfully addresses the issue of unequal importance of image
regions in VPR tasks and enhances the global reasoning ability of the model to
encode geometric configurations between images. The results on the challenging
Tokyo 24/7 show that the models with ClusVPR achieve significantly better
rank-1 recall compared to models with NetVLAD, with improvements of 4.8%,
4.2%, and 2.4% respectively, reaching 80.1%, 84.6%, and 87.5%.

OptLAD improves the precision-recall performance of our model compared to
those with NetVLAD, while having a significantly lower parameter count of
around one-fourth. For example, the models with OptLAD outperform those with
NetVLAD on the MSLS val, with improvements on rank-1 recall of 1.1%, 0.8%, and
1.1% respectively.

Pyramid self-supervised strategy has been shown to effectively improve
performance by exploiting scale-wise image information. On the Tokyo 24/7, the
models with PSS outperform those with S-Triplet, resulting in an improvement
on rank-1 recall of 4.7%, 3.0%, and 2.9%, respectively.

The impact of different down-sampling methods in the CWTNet is evaluated
through an ablation experiment. Specifically, We compared the performance of
our model with three different down-sampling methods (Average Pooling, Max
Pooling, and Center Pooling), as shown in Table IV. The results indicate that
adopting the average pooling achieved the best precision on both the Pitts250k
and Tokyo 24/7, with a marginal improvement of 0.6% and 0.8% on rank-1 recall
compared to the second-best method, respectively. This implies that the
average pooling is more effective in preserving the information from different
regions of the image.

TABLE IV: Results of the ablation experiment on the down-sampling methods. Method | Pitts250k | Tokyo 24/7
---|---|---
R@1 | R@5 | R@10 | R@1 | R@5 | R@10
Average Pooling | 92.4 | 96.9 | 97.6 | 87.5 | 91.5 | 93.9
Max Pooling | 91.8 | 96.6 | 97.6 | 86.7 | 90.9 | 93.3
Center Pooling | 91.1 | 96.4 | 97.3 | 85.9 | 90.2 | 92.7
Figure 7: Qualitative matching results of different models. The models are
evaluated on day, night, and rotated image pairs. Green lines indicate correct
matches.

###  IV-F Keypoints detection and matching

To further evaluate the effectiveness of our model, we conducted an experiment
using the DFM [48] pipeline by replacing the VGG backbone with our pre-trained
model. Since we only want to evaluate the performance of our pre-trained
backbone in the DFM pipeline, we did not fine-tune it again. Instead, we
simply replaced the feature extraction part with our pre-trained backbone and
evaluated its performance on day, night, and rotated image pairs, as shown in
Fig 7. We compared the results with those of three strong baseline models–
DFM, GIFT [49], and Superpoint [50]. The results demonstrate that our model
produces more accurate and dense matches than the other three methods,
indicating high generalization performance.

###  IV-G Results with different CNN backbones

To investigate whether using stronger backbones can improve the performance of
our model on VPR datasets, we conducted additional experiments by employing
VGG-19 [51] and ResNets-101/152 [5] as the backbones. We compare the results
obtained with these backbones to our prior results on the VPR datasets.

Figure 8: Recalls at N-top retrievals for our ClusVPR with different CNN
backbones.

The precision-recall curves of the models with different CNN backbones are
displayed in Fig 8. Table V shows the model complexity and detailed precision-
recall of the models. The results indicate that models with stronger backbones
have less than 0.5% rank-1 recall improvement compared to our ClusVPR with
VGG-16. The advantages of utilizing stronger backbones are more evident in the
Tokyo 24/7 dataset. For example, the model with ResNet-152 achieves 92.2%
rank-5 recall, a 0.7% accuracy gain over ClusVPR with VGG-16, at the cost of
44.5M additional parameters. Consequently, after carefully evaluating the
results of ClusVPR with different backbones, we conclude that using more
robust and stronger backbones can yield marginal improvements in the model’s
performance, but at the cost of additional complexity.

TABLE V: Model complexity and precision-recall of our ClusVPR with different CNN backbones on the Pitts250k and Tokyo 24/7 test sets. Method | Pitts250k | Tokyo 24/7 | Model Complexity
---|---|---|---
R@1 | R@5 | R@10 | R@15 | R@20 | R@25 | R@1 | R@5 | R@10 | R@15 | R@20 | R@25 | Params(M) | FLOPs(G)
ClusVPR-VGG-16 | 92.4 | 96.9 | 97.6 | 98.0 | 98.2 | 98.3 | 87.5 | 91.5 | 93.9 | 94.1 | 94.4 | 94.7 | 53.12 | 92.14
ClusVPR-VGG-19 | 92.5 | 96.9 | 97.7 | 97.9 | 98.3 | 98.4 | 87.6 | 91.4 | 94.0 | 94.2 | 94.5 | 94.8 | 58.46 | 121.97
ClusVPR-ResNet-101 | 92.1 | 96.9 | 97.7 | 98.1 | 98.3 | 98.5 | 87.3 | 91.4 | 94.0 | 94.3 | 94.5 | 94.8 | 81.98 | 50.61
ClusVPR-ResNet-152 | 92.5 | 96.9 | 97.7 | 98.2 | 98.3 | 98.6 | 87.8 | 92.2 | 94.3 | 94.7 | 94.7 | 94.9 | 97.62 | 73.49

##  V Conclusions

This paper proposed a novel ClusVPR model and CWTNet architecture that
improves the global representation of images for VPR tasks, addressing the
problem of the unequal importance of image regions. The CWTNet calculates
weights for image tokens and encodes global dependencies, allowing our model
to encode corresponding geometric configurations between images. The proposed
OptLAD layer enhances the efficiency of the model and outperforms the standard
NetVLAD. Additionally, a pyramid self-supervised strategy is introduced to
extract more precise information from scale-wise image patches. The extensive
experiments demonstrate the superior performance of our proposed model in
terms of qualitative, quantitative, and efficiency evaluations.

## References

  * [1] R. Arandjelović, P. Gronat, A. Torii, T. Pajdla, and J. Sivic, “Netvlad: Cnn architecture for weakly supervised place recognition,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 40, no. 6, pp. 1437–1451, 2018.
  * [2] B. Fan, Y. Yang, W. Feng, F. Wu, J. Lu, and H. Liu, “Seeing through darkness: Visual localization at night via weakly supervised learning of domain invariant features,” _IEEE Trans. Multimedia_ , 2022.
  * [3] A. Khaliq, S. Ehsan, Z. Chen, M. Milford, and K. McDonald-Maier, “A holistic visual place recognition approach using lightweight cnns for significant viewpoint and appearance changes,” _IEEE Trans. Robot._ , 2020.
  * [4] C. Häne, L. Heng, G. H. Lee, F. Fraundorfer, P. Furgale, T. Sattler, and M. Pollefeys, “3d visual perception for self-driving cars using a multi-camera system: Calibration, mapping, localization, and obstacle detection,” _Image Vis. Comput._ , 2017.
  * [5] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2016.
  * [6] C.-C. Hsu and C.-W. Lin, “Cnn-based joint clustering and representation learning with feature drift compensation for large-scale image data,” _IEEE Trans. Multimedia_ , 2018.
  * [7] A. Dosovitskiy, L. Beyer, A. Kolesnikov, and D. Weissenborn, “An image is worth 16x16 words: Transformers for image recognition at scale,” in _Int. Conf. Learn. Represent._ , 2021.
  * [8] X. Lin, S. Sun, W. Huang, B. Sheng, P. Li, and D. D. Feng, “Eapt: Efficient attention pyramid transformer for image processing,” _IEEE Trans. Multimedia_ , 2021.
  * [9] Y. Ge, H. Wang, F. Zhu, R. Zhao, and H. Li, “Self-supervising fine-grained region similarities for large-scale image localization,” in _Eur. Conf. Comput. Vis._ , 2020.
  * [10] y. hongji, X. Lu, ., and Y. Zhu, “Cross-view geo-localization with layer-to-layer transformer,” in _Adv. Neural Inform. Process. Syst._ , 2021\.
  * [11] S. Hausler, S. Garg, M. Xu, M. Milford, and T. Fischer, “Patch-netvlad: Multi-scale fusion of locally-global descriptors for place recognition,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2021.
  * [12] M. Leyva-Vallina, N. Strisciuglio, ., and N. Petkov, “Data-efficient large scale place recognition with graded similarity supervision,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2023.
  * [13] R. Arandjelović, P. Gronat, A. Torii, T. Pajdla, and J. Sivic, “Netvlad: Cnn architecture for weakly supervised place recognition,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2016.
  * [14] A. Ali-bey, Chaib-draa, Brahim, and P. Giguère, “Global proxy-based hard mining for visual place recognition,” _British Mach. Vis. Conf._ , 2023.
  * [15] A. El-Nouby, N. Neverova, I. Laptev, and H. Jégou, “Training vision transformers for image retrieval,” _arXiv preprint arXiv:2102.05644_ , 2021\.
  * [16] S. Zhu, L. Yang, C. Chen, M. Shah, X. Shen, and H. Wang, “$r^{2}$former: Unified $r$etrieval and $r$eranking transformer for place recognition,” _arXiv preprint arXiv:2304.03410_ , 2023.
  * [17] R. Child, S. Gray, A. Radford, and I. Sutskever, “Generating long sequences with sparse transformers,” _arXiv preprint arXiv:1904.10509_ , 2019.
  * [18] L. Liu, H. Li, and Y. Dai, “Stochastic attraction-repulsion embedding for large scale image localization,” in _Int. Conf. Comput. Vis._ , 2019, pp. 2570–2579.
  * [19] F. Schroff, D. Kalenichenko, ., and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2015.
  * [20] M. Cummins and P. Newman, “Fab-map: Probabilistic localization and mapping in the space of appearance,” _Int. J. Rob. Res._ , 2008.
  * [21] R. Mur-Artal and J. D. Tardós, “Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras,” _IEEE Trans. Robot._ , 2017.
  * [22] D. G. Lowe, “Object recognition from local scale-invariant features,” in _Int. Conf. Comput. Vis._ , 1999.
  * [23] H. Bay, T. Tuytelaars, and L. V. Gool, “Surf: Speeded up robust features,” in _Eur. Conf. Comput. Vis._ , 2006.
  * [24] Y. Xu, P. Shamsolmoali, and E. Granger, “Transvlad: Multi-scale attention-based global descriptors for visual geo-localization,” in _IEEE Winter Conf. Appl. Comput. Vis._ , 2023.
  * [25] B. Cao, A. Araujo, and J. Sim, “Unifying deep local and global features for image search,” in _Eur. Conf. Comput. Vis._ , 2020.
  * [26] Y. Zhu, J. Wang, L. Xie, and L. Zheng, “Attention-based pyramid aggregation network for visual place recognition,” in _Pro. ACM Int. Conf. Multimedia_ , 2018.
  * [27] F. Radenović, G. Tolias, ., and O. Chum, “Fine-tuning cnn image retrieval with no human annotation,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 41, no. 7, 2018.
  * [28] A. Gordo, J. Almazan, J. Revaud, and D. Larlus, “End-to-end learning of deep visual representations for image retrieval,” _Int. J. Comput. Vis._ , vol. 124, no. 2, 2017.
  * [29] G. Berton, R. Mereu, G. Trivigno, C. Masone, G. Csurka, T. Sattler, and B. Caputo, “Deep visual geo-localization benchmark,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2022.
  * [30] Y. Shen, S. Zhou, J. Fu, R. Wang, S. Chen, and N. Zheng, “Structvpr: Distill structural knowledge with weighting samples for visual place recognition,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2023.
  * [31] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Adv. Neural Inform. Process. Syst._ , 2017.
  * [32] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jegou, “Training data-efficient image transformers & amp; distillation through attention,” in _Proc. Int. Conf. Mach. Learn._ , 2021.
  * [33] R. Wang, Y. Shen, W. Zuo, S. Zhou, and N. Zheng, “Transvpr: Transformer-based place recognition with multi-level attention aggregation,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2022.
  * [34] L. Yan, Y. Cui, Y. Chen, and D. Liu, “Hierarchical attention fusion for geo-localization,” in _IEEE Int. Conf. Acoust. Speech. Signal. Process. Proc._ , 2021.
  * [35] H. Hu, Z. Qiao, M. Cheng, Z. Liu, and H. Wang, “Dasgil: Domain adaptation for semantic and geometric-aware image-based localization,” _IEEE Trans. Image Process._ , vol. 30, 2020.
  * [36] V. Paolicelli, A. Tavera, C. Masone, G. Berton, and B. Caputo, “Learning semantics for visual place recognition through multi-scale attention,” in _Int. Conf. Image Anal. Process._ , 2022.
  * [37] H. Wu, B. Xiao, N. Codella, M. Liu, X. Dai, L. Yuan, and L. Zhang, “Cvt: Introducing convolutions to vision transformers,” in _Int. Conf. Comput. Vis._ , 2021.
  * [38] D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),” _arXiv preprint arXiv:1606.08415_ , 2016.
  * [39] H. Jégou and O. Chum, “Negative evidences and co-occurences in image retrieval: The benefit of pca and whitening,” in _Eur. Conf. Comput. Vis._ , 2012.
  * [40] R. Arandjelović and A. Zisserman, “All about vlad,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2013.
  * [41] A. Torii, R. Arandjelović, J. Sivic, M. Okutomi, and T. Pajdla, “24/7 place recognition by view synthesis,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2015.
  * [42] F. Warburg, S. Hauberg, M. Lopez-Antequera, P. Gargallo, Y. Kuang, and J. Civera, “Mapillary street-level sequences: A dataset for lifelong place recognition,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2020.
  * [43] A. Torii, J. Sivic, T. Pajdla, and M. Okutomi, “Visual place recognition with repetitive structures,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2013\.
  * [44] J. Philbin, O. Chum, ., and M. Isard, “Lost in quantization: Improving particular object retrieval in large scale image databases,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2008.
  * [45] J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman, “Object retrieval with large vocabularies and fast spatial matching,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2007.
  * [46] H. Jegou, M. Douze, and C. Schmid, “Hamming embedding and weak geometric consistency for large scale image search,” in _Eur. Conf. Comput. Vis._ , 2008.
  * [47] A. Stylianou, R. Souvenir, ., and R. Pless, “Visualizing deep similarity networks,” in _IEEE Winter Conf. Appl. Comput. Vis._ , 2019.
  * [48] U. Efe, K. G. Ince, and A. Alatan, “Dfm: A performance baseline for deep feature matching,” in _IEEE Conf. Comput. Vis. Pattern Recog. Workshops_ , 2021.
  * [49] Y. Liu, Z. Shen, Z. Lin, S. Peng, H. Bao, and X. Zhou, “Gift: Learning transformation-invariant dense visual descriptors via group cnns,” in _Adv. Neural Inform. Process. Syst._ , 2019.
  * [50] D. DeTone, T. Malisiewicz, and A. Rabinovich, “Superpoint: Self-supervised interest point detection and description,” in _IEEE Conf. Comput. Vis. Pattern Recog. Workshops_ , 2018.
  * [51] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _Int. Conf. Learn. Represent._ , 2015\.