# MinkLoc++: Lidar and Monocular Image Fusion for Place Recognition
††thanks: The project was funded by POB Research Centre for Artificial
Intelligence and Robotics of Warsaw University of Technology within the
Excellence Initiative Program - Research University (ID-UB)

Jacek Komorowski1, Monika Wysoczańska1, Tomasz Trzcinski1,2 1Institute of
Computer Science, Warsaw University of Technology 2Tooploox
<EMAIL_ADDRESS>
###### Abstract

We introduce a discriminative multimodal descriptor based on a pair of sensor
readings: a point cloud from a LiDAR and an image from an RGB camera. Our
descriptor, named MinkLoc++, can be used for place recognition, re-
localization and loop closure purposes in robotics or autonomous vehicles
applications. We use late fusion approach, where each modality is processed
separately and fused in the final part of the processing pipeline. The
proposed method achieves state-of-the-art performance on standard place
recognition benchmarks. We also identify _dominating modality_ problem when
training a multimodal descriptor. The problem manifests itself when the
network focuses on a modality with a larger overfit to the training data. This
drives the loss down during the training but leads to suboptimal performance
on the evaluation set. In this work we describe how to detect and mitigate
such risk when using a deep metric learning approach to train a multimodal
neural network. Our code is publicly available on the project website.
111https://github.com/jac99/MinkLocMultimodal

###### Index Terms:

multimodal place recognition, global descriptor, deep metric learning

##  I Introduction

Place recognition is an important task in computer vision with a broad range
of practical applications, such as loop-closure in SLAM [1, 2] or autonomous
driving [3, 4]. Early methods [5, 6, 7, 8] mostly rely on images from RGB
cameras and are affected by illumination changes and different weather
conditions. To mitigate this problem and improve the robustness, autonomous
vehicles are nowadays equipped with additional sensors, such as LiDAR. LiDAR
provides information about the geometry of the observed scene in the form of a
sparse point cloud. It’s more robust to adverse environmental conditions than
RGB camera and can provide reliable data in limited visibility conditions
(e.g. at night). On the other hand, LiDARs cannot capture fine details of the
observed scene, such as a texture of visible buildings or other objects, that
might be helpful in localization task. The intuitive approach to improve place
recognition performance is to combine these two modalities. Such multimodal
approach is an interesting machine learning problem that has gained attention
recently but remains underexplored [9]. The main research question is how to
efficiently combine different modalities to fully exploit their complementary
character.

Figure 1: Multimodal place recognition. MinkLoc++ computes a global descriptor
from a pair of sensor readings: a 3D point cloud from LiDAR and an image from
a RGB camera. Localization is performed by searching the database for a geo-
tagged pair of readings with the closest descriptor.

Place recognition is often formulated as an instance retrieval problem. Given
a query instance, such as an RGB image, the aim is to find its closest match,
with a known location, in a large-scale database. This idea is illustrated in
Fig. 1. In this work, we focus on learning a discriminative global descriptor
for large-scale place recognition purposes using multimodal data: an image
from an RGB camera and a 3D point cloud from a LiDAR. We use a late fusion
approach to obtain our multimodal descriptor, where two modalities are
processed separately and merged in the final part of the processing pipeline.
Our method yields the state-of-the-art results on the Oxford RobotCar [10] and
KITTI datasets [11].

In our initial experiments we observed an interesting behaviour, when training
the multimodal network using late fusion of two unimodal descriptors. The
network trained with two modalities performs worse compared to using only one
modality. Further investigation shows, that this counter-intuitive result is
caused by different susceptibility to overfitting of two feature extraction
subnetworks. The network learns to focus on the modality giving a smaller
training error – but because of much larger overfitting, producing poor
results on the evaluation set. The same phenomenon is observed in the context
of training multimodal classifiers in [12], where authors propose so-called
Gradient-Blending approach based on overfitting-to-generalization ratio of
each modality. In this work we use deep metric learning approach with a
triplet loss and propose a similar solution based on extending the loss
function with additional terms based on individual modalities.

In summary the main contributions of our work are as follows.

  * •

We design a discriminative multimodal descriptor for place recognition
purposes based on two modalities: a 3D point cloud from a LiDAR scan and an
image from RGB camera. The proposed method advances state of the art.

  * •

We propose an effective solution to the problem of _dominating modality_ which
adversely affects the discriminability of a multimodal descriptor. The problem
happens when one modality exhibits significantly higher overfitting to the
training set than the other one.

The remainder of this paper is organized as follows: Sec.II gives a brief
overview of the related work. Sec.III introduces our method for multimodal
place recognition. Sec.IV shows the experimental results and Sec.V concludes
our work.

##  II Related work

###  II-A Image-based place recognition

The majority of image retrieval methods [13, 5] builds upon local features
aggregation techniques such as Fisher Vectors [14] or VLAD[15]. Previously
used handcrafted features are currently replaced with convolutional neural
networks (CNNs), such as VGG [16] or ResNet [17]. In [5] authors introduce
NetVLAD, an end-to-end learnable method explicitly targeting the large-scale
place recognition task. Its main component is a generalized VLAD layer for
local feature aggregation. It can be plugged into any CNN-based feature
extractor and trained using backpropagation. Similarly, Generalized Mean
Pooling (GeM) proposed in [18] is another aggregation method that can be
appended a CNN backbone and trained end-to-end. GeM layer generalizes max and
average pooling and proved successful in instance retrieval tasks.

###  II-B Point cloud-based place recognition

PointNetVLAD [19] is the first deep network to generate a global point cloud
descriptor for large-scale place recognition. The method uses PointNet
architecture [20] to extract local features, followed by NetVLAD aggregation
layer. However PointNet-based architecture is not well suited to capture local
geometric structures and extract informative local features. LPD-Net [21]
addresses this weakness by enhancing an input point cloud with handcrafted
features and using graph neural networks to extract local contextual
information. EPC-Net [22] improves upon LPD-Net by using proxy point
convolutional neural network. SOE-Net [23] improves extraction of local
contextual information by introducing the PointOE module, which captures local
geometric structures from eight spatial orientations.

All of the previously mentioned methods are based on unordered set of points
representation. MinkLoc3D [24] uses an alternative representation and is based
on sparse voxelized representation. It uses 3D convolutional architecture
modelled upon a Feature Pyramid Network (FPN) [25] design pattern to extract
informative local features. Local features are aggregated using GeM [18] layer
into a discriminative global descriptor. MinkLoc3D yields state-of-the-art
results on standard benchmarks outperforming other other point cloud-based
global descriptors.

###  II-C LiDAR and RGB camera fusion for multimodal place recognition

Methods for multimodal place recognition can be categorized by a modality
fusion strategy. The first group creates augmented images based on structural
(3D) information, thus approaches modality fusion problem in the early stages
of the processing pipeline. CORAL, a bi-modal descriptor presented in [26],
builds an elevation image from an input point cloud. Elevation image is
enhanced with projected RGB image features extracted at multiple scales and
processed using a deep neural network. Extracted local features are aggregated
using NetVLAD layer. In [2] an input RGB image is concatenated with LiDAR
intensity image and processed using a convolutional neural network.

The most common approach in multimodal place recognition is late fusion. In
[27] a point cloud descriptor is obtained using PointNetVLAD architecture,
with NetVLAD aggregation layer altered to ignore non-informative 3D points.
Image-based descriptor is extracted using ResNet50 [17] architecture. Two
unimodal descriptors are concatenated and processed by a fully-connected layer
to produce the multimodal descriptor. [28] enhances _late fusion_ with a
global channel attention layer (GCA). In [29] image-based place recognition is
augmented with structural cues constructed from RGB video stream using
structure-from-motion. The reconstructed 3D point segments are discretized
into a regular voxel grid and processed using 3D convolutional neural network.
Features extracted from a 3D point cloud and an RGB image are merged to form
the multimodal descriptor.

##  III Method description

This section presents our MinkLoc++ method to compute a discriminative global
descriptor for place recognition purposes. The descriptor is based on two
modalities: an image from an RGB camera and a point cloud from a LiDAR
installed on a moving vehicle. Place recognition is performed by searching the
database of geo-tagged pairs of sensor readings (RGB image and 3D point cloud)
for descriptors closest to the descriptor of the query pair. The idea is
illustrated in Fig. 1.

###  III-A Network architecture

Figure 2: High level MinkLoc++ architecture. Sensor readings (3D point cloud
and RGB image) are processed by separate descriptor extraction networks,
consisting of a local feature extraction block followed by a feature pooling
layer. Resultant point cloud descriptor $\mathcal{D}_{PC}$ and image
descriptor $\mathcal{D}_{RGB}$ are aggregated to produce a fused multimodal
descriptor $\mathcal{D}$. Green arrows show intermediary values used in the
different terms of the loss function.

We use _late fusion_ approach where modalities are processed separately and
combined in the final part of the processing pipeline. The _late fusion_
approach is highly flexible and robust if one sensor fails.

The high-level network architecture is shown in Fig. 2. The network consists
of two branches: the upper branch computes a point cloud descriptor
$\mathcal{D}_{PC}\in\mathbb{R}^{k}$ and the lower branch computes an RGB image
descriptor $\mathcal{D}_{RGB}\in\mathbb{R}^{k}$. In our experiments we use
$k=128$.

The upper branch uses an enhanced version of MinkLoc3D [24] architecture, that
proved successful in point cloud-based place recognition task. We improve the
design by incorporating ECA [30] channel attention mechanism originally
proposed for 2D convolutional networks. The input point cloud is quantized
into a sparse voxelized representation and processed by a _point cloud feature
extraction block_ , shown in Fig. 3. It is a 3D convolutional network based on
a Feature Pyramid Network [31] design pattern. First, the input point cloud is
processed in the bottom-up part direction producing 3D feature maps with
decreasing spatial resolution and increasing receptive field. Each block in
the bottom-up pass consists of a series of 3D convolutions with skip
connections followed by ECA [30] channel attention layer. See Fig. 4 for
details. Then, the transposed convolution upsamples the feature map generated
by the last convolutional block. Upsampled feature map is concatenated with
features from the corresponding layer of the bottom-up pass using a lateral
connection. This design produces a feature map with relatively high spatial
resolution and large receptive field. Our initial experiments proved, that
pooling 3D features from feature map with higher spatial resolution is
advantageous over using lower resolution maps computed by deeper networks. The
resultant sparse 3D feature map $\mathcal{F}_{PC}$ is pooled using a
generalized-mean (GeM) pooling [18] layer to produce a point cloud descriptor
$\mathcal{D}_{PC}\in\mathbb{R}^{k}$.

Figure 3: Architecture of a _point cloud feature extraction block_. The input point cloud is quantized into a sparse 3D tensor and processed by a 3D convolutional network with FPN [31] architecture, producing a point cloud feature map $\mathcal{F}_{PC}$ Numbers in parentheses (e.g. 1/32) denote a stride and a number of channels of a feature map produced by each layer.  Block | Details
---|---
Conv0 | $\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 32$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$5_{k}1_{s}$}$}\end{array}$
Conv1 | $\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 32$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$2_{k}2_{s}$}$}\end{array}\Big{\langle}\;\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 32$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$3_{k}1_{s}$}$}\end{array}\;\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 32$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$3_{k}1_{s}$}$}\end{array}\kern-6.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{A}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\Big{\rangle}$
Conv2 | $\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 64$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$2_{k}2_{s}$}$}\end{array}\Big{\langle}\;\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 64$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$3_{k}1_{s}$}$}\end{array}\;\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 64$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$3_{k}1_{s}$}$}\end{array}\kern-6.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{A}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\Big{\rangle}$
Conv3 | $\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 64$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$2_{k}2_{s}$}$}\end{array}\Big{\langle}\;\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 64$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$3_{k}1_{s}$}$}\end{array}\;\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 64$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$3_{k}1_{s}$}$}\end{array}\kern-6.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{A}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\Big{\rangle}$
1x1x1Conv2, 3 | $\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 128$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$1_{k}1_{s}$}$}\end{array}$
1x1x1Conv3 | $\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 128$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$1_{k}1_{s}$}$}\end{array}$
TConv3 | $\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$t~{}$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 128$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$2_{k}2_{s}$}$}\end{array}$
Figure 4: Layers in the _point cloud feature extraction block_. All
convolutions in Conv$0\ldots 3$ blocks are followed by batch norm and ReLU
non-linearity. $\mathbb{C}$ denotes a 3D convolution with a number of filters
given as the top-right index, $t$ decorator indicates a transposed
convolution, lower $k$ shows a filter size and lower $s$ a stride.
$\mathbb{A}$ is ECA [30] channel attention and $\langle\ldots\rangle$
enclosures a residual block with a skip connection.

The lower branch uses first four blocks of a pre-trained ResNet18 [17] to
extract a 2D feature map with 256 channels. The number of channels is reduced
to $k$ using a convolutional layer with 1x1 filter, producing an RGB image
feature map $\mathcal{F}_{RGB}$. The feature map is pooled using a GeM pooling
to produce an RGB image descriptor $\mathcal{D}_{RGB}\in\mathbb{R}^{k}$.

_Descriptor aggregation_ block combines a point cloud descriptor
$\mathcal{D}_{PC}$ and an image descriptor $\mathcal{D}_{RGB}$ to produce a
final $2k$-dimensional multimodal descriptor. We use a simple concatenation
along channels dimension to merge two descriptors. In ablation study we
evaluate other options and discuss why this simplistic aggregation strategy
gives better results than more sophisticated approaches.

###  III-B Network training

To train the network to produce discriminative global descriptors we use a
deep metric learning approach [32] with a triplet margin loss [33]. Each mini-
batch element is a pair of readings: a 3D point cloud and a corresponding RGB
image. Mini-batch elements are arranged into triplets consisting of an anchor,
a positive and a negative example. The positive example is a batch element
similar to the anchor and a negative example is a batch element dissimilar to
the anchor. Two elements are similar if they show the same place, that is if
they were acquired at locations at most 10 meters apart. Elements are
dissimilar, if they represent different locations, that is they were captured
at locations more than 50 meters apart. If a distance between locations is
within 10-50 meters range, the elements are considered neither similar nor
dissimilar. To increase the training efficiency, we use batch-hard negative
mining [33] to build informative triplets.

The loss function $\mathcal{L}$, defined by Eq. 1, is a sum of three terms:
the first is based on the fused multimodal descriptor $\mathcal{D}$, the
second on the a point cloud descriptor $\mathcal{D}_{PC}$ and third on an RGB
image descriptor $\mathcal{D}_{RGB}$.

$\mathcal{L}=(1-\alpha-\beta)\mathcal{L}_{F}+\alpha\mathcal{L}_{PC}+\beta\mathcal{L}_{RGB},$
(1)

where $\alpha,\beta,\gamma$ are experimentally chosen weights and each
component $\mathcal{L}_{F}$, $\mathcal{L}_{PC}$, $\mathcal{L}_{RGB}$ is a
triplet margin loss function [33] defined as:

$\mathcal{L}_{\square}(a_{i},p_{i},n_{i})=\max\left\\{d(a_{i},p_{i})-d(a_{i},n_{i})+m,0\right\\},$
(2)

where $d(x,y)=||x-y||_{2}$ is an Euclidean distance between descriptors $x$
and $y$; $a_{i},p_{i},n_{i}$ are descriptors of an anchor, a positive and a
negative element in $i$-th triplet and $m$ is a margin. For each batch,
$\mathcal{L}_{F}$ is computed from triplets constructed from multimodal
descriptors $\mathcal{D}$; $\mathcal{L}_{PC}$ from triplets constructed from
point cloud descriptors $\mathcal{D}_{PC}$; and $\mathcal{L}_{RGB}$ from
triplets constructed from RGB image descriptors $\mathcal{D}_{RGB}$. Fig. 2
shows the relation of each loss component to the processing pipeline.

The rationale for using multiple terms in the loss function is the _dominating
modality_ problem discovered during our initial experiments. We noticed that a
multimodal descriptor, trained with a standard triplet loss, had an inferior
performance on the evaluation set compared to the best unimodal descriptor.
This counter-intuitive result is caused by the fact that one modality (RGB
image in our case) performs much better on the training set and much worse on
the evaluation set than the other one (3D point cloud), due to much larger
overfitting. Thus, the network learns to focus on the modality with smaller
training error, but greater evaluation error, largely ignoring the other one.
We mitigate this problem by extending the loss function with additional terms
$\mathcal{L}_{PC}$ and $\mathcal{L}_{RGB}$ based on unimodal descriptors
$\mathcal{D}_{PC}$ and $\mathcal{D}_{RGB}$ produced by two unimodal
subnetworks (see Fig. 2). This simple solution allows balancing the influence
each modality and improves performance of the fused multimodal descriptor.

The same phenomenon was observed in the context of training multimodal
classifiers in [12]. It proposes Gradient-Blending, where weights of each loss
term are based on overfitting-to-generalization ratio of each single modality
subnetwork. This approach cannot be directly applied when training a
multimodal descriptor using a deep metric learning approach. When batch-hard
negative mining is used, the loss function often fluctuates or stagnates for a
number of epochs as harder and harder triplets are constructed. Instead, the
number of active triplets (i.e. triplets with non-zero loss) constructed from
each unimodal descriptor is much better indicator of overfitting. Hence, we
find the dominating modality (RGB image) by examining the ratios of active
triplets in training and validation set using each unimodal descriptor.
Weights $\alpha$ and $\beta$ in Eq. 1 are chosen to maximize the performance
on the validation set. It should be noted, that it’s sufficient to tune only
one parameter in Eq. 1, that is the weight for the non-dominating modality.
The weight for the dominating modality can be set to 0, as parameters of its
descriptor extraction subnetwork will nevertheless be optimized during the
learning due to their influence on the fused multimodal loss $\mathcal{L}_{F}$
term. See ablation study section for evidence how multi-head loss improves
discriminability of the multimodal descriptor.

The loss function is minimized using a stochastic gradient descent with Adam
optimizer. Similar to [24], we use dynamic batch sizing. The training starts
with a small batch size (32 elements) and the batch size is gradually
increased up to 128 elements as the training progress and the network learns
more discriminative embeddings. The number of active triplets (i.e. triplets
with non-zero loss) is monitored and when falls below the threshold, the batch
size is increased. This prevents the training process from collapse, which
could happen in early epochs when mining hard triplets from a large batch, and
allows using larger batch to mine more difficult triplets later on.

We use data augmentation to improve generalization and reduce the overfitting.
For RGB images, we use photometric distortions, random crop and random erasing
augmentation [34]. For 3D points clouds, we use a random jitter and random
points removal. We also adapted random erasing augmentation to remove 3D
points within the randomly selected fronto-parallel cuboid.

###  III-C Implementation details

In all experiments we quantize 3D point coordinates with 0.01 quantization
step. As point coordinates in the Oxford RobotCar dataset are normalized to be
within $\left[-1,1\right]$ range, this gives up to 200 voxels in each spatial
direction. The initial learning rate for network parameters in the _RGB image
feature extraction block_ is set to $10^{-4}$ and for all other parameters to
$10^{-3}$. In all experiments we train the network for 50 epochs, reducing the
learning rate by 10 at the end of 30 epoch. The dimensionality of point cloud
and RGB image descriptors is set to $k=128$, and the multimodal descriptor has
$2k=256$ dimensions. To prevent embedding collapse in early epochs of
training, we use a dynamic batch sizing strategy. The initial batch size is
set to 8. When the number of active triplets falls below 70% of the current
batch size, the batch is increased by 40% until the maximum size of 160
elements is reached. To limit overfitting we use $L_{2}$ weight regularization
with $\lambda=10^{-3}$ coefficient. The coefficients of the loss terms in Eq.
1 are $\alpha=0.5,\beta=0$.

All experiments are performed on a server with a single nVidia RTX 2080Ti GPU,
12 core AMD Ryzen Threadripper 1920X processor and 64 GB of RAM. We use
PyTorch 1.7 [35] deep learning framework, MinkowskiEngine 0.4.3 [36] auto-
differentiation library for sparse tensors and Pytorch Metric Learning library
0.9.96 [37].

##  IV Experimental results

This section describes results of an experimental evaluation of our global
descriptor and comparison with state of the art.

###  IV-A Datasets and evaluation methodology

Our method is trained and evaluated on a modified Oxford RobotCar dataset
[10]. The dataset is created using a suite sensors (RGB cameras, LiDAR
sensors, GPS/INS) mounted on the car travelling multiple times through the
same route in the city of Oxford at different times of day and year. We build
our dataset by enhancing a dataset of 3D point clouds introduced in [19],
constructed from RobotCar LiDAR scans. Point clouds are build from consecutive
2D LiDAR scans during the 20 m. drive of a vehicle. The ground plane is
removed and point clouds are downsampled to 4096 points. For each point cloud
we find in an original RobotCar dataset corresponding RGB images with the
closest timestamps taken from the center camera. Each image is downsampled
from 1280x960 to 320x200 resolution. During training, to increase data
diversity and limit overfitting, we randomly sample from 15 closest RGB
images. In evaluation we use only one RGB image with the closest timestamp.
Exemplary dataset elements are shown in Fig. 1. Elements are similar (positive
examples) if they are at most 10m apart and dissimilar (negative examples) if
they are at least 50m apart. Elements between 10 and 50m apart are neither
similar nor dissimilar. The dataset is split into disjoint training (21.7k
elements) and test (3k elements) areas based on UTM coordinates.

We follow the same evaluation protocol and use the same train/test split (so
called Baseline scenario), as introduced in [19] and used in later works [21,
24]. An element (point cloud with a corresponding RGB image) from a test
dataset is taken as a query. Pairs of point clouds and RGB images from other
traversals covering the same region but captured at different times form the
database. _Recall@N_ measures the percentage of correctly localized queries
using top-$N$ elements returned from the database. Localization is correct, if
at least one of top-$N$ retrieved database elements is within 25 meter
distance threshold from the ground truth position of the query element. In
addition to Average Recall@1 (AR@1) we report Average Recall@1% (AR@1%), which
is calculated taking into account top-$k$ matches, where $k$ is 1% of the
database size.

Additionally, we evaluate the generalization ability of our methods using
KITTI Odometry dataset. The dataset consists of RGB camera images, as well as
LIDAR scans acquired with Velodyne laser scanner. Therefore, the
characteristics of input point clouds differs significantly from the scans in
Oxford RobotCar dataset. We follow the same procedure as introduced in [26].
We take Sequence 00 which visits the same places repeatedly and construct the
reference database using the data gathered during the first 170 seconds. The
rest is used as localization queries.

###  IV-B Evaluation results

We compare the performance of our multimodal descriptor with point cloud-only
descriptors: PointNetVLAD [19], PCAN [38], LPD-Net [21], EPC-Net [22], SOE-Net
[23], MinkLoc3D [24] and multimodal descriptors based on a 3D point cloud and
an RGB image: CORAL [26], PIC-Net [28]. Where the source code or pretrained
model is available (PointNetVAD, PCAN, LPD-Net, MinkLoc3D) we run the
evaluation by ourselves, otherwise (EPC-Net, SOE-Net, CORAL, PIC-Net) we show
results reported by authors on the same dataset and with identical evaluation
protocol.

Evaluation results on the modified Oxford RobotCar dataset (Baseline scenario)
are shown in Tab. I. Our MinkLoc++ model outperforms other multimodal methods.
The improvement in AR@1% seems moderate (99.1 our method versus 98.2 for PIC-
Net) but this metric is close to 100% and there’s very little room for
improvement. We also compared MinkLoc++ trained using only 3D modality to a
number of unimodal, point cloud-based descriptors. Our method outperforms all
of them, including recently proposed EPC-Net and SOE-Net. It can be noticed,
that overall improvement from using multimodal approach compared to 3D-based
unimodal is limited. The best unimodal descriptor has AR@1 94.5 compared to
96.7 obtained by multimodal MinkLoc++. This can be explained by two factors.
First, the results are close to 100% and even 2 p.p. improvement is worth
noticing. Second, RobotCar dataset was acquired at varying environmental
conditions, with a number of images taken in limited visibility conditions (at
night, very sunny or rainy day). 3D modality based on LiDAR scans is more
reliable source of data than an RGB image.

TABLE I: Evaluation results of place recognition methods on Robot Car dataset.  |  | Modality
---
| AR@1%
---
| AR@1
---
PointNetVLAD [19] | 3D | 80.3 | 63.3
PCAN [38] | 3D | 83.8 | 70.7
DH3D-4096 [39] | 3D | 84.3 | 73.3
LPD-Net [21] | 3D | 94.9 | 86.4
EPC-Net [22] | 3D | 94.7 | 86.2
SOE-Net [23] | 3D | 96.4 | 89.3
MinkLoc3D [24] | 3D | 97.9 | 93.8
MinkLoc++ (our) | 3D | 98.2 | 93.9
CORAL [26] | 3D + RGB | 96.1 | 88.9
PIC-Net [28] | 3D + RGB | 98.2 | -
MinkLoc++ (our) | 3D + RGB | 99.1 | 96.7
TABLE II: Evaluation results of place recognition methods on KITTI Odometry dataset.  |  | Modality
---
| AR@1%
---
PointNetVLAD [19] | 3D | 72.4
LPD-Net [21] | 3D | 74.6
MinkLoc++ (our) | 3D | 72.6
MinkLoc++ (our) | RGB | 76.2
CORAL [26] | 3D + RGB | 76.4
MinkLoc++ (our) | 3D + RGB | 82.1

(a)

(b)

(c)

(d)

(a)

(b)

(c)

(d)

Figure 5: Example of successful retrieval using multimodal approach. (a) is
the query element, (b) incorrect retrieval using 3D modality and (c) incorrect
retrieval using RGB modality (d) correct retrieval using both modalities.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 6: Example of successful retrieval using multimodal approach. The top
left is the query example. The top right is a successful retrieval based on
multimodal input. The bottom left is incorrect retrieval using RGB input, and
the bottom right is the one based on 3D input.

(a)

(b)

(c)

(a)

(b)

(c)

Figure 7: Example of unsuccessful retrieval using both modalities. (a) is the
query element, (b) incorrect match to the query and (c) the closest true
match. Geometry of true and false matches is very similar and appearance of
the query differs from the true match due to lighting conditions.

To verify robustness of our method to different environmental conditions, we
run additional evaluation on RobotCar Seasons [40] benchmark. This is a
challenging image-based dataset, prepared from Oxford RobotCar, with different
splits corresponding to different atmospheric conditions and times of a day
(e.g. snow, rain, dawn or night). We extend this dataset with 3D information
to allow testing our multimodal descriptor. For each image, we find LiDAR
readings with corresponding timestamps in the in original RobotCar dataset
[10] and use them to construct 3D point clouds. We compute descriptors for
each element in the extended dataset and use them to retrieve elements from
each dataset split. We compare our method with two image-based and two point
cloud-based global descriptors. Tab. III reports the percentage of correctly
localized queries for each dataset split. The query is correctly localized if
the pose of best matching element from the database is within 5 meter and 10∘
threshold from the ground truth pose. In day conditions MinkLoc++ consistently
outperforms all unimodal methods, except for rainy weather. The rain adversely
affects LiDAR performance causing worse results. The improvement over image-
based methods in good visibility conditions is only moderate and additional
modality (3D) does not significantly improve the results. In night conditions,
point cloud-based methods and our multimodal method show their potential and
overcome image-based methods by a large margin.

TABLE III: Percentage of correctly localized queries on RobotCar Seasons benchmark. |  | day conditions | night conditions
---|---|---|---
|  |  | dawn
---
| dusk
---
| overcast
---
summer
| overcast
---
winter
| rain
---
| snow
---
| sun
---
| night
---
| night-rain
---
DenseVLAD [41] | RGB | 92.5 | 94.2 | 92.0 | 93.3 | 96.9 | 90.2 | 80.2 | 19.9 | 25.5
NetVLAD [5] | RGB | 82.6 | 92.9 | 95.2 | 92.6 | 96.0 | 91.8 | 86.7 | 15.5 | 16.4
LPD-Net [21] | 3D | 79.7 | 79.9 | 79.7 | 73.8 | - | - | 82.3 | 77.3 | 32.8
MinkLoc3D [24] | 3D | 89.2 | 88.3 | 90.3 | 83.1 | 66.3 | 86.3 | 87.4 | 86.1 | 58.0
MinkLoc++ (our) | 3D+RGB | 93.6 | 94.4 | 96.3 | 94.4 | 79.1 | 94.3 | 88.0 | 80.6 | 59.1

###  IV-C Generalization evaluation

To test the generalization ability of our method, we evaluate the model
trained on RobotCar dataset on KITTI dataset. We compare our results with [26]
by reproducing the evaluation procedure described in this paper. Our
multimodal MinkLoc++ descriptor outperforms other methods. Our method can
efficiently fuse two modalities to boost the performance. It must be noted,
that point clouds in Kitti dataset have very different characteristic than
point clouds in our training dataset. The former are build from a single
$360^{\circ}$ sweep of a 3D LiDAR, the latter are constructed by merging
multiple scans from a 2D LiDAR during the 20m. drive of a vehicle. This
explains worse performance of our method using solely 3D modality compared to
the results on RobotCar dataset. Fig. 6 shows an example of a successful
retrieval case.

###  IV-D Ablation study

In this section, we investigate the effects of the design choices on our
multimodal descriptor performance. In all experiments, unless otherwise noted,
the network is trained using the same dataset and hyperparameters as described
in previous sections.

First, we compare our multi-head loss function with a straightforward approach
where the loss is based only on the fused multimodal descriptor. Tab. IV shows
the number of active triplets, at the end of the training, constructed from
each unimodal descriptor and the performance of a fused multimodal descriptor,
for different combination of weights in the loss function given by Eq. 1.
Setting $\alpha$ and $\beta$ to 0 (first row), means that only one term
$\mathcal{L}_{F}$, based on the fused multimodal descriptor, is used. In this
case, when using a training data there’s much less ($-52$) active triplets for
RGB image modality than for 3D modality. On the contrary, on validation data,
there’s more ($+24$) active triplets for RGB modality than 3D. This suggests,
that the network overfits much more to RGB modality, which is the _dominating
modality_. Because of the larger overfit, during the training the network
concentrates on RGB modality, largely ignoring the other one. This leads to
suboptimal performance of the fused multimodal descriptor on the evaluation
set. The solution is to increase the weight of the loss term corresponding to
the non-dominating modality ($\mathcal{L}_{PC}$ in our case). This balances
both modalities during the training and increases performance of the fused
multimodal descriptor. Note, that the best performance of the multimodal
descriptor is achieved when the difference between the number of active
triplets for two unimodal descriptors is similar in both training and
validation sets (rows 2 and 3 in Tab. IV with $\alpha=0.5$ and $\alpha=0.8$).
Using multi-head loss function allows increasing the performance from 94.4%
AR@1 (for $\alpha=\beta=0$) to 96.7% (for $\alpha=0.5,\beta=0$). In earlier
experiments we tried limiting overfitting for RGB modality by regularization
or more aggressive data augmentation. But this had only a limited impact and
didn’t allow solving the above mentioned problem.

TABLE IV: Impact of weights of each unimodal loss term in Eq.1 on the number of active triplets for each modality and the performance (AR%1) of the multimodal descriptor. |  | Active triplets train | Active triplets val |
---|---|---|---|---
$\alpha$ | $\beta$ |  | $\mathcal{D}_{PC}$
---
| $\mathcal{D}_{RGB}$
---
|  | $\mathcal{D}_{PC}$
---
| $\mathcal{D}_{RGB}$
---
|  | AR@1
---
0 | 0 | 87 | 35 | -52 | 95 | 119 | +24 | 94.4
0.2 | 0 | 51 | 56 | +5 | 83 | 124 | +41 | 96.0
0.5 | 0 | 25 | 66 | +41 | 75 | 121 | +46 | 96.7
0.8 | 0 | 19 | 73 | +54 | 71 | 123 | +52 | 96.4

In Tab. V we compare discriminativity of the multimodal descriptor with its
unimodal counterparts. To make a fair comparison we increase dimensionality of
each unimodal descriptor to 256, the same size as the multimodal descriptor.
Using multimodal approach improves discriminability of the global descriptor
(96.7 AR@1) compared to its unimodal counterparts (86.8 for RGB and 94.5 for
3D). As expected, an RGB-based descriptor has the worst performance. Oxford
RobotCar is a challenging dataset and image acquisition conditions vary
significantly between different traversals. The same place visited during a
sunny day and at dusk or at night looks differently. The improvement over 3D
modality was more limited.

TABLE V: Impact of each modality on the discriminability of MinkLoc++ descriptor. | Modality
---
| AR@1%
---
| AR@1
---
RGB | 95.9 | 86.8
3D point cloud | 98.3 | 94.5
RGB + 3D | 99.1 | 96.7

Tab. VI shows the impact of a feature pooling method on the performance of
each unimodal descriptor. The following methods are evaluated: maximum
activations of convolutions (MAC) [42], sum-pooled convolutional features
(SPoC) [43], generalized-mean pooling (GeM) [18], NetVLAD [5] and NetVLAD with
Context Gating [44] (NetVLAD-CG). For RGB modality, all examined methods,
except for SPoC, performed similarly, with AR@1 varying between 86.1 and 86.9.
For 3D modality, generalized-main pooling, with one trainable parameter,
yields the best results (94.5 AR@1). Hence, we choose GeM as a feature pooling
method for both modalities. It can be noted that more sophisticated feature
aggregation methods (NetVLAD and NetVLAD with Context Gating) didn’t improved
the performance. This can be explained by the fact, that NetVLAD layer with a
large number of trainable parameters (about 6 mln. in our configuration)
increases overfitting which prevents an improvement on the evaluation set.

TABLE VI: Impact of a feature pooling method on a discriminability (AR@1) of unimodal descriptors. |  | SPoC
---
| MAC
---
| GeM
---
| NetVLAD
---
| NetVLAD-GC
---
RGB | 76.9 | 86.9 | 86.8 | 86.9 | 86.1
3D | 86.7 | 91.6 | 94.5 | 89.2 | 89.8

We also investigate different approaches for aggregation of two unimodal
descriptors to form the multimodal descriptor. Two basic strategies are
concatenation and summation of unimodal descriptors and the results are shown
in the second column (’-’) in Tab. VII. After this initial step, we optionally
process the merged descriptor using a fully connected layer (column FC in the
table) or two-layer MLP (column MLP). Interestingly, simple concatenation or
addition of unimodal descriptors gives the best results. When concatenation is
followed by a fully connected layer, AR@1 drops by 9 p.p. to 87.8. This
behavior can again be attributed to the _dominating modality_ problem. When a
fully connected layer or MLP is used to process merged descriptors, the
benefits or using multi-head loss function, with additional loss term for each
modality, are lost. Added layers learn to extract information from _dominating
modality_ (RGB image in our case) from merged data and focus on it. Due to
larger overfitting to the training set this leads to noticeably worse
performance on the evaluation set. It can be noted that adding 2-layer MLP
drops AR@1 to 85.0 (86.61), which is comparable with the performance of the
unimodal descriptor based on the dominating RGB modality (86.8 AR@1).

TABLE VII: Impact of unimodal descriptors aggregation method on the discriminability (AR@1) of the multimodal descriptor. (-) only concatenate/add, FC = with a fully connected layer, MLP = with 2 layer MLP. |  | -
---
| FC
---
| MLP
---
Concatenation | 96.7 | 87.8 | 86.6
Addition | 96.7 | 90.7 | 85.0

##  V Conclusion

In this paper we present MinkLoc++, a multimodal descriptor for place
recognition purposes based on an input from a LiDAR scanner and RGB camera.
Experimental evaluation proves that the proposed method outperforms state of
the art on standard benchmarks. The performance of our method can be
attributed to two factors. First, we use an efficient 3D convolutional
network, based on sparse voxelized representation, for producing a
discriminative point cloud descriptor. Enhancing MinkLoc3D [24] architecture
with a ECA channel attention [30] improves results on 3D modality. Second, we
show how to efficiently train multimodal descriptor avoiding _dominating
modality_ problem. Straightforward approach of computing the loss based only
on the fused multimodal descriptor leads to suboptimal results. This is caused
by the fact, that one modality (RGB image in our case) performs better on the
training set, and much worse on the validation set, due to larger overfitting.
The network learns to focus on the modality with better training performance
leading to worse results on the evaluation set. We mitigate this problem by
extending the loss function with additional terms based on unimodal
descriptors computed in intermediary steps of the processing pipeline. This
simple solution balances both modalities and improves performance of the fused
descriptor.

## References

  * [1] R. Dube, D. Dugas, E. Stumm, J. Nieto, R. Siegwart, and C. Cadena, “SegMatch: Segment based place recognition in 3d point clouds,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2017.
  * [2] K. Żywanowski, A. Banaszczyk, and M. R. Nowicki, “Comparison of camera-based and 3d lidar-based place recognition across weather conditions,” in _2020 16th International Conference on Control, Automation, Robotics and Vision (ICARCV)_ , 2020, pp. 886–891.
  * [3] S. Xie, C. Pan, Y. Peng, K. Liu, and S. Ying, “Large-scale place recognition based on camera-lidar fused descriptor,” _Sensors_ , vol. 20, no. 10, 2020\. [Online]. Available: https://www.mdpi.com/1424-8220/20/10/2870
  * [4] S. Ouerghi, R. Boutteau, X. Savatier, and F. Tlili, “Visual odometry and place recognition fusion for vehicle position tracking in urban environments,” _Sensors_ , vol. 18, no. 4, 2018. [Online]. Available: https://www.mdpi.com/1424-8220/18/4/939
  * [5] R. Arandjelovic, P. Gronat, A. Torii, T. Pajdla, and J. Sivic, “Netvlad: Cnn architecture for weakly supervised place recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 5297–5307.
  * [6] M. Cummins and P. Newman, “Fab-map: Probabilistic localization and mapping in the space of appearance,” _The International Journal of Robotics Research_ , vol. 27, no. 6, pp. 647–665, 2008. [Online]. Available: https://doi.org/10.1177/0278364908090961
  * [7] D. Galvez-López and J. D. Tardos, “Bags of binary words for fast place recognition in image sequences,” _IEEE Transactions on Robotics_ , vol. 28, no. 5, pp. 1188–1197, 2012.
  * [8] M. Gehrig, E. Stumm, T. Hinzmann, and R. Siegwart, “Visual place recognition with probabilistic voting,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , 2017, pp. 3192–3199.
  * [9] L.-P. Morency and T. Baltrušaitis, “Multimodal machine learning: Integrating language, vision and speech,” in _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts_. Vancouver, Canada: Association for Computational Linguistics, Jul. 2017, pp. 3–5. [Online]. Available: https://www.aclweb.org/anthology/P17-5002
  * [10] W. Maddern, G. Pascoe, C. Linegar, and P. Newman, “1 Year, 1000km: The Oxford RobotCar Dataset,” _The International Journal of Robotics Research (IJRR)_ , vol. 36, no. 1, pp. 3–15, 2017.
  * [11] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2012.
  * [12] W. Wang, D. Tran, and M. Feiszli, “What makes training multi-modal classification networks hard?” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 12 695–12 705.
  * [13] J. Revaud, J. Almazán, R. S. de Rezende, and C. R. de Souza, “Learning with average precision: Training image retrieval with a listwise loss,” _CoRR_ , vol. abs/1906.07589, 2019. [Online]. Available: http://arxiv.org/abs/1906.07589
  * [14] J. Sanchez, F. Perronnin, T. Mensink, and J. Verbeek, “Image Classification with the Fisher Vector: Theory and Practice,” _International Journal of Computer Vision_ , vol. 105, no. 3, pp. 222–245, Dec. 2013. [Online]. Available: https://hal.inria.fr/hal-00830491
  * [15] H. Jégou, M. Douze, C. Schmid, and P. Pérez, “Aggregating local descriptors into a compact image representation,” in _IEEE Conference on Computer Vision & Pattern Recognition_, jun 2010. [Online]. Available: http://lear.inrialpes.fr/pubs/2010/JDSP10
  * [16] S. Liu and W. Deng, “Very deep convolutional neural network based image classification using small training sample size,” in _2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR)_ , 2015, pp. 730–734.
  * [17] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [18] F. Radenović, G. Tolias, and O. Chum, “Fine-tuning cnn image retrieval with no human annotation,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 41, no. 7, pp. 1655–1668, 2018.
  * [19] M. Angelina Uy and G. Hee Lee, “Pointnetvlad: Deep point cloud based retrieval for large-scale place recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4470–4479.
  * [20] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 652–660.
  * [21] Z. Liu, S. Zhou, C. Suo, P. Yin, W. Chen, H. Wang, H. Li, and Y.-H. Liu, “Lpd-net: 3d point cloud learning for large-scale place recognition and environment analysis,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 2831–2840.
  * [22] L. Hui, M. Cheng, J. Xie, and J. Yang, “Efficient 3d point cloud feature learning for large-scale place recognition,” _arXiv preprint arXiv:2101.02374_ , 2021.
  * [23] Y. Xia, Y. Xu, S. Li, R. Wang, J. Du, D. Cremers, and U. Stilla, “Soe-net: A self-attention and orientation encoding network for point cloud based place recognition,” _arXiv preprint arXiv:2011.12430_ , 2020.
  * [24] J. Komorowski, “Minkloc3d: Point cloud based large-scale place recognition,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , January 2021, pp. 1790–1799.
  * [25] T. Lin, P. Dollár, R. B. Girshick, K. He, B. Hariharan, and S. J. Belongie, “Feature pyramid networks for object detection,” _CoRR_ , vol. abs/1612.03144, 2016. [Online]. Available: http://arxiv.org/abs/1612.03144
  * [26] Y. Pan, X. Xu, W. Li, Y. Wang, and R. Xiong, “Coral: Colored structural representation for bi-modal place recognition,” _arXiv preprint arXiv:2011.10934_ , 2020.
  * [27] S. Xie, C. Pan, Y. Peng, K. Liu, and S. Ying, “Large-scale place recognition based on camera-lidar fused descriptor,” _Sensors_ , vol. 20, no. 10, p. 2870, 2020.
  * [28] Y. Lu, F. Yang, F. Chen, and D. Xie, “Pic-net: Point cloud and image collaboration network for large-scale place recognition,” 2020.
  * [29] A. Oertel, T. Cieslewski, and D. Scaramuzza, “Augmenting visual place recognition with structural cues,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 4, pp. 5534–5541, 2020.
  * [30] Q. Wang, B. Wu, P. Zhu, P. Li, W. Zuo, and Q. Hu, “Eca-net: Efficient channel attention for deep convolutional neural networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [31] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2117–2125.
  * [32] J. Lu, J. Hu, and J. Zhou, “Deep metric learning for visual understanding: An overview of recent advances,” _IEEE Signal Processing Magazine_ , vol. 34, no. 6, pp. 76–84, 2017.
  * [33] A. Hermans, L. Beyer, and B. Leibe, “In defense of the triplet loss for person re-identification,” _arXiv preprint arXiv:1703.07737_ , 2017.
  * [34] Z. Zhong, L. Zheng, G. Kang, S. Li, and Y. Yang, “Random erasing data augmentation,” _arXiv preprint arXiv:1708.04896_ , 2017.
  * [35] A. Paszke _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” in _Advances in Neural Information Processing Systems 32_ , H. Wallach _et al._ , Eds. Curran Associates, Inc., 2019, pp. 8024–8035. [Online]. Available: http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * [36] C. Choy, J. Gwak, and S. Savarese, “4d spatio-temporal convnets: Minkowski convolutional neural networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3075–3084.
  * [37] K. Musgrave, S. Belongie, and S.-N. Lim, “A metric learning reality check,” _arXiv preprint arXiv:2003.08505_ , 2020.
  * [38] W. Zhang and C. Xiao, “Pcan: 3d attention map learning using contextual information for point cloud based retrieval,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 12 436–12 445.
  * [39] J. Du, R. Wang, and D. Cremers, “Dh3d: Deep hierarchical 3d descriptors for robust large-scale 6dof relocalization,” in _European Conference on Computer Vision (ECCV)_ , 2020.
  * [40] T. Sattler, W. Maddern, C. Toft, A. Torii, L. Hammarstrand, E. Stenborg, D. Safari, M. Okutomi, M. Pollefeys, J. Sivic _et al._ , “Benchmarking 6dof outdoor visual localization in changing conditions,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8601–8610.
  * [41] A. Torii, R. Arandjelovic, J. Sivic, M. Okutomi, and T. Pajdla, “24/7 place recognition by view synthesis,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 1808–1817.
  * [42] G. Tolias, R. Sicre, and H. Jégou, “Particular object retrieval with integral max-pooling of cnn activations,” in _ICL 2016-RInternational Conference on Learning Representations_ , 2016, pp. 1–12.
  * [43] A. B. Yandex and V. Lempitsky, “Aggregating local deep features for image retrieval,” in _2015 IEEE International Conference on Computer Vision (ICCV)_ , 2015, pp. 1269–1277.
  * [44] A. Miech, I. Laptev, and J. Sivic, “Learnable pooling with context gating for video classification,” _arXiv:1706.06905_ , 2017.