# MinkLoc3D: Point Cloud Based Large-Scale Place Recognition

Jacek Komorowski
Warsaw University of Technology
Warsaw, Poland
<EMAIL_ADDRESS>
###### Abstract

The paper presents a learning-based method for computing a discriminative 3D
point cloud descriptor for place recognition purposes. Existing methods, such
as PointNetVLAD, are based on unordered point cloud representation. They use
PointNet as the first processing step to extract local features, which are
later aggregated into a global descriptor. The PointNet architecture is not
well suited to capture local geometric structures. Thus, state-of-the-art
methods enhance vanilla PointNet architecture by adding different mechanism to
capture local contextual information, such as graph convolutional networks or
using hand-crafted features. We present an alternative approach, dubbed
MinkLoc3D, to compute a discriminative 3D point cloud descriptor, based on a
sparse voxelized point cloud representation and sparse 3D convolutions. The
proposed method has a simple and efficient architecture. Evaluation on
standard benchmarks proves that MinkLoc3D outperforms current state-of-the-
art. Our code is publicly available on the project website.
111https://github.com/jac99/MinkLoc3D

##  1 Introduction

Applying deep learning methods to solve 3D computer vision problems is an area
of active development. A number of methods for classification [28], semantic
segmentation [28, 7] and local [8] or global [1] features extraction from 3D
point clouds was recently proposed. We focus our attention on finding a
discriminative, low-dimensional 3D point cloud descriptor for place
recognition purposes. Localization is performed by searching the database for
geo-tagged point clouds with descriptors closest to the query point cloud
descriptor. The idea is illustrated in Fig. 1. Place recognition methods are
widely used in robotics, autonomous driving [23] and augmented reality [24].

Figure 1: Point cloud-based place recognition. MinkLoc3D computes a global
descriptor of a query point cloud. Localization is performed by searching the
database for geo-tagged point clouds with closest descriptors.

The first learning-based place recognition method operating on 3D point clouds
is PointNetVLAD [1]. It uses PointNet [28] architecture to extract local
features and NetVLAD [2] layer to aggregate them into a global descriptor.
While PointNet proved to be successful in many applications, it was originally
proposed to process point clouds representing single objects, not large and
complex scenes. It is not well suited extract informative local features. To
overcome this weakness, latter 3D place recognition methods enhance vanilla
PointNet architecture by adding different mechanism to capture local
contextual information. PCAN [41] uses sampling and grouping operation at
multiple scales. State-of-the-art LPD-Net [19] method uses rather complex
architecture and combines learning-based and handcrafted local features. 3D
points enhanced with pre-computed handcrafted features are processed by a
PointNet module, fed to a graph neural network to aggregate neighbourhood
information and further processed using Point Net architecture. Finally, a
global descriptor is computed using NetVLAD [2] layer. LPD-Net surpasses
previous state-of-the-art by a large margin. However, at the expense of
architectural and computational complexity.

Increasing complexity of recent 3D point cloud-based place recognition
methods, all based on unordered set of points representation, motivated us to
investigate feasibility of using alternate approach and network architecture.
We choose sparse voxelized representation and sparse convolutions, as recently
it proved successful in many 3D vision tasks, including local feature
extraction [8], semantic segmentation [7] and point cloud registration [6].

Our method, dubbed MinkLoc3D, has a simple, elegant and effective architecture
and outperforms prior state-of-the-art. MinkLoc3D consists of two parts, local
feature extraction network followed by feature aggregation layer. In order to
produce local features with richer semantic content, we adapted Feature
Pyramid Network (FPN) [18] architecture. The input point cloud is first
quantized into a sparse voxelized representation and processed by a local
feature extraction network. Different than prior methods we use a simple
Generalized-Mean pooling [18] layer, instead of NetVLAD [2] layer, to
aggregate local features into a discriminative global point cloud descriptor.

MinkLoc3D achieves state-of-the-art results on standard 3D place recognition
benchmarks. It outperforms PointNetVLAD [1] by a large margin. It improves
over the current state-of-the-art LPD-Net [19] despite having simpler
architecture and being more computationally effective. Comparison with vision-
based RobotCar Seasons [34] benchmark proves its robustness to challenging
environmental conditions.

Our main contribution is the development of a global point cloud descriptor
extraction method based on an alternate point cloud representation and network
architecture than prior state-of-the-art. Our MinkLoc3D method advances state-
of-the-art on the popular benchmarks. It proves the potential of using sparse
voxelized representation and sparse convolutions for efficient extraction of
discriminative features from 3D point clouds. We believe our work can spark
further improvements in the point cloud-based place recognition field by
showing promising development direction.

##  2 Related work

#### Point cloud representation for deep learning.

Early deep learning methods for 3D point cloud processing use volumetrically
discretized representations [22]. It’s a natural extension of 2D image
representation as a grid of pixels and 3D convolutions can be used to
effectively process such data. However, such representation is very
inefficient. The memory requirement grows cubically as spatial resolution
increases, making it inappropriate for processing larger point clouds.

PointNet [28] is the first deep learning architecture operating directly on
raw 3D point clouds. Each point is processed in isolation by multi-layer
perceptrons and point features are aggregated using a symmetric max pooling
function. This makes the architecture independent from input points ordering.
The drawback is that it cannot capture local geometric structures and has
limited ability to recognize fine-grained patterns. To alleviate this problem,
PointNet++ [29] enhances PointNet with hierarchical processing.

An alternative is to use sparse voxelized representation [12]. This allows
using 3D convolutions to effectively capture local structures and patterns,
similarly as 2D convolutions do in 2D images. However, naive implementations
are computationally inefficient. Recently, an auto-differentiation library for
sparse tensors, so called Minkowski Engine
222https://github.com/NVIDIA/MinkowskiEngine, was proposed [7]. It efficiently
implements sparse convolutions by using coordinate hashing. Sparse voxelized
representation proved successful and yield state-of-the-art results in many 3D
vision tasks, such as local feature extraction [8] and semantic segmentation
[7].

#### 3D point cloud-based place recognition using learned global features.

PointNetVLAD [1] is the first deep network for large-scale 3D point cloud
retrieval. It combines PointNet [28] architecture to extract local features
and NetVLAD [2] layer to aggregate local features and produce a discriminative
global descriptor. The main weakness of PointNetVlad is its reliance on
PointNet [28] for local feature extraction. PointNet architecture is weak at
capturing local geometric structures which adversely impacts discriminability
of the resultant global descriptor. To overcome this weakness, latter methods
enhance PointNetVlad, by adding different mechanism to extract local
contextual information.

PCAN [41] adds an attention mechanism to predict significance of each point
based on a local context. The input point cloud is first processed using
PointNet architecture to compute local features. Then, sampling and grouping
approach inspired by PointNet++ [29] is used to extract local contextual
information at multiple scales and produce per-point attention map. Finally,
NetVLAD [2] layer aggregates attention-weighted local features into a global
descriptor.

DAGC [36] combines dynamic graph CNN [39] architecture with dual attention
mechanism [11] to aggregate local contextual information at multiple scales.
Local features are aggregated using NetVLAD [2] layer to produce a global
descriptor.

LPD-Net [19] relies on handcrafted features and uses graph neural networks to
extract local contextual information. First, ten handcrafted features, such as
local curvature or point density, are computed for each point. Then, 3D points
enhanced with handcrafted features are processed using Point Net architecture,
fed to a graph neural network to aggregate neighbourhood features and further
processed using Point Net-like architecture. Finally, global descriptor is
computed using NetVLAD [2] layer. The method yields state-of-the-art results,
surpassing previously proposed solutions by a significant margin. However, at
the expense of architectural complexity and high computational cost.

DH3D [9] is a recent 6DoF relocalization method operating on 3D point clouds.
It unifies global place recognition and local 6DoF pose refinement by
inferring local and global 3D descriptors in a single pass through the
network. The local feature extraction module uses Flex Convolution (FlexConv)
[13] and Squeeze-and-Excitation (SE) [15] blocks to fuse multi-level spatial
contextual information and channel-wise feature correlations into local
descriptors. NetVLAD [2] layer aggregates attention-weighted local features
into a global point cloud descriptor.

#### Deep metric learning.

Deep metric learning [20] uses deep neural networks to compute a non-linear
mapping from a high dimensional data point space to a low-dimensional
Euclidean space, known as a representation or embedding space. The learned
mapping preserves semantic similarity between objects. This technique is
widely used in many recognition tasks in computer vision domain, such as
pedestrian re-identification [14] and image retrieval [17]. Early deep metric
learning methods use a Siamese architecture trained with a contrastive loss
[3]. Latter methods propose more complex loss functions, such as triplet [14]
or quadruplet [5] loss. Significant attention is put to a selection of an
effective sampling scheme to choose informative training samples, so called
hard negatives mining [40]. One of the most popular schemes is _batch hard_
negative mining proposed in [14], which constructs training triplets by
selecting the hardest positive and negative examples within each mini-batch.
In the last few years a number of more sophisticated loss function
formulations and sampling schemes was proposed [40, 38, 4]. However, recent
works [26, 31] suggest that their advantage over classic contrastive or
triplet margin loss is moderate at best. Based on these findings we choose
triplet margin loss when training our network.

##  3  MinkLoc3D: global point cloud descriptor for place recognition

Our goal is to compute a discriminative and generalizable global descriptor
from the input point cloud given as an unordered set of 3D coordinates. This
section describes the proposed architecture and training process of the
network computing such descriptor.

###  3.1 Network architecture

Our network has a very simple architecture shown in Fig. 2, yet it proved to
be more effective and efficient than state-of-the-art methods on standard
benchmarks. It consists of two parts: local feature extraction network and
generalized-mean (GeM) pooling [30] layer. Input point cloud
$P=\left\\{\left(x_{i},y_{i},z_{i}\right)\right\\}$, in the form of a set of
3D point coordinates, is first quantized into a single channel sparse tensor
$\hat{P}=\left\\{\left(\hat{x}_{i},\hat{y}_{i},\hat{z}_{i},1\right)\right\\}$.
The values of this single channel are set to one for non-empty voxels. The
sparse tensor if fed to the local feature extraction network, which produces a
sparse 3D feature map
$\hat{F}=\left\\{\left(\hat{x}_{j},\hat{y}_{j},\hat{z}_{j},f_{j}^{(1)},\ldots,f_{j}^{(c)}\right)\right\\}$,
where $c$ is a feature dimensionality (256 in our experiments),
$\hat{x}_{j},\hat{y}_{j},\hat{z}_{j}$ quantized coordinates and
$f_{j}^{(1)},\ldots,f_{j}^{(c)}$ features of $j$-th feature map element. The
sparse 3D feature map $\hat{F}$ is pooled using a generalized-mean (GeM)
pooling [30] layer, which produces a global descriptor vector
$\boldsymbol{g}$. GeM is generalization of a global max pooling and global
average pooling operators and is defined as:
$g^{(k)}=\left(\frac{1}{n}\sum_{j=1\ldots
n}\left(f_{j}^{(k)}\right)^{p}\right)^{\frac{1}{p}}$, where $g^{(k)}$ is
$k$-th element of the global descriptor vector $\boldsymbol{g}$, $n$ is a size
(number of non-zero elements) in the sparse local feature map $\hat{F}$,
$f_{j}^{(k)}$ is $k$-th feature of the $j$-th local feature map element and
$p$ is a learnable pooling parameter.

Figure 2: MinkLoc3D architecture. The input point cloud is quantized into a
sparse, single channel, 3D tensor. Local features are extracted using a 3D
Feature Pyramid Network [18] architecture. Generalized-mean (GeM) [30] pooling
produces a global point cloud descriptor. Numbers in local feature extraction
module (e.g. 1/32) denote a stride and number of channels of a feature map
produced by each block.

The design of the local feature extraction network is inspired by MinkowskiNet
[7] sparse convolutional network architecture, and Feature Pyramid Network
[18] design pattern. Bottom-up part part of the network contains four
convolutional blocks producing sparse 3D feature maps with decreasing spatial
resolution and increasing receptive field. The top-down part contains a
transposed convolution generating an upsampled feature map. Upsampled feature
map is concatenated with the skipped features from the corresponding layer of
the bottom-up pass using a lateral connection. Such design is intended to
produce a feature map with relatively high spatial resolution and large
receptive field. Our initial experiments proved its advantage over a simple
convolutional architecture without top-down processing.

Block | Layers
---|---
Conv0 | $\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 32$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$5_{k}1_{s}$}$}\end{array}$
Conv1 | $\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 32$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$2_{k}2_{s}$}$}\end{array}\Big{\langle}\;\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 32$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$3_{k}1_{s}$}$}\end{array}\;\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 32$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$3_{k}1_{s}$}$}\end{array}\Big{\rangle}$
Conv2 | $\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 64$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$2_{k}2_{s}$}$}\end{array}\Big{\langle}\;\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 64$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$3_{k}1_{s}$}$}\end{array}\;\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 64$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$3_{k}1_{s}$}$}\end{array}\Big{\rangle}$
Conv3 | $\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 64$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$2_{k}2_{s}$}$}\end{array}\Big{\langle}\;\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 64$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$3_{k}1_{s}$}$}\end{array}\;\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 64$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$3_{k}1_{s}$}$}\end{array}\Big{\rangle}$
1x1Conv2 | $\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 256$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$1_{k}1_{s}$}$}\end{array}$
1x1Conv3 | $\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 256$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$1_{k}1_{s}$}$}\end{array}$
TConv3 | $\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\ \kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$t$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt] \kern 8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\ 256$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$2_{k}2_{s}$}$}\end{array}$
Table 1: Details of a local feature extraction part of MinkLoc3D network. All
convolutions in bottom-up Conv$0\ldots 3$ blocks are followed by batch norm
and ReLU non-linearity. $\langle\ldots\rangle$ denotes a residual block.

Tab. 1 shows details of each convolutional block in a local feature extraction
network. We use notation introduced in [35], where
$\kern-4.0pt\begin{array}[]{r}\kern 0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\\\
\kern
0.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-12.0pt\begin{array}[]{c}\mbox{\Large$\mathbb{C}$}\\\\[-18.0pt]
\kern
8.0pt\raisebox{0.0pt}{$\mbox{\tiny$$}$}\end{array}\kern-8.0pt\begin{array}[]{l}\kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$\
c$}$}\\\ \kern-2.5pt\raisebox{0.0pt}{$\mbox{\tiny$a_{k}b_{s}$}$}\end{array}$
denotes a convolution with $c$ kernels of shape $a\times a\times a$ and stride
b. $t$ decorator is used to indicate a transposed convolution.
$\langle\ldots\rangle$ denotes a residual block with a skip connection,
defined as $\langle f\rangle\left(x\right)\doteq f\left(x\right)+x$. The first
convolutional block (Conv0) has bigger 5x5x5 kernels, in order to aggregate
information from a larger neighbourhood. Bottom-up blocks (Conv1, Conv2 and
Conv3) are made of a stride two convolution, which decreases spatial
resolution by two, followed by residual block consisting of two convolutional
layers with 3x3x3 kernel. All convolutional layers in bottom-up blocks are
followed by batch normalization [16] layer and ReLU non-linearity. Two 1x1Conv
blocks have the same structure, both contain a single convolutional layer with
1x1x1 kernel. The aim of these blocks is to unify the number of channels in
feature maps produced by bottom-up blocks, before they are merged in the top-
down pass through the network. The top-down part of the network consists of a
single transposed convolution layer (TConv3) with 2x2x2 kernel.

###  3.2 Network training

To train our network we use a deep metric learning approach [20] with a
triplet margin loss [14] defined as:

$L(a_{i},p_{i},n_{i})=\max\left\\{d(a_{i},p_{i})-d(a_{i},n_{i})+m,0\right\\},$

where $d(x,y)=||x-y||_{2}$ is an Euclidean distance between embeddings $x$ and
$y$; $a_{i},p_{i},n_{i}$ are embeddings of an anchor, a positive and a
negative elements in $i$-th training triplet and $m$ is a margin
hyperparameter. The loss function is minimized using a stochastic gradient
descent approach with Adam optimizer.

Previous methods, such as PointNetVLAD [1] and LPD-Net[19], use rather
inefficient training strategy. In order to construct informative triplets, for
each anchor point cloud they sample 2 positive and 18 negative candidates.
Embeddings of all candidate point clouds are calculated and only one hardest
positive and negative example is taken to construct a training triplet. Thus,
21 point clouds need to be processed to construct one triplet.

We developed an alternative, more efficient, training procedure based on batch
hard negative mining approach [14]. At the beginning of each epoch we randomly
partition a training set into batches. A batch of size $n$ is constructed by
sampling $n/2$ pairs of structurally similar elements. After a batch is
constructed, we compute two $n\times n$ boolean masks, one indicating
structurally similar pairs and the other structurally dissimilar. We use hash-
based indexing to efficiently check if two elements are structurally similar,
dissimilar or similarity is indefinite. Then, the batch is fed to the network
to compute embeddings. Using similarity and dissimilarity boolean masks and
computed embeddings, we mine hardest positive and hardest negative examples
and construct informative training triplets. In our approach, processing one
batch consisting of $n$ elements produces $n$ training triplets. This approach
brings down network training time from days to hours.

During experiments we noticed that with larger batch sizes, the training
process is prone to collapse, where all embeddings approach the same value. To
overcome this problem, we use a simple yet effective dynamic batch sizing
strategy. The training starts with a small batch size, say 16 examples. At the
end of each epoch, the average number of active triplets (i.e. triplets
producing non-zero loss) per batch is examined. If the ratio of active
triplets to all triplets in a batch falls below the predefined threshold
$\Theta$, the batch size is increased by a fixed batch expansion rate
$\alpha$.

To increase variability of the training data and reduce overfitting, we apply
on-the-fly data augmentation. It includes random jitter with a value drawn
from a normal distribution $\mathcal{N}\left(\mu=0,\sigma=0.001\right)$;
random translation by a value sampled from $0\ldots 0.01$ range; and removal
of randomly chosen points, where the percentage of points to remove is
uniformly sampled from $0\ldots 10\%$ range. We also adapted a random erasing
augmentation [42] and randomly remove all points within a fronto-parallel
cuboid with a random size and position.

##  4 Experimental results

In this section we describe the datasets and evaluation methodology, compare
our method to the state-of-the-art and conduct ablation study. We also compare
our method with image-based visual localization method on the standard visual
localization benchmark.

###  4.1 Datasets and evaluation methodology

The network is trained and evaluated using a modified Oxford RobotCar dataset
and three in-house datasets: University Sector (U.S.), Residential Area (R.A.)
, Business District (B.D.) introduced in [1]. The datasets are created using a
LiDAR sensor mounted on the car travelling through these four regions at
different times of day and year. Oxford RobotCar dataset is build using SICK
LMS-151 2D LiDAR scans and in-house dataset using Velodyne HDL-64 3D LiDAR.

All point clouds are preprocessed with the ground planes removed and
downsampled to 4096 points. The point coordinates are shifted and rescaled to
be zero mean and inside the $\left[-1,1\right]$ range. See Fig. 4 for
exemplary data items. Training tuples are generated using ground truth UTM
coordinates. Structurally similar point clouds are at most 10m apart.
Dissimilar point clouds are at least 50m apart. For point clouds between 10
and 50m apart similarity is indefinite. Each dataset is split into disjoint
training and test subsets. For more information please refer to [1].

Same as in previous works, we evaluate our network in two scenarios. In
_baseline_ scenario, the network is trained using the training subset of
Oxford dataset and evaluated on test splits of Oxford and in-house datasets.
In _refined_ scenario, the network is trained on the training subset of Oxford
and in-house datasets; and evaluated on test splits of Oxford and in-house
datasets. The number of training and test elements used in each scenario is
shown in Tab. 2.

| Baseline Dataset | Refined Dataset
---|---|---
|  | Training
---
| Test
---
| Training
---
| Test
---
Oxford | 21.7k | 3.0k | 21.7k | 3.0k
In-house |  | -
---
4.5k | 6.7k | 1.7k
Table 2: Number of elements in datasets used in _baseline_ and _refined_
evaluation scenarios.

#### Evaluation metrics

We follow the same evaluation protocol as in [1, 19]. A point cloud from a
testing dataset is taken as a query and point clouds from different traversals
that cover the same region form the database. The query point cloud is
successfully localized if at least one of the top $N$ retrieved database
clouds is within $d=25$ meters from the ground truth position of the query.
_Recall@N_ is defined as the percentage of correctly localized queries. As in
[1] we report Average Recall@1 (AR@1) and Average Recall@1% (AR@1%) metrics.

#### Implementation details.

In all experiments we quantize 3D point coordinates with 0.01 quantization
step. As point coordinates in the Baseline and Refined datasets are normalized
to be in $\left[-1,1\right]$ range, this produces up to 200 voxels in each
spatial direction. Other parameters of the training process are listed in
Tab.3. Initial learning rate is divided by 10 at the epoch given in LR
scheduler steps row. The Refined Dataset is larger and more diverse than
Baseline Dataset, hence in refined scenario the network is trained twice as
long. The dimensionality of the resultant global descriptor is set to 256 ,
same as in prior methods.

|  | Baseline
---
| Refined
---
Initial batch size | 32 | 16
Batch size limit | 256 | 256
Batch expansion threshold $(\Theta)$ | 0.7 | 0.7
Batch expansion rate $(\alpha)$ | 1.4 | 1.4
Number of epochs | 40 | 80
Initial learning rate | 1e-3 | 1e-3
LR scheduler steps | 30 | 60
$L_{2}$ weight decay | 1e-3 | 1e-3
Triplet loss margin $(m)$ | 0.2 | 0.2
Table 3: Parameters of the training process in _baseline_ and _refined_
evaluation scenarios.

All experiments are performed on a server with a single nVidia RTX 2080Ti GPU,
12 core AMD Ryzen Threadripper 1920X processor, 64 GB of RAM and SSD hard
drive. We use PyTorch 1.5 [27] deep learning framework, MinkowskiEngine 0.4.3
[7] auto-differentiation library for sparse tensors and PML Pytorch Metric
Learning library 0.9.88 [26].

###  4.2 Results and discussion

#### Comparison with state-of-the-art.

We compare performance our global descriptor with prior art: PointNetVLAD [1],
PCAN [41], DAGC [36] and LPD-Net [19]. We also include DH3D [9] method in an
evaluation. DH3D is a recent 6DOF localization method, which includes a global
point cloud descriptor computation as a part of the pose estimation pipeline.

Tab. 4 compares performance of our MinkLoc3D with state-of-the-art methods
trained on the Baseline Dataset. When evaluated on Oxford dataset, MinkLoc3D
wins with AR@1% 3.0 p.p. higher than the runner-up, LPD-Net. When evaluated on
three in-house datasets, it performs slightly worse compared than LPD-net (1.0
and 0.6 p.p. worse for U.S. and B.D. sets respectively and 0.7 p.p. better for
R.A.). It must be noted that Oxford dataset and three in-house datasets were
acquired using LiDARs with different characteristics. Even so, our method
yields comparable results to LPD-Net which relies on hand-crafted features.
MinkLoc3D discriminability and generalization capability is significantly
higher than all other fully learning based methods.

|  | Oxford
---
| U.S.
---
| R.A.
---
| B.D.
---
PointNetVLAD [1] | 80.3 | 72.6 | 60.3 | 65.3
PCAN [41] | 83.8 | 79.1 | 71.2 | 66.8
DH3D-4096 [9] | 84.3 | - | - | -
DAGC [36] | 87.5 | 83.5 | 75.7 | 71.2
LPD-Net [19] | 94.9 | 96.0 | 90.5 | 89.1
MinkLoc3D (our) | 97.9 | 95.0 | 91.2 | 88.5
Table 4: Evaluation results (Average Recall at 1%) of place recognition
methods trained on the Baseline Dataset.

Tab. 5 shows evaluation results of state-of-the-art methods trained on a
larger and more diverse Refined Dataset. For PointNetVLAD and PCAN we run the
evaluation using the trained models provided by authors. LPD-Net was trained
from scratch and evaluated on Refined Dataset using the open source code. Our
MinkLoc3D is a clear winner. Compared to the state-of-the-art LPD-Net, the
AR@1% is higher between 0.8 p.p. and 2.9 p.p. on all evaluation subsets. The
advantage over other methods is even higher, between 5-18 p.p. Average Recall
plots in Fig. 3 show that our method outperforms previous methods on all
evaluation subsets.

| Oxford | U.S. | R.A. | B.D.
---|---|---|---|---
|  | AR@1%
---
| AR@1
---
| AR@1%
---
| AR@1
---
| AR@1%
---
| AR@1
---
| AR@1%
---
| AR@1
---
PointNetVLAD [1] | 80.1 | 63.3 | 94.5 | 86.1 | 93.1 | 82.7 | 86.5 | 80.1
PCAN [41] | 86.4 | 70.7 | 94.1 | 83.7 | 92.3 | 82.3 | 87.0 | 80.3
DAGC [36] | 87.8 | 71.5 | 94.3 | 86.3 | 93.4 | 82.8 | 88.5 | 81.3
LPD-Net [19] | 94.9 | 86.6 | 98.9 | 94.4 | 96.4 | 90.8 | 94.4 | 90.8
MinkLoc3D (our) | 98.5 | 94.8 | 99.7 | 97.2 | 99.3 | 96.7 | 96.7 | 94.0
Table 5: Evaluation results (Average Recall at 1% and at 1) of place
recognition methods trained on the Refined Dataset.

(a) Oxford (b) U.S. (c) R.A. (d) B.D.

Figure 3: Average recall of place recognition methods trained on the Refined
Dataset.

Tab. 6 compares the number of trainable parameters and inference time (runtime
per cloud). Our MinkLoc3D method is significantly faster compared to LPD-Net.
LPD-Net requires time consuming preprocessing of the input point cloud to
compute 10 handcrafted features. Even without including hand-crafted feature
extraction time, LPD-Net has longer inference time compared to MinkLoc3D (26
vs 22 ms). Our model is also much lighter compared to prior methods. It has
only 1.5 million trainable parameters, whereas other methods have an order of
magnitude more. This can be explained by the fact, that our method produces
informative local features, that can be pooled using a simple Generalized-Mean
pooling [25] which has few learnable parameters. Other methods use NetVLAD [2]
aggregation layer with millions of learnable parameters.

|  | Parameters
---
| Runtime
---
per cloud
PointNetVLAD [1] | 19.8M | 15 ms
PCAN [41] | 20.4M | 55 ms
LPD-Net [19] | 19.8M | 26 ms
LPD-Net [19] with f.e. | 19.8M | 917 ms
MinkLoc3D (our) | 1.1M | 21 ms
Table 6: Computation time required by different methods. _LPD-Net with f.e._
includes hand-crafted features extraction time.

Figure 4 visualizes nearest neighbour search results using our MinkLoc3D
descriptor in Oxford evaluation subset. The leftmost column shows a query
point cloud and other columns show its five nearest neighbours. Figure 5 shows
failure cases. More visualizations of nearest neighbour search results can be
found in Supplementary Material.

(a) query

(b) dist=1.13 TP

(c) dist=1.40 TP

(d) dist=1.44 TP

(e) dist=1.45 TP

(f) dist=1.82 FP

Figure 4: Nearest neighbours search results in Oxford evaluation subset. The
leftmost column shows a query point cloud. Other columns show its five nearest
neighbours. _dist_ is an Euclidean distance in the descriptor space. TP
indicates true positive and FP false positive.

(a)

(b)

(c)

(a)

(b)

(c)

Figure 5: Failure cases. Examples of unsuccessful retrieval results using our
network. (a) is the query point cloud, (b) incorrect match to the query and
(c) the closest true match.

#### Ablation study.

In this section we investigate impact of the network design choices on the
discriminativity and generalization capability of our method. In all
experiments, the network is trained using the Baseline Dataset and evaluated
on Oxford and three in-house datasets (U.S., R.A. and B.D.).

Tab. 7 shows the impact of a feature aggregation method on the performance of
the global descriptor. The following methods are evaluated: global max pooling
(MAC), Generalized-Mean (GeM) pooling [30], NetVLAD [2] and NetVLAD with
Context Gating [25] (NetVLAD-CG). Surprisingly, a simple GeM layer with few
learnable parameters produces the most discriminative global descriptors and
has the best generalization capability. More sophisticated methods, NetVLAD
and NetVLAD with Context Gating, score similarly on Oxford dataset, but
noticeably worse on in-house datasets. This can be attributed to two factors.
First, our training datasets have a moderate size. Using NetVLAD layer, with
millions of trainable parameters, increases the risk of overfitting. Second,
our local feature extraction network works very well and produces informative
features, that can be effectively pooled using a simple GeM layer to produce a
discriminative global descriptor.

| Architecture
---
| Oxford
---
AR@1%
| In-house
---
AR@1%
MinkFPN+MAC | 97.3 | 92.4
MinkFPN+GeM | 97.9 | 93.2
MinkFPN+NetVLAD | 97.0 | 91.1
MinkFPN+NetVLAD-CG | 97.2 | 84.7
Table 7: Impact of a feature aggregation method on the discriminability of the
global descriptor. The network is trained on the Baseline Dataset. * indicates
MinkLoc3D architecture.

Tab. 8 shows impact of a descriptor size on the discriminability of the global
descriptor. The number of channels in lateral connections (1x1Conv2, 1x1Conv3
blocks) and in a transposed convolution TConv3 block is set to the same value
as the dimensionality of the final descriptor. Parameters of bottom-up
convolutional blocks remain unchanged. The network performance is relatively
similar with larger descriptor sizes (between 64 and 512) with AR@1% between
97.3 and 98.0% on Oxford dataset and between 90.3 and 93.2% on in-house
datasets. The performance deteriorates, when the descriptor size falls to 32.

| Descriptor
---
size
| Oxford
---
AR@1%
| In-house
---
AR@1%
512 | 98.0 | 92.6
256 | 97.9 | 93.2
128 | 97.5 | 91.5
64 | 97.3 | 90.3
32 | 95.8 | 86.4
Table 8: Impact of a descriptor size on the discriminability of the global
descriptor. The network is trained on the Baseline Dataset.

#### Comparison with image based methods.

In this paragraph we compare performance of our MinkLoc3D with state-of-the-
art image-based place recognition and visual localization methods in
challenging environmental conditions. The comparison is done using RobotCar
Seasons [34] dataset. It contains outdoor images captured in the city of
Oxford at various periods of a year in different atmospheric conditions, e.g.
snow, rain, dawn or night.

| day conditions | night conditions
---|---|---
|  | dawn
---
| dusk
---
| overcast
---
summer
| overcast
---
winter
| rain
---
| snow
---
| sun
---
| night
---
| night-rain
---
DenseVLAD [37] | 92.5 | 94.2 | 92.0 | 93.3 | 96.9 | 90.2 | 80.2 | 19.9 | 25.5
NetVLAD [2] | 82.6 | 92.9 | 95.2 | 92.6 | 96.0 | 91.8 | 86.7 | 15.5 | 16.4
NetVLAD+SP [32] | 90.3 | 96.7 | 98.1 | 96.2 | 97.6 | 95.9 | 94.1 | 35.4 | 33.4
DenseVLAD+D2-Net [10] | 94.4 | 95.9 | 98.3 | 96.2 | 96.9 | 94.9 | 91.1 | 53.9 | 56.1
NetVLAD+SP+SG [33] | 97.3 | 97.2 | 99.8 | 96.7 | 98.1 | 97.8 | 96.1 | 91.9 | 92.0
LPD-Net [19] | 79.7 | 79.9 | 79.7 | 73.8 | - | - | 82.3 | 77.3 | 32.8
MinkLoc3D (our) | 89.2 | 88.3 | 90.3 | 83.1 | 66.3 | 86.3 | 87.4 | 86.1 | 58.0
Table 9: Comparisons with 6DoF visual localization methods on RobotCar Seasons
dataset. We report percentage of queries correctly localized within 5 meter
and 10∘ threshold. Five top rows show performance of image-based methods and
two bottom rows LiDAR scan-based methods.

We compare MinkLoc3D performance against place recognition methods based on a
global image descriptor: DenseVLAD [37] and NetVLAD [2]; and against full 6DoF
(6 degree-of-freedom) relocalization methods: NetVLAD+SP [32],
DenseVLAD+D2-Net [10] and NetVLAD+SP+SG [33]. For each image in RobotCar
Seasons dataset, we find LiDAR readings with corresponding timestamps in the
in original RobotCar dataset [21] and construct the point cloud. Then, we use
MinkLoc3D network to compute a global point cloud descriptors and link this
descriptor with a corresponding image. To approximate a 6DoF pose of a query
image, we search for a database image which descriptor (computed from its
corresponding point cloud), is closest to the descriptor of a query image
(computed from a corresponding point cloud). Then, we return the known pose of
a database image as an approximation of the query image pose. Retrieved poses
are evaluated using the online evaluation service at _Long-term visual
localization_ site.333https://www.visuallocalization.net/

Results are shown in Tab. 9. In day conditions image-based methods perform
generally better, with up to 10 p.p. more correctly localized queries.
However, both LiDAR based methods (ours and LPD-Net) operate on relative
small, downsampled point clouds with 4096 points. Even with this small number
of points both methods perform reasonably well. Also both LiDAR based methods
approximate 6DoF pose by taking the pose of the closest nearest neighbour
found. Full 6DoF localization methods, NetVLAD+SP, DenseVLAD+D2 Net and
NetVLAD+SP+SG, employ much more sophisticated approach, where candidate
matches found using a global descriptor are filtered by matching local
features with geometric consistency criteria. In night conditions point cloud-
based method show their potential. Our method surpass all image-based methods
by a large margin with the exception of the latest NetVLAD+SP+SG, which has
slightly higher performance.

##  5 Conclusion

In this paper we present MinkLoc3D, a novel 3D point cloud descriptor, based
on a sparse voxelized point cloud representation and 3D FPN [18] architecture.
Extensive experimental evaluation proves that it outperforms prior cloud-based
place recognition methods. The success of our method can be attributed to two
factors. First, sparse convolutional architecture can produces informative
local features, that can be used to construct a discriminative global point
cloud descriptor. Second, improvements in the training process allows
efficient and effective training with larger batch size, which positively
affects discriminability and generalization capability of the resultant
descriptor. The natural next step is to enhance the proposed method to build
full 6DoF localization solution.

It should be also noted, that achieved results (AR@1% between 96.7% and 99.4%
when trained on Refined Dataset) show that standard benchmarks used to train
and evaluate point cloud-based place recognition methods are close to being
saturated and there’s a little room for improvement. Larger and more diverse
datasets would be needed to instigate further progress.

#### Acknowledgements

The project was funded by POB Research Centre for Artificial Intelligence and
Robotics of Warsaw University of Technology within the Excellence Initiative
Program - Research University (ID-UB).

## References

  * [1] Mikaela Angelina Uy and Gim Hee Lee.  Pointnetvlad: Deep point cloud based retrieval for large-scale place recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4470–4479, 2018.
  * [2] Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic.  Netvlad: Cnn architecture for weakly supervised place recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5297–5307, 2016.
  * [3] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah.  Signature verification using a” siamese” time delay neural network.  In Advances in neural information processing systems, pages 737–744, 1994.
  * [4] Fatih Cakir, Kun He, Xide Xia, Brian Kulis, and Stan Sclaroff.  Deep metric learning to rank.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1861–1870, 2019.
  * [5] Weihua Chen, Xiaotang Chen, Jianguo Zhang, and Kaiqi Huang.  Beyond triplet loss: a deep quadruplet network for person re-identification.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 403–412, 2017.
  * [6] Christopher Choy, Wei Dong, and Vladlen Koltun.  Deep global registration.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2514–2523, 2020.
  * [7] Christopher Choy, JunYoung Gwak, and Silvio Savarese.  4d spatio-temporal convnets: Minkowski convolutional neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3075–3084, 2019.
  * [8] Christopher Choy, Jaesik Park, and Vladlen Koltun.  Fully convolutional geometric features.  In Proceedings of the IEEE International Conference on Computer Vision, pages 8958–8966, 2019.
  * [9] Juan Du, Rui Wang, and Daniel Cremers.  Dh3d: Deep hierarchical 3d descriptors for robust large-scale 6dof relocalization.  In European Conference on Computer Vision (ECCV), 2020.
  * [10] Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler.  D2-net: A trainable cnn for joint description and detection of local features.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8092–8101, 2019.
  * [11] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu.  Dual attention network for scene segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3146–3154, 2019.
  * [12] Ben Graham.  Sparse 3d convolutional neural networks.  In Proceedings of the British Machine Vision Conference (BMVC), pages 150.1–150.9. BMVA Press, September 2015.
  * [13] Fabian Groh, Patrick Wieschollek, and Hendrik PA Lensch.  Flex-convolution.  In Asian Conference on Computer Vision, pages 105–122. Springer, 2018.
  * [14] Alexander Hermans, Lucas Beyer, and Bastian Leibe.  In defense of the triplet loss for person re-identification.  arXiv preprint arXiv:1703.07737, 2017.
  * [15] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018.
  * [16] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  volume 37 of Proceedings of Machine Learning Research, pages 448–456, Lille, France, 07–09 Jul 2015. PMLR.
  * [17] Jung-Eun Lee, Rong Jin, and Anil K Jain.  Rank-based distance metric learning: An application to image retrieval.  In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8. IEEE, 2008.
  * [18] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [19] Zhe Liu, Shunbo Zhou, Chuanzhe Suo, Peng Yin, Wen Chen, Hesheng Wang, Haoang Li, and Yun-Hui Liu.  Lpd-net: 3d point cloud learning for large-scale place recognition and environment analysis.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2831–2840, 2019.
  * [20] Jiwen Lu, Junlin Hu, and Jie Zhou.  Deep metric learning for visual understanding: An overview of recent advances.  IEEE Signal Processing Magazine, 34(6):76–84, 2017.
  * [21] Will Maddern, Geoff Pascoe, Chris Linegar, and Paul Newman.  1 Year, 1000km: The Oxford RobotCar Dataset.  The International Journal of Robotics Research (IJRR), 36(1):3–15, 2017.
  * [22] Daniel Maturana and Sebastian Scherer.  Voxnet: A 3d convolutional neural network for real-time object recognition.  In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 922–928. IEEE, 2015.
  * [23] Colin McManus, Winston Churchill, Will Maddern, Alexander D Stewart, and Paul Newman.  Shady dealings: Robust, long-term visual localisation using illumination invariance.  In 2014 IEEE international conference on robotics and automation (ICRA), pages 901–906. IEEE, 2014.
  * [24] Sven Middelberg, Torsten Sattler, Ole Untzelmann, and Leif Kobbelt.  Scalable 6-dof localization on mobile devices.  In European conference on computer vision, pages 268–283. Springer, 2014.
  * [25] Antoine Miech, Ivan Laptev, and Josef Sivic.  Learnable pooling with context gating for video classification.  arXiv:1706.06905, 2017.
  * [26] Kevin Musgrave, Serge Belongie, and Ser-Nam Lim.  A metric learning reality check.  arXiv preprint arXiv:2003.08505, 2020.
  * [27] Adam Paszke et al.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach et al., editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [28] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.
  * [29] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  In Advances in neural information processing systems, pages 5099–5108, 2017.
  * [30] Filip Radenović, Giorgos Tolias, and Ondřej Chum.  Fine-tuning cnn image retrieval with no human annotation.  IEEE transactions on pattern analysis and machine intelligence, 41(7):1655–1668, 2018.
  * [31] Karsten Roth, Timo Milbich, Samarth Sinha, Prateek Gupta, Björn Ommer, and Joseph Paul Cohen.  Revisiting training strategies and generalization performance in deep metric learning, 2020.
  * [32] Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk.  From coarse to fine: Robust hierarchical localization at large scale.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12716–12725, 2019.
  * [33] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superglue: Learning feature matching with graph neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4938–4947, 2020.
  * [34] Torsten Sattler, Will Maddern, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Marc Pollefeys, Josef Sivic, et al.  Benchmarking 6dof outdoor visual localization in changing conditions.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8601–8610, 2018.
  * [35] Władysław Skarbek.  Symbolic tensor neural networks for digital media–from tensor processing via bnf graph rules to creams applications.  Fundamenta Informaticae, 168(2-4):89–184, 2019.
  * [36] Qi Sun, Hongyan Liu, Jun He, Zhaoxin Fan, and Xiaoyong Du.  Dagc: Employing dual attention and graph convolution for point cloud based place recognition.  In Proceedings of the 2020 International Conference on Multimedia Retrieval, pages 224–232, 2020.
  * [37] Akihiko Torii, Relja Arandjelovic, Josef Sivic, Masatoshi Okutomi, and Tomas Pajdla.  24/7 place recognition by view synthesis.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1808–1817, 2015.
  * [38] Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott.  Multi-similarity loss with general pair weighting for deep metric learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5022–5030, 2019.
  * [39] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon.  Dynamic graph cnn for learning on point clouds.  Acm Transactions On Graphics (tog), 38(5):1–12, 2019.
  * [40] Chao-Yuan Wu, R Manmatha, Alexander J Smola, and Philipp Krahenbuhl.  Sampling matters in deep embedding learning.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2840–2848, 2017.
  * [41] Wenxiao Zhang and Chunxia Xiao.  Pcan: 3d attention map learning using contextual information for point cloud based retrieval.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12436–12445, 2019.
  * [42] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang.  Random erasing data augmentation.  arXiv preprint arXiv:1708.04896, 2017.


# Learning Soft-Attention Models for Tempo-invariant Audio-Sheet Music
Retrieval

###### Abstract

Connecting large libraries of digitized audio recordings to their
corresponding sheet music images has long been a motivation for researchers to
develop new cross-modal retrieval systems. In recent years, retrieval systems
based on embedding space learning with deep neural networks got a step closer
to fulfilling this vision. However, global and local tempo deviations in the
music recordings still require careful tuning of the amount of temporal
context given to the system. In this paper, we address this problem by
introducing an additional soft-attention mechanism on the audio input.
Quantitative and qualitative results on synthesized piano data indicate that
this attention increases the robustness of the retrieval system by focusing on
different parts of the input representation based on the tempo of the audio.
Encouraged by these results, we argue for the potential of attention models as
a very general tool for many MIR tasks.

##  1 Introduction

Figure 1: Illustration of the cross-modal retrieval application. (a) Audio
snippet serves as input query to the embedding network. The attention network
selects the “relevant” region of the audio spectrogram. (b) 2-dimensional
visualization of the embedding space. The nearest neighbors are selected as
candidates. (c) Ranked list of candidates sorted by their distances to the
query embedding.

Algorithms for content-based search and retrieval play an important role in
many applications that are based on large collections of music data. In this
paper, we re-visit a challenging cross-modal retrieval problem, namely
audio–sheet music retrieval: given a short audio excerpt, we are trying to
retrieve the corresponding score from a database of sheet music (stored as
images).

Traditional methods for audio–sheet retrieval are based on common mid-level
representations that allow for an easy comparison of time points in the audio
and positions in the sheet music, see for instance [14, 17, 21, 2]. Typical
examples are spectral features like pitch class profiles (chroma-features) or
symbolic representations. Unfortunately, deriving such mid-level
representations is an error-prone process, as this may involve preprocessing
steps such as music transcription [6, 9, 19, 26, 16, 18], Optical Music
Recognition [24, 7, 15, 25, 28], and sometimes both. For a recent overview of
different cross-modal retrieval approaches, see [22].

In [11], an alternative approach has been proposed that circumvents these
problematic preprocessing steps by learning embeddings directly from the
multi-modal data (see Figure 1 for a schematic overview). Given short snippets
of audio and their respective excerpts of sheet music images, a cross-modal
neural network is trained to learn an embedding space in which both modalities
are represented as 32-dimensional vectors (Figure 1a). The vectors can then be
easily compared in the embedding space by means of a distance function, e. g.,
the cosine distance (Figure 1b). The retrieval results are then selected by
sorting the candidates by the obtained distances (Figure 1c). A conceptually
similar retrieval approach for videos was presented in [1].

In essence, the neural network replaces the step of extracting mid-level
features by directly learning a transformation from the audio and from the
sheet music image data to a common vector space. A limitation of this approach
is that the temporal context (or field of view) of the data in both modalities
is of fixed size. For audio data, this implies that the actual musical content
of the window depends on the tempo a piece is played in. If it is played in a
fast tempo, the audio excerpt will contain a larger amount of musical content
(i. e., more notes) than if the same piece is played slowly. Obviously, this
can lead to large discrepancies between what the model has seen during
training, and the data it will see during test time.

A possible solution to this problem is to let the network decide the
appropriate temporal context for a given audio query by using a separate
attention mechanism [23, 8, 3, 27]. In a recent workshop paper [12], the
concept of using a soft-attention mechanism in a cross-modal retrieval
scenario was presented. This allowed the network to deal with a much larger
temporal context at the input than without attention. In this paper, we
substantially extend this idea with systematic and quantitative experiments
with respect to tempo robustness, as well as giving further details about the
used architecture and the training data. The remainder of this paper is
structured as follows. In Section 2, we introduce the retrieval task and
explain the necessary steps to approach it with end-to-end cross-modal
embedding learning. Section 3 reports on systematic experiments that show the
benefits of using a dedicated attention network on audio spectrograms, to
improve retrieval results compared to the current state of the art.
Furthermore, we present qualitative examples to highlight the intuitive
behavior of the attention network and conclude the paper in Section 4.

##  2 Audio–Sheet Retrieval

We consider a cross-modal retrieval scenario (Figure 1): given an audio
excerpt as a search query, we wish to retrieve the corresponding snippet of
sheet music of the respective piece. We approach this retrieval problem by
learning a low-dimensional multimodal embedding space ($32$ dimensions) for
both snippets of sheet music and excerpts of music audio (Figure 1a). We
desire for each modality a projection into a shared space where semantically
similar items of the two modalities are projected close together, and
dissimilar items far apart (Figure 1b). Once the inputs are embedded in such a
space, cross-modal retrieval is performed using simple distance measures and
nearest-neighbor search. Finally, the retrieval results are obtained by means
of a ranked list through sorting the distances in an ascending order (Figure
1c).

The embedding space is trained with convolutional neural networks (CNN).
Figure 1a sketches the network architecture. The baseline model (without
attention) consists of two convolutional pathways: one is responsible for
embedding the sheet music, and the other for embedding the audio excerpt. The
key part of the network is the canonically correlated (CCA) embedding layer
[13]. This layer forces the two pathways to learn representations that can be
projected into a shared space. The desired properties of this multimodal
embedding space are enforced by training with pairwise ranking loss (also
known as contrastive loss) [20]. This is the basic structure of the model
recently described and evaluated in [11]. This attention-less model serves as
a baseline in our experiments, i. e., the input audio (or sheet music) has to
be sliced into excerpts of a given size (e. g., 2 s). However, when processing
performed music, the temporal context captured by the fixed-size input
excerpts varies, based on the current tempo of the piece.

Audio (Spectrogram) | Sheet-Image
---|---
$92\times\\{42,84,168\\}$ | $160\times 200$
2x Conv($3$, pad-1)-$24$ \- BN |  }11120pt[Attention Network]
MaxPooling(2) |
2x Conv($3$, pad-1)-$48$ \- BN |
MaxPooling(2) |
2x Conv($3$, pad-1)-$96$ \- BN |
MaxPooling(2) |
2x Conv($3$, pad-1)-$96$ \- BN |
MaxPooling(2) |
Conv($1$, pad-0)-$32$ \- Linear |
GlobalAvgPooling + Softmax |
Mult(Spectrogram, Attention) |
2x Conv($3$, pad-1)-$24$ \- BN | 2x Conv($3$, pad-1)-$24$ \- BN
MaxPooling(2) | MaxPooling(2)
2x Conv($3$, pad-1)-$48$ \- BN | 2x Conv($3$, pad-1)-$48$ \- BN
MaxPooling(2) | MaxPooling(2)
2x Conv($3$, pad-1)-$96$ \- BN | 2x Conv($3$, pad-1)-$96$ \- BN
MaxPooling(2) | MaxPooling(2)
2x Conv($3$, pad-1)-$96$ \- BN | 2x Conv($3$, pad-1)-$96$ \- BN
MaxPooling(2) | MaxPooling(2)
Conv($1$, pad-0)-$32$ \- Linear - BN | Conv($1$, pad-0)-$32$ \- Linear - BN
Dense($32$) + Linear | Dense($32$) + Linear
Embedding Layer + Ranking Loss

Table 1: Overview of the network architecture. The upper part describes the
attention network, the lower part the embedding part. Conv($3$, pad-1)-$24$:
3$\times$3 convolution, 24 feature maps and zero-padding of 1. We use ELU
activation functions on all layers if not stated otherwise [10].

This attention-less model trains and operates on fixed-size input windows from
both modalities. In other words, the musical content provided to the CNN may
contain significantly less or more information—especially note onsets—than
excerpts it has been trained on. One may of course compensate this with data
augmentation, but a more general solution is to simply let the model decide
how much information is needed from the audio.

For this purpose, we explore a soft-attention mechanism. First, we
substantially increase the audio field of view (number of spectrogram frames),
up to a factor of four. Next, we add to the model the attention pathway $h$,
which should learn to restrict the audio input again by focusing only at those
parts that appear relevant for an efficient search query. As detailed in Table
1, this attention mechanism is implemented as a separate CNN. The output of
this CNN is a probability density function $a$ which has the same number of
frames as the audio spectrogram. Before feeding the spectrogram to the audio
embedding network $g$, we multiply each frame with its attention weight. This
enables the model to cancel out irrelevant parts of the query and focus on the
important information. In the following experiments, we show that adding this
attention network in combination with the longer temporal context,
substantially improves the results in the considered audio–sheet retrieval
task.

##  3 Experiments

This section reports on the conducted retrieval experiments. We start by
describing the data used for training and testing the models, and the data
augmentation steps applied during training. Afterwards, we present the results
for the two main experiments, both dealing with audio–sheet retrieval: given
an audio excerpt, retrieve the corresponding snippet of sheet music of the
respective piece. Finally, we look at the attention-layer’s behavior for five
examples and discuss benefits and limitations of the approach.

###  3.1 Data Description and Training

We use a dataset of synthesized classical piano music, called _MSMD_ [11]. In
version 1.1, MSMD comprises 467 pieces by 53 composers, including Bach,
Mozart, Beethoven and Chopin, totalling in over a thousand pages of sheet
music and 15+ hours of audio, with fine-grained cross-modal alignment between
note onsets and noteheads. The main changes from version 1.0 (as used in [11])
to version 1.1 are that we cleaned the test set from broken pieces and set all
note velocities to a value of $64$. The scores and audio are both synthesized
using the score engraving software LilyPond111http://www.lilypond.org and
FluidSynth.222http://www.fluidsynth.org

During training, we augment the sheet music by resizing and shifting the
image. For augmenting the audio, we vary the tempo between 95 and 110 % and
sample from a pool of three different piano soundfonts. For details of these
augmentation steps, we kindly refer the reader to the explanation given in
[11]. After training convergence, we refine the used CCA embedding layer on
the whole training set. The reason for this is that during training, the
covariance estimates used in the CCA projection are only based on the number
of samples contained in the mini-batch.

For testing, we select 10,000 audio–sheet music pairs from an unseen testset,
where the synthesized audio is rendered with a separate, hold-out piano
soundfont. The rendering procedure for the sheet music remains the same. The
above described augmentation steps are disabled during testing. In terms of
performance, the refinement step of the CCA layer yields an increase in
performance of around 0.07 up to 0.15 in terms of mean reciprocal rank (MRR)
on the test set (see Table 2 for details). The used dataset
,333https://github.com/CPJKU/msmd/tree/v1.1 as well as the implementation
along with trained models
,444https://github.com/CPJKU/audio_sheet_retrieval/tree/ismir-2019 are
publicly available.

###  3.2 Experiment 1: Attention

Model | R@1 | R@5 | R@25 | MRR | MR
---|---|---|---|---|---
BL1-SC [11] | 13.67 | 34.62 | 57.44 | 0.24 | 15
BL2-SC | 34.25 | 54.68 | 70.62 | 0.44 | 4
BL2-MC | 36.59 | 60.52 | 77.21 | 0.48 | 3
BL2-LC | 27.66 | 51.79 | 70.34 | 0.39 | 5
BL2-SC + AT | 38.41 | 59.95 | 74.36 | 0.48 | 3
BL2-MC + AT | 46.54 | 68.45 | 81.10 | 0.57 | 2
BL2-LC + AT | 53.16 | 74.42 | 85.35 | 0.63 | 1

(a) Un-refined CCA layer.

Model | R@1 | R@5 | R@25 | MRR | MR
---|---|---|---|---|---
BL1-SC [11] | 19.12 | 44.16 | 66.63 | 0.31 | 8
BL2-SC | 48.91 | 67.22 | 78.27 | 0.57 | 2
BL2-MC | 47.08 | 68.19 | 80.82 | 0.57 | 2
BL2-LC | 43.46 | 68.38 | 82.84 | 0.55 | 2
BL2-SC + AT | 55.43 | 72.64 | 81.05 | 0.63 | 1
BL2-MC + AT | 58.14 | 76.50 | 84.60 | 0.67 | 1
BL2-LC + AT | 66.71 | 84.43 | 91.19 | 0.75 | 1

(b) Refined CCA layer.

Table 2: Overview of the experimental results. (a) Lists the results without
and (b) with a refinement of the CCA layer. All experiments used 10,000
candidates and were conducted on MSMD-v1.1 (R@k = Recall@k, MRR = Mean
Reciprocal Rank, MR = Median Rank).

In the first experiment, we investigate the influence of the additional
attention network. We systematically increase the temporal context of the
audio representation from a short context (SC, 42 frames $=$ 2.1 s), over a
medium (MC, 84 frames $=$ 4.2 s), to a long context (LC, 168 frames $=$ 8.4
s). The results are summarized in Table 2. As evaluation metrics we use
different Recalls (R@1, R@5, R@25 $\in[0,100]$, higher is better), the mean
reciprocal rank (MRR $\in[0,1]$, higher is better), as well as the median rank
(MR $\in[1,10000]$, lower is better). In the following discussion, we focus on
the results of the refined CCA layer, as given in Table 2(b).

As a baseline (BL1-SC), we use the network architecture as described in [11],
which uses the short context (SC). For this model, approximately 20% of the
queries are on rank 1 (R@1 = $19.12$) and in almost 70% of the audio queries,
the relevant sheet image is within the first 25 ranks (R@25 = $66.63$). As a
second baseline (BL2-SC), we slightly adapt the original architecture by
exchanging the global average pooling layer (before the embedding layer) by a
dense layer for each modality (see the non-attention part of Table 1 for
details). With this adaptation, the results improve significantly to R@1 =
$48.91$, R@25 = $78.27$, and a median rank MR = 2, instead of MR = 8 for BL1.

By increasing the temporal context on BL2 to medium sized context (BL2-MC),
mean reciprocal rank (MRR = $0.57$) and median rank (MR = $2$) stay unchanged.
When increasing the temporal context to the long context (BL2-LC), the model
degrades in performance, e. g., R@1 drops from $48.91$% for SC to $43.46$% for
LC and the MRR from $0.57$ to $0.55$. Adding the attention network to the
audio input (BL2-SC + AT) improves the results by 7, 5, and 3% for the
recalls, as well as 0.05 for the MRR compared to BL2-SC. The more context is
given to the network, the better the performance metrics get, e. g., R@1
improves from $58.14$ (BL2-MC + AT) to $66.71$ (BL2-LC + AT). The MRR,
improves from $0.63$ (BL2-SC + AT), over $0.67$ (BL2-MC + AT), up to $0.75$
(BL2-LC + AT).

We derive two main observations from these results. First, optimizing the
network architecture is important; dropping the global average pooling in
favour of a fully-connected dense layer lifted the results to another level.
The reason could be that the fully-connected layer better retains the
structure of the input spectrogram than the average pooling and in addition
can be more selective on relevant input parts, e. g., by setting weights to
zero. Second, the attention network enables the network to deal with larger
temporal context sizes. From a signal processing perspective, one would expect
that more context (and thus longer queries) would always help since it
increases the specificity of the query. However, since we squash this
information into a 32-dimensional embedding vector, it seems that too much
information (e. g., too many onsets in the audio), actually harms the
retrieval quality when not using attention.

Model | $\rho=0.5$ | $\rho=0.66$ | $\rho=1$ | $\rho=1.33$ | $\rho=2$
---|---|---|---|---|---
BL1-SC [11] | 0.20 | 0.27 | 0.31 | 0.30 | 0.22
BL2-SC | 0.44 | 0.52 | 0.57 | 0.56 | 0.46
BL2-MC | 0.46 | 0.53 | 0.57 | 0.55 | 0.43
BL2-LC | 0.44 | 0.50 | 0.55 | 0.51 | 0.35
BL2-SC + AT | 0.55 | 0.63 | 0.63 | 0.64 | 0.56
BL2-MC + AT | 0.54 | 0.61 | 0.67 | 0.67 | 0.62
BL2-LC + AT | 0.64 | 0.69 | 0.75 | 0.73 | 0.64

Table 3: Mean Reciprocal Rank (MRR) for different models and different tempo
ratios $\rho\in\\{0.5,0.66,1,1.33,2\\}$. For example, $\rho=0.5$ stands for
half of the original tempo and $\rho=2$ for doubling the tempo. The listed
models correspond to the refined CCA models as listed in Table 2(b) with the
same test set size of 10,000 candidates. Figure 2: Example of tempo variations
within a piece by Johann André - Sonatine (Op. 34, I.). The three boxes below
the sheet music show the attention output, and the corresponding audio
spectrogram for the respective excerpt.

###  3.3 Experiment 2: Tempo Robustness

Figure 3: Illustration of the sheet music, input attention (normalized), and
spectrogram for five examples from the MSMD test set. (a) L. v. Beethoven -
Piano Sonata (Op. 79, 1st Mvt.), (b) J. S. Bach - Goldberg Variations:
Variatio 12 (BWV 988), (c) J. S. Bach - French Suite VI: Menuet (BWV 817), (d)
R. Schumann - Album for the Youth: Untitled (Op. 68, Nr. 26), and (e) M.
Mussorgsky - Pictures at an Exhibition VIII: Catacombae.

In a second experiment, we explicitly test the system’s robustness to global
tempo changes. For this purpose, we re-rendered the MSMD test dataset with
various tempo ratios $\rho\in\\{0.5,0.66,1,1.33,2\\}$, where $\rho=0.5$ stands
for halving and $\rho=2$ for doubling the original tempo. The results are
given in Table 3. For the sake of brevity, we only show the mean reciprocal
rank (MRR) for the refined CCA models.

In general, we observe a similar trend as in the first experiment. While the
original baseline approach (BL1-SC) performs rather poorly (0.20 to 0.31),
exchanging the global average pooling layer (BL2-SC) helps to improve the
performance (0.44 to 0.57). The best attention model (BL2-LC + AT) yields
values ranging from 0.64 to 0.75. In this experiment, we would have expected
that the improvement holds true for all testsets rendered at different global
tempi. However, the numbers tell a slightly different story. To understand
this, we had to go back to the generation of the MSMD dataset. Recall that the
dataset is generated from LilyPond files obtained from the Mutopia
Project.555https://www.mutopiaproject.org These files contain tempo
specifications which are retained in the synthesis pipeline. The specified
tempi vary in a range between 30 and 182 bpm (mean tempo = 106 bpm, std. dev.
= 25 bpm). This distribution of tempi implies that all experiments we perform
on the original MSMD testset ($\rho=1$) already test for a variety of tempi.
Synthesizing this dataset with different tempo factors—as done in our second
experiment—shifts and stretches this distribution. In the edge cases
($\rho=0.5$ and $\rho=2$), this leads to unrealistic tempi, e. g., 15 bpm or
364 bpm, producing input audio windows with absurdly low or high onset
density.

In summary, all of the tested models have in common that they work best for
the original testset tempo ($\rho=1$) and have similar relative performance
drops for the tempo variations. However, the attention model is able to keep
the retrieval results at a much higher level than all the other models. In the
following section, we take a closer look at some examples from the MSMD test
set to get a better intuition of the soft-attention mechanism and how it
reacts to tempo deviations.

###  3.4 Examples

In the experiments above, we have seen that the attention models improve the
retrieval results by a considerable margin. Another positive aspect of the
soft-attention mechanism is that its behavior is directly interpretable—a rare
case in the deep learning landscape. For all the presented examples, we
provide further videos with audio on an accompanying website, along with
detailed instructions for
reproduction.666http://www.cp.jku.at/resources/2019_ASR-TempoInv_ISMIR

Figure 2 shows the sheet music for Johann André’s Sonatine, Op.34. The piece
starts with a calm part with mainly legato quarter notes at 103 bpm. The first
(blue) box shows the corresponding attention output and the audio spectrogram
for measure five. The attention output is relatively flat with a small peak in
the middle of the audio excerpt. In bar 17, the note density changes, with
eighth notes in the left hand entering the scene. As shown by the second (red)
box, the attention layer reacts to this by building up a more distinct focus
around the center, placing less weight on the outer parts of the spectrogram.
In the third part beginning with measure 24, the “tempo” (more precisely: the
perceived “speed”, in terms of events per time unit) essentially doubles,
which is reflected in a high peak and narrow distribution in the attention
output (green right box). From these three examples, it also becomes clear
that tempo and note density are essentially two sides of the same coin for the
attention network.

In Figure 3, we show examples from the testset with different global
performance tempi, along with the sheet music excerpt, the attention weights,
and the spectrogram. The fastest piece, with around 250 bpm, is shown in
Figure 3a. The corresponding attention output is very focused on the middle
part of the audio spectrogram, trying to concentrate the attention to the
notes that actually appear in the sheet music snippet. The second example (b)
is rather slow with 95 bpm. However, through the use of 16th and 32th notes,
the main melody gets a double-time feel, thus the actual tempo is perceived at
around 190 bpm. In Figure 3c, the tempo is at 115 bpm. The attention output
starts to widen up, allowing more temporal context to reach the actual
embedding network. This trend goes in Figure 3d when the tempo 78 bpm. Figure
3e shows an extreme example where the piece mainly consists of chords with
long, dotted half notes. Here, the attention has to use the complete temporal
context of the audio spectrogram to match the score information with the
audio.

The examples demonstrate that depending on the spectrogram content, the model
indeed attends to whatever it believes is a representative counterpart of the
target sheet music snippet. Since the fixed-size sheet snippets contain
roughly similar numbers of notes, as the density of note heads on the printed
page tends to be independent of the tempo of the piece, attention is sharply
peaked when the density of onsets in the audio is high, and conversely is
distributed more evenly when there are fewer notes in the audio excerpt.

Figure 4: Entropy of the input attention vs. the number of onsets in the
respective audio frame.

##  4 Summary

In this paper, we have described a soft-attention mechanism that helps to
overcome the fixed window sizes uses in Convolutional Neural Networks. In our
end-to-end audio–sheet music retrieval application, the results improved
substantially compared to the state of the art. By looking at a number of
examples from the retrieval results, the soft-attention mechanism showed an
intuitive and interpretable behavior.

This appealing and intuitive behavior is summarized in Figure 4, which shows
the entropy of the attention distribution in relation to the number of onsets
contained in the audio excerpt, for all 10,000 test samples. The entropy is a
measure of flatness of the attention output: a flat function gives high
entropy, a very narrow function low entropy. The downward trend in the figure
confirms our observations from above: the more onsets in the respective audio
spectrogram, the narrower the attention distribution.

Given the improved retrieval performance and the intuitive behavior of the
attention model, we think this is a promising line of research for reducing
the sensitivity of cross-modal music retrieval models to the audio input
window size. To this end, our experiments were conducted on synthesized piano
recordings. However, results in [11] indicate that the embedding models
trained on this data generalize to real scores and performances. A possible
next step would be to investigate whether the attention mechanism reacts to
local tempo changes as occuring frequently in real performances (e. g.,
ritardandi and accelerandi). Furthermore, it would be interesting to leave the
piano music domain and extend the model to cope with differences in timbre,
(e. g., orchestral music) as done in [4] for the challenging
Barlow–Morgenstern scenario [5].

Acknowledgements
This project has received funding from the European Research Council (ERC)
under the European Union’s Horizon 2020 research and innovation programme
(grant agreement number 670035, project CON ESPRESSIONE, and the Marie
Skłodowsa-Curie grant agreement number 765068, MIP-Frontiers).

## References

  * [1] Relja Arandjelovic and Andrew Zisserman.  Look, listen and learn.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 609–617, Venice, Italy, 2017.
  * [2] Andreas Arzt, Sebastian Böck, and Gerhard Widmer.  Fast identification of piece and score position via symbolic fingerprinting.  In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), pages 433–438, Porto, Portugal, 2012\.
  * [3] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  In International Conference on Learning Representations (ICLR), pages 1–15, San Diego, CA, USA, 2015.
  * [4] Stefan Balke, Vlora Arifi-Müller, Lukas Lamprecht, and Meinard Müller.  Retrieving audio recordings using musical themes.  In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pages 281–285, Shanghai, China, 2016.
  * [5] Harold Barlow and Sam Morgenstern.  A Dictionary of Musical Themes.  Crown Publishers, Inc., revised edition edition, 1975.
  * [6] Sebastian Böck and Markus Schedl.  Polyphonic piano note transcription with recurrent neural networks.  In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 121–124, Kyoto, Japan, March 2012.
  * [7] Donald Byrd and Jakob G. Simonsen.  Towards a standard testbed for optical music recognition: Definitions, metrics, and page images.  Journal of New Music Research, 44(3):169–195, 2015.
  * [8] William Chan, Navdeep Jaitly, Quoc V. Le, and Oriol Vinyals.  Listen, attend and spell: A neural network for large vocabulary conversational speech recognition.  In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4960–4964, Shanghai, China, 2016.
  * [9] Tian Cheng, Matthias Mauch, Emmanouil Benetos, and Simon Dixon.  An attack/decay model for piano transcription.  In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), pages 584–590, New York City, United States, 2016.
  * [10] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter.  Fast and accurate deep network learning by exponential linear units (ELUs).  In Proceedings of the International Conference on Learning Representations (ICLR) (arXiv:1511.07289), 2016.
  * [11] Matthias Dorfer, Jan Hajič jr., Andreas Arzt, Harald Frostel, and Gerhard Widmer.  Learning audio–sheet music correspondences for cross-modal retrieval and piece identification.  Transactions of the International Society for Music Information Retrieval, 1(1), 2018.
  * [12] Matthias Dorfer, Jan Hajič jr., and Gerhard Widmer.  Attention as a perspective for learning tempo-invariant audio queries.  In Proceedings of the ICML Joint Workshop on Machine Learning for Music, Stockholm, Sweden, 2018.
  * [13] Matthias Dorfer, Jan Schlüter, Andreu Vall, Filip Korzeniowski, and Gerhard Widmer.  End-to-end cross-modality retrieval with CCA projections and pairwise ranking loss.  International Journal of Multimedia Information Retrieval, 7(2):117–128, Jun 2018.
  * [14] Christian Fremerey, Michael Clausen, Sebastian Ewert, and Meinard Müller.  Sheet music-audio identification.  In Proceedings of the International Conference on Music Information Retrieval (ISMIR), pages 645–650, Kobe, Japan, October 2009.
  * [15] Jan Hajič jr., Jiri Novotný, Pavel Pecina, and Jaroslav Pokorný.  Further steps towards a standard testbed for Optical Music Recognition.  In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), pages 157–163, New York City, United States, 2016.
  * [16] Curtis Hawthorne, Erich Elsen, Jialin Song, Adam Roberts, Ian Simon, Colin Raffel, Jesse Engel, Sageev Oore, and Douglas Eck.  Onsets and frames: Dual-objective piano transcription.  In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), pages 50–57, Paris, France, 2018\.
  * [17] Özgür Izmirli and Gyanendra Sharma.  Bridging printed music and audio through alignment using a mid-level score representation.  In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), pages 61–66, Porto, Portugal, 2012\.
  * [18] Rainer Kelz, Sebastian Böck, and Gerhard Widmer.  Deep polyphonic ADSR piano note transcription.  In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, United Kingdom, May 2019.
  * [19] Rainer Kelz, Matthias Dorfer, Filip Korzeniowski, Sebastian Böck, Andreas Arzt, and Gerhard Widmer.  On the potential of simple framewise approaches to piano transcription.  In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), pages 475–481, New York City, USA, 2016.
  * [20] Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel.  Unifying visual-semantic embeddings with multimodal neural language models.  arXiv preprint (arXiv:1411.2539), 2014.
  * [21] Frank Kurth, Meinard Müller, Christian Fremerey, Yoon ha Chang, and Michael Clausen.  Automated synchronization of scanned sheet music with audio recordings.  In Proceedings of the International Conference on Music Information Retrieval (ISMIR), pages 261–266, Vienna, Austria, September 2007\.
  * [22] Meinard Müller, Andreas Arzt, Stefan Balke, Matthias Dorfer, and Gerhard Widmer.  Cross-modal music retrieval and applications: An overview of key methodologies.  IEEE Signal Processing Magazine, 36(1):52–62, 2019.
  * [23] Chris Olah and Shan Carter.  Attention and augmented recurrent neural networks.  Distill, 2016.
  * [24] Christopher Raphael and Jingya Wang.  New approaches to optical music recognition.  In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), pages 305–310, Miami, Florida, USA, 2011.
  * [25] Ana Rebelo, Ichiro Fujinaga, Filipe Paszkiewicz, Andre R. S. Marcal, Carlos Guedes, and Jaime S. Cardoso.  Optical music recognition: state-of-the-art and open issues.  International Journal of Multimedia Information Retrieval, 1(3):173–190, 2012.
  * [26] Siddharth Sigtia, Emmanouil Benetos, and Simon Dixon.  An end-to-end neural network for polyphonic piano music transcription.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(5):927–939, 2016.
  * [27] Carl Southall, Ryan Stables, and Jason Hockman.  Automatic drum transcription for polyphonic recordings using soft attention mechanisms and convolutional neural networks.  In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), pages 606–612, Suzhou, China, 2017\.
  * [28] Cuihong Wen, Ana Rebelo, Jing Zhang, and Jaime S. Cardoso.  A new Optical Music Recognition system based on combined neural network.  Pattern Recognition Letters, 58:1–7, 2015.