# ScaleNet: A Shallow Architecture for Scale Estimation

Axel Barroso-Laguna Yurun Tian Krystian Mikolajczyk
Imperial College London
{axel.barroso17, yurun.tian<EMAIL_ADDRESS>
###### Abstract

In this paper, we address the problem of estimating scale factors between
images. We formulate the scale estimation problem as a prediction of a
probability distribution over scale factors. We design a new architecture,
ScaleNet, that exploits dilated convolutions as well as self- and cross-
correlation layers to predict the scale between images. We demonstrate that
rectifying images with estimated scales leads to significant performance
improvements for various tasks and methods. Specifically, we show how ScaleNet
can be combined with sparse local features and dense correspondence networks
to improve camera pose estimation, 3D reconstruction, or dense geometric
matching in different benchmarks and datasets. We provide an extensive
evaluation on several tasks, and analyze the computational overhead of
ScaleNet. The code, evaluation protocols, and trained models are publicly
available at https://github.com/axelBarroso/ScaleNet.

##  1 Introduction

Establishing correspondences is the very first step in many different 3D
pipelines. Advancing on this task will have a direct impact on the performance
of downstream applications such as camera pose estimation [40], autonomous
driving[6], or 3D reconstructions[42]. However, methods that search for
correspondences between images face significant challenges, and although some
solutions are more mature than others, the task still is far from being
solved.
As the field advances, even though the intermediate tasks in the
correspondence search remain the same, their methods are being revisited and
redesigned, e.g., keypoint detectors/descriptors [11, 12, 38], dense geometric
matchers [26, 54], or geometric verification techniques [32, 47, 41]. These
new approaches have shown that the downstream tasks can be pushed to new
performance levels through robust correspondences. The key objective of these
new methods is to handle more and more extreme cases where previous pipelines
failed, and although some methods are arguably application-specific [61],
their robustness to extreme conditions is the main reason for success.

Figure 1:  We propose a scale-aware system that helps establish
correspondences under strong scale changes, allowing to correctly match pairs
where previous pipelines were not successful. We introduce ScaleNet, a network
that estimates the scale distribution between two images so that regions of
interest can be corrected and exhibit the same scale factor (top). We display
(bottom) an indoor and outdoor example where matches from R2D2 with multi-
scale pyramid (left) increase if using ScaleNet rectification (right).

Inspired by the previous methods targeting visual robustness [59, 30, 31], we
address the problem of handling the scale change between images, which is a
long-standing challenge in computer vision [27, 24]. Scale robustness and
estimation have been the focus of much research in the area of handcrafted
feature extraction [55, 24, 61] as a reliable solution that can significantly
boost the performance of existing methods. Moreover, the scale change is
arguably the most challenging, and the most important parameter to estimate
compared to rotation, translation, or even local affine deformations [55].
There are several strategies to deal with scale changes, with the multi-scale
pyramid being one of the most popular solutions [24, 55, 12, 38, 4, 3, 53].
Although the multi-scale pyramid mitigates the problem of different scales, it
increases complexity and ambiguity as the matcher needs to establish
correspondences among multiple scale levels. Figure 1 (bottom) shows an
example of extreme pairs for which R2D2[38] with multi-scale pyramid can only
get a high number of correct matches once we rectify the scales. Besides the
added complexity, a multi-scale pyramid is not always a straightforward
solution to incorporate in some methods, such as dense correspondence networks
[54, 19, 46]. In contrast to multi-scale pyramids, some works aim at being
invariant to different scales through their learning process [35, 11],
however, as a side effect, they become progressively less discriminative [56].
Another possible direction, and popular strategy, is to estimate the local or
global transformations and rectify the images prior to establishing the
correspondences[59, 31, 51, 37].

The scale factor characterizes the relationship between pairs of images and,
in general, an accurate estimate can only be achieved when considering both
images at the same time. Using pairs of images as input may increase the
complexity beyond acceptable in some applications such as large-scale
retrieval and localization unless used in their final verification stage.
Nonetheless, solving the scale before the main analysis improves the
discriminative power and allows less robust but more efficient methods to be
used in challenging scenarios [35]. Hence, we propose a new approach that
estimates and corrects the scale factor between a given pair of images before
the correspondence search, which is illustrated at the top of figure 1. Our
scale predictor network, termed ScaleNet, is inspired by dense geometric
methods [26, 54, 52] and conditioned local features extractors [56, 15, 35].
ScaleNet extracts features from two low-resolution images and exploits CNN
correlation layers to predict the scale factor. Due to the non-linear nature
of scale changes, we formulate the scale regression problem as the estimation
of a probability distribution in logarithmic space. We show how ScaleNet can
be combined with different methods and demonstrate the improvements on
different tasks and datasets.

Our contributions include: 1) a scale-aware matching system based on a novel
scale predictor architecture, 2) a strategy to measure and label the scale
factor between two images, and 3) a learning scheme that tackles the non-
linear nature of scale changes.

##  2 Related work

Recent works have allowed significant progress in establishing good
correspondences between images. Although many works have focused on solving
entire tasks in an end-to-end manner [40, 5], there are lots of efforts
focused on identifying limitations and improving the robustness of individual
steps in modern pipelines [29, 3, 41].

Image rectification consists of predicting or applying a set of
transformations to the images so that the search for correspondences is done
in an optimum setting. Pioneering work on this area is ASIFT [59], which
applies multiple affine transformations to find less challenging image pairs
for matching with SIFT [24]. MODS [30] investigated this line of work by
introducing an iterative scheme to generate intermediate synthetic views
between images. MODS also proposed an adaptive system to avoid applying
synthetic transformations to easy-to-match pairs, being faster and more
versatile than previous ASIFT. Closer to our work is [61], which computes the
scale factor between a pair of images by detecting and matching exhaustively
SIFT features on multiple scale levels. Although it shows that they can deal
with strong scale changes, their method is tied to the need of visiting all
possible scaled images before the actual local feature matching stage. One
negative aspect of these methods is that they still require a blind
exploration of synthetic views to find the optimal image transformations. A
few works have tried to overcome the previous limitation and proposed to learn
such parameterizations directly from the images. One of the first attempts is
[58], where authors introduced a neural network to assign a canonical
orientation to every input image patch before the descriptor architecture. In
AffNet [31], authors follow this trend and learn a full affine shape estimator
to geometrically align input patches before the descriptor. Moreover, [37]
proposed a scene-specific overlapping estimator and showed that scale
rectification based on image overlapping can improve feature matching. Unlike
previously learned methods, our labeling strategy is based on keypoint
distance ratios, resulting in a scene-agnostic scale predictor that uses pairs
of images as input. Even though our ScaleNet tackles only the scale factor out
of several possible transformation parameters, we show in section 5 that the
scale factor is crucial for boosting the performance of current methods.

Figure 2: ScaleNet uses a multi-scale (M-S) feature extractor and a
combination of self- and cross-correlation layers to calculate the
relationship within each image and between them. The M-S extractor is composed
of a common CNN and an ASPP block [7, 8]. The ASPP module consists of three
3x3 convolutional layers with different dilation rates. Conv blocks perform
1x1 convolutions to learn the cross-channel interactions that go into the
self- and cross-correlation layers. Correlation volumes’ dimensionality is
reduced through a CNN, and its results are concatenated into a flat feature
vector. Finally, consecutive fully connected layers calculate the scale
distribution.

Visual robustness has been the focus of numerous works in the field of
correspondence search [25, 14, 4]. The rotation has been addressed by
correcting input patches [24, 14, 31] before extracting local descriptors [29,
50, 49], or by designing robust architectures [25, 4, 23]. However, in the
context of scale changes, the standard strategy is the well-known multi-scale
(M-S) pyramid approach, which applies methods at different re-scaled versions
of the image [38, 12, 48]. Even though M-S pyramids offer a versatile solution
for many applications, it does not provide a suitable approach for extreme
scale changes (cf. section 5), and thus, both, single and multi-scale feature
extractors, benefit from correcting the scale factor before extracting
features. Moreover, recent works show that there is growing interest in pair-
wise methods, i.e., that use two input images at the same time to establish
the local or dense correspondences [46, 19, 15, 26, 54, 56, 36], but, in that
scenario, there is no clear or effective strategy for dealing with scale
changes. Hence, given that such methods already take two images as inputs,
ScaleNet rectification offers a more natural and intuitive process towards
visual robustness than M-S pyramids.

##  3 Method

ScaleNet embraces several key concepts to deliver good performance in
practical settings. The first key aspect is that it is effective for low-
resolution images, which makes it more efficient. Another important idea is
the formulation of the scale estimation as a distribution prediction in
logarithmic space rather than a regression problem [31, 57, 18]. This allows
using a simple and shallow, yet effective architecture. Figure 2 presents our
ScaleNet architecture, and the following sections detail each of the aspects
of ScaleNet and its learning scheme.

###  3.1 ScaleNet architecture

Consider $A$ and $B$ as input images to ScaleNet. Low-resolution images $A$
and $B$ are processed by a multi-scale feature extractor block, which is
composed of a generic network, e.g., VGG, or ResNet, followed by the atrous
spatial pyramid pooling (ASPP) [7, 8]. The ASPP block achieves multi-scale
robustness by applying to the feature map $3\times 3$ dilated convolutions,
each with a different dilation rate, e.g., 2, 3, and 4. Thus, the ASPP block
allows the network to compute and fuse features from different receptive
fields. M-S features are concatenated and fed into a final $1\times 1$
convolution, which combines the features from local and global areas at a
minimum cost. We then apply self- and cross-correlation layers to multi-scale
features $f_{A}$ and $f_{B}$, and obtain the correlation maps, $c_{A}$,
$c_{B}$, and $c_{A-B}$, which contain the self- and cross-pairwise
similarities. As in [39], ReLU and L2-normalization are applied to the
correlation maps before the feature reduction blocks, which are composed of
four Conv-Batch-ReLU layers each. Finally, $c^{\prime}_{A}$, $c^{\prime}_{B}$,
and $c^{\prime}_{A-B}$ maps are flattened, concatenated, and fed into a set of
fully connected layers to predict the scale distribution $P$, with a final
softmax activation layer.

###  3.2 Predicting scale distributions

ScaleNet outputs a scale distribution rather than a regressed single scale
factor, which helps the network to converge to a reliable model. In contrast,
when tackling the problem as a regression task, the same network cannot
predict an accurate scale (cf. appendix C.1). We attribute this to the fact
that the network can learn and interpret the relationships between the
quantized scale ranges and solve an easier classification task, which requires
it to assign the weights to the predefined scale factors instead of predicting
its actual value.
We formulate the scale estimation as a problem of predicting the probability
distribution in a scale-space quantized into $L$ bins. Given images $A$ and
$B$, our objective function measures the distance between our computed scale
distribution, $P_{A\rightarrow B}$, and the ground-truth distribution
$P_{A\rightarrow B}^{gt}$:

$\begin{gathered}Loss(A,B)=KL(P_{A\rightarrow B},P_{A\rightarrow
B}^{gt}),\end{gathered}$ (1)

where $KL(\cdot,\cdot)$ is the Kullback-Leibler divergence loss. To obtain the
scale factor from the probability distribution $P_{A\rightarrow B}$, we
combine all scale levels using a soft-scale computation. It enables the
network to output scale factors that interpolate between the quantized scale
values, thus covering all possible scales between images $A$ and $B$. Soft-
assignment gives the architecture further flexibility and robustness when
inferring scales as shown in section 5.4.
The scale factor is a relative ratio operator, hence, it is non-linear. To
avoid a bias by high scale values when computing the soft-scale, we transform
the quantized scale classes, $s_{i}$, to logarithmic space. The logarithmic
transformation allows to calculate the soft-scale, $\bar{S}_{A\rightarrow B}$,
as a linear combination of the logarithmic scales, $\bar{s}_{i}$, weighted by
the predicted scale probabilities from softmax output $p_{i}$. The global
scale factor $\bar{S}_{A\rightarrow B}$ in log-scale is given as:

$\begin{gathered}\bar{S}_{A\rightarrow B}=\sum_{i=0}^{L-1}p_{i}\cdot
ln(s_{i}),\end{gathered}$ (2)

where $s_{i}=\sigma^{t}$ corresponds to the quantized scale for bin $i$,
$\sigma$ is our predefined base scale factor, integer
$t\in[-(L-1)/2,\ldots,0,\ldots,(L-1)/2]$, and $L$ is the total number of scale
bins. Moreover, we improve the robustness of our scale estimator by a simple
yet effective consistency check trick, where we compute the scale factor,
$\bar{S}_{A\rightarrow B}$, and its inverse, $\bar{S}_{B\rightarrow A}$, and
combine them as:

$\begin{gathered}\hat{S}_{A\rightarrow B}=\frac{\bar{S}_{A\rightarrow
B}-\bar{S}_{B\rightarrow A}}{2}\quad\textrm{and}\quad S_{A\rightarrow
B}=e^{\hat{S}_{A\rightarrow B}},\end{gathered}$ (3)

with $S_{A\rightarrow B}$ as the final scale factor between the images.

###  3.3 Dataset generation

Figure 3: In the dataset generation, we randomly pick pairs of corresponding
points in both views and compute their distances in the image planes. The
final scale ground-truth is the average of the distance ratios of all picked
keypoints.

ScaleNet is trained with synthetically generated image pairs as well as images
from real scenes.

Synthetic pairs. We define a set of planar affine transformations to map one
image into another. Such image pairs are easy to generate on-demand for any
ground-truth scale $S_{gt}$, however, they do not include the real noise from
different imaging conditions.

Real pairs present more challenging conditions than synthetic pairs, i.e.,
non-planar viewpoint changes, weather/illumination conditions, or occlusions,
among others. In addition, in contrast to the previous synthetic global
transformations, the scale between two real images may spatially vary, and
areas of different depth or strong perspective changes may include various
scale factors. Thus, we introduce a new approach for obtaining training data
by estimating the scale factors between real images. We use 3D reconstruction
datasets, where 3D point clouds and their corresponding projected 2D positions
on the images are available. First of all, given the 3D model, we find pairs
of images with an overlap higher than 10% computed as in [12, 33]. For each
pair of images $A$ and $B$, we identify the 3D points of the model that are
visible in both images. Using the registered 3D points from the model as
opposed to sampling random positions ensures that the regions used for
computing the scale are discriminative. Thus, given the covisible 3D points,
we query its projected 2D positions, $k_{A}$, and $k_{B}$, on image $A$ and
$B$. We randomly sample pairs of points, $k_{A}$ and $k_{B}$, and compute
their distances as shown in figure 3.

The scale factor between two images with keypoints $i$ and $j$ is calculated
as the ratio of their distances in image $A$ and $B$:

$\begin{gathered}S_{i-j}=\frac{\|k_{Bi}-k_{Bj}\|}{\|k_{Ai}-k_{Aj}\|}\quad\textrm{and}\quad
i,j\in[1,...,K],\end{gathered}$ (4)

with $K$ as the total number of covisible 3D points between images $A$ and
$B$. As different regions may exhibit different scale factors, we compute the
global scale factor as the average of ratios in logarithmic scale after
sampling $R$ different pairs of points:

$\begin{gathered}S_{A\rightarrow B}^{gt}=e^{\bar{S}_{A\rightarrow
B}}\quad\textrm{where}\quad\bar{S}_{A\rightarrow B}=\frac{1}{R}\sum_{i\neq
j}ln(S_{i-j}).\end{gathered}$ (5)

As detailed in section 3.2, the ScaleNet learning scheme minimizes the K-L
divergence between the predicted scales and ground-truth distributions. We,
therefore, build the ground-truth scale distribution, $P_{A\rightarrow
B}^{gt}$, such as it satisfies:

$\begin{gathered}ln(S_{A\rightarrow B}^{gt})=\bar{S}_{A\rightarrow
B}^{gt}=\sum_{i=0}^{L-1}p_{i}^{gt}*ln(s_{i}),\end{gathered}$ (6)

where $s_{i}$ are the quantized scale factors, and $P_{A\rightarrow
B}^{gt}=[p_{0}^{gt},...,p_{(L-1)}^{gt}]$ is the ground-truth distribution of
the scale estimated as a normalized histogram of point pairs.

##  4 Implementation notes

ScaleNet details.  ScaleNet predicts a scale distribution with $L=13$ bins and
$\sigma=\sqrt{2}$, giving possible scale factors in range $S\in[0.16,...,6]$.
ASPP module has 3 levels with dilation rates 2, 3, and 4. During training,
ScaleNet uses Adam Optimizer with a learning rate of $10^{-4}$ and a decay
factor of 0.1 every ten epochs. The training takes on average 40 epochs, 20
hours on a machine with an i7-7700 CPU running at 3.60GHz, and an NVIDIA
GeForce GTX 1080-Ti. ScaleNet model and training scripts are implemented in
PyTorch [34].

Dataset details.  We use Megadepth dataset [21] for generating our custom
training and testing dataset. We discard scenes that are in the PhotoTourism
test from our training set as in [41] to avoid overlap. We keep 10% of the
training scenes as our validation set. During data generation, we sample
$R=200$ random pairs of 3D points (cf. equation 5) for a robust scale
estimation between the two images. We sample pairs of images with scale
factors $S\in[0.16,...,6]$ and create a collection of 250,000 training and
25,000 validation pairs where all scale factors are well represented.
Synthetic pairs are generated on the fly from the Megadepth training images
during training. We include more details and examples of our training set in
appendix A.

##  5 Experiments

This section presents results for ScaleNet integrated with state-of-the-art
methods on several datasets and tasks. Refer to appendix for more experiments
and qualitative examples.

###  5.1 Preliminaries

| Pose estimation (AUC) | Time
---|---|---
| at 5° | at 10° | at 20° | (ms)
Baseline | 4.8 | 7.4 | 10.4 | -
VGG-16 | 5.4 | 8.3 | 11.8 | 8.3
ResNet-50 | 5.6 | 8.8 | 12.7 | 15.8
VGG w/ self-corr. | 6.2 | 9.3 | 12.5 | 11.3
VGG w/ ASPP | 7.3 | 11.1 | 15.7 | 16.6
Ours (VGG+self-corr+ASPP) | 8.4 | 12.3 | 17.6 | 19.4
Ours \+ Consistency check | 8.7 | 13.4 | 19.5 | 19.8
Table 1: Ablation study of the different ScaleNet’s design choices. Baseline
refers to SuperPoint [11] without scale correction.

Multi-scale pyramid & ScaleNet. Multi-scale (M-S) pyramids and ScaleNet aim at
making methods more robust against arbitrary scale changes. Although both
approaches tackle the same problem, each has its strengths, e.g., M-S pyramids
can compute a higher number of features by visiting multiple resized images,
and ScaleNet offers a more natural integration into tasks where two images are
given as input [15, 26, 54, 56]. Besides, we claim that ScaleNet improves not
only single-scale feature extractors but also multi-scale ones. We analyze in
figure 4(a) the robustness of methods against synthetic scale transformations
and show how the combination with ScaleNet benefits them. In this experiment,
we use 2,000 random images from the Megadepth dataset and scale them to create
the pairs. We measure the mean matching accuracy (MMA) computed as in [38]. As
expected, results indicate that the performance of single-scale methods,
Key.Net[3] and R2D2[38], drop notably even when images present small scale
perturbations ($s>1.5$). Meanwhile, M-S pyramid or ScaleNet strategies
mitigate the effect and lead to a better approach. Moreover, we observe that
the combination of M-S pyramids and ScaleNet achieves the top performance, and
proves that both strategies contribute and work well together.

Local vs global scale estimation. ScaleNet can predict a scale for each point
within the image, however, it is not straightforward to correct the scale
factor locally for networks that process the whole image, e.g., dense
correspondences networks [26, 54, 52], or dense local feature extractors [11,
12, 38]. Figure 4(b) shows the histogram of the average scale ratios between
the globally and the locally estimated scales per image. To compute the local
scale values, we restrict the random sampling to spatially neighboring
keypoints in equation 4 rather than points sampled across the whole image.
Figure 4(b) shows that in the majority of images the differences between
global and local estimations are small and within 1.0 and 1.2. A ratio $r$ of
1.0 indicates that the local and the global scales between the two images are
the same, and therefore, the global scale is valid across the whole image.
Based on results in figure 4(b), we argue that although local scales could
bring an extra benefit in scenes with strong viewpoint or perspective changes,
a single global scale will significantly contribute towards correcting images.

(a) Mean matching accuracy under synthetic scale transformations, $s$.

(b) Histogram of the scale ratios, $r$, between the globally and locally
estimated scales in image pairs used for training, with $r=s_{a}/s_{b}$, and
where $s_{a}=max(s_{local},s_{global})$, and
$s_{b}=min(s_{local},s_{global})$.

Figure 4: Analysis on the visual robustness against pure scale changes, and
comparison of local vs global scales.

We first compare the effect of different pre-trained feature extractors on
ImageNet [10] (cf. figure 2), VGG-16 [45], and ResNet-50 [16], and see that
the more complex ResNet representation contributes towards better poses with
the downside of a higher computational cost. Thus, to keep our method light
and fast, and given the similarity of the AUC scores, we use the VGG feature
extractor for following experiments. In addition to the extractors, we analyze
the effect of the self-correlation layers and the multi-scale ASPP component
and observe that both boost the performance. Self-correlations capture the
intra-image relationships and, therefore, give a better awareness of the
global content. Additionally, ASPP offers a mechanism to extract more global
features, thus, address larger scale changes. Furthermore, we show that the
consistency check offers a higher AUC at a low computation cost. Note that M-S
features and correlation layers only need to be computed once in inference
and, hence, the extra cost for consistency check is small and proportional to
running the dense layers twice.

###  5.2 Relative camera pose

Protocol. We first evaluate ScaleNet on the camera pose estimation task due to
its natural integration into the existing pipelines. Similar to the previous
ablation experiment, given a collection of image pairs, we calculate the AUC
of the camera pose error at $5^{\circ}$, $10^{\circ}$, and $20^{\circ}$ as in
[41, 46]. We use the test scenes from Megadepth and mine 4,000 images pairs
with small and strong scale changes such as $s\in[0.16,6]$. Appendix contains
more details and examples of our dataset. We study the effect of ScaleNet on
popular and publicly available local feature methods [11, 3, 29, 38, 24], and
refer to Key.Net/HardNet as Key.Net in the following tables and figures.

Results in table LABEL:tab:camera_pose_megadepth show that ScaleNet
corrections boost the performance of all methods. As discussed previously,
even though ScaleNet excels when combined with a single-scale method, e.g.,
SuperPoint, ScaleNet is also able to improve the pose estimation of multi-
scale extractors. We observe that the average improvements are 77% and 24% for
single and multi-scale methods, respectively. In addition, we report results
with SuperPoint and SuperGlue [41] and see that even this state-of-the-art
matcher benefits (+33% on average) from a scale correction prior to the
feature extraction.

###  5.3 Geometric matching

Results in table LABEL:tab:dense_matching_megadepth_v2 show the PCK scores
obtained with and without scale correction before the dense architectures,
DGC-Net and GLU-Net. Moreover, to highlight the benefit of ScaleNet for images
at different scale factors, besides reporting the results for the full dataset
(All), we create the Easy ($s>1.2$) and Hard ($s>1.8$) splits, where $s$
indicates scale distortions factor between images. Each split has 1,600, 627,
and 440 image pairs, respectively. We observe that ScaleNet integration does
not bring significant improvements in the All data, where the majority of
images have small scale variations, but it is important to note that it does
not hurt the performance either. Meanwhile, in extreme cases (Hard split),
current dense correspondence methods fail under severe scale changes, and
their integration with ScaleNet improves the results by 347% and 148% for DGC-
Net and GLU-Net.

###  5.4 3D reconstruction

Protocol. ScaleNet can be easily integrated into geometric correspondence or
relative camera pose pipelines, which are often a part of a more general 3D
reconstruction system. To evaluate ScaleNet in this scenario, we follow the
protocol proposed in the Local Feature Evaluation Benchmark [43] for building
3D reconstruction models. As ScaleNet can upsample one of the images, and that
could result in a higher number of candidate keypoints, we limit the number of
features to the top 2,048 keypoints based on the protocol proposed in [60]. We
present results for the test split from the IMC dataset [60], which includes
ten different scenes with 100 images each. IMC images pose significant
challenges, e.g., weather/illumination, perspective, scale, as well as strong
occlusions. As ScaleNet is applied to image pairs before the feature
extraction, the detectors and descriptors are recomputed every time the scale
is corrected. Hence, to reduce the computation time, we propose a discrete
variant of ScaleNet (D-ScaleNet) that makes the extraction process more
efficient. D-ScaleNet implements a hard-assignment by selecting the maximum
scale instead of the soft-scale and consistency check from equation 2 and 3.
Analogous to multi-scale pyramid approaches, we run the detectors/descriptors
at multiple resized images but then select the optimal set of pre-computed
features for matching based on D-ScaleNet estimation.

| Time (s)
---|---
| Extraction | Matching | Reconst. | Total
SuperPoint | 10.3 | 16.7 | 195.3 | 222.3
w/ ScaleNet | 980.5 | 46.3 | 208.9 | 1235.7
w/ D-ScaleNet | 141.8 | 161.8 | 205.6 | 509.2
Table 5: 3D reconstruction times on the British Museum scene (100 images) from
IMC dataset [60].

Results in table LABEL:tab:3d_results show the 3D reconstruction metrics of
state-of-the-art methods with and without ScaleNet. We notice that image
rectification by both, D-ScaleNet and ScaleNet, increases the number of
registered images and the total number of observations in the 3D models. Track
length is especially boosted by scale correction, meaning that the model was
able to match the same keypoint simultaneously in more images. This increase
of track length is particularly important since it proves that ScaleNet helps
current methods distinguish and link points that were not possible without it,
due to extreme view differences. On average, improvements added by ScaleNet
are greater than those produced by D-ScaleNet, however, D-ScaleNet still
brings a notable boost over baselines. On the opposite side, ScaleNet
increases the reprojection error (Rep. Error) of the reconstructions by 0.09
points on average. We attribute this to their longer track lengths since more
points are triangulated throughout the images, and thus, the reprojection
error increases, which has also been reported in [43]. Longer tracks will
benefit works that rely on complete and long tracks to refine the point
positions and reduce their reprojection errors [22, 13]. In table 5, we show a
comparison of the times taken to generate a 3D model when using ScaleNet and
D-ScaleNet, and display the benefits in terms of computational time that
D-ScaleNet provides.

###  5.5 Image matching

Figure 5: MMA on the full Viewpoint HPatches (left), scenes with easy (middle)
and hard (right) scale transformations. Dashed lines (- -) show ScaleNet
combined with other methods.

Protocol. We compute the Mean Matching Accuracy (MMA) [28] as the ratio of
correctly matched features within a threshold (5 pixels) and the total number
of features following the benchmark proposed in [12]. We report results for
the 59 viewpoint scenes from the HPatches dataset [1]. Similar to the 3D task,
we fix the number of features to top 1,000 as in [20] to eliminate the effect
of increased matching scores from a high number of keypoints.

Results in figure 5 (left) show that ScaleNet improves the robustness of
current methods overall, and excels for single-scale methods, e.g., Key.Net-SS
[3], or R2D2-SS [38]. However, in contrast to images used for statistics in
figure 4(b), most of the scenes in the HPatches Viewpoint contain strong
perspective changes, i.e., display different scale factors within the scene.
Hence, to highlight the effect of ScaleNet, we show results for the subset of
scenes with affine transformations, i.e., scenes with stretch and skew
transformations in addition to global scale and rotation. We select the splits
such that the Easy has $s\in[1.2,\ldots,1.8]$ and the Hard has $s>1.8$. When
the scale factor is global across the image, ScaleNet correction can deal with
planar scenes and improve the matching accuracy of single and multi-scale
methods. As expected, results show that ScaleNet scaling is more critical for
stronger scale changes in figure 5 (right). Only SIFT [24], which is specially
designed to be robust against scale changes on planar scenes, does not benefit
from ScaleNet correction in the Hard split. Moreover, to deal with scenes with
possible strong perspective scale changes, we introduce in appendix B Local-
ScaleNet, which infers local scale factors and offers a more robust and
functional alternative for scenes with perspective changes.

##  6 Discussion

Limitations. ScaleNet deals with arbitrary scale changes and, hence, it only
brings improvements if such changes are present in the images. This makes
ScaleNet useful in applications where those viewpoint changes prevent a
successful matching of images, e.g., extreme and sparse collection of images,
or ground-aerial applications. Nevertheless, even though ScaleNet does not
boost performance when there is no scale change, it does not hurt either (cf.
appendix LABEL:appendix_sec:experiments_camera). Another limitation comes when
ScaleNet needs to be applied to a large collection of images, e.g., 3D tasks.
ScaleNet works with pairs of images, hence, features can not be stored but
need to be computed every time a new image is presented, increasing the
feature extraction time as seen in table 5. Although we propose D-ScaleNet,
which mitigates the complexity time for such tasks, ScaleNet can be further
optimized for faster processing by replacing VGG with more compact models such
as MobileNet [17], or by only using ScaleNet for computing camera pose after a
restrictive retrieval search.

Societal impact. Image matching is a pivotal but small component within large
systems that facilitate technologies like AR, 3D reconstruction, navigation,
modeling, SLAM, among others. Hence, as we contribute towards more robust
matching pipelines, ScaleNet’s societal impact is tied to the applications
that rely on such technologies. Some applications may include smartphone apps,
AR headsets, or autonomous cars. However, as our method cannot work
independently of a larger system, the negative or ethical issues are not
directly associated with our approach but rather with the specific business
and final application where image matching may be used.

Reproducibility. The experiments are computed on standard and public datasets
and tasks, and hence, they can be reproduced. Moreover, we made public the
evaluation and training scripts, as well as our custom training dataset. In
addition, to encourage the research on scale estimation, we published the test
set of section 5.2 and splits of section 5.3 for easier comparison and support
of future works.

##  7 Conclusions

We introduced ScaleNet, an approach that estimates the scale change between
images and improves the performance of methods that search for correspondences
throughout different views of the same scene. We proposed a novel learning
scheme that formulates the problem of scale estimation as a prediction of a
probability distribution of scales. We demonstrated how to make use of images
from non-planar scenes to generate the training data. In addition to ScaleNet,
we also introduced D-ScaleNet, a discrete variant of the proposed approach,
and demonstrated its effectiveness in 3D-related tasks as well as
computational time. We proved that ScaleNet can improve the results of popular
pipelines in image matching for relative camera pose or 3D reconstruction
while not being limited only to these tasks.

Acknowledgements. This project was supported by Chist-Era EPSRC IPALM
EP/S032398/1 grant.

##  Appendix A Dataset details

Real pair images are obtained directly from the Megadepth dataset [21] as
detailed in the main paper. As a reference, we display in figure 6 examples of
our dataset with their labeled scale changes. Since our labeling strategy
relies on a 3D model, we can capture extreme scale changes between images if
the collection of images that was used for the 3D reconstruction was big.
While keypoint/descriptors may not be able to match correctly two images when
there are strong scale changes, we rely on the fact that 3D models can relate
two images if there were images in-between, i.e., we can compute the
relationship between image $A$ and $B$ if we have an easier-to-match image $C$
in between the two views. This idea is somehow similar to the strategy
proposed in MODS [30], where authors create synthetic images between two
different views to be able to relate them. In addition, we also use synthetic
pairs to train ScaleNet. Synthetic pairs are computed on the fly, and
therefore an unlimited number of examples can be created. We define a set of
synthetic transformations with affine parameters: scale $\in[0.16,6]$,
rotation $\in[-30$°$,30$°$]$ and skew $\in[-0.2,0.2]$. We use the random
transformation to wrap the input image and generate the training pair. The
ground-truth scale is directly the scale factor used to transform the image.

##  Appendix B Local-ScaleNet

Local-ScaleNet predicts a dense scale factor map between a pair of input
images. In contrast to ScaleNet, in which scale estimation is global, pixel-
wise Local-ScaleNet estimations offer a more suitable option when scale
transformation significantly varies across the images, e.g., images with
strong perspective distortion.

Figure 6: Image pairs examples of our dataset with their scale factor
annotations computed as the average on logarithmic space of local scale
changes. Figure 7: Local-ScaleNet uses a pre-trained VGG-16 and an ASPP block
as its multi-scale feature extractor. After features are computed, a
combination of self- and cross-correlation layers are used to calculate the
relationship within image $A$ and between images $A$ and $B$. Correlation
volumes’ dimensionality is reduced through a CNN, and its results are
concatenated into a single map. Finally, a dense scale distribution map is
calculated by the final dense prediction block. During inference time, the
scale distribution map is converted into a scale factor map and resized to the
original resolution of image $A$.

Architecture. Local-ScaleNet architecture follows the scheme proposed in
ScaleNet (cf. section 3) with small variations. Analogous to ScaleNet, Local-
ScaleNet takes two input images, $A$ and $B$, and computes features, $f_{A}$
and $f_{B}$, with the same multi-scale feature extractor block. Features,
$f_{A}$ and $f_{B}$, go into the correlation layers to calculate the self- and
cross-similarities, $c_{A}$ and $c_{A-B}$. In contrast to ScaleNet, Local-
ScaleNet does not compute the self-similarities in image $B$. Since Local-
ScaleNet estimates the scale factor locally, self-similarities in image $B$,
$c_{B}$, are not spatially correlated with image $A$, and therefore, $c_{B}$
is not used in the per-pixel scale estimation. Similar to ScaleNet, $c_{A-B}$
and $c_{A}$ go into our local feature reduction block to process and reduce
the channel dimension of the correlation maps. $c^{\prime}_{A}$ and
$c^{\prime}_{A-B}$ are then concatenated and fed into the final dense
prediction network which outputs the local distribution of scales between
images $A$ and $B$. Furthermore, during test time, we transform our scale
distribution into a scale factor map by following the pipeline presented in
the main paper (cf. equation 3) in a pixel-wise manner. Due to image $A$ and
$B$ not correlating spatially, we do not apply the consistency check from the
original ScaleNet. Finally, the scale factor map is resized into the original
image $A$ resolution. Full Local-ScaleNet architecture and pipeline are shown
in figure 7.

Figure 8: The local dataset generation pipeline, firstly, computes a random
homography transformation. The homography is applied to an input image to
generate a tuple of images. We refer to them as Source and Target images. The
same homography is used to calculate the source scale map. Moreover, we take
an extra random image from the dataset to fill those regions with zero values
in the Target image due to the homography transformation.

Dataset. As discussed, Local-ScaleNet generates a dense map of scale
distributions. To train it, we need pairs of images with their corresponding
pixel-wise scale ground-truths. We based our dataset on synthetic pairs of
images as displayed in figure 8. Even though real pairs of images could be
used to train Local-ScaleNet, real image scenes do not present strong local
distortions, i.e., global scale often reflects the scale changes in local
regions. Thus, we use synthetic pairs to ensure that the global scale differs
from the scales between local regions. A random homography transformation is
generated and applied to an input image, such that the input image, Source,
and the warped one, Target, are geometrically related by our homography. We
use the same parameters to build the homography matrix as those presented in
appendix A and add a perspective distortion to ensure different local scale
factors throughout the Target image. Moreover, due to strong homography
transformations, Target image may contain zero values in the non-overlapping
regions as seen in figure 8. To avoid the network being driven by those zero
values, we filled the non-overlapping regions between Source and Target images
with a randomly sampled image from the dataset. Therefore, our final training
pair is composed by the Source and our new Target-Filled image. Lastly, we use
the same homography to generate the ground-truth scale map between Source and
Target-Filled. As we have the synthetic transformation, we generate the source
scale map by sampling local points throughout the whole Source image and
computing the scale factor within their neighborhood region as the ratio of
their distances in the Source and Target-Filled images.

##  Appendix C Extended experiments

This section extends the experiments and results presented in the main paper.

###  C.1 Design choices

ScaleNet embraces some key ideas on its learning scheme and parameterization
that help towards delivering good scale estimations. We discuss them here and
highlight the importance of our decision during ScaleNet design. Table
LABEL:appendix_tab:ablation_study shows the performance of our baseline, which
uses SuperPoint network without any scale rectification. Moreover, we report
the results of ScaleNet without the consistency check for an easier comparison
against other designs.

Natural vs logarithmic space. The scale factor is a relative ratio operator,
hence, it is non-linear. We compute the scale factor as a soft-computation
based on quantized scale classes, $s_{i}$, and a probability distribution (cf.
equation 2). To avoid being biased by high scale values when computing the
soft-scale, we transform the quantized scale classes, $s_{i}$ , to logarithmic
space. Hence, to demonstrate the superiority of this parameterization, we
display in table LABEL:appendix_tab:ablation_study the results of doing the
soft-computation directly in natural space (Natural rep.). We see that even
though the performance of the natural representation model is over the
baseline, we can improve upon it by the simple and effective approach of
dealing with the scale factors in the logarithmic space.

Regression vs classification model. Even though predicting directly the scale
factor between images is theoretically possible, in the practical scenario, it
appears to be a much harder task. To prove it, we have trained a regression
model with the L2 loss, where our network had to predict a normalized scale
factor between the two images in logarithmic space. We use logarithmic space
since the previous experiment shows the benefits of this representation. As
shown in table LABEL:appendix_tab:ablation_study, the proposed ScaleNet
regressor model does not bring any benefit over the baseline accuracy (which
does not use any scale rectification). We claim that even though a regressor
model could be trained, learning and interpreting the relationships between
the quantized scale ranges is an easier task, and hence, we embrace the
decision of a classification model rather than a regressor.

Results in table LABEL:tab:camera_localization give the benefits of camera
pose error when using ScaleNet. The boost of ScaleNet when working together
with a single scale method shows the importance of rectifying the scale
difference between images. Even though multi-scale methods look for features
on multiple scaled images, the best results are obtained when ScaleNet
corrects one of the images before feature extraction. Moreover, ScaleNet
brings benefits to current pipelines when the scale change between image pairs
is extreme, and therefore, if the scale factor is not significant as in the
evaluated scenes, multi-scale feature extraction is sufficient to perform the
correspondence search.

###  C.4 Camera pose

| Pose estimation (AUC)
---|---
| at 5° | at 10° | at 20°
SP + SuperGlue | 35.2 | 54.7 | 71.6
w/ ScaleNet |  36.3 (+3%) |  55.7 (+2%) |  72.4 (+1%)
Table 9: Camera pose results on Megadepth test split from [46]. When scale
changes are small on the image pairs, ScaleNet brings more robustness without
hurting the performance.

Protocol. To test the accuracy of our scale predictions, we compute the mean
scale ratio, $r$, between the ground-truth, $S_{GT}$, and the estimated scale,
$s$. Specifically, we compute the ratio of a scale prediction as
$r=s_{a}/s_{b}$, $s_{a}=max(S_{GT},s)$, and $s_{b}=min(S_{GT},s)$. In addition
to ScaleNet predictions, we also propose two baselines as a reference: Random
makes a random prediction with $s\in[0.16,6]$, and Constant uses a fixed scale
value such as $s=1.0$. In addition, we also indicate the results of perfect
scale estimation in Perfect ($r=1.0$). We use the test set proposed in the
relative camera pose experiment of section 5.2.

Results. We report in table LABEL:appendix_tab:eval_scale the mean scale ratio
between ground-truth and predicted scales. ScaleNet predictions ($r=1.6$) are
accurate enough to bring images to scale discrepancies where current local
feature extractors can operate. However, even though scale prediction accuracy
is relevant, it is hard to quantify. Therefore, we believe that results on
downstream applications are needed to understand the impact of ScaleNet image
rectification.

###  C.7 Complexity time

Time (ms)
---
| DGC-Net | GLU-Net | R2D2 | Key.Net | ScaleNet
| 35.4 | 25.1 | 384.3 | 640.5 | 19.8
Table 12: Comparison of computational times in ms for several state-of-the-art
methods. We report the multi-scale extraction times for R2D2 and
Key.Net/HardNet.

Table 12 introduces the computational time of ScaleNet and other state-of-the-
art methods. Although ScaleNet has the fastest inference time, it only
predicts a single transformation parameter rather than the flow between images
or local keypoints/descriptors. However, we showed that ScaleNet largely
benefits other methods if scale changes are presented in the images and
increases only 19.8ms the computational time of matching two images. Besides
dense geometric matching models, DGC-Net (35.4ms) and GLU-Net (25.1ms), we
also provided the times of multi-scale feature extraction of R2D2-MS (384.3ms)
and Key.Net-MS/HardNet (640.5ms). As multi-scale methods require a minimum
image size to work properly, we report the times when extracting features from
an image of size $600\times 800$, and see that the time for ScaleNet is
significantly smaller than the feature extractors. Thus, we believe that if
scale robustness is needed, the improvements outweigh the computational
overhead introduced by our scale correction.

Besides, some strategies could reduce the impact of using scale correction. As
discussed in the main paper, ScaleNet could use faster and more efficient
feature extractors, i.e., MobileNet [17], or we could precompute ScaleNet
backbone features and store them. Catching ScaleNet backbone features would
allow us only to run correlation and fully connected layers for every new
pairs of images, reducing the complexity of some 3D system, e.g., 3D
reconstruction. As a reference, the time on inference when caching backbone
features is 1.8ms, meanwhile, the inference time of regular ScaleNet is
19.8ms.

##  Appendix D Qualitative examples

In this section, we provide some qualitative results from applying ScaleNet in
image pairs from the Megadepth dataset [21] that contain strong scale changes.
In figure 9, we show the matches that were obtained for each method. We only
plot the matches that survive the Lowe’s ratio test[24] and MAGSAC[2]
geometric fitting method. As in the paper, to avoid the effect of a higher
number of keypoints due to one image being upsampled, we only use the top
2,048 keypoint candidates. We observe that all methods benefit from the scale
correction when images present severe scale variations. Furthermore, examples
show difficult image pairs where the multi-scale methods, SIFT [24], R2D2
[38], and Key.Net/HardNet [3, 29], could not handle the extreme-scale change
and found few or non-matches between pairs. We see that on those examples,
methods benefit largely from scale rectification. Similarly, SuperPoint [11]
could only find correct matches if the scale factor between images is
corrected.

Figure 9: Comparison of matches without (red boxes) and with (green boxes) the
scale rectification proposed by ScaleNet. We only plot the matches that agree
with the camera pose computed by the methods, either with or without ScaleNet.
The examples display extreme scale conditions, and hence, we observe that the
number of matches notably increases when applying ScaleNet’s image
rectification.

## References

  * [1] Vassileios Balntas, Karel Lenc, Andrea Vedaldi, and Krystian Mikolajczyk.  Hpatches: A benchmark and evaluation of handcrafted and learned local descriptors.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5173–5182, 2017.
  * [2] Daniel Barath, Jiri Matas, and Jana Noskova.  Magsac: marginalizing sample consensus.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10197–10205, 2019.
  * [3] Axel Barroso-Laguna, Edgar Riba, Daniel Ponsa, and Krystian Mikolajczyk.  Key.net: Keypoint detection by handcrafted and learned cnn filters.  In Proceedings of the IEEE International Conference on Computer Vision, 2019.
  * [4] Axel Barroso-Laguna, Yannick Verdie, Benjamin Busam, and Krystian Mikolajczyk.  Hdd-net: Hybrid detector descriptor with mutual interactive learning.  In Proceedings of the Asian Conference on Computer Vision, 2020\.
  * [5] Eric Brachmann and Carsten Rother.  Neural-guided ransac: Learning where to sample model hypotheses.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4322–4331, 2019.
  * [6] Mathias Bürki, Lukas Schaupp, Marcin Dymczyk, Renaud Dubé, Cesar Cadena, Roland Siegwart, and Juan Nieto.  Vizard: Reliable visual localization for autonomous vehicles in urban outdoor environments.  In 2019 IEEE Intelligent Vehicles Symposium (IV), pages 1124–1130. IEEE, 2019.
  * [7] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848, 2017.
  * [8] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint arXiv:1706.05587, 2017.
  * [9] Ondrej Chum, Tomas Werner, and Jiri Matas.  Two-view geometry estimation unaffected by a dominant plane.  In Conference on Computer Vision and Pattern Recognition, volume 1, pages 772–779. IEEE, 2005.
  * [10] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A Large-Scale Hierarchical Image Database.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
  * [11] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superpoint: Self-supervised interest point detection and description.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 224–236, 2018.
  * [12] Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler.  D2-net: A trainable cnn for joint detection and description of local features.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  * [13] Mihai Dusmanu, Johannes L. Schönberger, and Marc Pollefeys.  Multi-View Optimization of Local Feature Geometry.  In Proceedings of the European Conference on Computer Vision, 2020\.
  * [14] Patrick Ebel, Anastasiia Mishchuk, Kwang Moo Yi, Pascal Fua, and Eduard Trulls.  Beyond cartesian representations for local descriptors.  In Proceedings of the IEEE International Conference on Computer Vision, pages 253–262, 2019.
  * [15] Hugo Germain, Guillaume Bourmaud, and Vincent Lepetit.  S2dnet: Learning accurate correspondences for sparse-to-dense feature matching.  In European Conference on Computer Vision, 2020.
  * [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [17] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  CoRR, abs/1704.04861, 2017.
  * [18] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al.  Spatial transformer networks.  In Advances in neural information processing systems, pages 2017–2025, 2015.
  * [19] Wei Jiang, Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, and Kwang Moo Yi.  Cotr: Correspondence transformer for matching across images.  arXiv preprint arXiv:2103.14167, 2021.
  * [20] Karel Lenc and Andrea Vedaldi.  Large scale evaluation of local image feature detectors on homography datasets.  In The British Machine Vision Conference (BMVC), 2018.
  * [21] Zhengqi Li and Noah Snavely.  Megadepth: Learning single-view depth prediction from internet photos.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3302–3312, 2018.
  * [22] Philipp Lindenberger, Paul-Edouard Sarlin, Viktor Larsson, and Marc Pollefeys.  Pixel-perfect structure-from-motion with featuremetric refinement.  arXiv preprint arXiv:2108.08291, 2021.
  * [23] Yuan Liu, Zehong Shen, Zhixuan Lin, Sida Peng, Hujun Bao, and Xiaowei Zhou.  Gift: Learning transformation-invariant dense visual descriptors via group cnns.  arXiv preprint arXiv:1911.05932, 2019.
  * [24] David G. Lowe.  Distinctive image features from scale-invariant keypoints.  In International booktitle of computer vision, 2004.
  * [25] Zixin Luo, Lei Zhou, Xuyang Bai, Hongkai Chen, Jiahui Zhang, Yao Yao, Shiwei Li, Tian Fang, , and Long Quan.  Aslfeat: Learning local features of accurate shape and localization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
  * [26] Iaroslav Melekhov, Aleksei Tiulpin, Torsten Sattler, Marc Pollefeys, Esa Rahtu, and Juho Kannala.  DGC-Net: Dense geometric correspondence network.  In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), 2019.
  * [27] Krystian Mikolajczyk and Cordelia Schmid.  Scale & affine invariant interest point detectors.  International journal of computer vision, 60(1):63–86, 2004.
  * [28] Krystian Mikolajczyk and Cordelia Schmid.  A performance evaluation of local descriptors.  In IEEE Transactions on Pattern Analysis and Machine Intelligence, number 10, pages 1615–1630. IEEE, 2005.
  * [29] Anastasiia Mishchuk, Dmytro Mishkin, Filip Radenovic, and Jiri Matas.  Working hard to know your neighbor’s margins: Local descriptor learning loss.  In Advances in Neural Information Processing Systems, pages 4826–4837, 2017.
  * [30] Dmytro Mishkin, Jiri Matas, and Michal Perdoch.  Mods: Fast and robust method for two-view matching.  In Computer Vision and Image Understanding, volume 141, pages 81–93. Elsevier, 2015.
  * [31] Dmytro Mishkin, Filip Radenovic, and Jiri Matas.  Repeatability is not enough: Learning affine regions via discriminability.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 284–300, 2018.
  * [32] Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua.  Learning to find good correspondences.  In In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2666–2674, 2018.
  * [33] Yuki Ono, Eduard Trulls, Pascal Fua, and Kwang Moo Yi.  Lf-net: learning local features from images.  In Advances in Neural Information Processing Systems, pages 6234–6244, 2018.
  * [34] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  2017\.
  * [35] Rémi Pautrat, Viktor Larsson, Martin R. Oswald, and Marc Pollefeys.  Online invariance selection for local feature descriptors.  In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  * [36] Rémi Pautrat, Viktor Larsson, Martin R. Oswald, and Marc Pollefeys.  Online invariance selection for local feature descriptors.  In European Conference on Computer Vision, 2020.
  * [37] Anita Rau, Guillermo Garcia-Hernando, Danail Stoyanov, Gabriel J Brostow, and Daniyar Turmukhambetov.  Predicting visual overlap of images through interpretable non-metric box embeddings.  In European Conference on Computer Vision, pages 629–646. Springer, 2020.
  * [38] Jerome Revaud, Philippe Weinzaepfel, César De Souza, and Martin Humenberger.  R2d2: Repeatable and reliable detector and descriptor.  In Advances in Neural Information Processing Systems, 2019.
  * [39] I. Rocco, R. Arandjelović, and J. Sivic.  Convolutional neural network architecture for geometric matching.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
  * [40] Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk.  From coarse to fine: Robust hierarchical localization at large scale.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [41] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superglue: Learning feature matching with graph neural networks.  In In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
  * [42] Johannes L Schonberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4104–4113, 2016.
  * [43] Johannes L Schonberger, Hans Hardmeier, Torsten Sattler, and Marc Pollefeys.  Comparative evaluation of hand-crafted and learned local features.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1482–1491, 2017.
  * [44] Xi Shen, François Darmon, Alexei A Efros, and Mathieu Aubry.  Ransac-flow: generic two-stage image alignment.  arXiv preprint arXiv:2004.01526, 2020.
  * [45] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In International Conference on Learning Representations, 2015.
  * [46] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou.  Loftr: Detector-free local feature matching with transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8922–8931, 2021.
  * [47] Weiwei Sun, Wei Jiang, Eduard Trulls, Andrea Tagliasacchi, and Kwang Moo Yi.  Acne: Attentive context normalization for robust permutation-equivariant learning.  In In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11286–11295, 2020.
  * [48] Yurun Tian, Vassileios Balntas, Tony Ng, Axel Barroso-Laguna, Yiannis Demiris, and Krystian Mikolajczyk.  D2d: Keypoint extraction with describe to detect approach.  In Proceedings of the Asian Conference on Computer Vision (ACCV), 2020.
  * [49] Yurun Tian, Axel Barroso-Laguna, Tony Ng, Vassileios Balntas, and Krystian Mikolajczyk.  Hynet: Local descriptor with hybrid similarity measure and triplet loss.  In Advances in neural information processing systems, 2020.
  * [50] Yurun Tian, Xin Yu, Bin Fan, Fuchao Wu, Huub Heijnen, and Vassileios Balntas.  Sosnet: Second order similarity regularization for local descriptor learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11016–11025, 2019.
  * [51] Carl Toft, Daniyar Turmukhambetov, Torsten Sattler, Fredrik Kahl, and Gabriel Brostow.  Single-image depth prediction makes feature matching easier, 2020.
  * [52] Prune Truong, Martin Danelljan, Luc Van Gool, and Radu Timofte.  Gocor: Bringing globally optimized correspondence volumes into your neural network, 2021.
  * [53] Prune Truong, Martin Danelljan, Luc Van Gool, and Radu Timofte.  Learning accurate dense correspondences and when to trust them, 2021.
  * [54] Prune Truong, Martin Danelljan, and Radu Timofte.  GLU-Net: Global-local universal network for dense flow and correspondences.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [55] Tinne Tuytelaars and Krystian Mikolajczyk.  Local invariant feature detectors: a survey.  In Foundations and Trends in Computer Graphics and Vision, 2008\.
  * [56] Olivia Wiles, Sebastien Ehrhardt, and Andrew Zisserman.  Co-attention for conditioned image matching.  In In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
  * [57] Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua.  Lift: Learned invariant feature transform.  In European Conference on Computer Vision, pages 467–483. Springer, 2016.
  * [58] Kwang Moo Yi, Yannick Verdie, Pascal Fua, and Vincent Lepetit.  Learning to assign orientations to feature points.  CoRR, abs/1511.04273, 2015.
  * [59] Guoshen Yu and Jean-Michel Morel.  Asift: An algorithm for fully affine invariant comparison.  Image Processing On Line, 1:11–38, 2011.
  * [60] Jin Yuhe, Dmytro Mishkin, Anastasiia Mishchuk, Jiri Matas, Pascal Fua, Kwang Moo Yi, and Eduard Trulls.  Image matching across wide baselines: From paper to practice.  In International Journal of Computer Vision, 2020.
  * [61] Lei Zhou, Siyu Zhu, Tianwei Shen, Jinglu Wang, Tian Fang, and Long Quan.  Progressive large scale-invariant image matching in scale space.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2362–2371, 2017.


# A Waveform Representation Framework for High-quality Statistical Parametric
Speech Synthesis

Bo Fan1, Siu Wa Lee2, Xiaohai Tian34, Lei Xie1 and Minghui Dong2 1 School of
Computer Science, Northwestern Polytechnical University, Xi’an, China
2 Human Language Technology Department, Institute for Infocomm Research,
Singapore
3 School of Computer Engineering, Nanyang Technological University (NTU),
Singapore
4 Joint NTU-UBC Research Center of Excellence in Active Living for the
Elderly, NTU, Singapore
Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

State-of-the-art statistical parametric speech synthesis (SPSS) generally uses
a vocoder to represent speech signals and parameterize them into features for
subsequent modeling. Magnitude spectrum has been a dominant feature over the
years. Although perceptual studies have shown that phase spectrum is essential
to the quality of synthesized speech, it is often ignored by using a minimum
phase filter during synthesis and the speech quality suffers. To bypass this
bottleneck in vocoded speech, this paper proposes a phase-embedded waveform
representation framework and establishes a magnitude-phase joint modeling
platform for high-quality SPSS. Our experiments on waveform reconstruction
show that the performance is better than that of the widely-used STRAIGHT.
Furthermore, the proposed modeling and synthesis platform outperforms a
leading-edge, vocoded, deep bidirectional long short-term memory recurrent
neural network (DBLSTM-RNN)-based baseline system in various objective
evaluation metrics conducted.

##  1 Introduction

Statistical parametric speech synthesis (SPSS) has been increasingly popular
due to its compact and flexible representation of voice characteristics [1].
Conventionally, in an SPSS system, we firstly extract parametric
representations of speech including spectral and excitation parameters from a
speech database and then model them with a set of models [2]. Several
statistical generative models have been applied to SPSS successfully, e.g.,
hidden Markov model (HMM)-based SPSS [2], deep neural network (DNN)-based SPSS
[3] and deep bidirectional long short-term memory recurrent neural network
(DBLSTM-RNN)-based SPSS [4].

To parameterize speech signals into features for subsequent synthesis
processes, vocoder has been typically used. It is based on the source-filter
model [5], which assumes a stationary speech segment is generated by passing a
sound source through a vocal tract filter. By using a vocoder, the resultant
speech features are regular and suitable for modeling. However, in [6], their
subjective listening test shows clear degradation of quality in vocoded
speech. It further indicates that the source and filter parameters have to be
jointly modelled for high-quality synthesis. Besides, to assure interframe
coherence [7], a minimum phase hypothesis [7] has been used in most vocoders,
which ignores the natural mixed-phase characteristics of speech signals,
resulting in apparent degradation of the speech waveform quality.

More and more works have reported the importance of phase information in
different speech processing applications, such as speech synthesis [8, 9],
iterative signal reconstruction [10], automatic speech recognition [11, 12],
speech coding [13] and pitch extraction [14]. Paliwal et al. [15] have
investigated the relative importance of short-time magnitude and phase spectra
on speech perception through human perception listening test. Results show
that phase spectrum clearly contributes to the speech intelligibility.
Sometimes its contribution is as much as the magnitude spectrum.
Koutsogiannaki et al. [16] have proposed the phase distortion deviation
feature, enabling to capture voice irregularities and highlights the
importance of the phase spectrum in voice quality assessment. These two works
indicate that phase information is important for both human perception and
voice quality assessment. Combining phase spectrum with magnitude spectrum in
frequency domain is equivalent to the speech waveform in time-domain.
Therefore, the phase information is focused in our speech waveform
representation framework.

There are some approaches of waveform representation directly in the time
domain. Time domain pitch-synchronous overlap-add (TD-PSOLA) [17] performs
pitch-synchronous analysis, modification and synthesis. During synthesis,
speech frames are summed up. The quality of the reconstructed waveform with
typical pitch or timing modification is similar to that of the original
waveform. Multi-band re-synthesis pitch synchronous overlap add (MBR-PSOLA)
[18] comments TD-PSOLA with three mismatches: phase mismatch, pitch mismatch,
spectral envelope mismatch. It further suggests to solve these mismatches by
re-synthesizing voiced parts of the speech database with constant phase and
constant pitch. The artificial processing in MBR-PSOLA decreases the quality
of speech and leads to buzzy sound [19]. Alternatively, there are a few recent
works for SPSS directly in the time domain. Tokuda et al. [20] have proposed
an approach to model cepstral coefficients to approximate the speech waveform.
In their framework, periodic, voiced components have not been properly
generated yet. In [21], complex cepstrum has been used to embed phase
information for hidden semi-Markov models (HSMM) speech modelling.

In this paper, we propose a phase-embedded waveform representation framework,
and establish a magnitude-phase joint modeling platform for SPSS. This work
uses glottal-synchronous overlap add approach for speech analysis and
synthesis where glottal closure instants (GCIs) are employed. GCIs refer to
the moments of most significant excitation that occur at the level of the
vocal folds during each glottal period [22]. Short-term segments are defined
as any two consecutive GCI periods. In order to produce smooth trajectories of
our features which are required in SPSS, we design a cost function with a
global smoothness constraint. The GCI locations selected are finally
determined by conducting dynamic programming over a list of probable GCI
candidates. Consequently, these segments will be very regular with stable
magnitude and matched phase spectrum. With this waveform representation
framework, the bottleneck suffered from vocoded speech is thus bypassed. This
framework is hence capable of delivering better quality speech over the
vocoded speech. Then we propose an approach for magnitude-phase joint spectrum
modeling. Full spectrum is used in this framework, which is in line with the
satisfactory performance in recent deep learning-based TTS [23]. To leverage
on the modeling power of deep learning, we use DBLSTM-RNN to learn magnitude
and phase spectrum simultaneously. Bidirectional recurrent connections can
fully exploit the speech contextual information in both forward and backward
directions. With purpose-built memory cells to store information, the long
short-term memory (LSTM) architecture does better in finding and taking
advantage of the long range context.

##  2 TD-PSOLA

Time domain pitch-synchronous overlap add (TD-PSOLA) is used for pitch and
timing modification of speech signals [17], [24]. It is also popular for
concatenation-based TTS. As no source-filter decomposition or vocoding is
performed, the quality of resultant speech after analysis and reconstruction
is highly similar to the original speech.

Given an arbitrary speech waveform signal $x(n)$, TD-PSOLA is carried out in
the time domain. It first decomposes $x(n)$ into a sequence of overlapping,
pitch-synchronized segments. Each segment $x_{s}(n)$ lasts for two pitch
periods, running from a pitch period before and another pitch period after the
segment centre. Then a window function $h_{s}(n)$, such as hanning window,
will be applied to each segment. Assuming $S$ denotes the total number of the
segments, where $s=1,2,...,S$,

$x_{s}(n)=h_{s}(n)x(n)$ (1)

$h_{s}(n)$ is non-zero during the above two-pitch period. This is how
$x_{s}(n)$ is extracted for voiced speech; for unvoiced speech, the segment
length is set to a constant. Any modification in pitch or timing can then be
performed on these extracted segments. Finally, modified segments are
overlapped and added to produce the speech output [24].

Although TD-PSOLA generates pitch- and timing-modified output signals with
satisfactory speech quality, using TD-PSOLA in speech synthesis where
statistical averaging, modeling or signal modification are common, is not
sufficient. This is because matched attributes on phase and pitch are needed
[18].

##  3 Waveform Representation Framework

In this work, a glottal-synchronous based waveform representation framework is
proposed for speech modelling. Similar to TD-PSOLA, glottal closure instants
(GCIs) represent both the pitch contours and the boundaries of individual
cycles of speech. Existing GCI detection approaches generally estimate the GCI
locations in a local manner, ignoring the resultant trajectories of various
acoustic attributes, i.e. segment length (representing fundamental frequency
($F0$)), magnitude and phase spectrum, exhibited in the utterance. As smooth
trajectories of these attributes are necessary for SPSS, we revise a state-of-
the-art GCI detection approach, so as to facilitate satisfactory modelling of
these attributes.

###  3.1 System Overview

Figure 1: Our proposed waveform representation framework.

The proposed framework, as shown in Fig. 1, consists of two parts: analysis
and synthesis. In the analysis stage, given an arbitrary waveform, firstly,
the GCI locations are detected by the following revised GCI detection module.
Then, the waveform is decomposed into overlapping short-term segments. Each
segment is defined by any two consecutive GCI periods. Finally, segment
lengths, magnitude and phase spectrum are used to represent these segments.

In the synthesis stage, given corresponding segment lengths, magnitude and
phase spectrum, we convert them into overlapping short-term segments. Then,
the waveform is reconstructed using the similar technique as TD-PSOLA [17].

###  3.2 Glottal Closure Instant Detection

The GCI positions determine the features including segment lengths, magnitude
and phase spectrum. Thus, the GCI detection method is of great importance.

Among the present GCIs detection techniques, the Speech Event Detection using
the Residual Excitation And a Mean-based Signal (SEDREAMS) algorithm [25] is
widely used. In [26], SEDREAMS was shown to have the highest robustness and
reliability. During the detection, SEDREAMS outputs only one GCI location for
each GCI segment [25]. This is a local estimation process, without considering
the GCI detection results in the neighborhood. However, SPSS requires smooth
trajectories of speech features, which are defined once GCI locations are
determined. By considering lists of probable GCI candidates and estimating the
optimal GCI locations in a global manner, the trajectories of these features
are stabilized.

Based on SEDREAMS, our modified GCI detection method contains the following
steps:

  * a)

Given a waveform $x(n)$ (Fig. 2(a)), calculate the moving average signal (Fig.
2(b)).

  * b)

Determine the intervals for possible GCI locations111For the detailed
implementations of the moving average filter and interval determination,
please refer to [25] (Fig. 2(c)).

  * c)

$M$ candidates are chosen, based on the top $M$ highest linear predictive
coding (LPC) residual values in the LPC residual signal (Fig. 2(d)), as the
possible GCI locations in each interval. Suppose there are $N$ intervals, the
$k$-th candidate of $i$-th interval denoted as $g_{i,k}$.

  * d)

Transfer all the possible segment lengths into $F0$. For the $i$-th segment,
the $j$-th $F0$ is expressed as

$F0_{i,j}=Fs/(g_{(i+1),s}-g_{i,t}),$ (2)

where $F_{s}$ is the sampling frequency, $i=1,2,...,N$, $j=1,2,...,M\times M$,
$s=1,2,...,M$ and $t=1,2,...,M$.

  * e)

Given the reference $F0^{\text{ref}}$, the optimal segment lengths are
determined by dynamic programming with the following constraint,

$E={\arg}\mathop{\min}\limits_{j}\sum\nolimits_{i=1}^{N}\left\|{F0^{\text{ref}}}-{F0_{i,j}}\right\|.$
(3)

  * f)

Finally, the GCI locations are deduced accordingly (Fig. 2(e)).

In our implementation, $M$ is five and the reference $F0$ is extracted by
STRAIGHT [27]. STRAIGHT is robust for $F0$ tracking and can generate a highly
accurate and smooth $F0$ trajectory. The $F0$ trajectory extracted from
STRAIGHT is robust The dynamic programming process is implemented by the
Viterbi algorithm. In a voiced segment, the pitch located in the middle is
more stable compared to the rest. Consequently, Viterbi search starts at this
middle position to both ends.

Figure 2: (a) A section of voiced waveform; (b) The corresponding moving
average signal; (c) Short intervals in the moving average signal; (d) LPC
residual signal in one interval with candidates marked with red cross; (e) The
final GCI locations marked with red stem.

A comparison of $F0$ trajectory between our GCI detection and SEDREAMS is
depicted in Fig. 3. From Fig. 3(a), it is observed that the $F0$ given by our
GCI detection is smoother than the one from SEDREAMS. And from Fig. 3(b), it
is clear that our GCI detection approach removes some abnormal jumps (around
the 247-th frame) of the $F0$ trajectory occurred in the SEDREAMS.

Figure 3: A comparison of $F0$ trajectory between our GCI detection and
SEDREAMS. (a) and (b) are two segments in the voiced parts. Figure 4: Static
and dynamic phase spectrum as well as their corresponding magnitude spectrum.

##  4 Waveform Modeling

State-of-the-art SPSS usually models the magnitude spectrum of speech signals
and discards the phase spectrum. During synthesis, a vocoder based on minimum-
phase or zero-phase filter is often used together with the generated magnitude
spectra to produce the synthesized output. Nevertheless, phase spectrum has
been recently found to be essential for speech perception. The speech quality
of vocoded outputs are found to be degraded from the original speech
recordings [6]. This may shed light on SPSS, where speech waveform with phase
information in addition to the existing magnitude spectrum, is modeled.

In our work, speech signals are modeled by the corresponding magnitude and
phase spectra, without the use of a vocoder. Consequently, reconstruction of
speech waveform is facilitated. We use a recently-emerging learning technique,
DBLSTM-RNN, to jointly model the two spectra. DBLSTM-RNN is well-suited for
learning sequential events apart from long time lags of unknown size [28].
Promising performance in various speech applications is observed [29], [4].

Our joint model of magnitude and phase is constructed as follows. We employ
line spectrum pair (LSP) as the feature representation of magnitude spectrum.
LSP, being an alternative LPC spectral representation, is robust and suitable
for interpolation and modeling [30], [31].

For phase spectrum, we propose to use the dynamic phase spectrum for this
waveform learning TTS framework. It is also called group delay: the group
delay $\tau_{k}(n)$ at time $n$ and frequency bin $k$ is calculated as the
frequency derivative of the instantaneous phase $\theta_{k}(n)$, i.e.

$\tau_{k}(n)=\theta_{k}(n)-\theta_{k-1}(n).$ (4)

To enable reconstruction of the phase spectrum after DBLSTM-RNN modeling, the
instantaneous phase at the first frequency bin is kept, together with the
group delays of the remaining frequency bins. In other words, our phase
representation consists of
$\theta_{1}(n),\tau_{2}(n),\tau_{3}(n),...,\tau_{K}(n)$, where $K$ is the
total number of frequency bins.

This group-delay-based phase representation is found to be stable and
facilitates statistical modeling in subsequent TTS process, as shown in Fig.
4. Comparing the spectra of static phase and dynamic phase, the distribution
of the dynamic phase often exhibits a smaller range. Comparing the log
magnitude spectrum with the dynamic phase spectrum, patterns of voiced and
unvoiced portions are consistent and spectral patterns of individual speech
sounds are quite similar in the log magnitude spectrum and the dynamic phase
spectrum. This is important and useful for our joint modeling. On the
contrary, there is no clear difference in the static phase spectrum for
individual speech sounds. When moving along the time-axis, the static phase
spectra look like the same.

##  5 Experiments

We conducted two experiments to assess the efficacy of our waveform
representation framework. In the experiment on waveform reconstruction,
objective and subjective evaluations were carried out to compare the
performance between our framework and other three vocoders: STRAIGHT, Tandem-
STRAIGHT [32] and AHOCoder [33] respectively. As we know, STRAIGHT is a very
popular vocoder used for speech analysis and reconstruction, and Tandem-
STRAIGHT is the upgrade version of STRAIGHT. AHOCoder is reported to be of
similar quality compared with STRAIGHT. In the experiment on waveform
modeling, we trained a text-to-speech (TTS) system based on our framework and
also a baseline TTS system [4] as a comparison. This baseline is a leading-
edge approach based on DBLSTM-RNN and generates high-quality synthesized
speech. It uses STRAIGHT as its vocoder.

A corpus with 4,936 Chinese utterances (around 6 hours) spoken by a native
male speaker in a neutral style was used in our experiments. Speech waveform
signals are sampled at 16kHz. The contextual labels are both phonetically and
prosodically rich, including quin-phone, prosody, tone and syllable
information. For TTS systems, the training, validation and test data consist
of 3,949, 494 and 493 utterances, respectively.

###  5.1 Experiment on Waveform Reconstruction

Speech waveform in the test set of the corpus was analyzed and re-synthesized
using our waveform representation framework and the three vocoders. The
reconstructed speech waveform was then used for objective and subjective
evaluations.

####  5.1.1 Objective Evaluation

In the objective evaluation, we calculated the root mean square error (RMSE)
between the reconstructed and original speech waveform signals in the voiced
parts (RMSE_voiced), the unvoiced parts (RMSE_unvoiced) and the entire
waveform (RMSE), respectively. The results are shown in Table 1. These
voiced/unvoiced results from our framework and the three vocoders generally
represent the performance on vowels/consonants respectively.

Table 1:  Reconstruction performance: our framework vs. the three vocoders

.

Methods Measures RMSE_voiced RMSE_unvoiced RMSE Our framework 0.026 0.042
0.031 STRAIGHT [27] 0.173 0.044 0.152 Tandem-STRAIGHT [32] 0.177 0.044 0.156
AHOCoder [33] 0.182 0.049 0.160

Figure 5: The reconstructed waveform using our framework and STRAIGHT.

The objective evaluation result shows that the performance of our framework is
much better than that of the three vocoders especially in the voiced parts.
The short-term segments are extracted at a constant rate in the unvoiced parts
from our framework which is similar to STRAIGHT. Taking the waveform in Fig. 5
around the 5000-th sample as an example, the absolute difference between (a)
and (b) is very close to that between (a) and (c). In the voiced parts, our
framework performs much better than STRAIGHT does. It is because our framework
retains the full phase spectrum, while STRAIGHT discards it and uses a
minimum-phase setting instead. We can see clearly from Fig. 5 that the
absolute difference between (a) and (b) is much smaller than that between (a)
and (c) in the voiced parts.

####  5.1.2 Subjective Evaluation

20 pairs of speech waveform are randomly selected from the reconstructed
waveforms. Then a group of 20 subjects were asked to perform the ABX
preference test. We put the original waveform into X, while we put the
waveform reconstructed using our framework and each of the three vocoders into
A and B randomly. Each subject was asked to answer which one(A or B) is more
similar to X. The third option Neutral means the subject has no preference on
A or B. The ABX result is shown in Fig. 6. We can clearly see that the
reconstructed speech waveform using our framework is significantly preferred
as compared with all of the three vocoders.

Figure 6: The ABX result of the reconstructed speech waveform using our
framework and the three vocoders. We conducted $t$-test using a significance
level of $p<0.05$ which is depicted with the error bars in orange.

###  5.2 Experiment on Waveform Modeling

In the baseline DBLSTM-RNN-based TTS [4], STRAIGHT is used to vocode the
speech waveform by a 25-ms moving window, and shifted every 5-ms. The
generated magnitude spectrum from STRAIGHT was converted into LSP. The
dimensionality of the input contextual label is 427. The output feature
contains voiced/unvoiced flag (1 dimension), log F0 (1 dimension), LSP (40
dimensions) and gain (1 dimension), totally 43 dimensions. As suggested in
[4], a neural network with two BLSTM layers sitting on two feed forward layers
with 256 nodes in each layer is employed to train the DBLSTM-RNN-based TTS.

For our TTS system, features were extracted from the short-term segments
specified by GCI locations. The format of the input label is the same as the
baseline. The segment length is transformed into $F0$. The output feature
comprises several components: voice/unvoiced flag (1 dimension), log $F0$ (1
dimension), LSP (40 dimensions), gain (1 dimension) and dynamic phase feature
(257 dimensions), totally 300 dimensions. The same network topology as
baseline is used to train our TTS system.

To evaluate the performance of these two TTS systems, five metrics are used
for objective evaluation:

  * •

RMSE_$F0$: root mean square error in $F0$ estimation;

  * •

Voiced/unvoiced (V/U) error rate;

  * •

Log spectral distance (LSD):

$LSD({\bf S}_{p},{\bf S}_{g})=\\\
\sqrt{\frac{1}{N}\sum_{j=1}^{N}(\sum_{k=1}^{M_{s}}[10log_{10}\,s_{p}(j,k)-10log_{10}\,s_{g}(j,k)]^{2})},$
(5)

where ${\bf S}_{p}$ and ${\bf S}_{g}$ are the predicted and ground-truth
magnitude spectrum, respectively. $N$ is the total number of frames in the
voiced parts and $M_{s}$ refers to the dimensionality of magnitude spectrum.
$s_{p}(j,k)$ is the the $k$-th value of magnitude in $j$-th frame;

  * •

Mel cepstral distance (MCD):

$MCD({\bf c}_{p},{\bf
c}_{g})=\frac{10}{ln\,10}{\sqrt{2\sum_{k=1}^{M_{c}}[c_{p}(k)-c_{g}(k)]^{2}}},$
(6)

where ${\bf c}_{p}$ and ${\bf c}_{g}$ are the predicted and ground-truth Mel
cepstrum coefficient vectors, respectively, and $M_{c}$ refers to the
dimensionality of Mel cepstrum coefficients;

  * •

Dynamic phase distance (DPD):

$DPD({\bf d}_{p},{\bf
d}_{g})=\sqrt{\sum_{k=1}^{M_{d}}[d_{p}(k)-d_{g}(k)]^{2}},$ (7)

where ${\bf d}_{p}$ and ${\bf d}_{g}$ are the predicted and ground-truth
dynamic phase feature vectors, respectively, and $M_{d}$ refers to the
dimensionality of the dynamic phase feature.

The synthesized speech waveform from the labels in the test set uses the
ground-truth durations. These five metrics are calculated at the GCIs level,
i.e., the short-term segments are specified by the GCIs locations. In order to
make the systems comparable, GCI detection is required for all speech
waveforms synthesized from any system under comparison. And after the GCI
detection, it should be aligned to the ground-truth GCIs by finding out the
closest one.

The objective evaluation result is shown in Table 2. It shows that our TTS
system is better than the baseline in terms of all the five metrics. In
particular, for DPD, the average absolute difference in one frequency bin is
about 0.70rad in our TTS system while 0.91rad for the baseline TTS system.

Table 2:  Objective evaluation on waveform modeling with $t$-test using a
significance level of $p<0.05$.

Measures Methods Our TTS system Baseline [4] RMSE_F0 (Hz) 23.6 $\pm$ 1.2 28.0
$\pm$ 2.3 V/U error rate (%) 5.9 $\pm$ 0.6 8.6 $\pm$ 1.0 LSD (dB) 59.2 $\pm$
0.8 63.9 $\pm$ 1.1 MCD (dB) 4.5 $\pm$ 0.1 4.8 $\pm$ 0.1 DPD (rad) 11.4 $\pm$
0.2 14.7 $\pm$ 0.3

##  6 Conclusions and Future Work

This paper proposed a glottal-synchronous based waveform representation
framework for high-quality statistical parametric speech synthesis. Speech
signal was represented by magnitude and phase full-spectral components,
without the the use of a vocoder. We revised the SEDREAMS GCI detection
approach to improve the feature stability for statistical modelling. Both
objective and subjective evaluations were conducted to assess the
reconstruction performance of our framework. Results indicate that, comparing
to the reconstructed signal obtained by three popular vocoders, the proposed
framework achieves promising results in RMSE in time domain speech waveform
and preference score.

We also proposed a platform for speech modelling. DBLSTM-RNN is applied to
jointly model the corresponding magnitude and phase spectra, and group delay-
based phase representation is used to facilitate statistical modelling.
Objective results show that, the TTS system based on the proposed framework
generates the features, specifically the phase feature, with lower distortion
as compared with a vocoder based system. Further works include studying the
speech quality of synthesized speech and the associated factors and
experiments on subjective evaluation.

## Acknowledgment

This work was supported by the National Natural Science Foundation of China
(61175018 and 61571363).

## References

  * [1] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura,  “Simultaneous modeling of spectrum, pitch and duration in HMM-based speech synthesis,”  in Proc. Eurospeech, 1999, pp. 2347–2350.
  * [2] H. Zen, K. Tokuda, and A. Black,  “Statistical parametric speech synthesis,”  Speech Communication, vol. 51, no. 11, pp. 1039–1064, 2009.
  * [3] H. Zen, A. Senior, and M. Schuster,  “Statistical parametric speech synthesis using deep neural networks,”  in Proc. ICASSP. IEEE, 2013, pp. 7962–7966.
  * [4] Y. Fan, Y. Qian, F. Xie, and F. Soong,  “TTS synthesis with bidirectional LSTM based recurrent neural networks,”  in Proc. Interspeech, 2014, pp. 1964–1968.
  * [5] G. Fant,  Acoustic Theory of Speech Production,  The Hague: Mouton, 1960.
  * [6] T. Merritt, T. Raitio, and S. King,  “Investigating source and filter contributions, and their interaction, to statistical parametric speech synthesis,”  in Proc. Interspeech, 2014, pp. 1509–1513.
  * [7] M. Crespo, P. Velasco, L. Serrano, and J. Sardina,  “On the use of a sinusoidal model for speech synthesis in text-to-speech,”  in Progress in Speech Synthesis, pp. 57–70. Springer, 1997.
  * [8] H. Banno, K. Takeda, and F. Itakura,  “The effect of group delay spectrum on timbre,”  Acoustical Science and Technology, vol. 23, no. 1, pp. 1–9, 2002\.
  * [9] G. Degottex, A. Roebel, and X. Rodet,  “Phase minimization for glottal model estimation,”  Audio, Speech, and Language Processing, IEEE Transactions on, vol. 19, no. 5, pp. 1080–1090, 2011.
  * [10] L. Alsteris and K. Paliwal,  “Iterative reconstruction of speech from short-time Fourier transform phase and magnitude spectra,”  Computer Speech & Language, vol. 21, no. 1, pp. 174–186, 2007\.
  * [11] R. Schluter and H. Ney,  “Using phase spectrum information for improved speech recognition performance,”  in Proc. ICASSP. IEEE, 2001, vol. 1, pp. 133–136.
  * [12] G. Shi, M. Shanechi, and P. Aarabi,  “On the importance of phase in human speech recognition,”  Audio, Speech, and Language Processing, IEEE Transactions on, vol. 14, no. 5, pp. 1867–1874, 2006.
  * [13] H. Pobloth and W. Kleijn,  “Squared error as a measure of perceived phase distortion,”  The Journal of the Acoustical Society of America, vol. 114, no. 2, pp. 1081–1094, 2003.
  * [14] T. Nakatani, T. Irino, and P. Zolfaghari,  “Dominance spectrum based v/uv classification and F0 estimation,”  in Proc. Eurospeech, 2003, pp. 2313–2316.
  * [15] K. Paliwal and L. Alsteris,  “Usefulness of phase spectrum in human speech perception,”  in Proc. Eurospeech, 2003, pp. 2117–2120.
  * [16] M. Koutsogiannaki, O. Simantiraki, G. Degottex, and Y. Stylianou,  “The importance of phase on voice quality assessment,”  in Proc. Interspeech, 2014, pp. 1653–1657.
  * [17] E. Moulines and F. Charpentier,  “Pitch-synchronous waveform processing techniques for text-to-speech synthesis using diphones,”  Speech communication, vol. 9, no. 5, pp. 453–467, 1990.
  * [18] T. Dutoit and H. Leich,  “MBR-PSOLA: Text-to-speech synthesis based on an MBE re-synthesis of the segments database,”  Speech Communication, vol. 13, no. 3, pp. 435–440, 1993.
  * [19] Y. Stylianou,  “Removing linear phase mismatches in concatenative speech synthesis,”  Speech and Audio Processing, IEEE Transactions on, vol. 9, no. 3, pp. 232–239, 2001.
  * [20] K. Tokuda and H. Zen,  “Directly modeling speech waveforms by neural networks for statistical parametric speech synthesis,”  in Proc. ICASSP, 2015, pp. 4215–4219.
  * [21] R. Maia, M. Akamine, and M. Gales,  “Complex cepstrum as phase information in statistical parametric speech synthesis,”  in Proc. ICASSP. IEEE, 2012, pp. 4581–4584.
  * [22] R. Smits and B. Yegnanarayana,  “Determination of instants of significant excitation in speech using group delay function,”  Speech and Audio Processing, IEEE Transactions on, vol. 3, no. 5, pp. 325–333, 1995.
  * [23] Z. Ling, L. Deng, and D. Yu,  “Modeling spectral envelopes using restricted Boltzmann machines and deep belief networks for statistical parametric speech synthesis,”  Audio, Speech, and Language Processing, IEEE Transactions on, vol. 21, no. 10, pp. 2129–2139, 2013.
  * [24] P. Taylor,  Text-to-Speech Synthesis,  United Kingdom: University of Cambridge, 2007.
  * [25] T. Drugman and T. Dutoit,  “Glottal closure and opening instant detection from speech signals,”  in Proc. Interspeech, 2009, pp. 2891–2894.
  * [26] T. Drugman, M. Thomas, J. Gudnason, P. Naylor, and T. Dutoit,  “Detection of glottal closure instants from speech signals: a quantitative review,”  Audio, Speech, and Language Processing, IEEE Transactions on, vol. 20, no. 3, pp. 994–1006, 2012.
  * [27] H. Kawahara, I. Masuda-Katsuse, and A. De Cheveigne,  “Restructuring speech representations using a pitch adaptive time-frequency smoothing and an instantaneous-frequency-based F0 extraction: Possible role of a repetitive structure in sounds,”  Speech communication, vol. 27, no. 3, pp. 187–207, 1999.
  * [28] A. Graves,  Supervised Sequence Labelling with Recurrent Neural Networks,  Springer, 2012.
  * [29] A. Graves, A. Mohamed, and G. Hinton,  “Speech recognition with deep recurrent neural networks,”  in Proc. ICASSP. IEEE, 2013, pp. 6645–6649.
  * [30] F. K. Soong and B.-H. Juang,  “Line spectrum pair (LSP) and speech data compression,”  in Proc. ICASSP. IEEE, 1984, pp. 37–40.
  * [31] F. Itakura,  “Line spectrum representation of linear predictor coefficients of speech signals,”  The Journal of the Acoustical Society of America, vol. 57, pp. S35, 1975.
  * [32] H. Kawahara, M. Morise, T. Takahashi, R. Nisimura, T. Irino, and H. Banno,  “TANDEM-STRAIGHT: A temporally stable power spectral representation for periodic signals and applications to interference-free spectrum, F0, and aperiodicity estimation,”  in Proc. ICASSP, 2008, pp. 3933–3936.
  * [33] D. Erro, I. Sainz, E. Navas, and I. Hernaez,  “Harmonics plus noise model based vocoder for statistical parametric speech synthesis,”  IEEE Journal of Selected Topics in Signal Processing, vol. 8, no. 2, pp. 184–194, 2014.