# Autoencoders for music sound modeling: a comparison of linear, shallow,
deep, recurrent and variational models

###### Abstract

This study investigates the use of non-linear unsupervised dimensionality
reduction techniques to compress a music dataset into a low-dimensional
representation which can be used in turn for the synthesis of new sounds. We
systematically compare (shallow) autoencoders (AEs), deep autoencoders (DAEs),
recurrent autoencoders (with Long Short-Term Memory cells – LSTM-AEs) and
variational autoencoders (VAEs) with principal component analysis (PCA) for
representing the high-resolution short-term magnitude spectrum of a large and
dense dataset of music notes into a lower-dimensional vector (and then convert
it back to a magnitude spectrum used for sound resynthesis). Our experiments
were conducted on the publicly available multi-instrument and multi-pitch
database NSynth. Interestingly and contrary to the recent literature on image
processing, we can show that PCA systematically outperforms shallow AE. Only
deep and recurrent architectures (DAEs and LSTM-AEs) lead to a lower
reconstruction error. The optimization criterion in VAEs being the sum of the
reconstruction error and a regularization term, it naturally leads to a lower
reconstruction accuracy than DAEs but we show that VAEs are still able to
outperform PCA while providing a low-dimensional latent space with nice
“usability” properties. We also provide corresponding objective measures of
perceptual audio quality (PEMO-Q scores), which generally correlate well with
the reconstruction error.

##  1 Introduction

Deep neural networks, and in particular those trained in an unsupervised (or
self-supervised) way such as autoencoders [1] or GANs [2], have shown nice
properties to extract latent representations from large and complex datasets.
Such latent representations can be sampled to generate new data. These types
of models are currently widely used for image and video generation [3, 4, 5].
In the context of a project aiming at designing a music sound synthesizer
driven by high-level control parameters and propelled by data-driven machine
learning, we investigate the use of such techniques for music sound generation
as an alternative to classical music sound synthesis techniques like additive
synthesis, subtractive synthesis, frequency modulation, wavetable synthesis or
physical modeling [6].

So far, only a few studies in audio processing have been proposed in this
line, with a general principle that is similar to image
synthesis/transformation: projection of the signal space into a low-
dimensional latent space (encoding or embedding), modification of the latent
coefficients, and inverse transformation of the modified latent coefficients
into the original signal space (decoding).

In [7, 8], the authors implemented this principle with autoencoders to process
normalized magnitude spectra. An autoencoder (AE) is a specific type of
artificial neural network (ANN) architecture which is trained to reconstruct
the input at the output layer, after passing through the latent space.
Evaluation was made by computing the mean squared error (MSE) between the
original and the reconstructed magnitude spectra.

In [9], NSynth, an audio synthesis method based on a time-domain autoencoder
inspired from the WaveNet speech synthesizer [10] was proposed. The authors
investigated the use of this model to find a high-level latent space well-
suited for interpolation between instruments. Their autoencoder is conditioned
on pitch and is fed with raw audio from their large-scale multi-instrument and
multi-pitch database (the NSynth dataset). This approach led to promising
results but has a high computational cost.

Another technique to synthesize data using deep learning is the so-called
variational autoencoder (VAE) originally proposed in [11], which is now
popular for image generation. A VAE can be seen as a probabilistic/generative
version of an AE. Importantly, in a VAE, a prior can be placed on the
distribution of the latent variables, so that they are well suited for the
control of the generation of new data. This has been recently exploited for
the modeling and transformation of speech signals [12, 13] and also for music
sounds synthesis [14], incorporating some fitting of the latent space with a
perceptual timbre space. VAEs have also been recently used for speech
enhancement [15, 16, 17].

Figure 1: Global diagram of the sound analysis-transformation-synthesis
process.

In line with the above-presented studies, the goal of the present paper is i)
to provide an extensive comparison of several autoencoder architectures
including shallow, deep, recurrent and variational autoencoders, with a
systematic comparison to a linear dimensionality reduction technique, in the
present case Principal Component Analysis (PCA) (to the best of our knowledge,
such comparison of non-linear approaches with a linear one has never been done
in previous studies). This is done using both an objective physical measure
(root mean squared error – RMSE) and an objective perceptual measure (PEMO-Q
[18]); ii) to compare the properties of the latent space in terms of
correlation between the extracted dimensions; and iii) to illustrate how
interpolation in the latent space can be performed to create interesting
hybrid sounds.

##  2 Methodology

The global methodology applied for (V)AE-based analysis-transformation-
synthesis of audio signals in this study is in line with previous works [7, 8,
12, 13]. It is illustrated in Fig. 1 and is described in the next subsections.

###  2.1 Analysis-Synthesis

First, a Short-Term Fourier Transform (STFT) analysis is performed on the
input audio signal. The magnitude spectra are sent to the model (encoder
input) on a frame-by-frame basis, and the phase spectra are stored for the
synthesis stage. After possible modifications of the extracted latent
variables (at the bottleneck layer output, see next subsection), the output
magnitude spectra is provided by the decoder. The output audio signal is
synthesized by combining the decoded magnitude spectra with the phase spectra,
and by applying inverse STFT with overlap-add. If the latent coefficients are
not modified in between encoding and decoding, the decoded magnitude spectra
are close to the original ones and the original phase spectra can be directly
used for good quality waveform reconstruction. If the latent coefficients are
modified so that the decoded magnitude spectra become different from the
original one, then the Griffin & Lim algorithm [19] is used to estimate/refine
the phase spectra (the original phase spectra are used for initialization) and
finally reconstruct the time-domain signal. A few more technical details
regarding data pre-processing are given in Section 3.2.

###  2.2 Dimensionality Reduction Techniques

Principal Component Analysis: As a baseline, we investigated the use of PCA to
reduce the dimensionality of the input vector $\mathbf{x}$. PCA is the optimal
linear orthogonal transformation that provides a new coordinate system (i.e.
the latent space) in which basis vectors follow modes of greatest variance in
the original data [20].

Autoencoder: An AE is a specific kind of ANN traditionally used for
dimensionality reduction thanks to its diabolo shape [21], see Fig. 2. It is
composed of an encoder and a decoder. The encoder maps a high-dimensional low-
level input vector $\mathbf{x}$ into a low-dimensional higher-level latent
vector $\mathbf{z}$, which is assumed to nicely encode properties or
attributes of $\mathbf{x}$. Similarly, the decoder reconstructs an estimate
$\mathbf{\hat{x}}$ of the input vector $\mathbf{x}$ from the latent vector
$\mathbf{z}$. The model is written as:

$\mathbf{z}=f_{\text{enc}}(\mathbf{W}_{\text{enc}}\mathbf{x}+\mathbf{b}_{\text{enc}})\quad\textnormal{
and
}\quad\mathbf{\hat{x}}=f_{\text{dec}}(\mathbf{W}_{\text{dec}}\mathbf{z}+\mathbf{b}_{\text{dec}}),$

where $f_{\text{enc}}$ and $f_{\text{dec}}$ are (entry-wise) non-linear
activation functions, $\mathbf{W}_{\text{enc}}$ and $\mathbf{W}_{\text{dec}}$
are weight matrices and $\mathbf{b}_{\text{enc}}$ and
$\mathbf{b}_{\text{dec}}$ are bias vectors. For regression tasks (such as the
one considered in this study), a linear activation function is generally used
for the output layer.

At training time, the weight matrices and the bias vectors are learned by
minimizing some cost function over a training dataset. Here we consider the
mean squared error (MSE) between the input $\mathbf{x}$ and the output
$\mathbf{\hat{x}}$.

The model can be extended by adding hidden layers in both the encoder and
decoder to create a so-called deep autoencoder (DAE), as illustrated in Fig.
2. This kind of architecture can be trained globally (end-to-end) or layer-by-
layer by considering the DAE as a stack of shallow AEs [1, 22].

Figure 2: General architecture of a (deep) autoencoder.

LSTM Autoencoder: In a general manner, a recurrent neural network (RNN) is an
ANN where the output of a given hidden layer does not depend only on the
output of the previous layer (as in a feedforward architecture) but also on
the internal state of the network. Such internal state can be defined as the
output of each hidden neuron when processing the previous input observations.
They are thus well-suited to process time series of data and capture their
time dependencies. Such networks are here expected to extract latent
representations that encode some aspects of the sound dynamics. Among
different existing RNN architectures, in this study we used the Long Short-
Term Memory (LSTM) network [23], which is known to tackle correctly the so-
called vanishing gradient problem in RNNs [24]. The structure of the model
depicted in Fig. 2 still holds while replacing the classical neuronal cells by
LSTM cells, leading to a LSTM-AE. The cost function to optimize remains the
same, i.e. the MSE between the input $\mathbf{x}$ and the output
$\mathbf{\hat{x}}$. However, the model is much more complex and has more
parameters to train [23].

Variational Autoencoder: A VAE can be seen as a probabilistic AE which
delivers a parametric model of the data distribution, such as:

$p_{\theta}(\mathbf{x},\mathbf{z})=p_{\theta}(\mathbf{x}|\mathbf{z})p_{\theta}(\mathbf{z}),$

where $\theta$ denotes the set of distribution parameters. In the present
context, the likelihood function $p_{\theta}(\mathbf{x}|\mathbf{z})$ plays the
role of a probabilistic decoder which models how the generation of observed
data $\mathbf{x}$ is conditioned on the latent data $\mathbf{z}$. The prior
distribution $p_{\theta}(\mathbf{z})$ is used to structure (or regularize) the
latent space. Typically a standard Gaussian distribution
$p_{\theta}(\mathbf{z})=\mathcal{N}(\mathbf{z};\mathbf{0},\mathbf{I})$ is
used, where $\mathbf{I}$ is the identity matrix [11]. This encourages the
latent coefficients to be mutually orthogonal and lie on a similar range. Such
properties may be of potential interest for using the extracted latent
coefficients as control parameters of a music sound generator. The likelihood
$p_{\theta}(\mathbf{x}|\mathbf{z})$ is defined as a Gaussian density:

$p_{\theta}(\mathbf{x}|\mathbf{z})=\mathcal{N}(\mathbf{x};\bm{\mu}_{\theta}(\mathbf{z}),\bm{\sigma}^{2}_{\theta}(\mathbf{z})),$

where $\bm{\mu}_{\theta}(\mathbf{z})$ and
$\bm{\sigma}^{2}_{\theta}(\mathbf{z})$ are the outputs of the decoder network
(hence $\theta=\\{\mathbf{W}_{\text{dec}},\mathbf{b}_{\text{dec}}\\}$). Note
that $\bm{\sigma}_{\theta}^{2}(\mathbf{z})$ indifferently denotes the
covariance matrix of the distribution, which is assumed diagonal, or the
vector of its diagonal entries.

The exact posterior distribution $p_{\theta}(\mathbf{z}|\mathbf{x})$
corresponding to the above model is intractable. It is approximated with a
tractable parametric model $q_{\phi}(\mathbf{z}|\mathbf{x})$ that will play
the role of the corresponding probabilistic encoder. This model generally has
a form similar to the decoder:

$q_{\phi}(\mathbf{z}|\mathbf{x})=\mathcal{N}(\mathbf{z};\tilde{\bm{\mu}}_{\phi}(\mathbf{x}),\tilde{\bm{\sigma}}^{2}_{\phi}(\mathbf{x})),$

where $\tilde{\bm{\mu}}_{\phi}(\mathbf{x})$ and
$\tilde{\bm{\sigma}}^{2}_{\phi}(\mathbf{x})$ are the outputs of the encoder
ANN (the parameter set $\phi$ is composed of $\mathbf{W}_{enc}$ and
$\mathbf{b}_{enc}$; $\tilde{\bm{\sigma}}_{\phi}^{2}(\mathbf{x})$ is a diagonal
covariance matrix or is the vector of its diagonal entries).

Training of the VAE model, i.e. estimation of $\theta$ and $\phi$, is done by
maximizing the marginal log-likelihood
$\textnormal{log}\,p_{\theta}(\mathbf{x})$ over a large training dataset of
vectors $\mathbf{x}$. It can be shown that the marginal log-likelihood can be
written as [11]:

$\textnormal{log}\
p_{\theta}(\mathbf{x})=\textnormal{D}_{\textnormal{KL}}(q_{\phi}(\mathbf{z}|\mathbf{x})|p_{\theta}(\mathbf{z}|\mathbf{x}))+\mathcal{L}(\phi,\theta,\mathbf{x}),$

where $\textnormal{D}_{\textnormal{KL}}\geq 0$ denotes the Kullback-Leibler
divergence (KLD) and $\mathcal{L}(\phi,\theta,\mathbf{x})$ is the variational
lower bound (VLB) given by:

$\mathcal{L}(\phi,\theta,\mathbf{x})=\underbrace{-\textnormal{D}_{\textnormal{KL}}(q_{\phi}(\mathbf{z}|\mathbf{x})|p_{\theta}(\mathbf{z}))}_{\textnormal{regularization}}+\underbrace{\mathbb{E}_{q_{\phi}(\mathbf{z}|\mathbf{x})}[\textnormal{log}\,p_{\theta}(\mathbf{x}|\mathbf{z})]}_{\textnormal{reconstruction
accuracy}}.$ (1)

In practice, the model is trained by maximizing
$\mathcal{L}(\phi,\theta,\mathbf{x})$ over the training dataset with respect
to parameters $\phi$ and $\theta$. We can see that the VLB is the sum of two
terms. The first term acts as a regularizer encouraging the approximate
posterior $q_{\phi}(\mathbf{z}|\mathbf{x})$ to be close to the prior
$p_{\theta}(\mathbf{z})$. The second term represents the average
reconstruction accuracy. Since the expectation w.r.t.
$q_{\phi}(\mathbf{z}|\mathbf{x})$ is difficult to compute analytically, it is
approximated using a Monte Carlo estimate and samples drawn from
$q_{\phi}(\mathbf{z}|\mathbf{x})$. For other technical details that are not
relevant here, the reader is referred to [11].

As discussed in [12] and [25], a weighting factor, denoted $\beta$, can be
introduced in (1) to balance the regularization and reconstruction terms:

$\displaystyle\mathcal{L}(\phi,\theta,\beta,\mathbf{x})$
$\displaystyle=-\beta\,\textnormal{D}_{\textnormal{KL}}(q_{\phi}(\mathbf{z}|\mathbf{x})|p_{\theta}(\mathbf{z}))$
$\displaystyle\qquad\quad+\mathbb{E}_{q_{\phi}(\mathbf{z}|\mathbf{x})}[\textnormal{log}\,p_{\theta}(\mathbf{x}|\mathbf{z})],$
(2)

This enables the user to better control the trade-off between output signal
quality and compactness/orthogonality of the latent coefficients $\mathbf{z}$.
Indeed, if the reconstruction term is too strong relatively to the
regularization term, then the distribution of the latent space will be poorly
constrained by the prior $p_{\theta}(\mathbf{z})$, turning the VAE into an AE.
Conversely, if it is too weak, then the model may focus too much on
constraining the latent coefficients to follow the prior distribution while
providing poor signal reconstruction [25]. In the present work we used this
type of $\beta$-VAE and we present the results obtained with different values
of $\beta$. These latter were selected manually after pilot experiments to
ensure that the values of the regularization and the reconstruction accuracy
terms in (2.2) are in the same range.

##  3 Experiments

###  3.1 Dataset

In this study, we used the NSynth dataset introduced in [9]. This is a large
database (more than 30 GB) of 4s long monophonic music sounds sampled at 16
kHz. They represent $1,\\!006$ different instruments generating notes with
different pitches (from MIDI 21 to 108) and different velocities (5 different
levels from 25 to 127). To generate these samples different methods were used:
Some acoustic and electronic instruments were recorded and some others were
synthesized. The dataset is labeled with: i) instrument family (e.g.,
keyboard, guitar, synth_lead, reed), ii) source (acoustic, electronic or
synthetic), iii) instrument index within the instrument family, iv) pitch
value, and v) velocity value. Some other labels qualitatively describe the
samples, e.g. brightness or distortion, but they were not used in our work.

To train our models, we used a subset of $10,\\!000$ different sounds randomly
chosen from this NSynth database, representing all families of instruments,
different pitches and different velocities. We split this dataset into a
training set (80%) and testing set (20%). During the training phase, 20% of
the training set was kept for validation. In order to have a statistically
robust evaluation, a $k$-fold cross-validation procedure with $k=5$ was used
to train and test all different models (we divided the dataset into $5$ folds,
used $4$ of them for training and the remaining one for test, and repeated
this procedure $5$ times so that each sound of the initial dataset was used
once for testing).

###  3.2 Data Pre-Processing

For magnitude and phase short-term spectra extraction, we applied a
$1,\\!024$-point STFT to the input signal using a sliding Hamming window with
$50$% overlap. Frames corresponding to silence segments were removed. The
corresponding $513$-point positive-frequency magnitude spectra were then
converted to log-scale and normalized in energy: We fixed the maximum of each
log-spectrum input vector to 0 dB (the energy coefficient was stored to be
used for signal reconstruction). Then, the log-spectra were thresholded, i.e.
every log-magnitude below a fixed threshold was set to the threshold value.
Finally they were normalized between $-1$ and $1$, which is a usual procedure
for ANN inputs. Three threshold values were tested: $-80$ dB, $-90$ dB and
$-100$ dB. Corresponding denormalization, log-to-linear conversion and energy
equalization were applied after the decoder, before signal reconstruction with
transmitted phases and inverse STFT with overlap-add.

###  3.3 Autoencoder Implementations

We tried different types of autoencoders: AE, DAE, LSTM-AE and VAE. For all
the models we investigated several values for the encoding dimension, i.e. the
size of the bottleneck layer / latent variable vector, from $enc=4$ to $100$
(with a fine-grained sampling for $enc\leq 16$). Different architectures were
tested for the DAEs: [513, 128, $enc$, 128, 513], [513, 256, $enc$, 256, 513]
and [513, 256, 128, $enc$, 128, 256, 513]. Concerning the LSTM-AE, our
implementation used two vanilla forward LSTM layers (one for the encoder and
one for the decoder) with non-linear activation functions giving the following
architecture: [513, $enc$, 513]. Both LSTM layers were designed for many-to-
many sequence learning, meaning that a sequence of inputs, i.e. of spectral
magnitude vectors, is encoded into a sequence of latent vectors of same
temporal size and then decoded back to a sequence of reconstructed spectral
magnitude vectors. The architecture we used for the VAE was [513, 128, $enc$,
128, 513] and we tested different values of the weight factor $\beta$. For all
the neural models, we tested different pairs of activation functions for the
hidden layers and output layer, respectively: (tanh, linear), (sigmoid,
linear) and (tanh, sigmoid).

AE, DAE, LSTM-AE and VAE models were implemented using the Keras toolkit [26]
(we used the scikit-learn [27] toolkit for the PCA). Training was performed
using the Adam optimizer [28] with a learning rate of $10^{-3}$ over $600$
epochs with early stopping criterion (with a patience of $30$ epochs) and a
batch size of $512$. The DAEs were trained in two different ways, with and
without layer-wise training.

###  3.4 Experimental Results for Analysis-Resynthesis

Figure 3: Reconstruction error (RMSE in dB) obtained with PCA, AE, DAE (with
and without layer-wise training) and LSTM-AE, as a function of latent space
dimension.

Figure 4: Reconstruction error (RMSE in dB) obtained with VAEs as a function
of latent space dimension (RMSE obtained with PCA is also recalled).

Fig. 4 shows the reconstruction error (RMSE in dB) obtained with PCA, AE, DAE
and LSTM-AE models on the test set (averaged over the 5 folds of the cross-
validation procedure), as a function of the dimension of the latent space. The
results obtained with the VAE (using the same protocol, and for different
$\beta$ values) are shown in Fig. 4. For the sake of clarity, we present here
only the results obtained for i) a threshold of $-100$ dB applied on the log-
spectra, and ii) a restricted set of the tested AE, DAE and VAE architectures
(listed in the legends of the figures). Similar trends were observed for other
thresholds and other tested architectures. For each considered dimension of
the latent space, a 95% confidence interval of each reconstruction error was
obtained by conducting paired t-test, considering each sound (i.e. each audio
file) of the test set as an independent sample.

RMSE provides a global measure of magnitude spectra reconstruction but can be
insufficiently correlated to perception depending on which spectral components
are correctly or poorly reconstructed. To address this classical issue in
audio processing, we also calculated objective measures of perceptual audio
quality, namely PEMO-Q scores [18]. The results are reported in Fig. 6 and
Fig. 6.

As expected, the RMSE decreases with the dimension of the latent space for all
methods. Interestingly, PCA systematically outperforms (or at worst equals)
shallow AE. This somehow contradicts recent studies on image compression for
which a better reconstruction is obtained with AE compared to PCA [1]. To
confirm this unexpected result, we replicated our PCA vs. AE experiment on the
MNIST image dataset [29], using the same AE implementation and a standard
image preprocessing (i.e. vectorization of each $28\times 28$ pixels gray-
scale image into a $784$-dimensional feature vector). In accordance with the
literature, the best performance was systematically obtained with AE (for any
considered dimension of the latent space). This difference of AE’s behavior
when considering audio and image data was unexpected and, to our knowledge, it
has never been reported in the literature.

Then, contrary to (shallow) AE, DAEs systematically outperform PCA (and thus
AE), with up to almost 20% improvement (for $enc=12$ and $enc=16$). Our
experiments did not reveal notable benefit of layer-by-layer DAE training over
end-to-end training. Importantly, for small dimensions of the latent space
(e.g. smaller than 16), RMSE obtained with DAE decreases much faster than with
PCA and AE. This is even more the case for LSTM-AE which shows an improvement
of the reconstruction error of more than 23% over PCA (for $enc=12$ and
$enc=16$). These results confirm the benefits of using a more complex
architecture than shallow AE, here deep or recurrent, to efficiently extract
high-level abstractions and compress the audio space. This is of great
interest for sound synthesis for which the latent space has to be kept as low-
dimensional as possible (while maintaining a good reconstruction accuracy) in
order to be “controlled” by a musician.

Fig. 4 shows that the overall performance of VAEs is in between the
performance of DAEs (even equals DAEs for lower encoding dimensions, say
smaller than 12) and the performances of PCA and AE. Let us recall that
minimizing the reconstruction accuracy is not the only goal of VAE which also
aims at constraining the distribution of the latent space. As shown in Fig. 4,
the parameter $\beta$, which balances regularization and reconstruction
accuracy in (2.2), plays a major role. As expected, high $\beta$ values foster
regularization at the expense of reconstruction accuracy. However, with
$\beta\leqslant 2.10^{-6}$ the VAE clearly outperforms PCA, e.g. up to $20\%$
for $enc=12$.

It can be noticed that when the encoding dimension is high ($enc=100$), PCA
seems to outperform all the other models. Hence, in that case, the simpler
(linear model) seems to be the best (we can conjecture that achieving the same
level of performance with autoencoders would require more training data, since
the number of free parameters of these model increases drastically). However,
using such high-dimensional latent space as control parameters of a music
sound generator is impractical.

Similar conclusions can be drawn from Fig. 6 and Fig. 6 in terms of audio
quality. Indeed, in a general manner, the PEMO-Q scores are well correlated
with RMSE measures in our experiments. PEMO-Q measures for PCA and AE are very
close, but PCA still slightly outperforms the shallow AE. The DAEs and the
VAEs both outperform the PCA (up to about $11\%$ for $enc=12$ and $enc=16$)
with the audio quality provided by the DAEs being a little better than for the
VAEs. Surprisingly, and contrary to RMSE scores, the LSTM-AE led to a
(slightly) lower PEMO-Q scores, for all considered latent dimensions. Further
investigations will be done to assess the relevance of such differences at the
perceptual level.

Figure 5: PEMO-Q measures obtained with PCA, AE, DAEs (with and without layer-
wise training) and LSTM-AE, as a function of latent space dimension.

Figure 6: PEMO-Q measures obtained with VAEs as a function of latent space
dimension (measures obtained with PCA are also recalled).

###  3.5 Decorrelation of the Latent Dimensions

Now we report further analyses aiming at investigating how the extracted
latent dimensions may be used as control parameters by the musician. In the
present sound synthesis framework, such control parameters are expected to
respect (at least) the following two constraints i) to be as decorrelated as
possible in order to limit the redundancy in the spectrum encoding, ii) to
have a clear and easy-to-understand perceptual meaning. In the present study,
we focus on the first constraint by comparing PCA, DAEs, LSTM-AE and VAEs in
terms of correlation of the latent dimensions. More specifically, the absolute
values of the correlation coefficient matrices of the latent vector
$\mathbf{z}$ were computed on each sound from the test dataset and Fig. 7
reports the mean values averaged over all the sounds of the test dataset. For
the sake of clarity, we present here these results only for a latent space of
dimension $16$ for one model of DAE ([513, 128, 16, 128, 513] (tanh & lin)
with end-to-end training) and for VAEs with the same architecture ([513, 128,
16, 128, 513] (tanh & lin)) and different values of $\beta$ (from $1.10^{-6}$
to $2.10^{-5}$).

As could be expected from the complexity of its structure, we can see that the
LSTM-AE extracts a latent space where the dimensions are significantly
correlated with each other. Such additional correlations may come from the
sound dynamics which provide redundancy in the prediction. We can also see
that PCA and VAEs present similar behaviors with much less correlation of the
latent dimensions, which is an implicit property of these models.
Interestingly, and in accordance with (2.2), we can notice that the higher the
$\beta$, the more regularized the VAE and hence the more decorrelated the
latent dimensions. Importantly, Fig. 7 clearly shows that for a well-chosen
$\beta$ value, the VAE can both extract latent dimensions that are much less
correlated than for corresponding DAEs, which makes it a better candidate for
extracting good control parameters, while allowing fair to good reconstruction
accuracy (see Fig. 4). The $\beta$ value has thus to be chosen wisely in order
to find the optimal trade-off between decorrelation of the latent dimensions
and reconstruction accuracy.

Figure 7: Correlation matrices of the latent dimensions (average absolute
correlation coefficients) for PCA, DAE, LSTM-AE and VAEs.

###  3.6 Examples of Sound Interpolation

As a first step towards the practical use of the extracted latent space for
navigating through the sound space and creating new sounds, we illustrate how
it can be used to interpolate between sounds, in the spirit of what was done
for instrument hybridization in [9]. We selected a series of pairs of sounds
from the NSynth dataset with the two sounds in a pair having different
characteristics. For each pair, we proceeded to separate encoding, entry-wise
linear interpolation of the two resulting latent vectors, decoding, and
finally individual signal reconstruction with inverse STFT and the Griffin and
Lim algorithm to reconstruct the phase spectrogram [19]. We experimented
different degrees of interpolation between the two sounds: $\hat{\mathbf{z}}=\
\alpha\ \mathbf{z}_{1}+(1-\alpha)\ \mathbf{z}_{2}$, with $\mathbf{z}_{i}$ the
latent vector of sound $i$, $\hat{\mathbf{z}}$ the new interpolated latent
vector, and $\alpha\in[0,0.25,0.5,0.75,1]$ (this interpolation is processed
independently on each pair of vectors of the time sequence). The same process
was applied using the different AE models we introduced earlier.

Fig. 8 displays one example of results obtained with PCA, with the LSTM-AE and
with the VAE (with $\beta=1.10^{-6}$), with an encoding dimension of $32$.
Qualitatively, we note that interpolations in the latent space lead to a
smooth transition between source and target sound. By increasing sequentially
the degree of interpolation, we can clearly go from one sound to another in a
consistent manner, and create interesting hybrid sounds. The results obtained
using PCA interpolation are (again qualitatively) below the quality of the
other models. The example spectrogram obtained with interpolated PCA
coefficients is blurrier around the harmonics and some audible artifacts
appear. On the opposite, the LSTM-AE seems to outperform the other models by
better preserving the note attacks (see comparison with VAE in Fig. 8). More
interpolation examples along with corresponding audio samples can be found at
https://goo.gl/Tvvb9e.

(a) Original samples - Left : bass_electronic_010-055-100, Right :
brass_acoustic_050-055-100

(b) PCA

(c) LSTM-AE

(d) VAE

Figure 8: Examples of decoded magnitude spectrograms after sound interpolation
of 2 samples (top) in the latent space using respectively PCA (2nd row), LSTM-
AE (3rd row) and VAE (bottom). A more detailed version of the figure can be
found at https://goo.gl/Tvvb9e.

##  4 Conclusions and Perspectives

In this study, we investigated dimensionality reduction based on autoencoders
to extract latent dimensions from a large music sound dataset. Our goal is to
provide a musician with a new way to generate sound textures by exploring a
low-dimensional space. From the experiments conducted on a subset of the
publicly available database NSynth, we can draw the following conclusions: i)
Contrary to the literature on image processing, shallow autoencoders (AEs) do
not here outperform principal component analysis (in terms of reconstruction
accuracy); ii) The best performance in terms of signal reconstruction is
always obtained with deep or recurrent autoencoders (DAEs or LSTM-AE); iii)
Variational autoencoders (VAEs) lead to a fair-to-good reconstruction accuracy
while constraining the statistical properties of the latent space, ensuring
some amount of decorrelation across latent coefficients and limiting their
range. These latter properties make the VAEs good candidates for our targeted
sound synthesis application.

In line with the last conclusion, future works will mainly focus on VAEs.
First, we will investigate recurrent architecture for VAE such as the one
proposed in [30]. Such approach may lead to latent dimensions encoding
separately the sound texture and its dynamics, which may be of potential
interest for the musician.

Then, we will address the crucial question of the perceptual meaning/relevance
of the latent dimensions. Indeed using a non-informative prior distribution of
$\mathbf{z}$ such as a standard normal distribution does not ensure that each
dimension of $\mathbf{z}$ represents an interesting perceptual dimension of
the sound space, although this is a desirable objective. In [14], the authors
recently proposed a first solution to this issue in the context of a
restricted set of acoustic instruments. They introduced in the variational
lower bound (2.2) of the VAE loss an additional regularization term
encouraging the latent space to respect the structure of the instrument
timbre. In the same spirit, our future works will investigate different
strategies to model the complex relationships between sound textures and their
perception, and introduce these models at the VAE latent space level.

##  5 Acknowledgment

The authors would like to thank Simon Leglaive for our fruitful discussions.
This work was supported by ANRT in the framework of the PhD program CIFRE
2016/0942.

## References

  * [1] G. Hinton and R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” _Science_ , vol. 313, no. 5786, pp. 504–507, 2006.
  * [2] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Advances in Neural Information Process. Systems_ , Montreal, Canada, 2014\.
  * [3] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee, “Generative adversarial text to image synthesis,” in _Proc. of the Int. Conf. on Machine Learning_ , New York, NY, 2016.
  * [4] S. Mehri, K. Kumar, I. Gulrajani, R. Kumar, S. Jain, J. Sotelo, A. Courville, and Y. Bengio, “SampleRNN: An unconditional end-to-end neural audio generation model.” in _Int. Conf. on Learning Representations_ , 2017.
  * [5] S. Tulyakov, M.-Y. Liu, X. Yang, and J. Kautz, “Mocogan: Decomposing motion and content for video generation,” in _Proc. of the IEEE conf. on computer vision and pattern recognition_ , 2018.
  * [6] E. Miranda, _Computer Sound Design: Synthesis Techniques and Programming_ , ser. Music Technology series. Focal Press, 2002.
  * [7] A. Sarroff and M. Casey, “Musical audio synthesis using autoencoding neural nets,” in _Joint Int. Computer Music Conf. and Sound and Music Computing Conf._ , Athens, Greece, 2014.
  * [8] J. Colonel, C. Curro, and S. Keene, “Improving neural net auto encoders for music synthesis,” in _Audio Engineering Society Convention_ , New-York, NY, 2017.
  * [9] J. Engel, C. Resnick, A. Roberts, S. Dieleman, D. Eck, K. Simonyan, and M. Norouzi, “Neural audio synthesis of musical notes with wavenet autoencoders,” _arXiv preprint arXiv:1704.01279_ , 2017.
  * [10] A. Van Den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,” _arXiv preprint arXiv:1609.03499_ , 2016.
  * [11] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv preprint arXiv:1312.6114_ , 2013.
  * [12] M. Blaauw and J. Bonada, “Modeling and transforming speech using variational autoencoders.” in _Conf. of the Int. Speech Comm. Association (Interspeech)_ , San Francisco, CA, 2016.
  * [13] W.-N. Hsu, Y. Zhang, and J. Glass, “Learning latent representations for speech generation and transformation,” _arXiv preprint arXiv:1704.04222_ , 2017\.
  * [14] P. Esling, A. Chemla-Romeu-Santos, and A. Bitton, “Generative timbre spaces with variational audio synthesis,” in _Proc. of the Int. Conf. on Digital Audio Effects 2018_ , Aveiro, Portugal, 2018.
  * [15] S. Leglaive, L. Girin, and R. Horaud, “A variance modeling framework based on variational autoencoders for speech enhancement,” in _IEEE Int. Workshop on Machine Learning for Signal Process._ , Aalborg, Denmark, 2018.
  * [16] ——, “Semi-supervised multichannel speech enhancement with variational autoencoders and non-negative matrix factorization,” in _IEEE Int. Conf. on Acoustics, Speech and Signal Process._ , Brighton, UK, 2019.
  * [17] L. Li, H. Kameoka, and S. Makino, “Fast MVAE: Joint separation and classification of mixed sources based on multichannel variational autoencoder with auxiliary classifier,” in _IEEE Int. Conf. on Acoustics, Speech and Signal Process._ , Brighton, UK, 2019.
  * [18] R. Huber and B. Kollmeier, “PEMO-Q: A new method for objective audio quality assessment using a model of auditory perception,” _IEEE Transactions on Audio, Speech, and Language Process._ , vol. 14, no. 6, pp. 1902–1911, 2006.
  * [19] D. Griffin and J. Lim, “Signal estimation from modified short-time fourier transform,” _IEEE Transactions on Acoustics, Speech, and Signal Process._ , vol. 32, no. 2, pp. 236–243, 1984.
  * [20] C. M. Bishop, _Pattern recognition and machine learning_. Springer, 2006.
  * [21] I. Goodfellow, Y. Bengio, and A. Courville, _Deep Learning_. MIT Press, 2016.
  * [22] Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, “Greedy layer-wise training of deep networks,” in _Advances in Neural Information Process. Systems_ , Vancouver, Canada, 2007.
  * [23] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [24] Y. Bengio, P. Simard, and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” _IEEE transactions on neural networks_ , vol. 5, no. 2, pp. 157–166, 1994.
  * [25] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner, “$\beta$-VAE: learning basic visual concepts with a constrained variational framework.” in _Int. Conf. on Learning Representations_ , 2017.
  * [26] F. Chollet _et al._ , “Keras,” https://keras.io, 2015.
  * [27] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” _Journal of machine learning research_ , vol. 12, no. Oct, pp. 2825–2830, 2011.
  * [28] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [29] Y. LeCun, C. Cortes, and C. Burges, “The MNIST database of handwritten digits,” http://yann.lecun.com/exdb/mnist/, 1998.
  * [30] J. Chung, K. Kastner, L. Dinh, K. Goel, A. C. Courville, and Y. Bengio, “A recurrent latent variable model for sequential data,” in _Advances in Neural Information Process. Systems_ , 2015.


# Anomalous Ising freezing times

J. Denholm and B. Hourahine  SUPA and Department of Physics, University of
Strathclyde, 107 Rottenrow East, Glasgow, G4 ONG<EMAIL_ADDRESS>
###### Abstract

We measure the relaxation time of a square lattice Ising ferromagnet that is
quenched to zero-temperature from supercritical initial conditions. We reveal
an anomalous and seemingly overlooked timescale associated with the relaxation
to “frozen” two-stripe states. While close to a power law of the form $\sim
L^{\nu}$, we argue this timescale actually grows as $L^{2}\ln{L}$, with $L$
the linear dimension of the system. We uncover the mechanism behind this
scaling form by using a synthetic initial condition that replicates the late
time ordering of two-stripe states, and subsequently explain it heuristically.

  *

##  1 Introduction

The zero-temperature coarsening of the 2D Ising ferromagnet was thought fully
understood through the dynamical scaling hypothesis [1]. From a symmetric and
unmagnetised initial state, magnetic domains nucleate and grow in length as
the square root of time, before ultimately engulfing the system [1,
2]—however, the dynamics is richer still.

Surprisingly, the ground state is not always reached. Instead, 34% of
realisations “freeze” into two-stripe states, which span in only a single
lattice dimension and are forever trapped at constant energy [3, 4].
Additionally, the ground state is only reached on a timescale of
$\mathcal{O}(L^{2})$ around $62\%$ of the time, with $L$ the linear dimension
of the system [3, 4]. Perhaps most interestingly, $4\%$ of the realisations
reach “diagonal” winding configurations, which dramatically forestall the
relaxation process by slowly decaying to homogeneity on a timescale of
$\mathcal{O}(L^{3.5})$ [3, 4].

The probability of observing each of these topologically distinct behaviours
is seemingly identical to the equivalent crossing probability in continuum
percolation, which correspondingly explains how the “fate” of the system is
sealed [5, 6, 7, 8, 9, 10]. The role of critical percolation in 2D curvature
driven coarsening has been studied extensively [11, 12, 13, 14, 15, 16, 17,
18, 19, 20].

In one spatial dimension, the ground state is always reached [21], and in
three-dimensions, the ground state is essentially never reached [22, 23, 24,
25]. The final states in three dimensions are a host of non-static
topologically complex configurations that remain at iso-energy _ad infinitum_
[24, 25].

In this work, we uncover an apparently overlooked timescale associated with
the relaxation to “frozen” two-stripe states. Our main result is that this
anomalous timescale appears to grow as $\sim L^{2}\ln L$—i.e. the dynamics is
slower than the standard coarsening time of $\sim L^{2}$. We detail the model
and method in Sec. 2, and introduce the new timescale in Sec. 3. In Sec. 4 we
analyse this new scaling form and explain its origin.

##  2 Zero-temperature Ising Model

We consider nearest-neighbour interactions between magnetic spins on
periodically bounded square lattices of length $L$, which is equivalent to
wrapping the square lattice around a torus. The spins are denoted by
$S_{i}=\pm 1$ and may point up or down respectively. The total energy of the
system is given by the Hamiltonian

$H=-J\sum\limits_{i,j}S_{i}S_{j},$ (1)

where ${J}>0$ is a ferromagnetic coupling constant controlling the strength of
the nearest-neighbour interactions, $i$ indexes each spin in the system and
$j$ the nearest-neighbours of each $S_{i}$.

Operationally we use single spin-flip zero-temperature Glauber dynamics:
energy lowering and conserving flips are accepted with probabilities of $1$
and $0.5$ respectively, while energy raising flips are forbidden [26, 21]. To
mimic infinite temperature initial conditions, we initialise each realisation
of the dynamics by randomly ordering a microstate of zero-magnetisation.

We implement the dynamics using a standard continuous-time rejection-free
Monte Carlo method [27, 28, 29, 30]. Each site has the chance to flip once, on
average, in a single Monte Carlo time step. Let the probability that a given
site $S_{i}$ should flip be $p_{i}$. The total probability in the system is
$P=\sum p_{i}$. To flip a spin, one selects a site with probability $p_{i}/P$
and reverses its orientation [27, 28, 29, 30]. Time then advances by $\Delta
t=-\ln r/P$, where $r\in(0,1)$ is a uniform random number and $\left<-\ln
r\right>=1$. This process is repeated until no flippable spins remain. The
advantage of this method is that one only visits flippable spins, thus
avoiding the simulation of null processes. Intuitively, $\Delta t$ is actually
the time between accepted spin flip events in the standard rejection-based
Monte Carlo method [27, 28, 29, 30].

The unique classes of spanning clusters that arise in the zero-temperature
Ising model are distinguished by the winding numbers of continuum percolation.
A cluster of winding number $(a,b)$ winds $a$ times horizontally (toroidally)
and $b$ times vertically (poloidally)—as shown in Fig. 1. In Fig. 1 (a), the
configuration exhibits the ground state topology, meaning it necessarily
collapses to the ground state. In Fig. 1 (b), the configuration is that of an
infinitely long-lived frozen two-stripe state. In Fig 1 (c)–(d), we see
unstable diagonal winding configurations that ultimately reach the ground
state.

Clusters of winding numbers $(\pm a,\pm b)$ are equivalent both physically and
in probability, so we set $a,b\geq 0$. This is also true of $(1,0)$ and
$(0,1)$ windings, so we denote all on-axis stripe states as $(1,0)$.

Figure 1: (a) Ground state topology: a _single_ domain winds in _both_ lattice
dimensions. (b) On-axis stripe topology: _both_ domains wind in a _single_
lattice dimension. (c) Diagonal stripe topology: _both_ domains wind in _both_
lattice directions. (d) Diagonal stripe topology: _both_ domains wind _once_
vertically and _twice_ horizontally. (c) and (d) are both unstable and
ultimately collapse to the ground state.

##  3 Overview of timescales

The extinction and survival probabilities are useful diagnostics for
highlighting the multiple timescales at play, and are defined as follows. Let
the extinction time be the time for a given realisation to reach its final
state. The distribution of the extinction time is the extinction probability,
$E(t)$, which upon integration, gives the probability of reaching a final
state within some time interval $\Delta t$.

The survival probability is simply the likelihood that the system is still
active at time $t$. Trivially, the survival probability is

$S(t)=1-\int\limits_{0}^{t}E(t)dt.$ (2)

The cumulative of $E(t)$ gives the fraction of realisations that have reached
their final state by time $t$, and the remaining fraction, or the survival
probability, is simply one minus the cumulative. Note, Eqn. 2 assumes $E(t)$
is a normalised probability density function.

For each of the distinct timescales, there is an associated relaxation time
$\tau(L)$ [3, 6]. Realisations reaching their final state throughout this time
cause the survival probability to undergo an exponential decay of the form
$\sim\exp(-t/\tau(L))$ [3, 6]. The two known timescales in the zero-
temperature Ising model should cause two exponential decay regimes in $S(t)$,
with decay constants $\tau(L)\sim L^{2}$ and $\tau(L)\sim L^{3.5}$ [3, 6]. On
a semi-logarithmic scale, these appear as linear decays with slopes of
$-1/\tau(L)$.

Consider now the survival probability in Fig. 2; although we expected two
decay regimes, there are actually _three_. Firstly, we see the standard
ground-state coarsening timescale of $\mathcal{O}(L^{2})$ (Fig. 2 (a)).
Secondly, we see an additional timescale that grows as $\sim L^{2}\ln L$,
which is actually associated with realisations reaching frozen two-stripe
states (Fig. 2 (a)). Finally, we see the large timescale of
$\mathcal{O}(L^{3.5})$ discovered by Spirin _et al._ [3], which causes the
slow decay in Fig. 2 (b). If one examines the survival probabilities presented
in Refs. [3, 6], there are subtle hints of this intermediate timescale, but
its influence on $S(t)$ is almost imperceptible in the small system sizes
used.

Figure 2: (a) Survival probability versus time for $0\leq t\leq 3\times
10^{5}$. (b) Slow decaying tail of $S(t)$ corresponding to the relaxation of
diagonal stripe states. The data are based on $10^{4}$ realisations with
$L=512$.

###  3.1 Survival analysis

With these three decay regimes in mind, we now examine the survival
probability as a function of $t/L^{2}$ for specified $L$ in Fig. 3. In the
first decay regime—which is caused by the ground-state timescale of order
$L^{2}$—we see data collapse in $S(t)$ (see Fig. 3 (a)). However, in the
second decay regime—which is caused by the anomalous timescale of order
$L^{2}\ln L$—the data collapse fails (see Fig. 3 (a)). Finally, we see the
long tails caused by the slow relaxation of diagonal-stripe states in Fig. 3
(b).

Figure 3: Survival probability versus $t/L^{2}$ for (a) $0\leq t/L^{2}\leq 1$
and (b) $0\leq t/L^{2}\leq 400$ from systems of length $L$. The data are based
on $10^{4}$ realisations.

Since we know the form of the survival probability, we can actually recover
the slopes quoted in Fig. 2 by fitting straight lines to $\ln S(t)$ versus $t$
over each region and computing how the slope scales with $L$. Distinguishing
between the $L^{2}$ and $L^{2}\ln L$ timescales is difficult due to their
overlap—particularly at small $L$—so we apply the analysis to the $L^{2}$ and
$L^{3.5}$ timescales only. We plot these estimates in Fig. 4.

Figure 4: Estimates of the slopes in $\ln S(t)$ versus $t$ over (a) the
ground-state decay and (b) the diagonal-stripe state decay versus $L$.

It is important to note that these estimates are necessarily crude because one
must first judge over which region to apply the fit.

###  3.2 Synthetic initial conditions

Figure 5: (a) Ground state freezing time from the cross initial condition in
Fig. 1 (a) with a power law fit. The data are based on $10^{4}$ realisations.

Before we turn our focus to the anomalous timescale, we recover the known
timescales from our simulations using the synthetic initial conditions shown
in Fig. 1. These conveniently allow us to select the topology of the evolution
we wish to study. We begin with the ground state freezing time, which grows
with the system size as $L^{2}$. We demonstrate this using the cross initial
condition in Fig. 1 (a)—the result of which we plot in Fig. 5. As $L$
increases, the data exhibit a subtle downward curvature that is indicative of
finite size effects.

Spirin _et al._ found the relaxation time of the diagonal stripe states to
scale as roughly $\sim L^{3.5}$ [3]. We confirm this by realising the dynamics
from the $(1,1)$ and $(2,1)$ initial conditions shown in Fig. 1 (c)–(d). We
plot the resulting times in Fig. 6, and obtain power law fits of exponents
$\nu=3.62$ and $\nu=3.58$, as expected. In both cases, there is a non-
monotonic curvature in the data that suggests these scaling estimates are pre-
asymptotic.

Figure 6: Diagonal stripe state relaxation times versus $L$ from the (a)
$(1,1)$ and (b) $(2,1)$ initial conditions shown in Fig. 1 (c)–(d). The data
are based on $10^{4}$ realisations.

##  4 Anomalous timescale

Here, we show that the relaxation time of frozen two-stripe states grows as
$\sim L^{2}\ln L$. Our argument is based on three key features: (i) The
scaling of the relaxation times using supercritical initial conditions; (ii)
the scaling using a synthetic “wedding cake” initial condition; (iii) an
argument based on annihilating random walkers.

###  4.1 Supercritical initial conditions

Figure 7: Data collapse in extinction probability for two-stripe states. The
distributions are based on two two-stripe state times from ensembles of
$5\times 10^{4}$ realisations.

We investigate the time scaling of the two-stripe states by realising the
dynamics in systems of $16\leq L\leq 1024$ from random initial conditions,
storing only the subset of times from instances that reach frozen two-stripe
states. To test if this relaxation is governed by a single timescale, we
consider the distribution of the two-stripe freezing times (see Fig. 7). In
each case, time is rescaled by the mean of the distribution. The data collapse
in Fig. 7 indicates the relaxation is governed by a _single_ timescale.

We plot the mean two-stripe state freezing time $T_{S}$ in Fig. 8 (a) as a
function of $L$ and obtain a power law fit (solid line) of exponent
$\nu=2.13$. Clearly, $T_{S}$ grows faster than $\sim L^{2}$, but the form is
not necessarily a simple power law; the near, but greater than integer
exponent, along with subtle downward curvature in the data, is evidence of a
logarithmic factor.

Figure 8: (a) Two-stripe state freezing time $T_{S}$ versus system size $L$.
(b) $T_{s}$ versus $L^{2}\ln L$. (c) $T_{S}/L^{2}$ versus $L$ (semi-
logarithmic scale). The data are based on the two-stripe states times from
ensembles of $5\times 10^{4}$ realisations.

There are two strong pieces of evidence for $L^{2}\ln L$ scaling. The first is
shown in Fig. 8 (b), where we plot $T_{S}$ versus $L^{2}\ln L$—the result of
which is linear. A power law fit to these data gives an exponent of
$\nu=0.96$, which is strikingly close to linear considering the presence of
finite size effects. The second piece of evidence is the recovery of the
logarithm in $T_{S}$: in Fig. 8 (c) we plot $T_{S}/L^{2}$ versus $\ln L$ on a
semi-logarithmic scale, and find that even though there is curvature due to
finite size effects, there is still discernible evidence of a logarithm. The
recovery of $\ln L$ from $T_{S}$ is an important piece of evidence because it
is the weakest part of the scaling form, and is discernible even in the
presence of finite size effects.

Frozen two-stripe states only occur with probability $\approx 0.34$, so to
study them one essentially discards $66\%$ of their simulations. To reduce the
role of finite size effects in our analysis and increase the number of
realisations used in our scaling estimates, we seek to study this process more
efficiently.

###  4.2 Synthetic initial condition

To further explore the two-stripe state relaxation, we use the synthetic
“wedding cake” initial condition (Fig. 9).

Figure 9: Maximally tiered “wedding cake” initial condition.

Realising the dynamics from this configuration is more practical as the
required CPU time is lesser, and 100% of the realisations are used. We can
therefore obtain $10^{5}$ realisations with $16\leq L\leq 2048$ using the
wedding cake initial condition.

Figure 10: (a) Wedding cake freezing time $T_{\mathrm{wc}}$ versus system size
$L$. (b) $T_{\mathrm{wc}}$ versus $L^{2}\ln L$. (c) $T_{\mathrm{wc}}/L^{2}$
versus $L$ (semi-logarithm scale). The data are based on $10^{5}$ realisations
from the wedding cake initial condition in Fig. 9.

We perform the same analysis as in Sec. 4.1 to show the wedding cake
relaxation scales as $L^{2}\ln L$. Firstly, we plot the wedding cake freezing
time $T_{\mathrm{wc}}$ versus $L$ in Fig. 10 (a) and obtain a power law fit of
exponent $\nu=2.14$. We again see subtle downward curvature in the data, and
the exponent is congruent with that of Fig. 8 (a). Secondly, we plot
$T_{\mathrm{wc}}$ versus $L^{2}\ln L$ in Fig. 10 (b) and obtain a power law
fit of exponent $0.98$, thus showing $T_{\mathrm{wc}}$ is linear with
$L^{2}\ln L$. Thirdly, we recover the logarithm in $T_{\mathrm{wc}}$ by
dividing out $L^{2}$ in Fig. 10 (c). On the semi-logarithmic scale,
$T_{\mathrm{wc}}/L^{2}$ is strikingly linear—especially at large $L$—given
that the form is still influenced by finite size effects.

Our analysis using the wedding cake configuration is cleaner compared to the
case of supercritical initial conditions as we obtain a greater number of
realisations per system size and are able to simulate larger systems.

###  4.3 Heuristic argument

Our simulations show $L^{2}\ln L$ scaling, so we now seek a theoretical
argument to solidify this claim. Here, we use the wedding cake as a simplistic
model to probe the origin of the anomalous timescale.

First, we cartoon the relaxation of the wedding cake. In the initial
condition, each tier has two misaligned spin pairs forming domain-wall
particles that behave as random walkers (see Fig. 11).

Figure 11: Random walker picture of the “wedding cake” initial condition. The
red circles cartoon misaligned spin pairs (domain-wall particles) as random
walkers that may hop left or right with equal probability, and are reflected
by the edges of their tiers.

The walkers may hop to the left or right with equal probability, and and
reflected by the boundaries of their tiers. When two walkers on the same tier
meet: they annihilate—at which point the tier has relaxed and the number of
active sites has reduced by four. The width of tier $k$ is $L_{k}=L-2(k-1)$,
and the height of the wedding cake is $h=L/2-1$ (see Fig. 11).

The walkers on tier $k=5$ are pinned between the positions of the walkers on
tiers $k=4$ and $k=6$, so they cannot meet. However, the walkers on the
topmost tier are unconstrained and therefore free to meet, so the entire
structure relaxes “top-down”.

Each of the tiers in the wedding cake can be modelled as annihilating random
walkers on intervals of length $L_{k}$. As the system relaxes from the top
down, we assume the relaxation of the topmost tier occurs independently of the
other tiers in the system. This is however a quasi-static approximation,
because the location of the left- and right-hand boundaries of the topmost
tier depends upon the positions of the walkers on the second top-most tier,
which do fluctuate.

When a spin is flipped, the corresponding walker hops to the left or right.
The _number of hops_ required for two random walkers on an interval of length
$L_{k}$ to collide is of order $L_{k}^{2}$, ergo the total number of hops
required for the entire structure to relax is

$N_{\mathrm{H}}\sim\sum_{k=1}^{k=h}L_{k}^{2}$ (3)

If our argument is to accurately describe the wedding cake relaxation, we
should be able to count the number of spin flips in our simulations and find
the same scaling as Eqn. 3. We show these quantities in Fig. 12, where one
sees the forms exhibit equivalent scaling. Eqn. 3 only gives the order of the
number hops, which explains why the forms in Figure 12 have the same slopes
but different magnitudes.

Figure 12: Number of flips in the wedding cake relaxation versus $L$ (circles)
and plot of Eqn. 3 (line). The data are based on $10^{4}$ realisations.

The congruence between the scaling of the number of hops required for the
system to relax and Eqn. 3 in Fig. 12 validates our treatment of each tier as
independent random walkers.

Now that we have established the number of flips required for the structure to
relax, we seek to relate this quantity to simulation time, which has units of
Monte Carlo time steps. Each spin in the system undergoes one microtrial per
time step; that is to say, each site has the chance to flip once, on average,
in a single time step. A successful microtrial is one in which the spin flip
is accepted. As time progresses during a zero-temperature quench, the number
of flippable spins decreases, so fewer and fewer spins flip in a given time
step. Consequently, the time between successful microtrials—that is, the time
between successful spin flips—increases. The mean _time between_ successful
microtrials is $\Delta t=1/P$ [27, 28, 29, 30]. $\Delta t$ is inversely
proportional to the number of flippable spins in the system, which in the
wedding cake is $4k$, so we denote the time increment as $\Delta t_{k}$ [27,
28, 29, 30].

We know the number of flips required for the tiers to relax, and we have an
expression for the time between individual flips, so we can write the total
relaxation time as

$t=\sum_{k=1}^{k=h}L_{k}^{2}\Delta t_{k},$ (4)

i.e. the relaxation time is the sum of the total number of hops (successful
flips) multiplied by the time step between each hop. As we use zero-
temperature Glauber dynamics, the probability that a given _active site_ in
the wedding cake configuration should flip—which is an energy conserving
move—is $0.5$, except in the very rare exception where two walkers collide,
which is energy lowering. Thus, $\Delta t_{k}=1/(0.5\times 4k)$, and Eqn. 4
becomes

$\displaystyle t$ $\displaystyle\sim$
$\displaystyle\frac{1}{2}\sum_{k=1}^{k=h}\frac{1}{k}L_{k}^{2},$
$\displaystyle\sim$
$\displaystyle\frac{1}{2}\sum_{k=1}^{k=h}\frac{L^{2}}{k}+\frac{4L+4}{k}-4L+4k-8.$

This form is dominated by the first term in the sum, which asymptotically is

$t\simeq L^{2}\ln L.$ (6)

Although this argument _is_ crude, in that it does not completely encapsulate
the relaxation process, it is a compelling justification for $L^{2}\ln L$
scaling.

##  5 Discussion & Conclusion

We identified a new timescale in the zero-temperature coarsening of the square
lattice Ising ferromagnet that grew as $L^{2}\ln L$ and arose from the
relaxation to “frozen” two-stripe states. Our argument for $L^{2}\ln L$
scaling was based on three key features: (i) the time scaling using
supercritical initial conditions; (ii) the scaling using a synthetic “wedding
cake” initial condition; (iii) an argument based on annihilating random
walkers.

Using both natural and synthetic initial conditions, we showed that the
relaxation times were linear with $L^{2}\ln L$, and by dividing out $L^{2}$ we
were able to recover the logarithm. The latter is an important piece of
evidence as the logarithm is the most subtle part of the form, and after
dividing out $L^{2}$ it was clearly discernible—even in the presence of finite
size effects.

We also recovered the known timescales of $\sim L^{2}$ and $\sim L^{3.5}$ as a
test of our simulations. Finite size effects hindered our estimates of the
exponents in each case; even the simple ground state relaxation, finite size
influences caused the _correct_ scaling form to yield a _poor fit_ at small
$L$.

This overlooked timescale is seemingly the same as the relaxation-time of so-
called three-hexagon states in the triangular lattice Potts model, and is
likely a general feature of edge ordering in kinetic spin systems [31]. One
question raised by this work is that of the relaxation timescales in the zero-
temperature Potts model, which are markedly different on the square and
triangular lattices [32, 31]. The relaxation timescales in the square-lattice
Potts model are non-trivial and not yet well understood [32].

## Acknowledgements

J.D. thanks Sid Redner for invaluable advice and direction throughout this
work. The authors thank Leticia Cugliandolo for providing insightful
criticisms and suggestions regarding this manuscript, and Gian-Luca Oppo for
helpful comments. J.D. acknowledges EPSRC DTA5 grant EP/N509760/1 for
financial support. The authors acknowledge ARCHIE-WeSt High Performance
Computer based at the University of Strathclyde as well as grant EP/P015719/1
for computer resources.

## References

  * [1] A. Bray, “Theory of phase ordering kinetics,” Physica A, vol. 194, no. 1, pp. 41 – 52, 1993.
  * [2] K. Humayun and A. J. Bray, “Non-equilibrium dynamics of the ising model for $T\leq T_{c}$,” Journal of Physics A: Mathematical and General, vol. 24, pp. 1915–1930, apr 1991.
  * [3] V. Spirin, P. L. Krapivsky, and S. Redner, “Fate of zero-temperature Ising ferromagnets,” Phys. Rev. E, vol. 63, p. 036118, Feb 2001.
  * [4] V. Spirin, P. L. Krapivsky, and S. Redner, “Freezing in Ising ferromagnets,” Phys. Rev. E, vol. 65, p. 016119, Dec 2001.
  * [5] K. Barros, P. L. Krapivsky, and S. Redner, “Freezing into stripe states in two-dimensional ferromagnets and crossing probabilities in critical percolation,” Phys. Rev. E, vol. 80, p. 040101(R), Oct 2009.
  * [6] J. Olejarz, P. L. Krapivsky, and S. Redner, “Fate of 2D kinetic ferromagnets and critical percolation crossing probabilities,” Phys. Rev. Lett., vol. 109, p. 195702, Nov 2012.
  * [7] J. L. Cardy, “Critical percolation in finite geometries,” J. Phys. A, vol. 25, pp. L201–L206, feb 1992.
  * [8] H. T. Pinson, “Critical percolation on the torus,” J. Stat. Phys, vol. 75, pp. 1167–1177, Jun 1994.
  * [9] G. Pruessner and N. R. Moloney, “Winding clusters in percolation on the torus and the möbius strip,” J. Stat. Phys., vol. 115, pp. 839–853, May 2004\.
  * [10] R. P. Langlands, C. Pichet, P. Pouliot, and Y. Saint-Aubin, “On the universality of crossing probabilities in two-dimensional percolation,” J. Stat. Phys, vol. 67, pp. 553–574, May 1992.
  * [11] J. J. Arenzon, A. J. Bray, L. F. Cugliandolo, and A. Sicilia, “Exact results for curvature-driven coarsening in two dimensions,” Phys. Rev. Lett., vol. 98, p. 145701, Apr 2007.
  * [12] T. Blanchard and M. Picco, “Frozen into stripes: Fate of the critical Ising model after a quench,” Phys. Rev. E, vol. 88, p. 032131, Sep 2013.
  * [13] T. Blanchard, F. Corberi, L. F. Cugliandolo, and M. Picco, “How soon after a zero-temperature quench is the fate of the Ising model sealed?,” EPL, vol. 106, p. 66001, jun 2014.
  * [14] L. F. Cugliandolo, “Critical percolation in bidimensional coarsening,” J. Stat. Mech, vol. 2016, p. 114001, nov 2016.
  * [15] T. Blanchard, L. F. Cugliandolo, M. Picco, and A. Tartaglia, “Critical percolation in the dynamics of the 2D ferromagnetic Ising model,” J. Stat. Mech, vol. 2017, p. 113201, nov 2017.
  * [16] F. Corberi, L. F. Cugliandolo, F. Insalata, and M. Picco, “Coarsening and percolation in a disordered ferromagnet,” Phys. Rev. E, vol. 95, p. 022101, Feb 2017.
  * [17] P. Mullick and P. Sen, “Zero-temperature coarsening in the Ising model with asymmetric second-neighbor interactions in two dimensions,” Phys. Rev. E, vol. 95, p. 052150, May 2017.
  * [18] C. Godrèche and M. Pleimling, “Freezing in stripe states for kinetic Ising models: a comparative study of three dynamics,” J. Stat. Mech, vol. 2018, p. 043209, apr 2018.
  * [19] H. Ricateau, L. F. Cugliandolo, and M. Picco, “Critical percolation in the slow cooling of the bi-dimensional ferromagnetic Ising model,” J. Stat. Mech, vol. 2018, p. 013201, jan 2018.
  * [20] A. Tartaglia, L. F. Cugliandolo, and M. Picco, “Coarsening and percolation in the kinetic 2D Ising model with spin exchange updates and the voter model,” Journal of Statistical Mechanics: Theory and Experiment, vol. 2018, p. 083202, aug 2018.
  * [21] P. L. Krapivsky, S. Redner, and E. Ben-Naim, A Kinetic View of Statistical Physics.  Cambridge University Press, 2010.
  * [22] C. M. Newman and D. L. Stein, “Blocking and persistence in the zero-temperature dynamics of homogeneous and disordered Ising models,” Phys. Rev. Lett., vol. 82, pp. 3944–3947, May 1999.
  * [23] C. Newman and D. Stein, “Zero-temperature dynamics of Ising spin systems following a deep quench: results and open problems,” Physica A: Statistical Mechanics and its Applications, vol. 279, no. 1, pp. 159 – 168, 2000\.
  * [24] J. Olejarz, P. L. Krapivsky, and S. Redner, “Zero-temperature freezing in the three-dimensional kinetic Ising model,” Phys. Rev. E, vol. 83, p. 030104(R), Mar 2011.
  * [25] J. Olejarz, P. L. Krapivsky, and S. Redner, “Zero-temperature relaxation of three-dimensional Ising ferromagnets,” Phys. Rev. E, vol. 83, p. 051104, May 2011.
  * [26] R. J. Glauber, “Time‐dependent statistics of the Ising model,” Journal of Mathematical Physics, vol. 4, no. 2, pp. 294–307, 1963.
  * [27] A. Bortz, M. Kalos, and J. Lebowitz, “A new algorithm for monte carlo simulation of Ising spin systems,” Journal of Computational Physics, vol. 17, no. 1, pp. 10 – 18, 1975.
  * [28] P. S. Sahni, D. J. Srolovitz, G. S. Grest, M. P. Anderson, and S. A. Safran, “Kinetics of ordering in two dimensions. II. Quenched systems,” Phys. Rev. B, vol. 28, pp. 2705–2716, Sep 1983.
  * [29] G. N. Hassold and E. A. Holm, “A fast serial algorithm for the finite temperature quenched Potts model,” Computers in Physics, vol. 7, no. 1, pp. 97–107, 1993.
  * [30] P. Landau and K. Binder, A guide to Monte–Carlo Simulations in Statistical Physics.  Cambridge University Press, 2009.
  * [31] J. Denholm and S. Redner, “Topology-controlled Potts coarsening,” Phys. Rev. E, vol. 99, p. 062142, Jun 2019.
  * [32] J. Olejarz, P. L. Krapivsky, and S. Redner, “Zero-temperature coarsening in the 2d Potts model,” vol. 2013, p. P06018, jun 2013.


SichenJin YoungmoonJung SeungjinLee JaeyoungRoh ChangwooHan HoonyoungCho

# CTC-aligned Audio-Text Embedding for Streaming Open-vocabulary Keyword
Spotting

###### Abstract

This paper introduces a novel approach for streaming open-vocabulary keyword
spotting (KWS) with text-based keyword enrollment. For every input frame, the
proposed method finds the optimal alignment ending at the frame using
connectionist temporal classification (CTC) and aggregates the frame-level
acoustic embedding (AE) to obtain higher-level (i.e., character, word, or
phrase) AE that aligns with the text embedding (TE) of the target keyword
text. After that, we calculate the similarity of the aggregated AE and the TE.
To the best of our knowledge, this is the first attempt to dynamically align
the audio and the keyword text on-the-fly to attain the joint audio-text
embedding for KWS. Despite operating in a streaming fashion, our approach
achieves competitive performance on the LibriPhrase dataset compared to the
non-streaming methods with a mere 155K model parameters and a decoding
algorithm with time complexity $O(U)$, where $U$ is the length of the target
keyword at inference time.

###### keywords:

streaming keyword spotting, open-vocabulary, audio-text embedding, CTC.

##  1 Introduction

Keyword Spotting (KWS) plays an important role in voice assistants by
detecting wake-up words and enabling hands-free activation. Over the years,
KWS has developed from fixed vocabulary [1, 2, 3], where the models are
trained with large amount of audio data of the same keywords, to open-
vocabulary systems where users are allowed to use customized keywords by going
through the enrollment process without re-training the models. Open-vocabulary
KWS can be challenging since it requires the system to be flexible and robust
to unseen keywords.

There are two possible ways of enrollment for customized keywords. One is
named Query-by-Example (QbyE) [4, 5, 6], where several example keyword
utterances are provided. The similarities between the input queries and the
enrolled examples are detected by aligning the frame-level acoustic embedding
(AE) using Dynamic Time warping (DTW) [6] or attention [4]. These methods work
well in controlled environments. However, their reliance solely on vocal
features makes them susceptible to reduced performance under varying
conditions and across diverse user voices. On the other hand, the enrollment
process of speaking the same keyword several times can feel inconvenient and
unnatural.

Another way of enrollment can be as simple as inserting keywords in text form.
This intuitive method can significantly enhance the user interface for its
simplicity. There has been several approaches to detect the spoken keywords by
mapping both text and speech to a shared latent encoding space. To handle the
difference in the lengths of the keywords and their spoken form, attention [7,
8] and alignment methods based on dynamic programming [9, 10] were explored to
match the corresponding text and audio frames sharing the same syntax. While
these methods are showing promising results, they often need computations on
the global context, making them hard to run in a streaming manner.

As the front-line user interface of voice assistants, the computations for KWS
need to be very small and use least possible information from the past in
order for them to be active all the time. Convolutional neural networks (CNN)
[11, 12] are often used in small-footprint KWS systems [2, 3] to yield better
performance with smaller models. Connectionist temporal classification (CTC)
[13, 14] is employed to get phoneme hypotheses for each audio frame for
streaming KWS. Building upon these works, it occurred to us that integrating
embedding-based methods could elevate small-footprint online KWS to a new
level.

In light of the requirements and the difficulties, we propose a novel
structure combining the advantages of CTC and embedding approaches: CTC for
instant information retrieval from each audio frame and embedding for
comparing the global information of the entire keyword. With CTC-aligned
Audio-Text (CTCAT) keyword detector, we provide an end-to-end training
strategy to learn the CTC alignment and the joint embedding space
simultaneously, together with an inference algorithm with time complexity
$O(U)$ where $U$ is the keyword length. Experimental results on the
LibriPhrase dataset show that the proposed KWS system achieves performance
comparable to that of its non-streaming counterparts, despite having a
significantly smaller model size (i.e., an acoustic encoder with just 155K
parameters) and faster inference times.

Figure 1: The depiction of the entire structure of the CTC-aligned Audio-Text
(CTCAT) keyword detector. The acoustic encoder and the text encoder encodes
the input audio sequence and the target keyword tokens into acoustic embedding
(AE) and text embedding (TE), and the CTC aligner aligns the embedding vectors
that share the same tokens.

##  2 Related Work

In this work, we attempt to merge the strengths of streaming and non-streaming
KWS techniques, introducing an innovative approach that leverages both the
frame-wise information using CTC and the global context using multi-view loss.
We explain these two methodologies in the following section.

###  2.1 CTC

CTC [13] is among the earliest attempts in ASR to automatically learn the
alignment between the speech frames and the transcript without frame-wise
labels. To deal with the length difference between the transcript and the
spoken utterance, a new blank token is added to the original vocabulary,
making $V^{\prime}=V\cup\\{Blank\\}$. At each time step $t$, the output of the
neural network $y_{k}^{t}$ is interpreted as the probability observing label
$k$ at time $t$. $V^{\prime T}$ is the set of length $T$ sequences over the
$V^{\prime}$ and $\pi$ is defined as an element of $V^{\prime T}$. The
distribution of one possible path $\pi$ of time $T$ can be interpreted as the
product of the probabilities of each token $\pi_{t}$ on the way.

$p(\pi|\mathbf{X})=\prod^{T}_{t=1}y^{t}_{\pi_{t}},\forall\pi\in V^{\prime T}$
(1)

In order to compute the probability of the target labels, a many-to-one map
$\mathcal{B}$ is defined where all blank tokens and repeated labels from the
paths are removed. Finally, the probability of the target label $\mathbf{l}$
is defined as the sum of the probabilities of all the paths corresponding to
it, as follows:

$p(\mathbf{l}|\mathbf{X})=\sum_{\pi\in\mathcal{B}^{-1}(\mathbf{l})}p(\pi|\mathbf{X}).$
(2)

The forward-backward algorithm is used to efficiently calculate the
probability $p(\mathbf{l}|\mathbf{X})$ and the loss function is defined as the
log probability of the target label:

$L_{\textrm{CTC}}=-ln(p(\mathbf{l}|\mathbf{X})).$ (3)

###  2.2 Multi-view Loss

Jung et al. [15] extended the multi-view learning method [16] using the
asymmetric proxy loss (AsyP) loss by setting the text embedding (TE) as
proxies. We refer to the AsyP loss as `multi-view loss' throughout this paper,
and we explain the loss using the same notations as in [15]. Let a mini-batch
consist of a set of $N$ data tuples, which is denoted as
$\\{(\bm{x}_{i},\bm{t}_{i},c_{i})|i=1,2,\cdots,N\\}$, where $\bm{x}_{i}$ is
the AE of the $i$-th speech segment, $\bm{t}_{i}$ is the TE of the
corresponding text, and $c_{i}$ is the word label. The multi-view loss is
defined as the sum of the anchor–positive (AP) and anchor–negative (AN) terms,
as follows:

$\displaystyle\mathcal{L}_{\textrm{multi-
view}}=\frac{1}{N}\sum_{i=1}^{N}\bigg{(}\frac{1}{\alpha}\underset{j\in\mathcal{P}_{i}}{\textrm{ELSE}}\,\alpha(\lambda-S(\bm{t}_{i},\bm{a}_{j}))$
$\displaystyle+\underset{k\in\mathcal{N}_{i}}{\textrm{MSP}}\,\beta(S(\bm{a}_{i},\bm{t}_{k})-\lambda)\bigg{)},$
(4)

where $\mathcal{P}_{i}=\\{j|y_{j}=y_{i}\\}$, $\mathcal{N}_{i}=\\{k|y_{k}\neq
y_{i}\\}$, and $S(\cdot,\cdot)$ are the set of positive indices, the set of
negative indices, and the cosine similarity, respectively. ELSE and MSP are
the Extended-LogSumExp [17] and Mean-Softplus [18] functions, respectively.
The AP term draws anchor $\bm{t}$ and positives $\bm{a}$ closer, while the AN
term pushes the anchor $\bm{a}$ and negatives $\bm{t}$ apart. $\alpha$,
$\beta$, and $\lambda$ are hyper-parameters determining the boundaries in the
embedding space or the severity of penalty for violations.

##  3 Proposed Method

###  3.1 Architecture

The overall system consists of three main parts: a text encoder, an acoustic
encoder and a CTC aligner as shown in Fig. 1.

The text encoder encodes the keyword text to latent vectors. First, the text
is tokenized into smaller units, such as characters, as the CTC training
requires. Then, the tokenized keyword $\mathbf{y}=\\{y_{u}|1\leq u\leq U\\}$
are fed into two bi-directional LSTM layers to get the token-level TE
$\mathbf{h}^{\mathrm{tokenTE}}_{y_{u}}$, where $u$ is the index of non-blank
tokens and $U$ is the total number of non-blank tokens.

The acoustic encoder operates in streaming mode to encode the input audio
stream $\mathbf{X}=\\{\mathbf{x}_{t}|1\leq t\leq T\\}$ to latent vectors.
After that, the latent vectors are fed into two projection blocks: CTC and AE
projection blocks, resulting in token distributions
$P(\mathbf{y}|\mathbf{x}_{t})$ and frame-level AE
$\mathbf{h}^{\mathrm{frameAE}}_{t}$ for each frame $t$, respectively. In this
work we stack mobilenet [12] blocks in order to keep the size small.

The structure of the CTC aligner is shown in Fig. 2. The states of the
decoding graph $\\{s_{l}|1\leq l\leq L\\}$ each corresponds with the keyword
tokens $\mathbf{y}$ adding the blank tokens in between
$\\{y_{1},-,y_{2},-,...,-,y_{U}\\}$, where $l$ is the state index and
$L=2U-1$. We will refer to the token corresponding to a state $l$ as $st_{l}$.
Then, we can say that $st_{l}=y_{2u-1}$ and
$y_{u}=st_{\lceil\frac{l}{2}\rceil}$ for each non-blank token $u$. The
notations will interchange throughout the paper. Each state stores the
accumulated CTC score $z^{l,t}_{\textrm{CTC}}$, the transition timing
information when the previous non-blank states on the path were first visited
$\mathbf{t}^{l,t}_{y_{1:U}}$, and the accumulated AE for the non-blank tokens
$\mathbf{h}^{l,t}_{y_{1:U}}$. Note that the graph both starts and ends with
non-blank tokens unlike the usual forward-backward algorithm since we are
interested only in the portion of the audio where the keyword is pronounced,
rather than aligning the entire input audio sequence with the given keyword.

Figure 2: The decoding graph made for the target keyword ``cat". In each
state, transition timings and the accumulated AE are stored. The portions
colored white in the state means the information is not (half) filled.

In practice, we add padding tokens to the beginning and the end of the keyword
tokens when training the CTC aligner. We found this helps the model converge
with lower CTC losses and the extra padding token at the end ensures each non-
blank token accumulates its corresponding frame-wise embeddings without any
missing. The blank tokens following a non-blank token are considered as the
continuation of the pronunciation of the non-blank token and thus the frame-
level AE of the blank tokens is added to the token-level AE of the leading
non-blank token. It is worth mentioning that the padding tokens are only used
in algorithms related to CTC alignments, whereas only the non-blank keyword
tokens are considered for the comparison of the matching AE and TE.

###  3.2 CTC Aligner

At every time step $t$, the transition probabilities of the decoding graph are
updated according to the output distribution over the vocabulary
$V^{*}=V\cup\\{Padding,Blank\\}$ calculated from the CTC projection block.
Then we can use the Viterbi algorithm [19] to get the best forced-aligned
paths for the target keyword that ends at the current frame. As the first
state $s_{1}$ is always set independent of other states, its accumulated score
and transition timing information are defined as follows:

$\begin{split}z^{1,t}_{\textrm{CTC}}&=p(y_{1}|\mathbf{x}_{t}),\\\
t^{1,t}_{y_{1}}&=t.\end{split}$ (5)

Since the probabilities of the incoming transitions of a state are all defined
as the probability of the corresponding token of the state at the time, we can
simply compare the scores of the upstream states and decide the source state.

$I^{l,t}=\begin{cases}\mathop{arg\,max}\limits_{i\in\\{l,l-1\\}}z^{i,t-1}_{\textrm{CTC}}&\text{if
}l\text{ is even (blank)}\\\
\mathop{arg\,max}\limits_{i\in\\{l,l-1,l-2\\}}z^{i,t-1}_{\textrm{CTC}}&\text{if
}l\text{ is odd (non-blank)}\end{cases}$ (6)

The transition timings and the accumulated AE are inherited from the source
state. For states corresponding to the non-blank tokens, the transition timing
for the corresponding token $t^{l,t}_{st_{l}}$ is updated as the current time
$t$, and the frame-level AE $\mathbf{h}^{\mathrm{frameAE}}_{t}$ is added to
the accumulated AE for the corresponding token $\mathbf{h}^{l,t}_{st_{l}}$.
For states corresponding to blank tokens, the frame-level AE is added to the
last non-blank token $\mathbf{h}^{l,t}_{st_{l-1}}$. The score of the state is
updated as follows:

$z^{l,t}_{\textrm{CTC}}=z^{I^{l,t},t-1}_{\textrm{CTC}}+logP(st_{l}|\mathbf{x}_{t}).$
(7)

The proposed aligner uses dynamic programming and does not require storing and
revisiting of the computation histories. For decoding only, the time
complexity required for one time step is $O(U)$ and the space complexity is
$O(U^{2})$ where $U$ is the length of the keyword. Since the length of the
keyword is much smaller than the length of audio input sequences, the
algorithm is much more efficient than the non-streaming algorithms.

###  3.3 Training Strategy

Without the need of a pre-trained aligner, we train the model end-to-end from
scratch to ensure the acoustic encoder learns both the CTC alignment and the
AE at the same time. During training however, the entire audio input sequence
of finite length is provided at once unlike the inference time. We skip
storing the AE for the storation itself will consume $O(TU^{2})$ memory space,
and only save the CTC scores and the transition timings of the final state for
every time step in the training sample. After the alignment is finished, we
choose the aligned keyword path at $t_{optimal}$ with the highest CTC score
among all the frames. The token-level AE for each non-blank token $y_{u}$ is
obtained by pooling the frame-level AE between the transition timings stored
in the state $t_{y_{u}}=t^{L,t_{optimal}}_{y_{u}}$:

$\mathbf{h}^{\mathrm{tokenAE}}_{y_{u}}=\dfrac{\sum_{f=t_{y_{u}}}^{t_{y_{u+1}}-1}\mathbf{h}^{\mathrm{frameAE}}_{f}}{t_{y_{u+1}}-t_{y_{u}}}.$
(8)

The objective function is defined as a combination of the $L_{\textrm{CTC}}$
in Eq. (3) and the $\mathcal{L}_{\textrm{multi-view}}$ in Eq. (4), which
measures the difference between the aligned AE and the TE:

$L=L_{\textrm{CTC}}+L_{\textrm{multi-
view}}(\mathbf{h}^{tokenAE}_{y_{1:U}},\mathbf{h}^{tokenTE}_{y_{1:U}}).$ (9)

In this work, we use character as the base tokens. For the Libriphrase [7]
dataset, we also try to combine the embedding vectors of multiple characters
to form higher level embedding: word and phrase levels. For word level, the
frame-level AE and the token-level TE are further accumulated until the space
character is encountered. Whereas for phrase level, the embedding vectors of
the entire keyword are all pooled to make one keyword embedding vector.

###  3.4 Inference

Notably, there is no need to re-visit the results from previous time steps,
for they are preserved in the final state via dynamic programming. Every time
step, all we need is to retrieve the AE from last state of the updated graph
and calculate the cosine similarities between the AE and TE of the target
keyword:

$\begin{split}z_{\textrm{CTC}}&=z^{l,t}_{\textrm{CTC}},\\\
z_{\textrm{embed}}&=\dfrac{1}{U}\sum_{u=1}^{U}cos(\mathbf{h}^{tokenAE}_{y_{u}},\mathbf{h}^{tokenTE}_{y_{u}}).\\\
\end{split}$ (10)

The final score can then be derived by linearly integrating the two scores:

$z=z_{\textrm{CTC}}+\lambda z_{\textrm{embed}},$ (11)

where $\lambda$ is a hyper-parameter that needs to be tuned on a development
set.

##  4 Experiment Result

We trained our CTCAT keyword detector with an acoustic encoder with only 155K
parameters. The detailed structure can be found in Fig. 1, where 12
Convolutional blocks were stacked whose kernel size for the depthwise
convolution and the feature map number for the pointwise convolution being 12
and 96. All the projection blocks consisted of one dense layer followed by a
batch normalization layer [20]. For the text encoder, the input tokens were
turned into 256-dimension vectors by a trainable look-up table and then fed
into two BLSTM layers with hidden size of 256. We used 28 tokens (English
alphabets, space, apostrophe) to train the text encoder, and added two more
tokens for padding and blank to train the CTC aligner. 80-channel filterbanks
were extracted from the input audio stream with a window of 25ms and a stride
of 10ms. For the multi-view loss in Eq. (4), we set $\alpha=2$, $\beta=50$,
and $\lambda=0.1$. Our mini-batches contained 1024 keywords, each of which was
comprised with two examples as the positive pair, and the examples for all
other keywords acted as the negative example. Adam optimizer [21] and cosine
decay [22] are used until 100 epochs was reached for the Libriphrase training
set. It took us two days to train the models on two A100 GPUs [23].

Table 1: EER (%) and AUC (%) on the Libriphrase evaluation set. CTC is trained only with the CTC loss. The aligned embedding vectors are trained in three different levels: character, word, and phrase level. `#Params' denotes the (estimated) number of parameters of models used for inference. Method | #Params | EER (%) | AUC (%)
---|---|---|---
$\mathbf{LP_{E}}$ | $\mathbf{LP_{H}}$ | $\mathbf{LP_{E}}$ | $\mathbf{LP_{H}}$
Attention [7] |  $\sim$420K | 8.24 | 32.90 | 96.70 | 73.58
DSP [9] | 3.7M | 7.36 | 23.36 | 97.83 | 84.21
CTC | 147K | 9.11 | 32.37 | 96.76 | 73.95
+character | 155K | 8.65 | 32.76 | 96.99 | 73.53
+word | 155K | 7.05 | 31.62 | 97.97 | 75.12
+phrase | 155K | 6.06 | 29.63 | 98.32 | 77.10

The Libriphrase [7] dataset was chosen for evaluation for a fair comparison
with other methods. Our system was trained from scratch with the training set
from Libriphrase which was generated from train-clean-100 and train-clean-360
with phrases with 1 to 4 words. To make our model more robust, we convolved
the clean speech signals with synthetic room impulse responses (RIRs) from the
OpenSLR dataset [24] and added the MUSAN noise dataset [25] at randomly
selected signal-to-noise ratios (SNRs) between -3 and 25 dB. The evaluation
set was generated from train-others-500 and the negative examples are divided
into two sets, easy ($\mathbf{LP_{E}}$) and hard ($\mathbf{LP_{H}}$), based on
Levenstein distances. [26] We used the same evaluation dataset which contains
4391, 2605, 467, and 56 episodes for anchor phrases of each length. Each
episode contains 3 positive and 3 negative pairs. The text of the anchors were
used for text enrollment and the audio of the comparison examples are used for
the evaluation of the keyword detection task. The $\lambda$ in Eq. 11 was
decided from 100k example that we randomly held out from the training set. The
final results were shown with $\lambda=6$.

We compared our system with the non-streaming open-vocabulary keyword spotting
systems which used the cross-attention [7] and a dynamic programming based
algorithm called Dynamic Sequence Partitioning (DSP) [9] to align the input
audio sequence and the target text. Using the combined scores of CTC and
embedding, our system outperformed the non-streaming solutions on
$\mathbf{LP_{E}}$ set, and showed competitive results on $\mathbf{LP_{H}}$
set. This is a very encouraging result considering the extremely small
acoustic encoder with 155K parameters and 6.91M FLOPs per time step and the
highly efficient inference algorithm with time complexity $O(U)$. The detailed
result is shown in Table 1.

Four models: trained with only CTC loss, adding joint embedding learnings for
token (character), word and phrase levels are all trained from scratch
individually. From the experiment results in Table 1, we can confirm that
using the embedding score on top of the CTC score enhances the performance by
utilizing the global context over the entire keyword. Moreover, a slight
performance drop on $\mathbf{LP_{H}}$ was noticed when character-level
embedding score was adopted whereas using higher level abstraction resulted in
huge performance gain. This indicates that considering more global context
help segregate the embedding vectors in the latent space better. Table 2 shows
the average number of frame-level AE accumulated to form a single AE in
Libriphrase evaluation set.

Table 2: The average number of frame-level acoustic embedding (AE) accumulated to form a single higher level AE. | character | word | phrase
---|---|---|---
average length | 5.6 | 31.7 | 49.2

We also visualized the correlations between the embedding vectors and the CTC
alignments at the same time to confirm the benefit of our joint training
strategy in Fig 3. We chose the model trained with the word-level embedding to
see both the correlations between the frame-level AE within the same word and
between different words. Higher correlations were found within the aligned
paths which indicates that the CTC score and the embedding score can work
coherently without interfering with each other.

(a) Audio-text embedding (b) Acoustic embedding

Figure 3: The correlation map between the audio-text embedding of the positive
examples for the target keyword ``said the king''. The blue lines denote the
aligned result from CTC for the audio sections of $\langle
silence\rangle,``said",``the",``king",\langle silence\rangle$

##  5 Conclusion

We have introduced a novel CTC-aligned Audio-Text (CTCAT) keyword detector to
obtain the joint audio-text embedding in streaming KWS. We achieved dynamic
aligning between the target keyword text and the input audio sequence using
CTC, and used the acoustic embedding along the paths to obtain the embedding
similarities. Experiments on the LibriPhrase dataset showed that our method
shows competitive performance compared to non-streaming methods with much
smaller model size and time consumption.

## References

  * [1] R. Rose and D. Paul, ``A hidden markov model based keyword recognition system,'' in _Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 1990, pp. 129–132.
  * [2] T. N. Sainath and C. Parada, ``Convolutional neural networks for small-footprint keyword spotting,'' in _Proc. Interspeech_ , 2015, pp. 1478–1482.
  * [3] G. Chen, C. Parada, and G. Heigold, ``Small-footprint keyword spotting using deep neural networks,'' in _Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2014.
  * [4] J. Huang, W. Gharbieh, H. S. Shim, and E. Kim, ``Query-by-example keyword spotting system using multi-head attention and soft-triple loss,'' in _Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 6858–6862.
  * [5] G. Chen, C. Parada, and T. N. Sainath, ``Query-by-example keyword spotting using long short-term memory networks,'' in _Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2015, pp. 5236–5240.
  * [6] L. Lugosch, S. Myer, and V. S. Tomar, ``DONUT: ctc-based query-by-example keyword spotting,'' _arXiv:1811.10736_ , 2018.
  * [7] H.-K. Shin, H. Han, D. Kim, S.-W. Chung, and H.-G. Kang, ``Learning audio-text agreement for open-vocabulary keyword spotting,'' in _Proc. Interspeech_ , 2022, pp. 1871–1875.
  * [8] Y.-H. Lee and N. Cho, ``PhonMatchNet: Phoneme-guided zero-shot keyword spotting for user-defined keywords,'' in _Proc. Interspeech_ , 2023, pp. 3964–3968.
  * [9] K. Nishu, M. Cho, and D. Naik, ``Matching latent encoding for audio-text based keyword spotting,'' in _Proc. Interspeech_ , 2023, pp. 1613–1617.
  * [10] K. Nishu, M. Cho, P. Dixon, and D. Naik, ``Flexible keyword spotting based on homogeneous audio-text embedding,'' _arXiv:2308.06472_ , 2023.
  * [11] Y. Lecun and Y. Bengio, ``Convolutional networks for images, speech, and time-series,'' _The handbook of brain theory and neural networks_ , 1995.
  * [12] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, ``Mobilenets: Efficient convolutional neural networks for mobile vision applications,'' _arXiv:1704.04861_ , 2017.
  * [13] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, ``Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,'' in _Proc. the 23rd International Conference on Machine Learning (ICML)_ , 2006, pp. 369–376.
  * [14] Y. Wang and Y. Long, ``Keyword spotting based on ctc and rnn for mandarin chinese speech,'' in _2018 11th International Symposium on Chinese Spoken Language Processing (ISCSLP)_ , 2018, pp. 374–378.
  * [15] M. Jung and H. Kim, ``Asymmetric proxy loss for multi-view acoustic word embeddings,'' in _Proc. Interspeech_ , 2022, pp. 5170–5174.
  * [16] W. He, W. Wang, and K. Livescu, ``Multi-view recurrent neural acoustic word embeddings,'' in _Proc. International Conference on Learning Representations (ICLR)_ , 2017.
  * [17] X. Wang, X. Han, W. Huang, D. Dong, and M. R. Scott, ``Multi-similarity loss with general pair weighting for deep metric learning,'' in _Proc. the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 5022–5030.
  * [18] D. Yi, Z. Lei, S. Liao, and S. Z. Li, ``Deep metric learning for person re-identification,'' in _Proc. International Conference on Pattern Recognition_ , 2014, pp. 34–39.
  * [19] A. Viterbi, ``Error bounds for convolutional codes and an asymptotically optimum decoding algorithm,'' _IEEE Transactions on Information Theory_ , vol. 13, no. 2, pp. 260–269, 1967.
  * [20] S. Ioffe and C. Szegedy, ``Batch normalization: Accelerating deep network training by reducing internal covariate shift,'' in _Proc. International Conference on Machine Learning (ICML)_ , 2015, pp. 448–456.
  * [21] D. P. Kingma and J. Ba, ``Adam: A method for stochastic optimization,'' _arXiv:1412.6980_ , 2014.
  * [22] I. Loshchilov and F. Hutter, ``SGDR: stochastic gradient descent with restarts,'' in _Proc. International Conference on Learning Representations (ICLR)_ , 2017.
  * [23] H. Anzt, Y. M. Tsai, A. Abdelfattah, T. Cojean, and J. Dongarra, ``Evaluating the performance of nvidia’s a100 ampere gpu for sparse and batched computations,'' in _IEEE/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS)_ , 2020, pp. 26–38.
  * [24] T. Ko, V. Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur, ``A study on data augmentation of reverberant speech for robust speech recognition,'' in _Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2017, pp. 5220–5224.
  * [25] D. Snyder, G. Chen, and D. Povey, ``Musan: A music, speech, and noise corpus,'' _arXiv:1510.08484v1_ , 2015.
  * [26] V. I. Levenshtein, ``Binary codes capable of correcting deletions, insertions, and reversals,'' _Soviet physics. Doklady_ , vol. 10, pp. 707–710, 1965. [Online]. Available: https://api.semanticscholar.org/CorpusID:60827152