# Unsupervised Word Segmentation from Discrete Speech Units
in Low-Resource Settings

###### Abstract

When documenting oral-languages, Unsupervised Word Segmentation (UWS) from
speech is a useful, yet challenging, task. It can be performed from phonetic
transcriptions, or in the absence of these, from the output of unsupervised
speech discretization models. These discretization models are trained using
raw speech only, producing discrete speech units which can be applied for
downstream (text-based) tasks. In this paper we compare five of these models:
three Bayesian and two neural approaches, with regards to the exploitability
of the produced units for UWS. Two UWS models are experimented with and we
report results for Finnish, Hungarian, Mboshi, Romanian and Russian in a low-
resource setting (using only 5k sentences). Our results suggest that neural
models for speech discretization are difficult to exploit in our setting, and
that it might be necessary to adapt them to limit sequence length. We obtain
our best UWS results by using the SHMM and H-SHMM Bayesian models, which
produce high quality, yet compressed, discrete representations of the input
speech signal.

Index Terms: unsupervised word segmentation, speech discretization, acoustic
unit discovery, low-resource conditions.

##  1 Introduction

Popular models for speech processing rely on the availability of large amounts
of speech data and their transcriptions, which reduces model applicability to
a limited subset of languages considered high-resource. This excludes a
considerable number of low-resource languages, including many from oral-
tradition. Besides, learning supervised representations from speech differs
from the unsupervised way infants learn language, hinting that it should be
possible to develop more data-efficient speech processing models. Recent
efforts for Zero Resource [1, 2, 3, 4, 5, 6] processing focus on building
speech systems using limited amounts of data (hence zero resource), and
without textual or linguistic resources, for increasingly challenging tasks
such as acoustic or lexical unit discovery. Such zero resource approaches also
stimulated interest for computational language documentation [7, 8, 9, 10] and
computational language acquisition [11].

In this paper we address a challenging task: Unsupervised Word Segmentation
(UWS) from speech, which consists of (without access to any supervision)
outputting time-stamps delimiting stretches of speech, associated with class
labels corresponding to word hypotheses. We build on the work presented in [9]
which proposed a cascaded model for UWS that first generates a discrete
sequence from the speech signal using the model from [12], and then segments
the discrete sequence into words using a Bayesian [13] or a neural [14]
approach. Since then, much progress has been made in automatic speech
discretization: efficient Bayesian models for Acoustic Unit Discovery (AUD)
emerged [15, 16], and neural approaches (typically made of an auto-encoder
structure with a discretization layer) were also introduced [17, 18, 19].

Therefore, in this work we revise [9], by empirically investigating the
exploitability of five recent approaches for speech discretization for the UWS
task in a rather low-resource scenario (using 5k sentences only). More
precisely, we compare three Bayesian speech discretization models (HMM [12],
SHMM [15] and H-SHMM [16]), and two neural models (VQ-VAE [17] and vq-wav2vec
[18]). We train and extract discrete speech units from them using only 4-5
hours of speech, and we perform UWS from the sequences produced. Moreover, for
investigating how generalizable these models are, we apply the same pipeline
to five languages: Finnish, Hungarian, Mboshi, Romanian and Russian. In our
experiments, we focus on the temporal aspect of the UWS task, evaluating the
quality of the boundaries produced by these models.

Our experiments show that neural models for speech discretization are
difficult to directly exploit for UWS, as they output very long
sequences.111In this work we define the sequence length of a speech
discretization output as the one obtained after merging consecutive 10ms
windows that share the same prediction. In contrast to that, the Bayesian
speech discretization approaches from [15, 16] are robust and generalizable,
producing high quality, yet compressed, discrete speech sequences from the
input utterances in all languages. We obtain our best results by using these
sequences for training the neural UWS model from [14]. Finally, the
contributions of this work are the following.

  * •

We compare five models for speech discretization in low-resource settings for
posterior application in the task of UWS. Our results shed light on the direct
exploitability of current discretization approaches.

  * •

We investigate the generalization of the speech discretization models by
training models in the following languages: Mboshi [20] (truly unwritten
language), and Finnish, Hungarian, Romanian and Russian (from the multilingual
dataset MaSS [21]).

##  2 Related Work

Some recent editions of the zero resource speech challenges [3, 4, 5] focused
on UWS [22, 23] and Unsupervised Term Discovery (UTD) [24, 25] directly from
speech.222The different between UWS and UTD is that the latter does not
produce a total segmentation of the speech signal. In contrast to that, in
this paper we focus on a pipeline approach for UWS: we first perform speech
discretization, followed by text-based UWS. We focus on the evaluation of the
direct exploitability of discrete speech units in low-resource settings, as
their output allow us to deploy (lighter) text-based UWS approaches. In this
setting, the UWS model can also take advantage of some extra supervision, such
as the translations used in [9, 26].

Focusing on the use of discrete speech units, [27] propose constraining the
VQ-VAE model in order to generate a more exploitable output representation for
direct application to the UWS task. Different from that, in this work we focus
on providing an empirical comparison of recent approaches, extending [9] and
providing results in low-resource settings, and in five different languages.

Moreover, this work falls into the category of computation language
documentation approaches. Recent works in this field include the use of
aligned translation for improving transcription quality [28], and for
obtaining bilingual-rooted UWS [8, 14]. There are pipelines for obtaining
manual [29] and automatic [30] transcriptions, and for aligning transcription
and audio [31]. Other examples are methods for low-resource segmentation [32,
33], and for lexical unit discovery without textual resources [34]. Finally,
direct speech-to-speech [35] and speech-to-text [7, 36] architectures could be
an option for the lack of transcription, but it remains to be seen how
exploitable these can be in low-resource settings.

##  3 Unsupervised Speech Discretization Models

Speech discretization consists in labeling the speech signal into discrete
speech units, which can correspond or not to the language phonetic inventory.
This problem can be formulated as the learning of a set of $U$ discrete units
with embeddings
$\bm{\mathbf{H}}=\\{\bm{\mathbf{\eta}}^{1},\dots,\bm{\mathbf{\eta}}^{U}\\}$
from a sequence of untranscribed acoustic features
$\bm{\mathbf{X}}=[\bm{\mathbf{x}}_{1},\dots,\bm{\mathbf{x}}_{N}]$, as well as
the assignment of frame to unit $\bm{\mathbf{z}}=[z_{1},\dots,z_{N}]$.
Depending on the approach, neural or Bayesian, the assumptions and the
inference regarding these three quantities will differ.

###  3.1 Neural (VQ-based) models

VQ-VAE. It comprises an encoder, a decoder and a set of unit-specific
embeddings $\bm{\mathbf{H}}$. The encoder is a neural network that transforms
the data into a continuous latent representation
$\bm{\mathbf{V}}=(\bm{\mathbf{v}}_{1},\dots,\bm{\mathbf{v}}_{N})$. Each frame
is then assigned to the closest embedding in the Euclidean sense (Eq. 1). The
decoder transforms the sequence of quantized vectors into parameters of the
conditional log-likelihood of the data
$p(\bm{\mathbf{x}}_{n}|\bm{\mathbf{z}})$ and the network is trained to
maximize this likelihood. Since the quantization step is not differentiable,
the encoder is trained with a straight through estimator [37]. In addition, a
pair of $\ell_{2}$ losses are used to minimize the quantization error, and the
overall objective function that is maximized is presented in Eq. 2, where
$\operatorname{sg}[\cdot]$ is the stop-gradient operator. We define the
likelihood
$p(\bm{\mathbf{x}}_{n}|z_{n})=\mathcal{N}(\bm{\mathbf{x}}_{n};\bm{\mathbf{\mu}}(\bm{\mathbf{\eta}}^{z_{n}}),\bm{\mathbf{I}})$.
Under this assumption, the log-likelihood reduces to the mean-squared error
$||\bm{\mathbf{x}}_{n}-\bm{\mathbf{\mu}}(\bm{\mathbf{\eta}}^{z_{n}})||_{2}^{2}$.

$\displaystyle
z_{n}=\operatorname*{arg\,min}_{u}||\bm{\mathbf{v}}_{n}-\bm{\mathbf{\eta}}^{u}||_{2}.$
(1) $\displaystyle\mathcal{L}=\frac{1}{N}\sum_{n=1}^{N}\Bigl{(}$
$\displaystyle\ln
p(\bm{\mathbf{x}}_{n}|z_{n})-k_{1}||\operatorname{sg}[\bm{\mathbf{\eta}}^{z_{n}}]-\bm{\mathbf{v}}_{n}||_{2}^{2}$
$\displaystyle-
k_{2}||\bm{\mathbf{\eta}}^{z_{n}}-\operatorname{sg}[\bm{\mathbf{v}}_{n}]||_{2}^{2}\Bigr{)},$
(2)

vq-wav2vec. This model is composed of an encoder
($f:\bm{\mathbf{X}}\xrightarrow{}\bm{\mathbf{Z}}$), a quantizer
($q:\bm{\mathbf{Z}}\xrightarrow{}\bm{\mathbf{\hat{Z}}}$) and an aggregator
($g:\bm{\mathbf{\hat{Z}}}\xrightarrow{}\bm{\mathbf{C}}$). The encoder is a CNN
which maps the raw speech input $\bm{\mathbf{X}}$ into the dense feature
representation $\bm{\mathbf{Z}}$. From this representation, the quantizer
produces discrete labels $\bm{\mathbf{{\hat{Z}}}}$ from a fixed size codebook
$\bm{\mathbf{e}}\in\mathbb{R}^{V\times d}$ with $V$ representations of size
$d$. Since replacing an encoder feature vector $\bm{\mathbf{z}}_{i}$ by a
single entry in the codebook makes the method prone to model collapse, the
authors independently quantize partitions of each feature vector by creating
multiple groups $G$, arranging the feature vector into a matrix form
$\bm{\mathbf{z^{\prime}}}\in\mathbb{R}^{G\times(d/G)}$. Considering each row
by an integer index, the full feature vector is represented by the indices
$\bm{\mathbf{i}}\in[V]^{G}$, with $V$ being the possible number of variables
for a given group, and each element $\bm{\mathbf{i}}_{j}$ corresponding to a
fixed codebook vector ($j\in|G|$). For each of the $G$ groups, the
quantization is performed by using Gumbel-Softmax [38] or online k-means
clustering. Finally, the aggregator combines multiple quantized feature vector
time-steps into a new representation $\bm{\mathbf{c}}_{i}$ for each time step
$i$. The model is trained to distinguish a sample $k$ steps in the future
$\bm{\mathbf{\hat{z}}}_{i+k}$ from distractor samples
$\bm{\mathbf{\tilde{z}}}$ drawn from a distribution $p_{n}$, by minimizing the
contrastive loss for steps $k=\\{1,\dots,K\\}$ as in Eq. 3, where $T$ is the
sequence length, $\sigma(x)=1/(1+exp(-x))$,
$\sigma(\bm{\mathbf{\hat{z}}}_{i+k}^{\intercal}h_{k}(\bm{\mathbf{c_{i}}})$ is
the probability of $\bm{\mathbf{\hat{z}}}_{i+k}$ being the true sample, and
$h_{k}(\bm{\mathbf{c}}_{i})$ is the step-specific affine transformation
$h_{k}(\bm{\mathbf{c}}_{i})=W_{k}\bm{\mathbf{c}}_{i}+b_{k}$. Finally, this
loss is accumulated over all $k$ steps
$\mathcal{L}=\sum_{k=1}^{K}\mathcal{L}_{k}$.

$\displaystyle\mathcal{L}_{k}=\sum_{i=1}^{T-k}\Bigl{(}$
$\displaystyle\log\sigma(\bm{\mathbf{\hat{z}}}_{i+k}^{\intercal}h_{k}(\bm{\mathbf{c_{i}}}))$
$\displaystyle+\lambda\mathbb{E}_{\bm{\mathbf{\tilde{z}}}\sim
p_{n}}[\log\sigma(-\bm{\mathbf{\tilde{z}}}^{\intercal}h_{k}(\bm{\mathbf{c}}_{i}))]\Bigl{)}$
(3)

Training. For VQ-VAE, the encoder has 4 Bi-LSTM layers each with output
dimension 128 followed by a 16-dimensional and a feed-forward decoder with one
hidden layer. The number of discovered units (quantization centroids) is set
to 50. This setting is unusually low but helps to reduce the length of the
output sequence. We set $k_{1}=2$ and $k_{2}=4$ (Eq. 2) and
train333Implementation available at: https://github.com/BUTSpeechFIT/vq-aud
with Adam with an initial learning rate of $2\times 10^{-3}$ which is halved
whenever the loss stagnates for two training epochs. For vq-wav2vec, we use
the small model444Implementation available at:
https://github.com/pytorch/fairseq/tree/master/examples/wav2vec from [18], but
with only 64 channels, residual scale of 0.2, and warm-up of 10k. For
vocabulary we adopted G=2 and experimented having both V=4, resulting in 16
units (VQ-W2V-V16), and V=6, resulting in 36 units (VQ-W2V-V36). Larger
vocabularies resulted in excessively long sequences which could not be used
for UWS.555The dpseg original implementation only processes sequences shorter
than 350 tokens. We also experimented reducing the representation by using
Byte Pair Encoding (BPE) [39], hypothesizing that phones were being modeled by
a combination of different units. In this setting, BPE serves as a method for
identifying and clustering these patterns. Surprisingly, we found that using
BPE resulted in a decrease in UWS performance, which shows that this model is
not very consistent during labeling process.

Table 1: Statistics for the datasets computed over the text (FR), or over the phonemic representation (*). |  | #Types | #Tokens |  | Avg Token
---
Length
| Avg #Tokens
---
per Sentence
MB-FR | MB* | 6,633 | 30,556 | 4.2 | 6.0
FR | 5,162 | 42,715 | 4.4 | 8.3
MaSS | FI* | 12,088 | 70,226 | 6.0 | 13.2
HU* | 12,993 | 69,755 | 5.9 | 13.1
RO* | 6,795 | 84,613 | 4.5 | 15.9
RU* | 10,624 | 67,176 | 6.2 | 12.6
FR | 7,226 | 94,527 | 4.1 | 17.8

###  3.2 Bayesian Generative Models

For generative models, each acoustic unit embedding $\bm{\mathbf{\eta}}_{i}$
represents the parameters of a probability distribution
$p(\bm{\mathbf{x}}_{n}|\bm{\mathbf{\eta}}_{z_{n}},z_{n})$ with latent
variables $\bm{\mathbf{z}}$. Discovering the units amounts to estimating the
posterior distribution over the embeddings $\bm{\mathbf{H}}$ and the
assignment variables $\bm{\mathbf{z}}$ given by:

$\displaystyle p(\bm{\mathbf{z}},\bm{\mathbf{H}}|\bm{\mathbf{X}})$
$\displaystyle\propto
p(\bm{\mathbf{X}}|\bm{\mathbf{z}},\bm{\mathbf{H}})p(\bm{\mathbf{z}}|\bm{\mathbf{H}})\prod_{u=1}^{U}p(\bm{\mathbf{\eta}}^{u}).$
(4)

From this, we describe three different approaches.

HMM. In this model each unit is a 3-state left-to-right HMM with parameters
$\bm{\mathbf{\eta}}^{i}$. Altogether, the set of units forms a large HMM
analog to a “phone-loop” recognition model. This model, described in [12],
serves as the backbone for the two subsequent models.

SHMM. The prior $p(\bm{\mathbf{\eta}})$ in Eq. 4 is the probability that a
sound, represented by an HMM with parameters $\bm{\mathbf{\eta}}$, is an
acoustic unit. For the former model, it is defined as a combination of
exponential family distributions forming a prior conjugate to the likelihood.
While mathematically convenient, this prior does not incorporate any knowledge
about phones, i.e. it considers all possible sounds as potential acoustic
units. In [15], they propose to remedy this shortcoming by defining the
parameters of each unit $u$ as in Eq. 5, where $\bm{\mathbf{e}}^{u}$ is a low-
dimensional unit embedding, $\bm{\mathbf{W}}$ and $\bm{\mathbf{b}}$ are the
parameters of the _phonetic subspace_ , and the function $f(\cdot)$ ensures
that the resulting vector $\bm{\mathbf{\eta}}^{u}$ dwells in the HMM parameter
space. The subspace, defined by $\bm{\mathbf{W}}$ and $\bm{\mathbf{b}}$, is
estimated from several labeled source languages. The prior
$p(\bm{\mathbf{\eta}})$ is defined over the low-dimensional embeddings
$p(\bm{\mathbf{e}})$ rather than $\bm{\mathbf{\eta}}$ directly, therefore
constraining the search of units in the relevant region of the parameter
space. This model is denoted as the Subspace HMM (SHMM).

$\displaystyle\bm{\mathbf{\eta}}^{u}=f(\bm{\mathbf{W}}\cdot\bm{\mathbf{e}}^{u}+\bm{\mathbf{b}})$
(5)

H-SHMM. While the SHMM significantly improves results over the HMM, it also
suffers from an unrealistic assumption: it assumes that the phonetic subspace
is the same for all languages. [16] relax this assumption by proposing to
adapt the subspace for each target language while learning the acoustic units.
Formally, for a given language $\lambda$, the subspace and the acoustic units’
parameters are constructed as in Eq. 6-8, where the matrices
$\bm{\mathbf{M}}_{0},\dots,\bm{\mathbf{M}}_{K}$ and vectors
$\bm{\mathbf{m}}_{0},\dots,\bm{\mathbf{m}}_{K}$ represent some “template”
phonetic subspaces linearly combined by a language embedding
$\bm{\mathbf{\alpha}}^{\lambda}=[\alpha^{\lambda}_{1},\alpha^{\lambda}_{2},\dots,\alpha^{\lambda}_{K}]^{\top}$.
The matrices $\bm{\mathbf{M}}_{i}$ and the vectors $\bm{\mathbf{m}}_{i}$ are
estimated from labeled languages, from multilingual transcribed speech dataset
for instance, while the acoustic units’ low-dimensional embeddings
$\\{\bm{\mathbf{e}}_{i}\\}$ and the language embedding $\bm{\mathbf{\alpha}}$
are learned on the target (unlabeled) speech data. We refer to this model as
the Hierarchical SHMM (H-SHMM).

$\displaystyle\bm{\mathbf{W}}^{\lambda}$
$\displaystyle=\bm{\mathbf{M}}_{0}+\sum_{k=1}^{K}\alpha^{\lambda}_{k}\bm{\mathbf{M_{k}}}$
(6) $\displaystyle\bm{\mathbf{b}}^{\lambda}$
$\displaystyle=\bm{\mathbf{m}}_{0}+\sum_{k=1}^{K}\alpha^{\lambda}_{k}\bm{\mathbf{m_{k}}}$
(7) $\displaystyle\bm{\mathbf{\eta}}^{\lambda,u}$
$\displaystyle=f(\bm{\mathbf{W}}^{\lambda}\cdot\bm{\mathbf{e}}^{\lambda,u}+\bm{\mathbf{b}}^{\lambda})$
(8)

Inference. For the three generative models, the posterior distribution is
intractable and cannot be estimated. Instead, one seeks for an approximate
posterior
$q(\\{\bm{\mathbf{\eta}}_{i}\\},\bm{\mathbf{z}})=q(\\{\bm{\mathbf{\eta}}_{i}\\})q(\bm{\mathbf{z}})$
which maximizes the variational lower-bound $\mathcal{L}[q]$. Concerning
estimation of $q(\bm{\mathbf{z}})$, the expectation step is identical for all
models and is achieved with a modified forward-backward algorithm described in
[12]. Estimation of $q(\bm{\mathbf{\eta}})$, the maximization step, is model-
specific and is described in [12] for the HMM, in [15] for SHMM models, and in
[16] for the H-SHMM model. Finally, the output of each system is obtained from
a modified Viterbi algorithm which uses the expectation of the log-likelihoods
with respect to $q(\\{\bm{\mathbf{\eta}}_{i}\\})$, instead of point estimates.

Training. The models666Implementation available at: https://github.com/beer-
asr/beer/tree/master/recipes/hshmm are trained with 4 Gaussians per HMM state
and using 100 for the Dirichlet process’ truncation parameter. SHMM and H-SHMM
use an embedding size of 100, and H-SHMM models have a 6-dimensional language
embedding. For the methods which use sub-spaces estimation (SHMM and H-SHMM),
this estimation uses the following languages: French, German, Spanish, Polish
from the Globalphone corpus [40], as well as Amharic [41], Swahili [42] and
Wolof [43] from the ALFFA project [44]. We use 2-3 hours subsets of each,
totaling around 19 hours.

##  4 Experimental Setup

Table 2: Statistics for the discrete speech units produced for the Mboshi,
with the difference between the produced and reference representation shown
between parentheses. RAW is the original output from speech discretization
models, +SIL is the result after silence post-processing. Other languages
follow the same trend.

|  | HMM | SHMM | H-SHMM
---|---|---|---|---
RAW | # Units | 77 (+9) | 76 (+8) | 49 (-19)
| Avg #Units
---
per sequence
27.5 (+8.7) | 24.0 (+5.2) | 21.7 (+2.9)
Max Length | 68 (+17) | 69 (+18) | 63 (+12)
+SIL | # Units | 75 (+7) | 75 (+7) | 47 (-21)
| Avg #units
---
per sequence
20.9 (+2.1) | 19.9 (+1.1) | 19.4 (+0.6)
Max Length | 69 (+18) | 62 (+11) | 60 (+9)
|  | VQ-VAE | VQ-W2V-16 | VQ-W2V-36
RAW | # Units | 50 (-18) | 16 (-52) | 36 (-32)
| Avg #units
---
per sequence
65.2 (+46.4) | 81.7 (+62.9) | 111.0 (+92.2)
Max Length | 217 (+166) | 289 (+238) | 361 (+310)
+SIL | # Units | 50 (-18) | 16 (-52) | 36 (-32)
| Avg #units
---
per sequence
43.4 (+24.6) | 52.6 (+33.8) | 76.2 (+57.4)
Max Length | 143 (+92) | 229 (+178) | 271 (+220)

From the discrete speech units produced by the presented speech discretization
models, we produce segmentation in the symbolic domain by using two UWS
models. A final speech segmentation is then inferred using the units’ time-
stamps and evaluated by using the Zero Resource Challenge 2017 evaluation
suite, track 2 [4]. We now detail the UWS models used in this work, which are
trained with the same parameters from [9], the datasets, and the post-
processing for the speech discrete units.

Bayesian UWS approach (monolingual). Non-parametric Bayesian models [13, 45]
are statistical approaches for UWS and morphological analysis, known to be
robust in low-resource settings [46]. In these models, words are generated by
a unigram or bigram model over an infinite inventory, through the use of a
Dirichlet process. In this work, we use the unigram model from dpseg [33],
which was shown to be superior than the bilingual model in low-resource
settings [47].

Neural UWS approach (bilingual). We follow the bilingual pipeline from [9].
The discrete speech units and their sentence-level translations are fed to an
attention-based Neural Machine Translation system that produces soft-alignment
probability matrices between source and target sequences. For each sentence
pair, its matrix is used for clustering together (segmenting) neighboring
phones whose alignment distribution peaks at the same source word. We refer to
this model as neural.

Datasets. We use the same dataset from [9]: the Mboshi-French parallel corpus
(MB-FR) [20], which is a 5,130 sentence corpus from the language documentation
process of Mboshi (Bantu C25), an oral language spoken in Congo-Brazzaville.
We also report results using an extract from the MaSS corpus [21], a
multilingual speech-to-speech and speech-to-text dataset. We use the down-
sampling from [48], which results in 5,324 aligned sentences. We exclude
English, French and Spanish, as these languages are present in the sub-space
prior from SHMM and H-SHMM models. We also exclude Basque as the sequences
produced were too long for UWS training. The final set of languages is:
Finnish (FI), Hungarian (HU), Romanian (RO) and Russian (RU). In all cases,
the French (FR) translations are used as supervision for the neural UWS
approach. Statistics are presented in Table 1.

Discrete Speech Units Post-processing. We experiment with reducing the
representation by removing units predicted in silence windows. For this, we
use the gold references’ silence annotations. Removing these allow us to focus
the investigation on the quality of the units generated in relevant portions
of the speech. We see in Table 2 that removing windows that we know correspond
to silence considerably reduces the number of units generated by all models.
Before UWS evaluation, the silence windows are reintroduced to insure that
their segmentation boundaries are taken into account.

##  5 Experiments

Table 3: UWS Boundary F-scores for the MB-FR dataset. |  | dpseg | neural
---|---|---|---
|  | RAW | +SIL | RAW | +SIL
1 | HMM | 32.4 | 59.9 | 35.1 | 61.2
2 | SHMM | 43.7 | 61.4 | 41.4 | 64.7
3 | H-SHMM | 45.3 | 61.4 | 44.8 | 63.9
4 | VQ-VAE | 39.0 | 52.7 | 32.1 | 60.1
5 | VQ-W2V-V16 | 37.4 | 52.2 | 32.0 | 50.6
6 | VQ-W2V-V36 | - | 48.0 | - | 49.8
7 | True Phones | - | 77.1 | - | 74.5

(a) VQ-VAE: Discrete speech units (top), and reference (bottom).

(b) H-SHMM: Discrete speech units (top), and reference (bottom).

Figure 1: Speech discrete units produced by two models for the same Mboshi
sentence. Black lines denote the true boundaries, while dashed white lines
denote the discovered units boundaries.

Table 3 presents UWS Boundary F-scores for UWS models (dpseg and neural)
trained using different discrete speech units for the MB-FR dataset. We
include results for both the direct output (RAW) and the post-processed
version (+SIL). The RAW VQ-W2V-V36 is not included as its output sequences
were excessively large for training our UWS models (Table 2). We observe that
in all cases, post-processing the discrete speech units with the silence
information (+SIL) creates easier representations for the UWS task. We believe
this is due to the considerable reduction in average length of the sequences
(Table 2). For Bayesian models, we also observe a reduction in the number of
units, meaning that some units were modelling silence windows, even though
these models already produce an independent token for silence, which we remove
before UWS training.

Looking at the results for UWS models trained using the output of VQ-based
models (rows 4-6), we see that the best segmentation result is achieved using
the one with the smallest average sequence length (VQ-VAE). In general, we
believe that all VQ-based models under-perform due to the excessively long
sequences produced, which are challenging for UWS. Figure 1 illustrates this
difference in representation length, by presenting the speech discrete units
produced by H-SHMM and VQ-VAE for a given utterance. Overall, we find that UWS
models trained using the discrete speech units from Bayesian models produce
better segmentation, with models trained with SHMM and H-SHMM presenting the
best results. A noticeable difference between these two models is the
compression level: H-SHMM uses 27 less units than SHMM. Regarding type
retrieval, the models scored 12.1% (SHMM), 10.7% (H-SHMM), and 31% (topline).
We also find that SHMM models produced more types and less tokens, reaching a
higher Type-Token Ratio (0.63) compared to H-SHMM (0.55).

Table 4: UWS Boundary F-scores for the MaSS dataset using Bayesian models
(+SIL only). Best results presented in bold.

| dpseg | neural
---|---|---
| FI | HU | RO | RU | FI | HU | RO | RU
HMM | 45.6 | 49.9 | 53.5 | 47.1 | 53.4 | 51.2 | 56.6 | 54.9
SHMM | 49.0 | 52.3 | 53.5 | 50.5 | 56.0 | 53.9 | 57.7 | 57.7
H-SHMM | 50.5 | 52.9 | 58.0 | 52.9 | 56.1 | 53.3 | 59.6 | 56.0
True Phones | 87.1 | 83.3 | 88.0 | 85.9 | 68.4 | 63.4 | 75.7 | 68.4

Focusing on the generalization of the presented speech discretization models,
we trained them using four languages from the MaSS dataset. However, we found
that due to the considerable larger average length of the sentences (Table 1),
the VQ-based models produced sequences we were unable to directly apply to UWS
training. This again highlights that these models need some constraining, or
post-processing, in order to be directly exploitable for UWS. Focusing on the
Bayesian models, Table 4 present UWS results. We omit results for RAW, as we
observe the same trend from Table 3. Looking at the results for the four
languages, we again observe competitive results for SHMM and H-SHMM models,
illustrating that these approaches generalize well to different languages.

Regarding the lower results for the languages from MaSS dataset (best result:
59.6) compared to the Mboshi (best result: 64.7), we highlight that the data
for the former comes from read text, and that the utterances correspond to
verses that are considerably longer than sentences (Table 1). This results in
a more challenging setting for UWS and explains the lower results. Lastly, our
results over five languages show that the neural UWS model produces better
segmentation results from discrete speech units than dpseg, which in turn
performs the best with the true phones (topline). This confirms the trend
observed by [9]. The neural UWS models have the advantage of their word-level
aligned translations for grounding the segmentation process, which might be
attenuating the challenge of the task in this noisier scenario, with longer
sequences and more units.

##  6 Conclusion

In this paper we compared five methods for speech discretization, two neural
models (VQ-VAE, VQ-WAV2VEC), and three Bayesian approaches (HMM, SHMM,
H-SHMM), with respect to their performance serving as direct input to the task
of Unsupervised Word Segmentation (UWS) in low-resource settings. Our
motivation for such a study lies on the need of processing oral and low-
resource languages, for which obtaining transcriptions is a known bottleneck
[49]. In our UWS setting, and using five different languages (Finnish,
Hungarian, Mboshi, Romanian and Russian), we find the Bayesian SHMM and H-SHMM
models to perform the best, as they produced concise yet highly exploitable
representations from just few hours of speech. Finally, this work updates [9]
by using more recent speech discretization models, and presenting better UWS
results for Mboshi.

## References

  * [1] J. Glass, “Towards unsupervised speech processing,” in _Information Science, Signal Processing and their Applications (ISSPA)_. IEEE, 2012.
  * [2] A. Jansen _et al._ , “A summary of the 2012 JH CLSP Workshop on zero resource speech technologies and models of early language acquisition,” in _ICASSP_ , 2013.
  * [3] M. Versteegh, X. Anguera, A. Jansen, and E. Dupoux, “The zero resource speech challenge 2015: Proposed approaches and results,” _Procedia Computer Science_ , vol. 81, pp. 67–72, 2016.
  * [4] E. Dunbar, X. N. Cao, J. Benjumea, J. Karadayi, M. Bernard, L. Besacier, X. Anguera, and E. Dupoux, “The zero resource speech challenge 2017,” in _2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2017, pp. 323–330.
  * [5] E. Dunbar, R. Algayres, J. Karadayi, M. Bernard, J. Benjumea, X.-N. Cao, L. Miskic, C. Dugrain, L. Ondel, A. W. Black, L. Besacier, S. Sakti, and E. Dupoux, “The Zero Resource Speech Challenge 2019: TTS Without T,” in _Proc. Interspeech 2019_ , 2019, pp. 1088–1092. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2019-2904
  * [6] E. Dunbar, J. Karadayi, M. Bernard, X.-N. Cao, R. Algayres, L. Ondel, L. Besacier, S. Sakti, and E. Dupoux, “The zero resource speech challenge 2020: Discovering discrete subword and word units,” _arXiv preprint arXiv:2010.05967_ , 2020.
  * [7] L. Besacier, B. Zhou, and Y. Gao, “Towards speech translation of non written languages,” in _Spoken Language Technology Workshop, 2006. IEEE_. IEEE, 2006, pp. 222–225.
  * [8] L. Duong, A. Anastasopoulos, D. Chiang, S. Bird, and T. Cohn, “An attentional model for speech translation without transcription,” in _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2016, pp. 949–959.
  * [9] P. Godard, M. Z. Boito, L. Ondel, A. Bérard, F. Yvon, A. Villavicencio, and L. Besacier, “Unsupervised word segmentation from speech with attention,” in _Proc. Interspeech 2018_ , 2018, pp. 2678–2682. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2018-1308
  * [10] S. Bird, “Sparse transcription,” _Computational Linguistics_ , 2021.
  * [11] E. Dupoux, “Cognitive science in the era of artificial intelligence: A roadmap for reverse-engineering the infant language-learner,” _Cognition_ , 2018\.
  * [12] L. Ondel, L. Burget, and J. Černockỳ, “Variational inference for acoustic unit discovery,” _Procedia Computer Science_ , vol. 81, pp. 80–86, 2016.
  * [13] S. J. Goldwater, “Nonparametric bayesian models of lexical acquisition,” Ph.D. dissertation, Citeseer, 2007.
  * [14] M. Z. Boito, A. Bérard, A. Villavicencio, and L. Besacier, “Unwritten languages demand attention too! word discovery with encoder-decoder models,” in _2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2017, pp. 458–465.
  * [15] L. Ondel, H. K. Vydana, L. Burget, and J. Černocký, “Bayesian Subspace Hidden Markov Model for Acoustic Unit Discovery,” in _Interspeech_ , 2019, pp. 261–265. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2019-2224
  * [16] B. Yusuf, L. Ondel, L. Burget, J. Cernocky, and M. Saraclar, “A hierarchical subspace model for language-attuned acoustic unit discovery,” _arXiv preprint arXiv:2011.03115_ , 2020.
  * [17] A. van den Oord, O. Vinyals, and k. kavukcuoglu, “Neural discrete representation learning,” in _Advances in Neural Information Processing Systems_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017, pp. 6306–6315. [Online]. Available: https://proceedings.neurips.cc/paper/2017/file/7a98af17e63a0ac09ce2e96d03992fbc-Paper.pdf
  * [18] A. Baevski, S. Schneider, and M. Auli, “vq-wav2vec: Self-supervised learning of discrete speech representations,” in _International Conference on Learning Representations (ICLR)_ , 2020.
  * [19] J. Chorowski, R. J. Weiss, S. Bengio, and A. van den Oord, “Unsupervised speech representation learning using wavenet autoencoders,” _IEEE/ACM transactions on audio, speech, and language processing_ , vol. 27, no. 12, pp. 2041–2053, 2019.
  * [20] P. Godard, G. Adda, M. Adda-Decker, J. Benjumea, L. Besacier, J. Cooper-Leavitt, G.-N. Kouarata, L. Lamel, H. Maynard, M. Mueller, A. Rialland, S. Stueker, F. Yvon, and M. Z. Boito, “A very low resource language speech corpus for computational language documentation experiments,” in _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_. Miyazaki, Japan: European Language Resources Association (ELRA), May 2018. [Online]. Available: https://www.aclweb.org/anthology/L18-1531
  * [21] M. Z. Boito, W. N. Havard, M. Garnerin, É. L. Ferrand, and L. Besacier, “Mass: A large and clean multilingual corpus of sentence-aligned spoken utterances extracted from the bible,” _Language Resources and Evaluation Conference (LREC)_ , 2020.
  * [22] H. Kamper, K. Livescu, and S. Goldwater, “An embedded segmental k-means model for unsupervised segmentation and clustering of speech,” in _2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2017, pp. 719–726.
  * [23] O. Räsänen and M. A. C. Blandón, “Unsupervised discovery of recurring speech patterns using probabilistic adaptive metrics,” _arXiv preprint arXiv:2008.00731_ , 2020.
  * [24] C.-y. Lee, T. J. O’Donnell, and J. Glass, “Unsupervised lexicon discovery from acoustic input,” _Transactions of the Association for Computational Linguistics_ , vol. 3, pp. 389–403, 2015.
  * [25] V. Lyzinski, G. Sell, and A. Jansen, “An evaluation of graph clustering methods for unsupervised term discovery,” in _Sixteenth Annual Conference of the International Speech Communication Association_ , 2015.
  * [26] M. Z. Boito, A. Villavicencio, and L. Besacier, “Empirical evaluation of sequence-to-sequence models for word discovery in low-resource settings,” in _Proc. Interspeech 2019_ , 2019, pp. 2688–2692. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2019-2029
  * [27] H. Kamper and B. van Niekerk, “Towards unsupervised phone and word segmentation using self-supervised vector-quantized neural networks,” _arXiv:2012.07551_ , 2020.
  * [28] A. Anastasopoulos and D. Chiang, “Leveraging translations for speech transcription in low-resource settings,” in _Proc. Interspeech 2018_ , 2018, pp. 1279–1283. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2018-2162
  * [29] B. Foley, J. Arnold, R. Coto-Solano, G. Durantin, E. Mark, D. van Esch, S. Heath, F. Kratochvil, Z. Maxwell-Smith, D. Nash _et al._ , “Building speech recognition systems for language documentation: the coedl endangered language pipeline and inference system (elpis),” 2018.
  * [30] A. Michaud, O. Adams, T. A. Cohn, G. Neubig, and S. Guillaume, “Integrating automatic transcription into the language documentation workflow: Experiments with na data and the persephone toolkit,” 2018.
  * [31] J. Strunk, F. Schiel, F. Seifart _et al._ , “Untrained forced alignment of transcriptions and audio for language documentation corpora using webmaus.” in _LREC_ , 2014, pp. 3940–3947.
  * [32] C. Lignos and C. Yang, “Recession segmentation: simpler online word segmentation using limited resources,” in _Proceedings of the fourteenth conference on computational natural language learning_ , 2010, pp. 88–97.
  * [33] S. Goldwater, T. L. Griffiths, and M. Johnson, “A bayesian framework for word segmentation: Exploring the effects of context,” _Cognition_ , vol. 112, no. 1, pp. 21–54, 2009.
  * [34] C. Bartels, W. Wang, V. Mitra, C. Richey, A. Kathol, D. Vergyri, H. Bratt, and C. Hung, “Toward human-assisted lexical unit discovery without text resources,” in _Spoken Language Technology Workshop (SLT), 2016 IEEE_. IEEE, 2016, pp. 64–70.
  * [35] A. Tjandra, S. Sakti, and S. Nakamura, “Speech-to-speech translation between untranscribed unknown languages,” _arXiv:1910.00795_ , 2019.
  * [36] A. Bérard, O. Pietquin, C. Servan, and L. Besacier, “Listen and translate: A proof of concept for end-to-end speech-to-text translation,” _arXiv:1612.01744_ , 2016.
  * [37] Y. Bengio, N. Léonard, and A. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” _arXiv:1308.3432_ , 2013.
  * [38] E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with gumbel-softmax,” in _ICLR_ , 2017.
  * [39] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” in _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Berlin, Germany: Association for Computational Linguistics, Aug. 2016, pp. 1715–1725. [Online]. Available: https://www.aclweb.org/anthology/P16-1162
  * [40] T. Schultz, N. T. Vu, and T. Schlippe, “Globalphone: A multilingual text & speech database in 20 languages,” in _International Conference on Acoustics, Speech and Signal Processing_. IEEE, 2013.
  * [41] S. T. Abate, W. Menzel, and B. Tafila, “An amharic speech corpus for large vocabulary continuous speech recognition,” in _Ninth European Conference on Speech Communication and Technology_ , 2005.
  * [42] H. Gelas, L. Besacier, and F. Pellegrino, “Developments of Swahili resources for an automatic speech recognition system,” in _SLTU - Workshop on Spoken Language Technologies for Under-Resourced Languages_ , Afrique Du Sud, 2012.
  * [43] E. Gauthier, L. Besacier, S. Voisin, M. Melese, and U. P. Elingui, “Collecting Resources in Sub-Saharan African Languages for Automatic Speech Recognition: a Case Study of Wolof,” _LREC_ , 2016.
  * [44] L. Besacier, E. Gauthier, M. Mangeot, P. Bretier, P. Bagshaw, O. Rosec, T. Moudenc, F. Pellegrino, S. Voisin, E. Marsico _et al._ , “Speech technologies for african languages: example of a multilingual calculator for education,” in _Sixteenth Annual Conference of the International Speech Communication Association_ , 2015.
  * [45] M. Johnson and S. Goldwater, “Improving nonparameteric bayesian inference: experiments on unsupervised word segmentation with adaptor grammars,” in _Proc. NAACL-HLT_. Association for Computational Linguistics, 2009, pp. 317–325.
  * [46] P. Godard, G. Adda, M. Adda-Decker, A. Allauzen, L. Besacier, H. Bonneau-Maynard, G.-N. Kouarata, K. Löser, A. Rialland, and F. Yvon, “Preliminary experiments on unsupervised word discovery in mboshi,” in _Proc. Interspeech_ , 2016.
  * [47] P. Godard, “Unsupervised word discovery for computational language documentation,” Ph.D. dissertation, Paris Saclay, 2019.
  * [48] M. Z. Boito, A. Villavicencio, and L. Besacier, “Investigating language impact in bilingual approaches for computational language documentation,” in _Proceedings of the 1st Joint SLTU and CCURL Workshop (SLTU-CCURL 2020)_ , 2020.
  * [49] C. Brinckmann, “Transcription bottleneck of speech corpus exploitation,” in _Proceedings of the Second Colloquium on Lesser Used Languages and Computer Linguistics (LULCL II). Combining efforts to foster computational support of minority languages._ , 2009.