# Detecting Multiple Speech Disfluencies using a Deep Residual Network with
Bidirectional Long Short-Term Memory

###### Abstract

Stuttering is a speech impediment affecting tens of millions of people on an
everyday basis. Even with its commonality, there is minimal data and research
on the identification and classification of stuttered speech. This paper
tackles the problem of detection and classification of different forms of
stutter. As opposed to most existing works that identify stutters with
language models, our work proposes a model that relies solely on acoustic
features, allowing for identification of several variations of stutter
disfluencies without the need for speech recognition. Our model uses a deep
residual network and bidirectional long short-term memory layers to classify
different types of stutters and achieves an average miss rate of 10.03%,
outperforming the state-of-the-art by almost 27%.

Index Terms—  Speech, stuttering, disfluency, deep learning, residual network,
LSTM.

##  1 Introduction

Speech disfluencies are inconsistencies and interruptions in the flow of
otherwise normal speech. Of these speech impediments, stuttering is one of the
most prominent, affecting over 70 million people, about one percent of the
global population [1]. 5-10% of children stutter at some point in their
childhood, with a quarter of these children maintaining their stutters
throughout their entire lives [2]. Common therapy methods often involve
helping the patient monitor and maintain awareness of their speaking patterns
in order to correct them [3]. Moreover, therapeutic success rates have been
reported to be over 80%, especially when detected and dealt with in early
stages [4]. Accordingly, with the recent advances in machine learning, deep
learning, and language/speech processing techniques, developing smart and
interactive tools for detection and therapy is now a real possibility.

In addition to interactive therapy purposes, other applications can be
realized for automated stutter recognition. Fluent speech is crucial and
influential in presentations such as talks and business communications [5].
There are currently a number of applications available to assist speakers in
monitoring and improving their presentation skills. For example, monitoring of
features like volume, rate of speech, and intonation, among others have been
explored in this context [6] [7]. However, detection and quantification of
stutters has not yet been fully explored for such applications.

Despite the many potential applications for automated stutter detection,
little research has been done in this area. This is partially due to the fact
that the notion of detecting and classifying the type and location of stutters
can be a difficult problem, especially when factoring in variables such as
gender, speech rate, accent, and phone-realization [8]. Existing works in the
area mostly rely on automatic speech recognition (ASR) to first convert audio
signals to text, and then utilize language models to detect and identify the
stutters [9] [10] [11]. While this approach has proven effective and achieved
promising results, the reliance on ASR can both be a potential source for
error, as well as an unnecessary additional computational step.

In this paper, we propose a model that directly utilizes audio speech signals
to detect and classify stutters, skipping the ASR step and the need for
language models. Our method uses spectrogram features to train a deep neural
network with residual layers followed by bidirectional long short-term memory
(Bi-LSTM) units to learn to locate and identify different types of stutters.
The overview of our method is presented in Figure 1. Our experiments show the
effectiveness of our approach in generalizing across multiple classes of
stutters while maintaining a high accuracy and strong consistency between
classes.

Fig. 1: Proposed stutter detection system diagram.

##  2 Related Work

Early studies on the topic focused on the feasibility of stutter
differentiation, with training and testing often being performed on a small
set of specific stuttered words. For example, a hidden Markov model (HMM) was
used to create a stutter recognition assistance tool [12]. Testing results
averaged to 96% and 90% accuracy on human and artificially generated stuttered
speech samples respectively for a single pre-determined word [12].

A number of assumptions are often made in order to simplify the problem of
disfluency detection. For example, as different disfluencies vary heavily by
nature, proposed solutions often tackle one single type of stutter (such as
interjections, prolongations, or repetitions) at a time. In [13], for
instance, sound repetition stutters were accurately detected on a small set of
trained words. Another common assumption used for simplification has been to
remove under-represented subject classes (for example based on gender or age)
[13], [14], [11].

As ASR and natural language processing (NLP) has evolved greatly in recent
years, such methods have become increasing popular for the problem of stutter
classification and recognition. One such method incorporated annotations from
speech language pathologist to a word lattice model, improving the baseline
method by a relative 7.5% [9]. Another model using Bi-LSTMs with condition
random fields (CRFs) to get an average F-score of 85.9% across all stutter
types [15]. The current state-of-the-art stutter classification method uses
task-oriented finite state transducer (FST) lattices to detect repetition
stutters with an average 37% miss rate across 4 different types of [11].

##  3 Proposed Network

Our proposed method first generates spectrogram feature vectors from the audio
clips. The spectrograms are then passed through a deep residual neural
network, mapping the spectrogram matrices to a linear vector. These are then
be passed through a bidirectional LSTM to learn the extracted feature
embeddings for different types of stutters. Following the different steps of
our proposed pipeline are described.

###  3.1 Feature extraction

Spectrograms are commonly used features in speech analysis in different
applications ranging from speech recognition to noise cancellation [16] [17].
We use spectrograms as the sole feature for our model. These features are
generated every 10 ms on a 25 ms window for each 4-second audio clip.

###  3.2 Feature embedding layers

We utilize a residual network [18] in our model in order to effectively learn
the stutter-specific features while avoiding issues such as the vanishing
gradient problem. The use of this type of network also allows a deep
architecture (a depth of 18 convolution layers) without overfitting,
especially when considering the relatively small size of the dataset.
Moreover, architectures with residual components have recently shown
considerable promise in speech analysis [19] [20]. In our proposed solution,
each group of 3 convolutional layers is referred to as a convolutional block.
Figure 2 presents the convolutional blocks and the stacked blocks in our mode.
We used batch normalization and ReLu activation functions in the model. Table
1 presents the hyperparameters of our network. The detection task for each
stutter is formulated as a binary problem, with the same architecture
mentioned being used for every disfluency type.

Fig. 2: The architecture of our model.

###  3.3 Recurrent layers

The learned feature embeddings are provided to 2 recurrent layers, each
consisting of 512 bidirectional LSTM units [21]. We utilized LSTM layers as
they have been proven to be effective in classification when dealing with
short sequential data, and are a popular approach in speech and NLP [22]. In
the context of the problem at hand, most stutters tend to be quick and last
only a fraction of the 4-second audio clip that they are contained in.
Therefore the LSTM layers don’t suffer from memory issues [23]. Lastly, the
use of bidirectional LSTMs allow the model to learn both past and future
embeddings, providing further context for our problem. Dropout rates of 0.2
and 0.4 are utilized after each recurrent layer.

##  4 Experiment Setup and Results

###  4.1 Data and annotation

Speech samples were collected from the University College London’s Archive of
Stuttered Speech (UCLASS) Release One [24] dataset, created by the Division of
Psychology and Language Sciences within the university. The dataset contains
samples of monologues from 139 participants, ranging between 8 and 18 years of
age, with known stuttered speech impediments of different severity. Of these
recordings, 25 unique participants were used due to the availability of their
orthographic transcriptions of the monologues.

Table 1: Our ResNet architecture and hyper-parameters.

Module | Input Spectrogram $(257\times T\times 1)$ | Output Size
---|---|---
Conv Module | $\text{Conv2D},\ 7\times 7,\ 64$ | $257\times T\times 64$
$\begin{bmatrix}\text{conv},\ 3\times 3,\ 32\\\ \text{conv},\ 3\times 3,\ 64\\\ \text{conv},\ 3\times 3,\ 64\end{bmatrix}$ | $128\times T/2\times 64$
$\begin{bmatrix}\text{conv},\ 3\times 3,\ 64\\\ \text{conv},\ 3\times 3,\ 128\\\ \text{conv},\ 3\times 3,\ 128\end{bmatrix}$ | $64\times T/4\times 128$
$\begin{bmatrix}\text{conv},\ 3\times 3,\ 128\\\ \text{conv},\ 3\times 3,\ 128\\\ \text{conv},\ 3\times 3,\ 128\end{bmatrix}$ | $64\times T/8\times 128$
$\begin{bmatrix}\text{conv},\ 3\times 3,\ 128\\\ \text{conv},\ 3\times 3,\ 64\\\ \text{conv},\ 3\times 3,\ 64\end{bmatrix}$ | $32\times T/16\times 64$
$\begin{bmatrix}\text{conv},\ 3\times 3,\ 64\\\ \text{conv},\ 3\times 3,\ 64\\\ \text{conv},\ 3\times 3,\ 32\end{bmatrix}$ | $16\times T/32\times 32$
$\begin{bmatrix}\text{conv},\ 3\times 3,\ 32\\\ \text{conv},\ 3\times 3,\ 16\\\ \text{conv},\ 3\times 3,\ 16\end{bmatrix}$ | $8\times T/64\times 16$

Forced time-alignment was used on the audio and transcriptions to generate a
timestamp for each word and stutter spoken [25]. The stutter annotation
approach is similar to previously used methods [26, 27]. We then manually
annotated each recording for one of 7 stutter disfluencies [28]: sound
repetition, word repetition, phrase repetition, revision, interjection, or
prolongation. A description of each type of stutter can be found in Table 2.
We leave out part-word repetition disfluencies as the dataset contained only
few samples of such stutters, preventing our deep learning approach from
properly learning the classification task. Each monologue recording was
segmented into 4-second samples, totalling to 800 labeled audio clips.

Table 2: Types of stutter considered. Label | Stutter Disfluency | Description | Example
---|---|---|---
S | Sound Repetition | Repetion of any phenome | th-th-this
W | Word Repetition | Repetition of any word | why why
PH | Phrase Repetition | Repetition of multiple successive words | I know I know that
R | Revision | Repetition of thought, rephrased mid sentence | I think that- I believe that
I | Interjection | Fabricated word or sounds, added to stall for time | um, uh
PR | Prolongation | Prolonged sounds | whoooooo is it

###  4.2 Implementation details

The model was built using TensorFlow’s Keras API [29]. It was trained with a
learning rate of $10^{-4}$ over 30 epochs, with minimal improvement in results
seen in following epochs. A root means square propagation (RMSProp) optimizer
was used, as well as the softmax loss function. An Nvidia 1080 Ti GPU was used
to perform the training.

Table 3: The percentage miss-rate (MR) and accuracy (Acc) for each stutter type is presented using LOSO validation. |  | S | W | PH | R | I | PR
---|---|---|---|---|---|---|---
Paper | Method | MR | Acc | MR | Acc | MR | Acc | MR | Acc | MR | Acc | MR | Acc
Alharbi et al. [11] | Word Lat. | 60 | – | 0 | – |  |  | 25 | – |  |  | 0 | –
Ours (baseline) | ResNet+LSTM | 20.13 | 83.20 | 3.40 | 95.60 | 4.93 | 95.07 | 3.00 | 96.99 | 25.31 | 80.80 | 6.12 | 93.88
Ours (proposed) | ResNet+Bi-LSTM | 18.10 | 84.10 | 3.20 | 96.60 | 4.46 | 95.54 | 2.86 | 97.14 | 25.12 | 81.40 | 5.92 | 94.08
Table 4: Average accuracy and miss-rate of stutter classification models. Paper | Method | Ave. MR | Ave. Acc
---|---|---|---
Alharbi et al. [11] | Word Lat. | 37% | –
Ours (baseline) | ResNet+LSTM | 10.45% | 90.96%
Ours (proposed) | ResNet+Bi-LSTM | 10.03% | 91.15%

###  4.3 Validation

To rigorously test our proposed model, leave-one-subject-out (LOSO) cross
validation was used: the model was trained on the speech of 24 of the UCLASS
participants, while the last subject’s audio was used for testing. The process
was repeated 25 times, testing the model on a different subject every time.
For this dataset, accuracy (Acc) and miss rate (MR) values have been reported
in prior work. Lastly, it should be noted that we target 6 categories of
stutter disfluencies, as opposed to most prior work where fewer classes are
considered.

###  4.4 Performance and Comparison

The results of our experiments for the UCLASS dataset is summarized in Table
3, where we compare our method to [11]. Additionally, to evaluate the need for
bidirectional LSTM as opposed to a unidirectional LSTM, we compare our results
to a baseline model where a ResNet with LSTM is used instead of our proposed
model. The table shows that our method outperforms the state-of-the-art in
detection of sound repetition and revisions by considerable margins (an
improvements of 41.90% and 22.14% respectively).

The statistical language models and task-oriented word lattices used in other
methods rely heavily on generating a strong orthographic transcriptions for
each speaker. As a result, while these methods struggle with sub-word stutters
such as sound repetition or revision, they perform well for word repetition or
prolongation. This can be observed in Table 3 as [11] performs better than our
method by a small margin (3.2%) for word repetition. Additionally, [11]
performs with a lower miss rate than ours for detection of prolongation
(5.92%). Since our method relies on spectrogram features as opposed to a
language model, some longer utterances can exceed the four-second windows or
suffer from alignment issues, causing those stutters to be misclassified.
Hence our approach produces slightly more false negatives in classification of
prolongation.

As shown in Table 3, our classifier is able to identify interjections with an
accuracy of 81.4%. Many other works on stutter classification tend to avoid
interjection disfluencies as a class, since interjection stutters tend to be
more diverse and lack the consistency of repetition and prolongation stutters,
making them more difficult to classify. While other works such as [30, 31]
were able to robustly detect interjections with mel-frequency cepstral
coefficients (MFCC), small subsets of the UCLASS dataset were used, preventing
us from performing a fair comparison to our model.

The comparison between our proposed method and the baseline approach (using
LSTM instead of bidirectional LSTM) fares similarly, with the bidirectional
LSTM having slightly better or similar results for every class. This lack of
significant difference between the two LSTM variations is most likely due to
the fact that the feature embeddings learned using the ResNet portion of our
pipeline are quite robust, accurately capturing the information required to
represent different stutters. While the difference in bidirectional and
unidirectional LSTMs is marginal, we opt to use the Bi-LSTM approach as the
additional computational cost for Bi-LSTM is not significant.

Table 4 presents the average performance of our model compared to [11] and the
baseline approach. It can be observed that our model achieves an improvement
of 26.97% lower miss rate on the UCLASS dataset over the previous state-of-
the-art. Moreover, as previously shown, our method slightly outperforms the
unidirectional LSTM baseline when averaged across all stutter types. Lastly,
Figure 3 shows the performance of our method for different stutter types
against different training epochs. It can be seen that after approximately 20
epochs, our model reaches a steady-state, indicating stable learning of
disfluency-related features throughout the learning phase.

Fig. 3: Average training accuracy for the considered types of stutter.

##  5 Conclusion and Future Work

We present a method for detection and classification of different types of
stutter disfluencies. Our model utilizes a residual network and bidirectional
LSTM units trained using input spectrogram features calculated from labeled
audio segments of stuttered speech. Six classes of stutter were considered in
this paper: sound repetition, word repetition, phrase repetition, revision,
interjection, and prolongation. Investigations show that our method performs
robustly across all classes and performs with very high average accuracy and
low average miss rate, achieving state-of-the-art with a significant
improvement over previous the previous state-of-the-art for stutter detection.

In future work, building upon the current model, we will conduct research on
multi-class learning of different stutter disfluencies. As multiple stutter
types may occur at once (e.g. ’I went to uh to to uh to’), this approach may
result in more robust classification of stutters.

##  6 Acknowledgements

The authors would like to thank Prof. Jim Hamilton for his support and
valuable feedback throughout the course of this of this work.

## References

  * [1] Stuttering Foundation,  “Stuttering foundation website,” https://www.stutteringhelp.org, 2019\.
  * [2] National Institute on Deafness and Other Communication Disorders,  “Stuttering,” https://www.nidcd.nih.gov, 2019.
  * [3] B. Guitar,  Stuttering: An Integrated Approach to Its Nature and Treatment,  Lippincott Williams & Wilkins, 4 edition, 2014.
  * [4] T. Saltuklaroglu and J. Kalinowski,  “How effective is therapy for childhood stuttering? dissecting and reinterpreting the evidence in light of spontaneous recovery rates,”  International Journal of Language & Communication Disorders, vol. 40, pp. 359–374, 2004.
  * [5] Sherwyn p Morreale, Michael m Osborn, and Judy c Peaston,  “Why communication is important: A rationale for the centrality of the study of communication,”  Journal of the Association for Communication Administration, vol. 29, pp. 1–25, 2000.
  * [6] H. Trinh, R. Asadi, D. Edge, and T. Bickmore,  “Robocop: A robotic coach for oral presentations,”  ACM Conference on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 1, no. 2, pp. 27, 2017.
  * [7] S. A. Kotz, M. Meyer, K. Alter, M. Besson, D. Y. von Cramon, and A. D. Friederici,  “On the lateralization of emotional prosody: an event-related functional mr investigation,”  Brain and Language, vol. 86, no. 3, pp. 366–376, 2003.
  * [8] H. Petkar,  “A review of challenges in automatic speech recognition,”  International Journal of Computer Applications, vol. 151, no. 3, pp. 23–26, 2016.
  * [9] P. A. Heeman, R. Lunsford, A. McMillin, and J. S. Yaruss,  “Using clinician annotations to improve automatic speech recognition of stuttered speech,”  INTERSPEECH, pp. 2651–2655, 2016.
  * [10] S. Alharbi, M. Hasan, A.J.H. Simons, S. Brumfitt, and P. Green,  “Detecting stuttering events in transcripts of children’s speech,”  International Conference on Statistical Language and Speech Processing, pp. 217–228, 2017.
  * [11] S. Alharbi, M. Hasan, A.J.H. Simons, S. Brumfitt, and P. Green,  “A lightly supervised approach to detect stuttering in children’s speech,”  INTERSPEECH, pp. 3433–3437, 2018.
  * [12] T. Tan, Helbin-Liboh, A. K. Ariff, C. Ting, and S. Salleh,  “Application of malay speech technology in malay speech therapy assistance tools,”  International Conference on Intelligent and Advanced Systems, pp. 330–334, 2007.
  * [13] H.C.Nagaraj K.M Ravikumar, R.Rajagopal,  “An approach for objective assessment of stuttered speech using mfcc features,”  Digital Signal Processing Journal, vol. 9, pp. 19–24, 2019.
  * [14] L. S. Chee, O. C.Ai, M. Hariharan, and S. Yaacob,  “Automatic detection of prolongations and repetitions using lpcc,”  International Conference for Technical Postgraduates, pp. 1–4, 2009.
  * [15] V. Zayats, M. Ostendorf, and H. Hajishirzi,  “Disfluency detection using a bidirectional lstm,”  INTERSPEECH, pp. 2523–2527, 2016.
  * [16] B. E. D. Kingsbury, N. Morgan, and S. Greenberg,  “Robust speech recognition using the modulation spectrogram,”  Speech Communication, vol. 25, pp. 117–132, 1998.
  * [17] E. Szabelska and A. Kruczyńska,  “Computer-based speech analysis in stutter,”  Applied Computer Science, vol. 9, no. 2, pp. 34–42, 2013.
  * [18] K. He, X. Zhang, S. Ren, and J. Sun,  “Deep residual learning for image recognition,”  IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778, 2016.
  * [19] W. Xiong, L. Wu, F. Alleva, J. Droppo, X. Huang, and A. Stolcke,  “The microsoft 2017 conversational speech recognition system,”  in IEEE International Conference on Acoustics, Speech and Signal Processing, 2018, pp. 5934–5938.
  * [20] A. Hajavi and A. Etemad,  “A deep neural network for short-segment speaker recognition,”  INTERSPEECH, pp. 2878–2882, 2019.
  * [21] M. Schuster and K. K. Paliwal,  “Bidirectional recurrent neural networks,”  IEEE Transactions on Signal Processing, vol. 45, no. 11, pp. 2673–2681, 1997.
  * [22] S. Hochreiter and J. Schmidhuber,  “Long short-term memory,”  Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997.
  * [23] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov,  “Dropout: a simple way to prevent neural networks from overfitting,”  The Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014.
  * [24] P. Howell, S. Davis, and J. Bartrip,  “The university college london archive of stuttered speech (uclass),”  Journal of Speech, Language, and Hearing Research, vol. 52, pp. 556–569, 2009.
  * [25] T. J. Hazen,  “Automatic alignment and error correction of human generated transcripts for long speech recordings,”  Ninth International Conference on Spoken Language Processing, 2006\.
  * [26] E. Yairi and N. G. Ambrose,  “Early childhood stuttering i: persistency and recovery rates,”  Journal of Speech, Language, and Hearing Research, vol. 42, 1999\.
  * [27] F. S. Juste and C. R. Furquim de Andrade,  “Speech disfluency types of fluent and stuttering individuals: Age effects,”  International Journal of Phoniatrics, Speech Therapy and Communication Pathology, vol. 63, 2011.
  * [28] J. S. Yaruss,  “Clinical measurement of stuttering behaviors,”  Contemporary Issues in Communication Science and Disorders, vol. 24, 1997.
  * [29] François Chollet et al.,  “Keras,” 2015.
  * [30] P. Mahesha and D. S. Vinod,  “Automatic segmentation and classification of dysfluencies in stuttering speech,”  Proceedings of the Second International Conference on Information and Communication Technology for Competitive Strategies, pp. 1–6, 2016.
  * [31] P. Mahesha and D. S. Vinod,  “Lp-hillbert transform based mfcc for effective discrimination of stuttering dysfluencies,”  International Conference on Wireless Communications, Signal Processing and Networking, pp. 2561–2565, 2017.