# Seq2seq for Automatic Paraphasia Detection in Aphasic Speech

Matthew Perez, Duc Le, Amrit Romana, Elise Jones, Keli Licata, Emily Mower
Provost

###### Abstract

Paraphasias are speech errors that are often characteristic of aphasia and
they represent an important signal in assessing disease severity and subtype.
Traditionally, clinicians manually identify paraphasias by transcribing and
analyzing speech-language samples, which can be a time-consuming and
burdensome process. Identifying paraphasias automatically can greatly help
clinicians with the transcription process and ultimately facilitate more
efficient and consistent aphasia assessment. Previous research has
demonstrated the feasibility of automatic paraphasia detection by training an
automatic speech recognition (ASR) model to extract transcripts and then
training a separate paraphasia detection model on a set of hand-engineered
features. In this paper, we propose a novel, sequence-to-sequence (seq2seq)
model that is trained end-to-end (E2E) to perform both ASR and paraphasia
detection tasks. We show that the proposed model outperforms the previous
state-of-the-art approach for both word-level and utterance-level paraphasia
detection tasks and provide additional follow-up evaluations to further
understand the proposed model behavior.

###### Index Terms:

paraphasia detection, multitask learning, seq2seq, aphasia, speech analysis.

##  I Introduction

Aphasia is a common language disorder that occurs as a result of damage to the
brain and can ultimately impair the communication abilities (both expressive
and receptive) of an individual. Aphasia affects over two million people in
the United States and nearly 180,000 acquire aphasia each year following a
medical event such as a traumatic brain injury or stroke [1]. Aphasia can
manifest in a variety of ways that can negatively impact speech production.
One example of this is through increased speech errors, such as paraphasias.

Paraphasias are a type of communication error and identifying paraphasias can
aid clinicians in characterizing an individual's aphasia and developing
targeted intervention strategies [2]. In this work, we focus on identifying
phonemic and neologistic paraphasias [3, 4].

  * •

_phonemic_ paraphasias involve substituting, omitting, or rearranging phonemes
(i.e. `shut' $\rightarrow$ `zut')

  * •

_neologistic_ paraphasias involve substituting a nonsensical word in place of
the target word (i.e. `bottle' $\rightarrow$ `flibber')

Clinical research has highlighted the impact that accurate paraphasia
detection plays in predicting recovery patterns and guiding treatment planning
[5, 6]. Importantly, automated tools that effectively identify the presence of
paraphasias in an individual's spoken output can ultimately allow for more
efficient and consistent assessment procedures for language disorders like
aphasia.

Previous automatic paraphasia detection work has used a pipeline consisting of
an automatic speech recognition (ASR) model, transcript-derived feature
extraction, and then a paraphasia classification model [7]. Although the
authors demonstrated the feasibility of automatic paraphasia detection, the
pipeline required three separate processes that had to be trained/computed
independently. In this work, we propose learning all these pipeline components
within a single model that is trained E2E.

We present a novel framework for automatic paraphasia detection that uses a
sequence-to-sequence (seq2seq) model to perform both ASR and paraphasia
detection tasks. We first acknowledge that paraphasia detection systems are
largely dependent on ASR performance and evaluate several E2E ASR
architectures (including seq2seq). We then evaluate the proposed seq2seq model
on a paraphasia detection task and compare against the previous state-of-the-
art (SOTA) approach. We investigate the effects of single-task learning (STL)
and multi-task learning (MTL) objectives on the proposed seq2seq model and
further analyze model performance using additional word-level paraphasia
detection metrics such as temporal distance and time tolerant recall to
supplement our findings. Lastly, we analyze the effect of tokenizer size on
paraphasia detection and present some example transcriptions from the model to
highlight some of the strengths and limitations of the proposed approach. The
research contributions of this paper are:

  * •

An evaluation of the seq2seq architecture for automatic aphasic speech
recognition.

  * •

The proposed seq2seq model for E2E paraphasia detection.

  * •

Assessment of the impact of pretraining on ASR and paraphasia detection tasks.

  * •

The effects of single-task (STL) and multi-task (MTL) learning objectives on
the proposed seq2seq paraphasia detection model.

  * •

An analysis of hyperparameters such as tokenizer size on the proposed model
performance.

  * •

An analysis and discussion of sampled output from the proposed paraphasia
detection model.

##  II Background

###  II-A Aphasia Assessment

Traditional aphasia assessment practices involve tasks designed to elicit
spontaneous speech-language samples, such as those involving descriptions of
multi-action pictures or responses to conversational questions [8, 9, 10].
These samples are obtained, transcribed, and analyzed by a speech-language
pathologist (SLP) and the resulting analyses can be used for aphasia
classification [11, 12], treatment planning [13], and progress monitoring
[14]. Ultimately, both transcription and analysis can consume a lot of a SLP's
already limited time. Machine learning systems that can automatically analyze
aphasic speech can greatly aid SLPs with the aphasia assessment process and
allow for more time to be devoted to patient contact and care. One form of
analysis that can be improved with machine learning is automatic paraphasia
detection. Clinical works have shown that paraphasias are a useful biomarker
in characterizing different types of aphasia and ultimately assessing aphasia
severity [2, 15, 6]. Ultimately, identifying certain types of paraphasia can
greatly aid SLPs with aphasia assessment efforts and the development of
targeted intervention strategies.

###  II-B Aphasia Treatment Planning

Improving assessment procedures by introducing the use of automated tools can
also facilitate treatment planning and support a SLP's selection of the most
appropriate therapy objectives for each PWA. Treatment for PWA can include a
variety of speech-language therapy approaches targeting functional
communication across language domains, including spoken language expression,
spoken language comprehension, reading comprehension, and written expression
[16, 17, 18, 19, 20]. Traditionally, speech therapy involves meeting regularly
with a SLP to help manage speech-language difficulties. However, given the
rise of ubiquitous computing and smart devices, some clinical works have
explored the use of digital technology to supplement treatment and
rehabilitation plans, particularly when traditional in-person therapy services
are limited due to geographic or financial constraints.

Ballard et al. investigated app-based speech therapy for those with apraxia of
speech and aphasia [19]. The app uses ASR to recognize input speech related to
a naming task and provide feedback to the user. Results showed that
participants exhibited increased word production accuracy over time. However,
one limitation of this app is that there were no means for providing feedback
regarding paraphasic errors, which could reinforce error patterns and/or
contribute to a PWA's limited awareness of errors [21]. For remote speech-
language therapy applications, feedback from automatic paraphasia detection
can be useful in guiding user-driven intervention. As SLP's tailor treatment
planning to the needs of PWA's, it's important the benefits of using app-based
technology with automatic error detection methods to supplement traditional
speech therapy approaches.

###  II-C Paraphasia Detection

Several works have demonstrated the ability to identify paraphasias from text
input [5, 22]. However, a limitation of these approaches is that they rely on
manual transcripts and are ultimately not fully automatic when considering
speech as an input signal.

One work, by Le et al. has investigated a fully automatic pipeline for
paraphasia detection [7], which relied on a hidden markov model-based
Multitask Learning Bidirectional Long Short-Term Memory (MTL-BLSTM) acoustic
model to first produce transcriptions. From these transcriptions, features
such as pronunciation, word and phone durations, and phoneme posterior
distance are extracted and used to train a donwstream paraphasia classifier.
The authors used two evaluation schemes: the first is augmented word error
rate (AWER), which is a word-level metric used to evaluate both transcription
and paraphasia label. The second is the average F1-score between the negative
and positive paraphasia classes, which is computed at the utterance-level.
Figure 1 shows an example of how a transcript is combined with paraphasia
labels for AWER evaluation. The authors present the first results for
automatic paraphasia detection on this set achieving AWERs of 53.5, 54.2, and
47.8 and F1 scores of 0.594, 0.611, and 0.604 for phonemic+neologistic,
phonemic, and neologistic paraphasia detection, respectively. To the best of
our knowledge, this work by Le et al. represents the closest approach to ours
for automatic paraphasia detection. In this paper, we focus on improving
automatic paraphasia detection using a novel seq2seq model that learns both
ASR and paraphasia detection tasks E2E.

$\begin{array}[]{l l}\text{CHAT Transcript:}&\text{I have efezi@u [: aphasia]
[* n:k]}\\\ \text{Post-processed Transcript:}&\text{I have efezia}\\\
\text{Paraphasia Labels:}&\text{0 \hskip 2.0pt 0 \hskip 15.0pt 1}\\\
\text{AWER Transription:}&\text{I/0 have/0 efezia/1}\end{array}$ Figure 1:
Example showcasing how text and paraphasia labels are concatenated for AWER
evaluation. Paraphasia labels are binary with 0=non-paraphasia and
1=paraphasia.

###  II-D Aphasic Speech Recognition

ASR often represents an important first step before automatic aphasic analysis
such as paraphasia detection and precious works have shown that poor ASR
transcription can negatively impact downstream analyses [23, 24]. With this in
mind, training ASR models that perform well on aphasic speech is critical for
automatic paraphasia detection. In this section, we review ASR research
focused on improving aphasic speech recognition. Previous works have focused
on overcoming challenges such as abnormal speech patterns, high speaker
variability, and data scarcity [25, 24, 26, 27]. These challenges make it
difficult to apply or adapt traditional off-the-shelf systems due to the data
mismatch that exists between speech from healthy controls, which is typically
used to train off-the-shelf systems, and disordered speech [28]. With this in
mind, many researchers opt to train in-domain ASR models for aphasic speech
recognition. However, training custom models using supervised learning
techniques is also difficult due to the aformentioned challenges and the
scarcity of labeled data for PWAs [24, 23, 29, 30, 31].

Some earlier aphasic research used traditional ASR model frameworks, which
have separate acoustic, language, and pronunciation models. These works
focused on improving the acoustic model which consisted of a hidden markov
model, deep neural network (HMM-DNN) [32, 24, 33, 26]. Previous work by Le et
al. has focused on using speaker-embeddings such as i-vectors, out-of-domain
training, and a Multitask Learning Bidirectional Long Short-Term Memory (MTL-
BLSTM) architecture to improve aphasic speech recognition [32, 24]. The BLSTM
layers of this model capture both forward and backward dependencies in the
input sequences, allowing for better context modeling. Additionally, multitask
learning of both senone and monophone labels allow for additional model
regularization and improved performance.

End-to-end (E2E) ASR systems focus on modeling word sequences directly from
acoustic frames without the need for an HMM. Additionally, these approaches
learn the traditional components of acoustic, language, and pronunciation
models all together in a single architecture. Some examples of E2E models are
Connectionist Temporal Classification (CTC) models or sequence-to-sequence
(seq2seq) models. Generally, these models are transformer-based and have been
pretrained on vast amounts of speech data before they are finetuned E2E for
aphasic speech recognition. For example, Torre et al. explored fine-tuning a
pretrained ASR model by adding an extra layer and optimizing with CTC loss for
multilingual Aphasic speech recognition. The pretrained model they used is
Wav2Vec2-XLSR and the authors were able to show that this approach
outperformed existing HMM-DNN approaches [29] on the Spanish and English
corpora for ApashiaBank.

Seq2seq is another approach for E2E ASR model training and involves an
encoder-decoder architecture. Another approach is to use a seq2seq ASR model,
which ignores the frame-independence assumption made by traditional HMM or CTC
learning approaches and is able to optimize word error rate more directly
[34]. The seq2seq approach models the speech recognition task as a machine
translation task, and, especially with the use of transformers, has shown much
success on traditional ASR benchmarks [35, 36]. A small body of work has
started to investigate seq2seq models for aphasic speech recognition [37, 38].
Peng et al. proposed an E-branchformer that achieves strong ASR performance
across a variety of datasets [38]. Tang et al. illustrated how seq2seq
frameworks can benefit from leveraging pretrained, self-supervised models. In
their work, they finetune a seq2seq model with a pretrained WavLM model and
perform multitask learning with ASR and aphasia detection [37].

In this work, we evaluate some of the methods explored above and investigate
the role of pretraining on aphasic speech recognition systems as a means of
improving automatic paraphasia detection. We then show how a seq2seq model can
be trained to consider both ASR and paraphasia detection tasks.

##  III Dataset

We use the AphasiaBank dataset, which is a collection of multimedia data for
the study of communication in aphasia [39]. The database is collected from
multiple institutions and contains data in several languages, however, we
specifically use the English audio data from the Protocol and the Scripts
(non-protocol) sets. The Protocol dataset is composed of both Aphasic and
Control data collected from 26 different institutions. The speech data
consists of a free-form discussion with an interviewer along with several
discourse tasks including free speech, picture descriptions, story narratives,
and procedural discourse. The Scripts dataset is composed of Aphasic data from
the Fridriksson subset and contains speech data consisting of read scripts on
different topics (advocacy, eggs, vast, and weather). The Scripts dataset is
particularly useful for paraphasia detection due to its increased frequency of
paraphasias, compared to the Protocol dataset, with phonemic and nelogistic
paraphasias representing 12% and 6.5% of all words. Participants were assessed
using the Western Aphasia Battery - Revised (WAB-R), which is a standard test
used for assessing aphasia [40]. We group PWAs into severity classes based on
the WAB-R Aphasia Quotient (AQ) following a similar approach to that outlined
in [41, 24] based on mild, moderate, severe, and very severe. Table I contains
the total time and the percentage of paraphasias for each dataset and severity
class.

##  IV Transcript Pre-processing

All utterances were transcribed following the CHAT transcription format and
included timestamps for both participant and interviewer speech segments [42].
We isolate participant speech and discard utterances that have labelled
unintelligible speech or overlapping speech between participant and clinician.

We preprocess both the Protocol and Scripts transcripts following the approach
described in [7]. Non-word phonological errors are transcribed in the
International Phonetic Alphabet (IPA) format and each IPA pronunciation is
heuristically mapped to a sequence of phones. We add additional heuristics
that convert this phonemic sequence into a pseuodo-word target. Figure 1 shows
an example of a non-word phonological error `aphasia' becoming `efezia'.
Lastly, we normalize the transcripts to lowercase and remove punctuations.

Dataset | Severity | Time (hrs) |  Paraphasia Token Representation (%)
---|---|---|---
Protocol | Control | 38.3 | 0.00
Mild | 36.0 | 0.01
Moderate | 19.3 | 0.02
Severe | 3.3 | 0.03
Very Severe | 0.5 | 0.08
| Total | 97.4 | 0.03
Scripts | Total | 3.0 | 0.24
TABLE I: Dataset Info in Hours Figure 2: Model Architecture for seq2seq speech
recognition. Our best results use a pretrained WavLM model as the encoder.

##  V Methods

###  V-A ASR Models

ASR is a critical first step in the automatic paraphasia detection pipeline.
With this in mind, we evaluate a variety of different ASR models, which
include HMM-based, encoder-only, and encoder-decoder (i.e. seq2seq)
architectures. We also, investigate the impact of pretraining on E2E ASR
models (i.e. encoder-only and seq2seq) using Wav2Vec2, HuBERT, Whisper, and
WavLM models.

Hybrid HMM-BLSTM
Hybrid HMM-DNN acoustic models are considered a more traditional tool for ASR
modeling. Le et al. used a variant of an HMM-DNN, referred to as the MTL-
BLSTM, to transcribe aphasic speech before paraphasia detection. The MTL-BLSTM
relied on a manually curated lexicon that is based on the CMU dictionary and
contains 39 phones. The MTL-BLSTM uses mel-filterbank coefficient (MFB)
features augmented with utterance-level i-vectors and is optimized to predict
both monophone and senone labels. This method decodes the MTL-BLSTM output
with a trigram language model that was built on the training set. We implement
this approach using pytorch-kaldi [43].

Wav2Vec2.0
Wav2Vec2.0 consists of a CNN-based speech feature encoder, a quantization
module, and a transformer network [44]. The input to Wav2Vec2.0 is raw audio,
which is passed through the feature encoder with a receptive field of 25ms and
a stride of 20ms. The output of the CNN is then passed to the transformer
network which learns contextualized speech representations. Wav2Vec2.0 was
pretrained to learn meaningful representations through a two-step process of
extracting pseudo-targets from the audio via a quantization module and then
learning to predict these targets with added noise (i.e. masked prediction).
In our work, we use a large Wav2Vec2.0 model that consists of 317M parameters
and was pretrained and finetuned on 960 hours of Libri-light and
Librispeech111https://huggingface.co/facebook/wav2vec2-large-960h-lv60-self.

HuBERT
HuBERT builds on the initial Wav2Vec2.0 pretraining process. Rather than
learning the pseudo-targets while training, these pseudo-targets are created
prior to training via k-mean clustering. Additionally, HuBERT uses embeddings
from the intermediate layers of the BERT encoder to generate better targets
throughout the learning process [45]. As a result, HuBERT has been shown to be
on par with or better than Wav2Vec2.0 for ASR benchmark tasks [46]. In our
experiments, we use a large HuBERT model that consists of 317M parameters and
has been pretrained on LibriLight and finetuned on 960h of Librispeech
data222https://huggingface.co/facebook/hubert-large-ls960-ft.

WavLM
WavLM extends the HuBERT framework by focusing on data augmentation during
pretraining in order to improve speaker representation learning and spoken
content modeling [47]. This is achieved by introducing denoising as an
objective learning task in addition to masked speech prediction. Additionally,
WavLM uses a gated relative position bias for the Transformer structure to
better capture the sequence ordering of speech input. As a result, WavLM
generalizes well to many downstream tasks and is currently the top-ranked
model on the SUPERB benchmark, which is designed to evaluate universal shared
representations for a diverse set of speech processing tasks [46]. The
implementation we use consists of 317M parameters and was trained on 60k hours
of LibriLight, 10k hours of GigaSpeech, and 24k hours of
VoxPopuli333https://huggingface.co/microsoft/wavlm-large.

Whisper
The Whisper architecture consists of an encoder-decoder Transformer that is
trained in an E2E fashion [48]. Audio representations are passed to the
encoder and the decoder is in charge of predicting text with the inclusion of
special tokens designed to direct learning for language identification,
phrase-level alignment, and multilingual transcription. Whisper is trained on
a large and diverse dataset and as a result can generalize well to unseen
datasets compared to other SSL models. OpenAI notes that Whisper makes 50%
fewer errors compared to other self-supervised learning (SSL) models for zero-
shot learning tasks. We consider Whisper in a zero-shot setting (off-the-shelf
application) in our experiments, in addition to fine-tuning this model. The
base model we use is trained on 680k hours of labeled data and has 74M
parameters444https://huggingface.co/openai/whisper-base. For our off-the-shelf
model we use the Whisper-X framework with a whisper base model [49].

Encoder-only
We investigate fine-tuning the pretrained models listed above in an E2E
fashion using encoder-only or encoder-decoder (seq2seq) architectures. For
encoder-only finetuning, we add three fully connected layers of sizes 1024,
1024, and 28 to the pretrained model, where 28 is the number of character
targets for English. We then train with CTC loss, which learns the alignment
between audio frames and characters [50].

Encoder-Decoder
For seq2seq finetuning, we use an encoder-decoder architecture where the
encoder is a $N$ layer Transformer, depending on the pretrained model used. If
pretraining is not used, then $N$=12. The decoder is a six-layer transformer,
following the default seq2seq architecture used in [36]. We use a
SentencePiece tokenizer with a unigram tokenization scheme, which is
instrumental in breaking down input text into manageable subword units,
facilitating more granular and accurate language processing. We use a token
size of 500 due to the constrained nature of the AphasiaBank dataset. The
seq2seq model is optimized using a joint CTC-attention loss criterion [51]
which is a weighted sum of CTC and CE loss following equation 1 where $\alpha$
in our experiments is set to a default value of 0.3.

$\mathcal{L}=\alpha*\mathcal{L}_{CTC}+(1-\alpha)\mathcal{L}_{CE}$ (1)

For both encoder-only and encoder-decoder approaches, we use SpecAugment [52]
to create perturbations in the time domain by resampling utterances at
different rates of [0.8, 0.9, 0.95, 1.0, 1.05, 1.1, 1.2]. Work by Green et.
al. has suggested that time masking within SpecAugment is more effective for
disordered speech recognition than frequency masking[27].

###  V-B Paraphasia Detection Model

The proposed paraphasia detection model is a transformer-based encoder-decoder
architecture and is outlined in Figure 2. The encoder output ($H_{enc}$) is
fed to a CTC layer. $H_{enc}$ is also fed to the decoder to predict both
subword token $y_{t}$ and paraphasia label $p_{t}$ in a sequence. We employ a
loss function based on the negative log-likelihood to compute the cross-
entropy loss for ASR prediction and paraphasia classification. Both cross-
entropy losses are summed together for a total loss shown in equation 2.

$\displaystyle Loss=-logP(y_{t}|y_{1},y_{2},...,y_{t-1},x)+$ (2)
$\displaystyle-logP(p_{t}|y_{1},y_{2},...,y_{t-1},x)$

During the training phase, subword paraphasia labels are generated by
assigning the word-level paraphasia label to each of the word's constituent
subwords. In the inference phase, we aggregate the subword paraphasia labels
for each word using an 'OR' function, meaning that if any subword of a word is
labeled as a paraphasia, the entire word is classified as a paraphasia.

For our proposed seq2seq model, we investigate both single-task learning (STL)
and multitask learning (MTL) objectives. The STL objective represents a
paraphasia detection pipeline similar to [7] which first optimizes for ASR and
then paraphasia detection. The MTL objective represents learning both of these
tasks simultaneously in the network.

For the STL models, we optimize for ASR-only on the Protocol dataset, then
finetune using an ASR-only objective on the Scripts dataset for five epochs
followed by a paraphasia detection-only objective for the remainder of the
training process. For the MTL models, we first optimize for both ASR and
paraphasia detection tasks on the Protocol dataset. This involves upsampling
the utterances with paraphasias to ensure balanced paraphasia representation
in our mini-batches since the protocol dataset has a very limited number of
paraphasias compared to the Scripts dataset. During this first tuning step on
the Protocol dataset we combine both phonemic and neologistic paraphasias into
a single class so that the model can learn to detect both types of
paraphasias. We then finetune using the Scripts dataset and optimize for both
ASR and paraphasia detection tasks. The code for our proposed model can be
found at the following github repository
555https://github.com/matthewkperez/speechbrain_Paraphasia_Detection.

##  VI Experiments

###  VI-A Aphasia ASR

We use the Protocol dataset and partition the data into speaker-independent
train, dev, and test sets using 70%, 10%, and 20% respectively. We focus
exclusively on the participant speech segments. We remove utterances that are
less than 0.75s due to poor alignment and utterances that are greater than 10s
due to the hardware constraints of training Transformer-based models. All
audio data are downsampled to 16kHz data.

We evaluate model performance using word error rate (WER) and provide a
detailed breakdown across speaker severity. Both the hybrid HMM-DNN and CTC
models make use of a trigram language model based on the training set. Before
decoding, we perform a hyperparameter sweep for controlling smoothing and
back-off where $alpha$=[0.4,0.5,0.6] and $beta$=[0.8,1.0,1.2]. For seq2seq
model decoding, we sweep over the $ctcweight$=[0.2,0.3,0.4]. These
hyperparameter sweeps were performed on the validation set and the optimal
values were used for decoding on the test set.

###  VI-B Paraphasia Detection

For the task of paraphasia detection, we compare our work against that of [7]
and investigate the E2E model training for automatic paraphasia detection. We
follow the same training and evaluation scheme as [7] for consistency. For all
models, we follow a two-step training approach that consists of first training
on the much larger Protocol dataset, and second finetuning to the domain of
Scripts dataset. We use the same Protocol dataset partitioning outlined in
section VI-A. The Scripts dataset is split into speaker-independent folds and
model training and evaluation is performed in a leave-one-subject-out fashion
following previous work. We aggregate our results across all test folds and
then compute evaluation metrics following the work of Le et. al. [7].
Paraphasia detection is treated as a binary classification task, so we train
independent models for detecting phonemic (p), neologistic (n), or phonemeic-
neologistic (p-n) paraphasias. We present the paraphasia detection results at
both the word- and utterance-levels on the Scripts dataset.

Figure 3: Example of $\omega$=1 is used when computing TTR for a misaligned
AWER transcript. The evaluation of the paraphasia label for the word `efezia'
with an $\omega$=1 results in a TP whereas an $\omega$=0 results in a FN

Evaluation Metrics
For word-level evaluation, we use augmented word error rate (AWER) which was
previously used by Le et al. [7]. We create ground truth and predicted AWER
transcripts by appending the paraphasia labels to each word in the
corresponding transcripts (see Figure 1 for an example), and then calculate
AWER as the WER between the ground truth and predicted AWER transcripts. AWER
represents a high evaluation standard since it requires both the word and
paraphasia label to be correctly recognized. A limitation of this metric is
that it does not allow us to focus the evaluation on just word-level
paraphasia detection. To focus our evaluation on just the paraphasia detection
performance, we can consider isolating paraphasia labels from the AWER
transcripts and comparing the ground truth sequence $Y$=[$y_{1}$,
$y_{2}$,…,$y_{G}$], $y\in\\{0,1\\}$ to the predicted sequence,
$\hat{Y}$=[$\hat{y_{1}}$,$\hat{y_{2}}$,…,$\hat{y_{P}}$],
$\hat{y}\in\\{0,1\\}$. We include two additional word-level metrics that are
designed for evaluating fixed-length sequences like word-level paraphasias.

The first metric is temporal distance (TD) [53], which is the sum of the
target-to-candidate (TTC) and candidate-to-target (CTT) distances, where a
lower TD indicates a better score. The TTC, outlined in equation 4, is the sum
of distances from each target paraphasia ($y_{i}$) to the closest predicted
paraphasia ($\hat{y}_{j}$). The CTT, outlined in equation 5, is the sum of
distances from each predicted paraphasia ($\hat{y}_{j}$) to the closest target
paraphasia ($y_{i}$). Effectively, TTC punishes false negatives (FN) and CTT
punishes false positives (FP). As a result, the effect on TD is that predicted
paraphasias that are close in proximity to target paraphasias will result in a
good metric.

$\text{TD}_{\downarrow}(Y,\hat{Y})=TTC(Y,\hat{Y})+CTT(Y,\hat{Y})$ (3)

$\text{TTC}(Y,\hat{Y})=\sum_{i=0}^{G}y_{i}*\left(\min_{0\leq
j<P}\\{|i-j|:y_{i}==\hat{y}_{j}\\}\right)$ (4)

$\text{CTT}(Y,\hat{Y})=\sum_{j=0}^{P}\hat{y_{j}}*\left(\min_{0\leq
i<G}\\{|i-j|:y_{i}==\hat{y}_{j}\\}\right)$ (5)

The second metric is time tolerant recall (TTR) following equation 6, which
involves computing true positives (TP), false negatives (FN) within a given
buffer or window [54]. Figure 3 illustrates an example of this, where the
window size is 1. Equation 7 and equation 8 show how TP and FN are computed,
where $\omega$ is the window size and $I$(c) evaluates to 1 if and only if the
condition $c$ is true.

$\text{TTR}(Y,\hat{Y})=\frac{\text{TP}(Y,\hat{Y})}{\text{TP}(Y,\hat{Y})+\text{FN}(Y,\hat{Y})}$
(6)

$\text{TP}(Y,\hat{Y})=\sum_{i=0}^{G}y_{i}*I\left(\sum_{j=\max(0,i-\omega)}^{\min(N,i+\omega)}\hat{y}_{j}>0\right)$
(7)

$\text{FN}(Y,\hat{Y})=\sum_{i=0}^{G}y_{i}*\left(1-I\left(\sum_{j=\max(0,i-\omega)}^{\min(P,i+\omega)}\hat{y}_{j}>0\right)\right)$
(8)

We also perform utterance-level evaluations using the average F1-score of both
the control and paraphasia classes, which was used in prior works [7].

Model | Mild | Moderate | Severe | Very Severe
---|---|---|---|---
MTL-BLSTM [24] | 39.4 | 42.8 | 49.7 | 55.3
Encoder-only
Whisper-small* | 32.6 | 40.6 | 43.7 | 65.3
Whisper-small | 31.1 | 33.8 | 41.5 | 52.1
Wav2Vec2 | 17.5 | 23.0 | 30.0 | 47.9
HuBERT | 16.7 | 22.0 | 28.7 | 50.7
WavLM | 16.2 | 22.2 | 28.9 | 46.6
Seq2Seq
Transformer-Transformer | 32.9 | 39.5 | 44.7 | 46.6
Wav2Vec2-Transformer | 17.8 | 27.6 | 32.6 | 69.9
HuBERT-Transformer | 16.4 | 24.7 | 29.4 | 61.6
WavLM-Transformer | 14.1 | 20.5 | 26.6 | 45.2
TABLE II: WER of ASR models. * indicates off-the-shelf system with no
finetuning

##  VII Results

###  VII-A ASR

Table II shows the WERs for each model we evaluated. We see that an off-the-
shelf Whisper model outperforms the previous MTL-BLSTM, achieving WERs of
32.6, 40.6, 43.7, and 65.3 for mild, moderate, severe, and very severe
aphasia, respectively. This highlights the advantage of using modern
architectures that have been pretrained on vast amounts data over traditional
HMM-DNN acoustic models.

We find that fine-tuning the Whisper model on the AphasiaBank dataset leads to
further improvements over the off-the-shelf model. Further, all of the
encoder-only models that finetune with CTC loss, outperform the off-the-shelf
Whisper model. HuBERT and WavLM achieve the lowest WERs out of these, with
HuBERT achieving WERs of 22.0 and 28.7 for moderate and severe aphasia
respectively and WavLM achieving WERs of 16.2 and 46.6 for mild and very
severe respectively. This highlights the benefit of using pretrained speech
models for aphasic speech recognition and finetuning them E2E with CTC loss.

Lastly, we find that a seq2seq model with a randomly initialized transformer
encoder achieves worse ASR performance than finetuned encoder-only models.
However, when the seq2seq encoder is pretrained, we see performance
improvements over the previous approaches, especially when a WavLM model is
used. The WavLM-Transformer model achieves the best performance across all
presented methods with WERs of 14.1, 20.5, 26.6, and 45.2 for mild, moderate,
severe, and very severe aphasia respectively. We note that when comparing E2E
finetuning approaches (encoder-only and seq2seq) across the pretrained speech
models, we see some minor performance differences in WER, specifically with
Wav2Vec2 and HuBERT.

These results highlight the importance of leveraging pretrained models for E2E
aphasic speech recognition. Additionally, finetuning these pretrained models
is critical likely due to the issue of data mismatch during pretraining.
Lastly, we saw how model design, pretraining, and optimization choices can
impact ASR performance as seen by the differences between CTC models and
seq2seq models. With these design considerations in mind, we will now explore
how the seq2seq model can be extended for paraphasia detection.

| Word-level | Utterance-level
---|---|---
| AWER | TD | F1-score
Method | Phn+Neo | Phn | Neo | Phn+Neo | Phn | Neo | Phn+Neo | Phn | Neo
Le et. al. [24] | 53.5 | 54.2 | 47.8 | - | - | - | .594 | .611 | .604
STL (proposed) |  |  |  |  |  |  |  |  |
Wav2Vec2-Transformer | 148.7 | 139.4 | 120.9 | 54.3 | 19.9 | 10.3 | .687 | .590 | .636
HuBERT-Transformer | 124.7 | 157.6 | 113.2 | 45.2 | 29.2 | 12.1 | .691 | .629 | .669
WavLM-Transformer | 117.0 | 125.4 | 148.5 | 56.3 | 19.5 | 13.2 | .706 | .638 | .640
MTL (proposed) |  |  |  |  |  |  |  |  |
Wav2Vec2-Transformer | 52.0 | 48.9 | 46.7 | 8.5 | 8.1 | 5.7 | .693 | .638 | .656
HuBERT-Transformer | 49.9 | 46.9 | 44.3 | 8.1 | 8.0 | 5.5 | .703 | .643 | .671
WavLM-Transformer | 48.4 | 45.0 | 30.4 | 8.5 | 8.1 | 5.0 | .688 | .635 | .688
TABLE III: Paraphasia detection results. Word-level evaluation is measured
with AWER. Utterance-level evaluation is measured with F1-score. Results are
aggregated over all speaker-independent folds.

###  VII-B Paraphasia Detection

Starting first with the word-level metrics, we can see that in table III, the
seq2seq models trained with STL have very large AWERs. We note that this is
due to the E2E model setup and that once the learning objective switches from
ASR to paraphasia detection, ASR performance begins to decrease since it is no
longer explicitly optimized. However, we see that when MTL is used and both
tasks are jointly optimized the resulting models have much lower AWERs
compared to STL models. The best-performing model in terms of AWER is the MTL
WavLM-Transformer, which provides significant performance improvements over
the previously established baseline when detecting phonemic, neologistic, and
phonemic+neologistic paraphasias. We achieve AWERs of 45.0, 30.4, and 48.4 for
phonemic, neologistic, and phonemic+neologistic paraphasias, respectively,
which represents performance improvements over the previous approach of 16.9%,
36.4%, and 9.5%.

When comparing the TD across seq2seq models, we observe a large performance
gap between the use of MTL and STL objectives. These performance gaps
demonstrate that STL models are not able to perform fine-grain paraphasia
detection to the same degree as MTL models. This performance gap is also
likely a result of poor ASR performance, which can result in poor alignment
and ultimately a larger TD metric. These results suggest that in a seq2seq
system, optimizing for both ASR performance and paraphasia detection
simultaneously leads to better word-level paraphasia detection.

When looking at utterance-level average F1-scores, we find that the seq2seq
paraphasia detection models outperform the previous state-of-the-art approach
for phonemic, neologistic, and phonemic+neologistic paraphasias. Our best-
performing models achieve an F1 of 0.643, 0.688, and 0.706 for phonemic,
neologistic, and phonemic+neologistic paraphasia detection, respectively. This
represents performance improvements of 5.2%, 13.9%, and 18.9% over the
previous SOTA method. When looking at the F1-scores for the seq2seq models we
observe minimal performance changes when comparing different pretrained models
and learning objectives.

Lastly, we explore the performance of word-level paraphasia detection using
TTR and investigate the impact of window size ($\omega$) on seq2seq MTL
models. In Figure 4, we observe large TTR improvements as $\omega$ increases
demonstrating that all these models exhibit high proximity to the ground truth
label index. We note that WavLM-Transformer achieves the highest TTR when
$\omega$=0, which could potentially occur due to better alignment as indicated
by the higher ASR performance shown in Table II. However, as $\omega$
increases, we see that Hubert-Transformer closes the performance gap and
outperforms WavLM-Transformer when $\omega$=2. These results demonstrate that
both HuBERT-Transformer and WavLM-Transformer approaches have high word-level
paraphasia recall given a buffer size of a few words.

Figure 4: The impact of window size ($\omega$) on Time Tolerant Recall when
using MTL seq2seq models.

Paraphasia Detection - Discussion
We've shown that the proposed seq2seq model achieves better word-level and
utterance-level paraphasia detection over prior work, highlighting the
advantages of E2E training. When comparing STL and MTL objectives we find that
while the STL approach does demonstrate adequate utterance-level paraphasia
detection, the performance of word-level paraphasia detection, as measured
with AWER and TD, is poor. This is likely due to the final STL models having
unoptimized ASR heads which in turn impair fine-grain word-level paraphasia
detection. In contrast, simultaneously optimizing both ASR and paraphasia
detection tasks lead to models that are more robust interms of word-level and
utterance-level paraphasia detection. We find that the seq2seq models trained
with MTL achieve the SOTA paraphasia detection for all word-level metrics and
some utterance-level metrics. With this in mind, we believe that the use of an
MTL objective is essential for training paraphasia detection E2E. Both the
HuBERT-Transformer and WavLM-Transformer trained with MTL achieve high
paraphasia detection performance depending on the paraphasia that is being
detected. The HuBERT-Transformer performs better at phonemic paraphasia
detection while the WavLM-Transformer performs better at neologistic
paraphasia detection. We find that either HuBERT-Transformer or WavLM-
Transformer models are viable for detecting the presence of paraphasias in a
given utterance and location of paraphasias within a few words. When thinking
about how automated paraphasia detection can be used to facilitate aphasia
assessment, we believe that this work demonstrates the efficacy of a seq2seq
model trained with MTL for both ASR and paraphasia detection.

| 100 | 500 | 1000 | 2000
---|---|---|---|---
AWER | 49.9 | 49.9 | 37.7 | 51.8
TD | 8.8 | 8.1 | 8.5 | 8.7
F1-score | 0.68 | 0.70 | 0.68 | 0.69
TABLE IV: Analysis of tokenizer size on the MTL HuBERT-Transformer. All
metrics were computed across all folds on the Scripts dataset.

###  VII-C Tokenizer Analysis

Selecting a reasonable vocabulary size is critical in the subword tokenization
process and can impact the performance of ASR and paraphasia detection. In
this section, we explore the impact of vocabulary size on the performance of
ASR and downstream paraphasia detection tasks. We use a SentencePiece
tokenizer with a unigram tokenization scheme and sweep over vocabulary sizes
of 100, 500, 1000, and 2000. We focus our analysis on the MTL WavLM-
Transformer model for phonemic+neologistic paraphasias, which was one the
best-performing model in section VII-B.

From table IV, we see that using a tokenizer size of 500 generally yields the
best performance according to both word-level and utterance-level paraphasia
detection metrics presented. The one metric that goes against this statement
is AWER, where a tokenizer size of 1000 achieves a noticeably low AWER. We
believe this could be due to the model converging at a point that is more
optimal for ASR, ultimately resulting in a lower AWER. We believe that for
paraphasia detection selecting an appropriate tokenizer size is important as
too large a vocabulary will result in sparse paraphasia labels, while too
small a vocabulary can result in too much overlap between smaller sets of
subtokens.

P1_B2_SA_C1-4
---
Ground Truth | fees/1 speak/0 directing/0 to/0 me/0 and/0 din/1 me/0 time/0 to/0 myunikat/1
Predicted | please/1 meek/1 directly/0 to/0 me/0 and/0 then/1 me/1 time/1 to/0 myunikat/1
P1_T4_SA_C2-0
Ground Truth | I/0 han/1 asferaja/1
Predicted | I/0 have/1 afasa/1
P3_T4_SA_C3-1
Ground Truth | jersit/1 $<$eps$>$ means/0 I/0 have/0 diferkli/1 vis/1 lanerj/1
Predicted | durs/1 it/0 means/0 I/0 have/0 diffritulti/1 landerj/1 $<$eps$>$
TABLE V: Transcription Analysis: MTL Hubert-Transformer for phonemic and
neologistic paraphasias

###  VII-D Transcription Analysis

Table V has some example AWER transcripts produced by the MTL HuBERT-
Transformer model for phonemic and neologistic paraphasias. By examining some
output, we can get a better understanding of ASR and paraphasia detection
performance at the word-level.

For utterance P1_B2_SA_C1-4, the alignment of the ground truth and predicted
AWER transcripts is good. The model is able to correctly identify 3
paraphasias and produces 3 false positives. The strength of this model is the
ability to detect the presence of paraphasias as well as which words they
belong to. Additionally, the locations of the false positives are near the
ground truth paraphasias, which can be acceptable for certain applications
like flagging paraphasia regions. We also note that for word recognition, the
ASR output does misrecognize some words.

For utterance P1_T4_SA_C2-0, we can another case of good alignment and the
paraphasia detection model is able to correctly predict all paraphasias. We do
see another instance though where the ASR head is unable to recognize the
paraphasic words. This highlights the importance of the additional metrics
like TD and TTR which focus the evaluation on just the paraphasia detection
output. This utterance highlights a common pattern of the model where the ASR
error for paraphasic words is very high. We believe this is in part due to the
high variability of the pseudo-word targets for some paraphasias (described in
section IV). This high label variability is also compounded by challenges such
as high speaker variability and data scarcity, particularly for paraphasias.

For utterance P3_T4_SA_C3-1, we can see some slight misalignment near the end
of the transcript most notably for the word `lanerj'. The levenshtein distance
that is used to align the AWER transcripts does not produce perfect alignments
for evaluating word-level paraphasia detection. This can produce misalignments
highlighted in this example and ultimately motivates the use of metrics like
TD and TTR that consider proximity.

Transcription Analysis - Discussion
This analysis highlights some of the strengths and weaknesses of the MTL
HuBERT-Transformer model as well as some of the challenges associated with
evaluating paraphasia detection. We can see the model performing well when
recognizing paraphasias at the word-level and generally good ASR performance.
However, one limitation is the poor ASR performance for paraphasic words,
which we believe is due to the high variability with how these pseudo-word
targets are generated. Another challenge is the issue of misalignment that we
see in utterance P3_T4_SA_C3-1 for the paraphasia `lanerj'. This example
highlights the importance of using metrics that take proximity into account,
like TD and TTR, when evaluating word-level paraphasia detection. These
examples highlight the challenges associated with this task and the need for
detailed evaluations that can help researchers better understand the strengths
and limitations of the resulting machine-learning systems.

In clinical settings, these models can provide more feedback to medical
professionals who are in the process of analyzing aphasic speech. One example
of this is in streamlining the annotation process, where the model is used to
flag paraphasic instances. With applications like this in mind, slight
misalignment issues can be overcome by the medical professional who has the
context of both the recognized word and predicted paraphasia label
(highlighted by the AWER transcript output).

##  VIII Conclusion

This work investigates different methods for improving paraphasia detection,
which can aid clinicians with traditional speech-language aphasic analyses and
be helpful for specific treatment planning such as supplemental, self-driven,
app-based therapy. We first begin by evaluating existing ASR architectures for
aphasic speech recognition. We find that leveraging pretrained speech models
is critical in low-resource domains such as aphasia and that fine-tuning with
either an encoder-only or seq2seq architecture led to improved performance.
Our best model is a seq2seq WavLM-Transformer model.

We then extend this approach and present a novel paraphasia detection model
that is trained E2E and performs both speech recognition and binary paraphasia
classification. We explore the proposed seq2seq model with both MTL and STL
objectives and compare against prior work on previously used word-level and
utterance-level paraphasia detection metrics as well as provide additional
follow-up evaluations for word-level paraphasia detection. We demonstrate that
either a HuBERT or WavLM seeded seq2seq model trained with MTL achieves state-
of-the-art paraphasia detection performance at the word- and utterance-levels.
We provide some analyses on the effects of tokenizer size on paraphasia
detection, which is a hyperparameter to consider for seq2seq models. Lastly,
we show some AWER output from MTL HuBERT-Transformer model to highlight some
of the common strengths and weaknesses observed in the model and discuss how
this could be used in clinical settings.

##  IX Acknowledgements

This research is based in part upon work supported by the National Science
Foundation (NSF IIS-RI 2006618, NSF IIS-RI 008860, Graduate Research
Fellowship Program). This research is supported in part through computational
resources and services provided by Advanced Research Computing (ARC), a
division of Information and Technology Services (ITS) at the University of
Michigan, Ann Arbor.

## References

  * [1] N. A. Association, https://www.aphasia.org/, [Online; accessed 10-May-2022].
  * [2] O. Spreen and A. H. Risser, _Assessment of aphasia_. Oxford University Press, 2003.
  * [3] N. Helm-Estabrooks and M. Albert, _Manual of Aphasia and Aphasia Therapy_. Pro-Ed, 2004. [Online]. Available: https://books.google.com/books?id=adYLAQAAMAAJ
  * [4] M. M. Saling, ``Chapter 3 - disorders of language,'' in _Neurology and Clinical Neuroscience_ , A. H. Schapira, E. Byrne, S. DiMauro, R. S. Frackowiak, R. T. Johnson, Y. Mizuno, M. A. Samuels, S. D. Silberstein, and Z. K. Wszolek, Eds. Philadelphia: Mosby, 2007, pp. 31–42. [Online]. Available: https://www.sciencedirect.com/science/article/pii/B9780323033541500079
  * [5] G. Fergadiotis, K. Gorman, and S. Bedrick, ``Algorithmic classification of five characteristic types of paraphasias,'' _American Journal of Speech-Language Pathology_ , vol. 25, no. 4S, pp. S776–S787, 2016.
  * [6] E. T. McKinnon, J. Fridriksson, A. Basilakos, G. Hickok, A. E. Hillis, M. V. Spampinato, E. Gleichgerrcht, C. Rorden, J. H. Jensen, J. A. Helpern _et al._ , ``Types of naming errors in chronic post-stroke aphasia are dissociated by dual stream axonal loss,'' _Scientific reports_ , vol. 8, no. 1, p. 14352, 2018.
  * [7] D. Le, K. Licata, and E. M. Provost, ``Automatic paraphasia detection from aphasic speech: A preliminary study.'' in _Interspeech_ , 2017, pp. 294–298.
  * [8] J. Mayer and L. Murray, ``Functional measures of naming in aphasia: Word retrieval in confrontation naming versus connected speech,'' _Aphasiology_ , vol. 17, no. 5, pp. 481–497, 2003.
  * [9] R. Prins and R. Bastiaanse, ``Review,'' _Aphasiology_ , vol. 18, no. 12, pp. 1075–1091, 2004. [Online]. Available: https://doi.org/10.1080/02687030444000534
  * [10] P. Jaecks, M. Hielscher-Fastabend, and P. Stenneken, ``Diagnosing residual aphasia using spontaneous speech analysis,'' _Aphasiology_ , vol. 26, no. 7, pp. 953–970, 2012.
  * [11] A. H. Risser and O. Spreen, ``The western aphasia battery,'' _Journal of clinical and experimental neuropsychology_ , vol. 7, no. 4, pp. 463–470, 1985.
  * [12] K. C. Fraser, J. A. Meltzer, N. L. Graham, C. Leonard, G. Hirst, S. E. Black, and E. Rochon, ``Automated classification of primary progressive aphasia subtypes from narrative speech transcripts,'' _cortex_ , vol. 55, pp. 43–60, 2014.
  * [13] R. Prins and R. Bastiaanse, ``Analyzing the spontaneous speech of aphasic speakers.'' _Aphasiology_ , 2004.
  * [14] M. Grande, K. Hussmann, E. Bay, S. Christoph, M. Piefke, K. Willmes, and W. Huber, ``Basic parameters of spontaneous speech as a sensitive method for measuring change during the course of aphasia,'' _International Journal of Language & Communication Disorders_, vol. 43, no. 4, pp. 408–426, 2008.
  * [15] H. Goodglass and A. Wingfield, ``Word-finding deficits in aphasia: Brain—behavior relations and clinical symptomatology,'' in _Anomia_. Elsevier, 1997, pp. 3–27.
  * [16] G. Denes, C. Perazzolo, A. Piani, and F. Piccione, ``Intensive versus regular speech therapy in global aphasia: A controlled study,'' _Aphasiology_ , vol. 10, no. 4, pp. 385–394, 1996.
  * [17] S. K. Bhogal, R. Teasell, and M. Speechley, ``Intensity of aphasia therapy, impact on recovery,'' _Stroke_ , vol. 34, no. 4, pp. 987–993, 2003.
  * [18] A. Abad, A. Pompili, A. Costa, I. Trancoso, J. Fonseca, G. Leal, L. Farrajota, and I. P. Martins, ``Automatic word naming recognition for an on-line aphasia treatment system,'' _Computer Speech & Language_, vol. 27, no. 6, pp. 1235–1248, 2013.
  * [19] K. J. Ballard, N. M. Etter, S. Shen, P. Monroe, and C. Tien Tan, ``Feasibility of automatic speech recognition for providing feedback during tablet-based treatment for apraxia of speech plus aphasia,'' _American journal of speech-language pathology_ , vol. 28, no. 2S, pp. 818–834, 2019.
  * [20] American Speech-Language-Hearing Association. (2023) Asha practice portal: Aphasia. [Online]. Available: https://www.asha.org/practice-portal/clinical-topics/aphasia/#collapse_6
  * [21] J. Kurland, A. R. Wilkins, and P. Stokes, ``ipractice: Piloting the effectiveness of a tablet-based home practice program in aphasia treatment,'' in _Seminars in speech and language_ , vol. 35, no. 01. Thieme Medical Publishers, 2014, pp. 051–064.
  * [22] S. Pai, N. Sachdeva, P. Sachdeva, and R. Shah, ``Unsupervised paraphasia classification in aphasic speech,'' in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop_ , 2020, pp. 13–19.
  * [23] K. C. Fraser, F. Rudzicz, N. Graham, and E. Rochon, ``Automatic speech recognition in the diagnosis of primary progressive aphasia,'' in _Proceedings of the fourth workshop on speech and language processing for assistive technologies_ , 2013, pp. 47–54.
  * [24] D. Le, K. Licata, and E. M. Provost, ``Automatic quantitative analysis of spontaneous aphasic speech,'' _Speech Communication_ , vol. 100, pp. 1–12, 2018.
  * [25] H. Christensen, I. Casanueva, S. Cunningham, P. Green, and T. Hain, ``Automatic selection of speakers for improved acoustic modelling: Recognition of disordered speech with sparse data,'' in _2014 IEEE Spoken Language Technology Workshop (SLT)_. IEEE, 2014, pp. 254–259.
  * [26] M. Perez, Z. Aldeneh, and E. M. Provost, ``Aphasic speech recognition using a mixture of speech intelligibility experts,'' _Proc. Interspeech 2020_ , pp. 4986–4990, 2020.
  * [27] J. R. Green, R. L. MacDonald, P.-P. Jiang, J. Cattiau, R. Heywood, R. Cave, K. Seaver, M. A. Ladewig, J. Tobin, M. P. Brenner _et al._ , ``Automatic speech recognition of disordered speech: Personalized models outperforming human listeners on short phrases.'' in _Interspeech_ , 2021, pp. 4778–4782.
  * [28] S. E. Gutz, K. L. Stipancic, Y. Yunusova, J. D. Berry, and J. R. Green, ``Validity of off-the-shelf automatic speech recognition for assessing speech intelligibility and speech severity in speakers with amyotrophic lateral sclerosis,'' _Journal of Speech, Language, and Hearing Research_ , vol. 65, no. 6, pp. 2128–2143, 2022.
  * [29] I. G. Torre, M. Romero, and A. Álvarez, ``Improving aphasic speech recognition by using novel semi-supervised learning methods on aphasiabank for english and spanish,'' _Applied Sciences_ , vol. 11, no. 19, p. 8872, 2021.
  * [30] T. Lee, Y. Liu, P.-W. Huang, J.-T. Chien, W. K. Lam, Y. T. Yeung, T. K. Law, K. Y. Lee, A. P.-H. Kong, and S.-P. Law, ``Automatic speech recognition for acoustical analysis and assessment of cantonese pathological voice and speech,'' in _2016 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2016, pp. 6475–6479.
  * [31] S. S. Mahmoud, R. F. Pallaud, A. Kumar, S. Faisal, Y. Wang, and Q. Fang, ``A comparative investigation of automatic speech recognition platforms for aphasia assessment batteries,'' _Sensors_ , vol. 23, no. 2, p. 857, 2023.
  * [32] D. Le and E. M. Provost, ``Improving automatic recognition of aphasic speech with aphasiabank.'' in _Interspeech_ , 2016, pp. 2681–2685.
  * [33] V. Pérez-Rosas, B. Kleinberg, A. Lefevre, and R. Mihalcea, ``Automatic detection of fake news,'' in _Proceedings of the 27th International Conference on Computational Linguistics_ , 2018, pp. 3391–3401.
  * [34] E. Battenberg, J. Chen, R. Child, A. Coates, Y. G. Y. Li, H. Liu, S. Satheesh, A. Sriram, and Z. Zhu, ``Exploring neural transducers for end-to-end speech recognition,'' in _2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2017, pp. 206–213.
  * [35] L. Dong, S. Xu, and B. Xu, ``Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition,'' in _2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2018, pp. 5884–5888.
  * [36] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, ``Attention is all you need,'' _Advances in neural information processing systems_ , vol. 30, 2017.
  * [37] J. Tang, W. Chen, X. Chang, S. Watanabe, and B. MacWhinney, ``A new benchmark of aphasia speech recognition and detection based on e-branchformer and multi-task learning,'' _arXiv preprint arXiv:2305.13331_ , 2023.
  * [38] Y. Peng, K. Kim, F. Wu, B. Yan, S. Arora, W. Chen, J. Tang, S. Shon, P. Sridhar, and S. Watanabe, ``A comparative study on e-branchformer vs conformer in speech recognition, translation, and understanding tasks,'' _arXiv preprint arXiv:2305.11073_ , 2023.
  * [39] B. MacWhinney, D. Fromm, M. Forbes, and A. Holland, ``Aphasiabank: Methods for studying discourse,'' _Aphasiology_ , vol. 25, no. 11, pp. 1286–1307, 2011.
  * [40] A. Kertesz, ``Western aphasia battery–revised,'' 2007.
  * [41] D. Le, K. Licata, C. Persad, and E. M. Provost, ``Automatic assessment of speech intelligibility for individuals with aphasia,'' _IEEE/ACM transactions on audio, speech, and language processing_ , vol. 24, no. 11, pp. 2187–2199, 2016.
  * [42] B. MacWhinney, _The CHILDES project: Tools for analyzing talk, Volume I: Transcription format and programs_. Psychology Press, 2014.
  * [43] M. Ravanelli, T. Parcollet, and Y. Bengio, ``The pytorch-kaldi speech recognition toolkit,'' in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 6465–6469.
  * [44] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, ``wav2vec 2.0: A framework for self-supervised learning of speech representations,'' _Advances in neural information processing systems_ , vol. 33, pp. 12 449–12 460, 2020.
  * [45] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``Hubert: Self-supervised speech representation learning by masked prediction of hidden units,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [46] S.-w. Yang, P.-H. Chi, Y.-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin _et al._ , ``Superb: Speech processing universal performance benchmark,'' _arXiv preprint arXiv:2105.01051_ , 2021.
  * [47] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao _et al._ , ``Wavlm: Large-scale self-supervised pre-training for full stack speech processing,'' _IEEE Journal of Selected Topics in Signal Processing_ , vol. 16, no. 6, pp. 1505–1518, 2022.
  * [48] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, ``Robust speech recognition via large-scale weak supervision,'' _arXiv preprint arXiv:2212.04356_ , 2022.
  * [49] M. Bain, J. Huh, T. Han, and A. Zisserman, ``Whisperx: Time-accurate speech transcription of long-form audio,'' _arXiv preprint arXiv:2303.00747_ , 2023.
  * [50] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, ``Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,'' in _Proceedings of the 23rd international conference on Machine learning_ , 2006, pp. 369–376.
  * [51] S. Kim, T. Hori, and S. Watanabe, ``Joint ctc-attention based end-to-end speech recognition using multi-task learning,'' in _2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2017, pp. 4835–4839.
  * [52] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, ``Specaugment: A simple data augmentation method for automatic speech recognition,'' _arXiv preprint arXiv:1904.08779_ , 2019.
  * [53] G. Kovács, G. Sebestyen, and A. Hangan, ``Evaluation metrics for anomaly detection algorithms in time-series,'' _Acta Universitatis Sapientiae, Informatica_ , vol. 11, no. 2, pp. 113–130, 2019.
  * [54] E. Scharwächter and E. Müller, ``Statistical evaluation of anomaly detectors for sequences,'' _arXiv preprint arXiv:2008.05788_ , 2020.

|  Matthew Perez is a Ph.D. student at the University of Michigan working with
Professor Emily Mower Provost. He received his B.S. degree in Computer Science
from the University of Notre Dame in 2017 and his M.S. degree in Computer
Science and Engineering from the University of Michigan in 2019. He was the
recipient of the GEM Fellowship award (2019-2020) and the National Science
Foundation Graduate Research Fellowship (2020-2023). His research interests
include machine learning for speech-based assistive technology, automatic
speech recognition, and computational paralinguistics. He is a member of IEEE
and ISCA.
---|---
|  Duc Le is a Research Lead at ByteDance, where he works on end-to-end music
generation and large-scale music understanding. Previously, he was a Research
Scientist Manager at Meta, specializing in automatic speech recognition and
spoken language understanding. He received his B.S. in Computer Science (summa
cum laude) from the University of Texas at Dallas, Richardson, TX in 2012 and
his M.S. and Ph.D. in Computer Science from the University of Michigan, Ann
Arbor, MI in 2014 and 2017, respectively. He is a member of IEEE and ISCA, and
has served as session chair for ICASSP and INTERSPEECH.
---|---
|  Amrit Romana received her B.S. degree in Mathematics, and M.S. degree in
Computer Science and Engineering from University of Michigan, in 2014 and
2020, respectively. She is currently a Ph.D. student in Computer Science and
Engineering at University of Michigan. Her research interests include speech
processing and machine learning with the goal of making speech-based
technologies more accessible. She is a member of IEEE and ISCA.
---|---
|  Elise Jones is a Senior Speech-Language Pathologist at the University
Center for Language and Literacy at the University of Michigan. She earned her
B.S. in Speech and Hearing Sciences from Purdue University in 2014 and her
M.A. in Speech-Language Pathology from Indiana University in 2016. She holds a
Certificate of Clinical Competence through the American Speech and Hearing
Association (ASHA), primarily providing assessment and treatment for adults
with aphasia through the University of Michigan Aphasia Program. Her clinical
interests include supporting individuals with aphasia and their families in
communication, engagement, and improving overall independence and quality of
life.
---|---
|  Keli Licata is a Senior Speech-Language Pathologist and the Education
Coordinator at the University Center for Language and Literacy (UCLL) at the
University of Michigan. She works primarily with individuals with aphasia and
their caregivers within the University of Michigan Aphasia Program (UMAP). She
earned her B.A. in Linguistics and Psychology from the University of Michigan
and her M.A. in Speech-Language Pathology from Indiana University. Keli holds
a Certificate of Clinical Competence from the American Speech-Language-Hearing
Association (ASHA) and is licensed to provide teletherapy services across
multiple states. Her clinical areas of interest include providing evidence-
based, person- and family-centered care for adults with aphasia. Her
collaborative research interests currently include characterizing the stress
experienced by caregivers for adults with aphasia. Keli will also begin
teaching the graduate-level aphasia course in the Department of Communication
Sciences and Disorders at Michigan State University in January, 2024.
---|---
|  Emily Mower Provost (M’11, SM'17) is a Professor in Computer Science and
Engineering at the University of Michigan. She received her Ph.D. in
Electrical Engineering from the University of Southern California (USC), Los
Angeles, CA in 2010. She is a Toyota Faculty Scholar (2020) and has been
awarded a National Science Foundation CAREER Award (2017), the Oscar Stern
Award for Depression Research (2015), a National Science Foundation Graduate
Research Fellowship (2004-2007). She is an Associate Editor for IEEE
Transactions on Affective Computing and the IEEE Open Journal of Signal
Processing. She has also served as Associate Editor for Computer Speech and
Language and ACM Transactions on Multimedia. She has received best paper
awards or finalist nominations for Interspeech 2008, ACM Multimedia 2014, ICMI
2016, and IEEE Transactions on Affective Computing. Among other organizational
duties, she has been Program Chair for ACII (2017, 2021), ICMI (2016, 2018).
Her research interests are in human-centered speech and video processing,
multimodal interfaces design, and speech-based assistive technology. The goals
of her research are motivated by the complexities of the perception and
expression of human behavior.
---|---