17D070059

Bachelor of Technology

Computer Science and Engineering

Prof. Preethi Jyothi Basil Abraham, Microsoft IDC Hyderabad

# Gujarati-English Code-Switching Speech Recognition using ensemble prediction
of spoken language

Yash Sharma

(May 21, 2021)

###### Abstract

An important and difficult task in code-switched speech recognition is to
recognize the language, as lots of words in two languages can sound similar,
especially in some accents. We focus on improving performance of end-to-end
Automatic Speech Recognition models by conditioning transformer layers on
language ID of words and character in the output in an per layer supervised
manner. To this end, we propose two methods of introducing language specific
parameters and explainability in the multi-head attention mechanism, and
implement a Temporal Loss that helps maintain continuity in input alignment.
Despite being unable to reduce WER significantly, our method shows promise in
predicting the correct language from just spoken data. We introduce
regularization in the language prediction by dropping LID in the sequence,
which helps align long repeated output sequences.

###### Contents

  1. 1 Introduction
    1. 1.1 Background and Previous Work
    2. 1.2 Motivation
    3. 1.3 Objective & Method Summary
  2. 2 Related Work
    1. 2.1 Multilingual Neural Machine Translation
    2. 2.2 Bi-Encoder Code-switched Speech Recognition
  3. 3 Method Details
    1. 3.1 Regular Self Attention
    2. 3.2 Method 1 - Pre-Attention Addition
    3. 3.3 Method 2 - Post-Attention Addition
  4. 4 Training, Results, and Analysis
    1. 4.1 Gating Loss
      1. 4.1.1 CTC Loss
      2. 4.1.2 Label Smoothing Loss (for the decoder, char-based LID only)
      3. 4.1.3 Seamless Temporal Classification (STC) Loss
      4. 4.1.4 CTC Loss with trimming
    2. 4.2 Experiments and Results
    3. 4.3 Other Evaluation Techniques
    4. 4.4 Qualitative Analysis
      1. 4.4.1 Analysis of gating weights
      2. 4.4.2 Similar output from language specific parameters
  5. 5 Efficiency Improvements in STC Loss
    1. 5.1 Algorithm
  6. 6 Summary and Conclusion
    1. 6.1 Contributions
    2. 6.2 Scope for future research
  7. A Pipeline of ESPnet v2.0
    1. A.1 How to load additional data
  8. B Incremental Optimization of Code
  9. Acknowledgments

## Chapter 1 Introduction

###  1.1 Background and Previous Work

Our task was to improve Automatic Speech Recognition performance on code-
switched Gujarati-English data, using 100 hours of monolingual Gujarati and
100 hours of code-switched Gujarati-English with Gujarati as the matrix
language. This training data is our baseline, as we use other variations of
added data as intuition suggests. As a baseline and a starting point for
further experiments, we use an end-to-end ASR model: a Transformer based
encoder-decoder hybrid network [13] with both label (smoothing) loss and CTC
loss[4]; with a beam search based decoding on the CTC output, decoder output
and a RNN based character language model. More details are in Chapter 4.

In order to improve the WER of the model, we look at transliteration first. We
built a WFST-based [5, 6] English to Gujarati (and hence the reverse)
transliteration model based on a common phoneme set and a large lexicon in
both languages.

The monolingual Gujarati data is pruned through the inverse WFST,
transliterating words to English for which a parallel word exists. We then ran
a number of experiments with different amounts of transliterated data and
tabulated our results, a summary of which is in table 1.1.

Table 1.1: WER(CER) in %: Transliteration Experiments System | test-cs | dev-mono
---|---|---
Baseline |  |
Guj 100 + CS 100 |  32.5 (16.4) |  38.0 (21.1)
Sequential - retraining |  |
mono, CS | 34.7 (17.5) | 45.4 (28.0)
mono, trans, CS | 34.4 (17.3) | 46.1 (28.5)
trans, CS | 34.6 (17.4) | 45.9 (28.9)
Sequential - pretraining |  |
trans, CS | 34.7 (17.6) | 44.7 (27.6)
mono, trans | 52.9 (38.9) | 41.8 (22.6)
Mixed data |  |
mono + exp-trans | 51.2 (38.4) | 41.9 (23.1)
mono + exp-trans + CS | 33.0 (21.5) | 38.6 (16.8)
Transliteration Modes |  |
trans + CS | 33.4 (16.7) | 45.5 (27.2)
ran-trans + CS | 33.7 (17.1) | 45.8 (27.7)
exp-trans + CS | 33.5 (16.9) | 45.8 (25.1)
Using CS + Guj LM |  |
trans, CS | 34.0 (17.2) | 43.0 (26.0)
exp-trans + CS | 33.1 (16.9) | 42.7 (25.1)

###  1.2 Motivation

We suspected that the baseline model is actually better at phoneme recognition
than the word error rate showed. We hypothesize that the model is predicting
some words in one language (e.g. proper nouns), while the reference for that
word is in the other language. We measured the WER after transliterating both
reference and prediction text to a common language using a standard
transliteration library [1], the WER reduced significantly. We even used a
phoneme based transliteration (to the same phoneme set used for WFST
construction), which gave us the same results. This confirms our hypothesis.
We need to improve the model’s prediction of the correct language, while
keeping it’s original capacity to translate speech to phonemes. The results on
a test set is in table 1.2.

Table 1.2: Normalized WER in % System | norm. mode | test-cs
---|---|---
Guj 100 + CS 100 | regular | 32.5
| phone norm. | 31.6
| guj WER | 32.0
| eng WER | 31.5
exp-trans + CS | regular | 33.2
| phone norm. | 32.2
| guj WER | 32.7
| eng WER | 32.1
Guj 100 | regular | 51.4
| phone norm. | 49.4
| guj WER | 48.5
| eng WER | 47.8

###  1.3 Objective & Method Summary

Our objective was to improve WER by better understanding the true language of
the spoken words, and try to achieve post-transliterated WER level
performance. We attempt to improve language identification and therefore
prediction by conditioning the encoder (and the decoder) to a predicted
language ID for each frame in the speech (and text) encoding. We train the
parameters with an additional temporal classification technique mapping from
language identity probability distribution over the temporal encodings to
language IDs of each word or character in the output.

## Chapter 2 Related Work

We’ll discuss two papers, one in the Multilingual Neural Machine Translation
task, and the other targeting code-switched speech recognition using a bi-
encoder model. There are many papers in this domain tackling a similar issues
[2, 9], but we’ll focus on just these two specifically, as they resonate the
closest to our contribution that follows.

###  2.1 Multilingual Neural Machine Translation

Zhang et al. [14] in their work use language specific and agnostic characters
as an alteration to a machine translation model, which makes use of an
additional language ID tokens augmented to their text in order to make use of
language specific and language agnostic parameters (see figure 2.1)

Figure 2.1: Language conditioned Neural Machine Translation with parameter
sharing

$\displaystyle z^{l+1}$ $\displaystyle=\text{LN}(z^{l}+\text{CLSR}(f(z^{l})))$
$\displaystyle\text{CLSR}(f(z^{l}))$
$\displaystyle=g(z^{l})\mathbf{h}^{\text{lang}}+(1-g(z^{l}))\mathbf{h}^{\text{shared}}$
$\displaystyle\text{with}\ \ \mathbf{h}^{\text{lang}}=$ $\displaystyle
f(z^{l})W^{\text{lang}},\ \ h^{\text{shared}}=f(z^{l})$ $\displaystyle
g(z^{l})=\sigma(G(z^{l})+\alpha(t)$ $\displaystyle\mathcal{N}(0,1)),\
G(z^{l})=\text{Relu}(z^{l}\mathbf{w_{1}}+\mathbf{b})\mathbf{w_{2}}$

A similar method can be used this for separate language specific parameters
based on input speech. A common observation is that even if someone does not
understand the language another person is speaking, if they’ve heard it
before, they’ll be able to identify the language. One could even pick out
words that come from a common language that both people know.

###  2.2 Bi-Encoder Code-switched Speech Recognition

Lu et al. [7] attempt to improve performance on English-Mandarin code-switched
speech recognition using a dual encoder model. They first pretrain each
encoder on separate models with separate data - monolingual data coming from
each language. Then, treating these encoders as “experts", they use an
interpolation of the respective encodings using a weight also calculated frame
by frame using the same encoding, with additional trainable parameters. The
new parameters for this weight are tuned by direct optimization on code-
switched data. They call this a mixture of experts model.

This leads us to a new idea - add language-ID context, especially on
parameters of the model which have only seen the acoustic features of the
input. We derive ideas from papers blah blah citations here, and noticed that
a lot of papers seem to model language dependence by making separate decoders,
or separate set of models and then get a weighted average of the outputs, like
experts in different languages accumulating their opinion on something.

So the language conditional model has to keep a few things in mind. The
model’s number of parameters should not scale up too much, and should utmost
be proportional to a constant times the number of languages in question. This
proportionality constant should be much less, 1-5% of the number of parameters
in the baseline, in order to be comparable in capacity. For e.g. if we just
decide to add two encoders to the model for two languages, this would blow up
the parameters by close to 50%, in which case we’ll also have to compare this
new model with performance from a bigger model, which is what Lu et al. [7] do
in their work.

## Chapter 3 Method Details

We use the example of a self-attention network for these details [11]. For
each attention layer - we already have 8 multi-head attention units - each
having roughly 3 million parameters of the form query, key and value
embeddings. We propose that in "gating" layers, a modified version we propose,
we have separate q,k,v embeddings for each language, in each head. The
encoding from the previous layer is passed to each attention head’s each
language segment, and fed through the FF embeddings.

We do a linear interpolation of our language specific embeddings, where the
weights for the interpolation comes from a special feedforward gating layer,
which is fed the same encoding from the previous layer.

###  3.1 Regular Self Attention

Let $z^{l-1}$ be time series encoding from the previous layer. Let $h$ be the
number of layers

$\displaystyle\forall i\in[h]\ \text{ head}_{i}$
$\displaystyle=\text{Attention}(z^{l-1}Q^{l}_{i},z^{l-1}K^{l}_{i},z^{l-1}W^{l}_{i})$
$\displaystyle z^{l}=\text{Con}$
$\displaystyle\text{cat}(\text{head}_{1},\text{head}_{2},...,\text{head}_{h})W^{l}$

We ignore layer normalization and residual connections, as they are not
relevant for this discussion.

###  3.2 Method 1 - Pre-Attention Addition

Let $L$ be the number of languages. We do the linear interpolation before the
multiplication and softmax. Once the interpolated query, key and value are
obtained, the regular process of attention calculation is continued.

$\displaystyle g^{l}_{1},g^{l}_{2},...,g^{l}_{L}$
$\displaystyle\xleftarrow[]{}G^{l}(z^{l-1})$ $\displaystyle\forall 1\leq t\leq
T$ $\displaystyle\sum\limits_{n=1}^{L}g^{l}_{nt}=1$ $\displaystyle\forall
n\in[L]\ \ q^{l}_{in}=z^{l-1}Q^{l}_{in},\ k^{l}_{in}$
$\displaystyle=z^{l-1}K^{l}_{in},\ v^{l}_{in}=z^{l-1}V^{l}_{in}$
$\displaystyle q^{l}_{i}=\sum\limits_{n=1}^{L}g^{l}_{n}*q^{l}_{in},\
k^{l}_{i}$ $\displaystyle=\sum\limits_{n=1}^{L}g^{l}_{n}*k^{l}_{in},\
v^{l}_{i}=\sum\limits_{n=1}^{L}g^{l}_{n}*v^{l}_{in}$ $\displaystyle\forall
i\in[h]\ \text{ head}_{i}$
$\displaystyle=\text{Attention}(q^{l}_{i},k^{l}_{i},v^{l}_{i})$ $\displaystyle
z^{l}=\text{Concat}(\text{he}$
$\displaystyle\text{ad}_{1},\text{head}_{2},...,\text{head}_{h})W^{l}$

###  3.3 Method 2 - Post-Attention Addition

A modification of this was tested. Instead of interpolating all Q, K, V
individually, we first let the individual language attend vectors and
calculate new encodings, and then interpolate these encodings.

$\displaystyle\forall n\in[L]\ \ q^{l}_{in}=z^{l-1}Q^{l}_{in},\ k^{l}_{in}$
$\displaystyle=z^{l-1}K^{l}_{in},\ v^{l}_{in}=z^{l-1}V^{l}_{in}$
$\displaystyle\forall i\in[h],\ \forall n\in[L]\ \text{ head}_{in}$
$\displaystyle=\text{Attention}(q^{l}_{in},k^{l}_{in},v^{l}_{in})$
$\displaystyle\forall n\in[L]\ \text{Lhead}_{n}$
$\displaystyle=\text{Concat}(\text{head}_{1n},\text{head}_{2n},...,\text{head}_{hn})$
$\displaystyle\forall n\in[L]\ g^{{}^{\prime}}_{n}$
$\displaystyle=G^{l}(\text{Lhead}_{n})$ $\displaystyle\forall n\in[L]\ g_{n}$
$\displaystyle=\text{Softmax}_{n}(g^{\prime})$ $\displaystyle\forall i\in[h]\
\text{ head}_{i}$ $\displaystyle=\sum\limits_{n=1}^{L}g_{n}*\text{head}_{in}$
$\displaystyle z^{l}=\text{Concat}(\text{head}_{1}$
$\displaystyle,\text{head}_{2},...,\text{head}_{h})W^{l}$

The difference in cross-attention will come only in method 1, where the gating
weights will be calculated separately for key and value using the encoder
encoding, and for query using the output embedding. The same set of feed-
forward parameters are used for all.

These gating weights are then trained jointly with the ASR network by using
word-level or character level language ID of each example. A reference
language ID sentence is passed, which would have the same length as of the
output text if it’s character-level (including spaces) or would have the
length of the number of words in the reference. In the case of word-level,
we’re assuming that mixing of languages within words doesn’t occur. How this
loss is designed is covered in Section 4.1.

We run a number of experiments with these methods of training gating
parameters, both by jointly training parameters in $G^{l}$ vs disconnecting it
from the end-to-end network and training it separately (treating the weights
themselves as constants during back-propagation, and updating them
separately).

We also experimented with unified parameters for each encoder layer (i.e.
$\forall l\ G^{l}=G$), as well as separate gating for every layer, and we
discarded the former because a) we wanted to give freedom for each layer to
attended for different parts of encoding differently, b) encoding passed to
each layer is different anyway, so same gating parameters for all layers may
not fit well, and c) each gating layer $G^{l}$ is only a few thousand
parameters (for a hidden dimension size 1024, it’s $1024*L$ number of
parameters), so this does not significantly change the number of parameters in
the model.

The number of parameters are majorly changed due to having separate
feedforward parameters $Q^{l}_{in}$ etc. instead of $Q^{l}_{i}$. We notice a
1.1% increase in the total number of parameters in the ASR model per layer.
This is significantly once multiplied with 18 layers in encoder + decoder, so
in later experiments we add gating only to a few layers of the encoder and
decoder.

All experimental results are covered in Section 4.2

## Chapter 4 Training, Results, and Analysis

In order to train the Language ID identification in the model in a supervised
fashion, we come up with a series of methods that we will cover in this next
section.

We then present a number of experiments utilizing these techniques, and the
Character Error Rate (CER) and Word Error Rate (WER) of every experiment on a
2 hour code-switched test set.

###  4.1 Gating Loss

Each spoken speech encoded frame can be associated with a language (or a
probability distribution over which language it belongs to) when we listen to
a code-switched speech. The gating weights that interpolate the attention
encodings represent exactly this. Additional tokens like
$\langle\text{SPACE}\rangle$, $\langle\text{UNK}\rangle$,
$\langle\text{BLANK}\rangle$, $\langle\text{EOS/SOS}\rangle$ are ignored. This
is reasonable since we’re not actually looking to match the output sequence
exactly, spaces and everything, but just maximizes the likelihood of this
output sequence and associate the language ID series with continuous parts of
the speech. However due to difference in temporal resolution, this cannot be
directly aligned with the language IDs, either character based or word-based.
In addition to this every encoder (and decoder) layer produces its own set of
probability distribution (henceforth referred to as gating weights) that needs
to be aligned, and not just one set of gating weights for each example passed
through the model.

We implement a temporal alignment of each of the layer’s gating weights
independently. Let’s take an example of a 20 length input sequence, and a 4
length output sequence. For the output sequence $\langle\text{GUJ}\rangle$
$\langle\text{GUJ}\rangle$ $\langle\text{ENG}\rangle$
$\langle\text{GUJ}\rangle$, one encoder layer can have most likely weights
$\langle\text{GUJ}\rangle^{10}\ \langle\text{ENG}\rangle^{3}\
\langle\text{GUJ}\rangle^{7}$, and another layer can have
$\langle\text{GUJ}\rangle^{9}\ \langle\text{ENG}\rangle^{7}\
\langle\text{GUJ}\rangle^{4}$. An unacceptable alignment would be
$\langle\text{GUJ}\rangle^{1}\ \langle\text{ENG}\rangle^{2}\
\langle\text{GUJ}\rangle^{13}\langle\text{ENG}\rangle^{4}$, as it does not
have sufficient attention to Gujarati in the beginning of the encoding, and
has attention to English parameters at the end. Each layer has independence of
how much of which part of speech it wants to attend to, while also maximizing
alignment likelihood with the output sequence. Hence in each method of
alignment, the loss is independently calculated for each layer, and
summed/averaged over the gating layers.

We use two representations of language ID per utterance - word level and
character level. Word level sequences are short and don’t have
$\langle\text{SPACE}\rangle$, only one language token per word. Character-
level sequences are as long as the character sequence output themselves, with
$\langle\text{SPACE}\rangle$ tokens as delimiters between character LID
tokens.

####  4.1.1 CTC Loss

#####  4.1.1.1 With Projection

This is the simplest of the examples. When aligning any multi-feature
distribution to a probability distribution over a vocabulary based time
series, Connectionist Temporal Classification (CTC) is the most commonly used
loss and alignment mechanism[4].

$CTCLoss(\mathbf{x},\mathbf{y})=-\text{log}(P(\mathbf{y}|\mathbf{x}))=-\text{log}\big{(}\sum\limits_{\mathbf{a}\in
B^{-1}(\mathbf{y})}P(\mathbf{a}|\mathbf{x})\big{)}$

$B^{-1}(y)$ is the inverse of the two-step projection: (a) remove all
consecutive repeating characters - (b) remove $\epsilon$

The features (of time series length N, and features D) are first projected to
a vocabulary dimension V with trainable feedforward parameters, and taken
softmax over each time frame independently (this is the independece
assumption). The obtained probability distribution is vector a, which is then
aligned to output sequence y using the CTC loss function. An advantage of this
is we can have a probability distribution over all the output tokens (non-
language ones too, as discussed in the next method), projecting from just two
features - gating weights for each language. A disadvantage is that the gating
weights may not directly correspond to the language IDs in this case,
depending on how the parameters end up getting trained.

#####  4.1.1.2 Without Projection

But why use projection when we already have a probability distribution from
out features x (gating weights) over a subset of the vocabulary? Our
vocabulary consists of the language IDs for each language in the task, a space
token $\langle\text{SPACE}\rangle$, a blank token
$\langle\text{BLANK}\rangle$, unknown token $\langle\text{UNK}\rangle$, and a
common token for end and start of sentence $\langle\text{EOS/SOS}\rangle$.

So for deriving a from x, we make a smoothing parameter $\alpha$, give
probability
a[i] $=\alpha$ * x[i] $\ \forall i\in\text{language tokens}$, and a[i’]
$=(1-\alpha)/4\ \ \forall\texttt{i'}\in\text{remaining tokens}$.
This means that the most likely output sequence will only have tokens from one
of the languages (for $\alpha>0.33$). We just want a close approximation of
which parts of speech should be encoded by which expert - in this case, a
choice between Gujarati and English.

####  4.1.2 Label Smoothing Loss (for the decoder, char-based LID only)

For the cross attention in the decoder, the encoding fed to the query FF is
the output embeddings, whereas the key and value is the speech encoding from
the last encoder layer. While the latter two have the length of the processed
speech, the output embedding has the length of the output text (number of
characters). Hence the gating weights also have the same resolution as a
character-based LID sequence. Therefore, cross-entropy per time frame can be
used, and we use KL Divergence of the two probability distribution as a proxy
for cross-entropy, with a small smoothing factor.

Again there are two methods to go about this, with and without projection,
from these gating weights to the vocabulary discussed in the previous method.

####  4.1.3 Seamless Temporal Classification (STC) Loss

As mentioned in the sub-section 4.1.2, the ideal one-to-one cross-entropy loss
is a very particular case, it can’t be used for the rest of the network. In
addition to this, there are some issues with the current way CTC is
implemented, especially in the context we’re using it.

  1. 1.

In order to align with the output sequence with repeated tokens, the input
sequence needs to predict blank tokens $\epsilon$, (represented earlier as
$\langle\text{BLANK}\rangle$) in between. For e.g. if output is
“aaaaabbbabbab", then one of the input alignments is
“a$\epsilon$a$\epsilon$a$\epsilon$a$\epsilon$ab$\epsilon$b$\epsilon$bab$\epsilon$bab",
and this input has the minimum number of $\epsilon$s that are needed. This is
not ideal for predicting languages per speech frame, as such frequent
interruptions between characters do not make sense.

  2. 2.

Due to the requirement of blank tokens, an output of length $N$, with a lot of
repetitions will require up to $2N-1$ tokens in the input that needs to align
with it. This is especially true for the 50% of our data, which is fully
monolingual, and the output will be all <GUJ> (with spaces in between). This
issue can be overstepped in the case of word-based LID mapping, but is
considerable in char-based LID mapping. This requirement is not necessarily
met by the speech encodings in the encoder, and definitely not by the output
embeddings in the decoder (which have the same length $N$. In this case, when
there are no inputs possible to align with the output, the likelihood
summation of CTC is 0, and hence the loss is positive infinite.

The reason why CTC requires $\epsilon$ is so that the projection function over
all input alignments $\in(\Sigma\cup\\{\epsilon\\})^{*}$ is an onto function
on $\Sigma^{*}$. If there were no blank tokens, the model will fail to predict
consecutive repeated tokens in it’s output space.

So we propose to get rid of the blank tokens for the meantime. For an output
sequence, the model should aim at predicting input that aligns with the output
by stretching the latter with different intensities at different points of
it’s temporal space. We maximize the likelihood of all such input alignments,
each log likelihood normalized with all possible other outputs they can align
to.

Formally,

$\mathbb{B}^{-1}_{n}(\mathbf{y}=a_{1}a_{2}...a_{m})=\\{\mathbf{a}=(a_{1})^{+}(a_{2})^{+}...(a_{m})^{+}\big{\rvert}|\mathbf{a}|=n\\}$

is the inverse of projection in STC formulation. Treat $(x)^{+}$ as a regular
expression, i.e., x repeated at least once. We fix the length of the input as
$n$ for a specific inverse function, as it goes unsaid in the likelihood
formula for CTC loss as well.

As a difference to CTC, we do not treat the inverse of $\mathbb{B}^{-1}$ as a
function because we no longer can. Instead it is a set of outputs stemming
from a single high temporal input $\mathbf{a}$.

$\mathbb{B}(\mathbf{a}=(a_{1})^{n_{1}}(a_{2})^{n_{2}}...(a_{k})^{n_{k}})=\\{\mathbf{y}=(a_{1})^{m_{1}}(a_{2})^{m_{2}}...(a_{k})^{m_{k}}\big{\rvert}\forall
i\ m_{i}\leq n_{i},\ m_{i}\geq 1\\}$

where all consecutive $a_{i}$ are distinct, and $n_{i}\geq 1$ is how many
times $a_{i}$ is repeated.

Thus we have $|\mathbb{B}(\mathbf{a})|=\Pi_{i=1}^{k}n_{i}$, the small output
is of length $k$, and the largest is of length $\Sigma_{i=1}^{k}n_{i}$

The STC loss for input x and output y hence is

$STCLoss(\mathbf{x},\mathbf{y})=-\text{log}(P(\mathbf{y}|\mathbf{x}))=-\text{log}\big{(}\sum\limits_{\mathbf{a}\in\mathbb{B}_{n}^{-1}(\mathbf{y})}\frac{P(\mathbf{a}|\mathbf{x})}{\mathbf{|}\mathbb{B}(\mathbf{a})\mathbf{|}}\big{)}$

where $n=|\mathbf{x}|$, $P(\mathbf{a}|\mathbf{x})$ is the same as before, with the independence assumption. The normalization $|\mathbb{B}(\mathbf{a})|$ allows us to define the quantity inside the log as a probability P(y | x).

The removal of blank tokens results in the inability to form an onto mapping
to $\Sigma^{*}$, as we no longer have a single "projection" from the most
likely input sequence. From the current formulation, each of the output
sequences from $\mathbb{B}(\mathbf{a})$ are deemed equally likely.

However, in our task, we just want each encoder to attend to language specific
parameters for specific parts of the encodings, and we want each layer to have
it’s own freedom in how long it wants to attend to which part of the sequence.
We also do not wish to check the most likely language characters, as this is
reflected also from the actual character output of the complete end-to-end
model. We do however, do a qualitative analysis of the most likely language
IDs from each layer, and also during the beam search via the decoder, in the
temporal resolution of the speech, which we will discuss in the later
sections.

STC Loss can be presented again with or without projection, but we focused on
experiments only without projection. This loss is then averaged from each
layer, and added to the hybrid loss using a weight ($w_{g}=1.0$ if
unspecified)

####  4.1.4 CTC Loss with trimming

We discussed a flaw of the CTC loss while introducing STC loss - in output
sequence such as in our task with large amounts of character (LID) repetition,
we may require up to twice the number of time frames in the input sequence,
which is not practically enforced by the model when pre-processing the
acoustic frames with 2D convolutions and mel spectrogram calculation.

However, if there is a lot of repetition in the output sequence, that also
brings redundancy. If we could trim down these sequences randomly dropping
some frames enough so the input can fit the modified output, they would still
roughly maintain their semantic form. Hence a simple algorithm is formulated
to conservatively trim the output sequence until its trivial (single frame
output) or fits in the input length. The latter is kept constant throughout.

The random trimming of output sequences on the fly further acts as a
regularizer over several epochs, which helped validation loss in some
experiments with this method go down upto twice as many epochs as regular
baseline or gating methods. The disadvantage is this increases training time
per epoch, due to the conservative trimming and retries as is evident from the
code.

###  4.2 Experiments and Results

Despite all the modifications proposed in our work, we were unable to beat the
performance of the baseline model. As defined in the previous semester, 100
hours of monolingual Gujarati speech 100 hours of code-switched English-
Gujarati speech (with text) put together was the ideal as well as realistic
mixture of supervised data for our experiments. Additionally, the LM (used for
beam-search decoding) is also trained on the same text). In experiments where
we add additional data or change the dataset for ASR training, we keep the LM
same as before.

The training is annotated with 12 hours of validation dataset comprising of
code-switched data. The model is evaluated on 3 different datasets - a 2 hour
subset of this validation set referred to as d-cs, an unseen 2 hour code-
switched test set (t-cs) and an unseen 2 hour monolingual dataset (d-mono).

Training takes around 40 epochs to complete, however some experiments
involving trimming based regularization took 60 or more epochs to converge.

First to make sure the baseline results are reproducible despite the changes
in code over the last few months, we redo the baseline experiment. The
architecture is as discussed before (Section 1.1, [13]). There are 12 encoder
layers and 6 decoder layers, each with 1024 hidden units, and 8 attention
heads. We then start two experiments, one with joint gating parameters
($G^{l}=G$) and the other separate. They use CTC Loss with projection (ref:
section 4.1.1.1). The language ID is word-based. The projection parameters are
shared. The gating parameters in both these experiments are disconnected from
the network and trained separately (see table 4.1).

Table 4.1: WER in %: Method 1 - Share or not? S.no | Name | d-cs | t-cs | d-mono
---|---|---|---|---
1 | (new) Baseline | 32.6 | 32.9 | 38.3
2 | Shared gating | 33.6 | 33.4 | 39
3 | Multi gating | 32.8 | 32.8 | 38.4

Sharing parameters was not justified (section 3.3), so all further experiments
shed this condition. Also there were very few projection parameters (in
general, it would be $\text{num\\_lang}*(\text{num\\_lang}+4)$, we also
separate these parameters for each layer. We also try joint training of the
gating parameters with the rest of the E2E ASR model, i.e., no disconnection
(see table 4.2).

Table 4.2: WER in %: Method 1 - multi projection and joint training S.no | Name | d-cs | t-cs | d-mono
---|---|---|---|---
1 | Baseline | 32.6 | 32.9 | 38.3
4 | Gating with multi-proj CTC | 33.7 | 33.6 | 39.0
5 | multi-proj CTC and joint training | 33.4 | 33.3 | 39.2

A missing piece in this that the parameters attend to the probability
distribution per time series only, and independently. We try another
experiment where the projection takes a fixed context ($k=5$ for e.g.) of
gating weights ($t-k$ to $t+k$). Further to this, we add a smoothing loss to
the experiment, as we see that the language will not change for several frames
at a time (see table 4.3

Table 4.3: WER in %: Method 1 - Fixed context vs smoothing S.no | Name | d-cs | t-cs | d-mono
---|---|---|---|---
1 | Baseline | 32.6 | 32.9 | 38.3
6 | multi-proj CTC + smoothing | 33.2 | 33.1 | 39.1
7 | Mutli gating + context | 33.3 | 32.9 | 38.8
8 | Mutli gating + smooth | 33.5 | 33.5 | 39.2

Given there is almost a 19% increase in the number of parameters, with no
improvement, we realised it was important for parameters to be both language
specific and shared parameters. So we re-implement gating to allow only a few
layers to be gated, and other layers to be regular attention (see section
3.1).

First, some terminology for clarity. Layers of the encoder that are closer to
the input speech are referred to as bottom layers, and closer to the decoder
as top layers. Similar convention for the decoder, data flow is from bottom to
top. Our analysis of a trained model showed the absolute CTC gating loss shows
a decreasing trend starting from the bottom to the top layers. This is
consistent for layers in the encoder and the decoder (see figure 4.1)

Figure 4.1: Average Gating Loss vs Layer #

This means that while later layers (that capture several hidden features of
the encoded speech and text) are able to fit better to the language ID
context, earlier layers act as conflicting parameters. Hence when applying
gating only to a few layers, we focus on applying it to only the top few
layers.

At this point of experiments we also focus on character-level gating. We take
this step due to reasons mentioned in our qualitative analysis of gating
weights (section 4.4.1). But this brings us a new set of challenges that we
discussed in section 4.1.3. With word-level LID, the output sequences were
small enough that could align using CTC loss. However character-level language
ID data is roughly 5-6 longer than word-level, and CTC often results in
nan/inf loss, especially in the decoder.

For ease of understanding the following abbreviations are used - L2 enc/dec is
last 2 layers of gating in encoder (or decoder respectively, F2 for first 2).
All experiments from here on use character-based LID, unless mentioned
otherwise. (table 4.4)

Table 4.4: WER in %: Method 2 - Partial gating, CTC trim S.no | Name | d-cs | t-cs | d-mono
---|---|---|---|---
1 | Baseline | 32.6 | 32.9 | 38.3
9 | Word LID, L2 enc + L2 dec | 32.8 | 32.5 | 38.5
10 | Char LID, L2 enc + L2 dec | 38 | 37.8 | 43
11 | only encoder | 34.8 | 34.5 | 40.1
12 | encoder CTC trim + decoder CTC trim | 33.1 | 33.3 | 38.9
13 | encoder CTC trim + decoder CE | 34.8 | 34.4 | 40

Since we did not see improvement in performance from Method 1, we move on to
Method 2. We continue use partial gating on the last 2 layers, and introduce
the STC loss into our network optimization. However we only perform a few
experiments with it, as it takes longer to train. Wherever CTC loss is used,
random trimming is also added to avoid inf loss issues. (table 4.5)

Table 4.5: WER in %: Method 2 - STC and gating loss weight S.no | Name | d-cs | t-cs | d-mono
---|---|---|---|---
1 | Baseline | 32.6 | 32.9 | 38.3
14 | L2 enc CTC + L2 dec CE ($w_{g}=0.2$) | 32.9 | 33.4 | 38.9
15 | L2 enc STC \+ L2 dec CE ($w_{g}=0.2$) | 33.3 | 33.6 | 39.4
16 | L2 enc CTC + L2 dec CTC ($w_{g}=0.2$) | 33.6 | 33.5 | 39.5
17 | L2 enc CTC + L2 dec STC ($w_{g}=0.2$) | 33.3 | 33.5 | 39.0
18 | L2 enc CTC + L2 dec CE ($w_{g}=0.5$) | 33.3 | 33.1 | 38.1
19 |  L4 enc CTC + L4 dec CE ($w_{g}=0.5$) | 33.2 | 33.0 | 38.7
20 | L2 enc CTC + F2 dec CE ($w_{g}=0.5$) | 33.4 | 33.3 | 39.2
21 | L2 enc CTC + L2 dec CE ($w_{g}=0.75$) | 34.1 | 34.1 | 39.8

The bad performance in experiment 10 can be explained by the inf loss (from
CTC). The experiments after that focus on mitigating this issue.

As our qualitative analysis will show ahead (section 4.4), the model was
successfully able to attend to specific parameters for their respective
languages using the no-projection technique, but no improvement in WER. One
suspect could be over-training and over optimization for the LID part of loss
term. So we attempt to reduce the weight of the loss term as training
progresses, in a step-wise manner. We’ll represent these two experiments in
the form of $w_{g}=x::y::z$ representing $w_{g}=x$ for 10 epochs, $w_{g}=y$
for another 10, and $w_{g}=z$ for the remaining time, etc. Both experiments
use CTC (with trimming) for the encoder, and CE loss for the decoder (table
4.6)

Table 4.6: WER in %: Method 2 - reduction of gating weight periodically S.no | Name | d-cs | t-cs | d-mono
---|---|---|---|---
1 | Baseline | 32.6 | 32.9 | 38.3
22 | L2 enc CTC + L2 dec CE ($w_{g}=1.::0.1::0.$) | 33.3 | 33.1 | 38.9
23 | L2 enc CTC + L2 dec CE ($w_{g}=1.::0.1$) | 32.9 | 32.8 | 38.7

Next we try unsupervised training of the gating weights, by setting $w_{g}$ to
zero, and letting the parameters automatically guess the language of the
speech, similar to [7]. However, qualitative analysis showed that the gating
weights were random for this experiment, and it did not show better
performance either (see table 4.7). This is different from Lu et al’s paper as
those gating weights had pre-trained expert encoders to rely on.

Table 4.7: WER in %: Method 2 - Supervised vs Unsupervised training S.no | Name | d-cs | t-cs | d-mono
---|---|---|---|---
1 | Baseline | 32.6 | 32.9 | 38.3
24 | L2 enc CTC + L2 dec CE ($w_{g}=1$) | 33.5 | 33.4 | 39.3
25 | L2 enc CTC + L2 dec CE ($w_{g}=0$) | 33.1 | 33.5 | 38.9

So learning from [7] and our experiments, we try out pre-training these models
using mixed monolingual data, and then finetuning with the 100 hours of code-
switched data. The monolingual data includes the 100 hours of Gujarati data we
were using before, along with 100 hours of English data. We also try training
with all 300 hours of data together. (see table 4.8 and 4.9)

Table 4.8: WER in %: Method 2 - pretraining with monolingual data S.no | Name | Mode | d-cs | t-cs | d-mono
---|---|---|---|---|---
26 | Baseline | pretraining | 56.9 | 55.8 | 45.6
|  | finetuning | 34.1 | 34.0 | 43.2
(old) | Baseline | (only CS) | 34.1 | 34.2 | 46.6
27 | L2 enc CTC + L2 dec CE | pretraining | 55.0 | 53.9 | 43.9
|  | finetuning | 36.5 | 36.3 | 42.4

The finetuning baseline seems to have similar WER on code-switched data in
comparison to experiments with only code-switched data. This indicates that in
this setting, the monolingual data is not helping the model in code-switched
setting at all. We do notice some improvement in monolingual (Gujarati)
performance when using gating and language specific parameters. We also note
that in both models, finetuning with code-switched speech only improves
performance on monolingual examples.

We finally attempt to increase performance with this additional data by
reducing the monolingual English data we give to the model. We retain only 27
hours of short samples of English speech, as this is much more representative
of the total amount of English data in the code-switched dataset (English
being the embedded language). $w_{g}=0.5$ (see table 4.9)

Table 4.9: WER in %: Method 2 - Limited English Data S.no | Name | d-cs | t-cs | d-mono
---|---|---|---|---
1 | Baseline (200 hours) | 32.6 | 32.9 | 38.3
28 | Baseline (300 hours) | 33.2 | 33.0 | 39.0
29 | Baseline (227 hours) | 32.6 | 32.8 | 39.0
30 | L2 enc CTC + L2 dec CE (227 hours) | 33.2 | 33.1 | 39.3
31 | L2 enc CTC + L2 dec CE (300 hours) | 33.6 | 33.5 | 39.0
32 | L6 enc CTC + L3 dec CE (227 hours) | 34.0 | 34.0 | 39.6

###  4.3 Other Evaluation Techniques

CM-WER [10] quantifies our task well. CM-WER first picks the indices of words
that precede or follow a word of a different language. Then the error rate is
calculated as the number of such words predicted incorrectly (substituted or
deleted or inserted) vs total number of such words.

Firstly, we clarify how the metric to represent word error rate. We know the
formula for WER is

$\text{WER}=\frac{S+D+I}{S+D+C}$

By the definition of CM-WER, we simply add the number of mistakes ($M=S+D+I$)
at code switched points, add the number of correct such positions ($N=C$) and
calculated it as

$\text{CM-WER}=\frac{M}{M+N}$

However, in case of code-switching points, we can have no insertions, as the
code-switch point is explicitly marked by a reference word in a particular
language, which cannot be empty. Hence with $I=0$, both these metrics are the
same.

Evaluations of several of our experiments show that there is worsening of
code-switch point performance in gating methods. We also saw an even worsening
of non-code switch point error rate (i.e. words in the reference which are
surrounded by other words of the same language). The numbers in table 4.10
shows high error-rate for non code-switch points as all insertion errors from
the model are included in this. The S.no can be matched to earlier
experiments, roughly sorted by type.

Table 4.10: WER (in %) at Code-Switch Points and non Code-Switch Points S.no | Name | d CM | d non-CM | t CM | t non-CM
---|---|---|---|---|---
1 | Baseline | 27.17 | 36.45 | 27.88 | 36.32
28 | Baseline (300 hours) | 27.70 | 37.08 | 27.99 | 36.40
29 | Baseline (227 hours) | 26.65 | 36.86 | 27.88 | 36.18
| Method 1 & Word LID gating |  |  |  |
2 | Shared gating | 28.88 | 36.95 | 28.81 | 36.57
3 | Multi gating | 28.01 | 36.24 | 28.00 | 36.15
4 | Gating with multi-proj CTC | 28.38 | 37.43 | 28.91 | 36.88
9 | L2 e-CTC + L2 d-CTC | 27.88 | 36.28 | 27.93 | 35.61
| Method 2 & Char LID gating (CTC with trim) |  |  |  |
18 | L2 e-CTC + L2 d-CE ($w_{g}=0.5$) | 28.02 | 36.98 | 28.54 | 36.18
30 | L2 e-CTC + L2 d-CE ($w_{g}=0.5$, 227 hours) | 28.39 | 36.65 | 28.75 | 36.12
31 | L2 e-CTC + L2 d-CE ($w_{g}=0.5$, 300 hours) | 27.99 | 37.53 | 28.74 | 36.79
24 | L2 e-CTC + L2 d-CE ($w_{g}=1$) | 27.79 | 37.47 | 28.63 | 36.58
25 | L2 e-CTC + L2 d-CE ($w_{g}=0$) | 28.25 | 36.57 | 28.48 | 36.99
14 | L2 e-CTC + L2 d-CE ($w_{g}=0.2)$ | 28.09 | 36.23 | 28.81 | 36.57
16 | L2 e-CTC + L2 d-CTC ($w_{g}=0.2$, trimmed) | 28.45 | 37.31 | 29.54 | 36.17
17 | L2 e-CTC + L2 d-STC ($w_{g}=0.2$) | 28.15 | 36.97 | 28.75 | 36.76
20 | L2 e-CTC + F2 d-CE ($w_{g}=0.5$) | 28.14 | 37.11 | 28.76 | 36.48
15 | L2 e-STC \+ L2 d-CE ($w_{g}=0.2$) | 28.28 | 36.92 | 28.63 | 37.04
22 | L2 e-CTC + L2 d-CE ($w_{g}=1.::0.1$) | 27.64 | 36.66 | 28.21 | 36.00
23 | L2 e-CTC + L2 d-CE ($w_{g}=1.::0.1::0.$) | 28.19 | 36.88 | 28.75 | 36.07
19 |  L4 e-CTC + L4 d-CE ($w_{g}=0.5$) | 28.12 | 36.76 | 28.57 | 36.10
32 |  L6 e-CTC + L3 d-CE (227 hours) | 29.24 | 37.44 | 29.64 | 37.04
| Method 2 - Pretraining + finetuning |  |  |  |
26.a | Baseline (200 hours monolingual) | 70.12 | 47.64 | 70.12 | 46.30
26.b | Baseline (finetuned on 100 hours CS) | 27.96 | 38.42 | 28.47 | 37.77
27.a | L2 e-CTC + L2 d-CE ($w_{g}=0.5$, 200 hours) | 69.26 | 44.93 | 69.35 | 43.39
27.b | L2 e-CTC + L2 d-CE ($w_{g}=0.5$, 100 hours CS) | 32.32 | 39.53 | 32.62 | 38.84

###  4.4 Qualitative Analysis

We see no significant improvement in WER using our methods. We suspect that
one reason for this is a noisy dataset. We listened to some samples from the
training data, and despite not knowing Gujarati, we all agreed that the speech
was very fast and at times incomprehensible. Some reference texts were
incomplete, and our model(s) often was able to fill in the remaining dialogue.
Inspite of this or as a result of this, we notice a few good and bad outcomes
from our model.

####  4.4.1 Analysis of gating weights

In the word-level LID experiments - we noticed that the probability
distribution had a huge bias towards predicting
$\langle\text{BLANK}\rangle(\epsilon)$ tokens, and learns to predict peaks for
each word in the output sequence. This led to likely output sequence like
"$\epsilon\epsilon\langle\text{G}\rangle\epsilon\epsilon\langle\text{E}\rangle\epsilon\epsilon\epsilon\langle\text{G}\rangle$"
for a 3 word prediction. This was not what was intended, and we attributed
this to two factors - smaller output sequences, and how CTC loss is designed.
Which is why we moved to longer character-level LID, and STC loss.

Here we notice that the gating weights from each gating layer (encoder) is
very well aligned to a rough interpolation of the LID. We do not show plots
for decoder as it is a beam search based model, but our analysis of a few
example showed similar knowledge of the language. This is significant because
these gating weights are calculated per frame from just the speech encoding.

Figure 4.2: Gating weights alignment to reference LID

The text for the example in figure 4.2 is:

which roughly translates to "The final selection will be after this test
match".

####  4.4.2 Similar output from language specific parameters

Despite the model learning how to predict the language of speech, it was
predicting the same text for either set of language specific parameters. We
run this experiment by setting the gating weights to all 1 and 0, then vice
versa, and see no noticeable change in the decoding logs. We conclude that
both set of language parameters are very close to each other, therefore model
is unable to use the LID prediction. Further to this, since each set of
parameters only see gradients from parts of the encoding, either of them are
not fully trained with all data, and hence often perform poorly.

## Chapter 5 Efficiency Improvements in STC Loss

The following is a naive algorithm for calculating STC loss. We split the loss
formula mentioned in the previous chapter into the first repeating prefix in
the output sequence $\mathbf{y}$ and the rest. We then get recursive loss
computations, transform them back to probability by negative exponentiation,
take a sum over all prefixes of input $\mathbf{a}$ that could fit, and take
the negative log to get the loss.

###  5.1 Algorithm

Listing 1: Naive algorithm for STC Loss

⬇

def STCLoss(W, Y):

assert len(W) >= len(Y)

if len(Y) == 0:

return inf if len(W) != 0 else 0

prefixLength, prefix = NumSamePrefix(Y)

lenRemaining = len(Y) - len(W)

losses = []

for x in range(len_remaining + 1):

negLogProb = sum(W[:prefixLength + x] == prefix)

remLoss = STCLoss(W[prefixLength + x:], Y[prefixLength:])

normLoss = log(prefixLength)

losses.append(negLogProb + remLoss + normLoss)

return -LogSumpExp(-losses)

here W is the negative log probabilities of each character in the input
alignment (-log(a)) and Y is the output character sequence. logSumExp is as
the name suggests, and numSamePrefix returns the next character in the output
sequence and how many times it repeats ($\geq 1$). W[i] == char is the
negative log probability of character ‘char’ occuring at position ‘i’.

There are two ways to implement this dynamically. Top down, and bottom up.

Top down or memoization is starting from the original task
($\text{loss}(x,y)$) and calling the sub-problems utmost once, storing the
values for repeated calls. This is at times inefficient depending on the
implementation language, but only computes the required sub-problems.

Bottom up or tabulation is enumerating and storing all subsequences of input
that can align with output. This may have extra sub-problem computations that
were not required by the task, but subsequently is heavily parallelizable on
the platform we’re concerned with - pyTorch.

Find additional details of optimization of this code in Appendix B.

## Chapter 6 Summary and Conclusion

Our task was to improve code-switching speech recognition on Gujarati-English
speech with roughly 200 hours of parallel data. We analysed new avenues of
improving accuracy and found that transliterating the model’s output and
reference lead to small but significant decrease in error. We experiment with
explainable ideas that help the model learn the spoken language, and not just
the phonetic characteristics from the speech. However, none of our experiments
manage to reduce error rate significantly.

###  6.1 Contributions

During the course of this work, we contribute the following methods and
analysis to the ASR community, despite the lack of results yet

  1. 1.

An early iteration of introducing language dependencies within the encoder
(and decoder) of an end-to-end model, and push towards explainability of these
models.

  2. 2.

A new soft temporal alignment loss: Seamless Temporal Classification, inspired
by CTC Loss.

  3. 3.

We made certain implementation and formula based changes in Code-Mixed Word
Error Rate.

We also add a detailed explanation of the pipeline of ESPnet (version 2) (in
Appendix A). It is discussed in the context of loading data into the pipeline,
and the modification needed for it.

###  6.2 Scope for future research

  * •

STC loss could be used for direct ASR optimization. This could include a
different “prior" over the projection $\mathbb{B}(a)$ rather than the uniform
distribution used. Or just output the smallest projection, as a lot of
languages (e.g. Hindi, Gujarati or Korean) rarely have repeating consecutive
characters, and hence separate modelling for them need not be necessary for
good predictions of spoken text. For e.g. consecutive repeating characters
make up 2.8% of a lexicon, while it makes up only 0.4% of a Gujarati lexicon.
Even if significant, it can be fixed by a reconstruction model [3].

  * •

Experiments with more than 2 languages might show better performance.

  * •

This kind of setup with the right tuning could be used for speech
translation/transliteration. However this is also achievable with traditional
transliteration modules cascaded on an ASR model.

  * •

Instead of interpolating features in method 2 (see section 3.3) we could also
sample from the probability distribution and select output accordingly,
dropping the other language per frame entirely.

##  Appendix A Pipeline of ESPnet v2.0

###  A.1 How to load additional data

The following is a detailed explanation of how to load additional data apart
from speech and text, in this case language ID text, in one of the commonly
used neural network based speech recognition toolkit - ESPnet [12].
Specifically, we used version 2 of the library. While it does derive a lot
from kaldi[8]’s style of data format, the difference in version 2 is that it
doesn’t depend on kaldi’s binaries for it’s pre-processing.

Before you read on further, keep in mind that this is simply section discusses
loading raw data using the existing loaders. Much of the process was
simplified as the task of loading lang_id is similar to the text file loaded
(kaldi style format)

train_one_epoch in espnet2/train/trainer.py is our center of attention. it is
here in a loop that batches are obtained from an iterator. The reporter is
simply a wrapper. This function is called from run, iterator is passed to it,
which is built from train_iterator_factory.build_iter, which is passed the
epoch # to set random seed.

train_iterator_factory is an object returned by build_multiple_iter_factory,
of cls (of type ASRTask). It’s of type MultipleIterFactory, which takes
build_funcs as an initialization argument.

build_funcs is a list of partial functions (similar to lambda functions, each
is fully given all arguments relating to speech file path, text file path, and
in this case, additionally, language id file). The partial is used to maintain
laziness. The number of build_functions in build_funcs is the number of splits
made in data processing.

Each build_func when called returns a iterator factory, built using a subset
of data - a bunch of speech, text, etc. examples. This build_iter returns a
torch.DataLoader.

The collate_fn to this DataLoader converts a list of dictionaries, each having
single speech, text, (and language id) tensors to a dictionary of batched
tensors of speech, text (and language id). The speech is pre-processed through
the frontend in Stage 9, and hence is not trained along with the rest of the
ASR model. The magic of data processing (on the fly) hence happens in the
ESPnetDataset class. The ESPnetDataset is constructed by taking the path list
(as described above) with their keys for the dictionary that will be
eventually made (e.g. [(‘wav.scp’, ‘speech’, ‘sound’)]). Here ‘sound’ is the
data type and ‘speech’ is the key.

A side note: post collate - the dictionary that is created is sent as keyword
arguments to the ESPnetModel’s forward. So a modification in what data is
being sent would require a modification in the template of the forward
function. The default in their code is “speech" and “text", which is
accompanied by “speech_lengths" and “text_lengths"

Accompanying the data info is a preprocess function, which is where the
conversion of string to numbers for the model happens. So to add data, one
would modify the preprocess function in order to accommodate our new data
type, and modify the asr.sh script to add this additional data in the format
shown in the examples.

##  Appendix B Incremental Optimization of Code

We compare the average time taken for calculating the CTC loss and the STC
loss on a batch size of 5, with a random input and repetitive language ID
style output. It takes 1.3 seconds for CTC to perform a forward and backward
50 times (50 iterations) on this input, with one simple FF layer.

We initially implemented the process top down, with KeyError in python’s
default dictionary helping us solve first-time computations. This proved to be
expensive. One forward loss takes roughly 30 seconds time for the same test
input.

We then moved to bottom up, tabulating results for all O(NM) sub-problems in
this task. This reduced the time taken to 3 seconds per forward pass.

This code however had 3 loops, one on the output sequence and one on the input
sequence (for each sub-problem), plus another on the input sequence within the
sub-problem itself. By parallelizing one of the input loops, we bring down the
time required for 50 iterations to 51 seconds. Reducing another loop on the
input sequence, this is brought down to 7 seconds. The loop on the output
sequence cannot be removed due to data dependency.

This was still severely inefficient compared to CTC, and on further inspection
using cProfile, the bottleneck was a very inefficient implementation of
torch.flip. Our use case for it was a reversed version of torch.cumsum, which
is also highlighted in Github - pyTorch issue # 333520. As in the solution
mentioned, we get rid of the use of torch.flip, and with a few other tweaks,
we managed to get the computation time down to 3 seconds.

It’s difficult to further reduce this, as we’re comparing performance to CTC
written in C++, pre-compiled and optimized. STC loss also fairs much poorly
when the number of repetitions are reduced in the output sequence. Despite all
the efficiency improvements, training time with STC loss is almost double in
comparison to CTC Loss, and it’s the bottleneck in the training routine.

## References

  * Bhat et al. [2015] Bhat, I. A., Mujadia, V., Tammewar, A., Bhat, R. A., Shrivastava, M., 2015. Iiit-h system submission for fire2014 shared task on transliterated search. In: Proceedings of the Forum for Information Retrieval Evaluation. FIRE ’14. ACM, New York, NY, USA, pp. 48–53.
URL http://doi.acm.org/10.1145/2824864.2824872

  * Dabre et al. [2020] Dabre, R., Chu, C., Kunchukuttan, A., 2020. A survey of multilingual neural machine translation. ACM Computing Surveys (CSUR) 53 (5), 1–38.
  * Diwan and Jyothi [2020] Diwan, A., Jyothi, P., 2020. Reduce and reconstruct: Improving low-resource end-to-end asr via reconstruction using reduced vocabularies. arXiv preprint arXiv:2010.09322.
  * Graves et al. [2006] Graves, A., Fernández, S., Gomez, F., Schmidhuber, J., 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of the 23rd international conference on Machine learning. pp. 369–376.
  * Karimi et al. [2011] Karimi, S., Scholer, F., Turpin, A., 2011. Machine transliteration survey. ACM Computing Surveys (CSUR) 43 (3), 1–46.
  * Knight and Graehl [1997] Knight, K., Graehl, J., 1997. Machine transliteration. arXiv preprint cmp-lg/9704003.
  * Lu et al. [2020] Lu, Y., Huang, M., Li, H., Guo, J., Qian, Y., 2020. Bi-encoder transformer network for mandarin-english code-switching speech recognition using mixture of experts. Proc. Interspeech 2020, 4766–4770.
  * Povey et al. [2011] Povey, D., Ghoshal, A., Boulianne, G., Burget, L., Glembek, O., Goel, N., Hannemann, M., Motlicek, P., Qian, Y., Schwarz, P., et al., 2011. The kaldi speech recognition toolkit. In: IEEE 2011 workshop on automatic speech recognition and understanding. No. CONF. IEEE Signal Processing Society.
  * Sitaram et al. [2019] Sitaram, S., Chandu, K. R., Rallabandi, S. K., Black, A. W., 2019. A survey of code-switched speech and language processing. arXiv preprint arXiv:1904.00784.
  * Taneja et al. [2019] Taneja, K., Guha, S., Jyothi, P., Abraham, B., 2019. Exploiting monolingual speech corpora for code-mixed speech recognition. In: INTERSPEECH.
  * Vaswani et al. [2017] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., Polosukhin, I., 2017. Attention is all you need. arXiv preprint arXiv:1706.03762.
  * Watanabe et al. [2018] Watanabe, S., Hori, T., Karita, S., Hayashi, T., Nishitoba, J., Unno, Y., Soplin, N. E. Y., Heymann, J., Wiesner, M., Chen, N., et al., 2018. Espnet: End-to-end speech processing toolkit. arXiv preprint arXiv:1804.00015.
  * Watanabe et al. [2017] Watanabe, S., Hori, T., Kim, S., Hershey, J. R., Hayashi, T., 2017. Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing 11 (8), 1240–1253.
  * Zhang et al. [2021] Zhang, B., Bapna, A., Sennrich, R., Firat, O., 2021. Share or not? learning to schedule language-specific ca-pacity for multilingual translation. In: International Conference on Learning Representations.

## Acknowledgments

I wish to record a deep sense of gratitude to Prof. Preethi Jyothi, my
supervisor, for her valuable guidance and constant support in all stages of my
research. I would also like to thank everyone at STCI at Microsoft IDC,
especially Basil and Niranjan, for their support, resources, and time over the
15 months of work put in this project and the previous one.

Since this is also my last project from IIT Bombay, I would like to thank my
friends for making life at college interesting and helping me focus on what’s
important.

My most sincere gratitude goes to my parents, for tolerating me over the phone
during my time at college and at home during the pandemic.