\begin{table}[ht]
\centering
\vspace{-0.3cm}

\caption{Average WER(\%) on MLS test sets.}
\begin{tabular}[t]{ l  c }
 
 \hline
 \textbf{Model} & \textbf{WER (\%)} \\ \hline \hline
 
 \textbf{Prior works}  \\ 
 Monolingual CTC \cite{Pratap2020} & 11.8 \\
 Monolingual CTC w/ 5-gram LM \cite{Pratap2020} & 10.7 \\
 XLS-R (ft. 10h) \cite{babu22_interspeech} & 13.8 \\
 RNN-T 1B (ft. all) \cite{li2021scaling} & 7.9 \\
 \hline
 \textbf{Zero-shot} \\
 {MO} & 13.7 \\ 
 {ML-SCW} & 9.8  \\  
 {ML-IO} & 9.5  \\ 
 \hline
 \textbf{Finetune all}  \\  
 {ML-SCW} & 7.7  \\  
 {ML-IO} & 7.5  \\ 
 \hline
\end{tabular}
\label{tab:mls}
\end{table}
\begin{abstract}
End-to-end multilingual ASR has become more appealing because of several reasons such as simplifying the training and deployment process and positive performance transfer from high-resource to low-resource languages. However, scaling up the number of languages, total hours, and number of unique tokens is not a trivial task. This paper explores large-scale multilingual ASR models on 70 languages. We inspect two architectures: (1) Shared embedding and output and (2) Multiple embedding and output model. In the shared model experiments, we show the importance of tokenization strategy across different languages. Later, we use our optimal tokenization strategy to train multiple embedding and output model to further improve our result. Our multilingual ASR achieves 13.9\%-15.6\% average WER relative improvement compared to monolingual models. We show that our multilingual ASR generalizes well on an unseen dataset and domain, achieving 9.5\% and 7.5\% WER on Multilingual Librispeech (MLS) with zero-shot and finetuning, respectively.
\end{abstract}

One key challenge of multilingual ASR in practice is when we scale up the number of languages, our vocabulary size grows larger. Prior works \cite{li2021scaling} trained a multilingual ASR for 15 languages by simply combining all graphemes together. \cite{joshi21_interspeech} proposed multilingual with multi-decoder output for 4 languages. \cite{Pratap2020} built multilingual ASR on top of 50 languages with total 16,000 hours training dataset. To the best of our knowledge, most of the existing works have less amount of languages and smaller unique graphemes size compared to our in-house video dataset.

\subsection{Model Architecture}
Our multilingual model is based on an end-to-end Transducer model \cite{graves2012sequence} that is composed by encoder, predictor, and joiner modules. Let $\mathbf{x} = [x_1,..,x_T]$ be an input speech features with length $T$ and $\mathbf{y}=[y_1,...y_U]$ be an output token sequence with length $U$. The encoder model process $\mathbf{x}$ and produce higher level acoustic representation $\mathbf{h^{enc}}=[h_1^{enc},..,h_T^{enc}]=enc(\mathbf{x})$. The prediction network is (usually) an autoregressive decoder that produces hidden states $\mathbf{h^{pred}} = [h_1^{pred},..,h_U^{pred}]$. Each hidden states $h_{u}^{pred} = pred(y_{1},..,y_{u-1})$ conditioned on previous output tokens $y_{<u}$. The joiner module combines both encoder $h_t^{enc}$ and predictor $h_u^{pred}$ representation to calculate the logits $z_{t,u} = joiner(h_t^{enc}, h_u^{pred})$. Lastly, we apply softmax function to calculate the probability given input and previous output tokens $P(y_{t,u} | \mathbf{x}_{[1..t]}, y_{<u}) = softmax(z_{t,u})$. Transducer loss is defined as a negative log-likelihood of output sequence given the input features $\mathcal{L(\mathbf{x}, \mathbf{y}; \theta)}=-\log P_{\theta}(\mathbf{y}|\mathbf{x})$ where $\theta = \{enc, pred, joiner\}$ parameters.

\subsection{Multiple Embedding and Output Architecture}
We explore another architecture with language-specific input embedding and output linear layer. Figure \ref{fig:shared_lang_rnnt}b shows the architecture differences compared to the shared language architecture. Let $d_{emb}$ be the embedding output dimension, $d_{hid}$ be the LSTM hidden dimension, and $\hat{\mathcal{V}}_{l}=\mathcal{V}_{l} \cup \O$ be the tokens for language $l$ plus a blank token $\O$. Then we have multiple embedding weights $\mathbf{E} = [E_1,..E_L]$ where $ E_{l} \in \mathbb R^{|\hat{\mathcal{V}}_{l}| \times d_{emb}}$ and multiple output linear weights $\mathbf{W} = [W_1,..,W_{L}]$ where $ W_{l} \in \mathbb R^{d_{hid} \times |\hat{\mathcal{V}}_{l}|}$ and $\mathbf{b} = [b_1,..,b_{L}]$ where $ b_{l} \in \mathbb R^{|\hat{\mathcal{V}}_{l}|}$. For each language input embedding and output layer, we use characters if that language contains $>512$ unique characters, otherwise, we use subwords with 512 tokens. One advantage of this language-specific architecture is that we could represent the same token between different languages with different embedding and weight matrices. Thus, we could disambiguate characters and subwords that look the same in the written space but sound different (e.g., `a' is spelled as `\textipa{eI}' in English and `\textipa{a}' in Indonesian).

We run data alignment and segmentation pipelines to remove long silences or low-quality transcription and segment original speech and transcripts into maximum of 10-second chunks.  To increase the amount of training data, we apply speed perturbation \cite{ko15_interspeech} with speed factors of 0.9x and 1.1x. 
Since there are some languages with low amount of data, we apply batch re-sampling \cite{babu22_interspeech} by sampling from $p_l \sim \left( \frac{n_l}{N}\right)^{\alpha}$ with $\alpha = 0.5$.
We do not apply any extra processing for the transcript and keep all casings and punctuations for our model to learn them in an end-to-end fashion.

Our multilingual models have around 1 billion parameters for all scenarios (with little variation depending on the vocabulary size). To improve training throughput and memory efficiency, we use several tricks following \cite{zheng22d_interspeech} such as: fully sharded data parallel (FSDP) \cite{FairScale2021}, activation checkpointing \cite{chen2016training} and mixed-precision training \cite{micikevicius2017mixed}. We use Adam optimizer \cite{kingma2014adam} with peak $lr=4e-4, \beta_1=0.9, \beta_2=0.98$. We train our model for up to 700,000 updates with 64 GPUs. We apply $lr$ warm-up for the first 20,000 updates and exponentially decay for the rest of the remaining updates into $0.1$ from the peak $lr$.

We report our results on in-house datasets \textit{vid-clean} and \textit{vid-noisy} in Table \ref{tab:summary}. The multilingual model with shared character strategy {ML-SC} shows inferior results with relative WER degradation of 5.9\%-7.3\% compared to {MO}. By using the shared char+subword strategy, {ML-SCW} shows relative WER improvement of 16.8\%-18\% compared to {ML-SC} and 11.9\%-12\% compared to {MO}. On top of that, by adding language-specific embedding and output layer, {ML-IO} achieved relative WER improvement of 2.2\%-4.1\% compared to {ML-SCW} and 13.9\%-15.6\% compared to {MO}. 

\subsection{Results on MLS: Zero-shot and Finetuning}
Here, we want to observe how our multilingual model generalizes on new unseen domains and datasets. Therefore, we perform a zero-shot task to see how our model performs directly without any additional training steps and adaptation. After that, we also want to measure how good our model is to be adapted into a new dataset by using it as a seed model. For this purpose, we use Multilingual Librispeech (MLS) \cite{pratap2020mls} dataset that consisted of 8 languages (English, German, Dutch, French, Spanish, Italian, Portuguese, and Polish). Table \ref{tab:mls} shows MLS average WER from prior works and our experiment. We achieve 9.5\% WER on zero-shot and 7.5\% WER after finetuning. To the best of our knowledge, our result is competitive with the state-of-the-art performance on MLS dataset.
{"value": "11.8", "type": "Result", "task": ["ASR", "automatic speech recognition"], "metric": ["WER", "word error rate"], "training data/set": "MLS", "test data/set": "MLS", "experimental settings": {"xx": "yy"}, "model": "Monolingual CTC", "model settings": {"xx": "yy"}} {"value": "10.7", "type": "Result", "task": ["ASR", "automatic speech recognition"], "metric": ["WER", "word error rate"], "training data/set": "MLS", "test data/set": "MLS", "experimental settings": {"xx": "yy"}, "model": "Monolingual CTC w/ 5-gram LM", "model settings": {"xx": "yy"}} {"value": "13.8", "type": "Result", "task": ["ASR", "automatic speech recognition"], "metric": ["WER", "word error rate"], "training data/set": "MLS", "test data/set": "MLS", "experimental settings": {"xx": "yy"}, "model": "XLS-R (ft. 10h)", "model settings": {"xx": "yy"}} {"value": "7.9", "type": "Result", "task": ["ASR", "automatic speech recognition"], "metric": ["WER", "word error rate"], "training data/set": "MLS", "test data/set": "MLS", "experimental settings": {"xx": "yy"}, "model": "RNN-T 1B (ft. all)", "model settings": {"xx": "yy"}} {"value": "13.7", "type": "Result", "task": ["ASR", "automatic speech recognition"], "metric": ["WER", "word error rate"], "training data/set": "in-house datasets", "test data/set": "MLS", "experimental settings": {"Zero-shot": "true"}, "model": "MO", "model settings": {"xx": "yy"}} {"value": "9.8", "type": "Result", "task": ["ASR", "automatic speech recognition"], "metric": ["WER", "word error rate"], "training data/set": "in-house datasets", "test data/set": "MLS", "experimental settings": {"Zero-shot": "true"}, "model": "ML-SCW", "model settings": {"xx": "yy"}} {"value": "9.5", "type": "Result", "task": ["ASR", "automatic speech recognition"], "metric": ["WER", "word error rate"], "training data/set": "in-house datasets", "test data/set": "MLS", "experimental settings": {"Zero-shot": "true"}, "model": "ML-IO", "model settings": {"xx": "yy"}} {"value": "7.7", "type": "Result", "task": ["ASR", "automatic speech recognition"], "metric": ["WER", "word error rate"], "training data/set": "MLS", "test data/set": "MLS", "experimental settings": {"Finetune all": "true"}, "model": "ML-SCW", "model settings": {"xx": "yy"}} {"value": "7.5", "type": "Result", "task": ["ASR", "automatic speech recognition"], "metric": ["WER", "word error rate"], "training data/set": "MLS", "test data/set": "MLS", "experimental settings": {"Finetune all": "true"}, "model": "ML-IO", "model settings": {"xx": "yy"}}