\begin{table}[ht]
\vspace{-0.3cm}
\centering
\caption{Average WER(\%) on test \textit{vid-clean} and \textit{vid-noisy}.}
\begin{tabular}[t]{ l  c  c }
 \hline
 \textbf{Model} & \textbf{\texttt{vid-}} & \textbf{\texttt{vid-}} \\ 
 & \textbf{\texttt{clean}} & \textbf{\texttt{noisy}} \\ \hline \hline
 Monolingual {(MO)}& 19.2 & 20.2 \\ 
 \hline
 Multilingual &  &  \\  
  \quad w/ shared char {(ML-SC)} & 20.6 & 21.4 \\
  \quad w/ shared char + subw. {(ML-SCW)}& 16.9 & 17.8 \\
  \quad w/ lang. spec. in\&out  {(ML-IO)}& 16.2 & 17.4 \\
 \hline
\end{tabular}
\vspace{-0.3cm}
\label{tab:summary}
\end{table}
\begin{abstract}
End-to-end multilingual ASR has become more appealing because of several reasons such as simplifying the training and deployment process and positive performance transfer from high-resource to low-resource languages. However, scaling up the number of languages, total hours, and number of unique tokens is not a trivial task. This paper explores large-scale multilingual ASR models on 70 languages. We inspect two architectures: (1) Shared embedding and output and (2) Multiple embedding and output model. In the shared model experiments, we show the importance of tokenization strategy across different languages. Later, we use our optimal tokenization strategy to train multiple embedding and output model to further improve our result. Our multilingual ASR achieves 13.9\%-15.6\% average WER relative improvement compared to monolingual models. We show that our multilingual ASR generalizes well on an unseen dataset and domain, achieving 9.5\% and 7.5\% WER on Multilingual Librispeech (MLS) with zero-shot and finetuning, respectively.
\end{abstract}

\subsection{Model Architecture}
Our multilingual model is based on an end-to-end Transducer model \cite{graves2012sequence} that is composed by encoder, predictor, and joiner modules. Let $\mathbf{x} = [x_1,..,x_T]$ be an input speech features with length $T$ and $\mathbf{y}=[y_1,...y_U]$ be an output token sequence with length $U$. The encoder model process $\mathbf{x}$ and produce higher level acoustic representation $\mathbf{h^{enc}}=[h_1^{enc},..,h_T^{enc}]=enc(\mathbf{x})$. The prediction network is (usually) an autoregressive decoder that produces hidden states $\mathbf{h^{pred}} = [h_1^{pred},..,h_U^{pred}]$. Each hidden states $h_{u}^{pred} = pred(y_{1},..,y_{u-1})$ conditioned on previous output tokens $y_{<u}$. The joiner module combines both encoder $h_t^{enc}$ and predictor $h_u^{pred}$ representation to calculate the logits $z_{t,u} = joiner(h_t^{enc}, h_u^{pred})$. Lastly, we apply softmax function to calculate the probability given input and previous output tokens $P(y_{t,u} | \mathbf{x}_{[1..t]}, y_{<u}) = softmax(z_{t,u})$. Transducer loss is defined as a negative log-likelihood of output sequence given the input features $\mathcal{L(\mathbf{x}, \mathbf{y}; \theta)}=-\log P_{\theta}(\mathbf{y}|\mathbf{x})$ where $\theta = \{enc, pred, joiner\}$ parameters.

\subsection{Multiple Embedding and Output Architecture}
We explore another architecture with language-specific input embedding and output linear layer. Figure \ref{fig:shared_lang_rnnt}b shows the architecture differences compared to the shared language architecture. Let $d_{emb}$ be the embedding output dimension, $d_{hid}$ be the LSTM hidden dimension, and $\hat{\mathcal{V}}_{l}=\mathcal{V}_{l} \cup \O$ be the tokens for language $l$ plus a blank token $\O$. Then we have multiple embedding weights $\mathbf{E} = [E_1,..E_L]$ where $ E_{l} \in \mathbb R^{|\hat{\mathcal{V}}_{l}| \times d_{emb}}$ and multiple output linear weights $\mathbf{W} = [W_1,..,W_{L}]$ where $ W_{l} \in \mathbb R^{d_{hid} \times |\hat{\mathcal{V}}_{l}|}$ and $\mathbf{b} = [b_1,..,b_{L}]$ where $ b_{l} \in \mathbb R^{|\hat{\mathcal{V}}_{l}|}$. For each language input embedding and output layer, we use characters if that language contains $>512$ unique characters, otherwise, we use subwords with 512 tokens. One advantage of this language-specific architecture is that we could represent the same token between different languages with different embedding and weight matrices. Thus, we could disambiguate characters and subwords that look the same in the written space but sound different (e.g., `a' is spelled as `\textipa{eI}' in English and `\textipa{a}' in Indonesian).

\section{Experimental Setup}
\subsection{Dataset}
We conduct the experiments on our in-house datasets. All in-house datasets are de-identified public videos with no personally identifiable information (PII). Figure \ref{fig:plot_hours} shows the amount of hours training data for each language. Overall, we have around 150,000 hours across 70 languages.

We run data alignment and segmentation pipelines to remove long silences or low-quality transcription and segment original speech and transcripts into maximum of 10-second chunks.  To increase the amount of training data, we apply speed perturbation \cite{ko15_interspeech} with speed factors of 0.9x and 1.1x. 
Since there are some languages with low amount of data, we apply batch re-sampling \cite{babu22_interspeech} by sampling from $p_l \sim \left( \frac{n_l}{N}\right)^{\alpha}$ with $\alpha = 0.5$.
We do not apply any extra processing for the transcript and keep all casings and punctuations for our model to learn them in an end-to-end fashion.

We evaluate our model with test data on two different domains, \textit{vid-clean} and \textit{vid-noisy}, for every language. The main difference between these two domains is that the data from \textit{vid-noisy} are more acoustically challenging compared to \textit{vid-clean}. The amount of data for each language ranges from 5-50 hours for \textit{vid-clean} and 12-50 hours for \textit{vid-noisy}.

Our multilingual models have around 1 billion parameters for all scenarios (with little variation depending on the vocabulary size). To improve training throughput and memory efficiency, we use several tricks following \cite{zheng22d_interspeech} such as: fully sharded data parallel (FSDP) \cite{FairScale2021}, activation checkpointing \cite{chen2016training} and mixed-precision training \cite{micikevicius2017mixed}. We use Adam optimizer \cite{kingma2014adam} with peak $lr=4e-4, \beta_1=0.9, \beta_2=0.98$. We train our model for up to 700,000 updates with 64 GPUs. We apply $lr$ warm-up for the first 20,000 updates and exponentially decay for the rest of the remaining updates into $0.1$ from the peak $lr$.

We report our results on in-house datasets \textit{vid-clean} and \textit{vid-noisy} in Table \ref{tab:summary}. The multilingual model with shared character strategy {ML-SC} shows inferior results with relative WER degradation of 5.9\%-7.3\% compared to {MO}. By using the shared char+subword strategy, {ML-SCW} shows relative WER improvement of 16.8\%-18\% compared to {ML-SC} and 11.9\%-12\% compared to {MO}. On top of that, by adding language-specific embedding and output layer, {ML-IO} achieved relative WER improvement of 2.2\%-4.1\% compared to {ML-SCW} and 13.9\%-15.6\% compared to {MO}. 
{"value": "19.2", "type": "Result", "training data/set": "in-house datasets", "test data/set": "vid-clean", "task": ["ASR", "automatic speech recognition"], "metric": ["WER", "word error rate"], "experimental settings": {"xx": "yy"}, "model": ["Transducer", "Monolingual", "MO"], "model settings": {"xx": "yy"}} {"value": "20.2", "type": "Result", "training data/set": "in-house datasets", "test data/set": "vid-noisy", "task": ["ASR", "automatic speech recognition"], "metric": ["WER", "word error rate"], "experimental settings": {"xx": "yy"}, "model": ["Transducer", "Monolingual", "MO"], "model settings": {"xx": "yy"}} {"value": "20.6", "type": "Result", "training data/set": "in-house datasets", "test data/set": "vid-clean", "task": ["ASR", "automatic speech recognition"], "metric": ["WER", "word error rate"], "experimental settings": {"xx": "yy"}, "model": ["Transducer", "Multilingual w/ shared char", "ML-SC"], "model settings": {"xx": "yy"}} {"value": "21.4", "type": "Result", "training data/set": "in-house datasets", "test data/set": "vid-noisy", "task": ["ASR", "automatic speech recognition"], "metric": ["WER", "word error rate"], "experimental settings": {"xx": "yy"}, "model": ["Transducer", "Multilingual w/ shared char", "ML-SC"], "model settings": {"xx": "yy"}} {"value": "16.9", "type": "Result", "training data/set": "in-house datasets", "test data/set": "vid-clean", "task": ["ASR", "automatic speech recognition"], "metric": ["WER", "word error rate"], "experimental settings": {"xx": "yy"}, "model": ["Transducer", "Multilingual w/ shared char + subw.", "ML-SCW"], "model settings": {"xx": "yy"}} {"value": "17.8", "type": "Result", "training data/set": "in-house datasets", "test data/set": "vid-noisy", "task": ["ASR", "automatic speech recognition"], "metric": ["WER", "word error rate"], "experimental settings": {"xx": "yy"}, "model": ["Transducer", "Multilingual w/ shared char + subw.", "ML-SCW"], "model settings": {"xx": "yy"}} {"value": "16.2", "type": "Result", "training data/set": "in-house datasets", "test data/set": "vid-clean", "task": ["ASR", "automatic speech recognition"], "metric": ["WER", "word error rate"], "experimental settings": {"xx": "yy"}, "model": ["Transducer", "Multilingual w/ lang. spec. in&out", "ML-IO"], "model settings": {"xx": "yy"}} {"value": "17.4", "type": "Result", "training data/set": "in-house datasets", "test data/set": "vid-noisy", "task": ["ASR", "automatic speech recognition"], "metric": ["WER", "word error rate"], "experimental settings": {"xx": "yy"}, "model": ["Transducer", "Multilingual w/ lang. spec. in&out", "ML-IO"], "model settings": {"xx": "yy"}}