# XTREME-S: Evaluating Cross-lingual Speech Representations

###### Abstract

We introduce XTREME-S, a new benchmark to evaluate universal cross-lingual
speech representations in many languages. XTREME-S covers four task families:
speech recognition, classification, speech-to-text translation and retrieval.
Covering 102 languages from 10+ language families, 3 different domains and 4
task families, XTREME-S aims to simplify multilingual speech representation
evaluation, as well as catalyze research in “universal” speech representation
learning. This paper describes the new benchmark and establishes the first
speech-only and speech-text baselines using XLS-R and mSLAM on all downstream
tasks. We motivate the design choices and detail how to use the benchmark.
Datasets and fine-tuning scripts are made easily accessible through the
HuggingFace platform.111https://hf.co/datasets/google/xtreme_s

##  1 Introduction

In the past two decades, the exploding amount of content on the Internet has
led to a pressing urgency to build systems that can understand text, speech,
and videos in all of the world’s approximately 6,900 languages. Making speech
technology available in all languages is especially important to give speakers
of under-represented languages an equal voice on the Internet, and the
possibility to make their content and culture known outside of their language
cluster. Building speech systems for such a large number of languages is
especially challenging but recent advances in self-supervised learning (SSL)
present great opportunities to achieve this goal.

Speech pre-training techniques like wav2vec 2.0 [1] have emerged as the
predominant approach for automatic speech recognition (ASR) and direct speech-
to-text translation (ST), and have made speech models much more data
efficient: ASR models can be learnt with as little as a few hours of labeled
data [2, 3]. Multilingual pre-training helps build better representations for
languages that lack unannotated data, and thus enables the same data-efficient
strategies for low-resource languages. Approaches like XLS-R [4, 5], for
example, have shown particularly strong results on several tasks, including
ASR on BABEL and multilingual LibriSpeech, and AST on CoVoST-2. Following a
recent trend in natural language processing, the speech community has made
these multilingual pre-trained models publicly available to accelerate
research in multilingual speech understanding.

To support this rapid development and to make better speech technology
available in all languages of the world, the community requires high-quality
datasets and a unified evaluation benchmark that is shared across researchers
and practitioners. There has been significant progress in the past few years
towards building publicly available multilingual evaluation datasets for
speech understanding [6, 7, 8]. Many research studies have, however, designed
models on different tasks, and evaluated on a small and often disparate set of
languages. This makes comparisons across methods difficult, slows down the
development of multilingual representations, and hinders the evaluation of the
generalization capabilities of such pre-trained models. The goal of this paper
is to structure the evaluation of multilingual speech representation learning.

To address these issues and incentivize the rapidly-evolving research on
general-purpose multilingual speech representation learning, we introduce
XTREME-S, the Cross-lingual Transfer Evaluation of Multilingual Encoders for
Speech benchmark. XTREME-S builds on top of the XTREME series of evaluation
benchmarks for text understanding, with XTREME [9] and XTREME-R [10], which
specialize in the evaluation of multilingual text representations and have
helped the community improve multilingual language understanding, with
impressive performance improvements on a variety of
tasks.222https://sites.research.google/xtreme

XTREME-S is meant to be a more exhaustive, thorough and complete evaluation of
learned speech representations. It covers 102 diverse languages spanning more
than 10 language families and includes four different task families:
recognition, translation, classification and retrieval. The seven downstream
tasks of XTREME-S also cover various domains, from read-speech to
parliamentary speech. It also includes a new general-purpose massively
multilingual evaluation dataset dubbed Fleurs in all of the 102 languages.

Figure 1: XTREME-S is a benchmark for evaluating multilingual speech
representation learning. It covers 4 task families, 3 speech domains and 102
diverse languages. Code and data publicly available at
https://hf.co/datasets/google/xtreme_s.

##  2 Related work

Multilingual representations Self-supervised learning methods like BERT [11],
wav2vec 2.0 [1] or w2v-BERT [12] have been extended to the cross-lingual
setting through mBERT [11], XLM-R [13] or XLS-R [14, 5]. These methods
demonstrate the effectiveness of multilingual understanding in improving low-
resource language representation through unsupervised cross-lingual transfer
from higher-resource languages. Combined with the few-shot learning capability
of wav2vec 2.0 [2], strong self-supervised speech representations can be built
in low-resource languages, enabling training speech recognition systems with
just a few hours of labeled data. XLS-R models demonstrate data-efficient
capabilities in both speech recognition and speech translation for low-
resource languages. Recently, mSLAM [15] built a pre-trained multilingual
model for both speech and text, leading to strong improvements on speech
translation and even better data efficiency in low-resource languages. mSLAM
is evaluated on text downstream tasks from XTREME [9] and tasks from our new
XTREME-S benchmark.

Multilingual speech evaluation There has been a significant body of work on
building trusted multilingual evaluation datasets for speech. IARPA introduced
BABEL [16] for evaluating speech models in low-resource languages. This
dataset has been widely used in the speech community and covers real-world
conversational telephone speech in 17 African and Asian low-resource
languages. Recent work revived this dataset with different preprocessing [17,
18, 19, 14]. The CommonVoice effort [20] offers a wide coverage of speech
recognition data in more than 70 languages, with read speech of Wikipedia and
other sentences. CommonVoice has been used namely for phoneme recognition
[21]. The Multilingual LibriSpeech [6] dataset extends the classical
LibriSpeech task [22] to seven other European languages. VoxPopuli builds
semi-supervised learning data from European Parliament session [7] in 23
languages, and includes speech transcriptions and translations for 16
languages, as well as speech-to-speech translations. With more than 400k hours
of unlabeled speech, VoxPopuli is also used as a public pre-training corpus
[5, 15]. In speech-to-text translation, CoVoST-2 [8] has become one of the go-
to datasets for multilingual evaluation, covering 21 language directions into
English and English into 15 languages. Europarl-ST [23], Must-C [24] and mTEDX
[25] also provide common evaluation of speech translation. LangID can be
evaluated using VoxLingua107 [26] on YouTube data in 107 languages, and CMU
Wilderness [27] on New Testament data in 700+ languages. Fleurs is a new
multilingual speech understanding evaluation dataset in 102 languages.

Multilingual benchmarks For text understanding, GLUE [28] and SuperGLUE [29]
provide common benchmarks for representation learning [30, 31, 32]. Methods
like BERT, or T5 leverage GLUE to show the generalization ability of self-
supervised learning on a variety of tasks. In the multilingual setting, new
evaluation datasets like XNLI [33], MLQA [34] or TyDi QA [35] are grouped in
the XTREME benchmarks [9, 10], on which methods like mBERT, XLM-R or mT5 show
their generalization capabilities across languages. SUPERB [36] attempts to
transpose GLUE to the speech setting, by grouping several common speech tasks
to evaluate English speech models while LeBenchmark [37] is designed for the
evaluation of French self-supervised speech models. Our new XTREME-S benchmark
groups several multilingual speech datasets and is the speech version of
XTREME. The choice of tasks in XTREME-S is motivated by several factors
explained in this work. Most tasks have been already used in previous work as
evaluation for multilingual speech SSL.

##  3 XTREME-S

In this section, we describe the design decisions we made that led to the
choice of tasks, domains and languages for our benchmark. Then we describe
task families and their corresponding datasets.

###  3.1 Design principles

Given XTREME’s goal of providing an accessible benchmark for the evaluation of
cross-lingual transfer learning on a diverse and representative set of tasks
and languages, we select the tasks and languages that make up the benchmark
based on the following principles:

Task difficulty Tasks should be sufficiently challenging that they are not
saturated by the strongest existing baselines. The data should also be
representative of the challenges faced by practitioners, under the constraint
that the data should be publicly accessible.

Diversity We aim for task, domain and language diversity. Tasks should be
diverse and cover several domains to provide a reliable evaluation of model
generalization and robustness to noisy naturally-occurring speech in different
environments. Languages should be diverse to ensure that models can adapt to a
wide range of linguistic and phonological phenomena. Language coverage should
not be unnecessarily large so as to avoid cumbersome evaluations. We note that
the tasks are focused particularly on linguistic aspects of speech, while
nonlinguistic/paralinguistic aspects of speech relevant to e.g. speech
synthesis or voice conversion are not evaluated.

Data efficiency The training sets of XTREME-S range from a few hours to a few
hundred hours of labeled data per language. This is a few-shot setting suited
for low-resource understanding. XTREME-S strongly encourages data-efficient
self-supervised representation learning.

Training efficiency Tasks should be trainable with a reasonable amount of time
(few days) and compute (few GPUs). We enforce that constraint by having
datasets focused on few-shot learning (e.g. Fleurs or MLS). This is to make
the benchmark accessible, in particular to practitioners working under
resource constraints. We also minimize the number of required fine-tuning runs
where we can, for instance by encouraging multilingual fine-tuning over
monolingual fine-tuning.

Monolingual data Unlabeled speech is available publicly through corpora
already used in past work (e.g. MLS, VoxPopuli, CommonVoice). Unlabeled text
data is available in all languages, for instance, through Common Crawl data as
in the mC4
dataset333https://www.tensorflow.org/datasets/catalog/c4#c4multilingual.
Speech data is however not abundant for all languages, so multilinguality is
important to build strong representations for those languages.

Accessibility Each task should be available under a permissive license that
allows the use and redistribution of the data for research purposes. When
needed, we provide scripts to download and easily reproduce the preprocessing
steps. Tasks have also been selected based on their usage by pre-existing
multilingual pre-trained models, for simplicity.

Reproducibility We encourage submissions that leverage publicly available
speech and text datasets. Users should detail which data they use. In general,
we encourage settings that can be reproduced by the community, but also
encourage the exploration of new frontiers for speech representation learning.

Task | Corpus |  $|$ Train $|$ |  $|$ Dev $|$ |  $|$ Test $|$ |  $|$ Lang. $|$ | Fine-tune |  $|$ Eval $|$ | Task | Metric | Domain
---|---|---|---|---|---|---|---|---|---|---
Speech recognition | FLEURS | 999h | 122h | 293h | 102 | Multi | 1 | ASR | CER | Read-speech
MLS | 80h | 10h | 10h | 8 | Multi | 1 | ASR | WER | Read-speech
VoxPopuli | 1300h | 240h | 240h | 14 | Multi | 1 | ASR | WER | Euro Parl
Speech translation | CoVoST-2 | 566h | 144h | 153h | 21 | Multi | 1 | AST | BLEU | Read-speech
Speech classification | FLEURS | 999h | 122h | 293h | 102 | Multi | 1 | LangID | Acc. | Read-speech
Minds-14 | 2h | 1h | 1h | 14 | Multi | 1 | Intent Cl. | Acc. | E-banking
Speech retrieval | FLEURS | 49h | 6h | 14h | 5 | Either | 1/5 | Mining | P@K | Read-speech

Table 1: Characteristics of the datasets in XTREME-S. We report the number of
hours for each train, dev and test set, and the number of languages. We
specify the type of fine-tuning (monolingual or multilingual), which coincides
with the number of fine-tuning runs. We also include the task, the metric and
the speech domain.

###  3.2 Tasks

We present in this section the four task families of XTREME-S and their
corresponding datasets.

####  3.2.1 Speech Recognition (ASR)

For speech recognition, we use three datasets: Fleurs, MLS and VoxPopuli,
which cover more than 100 languages.

Fleurs-ASR Fleurs is the speech version of the FLoRes machine translation
benchmark [38]. We use $2009$ n-way parallel sentences from the FLoRes dev and
devtest publicly available sets, in $102$ languages. We collect between one
and three recordings for each sentence ($2.3$ on average), and build new
train-dev-test splits with $1509$, $150$ and $350$ sentences for train, dev
and test respectively. Training sets have around 10 hours of supervision.
Speakers of the train sets are different than speakers from the dev/test sets.
Multilingual fine-tuning is used and ”unit error rate” (characters, signs) of
all languages is averaged. Languages and results are also grouped into seven
geographical areas: Western Europe (WE), Eastern Europe (EE), Central-
Asian/Middle-East/North-Africa (CMN), Sub-Saharan Africa (SSA), South Asia
(SA), South-Eastern Asia (SEA) and CJK languages (CJK), as reported in Table
8.

MLS The Multilingual LibriSpeech (MLS) dataset is a large corpus derived from
read audiobooks of Librivox and consists of 8 languages: Dutch (nl), English
(en), French (fr), German (de), Italian (it), Polish (pl), Portuguese (pt),
Spanish (es). The latest version of this corpus contains around 50k hours
including 44k hours in English. The task consists of the official 10-hour
splits provided by [6] to evaluate few-shot learning capabilities. We use
multilingual fine-tuning on all languages at once.

VoxPopuli VoxPopuli is a multilingual speech dataset for semi-supervised
learning [7] . It contains 400k hours of unannotated speech as well as speech
transcriptions and translations. We use the 14 languages with more than 10
hours of data from the ASR task. Models are fine-tuned on all 14 languages at
once, ranging from 543 hours of supervision for English to 10 hours for
Slovenian. Word Error Rate (WER) is reported. The language modeling data is
provided by VoxPopuli.

####  3.2.2 Speech Translation (ST)

For speech translation, we use all the 21 language pairs into English from the
CoVoST-2 dataset.

CoVoST-2 CoVoST-2 is a large-scale multilingual speech translation corpus
covering translations from 21 languages into English. This represents the
largest open dataset available to date from total volume and language coverage
perspective. We consider all languages to English, grouped into high/mid/low
labeled data directions. The task has been widely used in recent speech
representation learning [5, 15] and has been recently expanded to cover
speech-to-speech translation [39].

####  3.2.3 Speech classification

For speech classification, we include LangID and intent classification. After
hyperparameter tuning, we encourage reporting the average result over 5 random
seeds.

Fleurs-LangID We use Fleurs as a LangID dataset by using the same train, dev
and test splits as used for ASR. We report over classification accuracy over
the 102 languages.

Minds-14 MINDS-14 [40] is an intent classification task from spoken data. It
covers 14 intents extracted from the e-banking domain, with spoken examples in
14 language varieties. We merge monolingual datasets into a single
multilingual dataset, with a $30$-$20$-$50$% train-dev-test split.

####  3.2.4 Speech retrieval (Optional)

For speech-text ASR retrieval, we use the Fleurs dataset in 5 languages.
Because it is a new task, we mark it as optional.

Fleurs We define a new speech-text ASR retrieval task based on fixed-size
embeddings. For each speech query embedding, the embedding of the correct text
transcription should be retrieved using similarity search (e.g. cosine
similarity), as in bitext mining [41]. For each language, the pool of
transcription candidates is augmented with 100k sentences from Wikipedia. We
encourage the use of a ranking loss for fine-tuning. The average accuracy over
the five languages should be reported. This is an optional new task.

###  3.3 Languages

Our 102 languages cover various language families and geographical locations
(see Table 8), from Western Europe/Americas, Eastern Europe, Central-Asia,
Middle-East, North-Africa, Sub-Saharan Africa, South Asia, South-East Asia to
CJK languages. We have 36 languages covered by at least two evaluation
datasets. The language coverage provides a good estimate of the generalization
ability of multilingual models.

##  4 Results

In this section, we describe our baselines and the corresponding results. We
also comment on the specificities of each downstream task and offer remarks on
how results can be improved.

Model | Speech recognition | Speech translation | Speech classification | Speech retrieval | Avg
---|---|---|---|---|---
Fleurs | MLS | VoxPopuli | CoVoST-2 | Fleurs-LID | Minds-14 | Fleurs-R5
Metrics | WER | WER | WER | BLEU | Acc. | F1 | P@1 | -
XLS-R (0.3B) | - | 12.8 | 12.8 | 13.2 | - | - | - |
w2v-bert-51 (0.6B) | 14.1 | 9.9 | 9.3 | 20.4 | 71.4 | 82.7 | - | 59.1
mSLAM (0.6B) | 14.6 | 10.1 | 9.2 | 20.6 | 73.3 | 86.9 | - | 59.7

Table 2: Table of results for XTREME-S.

###  4.1 Baselines

We present two baselines. The first is a 600M parameter speech-only pre-
trained wav2vec-BERT model trained on 429k unlabeled data in 51 languages from
VoxPopuli, MLS, CommonVoice and BABEL, similar to XLS-R. The second is the
600m parameter mSLAM speech-text pre-trained model that leverages the same
speech data, as well more than 10TiB of unlabeled text data from mC4 and some
ASR supervision. More details on these baselines, including fine-tuning
details can be found in [15]. For some tasks, we also report results of the
XLS-R models from [5]. If capacity constraints become an issue, we encourage
practitioners to use same-capacity apples-to-apples comparisons with the
smaller XLS-R (0.3B) and w2v-bert-51 (0.6B) models.

###  4.2 Speech recognition

Table 3:  Speech Recognition \- Average Character Error Rate (CER) for Fleurs
and average word error rate for the VoxPopuli and MLS-10Hr datasets. Per-
language results can be found in Appendix Tables 4, 10 and 11 respectively.

Model | Fleurs | MLS | VoxPop
---|---|---|---
Prior work [5] |
XLS-R (0.3B) | - | 12.8 | 12.8
XLS-R (2B) | - | 11.0 | -
Our work: Speech-only |
w2v-bert-51 (0.6B) | 14.1 | 9.9 | 9.3
Our work: Speech + Text |
mSLAM (0.6B) | 14.6 | 10.1 | 9.2
mSLAM (2B) | - | 9.7 | 9.1

Table 4:  Speech recognition \- Fleurs massively multilingual ASR baselines,
reporting CER, by geographical group. Observe the discrepancy between European
and African languages.

Model | WE | EE | CMN | SSA | SA | SEA | CJK | All
---|---|---|---|---|---|---|---|---
Number of languages | 25 | 16 | 12 | 20 | 14 | 11 | 4 | 102
Our work: Speech-only, no LM
w2v-bert-51 (0.6B) | 10.7 | 9.9 | 14.5 | 15.6 | 17.4 | 14.7 | 24.6 | 14.1
Our work: Speech + Text, no LM
mSLAM (0.6B) | 10.6 | 10.0 | 14.8 | 16.4 | 19.2 | 14.9 | 25.0 | 14.6

In Table 3, we report average character and word error rates on Fleurs, MLS
and VoxPopuli. We see that mSLAM obtains the best performance on MLS and
VoxPopuli with 9.7 and 9.1 average WER. Pre-trained models obtain strong
performance across domains and on both high-data regimes datasets like
VoxPopuli as well as low-data regimes tasks like Fleurs and MLS. We observe in
Table 4 that results are much better on the Western European group (with 11.5
average WER) than on other groups like Sub-Saharan African (26.7 average WER)
or South Asian (20.7), which can be explained in part due to the larger
amounts of unlabeled data in WE languages from MLS and VoxPopuli. Reducing the
gaps across geographical groups is an important research direction for future
work building on XTREME-S. Per-language results for MLS and VoxPopuli can be
found in Appendix Tables 10 and 11.

###  4.3 Speech translation

Table 5:  Speech translation \- CoVoST 2 X$\rightarrow$En summarized results
in BLEU. Full per-language results are available in the Appendix Table 9.

X $\rightarrow$ English | high | mid | low | all
---|---|---|---|---
Prior work, mBART decoder init. [5]
XLS-R (0.3B) | 30.6 | 18.9 | 5.1 | 13.2
XLS-R (2B) | 36.1 | 27.7 | 15.1 | 22.1
Our Work: Speech Only
w2v-bert-51 (0.6B) | 35.6 | 25.3 | 13.4 | 20.4
Our Work: Speech + Text
mSLAM (0.6B) | 35.5 | 25.2 | 13.7 | 20.6
mSLAM (2B) | 36.3 | 27.5 | 15.6 | 22.4

Average speech translation results are reported in Table 5 and grouped by
high-, mid- and low-resource languages. We observe that baselines perform well
on different data regimes also significantly stronger on high-resource
languages. Unlike previous approaches [8, 42], large-scale pre-trained
multilingual models are able to obtain good performance on low-resource
languages, showing again their few-shot capabilities in the case of speech
translation. For most low-resource languages, only a couple of hours are
available as supervision. Specifically, w2v-bert-51 (0.6B) obtains 13.4 and
mSLAM obtains 15.6 average BLEU on low-resource languages, 35.6 and 36.3 on
high-resource languages. Overall, those models obtain 20.4 and 22.4 average
BLEU respectively on all languages. On this dataset, only one multilingual
fine-tuning run is done to simplify the evaluation. We encourage practitioners
to also try different language re-sampling techniques, or various pre-training
settings of the text decoder, as done for XLS-R. If using additional
supervision, we still encourage reporting results which only leverage the
supervision provided by the CoVoST-2 dataset.

###  4.4 Speech classification

Table 6: Speech Classification \- MINDS-14 speech intent classification and
Fleurs speech language identification accuracy.

Model | Fleurs-LID | Minds-14
---|---|---
Our work: Speech Only
w2v-bert-51 (0.6B) | 71.4 | 82.7
Our work: Speech + Text
mSLAM (0.6B) | 73.3 | 86.9
mSLAM (2B) | 77.7 | 86.6

We report our baselines on the two speech classification datasets in Table 6.
We see that the mSLAM model obtains the best performance overall. Each of
these datasets only require a single fine-tuning run; we build a multilingual
training set from Minds-14 to reduce its inherent variance. Although not
mandatory, we encourage the community to find the best hyperparameters for
their fine-tuning setting, then re-run fine-tuning several times with
different seeds, and report the average to minimize variance.

On Minds-14, mSLAM obtains around 86.6% accuracy, and 77.7% accuracy on Fleurs
LangID, while w2v-bert-51 (0.6B) obtains 82.7 and 71.4 respectively. We note
that on Fleurs-LangID, speakers are different between train sets and dev/test
sets. Avoiding overfitting on speaker ID for the LangID task is essential for
obtaining good performance. In general, speech classification tasks are prone
to overfitting given the discrepancy between the richness of the input signal
(speaker, domain, recording conditions) and the small number of output labels.

Table 7:  Speech retrieval \- FLEURS speech-text retrieval accuracy for
English, Amharic, Hindi, Japanese and Yoruba. Target transcriptions are
retrieved from pools of 100k in-language sentences from Wikipedia or
CommonCrawl.

Model | en | am | hi | ja | yo
---|---|---|---|---|---
Speech-text transcription retrieval |
mSLAM (0.6B) | - | - | - | - | -
Speech-text translation retrieval |
mSLAM (0.6B) | NA | - | - | - | -

###  4.5 Speech retrieval (optional)

Our speech-text ASR retrieval tasks consists of retrieving the correct
transcription or English translation from an input speech utterance. We use
the standard train/dev/test sets of the Fleurs data. The train set can be used
for fine-tuning a siamese network with a pre-trained text and a pre-trained
speech model. The [CLS] tokens of each model are used in the context of a
ranking loss that is trained to match embeddings corresponding to speech-
transcription pairs $(s,t)$ contrasted with negatives $s_{c},t_{c}$:

$max(0,\alpha\ -\ S(s,t)\ +\ 0.5*(S(s_{c},t)+S(s,t_{c})))$

where $S(s,t)$ is a similarity measure of the speech and text embeddings
$(s,t)$, e.g. the cosine similarity. At inference time, after models are fine-
tuned with this ranking loss (or another), all embeddings of the dev and test
sets are computed, as well as all the target text embeddings of 100k sentences
from Wikipedia in corresponding language. The accuracy corresponds to the
number of time the correct transcription/translation is retrieved through
nearest neighbor search from the pool of target sentences (which combine both
the ground-truth dev/test transcriptions and the additional sentences from
Wikipedia or CommonCrawl). Results on this task will be updated in the next
version of the paper. The XTREME-S HuggingFace Dataset tool already provides
the correct splits for this task. We hope this will create a new research path
for speech search and speech retrieval.

##  5 Discussion

In this section, we discuss several components of the XTREME-S benchmark.

On test sets:  Test sets are available in open-source and are not hidden to
the public. We trust practitioners to perform all hyperparameter search and
checkpoint selection on the dev set, and eventually report performance on the
test set. Results are however double-checked through the submission of the
predictions of the model for each task.

On speech data:  We encourage the community to use similar unlabeled speech
datasets across submissions when possible to encourage apple-to-apple
comparisons across models. We do encourage submissions that also use different
unlabeled speech, although preferably only in the case where there is a
substantial difference (e.g. much smaller or much larger, or from more diverse
sources, or using TTS-augmented data etc). Additional unlabeled speech data
can be used for pre-training but also for self-training and other methods.

On text data:  The mC4 and Wikipedia datasets should cover all the languages
of the XTREME-S benchmark, including low-resource ones. We encourage the use
of these datasets for learning language models, for training text-augmented
speech models, or using TTS augmentation for example. We hope the community
can also develop smarter ways to adapt these very large unlabeled text
datasets to each particular task and domain through filtering methods.

On language modeling:  The use of language model decoding is allowed. When
using LMs, results should also be reported without LM fusion for comparison.
The dataset and the type of LM used should be explicitly detailed in
submissions and papers for reproducibility. When doing smart filtering of
unlabeled text data, the technique should be explained clearly and the data
released in open-source when possible.

On the use of external supervision:  At fine-tuning time, we ask that
submissions leverage only the ASR supervision of each task. For instance,
leveraging 10s of thousands of hours of ASR labeled data and then fine-tuning
on MLS-10h English is not a valid submission. Submissions can potentially
leverage all three datasets at once in a multi-task fashion (including during
pre-training as in mSLAM). Additional unlabeled datasets can be used. For
speech translation, additional supervision can be used in the form of open-
sourced text-to-text machine translation data (e.g. from Opus) but any such
data should be detailed explicitly in the submission and paper for clear
comparisons to other methods. The TTS systems used to potentially augment the
training set from the MT data should be reproducible. For speech
classification, the text data of each task can be used at training time but
not at inference time. No other supervision is allowed. For speech retrieval,
we encourage submissions to build generic universal fixed-size speech and text
embeddings by leveraging all kinds of supervision (e.g. more ASR data). We
only ask that new methods be easily reproduced (e.g. they do not use an
unreasonable number of new datasets). In the exception of the exploration of
very large-scale speech pre-training using proprietary data, which is
encouraged and may be considered as a separate track, all extra supervision as
well as unlabeled data should be easily accessible by other teams. The goal of
the benchmark is not to prove that using more supervision leads to better
performance but to discover new speech methods that lead to better data-
efficient performance, in many languages. However, we believe giving more
freedom in the submissions will lead to more interesting discoveries.

On the average score:  We weight differently each task of the XTREME-S
benchmark. Speech recognition and translation each have a weight of 40%, and
speech classification has a weight of 20%. The average score is computed in
the following way:

$0.4*\left(100-\frac{\text{Fleurs}+\text{MLS}+\text{VP}}{3}\right)_{\text{(WER)}}+$
$0.4*\text{CoVoST-2}_{\text{(BLEU)}}+0.2*\left(\frac{\text{F-LID}+\text{M-14}}{2}\right)_{\text{(Acc)}}$

This is to give more importance to the core recognition and translation tasks.

On submission:  As previously mentioned, test sets are not hidden to the
public. This means users can have access to their test results at the end of
their hyperparameter tuning cycle on the dev sets. We ask users to be extra
careful in this process not to inadvertently overfit on the test set.
Additional test sets may be added in the future to confirm the generalization
ability of submissions. We will provide a submission form where results can be
double-checked for consistency before the submission is added to the
leaderboard. More details will be added on the XTREME-S Dataset
card444https://hf.co/datasets/google/xtreme_s.

##  6 Conclusion

We presented XTREME-S, an evaluation benchmark meant to evaluate the
generalization ability of multilingual speech pre-trained models. The
benchmark consists of four key task types: recognition, translation,
classification and retrieval. In total, XTREME-S covers 102 languages with
various language families, from high-resource to low-resource, and different
scripts. Tasks cover several domains and data regimes, from a few hours of
supervision to more than a thousand hours, and are all directly open-sourced
and made easily accessible. We presented two baselines: one speech-only pre-
trained model and one speech-text pre-trained model that obtain strong results
on each task. We believe there remains significant room for improvements on
those tasks, in particular when it comes to reducing the gap between various
language families or groups. We detailed in this paper the design choices of
the XTREME-S benchmark and set guidelines for submissions. We also built a new
dataset named Fleurs, in 102 languages, covering many low-resource languages.
We hope XTREME-S will enable the community to build better speech
representations in many languages, and enable rapid access to data-efficient
speech technology for all the world’s languages.

## References

  * [1] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in _Proc. of NeurIPS_ , 2020.
  * [2] Q. Xu, A. Baevski, T. Likhomanenko, P. Tomasello, A. Conneau, R. Collobert, G. Synnaeve, and M. Auli, “Self-training and pre-training are complementary for speech recognition,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 3030–3034.
  * [3] A. Baevski, W.-N. Hsu, A. Conneau, and M. Auli, “Unsupervised speech recognition,” _arXiv preprint arXiv:2105.11084_ , 2021.
  * [4] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, “Unsupervised cross-lingual representation learning at scale,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Online: Association for Computational Linguistics, Jul. 2020, pp. 8440–8451. [Online]. Available: https://www.aclweb.org/anthology/2020.acl-main.747
  * [5] A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino _et al._ , “Xls-r: Self-supervised cross-lingual speech representation learning at scale,” _arXiv preprint arXiv:2111.09296_ , 2021.
  * [6] V. Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert, “Mls: A large-scale multilingual dataset for speech research,” in _Proc. of Interspeech_ , 2020.
  * [7] C. Wang, M. Riviere, A. Lee, A. Wu, C. Talnikar, D. Haziza, M. Williamson, J. Pino, and E. Dupoux, “VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,” in _Proc. of ACL_ , 2021.
  * [8] C. Wang, A. Wu, and J. Pino, “Covost 2 and massively multilingual speech-to-text translation,” _arXiv_ , 2020.
  * [9] J. Hu, S. Ruder, A. Siddhant, G. Neubig, O. Firat, and M. Johnson, “Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 4411–4421.
  * [10] S. Ruder, N. Constant, J. Botha, A. Siddhant, O. Firat, J. Fu, P. Liu, J. Hu, G. Neubig, and M. Johnson, “Xtreme-r: Towards more challenging and nuanced multilingual evaluation,” _arXiv preprint arXiv:2104.07412_ , 2021.
  * [11] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186. [Online]. Available: https://www.aclweb.org/anthology/N19-1423
  * [12] Y.-A. Chung, Y. Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y. Wu, “W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,” _arXiv preprint arXiv:2108.06209_ , 2021.
  * [13] A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli, “Unsupervised cross-lingual representation learning for speech recognition,” _arXiv_ , vol. abs/2006.13979, 2020.
  * [14] ——, “Unsupervised cross-lingual representation learning for speech recognition,” in _Proc. of Interspeech_ , 2021.
  * [15] A. Bapna, C. Cherry, Y. Zhang, Y. Jia, M. Johnson, Y. Cheng, S. Khanuja, J. Riesa, and A. Conneau, “mslam: Massively multilingual joint pre-training for speech and text,” 2022.
  * [16] M. J. F. Gales, K. M. Knill, A. Ragni, and S. P. Rath, “Speech recognition and keyword spotting for low-resource languages: Babel project research at cued,” in _n Spoken Language Technologies for Under-Resourced Languages_ , 2014.
  * [17] T. Alumäe, D. Karakos, W. Hartmann, R. Hsiao, L. Zhang, L. Nguyen, S. Tsakalidis, and R. Schwartz, “The 2016 bbn georgian telephone speech keyword spotting system,” in _ICASSP_ , 2017.
  * [18] A. Ragni, Q. Li, M. J. F. Gales, and Y. Wang, “Confidence estimation and deletion prediction using bidirectional recurrent neural networks,” in _SLT_ , Athens, 2018.
  * [19] H. Inaguma, J. Cho, M. K. Baskar, T. Kawahara, and S. Watanabe, “Transfer learning of language-independent end-to-end asr with language model fusion,” in _ICASSP_ , 2019.
  * [20] R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” _Proc. of LREC_ , 2020.
  * [21] M. Rivière, A. Joulin, P.-E. Mazaré, and E. Dupoux, “Unsupervised pretraining transfers well across languages,” in _Proc. of ICASSP_ , 2020\.
  * [22] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in _Proc. of ICASSP_. IEEE, 2015, pp. 5206–5210.
  * [23] J. Iranzo-Sánchez, J. A. Silvestre-Cerdà, J. Jorge, N. Roselló, A. Giménez, A. Sanchis, J. Civera, and A. Juan, “Europarl-st: A multilingual corpus for speech translation of parliamentary debates,” in _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 8229–8233.
  * [24] M. A. Di Gangi, R. Cattoni, L. Bentivogli, M. Negri, and M. Turchi, “MuST-C: a Multilingual Speech Translation Corpus,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 2012–2017. [Online]. Available: https://www.aclweb.org/anthology/N19-1202
  * [25] E. Salesky, M. Wiesner, J. Bremerman, R. Cattoni, M. Negri, M. Turchi, D. W. Oard, and M. Post, “The multilingual tedx corpus for speech recognition and translation,” _arXiv preprint arXiv:2102.01757_ , 2021.
  * [26] J. Valk and T. Alumäe, “Voxlingua107: a dataset for spoken language recognition,” in _Proc. of SLT_ , 2020.
  * [27] A. W. Black, “Cmu wilderness multilingual speech dataset,” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 5971–5975.
  * [28] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” in _ICLR_ , 2019.
  * [29] A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Superglue: A stickier benchmark for general-purpose language understanding systems,” _arXiv preprint arXiv:1905.00537_ , 2019\.
  * [30] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _Proc. of NAACL_ , 2019.
  * [31] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” _arXiv_ , vol. abs/1906.08237, 2019.
  * [32] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” _arXiv preprint arXiv:1910.10683_ , 2019\.
  * [33] A. Conneau, R. Rinott, G. Lample, A. Williams, S. R. Bowman, H. Schwenk, and V. Stoyanov, “Xnli: Evaluating cross-lingual sentence representations,” in _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, 2018.
  * [34] P. Lewis, B. Oğuz, R. Rinott, S. Riedel, and H. Schwenk, “Mlqa: Evaluating cross-lingual extractive question answering,” _arXiv preprint arXiv:1910.07475_ , 2019.
  * [35] J. H. Clark, E. Choi, M. Collins, D. Garrette, T. Kwiatkowski, V. Nikolaev, and J. Palomaki, “Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages,” _Transactions of the Association for Computational Linguistics_ , 2020.
  * [36] S. Yang, P. Chi, Y. Chuang, C. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G. Lin, T. Huang, W. Tseng, K. Lee, D. Liu, Z. Huang, S. Dong, S. Li, S. Watanabe, A. Mohamed, and H. Lee, “SUPERB: speech processing universal performance benchmark,” _CoRR_ , vol. abs/2105.01051, 2021. [Online]. Available: https://arxiv.org/abs/2105.01051
  * [37] S. Evain, H. Nguyen, H. Le, M. Zanon Boito, S. Mdhaffar, S. Alisamir, Z. Tong, N. Tomashenko, M. Dinarelli, T. Parcollet, A. Allauzen, Y. Estève, B. Lecouteux, F. Portet, S. Rossato, F. Ringeval, D. Schwab, and L. Besacier, “Task Agnostic and Task Specific Self-Supervised Learning from Speech with LeBenchmark,” in _35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks_ , no. NeurIPS, 2021. [Online]. Available: https://github.com/pytorch/fairseq/blob/main/examples/wav2vec/config/
  * [38] N. Goyal, C. Gao, V. Chaudhary, P.-J. Chen, G. Wenzek, D. Ju, S. Krishnan, M. Ranzato, F. Guzmán, and A. Fan, “The flores-101 evaluation benchmark for low-resource and multilingual machine translation,” 2021.
  * [39] Y. Jia, M. T. Ramanovich, Q. Wang, and H. Zen, “CVSS corpus and massively multilingual speech-to-speech translation,” _arXiv preprint arXiv:2201.03713_ , 2022.
  * [40] D. Gerz, P.-H. Su, R. Kusztos, A. Mondal, M. Lis, E. Singhal, N. Mrkšić, T.-H. Wen, and I. Vulić, “Multilingual and cross-lingual intent detection from spoken data,” _arXiv preprint arXiv:2104.08524_ , 2021.
  * [41] M. Artetxe and H. Schwenk, “Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond,” _Transactions of the Association for Computational Linguistics_ , vol. 7, pp. 597–610, 2019.
  * [42] X. Li, C. Wang, Y. Tang, C. Tran, Y. Tang, J. Pino, A. Baevski, A. Conneau, and M. Auli, “Multilingual speech translation with efficient finetuning of pretrained models,” _arXiv_ , vol. abs/2010.12829, 2021.

Table 8: Characteristics of the 102 languages in XTREME-S, with their ISO
codes, language families, estimated number of speakers in millions (#S) and
number of hours of labeled data for each dataset: Fleurs (FLRS), Multilingual
LibriSpeech (MLS), Vox Populi (VP), CoVoST-2 (CV-2), and Minds-14 (M-14).
Languages are grouped geographically in Western Europe (WE), Eastern Europe
(EE), Central-Asia/Middle-East/North-Africa (CMN), Sub-Saharan Africa (SSA),
South Asia (SA), South-East Asia (SEA) and CJK languages.
Idx | Language | ISO 639-3 | ISO 639-1 | Family | Group | #S | FLRS | MLS | VP | CV-2 | M-14
---|---|---|---|---|---|---|---|---|---|---|---
1 | Afrikaans | afr | af | Indo-European | SSA | 17 |  $\simeq$10 |  |  |  |
2 | Amharic | amh | am | Afro-Asiatic | SSA | 22 |  $\simeq$10 |  |  |  |
3 | Arabic | ara | ar | Afro-Asiatic | CMN | 180 |  $\simeq$10 |  |  | 2 |
4 | Armenian | hye | hy | Indo-European | EE | 6 |  $\simeq$10 |  |  |  |
5 | Assamese | asm | as | Indo-European | SA | 13 |  $\simeq$10 |  |  |  |
6 | Asturian | ast | - | Indo-European | WE | 0.6 |  $\simeq$10 |  |  |  |
7 | Azerbaijani | azj | az | Turkic | CMN | 18 |  $\simeq$10 |  |  |  |
8 | Belarusian | bel | be | Indo-European | EE | 3 |  $\simeq$10 |  |  |  |
9 | Bengali | ben | bn | Indo-European | SA | 260 |  $\simeq$10 |  |  |  |
10 | Bosnian | bos | bs | Indo-European | WE | 9 |  $\simeq$10 |  |  |  |
11 | Bulgarian | bul | bg | Indo-European | EE | 7 |  $\simeq$10 |  |  |  |
12 | Burmese | mya | my | Sino-Tibetan | SEA | 33 |  $\simeq$10 |  |  |  |
13 | Cantonese Chinese | yue | - | Sino-Tibetan | CJK | 920 |  $\simeq$10 |  |  |  |
14 | Catalan | cat | ca | Indo-European | WE | 4 |  $\simeq$10 |  |  | 81 |
15 | Cebuano | ceb | - | Austronesian | SEA | 16 |  $\simeq$10 |  |  |  |
16 | Croatian | hrv | hr | Indo-European | WE | 4 |  $\simeq$10 |  | 43 |  |
17 | Czech | ces | cs | Indo-European | EE | 10 |  $\simeq$10 |  | 62 | 10 | 1
18 | Danish | dan | da | Indo-European | WE | 5 |  $\simeq$10 |  |  |  |
19 | Dutch | nld | nl | Indo-European | WE | 21 |  $\simeq$10 | 10 | 53 | 2 | 2
20 | English | eng | en | Indo-European | WE | 550 |  $\simeq$10 | 10 |  | 543 | 4
21 | Estonian | est | et | Uralic | EE | 1 |  $\simeq$10 |  | 3 | 3 |
22 | Filipino (Tagalog) | tgl | tl | Austronesian | SEA | 22 |  $\simeq$10 |  |  |  |
23 | Finnish | fin | fi | Uralic | WE | 5 |  $\simeq$10 |  | 27 |  |
24 | French | fra | fr | Indo-European | WE | 280 |  $\simeq$10 | 10 | 211 | 180 | 1
25 | Fula | ful | ff | Atlantic-Congo | SSA | 12 |  $\simeq$10 |  |  |  |
26 | Galician | glg | gl | Indo-European | WE | 2 |  $\simeq$10 |  |  |  |
27 | Ganda | lug | lg | Atlantic-Congo | SSA | 4 |  $\simeq$10 |  |  |  |
28 | Georgian | kat | ka | Kartvelian | EE | 4 |  $\simeq$10 |  |  |  |
29 | German | deu | de | Indo-European | WE | 83 |  $\simeq$10 | 10 | 282 | 119 | 2
30 | Greek | ell | el | Indo-European | WE | 13 |  $\simeq$10 |  |  |  |
31 | Gujarati | guj | gu | Indo-European | SA | 56 |  $\simeq$10 |  |  |  |
32 | Hausa | hau | ha | Afro-Asiatic | SSA | 70 |  $\simeq$10 |  |  |  |
33 | Hebrew | heb | he | Afro-Asiatic | CMN | 4 |  $\simeq$10 |  |  |  |
34 | Hindi | hin | hi | Indo-European | SA | 320 |  $\simeq$10 |  |  |  |
35 | Hungarian | hun | hu | Uralic | WE | 13 |  $\simeq$10 |  | 63 |  |
36 | Icelandic | isl | is | Indo-European | WE | 0.3 |  $\simeq$10 |  |  |  |
37 | Igbo | ibo | ig | Atlantic-Congo | SSA | 18 |  $\simeq$10 |  |  |  |
38 | Indonesian | ind | id | Austronesian | SEA | 200 |  $\simeq$10 |  |  | 1 |
39 | Irish | gle | ga | Indo-European | WE | 0.2 |  $\simeq$10 |  |  |  |
40 | Italian | ita | it | Indo-European | WE | 61 |  $\simeq$10 | 10 | 91 | 28 | 3
41 | Japanese | jpn | ja | Japonic | CJK | 130 |  $\simeq$10 |  |  | 1 |
42 | Javanese | jav | jv | Austronesian | SEA | 85 |  $\simeq$10 |  |  |  |
43 | Kabuverdianu | kea | - | Indo-European | WE | 0.9 |  $\simeq$10 |  |  |  |
44 | Kamba | kam | - | Atlantic-Congo | SSA | 4 |  $\simeq$10 |  |  |  |
45 | Kannada | kan | kn | Dravidian | SA | 43 |  $\simeq$10 |  |  |  |
46 | Kazakh | kaz | kk | Turkic | CMN | 11 |  $\simeq$10 |  |  |  |
47 | Khmer | khm | km | Austro-Asiatic | SEA | 16 |  $\simeq$10 |  |  |  |
48 | Korean | kor | ko | Koreanic | CJK | 52 |  $\simeq$10 |  |  |  | 1
49 | Kyrgyz | kir | ky | Turkic | CMN | 8 |  $\simeq$10 |  |  |  |
50 | Lao | lao | lo | Kra-Dai | SEA | 20 |  $\simeq$10 |  |  |  |
51 | Latvian | lav | lv | Indo-European | EE | 2 |  $\simeq$10 |  |  | 2 |
Idx | Language | ISO 639-3 | ISO 639-1 | Family | Group | #S | FLRS | MLS | VP | CV-2 | M-14
---|---|---|---|---|---|---|---|---|---|---|---
52 | Lingala | lin | ln | Atlantic-Congo | SSA | 15 |  $\simeq$10 |  |  |  |
53 | Lithuanian | lit | lt | Indo-European | EE | 2 |  $\simeq$10 |  | 2 |  |
54 | Luo | luo | - | Nilo-Saharan | SSA | 4 |  $\simeq$10 |  |  |  |
55 | Luxembourgish | ltz | lb | Indo-European | WE | 0.4 |  $\simeq$10 |  |  |  |
56 | Macedonian | mkd | mk | Indo-European | EE | 1 |  $\simeq$10 |  |  |  |
57 | Malay | msa | ms | Austronesian | SEA | 80 |  $\simeq$10 |  |  |  |
58 | Malayalam | mal | ml | Dravidian | SA | 77 |  $\simeq$10 |  |  |  |
59 | Maltese | mlt | mt | Afro-Asiatic | WE | 0.5 |  $\simeq$10 |  |  |  |
60 | Mandarin Chinese | cmn | - | Sino-Tibetan | CJK | 80 |  $\simeq$10 |  |  |  | 1
61 | Maori | mri | mi | Austronesian | SEA | 0.2 |  $\simeq$10 |  |  |  |
62 | Marathi | mar | mr | Indo-European | SA | 83 |  $\simeq$10 |  |  |  |
63 | Mongolian | mon | mn | Mongolic | CMN | 5 |  $\simeq$10 |  |  | 3 |
64 | Nepali | npi | ne | Indo-European | SA | 16 |  $\simeq$10 |  |  |  |
65 | Northern Sotho | nso | - | Atlantic-Congo | SSA | 14 |  $\simeq$10 |  |  |  |
66 | Norwegian | nob | nb | Indo-European | WE | 5 |  $\simeq$10 |  |  |  |
67 | Nyanja | nya | ny | Atlantic-Congo | SSA | 12 |  $\simeq$10 |  |  |  |
68 | Occitan | oci | oc | Indo-European | WE | 0.5 |  $\simeq$10 |  |  |  |
69 | Oriya | ory | or | Indo-European | SA | 35 |  $\simeq$10 |  |  |  |
70 | Oromo | orm | om | Afro-Asiatic | SSA | 24 |  $\simeq$10 |  |  |  |
71 | Pashto | pus | ps | Indo-European | CMN | 13 |  $\simeq$10 |  |  |  |
72 | Persian | fas | fa | Indo-European | CMN | 40 |  $\simeq$10 |  |  | 5 |
73 | Polish | pol | pl | Indo-European | EE | 38 |  $\simeq$10 | 10 | 111 |  | 3
74 | Portuguese (Brazil) | por | pt | Indo-European | WE | 220 |  $\simeq$10 | 10 |  | 7 | 3
75 | Punjabi | pan | pa | Indo-European | SA | 113 |  $\simeq$10 |  |  |  |
76 | Romanian | ron | ro | Indo-European | EE | 19 |  $\simeq$10 |  | 89 |  |
77 | Russian | rus | ru | Indo-European | EE | 150 |  $\simeq$10 |  |  | 16 | 1
78 | Serbian | srp | sr | Indo-European | EE | 6 |  $\simeq$10 |  |  |  |
79 | Shona | sna | sn | Atlantic-Congo | SSA | 9 |  $\simeq$10 |  |  |  |
80 | Sindhi | snd | sd | Indo-European | SA | 68 |  $\simeq$10 |  |  |  |
81 | Slovak | slk | sk | Indo-European | EE | 4 |  $\simeq$10 |  | 35 |  |
82 | Slovenian | slv | sl | Indo-European | EE | 2 |  $\simeq$10 |  | 10 | 2 |
83 | Somali | som | so | Afro-Asiatic | SSA | 24 |  $\simeq$10 |  |  |  |
84 | Sorani Kurdish | ckb | - | Indo-European | CMN | 7 |  $\simeq$10 |  |  |  |
85 | Spanish | spa | es | Indo-European | WE | 490 |  $\simeq$10 | 10 | 166 | 97 | 2
86 | Swahili | swh | sw | Atlantic-Congo | SSA | 24 |  $\simeq$10 |  |  |  |
87 | Swedish | swe | sv | Indo-European | WE | 8 |  $\simeq$10 |  |  | 2 |
88 | Tajik | tgk | tg | Indo-European | CMN | 8 |  $\simeq$10 |  |  |  |
89 | Tamil | tam | ta | Dravidian | SA | 76 |  $\simeq$10 |  |  | 2 |
90 | Telugu | tel | te | Dravidian | SA | 82 |  $\simeq$10 |  |  |  |
91 | Thai | tha | th | Kra-Dai | SEA | 20 |  $\simeq$10 |  |  |  |
92 | Turkish | tur | tr | Turkic | CMN | 82 |  $\simeq$10 |  |  | 2 |
93 | Ukrainian | ukr | uk | Indo-European | EE | 32 |  $\simeq$10 |  |  |  |
94 | Umbundu | umb | - | Atlantic-Congo | SSA | 6 |  $\simeq$10 |  |  |  |
95 | Urdu | urd | ur | Indo-European | SA | 120 |  $\simeq$10 |  |  |  |
96 | Uzbek | uzb | uz | Turkic | CMN | 57 |  $\simeq$10 |  |  |  |
97 | Vietnamese | vie | vi | Austro-Asiatic | SEA | 96 |  $\simeq$10 |  |  |  |
98 | Welsh | cym | cy | Indo-European | WE | 0.7 |  $\simeq$10 |  |  | 1 |
99 | Wolof | wol | wo | Atlantic-Congo | SSA | 4 |  $\simeq$10 |  |  |  |
100 | Xhosa | xho | xh | Atlantic-Congo | SSA | 19 |  $\simeq$10 |  |  |  |
101 | Yoruba | yor | yo | Atlantic-Congo | SSA | 21 |  $\simeq$10 |  |  |  |
102 | Zulu | zul | zu | Atlantic-Congo | SSA | 11 |  $\simeq$10 |  |  |  |
Table 9:  Speech translation \- CoVoST 2 X$\rightarrow$En full results in
BLEU.

| High-resource | Mid-resource | Low-resource
---|---|---|---
X $\rightarrow$ English | fr | de | es | ca | fa | it | ru | pt | zh | tr | ar | et
Train Hours | 264h | 184h | 113h | 136h | 49h | 44h | 18h | 10h | 10h | 4h | 2h | 3h
Prior work, mBART Decoder init. [5] |  |  |  |  |
XLS-R (0.3B) | 32.9 | 26.7 | 34.1 | 28.7 | 5.9 | 29.0 | 26.4 | 28.3 | 4.9 | 4.6 | 3.0 | 3.5
XLS-R (2B) | 37.6 | 33.6 | 39.2 | 33.8 | 12.9 | 34.9 | 39.5 | 41.8 | 9.4 | 16.7 | 17.1 | 11.1
Our Work: Speech Only |  |  |  |  |
w2v-bert-51 (0.6B) | 36.9 | 33.1 | 38.9 | 33.5 | 5.8 | 34.9 | 41.8 | 36.1 | 8.0 | 8.8 | 13.7 | 17.4
Our Work: Speech + Text |  |  |  |  |
mSLAM (0.6B) | 36.7 | 32.7 | 39.1 | 33.4 | 6.2 | 35.0 | 41.7 | 34.2 | 8.7 | 11.7 | 13.3 | 17.2
mSLAM (2B) | 37.6 | 33.8 | 39.5 | 34.4 | 8.8 | 36.1 | 43.6 | 42.0 | 7.1 | 19.7 | 15.8 | 18.6

| Low-resource | Average
---|---|---
X $\rightarrow$ English | mn | nl | sv | lv | sl | ta | ja | id | cy | high | mid | low | all
Train Hours | 3h | 7h | 2h | 2h | 2h | 2h | 2h | 2h | 2h |  |  |  |
Prior work [5] |  |  |  |  |  |
XLS-R (0.3B) | 0.4 | 22.0 | 10.3 | 6.0 | 6.6 | 0.2 | 0.6 | 1.4 | 2.5 | 30.6 | 18.9 | 5.1 | 13.2
XLS-R (2B) | 1.6 | 31.7 | 29.6 | 19.5 | 19.6 | 0.5 | 3.5 | 16.5 | 14.0 | 36.1 | 27.7 | 15.1 | 22.1
Our Work: Speech Only |  |  |  |  |  |
w2v-bert-51 (0.6B) | 0.3 | 33.8 | 33.9 | 16.0 | 25.5 | 0.3 | 0.9 | 3.5 | 6.2 | 35.6 | 25.3 | 13.4 | 20.4
Our Work: Speech + Text |  |  |  |  |  |
mSLAM (0.6B) | 0.5 | 32.5 | 32.1 | 18.6 | 25.0 | 0.3 | 1.7 | 3.7 | 6.8 | 35.5 | 25.2 | 13.7 | 20.6
mSLAM (2B) | 0.3 | 34.4 | 35.5 | 22.8 | 29.2 | 0.3 | 1.7 | 4.7 | 4.4 | 36.3 | 27.5 | 15.6 | 22.4

Table 10:  Speech recognition \- Multilingual LibriSpeech (MLS) ASR baselines
in 8 languages, reporting WER.

Model | en | de | nl | fr | es | it | pt | pl | Avg
---|---|---|---|---|---|---|---|---|---
Number of training hours | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | -
Prior work (monolingual fine-tuning) [5] |
XLS-R(0.3B) | 15.9 | 9.0 | 13.5 | 12.4 | 8.1 | 13.1 | 17.0 | 13.9 | 12.8
XLS-R(2B) | 14.0 | 7.6 | 11.8 | 10.0 | 6.9 | 12.1 | 15.6 | 9.8 | 11.0
Our work: Speech Only (multilingual fine-tuning) |
w2v-bert-51 (0.6B) | 12.7 | 7.0 | 12.6 | 8.9 | 5.9 | 10.3 | 14.6 | 6.9 | 9.9
Our work: Speech + Text (multilingual fine-tuning) |
mSLAM (0.6B) | 13.3 | 7.0 | 12.5 | 9.7 | 5.5 | 10.5 | 14.1 | 8.5 | 10.1
mSLAM (2B) | 11.9 | 6.6 | 12.4 | 8.5 | 5.8 | 9.8 | 15.2 | 7.7 | 9.7

Table 11:  Speech recognition  \- VoxPopuli ASR results in terms of WER. | en | de | it | fr | es | pl | ro | hu
---|---|---|---|---|---|---|---|---
Labeled data | 543h | 282h | 91h | 211h | 166h | 111h | 89h | 63h
Prior work [5] |
XLS-R (0.3B) | 10.2 | 13.0 | 19.2 | 12.6 | 9.8 | 9.6 | 7.9 | 11.6
XLS-R (1B) | 8.8 | 11.5 | 15.1 | 10.8 | 8.2 | 7.7 | 7.3 | 9.6
Our work: Speech-only |
w2v-bert-51 (0.6B) | 7.2 | 9.0 | 15.8 | 9.2 | 8.6 | 6.5 | 7.6 | 8.4
Our work: Speech + Text |
mSLAM (0.6B) | 7.1 | 8.9 | 15.6 | 9.3 | 8.6 | 6.5 | 8.5 | 8.1
mSLAM (2B) | 7.0 | 8.7 | 15.4 | 9.4 | 8.4 | 6.4 | 7.8 | 8.4
| nl | cs | sl | fi | hr | sk | Avg |
Labeled data | 53h | 62h | 10h | 27h | 43h | 35h |  |
Prior work [5] |
XLS-R (0.3B) | 14.8 | 10.5 | 24.5 | 14.2 | 12.3 | 8.9 | 12.8 |
XLS-R (1B) | 12.5 | 8.7 | 19.5 | 11.3 | 10.0 | 7.1 | 10.6 |
Our work: Speech-only |
w2v-bert-51 (0.6B) | 10.5 | 7.0 | 15.8 | 9.3 | 9.1 | 6.0 | 9.3 |
Our work: Speech + Text |
mSLAM (0.6B) | 10.3 | 7.0 | 14.2 | 9.2 | 9.1 | 5.9 | 9.2 |
mSLAM (2B) | 10.5 | 6.8 | 15.1 | 8.7 | 9.1 | 6.0 | 9.1 |
Table 12:  FLEURS full ASR results . We report per-language results for all
geographical language groups. FLEURS is a dataset that is complete at more
than 97%. Some slight improvements and changes may be done in a 2nd version of
the dataset (e.g. missing recordings, or replaced low-quality recordings).
Updates will be made on our platform. We expect average results not to change
significantly.

| Western European
---|---
Language | ast | bs | ca | hr | da | nl | en | fi | fr | gl | de | el | hu | is | ga
w2v-bert-51 (0.6B) | 8.7 | 5.8 | 4.3 | 9.3 | 11.3 | 6.0 | 17.2 | 3.0 | 9.6 | 8.6 | 8.0 | 11.7 | 24.9 | 11.9 | 39.5
mSLAM (0.6B) | 7.5 | 5.1 | 4.7 | 8.5 | 14.0 | 6.8 | 16.3 | 3.4 | 9.7 | 8.7 | 5.7 | 12.0 | 18.1 | 12.8 | 40.5

| Western European (WE) | Eastern European
---|---|---
Language | it | kea | lb | mt | nb | oc | pt | es | sv | cy | am | be | bg | cs | et
w2v-bert-51 (0.6B) | 2.6 | 4.9 | 19.4 | 17.3 | 5.8 | 11.7 | 4.2 | 3.7 | 7.6 | 11.1 | 17.2 | 9.1 | 4.8 | 10.3 | 3.1
mSLAM (0.6B) | 2.3 | 5.1 | 21.0 | 17.3 | 6.1 | 12.7 | 4.4 | 3.3 | 7.8 | 12.0 | 17.8 | 7.5 | 5.2 | 9.2 | 3.5

| Eastern European (EE) | Central-Asia and
---|---|---
Language | ka | lv | lt | mk | pl | ro | ru | sr | sk | sl | uk | ar | az | he | kk
w2v-bert-51 (0.6B) | 30.7 | 4.4 | 12.8 | 11.8 | 5.0 | 8.0 | 5.6 | 11.6 | 4.9 | 7.9 | 21.4 | 10.5 | 12.7 | 37.2 | 6.5
mSLAM (0.6B) | 31.0 | 4.5 | 11.6 | 9.8 | 6.3 | 8.4 | 6.6 | 12.2 | 4.8 | 10.3 | 21.4 | 11.0 | 15.9 | 42.5 | 5.7

| Middle-East and North-Africa (CMN) | Sub-Saharan Africa
---|---|---
Language | ky | mn | ps | fa | ckb | tg | tr | uz | af | am | ff | lg | ha | ig | kam
w2v-bert-51 (0.6B) | 8.3 | 15.2 | 20.4 | 15.7 | 15.1 | 7.1 | 8.5 | 16.8 | 9.5 | 17.2 | 27.8 | 12.4 | 9.8 | 18.1 | 13.5
mSLAM (0.6B) | 8.0 | 16.1 | 21.1 | 10.0 | 15.0 | 7.6 | 9.7 | 15.5 | 11.9 | 17.8 | 27.5 | 12.9 | 10.5 | 18.7 | 14.0

| Sub-Saharan Africa (SSA) | South-Asia
---|---|---
Language | ln | luo | nso | ny | om | sn | so | sw | umb | wo | xh | yo | zu | as | bn
w2v-bert-51 (0.6B) | 6.1 | 7.0 | 11.7 | 11.5 | 21.7 | 16.6 | 21.3 | 19.4 | 13.1 | 17.8 | 23.9 | 23.3 | 9.8 | 13.7 | 9.4
mSLAM (0.6B) | 6.8 | 7.4 | 11.9 | 12.4 | 22.6 | 17.6 | 23.4 | 20.2 | 14.0 | 18.8 | 25.1 | 23.2 | 10.8 | 14.0 | 9.7

| South-Asia (SA) | South-East Asia
---|---|---
Language | gu | hi | kn | ml | mr | ne | or | pa | sd | ta | te | ur | my | ceb | tl
w2v-bert-51 (0.6B) | 9.3 | 12.4 | 7.0 | 8.6 | 14.8 | 13.0 | 19.2 | 13.6 | 16.0 | 11.8 | 12.0 | 82.9 | 18.2 | 5.9 | 7.1
mSLAM (0.6B) | 9.6 | 15.2 | 9.6 | 12.2 | 18.9 | 14.8 | 20.7 | 15.2 | 20.8 | 13.2 | 12.3 | 83.1 | 18.8 | 6.2 | 7.6

| South-East Asia (SEA) | CJK |  |  |
---|---|---|---|---|---
Language | id | jv | km | lo | ms | mi | th | vi | yue | cmn | ja | ko | All |  |
w2v-bert-51 (0.6B) | 5.2 | 7.0 | 29.9 | 38.1 | 8.6 | 10.3 | 18.6 | 14.2 | 37.0 | 22.2 | 37.7 | 21.7 | 14.1 |  |
mSLAM (0.6B) | 5.6 | 7.1 | 30.2 | 37.5 | 7.2 | 11.2 | 20.1 | 14.3 | 39.8 | 23.1 | 39.2 | 22.4 | 14.6 |  |