# SLAM: A Unified Encoder for Speech and
Language Modeling via Speech-Text Joint
Pre-Training

Ankur Bapna∗1, Yu-An Chung∗1,2, Nan Wu1,3, Anmol Gulati1, Ye Jia1,
Jonathan H. Clark1, Melvin Johnson1, Jason Riesa1, Alexis Conneau∗1, Yu
Zhang∗1
1Google Research
2MIT Computer Science and Artificial Intelligence Laboratory
3Center for Data Science, New York University

###### Abstract

Unsupervised pre-training is now the predominant approach for both text and
speech understanding. Self-attention models pre-trained on large amounts of
unannotated data have been hugely successful when fine-tuned on downstream
tasks from a variety of domains and languages. This paper takes the
universality of unsupervised language pre-training one step further, by
unifying speech and text pre-training within a single model. We build a single
encoder with the BERT objective on unlabeled text together with the w2v-BERT
objective on unlabeled speech. To further align our model representations
across modalities, we leverage alignment losses, specifically Translation
Language Modeling (TLM) and Speech Text Matching (STM) that make use of
supervised speech-text recognition data. We demonstrate that incorporating
both speech and text data during pre-training can significantly improve
downstream quality on CoVoST 2 speech translation, by around 1 BLEU compared
to single-modality pre-trained models, while retaining close to SotA
performance on LibriSpeech and SpeechStew ASR tasks. On four GLUE tasks and
text-normalization, we observe evidence of capacity limitations and
interference between the two modalities, leading to degraded performance
compared to an equivalent text-only model, while still being competitive with
BERT. Through extensive empirical analysis we also demonstrate the importance
of the choice of objective function for speech pre-training, and the
beneficial effect of adding additional supervised signals on the quality of
the learned representations.

00footnotetext: ∗Equal contribution. Correspondence to {ankurbpn, aconneau,
<EMAIL_ADDRESS>
##  1 Introduction

Self-supervised learning of text and speech representations has been
particularly impactful in natural language processing and speech processing.
Since GPT (Radford et al., 2018), BERT (Devlin et al., 2019) and their
variations (Yang et al., 2019; Conneau & Lample, 2019; Lewis et al., 2020;
Raffel et al., 2020; Joshi et al., 2020), performance on natural language
understanding downstream tasks (Socher et al., 2013; Rajpurkar et al., 2016;
Agirre et al., 2007; Williams et al., 2018) and monolingual (e.g., GLUE (Wang
et al., 2019b), SuperGLUE (Wang et al., 2019a)) and multilingual (e.g., XTREME
(Hu et al., 2020), XTREME-R (Ruder et al., 2021)) benchmarks has largely
improved thanks to evolving pre-trained models, which leverage increasing
amounts of unannotated data (Radford et al., 2019; Liu et al., 2019; Conneau
et al., 2019; Wenzek et al., 2020; Xue et al., 2021b) and increased model
capacity (Brown et al., 2020; Xue et al., 2021b; Lepikhin et al., 2020; Fedus
et al., 2021). Similarly for speech, unsupervised pre-training has emerged as
a predominant approach. Wav2vec 2.0 (Baevski et al., 2020b) and newer variants
(Zhang et al., 2020) initially showed the strength of pre-training on speech
recognition (Panayotov et al., 2015; Kahn et al., 2020; Zhang et al., 2021) on
multiple domains (Hsu et al., 2021) and languages (Conneau et al., 2020).

Self-supervised learning methods in language understanding are designed to be
used universally, i.e. a single large pre-trained model for all domains and
languages. One big advantage of these universal models is the ability to
leverage data skew across domains, tasks and languages; the availability of
task or domain-specific data in one language can boost model performance for
several languages that the model was pre-trained on. Extending this
generalization capability across modalities by having neural networks
understand both text and speech at the same time is a natural next step.

Jointly pre-training models on speech and text is a natural choice for
multimodal self-supervised learning, given the similarities between the two
modalities and the abundance of unannotated text data compared to speech.
Recent work has also shown that self-supervised speech representations can be
aligned to text with little to no supervision (Baevski et al., 2021),
suggesting the possibility of learning both modalities within a single neural
network. However, past work in multilingual modeling in particular has
demonstrated the difficulty of learning representations of different data
structures, however similar, within a shared network, exposing the so-called
transfer interference problem (Arivazhagan et al., 2019). We show in this work
that this trade-off also applies to joint speech-text self-supervised
learning.

We study a new multimodal speech-text pre-training approach that leverages
data from one modality to improve representations of the other, but also
suffers from transfer interference and capacity dilution. Our _Speech and
LAnguage Model (SLAM)_ consists of a single Conformer (Gulati et al., 2020)
trained with the SpanBERT objective for text (Joshi et al., 2020) and the
w2v-BERT (Chung et al., 2021) objective for speech. We show that a model using
only self-supervised objectives leads to good performance on both modalities,
but is outperformed by mono-modal pre-trained models, suffering from
significant transfer interference. To reduce the gap, we leverage supervised
alignment losses, specifically a translation language model (Conneau & Lample,
2019; Zheng et al., 2021) and speech-text matching (Li et al., 2021) loss. We
train our model in a multi-task fashion with the self-supervised and alignment
losses. This leads to performance competitive with the state-of-the-art on
SpeechStew and LibriSpeech ASR and on CoVoST 2 speech translation tasks. On
speech translation, we demonstrate further quality improvements by continuing
pre-training on speech-only, outperforming previous approaches by 1 BLEU on
average. On text tasks, our joint model loses quality compared to equivalent
mono-modal pre-trained models, but remains competitive with initial BERT
results (Devlin et al., 2019), demonstrating the capacity limitations with
modeling two high-resource modalities simultaneously. To the best of our
knowledge, our work is the first to study and underline the benefits and
limitations of speech-text unsupervised pre-training over mono-modal models,
on various speech and text downstream tasks. Our initial results set a new
challenge in multimodal self-supervised language understanding.

##  2 Related Work

Self-supervised learning of language representations using neural networks has
a long history. In the deep learning era, word2vec (Mikolov et al., 2013)
initially trained word representations from unannotated data using noise
contrastive estimation (Gutmann & Hyvärinen, 2012; Mnih & Teh, 2012). Word2vec
was followed by a series of papers that expanded the approach to contextual
representations of sentences, including ELMo (Peters et al., 2018), GPT
(Radford et al., 2018), BERT (Devlin et al., 2019) and T5 (Raffel et al.,
2019). They rely on either generative language modelling (Bengio et al., 2003)
or masked language modeling (Taylor, 1953) (MLM) and these self-supervised
pre-training approaches have led to significant improvements on a wide variety
of downstream tasks (Wang et al., 2019b; Hu et al., 2020).

In parallel, similar approaches were explored in speech understanding. Chung
et al. (2016) follows the word2vec approach to learn vector representations of
variable-length audio segments. Oord et al. (2018) introduces contrastive
predictive coding (CPC) which leverages language modeling and negative
sampling to learn speech representations. The first wav2vec model (Schneider
et al., 2019) closely follows this architecture using a noise contrastive
binary classification task for unsupervised pre-training. vq-wav2vec (Baevski
et al., 2020a) proposes to add a vector quantizer similar to VQ-VAE (van den
Oord et al., 2018), using Gumbel softmax (Jang et al., 2016) or online k-means
clustering to quantize the dense speech representations (Eloff et al., 2019).
When quantized, speech utterances become sequences of discrete tokens
belonging to a fixed vocabulary, similar to text, on which BERT is applied.
wav2vec 2.0 merges those two separate steps (quantization and contrastive
learning) into a unified end-to-end learning procedure that pre-trains a
Transformer model. They show significant gains on LibriSpeech (Panayotov et
al., 2015) as well as on few-shot learning for low-resource languages (Conneau
et al., 2020). w2v-BERT (Chung et al., 2021) expands wav2vec 2.0 by combining
contrastive learning and MLM. Zhang et al. (2020) and BigSSL (Zhang et al.,
2021) explore the limits of large-scale semi-supervised learning with
Conformers (Gulati et al., 2020).

One approach to utilize data across modalities could involve synthetically
transforming the modality of the data; one example being Chen et al. (2021)
where the authors utilize text-to-speech (TTS) to transform text data into
speech, and utilize it for monomodal speech pre-training. Recent advances in
self-supervised learning for text, speech and images have led to a new
frontier: multimodal self-supervised learning, where a single model learns
representations of all modalities using both unannotated and aligned data.
VATT Transformer (Akbari et al., 2021) leverages datasets of more than 100M
video-audio-text triplets to learn representations on all modalities at once
with noise contrastive estimation. Li et al. (2021) jointly learns to do
masked language modeling on text as well as matching image representations to
text with parallel data through alignment losses. Jia et al. (2021) learns
language representation for text-to-speech synthesis by jointly training on
phoneme and grapheme representations with MLM. Perhaps most similar to our
work, Zheng et al. (2021) learn joint speech-text representations by adapting
a translation language modeling (TLM) loss (Conneau & Lample, 2019) to the
speech-text setting and studies downstream effect on speech translation.

This work investigates the possibility of developing truly multimodal pre-
trained models building on state-of-the-art speech and text pre-training
approaches, and highlights the advantages and challenges associated with
multimodal pre-trained models by evaluating on a variety of speech and text
downstream tasks.

##  3 Method

In this section, we describe each component of our speech-text pre-training
framework, SLAM, starting with the model architecture in Section 3.1. We then
present the pre-training objectives and our multi-stage pre-training strategy
in Sections 3.2 and 3.4, followed by introducing the pre-training data in
Section 4.1. Figure 1 illustrates the overall pre-training framework.

###  3.1 Model Architecture

Our model contains a speech encoder, a text encoder, and a multimodal encoder.
At a high level, the speech and text encoders take speech and text signals as
input respectively and extract latent features from them. The latent features
from the two modalities are then fed to the same multimodal encoder for
learning speech-text joint representations. Next, we describe each of these
components.

##### Speech Encoder

The speech encoder is composed of a convolutional feature encoder followed by
a stack of Conformer layers (Gulati et al., 2020), each of which is a series
of multi-headed self attention (Vaswani et al., 2017), depth-wise
convolutions, and feed-forward layers. Given an acoustic feature sequence
$A=(a_{1},a_{2},...,a_{N})$ (we use 80-dimensional log Mel spectrograms in
this work, i.e., $a_{i}\in\mathbb{R}^{80}$), the feature encoder—which
consists of two 2D-convolutional layers111We combine information along the
time and the Mel spectrogram frequency dimensions. both with strides
$(2,2)$—acts as a sub-sampling block that extracts latent speech
representations $X=(x_{1},x_{2},...,x_{T})$ with a 4x reduction in the
sequence length from the input $A$. A linear projection layer is then applied
to map the latent representations’ dimensionality to that of the subsequent
Conformer stack, which further extracts higher-level contextualized speech
representations $C=(c_{1},c_{2},...,c_{T})$ from the feature encoder output
$X$. For the Conformer stack, we follow the layout described in Gulati et al.
(2020), with a model dimension of $1024$, feedforward hidden dimension of
$4096$, convolution kernel size $5$ and $8$ attention heads. Different from
Gulati et al. (2020), we use group normalization instead of batch norm in our
convolution layers which was empirically found to perform better on multimodal
mixed batch training. For all our w2v-BERT experiments we use 8 layers in the
speech-specific conformer layer stack ($N=8$ in Figure 1). Outputs of the
speech encoder are layer normalized before being fed to the multimodal
encoder. When training with wav2vec 2.0, we skip the additional Conformer
block in the speech encoder.

##### Text Encoder

The text encoder is a simple token embedding layer that transforms input text
into a sequence of token vector embeddings
$W=(w_{1},w_{2},...,w_{T^{\prime}})$. 222We evaluated using a deep Transformer
or Conformer stack for the text encoder but did not find it empirically useful
for speech translation or ASR. The textual tokens are combined with sinusoidal
positional encodings and layer normalized before being fed to the multimodal
encoder. We utilize a SentencePiece model (Kudo & Richardson, 2018) with a
$32k$ token vocabulary.

##### Multimodal Encoder

The multimodal encoder is a deep stack of Conformer layers that can take
either just speech, or just text, or concatenated speech-text pairs as input.
The Conformer layers used in the multimodal encoder are identical to the ones
used in the speech encoder. When training with w2v-BERT we use $M=16$
Conformer layers in the multimodal stack, while we use $M=24$ layers when
training with wav2vec 2.0. Depending on the type of input - i.e. just speech,
text or a speech-text pair - the model is tasked to solve different self-
supervised pre-training objectives.

Figure 1: (Left) Our model consists of a text embedding layer and a speech-
specific stack similar to w2v-BERT, the latter consisting of a ConvNet and a
series of $N$ Conformer layers. Both the text and speech output embeddings are
fed to a series of $N$ shared Conformer layers. Our unsupervised speech-text
pre-training approach consists of self-supervised learning objectives (in
blue), including w2v-BERT masked language modeling and contrastive losses, as
well as the text BERT objective. This can be combined with supervised
alignment losses (in red) which leverage speech-text annotated pairs. We
leverage in particular the MLM variant of translation language modeling (TLM)
and the ranking loss of speech-text matching (STM). (Right) Once pre-trained,
the speech part of the shared architecture can be fine-tuned on speech
understanding datasets like recognition or translation. The text part of the
architecture can be fine-tuned on language understanding tasks.

###  3.2 Pre-Training Objectives

We pre-train the model with four objectives: SpanBERT (Joshi et al., 2020) on
unlabeled text, w2v-BERT (Chung et al., 2021) on unlabeled speech, Translation
Language Modeling (Conneau & Lample, 2019; Zheng et al., 2021) on paired
speech and text data, and Speech-Text Matching (Li et al., 2021) on paired and
non-paired speech and text data.

####  3.2.1 Self-supervised learning objectives

We use two self-supervised learning objectives that are trained on unannotated
text or speech data.

BERT is the self-supervised learning objective applied to unannotated text
input (Devlin et al., 2019). It aims to learn contextualized textual
representations via solving a masked language modeling (MLM) task. We mask
spans of text as in SpanBERT (Joshi et al., 2020).

##### w2v-BERT

is the self-supervised learning objective used for pre-training on unannotated
speech data (Chung et al., 2021). It combines contrastive learning and MLM,
where the former trains the model to discretize continuous speech signals into
a finite set of discriminative speech tokens, and the latter trains the model
to learn contextualized speech representations via solving a masked prediction
task consuming the discretized tokens.

####  3.2.2 Alignment losses

Without the presence of paired data, the only incentive for the model to learn
joint representation is the inductive bias of having a shared set of Conformer
layers. Because this is such a strong assumption, we also leverage alignment
losses, which use speech-text paired ASR data to explicitly incentivize the
model to share representations within the model. We will see below that this
leads to better alignment between the speech and text representations, as
indicated by better performance on downstream tasks.

##### Translation Language Modeling (TLM)

was first introduced to align representations between two languages within a
shared Transformer. With TLM, parallel sentences are concatenated and sent to
a Transformer MLM which predicts missing words, encouraging the model to
leverage context from both input languages. In this work, we concatenate
speech utterances with their transcriptions using ASR supervised data, similar
to Zheng et al. (2021). We then train the model to predict masked text or
speech spans with BERT or w2v-BERT, encouraging the use of cross-modal
context.

##### Speech-Text Matching (STM)

predicts whether a pair of speech and text is positive (matched) or negative
(not matched). We use the multimodal encoder’s output embedding of the [CLS]
token as the joint representation of the speech-text pair, and append a fully-
connected (FC) layer followed by softmax to predict a two-class probability
$p^{\text{STM}}$. The STM loss is:

$\mathcal{L}_{\text{STM}}=\mathbb{E}_{(I,T)\sim\mathcal{D}}\left[\text{H}\left(y^{\text{STM}},\hskip
4.0ptp^{\text{STM}}(I,T)\right)\right]$

where $y^{\text{STM}}$ is a 2-dimensional one-hot vector representing the
ground-truth label, and H is cross-entropy. The STM objective explicitly
trains the model to align speech-text pairs, a signal which self-supervised
learning cannot explicitly provide.

###  3.3 Implementation Details

When the input only contains speech, the latent speech features
$X=(x_{1},x_{2},...,x_{T})$ extracted by the speech encoder are directly fed
to the multimodal encoder as input. The speech branch of the model (i.e., the
speech encoder along with the multimodal encoder) is trained to optimize the
w2v-BERT objective. Following Chung et al. (2021), we mask approximately
$50\%$ of the speech frames with spans of length $10$. Analogously, when the
input only contains text, the latent textual features
$W=(w_{1},w_{2},...,w_{T^{\prime}})$ extracted by the text encoder are fed to
the multimodal encoder as input, and the text branch (i.e., the text encoder
along with the multimodal encoder) is trained to optimize the SpanBERT
objective. We mask $15\%$ of text tokens with spans of length $5$.

When the input is a speech-text pair, the latent speech and text
representations $C$ and $W$ extracted respectively by the speech and text
encoders are concatenated, forming a sequence with a total length of
$T+T^{\prime}$ that is fed to the multimodal encoder as input. The multimodal
encoder is then trained to simultaneously predict the masked speech features
(as in the w2v-BERT objective) and masked text features (SpanBERT). We use
more aggressive masking when using paired data to increase the difficulty-
level of the task, and to encourage the multimodal encoder to learn to extract
useful features across modalities. We mask a single span consisting of $50\%$
of text tokens, and multiple spans masking out $75\%$ of the speech features
when training with paired data.

We train the model simultaneously on all these objectives; at every training
step the model is trained on a batch of (i) unlabeled speech, (ii) unlabeled
text, and (iii) paired speech and text. The gradients of all objectives are
aggregated and used to update the model parameters.

###  3.4 Multi-Stage Pre-Training

In practice, we find that pre-training the model simultaneously with unpaired
and paired data results in the model overfitting to the relatively small
paired dataset. To avoid this we pre-train the model in a multi-stage fashion,
where we first pre-train the model just on unpaired text and speech, and then
optimize it with unpaired and paired data simultaneously. This multi-stage
pre-training approach achieves better downstream performance than optimizing
all four losses from scratch. Concretely, we train on 500k updates with the
self-supervised losses, and between 250k and 500k additional steps with the
alignment losses. We observed improvements of 0.1 to 0.2 WER on LibriSpeech
dev-other and 0.3 average BLEU on CoVoST 2 when using the multi-stage strategy
as against training with all losses from scratch. In all models that use TLM
and/or STM, we utilize multi-stage pre-training.

##  4 Experiments

We first describe our pre-training and fine-tuning setup, including the speech
and text datasets used for pre-training as well as all our downstream tasks.
We then present our results, including ablations of our approach and
comparisons between multimodal and mono-modal models.

###  4.1 Pre-training Data

Libri-light (speech only): The Libri-light (LL-60k) dataset contains 60k hours
of unlabeled speech and is used to pre-train all our Masked Speech Models
(MSM). LL-60k is the most widely used large unsupervised speech corpus for
various pre-training techniques. Each input speech sequence is constructed by
first randomly selecting 32-64 seconds segments from the original utterance.
From these segments, a contiguous 32 second region is extracted from a random
starting point on-the-fly during MSM pre-training as described in (Zhang et
al., 2020)

LibriLM (text only): The Librispeech text corpus comprises of nearly 803
million tokens from 40M utterances of filtered text derived from 14.5K Project
Gutenberg books (Panayotov et al., 2015).

mC4-En (text only): The mC4-En dataset (Xue et al., 2021a) consists of
multiple terabytes of English text data, mined from CommonCrawl. The dataset
is publicly available.333https://huggingface.co/datasets/mc4

LibriSpeech (paired data): We use LibriSpeech (Panayotov et al., 2015) fullset
(960h) as paired data for Translation Language Modeling (TLM) and Speech-Text
Matching (STM).

###  4.2 Downstream Tasks

We present results on publicly available, well-benchmarked downstream tasks
including speech recogntion, speech translation, text normalization and
language understanding.

##### Speech translation:

CoVoST 2 (Wang et al., 2021a) is a large-scale multilingual speech translation
corpus covering translations from 21 languages into English and from English
into 15 languages. This represents the largest open dataset available to date
from total volume and language coverage perspective. Following Wang et al.
(2021b), we use four English to X directions, specifically German, Catalan,
Arabic and Turkish. To evaluate our pre-trained encoders on speech-
translation, we fine-tune it as part of a sequence-to-sequence model with a
$4$-layer Transformer decoder. The decoder uses a $384$ embedding dimension,
$1536$ feed-forward hidden dimension, $4$ attention heads and a $8192$ token
multilingual sub-word vocabulary.

##### ASR:

SpeechStew has 6 public benchmarks, including LibriSpeech (Panayotov et al.,
2015), AMI (Carletta et al., 2005), TEDLIUM (Rousseau et al., 2012), Common
Voice (Ardila et al., 2019), Switchboard/Fisher (Cieri et al., 2003) and Wall
Street Journal (LDC93S6B, LDC94S13B). In our experiments, LibriSpeech is the
same domain as our pre-training speech data (Libri-light) and others are
evaluated as out-of-domain corpora. Following Chan et al. (2021a), the whole
dataset (approx. 5k hours) is used to perform our finetuning experiments. Each
dataset used is specific to a certain target data condition, for instance
LS-960 closely matches LL-60k, AMI dataset is distinct from the LL-60k
condition and it contains speech from two kinds of microphones: ($\mathrm{i}$)
independent head microphone (IHM); and ($\mathrm{ii}$) single distant
microphone (SDM). SpeechStew is composed of datasets chosen from multiple
conditions to create a mixed domain aggregate corpus. Details of its
processing are described in Chan et al. (2021a). To evaluate on ASR, we fine-
tune our encoder as a conformer-transducer model, following Chung et al.
(2021).

##### Language understanding:

We consider four main tasks from the GLUE natural language understanding
benchmark: the MNLI natural language inference benchmark (Williams et al.,
2018), the Quora Question Pair (QQP) classification
dataset,444https://www.kaggle.com/c/quora-question-pairs the QNLI question
answering task (Wang et al., 2019b) and the SST-2 sentiment analysis dataset
(Socher et al., 2013). We report accuracy on the dev sets of each dataset
(except SST-2 where we report test accuracy) and compare our results to BERT,
SpanBERT and RoBERTa.

##### Text normalization:

Text normalization — also referred to as text verbalization — is a core task
in the text-to-speech (TTS) community. Text normalization takes as input raw
unverbalized text—as typically found in written form—and produces a verbalized
form of that text, expanding numbers, dates, abbreviations, etc. The output of
this task is a word-for-word spoken transcript—the input format expected by
TTS systems. For example, “A 1951 Lancia V6” would become “a nineteen fifty
one lancia v six” while “Dial 1951 for 6V batteries” might become “dial one
nine five one for six volt batteries.”. We consider the English task data from
Sproat & Jaitly (2016) and compare our results to those of Stahlberg & Kumar
(2020), known to be previously state-of-the-art. We report sentence error-rate
on the test set for all our experiments. When evaluating on text
normalization, we fine-tune our encoder with a $3$-layer transformer decoder
with a model dimension of $512$, hidden dimension of $1024$ and $4$ attention
heads.

###  4.3 Main Results

In this section, we analyze the results of fine-tuning our models on speech
and text downstream tasks.

####  4.3.1 Speech Translation

We present our results on CoVoST 2 En-X translation in Table 1. We compare our
models against results from Wang et al. (2021b), specifically against fine-
tuning wav2vec 2.0 on speech-translation, with and without LM fusion. Our
speech-only baselines trained using wav2vec 2.0 improve over Wang et al.
(2021b) by over 1 BLEU, possible due to increased encoder capacity. Our
w2v-BERT speech-only encoder further improves performance by around 0.4 BLEU.

The addition of mC4-En data to pre-training results in a drop of around 1.3
BLEU for the w2v-conformer, a concrete example of the interference issue. In
comparison, the w2v-BERT speech-text model is only worse than its speech-only
counterpart by approximately 0.6 BLEU. The addition of alignment losses
results in the joint pre-trained model matching the speech-only baseline and
alleviates interference.

We continue training our TLM + STM joint model on unlabeled speech-only data
to alleviate the capacity limitation in the multimodal pre-trained model.
Fine-tuning this speech-adapted model on CoVoST results in a model that
outperforms our best speech-only model by almost 1 BLEU point, illustrating
positive cross-modal transfer and the advantages of multimodal pre-training.

Table 1: BLEU on CoVoST 2 speech translation comparing our speech-text pre-
trained models against speech-only pre-training and pre-existing baselines.

# | Model | # Params | Text data | En-De | En-Ca | En-Ar | En-Tr | Avg
---|---|---|---|---|---|---|---|---
| Prior Work |  |  |  |  |  |  |
1 |  wav2vec-2.0 (Wang et al., 2021b) | 300M | - | 23.8 | 32.4 | 17.4 | 15.4 | 22.3
2 |  wav2vec-2.0 + LM (Wang et al., 2021b) | - | - | 24.9 | 34.0 | 18.0 | 16.7 | 23.4
| Our Work Speech-only |  |  |  |  |  |  |
3 |  w2v-conformer | 600M | - | 27.1 | 33.1 | 18.8 | 15.6 | 23.7
4 |  w2v-bert | 600M | - | 27.4 | 33.9 | 19.0 | 15.9 | 24.1
| Our Work Speech-text |  |  |  |  |  |  |
5 |  w2v-conformer + bert | 600M | mC4-En | 25.4 | 30.5 | 18.5 | 15.2 | 22.4
6 |  w2v-bert + bert (SLAM) | 600M | mC4-En | 26.9 | 33.1 | 18.1 | 16.1 | 23.5
7 |  SLAM-TLM | 600M | mC4-En | 27.5 | 33.4 | 18.9 | 16.6 | 24.1
8 |  SLAM-TLM-STM | 600M | mC4-En | 27.2 | 33.3 | 18.5 | 16.8 | 24.0
| Our Work Speech-text $\rightarrow$ Speech-only |  |  |  |  |  |  |
9 |  SLAM-TLM-STM $\rightarrow$ w2v-bert | 600M | mC4-En | 27.1 | 34.2 | 21.2 | 17.5 | 25.0

####  4.3.2 Speech Recognition

Table 2: WERs (%) when using the LibriSpeech 960h as supervised data. For all
methods, both self-training and LM fusion are not used. References are where
the numbers are quoted from.

# | Model | #Params | Text data | dev | dev-other | test | test-other
---|---|---|---|---|---|---|---
| Prior Work (no LM) |  |  |  |  |  |
1 |  wav2vec 2.0-CTC (Baevski et al., 2020b) | 300M | - | 2.1 | 4.5 | 2.2 | 4.5
2 |  w2v-Conformer XL (Zhang et al., 2020) | 600M | - | 1.7 | 3.5 | 1.7 | 3.5
3 |  w2v-bert XL (Chung et al., 2021) | 600M | - | 1.5 | 2.9 | 1.5 | 2.9
4 |  w2v-bert XXL (Chung et al., 2021) | 1B | - | 1.5 | 2.7 | 1.5 | 2.8
| Our Work Speech-only |  |  |  |  |  |
5 |  w2v-bert XL (Group Norm) | 600M | - | 1.5 | 2.9 | 1.6 | 2.9
| Our Work Speech-text |  |  |  |  |  |
6 |  w2v-conformer + bert | 600M | LibriLM | 1.7 | 4.0 | 1.9 | 4.0
7 |  w2v-bert + bert (SLAM) | 600M | LibriLM | 1.6 | 3.2 | 1.7 | 3.3
9 |  SLAM-TLM (one stage) | 600M | LibriLM | 1.6 | 3.1 | 1.7 | 3.2
8 |  SLAM-TLM | 600M | LibriLM | 1.5 | 2.9 | 1.6 | 3.1
9 |  SLAM-TLM-STM | 600M | LibriLM | 1.7 | 3.0 | 1.6 | 3.2
10 |  SLAM-TLM-STM | 600M | C4 | 1.7 | 3.2 | 1.7 | 3.2
| Our Work Speech-text $\rightarrow$ Speech-only |  |  |  |  |  |
11 |  SLAM-TLM-STM $\rightarrow$ w2v-bert | 600M | mC4-En | 1.6 | 3.0 | 1.6 | 3.1

Table 3: WERs (%) across multiple tasks and settings from the SpeechStew
benchmark compared against pre-existing baselines. †We follow Likhomanenko et
al. (2020) and remove punctuations during evaluation.

# | Model | #Params | Text data | AMI | Common Voice† | Switchboard/Fisher | TED-LIUM | WSJ | Avg
---|---|---|---|---|---|---|---|---|---
|  |  |  | IHM | SDM1 |  | SWBD | CH |  | eval92 |
| Prior Work |  |  |  |  |  |  |  |  |  |
1 |  w2v-conformer (Chan et al., 2021b) | 1B | - | 9.5 | 22.7 | 8.4 | 4.8 | 10.6 | 5.7 | 1.3 | 9.1
| Our Work Speech-only |  |  |  |  |  |  |  |  |  |
2 |  w2v-conformer | 600M | - | 9.6 | 23.8 | 8.4 | 4.7 | 9.2 | 5.6 | 1.4 | 9.1
3 |  w2v-bert | 600M | - | 9.1 | 23.1 | 8.6 | 4.5 | 9.0 | 5.4 | 1.3 | 8.7
| Our Work Speech+text |  |  |  |  |  |  |  |  |  |
5 |  w2v-bert + bert (SLAM) | 600M | LibriLM | 9.4 | 24.3 | 9.7 | 4.8 | 11.0 | 5.9 | 1.5 | 9.5
6 |  SLAM-TLM | 600M | LibriLM | 9.2 | 23.8 | 8.7 | 4.9 | 8.9 | 5.8 | 1.3 | 9.3
7 |  SLAM-TLM-STM | 600M | LibriLM | 9.3 | 23.5 | 8.6 | 4.6 | 9.1 | 5.6 | 1.3 | 9.0
8 |  SLAM-TLM | 600M | mC4-En | 9.4 | 24.7 | 8.9 | 4.8 | 9.2 | 5.1 | 1.5 | 9.1
9 |  SLAM-TLM-STM | 600M | mC4-En | 9.5 | 25.3 | 9.0 | 4.6 | 9.0 | 5.3 | 1.5 | 9.2

In Table 2, we present our results on the Librispeech 960h ASR benchmark. We
compare our unified speech-text encoders to a number of state-of-the-art self-
supervised representation learning methods from the literature, including
wav2vec 2.0 (Baevski et al., 2020b) and w2v-bert (Chung et al., 2021).

As shown in Table 2, w2v-BERT is consistently better than w2v-Conformer with
the text modality by 17% relative (line 6 and 7). However, simply adding the
text modality with LibriLM data hurts ASR performance by 14% relative compared
to the speech-only model, from 2.9 to 3.3 average WER (line 5 and 7) on
testother. By adding TLM loss (line 8), we are able to reduce the interference
and we bridge most of this gap, matching performance on dev/devother/test, and
only 0.2% worse on testother compared to the mono-modal model. We conclude
that the alignment losses help the model align the two modalities, resulting
in better use of shared parameters and reduction in the interference between
the speech and text modalities. Further introducing STM loss does not improve
ASR performance (line 9), but it still performs better than the model without
alignment losses. As we increase the amount of text data from LibriLM to
mC4-En (line 10), we observed a regression on devother and testother. We
conclude that the model needs more capacity to learn from the out-of-domain
and larger text dataset. Similar to speech translation, if we further pre-
train the model with speech only data, there is 0.1% consistent improvement
over all the test sets (line 10 and 11).

In Table 3, we present our results on 5 ASR benchmarks using SpeechStew
supervised data. Note that the unified encoder model has not seen any paired
data during pre-training on these out-of-domain benchmarks. We notice that the
alignment losses still improve over the baseline multimodal model (line 5 to
7). Interestingly, mC4-En data improves performance on TED-LIUM but is worse
on AMI compared to pre-training on LibriLM.555TED-LIUM is clean speech from
the TED-talks domain and thus likely to benefit from more text data, whereas
AMI has natural speech from meetings and might benefit from additional
capacity devoted to acoustic modeling.

####  4.3.3 Natural Language Understanding

We report results on four natural language understanding tasks from GLUE in
Table 4. We compare our methods to the original BERT model of Devlin et al.
(2019) and its extended versions, SpanBERT (Joshi et al., 2020) and RoBERTa
(Liu et al., 2019) which are trained on comparable objectives and comparable
text data respectively. We report dev results for MNLI, QNLI and QQP, as test
sets are not available for these tasks. We see that our SpanBERT-conformer
text-only baseline obtains competitive results with SpanBERT but is
outperformed by RoBERTa, possibly because of the Conformer architecture and
the optimized pre-training and fine-tuning of the RoBERTa architecture. Doing
an apples-to-apples comparison of our text-only model and our speech-text
architectures, we observe significant decrease in performance when adding the
speech modality. On MNLI, for instance, we go from 87.9% accuracy (line 5)
down to 83.3% accuracy with our full model (line 8), or from 95.4 on SST-2 to
93.9%. We observe some gains in performance when using alignment losses over
the fully self-supervised learning approach (line 6) which only slightly
alleviates the interference problem. Given the large amount of data in both
speech and text for English, it is likely that the capacity of the model is a
limiting factor for understanding both modalities simultaneously. We believe
that alleviating capacity limitations by inducing better cross-modal
alignments is an important challenge. We leave the investigation of larger-
capacity models and lower-resource languages for future work.

####  4.3.4 Text Normalization

In addition to GLUE, we also evaluate and report sentence error rate for text
normalization and compare our approach to Stahlberg & Kumar (2020) in Table 4.
Our baseline text-only model improves over the previous state-of-the-art by
$0.25\%$ absolute (lines 5 and 6). Adding speech during pre-training results
in worse performance compared to our text-only pre-training, but the addition
of TLM and STM alignment losses is able to recover some of the lost quality
(lines 6 to 8). Based on this, we suspect that future work in cross-modality
alignment may yield improvements on this task.

Table 4: Performance on four GLUE tasks and text-normalization against text-
only baselines. We report dev accuracy scores for MNLI, QNLI and QQP, test
accuracy for SST-2 and test sentence-error-rate for Text-Norm.

# | Model | # Params | Text data | MNLI | QNLI | QQP | SST-2 | Text-Norm
---|---|---|---|---|---|---|---|---
| Prior Work |  |  |  |  |  |  |
1 |  BERT (Devlin et al., 2019) | 340M | Wiki+Books | 86.6 | 92.3 | 91.3 | 93.2 | -
2 |  SpanBERT (Joshi et al., 2020) | 340M | Wiki+Books | 87.0 | 93.3 | - | 94.8 | -
3 |  RoBERTa (Liu et al., 2019) | 340M | CC | 90.2 | 94.7 | 92.2 | 96.4 | -
4 |  Seq2Edits (Stahlberg & Kumar, 2020) | - | - | - | - | - | - | 1.36
| Our Work Text-only |  |  |  |  |  |  |
5 |  SpanBERT-conformer | 450M | mC4-En | 87.9 | 92.6 | 91.8 | 95.4 | 1.11
| Our Work Speech-text |  |  |  |  |  |  |
6 |  w2v-bert + bert (SLAM) | 450M | mC4-En | 82.3 | 88.4 | 90.6 | 94.2 | 1.30
7 |  SLAM-TLM | 450M | mC4-En | 83.6 | 90.1 | 91.0 | 94.3 | 1.28
8 |  SLAM-TLM-STM | 450M | mC4-En | 83.3 | 90.0 | 91.0 | 93.9 | 1.19

##  5 Discussion

In this work, we demonstrate that a single encoder model can be pre-trained to
learn strong contextualized representations of speech and text simultaneously.
We combine self-supervised learning objectives for text (BERT) and self-
supervised approaches for speech (w2v-BERT) to learn a joint Speech and
LAnguage Model (SLAM). Downstream evaluations on speech and language
understanding tasks, including LibriSpeech and SpeechStew ASR, CoVoST 2 speech
translation, four GLUE tasks, and text-normalization uncover significant
interference challenges when pre-training simultaneously on high-resource
modalities. Using alignment losses such as translation language modeling and
speech-text matching which leverage speech-text supervised aligned data, we
show that we can improve the cross-modal representation alignment and improve
over mono-modal models on the speech translation tasks, while maintaining
state-of-the-art performance on speech recognition. We hope that this work
would motivate further research on extending the universality of self-
supervised learning of language representations to the multimodal speech-text
setting.

## References

  * Agirre et al. (2007) Eneko Agirre, Lluís Màrquez, and Richard Wicentowski (eds.).  _Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007)_.  Association for Computational Linguistics, Prague, Czech Republic, June 2007.
  * Akbari et al. (2021) Hassan Akbari, Linagzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong.  Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text.  _arXiv preprint arXiv:2104.11178_ , 2021.
  * Ardila et al. (2019) Rosana Ardila et al.  Common voice: A massively-multilingual speech corpus.  _arXiv preprint arXiv:1912.06670_ , 2019.
  * Arivazhagan et al. (2019) Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al.  Massively multilingual neural machine translation in the wild: Findings and challenges.  _arXiv preprint arXiv:1907.05019_ , 2019.
  * Baevski et al. (2020a) Alexei Baevski, Steffen Schneider, and Michael Auli.  vq-wav2vec: Self-supervised learning of discrete speech representations, 2020a.
  * Baevski et al. (2020b) Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  In _NeurIPS_ , 2020b.
  * Baevski et al. (2021) Alexei Baevski, Wei-Ning Hsu, Alexis Conneau, and Michael Auli.  Unsupervised speech recognition.  _arXiv preprint arXiv:2105.11084_ , 2021.
  * Bengio et al. (2003) Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin.  A neural probabilistic language model.  _The journal of machine learning research_ , 3:1137–1155, 2003.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_ , 2020.
  * Carletta et al. (2005) Jean Carletta et al.  The ami meeting corpus: A pre-announcement.  In _International workshop on machine learning for multimodal interaction_. Springer, 2005.
  * Chan et al. (2021a) William Chan, Daniel Park, Chris Lee, Yu Zhang, Quoc Le, and Mohammad Norouzi.  SpeechStew: Simply mix all available speech recognition data to train one large neural network.  _arXiv preprint arXiv:2104.02133_ , 2021a.
  * Chan et al. (2021b) William Chan, Daniel Park, Chris Lee, Yu Zhang, Quoc Le, and Mohammad Norouzi.  Speechstew: Simply mix all available speech recognition data to train one large neural network.  _arXiv preprint arXiv:2104.02133_ , 2021b.
  * Chen et al. (2021) Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhuvana Ramabhadran, Gary Wang, and Pedro Moreno.  Injecting text in self-supervised speech pretraining, 2021.
  * Chung et al. (2016) Yu-An Chung, Chao-Chung Wu, Chia-Hao Shen, Hung-Yi Lee, and Lin-Shan Lee.  Audio word2vec: Unsupervised learning of audio segment representations using sequence-to-sequence autoencoder.  In _Interspeech_ , 2016.
  * Chung et al. (2021) Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu.  w2v-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre-training.  In _ASRU_ , 2021.
  * Cieri et al. (2003) Christopher Cieri, David Miller, and Kevin Walker.  From switchboard to fisher: Telephone collection protocols, their uses and yields.  In _Eurospeech_ , 2003.
  * Conneau & Lample (2019) Alexis Conneau and Guillaume Lample.  Cross-lingual language model pretraining.  In _NeurIPS_ , 2019.
  * Conneau et al. (2019) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.  Unsupervised cross-lingual representation learning at scale.  In _ACL_ , 2019.
  * Conneau et al. (2020) Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli.  Unsupervised cross-lingual representation learning for speech recognition.  _arXiv preprint arXiv:2006.13979_ , 2020.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT_ , 2019.
  * Eloff et al. (2019) Ryan Eloff, André Nortje, Benjamin van Niekerk, Avashna Govender, Leanne Nortje, Arnu Pretorius, Elan Van Biljon, Ewald van der Westhuizen, Lisa van Staden, and Herman Kamper.  Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks.  _arXiv preprint arXiv:1904.07556_ , 2019.
  * Fedus et al. (2021) William Fedus, Barret Zoph, and Noam Shazeer.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.
  * Gulati et al. (2020) Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang.  Conformer: Convolution-augmented transformer for speech recognition.  In _Interspeech_ , 2020.
  * Gutmann & Hyvärinen (2012) Michael U Gutmann and Aapo Hyvärinen.  Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics.  _Journal of Machine Learning Research_ , 13(2), 2012.
  * Hsu et al. (2021) Wei-Ning Hsu, Anuroop Sriram, Alexei Baevski, Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Jacob Kahn, Ann Lee, Ronan Collobert, Gabriel Synnaeve, et al.  Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training.  _arXiv preprint arXiv:2104.01027_ , 2021.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson.  Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.  In _International Conference on Machine Learning_ , pp. 4411–4421. PMLR, 2020.
  * Jang et al. (2016) Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparameterization with gumbel-softmax.  _arXiv preprint arXiv:1611.01144_ , 2016.
  * Jia et al. (2021) Ye Jia, Heiga Zen, Jonathan Shen, Yu Zhang, and Yonghui Wu.  Png bert: Augmented bert on phonemes and graphemes for neural tts.  In _Interspeech_ , 2021.
  * Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy.  Spanbert: Improving pre-training by representing and predicting spans.  _Transactions of the Association for Computational Linguistics_ , 8:64–77, 2020.
  * Kahn et al. (2020) J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P. E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux.  Libri-light: A benchmark for asr with limited or no supervision.  In _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 7669–7673, 2020.  https://github.com/facebookresearch/libri-light.
  * Kudo & Richardson (2018) Taku Kudo and John Richardson.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pp. 66–71, Brussels, Belgium, November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/D18-2012.  URL https://aclanthology.org/D18-2012.
  * Lepikhin et al. (2020) Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen.  Gshard: Scaling giant models with conditional computation and automatic sharding.  _arXiv preprint arXiv:2006.16668_ , 2020.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _ACL_ , 2020.
  * Li et al. (2021) Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi.  Align before fuse: Vision and language representation learning with momentum distillation, 2021.
  * Likhomanenko et al. (2020) Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Paden Tomasello, Jacob Kahn, Gilad Avidov, Ronan Collobert, and Gabriel Synnaeve.  Rethinking Evaluation in ASR: Are Our Models Robust Enough?  _arXiv preprint arXiv:2010.11745_ , 2020.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  RoBERTa: A robustly optimized BERT pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  In _NIPS_ , 2013.
  * Mnih & Teh (2012) Andriy Mnih and Yee Whye Teh.  A fast and simple algorithm for training neural probabilistic language models, 2012.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Panayotov et al. (2015) Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur.  LibriSpeech: An ASR corpus based on public domain audio books.  In _ICASSP_ , 2015.
  * Peters et al. (2018) Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  Deep contextualized word representations.  _arXiv preprint arXiv:1802.05365_ , 2018.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving language understanding by generative pre-training.  2018\.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_ , 2019.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21:1–67, 2020\.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of EMNLP_ , pp. 2383–2392. Association for Computational Linguistics, 2016.
  * Rousseau et al. (2012) Anthony Rousseau, Paul Deléglise, and Yannick Esteve.  Ted-lium: an automatic speech recognition dedicated corpus.  In _LREC_ , pp. 125–129, 2012.
  * Ruder et al. (2021) Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Graham Neubig, and Melvin Johnson.  Xtreme-r: Towards more challenging and nuanced multilingual evaluation.  _arXiv preprint arXiv:2104.07412_ , 2021.
  * Schneider et al. (2019) Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli.  wav2vec: Unsupervised pre-training for speech recognition.  _arXiv preprint arXiv:1904.05862_ , 2019.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of EMNLP_ , pp. 1631–1642, 2013.
  * Sproat & Jaitly (2016) Richard Sproat and Navdeep Jaitly.  RNN approaches to text normalization: A challenge.  _CoRR_ , abs/1611.00068, 2016.  URL http://arxiv.org/abs/1611.00068.
  * Stahlberg & Kumar (2020) Felix Stahlberg and Shankar Kumar.  Seq2Edits: Sequence transduction using span-level edit operations.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 5147–5159, Online, November 2020\. Association for Computational Linguistics.  URL https://aclanthology.org/2020.emnlp-main.418.
  * Taylor (1953) Wilson L Taylor.  “cloze procedure”: A new tool for measuring readability.  _Journalism quarterly_ , 30(4):415–433, 1953\.
  * van den Oord et al. (2018) Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu.  Neural discrete representation learning, 2018.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _NIPS_ , 2017.
  * Wang et al. (2019a) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _arXiv preprint arXiv:1905.00537_ , 2019a.
  * Wang et al. (2019b) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _ICLR_ , 2019b.
  * Wang et al. (2021a) Changhan Wang, Anne Wu, and Juan Pino.  CoVoST 2 and massively multilingual speech-to-text translation.  In _interspeech_ , 2021a.
  * Wang et al. (2021b) Changhan Wang, Anne Wu, Juan Pino, Alexei Baevski, Michael Auli, and Alexis Conneau.  Large-scale self-and semi-supervised learning for speech translation.  _arXiv preprint arXiv:2104.06678_ , 2021b.
  * Wenzek et al. (2020) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Édouard Grave.  CCNet: Extracting high quality monolingual datasets from web crawl data.  In _LREC_ , 2020.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel R. Bowman.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of NAACL-HLT_ , 2018.
  * Xue et al. (2021a) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel.  mt5: A massively multilingual pre-trained text-to-text transformer.  In _NAACL_ , 2021a.
  * Xue et al. (2021b) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 483–498, Online, June 2021b. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.41.  URL https://aclanthology.org/2021.naacl-main.41.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoc V. Le.  XLNet: Generalized autoregressive pretraining for language understanding.  In _NeurIPS_ , 2019.
  * Zhang et al. (2020) Yu Zhang, James Qin, Daniel S. Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc V. Le, and Yonghui Wu.  Pushing the limits of semi-supervised learning for automatic speech recognition.  _arXiv preprint arXiv:2010.10504_ , 2020.
  * Zhang et al. (2021) Yu Zhang, Daniel S Park, Wei Han, James Qin, Anmol Gulati, Joel Shor, Aren Jansen, Yuanzhong Xu, Yanping Huang, Shibo Wang, Zongwei Zhou, Bo Li, Min Ma, William Chan, Jiahui Yu, Yongqiang Wang, Liangliang Cao, Khe Chai Sim, Bhuvana Ramabhadran, Tara N. Sainath, Françoise Beaufays, Zhifeng Chen, Quoc V. Le, Chung-Cheng Chiu, Ruoming Pang, and Yonghui Wu.  BigSSL: Exploring the frontier of large-scale semi-supervised learning for automatic speech recognition.  _arXiv preprint arXiv:2109.13226_ , 2021.
  * Zheng et al. (2021) Renjie Zheng, Junkun Chen, Mingbo Ma, and Liang Huang.  Fused acoustic and text encoding for multimodal bilingual pretraining and speech translation, 2021.