# Improved Cross-Lingual Transfer Learning For Automatic Speech Translation

Sameer Khurana, Nauman Dawalatabad, Antoine Laurent, Luis Vicente, Pablo
Gimeno, Victoria Mingote,
James Glass Sameer Khurana (email: skhurana@mit.edu), Nauman Dawalatabad, and
James Glass are with MIT Computer Science and Artificial Intelligence
Laboratory, USA. Luis Vicente, Pablo Gimeno, and Victoria Mingote are with the
University of Zaragoza, Spain. Antoine Laurent is with LIUM University,
France. This work was partially performed using HPC resources from
GENCI–IDRIS, grants AD011012527 and AD011012565. This work has received
funding from the European Union’s Horizon 2020 research and innovation
programme under the Marie Skłodowska-Curie grant agreement No 101007666.

###### Abstract

Research in multilingual speech-to-text translation is topical. Having a
single model that supports multiple translation tasks is desirable. The goal
of this work it to improve cross-lingual transfer learning in multilingual
speech-to-text translation via semantic knowledge distillation. We show that
by initializing the encoder of the encoder-decoder sequence-to-sequence
translation model with $\tt SAMU\text{-}XLS\text{-}R$, a multilingual speech
transformer encoder trained using multi-modal (speech-text) semantic knowledge
distillation, we achieve significantly better cross-lingual task knowledge
transfer than the baseline $\tt XLS\text{-}R$, a multilingual speech
transformer encoder trained via self-supervised learning. We demonstrate the
effectiveness of our approach on two popular datasets, namely, CoVoST-2 and
Europarl. On the 21 translation tasks of the CoVoST-2 benchmark, we achieve an
average improvement of 12.8 BLEU points over the baselines. In the zero-shot
translation scenario, we achieve an average gain of 18.8 and 11.9 average BLEU
points on unseen medium and low-resource languages. We make similar
observations on Europarl speech translation benchmark.

###### Index Terms:

Cross-Lingual Transfer Learning, Automatic Speech Translation, Semantically
Aligned Cross-Lingual Speech Representations

##  I Introduction

Self-Supervised Representation Learning (SSRL) from speech [1, 2, 3, 4, 5, 6,
7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18] has improved tremendously over
the past few years due to the introduction of Contrastive Predictive Coding
(CPC) [19], a self-supervised representation learning method applied to
speech, text, and visual data. The introduction of the core idea of noise
contrastive estimation [20] in CPC has led to a series of papers in speech
SSRL, such as Wav2Vec [5], VQ-Wav2Vec [6], Wav2Vec-2.0 [9], Multilingual
Wav2Vec-2.0 [11], XLS-R ($\tt XLS\text{-}R$, a bigger version of the
multilingual wav2vec-2.0) [15]. Pre-trained SSRL speech encoders like $\tt
XLS\text{-}R$ are considered "foundation models" [21] for downstream
multilingual speech processing applications such as Multilingual Automatic
Speech Recognition [11, 22, 15], Multilingual Speech Translation [23, 15, 18],
and other para-linguistic property prediction tasks [24, 25]. This work
focuses on Multilingual Speech Translation.

Multilingual Speech Translation (MST) refers to translating speech in all the
source languages in set $\mathcal{X}{}$ to text in all the target languages in
set $\mathcal{Y}{}$, which implies a total of
$|\mathcal{T}{}|=|\mathcal{X}{}|\times|\mathcal{Y}{}|$ translation tasks. In
MST, we train a single model for all the translation tasks given by set
$\mathcal{T}{}$. The benefits of having a single model instead of individual
model for each task $t\in\mathcal{T}{}$ are two-fold: First, it is convenient
to maintain and share a single model that can perform multiple tasks rather
than having $|\mathcal{T}{}|$ separate models, and second, sharing model
parameters amongst $|\mathcal{T}{}|$ translation tasks could lead to knowledge
transfer across tasks, especially from high-resource to low-resource.

The standard neural network architecture used for MST is the encoder-decoder
model [26, 27]. Recently, MST has seen significant improvements owing to; (i)
better initialization of the translation model’s encoder and decoder with pre-
trained speech encoders, like $\tt XLS\text{-}R$ [15], and text decoders, like
$\tt MBART$ [28], (ii) better fine-tuning strategies [23], and (iii) parallel
speech-text translation corpora [29, 30]. However, as we demonstrate in
Section II, the performance on low-resource tasks remains poor, and in
particular, the performance gap (cross-lingual transfer gap) between high and
low-resource languages remains large. We hypothesize that this is because the
$\tt XLS\text{-}R$ speech encoder learns non-robust surface-level features
from unlabeled speech data, rather than the high-level linguistic knowledge
about semantics.

To inject semantic knowledge into the learned $\tt XLS\text{-}R$
representations, we turn to the recently introduced Semantically-Aligned
Multimodal Cross-Lingual Representation Learning framework, $\tt
SAMU\text{-}XLS\text{-}R$ [17]. $\tt SAMU\text{-}XLS\text{-}R$ (Section III-B)
is a knowledge-distillation framework that distills semantic knowledge from a
pre-trained text embedding model into the pre-trained multilingual $\tt
XLS\text{-}R$ speech encoder. The KD framework outputs the $\tt
SAMU\text{-}XLS\text{-}R$ speech encoder that learns a semantically structured
multilingual speech manifold, where a spoken utterance is close to its spoken
translations in several other languages that are present in the $\tt
SAMU\text{-}XLS\text{-}R$’s training pool (Details in Section III-B). We claim
that building MST model using the semantic representations learned by $\tt
SAMU\text{-}XLS\text{-}R$ would lead to better cross-lingual transfer from
high to low-resource tasks compared to the $\tt XLS\text{-}R$ and other non-
semantic multilingual speech encoder baselines. We verify our claim through
several experiments (Section VI). Through this work, we make the following
contributions:

  * •

We doubled the number of languages previously supported by $\tt
SAMU\text{-}XLS\text{-}R$ encoder from 25 to 53 (Section IV-A).

  * •

$\tt SAMU\text{-}XLS\text{-}R$ encoder provides speech embedding at the
utterance level. Previous works [31] have explored the use $\tt
SAMU\text{-}XLS\text{-}R$ embedding for automatic spoken language
understanding and semantic speech-text and speech-speech retrieval [17].
Differently, in this work, for the first time, we show that the fine-grained
contextual embeddings (corresponding to 20ms duration speech segment) that
precede the coarse utterance level representation (3-10s long speech segment)
learned by $\tt SAMU\text{-}XLS\text{-}R$ is well-suited for the sequence
generation task of multilingual speech translation. We empirically
demonstrate, through several experiments, the superiority of $\tt
SAMU\text{-}XLS\text{-}R$ representations over $\tt XLS\text{-}R$ and other
multilingual speech encoders (Section VI).

  * •

On the public CoVoST-2 $\mathcal{X}{}\rightarrow\text{English}$ MST benchmark
[30], we show that by switching the $\tt XLS\text{-}R$ encoder in the MST
model to the $\tt SAMU\text{-}XLS\text{-}R$ encoder, the performance improves
significantly on medium-resource $\mathcal{X}{}\rightarrow\text{English}$
language group by 15.6 BLEU points, and on low-resource language group by 18.9
BLEU points, and overall by 13.8 BLEU points (Section VI-A).

  * •

We also show the efficacy of the $\tt SAMU\text{-}XLS\text{-}R$ speech encoder
in Zero-Shot translation settings. In the Zero-Shot MST scenario, where we
train the $\tt SAMU\text{-}XLS\text{-}R$-based MST model only on high-resource
$\mathcal{X}{}$$\rightarrow$English tasks in CoVoST-2, we observe an
improvement on medium and low-resource tasks of 18.8 and 11.9 BLEU points over
the $\tt XLS\text{-}R$ baseline (Section VI-B).

  * •

Finally, we extend our studies on another MST benchmark, namely Europarl [29],
which consists of several $\mathcal{X}{}$$\rightarrow$$\mathcal{Y}{}$
translation tasks. In the zero-shot MST scenario, we observe an overall
improvement of 8.5 BLEU points average with $\tt SAMU\text{-}XLS\text{-}R$
encoder over $\tt XLS\text{-}R$, owing to the significant increase in
performance on unseen (during training) source languages of 17 BLEU points
(Section VI-C).

##  II Motivation: Cross-Lingual Transfer Gap

To motivate our work, we show the performance of the multilingual $\tt
XLS\text{-}R$ speech encoder on the CoVoST-2 speech-to-text translation
benchmark [30]. CoVoST-2 comprises 21 X$\rightarrow$EN speech-to-text
translation tasks, where X refers to the language of speech utterance, and EN
refers to the corresponding English text translation. $\tt XLS\text{-}R$
speech encoder is pre-trained via Self-Supervised Learning using unlabeled
speech data in 128 languages. Babu et al. [15] fine-tunes the pre-trained $\tt
XLS\text{-}R$ encoder combined with a pre-trained $\tt MBART$ text decoder
[28] simultaneously on 21 X$\rightarrow$EN translation tasks (multi-task
learning) in CoVoST-2 benchmark. We categorize the 21 translation tasks into
high, mid, and low-resource groups. A task is classified as high if it has
more than 100 hours of paired speech(X)-text(EN) translation training data,
mid if training data is between 10 and 100 hours, and low if training data is
less than 10 hours. There are four high, five mid, and 12 low-resource tasks.

Figure 1: We report translation performance on 21 X$\rightarrow$EN speech-to-
text translation tasks in CoVoST-2 benchmark with different sized pre-trained
XLS-R encoders fine-tuned on labeled speech translation data. The 21 tasks are
categorized into high, mid, and low resource tasks depending on the available
labeled training data for a task. We report average BLEU-4 scores in the three
categories. The important thing to consider is the performance gap or cross-
lingual transfer gap between high and low-resource translation tasks. We
address this large gap in this paper. Figure 2: $\tt SAMU\text{-}XLS\text{-}R$
semantic knowledge-distillation framework. The learning framework comprises a
speech and a text encoder. The speech encoder transforms a raw speech waveform
into an embedding vector. The text encoder transforms the transcript
corresponding to the speech utterance into an embedding. The text encoder is
initialized using the pre-trained Language-Agnostic BERT Sentence Embedding
model $\tt LaBSE$ [32]. The speech encoder below the pooling layer is
initialized using the pre-trained $\tt XLS\text{-}R$ speech encoder [15].

We report the $\tt XLS\text{-}R$ (speech encoder)$\rightarrow$$\tt MBART$
(text decoder) transformer model’s performance on the high, mid, and low-
resource translation groups in Fig. 1. We report the average BLEU-4 score on
each translation group. The vital thing to observe is the performance gap
(cross-lingual transfer gap) between high and low-resource translation groups
for different-sized $\tt XLS\text{-}R$ encoders ranging from 300M to 2B
parameters. So, increasing the model size from 300M to 2B, a more than 500%
increase leads to only a 16% reduction in the cross-lingual transfer gap.
Since the translation model is built on top of the pre-trained $\tt
XLS\text{-}R$ encoder’s representations, there is some missing ingredient in
the pre-trained representations that leads to a poor cross-lingual transfer
from high to low-resource tasks. This implies that the knowledge that the $\tt
XLS\text{-}R$-based translation model acquires while learning to perform high-
resource X$\rightarrow$EN translation tasks is not useful (or transferable)
for learning the low-resource tasks.

We claim that the missing piece is semantic knowledge. We hypothesize that
since $\tt SAMU\text{-}XLS\text{-}R$ is specifically trained to encode
semantic information in its internal representations, building a translation
model on top of $\tt SAMU\text{-}XLS\text{-}R$ would lead to better cross-
lingual transfer from high to low-resource translation tasks, thus reducing
the cross-lingual transfer gap mentioned above.

##  III Preliminaries

###  III-A XLS-R ($\tt XLS\text{-}R$)

This section discusses the architecture of the $\tt XLS\text{-}R$ encoder. The
$\tt XLS\text{-}R$ encoder consists of a Convolutional Neural Network (CNN)
[33] feature extractor, denoted by $h$, and a transformer encoder [27],
denoted by $g$.

####  III-A1 CNN Feature Extractor

$h:\mathbf{a}_{1:S}\rightarrow\mathbf{f}{}_{1:T}$ maps the speech waveform
$(\mathbf{a}_{1:S}\mid\mathbf{a}_{s}\in\mathbb{R})$ to an intermediate
representation $(\mathbf{f}_{1:T}\mid\mathbf{f}_{t}\in\mathbb{R}^{d})$, where
$T=S/r$, where $r$ is the factor by which $h$ downsamples $\mathbf{a}_{1:S}$,
and $d$ is the size of the feature dimension. For the $\tt XLS\text{-}R$
encoder, $r=320$, and $d=1024$. The feature extractor $h$ consists of seven
temporal Convolution layers (CONV). Each layer uses ${\tt GeLU}$ activation
function [34]. The feature maps outputted by each layer have $512$ channels.
The output of the last $\tt CONV$ layer in $h$ is up-projected using a fully-
connected layer ($\tt FC$) so that the feature vector $f_{t}$ has the same
dimension as the transformer encoder. Note that the receptive field of each
feature vector is 20ms of the input speech waveform. The feature extractor is
followed by another $\tt CONV$ layer that encodes the relative position of the
feature sequence $\mathbf{f}_{1:T}$, which is used as input to the transformer
encoder $g$.

####  III-A2 Transformer Encoder

$g:\mathbf{f}_{1:T}\rightarrow\mathbf{c}{}_{1:T}$ maps
$(\mathbf{f}_{1:T}\mid\mathbf{f}_{t}\in\mathbb{R}^{d})$ to contextualized
representation $(\mathbf{c}{}_{1:T}\mid\mathbf{c}{}_{t}\in\mathbb{R}^{d})$.
The receptive field of each context vector $\mathbf{c}{}_{t}$ is equal to the
length of the input speech waveform $\mathbf{a}_{1:S}$, which is usually
5-10s. The transformer encoder consists of 24 layers. Each transformer layer
consists of Multi-Headed Self-Attention ($\tt MHSA$), followed by two $\tt FC$
layers. Layer Normalization ($\tt LN$) is used at the input of $\tt MHSA$ and
$\tt FC$ blocks. Residual connections adds the input of $\tt MHSA$, with its
output, and the input of $\tt FC$ block with its output. The transformer
encoder layer dimension is 1024 (size of the feature vector outputted by a
transformer layer), and the $\tt FC$ layer dimension is 3072. Each $\tt MHSA$
block consists of 16 attention heads. The output of the first $\tt FC$ layer
is processed by a $\tt ReLU$ non-linearity [35].

####  III-A3 Training Details

$\tt XLS\text{-}R$ is pre-trained on approximately 400K hours of multilingual
unlabeled speech data segmented into utterances of size 3-10s. The training
data consists of unlabeled speech in 128 languages coming from the following
corpora: VoxPopuli [36], CommonVoice (CoVo) [37], Multilingual Speech (MLS)
[38], BABEL, and Voxlingua [39]. For full training details, see [15]. We use
the pre-trained $\tt XLS\text{-}R$ checkpoints publicly
released111https://github.com/facebookresearch/fairseq/tree/main/examples/wav2vec/xlsr.

As baselines for multilingual X$\rightarrow$EN speech translation experiments
(Section VI-A), we use the $\tt XLS\text{-}R$$\rightarrow$$\tt MBART$
translation models that are trained on CoVoST-2 X$\rightarrow$EN MST
benchmark, officially released
here222https://huggingface.co/facebook/wav2vec2-xls-r-300m-21-to-en. The
official release comprises three translation models corresponding to 0.3B, 1B,
and 2B parameter $\tt XLS\text{-}R$ speech encoders. The text decoder in all
three cases in the 400M parameter $\tt MBART$ decoder.

TABLE I: The number of hours of transcribed speech data available for training $\tt SAMU\text{-}XLS\text{-}R$ in each of the 53 languages from the CommonVoice-Version8 corpus. ar | be | bg | ca | cs | cy | da | de | el | en | eo
---|---|---|---|---|---|---|---|---|---|---
85.2 | 903.9 | 8.2 | 916.8 | 54.9 | 116.3 | 6.6 | 1062.8 | 15.9 | 2185.8 | 1407.9
es | et | eu | fa | fi | fr | fy-NL | ga-IE | gl | ha | hi
404.6 | 33.0 | 98.9 | 317.3 | 8.5 | 826.1 | 49.6 | 4.3 | 10.2 | 3.4 | 11.7
hu | id | it | ja | ka | kmr | ky | lt | lv | mn | mt
19.9 | 25.8 | 310.6 | 40.8 | 7.6 | 47.0 | 37.2 | 17.4 | 7.1 | 12.4 | 8.3
nl | pl | pt | ro | ru | rw | sk | sl | sv-SE | sw | ta
98.0 | 142.2 | 112.0 | 15.8 | 162.6 | 2000.7 | 17.7 | 9.6 | 40.8 | 146.8 | 217.7
th | tr | tt | ug | uk | uz | vi | zh-CN | zh-HK | zh-TW |
142.1 | 65.1 | 29.2 | 59.8 | 63.4 | 81.0 | 4.5 | 68.0 | 99.7 | 62.6 |

###  III-B SAMU-XLS-R ($\tt SAMU\text{-}XLS\text{-}R$)

$\tt XLS\text{-}R$ encoder’s representation manifold encodes low-level
linguistic knowledge, as evident by the non-existent cross-lingual semantic
speech retrieval capability when using $\tt XLS\text{-}R$’s learned
representations in [17]. $\tt SAMU\text{-}XLS\text{-}R$ is a bi-modal (speech
& text) knowledge distillation (KD) framework for expanding the representation
space of $\tt XLS\text{-}R$ speech encoder also to encode high-level semantic
knowledge. The $\tt SAMU\text{-}XLS\text{-}R$ learning framework uses
multilingual transcribed speech set
$\mathcal{D}=\\{\mathbf{a}^{n}_{1:S},\mathbf{y}^{n}_{1:L}\\}_{i=1}^{N}$
consisting of speech, $\mathbf{a}^{i}_{1:S}$ paired with its text transcript
$\mathbf{y}^{i}_{1:L}$. Note that speech and its transcript are in the same
language. The $\tt SAMU\text{-}XLS\text{-}R$ KD framework consists of a speech
and a text processing branch as illustrated in Figure 2, and detailed below.

####  III-B1 The Speech Branch

The speech branch maps the speech waveform $\mathbf{a}_{1:S}$ of duration
3-10s to a single embedding vector $\mathbf{e}\in\mathbb{R}^{768}$. The
mapping is performed in the following two steps: (i) The pre-trained $\tt
XLS\text{-}R$ speech encoder maps $\mathbf{a}_{1:S}$ to the contextual
representation $\mathbf{c}{}_{1:T}$, and (ii) An attention-based temporal
pooling [40], and a non-linear ($\tt tanh$) down-projection layer transforms
$\mathbf{c}{}_{1:T}$ to a single embedding $\mathbf{e}$ of size 768.

####  III-B2 The Text Branch

The text branch consists of a pre-trained $\tt LaBSE$ sentence embedding
encoder [32] that transforms the text transcript $\mathbf{y}_{1:L}$ to a
semantic embedding $\mathbf{z}\in\mathbb{R}^{d}$, of size $d=768$. $\tt LaBSE$
supports 109 written languages. It embeds sentences from different languages
in a shared semantic embedding space, i.e., a sentence and its translation lie
close together in the $\tt LaBSE$ embedding space. By regressing on the $\tt
LaBSE$ semantic embedding, the $\tt XLS\text{-}R$ speech encoder learns to
encode semantics hidden in the spoken utterance in its internal
representations. This new semantics-aware speech encoder is referred to as
$\tt SAMU\text{-}XLS\text{-}R$ encoder.

####  III-B3 Training Details.

The $\tt SAMU\text{-}XLS\text{-}R$ encoder is trained to minimize the cosine
distance between the speech and the text embedding $\mathbf{e}$, and
$\mathbf{z}$ respectively (see Figure 2). The loss function is given by:

$\mathcal{L}=\beta*\left(1.0-\frac{\mathbf{e}\cdot\mathbf{z}}{||\mathbf{e}||\
||\mathbf{z}||}\right)$ (1)

Where $\beta$ is used to scale up the magnitude of the cosine distance loss, a
small loss magnitude could lead to extremely small gradient update values and
significantly longer training times. Upscaling the loss by a constant $\beta$
could alleviate this problem. The parameters of the speech branch are tuned to
minimize the above-mentioned loss $\mathcal{L}$. The text encoder $\tt LaBSE$
remains fixed during training along with the feature extractor $h$ of the pre-
trained $\tt XLS\text{-}R$ encoder (See Figure 2 and [17] for training
details).

##  IV Method

###  IV-A Expanding SAMU-XLS-R

We double the number of languages supported by $\tt SAMU\text{-}XLS\text{-}R$
to 53 than the 25 supported previously. For training $\tt
SAMU\text{-}XLS\text{-}R$, we use speech utterances from multiple languages
annotated with their text transcripts in the same language. We refer to $\tt
SAMU\text{-}XLS\text{-}R$’s training data pool as Background (BKG) data. BKG
data depends on the downstream MST task domain. Ideally, we would like BKG
data from the same domain as the MST task (in-domain). We use the multilingual
transcribed speech from the CommonVoice-Version8 (CoVo-V8) corpus [37].
CoVo-V8 consists of transcribed speech in 87 languages (26 language families).
Around 53 languages overlap with the language set supported by Language-
Agnostic BERT Sentence Encoder (LaBSE), which provides semantic supervision
for training the SAMU-XLS-R speech encoder. The 53 languages are: Arabic (ar),
Belarusian (be), Bulgarian (bg), Catalan (ca), Czech (cs), Welsh (cy), Danish
(da), German (de), Greek (el), English (en), Estonian (eo), Spanish (es),
Estonian (et), Basque (eu), Persian (fa), Finnish (fi), French (fr), Western
Frisian (fy-NL), Gaelic (ga-IE), Galician (gl), Hausa (ha), Hindi (hi),
Hungarian (hu), Indonesian (id), Italian (it), Japanese (ja), Georgian (ka),
Khmer (kmr), Kyrgyz (ky), Lithuanian (lt), Latvian (lv), Mongolian (mn),
Maltese (mt), Dutch (nl), Polish (pl), Portuguese (pt), Romanian (ro), Russian
(ru), Kinyarwanda (rw), Slovak (sk), Slovenian (sl), Swedish (sv-SE), Swahili
(sw), Tamil (ta), Thai (th), Turkish (tr), Tatar (tt), Uyghur (ug), Ukaranian
(uk), Uzbek (uz), Vietnamese (vi), Chinese-Mandarin (zh-CN), Chinese-HongKong
(zh-HK), Chinese-Taiwanese (zh-TW). Table I presents the number of hours of
transcribed speech available in each of the 53 languages for $\tt
SAMU\text{-}XLS\text{-}R$ training. The total training hours is 12.7K. BKG
data is highly imbalanced, where the majority of the data comes from a few
high-resource languages. Following [17, Equation 3], we balance the BKG data
to have a proportionate representation of different languages in a training
mini-batch, which avoids under-fitting to low-resource languages. The up-
sampling process involves repeating utterances in low-resource languages.
Unlike previous work, we use offline data augmentation, namely speed
perturbation [41], to improve the robustness of the learned representations to
the data distribution shifts.

###  IV-B Translation Model

####  IV-B1 Overview

We use the standard encoder-decoder architecture for our translation model. We
initialize the encoder using pre-trained $\tt SAMU\text{-}XLS\text{-}R$.
Following [23], the decoder is initialized with the decoder of a pre-trained
text-to-text translation model, namely $\tt MBART$333
https://github.com/facebookresearch/fairseq/tree/main/examples/multilingual\\#mbart50-models.
The encoder-decoder model is trained using corpora that consist of tuples
$(\mathbf{a}_{1:S},\mathbf{y}_{1:L})$, where $\mathbf{y}_{1:L}$ is the text
translation sequence of the speech sequence $\mathbf{a}_{1:S}$.

####  IV-B2 Training

Given a speech-text translation pair $(\mathbf{a}_{1:S},\mathbf{y}_{1:L})$, we
tune the parameters of the translation model to maximize the log-probability
of $\mathbf{y}_{1:L}$ conditioned on speech utterance $\mathbf{a}_{1:S}$ (log-
likelihood). Also, we employ label-smoothing [42, 43], where a ground-truth
$\mathbf{y}_{l}$ label is randomly replaced with the label predicted by the
model $\hat{\mathbf{y}}_{l}$. We set the probability of token replacement as
0.3. We use the Adam optimizer [44] to maximize the log-likelihood with the
optimal (according to a development set) learning rate (lr) of 5e-4. We use a
three-phase learning rate scheduler [9] as follows; (i) warm-up lr to 5e-4 for
the first 10% of the training iterations, (ii) keep the lr constant for the
next 40% of iterations, and (iii) decay the lr linearly for the rest of
training. We use 28K training iterations and train the model on 8 A100 GPUs. A
single training iteration uses a batch size of 10 minutes of speech utterances
paired with their text translations. We use mixed-precision training style
[45]; most computations are performed on half-precision floating point numbers
except the final loss computation. We use the fairseq toolkit [46] for model
training.

Following [9], we mask (time and feature dimension) the feature sequence
$\mathbf{f}_{1:T}$, output by the CNN feature extractor of $\tt
SAMU\text{-}XLS\text{-}R$. The masking process is performed in the following
two steps: (i) with some probability, referred to as the masking probability,
choose a masking index, and (ii) mask $M$ consecutive indices starting from
the chosen index. $M$ is known as the masking span. For the time dimension, we
set 0.3 as the masking probability and the mask span of six. For the feature
dimension, we set the masking probability to 0.5 and 64 as the mask span. The
time and feature masking parameters are chosen according to a development set.
The above-mentioned data augmentation is akin to SpecAugment [47], a commonly
used speech spectrogram data augmentation method for training speech-to-text
generation models.

The translation model comprises 700 million trainable parameters (300M encoder
and 400M decoder parameters). We fine-tune only 75 million parameters. Most
encoder parameters are fixed to the pre-trained $\tt SAMU\text{-}XLS\text{-}R$
parameters, and the decoder parameters are fixed to the pre-trained $\tt
MBART$ decoder. Below we give details about encoder and decoder fine-tuning.

####  IV-B3 Encoder Fine-Tuning

We keep all the parameters of the speech encoder fixed to their pre-trained
$\tt SAMU\text{-}XLS\text{-}R$ values. Instead, we add a small number of
downstream task-specific parameters to each transformer layer of the speech
encoder following [48]. We insert adapters, a bottleneck Feed-Forward layer,
after the Multi-Headed Self-Attention ($\tt MHSA$) and Fully-Connected ($\tt
FC$) blocks in each transformer layer. An adapter comprises a single hidden
layer with $\tt ReLU$ activation. The input and the output layers have the
same size. In contrast, the hidden layer is a fraction of the input layer
size. We found the optimal (according to a development set) size of the hidden
layer to be a fourth of the input layer size.

The motivation for using adapters is two-fold: (i) Parameter Efficiency: We
only tune the adapter layer parameters, a fraction of the total parameters
that comprise the speech encoder. In our case, the $\tt
SAMU\text{-}XLS\text{-}R$ encoder consists of 300M parameters, and we add 75M
task-specific adapter parameters. Hence, we only fine-tune the 75M parameters
on the downstream task. (ii) Avoids Knowledge Forgetting: By freezing the
encoder’s parameters to pre-trained values, we preserve the essential semantic
knowledge encoded in the $\tt SAMU\text{-}XLS\text{-}R$ encoder that it
acquired during the semantic knowledge-distillation pre-training phase. We
later show (Section VI-D2) that preserving this semantic knowledge is
essential to achieving good cross-lingual transfer. Hence, adapters form a
crucial part of our cross-lingual transfer learning strategy. It must be noted
that research on adapters is ever-expanding. An in-depth analysis of specific
adapter choices on downstream task performance is out-of-scope for this paper.

####  IV-B4 Decoder Fine-Tuning

Like the encoder, we keep most parameters fixed to their pre-trained values
acquired during the multilingual text-to-text translation task. Unlike the
encoder, we do not introduce any new task-specific parameters. Instead, we
tune a fraction of the decoder’s parameters. Each layer in the transformer
decoder consists of the Masked (causal) Multi-Headed Self-Attention ($\tt
MMHSA$), Encoder-Decoder Cross-Attention ($\tt CA$), and two Fully-Connected
($\tt FC$) blocks. The inputs to the $\tt MMHSA$, $\tt CA$, and $\tt FC$
blocks are normalized via Layer Normalization ($\tt LN$) method presented in
[49]. Following [23], we only tune the parameters of the $\tt LN$ and $\tt CA$
blocks. We fine-tune $\tt CA$ because, previously, it is trained as part of a
decoder in a text-to-text translation pipeline. Hence, the $\tt CA$ module
needs to be fine-tuned again for the downstream MST task to make it amenable
to the input from the speech encoder. Moreover, we fine-tune $\tt LN$ because
it is task and dataset-specific and empirically improves the performance of
MST [23].

####  IV-B5 Inference

We use beam search, with a beam size of 5, to generate text translations for a
given speech utterance. The inference process is offline, i.e., the decoder
takes the full input speech utterance into account to generate the output
translation. The decoder generates translation in an autoregressive manner. We
do not use any external language model during inference.

##  V Evaluation

###  V-A Translation Scenarios

We tackle the following translation scenarios in this work.

####  V-A1 Multilingual Translation

We simultaneously train a translation model on several speech-to-text
translation tasks in this scenario. E.g., we train a single model for all 21
X$\rightarrow$EN translation tasks in the CoVoST-2 benchmark. Most of the
training data come from a few high-resource translation tasks such as
FR$\rightarrow$EN and DE$\rightarrow$EN, while most tasks such as
ID$\rightarrow$EN are low-resource. In this scenario, we compare different
translation models to test for cross-lingual translation task transfer from
high to low-resource translation tasks.

####  V-A2 Zero-Shot Multilingual Translation

Given a set of translation tasks, we train a translation model on a subset of
the tasks while keeping the rest hidden during training in this translation
scenario. E.g., we train an X$\rightarrow$EN speech-to-text translation model
using high-resource translation tasks in the CoVoST-2 benchmark while keeping
the mid and low-resource tasks unseen during training. We compare translation
models for zero-shot cross-lingual task transfer in this scenario from high to
mid and low-resource X$\rightarrow$EN translation tasks.

###  V-B Translation Tasks

Figure 3: Number of hours of labeled training data (Y-Axis) for all the 21
X$\rightarrow$EN translation tasks in the CoVoST-2 benchmark. TABLE II:
Training data (hours) for the 72 translation tasks X$\rightarrow$Y in the
Europarl Speech-to-Text translation benchmark.

SRC/TGT | FR | DE | IT | ES | PT | PL | RO | NL | EN
---|---|---|---|---|---|---|---|---|---
FR | - | 21 | 20 | 21 | 22 | 20 | 18 | 22 | 32
DE | 18 | - | 17 | 18 | 18 | 17 | 17 | 18 | 30
IT | 21 | 21 | - | 21 | 21 | 21 | 19 | 20 | 37
ES | 14 | 14 | 14 | - | 14 | 13 | 12 | 13 | 22
PT | 10 | 10 | 10 | 10 | - | 9 | 9 | 9 | 15
PL | 18 | 18 | 17 | 18 | 18 | - | 16 | 18 | 28
RO | 12 | 12 | 12 | 12 | 12 | 12 | - | 12 | 24
NL | 5 | 5 | 4 | 5 | 4 | 4 | 4 | - | 7
EN | 81 | 83 | 80 | 81 | 81 | 79 | 72 | 80 | -

We build translation models to tackle the following translation tasks in this
work.

####  V-B1 X$\rightarrow$EN Speech-to-Text Translation

We build translation models for the 21 X(Speech)$\rightarrow$EN(Text)
translation tasks in the CoVoST-2 translation benchmark [30]. The 21 spoken
languages in CoVoST-2 are French (fr), German (de), Spanish (es), Catalan
(ca), Italian (it), Russian (ru), Chinese (zh), Portuguese (pt), Persian (fa),
Estonian (et), Mongolian (mn), Dutch (nl), Turkish (tr), Arabic (ar), Swedish
(sv), Latvian (lv), Slovenian (sl), Tamil (ta), Japanese (ja), Indonesian
(id), and Welsh (cy). The 21 translation tasks are divided into high, mid, and
low-resource groups, depending on the amount of labeled data available for a
translation task. High-resource tasks have more than 100 hours of labeled
training data, mid-resource tasks have between 10 and 100 hours, and low-
resource have less than ten hours of labeled training data. CoVoST-2 has four
high-resource translation tasks corresponding to fr, de, es, and ca source
languages and five mid-resource tasks corresponding to it, ru, zh, pt, and fa
languages. The rest of the tasks are low-resource. Fig. 3 presents the
training data for each of the 21 translation tasks. Notice the data imbalance
among different tasks.

In the multilingual translation scenario, we simultaneously train translation
models on all 21 tasks mentioned above. We only train translation models on
the four high-resource tasks in the zero-shot scenario.

TABLE III: We compare our proposed $\tt SAMU\text{-}XLS\text{-}R$-300M translation model with several other translation models, whose encoders are initialized using differently sized pre-trained $\tt XLS\text{-}R$ multilingual unimodal speech encoders. The performance is measured using BLEU-4, Google-BLEU, ROUGE-L, METEOR, BERTScore, and NIST translation metrics. | BLEU-4 |  | Google-BLEU
---|---|---|---
Model | High | Mid | Low | TRFGap |  | High | Mid | Low | TRFGap
$\tt XLS\text{-}R$-300M | 30.6 | 18.9 | 5.1 | 25.1 |  | 0.36 | 0.24 | 0.10 | 0.26
$\tt XLS\text{-}R$-1B | 34.3 | 25.5 | 11.7 | 22.6 |  | 0.38 | 0.29 | 0.16 | 0.22
$\tt XLS\text{-}R$-2B | 36.1 | 27.7 | 15.1 | 21.0 |  | 0.39 | 0.31 | 0.20 | 0.19
$\tt SAMU\text{-}XLS\text{-}R$-300M | 34.4 | 31.1 | 20.3 | 14.1 |  | 0.38 | 0.34 | 0.24 | 0.13
Cascaded | 32.6 | 29.7 | 22.5 | 10.1 |  | 0.36 | 0.33 | 0.26 | 0.10
Transcripts | 36.4 | 34.2 | 27.9 | 8.5 |  | 0.39 | 0.37 | 0.32 | 0.08
| ROUGE-L |  | METEOR
Model | High | Mid | Low | TRFGap |  | High | Mid | Low | TRFGap
$\tt XLS\text{-}R$-300M | 0.60 | 0.44 | 0.23 | 0.37 |  | 0.62 | 0.45 | 0.24 | 0.38
$\tt XLS\text{-}R$-1B | 0.61 | 0.49 | 0.31 | 0.30 |  | 0.63 | 0.50 | 0.32 | 0.31
$\tt XLS\text{-}R$-2B | 0.63 | 0.53 | 0.38 | 0.25 |  | 0.65 | 0.53 | 0.39 | 0.26
$\tt SAMU\text{-}XLS\text{-}R$-300M | 0.62 | 0.58 | 0.42 | 0.19 |  | 0.64 | 0.59 | 0.45 | 0.19
Cascaded | 0.60 | 0.56 | 0.45 | 0.14 |  | 0.61 | 0.56 | 0.46 | 0.15
Transcripts | 0.64 | 0.61 | 0.52 | 0.11 |  | 0.66 | 0.62 | 0.54 | 0.12
| BERTScore |  | NIST
Model | High | Mid | Low | TRFGap |  | High | Mid | Low | TRFGap
$\tt XLS\text{-}R$-300M | 0.56 | 0.33 | 0.04 | 0.52 |  | 8.0 | 5.0 | 1.9 | 6.1
$\tt XLS\text{-}R$-1B | 0.58 | 0.41 | 0.16 | 0.42 |  | 8.3 | 5.9 | 3.0 | 5.3
$\tt XLS\text{-}R$-2B | 0.61 | 0.45 | 0.25 | 0.36 |  | 8.6 | 6.3 | 3.7 | 4.9
$\tt SAMU\text{-}XLS\text{-}R$-300M | 0.59 | 0.54 | 0.34 | 0.25 |  | 8.3 | 7.1 | 4.4 | 4.0
Cascaded | 0.56 | 0.50 | 0.37 | 0.20 |  | 8.2 | 6.9 | 4.9 | 3.3
Transcripts | 0.62 | 0.58 | 0.47 | 0.14 |  | 8.7 | 7.7 | 5.7 | 3.0
TABLE IV: We compare our proposed $\tt SAMU\text{-}XLS\text{-}R$-300M translation model with ${\tt mSLAM}$ translation models, whose encoders are initialized using differently sized pre-trained ${\tt mSLAM}$ multilingual multimodal speech encoders. The performance is measured using the BLEU-4 translation metric. | BLEU-4 |
---|---|---
Model | High | Mid | Low | TRFGap |
${\tt mSLAM}$-600M | 37.6 | 27.8 | 15.1 | 22.5 |
${\tt mSLAM}$-2B | 37.8 | 29.6 | 18.5 | 19.3 |
$\tt SAMU\text{-}XLS\text{-}R$-300M | 34.4 | 31.1 | 20.3 | 14.1 |
TABLE V: We compare our proposed $\tt SAMU\text{-}XLS\text{-}R$-300M translation model with several other translation models, whose encoders are initialized using differently sized pre-trained $\tt XLS\text{-}R$ multilingual unimodal speech encoders. The performance is measured using the BLEU-4 translation metric. Model | fr | de | es | ca | it | fa | ru | zh | pt
---|---|---|---|---|---|---|---|---|---
$\tt XLS\text{-}R$-300M | 34.9 | 29.3 | 35.9 | 30.6 | 30.9 | 6.3 | 30.0 | 5.2 | 30.8
$\tt XLS\text{-}R$-1B | 36.3 | 31.4 | 37.8 | 32.1 | 33.5 | 9.1 | 35.7 | 6.9 | 41.4
$\tt XLS\text{-}R$-2B | 37.6 | 33.6 | 39.1 | 33.9 | 35.0 | 13.0 | 39.5 | 9.4 | 41.8
$\tt SAMU\text{-}XLS\text{-}R$-300M | 36.1 | 31.7 | 37.9 | 31.9 | 34.0 | 22.0 | 42.1 | 13.1 | 44.2
Cascaded | 32.7 | 30.9 | 36.4 | 30.3 | 32.5 | 14.3 | 42.9 | 14.4 | 44.2
Transcripts | 38.4 | 34.1 | 41.1 | 32.1 | 36.4 | 24.2 | 46.0 | 17.4 | 46.9
Model | nl | tr | et | mn | ar | sv | lv | sl | ta
$\tt XLS\text{-}R$-300M | 25.1 | 6.7 | 4.0 | 0.2 | 3.9 | 11.3 | 7.4 | 8.3 | 0.0
$\tt XLS\text{-}R$-1B | 29.6 | 11.1 | 8.0 | 0.6 | 9.3 | 24.5 | 15.7 | 16.8 | 0.1
$\tt XLS\text{-}R$-2B | 31.6 | 16.9 | 11.2 | 1.5 | 17.1 | 29.7 | 19.7 | 19.0 | 0.5
$\tt SAMU\text{-}XLS\text{-}R$-300M | 34.9 | 28.4 | 11.6 | 3.4 | 36.6 | 28.5 | 1.9 | 12.9 | 4.0
Cascaded | 33.0 | 25.1 | 17.4 | 0.0 | 35.7 | 39.6 | 20.3 | 27.8 | 1.6
Transcripts | 36.3 | 28.8 | 25.6 | 3.8 | 44.6 | 46.4 | 29.2 | 38.4 | 2.2
Model | cy | ja | id |  |  |  |  |  |
$\tt XLS\text{-}R$-300M | 2.8 | 0.6 | 1.2 |  |  |  |  |  |
$\tt XLS\text{-}R$-1B | 6.61 | 1.3 | 7.8 |  |  |  |  |  |
$\tt XLS\text{-}R$-2B | 14.2 | 3.5 | 16.4 |  |  |  |  |  |
$\tt SAMU\text{-}XLS\text{-}R$-300M | 34.1 | 13.1 | 34.4 |  |  |  |  |  |
Cascaded | 6.4 | 19.4 | 43.6 |  |  |  |  |  |
Transcripts | 9.0 | 20.8 | 49.8 |  |  |  |  |  |

####  V-B2 X$\rightarrow$Y Speech-to-Text Translation

We develop translation models for the 72 X$\rightarrow$Y translation tasks in
the Europarl benchmark [29]. There are nine spoken languages in Europarl
namely, en, fr, de, it, es, pt, pl, ro, and nl. Speech utterances in each
spoken language are paired with their corresponding text translations in eight
other languages. In the zero-shot scenario, we train the translation models on
32 tasks corresponding to the following four source languages fr, de, es, and
it. Each of the four source languages is paired with eight target languages.
Table II shows each translation task’s labeled training data.

###  V-C Translation Models

####  V-C1 $\tt SAMU\text{-}XLS\text{-}R$-300M

We propose $\tt SAMU\text{-}XLS\text{-}R$-300M transformer model for
translation in this work. The encoder is initialized using the pre-trained
$\tt SAMU\text{-}XLS\text{-}R$ speech encoder (Section IV-A), and the decoder
is initialized using the pre-trained $\tt MBART$ text decoder. The suffix 300M
in $\tt SAMU\text{-}XLS\text{-}R$-300M refers to the model’s size of 300M
parameters.

####  V-C2 $\tt XLS\text{-}R$-(300M, 1B, 2B)

We compare $\tt SAMU\text{-}XLS\text{-}R$-300M with three $\tt XLS\text{-}R$
speech encoder based translation models namely, $\tt XLS\text{-}R$-300M, $\tt
XLS\text{-}R$-1B, and $\tt XLS\text{-}R$-2B. The three translation models
differ from $\tt SAMU\text{-}XLS\text{-}R$-300M model in that the encoder of
the translation model is initialized using pre-trained $\tt XLS\text{-}R$
speech encoders of different sizes ranging from 300M to 2B parameters. The
decoder is initialized using the pre-trained $\tt MBART$ decoder. Unlike our
multimodal $\tt SAMU\text{-}XLS\text{-}R$ speech encoder, $\tt XLS\text{-}R$
is only trained using unlabeled speech data. Also, $\tt
SAMU\text{-}XLS\text{-}R$ is specifically trained to learn semantic
representations, while $\tt XLS\text{-}R$ has no constraints imposed during
its training phase to encode semantic knowledge.

####  V-C3 mSLAM

We compare $\tt SAMU\text{-}XLS\text{-}R$-300M translation model against two
$\tt mSLAM$ [18] speech encoder based translation models namely, $\tt
mSLAM$-600M, and $\tt mSLAM$-2B. Like $\tt SAMU\text{-}XLS\text{-}R$, $\tt
mSLAM$ speech encoder is a multimodal speech-text encoder. Unlike $\tt
SAMU\text{-}XLS\text{-}R$, which is trained using semantic supervision from a
pre-trained semantic text encoder, $\tt mSLAM$ is not trained with explicit
semantic supervision.

####  V-C4 Cascaded Translation

We compare $\tt SAMU\text{-}XLS\text{-}R$-300M with a strong cascaded
translation system. We perform the translation in two steps: (i) Transcribe
the speech utterance using an ASR model, and (ii) Use a text-to-text
translation model to translate the ASR transcript to text in a target
language. We use whisper-large-v2 444https://huggingface.co/openai/whisper-
large-v2 [50] as the ASR model in the first step and $\tt MBART$ [28] text-to-
text translation model for the second step in the cascade. For
X$\rightarrow$EN cascade, we use $\tt MBART$-many-to-English text-to-text
translation model555https://huggingface.co/facebook/mbart-large-50-many-to-
one-mmt. The Whisper ASR model is multilingual that supports transcription of
around 93 languages. Since [50] shows that whisper achieves state-of-the-art
ASR performance on several public benchmarks, we choose Whisper for
automatically transcribing speech. $\tt MBART$-many-to-English translation
model can translate text from 50 languages to English.

####  V-C5 Transcripts

As a topline, we use the ground-truth text transcripts corresponding to speech
utterances and use $\tt MBART$-many-to-English to translate to English.

##  VI Results

###  VI-A Multilingual X$\rightarrow$EN Translation

Table III shows the performance of different translation models on the high,
mid, and low-resource translation groups in the CoVoST-2 speech-to-text
translation benchmark. We compare our proposed $\tt
SAMU\text{-}XLS\text{-}R$-300M translation model against $\tt
XLS\text{-}R$-300M, $\tt XLS\text{-}R$-1B, and $\tt XLS\text{-}R$-2B
translation models. CoVoST-2 comprises 21 X$\rightarrow$EN translation tasks,
and the translation models are trained simultaneously on all translation
tasks. See Section V-B for details about the translation tasks, and Section
V-C for details about the different translation models.

The model performance is measured using the standard translation metrics,
namely BLEU-4 [51], Google-BLEU [52], ROUGE-L [53], METEOR [54], BERTScore
[55], and NIST [56]. We make the following observations: (i) On High resource
tasks, the $\tt XLS\text{-}R$-2B model performs the best, with SAMU-XLS-R-300M
lagging a couple of points behind. Compared to the similar-sized $\tt
XLS\text{-}R$-300M model, $\tt SAMU\text{-}XLS\text{-}R$-300M performs 4 BLEU
points better. (ii) On Mid resource tasks, $\tt SAMU\text{-}XLS\text{-}R$-300M
outperforms all the models achieving a BLEU score of 31.1, which is
significantly better than $\tt XLS\text{-}R$-300M model’s BLEU score of 5.1.
$\tt SAMU\text{-}XLS\text{-}R$-300M even outperforms the much larger $\tt
XLS\text{-}R$-2B speech encoder by 3.3 BLEU points. (iii) On Low resource
tasks, $\tt SAMU\text{-}XLS\text{-}R$-300M performs the best. Compared to the
similar-sized $\tt XLS\text{-}R$-300M model, $\tt
SAMU\text{-}XLS\text{-}R$-300M does better by 15 BLEU points. It also
outperforms the much larger $\tt XLS\text{-}R$-2B by 5.2 BLEU points. The
cross-lingual transfer gap (TRFGap), which is the difference in performance
between high and low resource task groups, is significantly less (14.1 BLEU)
for $\tt SAMU\text{-}XLS\text{-}R$-300M model compared to other models. Second
to $\tt SAMU\text{-}XLS\text{-}R$-300M is $\tt XLS\text{-}R$-2B, which has a
TRFGap of 21 BLEU points while having 500% more parameters. Similar arguments
can be made using metrics other than BLEU-4. On all metrics, $\tt
SAMU\text{-}XLS\text{-}R$ is significantly better than the $\tt XLS\text{-}R$
baselines and comparable to the strong cascaded system, while significantly
worse than the model which uses the human transcripts corresponding to speech
utterances for translation into English using the $\tt MBART$-many-to-English
text translation model. For the subsequent experiments, we only report BLEU-4
scores.

Table IV compares $\tt SAMU\text{-}XLS\text{-}R$-300M translation model with
${\tt mSLAM}$-600M, and ${\tt mSLAM}$-2B models that use differently sized
pre-trained multimodal (speech-text) multilingual ${\tt mSLAM}$ speech
encoder. $\tt SAMU\text{-}XLS\text{-}R$-300M performs better on mid- and low-
resource translation tasks. Importantly, $\tt SAMU\text{-}XLS\text{-}R$-300M
has a lower cross-lingual transfer gap (TRFGap) between high and low resource
groups of 14.1 BLEU points compared to 22.5 for ${\tt mSLAM}$-600M, and 19.3
for ${\tt mSLAM}$-2B. The BLEU scores for ${\tt mSLAM}$ models are lifted from
[18]. Since mSLAM [18] report only BLEU scores on the CoVoST-2 benchmark, and
we do not have access to ${\tt mSLAM}$ models, we can not evaluate the model
using metrics other than BLEU-4.

The above observations validate our claims that building translation
technology on top of semantic speech representations would increase the cross-
lingual task knowledge transfer from high to low-resource languages. Similar
inferences can be reached by using metrics other than BLEU-4.

Table V shows the performance of different translation models on each of the
21 X$\rightarrow$EN speech-to-text translation tasks in the CoVoST-2
benchmark. We observe that $\tt SAMU\text{-}XLS\text{-}R$-300M significantly
outperforms the similarly sized $\tt XLS\text{-}R$-300M translation model on
several mid and low-resource languages. Some notable improvements are for the
following source languages: id (34.4 vs. 1.2 BLEU), cy (34.1 vs. 2.8 BLEU), ja
(13.1 vs 0.6 BLEU), sv (28.5 vs 11.3 BLEU), tr (28.4 vs 6.7 BLEU), ar (36.6
vs. 3.9 BLEU), fa (22.0 vs. 6.3 BLEU), and pt (44.2 vs. 30.8 BLEU). For some
source languages $\tt SAMU\text{-}XLS\text{-}R$-300M does not do well, such as
lv (1.9 BLEU), sl (12.9 BLEU), mn (3.4 BLEU), et (11.6 BLEU), ta (4.0), ja
(13.1 BLEU), and zh (13.1 BLEU). For ta, ja, mn, and zh, the topline (${\tt
Transcripts}$) model also performs poorly. For sl, and lv, the topline text-
to-text translation and cascaded models perform significantly better than the
$\tt SAMU\text{-}XLS\text{-}R$-300M model.

Poor performance of $\tt SAMU\text{-}XLS\text{-}R$-300M on lv, mn, and sl can
be explained by the lack of available transcribed speech data in these
languages for multimodal pre-training of $\tt SAMU\text{-}XLS\text{-}R$
(Section IV-A). We have 7 hours for lv, 12 hours for mn, and 9 hours of
transcribed speech for sl, compared to 317 hours for fa, 85 hours for ar, 116
hours for cy, 98 hours for nl, 40 hours for sv, 25.8 hours for id, 162 hours
for ru, 400 hours for es, and close to 1K hours for fr, de, and ca. However,
for ta, zh, and ja we have a decent amount of transcribed speech, but still,
the performance is relatively poor. This can be explained away by observing
the performance of the topline (${\tt Transcripts}$), where we use a pre-
trained $\tt MBART$-many-to-English text-to-text translation model (Details in
Section V-C) that translates the ground-truth transcripts corresponding to
speech utterances in language X to text in English. The topline performance
for zh, ja, and ta is relatively poor. Since we use the decoder of $\tt
MBART$-many-to-English model to initialize the decoder of our speech-to-text
translation model $\tt SAMU\text{-}XLS\text{-}R$-300M, we also observe poor
speech-text-translation performance on these tasks.

###  VI-B Zero-Shot X$\rightarrow$EN Speech-to-Text Translation.

Next, we train the translation models on four high-resource X$\rightarrow$EN
translation tasks in the CoVoST-2 benchmark (See Section V-B for details). We
evaluate the X$\rightarrow$EN translation models on the high, mid, and low
task groups to test for zero-shot cross-lingual transfer capability of $\tt
SAMU\text{-}XLS\text{-}R$-300M from high to mid and low-resource
X$\rightarrow$EN tasks. We compare $\tt SAMU\text{-}XLS\text{-}R$-300 with
$\tt XLS\text{-}R$-300M translation model. Figure 4 presents the results. We
observe that $\tt SAMU\text{-}XLS\text{-}R$-300M performs on average 18.8 BLEU
points better in the mid-resource and 11.9 BLEU points in the low-resource
group. The cross-lingual transfer gap between the high & mid and high & low
groups is significantly smaller for $\tt SAMU\text{-}XLS\text{-}R$-300M (9.0,
and 20.8) than $\tt XLS\text{-}R$-300M (25.2, and 30.1).

Figure 4: We report average BLEU-4 for the zero-shot X$\rightarrow$EN
multilingual speech-to-text translation scenario on the high, mid, and low
resource task groups in the CoVoST-2 benchmark. We compare our translation
model $\tt SAMU\text{-}XLS\text{-}R$-300M with the similarly sized $\tt
XLS\text{-}R$-300M translation model. The translation models are only trained
on high-resource groups, while the mid and low-resource groups are unseen
during training. Figure 5: Absolute BLEU score improvements using $\tt
SAMU\text{-}XLS\text{-}R$-300M over $\tt XLS\text{-}R$-300M baseline on the 72
X$\rightarrow$Y translation tasks in the Europarl benchmark. The translation
models are trained on a subset of 32 translation tasks, corresponding to four
source languages, while 40 tasks are unseen during training corresponding to
five source languages. Figure 6: We compare on the Europarl X$\rightarrow$EN
benchmark $\tt XLS\text{-}R$ and $\tt XLS\text{-}R$-CTC initialization of the
translation model’s encoder. $\tt XLS\text{-}R$ is pre-trained using unlabeled
speech via self-supervised learning, while $\tt XLS\text{-}R$-CTC refers to
the $\tt XLS\text{-}R$ encoder that is fine-tuned (after self-supervised pre-
training) using transcribed speech data. We report the BLEU-4 score for eight
source spoken languages. Speech utterances in each source language are paired
with its English text translations.

These results strengthen our claims that building speech translation
technology with semantic speech representations would improve cross-lingual
transfer across languages. Note that zero-shot implies that the translation
model during its training does not see any paired X$\rightarrow$EN translation
data for mid and low-resource languages X. Transcribed speech data was
available for these languages during multimodal pre-training of $\tt
SAMU\text{-}XLS\text{-}R$ speech encoder which is used to initialize the
encoder of the $\tt SAMU\text{-}XLS\text{-}R$-300M translation model.

###  VI-C Zero-Shot X$\rightarrow$Y Speech-to-Text Translation.

Finally, to further bolster our claims about the usefulness of semantic speech
representations for translation, we compare translation models on the 72
X$\rightarrow$Y translation tasks in the Europarl Benchmark. See Section V-B
for details about translation tasks and available data for training
translation models.

We train translation models $\tt SAMU\text{-}XLS\text{-}R$-300M and $\tt
XLS\text{-}R$-300M on a 32-task subset out of 72 tasks. The 32 tasks
correspond to four source spoken languages: fr, de, es, and it, while the
remaining 40 tasks correspond to five source languages: pl, pt, ro, nl, and
en. Speech utterances in each source language are paired with text
translations in eight other languages. Unlike X$\rightarrow$EN translation
models discussed above, we initialize the decoder of X$\rightarrow$Y
translation models with the decoder of $\tt MBART$-many-to-
many666https://huggingface.co/facebook/mbart-large-50-many-to-many-mmt model
instead of $\tt MBART$-many-to-English, since we have to generate translations
in multiple target languages.

Figure 5 compares the performance of $\tt XLS\text{-}R$-300M, and $\tt
SAMU\text{-}XLS\text{-}R$-300M on the 72 translation tasks. We report the
absolute BLEU-4 score improvement that $\tt SAMU\text{-}XLS\text{-}R$-300M
achieves over $\tt XLS\text{-}R$-300M baseline translation model. The darker
the cell in the figure, the greater the improvement in the BLEU score. We
observe that $\tt SAMU\text{-}XLS\text{-}R$-300M performs drastically better
than $\tt XLS\text{-}R$-300M on the 40 unseen (during translation model
training) translation tasks and at par on the 32 seen (during translation
model training) translation tasks. We observe the biggest improvements in
unseen tasks such as pl$\rightarrow$en (23.8), ro$\rightarrow$en (21.9),
pl$\rightarrow$en (19.9), pl$\rightarrow$fr (19.6), ro$\rightarrow$es (17.2),
etc. Overall, $\tt SAMU\text{-}XLS\text{-}R$-300M improves over $\tt
XLS\text{-}R$-300M baseline model by an average of 12 BLEU-4 points.

TABLE VI: We compare the translation model’s performance when using Adapter-
based fine-tuning vs. fine-tuning all the encoder parameters. The performance
is measured using the BLEU-4 translation metric.

Model | fr | de | es | it | pl | pt | ro | nl
---|---|---|---|---|---|---|---|---
$\tt XLS\text{-}R$-F | 33.0 | 25.4 | 29.4 | 30.3 | 2.4 | 12.7 | 7.6 | 5.2
$\tt XLS\text{-}R$-A | 29.8 | 22.4 | 25.5 | 27.2 | 3.2 | 15.6 | 9.1 | 8.2
$\tt SAMU\text{-}XLS\text{-}R$-F | 33.2 | 26.3 | 29.7 | 30.7 | 3.0 | 12.2 | 9.8 | 7.0
$\tt SAMU\text{-}XLS\text{-}R$-A | 32.3 | 25.5 | 29.3 | 28.7 | 24.3 | 18.4 | 29.9 | 24.1

###  VI-D Analysis

####  VI-D1 $\tt XLS\text{-}R$ vs. $\tt XLS\text{-}R$-CTC

The experiments above compare $\tt SAMU\text{-}XLS\text{-}R$ with $\tt
XLS\text{-}R$ multilingual speech encoder. $\tt XLS\text{-}R$ is trained using
multilingual unlabeled speech data in 128 languages via self-supervised
learning, while $\tt SAMU\text{-}XLS\text{-}R$ fine-tunes pre-trained $\tt
XLS\text{-}R$ using multilingual transcribed speech data via semantic
knowledge distillation. But, what if instead of fine-tuning pre-trained $\tt
XLS\text{-}R$ with semantic supervision from text transcriptions like $\tt
SAMU\text{-}XLS\text{-}R$ does, we fine-tune $\tt XLS\text{-}R$ on the task of
Automatic Speech Recognition using the same multilingual transcribed speech
data that was used for $\tt SAMU\text{-}XLS\text{-}R$ training? We compare the
translation performance of $\tt XLS\text{-}R$ and $\tt XLS\text{-}R$-CTC
encoders in Fig. 6. $\tt XLS\text{-}R$-CTC refers to the encoder we get after
supervised CTC-based fine-tuning of pre-trained $\tt XLS\text{-}R$ using
multilingual transcribed speech data. CTC [57] is a standard framework for
training ASR models end-to-end. We observe that the pre-trained $\tt
XLS\text{-}R$ encoder performs better than $\tt XLS\text{-}R$-CTC encoder.
$\tt XLS\text{-}R$-CTC encoder is slightly better on some translation tasks.
Still, the difference is so small that our previous observations on the
efficacy of our proposed semantic speech encoder $\tt
SAMU\text{-}XLS\text{-}R$ are not impacted, even though we compared $\tt
SAMU\text{-}XLS\text{-}R$ with the $\tt XLS\text{-}R$ encoder, which is
trained using unlabeled multilingual speech.

####  VI-D2 Adapter vs. Full Encoder Fine-Tuning

As mentioned in Section IV-B3, we perform Adapter-based fine-tuning of the
translation model’s encoder, where we insert new task-specific parameters in
the form of adapters in each encoder layer. We only fine-tune adapter layers
while keeping the rest of the layers frozen to their pre-trained values. Table
VI compares the translation model’s performance when using Adapter-based fine-
tuning vs. fine-tuning all the encoder parameters. We observe that using
adapter fine-tuning with $\tt XLS\text{-}R$-300M ($\tt XLS\text{-}R$-A)
translation model brings performance gains on low-resource tasks such as
ro$\rightarrow$en, nl$\rightarrow$en, pt$\rightarrow$en, and
pl$\rightarrow$en. In contrast, the performance degrades significantly for
higher-resource tasks compared to full encoder fine-tuning ($\tt
XLS\text{-}R$-F). We observe a similar trend with $\tt
SAMU\text{-}XLS\text{-}R$-300M translation model. But, the performance gains
for low-resource tasks are drastic with adapter-based fine-tuning of the
encoder ($\tt SAMU\text{-}XLS\text{-}R$-A). Again, this is due to our proposed
semantic speech encoder $\tt SAMU\text{-}XLS\text{-}R$, which results in a
significant cross-lingual transfer from high to low-resource translation
tasks. This result also shows that preserving semantic knowledge during
training for downstream translation tasks, learned by the $\tt
SAMU\text{-}XLS\text{-}R$ encoder as a result of our multimodal learning
framework, is essential.

##  VII Conclusion

This paper addresses the central question of cross-lingual transfer learning
in Natural Language Processing. We focus on the problem of multilingual spoken
language translation, which we model using the standard encoder-decoder model.
We analyze the impact of different encoder initializations on the downstream
translation task performance. We show that by initializing the encoder with an
encoder that we pre-train using the newly introduced semantic knowledge
distillation framework $\tt SAMU\text{-}XLS\text{-}R$, we achieve
significantly better cross-lingual transfer in the downstream speech-to-text
translation task than the baselines. The baseline translation models use the
state-of-the-art multilingual pre-trained speech encoder $\tt XLS\text{-}R$
and others for initialization.

To substantiate our claims, we perform multilingual translation on two public
benchmarks, CoVoST-2 and Europarl. On the 21 X$\rightarrow$English CoVoST-2
speech translation tasks, we achieve an average improvement of 12.8 BLEU
points. In the zero-shot scenario, where we train the translation model only
on the four high-resource languages while keeping the rest 17 languages unseen
(during training), we achieve an average improvement of 11.8 BLEU points over
the baseline $\tt XLS\text{-}R$ encoder initialization. In particular, we
achieve drastic improvements of 18.8 and 11.9 average BLEU points on medium
and low-resource languages, respectively. We made similar observations on the
Europarl X$\rightarrow$Y speech-to-text translation benchmark.

Our work has limitations. Currently, training $\tt SAMU\text{-}XLS\text{-}R$
requires access to multilingual transcribed data, which could be hard for many
spoken languages. Also, the dependence on a pre-trained text encoder hinders
expanding $\tt SAMU\text{-}XLS\text{-}R$ to more languages. Hence, future work
should focus on injecting semantic information via weakly supervised learning
using unaligned speech and text data and without using the $\tt LaBSE$ text
encoder.

## References

  * [1] D. Harwath, A. Torralba, and J. Glass, “Unsupervised learning of spoken language with visual context,” _Advances in Neural Information Processing Systems_ , vol. 29, 2016.
  * [2] W.-N. Hsu, Y. Zhang, and J. Glass, “Unsupervised learning of disentangled and interpretable representations from sequential data,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [3] S. Khurana, S. R. Joty, A. Ali, and J. Glass, “A factorial deep markov model for unsupervised disentangled representation learning from speech,” in _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2019, pp. 6540–6544.
  * [4] S. Pascual, M. Ravanelli, J. Serrà, A. Bonafonte, and Y. Bengio, “Learning problem-agnostic speech representations from multiple self-supervised tasks,” _arXiv:1904.03416_ , 2019.
  * [5] S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsupervised pre-training for speech recognition,” _arXiv:1904.05862_ , 2019. [Online]. Available: https://arxiv.org/abs/1904.05862
  * [6] A. Baevski, S. Schneider, and M. Auli, “vq-wav2vec: Self-supervised learning of discrete speech representations,” in _International Conference on Learning Representations_ , 2020. [Online]. Available: https://openreview.net/forum?id=rylwJxrYDS
  * [7] Y.-A. Chung and J. Glass, “Generative pre-training for speech with autoregressive predictive coding,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 3497–3501.
  * [8] S. Khurana, A. Laurent, W.-N. Hsu, J. Chorowski, A. Lancucki, R. Marxer, and J. Glass, “A convolutional deep markov model for unsupervised speech representation learning,” _arXiv:2006.02547_ , 2020. [Online]. Available: https://arxiv.org/abs/2006.02547
  * [9] A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” _arXiv:2006.11477_ , 2020.
  * [10] D. Harwath, W.-N. Hsu, and J. Glass, “Learning hierarchical discrete linguistic units from visually-grounded speech,” in _International Conference on Learning Representations_ , 2020. [Online]. Available: https://openreview.net/forum?id=B1elCp4KwH
  * [11] A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli, “Unsupervised cross-lingual representation learning for speech recognition,” _arXiv:2006.13979_ , 2020. [Online]. Available: https://arxiv.org/abs/2006.13979
  * [12] A. T. Liu, S.-W. Li, and H. yi Lee, “TERA: Self-supervised learning of transformer encoder representation for speech,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 2351–2366, 2021. [Online]. Available: https://doi.org/10.1109%2Ftaslp.2021.3095662
  * [13] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [14] Y.-A. Chung, Y. Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y. Wu, “W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,” _arXiv:2108.06209_ , 2021. [Online]. Available: https://arxiv.org/abs/2108.06209
  * [15] A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli, “Xls-r: Self-supervised cross-lingual speech representation learning at scale,” _arXiv:2111.09296_ , 2021. [Online]. Available: https://arxiv.org/abs/2111.09296
  * [16] A. H. Liu, S. Jin, C.-I. J. Lai, A. Rouditchenko, A. Oliva, and J. Glass, “Cross-modal discrete representation learning,” _arXiv:2106.05438_ , 2021\. [Online]. Available: https://arxiv.org/abs/2106.05438
  * [17] S. Khurana, A. Laurent, and J. Glass, “Samu-xlsr: Semantically-aligned multimodal utterance-level cross-lingual speech representation,” _IEEE Journal of Selected Topics in Signal Processing_ , pp. 1–13, 2022.
  * [18] A. Bapna, C. Cherry, Y. Zhang, Y. Jia, M. Johnson, Y. Cheng, S. Khanuja, J. Riesa, and A. Conneau, “mSLAM: Massively multilingual joint pre-training for speech and text,” _arXiv:2202.01374_ , 2022. [Online]. Available: https://arxiv.org/abs/2202.01374
  * [19] A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” _arXiv:1807.03748_ , 2018. [Online]. Available: https://arxiv.org/abs/1807.03748
  * [20] M. Gutmann and A. Hyvärinen, “Noise-contrastive estimation: A new estimation principle for unnormalized statistical models,” in _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_ , ser. Proceedings of Machine Learning Research, Y. W. Teh and M. Titterington, Eds., vol. 9. Chia Laguna Resort, Sardinia, Italy: PMLR, 13–15 May 2010, pp. 297–304. [Online]. Available: https://proceedings.mlr.press/v9/gutmann10a.html
  * [21] R. Bommasani and et. al., “On the opportunities and risks of foundation models,” _arXiv:2108.07258_ , 2021. [Online]. Available: https://arxiv.org/abs/2108.07258
  * [22] M. Rivière, A. Joulin, P.-E. Mazaré, and E. Dupoux, “Unsupervised pretraining transfers well across languages,” _arXiv:2002.02848_ , 2020.
  * [23] X. Li, C. Wang, Y. Tang, C. Tran, Y. Tang, J. Pino, A. Baevski, A. Conneau, and M. Auli, “Multilingual speech translation with efficient finetuning of pretrained models,” _arXiv:2010.12829_ , 2020. [Online]. Available: https://arxiv.org/abs/2010.12829
  * [24] J. Shor, A. Jansen, W. Han, D. Park, and Y. Zhang, “Universal paralinguistic speech representations using self-supervised conformers,” _arXiv:2110.04621_ , 2021. [Online]. Available: https://arxiv.org/abs/2110.04621
  * [25] S. wen Yang, P.-H. Chi, Y.-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin, T.-H. Huang, W.-C. Tseng, K. tik Lee, D.-R. Liu, Z. Huang, S. Dong, S.-W. Li, S. Watanabe, A. Mohamed, and H. yi Lee, “Superb: Speech processing universal performance benchmark,” _arXiv:2105.01051_ , 2021.
  * [26] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” _arXiv:1409.3215_ , 2014. [Online]. Available: https://arxiv.org/abs/1409.3215
  * [27] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” _arXiv:1706.03762_ , 2017. [Online]. Available: https://arxiv.org/abs/1706.03762
  * [28] Y. Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, and L. Zettlemoyer, “Multilingual denoising pre-training for neural machine translation,” _Transactions of the Association for Computational Linguistics_ , vol. 8, pp. 726–742, 2020. [Online]. Available: https://aclanthology.org/2020.tacl-1.47
  * [29] J. Iranzo-Sánchez, J. A. Silvestre-Cerdà, J. Jorge, N. Roselló, A. Giménez, A. Sanchis, J. Civera, and A. Juan, “Europarl-st: A multilingual corpus for speech translation of parliamentary debates,” _arXiv:1911.03167_ , 2019. [Online]. Available: https://arxiv.org/abs/1911.03167
  * [30] C. Wang, J. Pino, A. Wu, and J. Gu, “Covost: A diverse multilingual speech-to-text translation corpus,” _arXiv:2002.01320_ , 2020. [Online]. Available: https://arxiv.org/abs/2002.01320
  * [31] G. Laperrière, V. Pelloin, M. Rouvier, T. Stafylakis, and Y. Estève, “On the use of semantically-aligned speech representations for spoken language understanding,” _arXiv:2210.05291_ , 2022. [Online]. Available: https://arxiv.org/abs/2210.05291
  * [32] F. Feng, Y. Yang, D. Cer, N. Arivazhagan, and W. Wang, “Language-agnostic bert sentence embedding,” _arXiv:2007.01852_ , 2020. [Online]. Available: https://arxiv.org/abs/2007.01852
  * [33] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of the IEEE_ , vol. 86, no. 11, pp. 2278–2324, 1998.
  * [34] D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),” _arXiv:1606.08415_ , 2016. [Online]. Available: https://arxiv.org/abs/1606.08415
  * [35] A. F. Agarap, “Deep learning using rectified linear units (ReLU),” _arXiv:1803.08375_ , 2018. [Online]. Available: https://arxiv.org/abs/1803.08375
  * [36] C. Wang, M. Riviere, A. Lee, A. Wu, C. Talnikar, D. Haziza, M. Williamson, J. Pino, and E. Dupoux, “VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,” in _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Online: Association for Computational Linguistics, Aug. 2021, pp. 993–1003. [Online]. Available: https://aclanthology.org/2021.acl-long.80
  * [37] R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common Voice: A massively-multilingual speech corpus,” _arXiv:1912.06670_ , 2020.
  * [38] V. Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert, “Mls: A large-scale multilingual dataset for speech research,” _arXiv:2012.03411_ , 2020.
  * [39] J. Valk and T. Alumäe, “Voxlingua107: a dataset for spoken language recognition,” _arXiv:2011.12998_ , 2020. [Online]. Available: https://arxiv.org/abs/2011.12998
  * [40] P. Safari, M. India, and J. Hernando, “Self-attention encoding and pooling for speaker recognition,” _arXiv:2008.01077_ , 2020.
  * [41] T. Ko, V. Peddinti, D. Povey, and S. Khudanpur, “Audio augmentation for speech recognition,” in _Proc. Interspeech 2015_ , 2015, pp. 3586–3589.
  * [42] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 2818–2826.
  * [43] R. Müller, S. Kornblith, and G. Hinton, “When does label smoothing help?” _arXiv:1906.02629_ , 2019. [Online]. Available: https://arxiv.org/abs/1906.02629
  * [44] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv:1412.6980_ , 2014. [Online]. Available: https://arxiv.org/abs/1412.6980
  * [45] P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu, “Mixed precision training,” _arXiv:1710.03740_ , 2017. [Online]. Available: https://arxiv.org/abs/1710.03740
  * [46] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli, “fairseq: A fast, extensible toolkit for sequence modeling,” _arXiv:1904.01038_ , 2019. [Online]. Available: https://arxiv.org/abs/1904.01038
  * [47] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “SpecAugment: A simple data augmentation method for automatic speech recognition,” _arXiv:1904.08779_ , 2019.
  * [48] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for nlp,” in _Proc. ICML_ , 2019.
  * [49] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” _arXiv:1607.06450_ , 2016.
  * [50] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision.”
  * [51] M. Post, “A call for clarity in reporting BLEU scores,” in _Proceedings of the Third Conference on Machine Translation: Research Papers_. Belgium, Brussels: Association for Computational Linguistics, Oct. 2018, pp. 186–191. [Online]. Available: https://www.aclweb.org/anthology/W18-6319
  * [52] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, Łukasz Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corrado, M. Hughes, and J. Dean, “Google’s neural machine translation system: Bridging the gap between human and machine translation,” 2016.
  * [53] C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in _Text Summarization Branches Out_. Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, pp. 74–81. [Online]. Available: https://aclanthology.org/W04-1013
  * [54] S. Banerjee and A. Lavie, “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,” in _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_. Ann Arbor, Michigan: Association for Computational Linguistics, Jun. 2005, pp. 65–72. [Online]. Available: https://www.aclweb.org/anthology/W05-0909
  * [55] M. K. Eddine, G. Shang, A. J.-P. Tixier, and M. Vazirgiannis, “Frugalscore: Learning cheaper, lighter and faster evaluation metrics for automatic text generation,” _arXiv preprint arXiv:2110.08559_ , 2021.
  * [56] G. Doddington, “Automatic evaluation of machine translation quality using n-gram co-occurrence statistics,” in _Proceedings of the Second International Conference on Human Language Technology Research_ , ser. HLT ’02. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2002, p. 138–145.
  * [57] A. Graves, S. Fernández, F. J. Gomez, and J. Schmidhuber, “Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,” in _Proc. ICML_ , Jun. 2006.


with metastable orbits are shown in green. These fourteen KBOs are the ones
considered in support of the planet nine hypothesis in the review by Batygin
et al. (2019). The orbital elements are from the Minor Planet Database and are
given in Table I.  Figure 9: (Color online) Same as fig 8 with eight
additional KBO orbits plotted (pink curves). These are the KBOs that have been
added to the Minor Planet Database since the publication of the review by
Batygin et al. (2019). The fourteen objects plotted in fig 8 are shown in
purple (stable orbits) and green (metastable orbits).  Figure 10: Plot of the
$C(u)$ staircase (blue) for all twenty two KBOs listed in Table 1. These
objects meet the criteria used in the review article by Batygin et al. (2019)
to identify potential members of the Sedna family ($a\geq 250$ au, $q\geq 30$
au and $i<40^{\circ}$). Also shown is $K(u)$, the cumulative distribution
function for the null hypothesis (red dashed line).

Finally we can use the data in Table 1 to compute the values of $\omega_{K}$
and demonstrate that the six KBOs are clustered in the $(e_{K},\omega_{K})$
phase space and moreover the clusering is close to the approximate location of
the fixed point. To this end it is useful to first compute the orbit normal
$\hat{{\mathbf{l}}}_{K}$ in the $XYZ$ frame. When the orbit is in the
reference orientation $\hat{{\mathbf{l}}}_{K}$ is along the positive $Z$ axis.
After the orbit it brought to the appropriate orientation
$\hat{{\mathbf{l}}}_{K}$ is given by

$\hat{{\mathbf{l}}}_{K}=\left(\begin{array}[]{c}\sin\Omega\sin i\\\
-\cos\Omega\sin i\\\ \cos i\end{array}\right)$ (F8)

We see from eq (F8) that $\hat{{\mathbf{l}}}_{K}\cdot\hat{{\mathbf{Z}}}=\cos
i$. The corresponding formula for the frame used in the dynamical analysis is

$\cos i_{K}=\hat{{\mathbf{l}}}_{K}\cdot\hat{{\mathbf{n}}}_{G}.$ (F9)

Since $0\leq i_{K}\leq\pi$ it follows that $\sin i_{K}\geq 0$ and $\sin i_{K}$
can be unambiguously calculated from $\cos i_{K}$. Next we observe that
according to eq (F3)
$\hat{{\boldsymbol{\alpha}}}_{K}\cdot\hat{{\mathbf{Z}}}=\sin\omega\sin i$. It
follows that

$\sin\omega_{K}\sin
i_{K}=\hat{{\boldsymbol{\alpha}}}_{K}\cdot\hat{{\mathbf{n}}}_{G}.$ (F10)

By a similar argument we can also show

$\cos\omega_{K}\sin
i_{K}=(\hat{{\mathbf{l}}}_{K}\times\hat{{\boldsymbol{\alpha}}}_{K})\cdot\hat{{\mathbf{n}}}_{G}$
(F11)

Note that the right hand sides of eqs (F10) and (F11) can be computed from the
known forms of all three vectors in the ecliptic frame. Together eqs (F10) and
(F11) unambiguously determine $\omega_{K}$. It may be worth remarking both
$i_{K}$ and $\omega_{K}$ are fixed unambiguously once we have chosen the
reference plane to be perpendicular to $\hat{{\mathbf{n}}}_{G}$, the direction
to the center of the galaxy. This choice is dictated by the physics and is not
arbitrary. However our choice of axes in the reference plane is arbitrary and
$\Omega_{K}$ will be dependent on that choice. Hence it is not meaningful to
speak of clustering in $\Omega_{K}$ or $\varpi_{K}=\omega_{K}+\Omega_{K}$, but
it is meaningful to examine clustering in $\omega_{K}$, as we now proceed to
do.

Fig. 4 in the main body of the paper shows the clustering of the six KBOs in
the $(e_{K},\omega_{K})$ plane. The $\omega_{K}$ values were calculated using
eqs (F9), (F10) and (F11) and the $e_{K}$ values are the ones enumerated in
Table 1. These points are shown in blue. Also shown in red in the same plot
are the locations of the corresponding fixed points $(e_{C},3\pi/2)$ in the
quadrupole approximation. We see that the Sedna family of KBOs is clustered
close to the fixed points in phase space as predicted. Fig 11 in this section
shows that the clustering in phase space persists when we include all twenty
two KBOs enumerated in table I: the six with stable orbits shown in the main
body of the paper as well as the eight with metastable orbits and the eight
that have appeared in the Minor Planet Database since the publication of the
review by Batygin et al. (2019).

Figure 11: Plot of $(e_{K}^{2},\omega_{K}^{2})$ for all 22 KBOs of the Sedna
family listed in Table I (purple points for the six KBOs known to have stable
orbits, green points for the ones with metastable orbits, and pink points for
the eight added to the minor planet database since the publication of the
review by Batygin et al. (2019)). Also shown is the location of the
corresponding fixed point of the MOND orbital dynamics in the quadrupole
approximation (red points). The clustering shown in Fig 4 of the main body of
the paper for the six KBOs with known stable orbits is seen to persist for all
twenty two KBOs.

To strengthen the evidence for phase space clustering we must look to future
surveys to provide a larger observational sample of KBOs of the Sedna family.
Also numerical simulations of the dynamics will allow more precise location of
the fixed point and its domain of stability. These calculations are underway.