# Multilingual Sentence-Level Semantic Search
using Meta-Distillation Learning

Meryem M'hamdi1 , Jonathan May1, Franck Dernoncourt2,
Trung Bui2, and Seunghyun Yoon2
1Information Sciences Institute, University of Southern California
{meryem, and<EMAIL_ADDRESS>
2Adobe Research
{dernonco, bui, and<EMAIL_ADDRESS>

Work was conducted while the first author was a research scientist intern at
Adobe.

###### Abstract

Multilingual semantic search is the task of retrieving relevant contents to a
query expressed in different language combinations. This requires a better
semantic understanding of the user’s intent and its contextual meaning.
Multilingual semantic search is less explored and more challenging than its
monolingual or bilingual counterparts, due to the lack of multilingual
parallel resources for this task and the need to circumvent “language bias”.
In this work, we propose an alignment approach: MAML-Align,111We will release
our code repository in the camera-ready version. specifically for low-resource
scenarios. Our approach leverages meta-distillation learning based on MAML, an
optimization-based Model-Agnostic Meta-Learner. MAML-Align distills knowledge
from a Teacher meta-transfer model T-MAML, specialized in transferring from
monolingual to bilingual semantic search, to a Student model S-MAML, which
meta-transfers from bilingual to multilingual semantic search. To the best of
our knowledge, we are the first to extend meta-distillation to a multilingual
search application. Our empirical results show that on top of a strong
baseline based on sentence transformers, our meta-distillation approach boosts
the gains provided by MAML and significantly outperforms naive fine-tuning
methods. Furthermore, multilingual meta-distillation learning improves
generalization even to unseen languages.

##  1 Introduction

Figure 1:  A high-level diagram of our meta-distillation MAML-Align framework
for multilingual semantic search and some of its application scenarios. We use
LAReQA Roy et al. (2020) retrieval-based question answering as our benchmark,
where the task is to rank and retrieve the most relevant content. We gradually
transfer from most to least resourced variants of semantic search. We leverage
knowledge distillation to align between the teacher T-MAML Finn et al. (2017),
specialized in transferring from monolingual to bilingual, and the student
S-MAML specialized in transferring from bilingual to multilingual semantic
search. The applications can either be few-shot or zero-shot depending on the
language arrangements used in the evaluation and whether they are used at any
stage in MAML-Align.

Nowadays, the web offers a wealth of information from multiple sources and in
different languages. This makes it increasingly challenging to retrieve
reliable information efficiently and accurately. Users across the globe may
express the need to retrieve relevant content in languages different from the
language of the query or in multiple languages simultaneously. All this
burgeons the great demand for multilingual semantic search. Compared to
bilingual semantic search, often portrayed as cross-lingual information
retrieval Savoy and Braschler (2019); Grefenstette (1998), multilingual or
mixed-language semantic search is under-explored and more challenging. It
requires not only more semantic understanding but also a stronger alignment
between the languages of the query and the contents to be retrieved Roy et al.
(2020).

The new wave of multilingual semantic search focuses on reducing the need to
machine translation through transfer learning. Pre-trained multilingual
Transformer-based models such as M-BERT Devlin et al. (2019) and XLM-R Conneau
et al. (2020) have been used as off-the-shelf encoders in multilingual
semantic search. However, their performance, especially for ad-hoc semantic
search, is still lacking Litschko et al. (2022). Knowledge distillation and
contrastive-distillation learning approaches are considered as the de-facto
approaches to produce better-aligned multilingual sentence representations
with reduced need to parallel corpora Reimers and Gurevych (2020); Tan et al.
(2023). However, they still rely on medium-scaled data including monolingual
corpora and back-translation and yield mixed results. Meta-transfer learning,
another technique for low-resource learning, has been leveraged for retrieval
tasks; however, its application has been restricted to the monolingual case
Lin and Chen (2020); Laadan et al. (2019); Carvalho et al. (2008). Hybrid
approaches of meta-learning and knowledge distillation either involve using
meta-learning to improve the student-teacher feedback loop Zhou et al. (2022);
Liu et al. (2022), or leverage knowledge distillation to enhance the
portability of MAML networks Zhang et al. (2020). To the best of our
knowledge, we are the first to adapt multilingual meta-transfer learning and
to extend an approach based on meta-distillation learning to multilingual
semantic search and to a multilingual application in general.

In this paper, inspired by M’hamdi et al. (2021), which propose the X-METRA-
ADA algorithm to adapt meta-learning to cross-lingual transfer learning for
cross-lingual natural language understanding, we propose an adaptation of
meta-transfer learning to multilingual semantic search. Given the lack of
resourcefulness of semantic search especially in the multilingual case, this
encourages us to pursue a meta-learning direction based on MAML. We also
explore the combination of meta-learning and knowledge distillation and adapt
it to the task of multilingual semantic search (Figure 1). We do that in two
stages 1) from monolingual to bilingual and 2) from bilingual to multilingual
to create a more gradual feedback loop, which makes it easier to generalize to
the multilingual case. We conduct experiments on different semantic search
benchmarks on top of a strong baseline based on sentence transformers Reimers
and Gurevych (2019). Our findings confirm the benefits of the meta-
distillation approach compared to naive fine-tuning and MAML.

Our main contributions are:  (1) We are the first to propose a meta-learning
approach for multilingual semantic search (§4.4) and to curate meta-tasks for
that effect (§5.2).  (2) We are the first to propose a meta-distillation
approach to distill the transfer from monolingual to bilingual to the transfer
from bilingual to multilingual semantic search (§4.4).  (3) We systematically
compare between several few-shot transfer learning methods and show the gains
of our multilingual meta-distillation approach (§6.1).  (4) We also conduct
ablation studies involving different language arrangements and different
sampling approaches in the meta-task construction (§6.2).

##  2 Related Work

#### Transfer Learning for Multilingual Semantic Search

Most approaches to multilingual semantic search or cross-lingual information
retrieval rely on machine translation to reduce the problem to monolingual
search Lu et al. (2008); Nguyen et al. (2008); Jones et al. (2008). However,
such systems are inefficient for multilingual semantic search due to error
propagation and overheads from API calls. In addition to that, the number of
language combinations in the query and content to be retrieved can get
prohibitively large Savoy and Braschler (2019). More prominent approaches
leverage transfer learning with models like M-BERT and XLM used for question-
answer retrieval Yang et al. (2020), bitext mining Ziemski et al. (2016);
Zweigenbaum et al. (2018), and semantic textual similarity Hoogeveen et al.
(2015); Lei et al. (2016) and show that semantic specialization and pre-fine-
tuning on other auxiliary tasks helps.

#### Multilingual Meta-Learning

Meta-transfer learning, or “learning to learn” has found favor in cross-
lingual transfer learning for numerous downstream applications Gu et al.
(2018); Hsu et al. (2020); Winata et al. (2020); Chen et al. (2020); Xiao et
al. (2021). Most recent meta-learning work involving transferring between
different languages focuses on cross-lingual meta-learning Nooralahzadeh et
al. (2020); M’hamdi et al. (2021). Meta-transfer learning has been extended
multilingually by exploring joint multi-task and multi-lingual transfer
Tarunesh et al. (2021); van der Heijden et al. (2021).

#### Meta-Distillation Learning

Meta-learning has also been leveraged to improve the performance of knowledge
distillation to help the teacher transfer better to the student Zhou et al.
(2022); Liu et al. (2022). Inversely, knowledge distillation has been
leveraged to improve meta-learning, especially MAML, by making it more
portable Zhang et al. (2020). Xu et al. (2021) follow a gradual multi-stage
process which is different in scope and approach from our work in that it uses
fine-tuning for domain adaptation to interpolate between in-domain and out-
domain data. In contrast, we apply our approach to a multilingual semantic
search in an end-to-end meta-learning framework which gradually meta-transfers
between semantic search language variants. Moreover, we show that our approach
outperforms naive joint fine-tuning, advocating for a meta-learning approach
in the few-shot learning scenario.222More detailed related work can be found
in Appendix A.

##  3 Meta-Learning Background

Given a training dataset $\mathcal{D}$ made of instances:
$\\{(x_{1},y_{1}),\dotsc,(x_{n},y_{n})\\}$, the goal of a conventional machine
learning model is to find the most optimal parameters $\theta^{*}$ that
minimize the loss $\mathcal{L}$:

$\theta^{*}=\underset{\theta}{\arg\min}\mathcal{L}(\theta;\omega;\mathcal{D}),$
(1)

where $\omega$ is some already acquired prior knowledge or assumption on how
to learn Hospedales et al. (2020). There are two main distinctions between
this conventional machine-learning process and meta-learning. First, machine
learning focuses on one task at a time whereas meta-learning optimizes over a
distribution of many sub-tasks, referred to as ’meta-tasks’, sampled to
simulate a low-resource scenario. Second, meta-learning effectively learns the
prior knowledge jointly with the task by adding an extra layer of abstraction
to the process.

Each meta-task is defined as a tuple $T=(S,Q)$, where $S$ and $Q$ denote
support and query sets, respectively. $S$ and $Q$ are sampled to simulate the
train and test labeled subsets of instances. Following a bi-level optimization
abstraction (as in MAML), the meta-learning process is a sequence of inner
loops each followed by an outer loop. The inner loop is specialized in
learning task-specific optimizations over the support sets in a batch of meta-
tasks; the outer loop, on the other hand, learns the generalization over the
query sets in the same batch in a leader-follower manner. The goal is to learn
a proper initialization point to generalize to the domain of $Q$. Meta-
learning works with meta-training $\mathcal{D}_{\mbox{{meta-
train}}}=\\{\mathcal{D}^{\mbox{{train}}}_{\mbox{{support}}},\mathcal{D}^{\mbox{{train}}}_{\mbox{{query}}}\\}$,
meta-testing $\mathcal{D}_{\mbox{{meta-
test}}}=\\{\mathcal{D}^{\mbox{{test}}}_{\mbox{{support}}},\mathcal{D}^{\mbox{{test}}}_{\mbox{{query}}}\\}$,
and optionally meta-validation $\mathcal{D}_{\mbox{{meta-valid}}}$ datasets.
During meta-training, we start by learning the optimal prior knowledge
$\omega^{*}$:

$\omega^{*}=\underset{\omega}{\arg\min}\mathcal{L}(\omega|\mathcal{D}_{\mbox{{meta-
train}}}).$ (2)

This learned prior knowledge is leveraged along with the support set in the
meta-testing dataset $\mathcal{D}^{\mbox{{test}}}_{\mbox{{support}}}$ during
meta-testing to fastly adapt to $\mathcal{D}^{\mbox{{test}}}_{\mbox{{query}}}$
(without optimizing on it like in meta-training), as follows:

$\theta^{*}=\underset{\theta}{\arg\min}\mathcal{L}(\theta|\omega^{*},\mathcal{D}^{\mbox{{test}}}_{\mbox{{support}}}).$
(3)

##  4 Methodology

In this section, we start by defining the task of sentence-level semantic
search and its different categories (§4.1), its language variants (§4.2), and
supervision degrees (§4.3). Then, we present our optimization-based meta-
distillation learning algorithm MAML-Align and show how it extends from the
original MAML algorithm (§4.4).

###  4.1 Task Formulation

Our base task is sentence-level semantic search. Given a sentence query $q$
from a pool of queries $\mathscr{Q}$, the goal is to find relevant content $r$
from a pool of candidate contents $\mathscr{R}$. The queries are of sentence
length and retrieved contents are either sentences or small passages of few
sentences.

In terms of the format of the queries and contents, there are two main
categories of semantic search:  (1) Symmetric Semantic Search.Query $q$ and
relevant content $r$ have approximately the same length and format.  (2)
Asymmetric Semantic Search.$q$and $r$ are not of the same length or format.
For example, $q$ can be a question and $r$ a passage answering that.

###  4.2 Task Language Variants

In the context of languages, we distinguish between three variants of semantic
search at evaluation time (also illustrated in Figure 1):  (1) Monolingual
Semantic Search (mono).The pools of queries and candidate contents
$\mathscr{Q}$ and $\mathscr{R}$ are from the same known and fixed language
$\boldsymbol{\ell}_{\mathscr{Q}}=\boldsymbol{\ell}_{\mathscr{R}}\in\mathscr{L}$.
(2) Bilingual Semantic Search (bi).The pools of queries and candidate contents
are sampled from two different languages
$\\{\boldsymbol{\ell}_{\mathscr{Q}},\boldsymbol{\ell}_{\mathscr{R}}\\}\in\mathscr{L}^{2}$,
such that
$\boldsymbol{\ell}_{\mathscr{Q}}\neq\boldsymbol{\ell}_{\mathscr{R}}$.  (3)
Multilingual Semantic Search (multi).This is the problem of retrieving
relevant contents from a pool of candidates from a subset of multiple
languages $\mathscr{L}_{\mathscr{R}}\subseteq\mathscr{L}$ to a query expressed
in a subset of multiple languages
$\mathscr{L}_{\mathscr{Q}}\subseteq\mathscr{L}$.

Unlike monolingual and bilingual semantic search, multilingual semantic search
doesn’t restrict or condition on which languages can be used in the queries or
the candidate contents. Therefore, it is more challenging and requires
stronger multilingual alignment Roy et al. (2020).

###  4.3 Supervision Degrees

In the absence of enough training data for the task, we distinguish between
three degrees of supervision of semantic search:

  * •

Zero-Shot Learning. This resembles ad-hoc semantic search in that it doesn’t
involve any fine-tuning specific to the task of semantic search. Rather, off-
the-shelf pre-trained language models are used directly to find relevant
content to a specific query. This still uses some supervision in the form of
parallel sentences used to pre-train those off-the-shelf models. In the
context of multilingual semantic search, we include in the zero-shot learning
case any evaluation on languages not seen during fine-tuning.

  * •

Few-Shot Learning. Few-shot learning is used in the form of a small fine-
tuning dataset. In the context of multilingual semantic search, we talk about
a few-shot evaluation for any language seen either in the arrangement of the
query or the contents to be retrieved during fine-tuning.

###  4.4 Meta-Learning Models

#### Original MAML Algorithm.

Our first variant is a direct adaptation of MAML to multilingual semantic
search. We use the procedure outlined in Algorithm 1. We start by sampling a
batch of meta-tasks from a meta-dataset distribution
$\mathcal{D}{\mbox{{X$\shortrightarrow$ $X\prime$ }}}$, which simulates the
transfer from $X$ to $X\prime$. $X$ and $X\prime$ denote different task
language variants of semantic search (monolingual, bilingual, multilingual, or
any combination of that). We start by initializing our meta-learner parameters
$\theta$ with the pre-trained learner parameters $\theta_{B}$. For each meta-
batch, we perform an inner loop (Algorithm 2) over each meta-task
$T_{j}=(S_{j},Q_{j})$, separately, where we update $\theta_{j}$ using
$S_{j}^{X}$ for $n$ steps. At the end of the inner loop, we compute the
gradients with respect to the loss of $\theta_{j}$ on $Q_{j}^{X\prime}$. After
finishing a pass over all meta-tasks of the batch, we perform one outer loop
by summing over all pre-computed gradients and updating $\theta$.

Algorithm 1 MAML: Transfer Learning from X to $X\prime$ (X$\rightarrow$
$X\prime$)

1:Task set distribution $\mathcal{D}{\mbox{{X $\shortrightarrow X\prime$}}}$
simulating transfer from X to $X\prime$ task language variants, pre-trained
learner $B$ with parameters $\theta_{B}$, and meta-learner $M$ with parameters
($\theta$, $\alpha$, $\beta$, $n$).

2:Initialize $\theta\leftarrow\theta_{B}$

3:while not done do

4: Sample a batch of tasks $\mathcal{T}=\\{T_{1},\ldots
T_{b}\\}\sim\mathcal{D}{\mbox{{X$\shortrightarrow X\prime$}}}$

5:
$\mathcal{L}^{S_{j}^{X}}_{T_{j}},\mathcal{L}^{Q_{j}^{X\prime}}_{T_{j}}(B_{\theta_{j}})$=
INNER_LOOP($\mathcal{T}$, $\theta$, $\alpha$, $n$)

6: Outer Loop: Update
$\theta\leftarrow\theta-\beta\nabla_{\theta}\sum^{b}_{j=1}\mathcal{L}^{Q_{j}^{X\prime}}_{T_{j}}(B_{\theta_{j}})$

7:end while

Algorithm 2 INNER_LOOP

1:function INNER_LOOP($\mathcal{T}$, $\theta$, $\alpha$, $n$)

2: for each $T_{j}=(S_{j}^{X},Q_{j}^{X\prime})$ in $\mathcal{T}$ do

3: Initialize $\theta_{j}\leftarrow\theta$

4: for $t=1\ldots n$ do

5: Evaluate $\partial
B_{\theta_{j}}/\partial\theta_{j}=\nabla_{\theta_{j}}\mathcal{L}^{S_{j}^{X}}_{T_{j}}(B_{\theta_{j}})$

6: Update $\theta_{j}=\theta_{j}-\alpha\partial
B_{\theta_{j}}/\partial\theta_{j}$

7: end for

8: Evaluate query loss $\mathcal{L}^{Q_{j}^{X\prime}}_{T_{j}}(B_{\theta_{j}})$
and save it for outer loop

9: end for

10:end function

#### MAML-Align Algorithm.

The idea behind this extension is to use knowledge distillation to distill
T-MAML to S-MAML and improve the generalization of MAML in Algorithm 3 across
different modes of transfer. T-MAML is more high-resource than S-MAML. Given
meta-tasks from $\mathcal{D}{\mbox{{X$\shortrightarrow$Y}}}$ and
$\mathcal{D}{\mbox{{Y$\shortrightarrow$Z}}}$, the goal is to use that shared
mode of transfer $Y$ to align different modes of transfer of semantic search.
After executing the two inner loops of the two MAMLs (with more inner steps
for T-MAML than S-MAML), where the support sets are sampled from $X$ and $Y$,
respectively, we compute in the optimization process of the outer loop: the
weighted combination of $\mathcal{L}_{task}$, the average over the task-
specific losses on the query sets sampled from $Y$ and $Z$, and
$\mathcal{L}_{kd}$, the distillation loss on $Y$.

Algorithm 3 MAML-Align: Knowledge distillation to align two different MAMLs
(X$\rightarrow$Y$\rightarrow$Z)

1:Task set distribution $\mathcal{D}{\mbox{{X$\shortrightarrow$Y}}}$ and
$\mathcal{D}{\mbox{{Y$\shortrightarrow$Z}}}$ sharing the same $Y$, pre-trained
learner $B$ with parameters $\theta_{B}$, and meta-learners $M{\mbox{{X
$\shortrightarrow$ Y}}}$ with parameters ($\theta$, $\alpha$, $\beta$, $n$)
and $M{\mbox{{Y $\shortrightarrow$ Z}}}$ with parameters ($\theta\prime$,
$\alpha$, $\beta\prime$, $n\prime$), where $n\prime<n$.

2:Initialize $\theta\leftarrow\theta_{B}$

3:Initialize $\theta\prime\leftarrow\theta_{B}$

4:while not done do

5: Sample batch of tasks
$\mathcal{T_{\mbox{{X$\shortrightarrow$Y}}}}=\\{T_{1},\ldots
T_{b}\\}\sim\mathcal{D}{\mbox{{X$\shortrightarrow$Y}}}$

6: Sample batch of tasks
$\mathcal{T_{\mbox{{Y$\shortrightarrow$Z}}}}=\\{T_{1},\ldots
T_{b}\\}\sim\mathcal{D}{\mbox{{Y$\shortrightarrow$Z}}}$

7: $\mathcal{L}^{S_{j}^{X}}_{T_{j}},\mathcal{L}^{Q_{j}^{Y}}_{T_{j}}$ =
INNER_LOOP($\mathcal{T_{\mbox{{X$\shortrightarrow$Y}}}}$, $\theta$, $\alpha$,
$n$)

8: $\mathcal{L}^{S_{j}^{Y}}_{T_{j}},\mathcal{L}^{Q_{j}^{Z}}_{T_{j}}$ =
INNER_LOOP ($\mathcal{T_{\mbox{{Y$\shortrightarrow$Z}}}}$, $\theta\prime$,
$\alpha$, $n\prime$)

9:
$\mathcal{L}_{task}=\sum^{b}_{j=1}\frac{\mathcal{L}^{Q^{Y}_{j}}_{T_{j}}(B_{\theta_{j}})+\mathcal{L}^{Q^{Z}_{j}}_{T_{j}}(B_{\theta_{j}})}{2}$

10:
$\mathcal{L}_{kd}=KL(\sum^{b}_{j=1}\mathcal{L}^{Q_{j}^{Y}}_{T_{j}}(B_{\theta_{j}}),\sum^{b}_{j=1}\mathcal{L}^{S_{j}^{Y}}_{T_{j}}(B_{\theta_{j}}))$

11: Update
$\theta\leftarrow\theta-\beta\nabla_{\theta}(\mathcal{L}_{task}+\lambda\mathcal{L}_{kd})$

12:end while

Figure 2 shows a conceptual comparison between MAML-Align and MAML.

Figure 2:  A conceptual comparison between MAML-Align and the original meta-
learning baseline MAML. A single iteration of MAML involves one inner loop
optimizing over a batch of support sets from a source language variant of the
task followed up by an outer loop optimizing over the batch query sets curated
from the target task variant. In MAML-Align, on the other hand, we curate two
support sets and one query set, where the second support set is used as both a
query and support set in T-MAML and S-MAML, respectively. We perform two inner
loops and two forward passes. Then, in the outer loop, we optimize jointly
over the distillation and task-specific losses of the query sets.

##  5 Experimental Setup

In this section, we describe the downstream datasets and models (§5.1), their
formulation as meta-tasks (§5.2), and the different baselines and model
variants used in the evaluation (§5.3).

###  5.1 Downstream Benchmarks

We evaluate our proposed approaches over the following combination of
multilingual and bilingual sentence-level semantic search datasets for which
we describe the downstream models used:333More details on the base model
architectures can be found in Appendix B More experimental details on the
datasets are and hyperparameters used in Appendix C.

  * •

Asymmetric Semantic Search. We use LAReQA Roy et al. (2020), focusing on
XQuAD-R, which is a retrieval-based task reformulated from the span-based
question answering XQuAD Artetxe et al. (2020). This dataset covers 11
languages. In this work, we only use seven languages: Arabic, German, Greek,
Hindi (used for few-shot learning), Russian, Thai, and Turkish (kept for zero-
shot evaluation).444We download the data from https://github.com/google-
research-datasets/lareqa. We design a Transformer-based triplet-encoder
model(modified from the original dual encoder in Roy et al. (2020)) with three
towers encoding 1) the question, 2) its answer and its context, and 3) the
negative candidates and their contexts.

  * •

Symmetric Semantic Search. As there is no multilingual parallel benchmark for
symmetric search, we focus, in our few-shot learning experiments, on a small-
scale bilingual benchmark. We use STSBMulti from SemEval-2017 Task 1 Cer et
al. (2017).555We download SemEval-2017 evaluation and its ground truth scores
from https://alt.qcri.org/semeval2017/task1/index.php?id=data-and-tools, which
covers English-English, Arabic-Arabic, Spanish-Spanish, Arabic-English,
Spanish-English, and Turkish-English. This is a semantic similarity benchmark,
which consists of a collection of sentence pairs drawn mostly from news
headlines. Each sentence pair is scored between 1 and 5 to denote the extent
of their similarity. We use a Transformer-based dual-encoder model, which
encodes sentences 1 and 2 in each sentence pair using the same shared encoder
and computes the cosine similarity score.

###  5.2 Meta-Datasets

Following our formulation of semantic search downstream benchmarks, we
construct pseudo-meta-tasks by drawing from the available triplets or sentence
pairs to form the support set $S$, so that each support set consists of a
batch of $k\\_shot$ triplets or sentence pairs. Then, we form the triplets or
sentence pairs in the query set $Q$ by picking for each question or sentence
pair in $S$ either a similar or random question or sentence pair. Details of
the different transfer modes and their support and query set arrangements are
in Table 4 in Appendix C.2. We construct meta-datasets for different stages of
meta-learning where Train, Dev, and Test splits are used to sample
$\mathcal{D}_{\mbox{{meta-train}}}$, $\mathcal{D}_{\mbox{{meta-valid}}}$, and
$\mathcal{D}_{\mbox{{meta-test}}}$, respectively. The optimizations on
$\mathcal{D}_{\mbox{{meta-train}}}$ and $\mathcal{D}_{\mbox{{meta-test}}}$ are
as defined in §3 and the optimization on $\mathcal{D}_{\mbox{{meta-valid}}}$
is similar to that of $\mathcal{D}_{\mbox{{meta-train}}}$.

###  5.3 Baselines & Model Variants

Since we are the first, to the best of our knowledge, to explore meta-learning
for bilingual or multilingual information retrieval or semantic search, we
only compare with respect to our internal variants and include external non-
meta-learning baselines.

#### Baselines.

We design the following baselines:

  * •

BASE: This is our initial zero-shot approach based on an off-the-shelf pre-
trained language model. For the rest of our analysis, we use the best model on
our 5-fold cross-validation test splits, which is sentence-BERT (S-BERT)
paraphrase-multilingual-mpnet-base-v2, according to our preliminary evaluation
of different Sentence Transformers models.666https://huggingface.co/sentence-
transformers in Table 5 in Appendix C.

  * •

S-BERT+Fine-tune: On top of S-BERT, we fine-tune jointly and directly on the
support and query sets of each meta-task in $\mathcal{D}_{\mbox{{meta-
train}}}$ and $\mathcal{D}_{\mbox{{meta-valid}}}$. This few-shot baseline
makes for a fair comparison with the meta-learning approaches.

#### Internal Variants.

We design the following meta-learning variants:

  * •

S-BERT+MAML: On top of S-BERT, we apply MAML (following Algorithm 1). At each
episode, we conduct a meta-training followed by a meta-validation phase.

  * •

S-BERT+MAML-Align: On top of S-BERT, we apply MAML-Align (following Algorithm
3). Similarly, at each episode, we conduct a meta-training followed by a meta-
validation phase.

#### External Evaluation.

To assess the impact of using machine translation models with or without meta-
learning and the impact of machine translation from higher-resourced data, we
explore Translate-Train (T-Train), where we translate English data in
SQUADEN777We use the translate.pseudo-test provided for XQuAD dataset by
XTREME benchmark
https://console.cloud.google.com/storage/browser/xtreme_translations. and
STSBEN888We use the translated dataset from the original English STSB
https://github.com/PhilipMay/stsb-multi-mt/. to the evaluation languages. We
then either use translated data in all languages or in each language
separately as a data augmentation technique.

Model |  Train Language(s) Configuration | Test on LAReQA |  Train Language(s) Configuration | Test on STSBMulti
---|---|---|---|---
Multilingual | Bilingual | Monolingual | Bilingual | Monolingual
Zero-Shot Baselines
LaBSE | - | 48.7 $\pm$ 2.6 (6) | 73.0 $\pm$ 1.3 (6) | 77.7 $\pm$ 1.7 (6) | - | 72.3 $\pm$ 7.1 (7) | 77.0 $\pm$ 6.2 (8)
S-BERT | - |  57.0 $\pm$ 2.9 (4) |  77.5 $\pm$ 1.1 (2) |  80.7 $\pm$ 1.4 (2) | - |  80.2 $\pm$ 5.7 (3) |  82.6 $\pm$ 5.5 (4)
+Few-Shot Learning
S-BERT+Fine-tune | mono$\rightarrow$bi | 47.0 $\pm$ 4.2 (8) | 68.6 $\pm$ 2.1 (8) | 71.9 $\pm$ 2.2 (8) | mono$\rightarrow$bi | 77.1 $\pm$ 3.4 (8) |  82.8 $\pm$ 3.1 (2)
S-BERT+MAML(*) | trans | 57.2 $\pm$ 3.5 (3) | 77.1 $\pm$ 1.3 (4) | 80.0 $\pm$ 3.5 (5) | mono$\rightarrow$bi |  79.9 $\pm$ 2.9 (4) | 82.7 $\pm$ 3.2 (3)
S-BERT+MAML-Align(*) | mono$\rightarrow$bi$\rightarrow$multi |  57.6 $\pm$ 3.2 (2) |  77.4 $\pm$ 1.5 (3) |  80.6 $\pm$ 1.5 (3) | mono$\rightarrow$bi$\rightarrow$multi(**) | 79.5 $\pm$ 2.7 (5) |  85.4 $\pm$ 1.3 (1)
+Machine-Translation
S-BERT+T-Train+Fine-tune | Arabic |  47.3 $\pm$ 3.8 (7) |  69.1 $\pm$ 2.3 (7) |  72.6 $\pm$ 2.7 (7) | Turkish |  74.3 $\pm$ 6.5 (9) |  81.4 $\pm$ 7.1 (5)
Average over test languages | 46.1 $\pm$ 4.4 (9) | 67.7 $\pm$ 2.7 (9) | 71.1 $\pm$ 3.0 (9) | Average over test languages | 69.0 $\pm$ 10.7 (10) | 78.4 $\pm$ 9.1 (6)
Jointly over test languages | 45.0 $\pm$ 3.4 (10) | 66.1 $\pm$ 1.7 (10) | 70.6 $\pm$ 2.1 (10) | Jointly over test languages | 68.9 $\pm$ 8.3 (11) | 77.1 $\pm$ 10.1 (7)
S-BERT+T-Train+MAML(*) | Arabic |  58.0 $\pm$ 3.2 (1) |  78.1 $\pm$ 1.4 (1) |  81.1 $\pm$ 1.5 (1) | Turkish |  80.4 $\pm$ 5.7 (2) |  82.8 $\pm$ 5.5 (2)
Average over test languages | 57.0 $\pm$ 3.7 (4) | 77.4 $\pm$ 2.1 (3) | 80.4 $\pm$ 2.1 (4) | Average over test languages | 79.1 $\pm$ 5.8 (6) | 82.7 $\pm$ 6.2 (3)
Jointly over test languages | 56.4 $\pm$ 3.7 (5) | 77.0 $\pm$ 1.6 (5) | 80.0 $\pm$ 1.5 (5) | Jointly over test languages |  80.5 $\pm$ 5.7 (1) | 82.6 $\pm$ 5.6 (4)

Table 1:  This is a comparison of different few-shot learning, zero-shot
baselines, and machine translation models under a variety of language
configuration scenarios. For LAReQA and STSBMulti, we report mAP@20 and
Pearson’s r × 100, respectively. All results are evaluated over 5-fold cross-
validation and averaged over multiple language choices. The same model
checkpoint is used for all three task language evaluation variants for each
row and dataset (except when the average is reported). mono, bi, and
multistand for monolingual, bilingual, and multilingual semantic search. trans
denotes the meta-transfer mode that uses mono$\rightarrow$bi and
bi$\rightarrow$multi in meta-training and meta-validation, respectively.
Models in (*) are our main contribution. (**) means that we use machine-
translated data to do that experiment as STSBMulti is not a parallel corpus.
Best and second-best results for each benchmark and evaluation mode are
highlighted in bold and italicized respectively, whereas the best results
across each model category are underlined. Ranks from best to worst are given
in each model and evaluation mode.9

##  6 Results & Analysis

In this section, we present the results obtained using different meta-learning
model variants compared to the baselines in multilingual, bilingual, and
monolingual task language variants. All experiments are evaluated using 5-fold
cross-validation and then the mean and standard deviation are reported.
Following XTREME-R Ruder et al. (2021) and SemEval-2017 Cer et al. (2017),
scores are reported using mean average precision at 20 (mAP@20) and Pearson
correlation coefficient percentage (Pearson’s r × 100) for LAReQA and
STSBMulti, respectively.

###  6.1 Multilingual, Bilingual, and Monolingual Performance Evaluation

Table 1 summarizes multilingual, bilingual, and monolingual performances
across different baselines and model variants for both semantic search
benchmarks. On average, we notice that MAML-Align achieves better results than
MAML or S-BERT zero-shot base model and significantly better than Fine-tune.
It is worth noting that we report the results for MAML using trans mode, which
is trained over a combination of mono$\rightarrow$bi and bi$\rightarrow$multi
in the meta-training and meta-validation stages, respectively. This suggests
that MAML-Align helps more in bridging the gap between those transfer modes.
We observe that fine-tuning baselines are consistently weak compared to
different meta-learning model variants, especially for LAReQA. We conjecture
that fine-tuning is over-fitting to the small amounts of training data, unlike
meta-learning approaches which are more robust against that. However, for
STSBMulti, the gap between fine-tuning and meta-learning while still existing
and to the favor of meta-learning is a bit reduced. We hypothesize that even
meta-learning models are suffering from meta-overfitting to some degree in
this case for STSBMulti.

(a) Few-shot multilingual evaluation on Arabic in LAReQA.

(b) Zero-shot multilingual evaluation on Russian in LAReQA.

(c) Few-shot evaluation on Arabic-Arabic in STSBMulti.

(d) Few-shot evaluation on Turkish-English in STSBMulti.

Figure 3: mAP@20 and Pearson’s r × 100 5-fold cross-validated multilingual
performance evaluation evaluated on LAReQA and STSBMulti on the first and last
two subplots, respectively. The first two subplots show the performance
evaluation on Arabic and Russian used in few-shot and zero-shot evaluations,
respectively, whereas the two subplots in the second-row showcase monolingual
and bilingual performances on Arabic-Arabic and Turkish-English where Arabic,
Turkish, and English are all covered in few-shot learning. There are
consistent gains in favor of meta-learning and meta-distillation learning
compared to their fine-tuning counterparts on top of off-the-shelf model
(S-BERT only) for all types of evaluations.

We notice that MAML on top of machine-translated data boosts the performance
on LAReQA in all evaluation task language evaluation variants and reaches the
best compromise in terms of multilingual, bilingual, and monolingual
performances. At the same time, not all languages used in the machine-
translated data provide an equal boost to the performance, as shown by the
average performance, due to noisy translations for certain languages. Although
there is usually a correlation between different models in terms of their
monolingual, bilingual, and multilingual performances, there is a slight drop
in the monolingual and bilingual performances for MAML-Align compared to the
zero-shot baseline. This means that there is still a compromise and gaps
between multilingual, monolingual, and bilingual performances. This suggests
that we should advocate for a balanced evaluation over different modes to get
better insights into which models are more robust and consistent. Figure 3
highlights a more fine-grained comparison between different model categories
on two languages and language pairs for each benchmark.999More fine-grained
results for all languages and for both benchmarks can be found in Tables 7 and
8 in Appendix D. We notice that the gain in favor of meta-learning approaches
is consistent across different languages and language pairs and also applies
to languages used for zero-shot learning.

###  6.2 Ablation Studies

Due to the lack of parallelism in STSBMulti making a multilingual evaluation
not possible, we focus hereafter on LAReQA in the remaining analysis and
ablation studies. Figure 4 shows the results across different modes of
transfer for Fine-tune and MAML. Among all transfer modes, trans,
mono$\rightarrow$bi, and mono$\rightarrow$mono have the best gains, whereas
bi$\rightarrow$multi and mixt are the weakest forms of transfer. trans is the
best meta-transfer mode, especially for MAML and this suggests that curating
different transfer modes for different meta-learning processes is beneficial
and leads to better generalization than fine-tuning on them jointly. mixt is
weaker than trans and this implies that jointly optimizing different forms of
transfers of meta-tasks makes it harder for MAML to learn to converge or
generalize. MAML-Align is shown to be better for combining different
optimization objectives.

Figure 4:  mAP@20 multilingual 5-fold cross-validated performance on LAReQA
between different meta-transfer modes for Fine-tune and MAML models. The gap
is large between Fine-tune and MAML across all meta-transfer modes and is even
larger to the favor of MAML when trans mode (the composed mode that mixes
between mono$\rightarrow$bi and bi$\rightarrow$multi in the meta-training and
meta-validation, respectively) is used.

Figure 5 shows a multilingual performance comparison between different
sampling modes in meta-tasks constructions. In each meta-task, we either
sample the query set that is the most similar to its corresponding support set
(Similar) or randomly (Random). We hypothesize that the sampling approach
plays a role in stabilizing the convergence and generalization of meta-
learning. While we were expecting that sampling for each support set a query
set that is the most similar to it would help meta-learning converge faster
and thus generalize better, it generalized worse on the multilingual
performance in this case. On the other hand, random sampling generalizes
better to out-of-sample test distributions leading to lower biases between
languages in the multilingual evaluation mode.

Figure 5:  mAP@20 multilingual 5-fold cross-validated performance on LAReQA
between different query set sampling modes in meta-tasks for MAML and MAML-
Align. We notice that random query sampling has better generalization for both
models.

Figure 6 shows the results for different sampling modes of negative examples
in the triplet loss. For each support and query set in each meta-task, we
either sample random, hard, or semi-hard triplets to test the added value of
triplet sampling in few-shot learning. While we expect training with more hard
triplets to help converge the triplet loss in MAML, the multilingual
performance using this type of sampling falls short of random sampling. This
is due to the fact that more sophisticated ways of triplet loss sampling
usually require a more careful hyperparameter tuning to pick the right amount
of triplets. For few-shot learning applications, this usually results in a
significant reduction in the number of training examples, which could further
hurt the generalization performance. In future work, we plan to investigate
hybrid sampling approaches to monitor at which point in meta-learning the
training should focus more on hard or easy triplets. This could be done by
proposing a regime for making the sampling of meta-tasks dynamic and flexible
to also combat meta-over-fitting.

Figure 6:  mAP@20 5-fold cross-validated multilingual performance over
different triplet negative sampling modes on LAReQA tested on different
languages using MAML-Align. We provide both average numbers and standard
deviation intervals. Random sampling seems best on average for few-shot
learning, whereas hard sampling is more stable across cross-validation splits.

##  7 Conclusion

In this work, we adapt multilingual meta-transfer learning combining MAML and
knowledge distillation to multilingual semantic search. Our experiments show
that our multilingual meta-knowledge distillation approach outperforms both
vanilla MAML and fine-tuning approaches on top of a strong sentence
transformers model. We evaluate comprehensively on two types of multilingual
semantic search and show improvement over sentence transformers even for
languages not covered during meta-learning.

## Limitations

Due to the lack of time and resources, exploring different combinations of
languages in the construction of the query and the content to be retrieved is
not feasible. On top of that, performing extensive hyperparameters search for
different model variants, modes of transfer, language combinations, etc is not
feasible. We follow a consistent configuration of the hyperparameters for each
of the two downstream tasks which we deem to be a fair comparison across all
setups, model variants. The insights from this study are tied to the
experimental setup that we describe extensively in the main paper and
appendix. We also have memory constraints when it comes to training meta-
learning algorithms to deal with ranking and retrieval of sentences from
multiple languages at the same time for one query. Our memory constraints make
it challenging to explore more sophisticated state-of-the-art Sentence
Transformers such as sentence-T5 or GPT Sentence Embeddings SGPT Ni et al.
(2022); Muennighoff (2022). Applying MAML as an upstream model on top of
T5-based downstream model makes it even more computationally infeasible. Our
main goal is to show the advantage of meta-learning and since our upstream
approach is model-agnostic that can be continuously adapted to novel embedding
approaches as they evolve. There is also a shortage of large-scale
multilingual semantic search datasets, especially for the symmetric case and
especially at the phrase level. This makes our evaluation a bit restricted to
the bilingual and monolingual for symmetric semantic search. In future work,
we plan to construct and annotate semantic search for ambiguous short queries
aligned at the multilingual case.

## References

  * Arnold et al. (2020) Sébastien M. R. Arnold, Praateek Mahajan, Debajyoti Datta, Ian Bunner, and Konstantinos Saitas Zarkias. 2020.  learn2learn: A library for meta-learning research.
  * Artetxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020.  On the cross-lingual transferability of monolingual representations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4623–4637, Online. Association for Computational Linguistics.
  * Carvalho et al. (2008) Vitor R Carvalho, Jonathan L Elsas, William W Cohen, and Jaime G Carbonell. 2008\.  A meta-learning approach for robust rank learning.  In _SIGIR 2008 workshop on learning to rank for information retrieval_ , volume 1.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017\.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 1–14, Vancouver, Canada. Association for Computational Linguistics.
  * Chen et al. (2020) Yi-Chen Chen, Jui-Yang Hsu, Cheng-Kuang Lee, and Hung-yi Lee. 2020.  DARTS-ASR: differentiable architecture search for multilingual speech recognition and adaptation.  In _Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020_ , pages 1803–1807. ISCA.
  * Clark et al. (2020) Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020.  TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages.  _Transactions of the Association for Computational Linguistics_ , 8:454–470.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 1126–1135. PMLR.
  * Grefenstette (1998) Gregory Grefenstette. 1998.  Cross language information retrieval.  In _Proceedings of the Third Conference of the Association for Machine Translation in the Americas: Tutorial Descriptions_ , Langhorne, PA, USA. Springer.
  * Gu et al. (2018) Jiatao Gu, Yong Wang, Yun Chen, Victor O. K. Li, and Kyunghyun Cho. 2018.  Meta-learning for low-resource neural machine translation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3622–3631, Brussels, Belgium. Association for Computational Linguistics.
  * Hoogeveen et al. (2015) Doris Hoogeveen, Karin M. Verspoor, and Timothy Baldwin. 2015.  Cqadupstack: A benchmark data set for community question-answering research.  In _Proceedings of the 20th Australasian Document Computing Symposium, ADCS 2015, Parramatta, NSW, Australia, December 8-9, 2015_ , pages 3:1–3:8. ACM.
  * Hospedales et al. (2020) Timothy M. Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J. Storkey. 2020\.  Meta-learning in neural networks: A survey.  _CoRR_ , abs/2004.05439.
  * Hsu et al. (2020) Jui-Yang Hsu, Yuan-Jui Chen, and Hung-yi Lee. 2020.  Meta learning for end-to-end low-resource speech recognition.  In _2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020_ , pages 7844–7848. IEEE.
  * Jones et al. (2008) Gareth Jones, Fabio Fantino, Eamonn Newman, and Ying Zhang. 2008.  Domain-specific query translation for multilingual information access using machine translation augmented with dictionaries mined from Wikipedia.  In _Proceedings of the 2nd workshop on Cross Lingual Information Access (CLIA) Addressing the Information Need of Multilingual Societies_.
  * Laadan et al. (2019) Doron Laadan, Roman Vainshtein, Yarden Curiel, Gilad Katz, and Lior Rokach. 2019\.  Rankml: a meta learning-based approach for pre-ranking machine learning pipelines.  _ArXiv preprint_ , abs/1911.00108.
  * Langedijk et al. (2022) Anna Langedijk, Verna Dankers, Phillip Lippe, Sander Bos, Bryan Cardenas Guevara, Helen Yannakoudakis, and Ekaterina Shutova. 2022.  Meta-learning for fast cross-lingual adaptation in dependency parsing.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8503–8520, Dublin, Ireland. Association for Computational Linguistics.
  * Lee et al. (2022) Hung-yi Lee, Shang-Wen Li, and Thang Vu. 2022.  Meta learning for natural language processing: A survey.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 666–684, Seattle, United States. Association for Computational Linguistics.
  * Lei et al. (2016) Tao Lei, Hrishikesh Joshi, Regina Barzilay, Tommi Jaakkola, Kateryna Tymoshenko, Alessandro Moschitti, and Lluís Màrquez. 2016.  Semi-supervised question retrieval with gated convolutions.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1279–1289, San Diego, California. Association for Computational Linguistics.
  * Lewis et al. (2020) Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020\.  MLQA: Evaluating cross-lingual extractive question answering.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7315–7330, Online. Association for Computational Linguistics.
  * Lin and Chen (2020) Chong-En Lin and Kuan-Yu Chen. 2020.  A preliminary study on using meta-learning technique for information retrieval.  In _Proceedings of the 32nd Conference on Computational Linguistics and Speech Processing (ROCLING 2020)_ , pages 59–71, Taipei, Taiwan. The Association for Computational Linguistics and Chinese Language Processing (ACLCLP).
  * Litschko et al. (2022) Robert Litschko, Ivan Vulic, Simone Paolo Ponzetto, and Goran Glavas. 2022.  On cross-lingual retrieval with multilingual text encoders.  _Inf. Retr. J._ , 25(2):149–183.
  * Liu et al. (2022) Jihao Liu, Boxiao Liu, Hongsheng Li, and Yu Liu. 2022.  Meta knowledge distillation.  _ArXiv preprint_ , abs/2202.07940.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Lu et al. (2008) Chengye Lu, Yue Xu, and Shlomo Geva. 2008.  Web-based query translation for English-Chinese CLIR.  In _International Journal of Computational Linguistics & Chinese Language Processing, Volume 13, Number 1, March 2008: Special Issue on Cross-Lingual Information Retrieval and Question Answering_, pages 61–90.
  * M’hamdi et al. (2021) Meryem M’hamdi, Doo Soon Kim, Franck Dernoncourt, Trung Bui, Xiang Ren, and Jonathan May. 2021.  X-METRA-ADA: Cross-lingual meta-transfer learning adaptation to natural language understanding and question answering.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3617–3632, Online. Association for Computational Linguistics.
  * Muennighoff (2022) Niklas Muennighoff. 2022.  SGPT: GPT sentence embeddings for semantic search.  _ArXiv preprint_ , abs/2202.08904.
  * Nayak (2019) Pandu Nayak. 2019.  Understanding searches better than ever before.
  * Nguyen et al. (2008) Dong Nguyen, Arnold Overwijk, Claudia Hauff, Dolf Trieschnigg, Djoerd Hiemstra, and Franciska de Jong. 2008.  Wikitranslate: Query translation for cross-lingual information retrieval using only wikipedia.  In _Evaluating Systems for Multilingual and Multimodal Information Access, 9th Workshop of the Cross-Language Evaluation Forum, CLEF 2008, Aarhus, Denmark, September 17-19, 2008, Revised Selected Papers_ , volume 5706 of _Lecture Notes in Computer Science_ , pages 58–65. Springer.
  * Ni et al. (2022) Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. 2022.  Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1864–1874, Dublin, Ireland. Association for Computational Linguistics.
  * Nooralahzadeh et al. (2020) Farhad Nooralahzadeh, Giannis Bekoulis, Johannes Bjerva, and Isabelle Augenstein. 2020.  Zero-shot cross-lingual transfer with meta learning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4547–4562, Online. Association for Computational Linguistics.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Reimers and Gurevych (2020) Nils Reimers and Iryna Gurevych. 2020.  Making monolingual sentence embeddings multilingual using knowledge distillation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4512–4525, Online. Association for Computational Linguistics.
  * Robertson and Zaragoza (2009) Stephen E. Robertson and Hugo Zaragoza. 2009.  The probabilistic relevance framework: BM25 and beyond.  _Found. Trends Inf. Retr._ , 3(4):333–389.
  * Roy et al. (2020) Uma Roy, Noah Constant, Rami Al-Rfou, Aditya Barua, Aaron Phillips, and Yinfei Yang. 2020.  LAReQA: Language-agnostic answer retrieval from a multilingual pool.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5919–5930, Online. Association for Computational Linguistics.
  * Ruder et al. (2021) Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021\.  XTREME-R: Towards more challenging and nuanced multilingual evaluation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10215–10245, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Savoy and Braschler (2019) Jacques Savoy and Martin Braschler. 2019.  _Lessons Learnt from Experiments on the Ad Hoc Multilingual Test Collections at CLEF_, pages 177–200. Springer International Publishing, Cham.
  * Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015.  Facenet: A unified embedding for face recognition and clustering.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015_ , pages 815–823. IEEE Computer Society.
  * Schuster et al. (2019) Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2019.  Cross-lingual transfer learning for multilingual task oriented dialog.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3795–3805, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Tan et al. (2023) Weiting Tan, Kevin Heffernan, Holger Schwenk, and Philipp Koehn. 2023.  Multilingual representation distillation with contrastive learning.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 1477–1490, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Tarunesh et al. (2021) Ishan Tarunesh, Sushil Khyalia, Vishwajeet Kumar, Ganesh Ramakrishnan, and Preethi Jyothi. 2021.  Meta-learning for effective multi-task and multilingual modelling.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 3600–3612, Online. Association for Computational Linguistics.
  * van der Heijden et al. (2021) Niels van der Heijden, Helen Yannakoudakis, Pushkar Mishra, and Ekaterina Shutova. 2021.  Multilingual and cross-lingual document classification: A meta-learning approach.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1966–1976, Online. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Winata et al. (2020) Genta Indra Winata, Samuel Cahyawijaya, Zhaojiang Lin, Zihan Liu, Peng Xu, and Pascale Fung. 2020.  Meta-transfer learning for code-switched speech recognition.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3770–3776, Online. Association for Computational Linguistics.
  * Xiao et al. (2021) Yubei Xiao, Ke Gong, Pan Zhou, Guolin Zheng, Xiaodan Liang, and Liang Lin. 2021\.  Adversarial meta sampling for multilingual low-resource speech recognition.  In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_ , pages 14112–14120. AAAI Press.
  * Xu et al. (2021) Haoran Xu, Seth Ebner, Mahsa Yarmohammadi, Aaron Steven White, Benjamin Van Durme, and Kenton Murray. 2021.  Gradual fine-tuning for low-resource domain adaptation.  In _Proceedings of the Second Workshop on Domain Adaptation for NLP_ , pages 214–221, Kyiv, Ukraine. Association for Computational Linguistics.
  * Yang et al. (2020) Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2020.  Multilingual universal sentence encoder for semantic retrieval.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 87–94, Online. Association for Computational Linguistics.
  * Zhang et al. (2020) Min Zhang, Donglin Wang, and Sibo Gai. 2020.  Knowledge distillation for model-agnostic meta-learning.  In _ECAI 2020 - 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago de Compostela, Spain, August 29 - September 8, 2020 - Including 10th Conference on Prestigious Applications of Artificial Intelligence (PAIS 2020)_ , volume 325 of _Frontiers in Artificial Intelligence and Applications_ , pages 1355–1362. IOS Press.
  * Zhou et al. (2022) Wangchunshu Zhou, Canwen Xu, and Julian McAuley. 2022.  BERT learns to teach: Knowledge distillation with meta learning.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7037–7049, Dublin, Ireland. Association for Computational Linguistics.
  * Zhu et al. (2021) Jeffrey Zhu, Mingqin Li, Jason Li, and Cassandra Odoula. 2021.  Bing delivers more contextualized search using quantized transformer inference on nvidia gpus in azure.
  * Ziemski et al. (2016) Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016.  The United Nations parallel corpus v1.0.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 3530–3534, Portorož, Slovenia. European Language Resources Association (ELRA).
  * Zweigenbaum et al. (2018) Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2018.  Overview of the third BUCC shared task: Spotting parallel sentences in comparable corpora.  In _11th Workshop on Building and Using Comparable Corpora - Special Topic: Comparable Corpora for Asian Languages, BUCC@LREC 2018, Miyazaki, Japan, May 8, 2018_. European Language Resources Association.

##  Appendix A More Related Work

Given the scarcity of research on multilingual semantic search using meta-
learning and knowledge distillation, we analyze independently previous work in
the area of semantic search in general, multilingual semantic search, cross-
lingual meta-transfer learning, and meta-distillation learning before delving
into some applications of meta-transfer learning for retrieval ranking and how
our work applies meta-transfer and meta-distillation for multilingual semantic
search as a whole.

#### Textual Semantic Search

Textual semantic search is the task of retrieving semantically relevant
content for a given query. Unlike traditional keyword-matching information
retrieval, semantic search seeks to improve search accuracy by understanding
the searcher’s intent and disambiguating the contextual meaning of the terms
in the query Muennighoff (2022). Semantic search has broad applications in
search engines such as Google Nayak (2019), Bing Zhu et al. (2021), etc. They
rely on Transformers Vaswani et al. (2017) as their dominant architecture
going beyond non-semantic models such as BM25 Robertson and Zaragoza (2009).

#### Multilingual Semantic Search

Previous work which extends semantic search to different languages is often
focused on cross-lingual information retrieval. Progress in cross-lingual
information retrieval (CLIR) or semantic search has seen multiple waves
Grefenstette (1998). Traditionally, when we think of CLIR we automatically
think of machine translation (MT) as if they are two faces to the same coin.
The only difference is that translation tools are used to render documents
readable in the case of MT whereas CLIR focuses on rendering them searchable
if at the very core translation technology is what is used for CLIR and MT
rather than other paradigms such as transfer learning. Most approaches that
fall into this category translate queries into the language of the documents
and then perform monolingual search Lu et al. (2008); Nguyen et al. (2008);
Jones et al. (2008). While this is an efficient option, that might not be the
most effective approach as queries can be so short and ungrammatical making
them hard to translate accurately. So, in this case, translating all documents
or sentences to the target languages can be used leading to better accuracy
but less efficiency. This translation form is even more inefficient in the
case of multilingual semantic search where the number of possible language
combinations that can be used in the source and target languages can grow
exponentially. Those pipeline approaches suffer from error propagation of the
machine translation component into the downstream semantic search, especially
for low-resource languages.

More prominent approaches include transfer learning where both query and
documents or sentences are encoded into a shared space. The first class of
approaches in this category use pre-trained language models where both the
query and the documents are encoded into a shared space. The cross-lingual
ability of models like M-BERT and XLM has been analyzed for different
retrieval-based downstream applications including question-answer retrieval
Yang et al. (2020), bitext mining Ziemski et al. (2016); Zweigenbaum et al.
(2018), and semantic textual similarity Hoogeveen et al. (2015); Lei et al.
(2016). Litschko et al. (2022) systematic empirical study focused on the
suitability of SOTA multilingual encoders for cross-lingual document and
sentence retrieval tasks across a number of diverse language pairs. They
benchmark the performance in unsupervised ad-hoc (setup with no relevance
judgments for IR-specific fine-tuning) and supervised sentence and document-
level CLIR. In other words, they profile the suitability of SOTA pre-trained
multilingual encoders for different CLIR tasks and diverse language pairs
across unsupervised, supervised and transfer setups. They also propose
localized relevance matching for document-level CLIR (independently score a
query against document). For unsupervised document-level CLIR, they show that
pre-trained multilingual encoders on average fail to significantly outperform
earlier models based on CLWEs. They also show that the performance of those
multilingual encoders crucially depends on how one encodes semantic
information with the models (treating them as sentence/document encoders
directly versus averaging over constituent words and/or subwords).
Multilingual sentence encoders fine-tuned on labeled data from sentence pair
tasks like natural language inference or semantic text similarity as well as
using parallel sentences on the other hand are shown to substantially
outperform general-purpose models in sentence-level CLIR. The second class
focuses on training training models with information retrieval objectives but
it is not clear how they generalize to new languages. In our work, we
investigate ways to further improve the transfer of these off-purpose
sentences on top of semantic specialization in a data-efficient manner.

#### Multilingual Meta-Transfer Learning

Meta-learning has gained the attention of the NLP community recently with
applications in cross-domain, cross-problem, and cross-lingual transfer
learning Lee et al. (2022). Meta-learning has been leveraged for semantic
search-related tasks but only monolingually. Lin and Chen (2020) is the first
work of its kind to device a meta-learning algorithm for information retrieval
tasks. They leverage model-agnostic meta-learner (MAML) to learn an
initialization of model parameters for the re-ranker of documents by
reformulating the problem as a N-way K-Shot setup where query is a category
and the document corresponding to it as a positive example and four documents
not related to the query. They show that their approach improves over
baselines involving vanilla DSSM and Vector Space Models. They also show that
fine-tuning in addition to meta-learning lead to more gains. However, they use
meta-learning just at the level of the ranker and not for other components
like searcher in which they only use traditional approaches like Match 25 to
calculate the relationship between query documents and retrieval documents. It
is not clear whether meta-learning can be used more in an end-to-end fashion
or to improve other components. Other meta-learning work which focus on the
re-ranking component include Laadan et al. (2019); Carvalho et al. (2008) but
they all follow a pipelined approach.

Since there is no prior work leveraging meta-learning for cross-lingual or
multilingual semantic search, to the best of our knowledge, we describe in
this section. The first work of its kind using meta-learning for cross-lingual
transfer learning is Gu et al. (2018), which is applied to neural machine
translation. They extend MAMLFinn et al. (2017) to transfer from multilingual
high-resource language tasks to to low-resource languages. They show the
competitive advantages of cross-lingual meta-transfer learning compared to
other multilingual baselines. Other applications include speech recognition
Hsu et al. (2020); Winata et al. (2020); Chen et al. (2020); Xiao et al.
(2021), Natural Language Inference(XNLI) Conneau et al. (2018) and
Multilingual Question Answering(MLQA) Lewis et al. (2020) using X-MAML
Nooralahzadeh et al. (2020), task-oriented dialog Schuster et al. (2019) and
TyDiQA Clark et al. (2020) using X-METRA-ADA M’hamdi et al. (2021), dependency
parsing Langedijk et al. (2022).

Most recent work adapting meta-learning to applications involving different
languages focus on cross-lingual meta-learning. Multilingual meta-learning
differs from cross-lingual meta-transfer learning in its support for multiple
languages jointly. M’hamdi et al., for example, propose X-METRA-ADA which
performs few-shot learning on one single target language at a time and also
enable zero-shot learning on target languages not seen during meta-training or
meta-adaptation. Their approach shows gains compared to naive fine-tuning in
the few-shot more than the zero-shot learning scenario. Tarunesh et al. (2021)
propose a meta-learning framework for both multi-task and multi-lingual
transfer leveraging heuristic sampling approaches. They show that a joint
approach to multi-task and multilingual learning using meta-learning enables
effective sharing of parameters across multiple tasks and multiple languages
thus benefits deeper semantic analysis tasks such as QA, PAWS, NLI, etc. van
der Heijden et al. (2021) propose a meta-learning framework and show its
effectiveness in both the cross-lingual and multilingual training adaptation
settings of document classification. However, their multilingual evaluation is
focused on the scenario where the same target languages during meta-testing
can be also used as auxiliary languages during meta-training. This motivates
us to investigate in this paper more in the direction of multilingual meta-
transfer learning, where we test the generalizability of our meta-learning
model when it is learned by taking into consideration multiple languages
jointly for semantic search.

#### Meta-Distillation Learning

Previous works at the intersection of meta-learning and knowledge distillation
either use meta-learning as a more effective alternative to the more
traditional knowledge distillation methods. Recently, more work has started
adopting a meta-learning approach to knowledge distillation by consolidating a
feedback loop between the teacher and the student networks where the teacher
can learn to better transfer knowledge to the student network Zhou et al.
(2022) or by meta-learning the distillation hyperparameter tuning Liu et al.
(2022). Knowledge distillation has also been leveraged to enhance the
portability of MAML networks Zhang et al. (2020). It has been shown that a
portable MAML with a smaller capacity can further boost few-shot learning
better than vanilla MAML. To the best of our knowledge, we are the first to
explore knowledge distillation to bridge the gap between different cross-
lingual meta-transfer learning models and to enhance the alignment between
them.

##  Appendix B More Details on Base Models

For asymmetric semantic search, we use a Transformer-based triplet-encoder
model. In the original paper on the asymmetric benchmark we evaluate on Roy et
al. (2020), a dual-encoder model is trained using contrastive loss in the form
of an in-batch sampled softmax loss. This format reuses for each question
answers from other questions in the same batch (batched randomly) as negative
examples. Instead, we use triplet loss Schroff et al. (2015), which was also
shown to outperform contrastive loss in general. Triplet loss is shown to
surpass contrastive loss in general.101010As posited in
https://shorturl.at/ktvx9. Its strength derives not just from the nature of
its function but also from its sampling procedure. This sampling procedure
which merely requires positive instances to be closer to negative instances
doesn’t require gathering as many positive examples as contrastive loss
requires. This makes triplet loss more practical in our few-shot learning
multilingual/cross-lingual scenario, as it provides more freedom in terms of
constructing negative candidates to tweak different sampling techniques from
different languages. We thus define a _triplet encoder model_ (shown in Figure
7) with three towers encoding the question, its answer combined with its
context, and the negative candidates and their contexts. While those towers
are encoded separately, they still share the same Transformer encoder model
which is initialized with pre-trained Sentence Transformers. On top of that,
two dot products $d(q,p)$ and $d(q,n)$ are computed. $d(q,p)$ is the dot
product between the question $q$ and its answer $p$, whereas $d(q,n)$ is
between $q$ and its non-answer candidate. Triplet loss is computed as :
$\mathcal{L}=\max{(d(q,p)-d(q,n)+margin,0)}$ where $margin$ is a tun-able
hyperparameter to eventually make each triplet an easy one by pushing the
distance $d(a,p)$ closer to 0 and $d(a,n)$ to $d(a,p)+margin$.

Triplets $(q,p,n)$ can be sampled with different levels of difficulty, as
follows:

  * •

Easy triplets: $d(q,p)+margin<d(q,n)$.

  * •

Hard triplets: $d(q,n)<d(q,p)$.

  * •

Semi-hard triplets: $d(q,p)<d(q,n)<d(q,p)+margin$.

Figure 7:  Architecture of Transformer-based triplet encoder for asymmetric
semantic search. Figure 8:  Architecture of Transformer-based dual-encoder for
symmetric semantic search.

For symmetric search, we use a Transformer-based dual-encoder model (shown in
Figure 8), which encodes sentence 1 and sentence 2 in each sentence pair
separately using the same shared encoder. Then, the cosine similarity score is
computed for each sentence pair and the mean squared error (squared L2 norm)
is computed between that and the golden score. This is not a retrieval-based
task, but a semantic similarity task.

##  Appendix C More Experimental Setup Details

###  C.1 Downstream Datasets

Tables 2 and 3 show a summary of the statistics of LAReQA and STSBMulti per
language and split, respectively. XQuAD-R in LAReQA has been distributed under
the CC BY-SA 4.0 license, whereas STSBMulti has been released under the
Creative Commons Attribution-ShareAlike 4.0 International License. The
translated datasets from SQUADEN and STSBEN are shared under the same license
as the original datasets. SQUADEN is shared under XTREME benchmark Apache
License Version 2.0. STSBEN scores are under Creative Commons Attribution-
ShareAlike 3.0 Unported (CC BY-SA 3.0) and sentence pairs are shared under
Commons Attribution - Share Alike 4.0 International License).

Language | ISO | Train | Dev | Test
---|---|---|---|---
#Q | #C | #Q | #C | #Q | #C
Arabic | AR | 696 | 783 | 220 | 255 | 274 | 184
German | DE | 696 | 812 | 220 | 256 | 274 | 208
Greek | EL | 696 | 788 | 220 | 254 | 274 | 192
Hindi | HI | 696 | 808 | 220 | 252 | 274 | 184
Russian | RU | 696 | 774 | 220 | 262 | 274 | 183
Thai | TH | 696 | 528 | 220 | 178 | 274 | 146
Turkish | TR | 696 | 732 | 220 | 248 | 274 | 187

Table 2: Statistics of LAReQA in each 5-fold cross-validation split. #Q
denotes the number of question whereas #C denotes the number of candidates.

Language Pair | ISO | # Sentence Pairs
---|---|---
Train | Dev | Test
English-English | EN-EN | 150 | 50 | 50
Spanish-Spanish | ES-ES | 150 | 50 | 50
Spanish-English | ES-EN | 150 | 50 | 50
Arabic-Arabic | AR-AR | 150 | 50 | 50
Arabic-English | AR-EN | 150 | 50 | 50
Turkish-English* | TR-EN | 150 | 50 | 50

Table 3:  Statistics of the STSBMulti from SEM-Eval2007 in each 5-fold cross-
validation split. * means that for Turkish-English, there are only 250 ground
truth similarity scores, while there are 500 sentence pairs. We assume that
the ground truth scores are only for the first 250 sentence pairs. In addition
to that, we use 5749 train, 1500 dev, and 1379 test splits from the STSB
original English benchmark.

###  C.2 Upstream Meta-Tasks

We detail in Table 4 the arrangements of languages for the different meta-
tasks used in the meta-training $\mathcal{D}_{\mbox{{meta-train}}}{}$, meta-
validation $\mathcal{D}_{\mbox{{meta-valid}}}{}$, and meta-testing
$\mathcal{D}_{\mbox{{meta-test}}}{}$ datasets. To make the comparison fair and
consistent across different transfer modes, we use the same combination of
languages and tweak them to fit the transfer mode. By picking a high number of
meta-tasks during meta-training, meta-validation, and meta-testing, we make
sure that all transfer modes are exposed to the same number of questions and
candidates.

Transfer Mode | Phase | Support$\rightarrow$Query/Support1$\rightarrow$Support2$\rightarrow$Query
---|---|---
LAReQA | STSBMulti
mono$\rightarrow$mono | All |  EL_EL$\rightarrow$AR_AR HI_HI$\rightarrow$DE_DE | (EN_EN,AR_AR,ES_ES)$\rightarrow$(EN_EN,AR_AR,ES_ES)
mono$\rightarrow$bi | All |  EL_EL$\rightarrow$EL_AR HI_HI$\rightarrow$HI_DE | [EN_EN,AR_AR,ES_ES]$\rightarrow$[AR_EN,ES_EN,TR_EN]
mono$\rightarrow$multi | All |  EL_EL$\rightarrow$EL_{AR,EL} HI_HI$\rightarrow$HI_{DE,HI} | Not Applicable
bi$\rightarrow$multi | All |  EL_AR$\rightarrow$EL_{AR,EL} HI_DE$\rightarrow$HI_{DE,HI} | Not Applicable
mixt | All |  mono$\rightarrow$mono mono$\rightarrow$bi mono$\rightarrow$multi bi$\rightarrow$multi | Not Applicable
trans | Meta-train | mono$\rightarrow$bi | Not Applicable
Meta-valid | bi$\rightarrow$multi
Meta-test | mono$\rightarrow$multi
mono$\rightarrow$bi$\rightarrow$multi | All |  EL_EL$\rightarrow$EL_AR$\rightarrow$EL_{AR,EL,HI} HI_HI$\rightarrow$HI_DE$\rightarrow$HI_{AR,DE,HI} |  EN_EN$\rightarrow$AR_EN$\rightarrow$EN_{AR,EN,ES} AR_AR$\rightarrow$AR_ES$\rightarrow$AR_{AR,EN,ES} ES_ES$\rightarrow$ES_AR$\rightarrow$ES_{AR,EN,ES}

Table 4: Arrangements of languages for the different modes of transfer and
meta-learning stages for two standard benchmark datasets LAReQA and STSBMulti.
X$\rightarrow$Y denotes transfer from an X model (for example a monolingual
model) used to sample the support set to a Y model (for example bilingual
model) used to sample the query set. We denote a support or query set in
LAReQA by x_y where x and y are the ISO language codes of the question and the
candidate answers and x_y in STSBMulti where x and y are the ISO language
codes of sentence 1 and 2 respectively. We use parenthesis to mean that the
same language pairs cannot be used in both support and query sets, brackets to
denote non-exclusivity (or in other words the language pairs used as a support
can also be used as a query), and curled braces to mean the query set may be
sampled from more than one language. We do not experiment with
mono$\rightarrow$multi, bi$\rightarrow$multi, mixt, and trans for STSBMulti,
since it is not a multilingual parallel benchmark, but we still experiment
with mono$\rightarrow$bi$\rightarrow$multi using machine-translated data in
that case.

###  C.3 Hyperparameters

Sentence Transformers Model | mAP@20
---|---
LASER | 13.5 $\pm$ 0.7
LaBSE | 48.7 $\pm$ 2.6
M-BERT+SQUADEN | 37.9 $\pm$ 3.4
distilbert-multilingual-nli-stsb-quora-ranking | 44.1 $\pm$ 0.9
use-cmlm-multilingual | 36.8 $\pm$ 2.6
distiluse-base-multilingual-cased-v2 | 46.9 $\pm$ 2.5
paraphrase-multilingual-MiniLM-L12-v2 | 49.6 $\pm$ 2.7
multi-qa-distilbert-dot-v1 | 6.4 $\pm$ 0.3
paraphrase-multilingual-mpnet-base-v2 |  57.0 $\pm$ 2.9

Table 5: Comparison of mAP@20 multilingual 5-fold cross-validation evaluation
of different S-BERT models compared to M-BERT model. Best results are
highlighted in bold.

Based on our prior investigation of different sentence-transformer models in
Table 5, we notice that paraphrase-multilingual-mpnet-
base-v2111111https://huggingface.co/sentence-transformers/paraphrase-
multilingual-mpnet-base-v2., which maps sentences and paragraphs to a
768-dimensional dense vector space, performs the best for LAReQA, so we use it
in our S-BERT experiments on that dataset. The good initial performance of
this pre-trained model is not surprising since it was trained on parallel data
and is recommended for use in tasks like clustering or semantic search. For
pre-processing LAReQA and SQUADEN, we truncate/pad all questions to length 96
and all answer or negative candidates concatenated with their contexts to 256.
For pre-processing STSBMulti and STSBEN, we pad or truncate each sentence to
fit the maximum length of 100.

For both benchmarks, for Fine-tune baselines, following XTREME-R, we use AdamW
optimizer Loshchilov and Hutter (2019). We use a learning rate of $lr=5e-5$,
$\epsilon=1e-8$ and a weight decay of 0, with no decay on the bias and
LayerNorm weights. We use a batch size of 8 triplets or sentence pairs. For
LAReQA, we sample 3 negative examples per anchor and then project those to 3
triplets with one negative example and use a margin of 1. In STSBMulti, we use
just sets of sentence pairs composed of one source and one target sentence
each, where we don’t have negative examples so we don’t need to flatten the
dimensions of the negative examples. We sample 7,000, 2,000, and 1,000 meta-
tasks in the meta-training, meta-validation, and meta-testing phases
respectively. We use meta-batches of size $4$. In each meta-task, we randomly
sample $k=8$ and $q=4$ support and query triplets respectively. We use the
same meta-tasks and sampling regime in Fine-tune as well.

For MAML and MAML-Align in both benchmarks, we use learn2learn Arnold et al.
(2020) implementation to handle gradient updates, especially in the inner
loop. For the inner loop, we use learn2learn pre-built optimizer with a
learning rate $\alpha=1e-3$. The inner loop is repeated $n=5$ times for meta-
training and meta-validation and meta-testing. For the outer loop, we use the
same optimizer with the same learning rate $\beta=1e-5$ that we used in the
Fine-tune model. At the end of each epoch, we perform meta-validation
similarly to meta-training with the same hyperparameters described before. We
use the same hyperparameters for MAML-Align for both T-MAML and S-MAML except
that we run the gradient updates in the inner loop in S-MAML just once,
whereas for T-MAML we perform $n=5$ inner loop gradient updates. We jointly
optimize the outer loop losses weighting the knowledge distillation by
$\lambda=0.5$. We don’t use meta-testing but keep it for evaluation purposes.
For a consistent comparison, we don’t use meta-testing for our main evaluation
as we use standard testing cross-validation splits, but we will include those
meta-testing datasets to encourage future work on few-shot learning. All
experiments are run for one fixed initialization seed using a 5-fold cross-
validation. We observe a variance with respect to different seeds smaller than
the variance with respect to 5-fold cross-validation, so we report the latter
to have a better upper bound of the variance.

All experiments are conducted on the same computing infrastructure using _one_
NVIDIA A40 GPU with $46068$ MiB memory and _one_ TESLA P100-PCIE with $16384$
MiB memory of CUDA version 11.6 each. We use Pytorch version 1.11.1, Python
version 3.8.13, learn2learn version 0.1.7, Hugging Face transformers version
4.21.3 and Sentence-Transformers 2.2.2. For paraphrase-multilingual-mpnet-
base-v2 used in the experiments in the main paper, there are 278,043,648
parameters. For asymmetric and symmetric semantic search benchmarks, there are
three and two encoding towers, respectively. Therefore, there are 834,130,944
and 556,087,296 parameters used for asymmetric and symmetric semantic search
benchmarks, respectively.

For all experiments and model variants, we train for up to 20 epochs maximum
and we implement early stopping, where we run the experiment for as long as
there is an improvement on the Dev set performance. After 50 mini meta-task
batches of no improvement on the Dev set, the experiment stops running. We use
the multilingual performance on the Dev set averaged over all languages of the
query set as the early stopping evaluation criteria. Based on this early
stopping policy, we report in Table 6 the typical runtime for each upstream
model variant and baseline.

Model | Runtime
---|---
Fine-tune | 2 h 18 min
MAML | 3 h 19 min
MAML-Align | 19 h 29 min

Table 6: Runtime per model variant excluding evaluation.

##  Appendix D More Results

Model |  Train Language(s) Configuration | Testing Languages
---|---|---
Few-Shot Languages | Zero-Shot Languages |
Arabic | German | Greek | Hindi | Russian | Thai | Turkish | Mean
AR | DE | EL | HI | RU | TH | TR
Zero-Shot Baselines
LASER | - | 13.2 $\pm$ 5.1 | 15.1 $\pm$ 5.9 | 14.6 $\pm$ 5.6 | 9.4 $\pm$ 3.9 | 14.9 $\pm$ 5.6 | 13.0 $\pm$ 5.6 | 14.1 $\pm$ 6.0 | 13.5 $\pm$ 0.7
LaBSE | - | 44.7 $\pm$ 2.0 | 47.9 $\pm$ 3.8 | 53.0 $\pm$ 3.4 | 53.4 $\pm$ 3.2 | 53.1 $\pm$ 3.8 | 49.8 $\pm$ 2.8 | 48.1 $\pm$ 3.5 | 50.0 $\pm$ 2.8
S-BERT | - |  56.3 $\pm$ 2.7 |  54.6 $\pm$ 2.1 |  58.2 $\pm$ 3.8 |  57.2 $\pm$ 3.9 |  58.7 $\pm$ 3.3 |  60.2 $\pm$ 3.4 |  54.1 $\pm$ 2.5 |  57.0 $\pm$ 2.9
+Few-Shot Learning
S-BERT+Fine-tune | mono$\rightarrow$mono |  45.9 $\pm$ 2.4 | 46.3 $\pm$ 2.5 | 47.9 $\pm$ 2.6 | 45.4 $\pm$ 3.1 |  48.9 $\pm$ 2.7 |  49.7 $\pm$ 3.2 |  45.1 $\pm$ 1.6 |  47.0 $\pm$ 2.0
mono$\rightarrow$bi | 45.8 $\pm$ 4.0 |  46.5 $\pm$ 3.6 |  48.6 $\pm$ 4.7 |  45.0 $\pm$ 5.8 |  48.9 $\pm$ 4.2 | 49.4 $\pm$ 4.5 | 45.0 $\pm$ 3.1 |  47.0 $\pm$ 4.2
mono$\rightarrow$multi | 40.4 $\pm$ 3.9 | 42.5 $\pm$ 3.2 | 43.1 $\pm$ 4.6 | 37.8 $\pm$ 5.4 | 44.1 $\pm$ 4.3 | 44.3 $\pm$ 4.4 | 41.1 $\pm$ 3.1 | 41.9 $\pm$ 4.0
bi$\rightarrow$multi | 33.8 $\pm$ 4.9 | 35.6 $\pm$ 4.2 | 35.2 $\pm$ 6.2 | 32.4 $\pm$ 3.9 | 37.1 $\pm$ 5.3 | 37.2 $\pm$ 5.5 | 34.4 $\pm$ 4.3 | 35.1 $\pm$ 4.8
mixt | 38.3 $\pm$ 4.1 | 39.8 $\pm$ 4.6 | 40.7 $\pm$ 3.8 | 39.3 $\pm$ 5.2 | 41.9 $\pm$ 5.0 | 41.7 $\pm$ 5.1 | 38.7 $\pm$ 3.9 | 40.1 $\pm$ 4.4
trans | 38.7 $\pm$ 3.8 | 39.9 $\pm$ 4.8 | 41.8 $\pm$ 3.4 | 40.1 $\pm$ 3.8 | 42.6 $\pm$ 4.3 | 42.6 $\pm$ 3.8 | 39.4 $\pm$ 4.0 | 40.7 $\pm$ 3.8
S-BERT+MAML | mono$\rightarrow$mono |  56.3 $\pm$ 1.6 | 54.5 $\pm$ 2.0 | 58.5 $\pm$ 3.3 |  57.0 $\pm$ 2.5 |  59.3 $\pm$ 2.5 | 59.6 $\pm$ 2.7 | 53.8 $\pm$ 1.9 | 57.0 $\pm$ 2.3
mono$\rightarrow$bi | 55.9 $\pm$ 3.1 |  55.0 $\pm$ 3.0 | 58.4 $\pm$ 4.6 | 56.9 $\pm$ 4.0 | 58.8 $\pm$ 3.9 |  59.9 $\pm$ 3.4 | 54.2 $\pm$ 3.0 | 57.0 $\pm$ 3.5
mono$\rightarrow$multi | 54.9 $\pm$ 2.8 | 53.6 $\pm$ 3.4 | 57.0 $\pm$ 4.7 | 55.8 $\pm$ 3.9 | 57.7 $\pm$ 4.1 | 58.7 $\pm$ 3.4 | 53.1 $\pm$ 3.2 | 55.9 $\pm$ 3.5
bi$\rightarrow$multi | 54.5 $\pm$ 2.1 | 53.6 $\pm$ 2.1 | 56.6 $\pm$ 2.5 | 55.5 $\pm$ 1.7 | 57.3 $\pm$ 2.2 | 58.5 $\pm$ 1.9 | 52.8 $\pm$ 1.6 | 55.5 $\pm$ 1.7
mixt | 55.0 $\pm$ 3.1 | 53.9 $\pm$ 2.4 | 57.2 $\pm$ 3.9 | 55.3 $\pm$ 4.2 | 57.6 $\pm$ 3.7 | 58.7 $\pm$ 3.0 | 52.9 $\pm$ 3.0 | 55.8 $\pm$ 3.2
trans | 56.0 $\pm$ 3.7 | 54.8 $\pm$ 2.2 |  59.1 $\pm$ 4.2 |  57.0 $\pm$ 4.4 | 59.1 $\pm$ 4.1 |  59.9 $\pm$ 3.8 |  54.4 $\pm$ 3.0 |  57.2 $\pm$ 3.5
S-BERT+MAML-Align | mono$\rightarrow$bi$\rightarrow$multi |  57.0 $\pm$ 2.9 |  55.1 $\pm$ 2.4 |  59.2 $\pm$ 4.2 |  57.7 $\pm$ 4.5 |  59.5 $\pm$ 3.5 |  60.2 $\pm$ 3.7 |  54.6 $\pm$ 2.7 |  57.6 $\pm$ 3.3
+Machine Translation
S-BERT+T-Train+Fine-tune | AR_AR$\rightarrow$AR_AR |  46.6 $\pm$ 3.5 |  45.8 $\pm$ 3.4 |  48.8 $\pm$ 4.2 |  46.8 $\pm$ 4.2 |  49.3 $\pm$ 4.6 | 48.6 $\pm$ 3.8 |  44.9 $\pm$ 3.5 |  47.3 $\pm$ 3.8
DE_DE$\rightarrow$DE_DE | 45.9 $\pm$ 5.0 | 45.1 $\pm$ 4.4 | 48.2 $\pm$ 5.8 | 45.8 $\pm$ 6.5 | 49.0 $\pm$ 5.1 |  48.8 $\pm$ 6.8 | 44.5 $\pm$ 4.5 | 46.8 $\pm$ 5.4
EL_EL$\rightarrow$EL_EL | 43.5 $\pm$ 4.3 | 43.1 $\pm$ 4.5 | 43.8 $\pm$ 4.5 | 43.4 $\pm$ 4.1 | 46.5 $\pm$ 4.2 | 45.0 $\pm$ 3.5 | 41.7 $\pm$ 4.3 | 43.8 $\pm$ 4.0
HI_HI$\rightarrow$HI_HI | 46.5 $\pm$ 3.1 | 44.8 $\pm$ 2.9 | 47.1 $\pm$ 3.8 | 45.9 $\pm$ 4.1 | 48.4 $\pm$ 4.4 | 49.6 $\pm$ 3.7 | 43.7 $\pm$ 3.0 | 46.6 $\pm$ 3.4
All test languages | 44.8 $\pm$ 2.8 | 43.5 $\pm$ 3.2 | 46.9 $\pm$ 4.0 | 44.0 $\pm$ 4.5 | 47.0 $\pm$ 3.4 | 46.4 $\pm$ 3.9 | 42.1 $\pm$ 3.0 | 45.0 $\pm$ 3.4
S-BERT+T-Train+MAML | AR_AR$\rightarrow$AR_AR |  57.3 $\pm$ 3.2 |  55.3 $\pm$ 2.1 |  59.3 $\pm$ 4.2 |  58.3 $\pm$ 4.2 |  60.2 $\pm$ 3.6 |  60.7 $\pm$ 3.5 |  54.8 $\pm$ 2.3 |  58.0 $\pm$ 3.2
DE_DE$\rightarrow$DE_DE | 56.1 $\pm$ 2.7 | 54.4 $\pm$ 2.2 | 58.3 $\pm$ 3.9 | 57.1 $\pm$ 4.1 | 58.8 $\pm$ 3.9 | 59.8 $\pm$ 3.7 | 54.1 $\pm$ 2.7 | 56.9 $\pm$ 3.2
EL_EL$\rightarrow$EL_EL | 55.9 $\pm$ 3.4 | 53.1 $\pm$ 4.3 | 57.4 $\pm$ 5.2 | 56.3 $\pm$ 5.5 | 58.5 $\pm$ 4.6 | 59.2 $\pm$ 5.2 | 52.8 $\pm$ 4.4 | 56.2 $\pm$ 4.5
HI_HI$\rightarrow$HI_HI | 56.7 $\pm$ 3.6 | 54.0 $\pm$ 2.6 | 58.5 $\pm$ 4.6 | 57.1 $\pm$ 4.7 | 58.9 $\pm$ 4.1 | 60.3 $\pm$ 3.0 | 53.7 $\pm$ 3.3 | 57.0 $\pm$ 3.5
All test languages | 55.9 $\pm$ 3.9 | 53.8 $\pm$ 2.7 | 58.0 $\pm$ 4.9 | 56.6 $\pm$ 4.4 | 58.1 $\pm$ 4.3 | 59.2 $\pm$ 3.9 | 53.4 $\pm$ 3.0 | 56.4 $\pm$ 3.7

Table 7:  mAP@20 multilingual 5-fold cross-validated performance tested for
different languages. Best and second-best results for each language are
highlighted in bold and italicized respectively, whereas best results across
categories of models are underlined. Gains from meta-learning approaches are
consistent across few-shot and zero-shot languages.

Model |  Train Language(s) Configuration | Testing Languages
---|---|---
Arabic-Arabic | Arabic-English | Spanish-Spanish | Spanish-English | English-English | Turkish-English | Mean
AR-AR | AR-EN | ES-ES | ES-EN | EN-EN | TR-EN
Zero-Shot Learning
LASER | - | 22.5 $\pm$ 8.5 | 21.6 $\pm$ 8.4 | 33.1 $\pm$ 9.4 | 15.3 $\pm$ 15.7 | 31.1 $\pm$ 5.4 | 21.2 $\pm$ 13.7 | 24.1 $\pm$ 12.4
LaBSE | - | 71.6 $\pm$ 6.2 | 73.2 $\pm$ 4.0 | 83.2 $\pm$ 1.7 | 68.7 $\pm$ 10.1 | 76.3 $\pm$ 2.7 | 74.9 $\pm$ 3.3 | 74.6 $\pm$ 4.6
S-BERT | - |  77.6 $\pm$ 5.3 |  81.3 $\pm$ 3.2 |  84.6 $\pm$ 2.9 |  83.7 $\pm$ 6.7 |  85.5 $\pm$ 4.2 |  75.7 $\pm$ 3.1 |  81.4 $\pm$ 4.2
+Few-Shot learning
S-BERT+Fine-tune | mono$\rightarrow$bi | 77.2 $\pm$ 5.8 | 77.8 $\pm$ 3.8 | 86.2 $\pm$ 2.8 | 79.6 $\pm$ 8.3 | 85.0 $\pm$ 4.5 | 73.7 $\pm$ 4.6 | 79.9 $\pm$ 2.0
S-BERT+MAML | mono$\rightarrow$bi | 77.6 $\pm$ 5.3 |  80.9 $\pm$ 2.6 | 85.1 $\pm$ 2.4 |  83.5 $\pm$ 6.7 | 85.6 $\pm$ 4.8 | 75.5 $\pm$ 3.7 | 81.3 $\pm$ 1.4
S-BERT+MAML-Align | mono$\rightarrow$bi$\rightarrow$multi |  79.0 $\pm$ 5.2 | 80.6 $\pm$ 1.0 |  86.6 $\pm$ 2.1 | 81.5 $\pm$ 6.8 |  90.6 $\pm$ 1.1 |  76.3 $\pm$ 4.0 |  82.4 $\pm$ 1.4
+Machine Translation
S-BERT+T-Train+Fine-tune | AR_AR$\rightarrow$AR_AR | 59.5 $\pm$ 7.9 | 50.6 $\pm$ 12.5 | 82.7 $\pm$ 4.3 | 70.1 $\pm$ 11.9 | 82.4 $\pm$ 5.7 | 62.5 $\pm$ 5.9 | 68.0 $\pm$ 14.6
EN_EN$\rightarrow$EN_EN | 72.6 $\pm$ 6.8 | 73.1 $\pm$ 4.9 | 82.4 $\pm$ 2.9 | 72.2 $\pm$ 10.9 | 80.3 $\pm$ 6.8 |  68.8 $\pm$ 5.8 | 74.9 $\pm$ 8.3
ES_ES$\rightarrow$ES_ES |  74.2 $\pm$ 8.0 | 72.3 $\pm$ 8.0 | 82.3 $\pm$ 2.8 | 66.8 $\pm$ 12.1 | 79.7 $\pm$ 6.9 | 68.5 $\pm$ 4.8 | 73.9 $\pm$ 9.5
TR_TR$\rightarrow$TR_TR | 73.9 $\pm$ 6.3 |  74.6 $\pm$ 3.4 |  85.9 $\pm$ 2.0 |  79.6 $\pm$ 6.3 |  84.3 $\pm$ 4.7 | 68.5 $\pm$ 3.7 |  77.8 $\pm$ 7.7
| All test languages | 65.8 $\pm$ 9.0 | 63.0 $\pm$ 4.6 | 82.5 $\pm$ 3.0 | 75.8 $\pm$ 8.7 | 83.0 $\pm$ 4.7 | 67.8 $\pm$ 4.9 | 73.0 $\pm$ 10.1
S-BERT+T-Train+MAML | AR_AR$\rightarrow$AR_AR | 75.5 $\pm$ 6.0 | 80.5 $\pm$ 2.5 | 85.8 $\pm$ 2.1 | 83.1 $\pm$ 6.3 | 85.6 $\pm$ 3.9 | 75.0 $\pm$ 4.0 | 80.9 $\pm$ 6.2
EN_EN$\rightarrow$EN_EN |  77.8 $\pm$ 5.2 |  81.7 $\pm$ 3.0 | 85.1 $\pm$ 2.6 |  83.8 $\pm$ 6.6 |  85.7 $\pm$ 4.3 | 75.8 $\pm$ 3.5 |  81.6 $\pm$ 5.8
ES_ES$\rightarrow$ES_ES | 76.4 $\pm$ 6.4 | 79.4 $\pm$ 3.4 |  86.9 $\pm$ 1.6 | 80.4 $\pm$ 7.7 | 84.7 $\pm$ 4.7 | 74.1 $\pm$ 4.2 | 80.3 $\pm$ 6.7
TR_TR$\rightarrow$TR_TR | 77.2 $\pm$ 5.9 | 79.8 $\pm$ 3.8 |  87.3 $\pm$ 1.7 | 81.6 $\pm$ 6.4 | 84.5 $\pm$ 4.2 | 74.2 $\pm$ 2.2 | 80.8 $\pm$ 6.2
| All test languages | 77.6 $\pm$ 5.3 |  81.8 $\pm$ 2.5 | 84.7 $\pm$ 2.9 | 83.6 $\pm$ 6.7 | 85.6 $\pm$ 4.2 |  75.9 $\pm$ 3.4 | 81.5 $\pm$ 5.7

Table 8:  Pearson correlation Pearson’s r × 100 5-fold cross-validated
performance on STSBMulti benchmark using different models few-shot learned on
STSBMulti or its translation. Best and second-best results for each language
are highlighted in bold and italicized respectively, whereas best results
across categories of models are underlined.

Tables 7 and 8 show full fine-grained results for all languages and language
pairs for both semantic search benchmarks.