# Three-dimensional plasmoid-mediated reconnection and turbulence in Hall
magnetohydrodynamics

Yi-Min Huang<EMAIL_ADDRESS>Department of Astrophysical Sciences, New
Jersey 08540, USA  Amitava Bhattacharjee  Department of Astrophysical
Sciences, New Jersey 08540, USA

###### Abstract

Plasmoid instability accelerates reconnection in collisional plasmas by
transforming a laminar reconnection layer into numerous plasmoids connected by
secondary current sheets in two dimensions (2D) and by fostering self-
generated turbulent reconnection in three dimensions (3D). In large-scale
astrophysical and space systems, plasmoid instability likely initiates in the
collisional regime but may transition into the collisionless regime as the
fragmentation of the current sheet progresses toward kinetic scales. Hall MHD
models are widely regarded as a simplified yet effective representation of the
transition from collisional to collisionless reconnection. However, plasmoid
instability in 2D Hall MHD simulations often leads to a single-X-line
reconnection configuration, which significantly differs from fully kinetic
particle-in-cell simulation results. This study shows that single-X-line
reconnection is less likely to occur in 3D compared to 2D. Moreover, depending
on the Lundquist number and the ratio between the system size and the kinetic
scale, Hall MHD can also realize 3D self-generated turbulent reconnection. We
analyze the features of the self-generated turbulent state, including the
energy power spectra and the scale dependence of turbulent eddy anisotropy.

##  I Introduction

Magnetic reconnection is a fundamental process that alters the connectivity of
magnetic field lines, releasing stored magnetic energy. The magnetic energy
released by reconnection is transformed into kinetic, thermal, and non-thermal
energy of plasma in explosive events in nature and laboratories, such as
geomagnetic substorms, solar flares, coronal mass ejections (CMEs), gamma-ray
bursts, and sawtooth crashes in fusion plasmas.[1, 2, 3, 4, 5, 6, 7]

Magnetic reconnection occurs at current sheets. In recent years, there has
been growing evidence that large-scale reconnection current sheets will likely
become fragmented due to the plasmoid instability.[8, 9] The plasmoid
instability has been found in a wide range of plasma models, including
resistive magnetohydrodynamics (MHD), [10, 11, 8, 12, 9, 13, 14, 15, 16, 17,
18, 19, 20, 21, 22, 23] Hall MHD,[24, 16, 25] and fully kinetic particle-in-
cell (PIC) simulations. [26, 27, 28, 29, 30, 31]

High-resolution two-dimensional (2D) resistive MHD simulations indicate that
the plasmoid instability leads to a fast reconnection rate that is nearly
independent of the resistivity.[9, 14, 18] In three-dimensional (3D)
reconnection with a guide field, in addition to modes that are symmetric along
the out-of-plane direction, oblique tearing modes can become unstable.[32]
Interaction of oblique tearing modes leads to self-generated turbulent
reconnection without the need for external forcing.[28, 33, 34, 35]

Taking into account two-fluid and kinetic physics, the plasmoid instability
can trigger even faster collisionless reconnection if the fragmented current
sheets reach a kinetic scale $\delta_{i}$ that corresponds to the ion skin
depth $d_{i}$ or the ion sound Larmor radius $\rho_{s}$, the latter being for
cases with a strong guide field.[27, 24, 16] This effect leads to the
consideration of reconnection “phase diagram” in the literature.[36, 16, 21,
37] The phase diagram organizes various types of reconnection in the parameter
space of two dimensionless parameters: the Lundquist number $S\equiv
LV_{A}/\eta$ and the system size to the kinetic scale ratio $\Lambda\equiv
L/\delta_{i}$ . Here, $L$ is a characteristic length scale of the reconnection
layer along the outflow direction, $V_{A}$ is the Alfvén speed of the
reconnecting magnetic field component, and $\eta$ is the magnetic diffusivity.
Within resistive MHD, the reconnection current sheet becomes unstable when the
Lundquist number $S$ exceeds a critical value $S_{c}\sim 10^{4}$.[9, 38] Once
the current sheet becomes unstable, the fragmentation continues until the
secondary current sheets become marginally stable. This condition yields the
typical widths of the secondary current sheets to scale as $\delta_{c}\sim
LS_{c}^{1/2}/S$.[14] Comparing the secondary current sheet width $\delta_{c}$
with the kinetic scale $\delta_{i}$ yields a heuristic criterion
$\Lambda<S/S_{c}^{1/2}$ for the plasmoid-mediated onset of collisionless
reconnection.

Based on simple estimates, we can see that this onset criterion is satisfied
for numerous types of reconnection events in the solar atmosphere and
astrophysical systems.[36] For example, consider post-CME current sheets in
the corona[39, 40] and UV (ultraviolet) bursts in the transition region.[41,
42, 43] Typical lengths of post-CME current sheets are on the order of $\sim
10^{9}\text{m}$, Lundquist numbers $S\sim 10^{14}$, and $d_{i}\sim 1\text{m}$;
Hence, the ratio $\Lambda=L/d_{i}\sim 10^{9}$, which is smaller than
$S/S_{c}^{1/2}\sim 10^{12}$. For UV bursts, the lengths are estimated to be on
the order of $10^{5}\text{m}$, the Lundquist numbers $S\sim 10^{10}$, and
$d_{i}\sim 0.1\text{m}$. The ratio $\Lambda\sim 10^{6}$ is smaller than
$S/S_{c}^{1/2}\sim 10^{8}$. Therefore, plasmoid-mediated onset of
collisionless reconnection is potentially important for both types of
reconnection events.

Evidence of plasmoids in both types of reconnection events is abundant. In
post-CME current sheets, plasmoid-like structures have been regularly observed
as moving blobs. [45, 39, 40] In UV bursts, although plasmoids are not
directly visible, their existence may be inferred from the shapes of emission
line profiles, which provide information on reconnection outflow through the
Doppler shift. The emission line profiles of UV bursts typically have a
triangle shape, which is consistent with a highly fluctuating reconnection
outflow associated with plasmoids. In contrast, a single-X-line reconnection
geometry will predict a double-peak line profile, which is inconsistent with
observation.[41, 43]

To model the transition from collisional to collisionless reconnection
mediated by the plasmoid instability, particle-in-cell (PIC) simulation
provides a first-principles, fully kinetic description.[27, 31] However, PIC
simulations, especially in 3D, are usually limited to relatively small system
sizes due to the computational cost. To simulate systems of larger sizes, this
study employs a Hall MHD model incorporating two-fluid effects through the
Hall terms in the generalized Ohm’s law. Hall MHD is often regarded as a
minimal model that captures important aspects of collisonless or weakly
collisional reconnection beyond resistive MHD.[44] Previous 2D Hall MHD
studies have demonstrated the onset of Hall reconnection mediated by plasmoid
instability.[24, 16]

From the perspective of plasmoid formation, 2D Hall MHD reconnection is
somewhat anomalous: it tends to settle to a single-X-line geometry after
expelling all the plasmoids from the reconnection layer. This feature makes it
qualitatively different from fully kinetic PIC simulations, where new
plasmoids are constantly generated.[27] The single-X-line geometry of Hall
reconnection also appears to be inconsistent with observations of post-CME
current sheets and UV bursts, which indicate that plasmoids continue to form
throughout the entire period of the events. Although non-single-X-line 2D Hall
reconnection has been demonstrated, it appears to require a fairly large
system size ($L/d_{i}\gtrsim 5000$) and may only exist in a narrow parameter
space of the phase diagram.[16]

The observed persistence of plasmoid formation throughout solar reconnection
events raises an important question regarding whether Hall MHD can adequately
describe these phenomena. However, it is important to note that the preference
for single-X-line reconnection in Hall MHD may result from the assumed 2D
symmetry. Will 3D systems allow the formation of complex 3D structures and
significantly change the picture? Moreover, if 3D Hall MHD reconnection
geometry deviates from a single X-line, can it support self-generated
turbulent reconnection?

To address these questions, we perform 3D Hall MHD simulations of reconnection
and compare the results with the corresponding 2D simulations. We aim for
large system sizes, up to $L/d_{i}=1000$. To mimic natural reconnection
events, we start from an initial current sheet that is significantly thicker
than $d_{i}$ and let it thin down self-consistently. After the plasmoid
instability triggers the onset of Hall reconnection, we continue to follow the
evolution until it reaches a saturated phase.

This paper is organized as follows. Section II describes the governing
equations and the simulation setup in detail. In Section III, we present
simulation results of reconnection geometry and reconnection rate for various
2D and 3D runs. Section IV looks more deeply into the characteristics of the
fully developed reconnection state by examining the energy power spectra and
two-point structure functions of kinetic and magnetic fluctuations. Finally,
we discuss the implications of our findings for large-scale astrophysical
reconnection and conclude in Section V.

##  II Governing Equations and Model Setup

Figure 1: Initial magnetic field configuration. Black lines are stream lines
of the in-plane magnetic field component, and colormap shows the out-of-plane
component $B_{z}$.

We employ the visco-resistive Hall magnetohydrodynamics model with an
adiabatic equation of state. In normalized units, the equations are:

$\partial_{t}\rho+\nabla\cdot\left(\rho\boldsymbol{v}\right)=0,$ (1)
$\displaystyle\partial_{t}\left(\rho\boldsymbol{v}\right)+\nabla\cdot\left(\rho\boldsymbol{vv}\right)=$
$\displaystyle-\nabla\left(p+\frac{B^{2}}{2}\right)+\nabla\cdot\left(\boldsymbol{BB}\right)$
$\displaystyle+\nabla\cdot\left(\rho\nu\frac{\nabla\boldsymbol{v}+\nabla\boldsymbol{v}^{T}}{2}\right),$
(2)
$\partial_{t}p+\nabla\cdot(p\boldsymbol{v})=-(\gamma-1)p\nabla\cdot\boldsymbol{v},$
(3) $\partial_{t}\boldsymbol{B}=-\nabla\times\boldsymbol{E},$ (4)
$\boldsymbol{E=}-\boldsymbol{v}\times\boldsymbol{B}+d_{i}\frac{\boldsymbol{J}\times\boldsymbol{B}-\nabla
p_{e}}{\rho}+\eta\boldsymbol{J}-\eta_{H}\nabla^{2}\boldsymbol{J}.$ (5)

Here, standard notations are used: $\rho$ is the plasma density;
$\boldsymbol{v}$ is the ion velocity; $p$ is the total plasma pressure;
$p_{e}$ is the electron pressure; $\boldsymbol{B}$ is the magnetic field;
$\boldsymbol{E}$ is the electric field;
$\boldsymbol{J}=\nabla\times\boldsymbol{B}$ is the electric current density;
$\eta$ is the resistivity; $\eta_{H}$ is the hyper-resistivity;[46] $\nu$ is
the viscosity; and $d_{i}$ is the ion skin depth. For simplicity, we assume
that the ions and electrons are locally in thermal equilibrium. Therefore, the
ion pressure $p_{i}$ and the electron pressure $p_{e}$ are equal, and the
total plasma pressure $p=p_{i}+p_{e}=2p_{e}$. Electron inertia terms are
neglected in the generalized Ohm’s law, Eq. (5).

The normalization of Eqs. (1) – (5) is based on constant reference values of
the density $n_{0}$, and the magnetic field $B_{0}$. Lengths are normalized to
the system size $L$, and time is normalized to the global Alfvén time
$t_{A}=L/V_{A}$, where $V_{A}=B_{0}/\sqrt{4\pi n_{0}m_{i}}$ and $m_{i}$ is the
ion mass. The normalization of physical variables is given by (normalized
variables $\to$ expressions in Gaussian units): $\rho\to\rho/n_{0}m_{i}$,
$\boldsymbol{B}\to\boldsymbol{B}/B_{0}$, $\boldsymbol{E}\to
c\boldsymbol{E}/B_{0}V_{A}$, $\boldsymbol{v}\to\boldsymbol{v}/V_{A}$, $p\to
p/n_{0}m_{i}V_{A}^{2}$, $\boldsymbol{J}\to\boldsymbol{J}/(B_{0}c/4\pi L)$, and
$d_{i}\to d_{i}/L\equiv\sqrt{m_{i}c^{2}/4\pi n_{0}e^{2}}/L$.

The simulation setup is similar to that employed in previous studies.[14, 16,
20, 33, 47] In this setup, the coalescence of two magnetic flux tubes drives
magnetic reconnection. The simulation box is a 3D cube in the domain
$(x,y,z)\in[-1/2,1/2]\times[-1/2,1/2]\times[-1/2,1/2]$. In normalized units,
the initial magnetic field is given by
$\boldsymbol{B}=B_{z}\boldsymbol{\hat{z}}+\boldsymbol{\hat{z}}\times\nabla\psi$,
where $\psi=\tanh\left(y/h\right)\psi_{0}$ and $\psi_{0}=\cos\left(\pi
x\right)\sin\left(2\pi y\right)/2\pi$. Here, the parameter $h$ determines the
initial thickness of the current layer between the flux tubes. The out-of-
plane magnetic field component $B_{z}$ is non-uniform such that the initial
configuration is approximately force-balanced. Precisely, $B_{z}$ satisfies

$B_{z}^{2}=B_{z0}^{2}+5\pi^{2}\psi_{0}^{2}+\left(\partial_{y}\psi_{0}\right)^{2}-\left(\partial_{y}\psi\right)^{2},$
(6)

where the parameter $B_{z0}$ sets the guide field strength. In this study, the
guide field strength $B_{z0}$ is set to unity. The initial current sheet
thickness $h=0.01$. In the upstream region of the current layer, the
reconnecting component $B_{x}$ and the guide field $B_{z}$ are both
approximately equal to unity. Figure 1 shows the initial magnetic field
configuration.

The initial plasma density and pressure are both uniform, with $\rho=1$ and
$p_{i}=p_{e}=1$ in normalized units. The heat capacity ratio $\gamma=5/3$ is
assumed. Perfectly conducting and free-slip boundary conditions are imposed
along both $x$ and $y$ directions. Specifically, we have
$\boldsymbol{v}\cdot\hat{\boldsymbol{n}}=0$,
$\hat{\boldsymbol{n}}\cdot\nabla\left(\hat{\boldsymbol{n}}\times\boldsymbol{v}\right)=0$,
and $\boldsymbol{B}\cdot\hat{\boldsymbol{n}}=0$ on the boundaries. Here,
$\hat{\boldsymbol{n}}$ is the unit vector normal to the boundary. The $z$
direction is assumed to be periodic.

This model system is numerically solved with a massively parallel HMHD code.
The numerical algorithm [48] approximates spatial derivatives by finite
differences with a five-point stencil in each direction, augmented with a
fourth-order numerical dissipation for numerical stability. The time-stepping
scheme can be chosen from several options, including a second-order accurate
trapezoidal leapfrog method and various strong-stability-preserving
Runge–Kutta methods.[49, 50] We employ the second-order accurate trapezoidal
leapfrog method in this study.

We perform two sets of simulations with the ion skin depth $d_{i}=0.002$ and
$0.001$. The plasma resistivity and viscosity $\nu$ are both set to a fixed
value $5\times 10^{-6}$. Using the box size as the length scale $L$, the
system size to the ion skin depth ratio $L/d_{i}$ is $500$ and $1000$,
respectively. The Lundquist number $S=V_{A}L/\eta=2\times 10^{5}$ and the
magnetic Prandtl number $Pr_{m}\equiv\nu/\eta=1$. Because Hall MHD tends to
develop fine structures at small scales, we add a small hyper-resistivity
$\eta_{H}=10^{-13}$ to smooth fluctuations at grid scales. The initial
velocity is seeded with a random noise of amplitude $10^{-3}$ to trigger the
plasmoid instability. We perform 2D and 3D simulations with the same
parameters and compare the results.

The 3D simulation mesh size is $N_{x}\times N_{y}\times N_{z}=2000\times
1000\times 2000$ ($2000\times 1000$ for 2D), where the grid sizes are uniform
along both $x$ and $z$ directions, and packed along the $y$ direction around
the midplane to resolve the reconnection layer better. The grid size along the
$y$ direction is $\Delta y=10^{-4}$ near the midplane ($y=0$). Moving away
from the midplane, the grid size gradually increases and reaches $\Delta
y=0.005$ near the boundaries at $y=\pm 1/2$.

##  III Reconnection Geometry and Reconnection Rate in Two-dimension and
Three-Dimension Simulations

Figure 2: Time sequence for the 2D simulation with $d_{i}=0.002$ in the
reconnection layer. The colormap denotes the reconnection outflow velocity
$v_{x}$ (multimedia view).

Figure 2 and the associate animation show the time evolution of the 2D
simulation with $d_{i}=0.002$, corresponding to $L/d_{i}=500$. Here, the
colormap denotes the reconnection outflow velocity $v_{x}$. The initial
thinning of the current sheet is accompanied by the formation of reconnection
outflow jets. At around $t=0.6$, several plasmoids start to form in the
reconnection layer. Subsequently, the onset of Hall reconnection expels all
the plasmoids, and the reconnection geometry settles to a single-X-line
configuration at $t=1.5$. The 2D simulation with $d_{i}=0.001$ also exhibits
qualitatively similar behavior, as shown in the animation in the supplementary
material. These two runs demonstrate the strong tendency of 2D Hall MHD to
form a single-X-line reconnection geometry, similar to the results of previous
studies.[24, 16]

Figure 3: Time sequence for the 3D simulation with $d_{i}=0.002$. Here, the
two end plates at $z=\pm 0.5$ and the isosurfaces of the flow speed at $v=0.4$
are colored according to the magnitude of the current density $J$ in the
logarithmic scale. Samples of magnetic field lines are colored according to
the flow speed $v$. The arrows attached to field lines denote the local
velocity vector (multimedia view).  Figure 4: Time sequence for the 3D
simulation with $d_{i}=0.001$ (multimedia view).

Three-dimensional (3D) simulations of the same parameters show different
behaviors. In the time evolution of the first case with $d_{i}=0.002$, shown
in Figure 3, the thinning current sheet first becomes unstable, forming flux
ropes. The interaction between flux ropes leads to complex dynamics and the
formation of chaotic field lines. However, all the flux ropes are eventually
ejected and the magnetic field self-organizes into a nearly single-X-line
configuration. The final state is similar to its 2D counterpart, even though a
translational symmetry is not imposed. However, for the second case with
$d_{i}=0.001$ ($L/d_{i}=1000$), shown in Figure 4, the magnetic configuration
does not settle to a single-X-line. Instead, it appears to develop a self-
generated turbulent state similar to previous resistive MHD and PIC
simulations.

To quantify the 3D reconnection rate, we first average the magnetic field
along the $z$ direction to obtain the mean magnetic field
$\bar{\boldsymbol{B}}$, which now depends only on $x$ and $y$. We then use the
mean field to calculate the reconnection rate in the same manner as the
calculation for the 2D reconnection rate. Figure 5 shows the time histories of
reconnection rates for these 2D and 3D runs. The first three cases, all of
which end up in a single-X-line configuration, yield faster reconnection
rates. In comparison, reconnection in the fourth case, which does not settle
to a single-X-line configuration, is slower. For reference, we also add a 3D
resistive MHD (i.e., $d_{i}=0$) run, which gives a substantially slower
reconnection rate.

Figure 5: Time histories of the reconnection rates for different cases. Solid
lines correspond to 3D simulations, and dashed lines are 2D simulations.

##  IV Characteristics of the Self-Generated Turbulent State

Figure 6: Time sequences of the kinetic (upper panel) and the magnetic energy
(lower panel) spectra integrated over the range from $y=-0.05$ to $0.05$ for
the case $d_{i}=0.002$. The tick labels of the colorbar indicate the times
corresponding to the curves. Figure 7: Time sequences of the kinetic (upper
panel) and the magnetic energy (lower panel) spectra integrated over the range
from $y=-0.05$ to $0.05$ for the case $d_{i}=0.001$. The tick labels of the
colorbar indicate the times corresponding to the curves. Figure 8: Two-point
structure function analysis of the fully developed turbulent state for the
case with $d_{i}=0.001$ at $t=2.1$. The first column shows the magnitude of
the fluctuating part of the magnetic field $\tilde{B}$, overlaid with
streamlines of the in-plane magnetic field. The second column shows contours
of the structure function $F_{2}^{B}(r_{\parallel},r_{\perp})$. The third
column shows the relations between the semi-major axis $r_{\parallel}\sim
k_{\parallel}^{-1}$ and the semi-minor axis $r_{\perp}\sim k_{\perp}^{-1}$ of
contours of the structure functions $F_{2}^{B}(r_{\parallel},r_{\perp})$ and
$F_{2}^{w}(r_{\parallel},r_{\perp})$. These relationships quantify the scale
dependence of anisotropy in turbulent eddies. The two dashed lines represent
the scalings of $k_{\parallel}\sim k_{\perp}$ (scale-independent) and
$k_{\parallel}\sim k_{\perp}^{2/3}$ (Goldreich–Sridhar theory), for reference.
The first row shows the results at $y=0$, the second row at $y=0.002$, and the
third row at $y=0.005$, respectively.

We further investigate whether the reconnection layer evolves to a self-
generated turbulent state in the two 3D Hall MHD simulations by examining the
energy spectra and two-point structure functions of the kinetic and magnetic
fluctuations _within_ the layer. Special treatments are necessary for
calculating energy spectra and structure functions due to the strong
inhomogeneity and shear in both magnetic and velocity fields. We adopt the
procedure of Huang and Bhattacharjee (2016), [33] which is summarized below.

To calculate the kinetic energy spectrum, we first define a weighted velocity
field $\boldsymbol{w}\equiv\rho^{1/2}\boldsymbol{v}$ such that the kinetic
energy density is a quadratic form $w^{2}/2$. We then decompose
$\boldsymbol{w}$ into the sum of the mean field $\bar{\boldsymbol{w}}$ and the
fluctuation $\tilde{\boldsymbol{w}}$, where the mean field is defined as the
average along the $z$ direction. The total kinetic energy is the sum of the
mean-field kinetic energy $\bar{E_{k}}\equiv\int\bar{w}^{2}/2\,d^{3}x$ and the
fluctuation part of the kinetic energy
$\tilde{E_{k}}\equiv\int\tilde{w}^{2}/2\,d^{3}x$. Likewise, we decompose the
magnetic field $\boldsymbol{B}$ into the mean field $\bar{\boldsymbol{B}}$ and
the fluctuation $\tilde{\boldsymbol{B}}$. The total magnetic energy is the sum
of the mean-field magnetic energy $\bar{E_{m}}\equiv\int\bar{B}^{2}/2\,d^{3}x$
and the fluctuation part of the magnetic energy
$\tilde{E_{m}}\equiv\int\tilde{B}^{2}/2\,d^{3}x$. We only consider the
fluctuation parts for the calculation of the kinetic and magnetic energy
spectra.

Before we calculate the turbulence energy spectra within the reconnection
layer, we first multiply the fluctuating fields $\tilde{\boldsymbol{B}}$ and
$\tilde{w}$ by a $C^{\infty}$ Planck-taper window function,[51] which equals
unity within the range $-0.2\leq x\leq 0.2$ and tapers off smoothly to zero
over the ranges where $0.2\leq\left|x\right|\leq 0.4$. This step reduces the
influence of irrelevant fluctuations from downstream exhaust regions, allowing
us to focus on turbulence within the reconnection layer. Next, we compute the
discrete Fourier energy spectra of $\tilde{E_{k}}$ and $\tilde{E_{m}}$ using
the “windowed” variables on each constant-$y$ slice to obtain 2D energy
spectra as functions of the wave numbers $k_{x}$ and $k_{z}$. Then, we
integrate the 2D energy spectra over the reconnection inflow direction $y$,
which is also the direction of the strongest inhomogeneity, within the range
$-0.05\leq y\leq 0.05$. Finally, we calculate one-dimensional (1D) spectra as
functions of $k\equiv\sqrt{k_{x}^{2}+k_{z}^{2}}$ by integrating over the
azimuthal direction on the $k_{x}-k_{z}$ plane.

Figure 6 shows the time evolution of the resulting 1D spectra of
$\tilde{E_{k}}$ and $\tilde{E_{m}}$ for the case of $d_{i}=0.002$. Both
spectra exhibit qualitatively similar behaviors and remain close to
equipartition between the two. When the plasmoid instability occurs at about
$t=0.6$, the energy spectra peak at $k\simeq 100$. Subsequently, the energy
cascades in both forward and inverse directions to smaller and larger scales
and quickly fills a broad range of scales at around $t=1.0$. However, because
the reconnection geometry eventually evolves to a single-X-line configuration,
the energy spectra do not settle to a quasi-steady state. Instead, the
fluctuation parts of the energies peak at around $t=1.0$, then gradually decay
as the reconnection geometry becomes increasingly quasi-2D.

In comparison, the time evolution of the energy spectra for the case with
$d_{i}=0.001$, shown in Fig. 7, is qualitatively similar to the previous case
at an early stage. However, because this case does not settle to a single-X-
line configuration, the reconnection layer appears to realize a self-generated
turbulent state, where the energy spectra become nearly time-independent after
$t=1.7$.

During the quasi-steady phase, the kinetic and magnetic energy spectra in
Figure 6 approximately follow the $k^{-5/3}$ power law in the MHD range when
$kd_{i}<O(1)$. The spectra steepen in the sub-$d_{i}$ range when
$kd_{i}>O(1)$. As a reference, we plot the $k^{-11/3}$ power law in the
sub-$d_{i}$ range, which is the prediction of Galtier and Buchlin.[52]
However, because we do not have a sufficient separation between the ion skin
depth $d_{i}$ and the dissipation scale, the energy spectra do not exhibit a
definitive power law in the sub-$d_{i}$ range. The same two power laws are
also shown in Figure 6 for reference, even though the energy spectra do not
reach a quasi-steady phase. Notably, the energy spectra in this case also
steepen in the sub-$d_{i}$ range.

Next, we further investigate the alignment of turbulence eddies with the local
magnetic field by calculating two-point structure functions of the kinetic and
magnetic fluctuations in terms of the parallel displacement $r_{\parallel}$
and the perpendicular displacement $r_{\perp}$ relative to the local magnetic
field; i.e.,
$F_{2}^{w}(r_{\parallel},r_{\perp})\equiv\left\langle\left|\boldsymbol{w}\left(\boldsymbol{x}+\boldsymbol{r}\right)-\boldsymbol{w}\left(\boldsymbol{x}\right)\right|^{2}\right\rangle$
and
$F_{2}^{B}(r_{\parallel},r_{\perp})\equiv\left\langle\left|\boldsymbol{B}\left(\boldsymbol{x}+\boldsymbol{r}\right)-\boldsymbol{B}\left(\boldsymbol{x}\right)\right|^{2}\right\rangle$.
Structure functions allow us to measure the scale dependence of the anisotropy
of turbulence eddies. In MHD turbulence, Goldreich & Sridhar (GS) theory [54,
55] predicts that turbulence eddies become increasingly more elongated along
the magnetic field at smaller scales. More precisely, GS theory predicts a
scale-dependent relationship $k_{\parallel}\sim k_{\perp}^{2/3}$ between the
wavenumbers parallel ($k_{\parallel}$) and perpendicular ($k_{\perp}$).

The GS theory relies on the critical balance condition, which assumes that the
time scale for the nonlinear energy cascade of an eddy in directions
perpendicular to the magnetic field balances the Alfvén wave propagation time
scale of the eddy along the field. However, in Hall MHD, the dispersive
whistler waves at sub-$d_{i}$ scales alter the parallel wave propagation time
and may cause different anisotropic scaling relations.[ChoL2004] At MHD scales
when $k_{\parallel}d_{i}\ll 1$, the whistler waves become the shear Alfvén
wave, and the GS theory may still be applicable.

The GS scaling relation $k_{\parallel}\sim k_{\perp}^{2/3}$ has been confirmed
by Cho and Vishniac for homogeneous MHD turbulence using two-point structure
function analysis.[53] In self-generated turbulence within reconnection
layers, however, previous resistive MHD studies have shown deviations from the
GS scaling relation.[33, 34] Here, we explore whether similar behavior is
observed in MHD-scale eddies for Hall MHD turbulent reconnection. In
particular, we investigate whether our simulations of inhomogeneous
reconnection layers in Hall MHD also exhibit deviations from the GS scaling
relation.

Here we adopt the procedure of Huang and Bhattacharjee (2016),[33] which is a
modification of the procedure of Cho and Vishniac (2000)[53] to accommodate
the inhomogeneous background of the reconnection layer. Due to the strong
inhomogeneity along the $y$ direction in our system, we calculate the
structure functions for each $x$–$z$ plane, restricting displacements to lie
within these individual planes, rather than using the full 3D space. For a
pair of points, we define the local magnetic field as the average of the
magnetic fields from the two points. To ensure consistency with displacements
limited to individual $x$–$z$ planes, we measure the parallel and
perpendicular components of the displacement relative to the in-plane
component of the local magnetic field. We compute the structure functions by
averaging over $10^{9}$ random pairs of points; the $x$ coordinates of these
points are limited to the range $-0.25\leq x\leq 0.25$.

We focus on the case with $d_{i}=0.001$ because it appears to realize a self-
generated turbulent state in the reconnection layer. In the following
discussion, we present the results from the snapshot at $t=2.1$, but other
snapshots during the quasi-steady period show similar behaviors. Figure 8
summarizes the outcome of this analysis. Here, the rows correspond to
different $x$-$z$ slices. The first row shows the results at $y=0$, the second
row at $y=0.002$, and the third row at $y=0.005$, respectively. The first
column of the figure shows the magnitude of the magnetic field fluctuation
$\tilde{B}$, overlaid with streamlines of the in-plane magnetic field. The
magnetic field fluctuations form elongated eddies along the direction of the
local magnetic field. The second column shows contours of the structure
function $F_{2}^{B}(r_{\parallel},r_{\perp})$, which clearly show elongated
eddies along the local magnetic field direction. We have also performed the
same analysis for the weighted velocity field fluctuation $\tilde{w}$ and the
corresponding structure function $F_{2}^{w}(r_{\parallel},r_{\perp})$. The
results are qualitatively similar to those of the magnetic field fluctuations;
therefore, they are not shown. The third column shows the relations between
the semi-major axes $r_{\parallel}\sim k_{\parallel}^{-1}$ and the semi-minor
axes $r_{\perp}\sim k_{\perp}^{-1}$ of contours of the structure functions
$F_{2}^{B}(r_{\parallel},r_{\perp})$ and $F_{2}^{w}(r_{\parallel},r_{\perp})$.
These relations quantify the scale dependence of turbulent eddy anisotropy.
The two dashed lines represent the scalings of $k_{\parallel}\sim k_{\perp}$
(scale-independent) and $k_{\parallel}\sim k_{\perp}^{2/3}$ (GS theory), for
reference. In the first two rows at $y=0$ and $y=0.002$, relatively close to
the mid-plane, the relations between $r_{\parallel}$ and $r_{\perp}$ appear to
be approximately scale-independent. This result is similar to previous
resistive MHD results.[33, 34] Interestingly, as we move further away from the
mid-plane to $y=0.005$ (last row), the GS scaling relation $k_{\parallel}\sim
k_{\perp}^{2/3}$ is partially recovered and appears to be consistent with
eddies much larger than the $d_{i}$ scale.

##  V Discussion and Conclusion

In summary, our simulations show that the occurrence of single-X-line Hall
reconnection in 3D is less probable compared to 2D. Whether the reconnection
realizes a self-generated turbulent state or a single-X-line configuration in
3D Hall MHD depends on plasma parameters, including the Lundquist number $S$
and the system size to kinetic scale ratio $\Lambda=L/d_{i}$. Among all the
cases we have examined, the single-X-line configuration yields the highest
reconnection rate.

The self-generated turbulent reconnection in 3D Hall MHD exhibits significant
qualitative differences compared to resistive MHD. The turbulent reconnection
layer in Hall MHD is significantly broader, and the reconnection rate is much
faster. Regardless of whether the reconnection geometry settles to a single
X-line or becomes turbulent, the hallmarks of Hall reconnection remain
visible. Notably, the reconnection exhaust regions open up and form a
Petschek-like configuration (see Figures 3 and 4). We emphasize that the
resemblance with the Petschek-like configuration is just geometrical and
nothing more because the underlying Hall MHD dynamics is qualitatively
different from the resistive MHD dynamics. Nonetheless, our simulation results
and future research for even larger system sizes may shed light on recent
Parker Solar Probe in situ measurements of reconnection exhausts associated
with interplanetary coronal mass ejections and heliospheric current sheets,
where bifurcated current sheets resembling Petschek’s reconnection model with
a pair of slow-shocks or rotational discontinuities have been reported. [56]

Many questions remain open, particularly for applications to natural
reconnection events where the system sizes are much larger than those
considered in this study (i.e., $\Lambda=L/d_{i}\ggg 1$). Will a larger system
size make self-generated turbulent reconnection easier to realize? Do global
reconnection rate and geometry depend on the microscopic description of
reconnection physics (e.g., Hall MHD or fully kinetic PIC) when the scale
separation between $L$ and $d_{i}$ is large? In the future, the Hall MHD
description of self-generated turbulent reconnection should be further
compared with fully-kinetic PIC simulations or more elaborated fluid models
such as high-moment multi-fluid models,[57, 58] in particular in the regime
when the system size is much larger than kinetic scales.

The self-generated turbulence in the reconnection layer also poses new
challenges to turbulence theories and future numerical simulations. Our Hall
simulation shows that the turbulent state in the vicinity of the reconnection
mid-plane does not satisfy the Goldreich-Sridhar (GS) scaling relation of
turbulence eddy anisotropy. This result is similar to previous findings in
resistive MHD simulations. However, away from the mid-plane, the GS scaling
relation is partially recovered for large eddies. This phenomenon has not been
observed in resistive MHD simulations, and is not predicted by any theory.

Why the GS scaling relation is not satisfied in resistive MHD self-generated
turbulent reconnection but is partially recovered in Hall MHD remains an open
question. Previously, Huang and Bhattacharjee argued that the discrepancy may
be because the reconnection layer is highly inhomogeneous, with the
reconnection outflow and the magnetic field being strongly sheared.[33] In
Hall MHD turbulent reconnection, the turbulent region is significantly thicker
than the resistive MHD counterpart. Because the magnetic field is not as
strongly sheared at planes away from the mid-plane, this observation could
explain why the GS scaling relation is partially recovered in places away from
the mid-plane.

Even within the MHD regime, whether self-generated turbulent reconnection
deviates from the GS theory has been unsettled. In a previous study, Kowal et
al. showed that the scaling relation for large-scale eddies in turbulent
reconnection simulations satisfies the GS scaling, while small-scale eddies
are “contaminated” by reconnection and exhibit different scaling
behaviors.[KowalFLV2017] However, the initial and boundary conditions in their
study differ from those in the present study: the initial condition contains a
tangential discontinuity rather than a smooth magnetic field; the strength of
the guide field is considerably lower than that of the reconnecting component
of the magnetic field; the boundary conditions are periodic in the outflow
direction and open in the inflow direction. It is unclear how much of their
findings can be ascribed to these distinctions.

It is worth mentioning that the present study uses a compressible code. The
relatively high plasma $\beta$ ($\approx 4$ relative to the reconnecting
component of the magnetic field) and the presence of a guide field keep the
plasma close to, but not perfectly, incompressible. Throughout the
simulations, the root-mean-square density fluctuation is approximately $3\%$,
although the density can differ from the mean value by up to $10\%$ at some
locations. Even though the compressibility of the plasma is relatively low,
the compressible waves within the system may introduce subtle effects on the
energy cascade and eddy anisotropy.[Galtier2023]

In order to assess the effects of compressibility, turbulent reconnection
should be investigated with incompressible MHD and Hall MHD, as well as
compressible systems with varying plasma $\beta$. Furthermore, simulations
with varying guide field strengths will clarify magnetic shear effects. In a
broader context, it is also of great interest to further investigate how Hall
and kinetic effects affect plasmoid-mediated energy cascade in homogeneous
turbulence, which is an important topic of current research.[59, 60, 61, 62,
63]

###### Acknowledgements.

This research was supported by the U.S. Department of Energy, grant number DE-
SC0021205, the National Science Foundation, grant numbers AGS-2301337 and
2209471, and the National Aeronautics and Space Administration, grant number
80NSSC18K1285. Computations were performed on facilities at the National
Energy Research Scientific Computing Center and the National Center for
Atmospheric Research.

## References

  * Biskamp [2000] D. Biskamp,  _Magnetic Reconnection in Plasmas_ (Cambridge University Press, 2000).
  * Priest and Forbes [2000] E. R. Priest and T. Forbes,  _Magnetic reconnection : MHD theory and applications_ (Cambridge University Press, 2000).
  * Zweibel and Yamada [2009] E. G. Zweibel and M. Yamada, Annu. Rev. Astron. Astrophys. 47, 291 (2009).
  * Yamada  _et al._ [2010] M. Yamada, R. Kulsrud, and H. Ji, Rev. Mod. Phys. 82, 603 (2010).
  * Zweibel and Yamada [2016] E. G. Zweibel and M. Yamada, Proc. R Soc. A 472, 20160479 (2016).
  * Pontin and Priest [2022] D. I. Pontin and E. R. Priest, Living Reviews in Solar Physics 19, 1 (2022).
  * Ji  _et al._ [2022] H. Ji, W. Daughton, J. Jara-Almonte, A. Le, A. Stanier, and J. Yoo, Nature Reviews Physics 4, 263 (2022).
  * Loureiro  _et al._ [2007] N. F. Loureiro, A. A. Schekochihin, and S. C. Cowley, Phys. Plasmas 14, 100703 (2007).
  * Bhattacharjee  _et al._ [2009] A. Bhattacharjee, Y.-M. Huang, H. Yang, and B. Rogers, Phys. Plasmas 16, 112102 (2009).
  * Biskamp [1986] D. Biskamp, Phys. Fluids 29, 1520 (1986).
  * Shibata and Tanuma [2001] K. Shibata and S. Tanuma, Earth Planets Space 53, 473 (2001).
  * Lapenta [2008] G. Lapenta, Phys. Rev. Lett. 100, 235001 (2008).
  * Cassak  _et al._ [2009] P. A. Cassak, M. A. Shay, and J. F. Drake, Phys. Plasmas 16, 120702 (2009).
  * Huang and Bhattacharjee [2010] Y.-M. Huang and A. Bhattacharjee, Phys. Plasmas 17, 062104 (2010).
  * Bárta  _et al._ [2011] M. Bárta, J. Büchner, M. Karlický, and J. Skála, Astrophys. J. 737, 24 (2011).
  * Huang  _et al._ [2011] Y.-M. Huang, A. Bhattacharjee, and B. P. Sullivan, Phys. Plasmas 18, 072109 (2011).
  * Shen  _et al._ [2011] C. Shen, J. Lin, and N. A. Murphy, Astrophys. J. 737, 14 (2011).
  * Loureiro  _et al._ [2012] N. F. Loureiro, R. Samtaney, A. A. Schekochihin, and D. A. Uzdensky, Phys. Plasmas 19, 042303 (2012).
  * Ni  _et al._ [2012] L. Ni, U. Ziegler, Y.-M. Huang, J. Lin, and Z. Mei, Phys. Plasmas 19, 072902 (2012).
  * Huang and Bhattacharjee [2012] Y.-M. Huang and A. Bhattacharjee, Phys. Rev. Lett. 109, 265002 (2012), arXiv:1211.6708 .
  * Huang and Bhattacharjee [2013] Y.-M. Huang and A. Bhattacharjee, Phys. Plasmas 20, 055702 (2013).
  * Takamoto [2013] M. Takamoto, Astrophys. J. 775, 50 (2013).
  * Wyper and Pontin [2014] P. F. Wyper and D. I. Pontin, Phys. Plasmas 21, 082114 (2014).
  * Shepherd and Cassak [2010] L. S. Shepherd and P. A. Cassak, Phys. Rev. Lett. 105, 015004 (2010).
  * Baalrud  _et al._ [2011] S. D. Baalrud, A. Bhattacharjee, Y.-M. Huang, and K. Germaschewski, Phys. Plasmas 18, 092108 (2011).
  * Daughton  _et al._ [2006] W. Daughton, J. Scudder, and H. Karimabadi, Phys. Plasmas 13, 072101 (2006).
  * Daughton  _et al._ [2009] W. Daughton, V. Roytershteyn, B. J. Albright, H. Karimabadi, L. Yin, and K. J. Bowers, Phys. Rev. Lett. 103, 065004 (2009).
  * Daughton  _et al._ [2011] W. Daughton, V. Roytershteyn, H. Karimabadi, L. Yin, B. J. Albright, B. Bergen, and K. J. Bower, Nature Physics 7, 539 (2011).
  * Fermo  _et al._ [2012] R. L. Fermo, J. F. Drake, and M. Swisdak, Phys. Rev. Lett. 108, 255005 (2012).
  * Daughton  _et al._ [2014] W. Daughton, T. K. M. Nakamura, H. Karimabadi, V. Roytershteyn, and B. Loring, Phys. Plasmas 21, 052307 (2014).
  * Stanier  _et al._ [2019] A. Stanier, W. Daughton, A. Le, X. Li, and R. Bird, Physics of Plasmas 26, 072121 (2019).
  * Baalrud  _et al._ [2012] S. D. Baalrud, A. Bhattacharjee, and Y.-M. Huang, Phys. Plasmas 19, 022101 (2012).
  * Huang and Bhattacharjee [2016] Y.-M. Huang and A. Bhattacharjee, Astrophys. J. 818, 20 (2016).
  * Yang  _et al._ [2020] L. Yang, H. Li, F. Guo, X. Li, S. Li, J. He, L. Zhang, and X. Feng, The Astrophysical Journal 901, L22 (2020).
  * Beg  _et al._ [2022] R. Beg, A. J. B. Russell, and G. Hornig, The Astrophysical Journal 940, 94 (2022).
  * Ji and Daughton [2011] H. Ji and W. Daughton, Phys. Plasmas 18, 111207 (2011).
  * Cassak and Drake [2013] P. A. Cassak and J. F. Drake, Phys. Plasmas 20, 061207 (2013).
  * Huang  _et al._ [2019] Y.-M. Huang, L. Comisso, and A. Bhattacharjee, Physics of Plasmas 26, 092112 (2019).
  * Guo  _et al._ [2013] L.-J. Guo, A. Bhattacharjee, and Y.-M. Huang, Astrophys. J. Lett. 771, L14 (2013).
  * Lin  _et al._ [2015] J. Lin, N. A. Murphy, C. Shen, J. C. Raymond, K. K. Reeves, J. Zhong, N. Wu, and Y. Li, Space Sci. Rev. 194, 237 (2015).
  * Innes  _et al._ [2015] D. E. Innes, L.-J. Guo, Y.-M. Huang, and A. Bhattacharjee, Astrophys. J. 813, 86 (2015).
  * Young  _et al._ [2018] P. R. Young, H. Tian, H. Peter, R. J. Rutten, C. J. Nelson, Z. Huang, B. Schmieder, G. J. M. Vissers, S. Toriumi, L. H. M. R. van der Voort, M. S. Madjarska, S. Danilovic, A. Berlicki, L. P. Chitta, M. C. M. Cheung, C. Madsen, K. P. Reardon, Y. Katsukawa, and P. Heinzel, Space Science Reviews 214, 10.1007/s11214-018-0551-0 (2018).
  * Guo  _et al._ [2020] L.-J. Guo, B. D. Pontieu, Y.-M. Huang, H. Peter, and A. Bhattacharjee, The Astrophysical Journal 901, 148 (2020).
  * Birn  _et al._ [2001] J. Birn, J. F. Drake, M. A. Shay, B. N. Rogers, R. E. Denton, M. Hesse, M. Kuznetsova, Z. W. Ma, A. Bhattacharjee, A. Otto, and P. L. Pritchett, J. Geophys. Res. 106, 3715 (2001).
  * Lin  _et al._ [2008] J. Lin, S. R. Cranmer, and C. J. Farrugia, J. Geophys. Res. 113, A11107 (2008).
  * Huang  _et al._ [2013] Y.-M. Huang, A. Bhattacharjee, and T. G. Forbes, Phys. Plasmas 20, 082131 (2013), arXiv:1308.1871 .
  * Huang  _et al._ [2017] Y.-M. Huang, L. Comisso, and A. Bhattacharjee, Astrophys. J. 849, 75 (2017), arXiv:1707.01863 .
  * Guzdar  _et al._ [1993] P. N. Guzdar, J. F. Drake, D. McCarthy, A. B. Hassam, and C. S. Liu, Phys. Fluids B 5, 3712 (1993).
  * Gottlieb  _et al._ [2001] S. Gottlieb, C.-W. Shu, and E. Tadmor, SIAM Review 43, 89 (2001).
  * Spiteri and Ruuth [2002] R. J. Spiteri and S. J. Ruuth, SIAM J. Numer. Anal. 40, 469 (2002).
  * McKechan  _et al._ [2010] D. J. A. McKechan, C. Robinson, and B. S. Sathyaprakash, Class. Quantum Grav. 27, 084020 (2010).
  * Galtier and Buchlin [2007] S. Galtier and E. Buchlin, The Astrophysical Journal 656, 560 (2007).
  * Cho and Vishniac [2000] J. Cho and E. T. Vishniac, Astrophys. J. 539, 273 (2000).
  * Goldreich and Sridhar [1995] P. Goldreich and S. Sridhar, Astrophys. J. 438, 763 (1995).
  * Goldreich and Sridhar [1997] P. Goldreich and S. Sridhar, Astrophys. J. 485, 680 (1997).
  * Phan  _et al._ [2020] T. D. Phan, S. D. Bale, J. P. Eastwood, B. Lavraud, J. F. Drake, M. Oieroset, M. A. Shay, M. Pulupa, M. Stevens, R. J. MacDowall, A. W. Case, D. Larson, J. Kasper, P. Whittlesey, A. Szabo, K. E. Korreck, J. W. Bonnell, T. D. de Wit, K. Goetz, P. R. Harvey, T. S. Horbury, R. Livi, D. Malaspina, K. Paulson, N. E. Raouafi, and M. Velli, The Astrophysical Journal Supplement Series 246, 34 (2020).
  * Wang  _et al._ [2015] L. Wang, A. H. Hakim, A. Bhattacharjee, and K. Germaschewski, Phys. Plasmas , 012108 (2015).
  * Ng  _et al._ [2015] J. Ng, Y.-M. Huang, A. Hakim, A. Bhattacharjee, A. Stanier, W. Daughton, L. Wang, and K. Germaschewski, Phys. Plasmas 22, 112104 (2015).
  * Boldyrev and Loureiro [2017] S. Boldyrev and N. F. Loureiro, Astrophys. J. 844, 125 (2017).
  * Mallet  _et al._ [2017] A. Mallet, A. A. Schekochihin, and B. D. G. Chandran, Mon. Not. R. Astr. Soc. 468, 4862 (2017).
  * Comisso  _et al._ [2018] L. Comisso, Y.-M. Huang, M. Lingam, E. Hirvijoki, and A. Bhattacharjee, Astrophys. J. 854, 103 (2018).
  * Dong  _et al._ [2018] C. Dong, L. Wang, Y.-M. Huang, L. Comisso, and A. Bhattacharjee, Phys. Rev. Lett. 121, 165101 (2018).
  * Dong  _et al._ [2022] C. Dong, L. Wang, Y.-M. Huang, L. Comisso, T. A. Sandstrom, and A. Bhattacharjee, Science Advances 8, 10.1126/sciadv.abn7627 (2022).


# Making Monolingual Sentence Embeddings Multilingual using Knowledge
Distillation

Nils Reimers and Iryna Gurevych
Ubiquitous Knowledge Processing Lab (UKP-TUDA)
Department of Computer Science, Technische Universität Darmstadt
www.ukp.tu-darmstadt.de

###### Abstract

We present an easy and efficient method to extend existing sentence embedding
models to new languages. This allows to create multilingual versions from
previously monolingual models. The training is based on the idea that a
translated sentence should be mapped to the same location in the vector space
as the original sentence. We use the original (monolingual) model to generate
sentence embeddings for the source language and then train a new system on
translated sentences to mimic the original model. Compared to other methods
for training multilingual sentence embeddings, this approach has several
advantages: It is easy to extend existing models with relatively few samples
to new languages, it is easier to ensure desired properties for the vector
space, and the hardware requirements for training are lower. We demonstrate
the effectiveness of our approach for 50+ languages from various language
families. Code to extend sentence embeddings models to more than 400 languages
is publicly available.111Code, models, and datasets:
https://github.com/UKPLab/sentence-transformers

##  1 Introduction

Mapping sentences or short text paragraphs to a dense vector space, such that
similar sentences are close, has wide applications in NLP. However, most
existing sentence embeddings models are monolingual, usually only for English,
as applicable training data for other languages is scarce. For multi- and
cross-lingual scenarios, only few sentence embeddings models exist.

In this publication, we present a new method that allows us to extend existing
sentence embeddings models to new languages. We require a teacher model $M$
for source language $s$ and a set of parallel (translated) sentences
$((s_{1},t_{1}),...,(s_{n},t_{n}))$ with $t_{i}$ the translation of $s_{i}$.
Note, the $t_{i}$ can be in different languages. We train a new student model
$\hat{M}$ such that $\hat{M}(s_{i})\approx M(s_{i})$ and
$\hat{M}(t_{i})\approx M(s_{i})$ using mean squared loss. We call this
approach multilingual knowledge distillation, as the student $\hat{M}$
distills the knowledge of the teacher $M$ in a multilingual setup. We
demonstrate that this type of training works for various language combinations
as well as for multilingual setups. We observe an especially high improvement
of up to 40 accuracy points for low resource languages compared to LASER
Artetxe and Schwenk (2019b).

The student model $\hat{M}$ learns a multilingual sentence embedding space
with two important properties: 1) Vector spaces are aligned across languages,
i.e., identical sentences in different languages are close, 2) vector space
properties in the original source language from the teacher model $M$ are
adopted and transferred to other languages.

Figure 1: Given parallel data (e.g. English and German), train the student
model such that the produced vectors for the English and German sentences are
close to the teacher English sentence vector.

The presented approach has various advantages compared to other training
approaches for multilingual sentence embeddings. LASER Artetxe and Schwenk
(2019b) trains an encoder-decoder LSTM model using a translation task. The
output of the encoder is used as sentence embedding. While LASER works well
for identifying exact translations in different languages, it works less well
for assessing the similarity of sentences that are not exact translations.

Multilingual Universal Sentence Encoder (mUSE) Chidambaram et al. (2019); Yang
et al. (2019) was trained in a multi-task setup on SNLI Bowman et al. (2015)
and on over a billion question-answer pairs from popular online forums and QA
websites. In order to align the cross-lingual vector spaces, mUSE used a
translation ranking task. Given a translation pair $(s_{i},t_{i})$ and various
alternative (incorrect) translations, identify the correct translation. First,
multi-task learning is difficult since it can suffer from catastrophic
forgetting and balancing multiple tasks is not straight-forward. Further,
running the translation ranking task is complex and results in a huge
computational overhead. Selecting random alternative translations usually
leads to mediocre results. Instead, hard negatives Guo et al. (2018) are
required, i.e., alternative incorrect translations that have a high similarity
to the correct translation. To get these hard negatives, mUSE was first
trained with random negatives samples, then, this preliminary sentence encoder
was used to identify hard negative examples. They then re-trained the network.

In this work, we use Sentence-BERT (SBERT) Reimers and Gurevych (2019), which
achieves state-of-the-art performance for various sentence embeddings task.
SBERT is based on transformer models like BERT Devlin et al. (2018) and
applies mean pooling on the output. In our experiments we use XLM-R Conneau et
al. (2019), a pre-trained network on 100 languages, as student model. Note,
the described approach is not limited to be used with transformer models and
should also work with other network architectures.

##  2 Training

We require a teacher model $M$, that maps sentences in one or more source
languages $s$ to a dense vector space. Further, we need parallel (translated)
sentences $((s_{1},t_{1}),...,(s_{n},t_{n}))$ with $s_{i}$ a sentence in one
of the source languages and $t_{i}$ a sentence in one of the target languages.

We train a student model $\hat{M}$ such that $\hat{M}(s_{i})\approx M(s_{i})$
and $\hat{M}(t_{i})\approx M(s_{i})$. For a given mini-batch $\mathcal{B}$, we
minimize the mean-squared loss:

$\frac{1}{|\mathcal{B}|}\sum_{j\in\mathcal{B}}\left[(M(s_{j})-\hat{M}(s_{j}))^{2}+(M(s_{j})-\hat{M}(t_{j}))^{2}\right]$

$\hat{M}$ could have the structure and the weights of $M$, or it can be a
different network architecture. This training procedure is illustrated in
Figure 1. We denote trained models with $\hat{M}\leftarrow M$, as the student
model $\hat{M}$ learns the representation of the teacher model $M$.

In our experiments, we mainly use an English SBERT model as teacher model $M$
and use XLM-RoBERTa (XLM-R) as student model $\hat{M}$. The English BERT
models have a wordpiece vocabulary size of 30k mainly consisting of English
tokens. Using the English SBERT model as initialization for $\hat{M}$ would be
suboptimal, as most words in other latin-based languages would be broken down
to short character sequences, and words in non-latin alphabets would be mapped
to the UNK token. In contrast, XLM-R uses
SentencePiece222https://github.com/google/sentencepiece, which avoids language
specific pre-processing. Further, it uses a vocabulary with 250k entries from
100 different languages. This makes XLM-R much more suitable for the
initialization of the multilingual student model.

##  3 Training Data

In this section, we evaluate the importance of training data for making the
sentence embedding model multilingual. The OPUS website333http://opus.nlpl.eu/
Tiedemann (2012) provides parallel data for hundreds of language pairs. In our
experiments, we use the following datasets:

  * •

GlobalVoices: A parallel corpus of news stories from the web site Global
Voices.

  * •

TED2020: We crawled the translated subtitles for about 4,000 TED talks,
available in over 100 languages. This dataset is available in our repository.

  * •

NewsCommentary: Political and economic commentary crawled from the web site
Project Syndicate, provided by WMT.

  * •

WikiMatrix: Mined parallel sentences from Wikipedia in different languages
Schwenk et al. (2019). We only used pairs with scores above 1.05, as pairs
below this threshold were often of bad quality.

  * •

Tatoeba: Tatoeba444https://tatoeba.org/ is a large database of example
sentences and translations to support language learning.

  * •

Europarl: Parallel sentences extracted from the European Parliament website
Koehn (2005).

  * •

JW300: Mined, parallel sentences from the magazines Awake! and Watchtower Agić
and Vulić (2019).

  * •

OpenSubtitles2018: Translated movie subtitles from opensubtitles.org Lison and
Tiedemann (2016).

  * •

UNPC: Manually translated United Nations documents from 1994 - 2014 Ziemski et
al. (2016).

Getting parallel sentence data can be challenging for some low-resource
language pairs. Hence, we also experiment with bilingual dictionaries:

  * •

MUSE: MUSE555https://github.com/facebookresearch/MUSE provides 110 large-scale
ground-truth bilingual dictionaries created by an internal translation tool
Conneau et al. (2017b).

  * •

Wikititles: We use the Wikipedia database dumps to extract the article titles
from cross-language links between Wikipedia articles. For example, the page
”United States” links to the German page ”Vereinigte Staaten”. This gives a
dictionary covering a wide range of topics.

The data set sizes for English-German (EN-DE) and English-Arabic (EN-AR) are
depicted in Table 5. For training, we balance the data set sizes by drawing
for a mini batch roughly the same number of samples from each data set. Data
from smaller data sets is repeated.

We trained XLM-R as our student model and used SBERT fine-tuned on English NLI
and STS data666bert-base-nli-stsb-mean-tokens model from our repository as our
teacher model. We trained for a maximum of 20 epochs with batch size 64,
10,000 warm-up steps, and a learning rate of 2e-5. As development set, we
measured the MSE loss on hold-out parallel sentences.

In Reimers and Gurevych (2017, 2018), we showed that the random seed can have
a large impact on the performances of trained models, especially for small
datasets. In the following experiments, we have quite large datasets of up to
several million parallel sentences and we observed rather minor differences
($\sim$ 0.3 score points) between random seeds.

##  4 Experiments

In this section, we conduct experiments on three tasks: Multi- and cross-
lingual semantic textual similarity (STS), bitext retrieval, and cross-lingual
similarity search. STS assigns a score for a pair of sentences, while bitext
retrieval identifies parallel (translated) sentences from two large
monolingual corpora.

Note, evaluating the capability of different strategies to align vector spaces
across languages is non-trivial. The performance for cross-lingual tasks
depends on the ability to map sentences across languages to one vector space
(usually the vector space for English) as well as on the properties this
source vector space has. Differences in performance can then be due to a
better or worse alignment between the languages or due to different properties
of the (source) vector space.

We evaluate the following systems:

SBERT-nli-stsb: The output of the BERT-base model is combined with mean
pooling to create a fixed-sized sentence representation Reimers and Gurevych
(2019). It was fine-tuned on the English AllNLI (SNLI Bowman et al. (2015) and
Multi-NLI Williams et al. (2018)) dataset and on the English training set of
the STS benchmark Cer et al. (2017) using a siamese network structure.

mBERT / XLM-R mean: Mean pooling of the outputs for the pre-trained
multilingual BERT (mBERT) and XLM-R model. These models are pre-trained on
multilingual data and have a multilingual vocabulary. However, no parallel
data was used.

mBERT- / XLM-R-nli-stsb: We fine-tuned XLM-R and mBERT on the (English) AllNLI
and the (English) training set of the STS benchmark.

LASER: LASER Artetxe and Schwenk (2019b) uses max-pooling over the output of a
stacked LSTM-encoder. The encoder was trained in an encoder-decoder setup
(machine translation setup) on parallel corpora over 93 languages.

mUSE: Multilingual Universal Sentence Encoder Chidambaram et al. (2019) uses a
dual-encoder transformer architecture and was trained on mined question-answer
pairs, SNLI data, translated SNLI data, and parallel corpora over 16
languages.

LaBSE: Language-agnostic BERT Sentence Embedding (LaBSE) Feng et al. (2020)
was trained similar to mUSE with a dual-encoder transformer architecture based
on BERT with 6 Billion translation pairs for 109 languages.

mBERT- / DistilmBERT- / XLM-R $\leftarrow$ SBERT-nli-stsb: We learn mBERT,
DistilmBERT, and XLM-R to imitate the output of the English SBERT-nli-stsb
model.

XLM-R $\leftarrow$ SBERT-paraphrases: We train XLM-R to imitate SBERT-
paraphrases, a RoBERTa model trained on more than 50 Million English
paraphrase pairs.

For our multi-lingual knowledge distillation experiments, we trained a single
model with parallel data for 50 languages777ar, bg, ca, cs, da, de, el, es,
et, fa, fi, fr, fr-ca, gl, gu, he, hi, hr, hu, hy, id, it, ja, ka, ko, ku, lt,
lv, mk, mn, mr, ms, my, nb, nl, pl, pt, pt, pt-br, ro, ru, sk, sl, sq, sr, sv,
th, tr, uk, ur, vi, zh-cn, zh-tw.

###  4.1 Multilingual Semantic Textual Similarity

Model | EN-EN | ES-ES | AR-AR | Avg.
---|---|---|---|---
mBERT mean | 54.4 | 56.7 | 50.9 | 54.0
XLM-R mean | 50.7 | 51.8 | 25.7 | 42.7
mBERT-nli-stsb | 80.2 | 83.9 | 65.3 | 76.5
XLM-R-nli-stsb | 78.2 | 83.1 | 64.4 | 75.3
Knowledge Distillation
mBERT $\leftarrow$ SBERT-nli-stsb | 82.5 | 83.0 | 78.8 | 81.4
DistilmBERT $\leftarrow$ SBERT-nli-stsb | 82.1 | 84.0 | 77.7 | 81.2
XLM-R $\leftarrow$ SBERT-nli-stsb | 82.5 | 83.5 | 79.9 | 82.0
XLM-R $\leftarrow$ SBERT-paraphrases | 88.8 | 86.3 | 79.6 | 84.6
Other Systems
LASER | 77.6 | 79.7 | 68.9 | 75.4
mUSE | 86.4 | 86.9 | 76.4 | 83.2
LaBSE | 79.4 | 80.8 | 69.1 | 76.4
Table 1: Spearman rank correlation $\rho$ between the cosine similarity of sentence representations and the gold labels for STS 2017 dataset. Performance is reported by convention as $\rho\times 100$. Model | EN-AR | EN-DE | EN-TR | EN-ES | EN-FR | EN-IT | EN-NL | Avg.
---|---|---|---|---|---|---|---|---
mBERT mean | 16.7 | 33.9 | 16.0 | 21.5 | 33.0 | 34.0 | 35.6 | 27.2
XLM-R mean | 17.4 | 21.3 | 9.2 | 10.9 | 16.6 | 22.9 | 26.0 | 17.8
mBERT-nli-stsb | 30.9 | 62.2 | 23.9 | 45.4 | 57.8 | 54.3 | 54.1 | 46.9
XLM-R-nli-stsb | 44.0 | 59.5 | 42.4 | 54.7 | 63.4 | 59.4 | 66.0 | 55.6
Knowledge Distillation
mBERT $\leftarrow$ SBERT-nli-stsb | 77.2 | 78.9 | 73.2 | 79.2 | 78.8 | 78.9 | 77.3 | 77.6
DistilmBERT $\leftarrow$ SBERT-nli-stsb | 76.1 | 77.7 | 71.8 | 77.6 | 77.4 | 76.5 | 74.7 | 76.0
XLM-R $\leftarrow$ SBERT-nli-stsb | 77.8 | 78.9 | 74.0 | 79.7 | 78.5 | 78.9 | 77.7 | 77.9
XLM-R $\leftarrow$ SBERT-paraphrases | 82.3 | 84.0 | 80.9 | 83.1 | 84.9 | 86.3 | 84.5 | 83.7
Other Systems
LASER | 66.5 | 64.2 | 72.0 | 57.9 | 69.1 | 70.8 | 68.5 | 67.0
mUSE | 79.3 | 82.1 | 75.5 | 79.6 | 82.6 | 84.5 | 84.1 | 81.1
LaBSE | 74.5 | 73.8 | 72.0 | 65.5 | 77.0 | 76.9 | 75.1 | 73.5
Table 2: Spearman rank correlation $\rho$ between the cosine similarity of
sentence representations and the gold labels for STS 2017 dataset. Performance
is reported by convention as $\rho\times 100$.

The goal of semantic textual similarity (STS) is to assign for a pair of
sentences a score indicating their semantic similarity. For example, a score
of 0 indicates not related and 5 indicates semantically equivalent.

The multilingual STS 2017 dataset Cer et al. (2017) contains annotated pairs
for EN-EN, AR-AR, ES-ES, EN-AR, EN-ES, EN-TR. We extend this dataset by
translating one sentence of each pair in the EN-EN dataset to German. Further,
we use Google Translate to create the datasets EN-FR, EN-IT, and EN-NL.
Samples of these machine translated versions have been checked by humans
fluent in that language.

For the generate sentence embeddings we compute cosine similarity and, as
recommended in Reimers et al. (2016), compute the Spearman’s rank correlation
$\rho$ between the computed score and the gold score.

Table 1 shows the results for the monolingual setup and Table 2 the cross-
lingual setup.

As shown before Reimers and Gurevych (2019), using mBERT / XLM-R without fine-
tuning yields rather poor performance. Training on English NLI & STS data
(mBERT/XLM-nli-stsb) significantly improves the performance also for the other
languages. While in the monolingual setup (Table 1) the performance is quite
competitive, we observe a significant drop for the cross-lingual setup (Table
2). This indicates that the vectors spaces are not well aligned across
languages.

Using our multilingual knowledge distillation approach, we observe state-of-
the-art performances for mono- as well as for the cross-lingual setup,
significantly outperforming other state-of-the-art models (LASER, mUSE,
LaBSE). Even though SBERT-nli-stsb was trained on the STSbenchmark train set,
we observe the best performance by SBERT-paraphrase, which was not trained
with any STS dataset. Instead, it was trained on a large and broad paraphrase
corpus, mainly derived from Wikipedia, which generalizes well to various
topics.

In our experiments, XLM-R is slightly ahead of mBERT and DistilmBERT. mBERT
and DistilmBERT use different language-specific tokenization tools, making
those models more difficult to be used on raw text. In contrast, XLM-R uses a
SentencePiece model that can be applied directly on raw text data for all
languages. Hence, in the following experiments we only report results for
XLM-R.

###  4.2 BUCC: Bitext Retrieval

Model | DE-EN | FR-EN | RU-EN | ZH-EN | Avg.
---|---|---|---|---|---
mBERT mean | 44.1 | 47.2 | 38.0 | 37.4 | 41.7
XLM-R mean | 5.2 | 6.6 | 22.1 | 12.4 | 11.6
mBERT-nli-stsb | 38.9 | 39.5 | 26.4 | 30.2 | 33.7
XLM-R-nli-stsb | 44.0 | 51.0 | 51.5 | 44.0 | 47.6
Knowledge Distillation
XLM-R $\leftarrow$ SBERT-nli-stsb | 86.8 | 84.4 | 86.3 | 85.1 | 85.7
XLM-R $\leftarrow$ SBERT-paraphrase | 90.8 | 87.1 | 88.6 | 87.8 | 88.6
Other systems
mUSE | 88.5 | 86.3 | 89.1 | 86.9 | 87.7
LASER | 95.4 | 92.4 | 92.3 | 91.7 | 93.0
LaBSE | 95.9 | 92.5 | 92.4 | 93.0 | 93.5
Table 3: $F_{1}$ score on the BUCC bitext mining task.

Bitext retrieval aims to identify sentence pairs that are translations in two
corpora in different languages. Guo et al. (2018) showed that computing the
cosine similarity of all sentence embeddings and to use nearest neighbor
retrieval with a threshold has certain issues.

For our experiments, we use the BUCC bitext retrieval code from
LASER888https://github.com/facebookresearch/LASER/ with the scoring function
from Artetxe and Schwenk (2019a):

$\displaystyle\text{score}(x,y)=\text{margin}(\text{cos}(x,y),$
$\displaystyle\sum_{z\in\text{NN}_{k}(x)}\frac{cos(x,z)}{2k}+\sum_{z\in\text{NN}_{k}(y)}\frac{cos(y,z)}{2k}$

with $x,y$ the two sentence embeddings and $\text{NN}_{k}(x)$ denoting the $k$
nearest neighbors of x in the other language999retrieved using using faiss:
https://github.com/facebookresearch/faiss. As margin function, we use
$\text{margin}(a,b)=a/b$.

We use the dataset from the BUCC mining task Zweigenbaum et al. (2017, 2018),
with the goal of extracting parallel sentences between an English corpus and
four other languages: German, French, Russian, and Chinese. The corpora
consist of 150K - 1.2M sentences for each language with about 2-3% of the
sentences being parallel. The data is split into training and test sets. The
training set is used to find a threshold for the score function. Pairs above
the threshold are returned as parallel sentences. Performance is measured
using $F_{1}$ score.

Results are shown in Table 3. Using mean pooling directly on mBERT / XLM-R
produces low scores. While training on English NLI and STS data improves the
performance for XLM-R (XLM-R-nli-stsb), it reduces the performance for mBERT.
It is unclear why mBERT mean and XLM-R mean produce vastly different scores
and why training on NLI data improves the cross-lingual performance for XLM-R,
while reducing the performance for mBERT. As before, we observe that mBERT /
XLM-R do not have well aligned vector spaces and training only on English data
is not sufficient.

Using our multilingual knowledge distillation method, we were able to
significantly improve the performance compared to the mBERT / XLM-R model
trained only on English data.

While LASER and LaBSE only achieve mediocre results on the STS 2017 dataset,
they achieve state-of-the-art performances on BUCC outperforming mUSE and our
approach. LASER and LaBSE were specifically designed to identify perfect
translations across languages. However, as the STS 2017 results show, these
models have issues assigning meaningful similarity scores for sentence pairs
that don’t have identical meaning.

In contrast, mUSE and our approach creates vector spaces such that
semantically similar sentences are close. However, sentences with similar
meanings must not be translations of each other. For example, in the BUCC
setup, the following pair is not labeled as parallel text:

  * •

Olympischen Jugend-Sommerspiele fanden vom 16. bis 28. August 2014 in Nanjing
(China) statt. (en: Summer Youth Olympic Games took place from August 16 to
28, 2014 in Nanjing (China))

  * •

China hosted the 2014 Youth Olympic Games.

Both sentences are semantically similar, hence our model and mUSE assign a
high similarity score. But the pair is not a translation, as some details are
missing (exact dates and location).

These results stress the point that there is no single sentence vector space
universally suitable for every application. For finding translation pairs in
two corpora, LASER and LaBSE would be the best choice. However, for the task
of finding semantically similar sentence pairs, our approach and mUSE would be
the better choices.

We noticed that several positive pairs are missing in the BUCC dataset. We
analyzed for SBERT, mUSE, and LASER 20 false positive DE-EN pairs each, i.e.,
we analyzed pairs with high similarities according to the embeddings method
but which are not translations according to the dataset. For 57 out of 60
pairs, we would judge them as valid, high-quality translations. This issue
comes from the way BUCC was constructed: It consists of a parallel part, drawn
from the News Commentary dataset, and sentences drawn from Wikipedia, which
are judged as non-parallel. However, it is not ensured that the sentences from
Wikipedia are in fact non-parallel. The systems successfully returned parallel
pairs from the Wikipedia part of the dataset. Results based on the BUCC
dataset should be judged with care. It is unclear how many parallel sentences
are in the Wikipedia part of the dataset and how this affects the scores.

###  4.3 Tatoeba: Similarity Search

In this section, we evaluate the strategy for lower resource languages, where
it can be especially challenging to get well-aligned sentence embeddings. For
evaluation, we use the Tatoeba test set-up from LASER Artetxe and Schwenk
(2019b): The dataset consists of up to 1,000 English-aligned sentence pairs
for various languages. Evaluation is done by finding for all sentences the
most similar sentence in the other language using cosine similarity. Accuracy
is computed for both directions (English to the other language and back).

As before, we fine-tune XLM-R with SBERT-nli-stsb as teacher model. As
training data, we use JW300, which covers over 300 languages. To make our
results comparable to LASER, we reduce the training data to the same amount as
used by LASER. We selected four languages with rather small parallel datasets:
Georgian (KA, 296k parallel sentence pairs), Swahili (SW, 173k), Tagalog (TL,
36k), and Tatar (TT, 119k). Tagalog and Tatar were not part of the 100
languages XLM-R was pre-trained for, i.e., XLM-R has no specific vocabulary
and the language model was not tuned for these languages.

As Table 4 shows, we observe a significant accuracy improvement compared to
LASER, indicating much better aligned vector spaces between English and these
languages. Even though Tagalog had the smallest dataset with 36k pairs and
that XLM-R was not pre-trained for this language, we achieve high accuracy
scores of 86.2 and 84.0. We conclude that our strategy also works for low
resource languages and can yield a significant improvement. The results for
all languages in the Tatoeba test set can be found in the appendix.

Model | KA | SW | TL | TT
---|---|---|---|---
LASER |  |
en $\to$ xx | 39.7 | 54.4 | 52.6 | 28.0
xx $\to$ en | 32.2 | 60.8 | 48.5 | 34.3
XLM-R $\leftarrow$ SBERT-nli-stsb |  |
en $\to$ xx | 73.1 | 85.4 | 86.2 | 54.5
xx $\to$ en | 71.7 | 86.7 | 84.0 | 52.3
Table 4: Accuracy on the Tatoeba test set in both directions (en to target
language and vice versa).

##  5 Evaluation of Training Datasets

To evaluate the suitability of the different training sets, we trained
bilingual XLM-R models for EN-DE and EN-AR on the described training datasets.
English and German are fairly similar languages and have a large overlap in
their alphabets, while English and Arabic are dissimilar languages with
distinct alphabets. We evaluate the performance on the STS 2017 dataset.

The results for training on the full datasets are shown in Table 5. Table 6
shows the results for training only on the first $k$ sentences of the TED2020
dataset.

First, we observe that the bilingual models are slightly better than the model
trained for 10 languages (section 4.1): 2.2 points improvement for EN-DE and
1.2 points improvement for EN-AR. Conneau et al. (2019) calls this curse of
multilinguality, where adding more languages to a model can degrade the
performance as the capacity of the model remains the same.

For EN-DE we observe only minor differences between the datasets. It appears
that the domain of the training data (news, subtitles, parliamentary debates,
magazines) is of minor importance. Further, only little training data is
necessary.

For the dissimilar languages English and Arabic, the results are less
conclusive. Table 5 shows that more data does not necessarily lead to better
results. With the Tatoeba dataset (only 27,000 parallel sentences), we achieve
a score of 76.7, while with the UNPC dataset (over 8 Million sentences), we
achieve only a score of 66.1. The domain of the parallel sentences is of
higher importance. The results on the reduced TED2020 dataset (Table 6) show
that the score improves slower for EN-AR than for EN-DE with more data.

Dataset | #DE | EN-DE | #AR | EN-AR
---|---|---|---|---
XLM-R mean | - | 21.3 | - | 17.4
XLM-R-nli-stsb | - | 59.5 | - | 44.0
MUSE Dict | 101k | 75.8 | 27k | 68.8
Wikititles Dict | 545k | 71.4 | 748k | 67.9
MUSE + Wikititles | 646k | 76.0 | 775k | 69.1
GlobalVoices | 37k | 78.1 | 29k | 68.6
TED2020 | 483k | 80.4 | 774k | 78.0
NewsCommentary | 118k | 77.7 | 7k | 57.4
WikiMatrix | 276k | 79.4 | 385k | 75.4
Tatoeba | 303k | 79.5 | 27k | 76.7
Europarl | 736k | 78.7 | - | -
JW300 | 1,399k | 80.0 | 382k | 74.0
UNPC | - | - | 8M | 66.1
OpenSubtitles | 21M | 79.8 | 28M | 78.8
All datasets | 25M | 81.4 | 38M | 79.0
Table 5: Data set sizes for the EN-DE / EN-AR sections. Performance (Spearman rank correlation) of XLM-R $\leftarrow$ SBERT-nli-stsb on the STS 2017 dataset. Dataset size | EN-DE | EN-AR
---|---|---
XLM-R mean | 21.3 | 17.4
XLM-R-nli-stsb | 59.5 | 44.0
1k | 71.5 | 48.4
5k | 74.5 | 59.6
10k | 77.0 | 69.5
25k | 80.0 | 70.2
Full TED2020 | 80.4 | 78.0
Table 6: Performance on STS 2017 dataset when trained with reduced TED2020
dataset sizes.

##  6 Target Language Training

In this section we evaluate whether it is better to transfer an English model
to a certain target language or if training from-scratch on suitable datasets
in the target language yields better results.

For this, we use the KorNLI and KorSTS datasets from Ham et al. (2020). They
translated the English SNLI Bowman et al. (2015), MultiNLI Williams et al.
(2018), and STSbenchmark (STSb) Cer et al. (2017) datasets to Korean with an
internal machine translation system. The dev and tests were post-edited by
professional translators.

Ham et al. fine-tuned Korean RoBERTa and XLM-R on these datasets using the
SBERT framework. We use the translated sentences they provide and tuned XLM-R
using multilingual knowledge distillation. We use SBERT-nli-stsb as teacher
model. Results are shown in Table 7.

Model | KO-KO
---|---
LASER | 68.44
mUSE | 76.32
Trained on KorNLI & KorSTS |
Korean RoBERTa-base | 80.29
Korean RoBERTa-large | 80.49
XLM-R | 79.19
XLM-R-large | 81.84
Multiling. Knowledge Distillation |
XLM-R $\leftarrow$ SBERT-nli-stsb | 81.47
XLM-R-large $\leftarrow$ SBERT-large-nli-stsb | 83.00
Table 7: Spearman rank correlation on Korean STSbenchmark test-set Ham et al.
(2020).

We observe a slight improvement of using multilingual knowledge distillation
over training the models directly on the translated NLI and STS data. This is
great news: Training on the Korean datasets yields a model only for Korean,
while with multilingual knowledge distillation, we get a model for English and
Korean with aligned vector spaces. Further, we do not necessarily have a
performance drop if there is only training data for the sentence embedding
method in English available.

##  7 Language Bias

Roy et al. (2020) introduces the concept of language bias: A model prefers one
language or language pair over others. For example, a model would have a
language bias if it maps sentences in the same language closer in vector space
just because they are of the same language. Language bias can be an issue if
the task involves a multilingual sentence pool: certain language pairs might
get discriminated, potentially harming the overall performance for
multilingual sentence pools.

Figure 2 shows the plot of the first two principle components for different
multi-lingual sentence embeddings methods. In the plot, we encoded the English
premise sentences from XNLI Conneau et al. (2018) with their Russian
translation. The plot shows for the LaBSE model a drastic separation between
the two languages, indicating that the language significantly impacts the
resulting embedding vector.

The experiments in Section 4 used so far mono-lingual sentence pools, i.e.,
all sentences in the source / target pool were of the same language. Hence,
these benchmarks are not suited to measure a potential harmful effect from
language bias. In order to measure a potential negative effect from language
bias, we combine all sentence pairs from the multilingual STS dataset and
compute similarity scores as described in Section 4.1. Note, in Section 4.1,
the models had only to score e.g. EN-AR sentence pairs. Now, there are 10
language combinations in one joined set. A model without language bias would
achieve on this joined set a performance similar to the average of the
performances over the individual subsets. However, if a model has a language
bias, sentence pairs from specific language combinations will be ranked higher
than others, lowering the Spearman rank correlation for the joint set.

The results are depicted in Table 8. We observe that LaBSE has a difference of
-1.29 and LASER has a difference of -0.92. Both scores are statistically
significant with confidence $p<0.001$. LASER and LaBSE both have a language
bias which decrease the performance on multilingual pools compared to mono-
lingual pools. In contrast, mUSE and the proposed multilingual knowledge
distillation have a minor, statistically insignificant language bias. There,
the performance for the joined set only decreases by -0.19 and -0.11 compared
to the evaluation on the individual sets.

In summary, mUSE and the proposed multilingual knowledge distillation approach
can be used on multilingual sentence pools without a negative performance
impact from language bias, while LASER and LaBSE prefer certain language
combinations over other, impacting the overall result.

Model | Expected Score | Actual Score | Difference
---|---|---|---
LASER | 69.5 | 68.6 | -0.92
mUSE | 81.7 | 81.6 | -0.19
LaBSE | 74.4 | 73.1 | -1.29
XLM-R $\leftarrow$ SBERT-paraphrases | 84.0 | 83.9 | -0.11
Table 8: Spearman rank correlation for the multilingual STS dataset. Expected
score is the average over the performance on the individual sets (Table 1 &
2). Actual score is the correlation for one joined set of sentence pairs.
Models without language bias would score on the joined set similar to the
average over the individual sets. The difference shows the negative impact
from the language bias.

##  8 Related Work

Sentence embeddings are a well studied area with dozens of proposed methods
Kiros et al. (2015); Conneau et al. (2017a); Cer et al. (2018); Yang et al.
(2018). Most of the methods have in common that they were only trained on
English. Multilingual representations have attracted significant attention in
recent times. Most of it focuses on cross-lingual word embeddings Ruder
(2017). A common approach is to train word embeddings for each language
separately and to learn a linear transformation that maps them to a shared
space based on a bilingual dictionary Artetxe et al. (2018). This mapping can
also be learned without parallel data Conneau et al. (2017b); Lample et al.
(2018).

A straightforward approach for creating cross-lingual sentence embeddings is
to use a bag-of-words representation of cross-lingual word embeddings.
However, Conneau et al. (2018) showed that this approach works poorly in
practical cross-lingual transfer settings. LASER Artetxe and Schwenk (2019b)
uses a sequence-to-sequence encoder-decoder architecture Sutskever et al.
(2014) based on LSTM networks. It trains on parallel corpora from neural
machine translation. To create a fixed sized sentence representation, they
apply max-pooling over the output of the encoder. LASER was trained for 93
languages on 16 NVIDIA V100 GPUs for about 5 days.

Multilingual Universal Sentence Encoder
(mUSE)101010https://tfhub.dev/google/universal-sentence-encoder Chidambaram et
al. (2019); Yang et al. (2019) uses a dual-encoder architecture. It was
trained in a multi-task setup on SNLI Bowman et al. (2015) and over 1 Billion
crawled question-answer pairs from various communities. A translation ranking
task was applied: Given a sentence in the source language and a set of
sentences in the target languages, identify the correct translation pair. To
work well, hard negative examples (similar, but incorrect translations) must
be included in the ranking task. mUSE was trained for 16 languages with 30
million steps. LaBSE Feng et al. (2020) is based on a BERT architecture and
used masked language model and 6 Billion translation pairs for training. It
was trained similar to mUSE with a translation ranking loss, however, without
any other training data.

In this publication, we extended Sentence-BERT (SBERT) Reimers and Gurevych
(2019). SBERT is based on transformer models like BERT Devlin et al. (2018)
and fine-tunes those using a siamese network structure. By using the pre-
trained weights from BERT, suitable sentence embeddings methods can be trained
efficiently. Multilingual BERT (mBERT) was trained on 104 languages using
Wikipedia, while XLM-R Conneau et al. (2019) was trained on 100 languages
using CommonCrawl. mBERT and XLM-R were not trained on any parallel data,
hence, their vector spaces are not aligned.

##  9 Conclusion

We presented a method to make monolingual sentence embeddings multilingual
with aligned vector spaces between the languages. This was achieved by using
multilingual knowledge distillation. We demonstrated that this approach
successfully transfers properties from the source language vector space (in
our case English) to various target languages. Models can be extended to
multiple languages in the same training process.

This stepwise training approach has the advantage that an embedding model with
desired properties, for example for clustering, can first be created for a
high-resource language. Then, in an independent step, it can be extended to
support further languages. This decoupling significantly simplifies the
training procedure compared to previous approaches. Further, it minimizes the
potential language bias of the resulting model.

We extensively tested the approach for various languages from different
language families. We observe that LASER and LaBSE work well for retrieving
exact translations, however, they work less well assessing the similarity of
sentence pairs that are not exact translations. Further, we noticed that LASER
and LaBSE show a language bias, preferring some language combinations over
other.

## Acknowledgments

This work has been supported by the German Research Foundation through the
German-Israeli Project Cooperation (DIP, grant DA 1600/1-1 and grant GU
798/17-1) and has been funded by the German Federal Ministry of Education and
Research and the Hessen State Ministry for Higher Education, Research and the
Arts within their joint support of the National Research Center for Applied
Cybersecurity ATHENE.

## References

  * Agić and Vulić (2019) Željko Agić and Ivan Vulić. 2019.  JW300: A Wide-Coverage Parallel Corpus for Low-Resource Languages.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3204–3210, Florence, Italy. Association for Computational Linguistics.
  * Artetxe et al. (2018) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018.  Generalizing and improving bilingual word embedding mappings with a multi-step framework of linear transformations.  _AAAI Conference on Artificial Intelligence_.
  * Artetxe and Schwenk (2019a) Mikel Artetxe and Holger Schwenk. 2019a.  Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3197–3203, Florence, Italy. Association for Computational Linguistics.
  * Artetxe and Schwenk (2019b) Mikel Artetxe and Holger Schwenk. 2019b.  Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond.  _Transactions of the Association for Computational Linguistics_ , 7(0):597–610.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017\.  SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 1–14, Vancouver, Canada. Association for Computational Linguistics.
  * Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018.  Universal Sentence Encoder.  _arXiv preprint arXiv:1803.11175_.
  * Chidambaram et al. (2019) Muthu Chidambaram, Yinfei Yang, Daniel Cer, Steve Yuan, Yunhsuan Sung, Brian Strope, and Ray Kurzweil. 2019.  Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Model.  In _Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)_ , pages 250–259, Florence, Italy. Association for Computational Linguistics.
  * Conneau et al. (2019) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Unsupervised Cross-lingual Representation Learning at Scale.  _arXiv preprint arXiv:1911.02116_.
  * Conneau et al. (2017a) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017a.  Supervised Learning of Universal Sentence Representations from Natural Language Inference Data.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 670–680, Copenhagen, Denmark. Association for Computational Linguistics.
  * Conneau et al. (2017b) Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017b.  Word Translation Without Parallel Data.  _arXiv preprint arXiv:1710.04087_.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: Evaluating Cross-lingual Sentence Representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _arXiv preprint arXiv:1810.04805_.
  * Feng et al. (2020) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020\.  Language-agnostic BERT Sentence Embedding.  _arXiv preprint arXiv:2007.01852_.
  * Guo et al. (2018) Mandy Guo, Qinlan Shen, Yinfei Yang, Heming Ge, Daniel Cer, Gustavo Hernandez Abrego, Keith Stevens, Noah Constant, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018.  Effective Parallel Corpus Mining using Bilingual Sentence Embeddings.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 165–176, Brussels, Belgium. Association for Computational Linguistics.
  * Ham et al. (2020) Jiyeon Ham, Yo Joong Choe, Kyubyong Park, Ilji Choi, and Hyungjoon Soh. 2020.  KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understanding.  _arXiv preprint arXiv:2004.03289_.
  * Kiros et al. (2015) Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Skip-Thought Vectors.  In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, _Advances in Neural Information Processing Systems 28_ , pages 3294–3302. Curran Associates, Inc.
  * Koehn (2005) Philipp Koehn. 2005.  Europarl: A Parallel Corpus for Statistical Machine Translation.  In _Conference Proceedings: the tenth Machine Translation Summit_ , pages 79–86, Phuket, Thailand. AAMT, AAMT.
  * Lample et al. (2018) Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018\.  Unsupervised Machine Translation Using Monolingual Corpora Only.  In _International Conference on Learning Representations_.
  * Lison and Tiedemann (2016) Pierre Lison and Jörg Tiedemann. 2016.  OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016)_ , Paris, France. European Language Resources Association (ELRA).
  * Reimers et al. (2016) Nils Reimers, Philip Beyer, and Iryna Gurevych. 2016.  Task-Oriented Intrinsic Evaluation of Semantic Textual Similarity.  In _Proceedings of the 26th International Conference on Computational Linguistics (COLING)_ , pages 87–96.
  * Reimers and Gurevych (2017) Nils Reimers and Iryna Gurevych. 2017.  Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 338–348, Copenhagen, Denmark. Association for Computational Linguistics.
  * Reimers and Gurevych (2018) Nils Reimers and Iryna Gurevych. 2018.  Why Comparing Single Performance Scores Does Not Allow to Draw Conclusions About Machine Learning Approaches.  _arXiv preprint arXiv:1803.09578_ , abs/1803.09578.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Roy et al. (2020) Uma Roy, Noah Constant, Rami Al-Rfou, Aditya Barua, Aaron Phillips, and Yinfei Yang. 2020.  LAReQA: Language-agnostic answer retrieval from a multilingual pool.  _arXiv preprint arXiv:2004.05484_.
  * Ruder (2017) Sebastian Ruder. 2017.  A survey of cross-lingual embedding models.  _arXiv preprint arXiv:1706.04902_ , abs/1706.04902.
  * Schwenk et al. (2019) Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2019.  WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia.  _arXiv preprint arXiv:11907.05791_.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.  Sequence to Sequence Learning with Neural Networks.  In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 27_ , pages 3104–3112. Curran Associates, Inc.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel Data, Tools and Interfaces in OPUS.  In _Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12)_ , Istanbul, Turkey. European Language Resources Association (ELRA).
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122. Association for Computational Linguistics.
  * Yang et al. (2019) Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernández Ábrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2019.  Multilingual Universal Sentence Encoder for Semantic Retrieval.  _arXiv preprint arXiv:1907.04307_ , abs/1907.04307.
  * Yang et al. (2018) Yinfei Yang, Steve Yuan, Daniel Cer, Sheng-Yi Kong, Noah Constant, Petr Pilar, Heming Ge, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2018.  Learning Semantic Textual Similarity from Conversations.  In _Proceedings of The Third Workshop on Representation Learning for NLP_ , pages 164–174, Melbourne, Australia. Association for Computational Linguistics.
  * Ziemski et al. (2016) Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016.  The United Nations Parallel Corpus v1.0.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016)_ , Paris, France. European Language Resources Association (ELRA).
  * Zweigenbaum et al. (2018) Pierre Zweigenbaum, Serge Sharoff, and Reinhar Rapp. 2018.  Overview of the Third BUCC Shared Task: Spotting Parallel Sentences in Comparable Corpora.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Paris, France. European Language Resources Association (ELRA).
  * Zweigenbaum et al. (2017) Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2017.  Overview of the Second BUCC Shared Task: Spotting Parallel Sentences in Comparable Corpora.  In _Proceedings of the 10th Workshop on Building and Using Comparable Corpora_ , pages 60–67, Vancouver, Canada. Association for Computational Linguistics.

## Appendix

Figure 2: First two principle components for parallel sentences in English and
Russian. For LaBSE, we observe a strong separation between the vector spaces,
i.e., it is more biased for same language pairs. mUSE and the proposed
multilingual knowledge distillation approach show nearly no language bias.

##  Appendix A Tatoeba Similarity Search

LASER Artetxe and Schwenk (2019b) introduces the Tatoeba test set: It consists
of up to 1,000 English-aligned sentence pairs for various languages.
Evaluation is done by finding for all sentences the most similar sentence in
the other language using cosine similarity. Accuracy is computed for both
directions (English to the other language and back). As the two scores are
usually quite close, we report the average of the scores here.

For our multi-lingual knowledge distillation, we exclude all training data
from Tatoeba. This is in contrast to LASER, which used Tatoeba also for
training. As the overlap between the Tatoeba test and train set is enormous,
this bears the risk that the scores for LASER are artificially high. For mUSE
and LaBSE, it is unknown whether they used any parallel data crawled from the
Tatoeba website.

Table 9 shows the results for the languages we had parallel training data. We
achieve accuracy scores usually in the 90th. Note mUSE is only available for
16 languages, hence, we also tested multilingual knowledge distillation using
DistilmBERT as student and mUSE as teacher. We report this as DistilmBERT
$\leftarrow$ mUSE in the table.

For the languages which we did not use any parallel data (Table ), the scores
show a much larger variance. But overall, the scores for languages without
parallel data are in most cases low. This is expected, as the model did not
learn how to align for these languages. Note, LaBSE and LASER had seen
training data for most of the languages in Table .

Model | ara | bul | cat | ces | cmn | dan | deu | ell | est | fin | fra | glg | heb
---|---|---|---|---|---|---|---|---|---|---|---|---|---
LASER | 92.0 | 95.0 | 95.9 | 96.5 | 95.4 | 96.0 | 99.0 | 95.0 | 96.7 | 96.3 | 95.6 | 95.5 | 92.2
mUSE | 81.0 | 54.0 | 66.3 | 17.8 | 94.3 | 25.9 | 98.2 | 1.6 | 8.4 | 8.2 | 93.5 | 82.2 | 1.8
LaBSE | 91.0 | 95.7 | 96.5 | 97.5 | 96.2 | 96.4 | 99.4 | 96.6 | 97.7 | 97.0 | 96.0 | 97.2 | 93.0
XLM-R $\leftarrow$ SBERT-p | 87.7 | 94.0 | 96.4 | 96.3 | 95.0 | 96.2 | 98.7 | 95.5 | 95.8 | 96.4 | 94.7 | 96.0 | 88.4
DistilmBERT $\leftarrow$ mUSE | 86.8 | 93.3 | 95.8 | 94.6 | 95.3 | 94.6 | 98.6 | 93.1 | 93.7 | 91.9 | 94.0 | 95.2 | 85.0
Model | hin | hrv | hun | hye | ind | ita | jpn | kat | kor | lit | lvs | mar | mkd
LASER | 94.7 | 97.2 | 96.0 | 36.1 | 94.5 | 95.3 | 90.7 | 35.9 | 88.9 | 96.2 | 95.4 | 91.5 | 94.7
mUSE | 1.2 | 23.9 | 10.2 | 1.7 | 93.3 | 94.3 | 93.8 | 2.6 | 86.0 | 10.2 | 11.1 | 1.8 | 33.1
LaBSE | 97.7 | 97.8 | 97.2 | 95.0 | 95.3 | 94.6 | 96.4 | 95.9 | 93.5 | 97.3 | 96.8 | 94.8 | 94.8
XLM-R $\leftarrow$ SBERT-p | 96.4 | 97.0 | 94.7 | 91.3 | 94.1 | 94.9 | 94.2 | 91.4 | 90.1 | 95.8 | 96.4 | 91.0 | 92.2
DistilmBERT $\leftarrow$ mUSE | 94.2 | 95.1 | 91.3 | 88.0 | 93.5 | 93.1 | 92.7 | 82.7 | 89.5 | 94.2 | 92.3 | 86.4 | 91.1
Model | mon | nld | nob | pes | pol | por | ron | rus | slk | slv | spa | sqi | srp
LASER | 8.2 | 96.3 | 98.8 | 93.4 | 97.8 | 95.2 | 97.4 | 94.6 | 96.6 | 95.9 | 98.0 | 98.0 | 95.3
mUSE | 16.9 | 94.0 | 23.9 | 12.7 | 93.7 | 94.9 | 30.0 | 93.7 | 21.1 | 20.9 | 95.4 | 19.9 | 27.7
LaBSE | 96.6 | 97.2 | 98.9 | 96.0 | 97.8 | 95.6 | 97.8 | 95.3 | 97.3 | 96.7 | 98.4 | 97.6 | 96.2
XLM-R $\leftarrow$ SBERT-p | 91.7 | 96.0 | 98.0 | 94.8 | 97.0 | 94.8 | 96.4 | 93.5 | 96.2 | 95.5 | 98.0 | 97.5 | 93.8
DistilmBERT $\leftarrow$ mUSE | 90.6 | 95.8 | 95.8 | 90.0 | 95.3 | 94.5 | 94.7 | 94.4 | 95.4 | 94.8 | 95.5 | 95.6 | 93.2
Model | swe | tha | tur | ukr | urd | vie | yue | zsm |
LASER | 96.6 | 95.4 | 97.5 | 94.5 | 81.9 | 96.8 | 90.0 | 96.4 |
mUSE | 18.8 | 96.0 | 94.0 | 51.0 | 6.4 | 10.4 | 84.2 | 89.1 |
LaBSE | 96.5 | 97.1 | 98.4 | 95.2 | 95.3 | 97.8 | 92.1 | 96.9 |
XLM-R $\leftarrow$ SBERT-p | 95.7 | 96.3 | 97.2 | 94.3 | 92.2 | 97.2 | 84.4 | 95.6 |
DistilmBERT $\leftarrow$ mUSE | 94.3 | 93.2 | 96.5 | 92.5 | 87.9 | 95.3 | 81.0 | 95.2 |
Table 9: Tatoeba test set results for languages with parallel data for multilingual knowledge distillation. Model | afr | amh | ang | arq | arz | ast | awa | aze | bel | ben | ber | bos | bre
---|---|---|---|---|---|---|---|---|---|---|---|---|---
LASER | 89.5 | 42.0 | 37.7 | 39.5 | 68.9 | 86.2 | 36.1 | 66.0 | 66.1 | 89.6 | 68.2 | 96.5 | 15.8
mUSE | 63.5 | 2.1 | 38.1 | 28.2 | 59.6 | 81.5 | 2.4 | 42.2 | 40.3 | 0.7 | 8.3 | 30.1 | 10.2
LaBSE | 97.4 | 94.0 | 64.2 | 46.2 | 78.4 | 90.6 | 73.2 | 96.1 | 96.2 | 91.3 | 10.4 | 96.2 | 17.3
XLM-R $\leftarrow$ SBERT-p | 84.5 | 67.9 | 25.0 | 30.6 | 63.7 | 78.3 | 46.5 | 85.0 | 86.9 | 77.6 | 6.8 | 95.8 | 10.1
DistilmBERT $\leftarrow$ mUSE | 68.3 | 2.7 | 37.7 | 32.7 | 61.5 | 85.0 | 43.9 | 43.4 | 49.1 | 1.2 | 8.1 | 94.2 | 11.5
Model | cbk | ceb | cha | cor | csb | cym | dsb | dtp | epo | eus | fao | fry | gla
LASER | 77.0 | 15.7 | 29.2 | 7.5 | 43.3 | 8.6 | 48.0 | 7.2 | 97.2 | 94.6 | 71.6 | 51.7 | 3.7
mUSE | 76.1 | 13.7 | 33.6 | 6.4 | 37.4 | 13.1 | 35.1 | 8.4 | 36.8 | 19.4 | 18.7 | 52.3 | 6.9
LaBSE | 82.5 | 70.9 | 39.8 | 12.8 | 56.1 | 93.6 | 69.3 | 13.3 | 98.4 | 95.8 | 90.6 | 89.9 | 88.8
XLM-R $\leftarrow$ SBERT-p | 69.4 | 11.7 | 25.9 | 5.1 | 40.5 | 34.9 | 51.4 | 7.3 | 68.8 | 48.6 | 50.8 | 58.4 | 7.5
DistilmBERT $\leftarrow$ mUSE | 77.2 | 13.8 | 34.7 | 7.3 | 48.0 | 13.1 | 52.0 | 9.4 | 41.2 | 19.0 | 36.1 | 54.0 | 6.0
Model | gle | gsw | hsb | ido | ile | ina | isl | jav | kab | kaz | khm | kur | kzj
LASER | 5.2 | 44.4 | 54.5 | 83.7 | 86.2 | 95.2 | 95.6 | 22.9 | 58.1 | 18.6 | 20.6 | 17.2 | 7.2
mUSE | 7.7 | 39.3 | 33.3 | 55.5 | 73.3 | 86.7 | 10.3 | 38.3 | 3.7 | 15.3 | 1.5 | 21.7 | 10.2
LaBSE | 95.0 | 52.1 | 71.2 | 90.9 | 87.1 | 95.8 | 96.2 | 84.4 | 6.2 | 90.5 | 83.2 | 87.1 | 14.2
XLM-R $\leftarrow$ SBERT-p | 18.6 | 36.8 | 57.6 | 56.0 | 70.5 | 87.9 | 75.8 | 37.3 | 2.7 | 73.7 | 64.8 | 43.7 | 8.0
DistilmBERT $\leftarrow$ mUSE | 8.0 | 38.9 | 56.4 | 61.1 | 77.8 | 90.4 | 16.0 | 31.7 | 3.7 | 16.8 | 1.2 | 27.7 | 10.8
Model | lat | lfn | mal | max | mhr | nds | nno | nov | oci | orv | pam | pms | swg
LASER | 58.5 | 64.5 | 96.9 | 50.9 | 10.4 | 82.9 | 88.3 | 66.0 | 61.2 | 28.1 | 6.0 | 49.6 | 46.0
mUSE | 36.7 | 60.5 | 1.2 | 65.0 | 14.3 | 57.5 | 21.2 | 66.1 | 42.9 | 28.3 | 8.4 | 48.8 | 48.7
LaBSE | 82.0 | 71.2 | 98.9 | 71.1 | 19.2 | 81.2 | 95.9 | 78.2 | 69.9 | 46.8 | 13.6 | 67.0 | 65.2
XLM-R $\leftarrow$ SBERT-p | 28.0 | 57.7 | 94.0 | 58.5 | 11.9 | 50.7 | 89.3 | 58.8 | 52.4 | 33.4 | 7.0 | 44.3 | 33.9
DistilmBERT $\leftarrow$ mUSE | 42.9 | 65.3 | 1.0 | 60.9 | 14.2 | 59.5 | 78.8 | 69.8 | 54.7 | 27.8 | 9.6 | 52.6 | 51.8
Model | swh | tam | tat | tel | tgl | tuk | tzl | uig | uzb | war | wuu | xho | yid
LASER | 57.6 | 69.4 | 31.1 | 79.7 | 50.6 | 20.7 | 44.7 | 45.2 | 18.7 | 13.6 | 87.7 | 8.5 | 5.7
mUSE | 13.7 | 2.8 | 15.7 | 2.4 | 16.2 | 20.9 | 46.6 | 4.0 | 15.9 | 15.6 | 82.2 | 14.8 | 1.9
LaBSE | 88.6 | 90.7 | 87.9 | 98.3 | 97.4 | 80.0 | 63.0 | 93.7 | 86.8 | 65.3 | 90.3 | 91.9 | 91.0
XLM-R $\leftarrow$ SBERT-p | 27.6 | 85.7 | 17.8 | 89.1 | 32.4 | 24.1 | 41.3 | 65.5 | 32.6 | 11.4 | 82.7 | 11.6 | 52.7
DistilmBERT $\leftarrow$ mUSE | 13.8 | 2.3 | 13.5 | 1.9 | 15.7 | 27.6 | 45.2 | 4.5 | 18.7 | 15.4 | 82.2 | 13.4 | 6.2
Table 10: Tatoeba test set results for languages for languages without
parallel data for multilingual knowledge distillation. LaBSE and LASER had
training data for most of these languages.