# Source Localization by Multidimensional Steered Response Power Mapping with
Sparse Bayesian Learning

###### Abstract

We propose an advance Steered Response Power (SRP) method for localizing
multiple sources. While conventional SRP performs well in adverse conditions,
it remains to struggle in scenarios with closely neighboring sources,
resulting in ambiguous SRP maps. We address this issue by applying sparsity
optimization in SRP to obtain high-resolution maps. Our approach represents
SRP maps as multidimensional matrices to preserve time-frequency information
and further improve performance in unfavorable conditions. We use multi-
dictionary Sparse Bayesian Learning to localize sources without needing prior
knowledge of their quantity. We validate our method through practical
experiments with a 16-channel planar microphone array and compare against
three other SRP and sparsity-based methods. Our multidimensional SRP approach
outperforms conventional SRP and the current state-of-the-art sparse SRP
methods for localizing closely spaced sources in a reverberant room.

Index Terms—  Source Localization, Steered Response Power, Sparse
Representation, Sparse Bayesian Learning

##  1 Introduction

Acoustic source localization in reverberant environments is an active problem
in microphone signal processing. Several methods have been developed for
source localization, such as subspace-based approaches like MUltiple SIgnal
Classification (MUSIC) [1, 2] and ESPRIT [3], time differences of arrival
(TDOA) approaches like Generalized Cross-Correlation Phase Transform (GCC-
PHAT) [4, 5] and steered response power (SRP) [4, 6], sparsity-based
approaches like Orthogonal Matching Pursuit (OMP) [7], Sparse Bayesian
Learning (SBL) [8, 9], as well as learning-based approaches [10, 11].

The SRP method estimates sound source positions by the summation of cross-
correlation of all possible microphone pairs. Recently, several approaches
have focused on hierarchical search [12, 13] and real-time SRP [14] to reduce
errors and complexity. However, SRP still struggles with localization in
scenarios where multiple sources closely spaced in reverberant environments
due to an ambiguous SRP map. Some research has overcome this challenge by
iterative grid decomposition [15] or sparse fitting [16, 17]. However, the
former approach still requires careful selection for the grid resolution,
especially in scenarios with numerous sources. Whereas, the latter performs
well in low reverberation scenarios but degrades dramatically as reverberation
increases.

In this paper, we propose a method to improve localization performance by
following the sparse fitting approach. To reduce high localization errors
caused by reverberant environments, we represent the obtained SRP maps as
multidimensional matrices to preserve more time-frequency information. We
apply multi-dictionary SBL (M-SBL) [18] as the sparsity optimization method,
allowing the proposed method to operate without prior knowledge of the number
of sources, thereby adapting to adverse real-world conditions. We utilize two
sets of candidate grids with different resolutions for SRP mapping and
sparsity fitting to further improve efficiency. We validate the proposed
method through practical experiments with a planar microphone array,
demonstrating improvements in localizing multiple closely spaced sources in a
reverberant room compared against other conventional and state-of-the-art
methods. Results show our method enhances robustness, allowing for stable
performance even when localizing on short duration recordings.

##  2 Problem Formulation

Consider $J$ sound sources, $j{\,=\,}\\{1,{\cdots},J\\}$, located at positions
$\mathbf{y}_{j}$ that each emit signals $s_{j}$. Additionally, consider a
microphone array with $M$ elements each positioned at $\mathbf{x}_{m}$ for
$m{\,=\,}\\{1,{\cdots},M\\}$. Received signals can be expressed as

$p_{m}(\mathfrak{t})=s_{j}(\mathfrak{t})*g_{m}(\mathfrak{t},\mathbf{y}_{j})+v_{m}(\mathfrak{t}),$
(1)

where $p_{m}(\mathfrak{t})$ denotes the $m^{\text{th}}$-microphone’s signal at
time $\mathfrak{t}$, $g_{m}(\mathfrak{t},\mathbf{y}_{j})$ denotes the impulse
response from the $j^{\text{th}}$ source to the $m^{\text{th}}$ microphone,
and $v_{m}$ denotes a noise term.

The TDOA between the pair of microphones $(m,m^{\prime})$ due to the
$j^{\text{th}}$ sound source is given by [4]:

$\tau_{m,m^{\prime}}(\mathbf{y}_{j})=\frac{1}{c}\begin{cases}\left\|\mathbf{y}_{j}{-}\mathbf{x}_{m}\right\|-\left\|\mathbf{y}_{j}{-}\mathbf{x}_{m^{\prime}}\right\|&\text{for
NF,}\\\
(\mathbf{x}_{m}{-}\mathbf{x}_{m^{\prime}})\cdot\vec{\mathbf{y}}_{j}&\text{for
FF,}\end{cases}$ (2)

where $c$ is the speed of sound, NF and FF denote near-field and far-field
sound propagation, respectively.

The goal of TDOA-based sound source localization is to find the source
locations of $\mathbf{y}_{j}$ from estimated TDOAs (2) of a microphone array.

##  3 Source Localization by SRP

Two common TDOA-based localization methods are GCC-PHAT [4, 5] and SRP-PHAT
[6]. While GCC-PHAT maximizes the GCC function of a microphone pair
$(m,m^{\prime})$ to estimate TDOAs, SRP-PHAT maximizes the GCC functions of
all $L{\,=\,}\binom{M}{2}$ microphone pairs denoted by
$\ell=\\{1,{\cdots},L\\}$.

The GCC function at a time-frequency frame is defined as

$R_{m,m^{\prime}}(\tau,k,t)=\Psi_{m{,}m^{\prime}}(k{,}t)P_{m}(k{,}t)P_{m^{\prime}}^{*}(k{,}t)e^{j2\pi
k\tau},$ (3)

where $k=\\{1,{\cdots},K\\}$ and $t=\\{1,{\cdots},T\\}$ index the wave number
and the time frame in time-frequency domain, respectively, $K$ and $T$ denote
the totals, $P_{m}(k,t)$ is the short-time Fourier transform of
$p_{m}(\mathfrak{t})$, $(\cdot)^{*}$ is the complex conjugate,
$\Psi_{m,m^{\prime}}(k,t)=1/|P_{m}(k,t)P_{m^{\prime}}^{*}(k,t)|$ is the
frequency-dependent weighting function for GCC-PHAT.

Assuming all sound sources belong to a set of $N$ candidate positions
$\mathbf{y}_{n}\in\\{\mathbf{y}_{1},{\cdots},\mathbf{y}_{N}\\}$, where
$N{\,>\,}J$, the SRP function is given as

$z_{k,t}(\mathbf{y}_{n})=\Re\left({\textstyle\sum_{\ell=1}^{L}}R_{\ell}(\tau,k,t)\right),$
(4)

where $\ell\equiv(m,m^{\prime})$.

The SRP map is composed of the SRP functions (4) of all $N$ candidate
positions. When the obtained SRP reaches a local maximum, the candidate
position will be an estimated source location. In practice, the source
positions $\hat{\mathbf{y}}_{j}$ are estimated by averaging SRP over time and
frequency as follows:

$\hat{\mathbf{y}}_{j}=\arg\max_{n\in N}{z(\mathbf{y}_{n})}=\arg\max_{n\in
N}{\textstyle\sum_{k=1}^{K}\sum_{t=1}^{T}}{z_{k,t}(\mathbf{y}_{n})},$ (5)

where $z(\mathbf{y}_{n})$ denotes the averaged SRP function of the
$n^{\text{th}}$ candidate position.

The SRP method for localization is robust in adverse environments after time-
frequency averaging. However, multiple sources located close to each other can
still lead to an ambiguous SRP map and poor source localization. In the next
section we address this issue of closely spaced sources by employing a
sparsity-based SRP method with M-SBL [18].

##  4 Proposed Method

We propose optimizing SRP mapping through the use of M-SBL. We aim for a
sparse solution to enforce concentrated peaks in a high resolution SRP map,
which allows for distinction between multiple sources placed close to each
other.

We define another set of $Q$ candidate positions
$\mathbf{y}_{q}\in\\{\mathbf{y}_{1},{\cdots},\mathbf{y}_{Q}\\}$, the
microphone signal model from $\mathbf{y}_{q}$ to $\mathbf{x}_{m}$ can be
expressed as a linear regression model by STFT as follows:

$P_{m}(k,t)=G_{q,m}(k)S_{q}(k,t)+\mathcal{N}_{m}(k,t)$ (6)

where $P_{m}(k,t)$ denotes the $m^{\text{th}}$ microphone signal at the
$k^{\text{th}}$ wave number and the $t^{\text{th}}$ time frame, $G_{q,m}$
denotes the dictionary matrix, $S_{q}(k,t)$ denotes the source weight at
$\mathbf{y}_{q}$, and $\mathcal{N}_{m}(k,t)$ denotes the noise term.

We assume the reverberation of the environment is unknown, hence the
dictionary function $G_{q,m}(k)$ is modeled as Green’s functions in case of NF
or FF scenarios, respectively, as follows:

$G_{q,m}(k)=\begin{cases}e^{jk\|\mathbf{x}_{m}-\mathbf{y}_{q}\|}/({4\pi\|\mathbf{x}_{m}-\mathbf{y}_{q}\|})&\text{
for NF, }\\\ e^{-jk\vec{\mathbf{y}}_{q}\cdot\mathbf{x}_{m}}&\text{ for FF.
}\end{cases}$ (7)

By combining between equation (3) and equation (6), we can obtain the
corresponding GCC from this candidate position $\mathbf{y}_{n}$ as follows:

$\displaystyle R_{m,m^{\prime}}$ $\displaystyle(\tau{,}k{,}t)\approx$ (8)
$\displaystyle\frac{S_{q}(k{,}t)G_{q{,}m}(k)(S_{q}(k{,}t)G_{q{,}m^{\prime}}(k))^{*}}{|S_{q}(k{,}t)G_{q{,}m}(k)(S_{q}(k{,}t)G_{q{,}m^{\prime}}(k))^{*}|}e^{j2\pi
k\tau},$

where the TDOA $\tau$ corresponds one-to-one with the first set of $N$
candidate positions. It is worth noting to clarify the two sets of candidate
grids in the proposed method: $N$ is for SRP mapping while $Q$ is for the
subsequent sparsity fitting. Following equation (7), the TDOA between the
microphone pair $(m,m^{\prime})$ for the candidate position $\mathbf{y}_{n}$,
denoted as $\tau_{m,m^{\prime}}(\mathbf{y}_{n})$, can be expressed as follows:

$\tau_{m,m^{\prime}}(\mathbf{y}_{n})=\frac{1}{j2\pi
k}\ln\big{(}G_{n,m^{\prime}}(k)G^{*}_{n,m}(k)\big{)}.$ (9)

By combining equation (4), (8), and (9), the SRP can be reformulated as:

$z_{k,t}(\mathbf{y}_{n},\mathbf{y}_{q})\approx\Re\left(\sum_{\ell=1}^{L}H^{*}_{n,\ell}(k)\frac{H_{q,\ell}(k)}{|H_{q,\ell}(k)|}\right),$
(10)

where $H_{n,\ell}(k)=G_{n,m}(k)G^{*}_{n,m^{\prime}}(k)$ is the relative
transfer function of the microphone pair $\ell$ [19].

We define vectors $\mathbf{a}_{\ell}(k){\,\in\,}\mathbb{C}^{N{\times}1}$ and
$\mathbf{b}_{\ell}(k){\,\in\,}\mathbb{C}^{1{\times}Q}$ to represent the right
part of equation (10) for brevity, where

$\begin{split}&\mathbf{a}_{\ell}(k){=}\begin{array}[]{c}\left[H_{1,\ell}^{*}(k),{\cdots},H_{N,\ell}^{*}(k)\right]^{\top},\end{array}\\\
&\mathbf{b}_{\ell}(k){=}\begin{array}[]{c}\left[\frac{H_{1,\ell}(k)}{|H_{1,\ell}(k)|},{\cdots},\frac{H_{Q,\ell}(k)}{|H_{Q,\ell}(k)|}\right].\end{array}\end{split}$
(11)

Then, we can regard the SRP map of all candidate positions as the summation of
equation (10) as follows:

$\mathbf{z}_{k,t}=\Re\left(\mathbf{A}_{k}\mathbf{B}_{k}\mathbf{S}_{k,t}\right)+\boldsymbol{\mathcal{N}}_{k,t},$
(12)

where
$\mathbf{z}_{k{,}t}{\,=\,}[z_{k,t}(\mathbf{y}_{1}),{\cdots},z_{k,t}(\mathbf{y}_{N})]^{\top}{\in\,}\mathbb{R}^{N{\times}1}$
is the SRP map, matrices
$\mathbf{A}_{k}{\,=\,}[\mathbf{a}_{1}(k),{\cdots},\mathbf{a}_{L}(k)]{\,\in\,}\mathbb{C}^{N{\times}L}$
and
$\mathbf{B}_{k}=[\mathbf{b}_{1}(k),{\cdots},\mathbf{b}_{L}(k)]^{\top}{\in\,}\mathbb{C}^{L{\times}Q}$
are formed by vectors $\mathbf{a}_{\ell}$ and $\mathbf{b}_{\ell}$,
respectively,
$\mathbf{S}_{k{,}t}{\,=\,}[S_{1}(k{,}t),{\cdots},S_{Q}(k{,}t)]^{\top}{\in\,}\mathbb{C}^{Q{\times}1}$
denotes the source weight matrix of $Q$ candidate positions, and
$\boldsymbol{\mathcal{N}}_{k,t}{\in\,}\mathbb{R}^{N{\times}1}$ denotes the
noise term.

Since sources are spare in numbers, we here assume $\mathbf{S}_{k,t}$ should
have mostly zero entries. Under this assumption, we define equation (12) as an
underdetermined system where $Q{\,\geq\,}N$. This indicates that we represent
SRP maps using a higher-resolution source weight map. Hence, (12) is a sparse
representation of the SRP map for a single time frame and a single wave
number. Such a sparse solution of $\mathbf{S}_{k,t}$ can be obtained by
solving the following optimization problem:

$\underset{\mathbf{S}_{k,t}}{\min}\frac{1}{2}\left\|\mathbf{z}_{k,t}-\Re(\mathbf{A}_{k}\mathbf{B}_{k}\mathbf{S}_{k,t})\right\|_{2}^{2}+\lambda\left\|\Re(\mathbf{S}_{k,t})\right\|_{\mathfrak{p}}^{\mathfrak{p}},$
(13)

where $\left\|\cdot\right\|_{\mathfrak{p}}$ denotes the
$\ell_{\mathfrak{p}}$-norm as $0{\,<\,}\mathfrak{p}{\,\leq\,}1$.

To further enhance the robustness, we consider incorporating time-frequency
diversity from SRP inputs. We here represent the SRP function as a
multidimensional matrix $\mathbf{Z}{\,\in\,}\mathbb{R}^{N{\times}T{\times}K}$
instead of the vector form $\mathbf{z}{\,\in\,}\mathbb{R}^{N{\times}1}$ in
equation (5). We then express a group-sparse representation of the
multidimensional matrix $\mathbf{Z}$ as

$\mathbf{Z}=\Re(\mathbf{ABS})+\boldsymbol{\mathcal{N}},$ (14)

where $\mathbf{Z}\in\mathbb{R}^{N{\times}T{\times}K}$,
$\mathbf{A}\in\mathbb{C}^{N{\times}L{\times}K}$,
$\mathbf{B}\in\mathbb{C}^{L{\times}Q{\times}K}$,
$\mathbf{S}\in\mathbb{C}^{Q{\times}T{\times}K}$, and
$\boldsymbol{\mathcal{N}}\in\mathbb{R}^{N{\times}T{\times}K}$. Equation (14)
represents multidimensional matrix multiplication.

We solve the group-sparse representation (14) by introducing a
multidimensional mixed-norm penalty term [20] as

$\underset{\mathbf{S}}{\min}\frac{1}{2}\left\|\mathbf{Z}-\Re(\mathbf{A}\mathbf{B}\mathbf{S})\right\|_{2}^{2}+\lambda\mathcal{J}_{\mathfrak{p}_{1},\mathfrak{p}_{2},\mathfrak{p}_{3}}(\Re(\mathbf{S})),$
(15)

where $\mathfrak{p}_{1},\mathfrak{p}_{2},\mathfrak{p}_{3}$ denote the norms of
spatial, time, and frequency domain, respectively, following a descending
order where
$0{\,<\,}\mathfrak{p}_{1}{\,\leq\,}\mathfrak{p}_{2}{\,\leq\,}\mathfrak{p}_{3}{\,\leq\,}2$,
and
$\mathcal{J}_{\mathfrak{p}_{1},\mathfrak{p}_{2},\mathfrak{p}_{3}}(\Re(\mathbf{S}))$
denotes the penalty function of $\Re(\mathbf{S})$.

Here we assume source signals are non-sparse in time-frequency domain, hence
we set
$\mathfrak{p}_{1}{\,=\,}\mathfrak{p},\mathfrak{p}_{2}{\,=\,}2,\mathfrak{p}_{3}{\,=\,}2$,
which means the penalty term of (15) is a $\ell_{\mathfrak{p},2,2}$-norm
function.

We next apply M-SBL to optimize (15) since it is suitable for solving the
underdetermined problem without the prior knowledge of the source quantities
$J$. The M-SBL method is based on two assumptions. First, the noise term
$\boldsymbol{\mathcal{N}}$ is assumed to be the zero-mean Gaussian noise term
with density
$\mathcal{N}(\boldsymbol{\mathcal{N}};\mathbf{0},\sigma^{2}\mathbf{I})$.
Second, the source weight $\mathbf{S}$ is assumed to be the zero-mean complex
Gaussian with density $\mathcal{CN}(\mathbf{S};\mathbf{0},\mathbf{\Gamma})$,
where $\mathbf{\Gamma}=\text{diag}(\boldsymbol{\gamma})$ is the diagonal
matrix of hyperparameters
$\boldsymbol{\gamma}=[\gamma_{1},{\cdots},\gamma_{Q}]$.

In the M-SBL framework, the hyperparameters $\boldsymbol{\gamma}$ are assumed
to be unknown and learned iteratively through maximizing the evidence to reach
a sparse result. Here, the evidence model is also the zero-mean Gaussian with
density $\mathcal{N}(\mathbf{Z};\mathbf{0},\boldsymbol{\Sigma}_{k})$, where
$\boldsymbol{\Sigma}_{k}=\sigma_{k}^{2}\mathbf{I}+\mathbf{G}_{k}\mathbf{\Gamma}\mathbf{G}_{k}^{H}$.
Maximizing $\hat{\boldsymbol{\gamma}}$ is facilitated iteratively using
derivatives of the evidence. Further procedural details of M-SBL can be found
in [18]. Finally, the update equation is obtained as

${\gamma}^{\text{new}}_{q}=\frac{\gamma_{n}^{\text{old}}}{T}\frac{{\textstyle\sum_{k=1}^{K}}\left\|\mathbf{Z}_{k}^{H}\boldsymbol{\Sigma}_{k}^{-1}\mathbf{D}_{k,q}\right\|_{2}^{2}}{{\textstyle\sum_{k=1}^{K}}\mathbf{D}_{k,q}^{H}\boldsymbol{\Sigma}_{k}^{-1}\mathbf{D}_{k,q}},$
(16)

where $\mathbf{D}_{k,q}$ denotes the $q^{\text{th}}$ column of the dictionary
matrix $\mathbf{D}_{k}{\,=\,}\Re(\mathbf{A}_{k}\mathbf{B}_{k})$. The candidate
positions corresponding to the largest peaks of $\hat{\boldsymbol{\gamma}}$
will be the estimated source locations. It is worth noting that M-SBL
terminates iterations when the error falls below the convergence threshold,
thus removing the need to determine the number of sources.

Compared to the current state-of-the-art sparsity-based modeling method for
SRP maps [17], the proposed method represents obtained SRP maps as a
multidimentional matrix $\mathbf{Z}$ instead of a time-frequency-averaged
vector $\mathbf{z}$ to improve localization by retaining time-frequency
information. Additionally, we set two candidate grids for SRP mapping and
sparse fitting with different resolutions for efficiency. We will compare
these two methods and other localization methods through experiments in the
next section.

##  5 Experimental Validation

We conducted recordings in a large meeting room with a hard floor and ceiling
at $3.3\text{ m}$ height, with a $T_{60}{\,\approx\,}0.7\text{ s}$
($T_{20}{\,=\,}113\text{ ms}$ measured). We used a $M{\,=\,}16$-channel MEMS
microphone planar array (MiniDSP UMA-16 v2) and $J{\,=\,}3$ target
loudspeakers (Genelec 8030C) positioned in the room as detailed in Fig. 1. The
loudspeaker’s played speech signals from the MS-SNSD dataset [21]. The
recording and localization used 48 kHz sampling frequency and 1024 STFT frame
length with 50$\%$ overlap. We note that the ground truth positions and DOAs
of the speakers in Fig. 1 were obtained by hand and are thus prone to human
error.

We evaluate the DOA estimation performance of the proposed method through this
experiment. We therefore use FF propagation for equations (2) and (7). The
candidate DOAs are defined over a hemisphere since the planar array struggles
in distinguishing front-back symmetry. We select $N{\,=\,}247$ candidate DOAs
for the SRP map, with a $15^{\circ}$ resolution over elevation range
$\theta{\,\in\,}[-90^{\circ},90^{\circ}]$ and a $10^{\circ}$ resolution over
azimuth range $\phi{\,\in\,}[0^{\circ},180^{\circ}]$, $Q{\,=\,}8281$ candidate
DOAs for the M-SBL, with a $2^{\circ}$ resolution for elevation $\theta$ and a
$2^{\circ}$-resolution for azimuth $\phi$. The grids are with respect to the
center of the microphone array. We set the convergence error of M-SBL as
$10^{-3}$. We denote the proposed method as SRP-SBL in the results for
brevity.

Fig. 1: Illustration of experiment setup. Loudspeaker distance and azimuth
positions are drawn in (a). Loudspeaker and microphone heights are pictured in
(b).

Fig. 2: Normalized output maps of (a) SRP-PHAT and (b) SRP-SBL. Note that the
color scale of (b) is logarithmic to better illustrate the localized peaks.
The true DOAs are denoted by $(\circ)$.

We compare SRP-SBL to three methods: conventional SRP-PHAT, M-SBL, and SRP-
sparsity (SRP-S)[17] which is the current state-of-the-art sparsity-based
modeling method for SRP maps. The candidate DOAs for these methods are the
same as SRP-SBL with $Q{\,=\,}8281$. It is important to distinguish that SBL
utilizes microphone measurements $\mathbf{P}$ for sparsity optimization, as
opposed to SRP-SBL which uses the SRP maps $\mathbf{Z}$. Furthermore, the
sparsity optimization method for SRP-S in [17] utilized the ADMM solver.
However, due to the sensitivity of ADMM to adverse environments and the
heuristic setting of the regularization parameter, we instead nominated to use
Simultaneous OMP [22] as an alternative optimization method in our evaluation.

In Fig. 2 we demonstrate the output maps of conventional SRP-PHAT and our
proposed SRP-SBL. In Fig. 2(a) we observe that the generated SRP-PHAT map
exhibits an ambiguous peak at $(\theta{,}\phi)=(0^{\circ}{,}75^{\circ})$,
making it challenging to distinguish all three sources. In contrast, the SRP-
SBL map in Fig. 2(b) clearly distinguishes the individual three sources and
their relative locations. We note that the color-scale of this SRP-SBL map is
logarithmic.

Fig. 3: Normalized output maps of the four methods and ground truth along the
azimuth angles. The true DOAs are denoted by $(\circ)$.

Fig. 4: LE for varying recording durations.

Fig. 4 presents an azimuth slice of the Fig. 2 results along with the results
of M-SBL and SRP-S. We observe that M-SBL is unable to correctly localize the
sources. Whereas, SRP-S correctly localizes $\mathbf{y}_{1}$ and closely finds
$\mathbf{y}_{2}$ but misses the third source.

We evaluate the impact of recording duration on localization accuracy in Fig.
4. In some scenarios where one source peak significantly outweighs others,
leading to difficulties in estimating local maxima, we here impose a
constraint that each estimated DOA should be spaced $3^{\circ}$ apart. The
localization error is given as the average angles between the estimated DOA
$(\hat{\theta}_{j}{,}\hat{\phi}_{j})$ of the $J$ highest peaks and the nearest
true DOA $({\theta}_{j},{\phi}_{j})$ as
$\mathbf{LE}=\frac{1}{J}{\sum_{j\texttt{=}1}^{J}}\cos^{\shortminus
1}(\sin(\hat{\theta}_{j})\sin({\theta}_{j})\cos(\hat{\phi}_{j}{\shortminus}{\phi}_{j}){+}\cos(\hat{\theta}_{j})\cos({\theta}_{j}))$.
It can be observed that both SRP-PHAT and M-SBL yield significant
$\mathbf{LE}$ for all recording duration. The SRP-S method performs well for
recording duration greater than $1\text{ s}$ but exhibits instability for
duration ${\,\leq\,}1\text{ s}$. In contrast, the proposed SRP-SBL method is
robust across a range of recording duration due to the way we maintain time
and frequency diversity.

##  6 Conclusion

In this paper, we propose a sparsity-based optimized SRP method for localizing
multiple neighboring sources. The method utilizes a multidimensional SRP
matrix as input and optimizes it through multidimensional SBL (M-SBL).
Practical experiment results indicate that the proposed method outperforms
conventional SRP, M-SBL, and the current state-of-the-art SRP sparsity-based
method (SRP-S), maintaining stable localization performance in reverberant
environments. In contrast, multiple closely spaced sources cause both SRP and
M-SBL to consider all sources as a single point. While SRP-S enhances
localization performance, it still loses its robustness in low-recording-
duration scenarios.

## References

  * [1] R. Schmidt,  “Multiple emitter location and signal parameter estimation,”  IEEE Trans. Antennas Propag., vol. 34, no. 3, pp. 276–280, 1986.
  * [2] L. Birnie, T. D. Abhayapala, H. Chen, and P. N. Samarasinghe,  “Sound source localization in a reverberant room using harmonic based music,”  in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2019, pp. 651–655.
  * [3] R. Roy and T. Kailath,  “Esprit-estimation of signal parameters via rotational invariance techniques,”  IEEE Trans. Acoust., Speech, Signal Process., vol. 37, no. 7, pp. 984–995, 1989.
  * [4] J. H. Dibiase,  “A high-accuracy, low-latency technique for talker localization in reverberant environments using microphone arrays,”  Ph. D. Thesis, 2000.
  * [5] W. Manamperi, T. D. Abhayapala, J. Zhang, and P. N. Samarasinghe,  “Drone audition: Sound source localization using on-board microphones,”  IEEE Trans. Acoust., Speech, Signal Process., vol. 30, pp. 508–519, 2022.
  * [6] J. P. Dmochowski, J. Benesty, and S. Affes,  “A generalized steered response power method for computationally viable source localization,”  IEEE Trans. Audio, Speech, Language Process., vol. 15, no. 8, pp. 2510–2526, 2007.
  * [7] G. Chardon, T. Nowakowski, J. de Rosny, and L. Daudet,  “A blind dereverberation method for narrowband source localization,”  IEEE J. Selected Topics Signal Process., vol. 9, no. 5, pp. 815–824, 2015.
  * [8] D. P. Wipf and B. D. Rao,  “Sparse bayesian learning for basis selection,”  IEEE Trans. Signal Process., vol. 52, no. 8, pp. 2153–2164, 2004.
  * [9] K. L. Gemba, S. Nannuru, and P. Gerstoft,  “Robust ocean acoustic localization with sparse bayesian learning,”  IEEE J. Selected Topics Signal Process., vol. 13, no. 1, pp. 49–60, 2019.
  * [10] S. Chakrabarty and E. A. P. Habets,  “Broadband doa estimation using convolutional neural networks trained with noise signals,”  in Proc. IEEE Workshop Appl. Signal Process. Audio Acoust. IEEE, 2017, pp. 136–140.
  * [11] A. S. Roman, I. R. Roman, and J. P. Bello,  “Robust doa estimation from deep acoustic imaging,”  in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. IEEE, 2024, pp. 1321–1325.
  * [12] D. N. Zotkin and R. Duraiswami,  “Accelerated speech source localization via a hierarchical search of steered response power,”  IEEE Trans. Speech Audio Process., vol. 12, no. 5, pp. 499–508, 2004.
  * [13] L. O. Nunes, W. A. Martins, M. V. S. Lima, L. W. P. Biscainho, F. M. Gonçalves, A. Said, B. Lee, et al.,  “A steered-response power algorithm employing hierarchical search for acoustic source localization using microphone arrays,”  IEEE Trans. Signal Process., vol. 62, no. 19, pp. 5171–5183, 2014.
  * [14] H. Do, H. F. Silverman, and Y. Yu,  “A real-time srp-phat source location implementation using stochastic region contraction (src) on a large-aperture microphone array,”  in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. IEEE, 2007, vol. 1, pp. I–121.
  * [15] A. Marti, M. Cobos, J. J. Lopez, and J. Escolano,  “A steered response power iterative method for high-accuracy acoustic source localization,”  J. Acoust. Soc. Am., vol. 134, no. 4, pp. 2627–2630, 2013.
  * [16] J. Velasco, D. Pizarro, and J. Macias-Guarasa,  “Source localization with acoustic sensor arrays using generative model based fitting with sparse constraints,”  Sensors, vol. 12, no. 10, pp. 13781–13812, 2012.
  * [17] E. Tengan, T. Dietzen, F. Elvander, and T. Van Waterschoot,  “Multi-source direction-of-arrival estimation using group-sparse fitting of steered response power maps,”  in Proc. IEEE Workshop Appl. Signal Process. Audio Acoust. IEEE, 2023, pp. 1–5.
  * [18] P. Gerstoft, C. F. Mecklenbräuker, A. Xenaki, and S. Nannuru,  “Multisnapshot sparse bayesian learning for doa,”  IEEE Signal Process. Lett., vol. 23, no. 10, pp. 1469–1473, 2016.
  * [19] I. Cohen,  “Relative transfer function identification using speech signals,”  IEEE Trans. Speech Audio Process., vol. 12, no. 5, pp. 451–459, 2004.
  * [20] N. Murata, S. Koyama, N. Takamune, and H. Saruwatari,  “Sparse representation using multidimensional mixed-norm penalty with application to sound field decomposition,”  IEEE Trans. Signal Process., vol. 66, no. 12, pp. 3327–3338, 2018.
  * [21] C. K. A. Reddy, E. Beyrami, J. Pool, R. Cutler, S. Srinivasan, and J. Gehrke,  “A scalable noisy speech dataset and online subjective test framework,”  Proc. Interspeech, pp. 1816–1820, 2019.
  * [22] J. A. Tropp, A. C. Gilbert, and M. J. Strauss,  “Algorithms for simultaneous sparse approximation. part i: Greedy pursuit,”  Signal Process., vol. 86, no. 3, pp. 572–588, 2006.


11institutetext:  HLTCOE. Johns Hopkins University, Baltimore MD 21211, USA
11email<EMAIL_ADDRESS>22institutetext: University of
Maryland, College Park MD 20742, USA 22email<EMAIL_ADDRESS>
# Neural Approaches to Multilingual Information Retrieval

Dawn Lawrie 11 0000-0001-7347-7086 Eugene Yang 11 0000-0002-0051-1535
Douglas W. Oard 1122 0000-0002-1696-0407 James Mayfield 11 0000-0003-3866-3013

###### Abstract

Providing access to information across languages has been a goal of
Information Retrieval (IR) for decades. While progress has been made on Cross
Language IR (CLIR) where queries are expressed in one language and documents
in another, the multilingual (MLIR) task to create a single ranked list of
documents across many languages is considerably more challenging. This paper
investigates whether advances in neural document translation and pretrained
multilingual neural language models enable improvements in the state of the
art over earlier MLIR techniques. The results show that although combining
neural document translation with neural ranking yields the best Mean Average
Precision (MAP), 98% of that MAP score can be achieved with an 84% reduction
in indexing time by using a pretrained XLM-R multilingual language model to
index documents in their native language, and that 2% difference in
effectiveness is not statistically significant. Key to achieving these results
for MLIR is to fine-tune XLM-R using mixed-language batches from neural
translations of MS MARCO passages.

###### Keywords:

multilingual ad-hoc retrieval ColBERT-X DPR-X multilingual training of MPLM

##  1 Introduction

With advances in neural models for machine translation (MT) and Information
Retrieval (IR), it is time to revisit the problem of Multilingual IR (MLIR).
Soon after Cross-Language IR (CLIR) was proposed as an information retrieval
task, research began on MLIR [34]. MLIR seeks to produce a total ordering over
retrieved documents, regardless of language, such that the most useful
documents appear at the top of the ranking. Assuming a searcher can consume
multilingual information (either directly or using MT), the search engine
should be able to return useful information regardless of the language of the
document.

Much prior work on MLIR has involved subsetting documents by language,
performing CLIR on each document set, and merging the results [37]. The advent
of neural machine translation and neural IR using Multilingual Pretrained
Language Models (MPLMs) creates new opportunities for MLIR that we study here.

If MT were perfect, translating all documents into the query language and
searching monolingually might suffice. Indeed, our experiments confirm that
for the high-resource languages with which we have experimented (English,
French, German, Italian, and Spanish), using neural machine translation to
convert each document into the query language is effective when used with
neural ranking (in our experiments, ColBERT [26]) fine-tuned on MS MARCO [2].
However, using neural MT in that way incurs substantial indexing costs because
a GPU is required first to translate the document and then again to encode it
into dense vectors for neural IR. Alternatively, we can use translations of MS
MARCO to fine-tune an MPLM; that approach is nearly as effective, not
statistically different, and considerably faster at indexing time. Our use of
MS MARCO makes English a natural choice as the query language, but our
approach is extensible to any query language for which suitable fine-tuning
data exists.

This paper makes the following contributions: (1) Using a collection
containing five high-resource European languages, we show that neural MT with
neural IR achieves higher MAP and Precision at 10 scores than any other known
MLIR technique, but that reliance on neural MT greatly increases the time
required to index a collection. (2) We show that extending the ColBERT-X [32]
Translate-Train (TT) CLIR model to multiple languages achieves equivalent
retrieval effectiveness with less than half the indexing time when used with
mixed-language fine-tuning. (3) We show that some language bias in favor of
query-language documents is present with all approaches, but that query-
language bias is smaller with our Multilingual Translate-Train (MTT)
implementation of ColBERT-X.

##  2 Background

We provide an overview of MLIR, followed by a brief review of traditional and
neural IR. The term “multilingual” has been used in several ways in IR. Hull
and Grefenstette [22], for example, note that it has been used to describe
monolingual retrieval in multiple languages, as in Blloshmi et al. [5], and it
has also been used to describe CLIR tasks that are run separately in several
languages [27, 7, 8, 9, 31]. We adopt the Cross-Language Evaluation Forum
(CLEF)’s meaning of MLIR: using a query to construct one ranked list in which
each document is in one of several languages [36]. We note that this
definition excludes mixed-language queries and mixed-language documents, which
are yet other cases to which “multilingual” has been applied.

Five broad approaches to MLIR have been tried. Among the earliest, Rehder et
al. [39] represented English, German and French documents in a learned
trilingual embedding space, represented the query in the same embedding space,
and then computed query-document similarity in the embedding space. The
techniques and training data for creating multilingual embeddings were,
however, too limited at the time to get good results from that technique. More
recently, Sorg and Cimiano [44] garnered substantial attention by training
embeddings on topically-related Wikipedia pages in English, German, French and
Spanish. This paper extends this line of work.

A second approach by Nie and Jin [33] indexed terms from all documents in
their original language then created queries containing translations of the
query terms in all target languages. With many document languages, this can
lead to long queries. A third approach is to translate indexed terms into the
query language at indexing time; the original queries can then be used
directly to find similar (translated) content [29, 18, 38]. We experiment with
this approach as well. This approach is, however, only practical when just a
few query languages are to be supported. To address that limitation, the
second and third approaches can be combined to create a fourth approach in
which documents and query terms are each converted into one of a small number
of indexing languages. This has been called a “pivot language” approach,
because in the limit all documents and queries can be translated into a single
language.

The fifth, and most widely studied, approach is to first use monolingual or
bilingual retrieval to create a ranked list for each document language, and
then to merge those ranked lists to construct a single result list [37, 43,
45]. While this approach is architecturally similar to collection sharding, a
widely-used approach to address efficiency, differences in collection
statistics result in incompatible scores that require normalization prior to
late fusion. Unfortunately, normalizing scores for collections across
languages has been shown to be challenging [37].

Finally, one can simply show one ranked list per language to the user, as is
done in the 2lingual search engine.111https://www.2lingual.com/ This approach
does not scale well beyond a small number of languages, but it has the
advantage of making it fairly clear to the searcher what the search engine has
done.

Every MLIR ranked retrieval model must rank the indexed documents given a
query. Traditional ranking methods such as computing inner products between
the query and each indexed document containing a query term using sparse BM25
[40] term weights are fast, but neural IR methods yield better rankings [26,
24, 32] with more relevant documents earlier in the ranked list.

This paper focuses on tradeoffs between effectiveness and efficiency. Each
technique described in this paper achieves ranking latency sufficient for
interactive use (below 300 ms) on the collections that we experiment with, but
the time required to index the documents varies. Indexing time consists of
three components: text processing (e.g., casing and tokenization), machine
translation, and representation (e.g., McCarley [30] and Magdy and Jones
[29]). Of these, neural MT is the slowest, so IR methods that do not require
neural MT at indexing time have a substantial indexing time advantage (e.g.,
Aljlayl and Frieder [1]). Our principal MLIR result is that MPLMs can achieve
MAP close to the best results while producing substantial savings in indexing
time.

We achieve this by extending the ColBERT-X [32] CLIR model to perform MLIR.
ColBERT-X combines three key ideas. First, drawing insight from BERT [15], it
represents documents using contextual embeddings, which better represent
meaning than simple term occurrence. Second, using both multilinguality and
improved pretraining from either multilingual BERT [47] or XLM-R [11],
ColBERT-X generates similar contextual embeddings for terms with similar
meaning, regardless of language. Third, drawing its structure from ColBERT
[26], ColBERT-X limits ranking latency by separating query and document
transformer networks, allowing offline indexing. ColBERT scores documents by
focusing query term attention on the most similar contextual embedding in each
document. Our experiments confirm that this approach yields better MLIR MAP
than does computation of inner products between classification tokens for the
query and each document, an approach known as Dense Passage Retrieval (DPR)
[24].

##  3 Fine-Tuning MPLMs for MLIR

Following Nair et al. [32] we consider two high-level approaches to fine-
tuning for generalizing neural retrieval models to MLIR. Both approaches use
existing MPLMs such as XLM-R [11] to encode queries and documents in multiple
languages. We adapt the MPLM to MLIR via task-specific fine-tuning. These
approaches are applicable to any retrieval model that is able to encode text
using an MPLM.

Consider a set of queries in a source language $\mathbf{L}_{s}$ and a set of
documents in $m$ target languages
$\mathbf{L}_{t}=\cup_{i=i}^{m}\mathbf{L}_{i}$. We want to train a scoring
function $\mathcal{M}_{\Theta}(q_{(s)},d_{(t)})\rightarrow\mathbb{R}$ for
ranking documents with respect to a query. This paper denotes the language of
an instance as a subscript $\bullet_{(l)}$.

###  3.1 English Training (ET)

Since MPLMs can encode text from many languages, we follow Nair et al. [32]
and only fine-tune the model monolingually. When processing queries, we
transfer the model to MLIR zero-shot. Specifically, consider a loss function
$\mathcal{L}$ (for example, cross-entropy),

$\Theta=\operatorname*{arg\,min}_{\theta}\sum_{q,d}\mathcal{L}_{\theta}(q_{(s)},d_{(s)},r_{q,d})$

where $q_{(s)}$ and $d_{(s)}$ are representations of the queries and documents
and $r_{q,d}$ is the relevance judgment of document $d$ on query $q$, both in
language $\mathbf{L}_{s}$, encoded by an MPLM. We use English as our query
language because that is the language of MS MARCO. We refer to this approach
as “English Training” or ET. However, this approach could equally well use any
language for which similar extensive training data is available.

Despite only exposing the model to text in $\mathbf{L}_{s}$ during fine-
tuning, the multilingual model can transfer its task model to other languages,
as has been seen in prior CLIR work [32]. However, such zero-shot language
transfer is suboptimal because of (1) the lack of alignment objectives between
languages during pretraining [48]; and (2) differences in the representation
of each language by the MPLM, which has been called the curse of
multilinguality [11, 46]. As we show in Section 6.1, such zero-shot transfer
not only produces suboptimal retrieval effectiveness, it can also lead to
language bias.

###  3.2 Multilingual Translate Training (MTT)

To mitigate those issues, we propose a Multilingual Translate-Train (MTT)
approach that generalizes the CLIR Translate-Train (TT) approach to MLIR [42,
32]. To expose target languages $\mathbf{L}_{1}...\mathbf{L}_{m}$ to the
model, we translate the monolingual training documents into each target
language using MT. Specifically, the training objective can be expressed as

$\Theta=\operatorname*{arg\,min}_{\theta}\sum_{q,d}\sum_{l=1}^{m}\mathcal{L}_{\theta}(q_{(s)},d_{(l)},r_{q,d})$

This objective exposes the retrieval model to language pairs that it might see
when processing queries, resulting in a more effective, better-balanced model.
We experiment with two batching approaches. In Mixed-language (MTT-M), each
batch contains documents in multiple languages, which encourages the model to
learn similarity measures for all languages simultaneously.222Batches include
the same query paired with document passages translated into each language.
With Single-language (MTT-S), each batch contains only documents in one
language, helping the model to learn retrieval for one language pair at a
time. We found that MTT-M yields better retrieval effectiveness; thus, we
present MTT-M as our main result. Section 5.1 compares the two approaches. In
Section 6.1, we also demonstrate that MTT-M reduces language bias in MLIR.
Implementation details can be found in Appendix 0.A

Table 1: Dataset statistics of CLEF 2001, 2002, and 2003. CLEF 2001 and 2002
share the document collection but have different queries. Numbers in
parentheses are the number of topics in each query set. We report the number
of documents judged relevant over all the topics in a particular year.

Query | English | German | Spanish | French | Italian | Total
---|---|---|---|---|---|---
Set | # Rel. | # Docs | # Rel. | # Docs | # Rel. | # Docs | # Rel. | # Docs | # Rel. | # Docs | # Rel. | # Docs
2001 (50) | 856 | 113,005 | 2,130 | 225,371 | 2,694 | 215,738 | 1,212 | 87,191 | 1,246 | 108,578 | 8,138 | 749,883
2002 (50) | 821 | 1,938 | 2,854 | 1,383 | 1,072 | 8,068
2003 (60) | 1,006 | 169,477 | 1,825 | 294,809 | 2,367 | 454,045 | 946 | 129,806 | – | – | 6,144 | 1,048,137

##  4 Experiments

One of the few test collections that currently supports MLIR evaluation with
relevance judgments across multiple languages is from the the Cross-Language
Evaluation Forum (CLEF). Following Rahimi et al. [38] we use five document
languages in the CLEF 2001-2002 collections [7, 8] and four languages in the
CLEF 2003 collection [9]. Table 1 shows collection statistics. We report
performance for both title and title+description queries, also following
Rahimi et al. [38]. Because the number of query elements (subwords) is limited
when encoding a query for dense retrieval, we remove stop structure to ensure
that no query exceeds the length limit. Stop structure includes phrases such
as “Find documents" and a limited stop-word list including “on," “the," and
“and."333For a complete list:
https://github.com/hltcoe/ColBERT-X/blob/main/scripts/stopstructure.txt

###  4.1 Neural Retrieval Models

We evaluate our proposed training approaches on two retrieval models –
ColBERT-X [32] and DPR-X [48, 49], which are multilingual variants of ColBERT
[26] and DPR [24]. Nair et al. [32] generalized the ColBERT [26] model to
CLIR, calling it ColBERT-X, by modifying the vocabulary space and replacing
the monolingual pretrained language model with the MPLM XLM-RoBERTa (XLM-R)
Large (550M parameters) [11]. With proper training, ColBERT-X achieves state-
of-the-art effectiveness in CLIR. In this study, we integrate our proposed
fine-tuning approaches with the ColBERT-X XLM-R implementation, which is based
on the ColBERTv1 code base. We similarly adapted DPR [24, 48], a neural
retrieval model that matches a single dense query vector to a single dense
document vector. We name this model DPR-X. We use Tevatron [17], an open-
source implementation of several neural end-to-end retrieval models in Python,
for training, indexing, and retrieval.

For training data, we use MS MARCO-v1 [2], a commonly-used question-answering
collection in English for fine-tuning neural retrieval models. For MTT, we use
the publicly available mMARCO translations of MS MARCO [6], fine-tuning using
the “small training triple” (query, positive and negative document) file
released by mMARCO’s creators. We trained all retrieval models with four GPUs
(NVIDIA DGX and v100 with 32 GB Memory) with a per-GPU batch size of 32
triples for 200,000 update steps. All models are trained with half-precision
floating points and optimized by the AdamW optimizer with a learning rate of
$5\times 10^{-6}$.

During indexing, documents are separated into overlapping spans of 180 tokens
with a stride of 90 [32]. We aggregate by MaxP [3, 13], which takes the
maximum score among the passages in a document as the document score.

###  4.2 Evaluation

We report previously published results for the state-of-the-art MULM [38]
system as a baseline for models that do not perform MT on the full collection.
MULM is essentially an MLIR version of Probabilistic Structured Queries (PSQ)
[14]. PSQ maps term frequency vectors from document to query language using a
matrix of translation probabilities generated using statistical machine
translation. For MLIR, a translation matrix is created for each query-document
language pair. The query likelihood model is used to score documents. Three
key decisions led to good performance: (1) estimating collection statistics
based on translation probabilities; (2) estimating document length based on
the translation and using that for smoothing; and (3) truncating the
translation list at three. As another baseline, we use BM25 ($b=0.4$,
$k_{1}=0.9$) as implemented in Patapsco [12] over neural machine translated
documents (abbreviated ITD for Indexed Translated Documents). For BM25,
English queries and documents are tokenized by spaCy [21] and stemmed by the
NLTK [4] Porter stemmer (all supported by Patapsco).

For approaches that require document translation, we use directional MT models
built on a transformer architecture (6-layer encoder/decoder) using Sockeye 2
[16, 19]. Measured by BLEU [35], Sockeye 2 achieves state-of-the-art
effectiveness in each translation direction. Optimizations cut decoding time
in half compared to Sockeye 1 [20]. We chose Sockeye 2 for its good trade-off
between efficiency and effectiveness.

To evaluate effectiveness on multiple languages in CLEF 2001-2002 and CLEF
2003, we combine the relevance judgments (qrels) for all languages for each
query. In general, different languages have different numbers of relevant
documents for each query. To evaluate models trained with English training
data, we also translate the document sets into English with MT for indexing.
Our main effectiveness measures are Mean Average Precision (MAP) and Precision
at 10 (P@10). Both measures focus on the top of the rankings, and both were
used by Rahimi et al. [38], facilitating comparison between the neural
approaches presented herein and prior state-of-the-art results.

To evaluate language bias, we count the number of relevant documents for a
query across all languages, and calculate recall at that level. To compute the
measure for a specific language, we keep this level constant, but ignore all
documents in other languages (both in the MLIR results and in the relevance
judgments). We call the mean of this measure over all queries Recall@MLIR-
Relevant. When computing the mean, we omit from the calculation cases in which
no relevant documents in that language are known (recall is undefined in such
cases). This measure lies between 0 and 1, and values across that full range
are achievable. We use the open source ir-measures [28]444https://ir-
measur.es/ package to compute all effectiveness measures.

##  5 Results

Table 2: Configurations of experiments identifying the pre-trained language
model when applicable, the fine tuning data and process, the retrieval model,
and the language of the indexed documents. Under Fine-Tuning Data, MS MARCO
refers to English MS MARCOv1, while mMARCO includes the translations into the
various languages as well as the original English MS MARCOv1. A model that
lists either under its Indexing Language can index either machine translated
document (translation) or native documents in their various languages.

| Language | Fine-Tuning | Fine-Tuning | Retrieval | Indexing
---|---|---|---|---|---
Name | Model | Data | Process | Model | Language
MULM | – | – | – | PSQ | native
BM25-ITD | – | – | – | BM25 | translation
ColBERT-X(ET) | XLM-R | MS MARCO | ET | ColBERT-X | either
ColBERT-X(MTT-M) | XLM-R | mMARCO | MTT-M | ColBERT-X | either
ColBERT-X(MTT-S) | XLM-R | mMARCO | MTT-S | ColBERT-X | either
DPR-X(ET) | XLM-R | MS MARCO | ET | DPR-X | either
DPR-X(MTT-M) | XLM-R | mMARCO | MTT-M | DPR-X | either
ColBERT(ET) | BERT | MS MARCO | ET | ColBERT | translation

We experiment with the Multilingual Translation Training (MTT) using two
retrieval models and compare them to two strong baseline retrieval models:
BM25-ITD indexing translated documents and MULM indexing native documents;
these represent the state of the art on our test collections. Since per-query
results for MULM have not been published we perform significance tests only
between our systems and the BM25+ITD baseline (the stronger of the two
baselines). Table 2 summarizes the experiments that facilitate this analysis.
We first compare the effectiveness of our two batching strategies for MTT
before examining their effectiveness relative to the baselines. Finally, we
consider the trade-off between effectiveness and indexing time.

###  5.1 Multilingual Batching for Fine-Tuning

Table 3: ColBERT-X MTT for Multiple or Single language training batches, indexing documents in their native language using title+description queries. $\dagger$ indicates significant improvement over MTT-S by paired $t$-test with 3-test Bonferroni correction ($p<0.05$).  | MAP | P@10
---|---|---
| 2001 | 2002 | 2003 | 2001 | 2002 | 2003
MTT-M | 0.462$\dagger$ | 0.462$\dagger$ | 0.461$\dagger$ | 0.704 | 0.752 | 0.653
MTT-S | 0.422 | 0.405 | 0.433 | 0.696 | 0.702 | 0.649

We compare two alternatives for fine-tuning the MTT condition and summarize
the results with title+description queries in Table 3. In all cases, mixed-
language batches (MTT-M) produce more effective retrieval models than single-
language (MTT-S). This is likely because, in MLIR, the model must rank
documents from different languages together instead of transferring trained
models to other languages. The outcome might be different if our goal were to
perform CLIR over monolingual document collections.

###  5.2 Effectiveness Relative to Baselines

Table 4: MAP and P@10 on CLEF Title and Title+Description queries. Bold are
best among a year; italics are best in a row (i.e., with and without neural
machine translation), $\dagger$ indicates significant difference from BM25+ITD
by paired $t$-test with 16-test Bonferroni correction ($p<0.05$).

|  | MAP | P@10
---|---|---|---
Query |  | MULM | BM25 | ColBERT-X | DPR-X | MULM | BM25 | ColBERT-X | DPR-X
Set | ITD | MTT-M | ET | MTT-M | ET | MTT-M | ET | MTT-M | ET
Title Queries
2001 | ✓ | – | 0.398 | 0.377 | 0.391 | 0.338 | 0.344 | – | 0.648 | 0.612 | 0.596 | 0.548 | 0.584
✗ | 0.349 | – | 0.360 | 0.322 | 0.327 | 0.298$\dagger$ | 0.650 | – | 0.600 | 0.588 | 0.592 | 0.570
2002 | ✓ | – | 0.337 | 0.367 | 0.389 | 0.287 | 0.304 | – | 0.618 | 0.606 | 0.670 | 0.530 | 0.596
✗ | 0.276 | – | 0.352 | 0.333 | 0.282 | 0.277 | 0.592 | – | 0.614 | 0.622 | 0.544 | 0.556
2003 | ✓ | – | 0.349 | 0.337 | 0.349 | 0.276$\dagger$ | 0.266$\dagger$ | – | 0.595 | 0.542 | 0.573 | 0.517 | 0.497$\dagger$
✗ | 0.305 | – | 0.332 | 0.290 | 0.273$\dagger$ | 0.247$\dagger$ | 0.497 | – | 0.546 | 0.541 | 0.527 | 0.492$\dagger$
All | ✓ | – | 0.361 | 0.359 | 0.375 | 0.299$\dagger$ | 0.302$\dagger$ | – | 0.619 | 0.583 | 0.611 | 0.531$\dagger$ | 0.554$\dagger$
✗ | 0.310 | – | 0.347 | 0.314$\dagger$ | 0.293$\dagger$ | 0.273$\dagger$ | 0.575 | – | 0.584 | 0.581 | 0.553$\dagger$ | 0.536$\dagger$
Title + Description Queries
2001 | ✓ | – | 0.436 | 0.472 | 0.477 | 0.365 | 0.356 | – | 0.704 | 0.718 | 0.754 | 0.658 | 0.650
✗ | 0.387 | – | 0.462 | 0.405 | 0.358 | 0.324$\dagger$ | 0.700 | – | 0.704 | 0.744 | 0.658 | 0.644
2002 | ✓ | – | 0.398 | 0.470$\dagger$ | 0.480$\dagger$ | 0.347 | 0.332 | – | 0.696 | 0.774 | 0.770 | 0.664 | 0.620
✗ | 0.347 | – | 0.462 | 0.410 | 0.335 | 0.310 | 0.666 | – | 0.752 | 0.720 | 0.672 | 0.640
2003 | ✓ | – | 0.394 | 0.419 | 0.410 | 0.343 | 0.328$\dagger$ | – | 0.615 | 0.646 | 0.661 | 0.620 | 0.600
✗ | 0.376 | – | 0.409 | 0.358 | 0.338 | 0.302$\dagger$ | 0.563 | – | 0.653 | 0.637 | 0.622 | 0.575
All | ✓ | – | 0.408 | 0.451$\dagger$ | 0.453$\dagger$ | 0.351$\dagger$ | 0.338$\dagger$ | – | 0.669 | 0.709 | 0.725$\dagger$ | 0.646 | 0.622
✗ | 0.368 | – | 0.442 | 0.390 | 0.343$\dagger$ | 0.312$\dagger$ | 0.643 | – | 0.700 | 0.697 | 0.639 | 0.617

Our main effectiveness results are shown in Table 4. For ColBERT-X and DPR-X,
MTT-M consistently improves effectiveness when retrieving documents in their
native language (i.e., without document MT) compared to English Training (ET).
Such improvements are seen in all three query sets, and for both Title (T) and
Title+Description (T+D) queries. Differences are larger for MAP than P@10,
indicating that MTT-M affects more than just the top ranks.

ColBERT-X MTT-M numerically outperforms MULM for both query types and over all
collections in MAP and nearly all collections in P@10. With longer, more
fluent title+description queries, ColBERT-X MTT-M gives a larger improvement
over MULM in both MAP and P@10, indicating that XLM-R favors queries with more
context. Since DPR-X is less effective [48], MTT-M only brings its performance
up to par with MULM.

With modern MT models, we can improve MLIR effectiveness. A common, yet
strong, baseline of using BM25 to search over translated documents yields
substantial improvement over MULM in both MAP and P@10 with both query types.
We argue that BM25+ITD is a proper baseline to which future MLIR experiments
should be compared.

Table 5: Monolingual ColBERT model using BERT-Large trained with ET and
evaluated with translated documents.

| MAP | P@10
---|---|---
Queries | 2001 | 2002 | 2003 | All | 2001 | 2002 | 2003 | All
T | 0.397 | 0.367 | 0.362 | 0.375 | 0.592 | 0.646 | 0.583 | 0.606
T+D | 0.439 | 0.413 | 0.420 | 0.424 | 0.736 | 0.714 | 0.673 | 0.706

We can also reduce neural IR to the monolingual case, training our retrieval
model with English training and searching documents represented by English
machine translations. For both ColBERT and DPR, an English-trained model (ET)
indexing translated documents often yields better effectiveness than MTT-M
indexing translated documents (ITD). Furthermore, an English trained model
indexing translated documents yields better effectiveness than MTT-M indexing
documents in their native language; however, these differences are only
statistically significant for CLEF 2002 on Title queries using a paired
$t$-test with 3-test Bonferroni correction ($p<0.05$). We observe similar
results with ColBERT using the BERT-Large pretrained LM trained under the same
conditions except for using a learning rate of $3\times 10^{-6}$ (the value
suggested by the authors). Compare Table 5 to ColBERT-X with English training,
presented in Table 4.

###  5.3 Preprocessing and Indexing Time

Table 6: ColBERT-X GPU hours for translating and indexing. BM25 does not use GPU. |  | CLEF2001-2002 | CLEF2003
---|---|---|---
Model | ITD | Translation | Index | Total | Translation | Index | Total
BM25 | ✓ | 55.0 | – | 55.0 | 68.6 | – | 68.6
ET | ✓ | 55.0 | 9.3 | 64.3 | 68.6 | 12.3 | 80.9
✗ | – | 9.9 | 9.9 | – | 12.4 | 12.4
MTT-S | ✓ | 55.0 | 16.9 | 71.9 | 68.6 | 19.0 | 87.6
✗ | – | 16.7 | 16.7 | – | 21.9 | 21.9
MTT-M | ✓ | 55.0 | 17.3 | 72.3 | 68.6 | 20.1 | 88.7
✗ | – | 15.1 | 15.1 | – | 19.3 | 19.3
Figure 1: Effectiveness (MAP) vs. efficiency (per-document GPU indexing time
in seconds) trade-off on CLEF 2001-2003. MAP scores (y-axis) for Title and
Title+Description queries are disjoint ranges. The upper left is the optimal
part of the chart.

Applying machine translation to entire document collections is expensive.
Table 6 summarizes the cost for preprocessing and indexing the collection in
GPU-hours for ColBERT-X and BM25. We omit consideration of query latency here
since all of our systems are sufficiently fast at query time for interactive
use on collections of this size. We refer the interested reader to Santhanam
et al. [41].

This table reveals that differences in total indexing time between searching
native and translated documents range from four to 6.5 times depending on
collection size and model.555Although Marian [23] is faster than Sockeye 2,
benchmark results from Sockeye 1 [20] and Sockeye 2 [19] confirm that Sockeye
2 is within a factor of 2 to 3 of Marian’s speed, leaving our conclusions
unchanged. Despite that searching translated documents with monolingual
retrieval models is more effective, the computational cost of MT at indexing
time is significantly higher; one might choose not to bear this cost in
exchange for the small and not statistically significant numerical gain in
measured effectiveness over searching documents in their native language with
MTT-M fine-tuning for title+description queries.

We also see this trade-off on a per-document basis. Figure 1 shows that
ColBERT-X with English training searching translated documents
(ColBERT-X(ET)+ITD) achieves the best effectiveness with both title (0.375
MAP) and title+description (0.453 MAP) queries. However, it has a high
preprocessing cost of 0.32 seconds per document, whereas ColBERT-X trained
with MTT-M searching documents in their native languages (ColBERT-X(MTT-M))
requires under 0.05 seconds per document. This is an 84% reduction in
preprocessing cost at an apparent (but not statistically significant) cost of
only 2% in MAP with title+description queries.

##  6 Analysis

This section investigates our experimental results by breaking down the
collection in two ways – by document language, and by topic.

###  6.1 Language Bias

Figure 2: R@MLIR-Relevant of BM25 and ColBERT-X variants for each language in
CLEF2001-2002 with title+description queries. The yellow dashed line is the
average over all languages, i.e., the R-Precision in MLIR. Outliers are
defined as values beyond 1.5$\times$interquartile range. Horizontal black bars
indicate the median and white circles indicate the mean.

Since MPLMs are known to exhibit language biases [10, 25], we investigate how
retrieval models fine-tuned with our training schemes inherit or alleviate
these biases. In MLIR we consider a model biased if it ranks a language’s
documents systematically higher or lower than those of another language. While
MLIR is not a new task, we are not aware of prior work that has examined
language bias. Therefore we introduce two approaches to studying this
phenomenon. The first approach examines rates of relevant documents. Since
relevant documents are unevenly distributed across languages (e.g., Spanish
has more than three times as many known relevant documents as English among
the CLEF 2001 topics, averaging 54 vs. 17 relevant documents per topic,
respectively), meaningful comparisons require us to focus on rates rather than
on counts. In this analysis, we focus on Recall@MLIR-Relevant (see Section
4.2), illustrating our analysis using the 100 title+description queries in
CLEF 2001-2002 topics to characterize the coverage of relevant documents in
each language (results on CLEF 2003 topics are similar).

Figure 2 shows distributional statistics of Recall@MLIR-Relevant over topics
by language and condition that have at least one known relevant document in
that language (96 for German, 97 for Spanish, 94 for Italian, 90 for French,
73 for English). When transferring a ColBERT-X model fine-tuned zero-shot with
English training (i.e., ColBERT-X(ET)) to other languages, the model favors
English documents due to the fine-tuning condition. This results in a strong
language bias in the retrieval results. Such biases can be ameliorated by
fine-tuning with MTT. MTT-M appears to have more consistent behavior across
languages compared to MTT-S, although the small apparent difference is not
statistically significant. When indexing translated documents, Recall@MLIR-
Relevant tends to be lower for English compared to other languages (though
also not significantly). Since documents were translated sentence-by-sentence,
we hypothesize that indexing translated documents provides more synonym
variety when decoding similar terms, resulting in document expansion; this
hypothesis requires more investigation, which we leave for future work.

An alternative approach to investigating language bias is to assume that in a
bias-free approach to MLIR, the scores for relevant documents would be drawn
from the same underlying distribution. Using the 2-sample Kolmogorov-Smirnov
test, the null hypothesis is that the two samples are drawn from the same
distribution. For this analysis, we chose English as a reference and tested
each topic with at least three relevant documents in each language. We then
adjusted the p-values to account for multiple comparisons. We found that we
could reject the null hypothesis for all languages and all configurations,
indicating the document scores are not drawn from the same distribution based
on language. Although some of this difference could result from differences in
collection statistics (i.e., with some languages better supporting the queries
than others based on the numbers of relevant documents), the differences we
observe across retrieval models indicate that there are retrieval model
effects as well. Notably, ColBERT-X(ET) retrieving documents in the native
language has the largest percentage of topics with bias (from 15% to 30%
depending on language pair), while all other configurations have no more than
12% of topics exhibiting biased scores. This confirms the qualitative analysis
above, which revealed that ColBERT-X(ET) over the documents in their native
language had the most skewed rates of relevant documents. Future research will
need to address language bias in document scores.

###  6.2 Example Queries

Figure 3: Average Precision (AP) of BM25 and ColBERT-X on selected topics
using title queries.

For more insight into differences among the algorithms, we show effectiveness
on individual queries in Figure 3. Our query selection here is not meant to be
representative, but rather illustrative of phenomena that we see. For two
topics on which ColBERT-X outperformed BM25 (topics 158 and 118), the queries
include terms that likely benefit from ColBERT-X soft term-matching – “soccer”
and “commissioner” respectively. This term expansion effect has also been
observed in monolingual retrieval with ColBERT.

MT is particularly helpful for topics 63 and 88, likely due to the quality of
the translation for documents on these topics. Especially for topic 88,
English monolingual retrieval produces strong results. Such behaviors indicate
that the multilingual term matching in ColBERT-X is still not as effective on
less common concepts like “mad cow” as is machine translation.

Topic 58 is an outlier. The term “euthanasia” is tokenized as a single token
for BM25 but separated into _eu, thana, and sia by the XLM-R tokenizer;
combined with the minimal context provided by a query, this prevents ColBERT-X
from matching properly across languages. Such diverse behaviors suggest room
for further MLIR improvements using system combination.

##  7 Conclusion and Future Work

This paper proposes the MTT training approach to MLIR that uses translated MS
MARCO. When searching non-English documents, fine-tuning with MTT using mixed-
language batches (MTT-M) enables neural models such as ColBERT and DPR to be
more effective than if fine-tuned on English MS MARCO. ColBERT-X with MTT-M is
not statistically different from monolingual English models applied to neural
indexing-time translation of the collection into English, yet it achieves
substantially better indexing time efficiency. These results may not hold for
more diverse sets of languages or when MT is less effective; future work will
examine the multilingual topics from the TREC 2022 the NeuCLIR
track,666https://neuclir.github.io/ which judges the relevance of documents
written in Chinese, Persian, and Russian. Our observation that the retrieval
method that yields the best retrieval effectiveness is query-dependent
suggests future work on system combination, but our focus on efficiency and on
language bias also calls attention to issues beyond retrieval effectiveness
that will merit consideration in such a study.

## References

  * Aljlayl and Frieder [2001] Aljlayl, M., Frieder, O.: Effective Arabic-English cross-language information retrieval via machine-readable dictionaries and machine translation. In: Proceedings of the Tenth International Conference on Information and Knowledge Management, pp. 295–302 (2001)
  * Bajaj et al. [2016] Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al.: MS MARCO: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268 (2016)
  * Bendersky and Kurland [2008] Bendersky, M., Kurland, O.: Utilizing passage-based language models for document retrieval. In: European Conference on Information Retrieval, pp. 162–174, Springer (2008)
  * Bird et al. [2009] Bird, S., Klein, E., Loper, E.: Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. O’Reilly Media, Inc. (2009)
  * Blloshmi et al. [2021] Blloshmi, R., Pasini, T., Campolungo, N., Banerjee, S., Navigli, R., Pasi, G.: IR like a SIR: Sense-enhanced Information Retrieval for Multiple Languages. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1030–1041, Association for Computational Linguistics, Online and Punta Cana, Dominican Republic (Nov 2021), doi:10.18653/v1/2021.emnlp-main.79, URL https://aclanthology.org/2021.emnlp-main.79
  * Bonifacio et al. [2021] Bonifacio, L.H., Jeronymo, V., Abonizio, H.Q., Campiotti, I., Fadaee, M., , Lotufo, R., Nogueira, R.: mMARCO: A multilingual version of MS MARCO passage ranking dataset. arXiv preprint arXiv:2108.13897 (2021)
  * Braschler [2001] Braschler, M.: CLEF 2001—overview of results. In: Workshop of the Cross-Language Evaluation Forum for European Languages, pp. 9–26, Springer (2001)
  * Braschler [2002] Braschler, M.: CLEF 2002—overview of results. In: Workshop of the Cross-Language Evaluation Forum for European Languages, pp. 9–27, Springer (2002)
  * Braschler [2003] Braschler, M.: CLEF 2003–overview of results. In: Workshop of the Cross-Language Evaluation Forum for European Languages, pp. 44–63, Springer (2003)
  * Choudhury and Deshpande [2021] Choudhury, M., Deshpande, A.: How linguistically fair are multilingual pre-trained language models? In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 12710–12718 (2021)
  * Conneau et al. [2020] Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., Stoyanov, V.: Unsupervised cross-lingual representation learning at scale. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440–8451, Association for Computational Linguistics, Online (Jul 2020), URL https://aclanthology.org/2020.acl-main.747
  * Costello et al. [2022] Costello, C., Yang, E., Lawrie, D., Mayfield, J.: Patapasco: a Python framework for cross-language information retrieval experiments. In: European Conference on Information Retrieval, pp. 276–280, Springer (2022)
  * Dai and Callan [2019] Dai, Z., Callan, J.: Deeper text understanding for IR with contextual neural language modeling. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 985–988 (2019)
  * Darwish and Oard [2003] Darwish, K., Oard, D.W.: Probabilistic structured query methods. In: Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 338–344 (2003)
  * Devlin et al. [2019] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186, Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019), URL https://aclanthology.org/N19-1423
  * Domhan et al. [2020] Domhan, T., Denkowski, M., Vilar, D., Niu, X., Hieber, F., Heafield, K.: The Sockeye 2 neural machine translation toolkit at AMTA 2020. In: Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pp. 110–115, Association for Machine Translation in the Americas, Virtual (Oct 2020)
  * Gao et al. [2022] Gao, L., Ma, X., Lin, J.J., Callan, J.: Tevatron: An efficient and flexible toolkit for dense retrieval. arXiv preprint arXiv:2203.05765 (2022)
  * Granell [2014] Granell, X.: Multilingual information management: Information, technology and translators. Chandos Publishing (2014)
  * Hieber et al. [2020] Hieber, F., Domhan, T., Denkowski, M., Vilar, D.: Sockeye 2: A toolkit for neural machine translation. In: EAMT 2020 (2020), URL https://www.amazon.science/publications/sockeye-2-a-toolkit-for-neural-machine-translation
  * Hieber et al. [2017] Hieber, F., Domhan, T., Denkowski, M., Vilar, D., Sokolov, A., Clifton, A., Post, M.: Sockeye: A toolkit for neural machine translation. arXiv preprint arXiv:1712.05690 (2017)
  * Honnibal et al. [2020] Honnibal, M., Montani, I., Van Landeghem, S., Boyd, A.: spaCy: Industrial-strength Natural Language Processing in Python. Tech. rep., Explosion (2020)
  * Hull and Grefenstette [1996] Hull, D.A., Grefenstette, G.: Querying across languages: A dictionary-based approach to multilingual information retrieval. In: Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 49–57 (1996)
  * Junczys-Dowmunt et al. [2018] Junczys-Dowmunt, M., Heafield, K., Hoang, H., Grundkiewicz, R., Aue, A.: Marian: Cost-effective high-quality neural machine translation in C++. arXiv preprint arXiv:1805.12096 (2018)
  * Karpukhin et al. [2020] Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., Yih, W.t.: Dense passage retrieval for open-domain question answering. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769–6781, Association for Computational Linguistics, Online (Nov 2020), URL https://aclanthology.org/2020.emnlp-main.550
  * Kassner et al. [2021] Kassner, N., Dufter, P., Schütze, H.: Multilingual lama: Investigating knowledge in multilingual pretrained language models. arXiv preprint arXiv:2102.00894 (2021)
  * Khattab and Zaharia [2020] Khattab, O., Zaharia, M.: ColBERT: Efficient and effective passage search via contextualized late interaction over BERT. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 39–48 (2020)
  * Lawrie et al. [2022] Lawrie, D., Mayfield, J., Oard, D.W., Yang, E.: HC4: A new suite of test collections for ad hoc CLIR. In: Proceedings of the 44th European Conference on Information Retrieval (2022)
  * MacAvaney et al. [2022] MacAvaney, S., Macdonald, C., Ounis, I.: Streamlining evaluation with ir-measures. In: European Conference on Information Retrieval, pp. 305–310, Springer (2022)
  * Magdy and Jones [2011] Magdy, W., Jones, G.J.: Should MT systems be used as black boxes in CLIR? In: European Conference on Information Retrieval, pp. 683–686, Springer (2011)
  * McCarley [1999] McCarley, J.S.: Should we translate the documents or the queries in cross-language information retrieval? In: Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pp. 208–214 (1999)
  * Mitamura et al. [2008] Mitamura, T., Nyberg, E., Shima, H., Kato, T., Mori, T., Lin, C.Y., Song, R., Lin, C.J., Sakai, T., Ji, D., et al.: Overview of the NTCIR-7 ACLIA tasks: Advanced cross-lingual information access. In: NTCIR (2008)
  * Nair et al. [2022] Nair, S., Yang, E., Lawrie, D., Duh, K., McNamee, P., Murray, K., Mayfield, J., Oard, D.W.: Transfer learning approaches for building cross-language dense retrieval models. In: Proceedings of the 44th European Conference on Information Retrieval (2022)
  * Nie and Jin [2002] Nie, J.Y., Jin, F.: A multilingual approach to multilingual information retrieval. In: Workshop of the Cross-Language Evaluation Forum for European Languages, pp. 101–110, Springer (2002)
  * Oard and Dorr [1996] Oard, D.W., Dorr, B.J.: A survey of multilingual text retrieval. Tech. Rep. UMIACS-TR-96019 CS-TR-3615, UMIACS (1996)
  * Papineni et al. [2002] Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318, Association for Computational Linguistics, Philadelphia, Pennsylvania, USA (Jul 2002), doi:10.3115/1073083.1073135, URL https://aclanthology.org/P02-1040
  * Peters and Braschler [2002] Peters, C., Braschler, M.: The importance of evaluation for cross-language system development: the clef experience. In: LREC (2002)
  * Peters et al. [2012] Peters, C., Braschler, M., Clough, P.: Multilingual information retrieval: From research to practice. Springer (2012)
  * Rahimi et al. [2015] Rahimi, R., Shakery, A., King, I.: Multilingual information retrieval in the language modeling framework. Information Retrieval Journal 18(3), 246–281 (2015)
  * Rehder et al. [1997] Rehder, B., Littman, M.L., Dumais, S.T., Landauer, T.K.: Automatic 3-language cross-language information retrieval with latent semantic indexing. In: TREC, pp. 233–239, Citeseer (1997)
  * Robertson et al. [2009] Robertson, S., Zaragoza, H., et al.: The probabilistic relevance framework: BM25 and beyond. Foundations and Trends® in Information Retrieval 3(4), 333–389 (2009)
  * Santhanam et al. [2022] Santhanam, K., Khattab, O., Potts, C., Zaharia, M.: PLAID: An efficient engine for late interaction retrieval. arXiv preprint arXiv:2205.09707 (2022)
  * Shi and Lin [2019] Shi, P., Lin, J.: Cross-lingual relevance transfer for document retrieval. arXiv preprint arXiv:1911.02989 (2019)
  * Si et al. [2008] Si, L., Callan, J., Cetintas, S., Yuan, H.: An effective and efficient results merging strategy for multilingual information retrieval in federated search environments. Information Retrieval 11(1), 1–24 (2008)
  * Sorg and Cimiano [2012] Sorg, P., Cimiano, P.: Exploiting wikipedia for cross-lingual and multilingual information retrieval. Data & Knowledge Engineering 74, 26–45 (2012), ISSN 0169-023X, URL https://www.sciencedirect.com/science/article/pii/S0169023X12000213, applications of Natural Language to Information Systems
  * Tsai et al. [2008] Tsai, M.F., Wang, Y.T., Chen, H.H.: A study of learning a merge model for multilingual information retrieval. In: Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 195–202 (2008)
  * Xu et al. [2021] Xu, H., Van Durme, B., Murray, K.: BERT, mBERT, or BiBERT? a study on contextualized embeddings for neural machine translation. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6663–6675, Association for Computational Linguistics, Online and Punta Cana, Dominican Republic (Nov 2021), URL https://aclanthology.org/2021.emnlp-main.534
  * Xu [2021] Xu, Y.: Global divergence and local convergence of utterance semantic representations in dialogue. In: Proceedings of the Society for Computation in Linguistics 2021, pp. 116–124, Association for Computational Linguistics, Online (Feb 2021), URL https://aclanthology.org/2021.scil-1.11
  * Yang et al. [2022] Yang, E., Nair, S., Chandradevan, R., Iglesias-Flores, R., Oard, D.W.: C3: Continued pretraining with contrastive weak supervision for cross language ad-hoc retrieval. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (2022), URL https://arxiv.org/abs/2204.11989
  * Zhang et al. [2021] Zhang, X., Ma, X., Shi, P., Lin, J.: Mr. TyDi: A multi-lingual benchmark for dense retrieval. In: Proceedings of the 1st Workshop on Multilingual Representation Learning, pp. 127–137, Association for Computational Linguistics, Punta Cana, Dominican Republic (Nov 2021), URL https://aclanthology.org/2021.mrl-1.12

##  Appendix 0.A MTT Implementation Details

As described in Section 3.2, MTT-M consists of examples with different
languages in the training batches. We implement it by mixing the translated
MS-MARCO triples round-robin. Specifically, each triple consists of an English
query and positive and negative passages translated into the target languages.
We constructed such triples using the translated documents provided by mMARCO
[6]. Each language results in a triple file of the same structure as
triples.train.small.tar.gz.777https://msmarco.blob.core.windows.net/msmarcoranking/triples.train.small.tar.gz
The following Bash command creates a combined triple file that mixes all
languages:

⬇

<(cat ./mmarco/french/triples.train.small.tsv) \

<(cat ./mmarco/german/triples.train.small.tsv) \

<(cat ./mmarco/italian/triples.train.small.tsv) \

<(cat ./mmarco/spanish/triples.train.small.tsv) \

| cat > combined.tsv

Training with four GPUs and a per-GPU batch size of 32 triples guarantees that
each batch consists of examples in different languages based on
ColBERT-X’s888https://github.com/hltcoe/ColBERT-X/blob/main/xlmr_colbert/training/lazy_batcher.py
batching scheme.

For MTT-S, we modified the ColBERT-X batching mechanism to load multiple
triple files and supply a batch of examples from only one source file whenever
the training process requests one. After each request, we switch the source
triple file to ensure all languages are presented equally to the model during
training.