# Laser threshold magnetometry using green light absorption by diamond
nitrogen vacancies in an external cavity laser

James L. Webb<EMAIL_ADDRESS>Center for Macroscopic Quantum States
(bigQ), Department of Physics, Technical University of Denmark, Kgs. Lyngby,
Denmark  Andreas F. L. Poulsen  Center for Macroscopic Quantum States (bigQ),
Department of Physics, Technical University of Denmark, Kgs. Lyngby, Denmark
Robert Staacke  Division of Applied Quantum System, Felix Bloch Institute for
Solid State Physics,
Leipzig University, 04103, Leipzig, Germany  Jan Meijer  Division Applied
Quantum System, Felix Bloch Institute for Solid State Physics,
Leipzig University, 04103, Leipzig, Germany  Kirstine Berg-Sørensen
Department of Health Technology, Technical University of Denmark, Kgs. Lyngby,
Denmark  Ulrik Lund Andersen  Center for Macroscopic Quantum States (bigQ),
Department of Physics, Technical University of Denmark, Kgs. Lyngby, Denmark
Alexander Huck  Center for Macroscopic Quantum States (bigQ), Department of
Physics, Technical University of Denmark, Kgs. Lyngby, Denmark

###### Abstract

Nitrogen vacancy (NV) centers in diamond have attracted considerable recent
interest for use in quantum sensing, promising increased sensitivity for
applications ranging from geophysics to biomedicine. Conventional sensing
schemes involve monitoring the change in red fluorescence from the NV center
under green laser and microwave illumination. Due to the strong fluorescence
background from emission in the NV triplet state and low relative contrast of
any change in output, sensitivity is severely restricted by a high optical
shot noise level. Here, we propose a means to avoid this issue, by using the
change in green pump absorption through the diamond as part of a semiconductor
external cavity laser run close to lasing threshold. We show theoretical
sensitivity to magnetic field on the pT/$\sqrt{\rm Hz}$ level is possible
using a diamond with an optimal density of NV centers. We discuss the physical
requirements and limitations of the method, particularly the role of amplified
spontaneous emission near threshold and explore realistic implementations
using current technology.

††preprint: APS/123-QED

##  I Introduction

Optical manipulation of material defects represents an ideal method for
quantum sensing, exploiting properties such as entanglement and superposition
[1]. The nitrogen-vacancy (NV) center in diamond, possessing long quantum
coherence times at room temperature, has in particular drawn considerable
interest [2, 3, 4]. Diamond is an ideal material for sensing, being
mechanically hard, chemically stable, isotopically pure as well as
biocompatible [5, 6]. The negatively charged nitrogen-vacancy center (NV-) has
an energy level structure that results in optical properties that are highly
sensitive to temperature [7], strain (pressure) [8], electric field [9] and
particularly magnetic field. Sensing is conventionally performed by detecting
changes in the intensity of red fluorescence ($\approx$ 637-750 nm) under
irradiation with green light and resonant microwaves via a process termed
optically detected magnetic resonance (ODMR) spectroscopy [4, 10, 11, 12]. It
can be done using a continuous wave (CW) method [13], or by using short laser
and microwave pulses [14, 15].

However, measuring via red fluorescence suffers from two considerable physical
limitations. First, the signal to be measured has a very low contrast on
bright emission from decay in the NV- triplet state. Although for a single
NV-, spin dependent contrast can be up to 30$\%$ [16], for a large ensemble of
NVs suitable for a diamond sensor the contrast can be at most a few percent
[17, 18]. Sensitivity is therefore limited by this low contrast and the high
level of shot noise from the bright background rising from triplet state
fluorescence emission. The second physical limitation is the high refractive
index of diamond, which traps the majority of the fluorescence inside the
diamond. Microfabrication schemes have been proposed to mitigate this issue,
but have yet to deliver significant improvements [19, 20].

An alternative method is to use optical absorption of the pump light by the
NVs. Previous work has used the change in green absorption in an optical
cavity [21, 22] or by using changes in infrared (IR) absorption by the singlet
state [23]. These schemes are technically demanding, requiring an optical
cavity or unusual wavelength (1042 nm) laser. A promising alternative is laser
threshold sensing [24], using changes in optical absorption resulting from the
parameter to be sensed (e.g. magnetic field or temperature) to push a medium
across lasing threshold. This method eliminates the bright background that
limits sensitivity using conventional fluorescence detection. A further
attraction is wide applicability to any material with variable optical
absorption, including a wider range of defects in diamond, SiC and 2D
materials [25, 26].

Building on the work by Dumeige et al. [27] and our own previous work on
diamond absorption magnetometry [21, 22], we outline here a scheme to use
laser threshold sensing of magnetic field with green light in a standard
external cavity laser. We show it is possible to achieve high sensitivity in
the pT/$\sqrt{\rm Hz}$ range with realistic assumptions for key physical
parameters. Our proposal differs from that of previous work by using simpler
green pump absorption rather than IR absorption and by using an ordinary
current driven laser diode/gain chip medium without the need for an additional
pump laser. We show that this configuration, highly suitable for
miniaturization, can deliver high sensitivity, and we discuss the key physics
required to reach such sensitivity levels. Finally, we discuss and calculate
limiting factors that may prevent these levels from being reached in practice.
This includes factors that may not have been previously considered, such as
amplified spontaneous emission near to lasing threshold.

##  II Methods

###  II.1 External Cavity Laser Model

Figure 1: a) Schematic of the external cavity setup with a Fabry-Perot
semiconductor laser diode of cavity length $L_{m}$ and end facet power
reflectivities $R_{1}$ and $R_{2}$ coupled to an external cavity of length
$L_{r}$ via mirror $R_{3}$ and diamond of thickness $d$. Laser emission
(dashed line) is through $R_{1}$. We model our external cavity laser with the
diamond as a single cavity with equivalent end reflectance $R_{e}$ and the
diamond absorption loss $\alpha_{d}$ included in the total cavity loss
$\alpha_{t}$, b) Simplified schematic of the laser threshold process, where a
reduction in diamond absorption through application of resonant microwaves
reduces the threshold current to $I_{th}^{\rm on}$, producing lasing output
$P_{out}$ when driven at $I_{th}^{\rm off}$.

We place the diamond into a standard external cavity laser setup as described
schematically in Fig. 1,a). This consists of a Fabry-Perot semiconductor laser
diode or gain chip of length $L$ with end facet reflectivities $R_{1}$ and
$R_{2}$ coupled to an external cavity formed by mirror $R_{3}$ via an external
cavity of length $L_{r}$ containing the diamond of thickness $d$. We assume
normal incidence and that transmission through the diamond is high, with
minimal reflection from the diamond facets. For simplicity, we assume a single
optical mode at a single wavelength. We consider the optical loss due to
absorption in a diamond of thickness $d$. The change in laser intensity
$\hat{I}$ on a pass through the diamond is given by

$\Delta\hat{I}=\hat{I_{0}}-\hat{I_{0}}e^{-\alpha_{d}z},$ (1)

where $\hat{I_{0}}$ is the intensity of the laser emission with no diamond
present in the external cavity, $\alpha_{d}$ is the absorption coefficient in
the diamond and $z$ is the path length taken within the diamond. For normal
incidence $z$ = $d$ and the absorption coefficient can be derived from the
rate equation model given in the following section or can be measured
experimentally. For the semiconductor lasing medium between mirrors 1 and 2,
we assume a total cavity loss $\alpha_{t}$, given by the sum of intrinsic
cavity loss due to the gain medium $\alpha_{c}$ and losses from the mirrors
and end facets $\alpha_{m}$, giving a total loss $\alpha_{t}$

$\alpha_{t}=\alpha_{c}+\alpha_{m}=\alpha_{c}+\frac{1}{L_{m}}\ln\left(\frac{1}{\sqrt{R_{1}R_{2}}}\right).$
(2)

In order to simplify the analysis of the external cavity structure, we use the
three mirror model [28, 29, 30, 31] to treat the complete diode/external
cavity structure as a single cavity of length $L$ = $L_{m}$ \+ $L_{r}$, with
mirror $R_{2}$ replaced by an effective reflectivity $R_{e}$, with the single
cavity containing the optical losses of the external cavity and diamond, the
internal losses of the gain medium in the laser diode and loss from the cavity
through the mirrors. By assuming the losses due to the diamond are spread
evenly throughout, we redefine the loss coefficient due to the diamond as
$\alpha_{e}=(\alpha_{d}/L)d$, and our total cavity loss as

$\alpha_{t}=\alpha_{c}+\alpha_{e}+\frac{1}{L}\ln\left(\frac{1}{\sqrt{R_{1}R_{e}}}\right),$
(3)

where $R_{e}=\left|r_{e}\right|^{2}$ relates the power reflectivity to the
complex field reflectivity, $r_{e}$. We use the model for the effective
reflectivity by Voumard et al. [32], detailed further in the Supplementary
Information. Neglecting phase components, at threshold $R_{1}R_{e}e^{(\Gamma
g-\alpha_{t})2L}=1$, where $g$ = $g_{th}$ is the (threshold) gain coefficient.

For the full structure, the rate equations for photon ($S$) and carrier ($N$)
density are given by the standard equations for a laser diode as

$\frac{dN}{dt}=\frac{I}{qV}-\frac{N}{\tau_{N}}-GS,$ (4)

and

$\frac{dS}{dt}=GS-\frac{S}{\tau_{P}}+\frac{\beta N}{\tau_{N}}.$ (5)

Here $I$ is the drive current, $V$ the volume of the gain region, $G$ the gain
of the lasing medium and $q$ the electronic charge. The term $GS$ arises from
stimulated emission in the laser diode gain medium and $S/\tau_{P}$ includes
the cavity loss from the mirrors, gain medium and diamond. Further, $\tau_{P}$
is the photon lifetime in the cavity and $\tau_{N}$ the carrier lifetime in
the laser diode. Carriers are generated by a current $I$ in a volume $V$,
where $V=L\times w\times t_{h}$, where $t_{h}$ is the thickness and $w$ the
width of the laser diode active region. The term $\beta N/\tau_{N}$ relates to
spontaneous emission, governed by the spontaneous emission factor $\beta$.

We can define gain $G$ phenomenologically, in the form [33]

$G=\Gamma g=\Gamma a(N_{th}-N_{tr})(1-\epsilon S),$ (6)

where $\Gamma$ is the confinement factor and $\epsilon$ is the gain
compression factor that phenomenologically accounts for effects such as
spectral hole burning at higher optical power. The carrier density at
transparency is given by $N_{tr}$. The rate equations for photon and carrier
density can be solved for a steady state condition ($dS/dt$ = 0, $dN/dt$ = 0).
For carrier density $N$ = $N_{th}$ close to $N_{tr}$ and neglecting
spontaneous emission ($\beta$ = 0), the gain balances the cavity loss. The
factor $a$ is the differential gain coefficient, a material specific property
defining how well the semiconductor can generate carriers for population
inversion. Equation (6) is valid for heterostructure laser diodes and certain
quantum well structures where the threshold is close to the transparency
carrier density. Unless otherwise stated, we use the Eq. (6) model in this
work.

Using Eqs. (4-6) at lasing threshold, where $S=0$, $G$ = 1/$\tau_{p}$ and
$\Gamma g_{th}$ = $\alpha_{t}$ we can derive an equation for carrier density
at threshold $N_{th}$

$N_{th}=N_{tr}+\frac{\alpha_{t}}{\Gamma a},$ (7)

and inserting this result into the rate equation for carrier density - Eq. (4)
- allows us to calculate the threshold current

$I_{th}=\frac{qV}{\eta_{i}\tau_{N}}N_{th}=\frac{qV}{\eta_{i}\tau_{N}}\left(N_{tr}+\frac{\alpha_{t}}{\Gamma
a}\right).$ (8)

Here we introduce the quantum efficiency of carrier to photon conversion
$\eta_{i}$. By using Eq. (8) in the rate equations at $I>I_{th}$, we can
calculate the photon density at any current above the lasing threshold. We can
then calculate the laser light power that can be emitted from the left hand
side mirror $R_{1}$ using the factor $\eta_{o}$, the output coupling
efficiency, which is defined as the ratio of photons lost through the mirror
$R_{1}$ to the total cavity loss $\alpha_{t}$ =
$\alpha_{m}+\alpha_{c}+\alpha_{e}$

$P_{out}=\eta_{o}\frac{hc}{\lambda\tau_{p}}\frac{V}{\Gamma}S,$ (9)

where $V/\Gamma$ is the effective mode volume of the cavity, $\lambda$ the
wavelength and $h$ and $c$ Plank’s constant and the speed of light
respectively. In the limit of $\epsilon S\rightarrow 0$ where there is no
limiting effect on the gain, the power output can be rewritten directly in
terms of the threshold current

$P_{out}=\eta_{o}\eta_{i}\frac{hc}{q\lambda}(I-I_{th}).$ (10)

In both of these expressions

$\eta_{o}=\frac{\alpha_{m_{1}}}{\alpha_{m}+\alpha_{c}+\alpha_{e}}=\frac{ln\frac{1}{\sqrt{R_{1}}}}{ln\frac{1}{\sqrt{R_{1}R_{e}}}+(\alpha_{e}+\alpha_{c})L}.$
(11)

For larger finite values of $\epsilon$ well above threshold or including
finite spontaneous emission through nonzero $\beta$, we can numerically solve
the steady state rate equations (Eqs. (4) and (5)) to calculate $N$, $S$ and
the laser power output.

The total cavity absorption $\alpha_{t}$ will change when microwaves are
applied to the diamond at frequency equal to the splitting of the NV triplet
ground state levels, reducing the lasing threshold current $\Delta I_{th}$ =
$I_{th}^{\rm off}$ \- $I_{th}^{\rm on}$, where $I_{th}^{\rm on}$ is the
threshold current on microwave resonance, and $I_{th}^{\rm off}$ the threshold
current off resonance. By running at drive current equal to $I_{th}^{\rm
off}$, laser output is generated only while on microwave resonance. This is
shown schematically in Fig. 1,b).

###  II.2 Absorption model

We use the rate equation model from [34] in order to calculate the optical
absorption of green pump light by the diamond and the maximum change in
absorption when on microwave resonance. The parameters we use for the
transition rates are the same as those in [27], derived from [35, 36, 37, 38].
We calculate the normalized occupancies of each energy level with microwaves
supplied $n_{i}^{\rm on}$ and without microwaves $n_{i}^{\rm off}$, where
$\sum_{i}n_{i}$ = 1 and index $i$ = 1-8, where i=1 refers to the $m_{s}$=0
ground state level, i=2 the $m_{s}$=$\pm$1 ground state levels, i=3,4 the spin
triplet excited states, i=5,6 the spin singlet shelving states and i=7,8 the
ground and excited state of the NV0. We define a total NV- density $N_{NV}$ in
ppm. Off resonance, the total number density of NV- in each state $N^{\rm
off}_{i}$ are given by

$N^{\rm off}_{i}=N_{NV}\frac{n_{i}^{\rm off}}{\sum_{i}n_{i}^{\rm off}}.$ (12)

We define a measurement axis along one of the 4 possible crystallographic axes
for the NV. We calculate that when microwaves are applied, we drive only the
NVs aligned along one axis such that the total number density on resonance
$N^{\rm on}_{i}$ is given by

$N^{\rm on}_{i}=\frac{1}{4}N_{NV}\frac{n_{i}^{\rm on}}{\sum_{i}n_{i}^{\rm
on}}+\frac{3}{4}N_{NV}\frac{n_{i}^{\rm off}}{\sum_{i}n_{i}^{\rm off}}.$ (13)

We calculate the change in intensity on a single pass when on and off
microwave resonance as

$\begin{split}\hat{I_{\rm on}}=\hat{I_{0}}e^{-\alpha_{\rm on}d},\\\
\hat{I_{\rm off}}=\hat{I_{0}}e^{-\alpha_{\rm off}d},\end{split}$ (14)

where $d$ is the thickness of the diamond and the absorption coefficient
$\alpha$ on and off resonance is given by

$\alpha^{\rm on}=\sigma_{g}(N^{\rm on}_{1}+N^{\rm on}_{2})+\sigma_{g0}N^{\rm
on}_{7}+\sigma_{e}(N^{\rm on}_{3}+N^{on}_{4})+\sigma_{r}N^{\rm on}_{8},$ (15)

$\begin{split}\alpha^{\rm off}=\sigma_{g}(N^{\rm off}_{1}+N^{\rm
off}_{2})+\sigma_{g0}N^{\rm off}_{7}+\sigma_{e}(N^{\rm off}_{3}+\\\ N^{\rm
off}_{4})+\sigma_{r}N^{\rm off}_{8}.\end{split}$ (16)

Here $\sigma_{g}$ and $\sigma_{g0}$ are respectively the absorption cross
sections of green light for NV- and NV0 and $\sigma_{e}$, and $\sigma_{r}$ the
ionisation cross sections for transfer between the charged and uncharged
defect state. This allows us to calculate the change in absorption when the
diamond is present without microwaves $\hat{I_{\rm off}}/\hat{I_{0}}$, the
change when driven on microwave resonance $\hat{I_{\rm on}}/\hat{I_{0}}$ and
the change between these, which we term the absorption contrast

$C=(\hat{I_{\rm off}}/\hat{I_{0}})-(\hat{I_{\rm on}}/\hat{I_{0}}).$ (17)

###  II.3 Key Physical Parameters

The key physical parameters of the model can be divided into those that are
intrinsic to the semiconductor gain medium, those intrinsic to the diamond and
those defined by the setup. Examples of the latter include the mirror
reflectivities $R_{1}$, $R_{2}$, $R_{3}$, the cavity length $L$ and any other
losses, such as reflection out of the cavity or from absorption by other
optical components such as lenses, included in the cavity loss factor
$\alpha_{c}$. These factors will also influence the photon lifetime in the
cavity $\tau_{P}$. The maximum Rabi frequency $\Omega_{R}$ that can be reached
also depends on microwave power and how well the microwaves can be coupled
into the diamond.

Figure 2: Dephasing time $T_{2}^{*}$ vs NV- density $N_{NV}$ where both values
are given in other works (citations in the main text). $T_{2}^{*}$ in those
with low NV concentration are limited by interaction with 13C spin, with the
highest values given by diamonds isotropically purified with 12C spin during
growth. $T_{2}^{*}$ in those with high NV concentration is limited by dipolar
interaction between defects, including other subsitutional nitrogen defects
such as P1 centers. (Note: the NV- density for the work by Childress et al. is
an upper estimate made here assuming a 10$\%$ NV- fraction; total
substitutional nitrogen content for this diamond was given as $\leq$0.1 ppm.

The parameters which are intrinsic to the diamond are the diamond thickness
$d$, $NV^{-}$ density $N_{NV}$, ensemble dephasing time $T_{2}^{*}$ defining
the ODMR linewidth and absorption contrast $C$ arising from changes in pump
absorption on or off microwave resonance. These factors define the diamond
absorption factor $\alpha_{d}$.

A number of these parameters are interrelated. The ODMR linewidth is
proportional to the inverse of $T_{2}^{*}$, which in turn is dependent on
$N_{NV}$ concentration in the limit of high nitrogen content and the abundance
of 13C for low nitrogen content [39]. There is also a dependence on other
material properties such as strain [40], which makes the relationship between
the parameters difficult to determine. We therefore consider values in the
experimental literature as a guide. Fig. 2 shows a plot of $T_{2}^{*}$ versus
NV- density $N_{NV}$ for a range of diamonds from the literature [41, 36, 42,
43, 44, 45, 46]. Typical NV- densities range from 0.1ppb up to tens of
ppm[47]. In general, $T_{2}^{*}$ $<$ 1 $\mu$s for samples with natural
(1.1$\%$) 13C content [48, 49]. Experiments typically realize Rabi frequencies
$\Omega_{R}$ of 1-5 MHz, with up to 10 MHz using optimal antenna geometries
[50].

Parameter | Range | Ref.
---|---|---
Transp. carrier density, $N_{tr}$ |  $3\times 10^{18}$-$2\times 10^{19}$cm-3 | [51, 52, 53, 54, 55]
Carrier lifetime, $\tau_{N}$ | 1-5ns | [56, 57]
Differential gain factor, $a$ | 10-17 \- 10-22m2 | [58, 59]
Confinement factor, $\Gamma$ | 0.01-0.1 | [60, 61]
Spont. emission factor, $\beta$ | 10-5-10-2 | [62, 63]
Table 1: Typical ranges for the key semiconductor gain medium parameters. Here
$N_{tr}$, $\tau_{N}$ and $a$ are taken for typical III-nitride semiconductors.
The range for $\Gamma$ is given for laser diodes with a thin (sub-$\mu$m)
active layer and is typically no more than a few percent. The range of $\beta$
is given for literature values for a range of laser diodes where confinement
is not deliberately sought e.g. microcavities, where values several orders of
magnitude higher than the given range are possible [64].

Those parameters intrinsic to the laser diode/gain chip used are the carrier
density at transparency $N_{tr}$, the gain compression factor $\epsilon$ that
arises from effects that limit the gain well above threshold, the differential
gain coefficient $a$ that relates gain and carrier density, the threshold
carrier lifetime $\tau_{N}$ , the confinement factor $\Gamma$, the volume of
the gain medium $V$ and the spontaneous emission factor $\beta$. For our gain
medium we take a III-V semiconductor heterostructure device, such as the
nitride compounds capable of emission at green wavelengths (e.g. InGaN) [65].
Table 1 shows a typical range of values for each of these parameters. We take
the typical ranges shown based on experimental results from different
structures (quantum well, vertical cavity) and from calculations based on bulk
material properties such as effective mass. $N_{tr}$ effectively defines the
size of the lasing threshold current $I_{th}$. The desired change in threshold
current on change in absorption factor $\alpha_{t}$ is defined in particular
by $\Gamma$ and the gain coefficient $a$ in Eqs. (3) and (7).

##  III Results

###  III.1 Absorption contrast

Figure 3: a) Absorption contrast percentage calculated from the rate model for
Diamond D3. This is the maximum change in absorption between on microwave
resonance and off microwave resonance as a function of Rabi frequency
$\Omega_{R}$ and laser intensity $\hat{I}$ in W/m2. b)-d) Normalized level
occupancy for the NV- triplet ground state, responsible for green absorption,
the uncharged NV0 defect state and the NV- singlet state. At high laser
intensities, population transfer to NV0 limits the achievable absorption
contrast. For reference, 10 mW of laser power with a 1 mm diameter circular
beam on the diamond gives an intensity $\hat{I}$ = $10^{4}$ W/m2. The black
spot indicates the Rabi frequency and power for calculations later in this
work.

We first calculate from the rate model the fraction of incident pump light
which is absorbed by the diamond and the change in this absorption ($C$) when
on microwave resonance. We choose to model three different diamonds covering
different regimes: D1, D2 and D3 with parameters (NV density and $T_{2}^{*}$)
representative of the values seen in the literature (Fig. 2). For Diamond D1
we choose a low NV- concentration $N_{NV}$ = 0.001 ppm, high $T_{2}^{*}$ = 5
$\mu$s, representative of 12C enriched diamonds. For Diamond D2 we choose a
medium NV- concentration $N_{NV}$ = 0.1 ppm, $T_{2}^{*}$ = 0.75 $\mu$s,
representative of CVD-grown diamond with natural 13C abundance. For Diamond D3
we choose $N_{NV}$ = 10 ppm, $T_{2}^{*}$ = 0.1 $\mu$s, characteristic of high
nitrogen content high-pressure high-temperature (HPHT) diamond. We use a
diamond thickness $d$ = 500 $\mu$m for all, representative of commercially
available single crystal plates.

Using the rate model, we can calculate the absorption of light incident on the
diamond 1 - $\hat{I_{\rm off}}$/$\hat{I_{0}}$ where $\hat{I_{0}}$ is intensity
of the incident light and $\hat{I_{\rm off}}$ the intensity of the light after
the diamond (without supplying microwaves). The calculated absorption for
Diamonds D1 - D3 is 0.015$\%$, 1.498$\%$ and 77$\%$, as expected from
increasing NV density. We can also calculate the change in absorption when on
and off microwave resonance. This is shown in Fig. 3 as absorption contrast
$C$ for D3 as a function of microwave drive power (as Rabi frequency
$\Omega_{R}$) and laser output (as intensity). The equivalent plots for D1 and
D2 are given in the Supplementary Information. Maximum $C$ = 0.22$\%$ for D3
and lowest for D1 with the lowest NV density with $C$ = 10${}^{-4}\%$. This
contrast is comparable to our previous absorption experiments using a diamond
with equivalent ppb-level NV- density [21]. We note that at Rabi frequencies
above 100 kHz and laser outputs above 106 W/m2 the absorption contrast begins
to drop. This results from depopulation of the triplet ground state 3A2
(normalized occupancy in Fig. 3,b) in favor of the NV0 (Fig. 3,c) and the
singlet shelving state (Fig. 3,d). However, since we aim to operate near the
lasing threshold, laser intensity will be low in our scheme, avoiding this
issue and ensuring we remain in the region of highest contrast.

Figure 4: Experimental absorption contrast percentage as a function of
microwave drive frequency and microwave power before the amplifier
(Minicircuits ZHL-16W), measuring through the diamond with 100 mW of laser
light ($\hat{I}$ = 2.5x104 W/m2). The sample was used to test the absorption
model using estimates of NV density and $T_{2}^{*}$ from the observed
linewidth (values in the main text). Note: due to input loss we remain below
the maximum gain threshold of the amplifier for all microwave powers shown,
which is exceeded at +3dBm.

To further validate the absorption modeling, we have also measured diamond
absorption on a high density sample consisting of a 1 mm thick HPHT diamond
with 200 ppm nitrogen content, irradiated with 10 MeV electrons and annealed
at 900 ∘C. The estimated NV content for this sample was 10-20 ppm. The
absorption contrast for this sample is shown in Fig. 4. The sample was found
to be moderately polycrystalline and was therefore measured without an offset
field to produce a single central dip in fluorescence, with a number of
satellite features resulting from the polycrystalinity and residual magnetic
field in the laboratory. Here total off resonance diamond absorption was
90$\%$ of incident pump light and maximum absorption contrast $C$ = 0.13$\%$.
For comparison to experiment, we model absorption contrast with our model with
NV density of 15 ppm, $T_{2}^{*}$ = 100 ns, derived from an estimate of the
resonance linewidth, an estimated Rabi frequency of 1 MHz and the same 100 mW
laser power as used experimentally. This gives a total off resonance
absorption of 89$\%$ of the pump light and absorption contrast of $C$ =
0.14$\%$, in good agreement with our measurements.

###  III.2 Change in Threshold Current

Figure 5: a) Threshold current as a function of differential gain factor $a$
and confinement factor $\Gamma$ for Diamond D3. b) Change in threshold current
due to the diamond absorption contrast $C$ = 0.02$\%$ for Diamond D3. Here
external cavity length $L_{r}$ was 10 mm and output mirror reflectivity
$R_{1}$ = 0.9.

We first calculate the lasing threshold current $I_{th}$ with the diamond
absent from the cavity. To do this we fixed some of the parameters of the
semiconductor gain medium. We choose a transparency carrier density of
$N_{tr}$ = 1$\times$1025 m-3 in the range typical for InGaN laser structures
[51], a gain region volume of $V$ = 1.25$\times$10-16 m3 (25 $\mu$m x 100 nm x
100 $\mu$m). For zero total cavity absorption $\alpha_{t}$ = 0, a typical
differential gain factor $a$ = 5$\times$10-20 m2, a confinement factor
$\Gamma$ of 2$\%$ and a carrier lifetime $\tau_{N}$ = 4 ns, giving a
reasonable lasing threshold current of 50 mA [65]. We take the relation
between the gain and the carrier density to be linear, with the carrier
density close to transparency. We make the simplifying assumption that due to
the low power, running close to lasing threshold we do not encounter gain
compression effects, such that the factor $\epsilon\rightarrow$ 0\. We also
initially make the simplifying assumption that the spontaneous emission rate
is low, with $\beta\rightarrow$0 (the importance of this second assumption
will be tested in the final section of this work). These assumptions allow the
threshold current $I_{th}$ to be calculated easily from Eq. (8). We define
$L_{r}$ = 10 mm, sufficient to include the diamond and any necessary optics in
a practical implementation. We set mirror reflectivity $R_{3}$ = 0.99 and
collect laser output from transmission through mirror $R_{1}$. We calculate
reflectivity $R_{2}$ from the Fresnel equations assuming an InxGa1-xN/air
interface with refractive index $n$ $\approx$ 2.6-2.9 for InxGa1-xN [66].

We impose two feasibility limits on the threshold current $I_{th}$. The first
is that it should not exceed 300 mA, based on the limits discussed in
technical documentation, in order to maintain thermal stability and for
practical heatsinking for a miniaturized diode/gain chip medium. The second is
that the change in the threshold caused by the diamond absorption must exceed
the shot noise of the drive current. From Fig. 5, we can see that these are
mutually exclusive objectives. Setting an absorption contrast $C$ = 0.2$\%$
(Diamond D3) and output mirror reflectivity $R_{1}$ = 0.9 in order to achieve
laser output while keeping threshold current reasonably low, a low confinement
factor $\Gamma$ and high differential gain coefficient $a$ result in the
highest change in threshold current $\Delta I_{th}$ and thus strongest effect
for sensing, but for very high $I_{th}$. Conversely, a higher value of
$\Gamma$ or lower $a$ gives lower $I_{th}$, but $\Delta I_{th}$ shifts which
are too small to be resolved.

Although Ntr is a factor usually defined by the semiconductor material, we
note that the other parameters here which define $I_{th}$, $\Gamma$, $a$ and
total cavity loss $\alpha_{t}$ including the mirror reflectivity and cavity
output through $R_{1}$, are all factors which are well understood and can be
controlled and optimized at either the semiconductor growth stage or in the
external cavity design.

###  III.3 Simulated ODMR

Figure 6: a) Simulated ODMR for Diamond D3 at a range of differential gain
factors $a$=10-21 $\rightarrow$ 10-19.5 m2 (exponents given in legend)
measured by calculating external cavity laser output power $P$ as a function
of microwave frequency for a Lorentzian lineshape transition centered at 2.83
GHz and of linewidth defined by fl = $\frac{1}{\pi T_{2}^{*}}$ = 3.2 MHz. b)
Maximum laser power output $P_{max}$ on resonance as a function of gain
coefficient $a$ for confinement factor $\Gamma$ = 0.01, 0.05, 0.1, c) the
lasing threshold current for $I_{th}$ $<$ 300 mA, for the same three values of
$\Gamma$.

Here we calculate the ODMR spectrum that would be produced from the external
cavity laser. We model a single microwave resonance from a single $m_{s}$ = 0
$\rightarrow$ $m_{s}$ = $\pm$1 transition using a Lorentzian lineshape typical
of ODMR for diamond [17]. We center our resonance at 2.82 GHz, replicating an
ODMR resonance feature associated with a single NV axis, split from resonance
features from other axes by an arbitrary weak DC offset magnetic field. The
maximum amplitude is defined by the maximum change in threshold current
between on and off microwave resonance and full width half maximum linewidth
fl. For simplicity we assume that we can reach the pulsed readout linewidth
defined by $T_{2}^{*}$. We calculate the external cavity laser output power
using Eq. (10). Fig. 6,a) shows the simulated ODMR for Diamond D3, with a
laser power output in the mW range for reasonable values of $\Gamma$ $<$ 0.1
and $a$ = 10-17-10-21 m2. The equivalent plots for Diamond D1 and D2 are given
in the Supplementary Information, with maximum power outputs in the range of
nW and $\mu$W respectively. Unlike for conventional red fluorescence ODMR, the
spectrum using this method is a peak at microwave resonance with zero
background, rather than a small percentage change on a bright background.

###  III.4 Magnetic Field Sensitivity

Figure 7: a) Optical shot noise limited sensitivity for Diamond D3 within a
viable range for diode parameters $a$ and $\Gamma$. Sensitivity is in the
picotesla range, enabled by the elimination of the high noise from the
background in the conventional fluorescence detection scheme. b) The ultimate
sensitivity limit imposed by shot noise on the laser drive current, worse by
up to 2 orders of magnitude. These plots give no consideration for practical
viability, with threshold currents $>$4 A at low $\Gamma$.

We calculate sensitivity to magnetic field by taking the background noise
level, dividing by the maximum ODMR slope and by assuming a maximum frequency
shift of 28 Hz $\approx$ 1 nT [13]. In our model, the primary sources of noise
are readout from the photodetector and the noise on the drive current. The
ultimate limit on both of these is shot noise of the output laser light and
the drive current shot noise. We make no account for other direct sources of
noise which are difficult to quantify, such as vibration or temperature
fluctuations. Fig. 7 shows a plot of sensitivity for Diamond D3 versus laser
diode parameters for a) the optical shot noise and b) drive current shot noise
limited regimes, with best sensitivity of 50 pT/$\sqrt{\rm Hz}$ when limited
by the shot noise of the drive current. We note that in practice the shot
noise limited operation may be experimentally difficult to realize and include
an estimate based on a commercial current source with ppm-level noise in the
Supplementary Information.

Figure 8: Regions where the sensor can and cannot operate due to imposed
limitations. In Regions A and C, operation is constrained by having a change
in threshold less than the shot noise of the drive current and $I_{th}$ $>$
300 mA respectively. In Region B, operation is possible.

The noise limitations as a function of diode parameters are highlighted in
Fig. 8 for Diamond D3. Here the regions A and C represent where operation is
noise limited and the region B represents the region in which the system can
operate. In region A, the change in threshold current is less than the shot
noise of the laser drive current ($\Delta I_{th}$ $<$ $I_{sh}$). In region C,
the threshold current $I_{th}$ $>$ 300 mA exceeds a reasonable maximum drive
current in order to maintain thermal stability. The limitations we impose mean
Diamond D1 or D2 have no viable operating region. For completeness, their
sensitivity plots are included in the Supplementary Information.

Figure 9: Best field sensitivity optimizing variables listed in the main text
as a function of a) NV- density and b) $T_{2}^{*}$ in $\mu$s, with the inset
showing a zoomed plot at the highest simulated values of $T_{2}^{*}$. The best
sensitivity was observed at the highest $T_{2}^{*}$, for NV- density 104 ppb.
Above this the total absorption for the diamond was too high, limiting laser
output and sensitivity.

By solving the rate model and calculating for laser diode output, we can
calculate sensitivity to magnetic field for any valid physical parameters of
the system, regardless of whether a diamond can be created with the requisite
properties. This includes whether a value of $T_{2}^{*}$ can be realized for a
corresponding $N_{NV}$, making no assumption regarding the relation between
these parameters, or whether $N_{NV}$ can be realized experimentally. Here we
choose parameters $R_{1}$, $\Gamma$, $T_{2}^{*}$, $a$ and NV- density $N_{NV}$
as optimization variables, while fixing diamond thickness ($d$ = 500 $\mu$m),
Rabi frequency ($\Omega_{R}$=1 MHz), laser beam width (0.5 mm), power (200 mW)
and mirror reflectivities. We limit our laser power to 200 mW based on our
rate model calculations, to ensure the majority of light is absorbed by the
NV- defects. We optimize using standard gradient descent methods. Fig. 9 shows
a plot of optical shot noise limited field sensitivity as a function of
$T_{2}^{*}$ and NV density. Sensitivity increased with higher $T_{2}^{*}$ as
would be expected, with maximum sensitivity at NV density of 104 ppb, above
which high overall absorption by the diamond acted to excessively reduce laser
output. Sub-picotesla level sensitivity is predicted for $T_{2}^{*}$ $>$ 1
$\mu$s (0.3-0.02 pT/$\sqrt{\rm Hz}$ for $T_{2}^{*}$ = 1-10 $\mu$s). Here the
optimal parameters were $a$ = 1.6x10-20 m2, $R_{1}$ = 0.154 and $\Gamma$ =
0.025. These are parameters within the achievable range for a semiconductor
gain medium (see Table 1).

Figure 10: Calculated magnetic field sensitivity and laser diode threshold
current as a function of differential gain factor $a$ using an empirical model
for a quantum well laser diode. The lasing threshold current increases such
that sub-picotesla sensitivity is not reached at a feasible threshold current
($<$300 mA). Here we take confinement factor $\Gamma$=0.01 as an example of
the low values typical of a quantum well laser diode.

We note that in general, the highest sensitivity is realized for the lowest
differential gain factor $a$. A standard laser diode demands a large $a$,
maximizing gain vs carrier density (steeper output power vs drive current
slope). Our scheme requires the reverse: that a small change in gain produced
by the diamond on/off microwave resonance results in a large change in
$N_{th}$ and $I_{th}$. In this respect, a quantum well structure with a
flatter logarithmic relation between gain and carrier density would seem
preferable. However, as we demonstrate in Fig. 10, using our model, with
modifications to the phenomenological description of the medium gain (detailed
in the Supplementary Information) and with the same optimization methodology
as above results in the threshold current exponentially exceeding drive
current feasibility limits before sub-picotesla/$\sqrt{\rm Hz}$ sensitivity is
reached, for any typical value for $\Gamma$ in the low percentage range.

###  III.5 Effect of spontaneous emission

Figure 11: a) External cavity laser output $P$ as a function of semiconductor
laser medium drive current $I$, varying spontaneous emission factor $\beta$.
The result of increasing $\beta$ is that there is no longer a sharp lasing
cut-on at threshold. b) shows the effect on the achievable sensitivity of this
effect, with sensitivity considerably reduced by up to 2 orders of magnitude
for $\beta$ = 10-2.

In the previous sections and past literature, the physical role of spontaneous
emission in the semiconductor gain medium used was not considered. In order to
maximize sensitivity, it is necessary to operate at or close to the off-
resonance lasing threshold. Without spontaneous emission, this can be treated
as a step cut-on, with zero or near-zero emission before lasing begins at
$I_{th}$. With spontaneous emission included, modeled by finite $\beta$ in
Eqs. (4) and (5) above, the power-current relationship close to threshold
instead follows a shallow curve, resulting from weak amplification of
spontaneous emission near threshold producing light emission below $I_{th}$.
Fig. 11,a) shows this effect for varying $\beta$. This acts to severely limit
sensitivity (Fig. 11,b) by reducing the contrast and adding background shot
noise. Typical values of $\beta$ range from 10-3 to 10-5, depending on laser
diode structure. We estimate approximately an order of magnitude worse
sensitivity at the low end of this range than with $\beta$ = 0.

##  IV Conclusion

In this work, we propose a scheme for laser threshold sensing using an
external cavity laser configuration with a current driven semiconductor lasing
medium. Using the change in lasing threshold, light emission only occurs on
microwave resonance. This eliminates the bright background that limits
sensitivity using conventional red fluorescence emission. Predicted
sensitivities for magnetometry with realistic cavity parameters and intrinsic
material parameters are in the pT/$\sqrt{\rm Hz}$ range, offering a route to
improvement over existing methods. Our model has limitations: we base our
calculations on emission into a single laser mode and do not calculate the
dynamics of the system, such as rapid switching in a pulsed operation scheme.
Although beyond the focus of this work, we note that the latter may be a
promising route for future investigation. A scheme where the laser medium
could be initially pumped and then the pump shut off while retaining
population inversion during sensing, typical of a Q-switched setup, would only
be limited by the optical shot noise of any emitted laser light. This is
however challenging to achieve for a semiconductor laser due to the short
excited state (carrier) lifetime.

A key physical limitation of any laser threshold scheme is the role of
amplified spontaneous emission. This blurs the sharp lasing transition, giving
nonzero light emission even below threshold and compromising sensitivity. A
broad transition can be avoided by minimizing gain factor $\beta$, although
this is difficult for a semiconductor laser, particularly since $\beta$ can
scale inversely with the size of the gain medium [67]. Obtaining a gain chip
or antireflective coated laser diode with the right parameters is challenging,
especially for green wavelengths. This problem also exists for infrared
absorption, since the laser emission must match the 1042 nm gap in the singlet
state. In our scheme, running in the infrared could be achieved by extending
the external cavity design proposed here using a diffraction grating in a
Littrow and Littman–Metcalf configuration to create a tunable system.

We consider in this work a normal incidence beam path through a fixed diamond
thickness $d$=500$\mu$m. We note that a higher sensitivity within feasible
limits of threshold current could potentially be reached with a thinner
diamond with a very high NV density. However, in the limit of
$d$$\rightarrow$0, other effects not considered in our model may act to limit
performance, such as variation in NV density and the role of other types of
defects. Measurements of absorption and $T_{2}^{*}$ as a function of diamond
thickness would be extremely useful in determining behavior in this regime. We
also consider that it may be possible to reach higher sensitivity in the low
NV density regime using an extended beam path achieved through internal
reflection in a thicker diamond. This again requires new experimental
measurements to precisely quantify losses (due to reflection or absorption) in
such a geometry.

We note that the fundamental limit for the scheme is the level of contrast $C$
generated between the on/off microwave resonance states, very low for a large
diamond ensemble. However, the scheme is not specifically limited to diamond
and is broadly applicable for any material where a large enough, controllable
difference in optical absorption could be generated. The advantage of using
diamond is the ability to coherently manipulate the desired states in a
quantum sensing scheme. Our calculations indicate the scheme will likely only
work for diamonds with a high ($>$ 1 ppm) NV- density. Such diamonds have a
worse ensemble $T_{2}^{*}$ time, limited by nitrogen spin interaction. A
developing solution here may be to use optimal control methods in order to
better control the ensemble. Such methods are widely implemented for nuclear
magnetic resonance and electron spin resonance on bulk samples, but have yet
to be fully developed for sensing using diamond defects [68].

##  V Acknowledgments

The work presented here was funded by the Novo Nordisk foundation through the
synergy grant bioQ and the bigQ Center funded by the Danish National Research
Foundation (DNRF).

## References

  * Schleich et al. [2016] W. P. Schleich, K. S. Ranade, C. Anton, M. Arndt, M. Aspelmeyer, M. Bayer, G. Berg, T. Calarco, H. Fuchs, E. Giacobino, et al., Applied Physics B 122 (2016).
  * Gruber [1997] A. Gruber, Science 276, 2012 (1997).
  * Doherty et al. [2013] M. W. Doherty, N. B. Manson, P. Delaney, F. Jelezko, J. Wrachtrup, and L. C. Hollenberg, Physics Reports 528, 1 (2013).
  * Taylor et al. [2008] J. M. Taylor, P. Cappellaro, L. Childress, L. Jiang, D. Budker, P. R. Hemmer, A. Yacoby, R. Walsworth, and M. D. Lukin, Nature Physics 4, 810 (2008).
  * Schirhagl et al. [2014] R. Schirhagl, K. Chang, M. Loretz, and C. L. Degen, Annual Review of Physical Chemistry 65, 83 (2014).
  * Webb et al. [2020] J. L. Webb, L. Troise, N. W. Hansen, J. Achard, O. Brinza, R. Staacke, M. Kieschnick, J. Meijer, J.-F. Perrier, K. Berg-Sørensen, et al., Frontiers in Physics 8 (2020).
  * Kucsko et al. [2013] G. Kucsko, P. C. Maurer, N. Y. Yao, M. Kubo, H. J. Noh, P. K. Lo, H. Park, and M. D. Lukin, Nature 500, 54 (2013).
  * Knauer et al. [2020] S. Knauer, J. P. Hadden, and J. G. Rarity, npj Quantum Information 6 (2020).
  * Dolde et al. [2011] F. Dolde, H. Fedder, M. W. Doherty, T. Nöbauer, F. Rempp, G. Balasubramanian, T. Wolf, F. Reinhard, L. C. L. Hollenberg, F. Jelezko, et al., Nature Physics 7, 459 (2011).
  * Hong et al. [2013] S. Hong, M. S. Grinolds, L. M. Pham, D. L. Sage, L. Luan, R. L. Walsworth, and A. Yacoby, MRS Bulletin 38, 155 (2013).
  * Rondin et al. [2014] L. Rondin, J.-P. Tetienne, T. Hingant, J.-F. Roch, P. Maletinsky, and V. Jacques, Reports on Progress in Physics 77, 056503 (2014).
  * Barry et al. [2020] J. F. Barry, J. M. Schloss, E. Bauch, M. J. Turner, C. A. Hart, L. M. Pham, and R. L. Walsworth, Reviews of Modern Physics 92 (2020).
  * Webb et al. [2019] J. L. Webb, J. D. Clement, L. Troise, S. Ahmadi, G. J. Johansen, A. Huck, and U. L. Andersen, Applied Physics Letters 114, 231103 (2019).
  * Wolf et al. [2015] T. Wolf, P. Neumann, K. Nakamura, H. Sumiya, T. Ohshima, J. Isoya, and J. Wrachtrup, Physical Review X 5 (2015).
  * Bucher et al. [2019] D. B. Bucher, D. P. L. A. Craik, M. P. Backlund, M. J. Turner, O. B. Dor, D. R. Glenn, and R. L. Walsworth, Nature Protocols 14, 2707 (2019).
  * Osterkamp et al. [2019] C. Osterkamp, M. Mangold, J. Lang, P. Balasubramanian, T. Teraji, B. Naydenov, and F. Jelezko, Scientific Reports 9 (2019).
  * Levchenko et al. [2015] A. O. Levchenko, V. V. Vasil'ev, S. A. Zibrov, A. S. Zibrov, A. V. Sivak, and I. V. Fedotov, Applied Physics Letters 106, 102402 (2015).
  * Wojciechowski et al. [2018] A. M. Wojciechowski, M. Karadas, C. Osterkamp, S. Jankuhn, J. Meijer, F. Jelezko, A. Huck, and U. L. Andersen, Applied Physics Letters 113, 013502 (2018).
  * Liu et al. [2019] Z. Liu, H. N. Abbasi, T.-F. Zhu, Y.-F. Wang, J. Fu, F. Wen, W. Wang, S. Fan, K. Wang, and H.-X. Wang, AIP Advances 9, 125218 (2019).
  * Huang et al. [2019] T.-Y. Huang, R. R. Grote, S. A. Mann, D. A. Hopper, A. L. Exarhos, G. G. Lopez, G. R. Kaighn, E. C. Garnett, and L. C. Bassett, Nature Communications 10 (2019).
  * Ahmadi et al. [2018] S. Ahmadi, H. A. R. El-Ella, A. M. Wojciechowski, T. Gehring, J. O. B. Hansen, A. Huck, and U. L. Andersen, Physical Review B 97 (2018).
  * Ahmadi et al. [2017] S. Ahmadi, H. A. El-Ella, J. O. Hansen, A. Huck, and U. L. Andersen, Physical Review Applied 8 (2017).
  * Jensen et al. [2014] K. Jensen, N. Leefer, A. Jarmola, Y. Dumeige, V. Acosta, P. Kehayias, B. Patton, and D. Budker, Physical Review Letters 112 (2014).
  * Jeske et al. [2016] J. Jeske, J. H. Cole, and A. D. Greentree, New Journal of Physics 18, 013015 (2016).
  * Jeong et al. [2019] T. Y. Jeong, H. Kim, S.-J. Choi, K. Watanabe, T. Taniguchi, K. J. Yee, Y.-S. Kim, and S. Jung, Nature Communications 10 (2019).
  * Castelletto and Boretti [2020] S. Castelletto and A. Boretti, Journal of Physics: Photonics 2, 022001 (2020).
  * Dumeige et al. [2019] Y. Dumeige, J.-F. Roch, F. Bretenaker, T. Debuisschert, V. Acosta, C. Becher, G. Chatzidrosos, A. Wickenbrock, L. Bougas, A. Wilzewski, et al., Optics Express 27, 1706 (2019).
  * Cunyun [2004] Y. Cunyun, _Tunable External Cavity Diode Lasers_ (World Scientific, 2004).
  * Petermann [1988] K. Petermann, _Laser Diode Modulation and Noise_ (Springer Netherlands, 1988).
  * de Groot et al. [1988] P. J. de Groot, G. M. Gallatin, and S. H. Macomber, Applied Optics 27, 4475 (1988).
  * Li et al. [2017] J. Li, H. Niu, and Y. Niu, Optical Engineering 56, 050901 (2017).
  * Voumard et al. [1977] C. Voumard, R. Salathé, and H. Weber, Applied Physics 12, 369 (1977).
  * Allen [1994] J. W. Allen, Advanced Materials for Optics and Electronics 4, 51 (1994).
  * Robledo et al. [2011] L. Robledo, H. Bernien, T. van der Sar, and R. Hanson, New Journal of Physics 13, 025013 (2011).
  * Tetienne et al. [2012] J.-P. Tetienne, L. Rondin, P. Spinicelli, M. Chipaux, T. Debuisschert, J.-F. Roch, and V. Jacques, New Journal of Physics 14, 103033 (2012).
  * Acosta et al. [2010] V. M. Acosta, E. Bauch, A. Jarmola, L. J. Zipp, M. P. Ledbetter, and D. Budker, Applied Physics Letters 97, 174104 (2010).
  * Meirzada et al. [2018] I. Meirzada, Y. Hovav, S. A. Wolf, and N. Bar-Gill, Physical Review B 98 (2018).
  * Wee et al. [2007] T.-L. Wee, Y.-K. Tzeng, C.-C. Han, H.-C. Chang, W. Fann, J.-H. Hsu, K.-M. Chen, and Y.-C. Yu, The Journal of Physical Chemistry A 111, 9379 (2007).
  * Jahnke et al. [2012] K. D. Jahnke, B. Naydenov, T. Teraji, S. Koizumi, T. Umeda, J. Isoya, and F. Jelezko, Applied Physics Letters 101, 012405 (2012).
  * Kehayias et al. [2019] P. Kehayias, M. J. Turner, R. Trubko, J. M. Schloss, C. A. Hart, M. Wesson, D. R. Glenn, and R. L. Walsworth, Physical Review B 100 (2019).
  * Rubinas et al. [2018] R. Rubinas, V. V. Vorobyov, V. V. Soshenko, S. V. Bolshedvorskii, V. N. Sorokin, A. N. Smolyaninov, V. G. Vins, A. P. Yelisseyev, and A. V. Akimov, Journal of Physics Communications 2, 115003 (2018).
  * Acosta et al. [2009] V. M. Acosta, E. Bauch, M. P. Ledbetter, C. Santori, K.-M. C. Fu, P. E. Barclay, R. G. Beausoleil, H. Linget, J. F. Roch, F. Treussart, et al., Physical Review B 80 (2009).
  * Balasubramanian et al. [2019] P. Balasubramanian, C. Osterkamp, Y. Chen, X. Chen, T. Teraji, E. Wu, B. Naydenov, and F. Jelezko, Nano Letters 19, 6681 (2019).
  * Glenn et al. [2018] D. R. Glenn, D. B. Bucher, J. Lee, M. D. Lukin, H. Park, and R. L. Walsworth, Nature 555, 351 (2018).
  * Bauch et al. [2018] E. Bauch, C. A. Hart, J. M. Schloss, M. J. Turner, J. F. Barry, P. Kehayias, S. Singh, and R. L. Walsworth, Physical Review X 8 (2018).
  * Childress et al. [2006] L. Childress, M. V. G. Dutt, J. M. Taylor, A. S. Zibrov, F. Jelezko, J. Wrachtrup, P. R. Hemmer, and M. D. Lukin, Science 314, 281 (2006).
  * Su et al. [2013] L.-J. Su, C.-Y. Fang, Y.-T. Chang, K.-M. Chen, Y.-C. Yu, J.-H. Hsu, and H.-C. Chang, Nanotechnology 24, 315702 (2013), URL https://doi.org/10.1088/0957-4484/24/31/315702.
  * Bauch et al. [2019] E. Bauch, S. Singh, J. Lee, C. A. Hart, J. M. Schloss, M. J. Turner, J. F. Barry, L. Pham, N. Bar-Gill, S. F. Yelin, et al., _Decoherence of dipolar spin ensembles in diamond_ (2019), eprint 1904.08763.
  * Maze et al. [2009] J. R. Maze, P. Cappellaro, L. Childress, M. V. G. Dutt, J. S. Hodges, S. Hong, L. Jiang, P. L. Stanwix, J. M. Taylor, E. Togan, et al., in _Advanced Optical Concepts in Quantum Computing, Memory, and Communication II_ , edited by Z. U. Hasan, A. E. Craig, and P. R. Hemmer (SPIE, 2009).
  * Yaroshenko et al. [2020] V. Yaroshenko, V. Soshenko, V. Vorobyov, S. Bolshedvorskii, E. Nenasheva, I. Kotel’nikov, A. Akimov, and P. Kapitanova, Review of Scientific Instruments 91, 035003 (2020).
  * Fu et al. [2019] H. Fu, W. Sun, O. Ogidi-Ekoko, J. C. Goodrich, and N. Tansu, AIP Advances 9, 045013 (2019).
  * Nakamura [1999] S. Nakamura, Semiconductor Science and Technology 14, R27 (1999).
  * Sulmoni [2014] L. A. M. Sulmoni (2014).
  * Tabataba-Vakili et al. [2020] F. Tabataba-Vakili, C. Brimont, B. Alloing, B. Damilano, L. Doyennette, T. Guillet, M. E. Kurdi, S. Chenot, V. Brändli, E. Frayssinet, et al., Applied Physics Letters 117, 121103 (2020).
  * Farrell et al. [2011] R. M. Farrell, D. A. Haeger, P. S. Hsu, K. Fujito, D. F. Feezell, S. P. DenBaars, J. S. Speck, and S. Nakamura, Applied Physics Letters 99, 171115 (2011).
  * Lutgen et al. [2010] S. Lutgen, A. Avramescu, T. Lermer, D. Queren, J. Müller, G. Bruederl, and U. Strauss, physica status solidi (a) 207, 1318 (2010).
  * Nakamura [1997] S. Nakamura, IEEE Journal of Selected Topics in Quantum Electronics 3, 435 (1997).
  * Al-Ghamdi et al. [2019] M. S. Al-Ghamdi, A. Bakry, and M. Ahmed, Journal of the European Optical Society-Rapid Publications 15 (2019).
  * Frost et al. [2013] T. Frost, A. Banerjee, and P. Bhattacharya, Applied Physics Letters 103, 211111 (2013).
  * Stańczyk et al. [2013] S. Stańczyk, T. Czyszanowski, A. Kafar, J. Goss, S. Grzanka, E. Grzanka, R. Czernecki, A. Bojarska, G. Targowski, M. Leszczyński, et al., Applied Physics Letters 103, 261107 (2013).
  * Zhang et al. [2009] L. Q. Zhang, D. S. Jiang, J. J. Zhu, D. G. Zhao, Z. S. Liu, S. M. Zhang, and H. Yang, Journal of Applied Physics 105, 023104 (2009).
  * Cassidy [1991] D. T. Cassidy, Journal of the Optical Society of America B 8, 747 (1991).
  * Scheibenzuber et al. [2011] W. G. Scheibenzuber, U. T. Schwarz, L. Sulmoni, J. Dorsaz, J.-F. Carlin, and N. Grandjean, Journal of Applied Physics 109, 093106 (2011).
  * Kreinberg et al. [2017] S. Kreinberg, W. W. Chow, J. Wolters, C. Schneider, C. Gies, F. Jahnke, S. Höfling, M. Kamp, and S. Reitzenstein, Light: Science & Applications 6, e17030 (2017).
  * Yang et al. [2017] J. Yang, D. G. Zhao, D. S. Jiang, X. Li, F. Liang, P. Chen, J. J. Zhu, Z. S. Liu, S. T. Liu, L. Q. Zhang, et al., Optics Express 25, 9595 (2017).
  * Anani et al. [2007] M. Anani, H. Abid, Z. Chama, C. Mathieu, A. Sayede, and B. Khelifa, Microelectronics Journal 38, 262 (2007).
  * Ma and Oulton [2018] R.-M. Ma and R. F. Oulton, Nature Nanotechnology 14, 12 (2018).
  * Nöbauer et al. [2015] T. Nöbauer, A. Angerer, B. Bartels, M. Trupke, S. Rotter, J. Schmiedmayer, F. Mintert, and J. Majer, Physical Review Letters 115 (2015).


# Multi-EuP: The Multilingual European Parliament Dataset for Analysis of Bias
in Information Retrieval

Jinrui Yang* Timothy Baldwin*† Trevor Cohn*
*School of Computing & Information Systems, The University of Melbourne
†Mohamed bin Zayed University of Artificial Intelligence, UAE
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We present Multi-EuP, a new multilingual benchmark dataset, comprising 22K
multilingual documents collected from the European Parliament, spanning 24
languages. This dataset is designed to investigate fairness in a multilingual
information retrieval (IR) context to analyze both language and demographic
bias in a ranking context. It boasts an authentic multilingual corpus,
featuring topics translated into all 24 languages, as well as cross-lingual
relevance judgments. Furthermore, it offers rich demographic information
associated with its documents, facilitating the study of demographic bias. We
report the effectiveness of Multi-EuP for benchmarking both monolingual and
multilingual IR. We also conduct a preliminary experiment on language bias
caused by the choice of tokenization strategy.

##  1 Introduction

Information retrieval (IR) classically uses a retrieval model to query a
document collection and return a ranked list of documents which are predicted
to be (decreasingly) relevant to the query. Retrieval models have increasingly
been based on supervised learning, involving the annotation of documents with
relevance scores relative to a given query, and the training of models to
predict the relative association between a query and document (Karpukhin et
al., 2020; Khattab and Zaharia, 2020).

In parallel with these advances, the democratisation of the internet has led
to a surge of individual contributors serving as information disseminators,
hailing from various countries and regions, and posting in different
languages. This has created possibilities for exploration of cross-lingual and
multilingual text retrieval. Cross-lingual retrieval pertains to scenarios
where queries are formulated in one language but documents are retrieved from
another language. On the other hand, multilingual retrieval involves a query
in one language but retrieval of documents across multiple languages
simultaneously. An important consideration in any such work is both robustness
and fairness across different combinations of languages – for instance, are
results from one language consistently ranked higher than another for certain
types of query.

While progress towards multilingual retrieval through the release of datasets
such as Mr. TYDI (Zhang et al., 2021) and mMARCO (Bonifacio et al., 2021),
both are limited in that they evaluate monolingual retrieval for a range of
languages, rather than true multilingual retrieval, using multiple languages
simultaneously. Additionally, mMARCO was created by machine translation of MS
MARCO (Nguyen et al., 2016), introducing a confounding factor of translation
errors.

We present a multilingual dataset based on the European Parliament debate
archive with queries in 24 distinct languages, and relevance judgements also
across all 24 languages. This ensures the “multilingual” nature of the dataset
in terms of both query-to-document and document-to-query associations. We
additionally augment each document with comprehensive metadata of the author,
including gender, nationality, political affiliation, and age, for use in
exploring fairness with respect to protected attributes.

Our work contributes to the field in three main ways: (1) we construct and
release the Multi-EuP dataset, a resource for multilingual retrieval over 24
languages, effectively capturing the multilingual nature of both queries and
documents; (2) we explore language bias within the realm of multilingual
retrieval, revealing that multilingual IR using BM25 indeed exhibits notable
language bias; and (3) we supplement the dataset with rich author metadata to
enable research on fairness and demographic bias in IR.111The Multi-EuP
dataset is available for download from https://github.com/jrnlp/Multi-EuP.

##  2 Background and Related Work

The European Parliament (EP) serves as an important forum for political
debates and decision-making at the European Union level. Members of the
European Parliament (MEP) are elected in direct elections across the EU. The
European Parliament debate is presided over by the President, who guides MEPs
in discussing specific subjects.

EP debates have been the source of three key datasets. First, Europarl-2005
was crafted by Koehn (2005) by collecting EP debates documents from 1996 to
2011, and extracting translations as a parallel corpus for statistical machine
translation, enriched with attributes including debate date, chapter id, MEP
id, language, MEP name, and MEP party.

Later, Rabinovich et al. (2017) built Europarl-2017 upon Europarl-2005, by
introducing additional demographic attributes: MEP gender and MEP age. These
were sourced from sources such as Wikidata (Vrandečić and Krötzsch, 2014) and
automatic annotation tools such as Genderize222https://genderize.io/ and
AlchemyVision.333https://www.ibm.com/smarterplanet/us/en/ibmwatson/developercloud/alchemy-
vision.html However, Europarl-2017 is limited to only two language pairs:
English–German and English–French. Europarl-2018 (Vanmassenhove and Hardmeier,
2018) expanded upon Europarl-2017 to add twenty additional language pairs,
based on the manual translations in the EP archives. These corpora have been
used primarily for machine translation research.

Since 2020, the EU has publicly released raw debates in the form of
transcribed source-language speeches with rich multilingual topic index data,
along with the original video and audio recordings. This forms the basis of
the Multi-EuP dataset, with additional attributes for each speaking MEP such
as an image, birthplace, and nationality.

Zhang et al. (2021) introduced Mr. TYDI, an evaluation benchmark dataset for
dense retrieval assessment over 11 languages. This dataset is constructed from
TYDI (Clark et al., 2020), a question answering dataset. For each language,
annotators assign relevance scores as judgments for questions, derived from
Wikipedia articles. Notably, the questions for different languages are crafted
independently, and relevance judgements are provided in-language only. Based
on the dataset, the authors evaluate on monolingual retrieval tasks for non-
English languages using BM25 and mDPR as zero-shot baselines. However, Mr.
TYDI’s scope is limited in that it is not truly multilingual, in that queries
in a given language are only performed over documents in that language. This
is part of the void our work aims to address.

MS MARCO (Nguyen et al., 2016) is a widely-used dataset, sourced from Bing’s
search query logs, but for English queries and documents only. To mitigate
this, Bonifacio et al. (2021) introduced mMARCO, a multilingual variant of the
MS MARCO passage ranking dataset, spanning 13 languages and created through
machine translation, based on one open-source approach (Tiedemann and
Thottingal, 2020) and one commercial system in the form of Google
Translate.444https://cloud.google.com/translate Analysis of the authors’
results reveals a positive correlation between translation quality and
retrieval performance, with higher translation BLEU scores yielding improved
retrieval MRR outcomes. However, similar to Mr. TYDI, mMARCO focuses on in-
language retrieval only for multiple languages, rather than multilingual
retrieval.

Throughout the past few decades, numerous datasets and tasks pertaining to
multilingual retrieval have been developed for evaluation, through efforts
such as CLEF, TREC, and FIRE, each contributing standardized document
collections and evaluation procedures. These evaluation datasets facilitate
genuine multilingual IR research such as Rahimi et al. (2015) and Lawrie et
al. (2023). However, the scope of these datasets is generally limited to a
small number of queries. For example, in the case of CLEF 2001-2003, each
edition encompasses a mere few dozen queries. This limitation tends to confine
research predominantly to evaluation and not offer a resource for training a
multilingual ranking model. Our dataset is of a scale to accommodate both
large-scale training and evaluation of multilingual retrieval methods.

Compared with the related work above, our work augments the multilingual
mixture of queries and documents compared to Mr.TYDI, preserves the
authenticity of multilingual contexts compared to mMARCO’s translation-based
approach, and surpasses the query count limitations of tasks like CLEF.

##  3 Multi-EuP

In our approach, we consider the debate topics to be the queries, and the text
of each individual speech delivered by an MEP to be a document.

|  Language |  ISO code |  Countries where official lang. |  Native Usage |  Total Usage |  # Docs |  Words per Doc
---|---|---|---|---|---|---
English |  EN |  United Kingdom, Ireland, Malta |  13% |  51% |  7123 |  286/200
German |  DE |  Germany, Belgium, Luxembourg |  16% |  32% |  3433 |  180/164
French |  FR |  France, Belgium, Luxembourg |  12% |  26% |  2779 |  296/223
Italian |  IT |  Italy |  13% |  16% |  1829 |  190/175
Spanish |  ES |  Spain |  8% |  15% |  2371 |  232/198
Polish |  PL |  Poland |  8% |  9% |  1841 |  155/148
Romanian |  RO |  Romania |  5% |  5% |  794 |  186/172
Dutch |  NL |  Netherlands, Belgium |  4% |  5% |  897 |  184/170
Greek |  EL |  Greece, Cyprus |  3% |  4% |  707 |  209/205
Hungarian |  HU |  Hungary |  3% |  3% |  614 |  126/128
Portuguese |  PT |  Portugal |  2% |  3% |  1176 |  179/167
Czech |  CS |  Czech Republic |  2% |  3% |  397 |  167/149
Swedish |  SV |  Sweden |  2% |  3% |  531 |  175/165
Bulgarian |  BG |  Bulgaria |  2% |  2% |  408 |  196/178
Danish |  DA |  Denmark |  1% |  1% |  292 |  218/198
Finnish |  FI |  Finland |  1% |  1% |  405 |  94/87
Slovak |  SK |  Slovakia |  1% |  1% |  348 |  151/158
Lithuanian |  LT |  Lithuania |  1% |  1% |  115 |  142/127
Croatian |  HR |  Croatia |  <1% |  <1% |  524 |  183/164
Slovene |  SL |  Slovenia |  <1% |  <1% |  270 |  201/163
Estonian |  ET |  Estonia |  <1% |  <1% |  58 |  160/158
Latvian |  LV |  Latvia |  <1% |  <1% |  89 |  111/123
Maltese |  MT |  Malta |  <1% |  <1% |  178 |  117/115
Irish |  GA |  Ireland |  <1% |  <1% |  33 |  198/172
Table 1: Multi-EuP statistics, broken down by language: ISO language code; EU
member states using the language officially; proportion of the EU population
speaking the language (Chalkidis et al., 2021); number of debate speech
documents; and words per document (mean/median).

#### Topics

The topics are officially annotated by the EU, and professionally translated
into 24 different
languages.555https://www.europarl.europa.eu/translation/en/translation-at-the-
european-parliament/ During preprocessing, we filter out procedural debate
topics such as agenda, leaving 1.1K unique topics. They will serve as a
valuable resource for assessing language bias in multilingual ranking methods,
given that all the topics across different languages are semantically
consistent.

#### Documents

The 22K multilingual documents within the Multi-EuP dataset originate from MEP
speeches during parliamentary debates. Each document annotated with additional
metadata, including the date of the speech, the MEP ID, and a link to the
video recording for potential multimodal research but not used here. Table 1
shows a detailed breakdown of the language distribution and descriptive
statistics of the dataset. We include in our corpus documents only in the
original language, as spoken by the MEP, but not their translations into other
languages. Our only use of translations is the debate topics themselves.

#### Judgments

To assess the relevance of documents to a given query, we use a binary
relevance judgment, based on whether the speech was part of a debate on the
given topic, resulting in one positive relevance judgment per document,
meaning that the document collection is much less sparse than Mr. TYDI and MS
MARCO, for example.

#### Languages

Multi-EuP covers 24 EU languages from seven families (Germanic, Romance,
Slavic, Uralic, Baltic, Semitic, Hellenic), each of which is the official
language of one or more member states. Table 1 provides a breakdown of each
language’s EU usage, member state distribution, and population, using ISO-639
codes.

#### MEP

Figure 1: The gender and birth year distributions of the 705 MEPs in Multi-EuP
dataset. The birth year corresponds to the current age calculation.

Multi-EuP encompasses 705 members elected across the 27 member states of the
EU. We constructed the MEP dictionary by collecting MEP attributes such as
name, photo, id in EU, nationality, place of birth, party affiliation, and
spoken language. We further annotated MEPs with gender and their birthdate,
based on Wikipedia profiles and Rabinovich et al. (2017), and manually checked
if difference existing. Figure 1 illustrates the gender and age distribution
across MEPs, with male MEPs being more than twice as numerous as female MEPs,
and the majority falling within the 40–70 age range. This corpus is rare,
perhaps unique, due to its richly detailed speaker demographic information,
which enables research on fairness and bias in information retrieval.

#### Data Split

For data splitting, we select two sets with 100 language-specific and distinct
topics for development and test set in 24 languages, and keep the remaining
topics to the training set. This design choice was made to maintain an ample
supply of topics and judgment samples essential for the training of deep
learning models, and also facilitate subsequent cross-lingual comparative
research.

#### Supported Task

Similarly to Mr.TYDI (Zhang et al., 2021), Multi-EuP can be used for
monolingual retrieval in English as well as non-English languages (eg. Swedish
queries against Swedish documents). However, unlike Mr.TYDI, Multi-EuP
encompasses multilingual documents and identical multilingual topics, ensuring
that queries in different languages can be compared. Consequently, Multi-EuP
can support diverse information retrieval experimental tasks. These including
one-vs-one scenarios with single one language queries against single one
language documents, in other words, monolingual or cross-lingual IR, one-vs-
many scenarios with single-language queries against multilingual documents,
i.e., multilingual IR, and many-vs-many scenarios involving multilingual
queries against multilingual documents, i.e, mixed multilingual IR).

##  4 Experiments and Findings

|  | GERMANIC |  | ROMANCE |  | SLAVIC |  | URALIC |  |
---|---|---|---|---|---|---|---|---|---
| EN | DA | DE | NL | SV |  | RO | ES | FR | IT | PT |  | PL | BG | CS |  | HU | FI |  | EL
One-vs-one (Queries and documents in the same language.)
numq | 839 | 208 | 840 | 458 | 330 |  | 434 | 680 | 765 | 659 | 557 |  | 628 | 273 | 259 |  | 404 | 251 |  | 360
numd | 7123 | 268 | 3433 | 897 | 531 |  | 794 | 2371 | 2779 | 1829 | 1176 |  | 1841 | 408 | 397 |  | 614 | 405 |  | 707
numr | 7123 | 3433 | 3433 | 897 | 531 |  | 794 | 2371 | 2779 | 1829 | 1176 |  | 1841 | 408 | 397 |  | 614 | 405 |  | 707
MRR | 73.79 | 45.51 | 56.70 | 39.02 | 45.77 |  | 42.58 | 54.25 | 46.57 | 56.40 | 56.51 |  | 47.68 | 44.70 | 47.12 |  | 39.99 | 46.46 |  | 39.58
One-vs-many (A fixed set of queries in the given language, with documents in
all 24 languages.)
numq | 100 | 100 | 100 | 100 | 100 |  | 100 | 100 | 100 | 100 | 100 |  | 100 | 100 | 100 |  | 100 | 100 |  | 100
numd | 22K | 22K | 22K | 22K | 22K |  | 22K | 22K | 22K | 22K | 22K |  | 22K | 22K | 22K |  | 22K | 22K |  | 22K
numr | 2902 | 1997 | 1995 | 1992 | 1996 |  | 1996 | 1996 | 1924 | 1994 | 1997 |  | 1996 | 1996 | 1997 |  | 1996 | 1997 |  | 1996
MRR | 62.79 | 16.15 | 28.27 | 20.88 | 19.40 |  | 16.10 | 22.57 | 24.22 | 14.24 | 18.7 |  | 4.80 | 7.57 | 9.52 |  | 7.51 | 17.61 |  | 11.16
Table 2: Details of Multi-EuP for the 16 most widely spoken EU official
languages, in terms of the number of queries ($q$), documents ($d$) and
relevance judgements ($r$). Results are for BM25 in one-vs-one and one-vs-many
settings based on MRR$@$100 (%). See Table 3 in the Appendix for results
across all languages. Note that as each document has a unique topic which in
turn defines the relevance judgements, numd= numr in the one-vs-one setting.

We conduct preliminary experiments in both one-vs-one and one-vs-many
settings, as described above.

#### Methods

We base our experiments on BM25 with default settings ($k_{1}$ = 0.9 and $b$ =
0.4), a popular traditional information retrieval baseline. Our implementation
is based on Pyserini (Lin et al., 2021), which is built upon Lucene (Yang et
al., 2017). Notably, the latest LUCENE 8.5.1 API offers language-specific
tokenizers, 666Provided by the Anaylzer package in LUCENE.
https://lucene.apache.org/core/8_5_1/analyzers-common/index.html covering 19
out of the 24 languages present in Multi-EuP. For the remaining languages —
namely Polish (PL), Croatian (HR), Slovak (SK), Slovenian (SL), and Maltese
(MT) — we use a whitespace tokenizer.

#### Evaluation

Our primary evaluation metric is Mean Reciprocal Rank (MRR). For a single
query, the reciprocal rank is $RR=\frac{1}{\text{rank}}$ where rank is the
position of the highest-ranked relevant document. If no correct answer was
returned, then the reciprocal rank is defined to be 0. For multiple queries
$Q$, the MRR is the mean of the $Q$ reciprocal ranks.

$MRR=\frac{1}{Q}\sum_{i=1}^{Q}\frac{1}{\text{rank}_{i}}$

MRR@$k$ denotes MRR computed at a depth of $k$ results. Note that the higher
the number the better, and that a perfect retriever achieves an MRR of 1
(assuming every query has at least one relevant document). The choice of
setting $k=100$ aligns with prior endeavors over MS MARCO (Nguyen et al.,
2016).

###  4.1 Monolingual IR (one-vs-one)

#### Experimental Setup

We first present results over Multi-EuP in a monolingual setting across the 24
different languages. Specifically, we evaluate single-language queries against
documents in the same language. In this configuration, we partitioned our
original collection of 22K documents into 24 distinct language-specific sub-
collections. Table 2 presents the results broken down across languages.

#### Results and Findings

Table 2 presents the MRR@100 results for BM25 on Multi-EuP. There are two
high-level findings:

First, Multi-EuP is a relatively easy benchmark for monolingual information
retrieval, as the MRR@100 is always around 40 or greater (meaning that the
first relevant document is in the top-3 results on average). Indeed, the
average MRR across the 24 test languages is 49.61. While direct comparison is
not possible, it is noteworthy that for Mr. TYDI, the average MRR is 32.1
across 11 languages. Part of this difference can be attributed to the fact
that our relevance judgments are not as sparse as theirs.

Second, similar to Mr. TYDI, direct comparison of absolute scores between
languages is not meaningful in a monolingual setting, as the document
collection size differs.

###  4.2 Multilingual IR (one-vs-many)

#### Experimental Setup

In contrast to Mr. TYDI (Zhang et al., 2021), Multi-EuP supports one-vs-many
retrieval, and allows us to systematically explore the effect of querying the
same document collection with the same set of topics in different languages.
This is because we have translations of the topics in all languages, documents
span multiple languages, and judgments are cross-lingual (e.g., English
queries potentially yield relevant Polish documents). For this experiment, we
use the default whitespace tokenizer in the Pyserini library.

#### Results and Findings

Table 2 presents the MRR results for BM25 for multilingual information
retrieval on 100 topics from the Multi-EuP test set. It’s worth noting that
these topics have translation-equivalent content in the different languages.
Consequently, the one-vs-many approach allows us to analyze language bias. We
made several key observations:

First, unsurprisingly, having more relevance judgments tends to improve
ranking accuracy. Therefore, when comparing English topics with other
languages, English exhibits notably better MRR performance.

Second, despite there being consistency in the topics, document collection,
and relevance judgments, there is a significant disparity in MRR scores across
languages, an effect we investigate further in the next section.

Figure 2: Language correlation matrix between topics and the ranking output
top 100 relevant documents in a one-vs-many setting. The row is the topic
languages, the columns is the document languages. The left matrix displays
results using a language-specific tokenizer, while the right matrix represents
the experiment with a simple whitespace tokenizer. Both of them show strong
language bias between the language of the topic and the retrieved documents.

##  5 Language Bias Discussion

In light of our findings in a one-vs-many setting, we were keen to delve
further into the underlying causes of the disparity between languages.

###  5.1 Bias Detection

Language bias is likely if the query language aligns better with one document
language than another. As mentioned earlier, Pyserini supports different
tokenizers, specifically language-specific tokenizers or simple whitespace
tokenization. Therefore, in the one-vs-many setting, we analyze the
composition of the top-100 rankings for the 100 topics. During indexing of the
document collection, we used the simple whitespace tokenizer, given the
multilingual nature of the collection. However, over the queries during
retrieval, we employed two different tokenizers — a language-specific
tokenizer, and the whitespace tokenizer.

We conducted a correlation analysis between the language of the topics and the
language of the top 100 relevant documents. From Table 2, we can see that
relevance judgments in our test cases are consistent across languages,
ensuring uniformity in the correlation matrix within the test set. However,
Figure 2 reveals that both approaches generate strong language bias. In both
cases, the query language aligns better with documents in its own language
than others. The right plot appears to show that languages from the same
family has strong correlation (e.g., PL, CS) and (IT, ES) since they may have
some shared vocabulary.

Figure 3: Linear regression between MRR@100 and the number of documents per
language. The left plot is based on collection indexing with a whitespace
tokenizer but a language-specific tokenizer over the queries. The right plot
uses a whitespace tokenizer for both indexing the collection and the queries.
The higher $R^{2}$ for the right plot suggests that using a whitespace
tokenizer for both the collection and queries reduces language bias in
multilingual IR.

###  5.2 Collection Distribution Factors

Initially, we hypothesized that the disparity for each language may be a
contributing factor to this bias. Figure 3 presents the regression line
between the number of documents in a given language and MRR, which explains
much of the variation across languages.

However, note the outlier above the regression line (Polish: PL), which has a
substantial number of documents but surprisingly low MRR performance. We refer
to this phenomenon as a “BM25 unfriendly” language. According to Wojtasik et
al. (2023), the main reason for the low performance of Polish lies in its
highly-inflected morphology, giving rise to a a multitude of word forms per
lexeme, including inflections of proper names, and complex morphological
structure. In such cases, lexical matching is less effective than in other
morphologically-simpler languages. Furthermore, LUCENE 8.5.1 API does not have
a language-specific tokenizer for Polish. Conversely, languages below the
regression line can be termed “BM25 friendly” languages, as they require fewer
documents to achieve higher MRR in retrieval.

###  5.3 Language Tokenizer Factors

Secondly, we speculated that the choice of language-specific Analyzer in
LUCENE might be a contributing factor, as it influences word tokenization,
token filter, synonym expansion and other processing.
777https://lucene.apache.org/core/8_0_0/core/org/apache/lucene/analysis/package-
summary.html#package.description To investigate this, we conducted a
controlled experiment in the one-vs-many setting. When indexing the
collection, given the multilingual nature of the collection, we employed
whitespace as the tokenizer. However, over the queries, we experimented with
either a language-specific tokenizer or whitespace tokenizer. We then compared
the linear regression of MRR against the number of documents in Figure 3. On
the right side of the plot, we can see a strong correlation when using
whitespace tokenization for both the collection and the queries, reducing
language bias.

Furthermore, when transitioning from language-specific tokenizers to
whitespace tokenizers, the overall MRR across all languages declined modestly,
from 15.02 to 14.18. That is, the original performance level was largely
preserved, but language bias was diminished in using simple whitespace
tokenization.

##  6 Conclusion

In this paper, we introduce Multi-EuP, a novel dataset for multilingual
information retrieval across 24 languages, collected from European Parliament
debates. The demographic information provided by the Multi-EuP dataset serves
a dual purpose: not only does it contribute to multilingual retrieval tasks,
but it also holds significant potential for advancing research in the realm of
fairness and bias. This dataset can play a pivotal role in investigating
issues of equitable representations and mitigation of biases within document
ranking settings.

Multi-EuP facilitates diverse information retrieval (IR) scenarios,
encompassing one-vs-one, one-vs-many, and many-vs-many settings. We
demonstrated the utility of Multi-EuP as a benchmark for evaluating both
monolingual and multilingual IR. Our study reveals the presence of language
bias in multilingual IR when employing BM25. We further validate the
effectiveness of mitigating this bias through the strategic implementation of
whitespace as a language tokenizer.

We propose to conduct future work in three main areas. First, we intend to
expand our investigation of language bias to encompass a broader range of
ranking methods, including neural methods such as mDPR Zhang et al. (2021),
mColBERT (Lawrie et al., 2023) and PLAID-X(Santhanam et al., 2022). Second, we
will expand the dataset by developing an automated API to retrieve data
published by the European Parliament (EP), thereby ensuring real-time
synchronization of our dataset. Lastly, our current experiments have explored
language bias only, but we plan to further investigate gender bias, age bias,
and nationality bias.

## Limitations

The limitations of the Multi-EuP dataset are notable but navigable. Primarily,
the temporal coverage of the dataset is confined to the past three years. This
temporal constraint arises due to the fact that, preceding 2020, documents
released by the EU were predominantly available in mono-lingual versions only.
However, a potential remedy lies in the amalgamation of the Europarl (Koehn,
2005) collection, enabling a more comprehensive and holistic Multi-EuP
dataset.

Furthermore, it is worth noting the domain skew of the dataset, in that Multi-
EuP inevitably centers on political matters. While this presents challenges,
particularly in terms of the intricate nuances of political language, it
inherently serves as an excellent foundational stepping stone for delving into
the intricacies of multilingual retrieval. We believe, however, that this
dataset can serve as a launching pad for broader explorations encompassing
cross-domain and open-domain transfer learning scenarios, thus contributing to
the broader landscape of language understanding and retrieval.

## Ethics Statement

The dataset contains publicly-available EP data that does not include personal
or sensitive information, with the exception of information relating to public
officeholders, e.g., the names of the active members of the European
Parliament, European Council, or other official administration bodies. The
collected data is licensed under the Creative Commons Attribution 4.0
International licence. 888https://eur-lex.europa.eu/content/legal-
notice/legal-notice.html

## Acknowledgements

This research was funded by Melbourne Research Scholarship and undertaken
using the LIEF HPC-GPGPU Facility hosted at the University of Melbourne. This
facility was established with the assistance of LIEF Grant LE170100200. We
would like to thank George Buchanan for providing valuable feedback.

## References

  * Bonifacio et al. (2021) Luiz Henrique Bonifacio, Israel Campiotti, Roberto de Alencar Lotufo, and Rodrigo Frassetto Nogueira. 2021.  mMARCO: A multilingual version of MS MARCO passage ranking dataset.  _CoRR_ , abs/2108.13897.
  * Chalkidis et al. (2021) Ilias Chalkidis, Manos Fergadiotis, and Ion Androutsopoulos. 2021.  MultiEURLEX - a multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6974–6996, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Clark et al. (2020) Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020.  TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages.  _Transactions of the Association for Computational Linguistics_ , 8:454–470.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia. 2020.  Colbert: Efficient and effective passage search via contextualized late interaction over BERT.  _CoRR_ , abs/2004.12832.
  * Koehn (2005) Philipp Koehn. 2005.  Europarl: A parallel corpus for statistical machine translation.  In _Proceedings of Machine Translation Summit X: Papers_ , pages 79–86, Phuket, Thailand.
  * Lawrie et al. (2023) Dawn Lawrie, Eugene Yang, Douglas W. Oard, and James Mayfield. 2023.  Neural approaches to multilingual information retrieval.  arXiv cs.IR 2209.01335.
  * Lin et al. (2021) Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021.  Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations.  https://github.com/castorini/pyserini.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A human generated machine reading comprehension dataset.  _CoRR_ , abs/1611.09268.
  * Rabinovich et al. (2017) Ella Rabinovich, Raj Nath Patel, Shachar Mirkin, Lucia Specia, and Shuly Wintner. 2017.  Personalized machine translation: Preserving original author traits.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_ , pages 1074–1084, Valencia, Spain. Association for Computational Linguistics.
  * Rahimi et al. (2015) Razieh Rahimi, Azadeh Shakery, and Irwin King. 2015.  Multilingual information retrieval in the language modeling framework.  _Information Retrieval Journal_ , 18:246–281.
  * Santhanam et al. (2022) Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. 2022.  PLAID: An efficient engine for late interaction retrieval.  In _Proceedings of the 31st ACM International Conference on Information & Knowledge Management_, CIKM ’22, page 1747–1756, New York, NY, USA. Association for Computing Machinery.
  * Tiedemann and Thottingal (2020) Jörg Tiedemann and Santhosh Thottingal. 2020.  OPUS-MT – building open translation services for the world.  In _Proceedings of the 22nd Annual Conference of the European Association for Machine Translation_ , pages 479–480, Lisboa, Portugal. European Association for Machine Translation.
  * Vanmassenhove and Hardmeier (2018) Eva Vanmassenhove and Christian Hardmeier. 2018.  Europarl datasets with demographic speaker information.  In _Proceedings of the 21st Annual Conference of the European Association for Machine Translation_ , page 391, Alicante, Spain.
  * Vrandečić and Krötzsch (2014) Denny Vrandečić and Markus Krötzsch. 2014.  Wikidata: A free collaborative knowledge base.  _Communications of the ACM_ , 57:78–85.
  * Wojtasik et al. (2023) Konrad Wojtasik, Vadim Shishkin, Kacper Wołowiec, Arkadiusz Janz, and Maciej Piasecki. 2023.  BEIR-PL: Zero shot information retrieval benchmark for the Polish language.  arXiv cs.IR 2305.19840.
  * Yang et al. (2017) Peilin Yang, Hui Fang, and Jimmy Lin. 2017.  Anserini: Enabling the use of lucene for information retrieval research.  In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 1253–1256.
  * Zhang et al. (2021) Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. 2021.  Mr. TyDi: A multi-lingual benchmark for dense retrieval.  arXiv cs.CL 2108.08787.

##  Appendix A Appendix

| GERMANIC |  | ROMANCE |  | SLAVIC |  | URALIC |  | BALTIC | HELLENTICH | CELTC
---|---|---|---|---|---|---|---|---|---|---|---
| EN | DA | DE | NL | SV | MT |  | RO | ES | FR | IT | PT |  | PL | BG | CS | SK | SL | HR |  | HU | FI | ET |  | LV | LT | EL | GA
One-vs-one (Queries in one language against documents in the same language, test on the whole set.) |  |  |  |  |  |  |  |
numq | 839 | 208 | 840 | 458 | 330 | 138 |  | 434 | 680 | 765 | 659 | 557 |  | 628 | 273 | 259 | 236 | 205 | 311 |  | 404 | 251 | 52 |  | 75 | 99 | 360 | 14
numd | 7123 | 292 | 3433 | 897 | 531 | 178 |  | 794 | 2371 | 2779 | 1829 | 1176 |  | 1841 | 408 | 397 | 348 | 270 | 524 |  | 614 | 405 | 58 |  | 89 | 115 | 707 | 16
numr | 7123 | 292 | 3433 | 897 | 531 | 178 |  | 794 | 2371 | 2779 | 1829 | 1176 |  | 1841 | 408 | 397 | 348 | 270 | 524 |  | 614 | 405 | 58 |  | 89 | 115 | 707 | 16
MRR | 73.79 | 45.51 | 56.70 | 39.02 | 45.77 | 65.17 |  | 42.58 | 54.25 | 56.51 | 47.68 | 56.40 |  | 47.12 | 44.70 | 47.12 | 44.51 | 45.47 | 39.83 |  | 39.99 | 46.46 | 41.59 |  | 58.04 | 50.03 | 39.58 | 72.22
One-vs-many (Queries in one language against documents in many languages, test on semantically consistent topics.) |  |  |  |  |  |  |  |
numq | 100 | 100 | 100 | 100 | 100 | 100 |  | 100 | 100 | 100 | 100 | 100 |  | 100 | 100 | 100 | 100 | 100 | 100 |  | 100 | 100 | 100 |  | 100 | 100 | 100 | 100
numd | 22K | 22K | 22K | 22K | 22K | 22K |  | 22K | 22K | 22K | 22K | 22K |  | 22K | 22K | 22K | 22K | 22K | 22K |  | 22K | 22K | 22K |  | 22K | 22K | 22K | 22K
numr | 2902 | 1997 | 1996 | 1996 | 1996 | 1996 |  | 1996 | 1924 | 1996 | 1997 | 1996 |  | 1996 | 1996 | 1997 | 1996 | 1996 | 1923 |  | 1996 | 1997 | 1909 |  | 1996 | 1997 | 1996 | 1992
MRR | 62.79 | 16.15 | 28.27 | 20.88 | 19.40 | 0.40 |  | 16.10 | 22.57 | 24.22 | 14.24 | 18.70 |  | 4.80 | 7.57 | 9.52 | 12.35 | 6.46 | 10.41 |  | 7.51 | 17.61 | 7.31 |  | 7.95 | 8.94 | 11.16 | 5.21

Table 3: Details of Multi-EuP for the all 24 spoken EU official languages, in
terms of the number of queries ($q$), documents ($d$) and relevance judgements
($r$). Results are for BM25 in one-vs-one and one-vs-many settings based on
MRR$@$100 (%). Note that as each document has a unique topic which in turn
defines the relevance judgements, numd= numr in the one-vs-one setting.